问题描述
我是汁液杂草项目的新手，几周以来，我们遇到了一个问题，即杂草会在生产中自行关闭。
在生产中，我们有一个用于后台，cronjobs，import等的后端服务器，我们还有一个用于...以及前端的前端服务器。两者都是完整的hybris应用程序。另外，我们的solr放在专用的机器上。
前端服务器始终运行良好。只有我们备份的服务器会在没有任何正当理由的情况下关闭。
后端服务器如下：
- Java 8
- 混合动力6.7
- 36GB RAM
- java.mem为28GB
- Ubuntu 16.04.2
INFO | jvm 1 | main | 2020/08/27 09:04:50.411 | INFO [cluster-ClusterId{value='5f462504979dc05c35c5afff',description='null'}-111.22.12.16:27017] [connection] Opened connection [connectionId{localValue:33,serverValue:8662}] to 111.22.12.16:27017
INFO | jvm 1 | main | 2020/08/27 09:04:50.513 | INFO [cluster-ClusterId{value='5f462504979dc05c35c5afff',description='null'}-111.22.12.16:27017] [cluster] Monitor thread successfully connected to server with description ServerDescription{address=111.22.12.16:27017,type=STANDALONE,state=CONNECTED,ok=true,version=ServerVersion{versionList=[3,4,10]},minWireversion=0,maxWireversion=5,maxDocumentSize=16777216,roundtripTimeNanos=115245710}
INFO | jvm 1 | main | 2020/08/27 09:04:51.014 |
INFO | jvm 1 | main | 2020/08/27 09:05:01.224 | INFO [RemoveFavouriteListEntryWithProductEmptyCronJob::de.hybris.platform.servicelayer.internal.jalo.ScriptingJob] (RemoveFavouriteListEntryWithProductEmptyCronJob) [ScriptingJobPerformable] ### Starting executing script : model://removeFavouriteListEntryWithProductEmptyScript ###
INFO | jvm 1 | main | 2020/08/27 09:05:01.224 | INFO [update-shopOrderIndex-cronJob::de.hybris.platform.servicelayer.internal.jalo.ServicelayerJob] (update-shopOrderIndex-cronJob) [SolrIndexerJob] Started indexer cronjob.
INFO | jvm 1 | main | 2020/08/27 09:05:01.324 | INFO [RemoveFavouriteListEntryWithProductEmptyCronJob::de.hybris.platform.servicelayer.internal.jalo.ScriptingJob] (RemoveFavouriteListEntryWithProductEmptyCronJob) [ScriptingJobPerformable] de.hybris.platform.servicelayer.internal.model.impl.DefaultModelService$$EnhancerBySpringcglib$$d7ee97a3@7ca8259bFavoritesListEntry removed []done
INFO | jvm 1 | main | 2020/08/27 09:05:01.324 | INFO [RemoveFavouriteListEntryWithProductEmptyCronJob::de.hybris.platform.servicelayer.internal.jalo.ScriptingJob] (RemoveFavouriteListEntryWithProductEmptyCronJob) [ScriptingJobPerformable] ### Finished executing script,returned script result = null ###
INFO | jvm 1 | main | 2020/08/27 09:05:01.324 | INFO [update-shopStockItemIndex-cronJob::de.hybris.platform.servicelayer.internal.jalo.ServicelayerJob] (update-shopStockItemIndex-cronJob) [SolrIndexerJob] Started indexer cronjob.
INFO | jvm 1 | main | 2020/08/27 09:05:01.425 | INFO [update-shopQuoteIndex-cronJob::de.hybris.platform.servicelayer.internal.jalo.ServicelayerJob] (update-shopQuoteIndex-cronJob) [SolrIndexerJob] Started indexer cronjob.
INFO | jvm 1 | main | 2020/08/27 09:05:01.826 | INFO [update-shopQuoteIndex-cronJob::de.hybris.platform.servicelayer.internal.jalo.ServicelayerJob] (update-shopQuoteIndex-cronJob) [SolrIndexerJob] Finished indexer cronjob.
INFO | jvm 1 | main | 2020/08/27 09:05:02.127 | INFO [update-shopStockItemIndex-cronJob::de.hybris.platform.servicelayer.internal.jalo.ServicelayerJob] (update-shopStockItemIndex-cronJob) [SolrIndexerJob] Finished indexer cronjob.
INFO | jvm 1 | main | 2020/08/27 09:05:02.429 | INFO [update-shopOrderIndex-cronJob::de.hybris.platform.servicelayer.internal.jalo.ServicelayerJob] (update-shopOrderIndex-cronJob) [SolrIndexerJob] Finished indexer cronjob.
INFO | jvm 1 | main | 2020/08/27 09:05:02.931 |
INFO | jvm 1 | main | 2020/08/27 09:05:27.177 | WARN [jgroups-11,shoplive-broadcast,hybrisnode-1] [NAKACK2] JGRP000011: hybrisnode-1: dropped message batch from non-member hybrisnode-2 (view=[hybrisnode-1|2] (1) [hybrisnode-1])
INFO | jvm 1 | main | 2020/08/27 09:05:27.677 |
INFO | jvm 1 | main | 2020/08/27 09:05:49.412 | INFO [solrclient-cleanup-master] [DefaultSolrClientPool] Checking pooled Solr client [config=shopSolrserverConfig]
INFO | jvm 1 | main | 2020/08/27 09:05:49.912 |
INFO | jvm 1 | main | 2020/08/27 09:06:19.376 | INFO [cluster-ClusterId{value='5f462504979dc05c35c5afff',description='null'}-111.22.12.16:27017] [connection] Opened connection [connectionId{localValue:34,serverValue:8664}] to 111.22.12.16:27017
INFO | jvm 1 | main | 2020/08/27 09:06:19.877 |
INFO | jvm 1 | main | 2020/08/27 09:06:29.399 | WARN [jgroups-11,hybrisnode-1] [NAKACK2] JGRP000011: hybrisnode-1: dropped message batch from non-member hybrisnode-2 (view=[hybrisnode-1|2] (1) [hybrisnode-1]) (received 3 identical messages from hybrisnode-2 in the last 62247 ms)
INFO | jvm 1 | main | 2020/08/27 09:06:29.900 |
INFO | jvm 1 | main | 2020/08/27 09:06:49.436 | INFO [solrclient-cleanup-master] [DefaultSolrClientPool] Checking pooled Solr client [config=shopSolrserverConfig]
INFO | jvm 1 | main | 2020/08/27 09:06:49.937 |
INFO | jvm 1 | main | 2020/08/27 09:07:49.358 | INFO [solrclient-cleanup-master] [DefaultSolrClientPool] Checking pooled Solr client [config=shopSolrserverConfig]
INFO | jvm 1 | main | 2020/08/27 09:07:49.860 |
INFO | jvm 1 | main | 2020/08/27 09:07:51.064 | WARN [jgroups-25,hybrisnode-1] [NAKACK2] JGRP000011: hybrisnode-1: dropped message batch from non-member hybrisnode-2 (view=[hybrisnode-1|2] (1) [hybrisnode-1]) (received 2 identical messages from hybrisnode-2 in the last 81647 ms)
INFO | jvm 1 | main | 2020/08/27 09:07:51.565 |
INFO | jvm 1 | main | 2020/08/27 09:08:20.737 | INFO [cluster-ClusterId{value='5f462504979dc05c35c5afff',description='null'}-111.22.12.16:27017] [connection] Opened connection [connectionId{localValue:35,serverValue:8665}] to 111.22.12.16:27017
INFO | jvm 1 | main | 2020/08/27 09:08:21.238 |
STATUS | wrapper | main | 2020/08/27 09:08:35.081 | Pinging the JVM took 2 seconds to respond.
INFO | jvm 1 | main | 2020/08/27 09:08:49.617 | INFO [solrclient-cleanup-master] [DefaultSolrClientPool] Checking pooled Solr client [config=shopSolrserverConfig]
INFO | jvm 1 | main | 2020/08/27 09:08:50.118 |
INFO | jvm 1 | main | 2020/08/27 09:09:16.324 | INFO [cluster-ClusterId{value='5f462504979dc05c35c5afff',description='null'}-111.22.12.16:27017] [connection] Opened connection [connectionId{localValue:36,serverValue:8688}] to 111.22.12.16:27017
INFO | jvm 1 | main | 2020/08/27 09:09:16.825 |
STATUS | wrapper | main | 2020/08/27 09:09:46.483 | Pinging the JVM took 1 seconds to respond.
STATUS | wrapper | main | 2020/08/27 09:09:52.562 | JVM received a signal SIGKILL (9).
STATUS | wrapper | main | 2020/08/27 09:09:52.563 | JVM process is gone.
ERROR | wrapper | main | 2020/08/27 09:09:52.563 | JVM exited unexpectedly.
STATUS | wrapper | main | 2020/08/27 09:09:52.984 | Reloading Wrapper configuration...
STATUS | wrapper | main | 2020/08/27 09:09:53.209 | Launching a JVM...
INFO | jvm 2 | main | 2020/08/27 09:09:53.415 | Java HotSpot(TM) 64-Bit Server VM warning: INFO: os::commit_memory(0x00000000e9990000,29366878208,0) Failed; error='Cannot allocate memory' (errno=12)
INFO | jvm 2 | main | 2020/08/27 09:09:53.416 | #
INFO | jvm 2 | main | 2020/08/27 09:09:53.416 | # There is insufficient memory for the Java Runtime Environment to continue.
INFO | jvm 2 | main | 2020/08/27 09:09:53.416 | # Native memory allocation (mmap) Failed to map 29366878208 bytes for committing reserved memory.
INFO | jvm 2 | main | 2020/08/27 09:09:53.416 | # An error report file with more information is saved as:
INFO | jvm 2 | main | 2020/08/27 09:09:53.416 | # /home/user/hybris/bin/platform/tomcat/bin/hs_err_pid31526.log
ERROR | wrapper | main | 2020/08/27 09:09:53.459 | JVM exited while loading the application.
STATUS | wrapper | main | 2020/08/27 09:09:53.862 | Reloading Wrapper configuration...
STATUS | wrapper | main | 2020/08/27 09:09:54.076 | Launching a JVM...
ERROR | wrapper | main | 2020/08/27 09:09:54.117 | JVM exited while loading the application.
INFO | jvm 3 | main | 2020/08/27 09:09:54.118 | Java HotSpot(TM) 64-Bit Server VM warning: INFO: os::commit_memory(0x00000000e9990000,0) Failed; error='Cannot allocate memory' (errno=12)
INFO | jvm 3 | main | 2020/08/27 09:09:54.118 | #
INFO | jvm 3 | main | 2020/08/27 09:09:54.118 | # There is insufficient memory for the Java Runtime Environment to continue.
INFO | jvm 3 | main | 2020/08/27 09:09:54.118 | # Native memory allocation (mmap) Failed to map 29366878208 bytes for committing reserved memory.
INFO | jvm 3 | main | 2020/08/27 09:09:54.118 | # An error report file with more information is saved as:
INFO | jvm 3 | main | 2020/08/27 09:09:54.118 | # /home/user/hybris/bin/platform/tomcat/bin/hs_err_pid31536.log
STATUS | wrapper | main | 2020/08/27 09:09:54.521 | Reloading Wrapper configuration...
STATUS | wrapper | main | 2020/08/27 09:09:54.737 | Launching a JVM...
ERROR | wrapper | main | 2020/08/27 09:09:54.773 | JVM exited while loading the application.
INFO | jvm 4 | main | 2020/08/27 09:09:54.773 | Java HotSpot(TM) 64-Bit Server VM warning: INFO: os::commit_memory(0x00000000e9990000,0) Failed; error='Cannot allocate memory' (errno=12)
INFO | jvm 4 | main | 2020/08/27 09:09:54.773 | #
INFO | jvm 4 | main | 2020/08/27 09:09:54.773 | # There is insufficient memory for the Java Runtime Environment to continue.
INFO | jvm 4 | main | 2020/08/27 09:09:54.773 | # Native memory allocation (mmap) Failed to map 29366878208 bytes for committing reserved memory.
INFO | jvm 4 | main | 2020/08/27 09:09:54.773 | # An error report file with more information is saved as:
INFO | jvm 4 | main | 2020/08/27 09:09:54.773 | # /home/user/hybris/bin/platform/tomcat/bin/hs_err_pid31546.log
STATUS | wrapper | main | 2020/08/27 09:09:55.176 | Reloading Wrapper configuration...
STATUS | wrapper | main | 2020/08/27 09:09:55.385 | Launching a JVM...
ERROR | wrapper | main | 2020/08/27 09:09:55.425 | JVM exited while loading the application.
INFO | jvm 5 | main | 2020/08/27 09:09:55.425 | Java HotSpot(TM) 64-Bit Server VM warning: INFO: os::commit_memory(0x00000000e9990000,0) Failed; error='Cannot allocate memory' (errno=12)
INFO | jvm 5 | main | 2020/08/27 09:09:55.425 | #
INFO | jvm 5 | main | 2020/08/27 09:09:55.425 | # There is insufficient memory for the Java Runtime Environment to continue.
INFO | jvm 5 | main | 2020/08/27 09:09:55.425 | # Native memory allocation (mmap) Failed to map 29366878208 bytes for committing reserved memory.
INFO | jvm 5 | main | 2020/08/27 09:09:55.425 | # An error report file with more information is saved as:
INFO | jvm 5 | main | 2020/08/27 09:09:55.425 | # /home/user/hybris/bin/platform/tomcat/bin/hs_err_pid31556.log
STATUS | wrapper | main | 2020/08/27 09:09:55.827 | Reloading Wrapper configuration...
STATUS | wrapper | main | 2020/08/27 09:09:56.039 | Launching a JVM...
ERROR | wrapper | main | 2020/08/27 09:09:56.076 | JVM exited while loading the application.
INFO | jvm 6 | main | 2020/08/27 09:09:56.077 | Java HotSpot(TM) 64-Bit Server VM warning: INFO: os::commit_memory(0x00000000e9990000,0) Failed; error='Cannot allocate memory' (errno=12)
INFO | jvm 6 | main | 2020/08/27 09:09:56.077 | #
INFO | jvm 6 | main | 2020/08/27 09:09:56.077 | # There is insufficient memory for the Java Runtime Environment to continue.
INFO | jvm 6 | main | 2020/08/27 09:09:56.077 | # Native memory allocation (mmap) Failed to map 29366878208 bytes for committing reserved memory.
INFO | jvm 6 | main | 2020/08/27 09:09:56.077 | # An error report file with more information is saved as:
INFO | jvm 6 | main | 2020/08/27 09:09:56.077 | # /home/user/hybris/bin/platform/tomcat/bin/hs_err_pid31566.log
FATAL | wrapper | main | 2020/08/27 09:09:56.378 | There were 5 Failed launches in a row,each lasting less than 300 seconds. Giving up.
FATAL | wrapper | main | 2020/08/27 09:09:56.379 | There may be a configuration problem: please check the logs.
STATUS | wrapper | main | 2020/08/27 09:09:56.580 | <-- Wrapper Stopped
对我来说，服务器似乎无缘无故地关闭了，但是并没有完全关闭并保留了当前RAM，然后尝试再次启动，但是没有足够的RAM。
这也是我看到的第一个项目，hybry自行关闭。我已经在其他4个大型项目中工作过，从未遇到过此类问题。
解决方法
这对我来说似乎是非标准的终止。在日志中，您可以看到
JVM收到信号SIGKILL（9）
这是外部发送杀死-9的东西。如果包装器执行了此操作，则在包装器最终终止-9之前，您应该看到有关请求关闭＆然后关闭关闭操作的消息。该日志显示了kill -9直接起作用。
我想您需要查看外部系统和/或O / S-有人告诉我，如果系统处于严重的内存压力下，O / S可以自行执行此操作。我知道您说系统有足够的内存，但是可能还有其他事情要消耗其他地方的内存吗？或者这可能是实际硬件被超额预订的VM，因此VM没有足够的内存出现了吗？