I'm getting this error when the nodes are shutting down. What are the possible causes for this? A bug was marked for this a year ago or so, but was marked as resolved, it seems? http://apache-ignite-issues.70530.x6.nabble.com/jira-Updated-IGNITE-9826-Ignite-node-with-TcpDiscoveryS3IpFinder-can-hang-while-stopping-td75612.html
Ignite cache is empty. Shutting down... 2019-10-02 08:44:37 [main] INFO org.apache.ignite.internal.processors.rest.protocols.tcp.GridTcpRestProtocol.info(117) - Command protocol successfully stopped: TCP binary 2019-10-02 08:44:37 [main] INFO org.eclipse.jetty.server.AbstractConnector.doStop(332) - Stopped ServerConnector@70101687{HTTP/1.1,[http/1.1]}{0.0.0.0:8080} 2019-10-02 08:44:37 [main] INFO org.apache.ignite.internal.processors.rest.protocols.http.jetty.GridJettyRestProtocol.info(117) - Command protocol successfully stopped: Jetty REST 2019-10-02 08:44:37 [grid-timeout-worker-#79] INFO org.apache.ignite.internal.IgniteKernal.info(117) - Metrics for local node (to disable set 'metricsLogFrequency' to 0) ^-- Node [id=c51b8462, uptime=00:15:00.073] ^-- H/N/C [hosts=2, nodes=2, CPUs=72] ^-- CPU [cur=-100%, avg=-99.73%, GC=0%] ^-- PageMemory [pages=1226] ^-- Heap [used=112MB, free=99.59%, comm=516MB] ^-- Off-heap [used=4MB, free=99.97%, comm=336MB] ^-- sysMemPlc region [used=0MB, free=99.21%, comm=40MB] ^-- default region [used=4MB, free=99.97%, comm=256MB] ^-- TxLog region [used=0MB, free=100%, comm=40MB] ^-- Outbound messages queue [size=0] ^-- Public thread pool [active=0, idle=0, qSize=0] ^-- System thread pool [active=0, idle=6, qSize=0] 2019-10-02 08:44:37 [tcp-disco-sock-reader-#7] INFO org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi.info(117) - Finished serving remote node connection [rmtAddr=/10.0.11.180:49151, rmtPort=49151 2019-10-02 08:44:37 [tcp-disco-sock-reader-#4] INFO org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi.info(117) - Finished serving remote node connection [rmtAddr=/10.0.31.134:42413, rmtPort=42413 2019-10-02 08:44:37 [tcp-disco-sock-reader-#6] INFO org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi.info(117) - Finished serving remote node connection [rmtAddr=/10.0.21.167:60763, rmtPort=60763 2019-10-02 08:44:37 [tcp-disco-ip-finder-cleaner-#5] ERROR org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi.error(137) - Failed to clean IP finder up. org.apache.ignite.spi.IgniteSpiException: Failed to list objects in the bucket: ignite-configurations-production at org.apache.ignite.spi.discovery.tcp.ipfinder.s3.TcpDiscoveryS3IpFinder.getRegisteredAddresses(TcpDiscoveryS3IpFinder.java:192) at org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi.registeredAddresses(TcpDiscoverySpi.java:1900) at org.apache.ignite.spi.discovery.tcp.ServerImpl$IpFinderCleaner.cleanIpFinder(ServerImpl.java:1998) at org.apache.ignite.spi.discovery.tcp.ServerImpl$IpFinderCleaner.body(ServerImpl.java:1973) at org.apache.ignite.spi.IgniteSpiThread.run(IgniteSpiThread.java:62) Caused by: com.amazonaws.SdkClientException: Failed to sanitize XML document destined for handler class com.amazonaws.services.s3.model.transform.XmlResponsesSaxParser$ListBucketHandler at com.amazonaws.services.s3.model.transform.XmlResponsesSaxParser.sanitizeXmlDocument(XmlResponsesSaxParser.java:214) at com.amazonaws.services.s3.model.transform.XmlResponsesSaxParser.parseListBucketObjectsResponse(XmlResponsesSaxParser.java:298) at com.amazonaws.services.s3.model.transform.Unmarshallers$ListObjectsUnmarshaller.unmarshall(Unmarshallers.java:70) at com.amazonaws.services.s3.model.transform.Unmarshallers$ListObjectsUnmarshaller.unmarshall(Unmarshallers.java:59) at com.amazonaws.services.s3.internal.S3XmlResponseHandler.handle(S3XmlResponseHandler.java:62) at com.amazonaws.services.s3.internal.S3XmlResponseHandler.handle(S3XmlResponseHandler.java:31) at com.amazonaws.http.response.AwsResponseHandlerAdapter.handle(AwsResponseHandlerAdapter.java:70) at com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleResponse(AmazonHttpClient.java:1554) at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeOneRequest(AmazonHttpClient.java:1272) at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:1056) at com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:743) at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:717) at com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:699) at com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$500(AmazonHttpClient.java:667) at com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:649) at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:513) at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:4137) at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:4079) at com.amazonaws.services.s3.AmazonS3Client.listObjects(AmazonS3Client.java:819) at com.amazonaws.services.s3.AmazonS3Client.listObjects(AmazonS3Client.java:791) at org.apache.ignite.spi.discovery.tcp.ipfinder.s3.TcpDiscoveryS3IpFinder.getRegisteredAddresses(TcpDiscoveryS3IpFinder.java:146) ... 4 common frames omitted Caused by: com.amazonaws.AbortedException: at com.amazonaws.internal.SdkFilterInputStream.abortIfNeeded(SdkFilterInputStream.java:53) at com.amazonaws.internal.SdkFilterInputStream.read(SdkFilterInputStream.java:81) at com.amazonaws.event.ProgressInputStream.read(ProgressInputStream.java:180) at java.base/sun.nio.cs.StreamDecoder.readBytes(StreamDecoder.java:284) at java.base/sun.nio.cs.StreamDecoder.implRead(StreamDecoder.java:326) at java.base/sun.nio.cs.StreamDecoder.read(StreamDecoder.java:178) at java.base/java.io.InputStreamReader.read(InputStreamReader.java:185) at java.base/java.io.BufferedReader.read1(BufferedReader.java:210) at java.base/java.io.BufferedReader.read(BufferedReader.java:287) at java.base/java.io.Reader.read(Reader.java:229) at com.amazonaws.services.s3.model.transform.XmlResponsesSaxParser.sanitizeXmlDocument(XmlResponsesSaxParser.java:186) ... 24 common frames omitted 2019-10-02 08:44:40 [tcp-disco-msg-worker-#2] WARN org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi.warning(127) - Local node has detected failed nodes and started cluster-wide procedure. To speed up failure detection please see 'Failure Detection' section under javadoc for 'TcpDiscoverySpi'