Hi, When loading ignite cache, we saw the spark job went into hung state at this step. We see one of the executor task has been running for hours and below are the logs from this executor that had the failure.
Stdout log Launch class org.apache.spark.executor.CoarseGrainedExecutorBackend by calling co.cask.cdap.app.runtime.spark.distributed.SparkContainerLauncher.launch 13:12:58.115 [main] INFO c.c.c.l.a.LogAppenderInitializer - Initializing log appender KafkaLogAppender 13:12:58.679 [authorization-enforcement-service] INFO c.c.c.s.a.AbstractAuthorizationService - Started authorization enforcement service... 13:12:59.391 [main] INFO c.c.c.c.g.LocationRuntimeModule - HDFS namespace is /project/ecpprodcdap 13:12:59.438 [main] INFO c.c.c.a.r.s.d.SparkContainerLauncher - Launch main class org.apache.spark.executor.CoarseGrainedExecutorBackend.main([--driver-url, spark://CoarseGrainedScheduler@10.214.4.161:33947, --executor-id, 29, --hostname, c893ach.ecom.bigdata.int.thomsonreuters.com, --cores, 5, --app-id, application_1506331241975_7951, --user-class-path, file:/data/7/yarn/nm/usercache/bigdata-app-ecplegalanalytics-svc/appcache/application_1506331241975_7951/container_e28_1506331241975_7951_01_000030/__app__.jar]) 13:12:59.501 [main] WARN c.c.c.i.a.Classes - Cannot patch method obtainTokenForHiveMetastore in org.apache.spark.deploy.yarn.YarnSparkHadoopUtil due to non-void return type: (Lorg/apache/hadoop/conf/Configuration;)Lscala/Option; 13:12:59.501 [main] WARN c.c.c.i.a.Classes - Cannot patch method obtainTokenForHBase in org.apache.spark.deploy.yarn.YarnSparkHadoopUtil due to non-void return type: (Lorg/apache/hadoop/conf/Configuration;)Lscala/Option; 13:13:26.130 [Executor task launch worker-0] WARN o.a.s.e.CoarseGrainedExecutorBackend - 17/11/16 13:13:26 INFO dataloader.IgniteDataLoader: Starting the Ignite node on - 10.214.4.161 13:13:26.134 [Executor task launch worker-3] WARN o.a.s.e.CoarseGrainedExecutorBackend - 17/11/16 13:13:26 INFO dataloader.IgniteDataLoader: Starting the Ignite node on - 10.214.4.161 13:13:26.134 [Executor task launch worker-2] WARN o.a.s.e.CoarseGrainedExecutorBackend - 17/11/16 13:13:26 INFO dataloader.IgniteDataLoader: Starting the Ignite node on - 10.214.4.161 13:13:26.135 [Executor task launch worker-1] WARN o.a.s.e.CoarseGrainedExecutorBackend - 17/11/16 13:13:26 INFO dataloader.IgniteDataLoader: Starting the Ignite node on - 10.214.4.161 13:13:26.135 [Executor task launch worker-4] WARN o.a.s.e.CoarseGrainedExecutorBackend - 17/11/16 13:13:26 INFO dataloader.IgniteDataLoader: Starting the Ignite node on - 10.214.4.161 13:13:26.281 [Executor task launch worker-0] ERROR - Failed to resolve default logging config file: config/java.util.logging.properties 13:13:26.283 [Executor task launch worker-0] WARN o.a.s.e.CoarseGrainedExecutorBackend - Console logging handler is not configured. [13:13:26] __________ ________________ [13:13:26] / _/ ___/ |/ / _/_ __/ __/ [13:13:26] _/ // (7 7 // / / / / _/ [13:13:26] /___/\___/_/|_/___/ /_/ /___/ [13:13:26] [13:13:26] ver. 1.8.0#20161205-sha1:9ca40dbe [13:13:26] 2016 Copyright(C) Apache Software Foundation [13:13:26] [13:13:26] Ignite documentation: http://ignite.apache.org [13:13:26] [13:13:26] Quiet mode. [13:13:26] ^-- To see **FULL** console log here add -DIGNITE_QUIET=false or "-v" to ignite.{sh|bat} [13:13:26] [13:13:26] OS: Linux 3.10.0-514.16.1.el7.x86_64 amd64 [13:13:26] VM information: Java(TM) SE Runtime Environment 1.8.0_121-b13 Oracle Corporation Java HotSpot(TM) 64-Bit Server VM 25.121-b13 [13:13:26] Configured plugins: [13:13:26] ^-- None [13:13:26] [13:13:26] Security status [authentication=off, tls/ssl=off] [13:13:27] Topology snapshot [ver=3, servers=3, clients=0, CPUs=48, heap=96.0GB] [13:13:27] To start Console Management & Monitoring run ignitevisorcmd.{sh|bat} [13:13:27] [13:13:27] Ignite node started OK (id=e98b003d, grid=WCAGridapplication_1506331241975_7951) [13:13:27] Topology snapshot [ver=2, servers=2, clients=0, CPUs=48, heap=66.0GB] 13:13:27.660 [Executor task launch worker-0] WARN o.a.s.e.CoarseGrainedExecutorBackend - 17/11/16 13:13:27 INFO dataloader.IgniteDataLoader: Started the Ignite node on - 10.214.4.161 13:13:27.660 [Executor task launch worker-2] WARN o.a.s.e.CoarseGrainedExecutorBackend - 17/11/16 13:13:27 INFO dataloader.IgniteDataLoader: Started the Ignite node on - 10.214.4.161 13:13:27.661 [Executor task launch worker-3] WARN o.a.s.e.CoarseGrainedExecutorBackend - 17/11/16 13:13:27 INFO dataloader.IgniteDataLoader: Started the Ignite node on - 10.214.4.161 13:13:27.661 [Executor task launch worker-1] WARN o.a.s.e.CoarseGrainedExecutorBackend - 17/11/16 13:13:27 INFO dataloader.IgniteDataLoader: Started the Ignite node on - 10.214.4.161 13:13:27.661 [Executor task launch worker-4] WARN o.a.s.e.CoarseGrainedExecutorBackend - 17/11/16 13:13:27 INFO dataloader.IgniteDataLoader: Started the Ignite node on - 10.214.4.161 13:13:27.674 [Executor task launch worker-4] WARN o.a.s.e.CoarseGrainedExecutorBackend - 17/11/16 13:13:27 INFO dataloader.IgniteDataLoader: Starting the Ignite node on - 10.214.4.161 13:13:27.674 [Executor task launch worker-2] WARN o.a.s.e.CoarseGrainedExecutorBackend - 17/11/16 13:13:27 INFO dataloader.IgniteDataLoader: Starting the Ignite node on - 10.214.4.161 13:13:27.676 [Executor task launch worker-4] WARN o.a.s.e.CoarseGrainedExecutorBackend - 17/11/16 13:13:27 INFO dataloader.IgniteDataLoader: Started the Ignite node on - 10.214.4.161 13:13:27.676 [Executor task launch worker-3] WARN o.a.s.e.CoarseGrainedExecutorBackend - 17/11/16 13:13:27 INFO dataloader.IgniteDataLoader: Starting the Ignite node on - 10.214.4.161 13:13:27.677 [Executor task launch worker-2] WARN o.a.s.e.CoarseGrainedExecutorBackend - 17/11/16 13:13:27 INFO dataloader.IgniteDataLoader: Started the Ignite node on - 10.214.4.161 13:13:27.678 [Executor task launch worker-1] WARN o.a.s.e.CoarseGrainedExecutorBackend - 17/11/16 13:13:27 INFO dataloader.IgniteDataLoader: Starting the Ignite node on - 10.214.4.161 13:13:27.678 [Executor task launch worker-3] WARN o.a.s.e.CoarseGrainedExecutorBackend - 17/11/16 13:13:27 INFO dataloader.IgniteDataLoader: Started the Ignite node on - 10.214.4.161 13:13:27.679 [Executor task launch worker-1] WARN o.a.s.e.CoarseGrainedExecutorBackend - 17/11/16 13:13:27 INFO dataloader.IgniteDataLoader: Started the Ignite node on - 10.214.4.161 13:13:27.680 [Executor task launch worker-0] WARN o.a.s.e.CoarseGrainedExecutorBackend - 17/11/16 13:13:27 INFO dataloader.IgniteDataLoader: Starting the Ignite node on - 10.214.4.161 13:13:27.682 [Executor task launch worker-0] WARN o.a.s.e.CoarseGrainedExecutorBackend - 17/11/16 13:13:27 INFO dataloader.IgniteDataLoader: Started the Ignite node on - 10.214.4.161 13:13:27.689 [Executor task launch worker-1] WARN o.a.s.e.CoarseGrainedExecutorBackend - 17/11/16 13:13:27 INFO dataloader.IgniteDataLoader: Starting the Ignite node on - 10.214.4.161 13:13:27.690 [Executor task launch worker-0] WARN o.a.s.e.CoarseGrainedExecutorBackend - 17/11/16 13:13:27 INFO dataloader.IgniteDataLoader: Starting the Ignite node on - 10.214.4.161 13:13:27.691 [Executor task launch worker-1] WARN o.a.s.e.CoarseGrainedExecutorBackend - 17/11/16 13:13:27 INFO dataloader.IgniteDataLoader: Started the Ignite node on - 10.214.4.161 13:13:27.692 [Executor task launch worker-0] WARN o.a.s.e.CoarseGrainedExecutorBackend - 17/11/16 13:13:27 INFO dataloader.IgniteDataLoader: Started the Ignite node on - 10.214.4.161 13:13:27.692 [Executor task launch worker-3] WARN o.a.s.e.CoarseGrainedExecutorBackend - 17/11/16 13:13:27 INFO dataloader.IgniteDataLoader: Starting the Ignite node on - 10.214.4.161 13:13:27.694 [Executor task launch worker-3] WARN o.a.s.e.CoarseGrainedExecutorBackend - 17/11/16 13:13:27 INFO dataloader.IgniteDataLoader: Started the Ignite node on - 10.214.4.161 13:13:27.695 [Executor task launch worker-2] WARN o.a.s.e.CoarseGrainedExecutorBackend - 17/11/16 13:13:27 INFO dataloader.IgniteDataLoader: Starting the Ignite node on - 10.214.4.161 13:13:27.696 [Executor task launch worker-2] WARN o.a.s.e.CoarseGrainedExecutorBackend - 17/11/16 13:13:27 INFO dataloader.IgniteDataLoader: Started the Ignite node on - 10.214.4.161 13:13:27.717 [Executor task launch worker-2] WARN o.a.s.e.CoarseGrainedExecutorBackend - 17/11/16 13:13:27 INFO dataloader.IgniteDataLoader: Starting the Ignite node on - 10.214.4.161 13:13:27.718 [Executor task launch worker-3] WARN o.a.s.e.CoarseGrainedExecutorBackend - 17/11/16 13:13:27 INFO dataloader.IgniteDataLoader: Starting the Ignite node on - 10.214.4.161 13:13:27.719 [Executor task launch worker-2] WARN o.a.s.e.CoarseGrainedExecutorBackend - 17/11/16 13:13:27 INFO dataloader.IgniteDataLoader: Started the Ignite node on - 10.214.4.161 13:13:27.720 [Executor task launch worker-3] WARN o.a.s.e.CoarseGrainedExecutorBackend - 17/11/16 13:13:27 INFO dataloader.IgniteDataLoader: Started the Ignite node on - 10.214.4.161 13:13:27.725 [Executor task launch worker-4] WARN o.a.s.e.CoarseGrainedExecutorBackend - 17/11/16 13:13:27 INFO dataloader.IgniteDataLoader: Starting the Ignite node on - 10.214.4.161 13:13:27.725 [Executor task launch worker-1] WARN o.a.s.e.CoarseGrainedExecutorBackend - 17/11/16 13:13:27 INFO dataloader.IgniteDataLoader: Starting the Ignite node on - 10.214.4.161 13:13:27.726 [Executor task launch worker-0] WARN o.a.s.e.CoarseGrainedExecutorBackend - 17/11/16 13:13:27 INFO dataloader.IgniteDataLoader: Starting the Ignite node on - 10.214.4.161 13:13:27.726 [Executor task launch worker-4] WARN o.a.s.e.CoarseGrainedExecutorBackend - 17/11/16 13:13:27 INFO dataloader.IgniteDataLoader: Started the Ignite node on - 10.214.4.161 13:13:27.727 [Executor task launch worker-1] WARN o.a.s.e.CoarseGrainedExecutorBackend - 17/11/16 13:13:27 INFO dataloader.IgniteDataLoader: Started the Ignite node on - 10.214.4.161 13:13:27.728 [Executor task launch worker-0] WARN o.a.s.e.CoarseGrainedExecutorBackend - 17/11/16 13:13:27 INFO dataloader.IgniteDataLoader: Started the Ignite node on - 10.214.4.161 13:13:27.745 [Executor task launch worker-0] WARN o.a.s.e.CoarseGrainedExecutorBackend - 17/11/16 13:13:27 INFO dataloader.IgniteDataLoader: Starting the Ignite node on - 10.214.4.161 13:13:27.746 [Executor task launch worker-0] WARN o.a.s.e.CoarseGrainedExecutorBackend - 17/11/16 13:13:27 INFO dataloader.IgniteDataLoader: Started the Ignite node on - 10.214.4.161 13:13:27.747 [Executor task launch worker-1] WARN o.a.s.e.CoarseGrainedExecutorBackend - 17/11/16 13:13:27 INFO dataloader.IgniteDataLoader: Starting the Ignite node on - 10.214.4.161 13:13:27.749 [Executor task launch worker-1] WARN o.a.s.e.CoarseGrainedExecutorBackend - 17/11/16 13:13:27 INFO dataloader.IgniteDataLoader: Started the Ignite node on - 10.214.4.161 13:13:27.750 [Executor task launch worker-4] WARN o.a.s.e.CoarseGrainedExecutorBackend - 17/11/16 13:13:27 INFO dataloader.IgniteDataLoader: Starting the Ignite node on - 10.214.4.161 13:13:27.751 [Executor task launch worker-4] WARN o.a.s.e.CoarseGrainedExecutorBackend - 17/11/16 13:13:27 INFO dataloader.IgniteDataLoader: Started the Ignite node on - 10.214.4.161 13:13:27.753 [Executor task launch worker-3] WARN o.a.s.e.CoarseGrainedExecutorBackend - 17/11/16 13:13:27 INFO dataloader.IgniteDataLoader: Starting the Ignite node on - 10.214.4.161 13:13:27.755 [Executor task launch worker-1] WARN o.a.s.e.CoarseGrainedExecutorBackend - 17/11/16 13:13:27 INFO dataloader.IgniteDataLoader: Starting the Ignite node on - 10.214.4.161 13:13:27.755 [Executor task launch worker-3] WARN o.a.s.e.CoarseGrainedExecutorBackend - 17/11/16 13:13:27 INFO dataloader.IgniteDataLoader: Started the Ignite node on - 10.214.4.161 13:13:27.756 [Executor task launch worker-1] WARN o.a.s.e.CoarseGrainedExecutorBackend - 17/11/16 13:13:27 INFO dataloader.IgniteDataLoader: Started the Ignite node on - 10.214.4.161 13:13:27.772 [Executor task launch worker-1] WARN o.a.s.e.CoarseGrainedExecutorBackend - 17/11/16 13:13:27 INFO dataloader.IgniteDataLoader: Starting the Ignite node on - 10.214.4.161 13:13:27.774 [Executor task launch worker-1] WARN o.a.s.e.CoarseGrainedExecutorBackend - 17/11/16 13:13:27 INFO dataloader.IgniteDataLoader: Started the Ignite node on - 10.214.4.161 13:13:27.784 [Executor task launch worker-4] WARN o.a.s.e.CoarseGrainedExecutorBackend - 17/11/16 13:13:27 INFO dataloader.IgniteDataLoader: Starting the Ignite node on - 10.214.4.161 13:13:27.785 [Executor task launch worker-0] WARN o.a.s.e.CoarseGrainedExecutorBackend - 17/11/16 13:13:27 INFO dataloader.IgniteDataLoader: Starting the Ignite node on - 10.214.4.161 13:13:27.786 [Executor task launch worker-1] WARN o.a.s.e.CoarseGrainedExecutorBackend - 17/11/16 13:13:27 INFO dataloader.IgniteDataLoader: Starting the Ignite node on - 10.214.4.161 13:13:27.787 [Executor task launch worker-4] WARN o.a.s.e.CoarseGrainedExecutorBackend - 17/11/16 13:13:27 INFO dataloader.IgniteDataLoader: Started the Ignite node on - 10.214.4.161 13:13:27.787 [Executor task launch worker-0] WARN o.a.s.e.CoarseGrainedExecutorBackend - 17/11/16 13:13:27 INFO dataloader.IgniteDataLoader: Started the Ignite node on - 10.214.4.161 13:13:27.788 [Executor task launch worker-3] WARN o.a.s.e.CoarseGrainedExecutorBackend - 17/11/16 13:13:27 INFO dataloader.IgniteDataLoader: Starting the Ignite node on - 10.214.4.161 13:13:27.788 [Executor task launch worker-1] WARN o.a.s.e.CoarseGrainedExecutorBackend - 17/11/16 13:13:27 INFO dataloader.IgniteDataLoader: Started the Ignite node on - 10.214.4.161 13:13:27.789 [Executor task launch worker-3] WARN o.a.s.e.CoarseGrainedExecutorBackend - 17/11/16 13:13:27 INFO dataloader.IgniteDataLoader: Started the Ignite node on - 10.214.4.161 13:13:27.791 [Executor task launch worker-2] WARN o.a.s.e.CoarseGrainedExecutorBackend - 17/11/16 13:13:27 INFO dataloader.IgniteDataLoader: Starting the Ignite node on - 10.214.4.161 13:13:27.792 [Executor task launch worker-2] WARN o.a.s.e.CoarseGrainedExecutorBackend - 17/11/16 13:13:27 INFO dataloader.IgniteDataLoader: Started the Ignite node on - 10.214.4.161 13:13:27.803 [Executor task launch worker-2] WARN o.a.s.e.CoarseGrainedExecutorBackend - 17/11/16 13:13:27 INFO dataloader.IgniteDataLoader: Starting the Ignite node on - 10.214.4.161 13:13:27.804 [Executor task launch worker-2] WARN o.a.s.e.CoarseGrainedExecutorBackend - 17/11/16 13:13:27 INFO dataloader.IgniteDataLoader: Started the Ignite node on - 10.214.4.161 13:13:27.805 [Executor task launch worker-3] WARN o.a.s.e.CoarseGrainedExecutorBackend - 17/11/16 13:13:27 INFO dataloader.IgniteDataLoader: Starting the Ignite node on - 10.214.4.161 13:13:27.807 [Executor task launch worker-3] WARN o.a.s.e.CoarseGrainedExecutorBackend - 17/11/16 13:13:27 INFO dataloader.IgniteDataLoader: Started the Ignite node on - 10.214.4.161 13:13:27.808 [Executor task launch worker-4] WARN o.a.s.e.CoarseGrainedExecutorBackend - 17/11/16 13:13:27 INFO dataloader.IgniteDataLoader: Starting the Ignite node on - 10.214.4.161 13:13:27.809 [Executor task launch worker-1] WARN o.a.s.e.CoarseGrainedExecutorBackend - 17/11/16 13:13:27 INFO dataloader.IgniteDataLoader: Starting the Ignite node on - 10.214.4.161 13:13:27.810 [Executor task launch worker-4] WARN o.a.s.e.CoarseGrainedExecutorBackend - 17/11/16 13:13:27 INFO dataloader.IgniteDataLoader: Started the Ignite node on - 10.214.4.161 13:13:27.810 [Executor task launch worker-1] WARN o.a.s.e.CoarseGrainedExecutorBackend - 17/11/16 13:13:27 INFO dataloader.IgniteDataLoader: Started the Ignite node on - 10.214.4.161 13:13:27.812 [Executor task launch worker-0] WARN o.a.s.e.CoarseGrainedExecutorBackend - 17/11/16 13:13:27 INFO dataloader.IgniteDataLoader: Starting the Ignite node on - 10.214.4.161 13:13:27.814 [Executor task launch worker-0] WARN o.a.s.e.CoarseGrainedExecutorBackend - 17/11/16 13:13:27 INFO dataloader.IgniteDataLoader: Started the Ignite node on - 10.214.4.161 13:13:27.816 [Executor task launch worker-3] WARN o.a.s.e.CoarseGrainedExecutorBackend - 17/11/16 13:13:27 INFO dataloader.IgniteDataLoader: Starting the Ignite node on - 10.214.4.161 13:13:27.817 [Executor task launch worker-3] WARN o.a.s.e.CoarseGrainedExecutorBackend - 17/11/16 13:13:27 INFO dataloader.IgniteDataLoader: Started the Ignite node on - 10.214.4.161 13:13:27.819 [Executor task launch worker-4] WARN o.a.s.e.CoarseGrainedExecutorBackend - 17/11/16 13:13:27 INFO dataloader.IgniteDataLoader: Starting the Ignite node on - 10.214.4.161 13:13:27.821 [Executor task launch worker-4] WARN o.a.s.e.CoarseGrainedExecutorBackend - 17/11/16 13:13:27 INFO dataloader.IgniteDataLoader: Started the Ignite node on - 10.214.4.161 13:13:27.828 [Executor task launch worker-4] WARN o.a.s.e.CoarseGrainedExecutorBackend - 17/11/16 13:13:27 INFO dataloader.IgniteDataLoader: Starting the Ignite node on - 10.214.4.161 13:13:27.829 [Executor task launch worker-4] WARN o.a.s.e.CoarseGrainedExecutorBackend - 17/11/16 13:13:27 INFO dataloader.IgniteDataLoader: Started the Ignite node on - 10.214.4.161 13:13:27.830 [Executor task launch worker-3] WARN o.a.s.e.CoarseGrainedExecutorBackend - 17/11/16 13:13:27 INFO dataloader.IgniteDataLoader: Starting the Ignite node on - 10.214.4.161 13:13:27.832 [Executor task launch worker-3] WARN o.a.s.e.CoarseGrainedExecutorBackend - 17/11/16 13:13:27 INFO dataloader.IgniteDataLoader: Started the Ignite node on - 10.214.4.161 13:13:27.832 [Executor task launch worker-0] WARN o.a.s.e.CoarseGrainedExecutorBackend - 17/11/16 13:13:27 INFO dataloader.IgniteDataLoader: Starting the Ignite node on - 10.214.4.161 13:13:27.833 [Executor task launch worker-0] WARN o.a.s.e.CoarseGrainedExecutorBackend - 17/11/16 13:13:27 INFO dataloader.IgniteDataLoader: Started the Ignite node on - 10.214.4.161 13:13:27.845 [Executor task launch worker-0] WARN o.a.s.e.CoarseGrainedExecutorBackend - 17/11/16 13:13:27 INFO dataloader.IgniteDataLoader: Starting the Ignite node on - 10.214.4.161 13:13:27.846 [Executor task launch worker-0] WARN o.a.s.e.CoarseGrainedExecutorBackend - 17/11/16 13:13:27 INFO dataloader.IgniteDataLoader: Started the Ignite node on - 10.214.4.161 13:13:27.848 [Executor task launch worker-3] WARN o.a.s.e.CoarseGrainedExecutorBackend - 17/11/16 13:13:27 INFO dataloader.IgniteDataLoader: Starting the Ignite node on - 10.214.4.161 13:13:27.849 [Executor task launch worker-3] WARN o.a.s.e.CoarseGrainedExecutorBackend - 17/11/16 13:13:27 INFO dataloader.IgniteDataLoader: Started the Ignite node on - 10.214.4.161 13:13:27.856 [Executor task launch worker-0] WARN o.a.s.e.CoarseGrainedExecutorBackend - 17/11/16 13:13:27 INFO dataloader.IgniteDataLoader: Starting the Ignite node on - 10.214.4.161 13:13:27.857 [Executor task launch worker-0] WARN o.a.s.e.CoarseGrainedExecutorBackend - 17/11/16 13:13:27 INFO dataloader.IgniteDataLoader: Started the Ignite node on - 10.214.4.161 13:13:27.859 [Executor task launch worker-3] WARN o.a.s.e.CoarseGrainedExecutorBackend - 17/11/16 13:13:27 INFO dataloader.IgniteDataLoader: Starting the Ignite node on - 10.214.4.161 13:13:27.860 [Executor task launch worker-3] WARN o.a.s.e.CoarseGrainedExecutorBackend - 17/11/16 13:13:27 INFO dataloader.IgniteDataLoader: Started the Ignite node on - 10.214.4.161 13:13:27.864 [Executor task launch worker-0] WARN o.a.s.e.CoarseGrainedExecutorBackend - 17/11/16 13:13:27 INFO dataloader.IgniteDataLoader: Starting the Ignite node on - 10.214.4.161 13:13:27.865 [Executor task launch worker-0] WARN o.a.s.e.CoarseGrainedExecutorBackend - 17/11/16 13:13:27 INFO dataloader.IgniteDataLoader: Started the Ignite node on - 10.214.4.161 13:13:27.872 [Executor task launch worker-3] WARN o.a.s.e.CoarseGrainedExecutorBackend - 17/11/16 13:13:27 INFO dataloader.IgniteDataLoader: Starting the Ignite node on - 10.214.4.161 13:13:27.873 [Executor task launch worker-3] WARN o.a.s.e.CoarseGrainedExecutorBackend - 17/11/16 13:13:27 INFO dataloader.IgniteDataLoader: Started the Ignite node on - 10.214.4.161 13:13:27.874 [Executor task launch worker-0] WARN o.a.s.e.CoarseGrainedExecutorBackend - 17/11/16 13:13:27 INFO dataloader.IgniteDataLoader: Starting the Ignite node on - 10.214.4.161 13:13:27.875 [Executor task launch worker-0] WARN o.a.s.e.CoarseGrainedExecutorBackend - 17/11/16 13:13:27 INFO dataloader.IgniteDataLoader: Started the Ignite node on - 10.214.4.161 13:13:27.878 [Executor task launch worker-4] WARN o.a.s.e.CoarseGrainedExecutorBackend - 17/11/16 13:13:27 INFO dataloader.IgniteDataLoader: Starting the Ignite node on - 10.214.4.161 13:13:27.879 [Executor task launch worker-4] WARN o.a.s.e.CoarseGrainedExecutorBackend - 17/11/16 13:13:27 INFO dataloader.IgniteDataLoader: Started the Ignite node on - 10.214.4.161 13:13:27.887 [Executor task launch worker-0] WARN o.a.s.e.CoarseGrainedExecutorBackend - 17/11/16 13:13:27 INFO dataloader.IgniteDataLoader: Starting the Ignite node on - 10.214.4.161 13:13:27.888 [Executor task launch worker-0] WARN o.a.s.e.CoarseGrainedExecutorBackend - 17/11/16 13:13:27 INFO dataloader.IgniteDataLoader: Started the Ignite node on - 10.214.4.161 13:13:27.889 [Executor task launch worker-4] WARN o.a.s.e.CoarseGrainedExecutorBackend - 17/11/16 13:13:27 INFO dataloader.IgniteDataLoader: Starting the Ignite node on - 10.214.4.161 13:13:27.889 [Executor task launch worker-3] WARN o.a.s.e.CoarseGrainedExecutorBackend - 17/11/16 13:13:27 INFO dataloader.IgniteDataLoader: Starting the Ignite node on - 10.214.4.161 13:13:27.890 [Executor task launch worker-4] WARN o.a.s.e.CoarseGrainedExecutorBackend - 17/11/16 13:13:27 INFO dataloader.IgniteDataLoader: Started the Ignite node on - 10.214.4.161 13:13:27.891 [Executor task launch worker-3] WARN o.a.s.e.CoarseGrainedExecutorBackend - 17/11/16 13:13:27 INFO dataloader.IgniteDataLoader: Started the Ignite node on - 10.214.4.161 13:13:27.899 [Executor task launch worker-4] WARN o.a.s.e.CoarseGrainedExecutorBackend - 17/11/16 13:13:27 INFO dataloader.IgniteDataLoader: Starting the Ignite node on - 10.214.4.161 13:13:27.901 [Executor task launch worker-4] WARN o.a.s.e.CoarseGrainedExecutorBackend - 17/11/16 13:13:27 INFO dataloader.IgniteDataLoader: Started the Ignite node on - 10.214.4.161 [13:14:02] Topology snapshot [ver=4, servers=4, clients=0, CPUs=96, heap=130.0GB] [13:14:02] Topology snapshot [ver=5, servers=5, clients=0, CPUs=144, heap=160.0GB] [13:14:03] Topology snapshot [ver=6, servers=6, clients=0, CPUs=192, heap=190.0GB] [13:14:03] Topology snapshot [ver=7, servers=7, clients=0, CPUs=240, heap=220.0GB] [13:14:03] Topology snapshot [ver=8, servers=8, clients=0, CPUs=288, heap=250.0GB] [13:14:03] Topology snapshot [ver=9, servers=9, clients=0, CPUs=336, heap=280.0GB] [13:14:04] Topology snapshot [ver=10, servers=10, clients=0, CPUs=384, heap=310.0GB] [13:14:04] Topology snapshot [ver=11, servers=11, clients=0, CPUs=432, heap=340.0GB] [13:14:04] Topology snapshot [ver=12, servers=12, clients=0, CPUs=480, heap=370.0GB] [13:14:04] Topology snapshot [ver=13, servers=13, clients=0, CPUs=528, heap=400.0GB] [13:14:04] Topology snapshot [ver=14, servers=14, clients=0, CPUs=576, heap=430.0GB] [13:14:04] Topology snapshot [ver=15, servers=15, clients=0, CPUs=624, heap=460.0GB] [13:14:05] Topology snapshot [ver=16, servers=16, clients=0, CPUs=672, heap=490.0GB] [13:14:05] Topology snapshot [ver=17, servers=17, clients=0, CPUs=720, heap=520.0GB] [13:14:06] Topology snapshot [ver=18, servers=18, clients=0, CPUs=720, heap=550.0GB] [13:14:06] Topology snapshot [ver=19, servers=19, clients=0, CPUs=768, heap=580.0GB] [13:14:06] Topology snapshot [ver=20, servers=20, clients=0, CPUs=816, heap=610.0GB] [13:14:06] Topology snapshot [ver=21, servers=21, clients=0, CPUs=864, heap=640.0GB] [13:14:07] Topology snapshot [ver=22, servers=22, clients=0, CPUs=864, heap=670.0GB] [13:14:07] Topology snapshot [ver=23, servers=23, clients=0, CPUs=912, heap=700.0GB] [13:14:07] Topology snapshot [ver=24, servers=24, clients=0, CPUs=960, heap=730.0GB] [13:14:07] Topology snapshot [ver=25, servers=25, clients=0, CPUs=1008, heap=760.0GB] [13:14:08] Topology snapshot [ver=26, servers=26, clients=0, CPUs=1056, heap=790.0GB] [13:14:08] Topology snapshot [ver=27, servers=27, clients=0, CPUs=1104, heap=820.0GB] [13:14:08] Topology snapshot [ver=28, servers=28, clients=0, CPUs=1104, heap=850.0GB] [13:14:08] Topology snapshot [ver=29, servers=29, clients=0, CPUs=1152, heap=880.0GB] [13:14:09] Topology snapshot [ver=30, servers=30, clients=0, CPUs=1200, heap=910.0GB] [13:14:09] Topology snapshot [ver=31, servers=31, clients=0, CPUs=1248, heap=940.0GB] [13:14:09] Topology snapshot [ver=32, servers=32, clients=0, CPUs=1248, heap=970.0GB] [13:14:09] Topology snapshot [ver=33, servers=33, clients=0, CPUs=1296, heap=1000.0GB] [13:14:10] Topology snapshot [ver=34, servers=34, clients=0, CPUs=1344, heap=1000.0GB] [13:14:10] Topology snapshot [ver=35, servers=35, clients=0, CPUs=1344, heap=1100.0GB] [13:14:10] Topology snapshot [ver=36, servers=36, clients=0, CPUs=1392, heap=1100.0GB] [13:14:11] Topology snapshot [ver=37, servers=37, clients=0, CPUs=1440, heap=1100.0GB] [13:14:11] Topology snapshot [ver=38, servers=38, clients=0, CPUs=1440, heap=1100.0GB] [13:14:12] Topology snapshot [ver=39, servers=39, clients=0, CPUs=1440, heap=1200.0GB] [13:14:12] Topology snapshot [ver=40, servers=40, clients=0, CPUs=1488, heap=1200.0GB] [13:14:12] Topology snapshot [ver=41, servers=41, clients=0, CPUs=1536, heap=1200.0GB] [13:14:16] Topology snapshot [ver=42, servers=42, clients=0, CPUs=1536, heap=1300.0GB] [13:14:16] Topology snapshot [ver=43, servers=43, clients=0, CPUs=1584, heap=1300.0GB] [13:14:16] Topology snapshot [ver=44, servers=44, clients=0, CPUs=1632, heap=1300.0GB] [13:14:16] Topology snapshot [ver=45, servers=45, clients=0, CPUs=1632, heap=1400.0GB] [13:14:17] Topology snapshot [ver=46, servers=46, clients=0, CPUs=1680, heap=1400.0GB] [13:14:17] Topology snapshot [ver=47, servers=47, clients=0, CPUs=1680, heap=1400.0GB] [13:14:17] Topology snapshot [ver=48, servers=48, clients=0, CPUs=1680, heap=1400.0GB] [13:14:17] Topology snapshot [ver=49, servers=49, clients=0, CPUs=1680, heap=1500.0GB] [13:14:18] Topology snapshot [ver=50, servers=50, clients=0, CPUs=1728, heap=1500.0GB] [13:14:18] Topology snapshot [ver=51, servers=51, clients=0, CPUs=1728, heap=1500.0GB] [13:14:18] Topology snapshot [ver=52, servers=52, clients=0, CPUs=1776, heap=1600.0GB] [13:14:19] Topology snapshot [ver=53, servers=53, clients=0, CPUs=1776, heap=1600.0GB] [13:14:19] Topology snapshot [ver=54, servers=54, clients=0, CPUs=1824, heap=1600.0GB] [13:14:19] Topology snapshot [ver=55, servers=55, clients=0, CPUs=1872, heap=1700.0GB] [13:14:19] Topology snapshot [ver=56, servers=56, clients=0, CPUs=1872, heap=1700.0GB] [13:14:20] Topology snapshot [ver=57, servers=57, clients=0, CPUs=1920, heap=1700.0GB] [13:14:20] Topology snapshot [ver=58, servers=58, clients=0, CPUs=1968, heap=1700.0GB] [13:14:20] Topology snapshot [ver=59, servers=59, clients=0, CPUs=1968, heap=1800.0GB] [13:14:21] Topology snapshot [ver=60, servers=60, clients=0, CPUs=1968, heap=1800.0GB] [13:14:21] Topology snapshot [ver=61, servers=61, clients=0, CPUs=2016, heap=1800.0GB] [13:14:21] Topology snapshot [ver=62, servers=62, clients=0, CPUs=2064, heap=1900.0GB] [13:14:21] Topology snapshot [ver=63, servers=63, clients=0, CPUs=2064, heap=1900.0GB] 13:16:03.292 [driver-heartbeater] WARN o.a.s.r.n.NettyRpcEndpointRef - Error sending message [message = Heartbeat(29,[Lscala.Tuple2;@57dd5d40,BlockManagerId(29, c893ach.ecom.bigdata.int.thomsonreuters.com, 35301))] in 1 attempts org.apache.spark.rpc.RpcTimeoutException: Futures timed out after [10 seconds]. This timeout is controlled by spark.executor.heartbeatInterval at org.apache.spark.rpc.RpcTimeout.org$apache$spark$rpc$RpcTimeout$$createRpcTimeoutException(RpcTimeout.scala:48) ~[spark-assembly-1.6.0-cdh5.10.2-hadoop2.6.0-cdh5.10.2.jar:na] at org.apache.spark.rpc.RpcTimeout$$anonfun$addMessageIfTimeout$1.applyOrElse(RpcTimeout.scala:63) ~[spark-assembly-1.6.0-cdh5.10.2-hadoop2.6.0-cdh5.10.2.jar:na] at org.apache.spark.rpc.RpcTimeout$$anonfun$addMessageIfTimeout$1.applyOrElse(RpcTimeout.scala:59) ~[spark-assembly-1.6.0-cdh5.10.2-hadoop2.6.0-cdh5.10.2.jar:na] at scala.runtime.AbstractPartialFunction.apply(AbstractPartialFunction.scala:33) ~[spark-assembly-1.6.0-cdh5.10.2-hadoop2.6.0-cdh5.10.2.jar:na] at org.apache.spark.rpc.RpcTimeout.awaitResult(RpcTimeout.scala:76) ~[spark-assembly-1.6.0-cdh5.10.2-hadoop2.6.0-cdh5.10.2.jar:na] at org.apache.spark.rpc.RpcEndpointRef.askWithRetry(RpcEndpointRef.scala:101) ~[spark-assembly-1.6.0-cdh5.10.2-hadoop2.6.0-cdh5.10.2.jar:na] at org.apache.spark.executor.Executor.org$apache$spark$executor$Executor$$reportHeartBeat(Executor.scala:491) [spark-assembly-1.6.0-cdh5.10.2-hadoop2.6.0-cdh5.10.2.jar:na] at org.apache.spark.executor.Executor$$anon$1$$anonfun$run$1.apply$mcV$sp(Executor.scala:520) [spark-assembly-1.6.0-cdh5.10.2-hadoop2.6.0-cdh5.10.2.jar:na] at org.apache.spark.executor.Executor$$anon$1$$anonfun$run$1.apply(Executor.scala:520) [spark-assembly-1.6.0-cdh5.10.2-hadoop2.6.0-cdh5.10.2.jar:na] at org.apache.spark.executor.Executor$$anon$1$$anonfun$run$1.apply(Executor.scala:520) [spark-assembly-1.6.0-cdh5.10.2-hadoop2.6.0-cdh5.10.2.jar:na] at org.apache.spark.util.Utils$.logUncaughtExceptions(Utils.scala:1818) [spark-assembly-1.6.0-cdh5.10.2-hadoop2.6.0-cdh5.10.2.jar:na] at org.apache.spark.executor.Executor$$anon$1.run(Executor.scala:520) [spark-assembly-1.6.0-cdh5.10.2-hadoop2.6.0-cdh5.10.2.jar:na] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [na:1.8.0_121] at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) [na:1.8.0_121] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) [na:1.8.0_121] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) [na:1.8.0_121] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [na:1.8.0_121] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [na:1.8.0_121] at java.lang.Thread.run(Thread.java:745) [na:1.8.0_121] Caused by: java.util.concurrent.TimeoutException: Futures timed out after [10 seconds] at scala.concurrent.impl.Promise$DefaultPromise.ready(Promise.scala:219) ~[spark-assembly-1.6.0-cdh5.10.2-hadoop2.6.0-cdh5.10.2.jar:na] at scala.concurrent.impl.Promise$DefaultPromise.result(Promise.scala:223) ~[spark-assembly-1.6.0-cdh5.10.2-hadoop2.6.0-cdh5.10.2.jar:na] at scala.concurrent.Await$$anonfun$result$1.apply(package.scala:107) ~[spark-assembly-1.6.0-cdh5.10.2-hadoop2.6.0-cdh5.10.2.jar:na] at scala.concurrent.BlockContext$DefaultBlockContext$.blockOn(BlockContext.scala:53) ~[spark-assembly-1.6.0-cdh5.10.2-hadoop2.6.0-cdh5.10.2.jar:na] at scala.concurrent.Await$.result(package.scala:107) ~[spark-assembly-1.6.0-cdh5.10.2-hadoop2.6.0-cdh5.10.2.jar:na] at org.apache.spark.rpc.RpcTimeout.awaitResult(RpcTimeout.scala:75) ~[spark-assembly-1.6.0-cdh5.10.2-hadoop2.6.0-cdh5.10.2.jar:na] ... 14 common frames omitted 13:16:05.145 [shuffle-client-0] WARN o.a.s.n.c.TransportResponseHandler - Ignoring response for RPC 6032743701250525400 from c893ach.ecom.bigdata.int.thomsonreuters.com/10.214.4.161:33947 (81 bytes) since it is not outstanding 13:20:31.373 [zk-client-EventThread] WARN o.a.t.d.ZKDiscoveryService - ZK Session expired: c519xzf.ecom.bigdata.int.thomsonreuters.com:2181,c570ntw.ecom.bigdata.int.thomsonreuters.com:2181,c482btu.ecom.bigdata.int.thomsonreuters.com:2181/ecpprodcdap/discoverable [13:20:55] Topology snapshot [ver=64, servers=62, clients=0, CPUs=2064, heap=1900.0GB] [13:21:07] Topology snapshot [ver=65, servers=61, clients=0, CPUs=2064, heap=1800.0GB] 13:21:11.914 [Executor task launch worker-3] ERROR o.a.s.e.Executor - Exception in task 18.0 in stage 4.0 (TID 308) javax.cache.CacheException: class org.apache.ignite.IgniteInterruptedException: Node is stopping: WCAGridapplication_1506331241975_7951 at org.apache.ignite.internal.processors.cache.GridCacheUtils.convertToCacheException(GridCacheUtils.java:1440) ~[program.expanded.jar/:na] at org.apache.ignite.internal.processors.cache.IgniteCacheProxy.cacheException(IgniteCacheProxy.java:2183) ~[program.expanded.jar/:na] at org.apache.ignite.internal.processors.cache.IgniteCacheProxy.putAll(IgniteCacheProxy.java:1430) ~[program.expanded.jar/:na] at un.api.dataloader.IgniteDataLoader$$anonfun$loadCompanyData$13.apply(IgniteDataLoader.scala:207) ~[program.expanded.jar/:na] at un.api.dataloader.IgniteDataLoader$$anonfun$loadCompanyData$13.apply(IgniteDataLoader.scala:205) ~[program.expanded.jar/:na] at org.apache.spark.rdd.RDD$$anonfun$mapPartitions$1$$anonfun$apply$20.apply(RDD.scala:710) ~[spark-assembly-1.6.0-cdh5.10.2-hadoop2.6.0-cdh5.10.2.jar:na] at org.apache.spark.rdd.RDD$$anonfun$mapPartitions$1$$anonfun$apply$20.apply(RDD.scala:710) ~[spark-assembly-1.6.0-cdh5.10.2-hadoop2.6.0-cdh5.10.2.jar:na] at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38) ~[spark-assembly-1.6.0-cdh5.10.2-hadoop2.6.0-cdh5.10.2.jar:na] at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:306) ~[spark-assembly-1.6.0-cdh5.10.2-hadoop2.6.0-cdh5.10.2.jar:na] at org.apache.spark.rdd.RDD.iterator(RDD.scala:270) ~[spark-assembly-1.6.0-cdh5.10.2-hadoop2.6.0-cdh5.10.2.jar:na] at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:66) ~[spark-assembly-1.6.0-cdh5.10.2-hadoop2.6.0-cdh5.10.2.jar:na] at org.apache.spark.scheduler.Task.run(Task.scala:89) ~[spark-assembly-1.6.0-cdh5.10.2-hadoop2.6.0-cdh5.10.2.jar:na] at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:242) ~[spark-assembly-1.6.0-cdh5.10.2-hadoop2.6.0-cdh5.10.2.jar:na] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [na:1.8.0_121] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [na:1.8.0_121] at java.lang.Thread.run(Thread.java:745) [na:1.8.0_121] Caused by: org.apache.ignite.IgniteInterruptedException: Node is stopping: WCAGridapplication_1506331241975_7951 at org.apache.ignite.internal.util.IgniteUtils$3.apply(IgniteUtils.java:766) ~[program.expanded.jar/:na] at org.apache.ignite.internal.util.IgniteUtils$3.apply(IgniteUtils.java:764) ~[program.expanded.jar/:na] ... 16 common frames omitted [13:21:12] Topology snapshot [ver=66, servers=60, clients=0, CPUs=2064, heap=1800.0GB] [13:21:14] Topology snapshot [ver=67, servers=59, clients=0, CPUs=2016, heap=1800.0GB] [13:21:14] Topology snapshot [ver=68, servers=58, clients=0, CPUs=2016, heap=1700.0GB] [13:21:14] Topology snapshot [ver=69, servers=57, clients=0, CPUs=1968, heap=1700.0GB] [13:21:14] Topology snapshot [ver=70, servers=56, clients=0, CPUs=1968, heap=1700.0GB] [13:21:14] Topology snapshot [ver=71, servers=55, clients=0, CPUs=1968, heap=1700.0GB] [13:21:17] Topology snapshot [ver=72, servers=54, clients=0, CPUs=1968, heap=1600.0GB] [13:21:17] Topology snapshot [ver=73, servers=53, clients=0, CPUs=1968, heap=1600.0GB] [13:21:17] Topology snapshot [ver=74, servers=52, clients=0, CPUs=1968, heap=1600.0GB] [13:21:17] Topology snapshot [ver=75, servers=51, clients=0, CPUs=1968, heap=1500.0GB] [13:21:17] Topology snapshot [ver=76, servers=50, clients=0, CPUs=1968, heap=1500.0GB] [13:21:17] Topology snapshot [ver=77, servers=49, clients=0, CPUs=1968, heap=1500.0GB] [13:21:18] Topology snapshot [ver=78, servers=48, clients=0, CPUs=1968, heap=1400.0GB] [13:21:19] Topology snapshot [ver=79, servers=47, clients=0, CPUs=1920, heap=1400.0GB] [13:21:19] Topology snapshot [ver=80, servers=46, clients=0, CPUs=1872, heap=1400.0GB] [13:21:19] Topology snapshot [ver=81, servers=45, clients=0, CPUs=1824, heap=1400.0GB] [13:21:19] Topology snapshot [ver=82, servers=44, clients=0, CPUs=1776, heap=1300.0GB] [13:21:19] Topology snapshot [ver=83, servers=43, clients=0, CPUs=1776, heap=1300.0GB] [13:21:19] Topology snapshot [ver=84, servers=42, clients=0, CPUs=1728, heap=1300.0GB] [13:21:19] Topology snapshot [ver=85, servers=41, clients=0, CPUs=1680, heap=1200.0GB] [13:21:19] Topology snapshot [ver=86, servers=40, clients=0, CPUs=1632, heap=1200.0GB] [13:21:19] Topology snapshot [ver=87, servers=39, clients=0, CPUs=1632, heap=1200.0GB] [13:21:19] Topology snapshot [ver=88, servers=38, clients=0, CPUs=1584, heap=1100.0GB] [13:21:19] Topology snapshot [ver=89, servers=37, clients=0, CPUs=1536, heap=1100.0GB] [13:21:19] Topology snapshot [ver=90, servers=36, clients=0, CPUs=1488, heap=1100.0GB] [13:21:19] Topology snapshot [ver=91, servers=35, clients=0, CPUs=1440, heap=1100.0GB] [13:21:20] Topology snapshot [ver=92, servers=34, clients=0, CPUs=1440, heap=1000.0GB] [13:21:20] Topology snapshot [ver=93, servers=33, clients=0, CPUs=1440, heap=990.0GB] [13:21:20] Topology snapshot [ver=94, servers=32, clients=0, CPUs=1392, heap=960.0GB] [13:21:21] Topology snapshot [ver=95, servers=31, clients=0, CPUs=1344, heap=930.0GB] [13:21:22] Topology snapshot [ver=96, servers=30, clients=0, CPUs=1344, heap=900.0GB] [13:21:22] Topology snapshot [ver=97, servers=29, clients=0, CPUs=1296, heap=870.0GB] [13:21:22] Topology snapshot [ver=98, servers=28, clients=0, CPUs=1248, heap=840.0GB] [13:21:22] Topology snapshot [ver=99, servers=27, clients=0, CPUs=1200, heap=810.0GB] [13:21:22] Topology snapshot [ver=100, servers=26, clients=0, CPUs=1152, heap=780.0GB] [13:21:22] Topology snapshot [ver=101, servers=25, clients=0, CPUs=1104, heap=750.0GB] [13:21:22] Topology snapshot [ver=102, servers=24, clients=0, CPUs=1056, heap=720.0GB] [13:21:23] Topology snapshot [ver=103, servers=23, clients=0, CPUs=1008, heap=690.0GB] [13:21:23] Topology snapshot [ver=104, servers=22, clients=0, CPUs=960, heap=660.0GB] [13:21:23] Topology snapshot [ver=105, servers=21, clients=0, CPUs=912, heap=630.0GB] [13:21:23] Topology snapshot [ver=106, servers=20, clients=0, CPUs=864, heap=600.0GB] [13:21:23] Topology snapshot [ver=107, servers=19, clients=0, CPUs=816, heap=570.0GB] [13:21:23] Topology snapshot [ver=108, servers=18, clients=0, CPUs=768, heap=540.0GB] [13:21:23] Topology snapshot [ver=109, servers=17, clients=0, CPUs=720, heap=510.0GB] [13:21:23] Topology snapshot [ver=110, servers=16, clients=0, CPUs=672, heap=480.0GB] [13:21:23] Topology snapshot [ver=111, servers=15, clients=0, CPUs=624, heap=450.0GB] [13:21:23] Topology snapshot [ver=112, servers=14, clients=0, CPUs=576, heap=420.0GB] [13:21:23] Topology snapshot [ver=113, servers=13, clients=0, CPUs=528, heap=390.0GB] [13:21:25] Topology snapshot [ver=114, servers=12, clients=0, CPUs=480, heap=360.0GB] [13:21:25] Topology snapshot [ver=115, servers=11, clients=0, CPUs=432, heap=330.0GB] [13:21:25] Topology snapshot [ver=116, servers=10, clients=0, CPUs=432, heap=300.0GB] [13:21:25] Topology snapshot [ver=117, servers=9, clients=0, CPUs=384, heap=270.0GB] [13:21:25] Topology snapshot [ver=118, servers=8, clients=0, CPUs=336, heap=240.0GB] [13:21:25] Topology snapshot [ver=119, servers=7, clients=0, CPUs=288, heap=210.0GB] [13:21:25] Topology snapshot [ver=120, servers=6, clients=0, CPUs=288, heap=180.0GB] [13:21:25] Topology snapshot [ver=121, servers=5, clients=0, CPUs=240, heap=150.0GB] [13:21:25] Topology snapshot [ver=122, servers=4, clients=0, CPUs=192, heap=120.0GB] [13:21:25] Topology snapshot [ver=123, servers=3, clients=0, CPUs=144, heap=90.0GB] [13:21:25] Topology snapshot [ver=124, servers=2, clients=0, CPUs=96, heap=60.0GB] [13:21:26] Topology snapshot [ver=125, servers=1, clients=0, CPUs=48, heap=30.0GB] [13:21:36] Ignite node stopped OK [name=WCAGridapplication_1506331241975_7951, uptime=00:08:09:192] [13:21:45] __________ ________________ [13:21:45] / _/ ___/ |/ / _/_ __/ __/ [13:21:45] _/ // (7 7 // / / / / _/ [13:21:45] /___/\___/_/|_/___/ /_/ /___/ [13:21:45] [13:21:45] ver. 1.8.0#20161205-sha1:9ca40dbe [13:21:45] 2016 Copyright(C) Apache Software Foundation [13:21:45] [13:21:45] Ignite documentation: http://ignite.apache.org [13:21:45] [13:21:45] Quiet mode. [13:21:45] ^-- To see **FULL** console log here add -DIGNITE_QUIET=false or "-v" to ignite.{sh|bat} [13:21:45] [13:21:45] OS: Linux 3.10.0-514.16.1.el7.x86_64 amd64 [13:21:45] VM information: Java(TM) SE Runtime Environment 1.8.0_121-b13 Oracle Corporation Java HotSpot(TM) 64-Bit Server VM 25.121-b13 [13:21:46] Configured plugins: [13:21:46] ^-- None [13:21:46] [13:21:47] Security status [authentication=off, tls/ssl=off] 13:30:13.022 [driver-heartbeater] WARN o.a.s.r.n.NettyRpcEndpointRef - Error sending message [message = Heartbeat(29,[Lscala.Tuple2;@151ddb70,BlockManagerId(29, c893ach.ecom.bigdata.int.thomsonreuters.com, 35301))] in 1 attempts org.apache.spark.rpc.RpcTimeoutException: Futures timed out after [10 seconds]. This timeout is controlled by spark.executor.heartbeatInterval at org.apache.spark.rpc.RpcTimeout.org$apache$spark$rpc$RpcTimeout$$createRpcTimeoutException(RpcTimeout.scala:48) ~[spark-assembly-1.6.0-cdh5.10.2-hadoop2.6.0-cdh5.10.2.jar:na] at org.apache.spark.rpc.RpcTimeout$$anonfun$addMessageIfTimeout$1.applyOrElse(RpcTimeout.scala:63) ~[spark-assembly-1.6.0-cdh5.10.2-hadoop2.6.0-cdh5.10.2.jar:na] at org.apache.spark.rpc.RpcTimeout$$anonfun$addMessageIfTimeout$1.applyOrElse(RpcTimeout.scala:59) ~[spark-assembly-1.6.0-cdh5.10.2-hadoop2.6.0-cdh5.10.2.jar:na] at scala.runtime.AbstractPartialFunction.apply(AbstractPartialFunction.scala:33) ~[spark-assembly-1.6.0-cdh5.10.2-hadoop2.6.0-cdh5.10.2.jar:na] at org.apache.spark.rpc.RpcTimeout.awaitResult(RpcTimeout.scala:76) ~[spark-assembly-1.6.0-cdh5.10.2-hadoop2.6.0-cdh5.10.2.jar:na] at org.apache.spark.rpc.RpcEndpointRef.askWithRetry(RpcEndpointRef.scala:101) ~[spark-assembly-1.6.0-cdh5.10.2-hadoop2.6.0-cdh5.10.2.jar:na] at org.apache.spark.executor.Executor.org$apache$spark$executor$Executor$$reportHeartBeat(Executor.scala:491) [spark-assembly-1.6.0-cdh5.10.2-hadoop2.6.0-cdh5.10.2.jar:na] at org.apache.spark.executor.Executor$$anon$1$$anonfun$run$1.apply$mcV$sp(Executor.scala:520) [spark-assembly-1.6.0-cdh5.10.2-hadoop2.6.0-cdh5.10.2.jar:na] at org.apache.spark.executor.Executor$$anon$1$$anonfun$run$1.apply(Executor.scala:520) [spark-assembly-1.6.0-cdh5.10.2-hadoop2.6.0-cdh5.10.2.jar:na] at org.apache.spark.executor.Executor$$anon$1$$anonfun$run$1.apply(Executor.scala:520) [spark-assembly-1.6.0-cdh5.10.2-hadoop2.6.0-cdh5.10.2.jar:na] at org.apache.spark.util.Utils$.logUncaughtExceptions(Utils.scala:1818) [spark-assembly-1.6.0-cdh5.10.2-hadoop2.6.0-cdh5.10.2.jar:na] at org.apache.spark.executor.Executor$$anon$1.run(Executor.scala:520) [spark-assembly-1.6.0-cdh5.10.2-hadoop2.6.0-cdh5.10.2.jar:na] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [na:1.8.0_121] at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) [na:1.8.0_121] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) [na:1.8.0_121] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) [na:1.8.0_121] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [na:1.8.0_121] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [na:1.8.0_121] at java.lang.Thread.run(Thread.java:745) [na:1.8.0_121] Caused by: java.util.concurrent.TimeoutException: Futures timed out after [10 seconds] at scala.concurrent.impl.Promise$DefaultPromise.ready(Promise.scala:219) ~[spark-assembly-1.6.0-cdh5.10.2-hadoop2.6.0-cdh5.10.2.jar:na] at scala.concurrent.impl.Promise$DefaultPromise.result(Promise.scala:223) ~[spark-assembly-1.6.0-cdh5.10.2-hadoop2.6.0-cdh5.10.2.jar:na] at scala.concurrent.Await$$anonfun$result$1.apply(package.scala:107) ~[spark-assembly-1.6.0-cdh5.10.2-hadoop2.6.0-cdh5.10.2.jar:na] at scala.concurrent.BlockContext$DefaultBlockContext$.blockOn(BlockContext.scala:53) ~[spark-assembly-1.6.0-cdh5.10.2-hadoop2.6.0-cdh5.10.2.jar:na] at scala.concurrent.Await$.result(package.scala:107) ~[spark-assembly-1.6.0-cdh5.10.2-hadoop2.6.0-cdh5.10.2.jar:na] at org.apache.spark.rpc.RpcTimeout.awaitResult(RpcTimeout.scala:75) ~[spark-assembly-1.6.0-cdh5.10.2-hadoop2.6.0-cdh5.10.2.jar:na] ... 14 common frames omitted 13:30:16.071 [shuffle-client-0] WARN o.a.s.n.c.TransportResponseHandler - Ignoring response for RPC 8904876257727937012 from c893ach.ecom.bigdata.int.thomsonreuters.com/10.214.4.161:33947 (81 bytes) since it is not outstanding 13:43:03.022 [driver-heartbeater] WARN o.a.s.r.n.NettyRpcEndpointRef - Error sending message [message = Heartbeat(29,[Lscala.Tuple2;@5c8cc3b4,BlockManagerId(29, c893ach.ecom.bigdata.int.thomsonreuters.com, 35301))] in 1 attempts org.apache.spark.rpc.RpcTimeoutException: Futures timed out after [10 seconds]. This timeout is controlled by spark.executor.heartbeatInterval at org.apache.spark.rpc.RpcTimeout.org$apache$spark$rpc$RpcTimeout$$createRpcTimeoutException(RpcTimeout.scala:48) ~[spark-assembly-1.6.0-cdh5.10.2-hadoop2.6.0-cdh5.10.2.jar:na] at org.apache.spark.rpc.RpcTimeout$$anonfun$addMessageIfTimeout$1.applyOrElse(RpcTimeout.scala:63) ~[spark-assembly-1.6.0-cdh5.10.2-hadoop2.6.0-cdh5.10.2.jar:na] at org.apache.spark.rpc.RpcTimeout$$anonfun$addMessageIfTimeout$1.applyOrElse(RpcTimeout.scala:59) ~[spark-assembly-1.6.0-cdh5.10.2-hadoop2.6.0-cdh5.10.2.jar:na] at scala.runtime.AbstractPartialFunction.apply(AbstractPartialFunction.scala:33) ~[spark-assembly-1.6.0-cdh5.10.2-hadoop2.6.0-cdh5.10.2.jar:na] at org.apache.spark.rpc.RpcTimeout.awaitResult(RpcTimeout.scala:76) ~[spark-assembly-1.6.0-cdh5.10.2-hadoop2.6.0-cdh5.10.2.jar:na] at org.apache.spark.rpc.RpcEndpointRef.askWithRetry(RpcEndpointRef.scala:101) ~[spark-assembly-1.6.0-cdh5.10.2-hadoop2.6.0-cdh5.10.2.jar:na] at org.apache.spark.executor.Executor.org$apache$spark$executor$Executor$$reportHeartBeat(Executor.scala:491) [spark-assembly-1.6.0-cdh5.10.2-hadoop2.6.0-cdh5.10.2.jar:na] at org.apache.spark.executor.Executor$$anon$1$$anonfun$run$1.apply$mcV$sp(Executor.scala:520) [spark-assembly-1.6.0-cdh5.10.2-hadoop2.6.0-cdh5.10.2.jar:na] at org.apache.spark.executor.Executor$$anon$1$$anonfun$run$1.apply(Executor.scala:520) [spark-assembly-1.6.0-cdh5.10.2-hadoop2.6.0-cdh5.10.2.jar:na] at org.apache.spark.executor.Executor$$anon$1$$anonfun$run$1.apply(Executor.scala:520) [spark-assembly-1.6.0-cdh5.10.2-hadoop2.6.0-cdh5.10.2.jar:na] at org.apache.spark.util.Utils$.logUncaughtExceptions(Utils.scala:1818) [spark-assembly-1.6.0-cdh5.10.2-hadoop2.6.0-cdh5.10.2.jar:na] at org.apache.spark.executor.Executor$$anon$1.run(Executor.scala:520) [spark-assembly-1.6.0-cdh5.10.2-hadoop2.6.0-cdh5.10.2.jar:na] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [na:1.8.0_121] at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) [na:1.8.0_121] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) [na:1.8.0_121] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) [na:1.8.0_121] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [na:1.8.0_121] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [na:1.8.0_121] at java.lang.Thread.run(Thread.java:745) [na:1.8.0_121] Caused by: java.util.concurrent.TimeoutException: Futures timed out after [10 seconds] at scala.concurrent.impl.Promise$DefaultPromise.ready(Promise.scala:219) ~[spark-assembly-1.6.0-cdh5.10.2-hadoop2.6.0-cdh5.10.2.jar:na] at scala.concurrent.impl.Promise$DefaultPromise.result(Promise.scala:223) ~[spark-assembly-1.6.0-cdh5.10.2-hadoop2.6.0-cdh5.10.2.jar:na] at scala.concurrent.Await$$anonfun$result$1.apply(package.scala:107) ~[spark-assembly-1.6.0-cdh5.10.2-hadoop2.6.0-cdh5.10.2.jar:na] at scala.concurrent.BlockContext$DefaultBlockContext$.blockOn(BlockContext.scala:53) ~[spark-assembly-1.6.0-cdh5.10.2-hadoop2.6.0-cdh5.10.2.jar:na] at scala.concurrent.Await$.result(package.scala:107) ~[spark-assembly-1.6.0-cdh5.10.2-hadoop2.6.0-cdh5.10.2.jar:na] at org.apache.spark.rpc.RpcTimeout.awaitResult(RpcTimeout.scala:75) ~[spark-assembly-1.6.0-cdh5.10.2-hadoop2.6.0-cdh5.10.2.jar:na] ... 14 common frames omitted 13:43:08.960 [shuffle-client-0] WARN o.a.s.n.c.TransportResponseHandler - Ignoring response for RPC 8645866707163739584 from c893ach.ecom.bigdata.int.thomsonreuters.com/10.214.4.161:33947 (81 bytes) since it is not outstanding 14:13:03.023 [driver-heartbeater] WARN o.a.s.r.n.NettyRpcEndpointRef - Error sending message [message = Heartbeat(29,[Lscala.Tuple2;@7d0e3c98,BlockManagerId(29, c893ach.ecom.bigdata.int.thomsonreuters.com, 35301))] in 1 attempts org.apache.spark.rpc.RpcTimeoutException: Futures timed out after [10 seconds]. This timeout is controlled by spark.executor.heartbeatInterval at org.apache.spark.rpc.RpcTimeout.org$apache$spark$rpc$RpcTimeout$$createRpcTimeoutException(RpcTimeout.scala:48) ~[spark-assembly-1.6.0-cdh5.10.2-hadoop2.6.0-cdh5.10.2.jar:na] at org.apache.spark.rpc.RpcTimeout$$anonfun$addMessageIfTimeout$1.applyOrElse(RpcTimeout.scala:63) ~[spark-assembly-1.6.0-cdh5.10.2-hadoop2.6.0-cdh5.10.2.jar:na] at org.apache.spark.rpc.RpcTimeout$$anonfun$addMessageIfTimeout$1.applyOrElse(RpcTimeout.scala:59) ~[spark-assembly-1.6.0-cdh5.10.2-hadoop2.6.0-cdh5.10.2.jar:na] at scala.runtime.AbstractPartialFunction.apply(AbstractPartialFunction.scala:33) ~[spark-assembly-1.6.0-cdh5.10.2-hadoop2.6.0-cdh5.10.2.jar:na] at org.apache.spark.rpc.RpcTimeout.awaitResult(RpcTimeout.scala:76) ~[spark-assembly-1.6.0-cdh5.10.2-hadoop2.6.0-cdh5.10.2.jar:na] at org.apache.spark.rpc.RpcEndpointRef.askWithRetry(RpcEndpointRef.scala:101) ~[spark-assembly-1.6.0-cdh5.10.2-hadoop2.6.0-cdh5.10.2.jar:na] at org.apache.spark.executor.Executor.org$apache$spark$executor$Executor$$reportHeartBeat(Executor.scala:491) [spark-assembly-1.6.0-cdh5.10.2-hadoop2.6.0-cdh5.10.2.jar:na] at org.apache.spark.executor.Executor$$anon$1$$anonfun$run$1.apply$mcV$sp(Executor.scala:520) [spark-assembly-1.6.0-cdh5.10.2-hadoop2.6.0-cdh5.10.2.jar:na] at org.apache.spark.executor.Executor$$anon$1$$anonfun$run$1.apply(Executor.scala:520) [spark-assembly-1.6.0-cdh5.10.2-hadoop2.6.0-cdh5.10.2.jar:na] at org.apache.spark.executor.Executor$$anon$1$$anonfun$run$1.apply(Executor.scala:520) [spark-assembly-1.6.0-cdh5.10.2-hadoop2.6.0-cdh5.10.2.jar:na] at org.apache.spark.util.Utils$.logUncaughtExceptions(Utils.scala:1818) [spark-assembly-1.6.0-cdh5.10.2-hadoop2.6.0-cdh5.10.2.jar:na] at org.apache.spark.executor.Executor$$anon$1.run(Executor.scala:520) [spark-assembly-1.6.0-cdh5.10.2-hadoop2.6.0-cdh5.10.2.jar:na] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [na:1.8.0_121] at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) [na:1.8.0_121] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) [na:1.8.0_121] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) [na:1.8.0_121] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [na:1.8.0_121] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [na:1.8.0_121] at java.lang.Thread.run(Thread.java:745) [na:1.8.0_121] Caused by: java.util.concurrent.TimeoutException: Futures timed out after [10 seconds] at scala.concurrent.impl.Promise$DefaultPromise.ready(Promise.scala:219) ~[spark-assembly-1.6.0-cdh5.10.2-hadoop2.6.0-cdh5.10.2.jar:na] at scala.concurrent.impl.Promise$DefaultPromise.result(Promise.scala:223) ~[spark-assembly-1.6.0-cdh5.10.2-hadoop2.6.0-cdh5.10.2.jar:na] at scala.concurrent.Await$$anonfun$result$1.apply(package.scala:107) ~[spark-assembly-1.6.0-cdh5.10.2-hadoop2.6.0-cdh5.10.2.jar:na] at scala.concurrent.BlockContext$DefaultBlockContext$.blockOn(BlockContext.scala:53) ~[spark-assembly-1.6.0-cdh5.10.2-hadoop2.6.0-cdh5.10.2.jar:na] at scala.concurrent.Await$.result(package.scala:107) ~[spark-assembly-1.6.0-cdh5.10.2-hadoop2.6.0-cdh5.10.2.jar:na] at org.apache.spark.rpc.RpcTimeout.awaitResult(RpcTimeout.scala:75) ~[spark-assembly-1.6.0-cdh5.10.2-hadoop2.6.0-cdh5.10.2.jar:na] ... 14 common frames omitted 14:13:05.100 [shuffle-client-0] WARN o.a.s.n.c.TransportResponseHandler - Ignoring response for RPC 6998673354740181981 from c893ach.ecom.bigdata.int.thomsonreuters.com/10.214.4.161:33947 (81 bytes) since it is not outstanding 14:46:56.835 [zk-client-EventThread] WARN o.a.t.d.ZKDiscoveryService - ZK Session expired: c519xzf.ecom.bigdata.int.thomsonreuters.com:2181,c570ntw.ecom.bigdata.int.thomsonreuters.com:2181,c482btu.ecom.bigdata.int.thomsonreuters.com:2181/ecpprodcdap/discoverable [14:47:20] Topology snapshot [ver=66, servers=62, clients=0, CPUs=2064, heap=1900.0GB] [14:47:27] Topology snapshot [ver=67, servers=61, clients=0, CPUs=2064, heap=1800.0GB] [14:47:27] Topology snapshot [ver=68, servers=60, clients=0, CPUs=2064, heap=1800.0GB] [14:47:27] Topology snapshot [ver=69, servers=59, clients=0, CPUs=2016, heap=1800.0GB] [14:47:27] Topology snapshot [ver=70, servers=58, clients=0, CPUs=2016, heap=1700.0GB] [14:47:27] Topology snapshot [ver=71, servers=57, clients=0, CPUs=1968, heap=1700.0GB] [14:47:27] Topology snapshot [ver=72, servers=56, clients=0, CPUs=1968, heap=1700.0GB] [14:47:27] Topology snapshot [ver=73, servers=55, clients=0, CPUs=1968, heap=1700.0GB] [14:47:27] Topology snapshot [ver=74, servers=54, clients=0, CPUs=1968, heap=1600.0GB] [14:47:27] Topology snapshot [ver=75, servers=53, clients=0, CPUs=1968, heap=1600.0GB] [14:47:27] Topology snapshot [ver=76, servers=52, clients=0, CPUs=1968, heap=1600.0GB] [14:47:27] Topology snapshot [ver=77, servers=51, clients=0, CPUs=1968, heap=1500.0GB] [14:47:27] Topology snapshot [ver=78, servers=50, clients=0, CPUs=1968, heap=1500.0GB] [14:47:27] Topology snapshot [ver=79, servers=49, clients=0, CPUs=1968, heap=1500.0GB] [14:47:27] Topology snapshot [ver=80, servers=48, clients=0, CPUs=1968, heap=1400.0GB] [14:47:27] Topology snapshot [ver=81, servers=47, clients=0, CPUs=1920, heap=1400.0GB] [14:47:27] Topology snapshot [ver=82, servers=46, clients=0, CPUs=1872, heap=1400.0GB] [14:47:27] Topology snapshot [ver=83, servers=45, clients=0, CPUs=1824, heap=1400.0GB] [14:47:27] Topology snapshot [ver=84, servers=44, clients=0, CPUs=1776, heap=1300.0GB] [14:47:27] Topology snapshot [ver=85, servers=43, clients=0, CPUs=1776, heap=1300.0GB] [14:47:27] Topology snapshot [ver=86, servers=42, clients=0, CPUs=1728, heap=1300.0GB] [14:47:27] Topology snapshot [ver=87, servers=41, clients=0, CPUs=1680, heap=1200.0GB] [14:47:27] Topology snapshot [ver=88, servers=40, clients=0, CPUs=1632, heap=1200.0GB] [14:47:27] Topology snapshot [ver=89, servers=39, clients=0, CPUs=1632, heap=1200.0GB] [14:47:27] Topology snapshot [ver=90, servers=38, clients=0, CPUs=1584, heap=1100.0GB] [14:47:27] Topology snapshot [ver=91, servers=37, clients=0, CPUs=1536, heap=1100.0GB] [14:47:27] Topology snapshot [ver=92, servers=36, clients=0, CPUs=1488, heap=1100.0GB] [14:47:27] Topology snapshot [ver=93, servers=35, clients=0, CPUs=1440, heap=1100.0GB] [14:47:27] Topology snapshot [ver=94, servers=34, clients=0, CPUs=1440, heap=1000.0GB] [14:47:27] Topology snapshot [ver=95, servers=33, clients=0, CPUs=1440, heap=990.0GB] [14:47:27] Topology snapshot [ver=96, servers=32, clients=0, CPUs=1392, heap=960.0GB] [14:47:27] Topology snapshot [ver=97, servers=31, clients=0, CPUs=1344, heap=930.0GB] [14:47:27] Topology snapshot [ver=98, servers=30, clients=0, CPUs=1344, heap=900.0GB] [14:47:27] Topology snapshot [ver=99, servers=29, clients=0, CPUs=1296, heap=870.0GB] [14:47:27] Topology snapshot [ver=100, servers=28, clients=0, CPUs=1248, heap=840.0GB] [14:47:27] Topology snapshot [ver=101, servers=27, clients=0, CPUs=1200, heap=810.0GB] [14:47:27] Topology snapshot [ver=102, servers=26, clients=0, CPUs=1152, heap=780.0GB] [14:47:27] Topology snapshot [ver=103, servers=25, clients=0, CPUs=1104, heap=750.0GB] [14:47:27] Topology snapshot [ver=104, servers=24, clients=0, CPUs=1056, heap=720.0GB] [14:47:27] Topology snapshot [ver=105, servers=23, clients=0, CPUs=1008, heap=690.0GB] [14:47:27] Topology snapshot [ver=106, servers=22, clients=0, CPUs=960, heap=660.0GB] [14:47:27] Topology snapshot [ver=107, servers=21, clients=0, CPUs=912, heap=630.0GB] [14:47:27] Topology snapshot [ver=108, servers=20, clients=0, CPUs=864, heap=600.0GB] [14:47:27] Topology snapshot [ver=109, servers=19, clients=0, CPUs=816, heap=570.0GB] [14:47:27] Topology snapshot [ver=110, servers=18, clients=0, CPUs=768, heap=540.0GB] [14:47:27] Topology snapshot [ver=111, servers=17, clients=0, CPUs=720, heap=510.0GB] [14:47:27] Topology snapshot [ver=112, servers=16, clients=0, CPUs=672, heap=480.0GB] [14:47:27] Topology snapshot [ver=113, servers=15, clients=0, CPUs=624, heap=450.0GB] [14:47:27] Topology snapshot [ver=114, servers=14, clients=0, CPUs=576, heap=420.0GB] [14:47:27] Topology snapshot [ver=115, servers=13, clients=0, CPUs=528, heap=390.0GB] [14:47:27] Topology snapshot [ver=116, servers=12, clients=0, CPUs=480, heap=360.0GB] [14:47:27] Topology snapshot [ver=117, servers=11, clients=0, CPUs=432, heap=330.0GB] [14:47:27] Topology snapshot [ver=118, servers=10, clients=0, CPUs=432, heap=300.0GB] [14:47:27] Topology snapshot [ver=119, servers=9, clients=0, CPUs=384, heap=270.0GB] [14:47:27] Topology snapshot [ver=120, servers=8, clients=0, CPUs=336, heap=240.0GB] [14:47:27] Topology snapshot [ver=121, servers=7, clients=0, CPUs=288, heap=210.0GB] [14:47:27] Topology snapshot [ver=122, servers=6, clients=0, CPUs=288, heap=180.0GB] [14:47:28] Topology snapshot [ver=123, servers=5, clients=0, CPUs=240, heap=150.0GB] [14:47:28] Topology snapshot [ver=124, servers=4, clients=0, CPUs=192, heap=120.0GB] [14:47:28] Topology snapshot [ver=125, servers=3, clients=0, CPUs=144, heap=90.0GB] [14:47:28] Topology snapshot [ver=126, servers=2, clients=0, CPUs=96, heap=60.0GB] [14:47:28] Topology snapshot [ver=127, servers=1, clients=0, CPUs=48, heap=30.0GB] 14:50:23.042 [zk-client-EventThread] WARN o.a.t.d.ZKDiscoveryService - ZK Session expired: c519xzf.ecom.bigdata.int.thomsonreuters.com:2181,c570ntw.ecom.bigdata.int.thomsonreuters.com:2181,c482btu.ecom.bigdata.int.thomsonreuters.com:2181/ecpprodcdap/discoverable 14:57:51.026 [zk-client-EventThread] WARN o.a.t.d.ZKDiscoveryService - ZK Session expired: c519xzf.ecom.bigdata.int.thomsonreuters.com:2181,c570ntw.ecom.bigdata.int.thomsonreuters.com:2181,c482btu.ecom.bigdata.int.thomsonreuters.com:2181/ecpprodcdap/discoverable 16:43:03.022 [driver-heartbeater] WARN o.a.s.r.n.NettyRpcEndpointRef - Error sending message [message = Heartbeat(29,[Lscala.Tuple2;@221555a,BlockManagerId(29, c893ach.ecom.bigdata.int.thomsonreuters.com, 35301))] in 1 attempts org.apache.spark.rpc.RpcTimeoutException: Futures timed out after [10 seconds]. This timeout is controlled by spark.executor.heartbeatInterval at org.apache.spark.rpc.RpcTimeout.org$apache$spark$rpc$RpcTimeout$$createRpcTimeoutException(RpcTimeout.scala:48) ~[spark-assembly-1.6.0-cdh5.10.2-hadoop2.6.0-cdh5.10.2.jar:na] at org.apache.spark.rpc.RpcTimeout$$anonfun$addMessageIfTimeout$1.applyOrElse(RpcTimeout.scala:63) ~[spark-assembly-1.6.0-cdh5.10.2-hadoop2.6.0-cdh5.10.2.jar:na] at org.apache.spark.rpc.RpcTimeout$$anonfun$addMessageIfTimeout$1.applyOrElse(RpcTimeout.scala:59) ~[spark-assembly-1.6.0-cdh5.10.2-hadoop2.6.0-cdh5.10.2.jar:na] at scala.runtime.AbstractPartialFunction.apply(AbstractPartialFunction.scala:33) ~[spark-assembly-1.6.0-cdh5.10.2-hadoop2.6.0-cdh5.10.2.jar:na] at org.apache.spark.rpc.RpcTimeout.awaitResult(RpcTimeout.scala:76) ~[spark-assembly-1.6.0-cdh5.10.2-hadoop2.6.0-cdh5.10.2.jar:na] at org.apache.spark.rpc.RpcEndpointRef.askWithRetry(RpcEndpointRef.scala:101) ~[spark-assembly-1.6.0-cdh5.10.2-hadoop2.6.0-cdh5.10.2.jar:na] at org.apache.spark.executor.Executor.org$apache$spark$executor$Executor$$reportHeartBeat(Executor.scala:491) [spark-assembly-1.6.0-cdh5.10.2-hadoop2.6.0-cdh5.10.2.jar:na] at org.apache.spark.executor.Executor$$anon$1$$anonfun$run$1.apply$mcV$sp(Executor.scala:520) [spark-assembly-1.6.0-cdh5.10.2-hadoop2.6.0-cdh5.10.2.jar:na] at org.apache.spark.executor.Executor$$anon$1$$anonfun$run$1.apply(Executor.scala:520) [spark-assembly-1.6.0-cdh5.10.2-hadoop2.6.0-cdh5.10.2.jar:na] at org.apache.spark.executor.Executor$$anon$1$$anonfun$run$1.apply(Executor.scala:520) [spark-assembly-1.6.0-cdh5.10.2-hadoop2.6.0-cdh5.10.2.jar:na] at org.apache.spark.util.Utils$.logUncaughtExceptions(Utils.scala:1818) [spark-assembly-1.6.0-cdh5.10.2-hadoop2.6.0-cdh5.10.2.jar:na] at org.apache.spark.executor.Executor$$anon$1.run(Executor.scala:520) [spark-assembly-1.6.0-cdh5.10.2-hadoop2.6.0-cdh5.10.2.jar:na] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [na:1.8.0_121] at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) [na:1.8.0_121] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) [na:1.8.0_121] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) [na:1.8.0_121] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [na:1.8.0_121] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [na:1.8.0_121] at java.lang.Thread.run(Thread.java:745) [na:1.8.0_121] Caused by: java.util.concurrent.TimeoutException: Futures timed out after [10 seconds] at scala.concurrent.impl.Promise$DefaultPromise.ready(Promise.scala:219) ~[spark-assembly-1.6.0-cdh5.10.2-hadoop2.6.0-cdh5.10.2.jar:na] at scala.concurrent.impl.Promise$DefaultPromise.result(Promise.scala:223) ~[spark-assembly-1.6.0-cdh5.10.2-hadoop2.6.0-cdh5.10.2.jar:na] at scala.concurrent.Await$$anonfun$result$1.apply(package.scala:107) ~[spark-assembly-1.6.0-cdh5.10.2-hadoop2.6.0-cdh5.10.2.jar:na] at scala.concurrent.BlockContext$DefaultBlockContext$.blockOn(BlockContext.scala:53) ~[spark-assembly-1.6.0-cdh5.10.2-hadoop2.6.0-cdh5.10.2.jar:na] at scala.concurrent.Await$.result(package.scala:107) ~[spark-assembly-1.6.0-cdh5.10.2-hadoop2.6.0-cdh5.10.2.jar:na] at org.apache.spark.rpc.RpcTimeout.awaitResult(RpcTimeout.scala:75) ~[spark-assembly-1.6.0-cdh5.10.2-hadoop2.6.0-cdh5.10.2.jar:na] ... 14 common frames omitted 16:43:03.445 [shuffle-client-0] WARN o.a.s.n.c.TransportResponseHandler - Ignoring response for RPC 8335836454112163975 from c893ach.ecom.bigdata.int.thomsonreuters.com/10.214.4.161:33947 (81 bytes) since it is not outstanding 18:11:17.912 [zk-client-EventThread] WARN o.a.t.d.ZKDiscoveryService - ZK Session expired: c519xzf.ecom.bigdata.int.thomsonreuters.com:2181,c570ntw.ecom.bigdata.int.thomsonreuters.com:2181,c482btu.ecom.bigdata.int.thomsonreuters.com:2181/ecpprodcdap/discoverable 18:12:02.467 [zk-client-EventThread] WARN o.a.t.d.ZKDiscoveryService - ZK Session expired: c519xzf.ecom.bigdata.int.thomsonreuters.com:2181,c570ntw.ecom.bigdata.int.thomsonreuters.com:2181,c482btu.ecom.bigdata.int.thomsonreuters.com:2181/ecpprodcdap/discoverable Stderr log SLF4J: Class path contains multiple SLF4J bindings. SLF4J: Found binding in [jar:file:/data/9/yarn/nm/usercache/bigdata-app-ecplegalanalytics-svc/filecache/1480/cdap-spark.jar/lib/ch.qos.logback.logback-classic-1.0.9.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: Found binding in [jar:file:/hadoop/cloudera/parcels/CDH-5.10.2-1.cdh5.10.2.p0.5/jars/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation. SLF4J: Actual binding is of type [ch.qos.logback.classic.util.ContextSelectorStaticBinder] SLF4J: Class path contains multiple SLF4J bindings. SLF4J: Found binding in [jar:file:/data/9/yarn/nm/usercache/bigdata-app-ecplegalanalytics-svc/filecache/1480/cdap-spark.jar/lib/ch.qos.logback.logback-classic-1.0.9.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: Found binding in [jar:file:/hadoop/cloudera/parcels/CDH-5.10.2-1.cdh5.10.2.p0.5/jars/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation. SLF4J: Actual binding is of type [ch.qos.logback.classic.util.ContextSelectorStaticBinder] 17/11/16 13:12:58 INFO utils.VerifiableProperties: Verifying properties 17/11/16 13:12:58 INFO utils.VerifiableProperties: Property key.serializer.class is overridden to kafka.serializer.StringEncoder 17/11/16 13:12:58 WARN utils.VerifiableProperties: Property log.publish.num.partitions is not valid 17/11/16 13:12:58 INFO utils.VerifiableProperties: Property metadata.broker.list is overridden to kafka-cpsprodeage.int.thomsonreuters.com:9092 17/11/16 13:12:58 INFO utils.VerifiableProperties: Property partitioner.class is overridden to co.cask.cdap.logging.appender.kafka.StringPartitioner 17/11/16 13:12:58 INFO utils.VerifiableProperties: Property producer.type is overridden to sync 17/11/16 13:12:58 INFO utils.VerifiableProperties: Property queue.buffering.max.ms is overridden to 1000 17/11/16 13:12:58 INFO utils.VerifiableProperties: Property request.required.acks is overridden to 1 17/11/16 13:12:58 INFO utils.VerifiableProperties: Property serializer.class is overridden to kafka.serializer.DefaultEncoder 17/11/16 13:13:26 INFO dataloader.IgniteDataLoader: Starting the Ignite node on - 10.214.4.161 17/11/16 13:13:26 INFO dataloader.IgniteDataLoader: Starting the Ignite node on - 10.214.4.161 17/11/16 13:13:26 INFO dataloader.IgniteDataLoader: Starting the Ignite node on - 10.214.4.161 17/11/16 13:13:26 INFO dataloader.IgniteDataLoader: Starting the Ignite node on - 10.214.4.161 17/11/16 13:13:26 INFO dataloader.IgniteDataLoader: Starting the Ignite node on - 10.214.4.161 17/11/16 13:13:26 INFO client.ClientUtils$: Fetching metadata from broker id:0,host:kafka-cpsprodeage.int.thomsonreuters.com,port:9092 with correlation id 0 for 1 topic(s) Set(ecpprodcdap.logs.user-v2) 17/11/16 13:13:26 INFO producer.SyncProducer: Connected to kafka-cpsprodeage.int.thomsonreuters.com:9092 for producing 17/11/16 13:13:26 INFO producer.SyncProducer: Disconnecting from kafka-cpsprodeage.int.thomsonreuters.com:9092 17/11/16 13:13:26 INFO producer.SyncProducer: Connected to c307senkecppr.int.thomsonreuters.com:9092 for producing Console logging handler is not configured. 17/11/16 13:13:27 INFO dataloader.IgniteDataLoader: Started the Ignite node on - 10.214.4.161 17/11/16 13:13:27 INFO dataloader.IgniteDataLoader: Started the Ignite node on - 10.214.4.161 17/11/16 13:13:27 INFO dataloader.IgniteDataLoader: Started the Ignite node on - 10.214.4.161 17/11/16 13:13:27 INFO dataloader.IgniteDataLoader: Started the Ignite node on - 10.214.4.161 17/11/16 13:13:27 INFO dataloader.IgniteDataLoader: Started the Ignite node on - 10.214.4.161 17/11/16 13:13:27 INFO dataloader.IgniteDataLoader: Starting the Ignite node on - 10.214.4.161 17/11/16 13:13:27 INFO dataloader.IgniteDataLoader: Starting the Ignite node on - 10.214.4.161 17/11/16 13:13:27 INFO dataloader.IgniteDataLoader: Started the Ignite node on - 10.214.4.161 17/11/16 13:13:27 INFO dataloader.IgniteDataLoader: Starting the Ignite node on - 10.214.4.161 17/11/16 13:13:27 INFO dataloader.IgniteDataLoader: Started the Ignite node on - 10.214.4.161 17/11/16 13:13:27 INFO dataloader.IgniteDataLoader: Starting the Ignite node on - 10.214.4.161 17/11/16 13:13:27 INFO dataloader.IgniteDataLoader: Started the Ignite node on - 10.214.4.161 17/11/16 13:13:27 INFO dataloader.IgniteDataLoader: Started the Ignite node on - 10.214.4.161 17/11/16 13:13:27 INFO dataloader.IgniteDataLoader: Starting the Ignite node on - 10.214.4.161 17/11/16 13:13:27 INFO dataloader.IgniteDataLoader: Started the Ignite node on - 10.214.4.161 17/11/16 13:13:27 INFO dataloader.IgniteDataLoader: Starting the Ignite node on - 10.214.4.161 17/11/16 13:13:27 INFO dataloader.IgniteDataLoader: Starting the Ignite node on - 10.214.4.161 17/11/16 13:13:27 INFO dataloader.IgniteDataLoader: Started the Ignite node on - 10.214.4.161 17/11/16 13:13:27 INFO dataloader.IgniteDataLoader: Started the Ignite node on - 10.214.4.161 17/11/16 13:13:27 INFO dataloader.IgniteDataLoader: Starting the Ignite node on - 10.214.4.161 17/11/16 13:13:27 INFO dataloader.IgniteDataLoader: Started the Ignite node on - 10.214.4.161 17/11/16 13:13:27 INFO dataloader.IgniteDataLoader: Starting the Ignite node on - 10.214.4.161 17/11/16 13:13:27 INFO dataloader.IgniteDataLoader: Started the Ignite node on - 10.214.4.161 17/11/16 13:13:27 INFO dataloader.IgniteDataLoader: Starting the Ignite node on - 10.214.4.161 17/11/16 13:13:27 INFO dataloader.IgniteDataLoader: Starting the Ignite node on - 10.214.4.161 17/11/16 13:13:27 INFO dataloader.IgniteDataLoader: Started the Ignite node on - 10.214.4.161 17/11/16 13:13:27 INFO dataloader.IgniteDataLoader: Started the Ignite node on - 10.214.4.161 17/11/16 13:13:27 INFO dataloader.IgniteDataLoader: Starting the Ignite node on - 10.214.4.161 17/11/16 13:13:27 INFO dataloader.IgniteDataLoader: Starting the Ignite node on - 10.214.4.161 17/11/16 13:13:27 INFO dataloader.IgniteDataLoader: Starting the Ignite node on - 10.214.4.161 17/11/16 13:13:27 INFO dataloader.IgniteDataLoader: Started the Ignite node on - 10.214.4.161 17/11/16 13:13:27 INFO dataloader.IgniteDataLoader: Started the Ignite node on - 10.214.4.161 17/11/16 13:13:27 INFO dataloader.IgniteDataLoader: Started the Ignite node on - 10.214.4.161 17/11/16 13:13:27 INFO dataloader.IgniteDataLoader: Starting the Ignite node on - 10.214.4.161 17/11/16 13:13:27 INFO dataloader.IgniteDataLoader: Started the Ignite node on - 10.214.4.161 17/11/16 13:13:27 INFO dataloader.IgniteDataLoader: Starting the Ignite node on - 10.214.4.161 17/11/16 13:13:27 INFO dataloader.IgniteDataLoader: Started the Ignite node on - 10.214.4.161 17/11/16 13:13:27 INFO dataloader.IgniteDataLoader: Starting the Ignite node on - 10.214.4.161 17/11/16 13:13:27 INFO dataloader.IgniteDataLoader: Started the Ignite node on - 10.214.4.161 17/11/16 13:13:27 INFO dataloader.IgniteDataLoader: Starting the Ignite node on - 10.214.4.161 17/11/16 13:13:27 INFO dataloader.IgniteDataLoader: Starting the Ignite node on - 10.214.4.161 17/11/16 13:13:27 INFO dataloader.IgniteDataLoader: Started the Ignite node on - 10.214.4.161 17/11/16 13:13:27 INFO dataloader.IgniteDataLoader: Started the Ignite node on - 10.214.4.161 17/11/16 13:13:27 INFO dataloader.IgniteDataLoader: Starting the Ignite node on - 10.214.4.161 17/11/16 13:13:27 INFO dataloader.IgniteDataLoader: Started the Ignite node on - 10.214.4.161 17/11/16 13:13:27 INFO dataloader.IgniteDataLoader: Starting the Ignite node on - 10.214.4.161 17/11/16 13:13:27 INFO dataloader.IgniteDataLoader: Starting the Ignite node on - 10.214.4.161 17/11/16 13:13:27 INFO dataloader.IgniteDataLoader: Starting the Ignite node on - 10.214.4.161 17/11/16 13:13:27 INFO dataloader.IgniteDataLoader: Started the Ignite node on - 10.214.4.161 17/11/16 13:13:27 INFO dataloader.IgniteDataLoader: Started the Ignite node on - 10.214.4.161 17/11/16 13:13:27 INFO dataloader.IgniteDataLoader: Starting the Ignite node on - 10.214.4.161 17/11/16 13:13:27 INFO dataloader.IgniteDataLoader: Started the Ignite node on - 10.214.4.161 17/11/16 13:13:27 INFO dataloader.IgniteDataLoader: Started the Ignite node on - 10.214.4.161 17/11/16 13:13:27 INFO dataloader.IgniteDataLoader: Starting the Ignite node on - 10.214.4.161 17/11/16 13:13:27 INFO dataloader.IgniteDataLoader: Started the Ignite node on - 10.214.4.161 17/11/16 13:13:27 INFO dataloader.IgniteDataLoader: Starting the Ignite node on - 10.214.4.161 17/11/16 13:13:27 INFO dataloader.IgniteDataLoader: Started the Ignite node on - 10.214.4.161 17/11/16 13:13:27 INFO dataloader.IgniteDataLoader: Starting the Ignite node on - 10.214.4.161 17/11/16 13:13:27 INFO dataloader.IgniteDataLoader: Started the Ignite node on - 10.214.4.161 17/11/16 13:13:27 INFO dataloader.IgniteDataLoader: Starting the Ignite node on - 10.214.4.161 17/11/16 13:13:27 INFO dataloader.IgniteDataLoader: Starting the Ignite node on - 10.214.4.161 17/11/16 13:13:27 INFO dataloader.IgniteDataLoader: Started the Ignite node on - 10.214.4.161 17/11/16 13:13:27 INFO dataloader.IgniteDataLoader: Started the Ignite node on - 10.214.4.161 17/11/16 13:13:27 INFO dataloader.IgniteDataLoader: Starting the Ignite node on - 10.214.4.161 17/11/16 13:13:27 INFO dataloader.IgniteDataLoader: Started the Ignite node on - 10.214.4.161 17/11/16 13:13:27 INFO dataloader.IgniteDataLoader: Starting the Ignite node on - 10.214.4.161 17/11/16 13:13:27 INFO dataloader.IgniteDataLoader: Started the Ignite node on - 10.214.4.161 17/11/16 13:13:27 INFO dataloader.IgniteDataLoader: Starting the Ignite node on - 10.214.4.161 17/11/16 13:13:27 INFO dataloader.IgniteDataLoader: Started the Ignite node on - 10.214.4.161 17/11/16 13:13:27 INFO dataloader.IgniteDataLoader: Starting the Ignite node on - 10.214.4.161 17/11/16 13:13:27 INFO dataloader.IgniteDataLoader: Started the Ignite node on - 10.214.4.161 17/11/16 13:13:27 INFO dataloader.IgniteDataLoader: Starting the Ignite node on - 10.214.4.161 17/11/16 13:13:27 INFO dataloader.IgniteDataLoader: Started the Ignite node on - 10.214.4.161 17/11/16 13:13:27 INFO dataloader.IgniteDataLoader: Starting the Ignite node on - 10.214.4.161 17/11/16 13:13:27 INFO dataloader.IgniteDataLoader: Started the Ignite node on - 10.214.4.161 17/11/16 13:13:27 INFO dataloader.IgniteDataLoader: Starting the Ignite node on - 10.214.4.161 17/11/16 13:13:27 INFO dataloader.IgniteDataLoader: Started the Ignite node on - 10.214.4.161 17/11/16 13:13:27 INFO dataloader.IgniteDataLoader: Starting the Ignite node on - 10.214.4.161 17/11/16 13:13:27 INFO dataloader.IgniteDataLoader: Started the Ignite node on - 10.214.4.161 17/11/16 13:13:27 INFO dataloader.IgniteDataLoader: Starting the Ignite node on - 10.214.4.161 17/11/16 13:13:27 INFO dataloader.IgniteDataLoader: Started the Ignite node on - 10.214.4.161 17/11/16 13:13:27 INFO dataloader.IgniteDataLoader: Starting the Ignite node on - 10.214.4.161 17/11/16 13:13:27 INFO dataloader.IgniteDataLoader: Started the Ignite node on - 10.214.4.161 17/11/16 13:13:27 INFO dataloader.IgniteDataLoader: Starting the Ignite node on - 10.214.4.161 17/11/16 13:13:27 INFO dataloader.IgniteDataLoader: Started the Ignite node on - 10.214.4.161 17/11/16 13:13:27 INFO dataloader.IgniteDataLoader: Starting the Ignite node on - 10.214.4.161 17/11/16 13:13:27 INFO dataloader.IgniteDataLoader: Started the Ignite node on - 10.214.4.161 17/11/16 13:13:27 INFO dataloader.IgniteDataLoader: Starting the Ignite node on - 10.214.4.161 17/11/16 13:13:27 INFO dataloader.IgniteDataLoader: Started the Ignite node on - 10.214.4.161 17/11/16 13:13:27 INFO dataloader.IgniteDataLoader: Starting the Ignite node on - 10.214.4.161 17/11/16 13:13:27 INFO dataloader.IgniteDataLoader: Started the Ignite node on - 10.214.4.161 17/11/16 13:13:27 INFO dataloader.IgniteDataLoader: Starting the Ignite node on - 10.214.4.161 17/11/16 13:13:27 INFO dataloader.IgniteDataLoader: Started the Ignite node on - 10.214.4.161 17/11/16 13:13:27 INFO dataloader.IgniteDataLoader: Starting the Ignite node on - 10.214.4.161 17/11/16 13:13:27 INFO dataloader.IgniteDataLoader: Starting the Ignite node on - 10.214.4.161 17/11/16 13:13:27 INFO dataloader.IgniteDataLoader: Started the Ignite node on - 10.214.4.161 17/11/16 13:13:27 INFO dataloader.IgniteDataLoader: Started the Ignite node on - 10.214.4.161 17/11/16 13:13:27 INFO dataloader.IgniteDataLoader: Starting the Ignite node on - 10.214.4.161 17/11/16 13:13:27 INFO dataloader.IgniteDataLoader: Started the Ignite node on - 10.214.4.161 17/11/16 13:30:13 INFO client.ClientUtils$: Fetching metadata from broker id:0,host:kafka-cpsprodeage.int.thomsonreuters.com,port:9092 with correlation id 178 for 1 topic(s) Set(ecpprodcdap.logs.user-v2) 17/11/16 13:30:13 INFO producer.SyncProducer: Connected to kafka-cpsprodeage.int.thomsonreuters.com:9092 for producing 17/11/16 13:30:13 INFO producer.SyncProducer: Disconnecting from kafka-cpsprodeage.int.thomsonreuters.com:9092 17/11/16 13:30:13 INFO producer.SyncProducer: Disconnecting from c593drtkecppr.int.thomsonreuters.com:9092 17/11/16 13:30:13 INFO producer.SyncProducer: Disconnecting from c481bmckecppr.int.thomsonreuters.com:9092 17/11/16 13:30:13 INFO producer.SyncProducer: Disconnecting from c307senkecppr.int.thomsonreuters.com:9092 17/11/16 13:30:13 INFO producer.SyncProducer: Connected to c307senkecppr.int.thomsonreuters.com:9092 for producing I see RpcTimeoutException before ignite node being stopped. Does this mean executor is being lost from the spark cluster which is resulting in ignite node being terminated from the ignite cluster? Do you have any other findings from the log? -- Sent from: http://apache-ignite-users.70518.x6.nabble.com/
