2019-04-09 09:58:03.012 [flink-akka.actor.default-dispatcher-3] INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - SourceConversion(table:[builtin, default, sourceStreamTable, source: [KafkaJsonTableSource]], fields:(TI, EV, CS_HOST, DCS_ID)) -> Calc(select: (TI, EV, DCS_ID)) -> JoinTable(table: (JDBC[int_webtrends_prd_name], schema:{RowType{, types=[IntType, StringType, StringType], fieldNames=[prd_id, prd_name, ev]}}), joinType: LeftOuterJoin, join: (TI, EV, DCS_ID), on: (TI=prd_name, EV=ev)) -> Calc(select: (prd_id, prd_name, 'APP' AS chl_id, DCS_ID)) -> SinkConversion to Tuple2 -> Filter -> Map -> Sink: JDBCRetractTableSink(prd_id, prd_name, chl_id, DCS_ID) (2/2) - execution #0 is assigned resource container_1554366508934_0084_01_000004_3 with d5363785f9f7f1965851ba54de935e7d 2019-04-09 09:58:03.012 [flink-akka.actor.default-dispatcher-3] INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: KafkaJsonTableSource (3/4) - execution #0 is assigned resource container_1554366508934_0084_01_000004_1 with 1f1217c617a27211157883e437a6bd6e 2019-04-09 09:58:03.012 [flink-akka.actor.default-dispatcher-3] INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: KafkaJsonTableSource (2/4) - execution #0 is assigned resource container_1554366508934_0084_01_000004_4 with cd351063b4a86182ad17b7b025541bb4 2019-04-09 09:58:03.013 [flink-akka.actor.default-dispatcher-3] INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - SourceConversion(table:[builtin, default, sourceStreamTable, source: [KafkaJsonTableSource]], fields:(TI, EV, CS_HOST, DCS_ID)) -> Calc(select: (TI, EV, DCS_ID)) -> JoinTable(table: (JDBC[int_webtrends_prd_name], schema:{RowType{, types=[IntType, StringType, StringType], fieldNames=[prd_id, prd_name, ev]}}), joinType: LeftOuterJoin, join: (TI, EV, DCS_ID), on: (TI=prd_name, EV=ev)) -> Calc(select: (prd_id, prd_name, 'APP' AS chl_id, DCS_ID)) -> SinkConversion to Tuple2 -> Filter -> Map -> Sink: JDBCRetractTableSink(prd_id, prd_name, chl_id, DCS_ID) (1/2) - execution #0 is assigned resource container_1554366508934_0084_01_000004_5 with e6742f700171d110fe27258bf77a789f 2019-04-09 09:58:03.013 [flink-akka.actor.default-dispatcher-3] INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: KafkaJsonTableSource (4/4) - execution #0 is assigned resource container_1554366508934_0084_01_000004_2 with 408ccaad63574e30b5fe622b87822fc4 2019-04-09 09:58:03.013 [jobmanager-future-thread-3] INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: KafkaJsonTableSource (1/4) (e972cd212cc3845b114deb5b86a229b6) switched from SCHEDULED to DEPLOYING. 2019-04-09 09:58:03.015 [jobmanager-future-thread-3] INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Deploying Source: KafkaJsonTableSource (1/4) (attempt #0) to slot container_1554366508934_0084_01_000004_0 on bd129120 2019-04-09 09:58:03.016 [jobmanager-future-thread-3] INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: KafkaJsonTableSource (2/4) (96a8bf35b6a85502b2b1afa400c2f0bd) switched from SCHEDULED to DEPLOYING. 2019-04-09 09:58:03.016 [jobmanager-future-thread-3] INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Deploying Source: KafkaJsonTableSource (2/4) (attempt #0) to slot container_1554366508934_0084_01_000004_4 on bd129120 2019-04-09 09:58:03.016 [jobmanager-future-thread-3] INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: KafkaJsonTableSource (3/4) (1f69f483771696ca5352edb9e172de6f) switched from SCHEDULED to DEPLOYING. 2019-04-09 09:58:03.017 [jobmanager-future-thread-3] INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Deploying Source: KafkaJsonTableSource (3/4) (attempt #0) to slot container_1554366508934_0084_01_000004_1 on bd129120 2019-04-09 09:58:03.017 [jobmanager-future-thread-3] INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: KafkaJsonTableSource (4/4) (8c5bfc27c50c37587d5b4439ab3a4a26) switched from SCHEDULED to DEPLOYING. 2019-04-09 09:58:03.017 [jobmanager-future-thread-3] INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Deploying Source: KafkaJsonTableSource (4/4) (attempt #0) to slot container_1554366508934_0084_01_000004_2 on bd129120 2019-04-09 09:58:03.017 [jobmanager-future-thread-3] INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - SourceConversion(table:[builtin, default, sourceStreamTable, source: [KafkaJsonTableSource]], fields:(TI, EV, CS_HOST, DCS_ID)) -> Calc(select: (TI, EV, DCS_ID)) -> JoinTable(table: (JDBC[int_webtrends_prd_name], schema:{RowType{, types=[IntType, StringType, StringType], fieldNames=[prd_id, prd_name, ev]}}), joinType: LeftOuterJoin, join: (TI, EV, DCS_ID), on: (TI=prd_name, EV=ev)) -> Calc(select: (prd_id, prd_name, 'APP' AS chl_id, DCS_ID)) -> SinkConversion to Tuple2 -> Filter -> Map -> Sink: JDBCRetractTableSink(prd_id, prd_name, chl_id, DCS_ID) (1/2) (482b5c9e8f24afcac81e95372b1c02d3) switched from SCHEDULED to DEPLOYING. 2019-04-09 09:58:03.017 [jobmanager-future-thread-3] INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Deploying SourceConversion(table:[builtin, default, sourceStreamTable, source: [KafkaJsonTableSource]], fields:(TI, EV, CS_HOST, DCS_ID)) -> Calc(select: (TI, EV, DCS_ID)) -> JoinTable(table: (JDBC[int_webtrends_prd_name], schema:{RowType{, types=[IntType, StringType, StringType], fieldNames=[prd_id, prd_name, ev]}}), joinType: LeftOuterJoin, join: (TI, EV, DCS_ID), on: (TI=prd_name, EV=ev)) -> Calc(select: (prd_id, prd_name, 'APP' AS chl_id, DCS_ID)) -> SinkConversion to Tuple2 -> Filter -> Map -> Sink: JDBCRetractTableSink(prd_id, prd_name, chl_id, DCS_ID) (1/2) (attempt #0) to slot container_1554366508934_0084_01_000004_5 on bd129120 2019-04-09 09:58:03.017 [jobmanager-future-thread-3] INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - SourceConversion(table:[builtin, default, sourceStreamTable, source: [KafkaJsonTableSource]], fields:(TI, EV, CS_HOST, DCS_ID)) -> Calc(select: (TI, EV, DCS_ID)) -> JoinTable(table: (JDBC[int_webtrends_prd_name], schema:{RowType{, types=[IntType, StringType, StringType], fieldNames=[prd_id, prd_name, ev]}}), joinType: LeftOuterJoin, join: (TI, EV, DCS_ID), on: (TI=prd_name, EV=ev)) -> Calc(select: (prd_id, prd_name, 'APP' AS chl_id, DCS_ID)) -> SinkConversion to Tuple2 -> Filter -> Map -> Sink: JDBCRetractTableSink(prd_id, prd_name, chl_id, DCS_ID) (2/2) (4c0abbf6e5dc19b1bd51e650364a2c26) switched from SCHEDULED to DEPLOYING. 2019-04-09 09:58:03.018 [jobmanager-future-thread-3] INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Deploying SourceConversion(table:[builtin, default, sourceStreamTable, source: [KafkaJsonTableSource]], fields:(TI, EV, CS_HOST, DCS_ID)) -> Calc(select: (TI, EV, DCS_ID)) -> JoinTable(table: (JDBC[int_webtrends_prd_name], schema:{RowType{, types=[IntType, StringType, StringType], fieldNames=[prd_id, prd_name, ev]}}), joinType: LeftOuterJoin, join: (TI, EV, DCS_ID), on: (TI=prd_name, EV=ev)) -> Calc(select: (prd_id, prd_name, 'APP' AS chl_id, DCS_ID)) -> SinkConversion to Tuple2 -> Filter -> Map -> Sink: JDBCRetractTableSink(prd_id, prd_name, chl_id, DCS_ID) (2/2) (attempt #0) to slot container_1554366508934_0084_01_000004_3 on bd129120 2019-04-09 09:58:04.081 [flink-akka.actor.default-dispatcher-27] INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - SourceConversion(table:[builtin, default, sourceStreamTable, source: [KafkaJsonTableSource]], fields:(TI, EV, CS_HOST, DCS_ID)) -> Calc(select: (TI, EV, DCS_ID)) -> JoinTable(table: (JDBC[int_webtrends_prd_name], schema:{RowType{, types=[IntType, StringType, StringType], fieldNames=[prd_id, prd_name, ev]}}), joinType: LeftOuterJoin, join: (TI, EV, DCS_ID), on: (TI=prd_name, EV=ev)) -> Calc(select: (prd_id, prd_name, 'APP' AS chl_id, DCS_ID)) -> SinkConversion to Tuple2 -> Filter -> Map -> Sink: JDBCRetractTableSink(prd_id, prd_name, chl_id, DCS_ID) (1/2) (482b5c9e8f24afcac81e95372b1c02d3) switched from DEPLOYING to RUNNING. 2019-04-09 09:58:04.086 [flink-akka.actor.default-dispatcher-3] INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - SourceConversion(table:[builtin, default, sourceStreamTable, source: [KafkaJsonTableSource]], fields:(TI, EV, CS_HOST, DCS_ID)) -> Calc(select: (TI, EV, DCS_ID)) -> JoinTable(table: (JDBC[int_webtrends_prd_name], schema:{RowType{, types=[IntType, StringType, StringType], fieldNames=[prd_id, prd_name, ev]}}), joinType: LeftOuterJoin, join: (TI, EV, DCS_ID), on: (TI=prd_name, EV=ev)) -> Calc(select: (prd_id, prd_name, 'APP' AS chl_id, DCS_ID)) -> SinkConversion to Tuple2 -> Filter -> Map -> Sink: JDBCRetractTableSink(prd_id, prd_name, chl_id, DCS_ID) (2/2) (4c0abbf6e5dc19b1bd51e650364a2c26) switched from DEPLOYING to RUNNING. 2019-04-09 09:58:04.087 [flink-akka.actor.default-dispatcher-3] INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: KafkaJsonTableSource (2/4) (96a8bf35b6a85502b2b1afa400c2f0bd) switched from DEPLOYING to RUNNING. 2019-04-09 09:58:04.089 [flink-akka.actor.default-dispatcher-3] INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: KafkaJsonTableSource (3/4) (1f69f483771696ca5352edb9e172de6f) switched from DEPLOYING to RUNNING. 2019-04-09 09:58:04.091 [flink-akka.actor.default-dispatcher-3] INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: KafkaJsonTableSource (1/4) (e972cd212cc3845b114deb5b86a229b6) switched from DEPLOYING to RUNNING. 2019-04-09 09:58:04.092 [flink-akka.actor.default-dispatcher-3] INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: KafkaJsonTableSource (4/4) (8c5bfc27c50c37587d5b4439ab3a4a26) switched from DEPLOYING to RUNNING. 2019-04-09 09:58:07.508 [AMRM Callback Handler Thread] INFO org.apache.flink.yarn.YarnSessionResourceManager - Container container_1554366508934_0084_01_000003 finished with exit code 31 2019-04-09 09:58:07.508 [AMRM Callback Handler Thread] INFO org.apache.flink.yarn.YarnSessionResourceManager - Requesting new container with resources <memory:2048, vCores:6>. Number pending requests 2. 2019-04-09 09:58:07.509 [AMRM Callback Handler Thread] INFO org.apache.flink.yarn.YarnSessionResourceManager - Container container_1554366508934_0084_01_000005 finished with exit code 31 2019-04-09 09:58:07.509 [AMRM Callback Handler Thread] INFO org.apache.flink.yarn.YarnSessionResourceManager - Requesting new container with resources <memory:2048, vCores:6>. Number pending requests 3. 2019-04-09 09:58:08.012 [AMRM Callback Handler Thread] INFO org.apache.flink.yarn.YarnSessionResourceManager - Received new container: container_1554366508934_0084_01_000006 - Remaining pending container requests: 2 2019-04-09 09:58:08.013 [pool-1-thread-4] INFO org.apache.flink.yarn.YarnSessionResourceManager - TaskExecutor container_1554366508934_0084_01_000006 will be started with container size 2048 MB, JVM heap size 1920 MB, new generation size 480 MB, JVM direct memory limit 128 MB on cdh129139:8042 2019-04-09 09:58:08.015 [pool-1-thread-4] INFO org.apache.flink.yarn.YarnSessionResourceManager - Adding keytab hdfs://cdh129130:8020/user/hive/.flink/application_1554366508934_0084/hive.keytab to the AM container local resource bucket 2019-04-09 09:58:08.047 [pool-1-thread-4] INFO org.apache.flink.yarn.Utils - Use the beforehand copied resource hdfs://cdh129130:8020/user/hive/.flink/application_1554366508934_0084/taskmanager-conf.yaml (the corresponding local path: file:/home/yarn/nm/usercache/hive/appcache/application_1554366508934_0084/container_1554366508934_0084_01_000001/taskmanager-conf.yaml). Visibility: APPLICATION. 2019-04-09 09:58:08.157 [pool-1-thread-4] INFO org.apache.flink.yarn.YarnSessionResourceManager - Creating container launch context for TaskManagers 2019-04-09 09:58:08.158 [pool-1-thread-4] INFO org.apache.flink.yarn.YarnSessionResourceManager - Starting TaskManagers 2019-04-09 09:58:08.518 [AMRM Callback Handler Thread] INFO org.apache.flink.yarn.YarnSessionResourceManager - Received new container: container_1554366508934_0084_01_000007 - Remaining pending container requests: 1 2019-04-09 09:58:08.519 [pool-1-thread-3] INFO org.apache.flink.yarn.YarnSessionResourceManager - TaskExecutor container_1554366508934_0084_01_000007 will be started with container size 2048 MB, JVM heap size 1920 MB, new generation size 480 MB, JVM direct memory limit 128 MB on cdh129144:8042 2019-04-09 09:58:08.520 [AMRM Callback Handler Thread] INFO org.apache.flink.yarn.YarnSessionResourceManager - Received new container: container_1554366508934_0084_01_000009 - Remaining pending container requests: 0 2019-04-09 09:58:08.520 [pool-1-thread-3] INFO org.apache.flink.yarn.YarnSessionResourceManager - Adding keytab hdfs://cdh129130:8020/user/hive/.flink/application_1554366508934_0084/hive.keytab to the AM container local resource bucket 2019-04-09 09:58:08.521 [pool-1-thread-1] INFO org.apache.flink.yarn.YarnSessionResourceManager - TaskExecutor container_1554366508934_0084_01_000009 will be started with container size 2048 MB, JVM heap size 1920 MB, new generation size 480 MB, JVM direct memory limit 128 MB on cdh129136:8042 2019-04-09 09:58:08.523 [pool-1-thread-1] INFO org.apache.flink.yarn.YarnSessionResourceManager - Adding keytab hdfs://cdh129130:8020/user/hive/.flink/application_1554366508934_0084/hive.keytab to the AM container local resource bucket 2019-04-09 09:58:08.555 [pool-1-thread-1] INFO org.apache.flink.yarn.Utils - Use the beforehand copied resource hdfs://cdh129130:8020/user/hive/.flink/application_1554366508934_0084/taskmanager-conf.yaml (the corresponding local path: file:/home/yarn/nm/usercache/hive/appcache/application_1554366508934_0084/container_1554366508934_0084_01_000001/taskmanager-conf.yaml). Visibility: APPLICATION. 2019-04-09 09:58:08.555 [pool-1-thread-3] INFO org.apache.flink.yarn.Utils - Use the beforehand copied resource hdfs://cdh129130:8020/user/hive/.flink/application_1554366508934_0084/taskmanager-conf.yaml (the corresponding local path: file:/home/yarn/nm/usercache/hive/appcache/application_1554366508934_0084/container_1554366508934_0084_01_000001/taskmanager-conf.yaml). Visibility: APPLICATION. 2019-04-09 09:58:08.661 [pool-1-thread-1] INFO org.apache.flink.yarn.YarnSessionResourceManager - Creating container launch context for TaskManagers 2019-04-09 09:58:08.661 [pool-1-thread-3] INFO org.apache.flink.yarn.YarnSessionResourceManager - Creating container launch context for TaskManagers 2019-04-09 09:58:08.662 [pool-1-thread-1] INFO org.apache.flink.yarn.YarnSessionResourceManager - Starting TaskManagers 2019-04-09 09:58:08.662 [pool-1-thread-3] INFO org.apache.flink.yarn.YarnSessionResourceManager - Starting TaskManagers 2019-04-09 09:58:14.023 [AMRM Callback Handler Thread] INFO org.apache.flink.yarn.YarnSessionResourceManager - Container container_1554366508934_0084_01_000006 finished with exit code 31 2019-04-09 09:58:14.024 [AMRM Callback Handler Thread] INFO org.apache.flink.yarn.YarnSessionResourceManager - Requesting new container with resources <memory:2048, vCores:6>. Number pending requests 1. 2019-04-09 09:58:14.024 [AMRM Callback Handler Thread] INFO org.apache.flink.yarn.YarnSessionResourceManager - Container container_1554366508934_0084_01_000007 finished with exit code 31 2019-04-09 09:58:14.024 [AMRM Callback Handler Thread] INFO org.apache.flink.yarn.YarnSessionResourceManager - Requesting new container with resources <memory:2048, vCores:6>. Number pending requests 2. 2019-04-09 09:58:14.024 [AMRM Callback Handler Thread] INFO org.apache.flink.yarn.YarnSessionResourceManager - Container container_1554366508934_0084_01_000009 finished with exit code 31 2019-04-09 09:58:14.025 [AMRM Callback Handler Thread] INFO org.apache.flink.yarn.YarnSessionResourceManager - Requesting new container with resources <memory:2048, vCores:6>. Number pending requests 3. 2019-04-09 09:58:19.533 [AMRM Callback Handler Thread] INFO org.apache.flink.yarn.YarnSessionResourceManager - Received new container: container_1554366508934_0084_01_000014 - Remaining pending container requests: 2 2019-04-09 09:58:19.534 [pool-1-thread-2] INFO org.apache.flink.yarn.YarnSessionResourceManager - TaskExecutor container_1554366508934_0084_01_000014 will be started with container size 2048 MB, JVM heap size 1920 MB, new generation size 480 MB, JVM direct memory limit 128 MB on cdh129139:8042 2019-04-09 09:58:19.534 [AMRM Callback Handler Thread] INFO org.apache.flink.yarn.YarnSessionResourceManager - Received new container: container_1554366508934_0084_01_000015 - Remaining pending container requests: 1 2019-04-09 09:58:19.535 [AMRM Callback Handler Thread] INFO org.apache.flink.yarn.YarnSessionResourceManager - Received new container: container_1554366508934_0084_01_000016 - Remaining pending container requests: 0 2019-04-09 09:58:19.535 [pool-1-thread-2] INFO org.apache.flink.yarn.YarnSessionResourceManager - Adding keytab hdfs://cdh129130:8020/user/hive/.flink/application_1554366508934_0084/hive.keytab to the AM container local resource bucket 2019-04-09 09:58:19.535 [pool-1-thread-5] INFO org.apache.flink.yarn.YarnSessionResourceManager - TaskExecutor container_1554366508934_0084_01_000015 will be started with container size 2048 MB, JVM heap size 1920 MB, new generation size 480 MB, JVM direct memory limit 128 MB on cdh129144:8042 2019-04-09 09:58:19.535 [pool-1-thread-4] INFO org.apache.flink.yarn.YarnSessionResourceManager - TaskExecutor container_1554366508934_0084_01_000016 will be started with container size 2048 MB, JVM heap size 1920 MB, new generation size 480 MB, JVM direct memory limit 128 MB on cdh129136:8042 2019-04-09 09:58:19.536 [pool-1-thread-5] INFO org.apache.flink.yarn.YarnSessionResourceManager - Adding keytab hdfs://cdh129130:8020/user/hive/.flink/application_1554366508934_0084/hive.keytab to the AM container local resource bucket 2019-04-09 09:58:19.536 [pool-1-thread-4] INFO org.apache.flink.yarn.YarnSessionResourceManager - Adding keytab hdfs://cdh129130:8020/user/hive/.flink/application_1554366508934_0084/hive.keytab to the AM container local resource bucket 2019-04-09 09:58:19.565 [pool-1-thread-4] INFO org.apache.flink.yarn.Utils - Use the beforehand copied resource hdfs://cdh129130:8020/user/hive/.flink/application_1554366508934_0084/taskmanager-conf.yaml (the corresponding local path: file:/home/yarn/nm/usercache/hive/appcache/application_1554366508934_0084/container_1554366508934_0084_01_000001/taskmanager-conf.yaml). Visibility: APPLICATION. 2019-04-09 09:58:19.565 [pool-1-thread-5] INFO org.apache.flink.yarn.Utils - Use the beforehand copied resource hdfs://cdh129130:8020/user/hive/.flink/application_1554366508934_0084/taskmanager-conf.yaml (the corresponding local path: file:/home/yarn/nm/usercache/hive/appcache/application_1554366508934_0084/container_1554366508934_0084_01_000001/taskmanager-conf.yaml). Visibility: APPLICATION. 2019-04-09 09:58:19.565 [pool-1-thread-2] INFO org.apache.flink.yarn.Utils - Use the beforehand copied resource hdfs://cdh129130:8020/user/hive/.flink/application_1554366508934_0084/taskmanager-conf.yaml (the corresponding local path: file:/home/yarn/nm/usercache/hive/appcache/application_1554366508934_0084/container_1554366508934_0084_01_000001/taskmanager-conf.yaml). Visibility: APPLICATION. 2019-04-09 09:58:19.665 [pool-1-thread-2] INFO org.apache.flink.yarn.YarnSessionResourceManager - Creating container launch context for TaskManagers 2019-04-09 09:58:19.665 [pool-1-thread-4] INFO org.apache.flink.yarn.YarnSessionResourceManager - Creating container launch context for TaskManagers 2019-04-09 09:58:19.665 [pool-1-thread-5] INFO org.apache.flink.yarn.YarnSessionResourceManager - Creating container launch context for TaskManagers 2019-04-09 09:58:19.665 [pool-1-thread-2] INFO org.apache.flink.yarn.YarnSessionResourceManager - Starting TaskManagers 2019-04-09 09:58:19.665 [pool-1-thread-4] INFO org.apache.flink.yarn.YarnSessionResourceManager - Starting TaskManagers 2019-04-09 09:58:19.665 [pool-1-thread-5] INFO org.apache.flink.yarn.YarnSessionResourceManager - Starting TaskManagers 2019-04-09 09:58:25.037 [AMRM Callback Handler Thread] INFO org.apache.flink.yarn.YarnSessionResourceManager - Container container_1554366508934_0084_01_000014 finished with exit code 31 2019-04-09 09:58:25.038 [AMRM Callback Handler Thread] INFO org.apache.flink.yarn.YarnSessionResourceManager - Requesting new container with resources <memory:2048, vCores:6>. Number pending requests 1. 2019-04-09 09:58:25.038 [AMRM Callback Handler Thread] INFO org.apache.flink.yarn.YarnSessionResourceManager - Container container_1554366508934_0084_01_000015 finished with exit code 31 2019-04-09 09:58:25.039 [AMRM Callback Handler Thread] INFO org.apache.flink.yarn.YarnSessionResourceManager - Requesting new container with resources <memory:2048, vCores:6>. Number pending requests 2. 2019-04-09 09:58:25.039 [AMRM Callback Handler Thread] INFO org.apache.flink.yarn.YarnSessionResourceManager - Container container_1554366508934_0084_01_000016 finished with exit code 31 2019-04-09 09:58:25.039 [AMRM Callback Handler Thread] INFO org.apache.flink.yarn.YarnSessionResourceManager - Requesting new container with resources <memory:2048, vCores:6>. Number pending requests 3. 2019-04-09 09:58:30.547 [AMRM Callback Handler Thread] INFO org.apache.flink.yarn.YarnSessionResourceManager - Received new container: container_1554366508934_0084_01_000017 - Remaining pending container requests: 2 2019-04-09 09:58:30.548 [AMRM Callback Handler Thread] INFO org.apache.flink.yarn.YarnSessionResourceManager - Received new container: container_1554366508934_0084_01_000018 - Remaining pending container requests: 1 2019-04-09 09:58:30.548 [AMRM Callback Handler Thread] INFO org.apache.flink.yarn.YarnSessionResourceManager - Received new container: container_1554366508934_0084_01_000019 - Remaining pending container requests: 0 2019-04-09 09:58:30.548 [pool-1-thread-6] INFO org.apache.flink.yarn.YarnSessionResourceManager - TaskExecutor container_1554366508934_0084_01_000017 will be started with container size 2048 MB, JVM heap size 1920 MB, new generation size 480 MB, JVM direct memory limit 128 MB on cdh129139:8042 2019-04-09 09:58:30.549 [pool-1-thread-3] INFO org.apache.flink.yarn.YarnSessionResourceManager - TaskExecutor container_1554366508934_0084_01_000019 will be started with container size 2048 MB, JVM heap size 1920 MB, new generation size 480 MB, JVM direct memory limit 128 MB on cdh129136:8042 2019-04-09 09:58:30.548 [pool-1-thread-7] INFO org.apache.flink.yarn.YarnSessionResourceManager - TaskExecutor container_1554366508934_0084_01_000018 will be started with container size 2048 MB, JVM heap size 1920 MB, new generation size 480 MB, JVM direct memory limit 128 MB on cdh129144:8042 2019-04-09 09:58:30.549 [pool-1-thread-6] INFO org.apache.flink.yarn.YarnSessionResourceManager - Adding keytab hdfs://cdh129130:8020/user/hive/.flink/application_1554366508934_0084/hive.keytab to the AM container local resource bucket 2019-04-09 09:58:30.550 [pool-1-thread-3] INFO org.apache.flink.yarn.YarnSessionResourceManager - Adding keytab hdfs://cdh129130:8020/user/hive/.flink/application_1554366508934_0084/hive.keytab to the AM container local resource bucket 2019-04-09 09:58:30.550 [pool-1-thread-7] INFO org.apache.flink.yarn.YarnSessionResourceManager - Adding keytab hdfs://cdh129130:8020/user/hive/.flink/application_1554366508934_0084/hive.keytab to the AM container local resource bucket 2019-04-09 09:58:30.575 [pool-1-thread-3] INFO org.apache.flink.yarn.Utils - Use the beforehand copied resource hdfs://cdh129130:8020/user/hive/.flink/application_1554366508934_0084/taskmanager-conf.yaml (the corresponding local path: file:/home/yarn/nm/usercache/hive/appcache/application_1554366508934_0084/container_1554366508934_0084_01_000001/taskmanager-conf.yaml). Visibility: APPLICATION. 2019-04-09 09:58:30.576 [pool-1-thread-6] INFO org.apache.flink.yarn.Utils - Use the beforehand copied resource hdfs://cdh129130:8020/user/hive/.flink/application_1554366508934_0084/taskmanager-conf.yaml (the corresponding local path: file:/home/yarn/nm/usercache/hive/appcache/application_1554366508934_0084/container_1554366508934_0084_01_000001/taskmanager-conf.yaml). Visibility: APPLICATION. 2019-04-09 09:58:30.576 [pool-1-thread-7] INFO org.apache.flink.yarn.Utils - Use the beforehand copied resource hdfs://cdh129130:8020/user/hive/.flink/application_1554366508934_0084/taskmanager-conf.yaml (the corresponding local path: file:/home/yarn/nm/usercache/hive/appcache/application_1554366508934_0084/container_1554366508934_0084_01_000001/taskmanager-conf.yaml). Visibility: APPLICATION. 2019-04-09 09:58:30.661 [pool-1-thread-6] INFO org.apache.flink.yarn.YarnSessionResourceManager - Creating container launch context for TaskManagers 2019-04-09 09:58:30.661 [pool-1-thread-7] INFO org.apache.flink.yarn.YarnSessionResourceManager - Creating container launch context for TaskManagers 2019-04-09 09:58:30.661 [pool-1-thread-3] INFO org.apache.flink.yarn.YarnSessionResourceManager - Creating container launch context for TaskManagers 2019-04-09 09:58:30.662 [pool-1-thread-6] INFO org.apache.flink.yarn.YarnSessionResourceManager - Starting TaskManagers 2019-04-09 09:58:30.662 [pool-1-thread-7] INFO org.apache.flink.yarn.YarnSessionResourceManager - Starting TaskManagers 2019-04-09 09:58:30.662 [pool-1-thread-3] INFO org.apache.flink.yarn.YarnSessionResourceManager - Starting TaskManagers
________________________________ sean...@live.com 发件人: 苏 欣<mailto:sean...@live.com> 发送时间: 2019-04-09 10:36 收件人: user-zh<mailto:user-zh@flink.apache.org> 主题: 回复: 答复: blink提交yarn卡在一直重复分配container ________________________________ sean...@live.com 发件人: 苏 欣<mailto:sean...@live.com> 发送时间: 2019-04-09 10:30 收件人: user-zh@flink.apache.org<mailto:user-zh@flink.apache.org> 主题: 答复: blink提交yarn卡在一直重复分配container 不好意思,已补充yarn的日志文件。 出现问题的原因我已经找到了,在配置flink-conf.yaml中的下面三项后,会出现分配不了资源的问题 security.kerberos.login.use-ticket-cache: false security.kerberos.login.keytab: /home/hive.keytab security.kerberos.login.principal: hive/cdh129135@MYCDH 如果在客户机使用kinit命令后再提交,yarn资源可以正常分配。 现在我有几个问题请教大佬们: 1、 提交作业到带有kerberos认证的yarn,除了kinit方式之外还有其他方式吗,为什么读配置文件中的票据会出现code 31? 2、 taskmanager.cpu.core与slot数量在yarn上面他们是相等的吗?有没有一个core分配多个slot的情况? 发送自 Windows 10 版邮件<https://go.microsoft.com/fwlink/?LinkId=550986>应用 发件人: Zili Chen<mailto:wander4...@gmail.com> 发送时间: 2019年4月8日 19:29 收件人: user-zh@flink.apache.org<mailto:user-zh@flink.apache.org> 主题: Re: blink提交yarn卡在一直重复分配container 你好,apache 的邮件列表不支持内嵌图片,请以附件或链接方式引用。 Best, tison. 苏 欣 <sean...@live.com> 于2019年4月8日周一 上午10:17写道: > 我以per-job方式提交了一个作业到yarn上面,发现会出现不断重复分配container的现象。 > > 现象为从yarn的web ui上看一瞬间tm的container分配成功了,但是立刻变为只剩一个jm的container,接着会继续分配tm的 > container。不断的重复这个过程直到作业调度不到资源而失败。 > > 我查了一下exit code没找到31代表是什么意思,有没有大佬帮忙分析下,非常感谢! > > > > 发送自 Windows 10 版邮件 <https://go.microsoft.com/fwlink/?LinkId=550986>应用 > > >