Hi Anandha, If your hive query has completed and still the containers are running, you might want to set the param: tez.session.am.dag.submit.timeout.secs to a lower value , for the yarn application to be completely shutdown. In session mode, by default the AM container waits till 5 minutes for any other queries to be fired.
Thanks, Ravi On Sat, May 20, 2017 at 4:44 AM, Anandha L Ranganathan < [email protected]> wrote: > We hit with unique problem. Still the tez containers are running but Hive > moved the file from staging location to actual location. Is this bug in the > Tez engine? > > I have attached logs and you can see still Map1, Reducer3 , Reducer 4, > Reducer 7 are running but DAG is SUCCEEDED. > > > > [2017-04-21 18:08:57,108] {hive_hooks.py:130} INFO - INFO : Map 1: 0(+2)/8 > Map 12: 1/1 Map 5: 1/1 Map 8: 1/1 Map 9: 1/1 Reducer 11: > 0/1 Reducer 3: 0/308 Reducer 4: 0/2 Reducer 7: 0/1 > [2017-04-21 18:09:57,116] {hive_hooks.py:130} INFO - INFO : Map 1: 0(+2)/8 > Map 12: 1/1 Map 5: 1/1 Map 8: 1/1 Map 9: 1/1 Reducer 11: > 0/1 Reducer 3: 0/308 Reducer 4: 0/2 Reducer 7: 0/1 > [2017-04-21 18:09:57,116] {hive_hooks.py:130} INFO - INFO : Map 1: 0(+2)/8 > Map 12: 1/1 Map 5: 1/1 Map 8: 1/1 Map 9: 1/1 Reducer 11: > 0/1 Reducer 3: 0/308 Reducer 4: 0/2 Reducer 7: 0/1 > [2017-04-21 18:29:38,595] {hive_hooks.py:130} INFO - INFO : Map 1: 0(+2)/8 > Map 12: 1/1 Map 5: 1/1 Map 8: 1/1 Map 9: 1/1 Reducer 11: > 0/1 Reducer 3: 0/308 Reducer 4: 0/2 Reducer 7: 0/1 > [2017-04-21 18:29:38,595] {hive_hooks.py:130} INFO - INFO : > [2017-04-21 18:29:38,596] {hive_hooks.py:130} INFO - INFO : Status: DAG > finished successfully in 1560.07 seconds > [2017-04-21 18:29:38,596] {hive_hooks.py:130} INFO - INFO : > [2017-04-21 18:29:38,596] {hive_hooks.py:130} INFO - > [2017-04-21 18:29:38,596] {hive_hooks.py:130} INFO - INFO : METHOD > DURATION(ms) > [2017-04-21 18:29:38,596] {hive_hooks.py:130} INFO - INFO : parse > 0 > [2017-04-21 18:29:38,596] {hive_hooks.py:130} INFO - INFO : semanticAnalyze > 0 > [2017-04-21 18:29:38,596] {hive_hooks.py:130} INFO - INFO : TezBuildDag > 0 > [2017-04-21 18:29:38,596] {hive_hooks.py:130} INFO - INFO : > TezSubmitToRunningDag 12,957 > [2017-04-21 18:29:38,597] {hive_hooks.py:130} INFO - INFO : TotalPrepTime > 53,372 > [2017-04-21 18:29:38,597] {hive_hooks.py:130} INFO - > [2017-04-21 18:29:38,597] {hive_hooks.py:130} INFO - INFO : Loading data to > table dw_insight.d_experiment from > hdfs://dfs-nameservices/data/insight/dwuser/d_experiment/.hive-staging_hive_2017-04-21_18-02-31_741_4202396962843353301-4404/-ext-10000 > [2017-04-21 18:29:38,597] {hive_hooks.py:130} INFO - INFO : Table > dw_insight.d_experiment stats: [numFiles=0, numRows=0, totalSize=0, > rawDataSize=0] > [2017-04-21 18:29:38,597] {hive_hooks.py:130} INFO - No rows affected > (1626.877 seconds) > [2017-04-21 18:29:38,604] {hive_hooks.py:130} INFO - 0: > jdbc:hive2://usw2prdpmn01:10000/dw_insight> > [2017-04-21 18:29:38,611] {hive_hooks.py:130} INFO - 0: > jdbc:hive2://usw2prdpmn01:10000/dw_insight> ANALYZE TABLE ${hiveconf:target_t > >
