[jira] [Commented] (HIVE-14111) better concurrency handling for TezSessionState - part I
[ https://issues.apache.org/jira/browse/HIVE-14111?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15373587#comment-15373587 ] Sergey Shelukhin commented on HIVE-14111: - Probably not: {noformat} org.apache.hadoop.ipc.RemoteException: Cannot renew lease for DFSClient_NONMAPREDUCE_-930074842_1. Name node is in safe mode. {noformat} > better concurrency handling for TezSessionState - part I > > > Key: HIVE-14111 > URL: https://issues.apache.org/jira/browse/HIVE-14111 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > Attachments: HIVE-14111.01.patch, HIVE-14111.02.patch, > HIVE-14111.03.patch, HIVE-14111.04.patch, HIVE-14111.05.patch, > HIVE-14111.06.patch, HIVE-14111.patch, sessionPoolNotes.txt > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-14111) better concurrency handling for TezSessionState - part I
[ https://issues.apache.org/jira/browse/HIVE-14111?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15373545#comment-15373545 ] Siddharth Seth commented on HIVE-14111: --- +1. Is the test failure on TestMiniTezCliDriver-tez_self_join.q-filter_join_breaktask.q-vector_decimal_precision.q-and-12-more - did not produce a TEST-*.xml file related ? Looks new. > better concurrency handling for TezSessionState - part I > > > Key: HIVE-14111 > URL: https://issues.apache.org/jira/browse/HIVE-14111 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > Attachments: HIVE-14111.01.patch, HIVE-14111.02.patch, > HIVE-14111.03.patch, HIVE-14111.04.patch, HIVE-14111.05.patch, > HIVE-14111.06.patch, HIVE-14111.patch, sessionPoolNotes.txt > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-14111) better concurrency handling for TezSessionState - part I
[ https://issues.apache.org/jira/browse/HIVE-14111?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15373466#comment-15373466 ] Sergey Shelukhin commented on HIVE-14111: - [~sseth] ping? > better concurrency handling for TezSessionState - part I > > > Key: HIVE-14111 > URL: https://issues.apache.org/jira/browse/HIVE-14111 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > Attachments: HIVE-14111.01.patch, HIVE-14111.02.patch, > HIVE-14111.03.patch, HIVE-14111.04.patch, HIVE-14111.05.patch, > HIVE-14111.06.patch, HIVE-14111.patch, sessionPoolNotes.txt > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-14111) better concurrency handling for TezSessionState - part I
[ https://issues.apache.org/jira/browse/HIVE-14111?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15372141#comment-15372141 ] Hive QA commented on HIVE-14111: Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12817215/HIVE-14111.06.patch {color:green}SUCCESS:{color} +1 due to 2 test(s) being added or modified. {color:red}ERROR:{color} -1 due to 9 failed/errored test(s), 10294 tests executed *Failed tests:* {noformat} TestMiniTezCliDriver-tez_self_join.q-filter_join_breaktask.q-vector_decimal_precision.q-and-12-more - did not produce a TEST-*.xml file org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_acid_globallimit org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_list_bucket_dml_12 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_stats_list_bucket org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_subquery_multiinsert org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_vector_complex_all org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_vector_complex_join org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_acid_globallimit org.apache.hadoop.hive.cli.TestMinimrCliDriver.org.apache.hadoop.hive.cli.TestMinimrCliDriver {noformat} Test results: https://builds.apache.org/job/PreCommit-HIVE-MASTER-Build/476/testReport Console output: https://builds.apache.org/job/PreCommit-HIVE-MASTER-Build/476/console Test logs: http://ec2-204-236-174-241.us-west-1.compute.amazonaws.com/logs/PreCommit-HIVE-MASTER-Build-476/ Messages: {noformat} Executing org.apache.hive.ptest.execution.TestCheckPhase Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase Tests exited with: TestsFailedException: 9 tests failed {noformat} This message is automatically generated. ATTACHMENT ID: 12817215 - PreCommit-HIVE-MASTER-Build > better concurrency handling for TezSessionState - part I > > > Key: HIVE-14111 > URL: https://issues.apache.org/jira/browse/HIVE-14111 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > Attachments: HIVE-14111.01.patch, HIVE-14111.02.patch, > HIVE-14111.03.patch, HIVE-14111.04.patch, HIVE-14111.05.patch, > HIVE-14111.06.patch, HIVE-14111.patch, sessionPoolNotes.txt > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-14111) better concurrency handling for TezSessionState - part I
[ https://issues.apache.org/jira/browse/HIVE-14111?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15369233#comment-15369233 ] Hive QA commented on HIVE-14111: Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12816891/HIVE-14111.05.patch {color:red}ERROR:{color} -1 due to build exiting with an error Test results: https://builds.apache.org/job/PreCommit-HIVE-MASTER-Build/443/testReport Console output: https://builds.apache.org/job/PreCommit-HIVE-MASTER-Build/443/console Test logs: http://ec2-204-236-174-241.us-west-1.compute.amazonaws.com/logs/PreCommit-HIVE-MASTER-Build-443/ Messages: {noformat} Executing org.apache.hive.ptest.execution.TestCheckPhase Executing org.apache.hive.ptest.execution.PrepPhase Tests exited with: NonZeroExitCodeException Command 'bash /data/hive-ptest/working/scratch/source-prep.sh' failed with exit status 1 and output '+ [[ -n /usr/java/jdk1.8.0_25 ]] + export JAVA_HOME=/usr/java/jdk1.8.0_25 + JAVA_HOME=/usr/java/jdk1.8.0_25 + export PATH=/usr/java/jdk1.8.0_25/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games + PATH=/usr/java/jdk1.8.0_25/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games + export 'ANT_OPTS=-Xmx1g -XX:MaxPermSize=256m ' + ANT_OPTS='-Xmx1g -XX:MaxPermSize=256m ' + export 'M2_OPTS=-Xmx1g -XX:MaxPermSize=256m -Dhttp.proxyHost=localhost -Dhttp.proxyPort=3128' + M2_OPTS='-Xmx1g -XX:MaxPermSize=256m -Dhttp.proxyHost=localhost -Dhttp.proxyPort=3128' + cd /data/hive-ptest/working/ + tee /data/hive-ptest/logs/PreCommit-HIVE-MASTER-Build-443/source-prep.txt + [[ false == \t\r\u\e ]] + mkdir -p maven ivy + [[ git = \s\v\n ]] + [[ git = \g\i\t ]] + [[ -z master ]] + [[ -d apache-github-source-source ]] + [[ ! -d apache-github-source-source/.git ]] + [[ ! -d apache-github-source-source ]] + cd apache-github-source-source + git fetch origin >From https://github.com/apache/hive 5cf0c1c..667952d master -> origin/master f361403..5d8df7c branch-2.1 -> origin/branch-2.1 + git reset --hard HEAD HEAD is now at 5cf0c1c HIVE-13901 : Hivemetastore add partitions can be slow depending on filesystems (Rajesh Balamohan via Sergey Shelukhin) + git clean -f -d + git checkout master Already on 'master' Your branch is behind 'origin/master' by 1 commit, and can be fast-forwarded. (use "git pull" to update your local branch) + git reset --hard origin/master HEAD is now at 667952d HIVE-14184: Adding test for limit pushdown in presence of grouping sets (Jesus Camacho Rodriguez, reviewed by Ashutosh Chauhan) + git merge --ff-only origin/master Already up-to-date. + git gc + patchCommandPath=/data/hive-ptest/working/scratch/smart-apply-patch.sh + patchFilePath=/data/hive-ptest/working/scratch/build.patch + [[ -f /data/hive-ptest/working/scratch/build.patch ]] + chmod +x /data/hive-ptest/working/scratch/smart-apply-patch.sh + /data/hive-ptest/working/scratch/smart-apply-patch.sh /data/hive-ptest/working/scratch/build.patch The patch does not appear to apply with p0, p1, or p2 + exit 1 ' {noformat} This message is automatically generated. ATTACHMENT ID: 12816891 - PreCommit-HIVE-MASTER-Build > better concurrency handling for TezSessionState - part I > > > Key: HIVE-14111 > URL: https://issues.apache.org/jira/browse/HIVE-14111 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > Attachments: HIVE-14111.01.patch, HIVE-14111.02.patch, > HIVE-14111.03.patch, HIVE-14111.04.patch, HIVE-14111.05.patch, > HIVE-14111.patch, sessionPoolNotes.txt > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-14111) better concurrency handling for TezSessionState - part I
[ https://issues.apache.org/jira/browse/HIVE-14111?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15366550#comment-15366550 ] Siddharth Seth commented on HIVE-14111: --- Lets wait for HiveQA - it's identified a bunch of issues already. That said, it seems to be identifying issues with QTestUtil rather than the main code. Hopefully some tests are covering the main flow. > better concurrency handling for TezSessionState - part I > > > Key: HIVE-14111 > URL: https://issues.apache.org/jira/browse/HIVE-14111 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > Attachments: HIVE-14111.01.patch, HIVE-14111.02.patch, > HIVE-14111.03.patch, HIVE-14111.04.patch, HIVE-14111.patch, > sessionPoolNotes.txt > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-14111) better concurrency handling for TezSessionState - part I
[ https://issues.apache.org/jira/browse/HIVE-14111?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15365552#comment-15365552 ] Sergey Shelukhin commented on HIVE-14111: - HiveQA is dead for now. [~sseth] do you like the latest patch? :) > better concurrency handling for TezSessionState - part I > > > Key: HIVE-14111 > URL: https://issues.apache.org/jira/browse/HIVE-14111 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > Attachments: HIVE-14111.01.patch, HIVE-14111.02.patch, > HIVE-14111.03.patch, HIVE-14111.04.patch, HIVE-14111.patch, > sessionPoolNotes.txt > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-14111) better concurrency handling for TezSessionState - part I
[ https://issues.apache.org/jira/browse/HIVE-14111?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15359906#comment-15359906 ] Hive QA commented on HIVE-14111: Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12815562/HIVE-14111.03.patch {color:green}SUCCESS:{color} +1 due to 2 test(s) being added or modified. {color:red}ERROR:{color} -1 due to 70 failed/errored test(s), 10287 tests executed *Failed tests:* {noformat} org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_list_bucket_dml_13 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_stats_list_bucket org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_subquery_multiinsert org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_vector_complex_all org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_vector_complex_join org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_dynamic_partition_pruning org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_script_env_var1 org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_tez_fsstat org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_union9 org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_update_two_cols org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_vector_interval_2 org.apache.hadoop.hive.cli.TestPerfCliDriver.testPerfCliDriver_query13 org.apache.hadoop.hive.cli.TestPerfCliDriver.testPerfCliDriver_query15 org.apache.hadoop.hive.cli.TestPerfCliDriver.testPerfCliDriver_query17 org.apache.hadoop.hive.cli.TestPerfCliDriver.testPerfCliDriver_query18 org.apache.hadoop.hive.cli.TestPerfCliDriver.testPerfCliDriver_query19 org.apache.hadoop.hive.cli.TestPerfCliDriver.testPerfCliDriver_query20 org.apache.hadoop.hive.cli.TestPerfCliDriver.testPerfCliDriver_query21 org.apache.hadoop.hive.cli.TestPerfCliDriver.testPerfCliDriver_query22 org.apache.hadoop.hive.cli.TestPerfCliDriver.testPerfCliDriver_query25 org.apache.hadoop.hive.cli.TestPerfCliDriver.testPerfCliDriver_query26 org.apache.hadoop.hive.cli.TestPerfCliDriver.testPerfCliDriver_query27 org.apache.hadoop.hive.cli.TestPerfCliDriver.testPerfCliDriver_query28 org.apache.hadoop.hive.cli.TestPerfCliDriver.testPerfCliDriver_query29 org.apache.hadoop.hive.cli.TestPerfCliDriver.testPerfCliDriver_query3 org.apache.hadoop.hive.cli.TestPerfCliDriver.testPerfCliDriver_query31 org.apache.hadoop.hive.cli.TestPerfCliDriver.testPerfCliDriver_query32 org.apache.hadoop.hive.cli.TestPerfCliDriver.testPerfCliDriver_query34 org.apache.hadoop.hive.cli.TestPerfCliDriver.testPerfCliDriver_query39 org.apache.hadoop.hive.cli.TestPerfCliDriver.testPerfCliDriver_query40 org.apache.hadoop.hive.cli.TestPerfCliDriver.testPerfCliDriver_query42 org.apache.hadoop.hive.cli.TestPerfCliDriver.testPerfCliDriver_query43 org.apache.hadoop.hive.cli.TestPerfCliDriver.testPerfCliDriver_query45 org.apache.hadoop.hive.cli.TestPerfCliDriver.testPerfCliDriver_query46 org.apache.hadoop.hive.cli.TestPerfCliDriver.testPerfCliDriver_query48 org.apache.hadoop.hive.cli.TestPerfCliDriver.testPerfCliDriver_query50 org.apache.hadoop.hive.cli.TestPerfCliDriver.testPerfCliDriver_query51 org.apache.hadoop.hive.cli.TestPerfCliDriver.testPerfCliDriver_query52 org.apache.hadoop.hive.cli.TestPerfCliDriver.testPerfCliDriver_query54 org.apache.hadoop.hive.cli.TestPerfCliDriver.testPerfCliDriver_query55 org.apache.hadoop.hive.cli.TestPerfCliDriver.testPerfCliDriver_query58 org.apache.hadoop.hive.cli.TestPerfCliDriver.testPerfCliDriver_query64 org.apache.hadoop.hive.cli.TestPerfCliDriver.testPerfCliDriver_query65 org.apache.hadoop.hive.cli.TestPerfCliDriver.testPerfCliDriver_query66 org.apache.hadoop.hive.cli.TestPerfCliDriver.testPerfCliDriver_query67 org.apache.hadoop.hive.cli.TestPerfCliDriver.testPerfCliDriver_query68 org.apache.hadoop.hive.cli.TestPerfCliDriver.testPerfCliDriver_query7 org.apache.hadoop.hive.cli.TestPerfCliDriver.testPerfCliDriver_query70 org.apache.hadoop.hive.cli.TestPerfCliDriver.testPerfCliDriver_query71 org.apache.hadoop.hive.cli.TestPerfCliDriver.testPerfCliDriver_query72 org.apache.hadoop.hive.cli.TestPerfCliDriver.testPerfCliDriver_query73 org.apache.hadoop.hive.cli.TestPerfCliDriver.testPerfCliDriver_query75 org.apache.hadoop.hive.cli.TestPerfCliDriver.testPerfCliDriver_query76 org.apache.hadoop.hive.cli.TestPerfCliDriver.testPerfCliDriver_query79 org.apache.hadoop.hive.cli.TestPerfCliDriver.testPerfCliDriver_query80 org.apache.hadoop.hive.cli.TestPerfCliDriver.testPerfCliDriver_query82 org.apache.hadoop.hive.cli.TestPerfCliDriver.testPerfCliDriver_query84 org.apache.hadoop.hive.cli.TestPerfCliDriver.testPerfCliDriver_query85 org.apache.hadoop.hive.cli.TestPerfCliDriver.testPerfCliDriver_query87 org.apache.hadoop.hive.cli.TestPerfCliDriver.testPerfCliDriver_query88 org.apache.hadoop.hive.cli.TestPerfCliDriver.testPerfCliDriver_query89
[jira] [Commented] (HIVE-14111) better concurrency handling for TezSessionState - part I
[ https://issues.apache.org/jira/browse/HIVE-14111?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15357992#comment-15357992 ] Sergey Shelukhin commented on HIVE-14111: - [~sseth] I don't think that's a problem per current code... some other code reuses the object on failure too. So, the session will be closed and nulled out but the same object could be reused and would be reopened later in a different TezTask, right? I added a comment suggesting that we remove that pattern and only use the object once, we can do that in a follow-up > better concurrency handling for TezSessionState - part I > > > Key: HIVE-14111 > URL: https://issues.apache.org/jira/browse/HIVE-14111 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > Attachments: HIVE-14111.01.patch, HIVE-14111.02.patch, > HIVE-14111.03.patch, HIVE-14111.patch, sessionPoolNotes.txt > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-14111) better concurrency handling for TezSessionState - part I
[ https://issues.apache.org/jira/browse/HIVE-14111?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15357921#comment-15357921 ] Siddharth Seth commented on HIVE-14111: --- [~sershe] - I didn't meant to add complexity to the patch. The one I attached contained comments from me walking through the code - some related to this jira, other comments in general. We could check in the comments (which are added in your second patch). It would be better to create jiras to track them though - especially ones which you think are valid (I think some of the comments in the file I uploaded were questions, and some may not have been valid). bq. What is session.destroy? Rather what object is this; do you mean TezClient::stop? In TezTask.submit -> TezSessionPoolManager.getInstance().destroySession(sessionState); I think that's the main bit which needs to be looked at from your original patch, to get past the core problem here of a session getting lost in case of an error. > better concurrency handling for TezSessionState - part I > > > Key: HIVE-14111 > URL: https://issues.apache.org/jira/browse/HIVE-14111 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > Attachments: HIVE-14111.01.patch, HIVE-14111.02.patch, > HIVE-14111.patch, sessionPoolNotes.txt > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-14111) better concurrency handling for TezSessionState - part I
[ https://issues.apache.org/jira/browse/HIVE-14111?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15357198#comment-15357198 ] Hive QA commented on HIVE-14111: Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12815268/HIVE-14111.02.patch {color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified. {color:red}ERROR:{color} -1 due to 441 failed/errored test(s), 10286 tests executed *Failed tests:* {noformat} org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_list_bucket_dml_12 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_schema_evol_orc_nonvec_mapwork_part org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_stats_list_bucket org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_subquery_multiinsert org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_bucket_map_join_tez1 org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_bucket_map_join_tez2 org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_constprog_dpp org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_cte_1 org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_cte_2 org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_cte_3 org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_cte_4 org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_cte_5 org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_cte_mat_1 org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_cte_mat_2 org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_cte_mat_3 org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_cte_mat_4 org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_cte_mat_5 org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_dynamic_partition_pruning org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_dynamic_partition_pruning_2 org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_hybridgrace_hashjoin_1 org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_hybridgrace_hashjoin_2 org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_llap_nullscan org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_llap_udf org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_llapdecider org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_lvj_mapjoin org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_mapjoin_decimal org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_mrr org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_orc_llap_nonvector org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_orc_ppd_basic org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_tez_bmj_schema_evolution org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_tez_dml org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_tez_dynpart_hashjoin_1 org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_tez_dynpart_hashjoin_2 org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_tez_fsstat org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_tez_insert_overwrite_local_directory_1 org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_tez_join org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_tez_join_hash org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_tez_join_result_complex org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_tez_join_tests org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_tez_joins_explain org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_tez_multi_union org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_tez_schema_evolution org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_tez_self_join org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_tez_smb_1 org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_tez_smb_main org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_tez_union org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_tez_union2 org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_tez_union_decimal org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_tez_union_dynamic_partition org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_tez_union_group_by org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_tez_union_multiinsert org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_tez_vector_dynpart_hashjoin_1 org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_tez_vector_dynpart_hashjoin_2 org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_vector_complex_all org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_vector_complex_join org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_vector_include_no_sel
[jira] [Commented] (HIVE-14111) better concurrency handling for TezSessionState - part I
[ https://issues.apache.org/jira/browse/HIVE-14111?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15353807#comment-15353807 ] Sergey Shelukhin commented on HIVE-14111: - Wrt close(), I think it should be ok. Right now the session object is often being reused after close; it should work with that. I added another TODO > better concurrency handling for TezSessionState - part I > > > Key: HIVE-14111 > URL: https://issues.apache.org/jira/browse/HIVE-14111 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > Attachments: HIVE-14111.patch, sessionPoolNotes.txt > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-14111) better concurrency handling for TezSessionState - part I
[ https://issues.apache.org/jira/browse/HIVE-14111?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15353780#comment-15353780 ] Sergey Shelukhin commented on HIVE-14111: - {noformat} // TODO Lots of exceptions can come out of this one invocation. If not returned in the finally block - things blow up. {noformat} It's inside the try...block. What do you mean? > better concurrency handling for TezSessionState - part I > > > Key: HIVE-14111 > URL: https://issues.apache.org/jira/browse/HIVE-14111 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > Attachments: HIVE-14111.patch, sessionPoolNotes.txt > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-14111) better concurrency handling for TezSessionState - part I
[ https://issues.apache.org/jira/browse/HIVE-14111?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15353787#comment-15353787 ] Sergey Shelukhin commented on HIVE-14111: - What is session.destroy? Rather what object is this; do you mean TezClient::stop? > better concurrency handling for TezSessionState - part I > > > Key: HIVE-14111 > URL: https://issues.apache.org/jira/browse/HIVE-14111 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > Attachments: HIVE-14111.patch, sessionPoolNotes.txt > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-14111) better concurrency handling for TezSessionState - part I
[ https://issues.apache.org/jira/browse/HIVE-14111?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15353774#comment-15353774 ] Sergey Shelukhin commented on HIVE-14111: - {noformat} // TODO What happens in this case - no new sessions launched ? {noformat} Yes. > better concurrency handling for TezSessionState - part I > > > Key: HIVE-14111 > URL: https://issues.apache.org/jira/browse/HIVE-14111 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > Attachments: HIVE-14111.patch, sessionPoolNotes.txt > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-14111) better concurrency handling for TezSessionState - part I
[ https://issues.apache.org/jira/browse/HIVE-14111?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15353770#comment-15353770 ] Sergey Shelukhin commented on HIVE-14111: - {noformat} if (session == null || conf == null) { // TODO Will the session being passed in here be null if this is being used from the set of default sessions. // Really don't see why - since SessionState can be re-used - in which case the session will be re-used. return false; } {noformat} Because it's supposed to be returned to the pool and unset. Same for some other comments > better concurrency handling for TezSessionState - part I > > > Key: HIVE-14111 > URL: https://issues.apache.org/jira/browse/HIVE-14111 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > Attachments: HIVE-14111.patch, sessionPoolNotes.txt > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-14111) better concurrency handling for TezSessionState - part I
[ https://issues.apache.org/jira/browse/HIVE-14111?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15353766#comment-15353766 ] Sergey Shelukhin commented on HIVE-14111: - Actually I think there are many tangentially related TODOs that can be addressed separately. > better concurrency handling for TezSessionState - part I > > > Key: HIVE-14111 > URL: https://issues.apache.org/jira/browse/HIVE-14111 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > Attachments: HIVE-14111.patch, sessionPoolNotes.txt > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-14111) better concurrency handling for TezSessionState - part I
[ https://issues.apache.org/jira/browse/HIVE-14111?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15353762#comment-15353762 ] Sergey Shelukhin commented on HIVE-14111: - {noformat} // TODO Define a helper class which can accept and validate potential values HIVE_EXECUTION_MODE("hive.execution.mode", "container", new StringSet("container", "llap"), "Chooses whether query fragments will run in container or in llap"), {noformat} This is completely unrelated. I'll just ignore other unrelated comments here ;) > better concurrency handling for TezSessionState - part I > > > Key: HIVE-14111 > URL: https://issues.apache.org/jira/browse/HIVE-14111 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > Attachments: HIVE-14111.patch, sessionPoolNotes.txt > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-14111) better concurrency handling for TezSessionState - part I
[ https://issues.apache.org/jira/browse/HIVE-14111?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15353605#comment-15353605 ] Sergey Shelukhin commented on HIVE-14111: - No, all changes are part of this patch... TODO comment removal needs to be done in some patch, that's the only unrelated one > better concurrency handling for TezSessionState - part I > > > Key: HIVE-14111 > URL: https://issues.apache.org/jira/browse/HIVE-14111 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > Attachments: HIVE-14111.patch, sessionPoolNotes.txt > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-14111) better concurrency handling for TezSessionState - part I
[ https://issues.apache.org/jira/browse/HIVE-14111?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15352351#comment-15352351 ] Siddharth Seth commented on HIVE-14111: --- FileSinkOperator, Operation, MetadataOperation, SQLOperation changes carried forward from some other patch ? The try { } finally {} in TezTask needs to deal with one special condition. An attempt to submit() dag can result in session.destroy being invoked - in which case I believe the session is lost again, and will never be added back to the queue. Also - while going through this code, I noticed some other interesting bits. Will post that as a diff - let me know what you think. Definitely seems like a part2 will be required, and unit tests. > better concurrency handling for TezSessionState - part I > > > Key: HIVE-14111 > URL: https://issues.apache.org/jira/browse/HIVE-14111 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > Attachments: HIVE-14111.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-14111) better concurrency handling for TezSessionState - part I
[ https://issues.apache.org/jira/browse/HIVE-14111?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15351988#comment-15351988 ] Sergey Shelukhin commented on HIVE-14111: - [~sseth] fyi > better concurrency handling for TezSessionState - part I > > > Key: HIVE-14111 > URL: https://issues.apache.org/jira/browse/HIVE-14111 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > Attachments: HIVE-14111.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)