[jira] [Commented] (HIVE-13730) hybridgrace_hashjoin_1.q test gets stuck
[ https://issues.apache.org/jira/browse/HIVE-13730?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15289292#comment-15289292 ] Vikram Dixit K commented on HIVE-13730: --- +1 > hybridgrace_hashjoin_1.q test gets stuck > > > Key: HIVE-13730 > URL: https://issues.apache.org/jira/browse/HIVE-13730 > Project: Hive > Issue Type: Bug > Components: Tez >Affects Versions: 2.1.0 >Reporter: Vikram Dixit K >Assignee: Wei Zheng >Priority: Blocker > Attachments: HIVE-13730.1.patch, HIVE-13730.2.patch, > HIVE-13730.3.patch, HIVE-13730.4.patch > > > I am seeing hybridgrace_hashjoin_1.q getting stuck on master. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13730) hybridgrace_hashjoin_1.q test gets stuck
[ https://issues.apache.org/jira/browse/HIVE-13730?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15289258#comment-15289258 ] Wei Zheng commented on HIVE-13730: -- I ran the three sets of tests that failed to produce TEST-*.xml files, as commented above by Hive QA. All of them passed locally. So Should have nothing to do with patch 4. {code} enforce_order.q,vector_partition_diff_num_cols.q,unionDistinct_1.q,tez_smb_empty.q,vectorized_timestamp.q,vectorized_rcfile_columnar.q,tez_dml.q,vector_join_nulls.q,delete_tmp_table.q,schema_evol_orc_nonvec_fetchwork_part.q,vectorization_part_varchar.q,load_dyn_part1.q,auto_sortmerge_join_3.q,vector_reduce_groupby_decimal.q,union_type_chk.q groupby2.q,tez_dynpart_hashjoin_1.q,custom_input_output_format.q,schema_evol_orc_nonvec_fetchwork_table.q,schema_evol_orc_nonvec_mapwork_part_all_complex.q,tez_multi_union.q,vector_between_in.q,vector_char_4.q,dynamic_partition_pruning_2.q,vector_decimal_math_funcs.q,union7.q,vector_char_simple.q,auto_sortmerge_join_8.q,schema_evol_orc_nonvec_mapwork_table.q,merge2.q vectorization_16.q,vector_decimal_round.q,orc_merge6.q,vector_multi_insert.q,tez_union.q,vector_decimal_precision.q,alter_merge_2_orc.q,auto_sortmerge_join_14.q,vector_aggregate_9.q,vector_reduce1.q,vector_count_distinct.q,auto_join0.q,cross_join.q,vector_coalesce_2.q,vector_varchar_simple.q {code} > hybridgrace_hashjoin_1.q test gets stuck > > > Key: HIVE-13730 > URL: https://issues.apache.org/jira/browse/HIVE-13730 > Project: Hive > Issue Type: Bug > Components: Tez >Affects Versions: 2.1.0 >Reporter: Vikram Dixit K >Assignee: Wei Zheng >Priority: Blocker > Attachments: HIVE-13730.1.patch, HIVE-13730.2.patch, > HIVE-13730.3.patch, HIVE-13730.4.patch > > > I am seeing hybridgrace_hashjoin_1.q getting stuck on master. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13730) hybridgrace_hashjoin_1.q test gets stuck
[ https://issues.apache.org/jira/browse/HIVE-13730?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15289201#comment-15289201 ] Ashutosh Chauhan commented on HIVE-13730: - We don't have a good run for TestMiniLlapCliDriver & for few batches of TestMiniTezCliDriver for few weeks now. Shall we disable this test, if this gonna take more time to review and commit? cc: [~vikram.dixit] > hybridgrace_hashjoin_1.q test gets stuck > > > Key: HIVE-13730 > URL: https://issues.apache.org/jira/browse/HIVE-13730 > Project: Hive > Issue Type: Bug > Components: Tez >Affects Versions: 2.1.0 >Reporter: Vikram Dixit K >Assignee: Wei Zheng >Priority: Blocker > Attachments: HIVE-13730.1.patch, HIVE-13730.2.patch, > HIVE-13730.3.patch, HIVE-13730.4.patch > > > I am seeing hybridgrace_hashjoin_1.q getting stuck on master. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13730) hybridgrace_hashjoin_1.q test gets stuck
[ https://issues.apache.org/jira/browse/HIVE-13730?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15286958#comment-15286958 ] Wei Zheng commented on HIVE-13730: -- Test failures are not related. [~vikram.dixit] Can you review please? > hybridgrace_hashjoin_1.q test gets stuck > > > Key: HIVE-13730 > URL: https://issues.apache.org/jira/browse/HIVE-13730 > Project: Hive > Issue Type: Bug > Components: Tez >Affects Versions: 2.1.0 >Reporter: Vikram Dixit K >Assignee: Wei Zheng >Priority: Blocker > Attachments: HIVE-13730.1.patch, HIVE-13730.2.patch, > HIVE-13730.3.patch, HIVE-13730.4.patch > > > I am seeing hybridgrace_hashjoin_1.q getting stuck on master. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13730) hybridgrace_hashjoin_1.q test gets stuck
[ https://issues.apache.org/jira/browse/HIVE-13730?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15286393#comment-15286393 ] Hive QA commented on HIVE-13730: Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12804208/HIVE-13730.4.patch {color:red}ERROR:{color} -1 due to no test(s) being added or modified. {color:red}ERROR:{color} -1 due to 82 failed/errored test(s), 10042 tests executed *Failed tests:* {noformat} TestHWISessionManager - did not produce a TEST-*.xml file TestMiniTezCliDriver-enforce_order.q-vector_partition_diff_num_cols.q-unionDistinct_1.q-and-12-more - did not produce a TEST-*.xml file TestMiniTezCliDriver-groupby2.q-tez_dynpart_hashjoin_1.q-custom_input_output_format.q-and-12-more - did not produce a TEST-*.xml file TestMiniTezCliDriver-vectorization_16.q-vector_decimal_round.q-orc_merge6.q-and-12-more - did not produce a TEST-*.xml file org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_ivyDownload org.apache.hadoop.hive.cli.TestHBaseCliDriver.testCliDriver_hbase_queries org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_tez_join_result_complex org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_bucket4 org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_bucket5 org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_bucket6 org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_disable_merge_for_bucketing org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_index_bitmap3 org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_infer_bucket_sort_map_operators org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_infer_bucket_sort_num_buckets org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_infer_bucket_sort_reducers_power_two org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_list_bucket_dml_10 org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_orc_merge1 org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_orc_merge2 org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_orc_merge9 org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_orc_merge_diff_fs org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_reduce_deduplicate org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_vector_outer_join1 org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_vector_outer_join2 org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_vector_outer_join3 org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_vector_outer_join4 org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_vector_outer_join5 org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_auto_join0 org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_groupby5_noskew org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_groupby_complex_types org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_groupby_map_ppr_multi_distinct org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_innerjoin org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_input12 org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_input13 org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_input18 org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_input1_limit org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_join16 org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_join_alt_syntax org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_join_vc org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_load_dyn_part7 org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_multi_insert_gby3 org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_multi_insert_mixed org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_sample1 org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_union3 org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_union33 org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_vectorization_16 org.apache.hadoop.hive.llap.daemon.impl.TestLlapDaemonProtocolServerImpl.test org.apache.hadoop.hive.llap.daemon.impl.comparator.TestShortestJobFirstComparator.testWaitQueueComparatorWithinDagPriority org.apache.hadoop.hive.llap.tez.TestConverters.testFragmentSpecToTaskSpec org.apache.hadoop.hive.llap.tezplugins.TestLlapTaskCommunicator.testFinishableStateUpdateFailure org.apache.hadoop.hive.metastore.TestAuthzApiEmbedAuthorizerInRemote.org.apache.hadoop.hive.metastore.TestAuthzApiEmbedAuthorizerInRemote org.apache.hadoop.hive.metastore.TestFilterHooks.org.apache.hadoop.hive.metastore.TestFilterHooks org.apache.hadoop.hive.metastore.TestHiveMetaStorePartitionSpecs.org.apache.had
[jira] [Commented] (HIVE-13730) hybridgrace_hashjoin_1.q test gets stuck
[ https://issues.apache.org/jira/browse/HIVE-13730?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15283760#comment-15283760 ] Hive QA commented on HIVE-13730: Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12803966/HIVE-13730.3.patch {color:red}ERROR:{color} -1 due to build exiting with an error Test results: http://ec2-54-177-240-2.us-west-1.compute.amazonaws.com/job/PreCommit-HIVE-MASTER-Build/287/testReport Console output: http://ec2-54-177-240-2.us-west-1.compute.amazonaws.com/job/PreCommit-HIVE-MASTER-Build/287/console Test logs: http://ec2-50-18-27-0.us-west-1.compute.amazonaws.com/logs/PreCommit-HIVE-MASTER-Build-287/ Messages: {noformat} This message was trimmed, see log for full details [INFO] Excluding org.apache.spark:spark-core_2.10:jar:1.6.0 from the shaded jar. [INFO] Excluding com.twitter:chill_2.10:jar:0.5.0 from the shaded jar. [INFO] Excluding com.twitter:chill-java:jar:0.5.0 from the shaded jar. [INFO] Excluding org.apache.xbean:xbean-asm5-shaded:jar:4.4 from the shaded jar. [INFO] Excluding org.apache.hadoop:hadoop-client:jar:2.6.0 from the shaded jar. [INFO] Excluding org.apache.hadoop:hadoop-mapreduce-client-app:jar:2.6.0 from the shaded jar. [INFO] Excluding org.apache.hadoop:hadoop-mapreduce-client-shuffle:jar:2.6.0 from the shaded jar. [INFO] Excluding org.apache.hadoop:hadoop-mapreduce-client-jobclient:jar:2.6.0 from the shaded jar. [INFO] Excluding org.apache.spark:spark-launcher_2.10:jar:1.6.0 from the shaded jar. [INFO] Excluding org.apache.spark:spark-network-common_2.10:jar:1.6.0 from the shaded jar. [INFO] Excluding org.apache.spark:spark-network-shuffle_2.10:jar:1.6.0 from the shaded jar. [INFO] Excluding org.apache.spark:spark-unsafe_2.10:jar:1.6.0 from the shaded jar. [INFO] Excluding org.slf4j:jul-to-slf4j:jar:1.7.10 from the shaded jar. [INFO] Excluding org.slf4j:jcl-over-slf4j:jar:1.7.10 from the shaded jar. [INFO] Excluding com.ning:compress-lzf:jar:1.0.3 from the shaded jar. [INFO] Excluding net.jpountz.lz4:lz4:jar:1.3.0 from the shaded jar. [INFO] Excluding com.typesafe.akka:akka-remote_2.10:jar:2.3.11 from the shaded jar. [INFO] Excluding com.typesafe.akka:akka-actor_2.10:jar:2.3.11 from the shaded jar. [INFO] Excluding com.typesafe:config:jar:1.2.1 from the shaded jar. [INFO] Excluding org.uncommons.maths:uncommons-maths:jar:1.2.2a from the shaded jar. [INFO] Excluding com.typesafe.akka:akka-slf4j_2.10:jar:2.3.11 from the shaded jar. [INFO] Excluding org.scala-lang:scala-library:jar:2.10.4 from the shaded jar. [INFO] Excluding org.json4s:json4s-jackson_2.10:jar:3.2.10 from the shaded jar. [INFO] Excluding org.json4s:json4s-core_2.10:jar:3.2.10 from the shaded jar. [INFO] Excluding org.json4s:json4s-ast_2.10:jar:3.2.10 from the shaded jar. [INFO] Excluding org.scala-lang:scalap:jar:2.10.0 from the shaded jar. [INFO] Excluding org.scala-lang:scala-compiler:jar:2.10.0 from the shaded jar. [INFO] Excluding org.apache.mesos:mesos:jar:shaded-protobuf:0.21.1 from the shaded jar. [INFO] Excluding com.clearspring.analytics:stream:jar:2.7.0 from the shaded jar. [INFO] Excluding io.dropwizard.metrics:metrics-graphite:jar:3.1.2 from the shaded jar. [INFO] Excluding com.fasterxml.jackson.module:jackson-module-scala_2.10:jar:2.4.4 from the shaded jar. [INFO] Excluding org.scala-lang:scala-reflect:jar:2.10.4 from the shaded jar. [INFO] Excluding oro:oro:jar:2.0.8 from the shaded jar. [INFO] Excluding org.tachyonproject:tachyon-client:jar:0.8.2 from the shaded jar. [INFO] Excluding org.tachyonproject:tachyon-underfs-hdfs:jar:0.8.2 from the shaded jar. [INFO] Excluding org.tachyonproject:tachyon-underfs-s3:jar:0.8.2 from the shaded jar. [INFO] Excluding org.tachyonproject:tachyon-underfs-local:jar:0.8.2 from the shaded jar. [INFO] Excluding net.razorvine:pyrolite:jar:4.9 from the shaded jar. [INFO] Excluding net.sf.py4j:py4j:jar:0.9 from the shaded jar. [INFO] Excluding org.spark-project.spark:unused:jar:1.0.0 from the shaded jar. [INFO] Excluding org.slf4j:slf4j-api:jar:1.7.10 from the shaded jar. [INFO] Replacing original artifact with shaded artifact. [INFO] Replacing /data/hive-ptest/working/apache-github-source-source/ql/target/hive-exec-2.1.0-SNAPSHOT.jar with /data/hive-ptest/working/apache-github-source-source/ql/target/hive-exec-2.1.0-SNAPSHOT-shaded.jar [INFO] Dependency-reduced POM written at: /data/hive-ptest/working/apache-github-source-source/ql/dependency-reduced-pom.xml [INFO] Dependency-reduced POM written at: /data/hive-ptest/working/apache-github-source-source/ql/dependency-reduced-pom.xml [INFO] Dependency-reduced POM written at: /data/hive-ptest/working/apache-github-source-source/ql/dependency-reduced-pom.xml [INFO] Dependency-reduced POM written at: /data/hive-ptest/working/apache-github-source-source/ql/dependency-reduced-pom.xml [INFO] Dependency-reduced POM written at: /dat
[jira] [Commented] (HIVE-13730) hybridgrace_hashjoin_1.q test gets stuck
[ https://issues.apache.org/jira/browse/HIVE-13730?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15283217#comment-15283217 ] Wei Zheng commented on HIVE-13730: -- As I dig deeper, it turns out that the issue is actually due to spilling the same hash partition twice. In HybridHashTableContainer.internalPutRow, once isMemoryFull() returns true, we will pick the biggest partition in memory so far by calling biggestPartition(). This method is problematic. {code} private int biggestPartition() { int res = 0; int maxSize = 0; // If a partition has been spilled to disk, its size will be 0, i.e. it won't be picked for (int i = 0; i < hashPartitions.length; i++) { int size; if (isOnDisk(i)) { continue; } else { size = hashPartitions[i].hashMap.getNumValues(); } if (size > maxSize) { maxSize = size; res = i; } } return res; } {code} If all in-memory partitions have size 0, then the default initial value 0 will be returned. But what if partition 0 has already been spilled previously? This will spill partition 0 again, which is not expected. > hybridgrace_hashjoin_1.q test gets stuck > > > Key: HIVE-13730 > URL: https://issues.apache.org/jira/browse/HIVE-13730 > Project: Hive > Issue Type: Bug > Components: Tez >Affects Versions: 2.1.0 >Reporter: Vikram Dixit K >Assignee: Wei Zheng >Priority: Blocker > Attachments: HIVE-13730.1.patch, HIVE-13730.2.patch, > HIVE-13730.3.patch > > > I am seeing hybridgrace_hashjoin_1.q getting stuck on master. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13730) hybridgrace_hashjoin_1.q test gets stuck
[ https://issues.apache.org/jira/browse/HIVE-13730?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15282770#comment-15282770 ] Hive QA commented on HIVE-13730: Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12803806/HIVE-13730.2.patch {color:red}ERROR:{color} -1 due to no test(s) being added or modified. {color:red}ERROR:{color} -1 due to 63 failed/errored test(s), 10003 tests executed *Failed tests:* {noformat} TestHWISessionManager - did not produce a TEST-*.xml file TestMiniTezCliDriver-vector_grouping_sets.q-update_all_partitioned.q-cte_5.q-and-12-more - did not produce a TEST-*.xml file TestMiniTezCliDriver-vector_interval_2.q-schema_evol_text_nonvec_mapwork_part_all_primitive.q-tez_fsstat.q-and-12-more - did not produce a TEST-*.xml file TestMiniTezCliDriver-vectorization_16.q-vector_decimal_round.q-orc_merge6.q-and-12-more - did not produce a TEST-*.xml file TestMiniTezCliDriver-vectorized_parquet.q-insert_values_non_partitioned.q-schema_evol_orc_nonvec_mapwork_part.q-and-12-more - did not produce a TEST-*.xml file TestSparkCliDriver-skewjoinopt15.q-join39.q-avro_joins_native.q-and-12-more - did not produce a TEST-*.xml file TestSparkClient - did not produce a TEST-*.xml file org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_ivyDownload org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_tez_join_result_complex org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_bucket4 org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_bucket5 org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_bucket6 org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_disable_merge_for_bucketing org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_index_bitmap3 org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_infer_bucket_sort_map_operators org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_infer_bucket_sort_num_buckets org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_infer_bucket_sort_reducers_power_two org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_list_bucket_dml_10 org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_orc_merge1 org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_orc_merge2 org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_orc_merge9 org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_orc_merge_diff_fs org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_reduce_deduplicate org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_vector_outer_join1 org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_vector_outer_join2 org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_vector_outer_join3 org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_vector_outer_join4 org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_vector_outer_join5 org.apache.hadoop.hive.cli.TestMiniTezCliDriver.org.apache.hadoop.hive.cli.TestMiniTezCliDriver org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_dynamic_partition_pruning org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_auto_sortmerge_join_9 org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_bucketsortoptimize_insert_7 org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_mapreduce1 org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_smb_mapjoin_15 org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_union_remove_19 org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_union_remove_4 org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_union_remove_8 org.apache.hadoop.hive.llap.daemon.impl.TestTaskExecutorService.testPreemptionQueueComparator org.apache.hadoop.hive.llap.tez.TestConverters.testFragmentSpecToTaskSpec org.apache.hadoop.hive.llap.tezplugins.TestLlapTaskCommunicator.testFinishableStateUpdateFailure org.apache.hadoop.hive.metastore.TestAuthzApiEmbedAuthorizerInRemote.org.apache.hadoop.hive.metastore.TestAuthzApiEmbedAuthorizerInRemote org.apache.hadoop.hive.metastore.TestFilterHooks.org.apache.hadoop.hive.metastore.TestFilterHooks org.apache.hadoop.hive.metastore.TestHiveMetaStoreStatsMerge.testStatsMerge org.apache.hadoop.hive.metastore.TestMetaStoreEventListenerOnlyOnCommit.testEventStatus org.apache.hadoop.hive.metastore.TestMetaStoreInitListener.testMetaStoreInitListener org.apache.hadoop.hive.metastore.TestMetaStoreMetrics.org.apache.hadoop.hive.metastore.TestMetaStoreMetrics org.apache.hadoop.hive.metastore.TestPartitionNameWhitelistValidation.testAddPartitionWithValidPartVal org.apache.hadoop.hive.metastore.TestPartitionNameWhitelistValidation.testAppendPartitionWithCommas org.apache.hadoop.hive.metas
[jira] [Commented] (HIVE-13730) hybridgrace_hashjoin_1.q test gets stuck
[ https://issues.apache.org/jira/browse/HIVE-13730?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15282431#comment-15282431 ] Wei Zheng commented on HIVE-13730: -- Here's an todo item after HIVE-13755 is fixed. Right now memory manager doesn't guarantee to allocate enough memory for each table in n-way join case. After fixing that issue, this assert below can be put into HybridHashTableContainer's cstr after the variables have been determined. {code} assert writeBufferSize * (numPartitions - numPartitionsSpilledOnCreation) <= memoryThreshold : "hive.auto.convert.join.noconditionaltask.size is set too low. It's not enough to " + "allocate " + (numPartitions - numPartitionsSpilledOnCreation) + " partitions (each " + " of size " + writeBufferSize; {code} > hybridgrace_hashjoin_1.q test gets stuck > > > Key: HIVE-13730 > URL: https://issues.apache.org/jira/browse/HIVE-13730 > Project: Hive > Issue Type: Bug > Components: Tez >Affects Versions: 2.1.0 >Reporter: Vikram Dixit K >Assignee: Wei Zheng >Priority: Blocker > Attachments: HIVE-13730.1.patch > > > I am seeing hybridgrace_hashjoin_1.q getting stuck on master. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13730) hybridgrace_hashjoin_1.q test gets stuck
[ https://issues.apache.org/jira/browse/HIVE-13730?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15279707#comment-15279707 ] Wei Zheng commented on HIVE-13730: -- This issue is caused/exposed by HIVE-12837. > hybridgrace_hashjoin_1.q test gets stuck > > > Key: HIVE-13730 > URL: https://issues.apache.org/jira/browse/HIVE-13730 > Project: Hive > Issue Type: Bug > Components: Tez >Affects Versions: 2.1.0 >Reporter: Vikram Dixit K >Assignee: Wei Zheng >Priority: Blocker > Attachments: HIVE-13730.1.patch > > > I am seeing hybridgrace_hashjoin_1.q getting stuck on master. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13730) hybridgrace_hashjoin_1.q test gets stuck
[ https://issues.apache.org/jira/browse/HIVE-13730?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15279701#comment-15279701 ] Wei Zheng commented on HIVE-13730: -- The root cause is that in some cases (e.g. when hive.auto.convert.join.noconditionaltask.size is set very small), a hash partition can be empty when it's being spilled (the memory estimation logic is conservative and strict, so even without loading any row into a hash partition, it can still assume the memory is about to get full, thus choose a partition to spill). Still, spilling an empty hash partition is OK. The problem happens during deserialization of the spilled hash partition (BytesBytesMultiHashMap). If the hash partition is empty, it will result in the refs array in the hashmap to have a length of only 1. This causes problem of putRow as the backtrace above shows, because it couldn't find a proper slot for inserting. The solution is to instantiate a new BytesBytesMultiHashMap manually if we figure out the deserialized hashmap is empty. This way we can have a properly constructed refs array for it. > hybridgrace_hashjoin_1.q test gets stuck > > > Key: HIVE-13730 > URL: https://issues.apache.org/jira/browse/HIVE-13730 > Project: Hive > Issue Type: Bug > Components: Tez >Affects Versions: 2.1.0 >Reporter: Vikram Dixit K >Assignee: Wei Zheng >Priority: Blocker > > I am seeing hybridgrace_hashjoin_1.q getting stuck on master. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13730) hybridgrace_hashjoin_1.q test gets stuck
[ https://issues.apache.org/jira/browse/HIVE-13730?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15279098#comment-15279098 ] Wei Zheng commented on HIVE-13730: -- It's stuck in an infinite while loop in BytesBytesMultiHashMap.findKeySlotToWrite(). {code} $ jps 90673 TezChild 90976 TezChild 90855 TezChild 91225 Jps 82923 RemoteMavenServer 90205 surefirebooter3625226115924096543.jar 90191 Launcher 90542 DAGAppMaster $ jstack 90673 2016-05-10 15:13:47 Full thread dump Java HotSpot(TM) 64-Bit Server VM (25.74-b02 mixed mode): "Attach Listener" #138 daemon prio=9 os_prio=31 tid=0x7feea480 nid=0x3d3b waiting on condition [0x] java.lang.Thread.State: RUNNABLE "TezTaskEventRouter{attempt_1462916018098_0001_32_01_00_0}" #134 daemon prio=5 os_prio=31 tid=0x7feea684f000 nid=0x692f waiting on condition [0x71be7000] java.lang.Thread.State: WAITING (parking) at sun.misc.Unsafe.park(Native Method) - parking to wait for <0x0007bc9d6490> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) at org.apache.tez.runtime.LogicalIOProcessorRuntimeTask$1.runInternal(LogicalIOProcessorRuntimeTask.java:773) at org.apache.tez.common.RunnableWithNdc.run(RunnableWithNdc.java:35) at java.lang.Thread.run(Thread.java:745) "org.apache.hadoop.hdfs.PeerCache@35f41fc9" #22 daemon prio=5 os_prio=31 tid=0x7feea686d800 nid=0x6a03 waiting on condition [0x71cea000] java.lang.Thread.State: TIMED_WAITING (sleeping) at java.lang.Thread.sleep(Native Method) at org.apache.hadoop.hdfs.PeerCache.run(PeerCache.java:244) at org.apache.hadoop.hdfs.PeerCache.access$000(PeerCache.java:41) at org.apache.hadoop.hdfs.PeerCache$1.run(PeerCache.java:119) at java.lang.Thread.run(Thread.java:745) "TaskHeartbeatThread" #15 daemon prio=5 os_prio=31 tid=0x7feea310c000 nid=0x6403 waiting on condition [0x719e1000] java.lang.Thread.State: TIMED_WAITING (parking) at sun.misc.Unsafe.park(Native Method) - parking to wait for <0x0007bcb6aa40> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2163) at org.apache.tez.runtime.task.TaskReporter$HeartbeatCallable.call(TaskReporter.java:200) at org.apache.tez.runtime.task.TaskReporter$HeartbeatCallable.call(TaskReporter.java:128) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) "IPC Parameter Sending Thread #0" #14 daemon prio=5 os_prio=31 tid=0x7feea0979000 nid=0x6203 waiting on condition [0x718de000] java.lang.Thread.State: TIMED_WAITING (parking) at sun.misc.Unsafe.park(Native Method) - parking to wait for <0x00078df78428> (a java.util.concurrent.SynchronousQueue$TransferStack) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460) at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362) at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) "IPC Client (1617838096) connection to /10.22.27.129:64289 from application_1462916018098_0001" #13 daemon prio=5 os_prio=31 tid=0x7feea11f6800 nid=0x6003 in Object.wait() [0x717db000] java.lang.Thread.State: TIMED_WAITING (on object monitor) at java.lang.Object.wait(Native Method) at org.apache.hadoop.ipc.Client$Connection.waitForWork(Client.java:920) - locked <0x00078df52318> (a org.apache.hadoop.ipc.Client$Connection) at org.apache.hadoop.ipc.Client$Connection.run(Client.java:965) "TezChild" #12 daemon prio=5 os_prio=31 tid=0x7feea0a65000 nid=0x5e07 runnable [0x716d7000] java.lang.Thread.State: RUNNA
[jira] [Commented] (HIVE-13730) hybridgrace_hashjoin_1.q test gets stuck
[ https://issues.apache.org/jira/browse/HIVE-13730?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15278987#comment-15278987 ] Wei Zheng commented on HIVE-13730: -- Looking into it.. > hybridgrace_hashjoin_1.q test gets stuck > > > Key: HIVE-13730 > URL: https://issues.apache.org/jira/browse/HIVE-13730 > Project: Hive > Issue Type: Bug > Components: Tez >Affects Versions: 2.1.0 >Reporter: Vikram Dixit K >Assignee: Wei Zheng >Priority: Blocker > > I am seeing hybridgrace_hashjoin_1.q getting stuck on master. -- This message was sent by Atlassian JIRA (v6.3.4#6332)