[jira] [Commented] (HIVE-13240) GroupByOperator: Drop the hash aggregates when closing operator
[ https://issues.apache.org/jira/browse/HIVE-13240?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15186370#comment-15186370 ] Sergey Shelukhin commented on HIVE-13240: - +1 > GroupByOperator: Drop the hash aggregates when closing operator > --- > > Key: HIVE-13240 > URL: https://issues.apache.org/jira/browse/HIVE-13240 > Project: Hive > Issue Type: Bug > Components: Query Processor >Affects Versions: 1.3.0, 1.2.1, 2.0.0 >Reporter: Gopal V >Assignee: Gopal V > Attachments: HIVE-13240.1.patch > > > GroupByOperator holds onto the Hash aggregates accumulated when the plan is > cached. > Drop the hashAggregates in case of error during forwarding to the next > operator. > Added for PTF, TopN and all GroupBy cases. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13240) GroupByOperator: Drop the hash aggregates when closing operator
[ https://issues.apache.org/jira/browse/HIVE-13240?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15186376#comment-15186376 ] Gopal V commented on HIVE-13240: [~wyukawa]: take a look at this patch for your Tez OOM issue. My primary suspect is that you need HIVE-10529 to fix that issue, but to confirm this issue or the other - you need to confirm that this issue goes away when you do hive --hiveconf tez.am.container.reuse.enabled=false; (note, has to be in the cli --hiveconf, doing set in CLI doesn't work) > GroupByOperator: Drop the hash aggregates when closing operator > --- > > Key: HIVE-13240 > URL: https://issues.apache.org/jira/browse/HIVE-13240 > Project: Hive > Issue Type: Bug > Components: Query Processor >Affects Versions: 1.3.0, 1.2.1, 2.0.0 >Reporter: Gopal V >Assignee: Gopal V > Attachments: HIVE-13240.1.patch > > > GroupByOperator holds onto the Hash aggregates accumulated when the plan is > cached. > Drop the hashAggregates in case of error during forwarding to the next > operator. > Added for PTF, TopN and all GroupBy cases. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13240) GroupByOperator: Drop the hash aggregates when closing operator
[ https://issues.apache.org/jira/browse/HIVE-13240?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15186443#comment-15186443 ] Wataru Yukawa commented on HIVE-13240: -- Thank you for your comment. But I switch to Hive on MapReduce in production environment, so now it's difficult to test. > GroupByOperator: Drop the hash aggregates when closing operator > --- > > Key: HIVE-13240 > URL: https://issues.apache.org/jira/browse/HIVE-13240 > Project: Hive > Issue Type: Bug > Components: Query Processor >Affects Versions: 1.3.0, 1.2.1, 2.0.0 >Reporter: Gopal V >Assignee: Gopal V > Attachments: HIVE-13240.1.patch > > > GroupByOperator holds onto the Hash aggregates accumulated when the plan is > cached. > Drop the hashAggregates in case of error during forwarding to the next > operator. > Added for PTF, TopN and all GroupBy cases. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13240) GroupByOperator: Drop the hash aggregates when closing operator
[ https://issues.apache.org/jira/browse/HIVE-13240?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15191083#comment-15191083 ] Hive QA commented on HIVE-13240: Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12792174/HIVE-13240.1.patch {color:red}ERROR:{color} -1 due to no test(s) being added or modified. {color:red}ERROR:{color} -1 due to 1921 failed/errored test(s), 9206 tests executed *Failed tests:* {noformat} TestCliDriver-columnStatsUpdateForStatsOptimizer_2.q-auto_sortmerge_join_13.q-alter_partition_clusterby_sortby.q-and-12-more - did not produce a TEST-*.xml file TestCliDriver-skewjoin_mapjoin11.q-udf_least.q-sample2.q-and-12-more - did not produce a TEST-*.xml file TestJdbcWithMiniHS2 - did not produce a TEST-*.xml file TestSparkCliDriver-auto_join11.q-vector_groupby_3.q-smb_mapjoin_8.q-and-3-more - did not produce a TEST-*.xml file TestSparkCliDriver-auto_join18.q-union_remove_23.q-input1_limit.q-and-12-more - did not produce a TEST-*.xml file TestSparkCliDriver-auto_join30.q-scriptfile1.q-join_reorder3.q-and-12-more - did not produce a TEST-*.xml file TestSparkCliDriver-auto_join9.q-bucketmapjoin10.q-skewjoinopt19.q-and-12-more - did not produce a TEST-*.xml file TestSparkCliDriver-auto_join_reordering_values.q-auto_sortmerge_join_7.q-multigroupby_singlemr.q-and-12-more - did not produce a TEST-*.xml file TestSparkCliDriver-avro_decimal_native.q-bucketmapjoin12.q-ppd_outer_join2.q-and-12-more - did not produce a TEST-*.xml file TestSparkCliDriver-avro_joins.q-list_bucket_dml_2.q-groupby_ppr.q-and-12-more - did not produce a TEST-*.xml file TestSparkCliDriver-bucketsortoptimize_insert_7.q-enforce_order.q-join36.q-and-12-more - did not produce a TEST-*.xml file TestSparkCliDriver-escape_distributeby1.q-union_remove_7.q-skewjoin_union_remove_2.q-and-12-more - did not produce a TEST-*.xml file TestSparkCliDriver-groupby2_noskew_multi_distinct.q-skewjoin_noskew.q-vector_data_types.q-and-12-more - did not produce a TEST-*.xml file TestSparkCliDriver-groupby3_map.q-sample2.q-auto_join14.q-and-12-more - did not produce a TEST-*.xml file TestSparkCliDriver-groupby4.q-timestamp_null.q-auto_join23.q-and-12-more - did not produce a TEST-*.xml file TestSparkCliDriver-groupby_complex_types.q-vectorization_10.q-join4.q-and-12-more - did not produce a TEST-*.xml file TestSparkCliDriver-groupby_grouping_id2.q-bucketmapjoin4.q-groupby7.q-and-12-more - did not produce a TEST-*.xml file TestSparkCliDriver-groupby_map_ppr_multi_distinct.q-table_access_keys_stats.q-groupby4_noskew.q-and-12-more - did not produce a TEST-*.xml file TestSparkCliDriver-join13.q-auto_join24.q-vector_decimal_aggregate.q-and-12-more - did not produce a TEST-*.xml file TestSparkCliDriver-join2.q-script_pipe.q-groupby_map_ppr.q-and-12-more - did not produce a TEST-*.xml file TestSparkCliDriver-join39.q-stats12.q-union27.q-and-12-more - did not produce a TEST-*.xml file TestSparkCliDriver-join_cond_pushdown_3.q-groupby7_noskew.q-auto_join13.q-and-12-more - did not produce a TEST-*.xml file TestSparkCliDriver-join_cond_pushdown_unqual4.q-vectorization_16.q-groupby10.q-and-12-more - did not produce a TEST-*.xml file TestSparkCliDriver-join_rc.q-insert1.q-vectorized_rcfile_columnar.q-and-12-more - did not produce a TEST-*.xml file TestSparkCliDriver-load_dyn_part12.q-nullgroup4_multi_distinct.q-union14.q-and-12-more - did not produce a TEST-*.xml file TestSparkCliDriver-load_dyn_part5.q-skewjoinopt8.q-groupby1_noskew.q-and-12-more - did not produce a TEST-*.xml file TestSparkCliDriver-parallel_join0.q-union_remove_9.q-smb_mapjoin_21.q-and-12-more - did not produce a TEST-*.xml file TestSparkCliDriver-parallel_join1.q-ptf_general_queries.q-ptf_rcfile.q-and-12-more - did not produce a TEST-*.xml file TestSparkCliDriver-ppd_gby_join.q-stats2.q-groupby_rollup1.q-and-12-more - did not produce a TEST-*.xml file TestSparkCliDriver-ppd_join4.q-join9.q-ppd_join3.q-and-12-more - did not produce a TEST-*.xml file TestSparkCliDriver-ppd_transform.q-ptf_seqfile.q-join11.q-and-12-more - did not produce a TEST-*.xml file TestSparkCliDriver-skewjoinopt15.q-bucketmapjoin3.q-auto_join10.q-and-12-more - did not produce a TEST-*.xml file TestSparkCliDriver-skewjoinopt16.q-join18.q-union11.q-and-12-more - did not produce a TEST-*.xml file TestSparkCliDriver-skewjoinopt3.q-multi_insert_mixed.q-runtime_skewjoin_mapjoin_spark.q-and-12-more - did not produce a TEST-*.xml file TestSparkCliDriver-smb_mapjoin_15.q-auto_sortmerge_join_13.q-auto_join18_multi_distinct.q-and-12-more - did not produce a TEST-*.xml file TestSparkCliDriver-smb_mapjoin_4.q-auto_join19.q-mapreduce1.q-and-12-more - did not produce a TEST-*.xml file TestSparkCliDriver-stats13.q-groupby6_map.q-join_casesensitive.q-and-12-more - did not produce a TEST-*.xml file TestSparkCliDriver-tez_joins_explain.q-input17.q-union29.q-and-12
[jira] [Commented] (HIVE-13240) GroupByOperator: Drop the hash aggregates when closing operator
[ https://issues.apache.org/jira/browse/HIVE-13240?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15203468#comment-15203468 ] Hive QA commented on HIVE-13240: Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12794151/HIVE-13240.2.patch {color:red}ERROR:{color} -1 due to no test(s) being added or modified. {color:red}ERROR:{color} -1 due to 20 failed/errored test(s), 9838 tests executed *Failed tests:* {noformat} TestSparkCliDriver-groupby3_map.q-sample2.q-auto_join14.q-and-12-more - did not produce a TEST-*.xml file TestSparkCliDriver-groupby_map_ppr_multi_distinct.q-table_access_keys_stats.q-groupby4_noskew.q-and-12-more - did not produce a TEST-*.xml file TestSparkCliDriver-join_rc.q-insert1.q-vectorized_rcfile_columnar.q-and-12-more - did not produce a TEST-*.xml file TestSparkCliDriver-ppd_join4.q-join9.q-ppd_join3.q-and-12-more - did not produce a TEST-*.xml file org.apache.hadoop.hive.cli.TestMiniTezCliDriver.org.apache.hadoop.hive.cli.TestMiniTezCliDriver org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_auto_join_nulls org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_delete_tmp_table org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_empty_join org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_explainuser_4 org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_mapreduce1 org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_orc_merge4 org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_orc_merge6 org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_schema_evol_text_mapwork_table org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_temp_table org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_tez_vector_dynpart_hashjoin_2 org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_transform1 org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_vector_decimal_round_2 org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_vector_decimal_udf org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_vector_string_concat org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_vectorized_distinct_gby {noformat} Test results: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/7321/testReport Console output: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/7321/console Test logs: http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-7321/ Messages: {noformat} Executing org.apache.hive.ptest.execution.TestCheckPhase Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase Tests exited with: TestsFailedException: 20 tests failed {noformat} This message is automatically generated. ATTACHMENT ID: 12794151 - PreCommit-HIVE-TRUNK-Build > GroupByOperator: Drop the hash aggregates when closing operator > --- > > Key: HIVE-13240 > URL: https://issues.apache.org/jira/browse/HIVE-13240 > Project: Hive > Issue Type: Bug > Components: Query Processor >Affects Versions: 1.3.0, 1.2.1, 2.0.0 >Reporter: Gopal V >Assignee: Gopal V > Attachments: HIVE-13240.1.patch, HIVE-13240.2.patch > > > GroupByOperator holds onto the Hash aggregates accumulated when the plan is > cached. > Drop the hashAggregates in case of error during forwarding to the next > operator. > Added for PTF, TopN and all GroupBy cases. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13240) GroupByOperator: Drop the hash aggregates when closing operator
[ https://issues.apache.org/jira/browse/HIVE-13240?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15234022#comment-15234022 ] Hive QA commented on HIVE-13240: Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12797623/HIVE-13240.03.patch {color:red}ERROR:{color} -1 due to no test(s) being added or modified. {color:red}ERROR:{color} -1 due to 49 failed/errored test(s), 9918 tests executed *Failed tests:* {noformat} TestMiniTezCliDriver-auto_join30.q-vector_data_types.q-tez_join.q-and-12-more - did not produce a TEST-*.xml file TestMiniTezCliDriver-vector_acid3.q-vector_decimal_trailing.q-lvj_mapjoin.q-and-12-more - did not produce a TEST-*.xml file TestMiniTezCliDriver-vector_interval_2.q-bucket3.q-vectorization_7.q-and-12-more - did not produce a TEST-*.xml file org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_ivyDownload org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_index_bitmap3 org.apache.hadoop.hive.cli.TestNegativeMinimrCliDriver.testNegativeCliDriver_minimr_broken_pipe org.apache.hadoop.hive.llap.tezplugins.TestLlapTaskSchedulerService.testForcedLocalityPreemption org.apache.hadoop.hive.metastore.TestHiveMetaStorePartitionSpecs.testAddPartitions org.apache.hadoop.hive.metastore.TestHiveMetaStorePartitionSpecs.testFetchingPartitionsWithDifferentSchemas org.apache.hadoop.hive.metastore.TestHiveMetaStorePartitionSpecs.testGetPartitionSpecs_WithAndWithoutPartitionGrouping org.apache.hadoop.hive.metastore.TestMetaStoreEndFunctionListener.testEndFunctionListener org.apache.hadoop.hive.metastore.TestMetaStoreInitListener.testMetaStoreInitListener org.apache.hadoop.hive.metastore.TestPartitionNameWhitelistValidation.testAddPartitionWithCommas org.apache.hadoop.hive.metastore.TestPartitionNameWhitelistValidation.testAddPartitionWithUnicode org.apache.hadoop.hive.metastore.TestPartitionNameWhitelistValidation.testAddPartitionWithValidPartVal org.apache.hadoop.hive.metastore.TestPartitionNameWhitelistValidation.testAppendPartitionWithCommas org.apache.hadoop.hive.metastore.TestPartitionNameWhitelistValidation.testAppendPartitionWithUnicode org.apache.hadoop.hive.metastore.TestPartitionNameWhitelistValidation.testAppendPartitionWithValidCharacters org.apache.hadoop.hive.ql.exec.tez.TestHostAffinitySplitLocationProvider.testOrcSplitsLocationAffinity org.apache.hadoop.hive.ql.lockmgr.TestDbTxnManager.concurrencyFalse org.apache.hadoop.hive.ql.lockmgr.TestDbTxnManager.testDDLExclusive org.apache.hadoop.hive.ql.lockmgr.TestDbTxnManager.testDelete org.apache.hadoop.hive.ql.lockmgr.TestDbTxnManager.testLockTimeout org.apache.hadoop.hive.ql.lockmgr.TestDbTxnManager.testRollback org.apache.hadoop.hive.ql.lockmgr.TestDbTxnManager.testSingleReadPartition org.apache.hadoop.hive.ql.lockmgr.TestDbTxnManager.testSingleWriteTable org.apache.hadoop.hive.ql.lockmgr.TestDbTxnManager.testUpdate org.apache.hadoop.hive.ql.lockmgr.TestDbTxnManager.testWriteDynamicPartition org.apache.hadoop.hive.ql.security.TestClientSideAuthorizationProvider.testSimplePrivileges org.apache.hadoop.hive.ql.security.TestFolderPermissions.org.apache.hadoop.hive.ql.security.TestFolderPermissions org.apache.hadoop.hive.ql.security.TestMetastoreAuthorizationProvider.testSimplePrivileges org.apache.hadoop.hive.ql.security.TestMultiAuthorizationPreEventListener.org.apache.hadoop.hive.ql.security.TestMultiAuthorizationPreEventListener org.apache.hadoop.hive.ql.security.TestStorageBasedClientSideAuthorizationProvider.testSimplePrivileges org.apache.hadoop.hive.ql.security.TestStorageBasedMetastoreAuthorizationDrops.testDropDatabase org.apache.hadoop.hive.ql.security.TestStorageBasedMetastoreAuthorizationDrops.testDropPartition org.apache.hadoop.hive.ql.security.TestStorageBasedMetastoreAuthorizationDrops.testDropTable org.apache.hadoop.hive.ql.security.TestStorageBasedMetastoreAuthorizationProvider.testSimplePrivileges org.apache.hadoop.hive.ql.security.TestStorageBasedMetastoreAuthorizationProviderWithACL.testSimplePrivileges org.apache.hadoop.hive.ql.security.TestStorageBasedMetastoreAuthorizationReads.testReadDbFailure org.apache.hadoop.hive.ql.security.TestStorageBasedMetastoreAuthorizationReads.testReadDbSuccess org.apache.hadoop.hive.ql.security.TestStorageBasedMetastoreAuthorizationReads.testReadTableFailure org.apache.hadoop.hive.thrift.TestHadoopAuthBridge23.testDelegationTokenSharedStore org.apache.hadoop.hive.thrift.TestHadoopAuthBridge23.testMetastoreProxyUser org.apache.hadoop.hive.thrift.TestHadoopAuthBridge23.testSaslWithHiveMetaStore org.apache.hive.hcatalog.api.repl.commands.TestCommands.org.apache.hive.hcatalog.api.repl.commands.TestCommands org.apache.hive.hcatalog.mapreduce.TestHCatPartitionPublish.org.apache.hive.hcatalog.mapreduce.TestHCatPartitionPublish org.apache.hive.jdbc.TestSSL.testSSLFetchHttp org.apache.hive.service.TestHS2ImpersonationWithRemoteMS.org.apache
[jira] [Commented] (HIVE-13240) GroupByOperator: Drop the hash aggregates when closing operator
[ https://issues.apache.org/jira/browse/HIVE-13240?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15234278#comment-15234278 ] Gopal V commented on HIVE-13240: Failed tests have NameNode errors {code} 2016-04-10T00:47:48,456 WARN [org.apache.hadoop.hdfs.server.namenode.FSNamesystem$NameNodeResourceMonitor@773c7b73[]]: namenode.NameNodeResourceChecker (NameNodeResourceChecker.java:isResourceAvailable(89)) - Space available on volume '/dev/xvde1' is 0, which is below the configured reserved amount 104857600 2016-04-10T00:47:48,456 WARN [org.apache.hadoop.hdfs.server.namenode.FSNamesystem$NameNodeResourceMonitor@773c7b73[]]: namenode.FSNamesystem (FSNamesystem.java:run(5159)) - NameNode low on available disk space. Entering safe mode. 2016-04-10T00:47:48,456 INFO [org.apache.hadoop.hdfs.server.namenode.FSNamesystem$NameNodeResourceMonitor@773c7b73[]]: hdfs.StateChange (FSNamesystem.java:reportStatus(6003)) - STATE* Safe mode is ON. Resources are low on NN. Please add or free up more resources then turn off safe mode manually. NOTE: If you turn off safe mode before adding resources, the NN will immediately return to safe mode. Use "hdfs dfsadmin -safemode leave" to turn safe mode off. 2016-04-10T00:47:54,612 WARN [LeaseRenewer:hiveptest@localhost:35563[]]: hdfs.LeaseRenewer (LeaseRenewer.java:run(458)) - Failed to renew lease for [DFSClient_NONMAPREDUCE_623273627_1] for 30 seconds. Will retry shortly ... org.apache.hadoop.ipc.RemoteException: Cannot renew lease for DFSClient_NONMAPREDUCE_623273627_1. Name node is in safe mode. Resources are low on NN. Please add or free up more resources then turn off safe mode manually. NOTE: If you turn off safe mode before adding resources, the NN will immediately return to safe mode. Use "hdfs dfsadmin -safemode leave" to turn safe mode off. {code} > GroupByOperator: Drop the hash aggregates when closing operator > --- > > Key: HIVE-13240 > URL: https://issues.apache.org/jira/browse/HIVE-13240 > Project: Hive > Issue Type: Bug > Components: Query Processor >Affects Versions: 1.3.0, 1.2.1, 2.0.0 >Reporter: Gopal V >Assignee: Gopal V > Attachments: HIVE-13240.03.patch, HIVE-13240.1.patch, > HIVE-13240.2.patch > > > GroupByOperator holds onto the Hash aggregates accumulated when the plan is > cached. > Drop the hashAggregates in case of error during forwarding to the next > operator. > Added for PTF, TopN and all GroupBy cases. -- This message was sent by Atlassian JIRA (v6.3.4#6332)