[jira] [Commented] (HIVE-13240) GroupByOperator: Drop the hash aggregates when closing operator
[ https://issues.apache.org/jira/browse/HIVE-13240?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15186370#comment-15186370 ] Sergey Shelukhin commented on HIVE-13240: - +1 > GroupByOperator: Drop the hash aggregates when closing operator > --- > > Key: HIVE-13240 > URL: https://issues.apache.org/jira/browse/HIVE-13240 > Project: Hive > Issue Type: Bug > Components: Query Processor >Affects Versions: 1.3.0, 1.2.1, 2.0.0 >Reporter: Gopal V >Assignee: Gopal V > Attachments: HIVE-13240.1.patch > > > GroupByOperator holds onto the Hash aggregates accumulated when the plan is > cached. > Drop the hashAggregates in case of error during forwarding to the next > operator. > Added for PTF, TopN and all GroupBy cases. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13240) GroupByOperator: Drop the hash aggregates when closing operator
[ https://issues.apache.org/jira/browse/HIVE-13240?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15186376#comment-15186376 ] Gopal V commented on HIVE-13240: [~wyukawa]: take a look at this patch for your Tez OOM issue. My primary suspect is that you need HIVE-10529 to fix that issue, but to confirm this issue or the other - you need to confirm that this issue goes away when you do hive --hiveconf tez.am.container.reuse.enabled=false; (note, has to be in the cli --hiveconf, doing set in CLI doesn't work) > GroupByOperator: Drop the hash aggregates when closing operator > --- > > Key: HIVE-13240 > URL: https://issues.apache.org/jira/browse/HIVE-13240 > Project: Hive > Issue Type: Bug > Components: Query Processor >Affects Versions: 1.3.0, 1.2.1, 2.0.0 >Reporter: Gopal V >Assignee: Gopal V > Attachments: HIVE-13240.1.patch > > > GroupByOperator holds onto the Hash aggregates accumulated when the plan is > cached. > Drop the hashAggregates in case of error during forwarding to the next > operator. > Added for PTF, TopN and all GroupBy cases. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13240) GroupByOperator: Drop the hash aggregates when closing operator
[ https://issues.apache.org/jira/browse/HIVE-13240?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15186443#comment-15186443 ] Wataru Yukawa commented on HIVE-13240: -- Thank you for your comment. But I switch to Hive on MapReduce in production environment, so now it's difficult to test. > GroupByOperator: Drop the hash aggregates when closing operator > --- > > Key: HIVE-13240 > URL: https://issues.apache.org/jira/browse/HIVE-13240 > Project: Hive > Issue Type: Bug > Components: Query Processor >Affects Versions: 1.3.0, 1.2.1, 2.0.0 >Reporter: Gopal V >Assignee: Gopal V > Attachments: HIVE-13240.1.patch > > > GroupByOperator holds onto the Hash aggregates accumulated when the plan is > cached. > Drop the hashAggregates in case of error during forwarding to the next > operator. > Added for PTF, TopN and all GroupBy cases. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13240) GroupByOperator: Drop the hash aggregates when closing operator
[
https://issues.apache.org/jira/browse/HIVE-13240?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15191083#comment-15191083
]
Hive QA commented on HIVE-13240:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12792174/HIVE-13240.1.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 1921 failed/errored test(s), 9206 tests
executed
*Failed tests:*
{noformat}
TestCliDriver-columnStatsUpdateForStatsOptimizer_2.q-auto_sortmerge_join_13.q-alter_partition_clusterby_sortby.q-and-12-more
- did not produce a TEST-*.xml file
TestCliDriver-skewjoin_mapjoin11.q-udf_least.q-sample2.q-and-12-more - did not
produce a TEST-*.xml file
TestJdbcWithMiniHS2 - did not produce a TEST-*.xml file
TestSparkCliDriver-auto_join11.q-vector_groupby_3.q-smb_mapjoin_8.q-and-3-more
- did not produce a TEST-*.xml file
TestSparkCliDriver-auto_join18.q-union_remove_23.q-input1_limit.q-and-12-more -
did not produce a TEST-*.xml file
TestSparkCliDriver-auto_join30.q-scriptfile1.q-join_reorder3.q-and-12-more -
did not produce a TEST-*.xml file
TestSparkCliDriver-auto_join9.q-bucketmapjoin10.q-skewjoinopt19.q-and-12-more -
did not produce a TEST-*.xml file
TestSparkCliDriver-auto_join_reordering_values.q-auto_sortmerge_join_7.q-multigroupby_singlemr.q-and-12-more
- did not produce a TEST-*.xml file
TestSparkCliDriver-avro_decimal_native.q-bucketmapjoin12.q-ppd_outer_join2.q-and-12-more
- did not produce a TEST-*.xml file
TestSparkCliDriver-avro_joins.q-list_bucket_dml_2.q-groupby_ppr.q-and-12-more -
did not produce a TEST-*.xml file
TestSparkCliDriver-bucketsortoptimize_insert_7.q-enforce_order.q-join36.q-and-12-more
- did not produce a TEST-*.xml file
TestSparkCliDriver-escape_distributeby1.q-union_remove_7.q-skewjoin_union_remove_2.q-and-12-more
- did not produce a TEST-*.xml file
TestSparkCliDriver-groupby2_noskew_multi_distinct.q-skewjoin_noskew.q-vector_data_types.q-and-12-more
- did not produce a TEST-*.xml file
TestSparkCliDriver-groupby3_map.q-sample2.q-auto_join14.q-and-12-more - did not
produce a TEST-*.xml file
TestSparkCliDriver-groupby4.q-timestamp_null.q-auto_join23.q-and-12-more - did
not produce a TEST-*.xml file
TestSparkCliDriver-groupby_complex_types.q-vectorization_10.q-join4.q-and-12-more
- did not produce a TEST-*.xml file
TestSparkCliDriver-groupby_grouping_id2.q-bucketmapjoin4.q-groupby7.q-and-12-more
- did not produce a TEST-*.xml file
TestSparkCliDriver-groupby_map_ppr_multi_distinct.q-table_access_keys_stats.q-groupby4_noskew.q-and-12-more
- did not produce a TEST-*.xml file
TestSparkCliDriver-join13.q-auto_join24.q-vector_decimal_aggregate.q-and-12-more
- did not produce a TEST-*.xml file
TestSparkCliDriver-join2.q-script_pipe.q-groupby_map_ppr.q-and-12-more - did
not produce a TEST-*.xml file
TestSparkCliDriver-join39.q-stats12.q-union27.q-and-12-more - did not produce a
TEST-*.xml file
TestSparkCliDriver-join_cond_pushdown_3.q-groupby7_noskew.q-auto_join13.q-and-12-more
- did not produce a TEST-*.xml file
TestSparkCliDriver-join_cond_pushdown_unqual4.q-vectorization_16.q-groupby10.q-and-12-more
- did not produce a TEST-*.xml file
TestSparkCliDriver-join_rc.q-insert1.q-vectorized_rcfile_columnar.q-and-12-more
- did not produce a TEST-*.xml file
TestSparkCliDriver-load_dyn_part12.q-nullgroup4_multi_distinct.q-union14.q-and-12-more
- did not produce a TEST-*.xml file
TestSparkCliDriver-load_dyn_part5.q-skewjoinopt8.q-groupby1_noskew.q-and-12-more
- did not produce a TEST-*.xml file
TestSparkCliDriver-parallel_join0.q-union_remove_9.q-smb_mapjoin_21.q-and-12-more
- did not produce a TEST-*.xml file
TestSparkCliDriver-parallel_join1.q-ptf_general_queries.q-ptf_rcfile.q-and-12-more
- did not produce a TEST-*.xml file
TestSparkCliDriver-ppd_gby_join.q-stats2.q-groupby_rollup1.q-and-12-more - did
not produce a TEST-*.xml file
TestSparkCliDriver-ppd_join4.q-join9.q-ppd_join3.q-and-12-more - did not
produce a TEST-*.xml file
TestSparkCliDriver-ppd_transform.q-ptf_seqfile.q-join11.q-and-12-more - did not
produce a TEST-*.xml file
TestSparkCliDriver-skewjoinopt15.q-bucketmapjoin3.q-auto_join10.q-and-12-more -
did not produce a TEST-*.xml file
TestSparkCliDriver-skewjoinopt16.q-join18.q-union11.q-and-12-more - did not
produce a TEST-*.xml file
TestSparkCliDriver-skewjoinopt3.q-multi_insert_mixed.q-runtime_skewjoin_mapjoin_spark.q-and-12-more
- did not produce a TEST-*.xml file
TestSparkCliDriver-smb_mapjoin_15.q-auto_sortmerge_join_13.q-auto_join18_multi_distinct.q-and-12-more
- did not produce a TEST-*.xml file
TestSparkCliDriver-smb_mapjoin_4.q-auto_join19.q-mapreduce1.q-and-12-more - did
not produce a TEST-*.xml file
TestSparkCliDriver-stats13.q-groupby6_map.q-join_casesensitive.q-and-12-more -
did not produce a TEST-*.xml file
TestSparkCliDriver-tez_joins_explain.q-input17.q-union29.q-and-12
[jira] [Commented] (HIVE-13240) GroupByOperator: Drop the hash aggregates when closing operator
[
https://issues.apache.org/jira/browse/HIVE-13240?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15203468#comment-15203468
]
Hive QA commented on HIVE-13240:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12794151/HIVE-13240.2.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 20 failed/errored test(s), 9838 tests
executed
*Failed tests:*
{noformat}
TestSparkCliDriver-groupby3_map.q-sample2.q-auto_join14.q-and-12-more - did not
produce a TEST-*.xml file
TestSparkCliDriver-groupby_map_ppr_multi_distinct.q-table_access_keys_stats.q-groupby4_noskew.q-and-12-more
- did not produce a TEST-*.xml file
TestSparkCliDriver-join_rc.q-insert1.q-vectorized_rcfile_columnar.q-and-12-more
- did not produce a TEST-*.xml file
TestSparkCliDriver-ppd_join4.q-join9.q-ppd_join3.q-and-12-more - did not
produce a TEST-*.xml file
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.org.apache.hadoop.hive.cli.TestMiniTezCliDriver
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_auto_join_nulls
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_delete_tmp_table
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_empty_join
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_explainuser_4
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_mapreduce1
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_orc_merge4
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_orc_merge6
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_schema_evol_text_mapwork_table
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_temp_table
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_tez_vector_dynpart_hashjoin_2
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_transform1
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_vector_decimal_round_2
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_vector_decimal_udf
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_vector_string_concat
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_vectorized_distinct_gby
{noformat}
Test results:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/7321/testReport
Console output:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/7321/console
Test logs:
http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-7321/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 20 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12794151 - PreCommit-HIVE-TRUNK-Build
> GroupByOperator: Drop the hash aggregates when closing operator
> ---
>
> Key: HIVE-13240
> URL: https://issues.apache.org/jira/browse/HIVE-13240
> Project: Hive
> Issue Type: Bug
> Components: Query Processor
>Affects Versions: 1.3.0, 1.2.1, 2.0.0
>Reporter: Gopal V
>Assignee: Gopal V
> Attachments: HIVE-13240.1.patch, HIVE-13240.2.patch
>
>
> GroupByOperator holds onto the Hash aggregates accumulated when the plan is
> cached.
> Drop the hashAggregates in case of error during forwarding to the next
> operator.
> Added for PTF, TopN and all GroupBy cases.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-13240) GroupByOperator: Drop the hash aggregates when closing operator
[
https://issues.apache.org/jira/browse/HIVE-13240?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15234022#comment-15234022
]
Hive QA commented on HIVE-13240:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12797623/HIVE-13240.03.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 49 failed/errored test(s), 9918 tests
executed
*Failed tests:*
{noformat}
TestMiniTezCliDriver-auto_join30.q-vector_data_types.q-tez_join.q-and-12-more -
did not produce a TEST-*.xml file
TestMiniTezCliDriver-vector_acid3.q-vector_decimal_trailing.q-lvj_mapjoin.q-and-12-more
- did not produce a TEST-*.xml file
TestMiniTezCliDriver-vector_interval_2.q-bucket3.q-vectorization_7.q-and-12-more
- did not produce a TEST-*.xml file
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_ivyDownload
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_index_bitmap3
org.apache.hadoop.hive.cli.TestNegativeMinimrCliDriver.testNegativeCliDriver_minimr_broken_pipe
org.apache.hadoop.hive.llap.tezplugins.TestLlapTaskSchedulerService.testForcedLocalityPreemption
org.apache.hadoop.hive.metastore.TestHiveMetaStorePartitionSpecs.testAddPartitions
org.apache.hadoop.hive.metastore.TestHiveMetaStorePartitionSpecs.testFetchingPartitionsWithDifferentSchemas
org.apache.hadoop.hive.metastore.TestHiveMetaStorePartitionSpecs.testGetPartitionSpecs_WithAndWithoutPartitionGrouping
org.apache.hadoop.hive.metastore.TestMetaStoreEndFunctionListener.testEndFunctionListener
org.apache.hadoop.hive.metastore.TestMetaStoreInitListener.testMetaStoreInitListener
org.apache.hadoop.hive.metastore.TestPartitionNameWhitelistValidation.testAddPartitionWithCommas
org.apache.hadoop.hive.metastore.TestPartitionNameWhitelistValidation.testAddPartitionWithUnicode
org.apache.hadoop.hive.metastore.TestPartitionNameWhitelistValidation.testAddPartitionWithValidPartVal
org.apache.hadoop.hive.metastore.TestPartitionNameWhitelistValidation.testAppendPartitionWithCommas
org.apache.hadoop.hive.metastore.TestPartitionNameWhitelistValidation.testAppendPartitionWithUnicode
org.apache.hadoop.hive.metastore.TestPartitionNameWhitelistValidation.testAppendPartitionWithValidCharacters
org.apache.hadoop.hive.ql.exec.tez.TestHostAffinitySplitLocationProvider.testOrcSplitsLocationAffinity
org.apache.hadoop.hive.ql.lockmgr.TestDbTxnManager.concurrencyFalse
org.apache.hadoop.hive.ql.lockmgr.TestDbTxnManager.testDDLExclusive
org.apache.hadoop.hive.ql.lockmgr.TestDbTxnManager.testDelete
org.apache.hadoop.hive.ql.lockmgr.TestDbTxnManager.testLockTimeout
org.apache.hadoop.hive.ql.lockmgr.TestDbTxnManager.testRollback
org.apache.hadoop.hive.ql.lockmgr.TestDbTxnManager.testSingleReadPartition
org.apache.hadoop.hive.ql.lockmgr.TestDbTxnManager.testSingleWriteTable
org.apache.hadoop.hive.ql.lockmgr.TestDbTxnManager.testUpdate
org.apache.hadoop.hive.ql.lockmgr.TestDbTxnManager.testWriteDynamicPartition
org.apache.hadoop.hive.ql.security.TestClientSideAuthorizationProvider.testSimplePrivileges
org.apache.hadoop.hive.ql.security.TestFolderPermissions.org.apache.hadoop.hive.ql.security.TestFolderPermissions
org.apache.hadoop.hive.ql.security.TestMetastoreAuthorizationProvider.testSimplePrivileges
org.apache.hadoop.hive.ql.security.TestMultiAuthorizationPreEventListener.org.apache.hadoop.hive.ql.security.TestMultiAuthorizationPreEventListener
org.apache.hadoop.hive.ql.security.TestStorageBasedClientSideAuthorizationProvider.testSimplePrivileges
org.apache.hadoop.hive.ql.security.TestStorageBasedMetastoreAuthorizationDrops.testDropDatabase
org.apache.hadoop.hive.ql.security.TestStorageBasedMetastoreAuthorizationDrops.testDropPartition
org.apache.hadoop.hive.ql.security.TestStorageBasedMetastoreAuthorizationDrops.testDropTable
org.apache.hadoop.hive.ql.security.TestStorageBasedMetastoreAuthorizationProvider.testSimplePrivileges
org.apache.hadoop.hive.ql.security.TestStorageBasedMetastoreAuthorizationProviderWithACL.testSimplePrivileges
org.apache.hadoop.hive.ql.security.TestStorageBasedMetastoreAuthorizationReads.testReadDbFailure
org.apache.hadoop.hive.ql.security.TestStorageBasedMetastoreAuthorizationReads.testReadDbSuccess
org.apache.hadoop.hive.ql.security.TestStorageBasedMetastoreAuthorizationReads.testReadTableFailure
org.apache.hadoop.hive.thrift.TestHadoopAuthBridge23.testDelegationTokenSharedStore
org.apache.hadoop.hive.thrift.TestHadoopAuthBridge23.testMetastoreProxyUser
org.apache.hadoop.hive.thrift.TestHadoopAuthBridge23.testSaslWithHiveMetaStore
org.apache.hive.hcatalog.api.repl.commands.TestCommands.org.apache.hive.hcatalog.api.repl.commands.TestCommands
org.apache.hive.hcatalog.mapreduce.TestHCatPartitionPublish.org.apache.hive.hcatalog.mapreduce.TestHCatPartitionPublish
org.apache.hive.jdbc.TestSSL.testSSLFetchHttp
org.apache.hive.service.TestHS2ImpersonationWithRemoteMS.org.apache
[jira] [Commented] (HIVE-13240) GroupByOperator: Drop the hash aggregates when closing operator
[
https://issues.apache.org/jira/browse/HIVE-13240?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15234278#comment-15234278
]
Gopal V commented on HIVE-13240:
Failed tests have NameNode errors
{code}
2016-04-10T00:47:48,456 WARN
[org.apache.hadoop.hdfs.server.namenode.FSNamesystem$NameNodeResourceMonitor@773c7b73[]]:
namenode.NameNodeResourceChecker
(NameNodeResourceChecker.java:isResourceAvailable(89)) - Space available on
volume '/dev/xvde1' is 0, which is below the configured reserved amount
104857600
2016-04-10T00:47:48,456 WARN
[org.apache.hadoop.hdfs.server.namenode.FSNamesystem$NameNodeResourceMonitor@773c7b73[]]:
namenode.FSNamesystem (FSNamesystem.java:run(5159)) - NameNode low on
available disk space. Entering safe mode.
2016-04-10T00:47:48,456 INFO
[org.apache.hadoop.hdfs.server.namenode.FSNamesystem$NameNodeResourceMonitor@773c7b73[]]:
hdfs.StateChange (FSNamesystem.java:reportStatus(6003)) - STATE* Safe mode is
ON.
Resources are low on NN. Please add or free up more resources then turn off
safe mode manually. NOTE: If you turn off safe mode before adding resources,
the NN will immediately return to safe mode. Use "hdfs dfsadmin -safemode
leave" to turn safe mode off.
2016-04-10T00:47:54,612 WARN [LeaseRenewer:hiveptest@localhost:35563[]]:
hdfs.LeaseRenewer (LeaseRenewer.java:run(458)) - Failed to renew lease for
[DFSClient_NONMAPREDUCE_623273627_1] for 30 seconds. Will retry shortly ...
org.apache.hadoop.ipc.RemoteException: Cannot renew lease for
DFSClient_NONMAPREDUCE_623273627_1. Name node is in safe mode.
Resources are low on NN. Please add or free up more resources then turn off
safe mode manually. NOTE: If you turn off safe mode before adding resources,
the NN will immediately return to safe mode. Use "hdfs dfsadmin -safemode
leave" to turn safe mode off.
{code}
> GroupByOperator: Drop the hash aggregates when closing operator
> ---
>
> Key: HIVE-13240
> URL: https://issues.apache.org/jira/browse/HIVE-13240
> Project: Hive
> Issue Type: Bug
> Components: Query Processor
>Affects Versions: 1.3.0, 1.2.1, 2.0.0
>Reporter: Gopal V
>Assignee: Gopal V
> Attachments: HIVE-13240.03.patch, HIVE-13240.1.patch,
> HIVE-13240.2.patch
>
>
> GroupByOperator holds onto the Hash aggregates accumulated when the plan is
> cached.
> Drop the hashAggregates in case of error during forwarding to the next
> operator.
> Added for PTF, TopN and all GroupBy cases.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
