[jira] [Commented] (HIVE-16133) Footer cache in Tez AM can take too much memory
[ https://issues.apache.org/jira/browse/HIVE-16133?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16130250#comment-16130250 ] Janos Gub commented on HIVE-16133: -- I think there is an error in the computation of the cache size. I opened a new jira ticket:https://issues.apache.org/jira/browse/HIVE-17344 > Footer cache in Tez AM can take too much memory > --- > > Key: HIVE-16133 > URL: https://issues.apache.org/jira/browse/HIVE-16133 > Project: Hive > Issue Type: Bug >Reporter: Siddharth Seth >Assignee: Sergey Shelukhin > Labels: TODOC2.2 > Fix For: 2.3.0 > > Attachments: HIVE-16133.01.patch, HIVE-16133.02.patch, > HIVE-16133.02.patch, HIVE-16133.03.patch, HIVE-16133.04.patch, > HIVE-16133.patch > > -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HIVE-16133) Footer cache in Tez AM can take too much memory
[ https://issues.apache.org/jira/browse/HIVE-16133?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15906462#comment-15906462 ] Lefty Leverenz commented on HIVE-16133: --- Doc note: This removes *hive.orc.cache.stripe.details.size* from HiveConf.java and adds *hive.orc.cache.stripe.details.mem.size*, so the wiki needs to be updated. * [Configuration Properties -- hive.orc.cache.stripe.details.size | https://cwiki.apache.org/confluence/display/Hive/Configuration+Properties#ConfigurationProperties-hive.orc.cache.stripe.details.size] ORC note: *hive.orc.cache.stripe.details.size* is also documented in the ORC wiki, so that should be updated too: * [ORC -- Hive Configuration -- Configuration Properties | https://orc.apache.org/docs/hive-config.html#configuration-properties] Added a TODOC2.2 label. > Footer cache in Tez AM can take too much memory > --- > > Key: HIVE-16133 > URL: https://issues.apache.org/jira/browse/HIVE-16133 > Project: Hive > Issue Type: Bug >Reporter: Siddharth Seth >Assignee: Sergey Shelukhin > Labels: TODOC2.2 > Fix For: 2.2.0 > > Attachments: HIVE-16133.01.patch, HIVE-16133.02.patch, > HIVE-16133.02.patch, HIVE-16133.03.patch, HIVE-16133.04.patch, > HIVE-16133.patch > > -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HIVE-16133) Footer cache in Tez AM can take too much memory
[ https://issues.apache.org/jira/browse/HIVE-16133?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15906089#comment-15906089 ] Hive QA commented on HIVE-16133: Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12857431/HIVE-16133.04.patch {color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified. {color:green}SUCCESS:{color} +1 due to 10339 tests passed Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/4084/testReport Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/4084/console Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-4084/ Messages: {noformat} Executing org.apache.hive.ptest.execution.TestCheckPhase Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase {noformat} This message is automatically generated. ATTACHMENT ID: 12857431 - PreCommit-HIVE-Build > Footer cache in Tez AM can take too much memory > --- > > Key: HIVE-16133 > URL: https://issues.apache.org/jira/browse/HIVE-16133 > Project: Hive > Issue Type: Bug >Reporter: Siddharth Seth >Assignee: Sergey Shelukhin > Attachments: HIVE-16133.01.patch, HIVE-16133.02.patch, > HIVE-16133.02.patch, HIVE-16133.03.patch, HIVE-16133.04.patch, > HIVE-16133.patch > > -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HIVE-16133) Footer cache in Tez AM can take too much memory
[ https://issues.apache.org/jira/browse/HIVE-16133?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15905869#comment-15905869 ] Sergey Shelukhin commented on HIVE-16133: - Still can;t repro, but I can see errors in the logs of the last run. Apparently sometimes footer in cache gets corrupted when it's stored as a buffer. Not sure how it happens, probably the buffer object, or data, is reused somewhere and needs to be copied. > Footer cache in Tez AM can take too much memory > --- > > Key: HIVE-16133 > URL: https://issues.apache.org/jira/browse/HIVE-16133 > Project: Hive > Issue Type: Bug >Reporter: Siddharth Seth >Assignee: Sergey Shelukhin > Attachments: HIVE-16133.01.patch, HIVE-16133.02.patch, > HIVE-16133.02.patch, HIVE-16133.03.patch, HIVE-16133.patch > > -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HIVE-16133) Footer cache in Tez AM can take too much memory
[ https://issues.apache.org/jira/browse/HIVE-16133?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15905754#comment-15905754 ] Hive QA commented on HIVE-16133: Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12857383/HIVE-16133.03.patch {color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified. {color:red}ERROR:{color} -1 due to 6 failed/errored test(s), 10336 tests executed *Failed tests:* {noformat} org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[tez_dynpart_hashjoin_1] (batchId=155) org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vector_join30] (batchId=146) org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vector_nullsafe_join] (batchId=156) org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vector_outer_join1] (batchId=148) org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vector_outer_join4] (batchId=155) org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver[explainanalyze_4] (batchId=95) {noformat} Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/4076/testReport Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/4076/console Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-4076/ Messages: {noformat} Executing org.apache.hive.ptest.execution.TestCheckPhase Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase Tests exited with: TestsFailedException: 6 tests failed {noformat} This message is automatically generated. ATTACHMENT ID: 12857383 - PreCommit-HIVE-Build > Footer cache in Tez AM can take too much memory > --- > > Key: HIVE-16133 > URL: https://issues.apache.org/jira/browse/HIVE-16133 > Project: Hive > Issue Type: Bug >Reporter: Siddharth Seth >Assignee: Sergey Shelukhin > Attachments: HIVE-16133.01.patch, HIVE-16133.02.patch, > HIVE-16133.02.patch, HIVE-16133.03.patch, HIVE-16133.patch > > -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HIVE-16133) Footer cache in Tez AM can take too much memory
[ https://issues.apache.org/jira/browse/HIVE-16133?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15904331#comment-15904331 ] Hive QA commented on HIVE-16133: Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12857116/HIVE-16133.02.patch {color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified. {color:red}ERROR:{color} -1 due to 5 failed/errored test(s), 10336 tests executed *Failed tests:* {noformat} org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[tez_vector_dynpart_hashjoin_1] (batchId=153) org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vector_join_nulls] (batchId=144) org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vectorized_ptf] (batchId=152) org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver[hybridgrace_hashjoin_1] (batchId=94) org.apache.hive.beeline.TestBeeLineWithArgs.testQueryProgress (batchId=213) {noformat} Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/4059/testReport Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/4059/console Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-4059/ Messages: {noformat} Executing org.apache.hive.ptest.execution.TestCheckPhase Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase Tests exited with: TestsFailedException: 5 tests failed {noformat} This message is automatically generated. ATTACHMENT ID: 12857116 - PreCommit-HIVE-Build > Footer cache in Tez AM can take too much memory > --- > > Key: HIVE-16133 > URL: https://issues.apache.org/jira/browse/HIVE-16133 > Project: Hive > Issue Type: Bug >Reporter: Siddharth Seth >Assignee: Sergey Shelukhin > Attachments: HIVE-16133.01.patch, HIVE-16133.02.patch, > HIVE-16133.02.patch, HIVE-16133.patch > > -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HIVE-16133) Footer cache in Tez AM can take too much memory
[ https://issues.apache.org/jira/browse/HIVE-16133?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15904330#comment-15904330 ] Sergey Shelukhin commented on HIVE-16133: - Looking at test failures > Footer cache in Tez AM can take too much memory > --- > > Key: HIVE-16133 > URL: https://issues.apache.org/jira/browse/HIVE-16133 > Project: Hive > Issue Type: Bug >Reporter: Siddharth Seth >Assignee: Sergey Shelukhin > Attachments: HIVE-16133.01.patch, HIVE-16133.02.patch, > HIVE-16133.02.patch, HIVE-16133.patch > > -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HIVE-16133) Footer cache in Tez AM can take too much memory
[ https://issues.apache.org/jira/browse/HIVE-16133?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15904208#comment-15904208 ] Hive QA commented on HIVE-16133: Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12857116/HIVE-16133.02.patch {color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified. {color:red}ERROR:{color} -1 due to 7 failed/errored test(s), 10336 tests executed *Failed tests:* {noformat} org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vector_auto_smb_mapjoin_14] (batchId=145) org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vector_join30] (batchId=146) org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vector_join_filters] (batchId=151) org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vector_join_nulls] (batchId=144) org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vector_nullsafe_join] (batchId=156) org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vector_outer_join1] (batchId=148) org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver[explainanalyze_4] (batchId=95) {noformat} Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/4057/testReport Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/4057/console Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-4057/ Messages: {noformat} Executing org.apache.hive.ptest.execution.TestCheckPhase Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase Tests exited with: TestsFailedException: 7 tests failed {noformat} This message is automatically generated. ATTACHMENT ID: 12857116 - PreCommit-HIVE-Build > Footer cache in Tez AM can take too much memory > --- > > Key: HIVE-16133 > URL: https://issues.apache.org/jira/browse/HIVE-16133 > Project: Hive > Issue Type: Bug >Reporter: Siddharth Seth >Assignee: Sergey Shelukhin > Attachments: HIVE-16133.01.patch, HIVE-16133.02.patch, > HIVE-16133.02.patch, HIVE-16133.patch > > -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HIVE-16133) Footer cache in Tez AM can take too much memory
[ https://issues.apache.org/jira/browse/HIVE-16133?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15903876#comment-15903876 ] Sergey Shelukhin commented on HIVE-16133: - Nm, tests actually failed because it doesn't support both maximum size and weight. Why don't we want to end up with it taking more? 10K would take unpredictable amount of memory as far as I understand, due to wide variety in protobuf object sizes. 500Mb seems reasonable for the usual AM size (2-4Gb I assume). Should it be less? 200Mb? cc [~prasanth_j] > Footer cache in Tez AM can take too much memory > --- > > Key: HIVE-16133 > URL: https://issues.apache.org/jira/browse/HIVE-16133 > Project: Hive > Issue Type: Bug >Reporter: Siddharth Seth >Assignee: Sergey Shelukhin > Attachments: HIVE-16133.01.patch, HIVE-16133.patch > > -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HIVE-16133) Footer cache in Tez AM can take too much memory
[ https://issues.apache.org/jira/browse/HIVE-16133?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15903805#comment-15903805 ] Sergey Shelukhin commented on HIVE-16133: - [~sseth] both the defaults would now be enforced > Footer cache in Tez AM can take too much memory > --- > > Key: HIVE-16133 > URL: https://issues.apache.org/jira/browse/HIVE-16133 > Project: Hive > Issue Type: Bug >Reporter: Siddharth Seth >Assignee: Sergey Shelukhin > Attachments: HIVE-16133.01.patch, HIVE-16133.patch > > -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HIVE-16133) Footer cache in Tez AM can take too much memory
[ https://issues.apache.org/jira/browse/HIVE-16133?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15902555#comment-15902555 ] Hive QA commented on HIVE-16133: Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12856897/HIVE-16133.01.patch {color:red}ERROR:{color} -1 due to no test(s) being added or modified. {color:red}ERROR:{color} -1 due to 380 failed/errored test(s), 9758 tests executed *Failed tests:* {noformat} org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[acid_join] (batchId=15) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[acid_mapjoin] (batchId=10) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[acid_subquery] (batchId=36) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[acid_table_stats] (batchId=49) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[acid_vectorization] (batchId=60) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[acid_vectorization_partition] (batchId=67) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[acid_vectorization_project] (batchId=18) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[authorization_delete] (batchId=76) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[authorization_delete_own_table] (batchId=61) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[authorization_update] (batchId=8) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[authorization_update_own_table] (batchId=72) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[autoColumnStats_4] (batchId=12) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[char_serde] (batchId=49) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[date_serde] (batchId=41) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[decimal_join2] (batchId=36) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[delete_all_non_partitioned] (batchId=27) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[delete_all_partitioned] (batchId=26) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[delete_orig_table] (batchId=37) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[delete_tmp_table] (batchId=47) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[delete_where_no_match] (batchId=27) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[delete_where_non_partitioned] (batchId=36) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[delete_where_partitioned] (batchId=37) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[delete_whole_partition] (batchId=9) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[implicit_cast_during_insert] (batchId=47) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[insert_acid_dynamic_partition] (batchId=19) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[insert_acid_not_bucketed] (batchId=64) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[insert_nonacid_from_acid] (batchId=68) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[insert_orig_table] (batchId=57) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[insert_update_delete] (batchId=78) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[insert_values_acid_not_bucketed] (batchId=17) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[insert_values_dynamic_partitioned] (batchId=69) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[insert_values_non_partitioned] (batchId=19) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[insert_values_orig_table] (batchId=53) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[insert_values_orig_table_use_metadata] (batchId=58) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[insert_values_partitioned] (batchId=70) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[insert_values_tmp_table] (batchId=4) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[llap_acid] (batchId=74) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[llap_reader] (batchId=7) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[masking_acid_no_masking] (batchId=22) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[materialized_view_authorization_sqlstd] (batchId=43) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[materialized_view_create] (batchId=65) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[materialized_view_create_rewrite] (batchId=2) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[materialized_view_create_rewrite_multi_db] (batchId=62) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[materialized_view_describe] (batchId=63) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[orc_createas1] (batchId=80) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[orc_dictionary_threshold] (batchId=20) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[orc_diff_part_cols2] (batchId=23) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[orc_diff_part_cols]
[jira] [Commented] (HIVE-16133) Footer cache in Tez AM can take too much memory
[ https://issues.apache.org/jira/browse/HIVE-16133?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15902418#comment-15902418 ] Siddharth Seth commented on HIVE-16133: --- The old default was 10K entries. Do we know how much memory that would take, on average anyway. Don't want to end up with the 500MB default being larger than what 10K records would typically take. > Footer cache in Tez AM can take too much memory > --- > > Key: HIVE-16133 > URL: https://issues.apache.org/jira/browse/HIVE-16133 > Project: Hive > Issue Type: Bug >Reporter: Siddharth Seth >Assignee: Sergey Shelukhin > Attachments: HIVE-16133.01.patch, HIVE-16133.patch > > -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HIVE-16133) Footer cache in Tez AM can take too much memory
[ https://issues.apache.org/jira/browse/HIVE-16133?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15901938#comment-15901938 ] Gunther Hagleitner commented on HIVE-16133: --- LGTM +1 > Footer cache in Tez AM can take too much memory > --- > > Key: HIVE-16133 > URL: https://issues.apache.org/jira/browse/HIVE-16133 > Project: Hive > Issue Type: Bug >Reporter: Siddharth Seth >Assignee: Sergey Shelukhin > Attachments: HIVE-16133.patch > > -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HIVE-16133) Footer cache in Tez AM can take too much memory
[ https://issues.apache.org/jira/browse/HIVE-16133?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15901900#comment-15901900 ] Sergey Shelukhin commented on HIVE-16133: - [~hagleitn] perhaps you can review? > Footer cache in Tez AM can take too much memory > --- > > Key: HIVE-16133 > URL: https://issues.apache.org/jira/browse/HIVE-16133 > Project: Hive > Issue Type: Bug >Reporter: Siddharth Seth >Assignee: Sergey Shelukhin > Attachments: HIVE-16133.patch > > -- This message was sent by Atlassian JIRA (v6.3.15#6346)