[jira] [Commented] (HIVE-13563) Hive Streaming does not honor orc.compress.size and orc.stripe.size table properties
[ https://issues.apache.org/jira/browse/HIVE-13563?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15299842#comment-15299842 ] Jesus Camacho Rodriguez commented on HIVE-13563: [~wzheng], will this go into 2.1.0? Thanks > Hive Streaming does not honor orc.compress.size and orc.stripe.size table > properties > > > Key: HIVE-13563 > URL: https://issues.apache.org/jira/browse/HIVE-13563 > Project: Hive > Issue Type: Bug > Components: ORC >Affects Versions: 2.1.0 >Reporter: Wei Zheng >Assignee: Wei Zheng > Labels: TODOC2.1 > Attachments: HIVE-13563.1.patch > > > According to the doc: > https://cwiki.apache.org/confluence/display/Hive/LanguageManual+ORC#LanguageManualORC-HiveQLSyntax > One should be able to specify tblproperties for many ORC options. > But the settings for orc.compress.size and orc.stripe.size don't take effect. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13563) Hive Streaming does not honor orc.compress.size and orc.stripe.size table properties
[ https://issues.apache.org/jira/browse/HIVE-13563?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15300196#comment-15300196 ] Wei Zheng commented on HIVE-13563: -- [~jcamachorodriguez] Yes this is going in > Hive Streaming does not honor orc.compress.size and orc.stripe.size table > properties > > > Key: HIVE-13563 > URL: https://issues.apache.org/jira/browse/HIVE-13563 > Project: Hive > Issue Type: Bug > Components: ORC >Affects Versions: 2.1.0 >Reporter: Wei Zheng >Assignee: Wei Zheng > Labels: TODOC2.1 > Attachments: HIVE-13563.1.patch > > > According to the doc: > https://cwiki.apache.org/confluence/display/Hive/LanguageManual+ORC#LanguageManualORC-HiveQLSyntax > One should be able to specify tblproperties for many ORC options. > But the settings for orc.compress.size and orc.stripe.size don't take effect. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13563) Hive Streaming does not honor orc.compress.size and orc.stripe.size table properties
[ https://issues.apache.org/jira/browse/HIVE-13563?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15300429#comment-15300429 ] Prasanth Jayachandran commented on HIVE-13563: -- [~owen.omalley] I don't understand how the ratio is related to triggering of minor compaction. With increase in stripe size to 128MB we are going to get less parallelism which may lead to jobs running slow. > Hive Streaming does not honor orc.compress.size and orc.stripe.size table > properties > > > Key: HIVE-13563 > URL: https://issues.apache.org/jira/browse/HIVE-13563 > Project: Hive > Issue Type: Bug > Components: ORC >Affects Versions: 2.1.0 >Reporter: Wei Zheng >Assignee: Wei Zheng > Labels: TODOC2.1 > Attachments: HIVE-13563.1.patch > > > According to the doc: > https://cwiki.apache.org/confluence/display/Hive/LanguageManual+ORC#LanguageManualORC-HiveQLSyntax > One should be able to specify tblproperties for many ORC options. > But the settings for orc.compress.size and orc.stripe.size don't take effect. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13563) Hive Streaming does not honor orc.compress.size and orc.stripe.size table properties
[ https://issues.apache.org/jira/browse/HIVE-13563?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15308390#comment-15308390 ] Jesus Camacho Rodriguez commented on HIVE-13563: Removing 2.1.0 target as issue is not tagged as Critical/Blocker and the RC will be created tomorrow. Please feel free to commit to branch-2.1 anyway and fix for 2.1.0 if this happens before the release. > Hive Streaming does not honor orc.compress.size and orc.stripe.size table > properties > > > Key: HIVE-13563 > URL: https://issues.apache.org/jira/browse/HIVE-13563 > Project: Hive > Issue Type: Bug > Components: ORC >Affects Versions: 2.1.0 >Reporter: Wei Zheng >Assignee: Wei Zheng > Labels: TODOC2.1 > Attachments: HIVE-13563.1.patch > > > According to the doc: > https://cwiki.apache.org/confluence/display/Hive/LanguageManual+ORC#LanguageManualORC-HiveQLSyntax > One should be able to specify tblproperties for many ORC options. > But the settings for orc.compress.size and orc.stripe.size don't take effect. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13563) Hive Streaming does not honor orc.compress.size and orc.stripe.size table properties
[
https://issues.apache.org/jira/browse/HIVE-13563?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15319811#comment-15319811
]
Prasanth Jayachandran commented on HIVE-13563:
--
looks good, +1. Minor comment:
{code}
AcidOutputFormat.Options optionsCloneForDelta = null;
{code}
can be moved inside else part as we are not referencing it outside the condition
{code}
else {
AcidOutputFormat.Options optionsCloneForDelta = options.clone();
}
{code}
> Hive Streaming does not honor orc.compress.size and orc.stripe.size table
> properties
>
>
> Key: HIVE-13563
> URL: https://issues.apache.org/jira/browse/HIVE-13563
> Project: Hive
> Issue Type: Bug
> Components: ORC
>Affects Versions: 2.1.0
>Reporter: Wei Zheng
>Assignee: Wei Zheng
> Labels: TODOC2.1
> Attachments: HIVE-13563.1.patch, HIVE-13563.2.patch
>
>
> According to the doc:
> https://cwiki.apache.org/confluence/display/Hive/LanguageManual+ORC#LanguageManualORC-HiveQLSyntax
> One should be able to specify tblproperties for many ORC options.
> But the settings for orc.compress.size and orc.stripe.size don't take effect.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-13563) Hive Streaming does not honor orc.compress.size and orc.stripe.size table properties
[
https://issues.apache.org/jira/browse/HIVE-13563?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15320206#comment-15320206
]
Hive QA commented on HIVE-13563:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12808840/HIVE-13563.3.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 12 failed/errored test(s), 10223 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_acid_globallimit
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_create_func1
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_list_bucket_dml_12
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_list_bucket_dml_13
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_stats_list_bucket
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_subquery_multiinsert
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_constprog_partitioner
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_index_bitmap3
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_acid_globallimit
org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_schemeAuthority
org.apache.hive.jdbc.TestJdbcWithLocalClusterSpark.testPermFunc
org.apache.hive.jdbc.TestJdbcWithMiniMr.testPermFunc
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-MASTER-Build/47/testReport
Console output:
https://builds.apache.org/job/PreCommit-HIVE-MASTER-Build/47/console
Test logs:
http://ec2-50-18-27-0.us-west-1.compute.amazonaws.com/logs/PreCommit-HIVE-MASTER-Build-47/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 12 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12808840 - PreCommit-HIVE-MASTER-Build
> Hive Streaming does not honor orc.compress.size and orc.stripe.size table
> properties
>
>
> Key: HIVE-13563
> URL: https://issues.apache.org/jira/browse/HIVE-13563
> Project: Hive
> Issue Type: Bug
> Components: ORC
>Affects Versions: 2.1.0
>Reporter: Wei Zheng
>Assignee: Wei Zheng
> Labels: TODOC2.1
> Attachments: HIVE-13563.1.patch, HIVE-13563.2.patch,
> HIVE-13563.3.patch
>
>
> According to the doc:
> https://cwiki.apache.org/confluence/display/Hive/LanguageManual+ORC#LanguageManualORC-HiveQLSyntax
> One should be able to specify tblproperties for many ORC options.
> But the settings for orc.compress.size and orc.stripe.size don't take effect.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-13563) Hive Streaming does not honor orc.compress.size and orc.stripe.size table properties
[
https://issues.apache.org/jira/browse/HIVE-13563?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15320992#comment-15320992
]
Wei Zheng commented on HIVE-13563:
--
For the two failures that have age==1
{code}
Test Name
Duration
Age
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_acid_globallimit
13 sec 1
org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_schemeAuthority
2 min 47 sec1
{code}
I updated golden file for acid_globallimit in patch 4. The other test
schemeAuthority passed locally.
> Hive Streaming does not honor orc.compress.size and orc.stripe.size table
> properties
>
>
> Key: HIVE-13563
> URL: https://issues.apache.org/jira/browse/HIVE-13563
> Project: Hive
> Issue Type: Bug
> Components: ORC
>Affects Versions: 2.1.0
>Reporter: Wei Zheng
>Assignee: Wei Zheng
> Labels: TODOC2.1
> Attachments: HIVE-13563.1.patch, HIVE-13563.2.patch,
> HIVE-13563.3.patch, HIVE-13563.4.patch
>
>
> According to the doc:
> https://cwiki.apache.org/confluence/display/Hive/LanguageManual+ORC#LanguageManualORC-HiveQLSyntax
> One should be able to specify tblproperties for many ORC options.
> But the settings for orc.compress.size and orc.stripe.size don't take effect.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-13563) Hive Streaming does not honor orc.compress.size and orc.stripe.size table properties
[
https://issues.apache.org/jira/browse/HIVE-13563?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15321720#comment-15321720
]
Hive QA commented on HIVE-13563:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12808978/HIVE-13563.4.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 8 failed/errored test(s), 10210 tests
executed
*Failed tests:*
{noformat}
TestMiniTezCliDriver-transform_ppr2.q-vector_outer_join0.q-vector_bround.q-and-10-more
- did not produce a TEST-*.xml file
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_acid_globallimit
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_list_bucket_dml_12
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_list_bucket_dml_13
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_stats_list_bucket
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_subquery_multiinsert
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_constprog_partitioner
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_index_bitmap3
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-MASTER-Build/53/testReport
Console output:
https://builds.apache.org/job/PreCommit-HIVE-MASTER-Build/53/console
Test logs:
http://ec2-50-18-27-0.us-west-1.compute.amazonaws.com/logs/PreCommit-HIVE-MASTER-Build-53/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 8 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12808978 - PreCommit-HIVE-MASTER-Build
> Hive Streaming does not honor orc.compress.size and orc.stripe.size table
> properties
>
>
> Key: HIVE-13563
> URL: https://issues.apache.org/jira/browse/HIVE-13563
> Project: Hive
> Issue Type: Bug
> Components: ORC
>Affects Versions: 2.1.0
>Reporter: Wei Zheng
>Assignee: Wei Zheng
> Labels: TODOC2.1
> Attachments: HIVE-13563.1.patch, HIVE-13563.2.patch,
> HIVE-13563.3.patch, HIVE-13563.4.patch
>
>
> According to the doc:
> https://cwiki.apache.org/confluence/display/Hive/LanguageManual+ORC#LanguageManualORC-HiveQLSyntax
> One should be able to specify tblproperties for many ORC options.
> But the settings for orc.compress.size and orc.stripe.size don't take effect.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-13563) Hive Streaming does not honor orc.compress.size and orc.stripe.size table properties
[ https://issues.apache.org/jira/browse/HIVE-13563?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15323983#comment-15323983 ] Lefty Leverenz commented on HIVE-13563: --- Doc note: This adds *hive.exec.orc.base.delta.ratio* to HiveConf.java, so it will need to be documented in the wiki. * [Configuration Properties -- ORC File Format | https://cwiki.apache.org/confluence/display/Hive/Configuration+Properties#ConfigurationProperties-ORCFileFormat] > Hive Streaming does not honor orc.compress.size and orc.stripe.size table > properties > > > Key: HIVE-13563 > URL: https://issues.apache.org/jira/browse/HIVE-13563 > Project: Hive > Issue Type: Bug > Components: ORC >Affects Versions: 2.1.0 >Reporter: Wei Zheng >Assignee: Wei Zheng > Labels: TODOC2.1 > Fix For: 1.3.0, 2.1.0 > > Attachments: HIVE-13563.1.patch, HIVE-13563.2.patch, > HIVE-13563.3.patch, HIVE-13563.4.patch, HIVE-13563.branch-1.patch > > > According to the doc: > https://cwiki.apache.org/confluence/display/Hive/LanguageManual+ORC#LanguageManualORC-HiveQLSyntax > One should be able to specify tblproperties for many ORC options. > But the settings for orc.compress.size and orc.stripe.size don't take effect. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13563) Hive Streaming does not honor orc.compress.size and orc.stripe.size table properties
[ https://issues.apache.org/jira/browse/HIVE-13563?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15421784#comment-15421784 ] Wei Zheng commented on HIVE-13563: -- [~leftylev] Wiki has been updated. > Hive Streaming does not honor orc.compress.size and orc.stripe.size table > properties > > > Key: HIVE-13563 > URL: https://issues.apache.org/jira/browse/HIVE-13563 > Project: Hive > Issue Type: Bug > Components: ORC >Affects Versions: 2.1.0 >Reporter: Wei Zheng >Assignee: Wei Zheng > Labels: TODOC2.1 > Fix For: 1.3.0, 2.1.0 > > Attachments: HIVE-13563.1.patch, HIVE-13563.2.patch, > HIVE-13563.3.patch, HIVE-13563.4.patch, HIVE-13563.branch-1.patch > > > According to the doc: > https://cwiki.apache.org/confluence/display/Hive/LanguageManual+ORC#LanguageManualORC-HiveQLSyntax > One should be able to specify tblproperties for many ORC options. > But the settings for orc.compress.size and orc.stripe.size don't take effect. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13563) Hive Streaming does not honor orc.compress.size and orc.stripe.size table properties
[ https://issues.apache.org/jira/browse/HIVE-13563?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15429614#comment-15429614 ] Lefty Leverenz commented on HIVE-13563: --- Thanks [~wzheng], I removed the TODOC2.1 label. Here's the link to the doc: * [Configuration Properties -- hive.exec.orc.base.delta.ratio | https://cwiki.apache.org/confluence/display/Hive/Configuration+Properties#ConfigurationProperties-hive.exec.orc.base.delta.ratio] > Hive Streaming does not honor orc.compress.size and orc.stripe.size table > properties > > > Key: HIVE-13563 > URL: https://issues.apache.org/jira/browse/HIVE-13563 > Project: Hive > Issue Type: Bug > Components: ORC >Affects Versions: 2.1.0 >Reporter: Wei Zheng >Assignee: Wei Zheng > Fix For: 1.3.0, 2.1.0 > > Attachments: HIVE-13563.1.patch, HIVE-13563.2.patch, > HIVE-13563.3.patch, HIVE-13563.4.patch, HIVE-13563.branch-1.patch > > > According to the doc: > https://cwiki.apache.org/confluence/display/Hive/LanguageManual+ORC#LanguageManualORC-HiveQLSyntax > One should be able to specify tblproperties for many ORC options. > But the settings for orc.compress.size and orc.stripe.size don't take effect. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13563) Hive Streaming does not honor orc.compress.size and orc.stripe.size table properties
[ https://issues.apache.org/jira/browse/HIVE-13563?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15254895#comment-15254895 ] Prasanth Jayachandran commented on HIVE-13563: -- 256KB is the default buffer size for base which will make 64KB for delta files which I think is huge when multiple delta files are open. Either we have to scale down the default buffer size to 64KB or change the ratio so that delta files gets 16KB which was previous default. > Hive Streaming does not honor orc.compress.size and orc.stripe.size table > properties > > > Key: HIVE-13563 > URL: https://issues.apache.org/jira/browse/HIVE-13563 > Project: Hive > Issue Type: Bug > Components: ORC >Affects Versions: 2.1.0 >Reporter: Wei Zheng >Assignee: Wei Zheng > Labels: TODOC2.1 > Attachments: HIVE-13563.1.patch > > > According to the doc: > https://cwiki.apache.org/confluence/display/Hive/LanguageManual+ORC#LanguageManualORC-HiveQLSyntax > One should be able to specify tblproperties for many ORC options. > But the settings for orc.compress.size and orc.stripe.size don't take effect. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13563) Hive Streaming does not honor orc.compress.size and orc.stripe.size table properties
[ https://issues.apache.org/jira/browse/HIVE-13563?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15254900#comment-15254900 ] Prasanth Jayachandran commented on HIVE-13563: -- I would prefer scaling down the default 256KB buffer size to 64KB. [~owen.omalley] Thoughts? > Hive Streaming does not honor orc.compress.size and orc.stripe.size table > properties > > > Key: HIVE-13563 > URL: https://issues.apache.org/jira/browse/HIVE-13563 > Project: Hive > Issue Type: Bug > Components: ORC >Affects Versions: 2.1.0 >Reporter: Wei Zheng >Assignee: Wei Zheng > Labels: TODOC2.1 > Attachments: HIVE-13563.1.patch > > > According to the doc: > https://cwiki.apache.org/confluence/display/Hive/LanguageManual+ORC#LanguageManualORC-HiveQLSyntax > One should be able to specify tblproperties for many ORC options. > But the settings for orc.compress.size and orc.stripe.size don't take effect. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13563) Hive Streaming does not honor orc.compress.size and orc.stripe.size table properties
[
https://issues.apache.org/jira/browse/HIVE-13563?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15255934#comment-15255934
]
Hive QA commented on HIVE-13563:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12800332/HIVE-13563.1.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 7 failed/errored test(s), 9992 tests executed
*Failed tests:*
{noformat}
TestHWISessionManager - did not produce a TEST-*.xml file
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_acid_globallimit
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_index_bitmap3
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_acid_globallimit
org.apache.hive.beeline.TestSchemaTool.testSchemaInit
org.apache.hive.hcatalog.api.repl.commands.TestCommands.org.apache.hive.hcatalog.api.repl.commands.TestCommands
org.apache.hive.hcatalog.hbase.TestPigHBaseStorageHandler.org.apache.hive.hcatalog.hbase.TestPigHBaseStorageHandler
{noformat}
Test results:
http://ec2-54-177-240-2.us-west-1.compute.amazonaws.com/job/PreCommit-HIVE-MASTER-Build/70/testReport
Console output:
http://ec2-54-177-240-2.us-west-1.compute.amazonaws.com/job/PreCommit-HIVE-MASTER-Build/70/console
Test logs:
http://ec2-50-18-27-0.us-west-1.compute.amazonaws.com/logs/PreCommit-HIVE-MASTER-Build-70/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 7 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12800332 - PreCommit-HIVE-MASTER-Build
> Hive Streaming does not honor orc.compress.size and orc.stripe.size table
> properties
>
>
> Key: HIVE-13563
> URL: https://issues.apache.org/jira/browse/HIVE-13563
> Project: Hive
> Issue Type: Bug
> Components: ORC
>Affects Versions: 2.1.0
>Reporter: Wei Zheng
>Assignee: Wei Zheng
> Labels: TODOC2.1
> Attachments: HIVE-13563.1.patch
>
>
> According to the doc:
> https://cwiki.apache.org/confluence/display/Hive/LanguageManual+ORC#LanguageManualORC-HiveQLSyntax
> One should be able to specify tblproperties for many ORC options.
> But the settings for orc.compress.size and orc.stripe.size don't take effect.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-13563) Hive Streaming does not honor orc.compress.size and orc.stripe.size table properties
[ https://issues.apache.org/jira/browse/HIVE-13563?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15259235#comment-15259235 ] Wei Zheng commented on HIVE-13563: -- ping [~owen.omalley].. > Hive Streaming does not honor orc.compress.size and orc.stripe.size table > properties > > > Key: HIVE-13563 > URL: https://issues.apache.org/jira/browse/HIVE-13563 > Project: Hive > Issue Type: Bug > Components: ORC >Affects Versions: 2.1.0 >Reporter: Wei Zheng >Assignee: Wei Zheng > Labels: TODOC2.1 > Attachments: HIVE-13563.1.patch > > > According to the doc: > https://cwiki.apache.org/confluence/display/Hive/LanguageManual+ORC#LanguageManualORC-HiveQLSyntax > One should be able to specify tblproperties for many ORC options. > But the settings for orc.compress.size and orc.stripe.size don't take effect. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13563) Hive Streaming does not honor orc.compress.size and orc.stripe.size table properties
[
https://issues.apache.org/jira/browse/HIVE-13563?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15259451#comment-15259451
]
Hive QA commented on HIVE-13563:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12800332/HIVE-13563.1.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 46 failed/errored test(s), 9935 tests
executed
*Failed tests:*
{noformat}
TestHWISessionManager - did not produce a TEST-*.xml file
TestMiniTezCliDriver-auto_join1.q-vector_complex_join.q-vectorization_limit.q-and-12-more
- did not produce a TEST-*.xml file
TestMiniTezCliDriver-vector_decimal_2.q-explainuser_1.q-explainuser_3.q-and-12-more
- did not produce a TEST-*.xml file
TestMiniTezCliDriver-vector_varchar_4.q-smb_cache.q-tez_join_hash.q-and-8-more
- did not produce a TEST-*.xml file
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_acid_globallimit
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_nomore_ambiguous_table_col
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_regexp_extract
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_index_bitmap3
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_acid_globallimit
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_clustern3
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_clustern4
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_nonkey_groupby
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_selectDistinctStarNeg_2
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_subquery_shared_alias
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_udtf_not_supported1
org.apache.hadoop.hive.metastore.TestAuthzApiEmbedAuthorizerInRemote.org.apache.hadoop.hive.metastore.TestAuthzApiEmbedAuthorizerInRemote
org.apache.hadoop.hive.metastore.TestFilterHooks.org.apache.hadoop.hive.metastore.TestFilterHooks
org.apache.hadoop.hive.metastore.TestHiveMetaStorePartitionSpecs.testAddPartitions
org.apache.hadoop.hive.metastore.TestHiveMetaStorePartitionSpecs.testFetchingPartitionsWithDifferentSchemas
org.apache.hadoop.hive.metastore.TestHiveMetaStorePartitionSpecs.testGetPartitionSpecs_WithAndWithoutPartitionGrouping
org.apache.hadoop.hive.metastore.TestMetaStoreEndFunctionListener.testEndFunctionListener
org.apache.hadoop.hive.metastore.TestMetaStoreEventListenerOnlyOnCommit.testEventStatus
org.apache.hadoop.hive.metastore.TestMetaStoreInitListener.testMetaStoreInitListener
org.apache.hadoop.hive.metastore.TestMetaStoreMetrics.org.apache.hadoop.hive.metastore.TestMetaStoreMetrics
org.apache.hadoop.hive.metastore.TestPartitionNameWhitelistValidation.testAddPartitionWithUnicode
org.apache.hadoop.hive.metastore.TestPartitionNameWhitelistValidation.testAddPartitionWithValidPartVal
org.apache.hadoop.hive.metastore.TestPartitionNameWhitelistValidation.testAppendPartitionWithCommas
org.apache.hadoop.hive.metastore.TestPartitionNameWhitelistValidation.testAppendPartitionWithUnicode
org.apache.hadoop.hive.metastore.TestPartitionNameWhitelistValidation.testAppendPartitionWithValidCharacters
org.apache.hadoop.hive.metastore.TestRetryingHMSHandler.testRetryingHMSHandler
org.apache.hadoop.hive.ql.security.TestClientSideAuthorizationProvider.testSimplePrivileges
org.apache.hadoop.hive.ql.security.TestFolderPermissions.org.apache.hadoop.hive.ql.security.TestFolderPermissions
org.apache.hadoop.hive.ql.security.TestMetastoreAuthorizationProvider.testSimplePrivileges
org.apache.hadoop.hive.ql.security.TestMultiAuthorizationPreEventListener.org.apache.hadoop.hive.ql.security.TestMultiAuthorizationPreEventListener
org.apache.hadoop.hive.ql.security.TestStorageBasedClientSideAuthorizationProvider.testSimplePrivileges
org.apache.hadoop.hive.ql.security.TestStorageBasedMetastoreAuthorizationDrops.testDropDatabase
org.apache.hadoop.hive.ql.security.TestStorageBasedMetastoreAuthorizationDrops.testDropPartition
org.apache.hadoop.hive.ql.security.TestStorageBasedMetastoreAuthorizationDrops.testDropTable
org.apache.hadoop.hive.ql.security.TestStorageBasedMetastoreAuthorizationProvider.testSimplePrivileges
org.apache.hadoop.hive.ql.security.TestStorageBasedMetastoreAuthorizationProviderWithACL.testSimplePrivileges
org.apache.hadoop.hive.ql.security.TestStorageBasedMetastoreAuthorizationReads.testReadDbSuccess
org.apache.hadoop.hive.ql.security.TestStorageBasedMetastoreAuthorizationReads.testReadTableFailure
org.apache.hadoop.hive.thrift.TestHadoopAuthBridge23.testSaslWithHiveMetaStore
org.apache.hive.hcatalog.listener.TestDbNotificationListener.sqlInsertPartition
org.apache.hive.minikdc.TestJdbcWithDBTokenStore.org.apache.hive.minikdc.TestJdbcWithDBTokenStore
org.apache.hive.service.TestHS2ImpersonationWithRemoteMS.org.apache.hive.service.TestHS2Impe
[jira] [Commented] (HIVE-13563) Hive Streaming does not honor orc.compress.size and orc.stripe.size table properties
[ https://issues.apache.org/jira/browse/HIVE-13563?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15280339#comment-15280339 ] Owen O'Malley commented on HIVE-13563: -- Actually, let's split the difference. I'd propose: default ratio: 8 default compression chunk: 128k default stripe size: 64MB delta compression: 16k delta stripe: 8MB thoughts? > Hive Streaming does not honor orc.compress.size and orc.stripe.size table > properties > > > Key: HIVE-13563 > URL: https://issues.apache.org/jira/browse/HIVE-13563 > Project: Hive > Issue Type: Bug > Components: ORC >Affects Versions: 2.1.0 >Reporter: Wei Zheng >Assignee: Wei Zheng > Labels: TODOC2.1 > Attachments: HIVE-13563.1.patch > > > According to the doc: > https://cwiki.apache.org/confluence/display/Hive/LanguageManual+ORC#LanguageManualORC-HiveQLSyntax > One should be able to specify tblproperties for many ORC options. > But the settings for orc.compress.size and orc.stripe.size don't take effect. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13563) Hive Streaming does not honor orc.compress.size and orc.stripe.size table properties
[ https://issues.apache.org/jira/browse/HIVE-13563?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15280508#comment-15280508 ] Prasanth Jayachandran commented on HIVE-13563: -- IMO, we are often hit by OOM because of compression buffer size and not stripe size. How about this? default ratio: 4 *Base files:* default compression chunk: 64k default stripe size: 64MB *Delta files:* delta compression: 16k delta stripe: 16MB > Hive Streaming does not honor orc.compress.size and orc.stripe.size table > properties > > > Key: HIVE-13563 > URL: https://issues.apache.org/jira/browse/HIVE-13563 > Project: Hive > Issue Type: Bug > Components: ORC >Affects Versions: 2.1.0 >Reporter: Wei Zheng >Assignee: Wei Zheng > Labels: TODOC2.1 > Attachments: HIVE-13563.1.patch > > > According to the doc: > https://cwiki.apache.org/confluence/display/Hive/LanguageManual+ORC#LanguageManualORC-HiveQLSyntax > One should be able to specify tblproperties for many ORC options. > But the settings for orc.compress.size and orc.stripe.size don't take effect. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13563) Hive Streaming does not honor orc.compress.size and orc.stripe.size table properties
[ https://issues.apache.org/jira/browse/HIVE-13563?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15281758#comment-15281758 ] Owen O'Malley commented on HIVE-13563: -- I think the ratio is better at 8 since we have the minor compaction set to run when there are 10 deltas, so 8 is a better match to the increase in work load. How about: ratio: 8 base compression 128k stripe 128mb delta compression 16k stripe 16mb > Hive Streaming does not honor orc.compress.size and orc.stripe.size table > properties > > > Key: HIVE-13563 > URL: https://issues.apache.org/jira/browse/HIVE-13563 > Project: Hive > Issue Type: Bug > Components: ORC >Affects Versions: 2.1.0 >Reporter: Wei Zheng >Assignee: Wei Zheng > Labels: TODOC2.1 > Attachments: HIVE-13563.1.patch > > > According to the doc: > https://cwiki.apache.org/confluence/display/Hive/LanguageManual+ORC#LanguageManualORC-HiveQLSyntax > One should be able to specify tblproperties for many ORC options. > But the settings for orc.compress.size and orc.stripe.size don't take effect. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
