[jira] [Commented] (HIVE-5504) OrcOutputFormat honors compression properties only from within hive
[
https://issues.apache.org/jira/browse/HIVE-5504?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13918392#comment-13918392
]
Sushanth Sowmyan commented on HIVE-5504:
Looks like the tests finally ran! :)
And the error reported by the pre-commit tests do not seem related to this
patch - the table in question that's failing on that test is not even created
as an orc table.
> OrcOutputFormat honors compression properties only from within hive
> -
>
> Key: HIVE-5504
> URL: https://issues.apache.org/jira/browse/HIVE-5504
> Project: Hive
> Issue Type: Bug
> Components: HCatalog
>Affects Versions: 0.11.0, 0.12.0, 0.13.0
>Reporter: Venkat Ranganathan
>Assignee: Sushanth Sowmyan
> Attachments: HIVE-5504.2.patch, HIVE-5504.patch
>
>
> When we import data into a HCatalog table created with the following storage
> description
> .. stored as orc tblproperties ("orc.compress"="SNAPPY")
> the resultant orc file still uses the default zlib compression
> It looks like HCatOutputFormat is ignoring the tblproperties specified.
> show tblproperties shows that the table indeed has the properties properly
> saved.
> An insert/select into the table has the resulting orc file honor the tbl
> property.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HIVE-5504) OrcOutputFormat honors compression properties only from within hive
[
https://issues.apache.org/jira/browse/HIVE-5504?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13918146#comment-13918146
]
Hive QA commented on HIVE-5504:
---
{color:red}Overall{color}: -1 at least one tests failed
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12631095/HIVE-5504.2.patch
{color:red}ERROR:{color} -1 due to 1 failed/errored test(s), 5218 tests executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_bucket_num_reducers
{noformat}
Test results:
http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/1604/testReport
Console output:
http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/1604/console
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 1 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12631095
> OrcOutputFormat honors compression properties only from within hive
> -
>
> Key: HIVE-5504
> URL: https://issues.apache.org/jira/browse/HIVE-5504
> Project: Hive
> Issue Type: Bug
> Components: HCatalog
>Affects Versions: 0.11.0, 0.12.0, 0.13.0
>Reporter: Venkat Ranganathan
>Assignee: Sushanth Sowmyan
> Attachments: HIVE-5504.2.patch, HIVE-5504.patch
>
>
> When we import data into a HCatalog table created with the following storage
> description
> .. stored as orc tblproperties ("orc.compress"="SNAPPY")
> the resultant orc file still uses the default zlib compression
> It looks like HCatOutputFormat is ignoring the tblproperties specified.
> show tblproperties shows that the table indeed has the properties properly
> saved.
> An insert/select into the table has the resulting orc file honor the tbl
> property.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HIVE-5504) OrcOutputFormat honors compression properties only from within hive
[
https://issues.apache.org/jira/browse/HIVE-5504?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13912302#comment-13912302
]
Thejas M Nair commented on HIVE-5504:
-
+1
> OrcOutputFormat honors compression properties only from within hive
> -
>
> Key: HIVE-5504
> URL: https://issues.apache.org/jira/browse/HIVE-5504
> Project: Hive
> Issue Type: Bug
> Components: HCatalog
>Affects Versions: 0.11.0, 0.12.0
>Reporter: Venkat Ranganathan
>Assignee: Sushanth Sowmyan
> Attachments: HIVE-5504.2.patch, HIVE-5504.patch
>
>
> When we import data into a HCatalog table created with the following storage
> description
> .. stored as orc tblproperties ("orc.compress"="SNAPPY")
> the resultant orc file still uses the default zlib compression
> It looks like HCatOutputFormat is ignoring the tblproperties specified.
> show tblproperties shows that the table indeed has the properties properly
> saved.
> An insert/select into the table has the resulting orc file honor the tbl
> property.
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
[jira] [Commented] (HIVE-5504) OrcOutputFormat honors compression properties only from within hive
[
https://issues.apache.org/jira/browse/HIVE-5504?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13911321#comment-13911321
]
Sushanth Sowmyan commented on HIVE-5504:
[~thejas], could I please get a review of this jira? (Daniel has reviewed the
hcat part of it, but this still needs hive-side verification).
> OrcOutputFormat honors compression properties only from within hive
> -
>
> Key: HIVE-5504
> URL: https://issues.apache.org/jira/browse/HIVE-5504
> Project: Hive
> Issue Type: Bug
> Components: HCatalog
>Affects Versions: 0.11.0, 0.12.0
>Reporter: Venkat Ranganathan
>Assignee: Sushanth Sowmyan
> Attachments: HIVE-5504.patch
>
>
> When we import data into a HCatalog table created with the following storage
> description
> .. stored as orc tblproperties ("orc.compress"="SNAPPY")
> the resultant orc file still uses the default zlib compression
> It looks like HCatOutputFormat is ignoring the tblproperties specified.
> show tblproperties shows that the table indeed has the properties properly
> saved.
> An insert/select into the table has the resulting orc file honor the tbl
> property.
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
[jira] [Commented] (HIVE-5504) OrcOutputFormat honors compression properties only from within hive
[
https://issues.apache.org/jira/browse/HIVE-5504?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13906384#comment-13906384
]
Daniel Dai commented on HIVE-5504:
--
+1 on hcat part.
> OrcOutputFormat honors compression properties only from within hive
> -
>
> Key: HIVE-5504
> URL: https://issues.apache.org/jira/browse/HIVE-5504
> Project: Hive
> Issue Type: Bug
> Components: HCatalog
>Affects Versions: 0.11.0, 0.12.0
>Reporter: Venkat Ranganathan
>Assignee: Sushanth Sowmyan
> Attachments: HIVE-5504.patch
>
>
> When we import data into a HCatalog table created with the following storage
> description
> .. stored as orc tblproperties ("orc.compress"="SNAPPY")
> the resultant orc file still uses the default zlib compression
> It looks like HCatOutputFormat is ignoring the tblproperties specified.
> show tblproperties shows that the table indeed has the properties properly
> saved.
> An insert/select into the table has the resulting orc file honor the tbl
> property.
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
[jira] [Commented] (HIVE-5504) OrcOutputFormat honors compression properties only from within hive
[
https://issues.apache.org/jira/browse/HIVE-5504?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13901888#comment-13901888
]
Sushanth Sowmyan commented on HIVE-5504:
[~thejas]/[~ashutoshc] , could I please get a review of this patch?
> OrcOutputFormat honors compression properties only from within hive
> -
>
> Key: HIVE-5504
> URL: https://issues.apache.org/jira/browse/HIVE-5504
> Project: Hive
> Issue Type: Bug
> Components: HCatalog
>Affects Versions: 0.11.0, 0.12.0
>Reporter: Venkat Ranganathan
>Assignee: Sushanth Sowmyan
> Attachments: HIVE-5504.patch
>
>
> When we import data into a HCatalog table created with the following storage
> description
> .. stored as orc tblproperties ("orc.compress"="SNAPPY")
> the resultant orc file still uses the default zlib compression
> It looks like HCatOutputFormat is ignoring the tblproperties specified.
> show tblproperties shows that the table indeed has the properties properly
> saved.
> An insert/select into the table has the resulting orc file honor the tbl
> property.
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
[jira] [Commented] (HIVE-5504) OrcOutputFormat honors compression properties only from within hive
[
https://issues.apache.org/jira/browse/HIVE-5504?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13900750#comment-13900750
]
Sushanth Sowmyan commented on HIVE-5504:
Actually, reading further, the defaults are taken care of in the WriterOptions
constructor, and thus, not a problem for this patch. And the parameters names
are still in OrcFile, it's only the default values that were moved to HiveConf.
This patch works as-is, and is compatible with HIVE-5728 and will respect the
new defaults behaviour as well. Resetting the patch as available.
> OrcOutputFormat honors compression properties only from within hive
> -
>
> Key: HIVE-5504
> URL: https://issues.apache.org/jira/browse/HIVE-5504
> Project: Hive
> Issue Type: Bug
> Components: HCatalog
>Affects Versions: 0.11.0, 0.12.0
>Reporter: Venkat Ranganathan
>Assignee: Sushanth Sowmyan
> Attachments: HIVE-5504.patch
>
>
> When we import data into a HCatalog table created with the following storage
> description
> .. stored as orc tblproperties ("orc.compress"="SNAPPY")
> the resultant orc file still uses the default zlib compression
> It looks like HCatOutputFormat is ignoring the tblproperties specified.
> show tblproperties shows that the table indeed has the properties properly
> saved.
> An insert/select into the table has the resulting orc file honor the tbl
> property.
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
[jira] [Commented] (HIVE-5504) OrcOutputFormat honors compression properties only from within hive
[
https://issues.apache.org/jira/browse/HIVE-5504?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13900734#comment-13900734
]
Sushanth Sowmyan commented on HIVE-5504:
Yeah, some parameters that were depended upon in OrcFile moved to HiveConf, so
at the very least, this patch requires regeneration.
One good thing at least is that OrcOutputFormat itself did not change, so that
is still an easy merge.
The one weird thing is that HIVE-5728 introduces logic to override the default
behaviour using hive-site.xml based parameters, and removes certain default
parameters from OrcFile. We need to change this patch to respect that.
> OrcOutputFormat honors compression properties only from within hive
> -
>
> Key: HIVE-5504
> URL: https://issues.apache.org/jira/browse/HIVE-5504
> Project: Hive
> Issue Type: Bug
> Components: HCatalog
>Affects Versions: 0.11.0, 0.12.0
>Reporter: Venkat Ranganathan
>Assignee: Sushanth Sowmyan
> Attachments: HIVE-5504.patch
>
>
> When we import data into a HCatalog table created with the following storage
> description
> .. stored as orc tblproperties ("orc.compress"="SNAPPY")
> the resultant orc file still uses the default zlib compression
> It looks like HCatOutputFormat is ignoring the tblproperties specified.
> show tblproperties shows that the table indeed has the properties properly
> saved.
> An insert/select into the table has the resulting orc file honor the tbl
> property.
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
[jira] [Commented] (HIVE-5504) OrcOutputFormat honors compression properties only from within hive
[
https://issues.apache.org/jira/browse/HIVE-5504?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13900607#comment-13900607
]
Sushanth Sowmyan commented on HIVE-5504:
Hrm, HIVE-5728 just hit, we should check if it conflicts with this patch (very
likely).
> OrcOutputFormat honors compression properties only from within hive
> -
>
> Key: HIVE-5504
> URL: https://issues.apache.org/jira/browse/HIVE-5504
> Project: Hive
> Issue Type: Bug
> Components: HCatalog
>Affects Versions: 0.11.0, 0.12.0
>Reporter: Venkat Ranganathan
>Assignee: Sushanth Sowmyan
> Attachments: HIVE-5504.patch
>
>
> When we import data into a HCatalog table created with the following storage
> description
> .. stored as orc tblproperties ("orc.compress"="SNAPPY")
> the resultant orc file still uses the default zlib compression
> It looks like HCatOutputFormat is ignoring the tblproperties specified.
> show tblproperties shows that the table indeed has the properties properly
> saved.
> An insert/select into the table has the resulting orc file honor the tbl
> property.
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
[jira] [Commented] (HIVE-5504) OrcOutputFormat honors compression properties only from within hive
[
https://issues.apache.org/jira/browse/HIVE-5504?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13900488#comment-13900488
]
Sushanth Sowmyan commented on HIVE-5504:
Attached reviewboard link : https://reviews.apache.org/r/18085/
> OrcOutputFormat honors compression properties only from within hive
> -
>
> Key: HIVE-5504
> URL: https://issues.apache.org/jira/browse/HIVE-5504
> Project: Hive
> Issue Type: Bug
> Components: HCatalog
>Affects Versions: 0.11.0, 0.12.0
>Reporter: Venkat Ranganathan
>Assignee: Sushanth Sowmyan
> Attachments: HIVE-5504.patch
>
>
> When we import data into a HCatalog table created with the following storage
> description
> .. stored as orc tblproperties ("orc.compress"="SNAPPY")
> the resultant orc file still uses the default zlib compression
> It looks like HCatOutputFormat is ignoring the tblproperties specified.
> show tblproperties shows that the table indeed has the properties properly
> saved.
> An insert/select into the table has the resulting orc file honor the tbl
> property.
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
[jira] [Commented] (HIVE-5504) OrcOutputFormat honors compression properties only from within hive
[
https://issues.apache.org/jira/browse/HIVE-5504?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13899412#comment-13899412
]
Sushanth Sowmyan commented on HIVE-5504:
The error reported by the precommit test seems to be unrelated to this fix.
> OrcOutputFormat honors compression properties only from within hive
> -
>
> Key: HIVE-5504
> URL: https://issues.apache.org/jira/browse/HIVE-5504
> Project: Hive
> Issue Type: Bug
> Components: HCatalog
>Affects Versions: 0.11.0, 0.12.0
>Reporter: Venkat Ranganathan
>Assignee: Sushanth Sowmyan
> Attachments: HIVE-5504.patch
>
>
> When we import data into a HCatalog table created with the following storage
> description
> .. stored as orc tblproperties ("orc.compress"="SNAPPY")
> the resultant orc file still uses the default zlib compression
> It looks like HCatOutputFormat is ignoring the tblproperties specified.
> show tblproperties shows that the table indeed has the properties properly
> saved.
> An insert/select into the table has the resulting orc file honor the tbl
> property.
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
[jira] [Commented] (HIVE-5504) OrcOutputFormat honors compression properties only from within hive
[
https://issues.apache.org/jira/browse/HIVE-5504?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13898765#comment-13898765
]
Hive QA commented on HIVE-5504:
---
{color:red}Overall{color}: -1 at least one tests failed
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12628073/HIVE-5504.patch
{color:red}ERROR:{color} -1 due to 1 failed/errored test(s), 5086 tests executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_auto_sortmerge_join_16
{noformat}
Test results:
http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/1276/testReport
Console output:
http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/1276/console
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 1 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12628073
> OrcOutputFormat honors compression properties only from within hive
> -
>
> Key: HIVE-5504
> URL: https://issues.apache.org/jira/browse/HIVE-5504
> Project: Hive
> Issue Type: Bug
> Components: HCatalog
>Affects Versions: 0.11.0, 0.12.0
>Reporter: Venkat Ranganathan
>Assignee: Sushanth Sowmyan
> Attachments: HIVE-5504.patch
>
>
> When we import data into a HCatalog table created with the following storage
> description
> .. stored as orc tblproperties ("orc.compress"="SNAPPY")
> the resultant orc file still uses the default zlib compression
> It looks like HCatOutputFormat is ignoring the tblproperties specified.
> show tblproperties shows that the table indeed has the properties properly
> saved.
> An insert/select into the table has the resulting orc file honor the tbl
> property.
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
[jira] [Commented] (HIVE-5504) OrcOutputFormat honors compression properties only from within hive
[
https://issues.apache.org/jira/browse/HIVE-5504?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13898202#comment-13898202
]
Sushanth Sowmyan commented on HIVE-5504:
[~thejas]/[~prasanth_j] , could I bug either of you for a review of this?
> OrcOutputFormat honors compression properties only from within hive
> -
>
> Key: HIVE-5504
> URL: https://issues.apache.org/jira/browse/HIVE-5504
> Project: Hive
> Issue Type: Bug
> Components: HCatalog
>Affects Versions: 0.11.0, 0.12.0
>Reporter: Venkat Ranganathan
>Assignee: Sushanth Sowmyan
> Attachments: HIVE-5504.patch
>
>
> When we import data into a HCatalog table created with the following storage
> description
> .. stored as orc tblproperties ("orc.compress"="SNAPPY")
> the resultant orc file still uses the default zlib compression
> It looks like HCatOutputFormat is ignoring the tblproperties specified.
> show tblproperties shows that the table indeed has the properties properly
> saved.
> An insert/select into the table has the resulting orc file honor the tbl
> property.
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
[jira] [Commented] (HIVE-5504) OrcOutputFormat honors compression properties only from within hive
[
https://issues.apache.org/jira/browse/HIVE-5504?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13897099#comment-13897099
]
Sushanth Sowmyan commented on HIVE-5504:
Testing related note : This bug is interesting in that hive as well as pig are
able to read data irrespective of what compression format was actually used.
i.e., the bug is that when we write to a compressed orc table while specifying
a compression of SNAPPY,say, pig using HCat will write out the table using the
default orc compression, which is ZLIB, irrespective of what the metadata
indicates. This, however, is not a problem for hive in that the end data is
still readable via hive and hcatalog/pig, so we don't get a read error. The
read error occurs when external tools that are expecting the file to be
snappy-compressed find that it is actually zlib compressed. It can also be a
performance/size issue if snappy is desired over zlib, but we still retain
zlib. Thus, testing by virtue of readability/non-readability or by way of
checking for errors is not possible here.
Instead, to test, end-to-end tests are the way to go here, and I've done the
following for this:
a) Create table using hive -e, specifying orc.compress=SNAPPY
b) use pig -useHCatalog, and write to the aforesaid table.
c) use hive --service orcfiledump on the file inside the table, it will show
what compression format it sees. Without this patch, it indicates ZLIB, and
with it, it indicates SNAPPY.
In addition, no other previous tests fail (there are no regressions)
> OrcOutputFormat honors compression properties only from within hive
> -
>
> Key: HIVE-5504
> URL: https://issues.apache.org/jira/browse/HIVE-5504
> Project: Hive
> Issue Type: Bug
> Components: HCatalog
>Affects Versions: 0.11.0, 0.12.0
>Reporter: Venkat Ranganathan
> Attachments: HIVE-5504.patch
>
>
> When we import data into a HCatalog table created with the following storage
> description
> .. stored as orc tblproperties ("orc.compress"="SNAPPY")
> the resultant orc file still uses the default zlib compression
> It looks like HCatOutputFormat is ignoring the tblproperties specified.
> show tblproperties shows that the table indeed has the properties properly
> saved.
> An insert/select into the table has the resulting orc file honor the tbl
> property.
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
