[jira] [Updated] (HIVE-998) Cleanup UDF testcases
[ https://issues.apache.org/jira/browse/HIVE-998?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Carl Steinbach updated HIVE-998: Assignee: (was: Carl Steinbach) > Cleanup UDF testcases > - > > Key: HIVE-998 > URL: https://issues.apache.org/jira/browse/HIVE-998 > Project: Hive > Issue Type: Test >Reporter: Carl Steinbach > > * For every UDF x there should be a corresponding udf_x.q testcase. > * Every udf_x.q file should begin with "DESCRIBE FUNCTION X", and "DESCRIBE > EXTENDED FUNCTION X". -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (HIVE-2896) Ant eclipse target should trigger generation of CliDriver tests
[ https://issues.apache.org/jira/browse/HIVE-2896?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Carl Steinbach resolved HIVE-2896. -- Resolution: Won't Fix > Ant eclipse target should trigger generation of CliDriver tests > --- > > Key: HIVE-2896 > URL: https://issues.apache.org/jira/browse/HIVE-2896 > Project: Hive > Issue Type: Bug > Components: Testing Infrastructure >Reporter: Carl Steinbach >Assignee: Carl Steinbach > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (HIVE-2437) update project website navigation links
[ https://issues.apache.org/jira/browse/HIVE-2437?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Carl Steinbach resolved HIVE-2437. -- Resolution: Fixed > update project website navigation links > --- > > Key: HIVE-2437 > URL: https://issues.apache.org/jira/browse/HIVE-2437 > Project: Hive > Issue Type: Sub-task >Reporter: John Sichi > > http://www.apache.org/foundation/marks/pmcs.html#navigation -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (HIVE-1906) Fix Eclipse classpath and add Eclipse launch configurations for HiveServer and MetaStoreServer
[ https://issues.apache.org/jira/browse/HIVE-1906?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Carl Steinbach resolved HIVE-1906. -- Resolution: Won't Fix > Fix Eclipse classpath and add Eclipse launch configurations for HiveServer > and MetaStoreServer > -- > > Key: HIVE-1906 > URL: https://issues.apache.org/jira/browse/HIVE-1906 > Project: Hive > Issue Type: Task > Components: Build Infrastructure >Reporter: Carl Steinbach >Assignee: Carl Steinbach > Attachments: HIVE-1906.1.patch.txt, HIVE-1906.2.patch.txt > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (HIVE-2697) Ant compile-test target should be triggered from subprojects, not from top-level targets
[ https://issues.apache.org/jira/browse/HIVE-2697?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Carl Steinbach resolved HIVE-2697. -- Resolution: Won't Fix > Ant compile-test target should be triggered from subprojects, not from > top-level targets > > > Key: HIVE-2697 > URL: https://issues.apache.org/jira/browse/HIVE-2697 > Project: Hive > Issue Type: Improvement > Components: Build Infrastructure >Affects Versions: 0.8.0 >Reporter: Carl Steinbach >Assignee: Carl Steinbach > Attachments: HIVE-2697.1.patch.txt > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (HIVE-2947) Arc test -- IGNORE
[ https://issues.apache.org/jira/browse/HIVE-2947?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Carl Steinbach resolved HIVE-2947. -- Resolution: Invalid > Arc test -- IGNORE > -- > > Key: HIVE-2947 > URL: https://issues.apache.org/jira/browse/HIVE-2947 > Project: Hive > Issue Type: Bug >Reporter: Carl Steinbach >Assignee: Carl Steinbach > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (HIVE-2878) TestHBaseMinimrCliDriver hbase_bulk.m fails on 0.23
[ https://issues.apache.org/jira/browse/HIVE-2878?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Carl Steinbach resolved HIVE-2878. -- Resolution: Won't Fix > TestHBaseMinimrCliDriver hbase_bulk.m fails on 0.23 > --- > > Key: HIVE-2878 > URL: https://issues.apache.org/jira/browse/HIVE-2878 > Project: Hive > Issue Type: Bug > Components: HBase Handler, Tests >Affects Versions: 0.8.1 >Reporter: Carl Steinbach >Assignee: Carl Steinbach > Attachments: HIVE-2878.br08.1.patch.txt > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (HIVE-2877) TABLESAMPLE(x PERCENT) tests fail on 0.22/0.23
[ https://issues.apache.org/jira/browse/HIVE-2877?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Carl Steinbach resolved HIVE-2877. -- Resolution: Won't Fix > TABLESAMPLE(x PERCENT) tests fail on 0.22/0.23 > -- > > Key: HIVE-2877 > URL: https://issues.apache.org/jira/browse/HIVE-2877 > Project: Hive > Issue Type: Bug > Components: Query Processor >Reporter: Carl Steinbach >Assignee: Carl Steinbach > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (HIVE-3150) Ivy checkModified setting should be configurable
[ https://issues.apache.org/jira/browse/HIVE-3150?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Carl Steinbach resolved HIVE-3150. -- Resolution: Won't Fix Mavenization made this ticket irrelevant. > Ivy checkModified setting should be configurable > > > Key: HIVE-3150 > URL: https://issues.apache.org/jira/browse/HIVE-3150 > Project: Hive > Issue Type: Bug > Components: Build Infrastructure >Reporter: Carl Steinbach >Assignee: Carl Steinbach > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (HIVE-2672) CLI fails to start when run on Hadoop 0.23.0
[ https://issues.apache.org/jira/browse/HIVE-2672?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Carl Steinbach resolved HIVE-2672. -- Resolution: Won't Fix No longer relevant. > CLI fails to start when run on Hadoop 0.23.0 > > > Key: HIVE-2672 > URL: https://issues.apache.org/jira/browse/HIVE-2672 > Project: Hive > Issue Type: Bug > Components: CLI, Shims >Reporter: Carl Steinbach >Assignee: Carl Steinbach > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (HIVE-3201) Thrift build target should clean generated source directories
[ https://issues.apache.org/jira/browse/HIVE-3201?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Carl Steinbach resolved HIVE-3201. -- Resolution: Invalid This ticket was made irrelevant by the switch to Maven. > Thrift build target should clean generated source directories > - > > Key: HIVE-3201 > URL: https://issues.apache.org/jira/browse/HIVE-3201 > Project: Hive > Issue Type: Bug > Components: Build Infrastructure >Reporter: Carl Steinbach >Assignee: Carl Steinbach > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-8933) Check release builds for SNAPSHOT dependencies
[ https://issues.apache.org/jira/browse/HIVE-8933?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Carl Steinbach updated HIVE-8933: - Resolution: Fixed Fix Version/s: 0.14.1 0.15.0 Hadoop Flags: Reviewed Status: Resolved (was: Patch Available) Alan, thanks for the review. Committed to trunk and branch-0.14. > Check release builds for SNAPSHOT dependencies > -- > > Key: HIVE-8933 > URL: https://issues.apache.org/jira/browse/HIVE-8933 > Project: Hive > Issue Type: Improvement > Components: Build Infrastructure >Reporter: Carl Steinbach >Assignee: Carl Steinbach > Fix For: 0.15.0, 0.14.1 > > Attachments: HIVE-8933.1.patch.txt > > > Hive 0.14.0 was released with SNAPSHOT dependencies. We should use the maven > enforcer plugin to prevent this from happening again in the future. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-8933) Check release builds for SNAPSHOT dependencies
[ https://issues.apache.org/jira/browse/HIVE-8933?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Carl Steinbach updated HIVE-8933: - Status: Patch Available (was: Open) RB: https://reviews.apache.org/r/28308/ [~alangates], would you mind reviewing this? Thanks! > Check release builds for SNAPSHOT dependencies > -- > > Key: HIVE-8933 > URL: https://issues.apache.org/jira/browse/HIVE-8933 > Project: Hive > Issue Type: Improvement > Components: Build Infrastructure >Reporter: Carl Steinbach >Assignee: Carl Steinbach > Attachments: HIVE-8933.1.patch.txt > > > Hive 0.14.0 was released with SNAPSHOT dependencies. We should use the maven > enforcer plugin to prevent this from happening again in the future. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-8933) Check release builds for SNAPSHOT dependencies
[ https://issues.apache.org/jira/browse/HIVE-8933?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Carl Steinbach updated HIVE-8933: - Attachment: HIVE-8933.1.patch.txt > Check release builds for SNAPSHOT dependencies > -- > > Key: HIVE-8933 > URL: https://issues.apache.org/jira/browse/HIVE-8933 > Project: Hive > Issue Type: Improvement > Components: Build Infrastructure >Reporter: Carl Steinbach >Assignee: Carl Steinbach > Attachments: HIVE-8933.1.patch.txt > > > Hive 0.14.0 was released with SNAPSHOT dependencies. We should use the maven > enforcer plugin to prevent this from happening again in the future. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HIVE-8933) Check release builds for SNAPSHOT dependencies
Carl Steinbach created HIVE-8933: Summary: Check release builds for SNAPSHOT dependencies Key: HIVE-8933 URL: https://issues.apache.org/jira/browse/HIVE-8933 Project: Hive Issue Type: Improvement Components: Build Infrastructure Reporter: Carl Steinbach Assignee: Carl Steinbach Hive 0.14.0 was released with SNAPSHOT dependencies. We should use the maven enforcer plugin to prevent this from happening again in the future. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-8873) Switch to calcite 0.9.2
[ https://issues.apache.org/jira/browse/HIVE-8873?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Carl Steinbach updated HIVE-8873: - Affects Version/s: 0.14.0 > Switch to calcite 0.9.2 > --- > > Key: HIVE-8873 > URL: https://issues.apache.org/jira/browse/HIVE-8873 > Project: Hive > Issue Type: Bug >Affects Versions: 0.14.0 >Reporter: Gunther Hagleitner >Assignee: Gunther Hagleitner > Fix For: 0.14.1 > > Attachments: HIVE-8873.1.patch > > > Calcite release 0.9.2 is out. We should update pom to use it. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-8845) Switch to Tez 0.5.2
[ https://issues.apache.org/jira/browse/HIVE-8845?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Carl Steinbach updated HIVE-8845: - Affects Version/s: 0.14.0 > Switch to Tez 0.5.2 > --- > > Key: HIVE-8845 > URL: https://issues.apache.org/jira/browse/HIVE-8845 > Project: Hive > Issue Type: Bug >Affects Versions: 0.14.0 >Reporter: Gunther Hagleitner >Assignee: Gunther Hagleitner > Fix For: 0.14.1 > > Attachments: HIVE-8845.1.patch > > > Tez 0.5.2 has been released, we should switch our pom to that. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HIVE-8906) Hive 0.14.0 release depends on Tez and Calcite SNAPSHOT artifacts
Carl Steinbach created HIVE-8906: Summary: Hive 0.14.0 release depends on Tez and Calcite SNAPSHOT artifacts Key: HIVE-8906 URL: https://issues.apache.org/jira/browse/HIVE-8906 Project: Hive Issue Type: Bug Reporter: Carl Steinbach The Hive 0.14.0 release depends on SNAPSHOT versions of tez-0.5.2 and calcite-0.9.2. I believe this violates Apache release policy (can't find the reference, but I seem to remember this being a problem with HCatalog before the merger), and it implies that the folks who tested the release weren't necessarily testing the same thing. It also means that people who try to build Hive using the 0.14.0 src release will encounter errors unless they configure Maven to pull artifacts from the snapshot repository. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-2390) Add UNIONTYPE serialization support to LazyBinarySerDe
[
https://issues.apache.org/jira/browse/HIVE-2390?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14128124#comment-14128124
]
Carl Steinbach commented on HIVE-2390:
--
I updated the description of this ticket to accurately reflect the change that
was made in this patch.
My impression is that this patch doesn't really change the situation in Hive
with respect to UNIONTYPEs -- this feature is still unusable. If I'm wrong
about this I would appreciate someone setting me straight.
> Add UNIONTYPE serialization support to LazyBinarySerDe
> --
>
> Key: HIVE-2390
> URL: https://issues.apache.org/jira/browse/HIVE-2390
> Project: Hive
> Issue Type: Bug
>Affects Versions: 0.13.1
>Reporter: Jakob Homan
>Assignee: Suma Shivaprasad
> Labels: TODOC14, uniontype
> Fix For: 0.14.0
>
> Attachments: HIVE-2390.1.patch, HIVE-2390.patch
>
>
> When the union type was introduced, full support for it wasn't provided. For
> instance, when working with a union that gets passed to LazyBinarySerde:
> {noformat}Caused by: java.lang.RuntimeException: Unrecognized type: UNION
> at
> org.apache.hadoop.hive.serde2.lazybinary.LazyBinarySerDe.serialize(LazyBinarySerDe.java:468)
> at
> org.apache.hadoop.hive.serde2.lazybinary.LazyBinarySerDe.serializeStruct(LazyBinarySerDe.java:230)
> at
> org.apache.hadoop.hive.serde2.lazybinary.LazyBinarySerDe.serialize(LazyBinarySerDe.java:184)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-2390) Add UNIONTYPE serialization support to LazyBinarySerDe
[
https://issues.apache.org/jira/browse/HIVE-2390?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Carl Steinbach updated HIVE-2390:
-
Summary: Add UNIONTYPE serialization support to LazyBinarySerDe (was:
Expand support for union types)
> Add UNIONTYPE serialization support to LazyBinarySerDe
> --
>
> Key: HIVE-2390
> URL: https://issues.apache.org/jira/browse/HIVE-2390
> Project: Hive
> Issue Type: Bug
>Affects Versions: 0.13.1
>Reporter: Jakob Homan
>Assignee: Suma Shivaprasad
> Labels: TODOC14, uniontype
> Fix For: 0.14.0
>
> Attachments: HIVE-2390.1.patch, HIVE-2390.patch
>
>
> When the union type was introduced, full support for it wasn't provided. For
> instance, when working with a union that gets passed to LazyBinarySerde:
> {noformat}Caused by: java.lang.RuntimeException: Unrecognized type: UNION
> at
> org.apache.hadoop.hive.serde2.lazybinary.LazyBinarySerDe.serialize(LazyBinarySerDe.java:468)
> at
> org.apache.hadoop.hive.serde2.lazybinary.LazyBinarySerDe.serializeStruct(LazyBinarySerDe.java:230)
> at
> org.apache.hadoop.hive.serde2.lazybinary.LazyBinarySerDe.serialize(LazyBinarySerDe.java:184)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-4629) HS2 should support an API to retrieve query logs
[ https://issues.apache.org/jira/browse/HIVE-4629?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14106098#comment-14106098 ] Carl Steinbach commented on HIVE-4629: -- bq. I tried posting this on RB but it went down. Thank you very much for removing the thrift enum compatibility problem! I had another comment with regards to the method signature which I think I did not explain well. I think the new method should be... [~brocknoland], I totally agree with this, but I didn't see this in the patch. Are you referring to something in the Thrift IDL file or something else? > HS2 should support an API to retrieve query logs > > > Key: HIVE-4629 > URL: https://issues.apache.org/jira/browse/HIVE-4629 > Project: Hive > Issue Type: Sub-task > Components: HiveServer2 >Reporter: Shreepadma Venugopalan >Assignee: Dong Chen > Attachments: HIVE-4629-no_thrift.1.patch, HIVE-4629.1.patch, > HIVE-4629.2.patch, HIVE-4629.3.patch.txt, HIVE-4629.4.patch, > HIVE-4629.5.patch, HIVE-4629.6.patch, HIVE-4629.7.patch > > > HiveServer2 should support an API to retrieve query logs. This is > particularly relevant because HiveServer2 supports async execution but > doesn't provide a way to report progress. Providing an API to retrieve query > logs will help report progress to the client. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HIVE-7420) Parameterize tests for HCatalog Pig interfaces for testing against all storage formats
[ https://issues.apache.org/jira/browse/HIVE-7420?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Carl Steinbach updated HIVE-7420: - Status: Open (was: Patch Available) Hi [~davidzchen], I left some comments on reviewboard, but also want to mention a couple issues here on JIRA. Please excuse the bullet points: * The automated tests flagged failures in TestHCatLoader, TestHCatLoaderComplexSchema, TestHCatStorer, and TestHCatStorerMulti. We can't commit this patch until these test failures are resolved. * This patch adds support for disabling all of the test methods in a particular testcase on a per-storageformat basis, but I think we really need the ability to disable individual test methods on a per-storageformat basis. It should be possible to do this by maintaining a Map> and using reflection to find the name of test method. * For parameterized tests that fail, JUnit reports the parameter index, but not the actual parameter values (see report above). The latter would be a lot nicer. There's a [JUnitParams|https://github.com/Pragmatists/junitparams] project on Github that doesn't suck (according to the author), and which I think may report the actual parameter values as opposed to the parameter indexes. If you have time please take a look at this and see if it's worth using here. > Parameterize tests for HCatalog Pig interfaces for testing against all > storage formats > -- > > Key: HIVE-7420 > URL: https://issues.apache.org/jira/browse/HIVE-7420 > Project: Hive > Issue Type: Sub-task > Components: HCatalog >Reporter: David Chen >Assignee: David Chen > Attachments: HIVE-7420-without-HIVE-7457.2.patch, > HIVE-7420-without-HIVE-7457.3.patch, HIVE-7420.1.patch, HIVE-7420.2.patch, > HIVE-7420.3.patch > > > Currently, HCatalog tests only test against RCFile with a few testing against > ORC. The tests should be covering other Hive storage formats as well. > HIVE-7286 turns HCatMapReduceTest into a test fixture that can be run with > all Hive storage formats and with that patch, all test suites built on > HCatMapReduceTest are running and passing against Sequence File, Text, and > ORC in addition to RCFile. > Similar changes should be made to make the tests for HCatLoader and > HCatStorer generic so that they can be run against all Hive storage formats. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HIVE-7457) Minor HCatalog Pig Adapter test clean up
[ https://issues.apache.org/jira/browse/HIVE-7457?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Carl Steinbach updated HIVE-7457: - Status: Open (was: Patch Available) [~davidzchen], the TestHCatLoader test failure needs some investigation. Also, can you please summarize the changes you made to TestOrcHCatPigStorer/TestOrcHCatStorerMulti/TestOrcHCatStorer? I'm not sure RB is comparing the right files in the diff. Thanks. > Minor HCatalog Pig Adapter test clean up > > > Key: HIVE-7457 > URL: https://issues.apache.org/jira/browse/HIVE-7457 > Project: Hive > Issue Type: Sub-task >Reporter: David Chen >Assignee: David Chen >Priority: Minor > Attachments: HIVE-7457.1.patch, HIVE-7457.2.patch, HIVE-7457.3.patch, > HIVE-7457.4.patch > > > Minor cleanup to the HCatalog Pig Adapter tests in preparation for HIVE-7420: > * Run through Hive Eclipse formatter. > * Convert JUnit 3-style tests to follow JUnit 4 conventions. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HIVE-7286) Parameterize HCatMapReduceTest for testing against all Hive storage formats
[ https://issues.apache.org/jira/browse/HIVE-7286?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Carl Steinbach updated HIVE-7286: - Resolution: Fixed Fix Version/s: 0.14.0 Status: Resolved (was: Patch Available) Committed to trunk. Thanks David! > Parameterize HCatMapReduceTest for testing against all Hive storage formats > --- > > Key: HIVE-7286 > URL: https://issues.apache.org/jira/browse/HIVE-7286 > Project: Hive > Issue Type: Sub-task > Components: HCatalog >Reporter: David Chen >Assignee: David Chen > Fix For: 0.14.0 > > Attachments: HIVE-7286.1.patch, HIVE-7286.10.patch, > HIVE-7286.11.patch, HIVE-7286.2.patch, HIVE-7286.3.patch, HIVE-7286.4.patch, > HIVE-7286.5.patch, HIVE-7286.6.patch, HIVE-7286.7.patch, HIVE-7286.8.patch, > HIVE-7286.9.patch > > > Currently, HCatMapReduceTest, which is extended by the following test suites: > * TestHCatDynamicPartitioned > * TestHCatNonPartitioned > * TestHCatPartitioned > * TestHCatExternalDynamicPartitioned > * TestHCatExternalNonPartitioned > * TestHCatExternalPartitioned > * TestHCatMutableDynamicPartitioned > * TestHCatMutableNonPartitioned > * TestHCatMutablePartitioned > These tests run against RCFile. Currently, only TestHCatDynamicPartitioned is > run against any other storage format (ORC). > Ideally, HCatalog should be tested against all storage formats supported by > Hive. The easiest way to accomplish this is to turn HCatMapReduceTest into a > parameterized test fixture that enumerates all Hive storage formats. Until > HIVE-5976 is implemented, we would need to manually create the mapping of > SerDe to InputFormat and OutputFormat. This way, we can explicitly keep track > of which storage formats currently work with HCatalog or which ones are > untested or have test failures. The test fixture should also use Reflection > to find all classes in the classpath that implements the SerDe interface and > raise a failure if any of them are not enumerated. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HIVE-6806) CREATE TABLE should support STORED AS AVRO
[ https://issues.apache.org/jira/browse/HIVE-6806?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Carl Steinbach updated HIVE-6806: - Resolution: Fixed Fix Version/s: 0.14.0 Hadoop Flags: Reviewed Status: Resolved (was: Patch Available) Committed to trunk. Thanks Ashish! > CREATE TABLE should support STORED AS AVRO > -- > > Key: HIVE-6806 > URL: https://issues.apache.org/jira/browse/HIVE-6806 > Project: Hive > Issue Type: New Feature > Components: Serializers/Deserializers >Affects Versions: 0.12.0 >Reporter: Jeremy Beard >Assignee: Ashish Kumar Singh >Priority: Minor > Labels: Avro > Fix For: 0.14.0 > > Attachments: HIVE-6806.1.patch, HIVE-6806.2.patch, HIVE-6806.3.patch, > HIVE-6806.patch > > > Avro is well established and widely used within Hive, however creating > Avro-backed tables requires the messy listing of the SerDe, InputFormat and > OutputFormat classes. > Similarly to HIVE-5783 for Parquet, Hive would be easier to use if it had > native Avro support. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HIVE-6806) CREATE TABLE should support STORED AS AVRO
[ https://issues.apache.org/jira/browse/HIVE-6806?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14071321#comment-14071321 ] Carl Steinbach commented on HIVE-6806: -- [~tomwhite], [~lars_francke], [~brocknoland]: Are you guys satisfied with the current version of the patch? If so I'll plan to +1 it and get it committed after another round of automated tests. Thanks. > CREATE TABLE should support STORED AS AVRO > -- > > Key: HIVE-6806 > URL: https://issues.apache.org/jira/browse/HIVE-6806 > Project: Hive > Issue Type: New Feature > Components: Serializers/Deserializers >Affects Versions: 0.12.0 >Reporter: Jeremy Beard >Assignee: Ashish Kumar Singh >Priority: Minor > Labels: Avro > Attachments: HIVE-6806.1.patch, HIVE-6806.2.patch, HIVE-6806.patch > > > Avro is well established and widely used within Hive, however creating > Avro-backed tables requires the messy listing of the SerDe, InputFormat and > OutputFormat classes. > Similarly to HIVE-5783 for Parquet, Hive would be easier to use if it had > native Avro support. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HIVE-6806) CREATE TABLE should support STORED AS AVRO
[ https://issues.apache.org/jira/browse/HIVE-6806?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Carl Steinbach updated HIVE-6806: - Summary: CREATE TABLE should support STORED AS AVRO (was: Native Avro support in Hive) > CREATE TABLE should support STORED AS AVRO > -- > > Key: HIVE-6806 > URL: https://issues.apache.org/jira/browse/HIVE-6806 > Project: Hive > Issue Type: New Feature > Components: Serializers/Deserializers >Affects Versions: 0.12.0 >Reporter: Jeremy Beard >Assignee: Ashish Kumar Singh >Priority: Minor > Labels: Avro > Attachments: HIVE-6806.1.patch, HIVE-6806.2.patch, HIVE-6806.patch > > > Avro is well established and widely used within Hive, however creating > Avro-backed tables requires the messy listing of the SerDe, InputFormat and > OutputFormat classes. > Similarly to HIVE-5783 for Parquet, Hive would be easier to use if it had > native Avro support. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HIVE-6806) Native Avro support in Hive
[ https://issues.apache.org/jira/browse/HIVE-6806?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14060995#comment-14060995 ] Carl Steinbach commented on HIVE-6806: -- Does anyone object to changing the summary of this ticket to "CREATE TABLE should support STORED AS AVRO"? The current description can be misinterpreted to mean that this patch is adding the AvroSerDe. > Native Avro support in Hive > --- > > Key: HIVE-6806 > URL: https://issues.apache.org/jira/browse/HIVE-6806 > Project: Hive > Issue Type: New Feature > Components: Serializers/Deserializers >Affects Versions: 0.12.0 >Reporter: Jeremy Beard >Assignee: Ashish Kumar Singh >Priority: Minor > Labels: Avro > Attachments: HIVE-6806.patch > > > Avro is well established and widely used within Hive, however creating > Avro-backed tables requires the messy listing of the SerDe, InputFormat and > OutputFormat classes. > Similarly to HIVE-5783 for Parquet, Hive would be easier to use if it had > native Avro support. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HIVE-5976) Decouple input formats from STORED as keywords
[ https://issues.apache.org/jira/browse/HIVE-5976?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Carl Steinbach updated HIVE-5976: - Status: Open (was: Patch Available) [~davidzchen], I'd like to review this and get it committed. Can you post an RB? Thanks! > Decouple input formats from STORED as keywords > -- > > Key: HIVE-5976 > URL: https://issues.apache.org/jira/browse/HIVE-5976 > Project: Hive > Issue Type: Task >Reporter: Brock Noland >Assignee: Brock Noland > Attachments: HIVE-5976.2.patch, HIVE-5976.patch, HIVE-5976.patch, > HIVE-5976.patch, HIVE-5976.patch > > > As noted in HIVE-5783, we hard code the input formats mapped to keywords. > It'd be nice if there was a registration system so we didn't need to do that. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HIVE-7094) Separate out static/dynamic partitioning code in FileRecordWriterContainer
[
https://issues.apache.org/jira/browse/HIVE-7094?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Carl Steinbach updated HIVE-7094:
-
Resolution: Fixed
Fix Version/s: 0.14.0
Hadoop Flags: Reviewed
Status: Resolved (was: Patch Available)
Committed to trunk. Thanks David!
> Separate out static/dynamic partitioning code in FileRecordWriterContainer
> --
>
> Key: HIVE-7094
> URL: https://issues.apache.org/jira/browse/HIVE-7094
> Project: Hive
> Issue Type: Sub-task
> Components: HCatalog
>Reporter: David Chen
>Assignee: David Chen
> Fix For: 0.14.0
>
> Attachments: HIVE-7094.1.patch, HIVE-7094.3.patch, HIVE-7094.4.patch,
> HIVE-7094.5.patch
>
>
> There are two major places in FileRecordWriterContainer that have the {{if
> (dynamicPartitioning)}} condition: the constructor and write().
> This is the approach that I am taking:
> # Move the DP and SP code into two subclasses:
> DynamicFileRecordWriterContainer and StaticFileRecordWriterContainer.
> # Make FileRecordWriterContainer an abstract class that contains the common
> code for both implementations. For write(), FileRecordWriterContainer will
> call an abstract method that will provide the local RecordWriter,
> ObjectInspector, SerDe, and OutputJobInfo.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HIVE-7094) Separate out static/dynamic partitioning code in FileRecordWriterContainer
[
https://issues.apache.org/jira/browse/HIVE-7094?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14037893#comment-14037893
]
Carl Steinbach commented on HIVE-7094:
--
[~sushanth]: I'm planning to commit this patch tonight. Please let me know if I
should hold off. Thanks.
> Separate out static/dynamic partitioning code in FileRecordWriterContainer
> --
>
> Key: HIVE-7094
> URL: https://issues.apache.org/jira/browse/HIVE-7094
> Project: Hive
> Issue Type: Sub-task
> Components: HCatalog
>Reporter: David Chen
>Assignee: David Chen
> Attachments: HIVE-7094.1.patch, HIVE-7094.3.patch, HIVE-7094.4.patch,
> HIVE-7094.5.patch
>
>
> There are two major places in FileRecordWriterContainer that have the {{if
> (dynamicPartitioning)}} condition: the constructor and write().
> This is the approach that I am taking:
> # Move the DP and SP code into two subclasses:
> DynamicFileRecordWriterContainer and StaticFileRecordWriterContainer.
> # Make FileRecordWriterContainer an abstract class that contains the common
> code for both implementations. For write(), FileRecordWriterContainer will
> call an abstract method that will provide the local RecordWriter,
> ObjectInspector, SerDe, and OutputJobInfo.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Updated] (HIVE-7094) Separate out static/dynamic partitioning code in FileRecordWriterContainer
[
https://issues.apache.org/jira/browse/HIVE-7094?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Carl Steinbach updated HIVE-7094:
-
Status: Open (was: Patch Available)
Looks like some tests failed. [~davidzchen], can you please take a look? Thanks.
> Separate out static/dynamic partitioning code in FileRecordWriterContainer
> --
>
> Key: HIVE-7094
> URL: https://issues.apache.org/jira/browse/HIVE-7094
> Project: Hive
> Issue Type: Sub-task
> Components: HCatalog
>Reporter: David Chen
>Assignee: David Chen
> Attachments: HIVE-7094.1.patch, HIVE-7094.3.patch, HIVE-7094.4.patch
>
>
> There are two major places in FileRecordWriterContainer that have the {{if
> (dynamicPartitioning)}} condition: the constructor and write().
> This is the approach that I am taking:
> # Move the DP and SP code into two subclasses:
> DynamicFileRecordWriterContainer and StaticFileRecordWriterContainer.
> # Make FileRecordWriterContainer an abstract class that contains the common
> code for both implementations. For write(), FileRecordWriterContainer will
> call an abstract method that will provide the local RecordWriter,
> ObjectInspector, SerDe, and OutputJobInfo.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HIVE-7094) Separate out static/dynamic partitioning code in FileRecordWriterContainer
[
https://issues.apache.org/jira/browse/HIVE-7094?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14031188#comment-14031188
]
Carl Steinbach commented on HIVE-7094:
--
[~davidzchen]: +1. Can you please attach a new version of the patch to trigger
testing? If everything passes I will commit.
> Separate out static/dynamic partitioning code in FileRecordWriterContainer
> --
>
> Key: HIVE-7094
> URL: https://issues.apache.org/jira/browse/HIVE-7094
> Project: Hive
> Issue Type: Sub-task
> Components: HCatalog
>Reporter: David Chen
>Assignee: David Chen
> Attachments: HIVE-7094.1.patch
>
>
> There are two major places in FileRecordWriterContainer that have the {{if
> (dynamicPartitioning)}} condition: the constructor and write().
> This is the approach that I am taking:
> # Move the DP and SP code into two subclasses:
> DynamicFileRecordWriterContainer and StaticFileRecordWriterContainer.
> # Make FileRecordWriterContainer an abstract class that contains the common
> code for both implementations. For write(), FileRecordWriterContainer will
> call an abstract method that will provide the local RecordWriter,
> ObjectInspector, SerDe, and OutputJobInfo.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Updated] (HIVE-7094) Separate out static/dynamic partitioning code in FileRecordWriterContainer
[
https://issues.apache.org/jira/browse/HIVE-7094?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Carl Steinbach updated HIVE-7094:
-
Status: Open (was: Patch Available)
[~davidzchen]: I left some comments on rb. Thanks.
> Separate out static/dynamic partitioning code in FileRecordWriterContainer
> --
>
> Key: HIVE-7094
> URL: https://issues.apache.org/jira/browse/HIVE-7094
> Project: Hive
> Issue Type: Sub-task
> Components: HCatalog
>Reporter: David Chen
>Assignee: David Chen
> Attachments: HIVE-7094.1.patch
>
>
> There are two major places in FileRecordWriterContainer that have the {{if
> (dynamicPartitioning)}} condition: the constructor and write().
> This is the approach that I am taking:
> # Move the DP and SP code into two subclasses:
> DynamicFileRecordWriterContainer and StaticFileRecordWriterContainer.
> # Make FileRecordWriterContainer an abstract class that contains the common
> code for both implementations. For write(), FileRecordWriterContainer will
> call an abstract method that will provide the local RecordWriter,
> ObjectInspector, SerDe, and OutputJobInfo.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Updated] (HIVE-4734) Use custom ObjectInspectors for AvroSerde
[ https://issues.apache.org/jira/browse/HIVE-4734?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Carl Steinbach updated HIVE-4734: - Status: Open (was: Patch Available) [~mwagner]: I'd like to get this committed. Would you mind rebasing the patch against trunk? Thanks. > Use custom ObjectInspectors for AvroSerde > - > > Key: HIVE-4734 > URL: https://issues.apache.org/jira/browse/HIVE-4734 > Project: Hive > Issue Type: Improvement > Components: Serializers/Deserializers >Reporter: Mark Wagner >Assignee: Mark Wagner > Attachments: HIVE-4734.1.patch, HIVE-4734.2.patch, HIVE-4734.3.patch, > HIVE-4734.4.patch, HIVE-4734.5.patch > > > Currently, the AvroSerde recursively copies all fields of a record from the > GenericRecord to a List row object and provides the standard > ObjectInspectors. Performance can be improved by providing ObjectInspectors > to the Avro record itself. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HIVE-7094) Separate out static/dynamic partitioning code in FileRecordWriterContainer
[
https://issues.apache.org/jira/browse/HIVE-7094?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Carl Steinbach updated HIVE-7094:
-
Component/s: HCatalog
> Separate out static/dynamic partitioning code in FileRecordWriterContainer
> --
>
> Key: HIVE-7094
> URL: https://issues.apache.org/jira/browse/HIVE-7094
> Project: Hive
> Issue Type: Sub-task
> Components: HCatalog
>Reporter: David Chen
>Assignee: David Chen
> Attachments: HIVE-7094.1.patch
>
>
> There are two major places in FileRecordWriterContainer that have the {{if
> (dynamicPartitioning)}} condition: the constructor and write().
> This is the approach that I am taking:
> # Move the DP and SP code into two subclasses:
> DynamicFileRecordWriterContainer and StaticFileRecordWriterContainer.
> # Make FileRecordWriterContainer an abstract class that contains the common
> code for both implementations. For write(), FileRecordWriterContainer will
> call an abstract method that will provide the local RecordWriter,
> ObjectInspector, SerDe, and OutputJobInfo.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Updated] (HIVE-7110) TestHCatPartitionPublish test failure: No FileSystem or scheme: pfile
[
https://issues.apache.org/jira/browse/HIVE-7110?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Carl Steinbach updated HIVE-7110:
-
Status: Open (was: Patch Available)
[~davidzchen]: Thanks for looking into this. Before we fix this problem I think
it's important to determine whether or not the test ever worked, and if so,
when and why it stopped working. I'm particularly curious to know if these
properties were previously getting set by Maven in a manner that made them
accessible to all HCatalog tests. Can you please look into this? Thanks.
> TestHCatPartitionPublish test failure: No FileSystem or scheme: pfile
> -
>
> Key: HIVE-7110
> URL: https://issues.apache.org/jira/browse/HIVE-7110
> Project: Hive
> Issue Type: Bug
> Components: HCatalog
>Reporter: David Chen
>Assignee: David Chen
> Attachments: HIVE-7110.1.patch, HIVE-7110.2.patch, HIVE-7110.3.patch,
> HIVE-7110.4.patch
>
>
> I got the following TestHCatPartitionPublish test failure when running all
> unit tests against Hadoop 1. This also appears when testing against Hadoop 2.
> {code}
> Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 26.06 sec
> <<< FAILURE! - in org.apache.hive.hcatalog.mapreduce.TestHCatPartitionPublish
> testPartitionPublish(org.apache.hive.hcatalog.mapreduce.TestHCatPartitionPublish)
> Time elapsed: 1.361 sec <<< ERROR!
> org.apache.hive.hcatalog.common.HCatException:
> org.apache.hive.hcatalog.common.HCatException : 2001 : Error setting output
> information. Cause : java.io.IOException: No FileSystem for scheme: pfile
> at
> org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:1443)
> at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:67)
> at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:1464)
> at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:263)
> at org.apache.hadoop.fs.Path.getFileSystem(Path.java:187)
> at
> org.apache.hive.hcatalog.mapreduce.HCatOutputFormat.setOutput(HCatOutputFormat.java:212)
> at
> org.apache.hive.hcatalog.mapreduce.HCatOutputFormat.setOutput(HCatOutputFormat.java:70)
> at
> org.apache.hive.hcatalog.mapreduce.TestHCatPartitionPublish.runMRCreateFail(TestHCatPartitionPublish.java:191)
> at
> org.apache.hive.hcatalog.mapreduce.TestHCatPartitionPublish.testPartitionPublish(TestHCatPartitionPublish.java:155)
> {code}
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HIVE-7094) Separate out static/dynamic partitioning code in FileRecordWriterContainer
[
https://issues.apache.org/jira/browse/HIVE-7094?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14016200#comment-14016200
]
Carl Steinbach commented on HIVE-7094:
--
[~davidzchen]: Is this ready for review now?
> Separate out static/dynamic partitioning code in FileRecordWriterContainer
> --
>
> Key: HIVE-7094
> URL: https://issues.apache.org/jira/browse/HIVE-7094
> Project: Hive
> Issue Type: Sub-task
>Reporter: David Chen
>Assignee: David Chen
> Attachments: HIVE-7094.1.patch
>
>
> There are two major places in FileRecordWriterContainer that have the {{if
> (dynamicPartitioning)}} condition: the constructor and write().
> This is the approach that I am taking:
> # Move the DP and SP code into two subclasses:
> DynamicFileRecordWriterContainer and StaticFileRecordWriterContainer.
> # Make FileRecordWriterContainer an abstract class that contains the common
> code for both implementations. For write(), FileRecordWriterContainer will
> call an abstract method that will provide the local RecordWriter,
> ObjectInspector, SerDe, and OutputJobInfo.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HIVE-4629) HS2 should support an API to retrieve query logs
[ https://issues.apache.org/jira/browse/HIVE-4629?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14014172#comment-14014172 ] Carl Steinbach commented on HIVE-4629: -- Does anyone think this is the right way to implement this feature? > HS2 should support an API to retrieve query logs > > > Key: HIVE-4629 > URL: https://issues.apache.org/jira/browse/HIVE-4629 > Project: Hive > Issue Type: Sub-task > Components: HiveServer2 >Reporter: Shreepadma Venugopalan >Assignee: Shreepadma Venugopalan > Attachments: HIVE-4629-no_thrift.1.patch, HIVE-4629.1.patch, > HIVE-4629.2.patch, HIVE-4629.3.patch.txt > > > HiveServer2 should support an API to retrieve query logs. This is > particularly relevant because HiveServer2 supports async execution but > doesn't provide a way to report progress. Providing an API to retrieve query > logs will help report progress to the client. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HIVE-7094) Separate out static/dynamic partitioning code in FileRecordWriterContainer
[
https://issues.apache.org/jira/browse/HIVE-7094?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14013214#comment-14013214
]
Carl Steinbach commented on HIVE-7094:
--
[~davidzchen]: Is this patch ready for review?
> Separate out static/dynamic partitioning code in FileRecordWriterContainer
> --
>
> Key: HIVE-7094
> URL: https://issues.apache.org/jira/browse/HIVE-7094
> Project: Hive
> Issue Type: Sub-task
>Reporter: David Chen
>Assignee: David Chen
> Attachments: HIVE-7094.1.patch
>
>
> There are two major places in FileRecordWriterContainer that have the {{if
> (dynamicPartitioning)}} condition: the constructor and write().
> This is the approach that I am taking:
> # Move the DP and SP code into two subclasses:
> DynamicFileRecordWriterContainer and StaticFileRecordWriterContainer.
> # Make FileRecordWriterContainer an abstract class that contains the common
> code for both implementations. For write(), FileRecordWriterContainer will
> call an abstract method that will provide the local RecordWriter,
> ObjectInspector, SerDe, and OutputJobInfo.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Updated] (HIVE-5857) Reduce tasks do not work in uber mode in YARN
[
https://issues.apache.org/jira/browse/HIVE-5857?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Carl Steinbach updated HIVE-5857:
-
Affects Version/s: 0.13.1
0.13.0
> Reduce tasks do not work in uber mode in YARN
> -
>
> Key: HIVE-5857
> URL: https://issues.apache.org/jira/browse/HIVE-5857
> Project: Hive
> Issue Type: Bug
> Components: Query Processor
>Affects Versions: 0.12.0, 0.13.0, 0.13.1
>Reporter: Adam Kawa
>Assignee: Adam Kawa
>Priority: Critical
> Labels: plan, uber-jar, uberization, yarn
> Attachments: HIVE-5857.1.patch.txt
>
>
> A Hive query fails when it tries to run a reduce task in uber mode in YARN.
> The NullPointerException is thrown in the ExecReducer.configure method,
> because the plan file (reduce.xml) for a reduce task is not found.
> The Utilities.getBaseWork method is expected to return BaseWork object, but
> it returns NULL due to FileNotFoundException.
> {code}
> // org.apache.hadoop.hive.ql.exec.Utilities
> public static BaseWork getBaseWork(Configuration conf, String name) {
> ...
> try {
> ...
> if (gWork == null) {
> Path localPath;
> if (ShimLoader.getHadoopShims().isLocalMode(conf)) {
> localPath = path;
> } else {
> localPath = new Path(name);
> }
> InputStream in = new FileInputStream(localPath.toUri().getPath());
> BaseWork ret = deserializePlan(in);
>
> }
> return gWork;
> } catch (FileNotFoundException fnf) {
> // happens. e.g.: no reduce work.
> LOG.debug("No plan file found: "+path);
> return null;
> } ...
> }
> {code}
> It happens because, the ShimLoader.getHadoopShims().isLocalMode(conf)) method
> returns true, because immediately before running a reduce task,
> org.apache.hadoop.mapred.LocalContainerLauncher changes its configuration to
> local mode ("mapreduce.framework.name" is changed from" "yarn" to "local").
> On the other hand map tasks run successfully, because its configuration is
> not changed and still remains "yarn".
> {code}
> // org.apache.hadoop.mapred.LocalContainerLauncher
> private void runSubtask(..) {
> ...
> conf.set(MRConfig.FRAMEWORK_NAME, MRConfig.LOCAL_FRAMEWORK_NAME);
> conf.set(MRConfig.MASTER_ADDRESS, "local"); // bypass shuffle
> ReduceTask reduce = (ReduceTask)task;
> reduce.setConf(conf);
> reduce.run(conf, umbilical);
> }
> {code}
> A super quick fix could just an additional if-branch, where we check if we
> run a reduce task in uber mode, and then look for a plan file in a different
> location.
> *Java stacktrace*
> {code}
> 2013-11-20 00:50:56,862 INFO [uber-SubtaskRunner]
> org.apache.hadoop.hive.ql.exec.Utilities: No plan file found:
> hdfs://namenode.c.lon.spotify.net:54310/var/tmp/kawaa/hive_2013-11-20_00-50-43_888_3938384086824086680-2/-mr-10003/e3caacf6-15d6-4987-b186-d2906791b5b0/reduce.xml
> 2013-11-20 00:50:56,862 WARN [uber-SubtaskRunner]
> org.apache.hadoop.mapred.LocalContainerLauncher: Exception running local
> (uberized) 'child' : java.lang.RuntimeException: Error in configuring object
> at
> org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:109)
> at
> org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:75)
> at
> org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:133)
> at
> org.apache.hadoop.mapred.ReduceTask.runOldReducer(ReduceTask.java:427)
> at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:408)
> at
> org.apache.hadoop.mapred.LocalContainerLauncher$SubtaskRunner.runSubtask(LocalContainerLauncher.java:340)
> at
> org.apache.hadoop.mapred.LocalContainerLauncher$SubtaskRunner.run(LocalContainerLauncher.java:225)
> at java.lang.Thread.run(Thread.java:662)
> Caused by: java.lang.reflect.InvocationTargetException
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at
> org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:106)
> ... 7 more
> Caused by: java.lang.NullPointerException
> at
> org.apache.hadoop.hive.ql.exec.mr.ExecReducer.configure(ExecReducer.java:116)
> ... 12 more
> 2013-11-20 00:50:56,862 INFO [uber-SubtaskRunner]
> org.apache.hadoop.mapred.TaskAttemptListenerImpl: Status update from
> attempt_1384392632998_34791_r_00_0
> 2013-11-20 00:50:56,862 INFO [uber-SubtaskRunner]
> org.apache.hadoop.mapred.TaskAttemptListenerImpl: Progress of TaskAtte
[jira] [Updated] (HIVE-5857) Reduce tasks do not work in uber mode in YARN
[
https://issues.apache.org/jira/browse/HIVE-5857?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Carl Steinbach updated HIVE-5857:
-
Assignee: Adam Kawa
> Reduce tasks do not work in uber mode in YARN
> -
>
> Key: HIVE-5857
> URL: https://issues.apache.org/jira/browse/HIVE-5857
> Project: Hive
> Issue Type: Bug
> Components: Query Processor
>Affects Versions: 0.12.0, 0.13.0, 0.13.1
>Reporter: Adam Kawa
>Assignee: Adam Kawa
>Priority: Critical
> Labels: plan, uber-jar, uberization, yarn
> Attachments: HIVE-5857.1.patch.txt
>
>
> A Hive query fails when it tries to run a reduce task in uber mode in YARN.
> The NullPointerException is thrown in the ExecReducer.configure method,
> because the plan file (reduce.xml) for a reduce task is not found.
> The Utilities.getBaseWork method is expected to return BaseWork object, but
> it returns NULL due to FileNotFoundException.
> {code}
> // org.apache.hadoop.hive.ql.exec.Utilities
> public static BaseWork getBaseWork(Configuration conf, String name) {
> ...
> try {
> ...
> if (gWork == null) {
> Path localPath;
> if (ShimLoader.getHadoopShims().isLocalMode(conf)) {
> localPath = path;
> } else {
> localPath = new Path(name);
> }
> InputStream in = new FileInputStream(localPath.toUri().getPath());
> BaseWork ret = deserializePlan(in);
>
> }
> return gWork;
> } catch (FileNotFoundException fnf) {
> // happens. e.g.: no reduce work.
> LOG.debug("No plan file found: "+path);
> return null;
> } ...
> }
> {code}
> It happens because, the ShimLoader.getHadoopShims().isLocalMode(conf)) method
> returns true, because immediately before running a reduce task,
> org.apache.hadoop.mapred.LocalContainerLauncher changes its configuration to
> local mode ("mapreduce.framework.name" is changed from" "yarn" to "local").
> On the other hand map tasks run successfully, because its configuration is
> not changed and still remains "yarn".
> {code}
> // org.apache.hadoop.mapred.LocalContainerLauncher
> private void runSubtask(..) {
> ...
> conf.set(MRConfig.FRAMEWORK_NAME, MRConfig.LOCAL_FRAMEWORK_NAME);
> conf.set(MRConfig.MASTER_ADDRESS, "local"); // bypass shuffle
> ReduceTask reduce = (ReduceTask)task;
> reduce.setConf(conf);
> reduce.run(conf, umbilical);
> }
> {code}
> A super quick fix could just an additional if-branch, where we check if we
> run a reduce task in uber mode, and then look for a plan file in a different
> location.
> *Java stacktrace*
> {code}
> 2013-11-20 00:50:56,862 INFO [uber-SubtaskRunner]
> org.apache.hadoop.hive.ql.exec.Utilities: No plan file found:
> hdfs://namenode.c.lon.spotify.net:54310/var/tmp/kawaa/hive_2013-11-20_00-50-43_888_3938384086824086680-2/-mr-10003/e3caacf6-15d6-4987-b186-d2906791b5b0/reduce.xml
> 2013-11-20 00:50:56,862 WARN [uber-SubtaskRunner]
> org.apache.hadoop.mapred.LocalContainerLauncher: Exception running local
> (uberized) 'child' : java.lang.RuntimeException: Error in configuring object
> at
> org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:109)
> at
> org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:75)
> at
> org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:133)
> at
> org.apache.hadoop.mapred.ReduceTask.runOldReducer(ReduceTask.java:427)
> at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:408)
> at
> org.apache.hadoop.mapred.LocalContainerLauncher$SubtaskRunner.runSubtask(LocalContainerLauncher.java:340)
> at
> org.apache.hadoop.mapred.LocalContainerLauncher$SubtaskRunner.run(LocalContainerLauncher.java:225)
> at java.lang.Thread.run(Thread.java:662)
> Caused by: java.lang.reflect.InvocationTargetException
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at
> org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:106)
> ... 7 more
> Caused by: java.lang.NullPointerException
> at
> org.apache.hadoop.hive.ql.exec.mr.ExecReducer.configure(ExecReducer.java:116)
> ... 12 more
> 2013-11-20 00:50:56,862 INFO [uber-SubtaskRunner]
> org.apache.hadoop.mapred.TaskAttemptListenerImpl: Status update from
> attempt_1384392632998_34791_r_00_0
> 2013-11-20 00:50:56,862 INFO [uber-SubtaskRunner]
> org.apache.hadoop.mapred.TaskAttemptListenerImpl: Progress of TaskAttempt
> attempt_1384392632998_34791_r
[jira] [Updated] (HIVE-5857) Reduce tasks do not work in uber mode in YARN
[
https://issues.apache.org/jira/browse/HIVE-5857?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Carl Steinbach updated HIVE-5857:
-
Status: Open (was: Patch Available)
[~kawaa]: Thanks for providing a fix. In order to get committed this patch
needs a test case. Can you please add a qfile test that enables ubertask mode
before running an aggregation query? Thanks.
> Reduce tasks do not work in uber mode in YARN
> -
>
> Key: HIVE-5857
> URL: https://issues.apache.org/jira/browse/HIVE-5857
> Project: Hive
> Issue Type: Bug
> Components: Query Processor
>Affects Versions: 0.12.0
>Reporter: Adam Kawa
>Priority: Critical
> Labels: plan, uber-jar, uberization, yarn
> Attachments: HIVE-5857.1.patch.txt
>
>
> A Hive query fails when it tries to run a reduce task in uber mode in YARN.
> The NullPointerException is thrown in the ExecReducer.configure method,
> because the plan file (reduce.xml) for a reduce task is not found.
> The Utilities.getBaseWork method is expected to return BaseWork object, but
> it returns NULL due to FileNotFoundException.
> {code}
> // org.apache.hadoop.hive.ql.exec.Utilities
> public static BaseWork getBaseWork(Configuration conf, String name) {
> ...
> try {
> ...
> if (gWork == null) {
> Path localPath;
> if (ShimLoader.getHadoopShims().isLocalMode(conf)) {
> localPath = path;
> } else {
> localPath = new Path(name);
> }
> InputStream in = new FileInputStream(localPath.toUri().getPath());
> BaseWork ret = deserializePlan(in);
>
> }
> return gWork;
> } catch (FileNotFoundException fnf) {
> // happens. e.g.: no reduce work.
> LOG.debug("No plan file found: "+path);
> return null;
> } ...
> }
> {code}
> It happens because, the ShimLoader.getHadoopShims().isLocalMode(conf)) method
> returns true, because immediately before running a reduce task,
> org.apache.hadoop.mapred.LocalContainerLauncher changes its configuration to
> local mode ("mapreduce.framework.name" is changed from" "yarn" to "local").
> On the other hand map tasks run successfully, because its configuration is
> not changed and still remains "yarn".
> {code}
> // org.apache.hadoop.mapred.LocalContainerLauncher
> private void runSubtask(..) {
> ...
> conf.set(MRConfig.FRAMEWORK_NAME, MRConfig.LOCAL_FRAMEWORK_NAME);
> conf.set(MRConfig.MASTER_ADDRESS, "local"); // bypass shuffle
> ReduceTask reduce = (ReduceTask)task;
> reduce.setConf(conf);
> reduce.run(conf, umbilical);
> }
> {code}
> A super quick fix could just an additional if-branch, where we check if we
> run a reduce task in uber mode, and then look for a plan file in a different
> location.
> *Java stacktrace*
> {code}
> 2013-11-20 00:50:56,862 INFO [uber-SubtaskRunner]
> org.apache.hadoop.hive.ql.exec.Utilities: No plan file found:
> hdfs://namenode.c.lon.spotify.net:54310/var/tmp/kawaa/hive_2013-11-20_00-50-43_888_3938384086824086680-2/-mr-10003/e3caacf6-15d6-4987-b186-d2906791b5b0/reduce.xml
> 2013-11-20 00:50:56,862 WARN [uber-SubtaskRunner]
> org.apache.hadoop.mapred.LocalContainerLauncher: Exception running local
> (uberized) 'child' : java.lang.RuntimeException: Error in configuring object
> at
> org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:109)
> at
> org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:75)
> at
> org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:133)
> at
> org.apache.hadoop.mapred.ReduceTask.runOldReducer(ReduceTask.java:427)
> at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:408)
> at
> org.apache.hadoop.mapred.LocalContainerLauncher$SubtaskRunner.runSubtask(LocalContainerLauncher.java:340)
> at
> org.apache.hadoop.mapred.LocalContainerLauncher$SubtaskRunner.run(LocalContainerLauncher.java:225)
> at java.lang.Thread.run(Thread.java:662)
> Caused by: java.lang.reflect.InvocationTargetException
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at
> org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:106)
> ... 7 more
> Caused by: java.lang.NullPointerException
> at
> org.apache.hadoop.hive.ql.exec.mr.ExecReducer.configure(ExecReducer.java:116)
> ... 12 more
> 2013-11-20 00:50:56,862 INFO [uber-SubtaskRunner]
> org.apache.hadoop.mapred.TaskAttemptListenerImpl: Status update from
> attempt_1384392632998_34791_r_0
[jira] [Updated] (HIVE-7066) hive-exec jar is missing avro core
[
https://issues.apache.org/jira/browse/HIVE-7066?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Carl Steinbach updated HIVE-7066:
-
Resolution: Fixed
Fix Version/s: 0.14.0
Hadoop Flags: Reviewed
Status: Resolved (was: Patch Available)
Committed to trunk. Thanks David!
> hive-exec jar is missing avro core
> --
>
> Key: HIVE-7066
> URL: https://issues.apache.org/jira/browse/HIVE-7066
> Project: Hive
> Issue Type: Bug
> Components: Build Infrastructure
>Affects Versions: 0.13.0, 0.13.1
>Reporter: David Chen
>Assignee: David Chen
> Fix For: 0.14.0
>
> Attachments: HIVE-7066.1.patch
>
>
> Running a simple query that reads an Avro table caused the following
> exception to be thrown on the cluster side:
> {code}
> java.lang.RuntimeException:
> org.apache.hive.com.esotericsoftware.kryo.KryoException:
> java.lang.IllegalArgumentException: Unable to create serializer
> "org.apache.hive.com.esotericsoftware.kryo.serializers.FieldSerializer" for
> class: org.apache.hadoop.hive.ql.io.avro.AvroContainerOutputFormat
> Serialization trace:
> outputFileFormatClass (org.apache.hadoop.hive.ql.plan.PartitionDesc)
> aliasToPartnInfo (org.apache.hadoop.hive.ql.plan.MapWork)
> at
> org.apache.hadoop.hive.ql.exec.Utilities.getBaseWork(Utilities.java:365)
> at
> org.apache.hadoop.hive.ql.exec.Utilities.getMapWork(Utilities.java:276)
> at
> org.apache.hadoop.hive.ql.io.HiveInputFormat.init(HiveInputFormat.java:254)
> at
> org.apache.hadoop.hive.ql.io.HiveInputFormat.pushProjectionsAndFilters(HiveInputFormat.java:445)
> at
> org.apache.hadoop.hive.ql.io.HiveInputFormat.pushProjectionsAndFilters(HiveInputFormat.java:438)
> at
> org.apache.hadoop.hive.ql.io.CombineHiveInputFormat.getRecordReader(CombineHiveInputFormat.java:587)
> at
> org.apache.hadoop.mapred.MapTask$TrackedRecordReader.(MapTask.java:191)
> at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:412)
> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:366)
> at org.apache.hadoop.mapred.Child$4.run(Child.java:255)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:394)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1190)
> at org.apache.hadoop.mapred.Child.main(Child.java:249)
> Caused by: org.apache.hive.com.esotericsoftware.kryo.KryoException:
> java.lang.IllegalArgumentException: Unable to create serializer
> "org.apache.hive.com.esotericsoftware.kryo.serializers.FieldSerializer" for
> class: org.apache.hadoop.hive.ql.io.avro.AvroContainerOutputFormat
> Serialization trace:
> outputFileFormatClass (org.apache.hadoop.hive.ql.plan.PartitionDesc)
> aliasToPartnInfo (org.apache.hadoop.hive.ql.plan.MapWork)
> at
> org.apache.hive.com.esotericsoftware.kryo.serializers.ObjectField.read(ObjectField.java:125)
> at
> org.apache.hive.com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:507)
> at
> org.apache.hive.com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:776)
> at
> org.apache.hive.com.esotericsoftware.kryo.serializers.MapSerializer.read(MapSerializer.java:139)
> at
> org.apache.hive.com.esotericsoftware.kryo.serializers.MapSerializer.read(MapSerializer.java:17)
> at
> org.apache.hive.com.esotericsoftware.kryo.Kryo.readObject(Kryo.java:694)
> at
> org.apache.hive.com.esotericsoftware.kryo.serializers.ObjectField.read(ObjectField.java:106)
> at
> org.apache.hive.com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:507)
> at
> org.apache.hive.com.esotericsoftware.kryo.Kryo.readObject(Kryo.java:672)
> at
> org.apache.hadoop.hive.ql.exec.Utilities.deserializeObjectByKryo(Utilities.java:942)
> at
> org.apache.hadoop.hive.ql.exec.Utilities.deserializePlan(Utilities.java:850)
> at
> org.apache.hadoop.hive.ql.exec.Utilities.deserializePlan(Utilities.java:864)
> at
> org.apache.hadoop.hive.ql.exec.Utilities.getBaseWork(Utilities.java:334)
> ... 13 more
> Caused by: java.lang.IllegalArgumentException: Unable to create serializer
> "org.apache.hive.com.esotericsoftware.kryo.serializers.FieldSerializer" for
> class: org.apache.hadoop.hive.ql.io.avro.AvroContainerOutputFormat
> at
> org.apache.hive.com.esotericsoftware.kryo.factories.ReflectionSerializerFactory.makeSerializer(ReflectionSerializerFactory.java:45)
> at
> org.apache.hive.com.esotericsoftware.kryo.factories.ReflectionSerializerFactory.makeSerializer(ReflectionSerializerFactory.java:26)
> at
> org.apache.hive.com.esotericsoftware.kryo.Kryo.newDefaultSerializer(Kryo.java:343)
> at
> org.apache.
[jira] [Updated] (HIVE-7066) hive-exec jar is missing avro core
[
https://issues.apache.org/jira/browse/HIVE-7066?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Carl Steinbach updated HIVE-7066:
-
Affects Version/s: 0.13.1
0.13.0
> hive-exec jar is missing avro core
> --
>
> Key: HIVE-7066
> URL: https://issues.apache.org/jira/browse/HIVE-7066
> Project: Hive
> Issue Type: Bug
> Components: Build Infrastructure
>Affects Versions: 0.13.0, 0.13.1
>Reporter: David Chen
>Assignee: David Chen
> Attachments: HIVE-7066.1.patch
>
>
> Running a simple query that reads an Avro table caused the following
> exception to be thrown on the cluster side:
> {code}
> java.lang.RuntimeException:
> org.apache.hive.com.esotericsoftware.kryo.KryoException:
> java.lang.IllegalArgumentException: Unable to create serializer
> "org.apache.hive.com.esotericsoftware.kryo.serializers.FieldSerializer" for
> class: org.apache.hadoop.hive.ql.io.avro.AvroContainerOutputFormat
> Serialization trace:
> outputFileFormatClass (org.apache.hadoop.hive.ql.plan.PartitionDesc)
> aliasToPartnInfo (org.apache.hadoop.hive.ql.plan.MapWork)
> at
> org.apache.hadoop.hive.ql.exec.Utilities.getBaseWork(Utilities.java:365)
> at
> org.apache.hadoop.hive.ql.exec.Utilities.getMapWork(Utilities.java:276)
> at
> org.apache.hadoop.hive.ql.io.HiveInputFormat.init(HiveInputFormat.java:254)
> at
> org.apache.hadoop.hive.ql.io.HiveInputFormat.pushProjectionsAndFilters(HiveInputFormat.java:445)
> at
> org.apache.hadoop.hive.ql.io.HiveInputFormat.pushProjectionsAndFilters(HiveInputFormat.java:438)
> at
> org.apache.hadoop.hive.ql.io.CombineHiveInputFormat.getRecordReader(CombineHiveInputFormat.java:587)
> at
> org.apache.hadoop.mapred.MapTask$TrackedRecordReader.(MapTask.java:191)
> at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:412)
> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:366)
> at org.apache.hadoop.mapred.Child$4.run(Child.java:255)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:394)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1190)
> at org.apache.hadoop.mapred.Child.main(Child.java:249)
> Caused by: org.apache.hive.com.esotericsoftware.kryo.KryoException:
> java.lang.IllegalArgumentException: Unable to create serializer
> "org.apache.hive.com.esotericsoftware.kryo.serializers.FieldSerializer" for
> class: org.apache.hadoop.hive.ql.io.avro.AvroContainerOutputFormat
> Serialization trace:
> outputFileFormatClass (org.apache.hadoop.hive.ql.plan.PartitionDesc)
> aliasToPartnInfo (org.apache.hadoop.hive.ql.plan.MapWork)
> at
> org.apache.hive.com.esotericsoftware.kryo.serializers.ObjectField.read(ObjectField.java:125)
> at
> org.apache.hive.com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:507)
> at
> org.apache.hive.com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:776)
> at
> org.apache.hive.com.esotericsoftware.kryo.serializers.MapSerializer.read(MapSerializer.java:139)
> at
> org.apache.hive.com.esotericsoftware.kryo.serializers.MapSerializer.read(MapSerializer.java:17)
> at
> org.apache.hive.com.esotericsoftware.kryo.Kryo.readObject(Kryo.java:694)
> at
> org.apache.hive.com.esotericsoftware.kryo.serializers.ObjectField.read(ObjectField.java:106)
> at
> org.apache.hive.com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:507)
> at
> org.apache.hive.com.esotericsoftware.kryo.Kryo.readObject(Kryo.java:672)
> at
> org.apache.hadoop.hive.ql.exec.Utilities.deserializeObjectByKryo(Utilities.java:942)
> at
> org.apache.hadoop.hive.ql.exec.Utilities.deserializePlan(Utilities.java:850)
> at
> org.apache.hadoop.hive.ql.exec.Utilities.deserializePlan(Utilities.java:864)
> at
> org.apache.hadoop.hive.ql.exec.Utilities.getBaseWork(Utilities.java:334)
> ... 13 more
> Caused by: java.lang.IllegalArgumentException: Unable to create serializer
> "org.apache.hive.com.esotericsoftware.kryo.serializers.FieldSerializer" for
> class: org.apache.hadoop.hive.ql.io.avro.AvroContainerOutputFormat
> at
> org.apache.hive.com.esotericsoftware.kryo.factories.ReflectionSerializerFactory.makeSerializer(ReflectionSerializerFactory.java:45)
> at
> org.apache.hive.com.esotericsoftware.kryo.factories.ReflectionSerializerFactory.makeSerializer(ReflectionSerializerFactory.java:26)
> at
> org.apache.hive.com.esotericsoftware.kryo.Kryo.newDefaultSerializer(Kryo.java:343)
> at
> org.apache.hive.com.esotericsoftware.kryo.Kryo.getDefaultSerializer(Kryo.java:336)
> at
> org.apache.hive.com.esotericsoftware.kryo.util.Defau
[jira] [Updated] (HIVE-7066) hive-exec jar is missing avro core
[
https://issues.apache.org/jira/browse/HIVE-7066?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Carl Steinbach updated HIVE-7066:
-
Component/s: Build Infrastructure
> hive-exec jar is missing avro core
> --
>
> Key: HIVE-7066
> URL: https://issues.apache.org/jira/browse/HIVE-7066
> Project: Hive
> Issue Type: Bug
> Components: Build Infrastructure
>Affects Versions: 0.13.0, 0.13.1
>Reporter: David Chen
>Assignee: David Chen
> Attachments: HIVE-7066.1.patch
>
>
> Running a simple query that reads an Avro table caused the following
> exception to be thrown on the cluster side:
> {code}
> java.lang.RuntimeException:
> org.apache.hive.com.esotericsoftware.kryo.KryoException:
> java.lang.IllegalArgumentException: Unable to create serializer
> "org.apache.hive.com.esotericsoftware.kryo.serializers.FieldSerializer" for
> class: org.apache.hadoop.hive.ql.io.avro.AvroContainerOutputFormat
> Serialization trace:
> outputFileFormatClass (org.apache.hadoop.hive.ql.plan.PartitionDesc)
> aliasToPartnInfo (org.apache.hadoop.hive.ql.plan.MapWork)
> at
> org.apache.hadoop.hive.ql.exec.Utilities.getBaseWork(Utilities.java:365)
> at
> org.apache.hadoop.hive.ql.exec.Utilities.getMapWork(Utilities.java:276)
> at
> org.apache.hadoop.hive.ql.io.HiveInputFormat.init(HiveInputFormat.java:254)
> at
> org.apache.hadoop.hive.ql.io.HiveInputFormat.pushProjectionsAndFilters(HiveInputFormat.java:445)
> at
> org.apache.hadoop.hive.ql.io.HiveInputFormat.pushProjectionsAndFilters(HiveInputFormat.java:438)
> at
> org.apache.hadoop.hive.ql.io.CombineHiveInputFormat.getRecordReader(CombineHiveInputFormat.java:587)
> at
> org.apache.hadoop.mapred.MapTask$TrackedRecordReader.(MapTask.java:191)
> at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:412)
> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:366)
> at org.apache.hadoop.mapred.Child$4.run(Child.java:255)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:394)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1190)
> at org.apache.hadoop.mapred.Child.main(Child.java:249)
> Caused by: org.apache.hive.com.esotericsoftware.kryo.KryoException:
> java.lang.IllegalArgumentException: Unable to create serializer
> "org.apache.hive.com.esotericsoftware.kryo.serializers.FieldSerializer" for
> class: org.apache.hadoop.hive.ql.io.avro.AvroContainerOutputFormat
> Serialization trace:
> outputFileFormatClass (org.apache.hadoop.hive.ql.plan.PartitionDesc)
> aliasToPartnInfo (org.apache.hadoop.hive.ql.plan.MapWork)
> at
> org.apache.hive.com.esotericsoftware.kryo.serializers.ObjectField.read(ObjectField.java:125)
> at
> org.apache.hive.com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:507)
> at
> org.apache.hive.com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:776)
> at
> org.apache.hive.com.esotericsoftware.kryo.serializers.MapSerializer.read(MapSerializer.java:139)
> at
> org.apache.hive.com.esotericsoftware.kryo.serializers.MapSerializer.read(MapSerializer.java:17)
> at
> org.apache.hive.com.esotericsoftware.kryo.Kryo.readObject(Kryo.java:694)
> at
> org.apache.hive.com.esotericsoftware.kryo.serializers.ObjectField.read(ObjectField.java:106)
> at
> org.apache.hive.com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:507)
> at
> org.apache.hive.com.esotericsoftware.kryo.Kryo.readObject(Kryo.java:672)
> at
> org.apache.hadoop.hive.ql.exec.Utilities.deserializeObjectByKryo(Utilities.java:942)
> at
> org.apache.hadoop.hive.ql.exec.Utilities.deserializePlan(Utilities.java:850)
> at
> org.apache.hadoop.hive.ql.exec.Utilities.deserializePlan(Utilities.java:864)
> at
> org.apache.hadoop.hive.ql.exec.Utilities.getBaseWork(Utilities.java:334)
> ... 13 more
> Caused by: java.lang.IllegalArgumentException: Unable to create serializer
> "org.apache.hive.com.esotericsoftware.kryo.serializers.FieldSerializer" for
> class: org.apache.hadoop.hive.ql.io.avro.AvroContainerOutputFormat
> at
> org.apache.hive.com.esotericsoftware.kryo.factories.ReflectionSerializerFactory.makeSerializer(ReflectionSerializerFactory.java:45)
> at
> org.apache.hive.com.esotericsoftware.kryo.factories.ReflectionSerializerFactory.makeSerializer(ReflectionSerializerFactory.java:26)
> at
> org.apache.hive.com.esotericsoftware.kryo.Kryo.newDefaultSerializer(Kryo.java:343)
> at
> org.apache.hive.com.esotericsoftware.kryo.Kryo.getDefaultSerializer(Kryo.java:336)
> at
> org.apache.hive.com.esotericsoftware.kryo.util.DefaultClassResolver.regist
[jira] [Updated] (HIVE-7104) Unit tests are disabled
[
https://issues.apache.org/jira/browse/HIVE-7104?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Carl Steinbach updated HIVE-7104:
-
Resolution: Fixed
Fix Version/s: 0.14.0
Hadoop Flags: Reviewed
Status: Resolved (was: Patch Available)
Committed to trunk. Thanks David!
> Unit tests are disabled
> ---
>
> Key: HIVE-7104
> URL: https://issues.apache.org/jira/browse/HIVE-7104
> Project: Hive
> Issue Type: Bug
>Reporter: David Chen
>Assignee: David Chen
> Fix For: 0.14.0
>
> Attachments: HIVE-7104.1.patch
>
>
> When I run {{mvn clean test -Phadoop-1|2}}, none of the unit tests are run. I
> did a binary search through the commit logs and found that the change that
> caused the unit tests to be disabled was the the change to the root pom.xml
> in the patch for HIVE-7067 (e77f38dc44de5a9b10bce8e0a2f1f5452f6921ed).
> Removing that change allowed the unit tests to be run again.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HIVE-3159) Update AvroSerde to determine schema of new tables
[ https://issues.apache.org/jira/browse/HIVE-3159?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13998544#comment-13998544 ] Carl Steinbach commented on HIVE-3159: -- bq. Recently committed HIVE-5823, added some bug. [~kamrul]: HIVE-5823 was resolved as WONTFIX. > Update AvroSerde to determine schema of new tables > -- > > Key: HIVE-3159 > URL: https://issues.apache.org/jira/browse/HIVE-3159 > Project: Hive > Issue Type: Improvement > Components: Serializers/Deserializers >Affects Versions: 0.12.0 >Reporter: Jakob Homan >Assignee: Mohammad Kamrul Islam > Attachments: HIVE-3159.10.patch, HIVE-3159.4.patch, > HIVE-3159.5.patch, HIVE-3159.6.patch, HIVE-3159.7.patch, HIVE-3159.9.patch, > HIVE-3159v1.patch > > > Currently when writing tables to Avro one must manually provide an Avro > schema that matches what is being delivered by Hive. It'd be better to have > the serde infer this schema by converting the table's TypeInfo into an > appropriate AvroSchema. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HIVE-6974) Make Metastore Version Check work with Custom version suffixes
[ https://issues.apache.org/jira/browse/HIVE-6974?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13995705#comment-13995705 ] Carl Steinbach commented on HIVE-6974: -- Looks like this was already fixed in HIVE-5484. > Make Metastore Version Check work with Custom version suffixes > -- > > Key: HIVE-6974 > URL: https://issues.apache.org/jira/browse/HIVE-6974 > Project: Hive > Issue Type: Bug > Components: Metastore >Reporter: Carl Steinbach > > HIVE-3764 added support for doing a version consistency check between the > Hive JARs on the classpath and the metastore schema in the backend database. > This is a nice feature, but it currently doesn't work for well for folks who > appending the release version with their own suffixes, e.g. "0.12.0.li_20". > We can fix this problem by modifying > MetaStoreSchemaInfo.getHiveSchemaVersion() to match against ^\d+\.\d+\.\d+ > and ignore anything that remains. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HIVE-3159) Update AvroSerde to determine schema of new tables
[ https://issues.apache.org/jira/browse/HIVE-3159?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Carl Steinbach updated HIVE-3159: - Status: Open (was: Patch Available) Looks like a couple tests failed. > Update AvroSerde to determine schema of new tables > -- > > Key: HIVE-3159 > URL: https://issues.apache.org/jira/browse/HIVE-3159 > Project: Hive > Issue Type: Improvement > Components: Serializers/Deserializers >Affects Versions: 0.12.0 >Reporter: Jakob Homan >Assignee: Mohammad Kamrul Islam > Attachments: HIVE-3159.10.patch, HIVE-3159.4.patch, > HIVE-3159.5.patch, HIVE-3159.6.patch, HIVE-3159.7.patch, HIVE-3159.9.patch, > HIVE-3159v1.patch > > > Currently when writing tables to Avro one must manually provide an Avro > schema that matches what is being delivered by Hive. It'd be better to have > the serde infer this schema by converting the table's TypeInfo into an > appropriate AvroSchema. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HIVE-6187) Add test to verify that DESCRIBE TABLE works with quoted table names
[
https://issues.apache.org/jira/browse/HIVE-6187?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Carl Steinbach updated HIVE-6187:
-
Status: Patch Available (was: Open)
> Add test to verify that DESCRIBE TABLE works with quoted table names
>
>
> Key: HIVE-6187
> URL: https://issues.apache.org/jira/browse/HIVE-6187
> Project: Hive
> Issue Type: Bug
>Affects Versions: 0.10.0
>Reporter: Andy Mok
> Attachments: HIVE-6187.1.patch
>
>
> Backticks around tables named after special keywords, such as items, allow us
> to create, drop, and alter the table. For example
> {code:sql}
> CREATE TABLE foo.`items` (bar INT);
> DROP TABLE foo.`items`;
> ALTER TABLE `items` RENAME TO `items_`;
> {code}
> However, we cannot call
> {code:sql}
> DESCRIBE foo.`items`;
> DESCRIBE `items`;
> {code}
> The DESCRIBE query does not permit backticks to surround table names. The
> error returned is
> {code:sql}
> FAILED: SemanticException [Error 10001]: Table not found `items`
> {code}
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Resolved] (HIVE-6974) Make Metastore Version Check work with Custom version suffixes
[ https://issues.apache.org/jira/browse/HIVE-6974?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Carl Steinbach resolved HIVE-6974. -- Resolution: Duplicate > Make Metastore Version Check work with Custom version suffixes > -- > > Key: HIVE-6974 > URL: https://issues.apache.org/jira/browse/HIVE-6974 > Project: Hive > Issue Type: Bug > Components: Metastore >Reporter: Carl Steinbach > > HIVE-3764 added support for doing a version consistency check between the > Hive JARs on the classpath and the metastore schema in the backend database. > This is a nice feature, but it currently doesn't work for well for folks who > appending the release version with their own suffixes, e.g. "0.12.0.li_20". > We can fix this problem by modifying > MetaStoreSchemaInfo.getHiveSchemaVersion() to match against ^\d+\.\d+\.\d+ > and ignore anything that remains. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HIVE-6187) Add test to verify that DESCRIBE TABLE works with quoted table names
[
https://issues.apache.org/jira/browse/HIVE-6187?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Carl Steinbach updated HIVE-6187:
-
Summary: Add test to verify that DESCRIBE TABLE works with quoted table
names (was: Cannot use backticks around table name when using DESCRIBE query)
> Add test to verify that DESCRIBE TABLE works with quoted table names
>
>
> Key: HIVE-6187
> URL: https://issues.apache.org/jira/browse/HIVE-6187
> Project: Hive
> Issue Type: Bug
>Affects Versions: 0.10.0
>Reporter: Andy Mok
> Attachments: HIVE-6187.1.patch
>
>
> Backticks around tables named after special keywords, such as items, allow us
> to create, drop, and alter the table. For example
> {code:sql}
> CREATE TABLE foo.`items` (bar INT);
> DROP TABLE foo.`items`;
> ALTER TABLE `items` RENAME TO `items_`;
> {code}
> However, we cannot call
> {code:sql}
> DESCRIBE foo.`items`;
> DESCRIBE `items`;
> {code}
> The DESCRIBE query does not permit backticks to surround table names. The
> error returned is
> {code:sql}
> FAILED: SemanticException [Error 10001]: Table not found `items`
> {code}
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Updated] (HIVE-6187) Cannot use backticks around table name when using DESCRIBE query
[
https://issues.apache.org/jira/browse/HIVE-6187?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Carl Steinbach updated HIVE-6187:
-
Attachment: HIVE-6187.1.patch
Attaching a patch that adds several quoted testcases to describe_table.q.
> Cannot use backticks around table name when using DESCRIBE query
>
>
> Key: HIVE-6187
> URL: https://issues.apache.org/jira/browse/HIVE-6187
> Project: Hive
> Issue Type: Bug
>Affects Versions: 0.10.0
>Reporter: Andy Mok
> Attachments: HIVE-6187.1.patch
>
>
> Backticks around tables named after special keywords, such as items, allow us
> to create, drop, and alter the table. For example
> {code:sql}
> CREATE TABLE foo.`items` (bar INT);
> DROP TABLE foo.`items`;
> ALTER TABLE `items` RENAME TO `items_`;
> {code}
> However, we cannot call
> {code:sql}
> DESCRIBE foo.`items`;
> DESCRIBE `items`;
> {code}
> The DESCRIBE query does not permit backticks to surround table names. The
> error returned is
> {code:sql}
> FAILED: SemanticException [Error 10001]: Table not found `items`
> {code}
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HIVE-6187) Cannot use backticks around table name when using DESCRIBE query
[
https://issues.apache.org/jira/browse/HIVE-6187?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13994965#comment-13994965
]
Carl Steinbach commented on HIVE-6187:
--
I can confirm that this functionality is currently working on trunk, and also
that it's broken in the 0.12.0 release. I'm not sure when it was fixed, and
there doesn't appear to be any test coverage that will prevent someone from
breaking it again in the future.
> Cannot use backticks around table name when using DESCRIBE query
>
>
> Key: HIVE-6187
> URL: https://issues.apache.org/jira/browse/HIVE-6187
> Project: Hive
> Issue Type: Bug
>Affects Versions: 0.10.0
>Reporter: Andy Mok
>
> Backticks around tables named after special keywords, such as items, allow us
> to create, drop, and alter the table. For example
> {code:sql}
> CREATE TABLE foo.`items` (bar INT);
> DROP TABLE foo.`items`;
> ALTER TABLE `items` RENAME TO `items_`;
> {code}
> However, we cannot call
> {code:sql}
> DESCRIBE foo.`items`;
> DESCRIBE `items`;
> {code}
> The DESCRIBE query does not permit backticks to surround table names. The
> error returned is
> {code:sql}
> FAILED: SemanticException [Error 10001]: Table not found `items`
> {code}
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HIVE-3159) Update AvroSerde to determine schema of new tables
[ https://issues.apache.org/jira/browse/HIVE-3159?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13980529#comment-13980529 ] Carl Steinbach commented on HIVE-3159: -- [~kamrul]: When we last chatted about this a couple months back you said you were planning to update the patch with some fixes. Are you still planning to do that? Thanks. > Update AvroSerde to determine schema of new tables > -- > > Key: HIVE-3159 > URL: https://issues.apache.org/jira/browse/HIVE-3159 > Project: Hive > Issue Type: Improvement > Components: Serializers/Deserializers >Reporter: Jakob Homan >Assignee: Mohammad Kamrul Islam > Attachments: HIVE-3159.4.patch, HIVE-3159.5.patch, HIVE-3159.6.patch, > HIVE-3159.7.patch, HIVE-3159v1.patch > > > Currently when writing tables to Avro one must manually provide an Avro > schema that matches what is being delivered by Hive. It'd be better to have > the serde infer this schema by converting the table's TypeInfo into an > appropriate AvroSchema. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HIVE-6974) Make Metastore Version Check work with Custom version suffixes
[ https://issues.apache.org/jira/browse/HIVE-6974?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Carl Steinbach updated HIVE-6974: - Description: HIVE-3764 added support for doing a version consistency check between the Hive JARs on the classpath and the metastore schema in the backend database. This is a nice feature, but it currently doesn't work for well for folks who appending the release version with their own suffixes, e.g. "0.12.0.li_20". We can fix this problem by modifying MetaStoreSchemaInfo.getHiveSchemaVersion() to match against ^\d+\.\d+\.\d+ and ignore anything that remains. > Make Metastore Version Check work with Custom version suffixes > -- > > Key: HIVE-6974 > URL: https://issues.apache.org/jira/browse/HIVE-6974 > Project: Hive > Issue Type: Bug > Components: Metastore >Reporter: Carl Steinbach > > HIVE-3764 added support for doing a version consistency check between the > Hive JARs on the classpath and the metastore schema in the backend database. > This is a nice feature, but it currently doesn't work for well for folks who > appending the release version with their own suffixes, e.g. "0.12.0.li_20". > We can fix this problem by modifying > MetaStoreSchemaInfo.getHiveSchemaVersion() to match against ^\d+\.\d+\.\d+ > and ignore anything that remains. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Created] (HIVE-6974) Make Metastore Version Check work with Custom version suffixes
Carl Steinbach created HIVE-6974: Summary: Make Metastore Version Check work with Custom version suffixes Key: HIVE-6974 URL: https://issues.apache.org/jira/browse/HIVE-6974 Project: Hive Issue Type: Bug Components: Metastore Reporter: Carl Steinbach -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HIVE-6835) Reading of partitioned Avro data fails if partition schema does not match table schema
[
https://issues.apache.org/jira/browse/HIVE-6835?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13972241#comment-13972241
]
Carl Steinbach commented on HIVE-6835:
--
[~ashutoshc]: Thanks for catching the Thrift codegen problem.
[~erwaman]: Updated patch looks good. +1
> Reading of partitioned Avro data fails if partition schema does not match
> table schema
> --
>
> Key: HIVE-6835
> URL: https://issues.apache.org/jira/browse/HIVE-6835
> Project: Hive
> Issue Type: Bug
>Affects Versions: 0.12.0
>Reporter: Anthony Hsu
>Assignee: Anthony Hsu
> Attachments: HIVE-6835.1.patch, HIVE-6835.2.patch, HIVE-6835.3.patch
>
>
> To reproduce:
> {code}
> create table testarray (a array);
> load data local inpath '/home/ahsu/test/array.txt' into table testarray;
> # create partitioned Avro table with one array column
> create table avroarray partitioned by (y string) row format serde
> 'org.apache.hadoop.hive.serde2.avro.AvroSerDe' with serdeproperties
> ('avro.schema.literal'='{"namespace":"test","name":"avroarray","type":
> "record", "fields": [ { "name":"a", "type":{"type":"array","items":"string"}
> } ] }') STORED as INPUTFORMAT
> 'org.apache.hadoop.hive.ql.io.avro.AvroContainerInputFormat' OUTPUTFORMAT
> 'org.apache.hadoop.hive.ql.io.avro.AvroContainerOutputFormat';
> insert into table avroarray partition(y=1) select * from testarray;
> # add an int column with a default value of 0
> alter table avroarray set serde
> 'org.apache.hadoop.hive.serde2.avro.AvroSerDe' with
> serdeproperties('avro.schema.literal'='{"namespace":"test","name":"avroarray","type":
> "record", "fields": [ {"name":"intfield","type":"int","default":0},{
> "name":"a", "type":{"type":"array","items":"string"} } ] }');
> # fails with ClassCastException
> select * from avroarray;
> {code}
> The select * fails with:
> {code}
> Failed with exception java.io.IOException:java.lang.ClassCastException:
> org.apache.hadoop.hive.serde2.objectinspector.StandardListObjectInspector
> cannot be cast to
> org.apache.hadoop.hive.serde2.objectinspector.PrimitiveObjectInspector
> {code}
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Comment Edited] (HIVE-6835) Reading of partitioned Avro data fails if partition schema does not match table schema
[
https://issues.apache.org/jira/browse/HIVE-6835?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13969281#comment-13969281
]
Carl Steinbach edited comment on HIVE-6835 at 4/15/14 7:00 AM:
---
+1. Will wait for tests to pass before committing.
was (Author: cwsteinbach):
+1
> Reading of partitioned Avro data fails if partition schema does not match
> table schema
> --
>
> Key: HIVE-6835
> URL: https://issues.apache.org/jira/browse/HIVE-6835
> Project: Hive
> Issue Type: Bug
>Affects Versions: 0.12.0
>Reporter: Anthony Hsu
>Assignee: Anthony Hsu
> Attachments: HIVE-6835.1.patch, HIVE-6835.2.patch
>
>
> To reproduce:
> {code}
> create table testarray (a array);
> load data local inpath '/home/ahsu/test/array.txt' into table testarray;
> # create partitioned Avro table with one array column
> create table avroarray partitioned by (y string) row format serde
> 'org.apache.hadoop.hive.serde2.avro.AvroSerDe' with serdeproperties
> ('avro.schema.literal'='{"namespace":"test","name":"avroarray","type":
> "record", "fields": [ { "name":"a", "type":{"type":"array","items":"string"}
> } ] }') STORED as INPUTFORMAT
> 'org.apache.hadoop.hive.ql.io.avro.AvroContainerInputFormat' OUTPUTFORMAT
> 'org.apache.hadoop.hive.ql.io.avro.AvroContainerOutputFormat';
> insert into table avroarray partition(y=1) select * from testarray;
> # add an int column with a default value of 0
> alter table avroarray set serde
> 'org.apache.hadoop.hive.serde2.avro.AvroSerDe' with
> serdeproperties('avro.schema.literal'='{"namespace":"test","name":"avroarray","type":
> "record", "fields": [ {"name":"intfield","type":"int","default":0},{
> "name":"a", "type":{"type":"array","items":"string"} } ] }');
> # fails with ClassCastException
> select * from avroarray;
> {code}
> The select * fails with:
> {code}
> Failed with exception java.io.IOException:java.lang.ClassCastException:
> org.apache.hadoop.hive.serde2.objectinspector.StandardListObjectInspector
> cannot be cast to
> org.apache.hadoop.hive.serde2.objectinspector.PrimitiveObjectInspector
> {code}
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HIVE-6835) Reading of partitioned Avro data fails if partition schema does not match table schema
[
https://issues.apache.org/jira/browse/HIVE-6835?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13969281#comment-13969281
]
Carl Steinbach commented on HIVE-6835:
--
+1
> Reading of partitioned Avro data fails if partition schema does not match
> table schema
> --
>
> Key: HIVE-6835
> URL: https://issues.apache.org/jira/browse/HIVE-6835
> Project: Hive
> Issue Type: Bug
>Affects Versions: 0.12.0
>Reporter: Anthony Hsu
>Assignee: Anthony Hsu
> Attachments: HIVE-6835.1.patch, HIVE-6835.2.patch
>
>
> To reproduce:
> {code}
> create table testarray (a array);
> load data local inpath '/home/ahsu/test/array.txt' into table testarray;
> # create partitioned Avro table with one array column
> create table avroarray partitioned by (y string) row format serde
> 'org.apache.hadoop.hive.serde2.avro.AvroSerDe' with serdeproperties
> ('avro.schema.literal'='{"namespace":"test","name":"avroarray","type":
> "record", "fields": [ { "name":"a", "type":{"type":"array","items":"string"}
> } ] }') STORED as INPUTFORMAT
> 'org.apache.hadoop.hive.ql.io.avro.AvroContainerInputFormat' OUTPUTFORMAT
> 'org.apache.hadoop.hive.ql.io.avro.AvroContainerOutputFormat';
> insert into table avroarray partition(y=1) select * from testarray;
> # add an int column with a default value of 0
> alter table avroarray set serde
> 'org.apache.hadoop.hive.serde2.avro.AvroSerDe' with
> serdeproperties('avro.schema.literal'='{"namespace":"test","name":"avroarray","type":
> "record", "fields": [ {"name":"intfield","type":"int","default":0},{
> "name":"a", "type":{"type":"array","items":"string"} } ] }');
> # fails with ClassCastException
> select * from avroarray;
> {code}
> The select * fails with:
> {code}
> Failed with exception java.io.IOException:java.lang.ClassCastException:
> org.apache.hadoop.hive.serde2.objectinspector.StandardListObjectInspector
> cannot be cast to
> org.apache.hadoop.hive.serde2.objectinspector.PrimitiveObjectInspector
> {code}
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Updated] (HIVE-6835) Reading of partitioned Avro data fails if partition schema does not match table schema
[
https://issues.apache.org/jira/browse/HIVE-6835?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Carl Steinbach updated HIVE-6835:
-
Status: Open (was: Patch Available)
[~erwaman]: Please see my comments on reviewboard. Thanks.
> Reading of partitioned Avro data fails if partition schema does not match
> table schema
> --
>
> Key: HIVE-6835
> URL: https://issues.apache.org/jira/browse/HIVE-6835
> Project: Hive
> Issue Type: Bug
>Affects Versions: 0.12.0
>Reporter: Anthony Hsu
>Assignee: Anthony Hsu
> Attachments: HIVE-6835.1.patch
>
>
> To reproduce:
> {code}
> create table testarray (a array);
> load data local inpath '/home/ahsu/test/array.txt' into table testarray;
> # create partitioned Avro table with one array column
> create table avroarray partitioned by (y string) row format serde
> 'org.apache.hadoop.hive.serde2.avro.AvroSerDe' with serdeproperties
> ('avro.schema.literal'='{"namespace":"test","name":"avroarray","type":
> "record", "fields": [ { "name":"a", "type":{"type":"array","items":"string"}
> } ] }') STORED as INPUTFORMAT
> 'org.apache.hadoop.hive.ql.io.avro.AvroContainerInputFormat' OUTPUTFORMAT
> 'org.apache.hadoop.hive.ql.io.avro.AvroContainerOutputFormat';
> insert into table avroarray partition(y=1) select * from testarray;
> # add an int column with a default value of 0
> alter table avroarray set serde
> 'org.apache.hadoop.hive.serde2.avro.AvroSerDe' with
> serdeproperties('avro.schema.literal'='{"namespace":"test","name":"avroarray","type":
> "record", "fields": [ {"name":"intfield","type":"int","default":0},{
> "name":"a", "type":{"type":"array","items":"string"} } ] }');
> # fails with ClassCastException
> select * from avroarray;
> {code}
> The select * fails with:
> {code}
> Failed with exception java.io.IOException:java.lang.ClassCastException:
> org.apache.hadoop.hive.serde2.objectinspector.StandardListObjectInspector
> cannot be cast to
> org.apache.hadoop.hive.serde2.objectinspector.PrimitiveObjectInspector
> {code}
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Updated] (HIVE-4329) HCatalog clients can't write to AvroSerde backed tables
[
https://issues.apache.org/jira/browse/HIVE-4329?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Carl Steinbach updated HIVE-4329:
-
Assignee: David Chen
> HCatalog clients can't write to AvroSerde backed tables
> ---
>
> Key: HIVE-4329
> URL: https://issues.apache.org/jira/browse/HIVE-4329
> Project: Hive
> Issue Type: Bug
> Components: HCatalog, Serializers/Deserializers
>Affects Versions: 0.10.0
> Environment: discovered in Pig, but it looks like the root cause
> impacts all non-Hive users
>Reporter: Sean Busbey
>Assignee: David Chen
>
> Attempting to write to a HCatalog defined table backed by the AvroSerde fails
> with the following stacktrace:
> {code}
> java.lang.ClassCastException: org.apache.hadoop.io.NullWritable cannot be
> cast to org.apache.hadoop.io.LongWritable
> at
> org.apache.hadoop.hive.ql.io.avro.AvroContainerOutputFormat$1.write(AvroContainerOutputFormat.java:84)
> at
> org.apache.hcatalog.mapreduce.FileRecordWriterContainer.write(FileRecordWriterContainer.java:253)
> at
> org.apache.hcatalog.mapreduce.FileRecordWriterContainer.write(FileRecordWriterContainer.java:53)
> at
> org.apache.hcatalog.pig.HCatBaseStorer.putNext(HCatBaseStorer.java:242)
> at org.apache.hcatalog.pig.HCatStorer.putNext(HCatStorer.java:52)
> at
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigOutputFormat$PigRecordWriter.write(PigOutputFormat.java:139)
> at
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigOutputFormat$PigRecordWriter.write(PigOutputFormat.java:98)
> at
> org.apache.hadoop.mapred.MapTask$NewDirectOutputCollector.write(MapTask.java:559)
> at
> org.apache.hadoop.mapreduce.task.TaskInputOutputContextImpl.write(TaskInputOutputContextImpl.java:85)
> {code}
> The proximal cause of this failure is that the AvroContainerOutputFormat's
> signature mandates a LongWritable key and HCat's FileRecordWriterContainer
> forces a NullWritable. I'm not sure of a general fix, other than redefining
> HiveOutputFormat to mandate a WritableComparable.
> It looks like accepting WritableComparable is what's done in the other Hive
> OutputFormats, and there's no reason AvroContainerOutputFormat couldn't also
> be changed, since it's ignoring the key. That way fixing things so
> FileRecordWriterContainer can always use NullWritable could get spun into a
> different issue?
> The underlying cause for failure to write to AvroSerde tables is that
> AvroContainerOutputFormat doesn't meaningfully implement getRecordWriter, so
> fixing the above will just push the failure into the placeholder RecordWriter.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HIVE-4629) HS2 should support an API to retrieve query logs
[ https://issues.apache.org/jira/browse/HIVE-4629?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13925491#comment-13925491 ] Carl Steinbach commented on HIVE-4629: -- Does the new version of the patch address any of the API design issues I mentioned earlier? > HS2 should support an API to retrieve query logs > > > Key: HIVE-4629 > URL: https://issues.apache.org/jira/browse/HIVE-4629 > Project: Hive > Issue Type: Sub-task > Components: HiveServer2 >Reporter: Shreepadma Venugopalan >Assignee: Shreepadma Venugopalan > Attachments: HIVE-4629-no_thrift.1.patch, HIVE-4629.1.patch, > HIVE-4629.2.patch > > > HiveServer2 should support an API to retrieve query logs. This is > particularly relevant because HiveServer2 supports async execution but > doesn't provide a way to report progress. Providing an API to retrieve query > logs will help report progress to the client. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HIVE-6481) Add .reviewboardrc file
[ https://issues.apache.org/jira/browse/HIVE-6481?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13925358#comment-13925358 ] Carl Steinbach commented on HIVE-6481: -- I updated the wiki some instructions for using rbt. > Add .reviewboardrc file > --- > > Key: HIVE-6481 > URL: https://issues.apache.org/jira/browse/HIVE-6481 > Project: Hive > Issue Type: Improvement > Components: Build Infrastructure >Reporter: Carl Steinbach >Assignee: Carl Steinbach > Fix For: 0.13.0 > > Attachments: HIVE-6481.1.patch, HIVE-6481.2.patch > > > We should add a .reviewboardrc file to trunk in order to streamline the > review process. > Used in conjunction with RBTools this file makes posting a review request as > simple as executing the following command: > % rbt post -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HIVE-6570) Hive variable substitution does not work with the "source" command
[
https://issues.apache.org/jira/browse/HIVE-6570?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Carl Steinbach updated HIVE-6570:
-
Assignee: Anthony Hsu
> Hive variable substitution does not work with the "source" command
> --
>
> Key: HIVE-6570
> URL: https://issues.apache.org/jira/browse/HIVE-6570
> Project: Hive
> Issue Type: Bug
>Reporter: Anthony Hsu
>Assignee: Anthony Hsu
>
> The following does not work:
> {code}
> source ${hivevar:test-dir}/test.q;
> {code}
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HIVE-6482) Fix NOTICE file: pre release task
[ https://issues.apache.org/jira/browse/HIVE-6482?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13917696#comment-13917696 ] Carl Steinbach commented on HIVE-6482: -- [~rhbutani] Looks good to me. > Fix NOTICE file: pre release task > - > > Key: HIVE-6482 > URL: https://issues.apache.org/jira/browse/HIVE-6482 > Project: Hive > Issue Type: Bug >Reporter: Harish Butani >Assignee: Harish Butani >Priority: Trivial > Attachments: HIVE-6482.1.patch, HIVE-6482.2.patch > > > As per steps in Release doc: > https://cwiki.apache.org/confluence/display/Hive/HowToRelease > Removed projects with Apache license as per [~thejas] suggestion. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HIVE-6024) Load data local inpath unnecessarily creates a copy task
[
https://issues.apache.org/jira/browse/HIVE-6024?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13915991#comment-13915991
]
Carl Steinbach commented on HIVE-6024:
--
[~ashutoshc] The qfile test that is added in this patch doesn't seem to
demonstrate anything. I think it should be removed. What do you think?
> Load data local inpath unnecessarily creates a copy task
>
>
> Key: HIVE-6024
> URL: https://issues.apache.org/jira/browse/HIVE-6024
> Project: Hive
> Issue Type: Improvement
> Components: Query Processor
>Reporter: Ashutosh Chauhan
>Assignee: Mohammad Kamrul Islam
> Attachments: HIVE-6024.1.patch, HIVE-6024.2.patch, HIVE-6024.3.patch
>
>
> Load data command creates an additional copy task only when its loading from
> {{local}} It doesn't create this additional copy task while loading from DFS
> though.
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
[jira] [Updated] (HIVE-6024) Load data local inpath unnecessarily creates a copy task
[
https://issues.apache.org/jira/browse/HIVE-6024?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Carl Steinbach updated HIVE-6024:
-
Status: Open (was: Patch Available)
I left a comment on RB. Thanks.
> Load data local inpath unnecessarily creates a copy task
>
>
> Key: HIVE-6024
> URL: https://issues.apache.org/jira/browse/HIVE-6024
> Project: Hive
> Issue Type: Improvement
> Components: Query Processor
>Reporter: Ashutosh Chauhan
>Assignee: Mohammad Kamrul Islam
> Attachments: HIVE-6024.1.patch, HIVE-6024.2.patch, HIVE-6024.3.patch
>
>
> Load data command creates an additional copy task only when its loading from
> {{local}} It doesn't create this additional copy task while loading from DFS
> though.
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
[jira] [Updated] (HIVE-6481) Add .reviewboardrc file
[ https://issues.apache.org/jira/browse/HIVE-6481?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Carl Steinbach updated HIVE-6481: - Resolution: Fixed Fix Version/s: 0.13.0 Hadoop Flags: Reviewed Status: Resolved (was: Patch Available) Committed to trunk. > Add .reviewboardrc file > --- > > Key: HIVE-6481 > URL: https://issues.apache.org/jira/browse/HIVE-6481 > Project: Hive > Issue Type: Improvement > Components: Build Infrastructure >Reporter: Carl Steinbach >Assignee: Carl Steinbach > Fix For: 0.13.0 > > Attachments: HIVE-6481.1.patch, HIVE-6481.2.patch > > > We should add a .reviewboardrc file to trunk in order to streamline the > review process. > Used in conjunction with RBTools this file makes posting a review request as > simple as executing the following command: > % rbt post -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Updated] (HIVE-6481) Add .reviewboardrc file
[ https://issues.apache.org/jira/browse/HIVE-6481?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Carl Steinbach updated HIVE-6481: - Attachment: HIVE-6481.2.patch Attaching a new version of the patch that includes the ASF license header in the .reviewboardrc file. > Add .reviewboardrc file > --- > > Key: HIVE-6481 > URL: https://issues.apache.org/jira/browse/HIVE-6481 > Project: Hive > Issue Type: Improvement > Components: Build Infrastructure >Reporter: Carl Steinbach >Assignee: Carl Steinbach > Attachments: HIVE-6481.1.patch, HIVE-6481.2.patch > > > We should add a .reviewboardrc file to trunk in order to streamline the > review process. > Used in conjunction with RBTools this file makes posting a review request as > simple as executing the following command: > % rbt post -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Updated] (HIVE-6481) Add .reviewboardrc file
[ https://issues.apache.org/jira/browse/HIVE-6481?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Carl Steinbach updated HIVE-6481: - Description: We should add a .reviewboardrc file to trunk in order to streamline the review process. Used in conjunction with RBTools this file makes posting a review request as simple as executing the following command: % rbt post was:We should add a .reviewboardrc file to trunk in order to streamline the review process. > Add .reviewboardrc file > --- > > Key: HIVE-6481 > URL: https://issues.apache.org/jira/browse/HIVE-6481 > Project: Hive > Issue Type: Improvement > Components: Build Infrastructure >Reporter: Carl Steinbach >Assignee: Carl Steinbach > Attachments: HIVE-6481.1.patch > > > We should add a .reviewboardrc file to trunk in order to streamline the > review process. > Used in conjunction with RBTools this file makes posting a review request as > simple as executing the following command: > % rbt post -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Updated] (HIVE-6481) Add .reviewboardrc file
[ https://issues.apache.org/jira/browse/HIVE-6481?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Carl Steinbach updated HIVE-6481: - Status: Patch Available (was: Open) Review request: https://reviews.apache.org/r/18377/ > Add .reviewboardrc file > --- > > Key: HIVE-6481 > URL: https://issues.apache.org/jira/browse/HIVE-6481 > Project: Hive > Issue Type: Improvement > Components: Build Infrastructure >Reporter: Carl Steinbach >Assignee: Carl Steinbach > Attachments: HIVE-6481.1.patch > > > We should add a .reviewboardrc file to trunk in order to streamline the > review process. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Updated] (HIVE-6481) Add .reviewboardrc file
[ https://issues.apache.org/jira/browse/HIVE-6481?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Carl Steinbach updated HIVE-6481: - Attachment: HIVE-6481.1.patch > Add .reviewboardrc file > --- > > Key: HIVE-6481 > URL: https://issues.apache.org/jira/browse/HIVE-6481 > Project: Hive > Issue Type: Improvement > Components: Build Infrastructure >Reporter: Carl Steinbach >Assignee: Carl Steinbach > Attachments: HIVE-6481.1.patch > > > We should add a .reviewboardrc file to trunk in order to streamline the > review process. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Created] (HIVE-6481) Add .reviewboardrc file
Carl Steinbach created HIVE-6481: Summary: Add .reviewboardrc file Key: HIVE-6481 URL: https://issues.apache.org/jira/browse/HIVE-6481 Project: Hive Issue Type: Improvement Components: Build Infrastructure Reporter: Carl Steinbach Assignee: Carl Steinbach We should add a .reviewboardrc file to trunk in order to streamline the review process. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Updated] (HIVE-5998) Add vectorized reader for Parquet files
[ https://issues.apache.org/jira/browse/HIVE-5998?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Carl Steinbach updated HIVE-5998: - Component/s: Vectorization Serializers/Deserializers > Add vectorized reader for Parquet files > --- > > Key: HIVE-5998 > URL: https://issues.apache.org/jira/browse/HIVE-5998 > Project: Hive > Issue Type: Sub-task > Components: Serializers/Deserializers, Vectorization >Reporter: Remus Rusanu >Assignee: Remus Rusanu >Priority: Minor > Labels: Parquet > Attachments: HIVE-5998.1.patch > > > HIVE-5783 is adding native Parquet support in Hive. As Parquet is a columnar > format, it makes sense to provide a vectorized reader, similar to how RC and > ORC formats have, to benefit from vectorized execution engine. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Updated] (HIVE-6414) ParquetInputFormat provides data values that do not match the object inspectors
[
https://issues.apache.org/jira/browse/HIVE-6414?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Carl Steinbach updated HIVE-6414:
-
Labels: Parquet (was: )
> ParquetInputFormat provides data values that do not match the object
> inspectors
> ---
>
> Key: HIVE-6414
> URL: https://issues.apache.org/jira/browse/HIVE-6414
> Project: Hive
> Issue Type: Bug
> Components: Serializers/Deserializers
>Reporter: Remus Rusanu
>Assignee: Justin Coffey
> Labels: Parquet
>
> While working on HIVE-5998 I noticed that the ParquetRecordReader returns
> IntWritable for all 'int like' types, in disaccord with the row object
> inspectors. I though fine, and I worked my way around it. But I see now that
> the issue trigger failuers in other places, eg. in aggregates:
> {noformat}
> Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime
> Error while processing row
> {"cint":528534767,"ctinyint":31,"csmallint":4963,"cfloat":31.0,"cdouble":4963.0,"cstring1":"cvLH6Eat2yFsyy7p"}
> at
> org.apache.hadoop.hive.ql.exec.MapOperator.process(MapOperator.java:534)
> at
> org.apache.hadoop.hive.ql.exec.mr.ExecMapper.map(ExecMapper.java:177)
> ... 8 more
> Caused by: org.apache.hadoop.hive.ql.metadata.HiveException:
> java.lang.ClassCastException: org.apache.hadoop.io.IntWritable cannot be cast
> to java.lang.Short
> at
> org.apache.hadoop.hive.ql.exec.GroupByOperator.processOp(GroupByOperator.java:808)
> at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:790)
> at
> org.apache.hadoop.hive.ql.exec.SelectOperator.processOp(SelectOperator.java:87)
> at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:790)
> at
> org.apache.hadoop.hive.ql.exec.TableScanOperator.processOp(TableScanOperator.java:92)
> at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:790)
> at
> org.apache.hadoop.hive.ql.exec.MapOperator.process(MapOperator.java:524)
> ... 9 more
> Caused by: java.lang.ClassCastException: org.apache.hadoop.io.IntWritable
> cannot be cast to java.lang.Short
> at
> org.apache.hadoop.hive.serde2.objectinspector.primitive.JavaShortObjectInspector.get(JavaShortObjectInspector.java:41)
> at
> org.apache.hadoop.hive.serde2.objectinspector.ObjectInspectorUtils.compare(ObjectInspectorUtils.java:671)
> at
> org.apache.hadoop.hive.serde2.objectinspector.ObjectInspectorUtils.compare(ObjectInspectorUtils.java:631)
> at
> org.apache.hadoop.hive.ql.udf.generic.GenericUDAFMin$GenericUDAFMinEvaluator.merge(GenericUDAFMin.java:109)
> at
> org.apache.hadoop.hive.ql.udf.generic.GenericUDAFMin$GenericUDAFMinEvaluator.iterate(GenericUDAFMin.java:96)
> at
> org.apache.hadoop.hive.ql.udf.generic.GenericUDAFEvaluator.aggregate(GenericUDAFEvaluator.java:183)
> at
> org.apache.hadoop.hive.ql.exec.GroupByOperator.updateAggregations(GroupByOperator.java:641)
> at
> org.apache.hadoop.hive.ql.exec.GroupByOperator.processHashAggr(GroupByOperator.java:838)
> at
> org.apache.hadoop.hive.ql.exec.GroupByOperator.processKey(GroupByOperator.java:735)
> at
> org.apache.hadoop.hive.ql.exec.GroupByOperator.processOp(GroupByOperator.java:803)
> ... 15 more
> {noformat}
> My test is (I'm writing a test .q from HIVE-5998, but the repro does not
> involve vectorization):
> {noformat}
> create table if not exists alltypes_parquet (
> cint int,
> ctinyint tinyint,
> csmallint smallint,
> cfloat float,
> cdouble double,
> cstring1 string) stored as parquet;
> insert overwrite table alltypes_parquet
> select cint,
> ctinyint,
> csmallint,
> cfloat,
> cdouble,
> cstring1
> from alltypesorc;
> explain select * from alltypes_parquet limit 10; select * from
> alltypes_parquet limit 10;
> explain select ctinyint,
> max(cint),
> min(csmallint),
> count(cstring1),
> avg(cfloat),
> stddev_pop(cdouble)
> from alltypes_parquet
> group by ctinyint;
> select ctinyint,
> max(cint),
> min(csmallint),
> count(cstring1),
> avg(cfloat),
> stddev_pop(cdouble)
> from alltypes_parquet
> group by ctinyint;
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
[jira] [Updated] (HIVE-6394) Implement Timestmap in ParquetSerde
[ https://issues.apache.org/jira/browse/HIVE-6394?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Carl Steinbach updated HIVE-6394: - Labels: Parquet (was: ) > Implement Timestmap in ParquetSerde > --- > > Key: HIVE-6394 > URL: https://issues.apache.org/jira/browse/HIVE-6394 > Project: Hive > Issue Type: Sub-task > Components: Serializers/Deserializers >Reporter: Jarek Jarcec Cecho > Labels: Parquet > > This JIRA is to implement timestamp support in Parquet SerDe. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Updated] (HIVE-6367) Implement Decimal in ParquetSerde
[ https://issues.apache.org/jira/browse/HIVE-6367?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Carl Steinbach updated HIVE-6367: - Labels: Parquet (was: ) > Implement Decimal in ParquetSerde > - > > Key: HIVE-6367 > URL: https://issues.apache.org/jira/browse/HIVE-6367 > Project: Hive > Issue Type: Sub-task > Components: Serializers/Deserializers >Reporter: Brock Noland >Assignee: Xuefu Zhang > Labels: Parquet > > Some code in the Parquet Serde deals with decimal and other does not. For > example in ETypeConverter we convert Decimal to double (which is invalid) > whereas in DataWritableWriter and other locations we throw an exception if > decimal is used. > This JIRA is to implement decimal support. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Updated] (HIVE-6384) Implement all Hive data types in Parquet
[ https://issues.apache.org/jira/browse/HIVE-6384?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Carl Steinbach updated HIVE-6384: - Labels: Parquet (was: ) > Implement all Hive data types in Parquet > > > Key: HIVE-6384 > URL: https://issues.apache.org/jira/browse/HIVE-6384 > Project: Hive > Issue Type: Task >Reporter: Brock Noland > Labels: Parquet > > Uber JIRA to track implementation of binary, timestamp, date, char, varchar > or decimal. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Updated] (HIVE-6368) Document parquet on hive wiki
[ https://issues.apache.org/jira/browse/HIVE-6368?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Carl Steinbach updated HIVE-6368: - Labels: Parquet (was: ) > Document parquet on hive wiki > - > > Key: HIVE-6368 > URL: https://issues.apache.org/jira/browse/HIVE-6368 > Project: Hive > Issue Type: Task > Components: Serializers/Deserializers >Reporter: Brock Noland >Assignee: Brock Noland >Priority: Critical > Labels: Parquet > Fix For: 0.13.0 > > -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Updated] (HIVE-6367) Implement Decimal in ParquetSerde
[ https://issues.apache.org/jira/browse/HIVE-6367?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Carl Steinbach updated HIVE-6367: - Component/s: Serializers/Deserializers > Implement Decimal in ParquetSerde > - > > Key: HIVE-6367 > URL: https://issues.apache.org/jira/browse/HIVE-6367 > Project: Hive > Issue Type: Sub-task > Components: Serializers/Deserializers >Reporter: Brock Noland >Assignee: Xuefu Zhang > > Some code in the Parquet Serde deals with decimal and other does not. For > example in ETypeConverter we convert Decimal to double (which is invalid) > whereas in DataWritableWriter and other locations we throw an exception if > decimal is used. > This JIRA is to implement decimal support. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Updated] (HIVE-6368) Document parquet on hive wiki
[ https://issues.apache.org/jira/browse/HIVE-6368?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Carl Steinbach updated HIVE-6368: - Component/s: Serializers/Deserializers > Document parquet on hive wiki > - > > Key: HIVE-6368 > URL: https://issues.apache.org/jira/browse/HIVE-6368 > Project: Hive > Issue Type: Task > Components: Serializers/Deserializers >Reporter: Brock Noland >Assignee: Brock Noland >Priority: Critical > Labels: Parquet > Fix For: 0.13.0 > > -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Updated] (HIVE-6414) ParquetInputFormat provides data values that do not match the object inspectors
[
https://issues.apache.org/jira/browse/HIVE-6414?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Carl Steinbach updated HIVE-6414:
-
Component/s: Serializers/Deserializers
> ParquetInputFormat provides data values that do not match the object
> inspectors
> ---
>
> Key: HIVE-6414
> URL: https://issues.apache.org/jira/browse/HIVE-6414
> Project: Hive
> Issue Type: Bug
> Components: Serializers/Deserializers
>Reporter: Remus Rusanu
>Assignee: Justin Coffey
>
> While working on HIVE-5998 I noticed that the ParquetRecordReader returns
> IntWritable for all 'int like' types, in disaccord with the row object
> inspectors. I though fine, and I worked my way around it. But I see now that
> the issue trigger failuers in other places, eg. in aggregates:
> {noformat}
> Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime
> Error while processing row
> {"cint":528534767,"ctinyint":31,"csmallint":4963,"cfloat":31.0,"cdouble":4963.0,"cstring1":"cvLH6Eat2yFsyy7p"}
> at
> org.apache.hadoop.hive.ql.exec.MapOperator.process(MapOperator.java:534)
> at
> org.apache.hadoop.hive.ql.exec.mr.ExecMapper.map(ExecMapper.java:177)
> ... 8 more
> Caused by: org.apache.hadoop.hive.ql.metadata.HiveException:
> java.lang.ClassCastException: org.apache.hadoop.io.IntWritable cannot be cast
> to java.lang.Short
> at
> org.apache.hadoop.hive.ql.exec.GroupByOperator.processOp(GroupByOperator.java:808)
> at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:790)
> at
> org.apache.hadoop.hive.ql.exec.SelectOperator.processOp(SelectOperator.java:87)
> at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:790)
> at
> org.apache.hadoop.hive.ql.exec.TableScanOperator.processOp(TableScanOperator.java:92)
> at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:790)
> at
> org.apache.hadoop.hive.ql.exec.MapOperator.process(MapOperator.java:524)
> ... 9 more
> Caused by: java.lang.ClassCastException: org.apache.hadoop.io.IntWritable
> cannot be cast to java.lang.Short
> at
> org.apache.hadoop.hive.serde2.objectinspector.primitive.JavaShortObjectInspector.get(JavaShortObjectInspector.java:41)
> at
> org.apache.hadoop.hive.serde2.objectinspector.ObjectInspectorUtils.compare(ObjectInspectorUtils.java:671)
> at
> org.apache.hadoop.hive.serde2.objectinspector.ObjectInspectorUtils.compare(ObjectInspectorUtils.java:631)
> at
> org.apache.hadoop.hive.ql.udf.generic.GenericUDAFMin$GenericUDAFMinEvaluator.merge(GenericUDAFMin.java:109)
> at
> org.apache.hadoop.hive.ql.udf.generic.GenericUDAFMin$GenericUDAFMinEvaluator.iterate(GenericUDAFMin.java:96)
> at
> org.apache.hadoop.hive.ql.udf.generic.GenericUDAFEvaluator.aggregate(GenericUDAFEvaluator.java:183)
> at
> org.apache.hadoop.hive.ql.exec.GroupByOperator.updateAggregations(GroupByOperator.java:641)
> at
> org.apache.hadoop.hive.ql.exec.GroupByOperator.processHashAggr(GroupByOperator.java:838)
> at
> org.apache.hadoop.hive.ql.exec.GroupByOperator.processKey(GroupByOperator.java:735)
> at
> org.apache.hadoop.hive.ql.exec.GroupByOperator.processOp(GroupByOperator.java:803)
> ... 15 more
> {noformat}
> My test is (I'm writing a test .q from HIVE-5998, but the repro does not
> involve vectorization):
> {noformat}
> create table if not exists alltypes_parquet (
> cint int,
> ctinyint tinyint,
> csmallint smallint,
> cfloat float,
> cdouble double,
> cstring1 string) stored as parquet;
> insert overwrite table alltypes_parquet
> select cint,
> ctinyint,
> csmallint,
> cfloat,
> cdouble,
> cstring1
> from alltypesorc;
> explain select * from alltypes_parquet limit 10; select * from
> alltypes_parquet limit 10;
> explain select ctinyint,
> max(cint),
> min(csmallint),
> count(cstring1),
> avg(cfloat),
> stddev_pop(cdouble)
> from alltypes_parquet
> group by ctinyint;
> select ctinyint,
> max(cint),
> min(csmallint),
> count(cstring1),
> avg(cfloat),
> stddev_pop(cdouble)
> from alltypes_parquet
> group by ctinyint;
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
[jira] [Updated] (HIVE-6366) Refactor some items in Hive Parquet
[ https://issues.apache.org/jira/browse/HIVE-6366?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Carl Steinbach updated HIVE-6366: - Labels: Parquet (was: ) > Refactor some items in Hive Parquet > --- > > Key: HIVE-6366 > URL: https://issues.apache.org/jira/browse/HIVE-6366 > Project: Hive > Issue Type: Task >Reporter: Brock Noland > Labels: Parquet > > [~jcoffey] and myself have discussed some re-factoring to the Parquet Serde > we'd live to do post commit. > Specifically > * Clean up the labeled TODO items in the parquet code > * Understand if the paths need to be convered to schemless paths and then > cleanup the handling of paths in ProjectPusher > * Object inspectors are written such that they can inspect inspected results > (e.g. "if Hive decides to inspect the result of an inspection") which should > not happen > * BinaryWritable can be removed -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Updated] (HIVE-6375) Implement CTAS and column rename for parquet
[ https://issues.apache.org/jira/browse/HIVE-6375?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Carl Steinbach updated HIVE-6375: - Labels: Parquet (was: ) > Implement CTAS and column rename for parquet > > > Key: HIVE-6375 > URL: https://issues.apache.org/jira/browse/HIVE-6375 > Project: Hive > Issue Type: Bug >Reporter: Brock Noland >Priority: Critical > Labels: Parquet > > More details here: > https://github.com/Parquet/parquet-mr/issues/272 -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Updated] (HIVE-6394) Implement Timestmap in ParquetSerde
[ https://issues.apache.org/jira/browse/HIVE-6394?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Carl Steinbach updated HIVE-6394: - Component/s: Serializers/Deserializers > Implement Timestmap in ParquetSerde > --- > > Key: HIVE-6394 > URL: https://issues.apache.org/jira/browse/HIVE-6394 > Project: Hive > Issue Type: Sub-task > Components: Serializers/Deserializers >Reporter: Jarek Jarcec Cecho > Labels: Parquet > > This JIRA is to implement timestamp support in Parquet SerDe. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Updated] (HIVE-5783) Native Parquet Support in Hive
[ https://issues.apache.org/jira/browse/HIVE-5783?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Carl Steinbach updated HIVE-5783: - Labels: Parquet (was: ) > Native Parquet Support in Hive > -- > > Key: HIVE-5783 > URL: https://issues.apache.org/jira/browse/HIVE-5783 > Project: Hive > Issue Type: New Feature > Components: Serializers/Deserializers >Reporter: Justin Coffey >Assignee: Justin Coffey >Priority: Minor > Labels: Parquet > Fix For: 0.13.0 > > Attachments: HIVE-5783.noprefix.patch, HIVE-5783.noprefix.patch, > HIVE-5783.patch, HIVE-5783.patch, HIVE-5783.patch, HIVE-5783.patch, > HIVE-5783.patch, HIVE-5783.patch, HIVE-5783.patch, HIVE-5783.patch, > HIVE-5783.patch, HIVE-5783.patch, HIVE-5783.patch, HIVE-5783.patch, > HIVE-5783.patch, HIVE-5783.patch, HIVE-5783.patch, HIVE-5783.patch, > HIVE-5783.patch > > > Problem Statement: > Hive would be easier to use if it had native Parquet support. Our > organization, Criteo, uses Hive extensively. Therefore we built the Parquet > Hive integration and would like to now contribute that integration to Hive. > About Parquet: > Parquet is a columnar storage format for Hadoop and integrates with many > Hadoop ecosystem tools such as Thrift, Avro, Hadoop MapReduce, Cascading, > Pig, Drill, Crunch, and Hive. Pig, Crunch, and Drill all contain native > Parquet integration. > Changes Details: > Parquet was built with dependency management in mind and therefore only a > single Parquet jar will be added as a dependency. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Updated] (HIVE-5998) Add vectorized reader for Parquet files
[ https://issues.apache.org/jira/browse/HIVE-5998?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Carl Steinbach updated HIVE-5998: - Labels: Parquet (was: ) > Add vectorized reader for Parquet files > --- > > Key: HIVE-5998 > URL: https://issues.apache.org/jira/browse/HIVE-5998 > Project: Hive > Issue Type: Sub-task >Reporter: Remus Rusanu >Assignee: Remus Rusanu >Priority: Minor > Labels: Parquet > Attachments: HIVE-5998.1.patch > > > HIVE-5783 is adding native Parquet support in Hive. As Parquet is a columnar > format, it makes sense to provide a vectorized reader, similar to how RC and > ORC formats have, to benefit from vectorized execution engine. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Updated] (HIVE-4445) Fix the Hive unit test failures on Windows when Linux scripts or commands are used in test cases
[ https://issues.apache.org/jira/browse/HIVE-4445?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Carl Steinbach updated HIVE-4445: - Assignee: Xi Fang > Fix the Hive unit test failures on Windows when Linux scripts or commands are > used in test cases > > > Key: HIVE-4445 > URL: https://issues.apache.org/jira/browse/HIVE-4445 > Project: Hive > Issue Type: Bug > Components: Tests >Affects Versions: 0.11.0 > Environment: Windows >Reporter: Xi Fang >Assignee: Xi Fang > Attachments: HIVE-4445.1.patch > > > The following unit tests fail on Windows because Linux scripts or commands > are used in the test cases or .q files: > 1. TestMinimrCliDriver: scriptfile1.q > 2. TestNegativeMinimrCliDriver: mapreduce_stack_trace_hadoop20.q, > minimr_broken_pipe.q > 3. TestCliDriver: hiveprofiler_script0.q -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Updated] (HIVE-3129) Create windows native scripts (CMD files) to run hive on windows without Cygwin
[ https://issues.apache.org/jira/browse/HIVE-3129?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Carl Steinbach updated HIVE-3129: - Assignee: Xi Fang > Create windows native scripts (CMD files) to run hive on windows without > Cygwin > > > Key: HIVE-3129 > URL: https://issues.apache.org/jira/browse/HIVE-3129 > Project: Hive > Issue Type: Bug > Components: CLI, Windows >Affects Versions: 0.11.0 >Reporter: Kanna Karanam >Assignee: Xi Fang > Labels: Windows > Attachments: HIVE-3129.1.patch, HIVE-3129.2.patch, > HIVE-3129.unittest.2.patch, HIVE-3129.unittest.patch > > > Create the cmd files equivalent to > a)Bin\hive > b)Bin\hive-config.sh > c)Bin\Init-hive-dfs.sh > d)Bin\ext\cli.sh > e)Bin\ext\debug.sh > f)Bin\ext\help.sh > g)Bin\ext\hiveserver.sh > h)Bin\ext\jar.sh > i)Bin\ext\hwi.sh > j)Bin\ext\lineage.sh > k)Bin\ext\metastore.sh > l)Bin\ext\rcfilecat.sh -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Updated] (HIVE-4349) Fix the Hive unit test failures when the Hive enlistment root path is longer than ~12 characters
[ https://issues.apache.org/jira/browse/HIVE-4349?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Carl Steinbach updated HIVE-4349: - Assignee: Xi Fang > Fix the Hive unit test failures when the Hive enlistment root path is longer > than ~12 characters > > > Key: HIVE-4349 > URL: https://issues.apache.org/jira/browse/HIVE-4349 > Project: Hive > Issue Type: Bug >Affects Versions: 0.11.0 >Reporter: Xi Fang >Assignee: Xi Fang > Attachments: HIVE-4349.1.patch > > > If the Hive enlistment root path is longer than 12 chars then test classpath > “hadoop.testcp” is exceeding the 8K chars so we are unable to run most of the > Hive unit tests on Windows. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Updated] (HIVE-4558) mapreduce_stack_trace_hadoop20 in TestNegativeMinimrCliDriver fails on Windows
[
https://issues.apache.org/jira/browse/HIVE-4558?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Carl Steinbach updated HIVE-4558:
-
Assignee: Xi Fang
> mapreduce_stack_trace_hadoop20 in TestNegativeMinimrCliDriver fails on Windows
> --
>
> Key: HIVE-4558
> URL: https://issues.apache.org/jira/browse/HIVE-4558
> Project: Hive
> Issue Type: Bug
> Components: Tests
>Affects Versions: 0.11.0
> Environment: Windows
>Reporter: Xi Fang
>Assignee: Xi Fang
> Attachments: HIVE-4558.1.patch
>
>
> testNegativeCliDriver_mapreduce_stack_trace_hadoop20 fails because group
> information is printed out on Windows. Here is the example of
> mapreduce_stack_trace_hadoop20.q.out.orig:
> --
> PREHOOK: query: FROM src SELECT TRANSFORM(key, value) USING
> 'script_does_not_exist' AS (key, value)
> PREHOOK: type: QUERY
> PREHOOK: Input: default@src
> PREHOOK: Output:
> hdfs://127.0.0.1:25477/code/HWX/hive-monarch/build/ql/scratchdir/hive_2013-05-14_15-21-00_075_593034964465269090/-mr-1
> Ended Job = job_20130514152027587_0001 with errors
> FATAL ExecMapper: org.apache.hadoop.hive.ql.metadata.HiveException: Hive
> Runtime Error while processing row {"key":"238","value":"val_238"}
> groups found for user Administrators
> Hive Runtime Error while processing row {"key":"238","value":"val_238"}
> --
> However, it is supposed to look like:
> --
> PREHOOK: query: FROM src SELECT TRANSFORM(key, value) USING
> 'script_does_not_exist' AS (key, value)
> PREHOOK: type: QUERY
> PREHOOK: Input: default@src
> \ A masked pattern was here
> FATAL ExecMapper: org.apache.hadoop.hive.ql.metadata.HiveException: Hive
> Runtime Error while processing row {"key":"238","value":"val_238"}
> Hive Runtime Error while processing row {"key":"238","value":"val_238"}
> --
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
[jira] [Updated] (HIVE-4558) mapreduce_stack_trace_hadoop20 in TestNegativeMinimrCliDriver fails on Windows
[
https://issues.apache.org/jira/browse/HIVE-4558?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Carl Steinbach updated HIVE-4558:
-
Assignee: (was: Carl Steinbach)
> mapreduce_stack_trace_hadoop20 in TestNegativeMinimrCliDriver fails on Windows
> --
>
> Key: HIVE-4558
> URL: https://issues.apache.org/jira/browse/HIVE-4558
> Project: Hive
> Issue Type: Bug
> Components: Tests
>Affects Versions: 0.11.0
> Environment: Windows
>Reporter: Xi Fang
> Attachments: HIVE-4558.1.patch
>
>
> testNegativeCliDriver_mapreduce_stack_trace_hadoop20 fails because group
> information is printed out on Windows. Here is the example of
> mapreduce_stack_trace_hadoop20.q.out.orig:
> --
> PREHOOK: query: FROM src SELECT TRANSFORM(key, value) USING
> 'script_does_not_exist' AS (key, value)
> PREHOOK: type: QUERY
> PREHOOK: Input: default@src
> PREHOOK: Output:
> hdfs://127.0.0.1:25477/code/HWX/hive-monarch/build/ql/scratchdir/hive_2013-05-14_15-21-00_075_593034964465269090/-mr-1
> Ended Job = job_20130514152027587_0001 with errors
> FATAL ExecMapper: org.apache.hadoop.hive.ql.metadata.HiveException: Hive
> Runtime Error while processing row {"key":"238","value":"val_238"}
> groups found for user Administrators
> Hive Runtime Error while processing row {"key":"238","value":"val_238"}
> --
> However, it is supposed to look like:
> --
> PREHOOK: query: FROM src SELECT TRANSFORM(key, value) USING
> 'script_does_not_exist' AS (key, value)
> PREHOOK: type: QUERY
> PREHOOK: Input: default@src
> \ A masked pattern was here
> FATAL ExecMapper: org.apache.hadoop.hive.ql.metadata.HiveException: Hive
> Runtime Error while processing row {"key":"238","value":"val_238"}
> Hive Runtime Error while processing row {"key":"238","value":"val_238"}
> --
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
[jira] [Updated] (HIVE-4558) mapreduce_stack_trace_hadoop20 in TestNegativeMinimrCliDriver fails on Windows
[
https://issues.apache.org/jira/browse/HIVE-4558?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Carl Steinbach updated HIVE-4558:
-
Assignee: Carl Steinbach
> mapreduce_stack_trace_hadoop20 in TestNegativeMinimrCliDriver fails on Windows
> --
>
> Key: HIVE-4558
> URL: https://issues.apache.org/jira/browse/HIVE-4558
> Project: Hive
> Issue Type: Bug
> Components: Tests
>Affects Versions: 0.11.0
> Environment: Windows
>Reporter: Xi Fang
>Assignee: Carl Steinbach
> Attachments: HIVE-4558.1.patch
>
>
> testNegativeCliDriver_mapreduce_stack_trace_hadoop20 fails because group
> information is printed out on Windows. Here is the example of
> mapreduce_stack_trace_hadoop20.q.out.orig:
> --
> PREHOOK: query: FROM src SELECT TRANSFORM(key, value) USING
> 'script_does_not_exist' AS (key, value)
> PREHOOK: type: QUERY
> PREHOOK: Input: default@src
> PREHOOK: Output:
> hdfs://127.0.0.1:25477/code/HWX/hive-monarch/build/ql/scratchdir/hive_2013-05-14_15-21-00_075_593034964465269090/-mr-1
> Ended Job = job_20130514152027587_0001 with errors
> FATAL ExecMapper: org.apache.hadoop.hive.ql.metadata.HiveException: Hive
> Runtime Error while processing row {"key":"238","value":"val_238"}
> groups found for user Administrators
> Hive Runtime Error while processing row {"key":"238","value":"val_238"}
> --
> However, it is supposed to look like:
> --
> PREHOOK: query: FROM src SELECT TRANSFORM(key, value) USING
> 'script_does_not_exist' AS (key, value)
> PREHOOK: type: QUERY
> PREHOOK: Input: default@src
> \ A masked pattern was here
> FATAL ExecMapper: org.apache.hadoop.hive.ql.metadata.HiveException: Hive
> Runtime Error while processing row {"key":"238","value":"val_238"}
> Hive Runtime Error while processing row {"key":"238","value":"val_238"}
> --
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
[jira] [Commented] (HIVE-5783) Native Parquet Support in Hive
[ https://issues.apache.org/jira/browse/HIVE-5783?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13881624#comment-13881624 ] Carl Steinbach commented on HIVE-5783: -- I noticed that this SerDe doesn't support several of Hive's types: binary, timestamp, date, and probably a couple others as well. If there other known limitations it would be helpful to list them. > Native Parquet Support in Hive > -- > > Key: HIVE-5783 > URL: https://issues.apache.org/jira/browse/HIVE-5783 > Project: Hive > Issue Type: New Feature > Components: Serializers/Deserializers >Reporter: Justin Coffey >Assignee: Justin Coffey >Priority: Minor > Fix For: 0.13.0 > > Attachments: HIVE-5783.patch, HIVE-5783.patch, HIVE-5783.patch, > HIVE-5783.patch, HIVE-5783.patch, HIVE-5783.patch, HIVE-5783.patch > > > Problem Statement: > Hive would be easier to use if it had native Parquet support. Our > organization, Criteo, uses Hive extensively. Therefore we built the Parquet > Hive integration and would like to now contribute that integration to Hive. > About Parquet: > Parquet is a columnar storage format for Hadoop and integrates with many > Hadoop ecosystem tools such as Thrift, Avro, Hadoop MapReduce, Cascading, > Pig, Drill, Crunch, and Hive. Pig, Crunch, and Drill all contain native > Parquet integration. > Changes Details: > Parquet was built with dependency management in mind and therefore only a > single Parquet jar will be added as a dependency. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Updated] (HIVE-6043) Document incompatible changes in Hive 0.12 and trunk
[ https://issues.apache.org/jira/browse/HIVE-6043?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Carl Steinbach updated HIVE-6043: - Description: We need to document incompatible changes. For example * HIVE-5372 changed object inspector hierarchy breaking most if not all custom serdes * HIVE-1511/HIVE-5263 serializes ObjectInspectors with Kryo so all custom serdes (fixed by HIVE-5380) * Hive 0.12 (HIVE-4825) separates MapredWork into MapWork and ReduceWork which is used by Serdes * HIVE-5411 serializes expressions with Kryo which are used by custom serdes * HIVE-4827 removed the flag of "hive.optimize.mapjoin.mapreduce" (This flag was introduced in Hive 0.11 by HIVE-3952). was: We need to document incompatible changes. For example * HIVE-5372 changed object inspector hierarchy breaking most if not all custom serdes * HIVE-1511/HIVE-5263 serializes ObjectInspectors with Kryo so all custom serdes (fixed by HIVE-5380) * Hive 0.12 separates MapredWork into MapWork and ReduceWork which is used by Serdes * HIVE-5411 serializes expressions with Kryo which are used by custom serdes * HIVE-4827 removed the flag of "hive.optimize.mapjoin.mapreduce" (This flag was introduced in Hive 0.11 by HIVE-3952). > Document incompatible changes in Hive 0.12 and trunk > > > Key: HIVE-6043 > URL: https://issues.apache.org/jira/browse/HIVE-6043 > Project: Hive > Issue Type: Task >Reporter: Brock Noland >Priority: Blocker > > We need to document incompatible changes. For example > * HIVE-5372 changed object inspector hierarchy breaking most if not all > custom serdes > * HIVE-1511/HIVE-5263 serializes ObjectInspectors with Kryo so all custom > serdes (fixed by HIVE-5380) > * Hive 0.12 (HIVE-4825) separates MapredWork into MapWork and ReduceWork > which is used by Serdes > * HIVE-5411 serializes expressions with Kryo which are used by custom serdes > * HIVE-4827 removed the flag of "hive.optimize.mapjoin.mapreduce" (This flag > was introduced in Hive 0.11 by HIVE-3952). -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (HIVE-5783) Native Parquet Support in Hive
[ https://issues.apache.org/jira/browse/HIVE-5783?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13875419#comment-13875419 ] Carl Steinbach commented on HIVE-5783: -- I noticed that many of the source files contain Criteo copyright notices. The ASF has a policy on this which is documented here: https://www.apache.org/legal/src-headers.html Since this patch was submitted directly to the ASF by the copyright owner or owner's agent it sounds like we have three options for handling this: # Remove the notices # move them to the NOTICE file associated with each applicable project release, or # provide written permission for the ASF to make such removal or relocation of the notices [~jcoffey] [~rusanu] Do you guys have preference? > Native Parquet Support in Hive > -- > > Key: HIVE-5783 > URL: https://issues.apache.org/jira/browse/HIVE-5783 > Project: Hive > Issue Type: New Feature > Components: Serializers/Deserializers >Reporter: Justin Coffey >Assignee: Justin Coffey >Priority: Minor > Attachments: HIVE-5783.patch, HIVE-5783.patch, > hive-0.11-parquet.patch, parquet-hive.patch > > > Problem Statement: > Hive would be easier to use if it had native Parquet support. Our > organization, Criteo, uses Hive extensively. Therefore we built the Parquet > Hive integration and would like to now contribute that integration to Hive. > About Parquet: > Parquet is a columnar storage format for Hadoop and integrates with many > Hadoop ecosystem tools such as Thrift, Avro, Hadoop MapReduce, Cascading, > Pig, Drill, Crunch, and Hive. Pig, Crunch, and Drill all contain native > Parquet integration. > Changes Details: > Parquet was built with dependency management in mind and therefore only a > single Parquet jar will be added as a dependency. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Updated] (HIVE-3159) Update AvroSerde to determine schema of new tables
[ https://issues.apache.org/jira/browse/HIVE-3159?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Carl Steinbach updated HIVE-3159: - Status: Open (was: Patch Available) I left some comments on reviewboard. Thanks. > Update AvroSerde to determine schema of new tables > -- > > Key: HIVE-3159 > URL: https://issues.apache.org/jira/browse/HIVE-3159 > Project: Hive > Issue Type: Improvement > Components: Serializers/Deserializers >Reporter: Jakob Homan >Assignee: Mohammad Kamrul Islam > Attachments: HIVE-3159.4.patch, HIVE-3159.5.patch, HIVE-3159.6.patch, > HIVE-3159.7.patch, HIVE-3159v1.patch > > > Currently when writing tables to Avro one must manually provide an Avro > schema that matches what is being delivered by Hive. It'd be better to have > the serde infer this schema by converting the table's TypeInfo into an > appropriate AvroSchema. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Updated] (HIVE-5829) Rewrite Trim and Pad UDFs based on GenericUDF
[ https://issues.apache.org/jira/browse/HIVE-5829?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Carl Steinbach updated HIVE-5829: - Fix Version/s: 0.13.0 Committed to trunk. Thanks Mohammad! > Rewrite Trim and Pad UDFs based on GenericUDF > - > > Key: HIVE-5829 > URL: https://issues.apache.org/jira/browse/HIVE-5829 > Project: Hive > Issue Type: Bug > Components: UDF >Reporter: Mohammad Kamrul Islam >Assignee: Mohammad Kamrul Islam > Fix For: 0.13.0 > > Attachments: HIVE-5829.1.patch, HIVE-5829.2.patch, HIVE-5829.3.patch, > HIVE-5829.4.patch, tmp.HIVE-5829.patch > > > This JIRA includes following UDFs: > 1. trim() > 2. ltrim() > 3. rtrim() > 4. lpad() > 5. rpad() -- This message was sent by Atlassian JIRA (v6.1.5#6160)
