Re: Review Request 18179: Support more generic way of using composite key for HBaseHandler
On April 6, 2014, 4:32 a.m., Xuefu Zhang wrote: hbase-handler/src/java/org/apache/hadoop/hive/hbase/CompositeHBaseKeyFactory.java, line 98 https://reviews.apache.org/r/18179/diff/8/?file=544997#file544997line98 Can we pass those as constructor arguments instead of individual set methods? It was like that long before and not fixed even in your refactoring patch. I cannot sure it's meaningful to change those columns to be arguments of constructor, if the analyzer has add method. Or you mean we should remove add method altogether? On April 6, 2014, 4:32 a.m., Xuefu Zhang wrote: ql/src/java/org/apache/hadoop/hive/ql/index/IndexPredicateAnalyzer.java, line 212 https://reviews.apache.org/r/18179/diff/8/?file=545019#file545019line212 What is this change for? Currently, IndexSearchCondition does not allow field, so I've improvised string[] to describe fields not hurting other codes. Wish I could have a better idea. On April 6, 2014, 4:32 a.m., Xuefu Zhang wrote: ql/src/java/org/apache/hadoop/hive/ql/plan/ExprNodeDescUtils.java, line 264 https://reviews.apache.org/r/18179/diff/8/?file=545025#file545025line264 Can we have some comments here describing what we are doing? sure On April 6, 2014, 4:32 a.m., Xuefu Zhang wrote: ql/src/java/org/apache/hadoop/hive/ql/plan/ExprNodeDescUtils.java, line 277 https://reviews.apache.org/r/18179/diff/8/?file=545025#file545025line277 Same as above. sure On April 6, 2014, 4:32 a.m., Xuefu Zhang wrote: ql/src/java/org/apache/hadoop/hive/ql/plan/TableScanDesc.java, line 73 https://reviews.apache.org/r/18179/diff/8/?file=545026#file545026line73 I don't quite follow the need of serializing filter object. In the HIVE-6290, analyzer makes a Scan object rather than ExprDesc. So made generic way of handling this kind of special predicates. I used similar way for handling over predicate object into non-native system and felt it's quite useful. On April 6, 2014, 4:32 a.m., Xuefu Zhang wrote: serde/src/java/org/apache/hadoop/hive/serde2/lazy/LazySimpleSerDe.java, line 174 https://reviews.apache.org/r/18179/diff/8/?file=545034#file545034line174 I don't see any use of this method. Remnant of previous patch. removed. On April 6, 2014, 4:32 a.m., Xuefu Zhang wrote: hbase-handler/src/java/org/apache/hadoop/hive/hbase/ColumnMappings.java, line 40 https://reviews.apache.org/r/18179/diff/8/?file=544996#file544996line40 I think I had a previous comment that hasn't been addressed. It' scopied here: Can we use List? I don't see any advantages of array in this case. Rather, it brings additional code handling the list-array conversion. commented above as I prefer arrays than lists for non-mutables, which implies it's ready and would not be added or removed further Yes, I just prefer array. If you don't like it, I'll change. On April 6, 2014, 4:32 a.m., Xuefu Zhang wrote: hbase-handler/src/java/org/apache/hadoop/hive/hbase/CompositeHBaseKeyFactory.java, line 93 https://reviews.apache.org/r/18179/diff/8/?file=544997#file544997line93 Can we have come comments describe what we achieve here functionally? It's hard to understand what's going on here. I didn't see this part of code in previous review. Much of codes are included from the patch of HIVE-6290, cause I didn't bother Swarnim to rebase it on this. I should do some investigation for proper comments on it. On April 6, 2014, 4:32 a.m., Xuefu Zhang wrote: ql/src/java/org/apache/hadoop/hive/ql/index/IndexPredicateAnalyzer.java, line 190 https://reviews.apache.org/r/18179/diff/8/?file=545019#file545019line190 Is this to fix some bug? Regretfully, I've appended null marker to discern const-column between column-const in HIVE-3617. That became more complicated by supporting ExprNodeFieldDesc to be analyzed. Now we know the order by comparing the first ExprDesc of array is constant. On April 6, 2014, 4:32 a.m., Xuefu Zhang wrote: ql/src/java/org/apache/hadoop/hive/ql/index/IndexPredicateAnalyzer.java, line 281 https://reviews.apache.org/r/18179/diff/8/?file=545019#file545019line281 There seems to have some indention problem. You mean 100 columns limit? I thought anyone respects code convention in theses days. I'll fix that. On April 6, 2014, 4:32 a.m., Xuefu Zhang wrote: serde/src/java/org/apache/hadoop/hive/serde2/lazy/LazyObjectBase.java, line 21 https://reviews.apache.org/r/18179/diff/8/?file=545033#file545033line21 The class name doesn't sound like an interface. When this be done, users would implement or extend this class. From the aspect of freedom provided, I thought interface would be (much) better for them. Wanted the name 'LazyObject', but it's already occupied and couldn't find excuses for further changes. - Navis --- This is an
[jira] [Updated] (HIVE-6809) Support bulk deleting directories for partition drop with partial spec
[ https://issues.apache.org/jira/browse/HIVE-6809?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Navis updated HIVE-6809: Attachment: HIVE-6809.3.patch.txt Support bulk deleting directories for partition drop with partial spec -- Key: HIVE-6809 URL: https://issues.apache.org/jira/browse/HIVE-6809 Project: Hive Issue Type: Improvement Components: Query Processor Reporter: Navis Assignee: Navis Attachments: HIVE-6809.1.patch.txt, HIVE-6809.2.patch.txt, HIVE-6809.3.patch.txt In busy hadoop system, dropping many of partitions takes much more time than expected. In hive-0.11.0, removing 1700 partitions by single partial spec took 90 minutes, which is reduced to 3 minutes when deleteData is set false. I couldn't test this in recent hive, which has HIVE-6256 but if the time-taking part is mostly from removing directories, it seemed not helpful to reduce whole processing time. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HIVE-6245) HS2 creates DBs/Tables with wrong ownership when HMS setugi is true
[ https://issues.apache.org/jira/browse/HIVE-6245?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13961755#comment-13961755 ] Sergey commented on HIVE-6245: -- We do run ALTER TABLE ADD PARTITION ... LOCATION 'hdfs://bla-bla' through JDBC as oozie-java-action. Sometimes it works, sometimes it doesn't work. HiveServer2 'looses' passed username and sets username='anonymous' and refuses to add partition because user named 'anonymous' has no access right to partition location. Whe does it happen from time to time? For example we have an oozie coordinator which adds 24 partitions during a day. It can work for a week and then suddenly fails with this 'anonymous' user. HS2 creates DBs/Tables with wrong ownership when HMS setugi is true --- Key: HIVE-6245 URL: https://issues.apache.org/jira/browse/HIVE-6245 Project: Hive Issue Type: Bug Components: HiveServer2 Affects Versions: 0.12.0 Reporter: Chaoyu Tang Assignee: Chaoyu Tang Attachments: HIVE-6245.patch The case with following settings is valid but does not work correctly in current HS2: == hive.server2.authentication=NONE (or LDAP) hive.server2.enable.doAs= true hive.metastore.sasl.enabled=false hive.metastore.execute.setugi=true == Ideally, HS2 is able to impersonate the logged in user (from Beeline, or JDBC application) and create DBs/Tables with user's ownership. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HIVE-6809) Support bulk deleting directories for partition drop with partial spec
[
https://issues.apache.org/jira/browse/HIVE-6809?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13961767#comment-13961767
]
Hive QA commented on HIVE-6809:
---
{color:red}Overall{color}: -1 at least one tests failed
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12638949/HIVE-6809.3.patch.txt
{color:red}ERROR:{color} -1 due to 2 failed/errored test(s), 5549 tests executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_reduce_deduplicate
org.apache.hcatalog.security.TestHdfsAuthorizationProvider.testTableOps
{noformat}
Test results:
http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/2160/testReport
Console output:
http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/2160/console
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 2 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12638949
Support bulk deleting directories for partition drop with partial spec
--
Key: HIVE-6809
URL: https://issues.apache.org/jira/browse/HIVE-6809
Project: Hive
Issue Type: Improvement
Components: Query Processor
Reporter: Navis
Assignee: Navis
Attachments: HIVE-6809.1.patch.txt, HIVE-6809.2.patch.txt,
HIVE-6809.3.patch.txt
In busy hadoop system, dropping many of partitions takes much more time than
expected. In hive-0.11.0, removing 1700 partitions by single partial spec
took 90 minutes, which is reduced to 3 minutes when deleteData is set false.
I couldn't test this in recent hive, which has HIVE-6256 but if the
time-taking part is mostly from removing directories, it seemed not helpful
to reduce whole processing time.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Resolved] (HIVE-3032) Msck query execution is failing giving NullPointer Exception
[ https://issues.apache.org/jira/browse/HIVE-3032?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chinna Rao Lalam resolved HIVE-3032. Resolution: Not a Problem With the trunk code base its not an issue. Msck query execution is failing giving NullPointer Exception Key: HIVE-3032 URL: https://issues.apache.org/jira/browse/HIVE-3032 Project: Hive Issue Type: Bug Components: Metastore Affects Versions: 0.8.1, 0.9.0, 0.9.1 Reporter: Kristam Subba Swathi Assignee: Chinna Rao Lalam Attachments: HIVE-3032.patch -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HIVE-6846) allow safe set commands with sql standard authorization
[ https://issues.apache.org/jira/browse/HIVE-6846?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thejas M Nair updated HIVE-6846: Attachment: HIVE-6846.1.patch allow safe set commands with sql standard authorization --- Key: HIVE-6846 URL: https://issues.apache.org/jira/browse/HIVE-6846 Project: Hive Issue Type: Bug Components: Authorization Affects Versions: 0.13.0 Reporter: Thejas M Nair Assignee: Thejas M Nair Attachments: HIVE-6846.1.patch HIVE-6827 disables all set commands when SQL standard authorization is turned on, but not all set commands are unsafe. We should allow safe set commands. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HIVE-6846) allow safe set commands with sql standard authorization
[ https://issues.apache.org/jira/browse/HIVE-6846?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13961792#comment-13961792 ] Thejas M Nair commented on HIVE-6846: - The patch also cleans up setting of various configuration options when sql std auth is used, and moves all that to one place. allow safe set commands with sql standard authorization --- Key: HIVE-6846 URL: https://issues.apache.org/jira/browse/HIVE-6846 Project: Hive Issue Type: Bug Components: Authorization Affects Versions: 0.13.0 Reporter: Thejas M Nair Assignee: Thejas M Nair Attachments: HIVE-6846.1.patch HIVE-6827 disables all set commands when SQL standard authorization is turned on, but not all set commands are unsafe. We should allow safe set commands. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HIVE-6846) allow safe set commands with sql standard authorization
[ https://issues.apache.org/jira/browse/HIVE-6846?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thejas M Nair updated HIVE-6846: Fix Version/s: 0.13.0 Status: Patch Available (was: Open) [~rhbutani] This removes some of the restrictions placed in HIVE-6827. I think this is extremely valuable for usability of sql standard authorization. I think it makes sense to include it for 0.13 release. allow safe set commands with sql standard authorization --- Key: HIVE-6846 URL: https://issues.apache.org/jira/browse/HIVE-6846 Project: Hive Issue Type: Bug Components: Authorization Affects Versions: 0.13.0 Reporter: Thejas M Nair Assignee: Thejas M Nair Fix For: 0.13.0 Attachments: HIVE-6846.1.patch HIVE-6827 disables all set commands when SQL standard authorization is turned on, but not all set commands are unsafe. We should allow safe set commands. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HIVE-6846) allow safe set commands with sql standard authorization
[
https://issues.apache.org/jira/browse/HIVE-6846?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13961854#comment-13961854
]
Hive QA commented on HIVE-6846:
---
{color:red}Overall{color}: -1 at least one tests failed
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12638964/HIVE-6846.1.patch
{color:red}ERROR:{color} -1 due to 27 failed/errored test(s), 5464 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_parallel_orderby
org.apache.hadoop.hive.ql.parse.authorization.TestSessionUserName.testSessionConstructorUser
org.apache.hadoop.hive.ql.parse.authorization.TestSessionUserName.testSessionDefaultUser
org.apache.hive.beeline.TestBeeLineWithArgs.org.apache.hive.beeline.TestBeeLineWithArgs
org.apache.hive.jdbc.TestJdbcDriver2.org.apache.hive.jdbc.TestJdbcDriver2
org.apache.hive.jdbc.TestJdbcWithMiniHS2.org.apache.hive.jdbc.TestJdbcWithMiniHS2
org.apache.hive.jdbc.TestJdbcWithMiniMr.org.apache.hive.jdbc.TestJdbcWithMiniMr
org.apache.hive.jdbc.TestSSL.testConnectionMismatch
org.apache.hive.jdbc.TestSSL.testInvalidConfig
org.apache.hive.jdbc.TestSSL.testSSLConnectionWithProperty
org.apache.hive.jdbc.TestSSL.testSSLConnectionWithURL
org.apache.hive.jdbc.TestSSL.testSSLFetch
org.apache.hive.jdbc.TestSSL.testSSLFetchHttp
org.apache.hive.jdbc.authorization.TestJdbcWithSQLAuthorization.org.apache.hive.jdbc.authorization.TestJdbcWithSQLAuthorization
org.apache.hive.jdbc.miniHS2.TestHiveServer2.testConnection
org.apache.hive.jdbc.miniHS2.TestHiveServer2.testGetVariableValue
org.apache.hive.jdbc.miniHS2.TestMiniHS2.testConfInSession
org.apache.hive.service.auth.TestCustomAuthentication.org.apache.hive.service.auth.TestCustomAuthentication
org.apache.hive.service.auth.TestPlainSaslHelper.testDoAsSetting
org.apache.hive.service.cli.TestEmbeddedThriftBinaryCLIService.org.apache.hive.service.cli.TestEmbeddedThriftBinaryCLIService
org.apache.hive.service.cli.TestScratchDir.testLocalScratchDirs
org.apache.hive.service.cli.TestScratchDir.testResourceDirs
org.apache.hive.service.cli.TestScratchDir.testScratchDirs
org.apache.hive.service.cli.session.TestSessionHooks.testProxyUser
org.apache.hive.service.cli.session.TestSessionHooks.testSessionHook
org.apache.hive.service.cli.thrift.TestThriftBinaryCLIService.org.apache.hive.service.cli.thrift.TestThriftBinaryCLIService
org.apache.hive.service.cli.thrift.TestThriftHttpCLIService.org.apache.hive.service.cli.thrift.TestThriftHttpCLIService
{noformat}
Test results:
http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/2161/testReport
Console output:
http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/2161/console
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 27 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12638964
allow safe set commands with sql standard authorization
---
Key: HIVE-6846
URL: https://issues.apache.org/jira/browse/HIVE-6846
Project: Hive
Issue Type: Bug
Components: Authorization
Affects Versions: 0.13.0
Reporter: Thejas M Nair
Assignee: Thejas M Nair
Fix For: 0.13.0
Attachments: HIVE-6846.1.patch
HIVE-6827 disables all set commands when SQL standard authorization is turned
on, but not all set commands are unsafe. We should allow safe set commands.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HIVE-6846) allow safe set commands with sql standard authorization
[ https://issues.apache.org/jira/browse/HIVE-6846?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13961870#comment-13961870 ] Harish Butani commented on HIVE-6846: - +1 for 0.13 allow safe set commands with sql standard authorization --- Key: HIVE-6846 URL: https://issues.apache.org/jira/browse/HIVE-6846 Project: Hive Issue Type: Bug Components: Authorization Affects Versions: 0.13.0 Reporter: Thejas M Nair Assignee: Thejas M Nair Fix For: 0.13.0 Attachments: HIVE-6846.1.patch HIVE-6827 disables all set commands when SQL standard authorization is turned on, but not all set commands are unsafe. We should allow safe set commands. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HIVE-6757) Remove deprecated parquet classes from outside of org.apache package
[ https://issues.apache.org/jira/browse/HIVE-6757?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13961892#comment-13961892 ] Justin Coffey commented on HIVE-6757: - much appreciated Harish! Remove deprecated parquet classes from outside of org.apache package Key: HIVE-6757 URL: https://issues.apache.org/jira/browse/HIVE-6757 Project: Hive Issue Type: Bug Reporter: Owen O'Malley Assignee: Owen O'Malley Priority: Blocker Fix For: 0.13.0 Attachments: HIVE-6757.2.patch, HIVE-6757.patch, parquet-hive.patch Apache shouldn't release projects with files outside of the org.apache namespace. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HIVE-5687) Streaming support in Hive
[ https://issues.apache.org/jira/browse/HIVE-5687?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13961928#comment-13961928 ] Lars Francke commented on HIVE-5687: To add to Owen's style guidelines thing: Just throwing this patch in my IDE gives me a lot of warnings and errors. Things like: * missorted modifiers (static final private - private static final) * Unnecessary package-level visibility * Redundant exceptions in throws clauses * Some very weird formatting * Call to simple getters from within class * for loop without initializer that can be a while loop * Unused variables * Conditions that are always true or false * Empty Javadoc tags * Unnecessary this * Missing @Override annotations * StringBuffer usage * Modifiers in interfaces (public) etc. I'm happy to do a full review on ReviewBoard but these are all things that Eclipse and IntelliJ can show you out of the box. So I'd appreciate it if you could set your IDE up to show these things and fix them in addition to using proper code formatting. Contact me if I can help in any way. Streaming support in Hive - Key: HIVE-5687 URL: https://issues.apache.org/jira/browse/HIVE-5687 Project: Hive Issue Type: Sub-task Reporter: Roshan Naik Assignee: Roshan Naik Labels: ACID, Streaming Fix For: 0.13.0 Attachments: 5687-api-spec4.pdf, 5687-draft-api-spec.pdf, 5687-draft-api-spec2.pdf, 5687-draft-api-spec3.pdf, HIVE-5687-unit-test-fix.patch, HIVE-5687.patch, HIVE-5687.v2.patch, HIVE-5687.v3.patch, HIVE-5687.v4.patch, HIVE-5687.v5.patch, Hive Streaming Ingest API for v3 patch.pdf, Hive Streaming Ingest API for v4 patch.pdf Implement support for Streaming data into HIVE. - Provide a client streaming API - Transaction support: Clients should be able to periodically commit a batch of records atomically - Immediate visibility: Records should be immediately visible to queries on commit - Should not overload HDFS with too many small files Use Cases: - Streaming logs into HIVE via Flume - Streaming results of computations from Storm -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Created] (HIVE-6853) show create table for hbase tables should exclude LOCATION
Miklos Christine created HIVE-6853: -- Summary: show create table for hbase tables should exclude LOCATION Key: HIVE-6853 URL: https://issues.apache.org/jira/browse/HIVE-6853 Project: Hive Issue Type: Bug Components: StorageHandler Affects Versions: 0.10.0 Reporter: Miklos Christine If you create a table on top of hbase in hive and issue a show create table hbase_table, it gives a bad DDL. It should not show LOCATION: [hive]$ cat /tmp/test_create.sql CREATE EXTERNAL TABLE nba_twitter.hbase2( key string COMMENT 'from deserializer', name string COMMENT 'from deserializer', pdt string COMMENT 'from deserializer', service string COMMENT 'from deserializer', term string COMMENT 'from deserializer', update1 string COMMENT 'from deserializer') ROW FORMAT SERDE 'org.apache.hadoop.hive.hbase.HBaseSerDe' STORED BY 'org.apache.hadoop.hive.hbase.HBaseStorageHandler' WITH SERDEPROPERTIES ( 'serialization.format'='1', 'hbase.columns.mapping'=':key,srv:name,srv:pdt,srv:service,srv:term,srv:update') LOCATION 'hdfs://nameservice1/user/hive/warehouse/nba_twitter.db/hbase' TBLPROPERTIES ( 'hbase.table.name'='NBATwitter', 'transient_lastDdlTime'='1386172188') Trying to create a table using the above fails: [hive]$ hive -f /tmp/test_create.sql cli -f /tmp/test_create.sql Logging initialized using configuration in jar:file:/opt/cloudera/parcels/CDH-4.4.0-1.cdh4.4.0.p0.39/lib/hive/lib/hive-common-0.10.0-cdh4.4.0.jar!/hive-log4j.properties FAILED: Error in metadata: MetaException(message:LOCATION may not be specified for HBase.) FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.DDLTask However, if I remove the LOCATION, then the DDL is valid. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HIVE-6853) show create table for hbase tables should exclude LOCATION
[ https://issues.apache.org/jira/browse/HIVE-6853?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Miklos Christine updated HIVE-6853: --- Attachment: HIVE-6853-0.patch show create table for hbase tables should exclude LOCATION --- Key: HIVE-6853 URL: https://issues.apache.org/jira/browse/HIVE-6853 Project: Hive Issue Type: Bug Components: StorageHandler Affects Versions: 0.10.0 Reporter: Miklos Christine Attachments: HIVE-6853-0.patch If you create a table on top of hbase in hive and issue a show create table hbase_table, it gives a bad DDL. It should not show LOCATION: [hive]$ cat /tmp/test_create.sql CREATE EXTERNAL TABLE nba_twitter.hbase2( key string COMMENT 'from deserializer', name string COMMENT 'from deserializer', pdt string COMMENT 'from deserializer', service string COMMENT 'from deserializer', term string COMMENT 'from deserializer', update1 string COMMENT 'from deserializer') ROW FORMAT SERDE 'org.apache.hadoop.hive.hbase.HBaseSerDe' STORED BY 'org.apache.hadoop.hive.hbase.HBaseStorageHandler' WITH SERDEPROPERTIES ( 'serialization.format'='1', 'hbase.columns.mapping'=':key,srv:name,srv:pdt,srv:service,srv:term,srv:update') LOCATION 'hdfs://nameservice1/user/hive/warehouse/nba_twitter.db/hbase' TBLPROPERTIES ( 'hbase.table.name'='NBATwitter', 'transient_lastDdlTime'='1386172188') Trying to create a table using the above fails: [hive]$ hive -f /tmp/test_create.sql cli -f /tmp/test_create.sql Logging initialized using configuration in jar:file:/opt/cloudera/parcels/CDH-4.4.0-1.cdh4.4.0.p0.39/lib/hive/lib/hive-common-0.10.0-cdh4.4.0.jar!/hive-log4j.properties FAILED: Error in metadata: MetaException(message:LOCATION may not be specified for HBase.) FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.DDLTask However, if I remove the LOCATION, then the DDL is valid. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HIVE-6853) show create table for hbase tables should exclude LOCATION
[ https://issues.apache.org/jira/browse/HIVE-6853?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Miklos Christine updated HIVE-6853: --- Status: Patch Available (was: Open) Attaching a patch that checks if the table uses the HBaseStorageHandler and removes the LOCATION field in the show create table method. show create table for hbase tables should exclude LOCATION --- Key: HIVE-6853 URL: https://issues.apache.org/jira/browse/HIVE-6853 Project: Hive Issue Type: Bug Components: StorageHandler Affects Versions: 0.10.0 Reporter: Miklos Christine Attachments: HIVE-6853-0.patch If you create a table on top of hbase in hive and issue a show create table hbase_table, it gives a bad DDL. It should not show LOCATION: [hive]$ cat /tmp/test_create.sql CREATE EXTERNAL TABLE nba_twitter.hbase2( key string COMMENT 'from deserializer', name string COMMENT 'from deserializer', pdt string COMMENT 'from deserializer', service string COMMENT 'from deserializer', term string COMMENT 'from deserializer', update1 string COMMENT 'from deserializer') ROW FORMAT SERDE 'org.apache.hadoop.hive.hbase.HBaseSerDe' STORED BY 'org.apache.hadoop.hive.hbase.HBaseStorageHandler' WITH SERDEPROPERTIES ( 'serialization.format'='1', 'hbase.columns.mapping'=':key,srv:name,srv:pdt,srv:service,srv:term,srv:update') LOCATION 'hdfs://nameservice1/user/hive/warehouse/nba_twitter.db/hbase' TBLPROPERTIES ( 'hbase.table.name'='NBATwitter', 'transient_lastDdlTime'='1386172188') Trying to create a table using the above fails: [hive]$ hive -f /tmp/test_create.sql cli -f /tmp/test_create.sql Logging initialized using configuration in jar:file:/opt/cloudera/parcels/CDH-4.4.0-1.cdh4.4.0.p0.39/lib/hive/lib/hive-common-0.10.0-cdh4.4.0.jar!/hive-log4j.properties FAILED: Error in metadata: MetaException(message:LOCATION may not be specified for HBase.) FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.DDLTask However, if I remove the LOCATION, then the DDL is valid. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HIVE-6843) INSTR for UTF-8 returns incorrect position
[
https://issues.apache.org/jira/browse/HIVE-6843?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13961957#comment-13961957
]
Clif Kranish commented on HIVE-6843:
Sorry, copy/paste got me. They _look_ the same. And sorry about the curly
quotes, I don't know where they came from.
The real issue is that for UTF-8 INSTR returns the position in bytes instead of
characters. So this reutrns a 9 where by my count it should be a 5. Thank you
for your support.
select INSTR ('НАСТРОЕние', 'Р') from
INSTR for UTF-8 returns incorrect position
--
Key: HIVE-6843
URL: https://issues.apache.org/jira/browse/HIVE-6843
Project: Hive
Issue Type: Bug
Components: UDF
Affects Versions: 0.11.0, 0.12.0
Reporter: Clif Kranish
Assignee: Szehon Ho
Priority: Minor
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Created] (HIVE-6854) Add unit test for Reimport use case
Harish Butani created HIVE-6854: --- Summary: Add unit test for Reimport use case Key: HIVE-6854 URL: https://issues.apache.org/jira/browse/HIVE-6854 Project: Hive Issue Type: Bug Reporter: Harish Butani AS a followup to HIVE-6848 -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HIVE-6848) Importing into an existing table fails
[ https://issues.apache.org/jira/browse/HIVE-6848?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13962030#comment-13962030 ] Harish Butani commented on HIVE-6848: - Sure, added a new jira for this HIVE-6854 Importing into an existing table fails -- Key: HIVE-6848 URL: https://issues.apache.org/jira/browse/HIVE-6848 Project: Hive Issue Type: Bug Components: Query Processor Reporter: Arpit Gupta Assignee: Harish Butani Fix For: 0.13.0 Attachments: HIVE-6848.1.patch This is because ImportSemanticAnalyzer:checkTable doesn't account for the renaming of OutputFormat class and the setting of a default value for Serialization.Format -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Created] (HIVE-6855) A couple of errors in MySQL db creation script for transaction tables
Alan Gates created HIVE-6855: Summary: A couple of errors in MySQL db creation script for transaction tables Key: HIVE-6855 URL: https://issues.apache.org/jira/browse/HIVE-6855 Project: Hive Issue Type: Bug Components: Metastore Affects Versions: 0.13.0 Reporter: Alan Gates Assignee: Alan Gates There are a few small issues in the database creation scripts for mysql. A couple of the tables don't set the engine to InnoDB. None of the tables set default character set to latin1. And the syntax CREATE INDEX...USING HASH doesn't work on older versions of MySQL. Instead the index creation should be done without specifying a method (no USING clause). -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HIVE-6855) A couple of errors in MySQL db creation script for transaction tables
[ https://issues.apache.org/jira/browse/HIVE-6855?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alan Gates updated HIVE-6855: - Status: Patch Available (was: Open) NO PRECOMMIT TESTS Updated the mysql scripts. A couple of errors in MySQL db creation script for transaction tables - Key: HIVE-6855 URL: https://issues.apache.org/jira/browse/HIVE-6855 Project: Hive Issue Type: Bug Components: Metastore Affects Versions: 0.13.0 Reporter: Alan Gates Assignee: Alan Gates Attachments: HIVE-6855.patch There are a few small issues in the database creation scripts for mysql. A couple of the tables don't set the engine to InnoDB. None of the tables set default character set to latin1. And the syntax CREATE INDEX...USING HASH doesn't work on older versions of MySQL. Instead the index creation should be done without specifying a method (no USING clause). -- This message was sent by Atlassian JIRA (v6.2#6252)
Re: Review Request 20051: HIVE-4904: A little more CP crossing RS boundaries
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/20051/#review39706 --- I took a look at this change; my knowledge of hive code is rather limited. 1. Column Pruner doesn't cross Script operator boundary. Theoretically you could prune above and below the script op separately. 2. It seems column pruner assumes that parent of UDTF is always select; but we haven't formalized this assumption. Other processors should throw exception if it ever come across a child that is UDTF. Theoretically you can push down certain filters below builtin UDTF. We may not be doing that today. 3. In Select Pruner it seems like there is no difference between 'prunedCols' and 'columns'. - John Pullokkaran On April 6, 2014, 1:33 a.m., Harish Butani wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/20051/ --- (Updated April 6, 2014, 1:33 a.m.) Review request for hive, Ashutosh Chauhan and Navis Ryu. Bugs: HIVE-4904 https://issues.apache.org/jira/browse/HIVE-4904 Repository: hive-git Description --- Currently, CP context cannot be propagated over RS except for JOIN/EXT. A little more CP is possible. Diffs - ql/src/java/org/apache/hadoop/hive/ql/optimizer/ColumnPruner.java 58a9b59 ql/src/java/org/apache/hadoop/hive/ql/optimizer/ColumnPrunerProcCtx.java db36151 ql/src/java/org/apache/hadoop/hive/ql/optimizer/ColumnPrunerProcFactory.java 0690fb7 ql/src/java/org/apache/hadoop/hive/ql/optimizer/Optimizer.java 3f16dc2 ql/src/java/org/apache/hadoop/hive/ql/optimizer/correlation/CorrelationUtilities.java 94224b3 ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java 3b33dc2 ql/src/test/queries/clientpositive/order_within_subquery.q PRE-CREATION ql/src/test/results/clientpositive/annotate_stats_select.q.out 1e982e6 ql/src/test/results/clientpositive/auto_join18.q.out b8677f4 ql/src/test/results/clientpositive/auto_join27.q.out a576190 ql/src/test/results/clientpositive/auto_join30.q.out 8709198 ql/src/test/results/clientpositive/auto_join31.q.out 1936e45 ql/src/test/results/clientpositive/auto_join32.q.out 05f53e6 ql/src/test/results/clientpositive/auto_sortmerge_join_10.q.out 8882aac ql/src/test/results/clientpositive/count.q.out eb048b6 ql/src/test/results/clientpositive/distinct_stats.q.out f715ea3 ql/src/test/results/clientpositive/groupby2_map.q.out 291f196 ql/src/test/results/clientpositive/groupby2_map_skew.q.out d005b6c ql/src/test/results/clientpositive/groupby3_map.q.out 1dfee08 ql/src/test/results/clientpositive/groupby3_map_skew.q.out 7af59bc ql/src/test/results/clientpositive/groupby_cube1.q.out 92d81f4 ql/src/test/results/clientpositive/groupby_distinct_samekey.q.out b405978 ql/src/test/results/clientpositive/groupby_map_ppr.q.out 27eff75 ql/src/test/results/clientpositive/groupby_multi_insert_common_distinct.q.out ad76252 ql/src/test/results/clientpositive/groupby_multi_single_reducer3.q.out 51a70c4 ql/src/test/results/clientpositive/groupby_position.q.out 727bccb ql/src/test/results/clientpositive/groupby_rollup1.q.out 36bf966 ql/src/test/results/clientpositive/groupby_sort_11.q.out 8ee7571 ql/src/test/results/clientpositive/groupby_sort_8.q.out a27cfaa ql/src/test/results/clientpositive/join18.q.out 7975c79 ql/src/test/results/clientpositive/limit_pushdown.q.out 9c93ada ql/src/test/results/clientpositive/limit_pushdown_negative.q.out 115b171 ql/src/test/results/clientpositive/metadataonly1.q.out 917efdf ql/src/test/results/clientpositive/multi_insert_gby2.q.out ab758cb ql/src/test/results/clientpositive/multi_insert_gby3.q.out 23ccebb ql/src/test/results/clientpositive/multi_insert_lateral_view.q.out 35e70b4 ql/src/test/results/clientpositive/nullgroup.q.out 2ac7dea ql/src/test/results/clientpositive/nullgroup2.q.out cf31dc1 ql/src/test/results/clientpositive/nullgroup4.q.out feae138 ql/src/test/results/clientpositive/nullgroup4_multi_distinct.q.out 2ee357f ql/src/test/results/clientpositive/order_within_subquery.q.out PRE-CREATION ql/src/test/results/clientpositive/reduce_deduplicate_extended.q.out 9c6d14e ql/src/test/results/clientpositive/udf_count.q.out fb45708 ql/src/test/results/clientpositive/union11.q.out f226f35 ql/src/test/results/clientpositive/union14.q.out a6d349b ql/src/test/results/clientpositive/union15.q.out 88c9553 ql/src/test/results/clientpositive/union16.q.out 2bd8d5e ql/src/test/results/clientpositive/union2.q.out 0fac9d9 ql/src/test/results/clientpositive/union25.q.out 1ebe682
[jira] [Updated] (HIVE-6855) A couple of errors in MySQL db creation script for transaction tables
[ https://issues.apache.org/jira/browse/HIVE-6855?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alan Gates updated HIVE-6855: - Attachment: HIVE-6855.patch A couple of errors in MySQL db creation script for transaction tables - Key: HIVE-6855 URL: https://issues.apache.org/jira/browse/HIVE-6855 Project: Hive Issue Type: Bug Components: Metastore Affects Versions: 0.13.0 Reporter: Alan Gates Assignee: Alan Gates Attachments: HIVE-6855.patch There are a few small issues in the database creation scripts for mysql. A couple of the tables don't set the engine to InnoDB. None of the tables set default character set to latin1. And the syntax CREATE INDEX...USING HASH doesn't work on older versions of MySQL. Instead the index creation should be done without specifying a method (no USING clause). -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HIVE-6855) A couple of errors in MySQL db creation script for transaction tables
[ https://issues.apache.org/jira/browse/HIVE-6855?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13962077#comment-13962077 ] Ashutosh Chauhan commented on HIVE-6855: Seems like we also need to update hive-schema-0.14.0.mysql.sql A couple of errors in MySQL db creation script for transaction tables - Key: HIVE-6855 URL: https://issues.apache.org/jira/browse/HIVE-6855 Project: Hive Issue Type: Bug Components: Metastore Affects Versions: 0.13.0 Reporter: Alan Gates Assignee: Alan Gates Attachments: HIVE-6855.patch There are a few small issues in the database creation scripts for mysql. A couple of the tables don't set the engine to InnoDB. None of the tables set default character set to latin1. And the syntax CREATE INDEX...USING HASH doesn't work on older versions of MySQL. Instead the index creation should be done without specifying a method (no USING clause). -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HIVE-6739) Hive HBase query fails on Tez due to missing jars and then due to NPE in getSplits
[ https://issues.apache.org/jira/browse/HIVE-6739?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HIVE-6739: --- Attachment: HIVE-6739.04.patch The previous patch was incomplete, updating. The tests that failed on 02 pass locally for me, which stands to reason as only the Tez path is changed here. This patch does not need Tez 0.4 Hive HBase query fails on Tez due to missing jars and then due to NPE in getSplits -- Key: HIVE-6739 URL: https://issues.apache.org/jira/browse/HIVE-6739 Project: Hive Issue Type: Bug Reporter: Sergey Shelukhin Assignee: Sergey Shelukhin Fix For: 0.13.0 Attachments: HIVE-6739.01.patch, HIVE-6739.02.patch, HIVE-6739.03.patch, HIVE-6739.04.patch, HIVE-6739.patch, HIVE-6739.preliminary.patch Tez paths in Hive never call configure on the input/output operators, so (among other things, potentially) requisite files never get added to the job -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HIVE-6739) Hive HBase query fails on Tez due to missing jars and then due to NPE in getSplits
[ https://issues.apache.org/jira/browse/HIVE-6739?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13962087#comment-13962087 ] Sergey Shelukhin commented on HIVE-6739: Will commit later today Hive HBase query fails on Tez due to missing jars and then due to NPE in getSplits -- Key: HIVE-6739 URL: https://issues.apache.org/jira/browse/HIVE-6739 Project: Hive Issue Type: Bug Reporter: Sergey Shelukhin Assignee: Sergey Shelukhin Fix For: 0.13.0 Attachments: HIVE-6739.01.patch, HIVE-6739.02.patch, HIVE-6739.03.patch, HIVE-6739.04.patch, HIVE-6739.patch, HIVE-6739.preliminary.patch Tez paths in Hive never call configure on the input/output operators, so (among other things, potentially) requisite files never get added to the job -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HIVE-6846) allow safe set commands with sql standard authorization
[ https://issues.apache.org/jira/browse/HIVE-6846?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thejas M Nair updated HIVE-6846: Attachment: HIVE-6846.2.patch Fixing tests failures, added another jdbc test. allow safe set commands with sql standard authorization --- Key: HIVE-6846 URL: https://issues.apache.org/jira/browse/HIVE-6846 Project: Hive Issue Type: Bug Components: Authorization Affects Versions: 0.13.0 Reporter: Thejas M Nair Assignee: Thejas M Nair Fix For: 0.13.0 Attachments: HIVE-6846.1.patch, HIVE-6846.2.patch HIVE-6827 disables all set commands when SQL standard authorization is turned on, but not all set commands are unsafe. We should allow safe set commands. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HIVE-6841) Vectorized execution throws NPE for partitioning columns with __HIVE_DEFAULT_PARTITION__
[ https://issues.apache.org/jira/browse/HIVE-6841?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13962108#comment-13962108 ] Jitendra Nath Pandey commented on HIVE-6841: [~rhbutani] This is a critical issue in hive-0.13 and fails many queries on partitioned tables in vectorized execution. It should be fixed in branch-0.13 as well. Vectorized execution throws NPE for partitioning columns with __HIVE_DEFAULT_PARTITION__ Key: HIVE-6841 URL: https://issues.apache.org/jira/browse/HIVE-6841 Project: Hive Issue Type: Bug Affects Versions: 0.13.0 Reporter: Jitendra Nath Pandey Assignee: Jitendra Nath Pandey Priority: Critical Attachments: HIVE-6841.1.patch, HIVE-6841.2.patch, HIVE-6841.3.patch If partitioning columns have __HIVE_DEFAULT_PARTITION__ or null, vectorized execution throws NPE. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HIVE-6855) A couple of errors in MySQL db creation script for transaction tables
[ https://issues.apache.org/jira/browse/HIVE-6855?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13962113#comment-13962113 ] Alan Gates commented on HIVE-6855: -- The transaction tables aren't currently in hive-schema-0.14.0.mysql.sql because we wanted to figure out a better method than adding them by hand for 0.14. A couple of errors in MySQL db creation script for transaction tables - Key: HIVE-6855 URL: https://issues.apache.org/jira/browse/HIVE-6855 Project: Hive Issue Type: Bug Components: Metastore Affects Versions: 0.13.0 Reporter: Alan Gates Assignee: Alan Gates Attachments: HIVE-6855.patch There are a few small issues in the database creation scripts for mysql. A couple of the tables don't set the engine to InnoDB. None of the tables set default character set to latin1. And the syntax CREATE INDEX...USING HASH doesn't work on older versions of MySQL. Instead the index creation should be done without specifying a method (no USING clause). -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HIVE-6846) allow safe set commands with sql standard authorization
[ https://issues.apache.org/jira/browse/HIVE-6846?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13962115#comment-13962115 ] Lefty Leverenz commented on HIVE-6846: -- What documentation does this need? allow safe set commands with sql standard authorization --- Key: HIVE-6846 URL: https://issues.apache.org/jira/browse/HIVE-6846 Project: Hive Issue Type: Bug Components: Authorization Affects Versions: 0.13.0 Reporter: Thejas M Nair Assignee: Thejas M Nair Fix For: 0.13.0 Attachments: HIVE-6846.1.patch, HIVE-6846.2.patch HIVE-6827 disables all set commands when SQL standard authorization is turned on, but not all set commands are unsafe. We should allow safe set commands. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HIVE-6841) Vectorized execution throws NPE for partitioning columns with __HIVE_DEFAULT_PARTITION__
[ https://issues.apache.org/jira/browse/HIVE-6841?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13962126#comment-13962126 ] Harish Butani commented on HIVE-6841: - +1 for 0.13 Vectorized execution throws NPE for partitioning columns with __HIVE_DEFAULT_PARTITION__ Key: HIVE-6841 URL: https://issues.apache.org/jira/browse/HIVE-6841 Project: Hive Issue Type: Bug Affects Versions: 0.13.0 Reporter: Jitendra Nath Pandey Assignee: Jitendra Nath Pandey Priority: Critical Attachments: HIVE-6841.1.patch, HIVE-6841.2.patch, HIVE-6841.3.patch If partitioning columns have __HIVE_DEFAULT_PARTITION__ or null, vectorized execution throws NPE. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HIVE-6841) Vectorized execution throws NPE for partitioning columns with __HIVE_DEFAULT_PARTITION__
[ https://issues.apache.org/jira/browse/HIVE-6841?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jitendra Nath Pandey updated HIVE-6841: --- Resolution: Fixed Fix Version/s: 0.13.0 Status: Resolved (was: Patch Available) The failed tests are not related to the patch and passed when run locally. Committed to trunk and branch-0.13. Vectorized execution throws NPE for partitioning columns with __HIVE_DEFAULT_PARTITION__ Key: HIVE-6841 URL: https://issues.apache.org/jira/browse/HIVE-6841 Project: Hive Issue Type: Bug Affects Versions: 0.13.0 Reporter: Jitendra Nath Pandey Assignee: Jitendra Nath Pandey Priority: Critical Fix For: 0.13.0 Attachments: HIVE-6841.1.patch, HIVE-6841.2.patch, HIVE-6841.3.patch If partitioning columns have __HIVE_DEFAULT_PARTITION__ or null, vectorized execution throws NPE. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HIVE-1996) LOAD DATA INPATH fails when the table already contains a file of the same name
[
https://issues.apache.org/jira/browse/HIVE-1996?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chinna Rao Lalam updated HIVE-1996:
---
Resolution: Duplicate
Status: Resolved (was: Patch Available)
This issue is solved as part of HIVE-3300
LOAD DATA INPATH fails when the table already contains a file of the same
name
Key: HIVE-1996
URL: https://issues.apache.org/jira/browse/HIVE-1996
Project: Hive
Issue Type: Bug
Affects Versions: 0.7.0, 0.8.1
Reporter: Kirk True
Assignee: Chinna Rao Lalam
Attachments: HIVE-1996.1.Patch, HIVE-1996.2.Patch, HIVE-1996.Patch
Steps:
1. From the command line copy the kv2.txt data file into the current user's
HDFS directory:
{{$ hadoop fs -copyFromLocal /path/to/hive/sources/data/files/kv2.txt
kv2.txt}}
2. In Hive, create the table:
{{create table tst_src1 (key_ int, value_ string);}}
3. Load the data into the table from HDFS:
{{load data inpath './kv2.txt' into table tst_src1;}}
4. Repeat step 1
5. Repeat step 3
Expected:
To have kv2.txt renamed in HDFS and then copied to the destination as per
HIVE-307.
Actual:
File is renamed, but {{Hive.copyFiles}} doesn't see the change in {{srcs}}
as it continues to use the same array elements (with the un-renamed, old file
names). It crashes with this error:
{noformat}
java.lang.NullPointerException
at org.apache.hadoop.hive.ql.metadata.Hive.copyFiles(Hive.java:1725)
at org.apache.hadoop.hive.ql.metadata.Table.copyFiles(Table.java:541)
at org.apache.hadoop.hive.ql.metadata.Hive.loadTable(Hive.java:1173)
at org.apache.hadoop.hive.ql.exec.MoveTask.execute(MoveTask.java:197)
at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:130)
at
org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:57)
at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:1060)
at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:897)
at org.apache.hadoop.hive.ql.Driver.run(Driver.java:745)
at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:164)
at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:241)
at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:456)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.util.RunJar.main(RunJar.java:156)
{noformat}
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Created] (HIVE-6856) ddl commands fail with permissions issue when running using webhcat in secure Tez cluster
Eugene Koifman created HIVE-6856: Summary: ddl commands fail with permissions issue when running using webhcat in secure Tez cluster Key: HIVE-6856 URL: https://issues.apache.org/jira/browse/HIVE-6856 Project: Hive Issue Type: Bug Components: WebHCat Affects Versions: 0.13.0 Reporter: Eugene Koifman Assignee: Eugene Koifman curl -u : --negotiate -d exec=show tables; -X POST http://server:50111/templeton/v1/ddl results in (when Tez is enabled in Secure cluster) Exception in thread main java.lang.RuntimeException: org.apache.hadoop.security.AccessControlException: Permission denied: user=hrt_qa, access=WRITE, inode=/user/hcat:hcat:hcat:drwxr-xr-x at org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.checkFsPermission(FSPermissionChecker.java:265) at org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.check(FSPermissionChecker.java:251) at org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.check(FSPermissionChecker.java:232) at org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.checkPermission(FSPermissionChecker.java:176) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkPermission(FSNamesystem.java:5497) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkPermission(FSNamesystem.java:5479) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkAncestorAccess(FSNamesystem.java:5453) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.mkdirsInternal(FSNamesystem.java:3596) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.mkdirsInt(FSNamesystem.java:3566) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.mkdirs(FSNamesystem.java:3540) at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.mkdirs(NameNodeRpcServer.java:754) at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.mkdirs(ClientNamenodeProtocolServerSideTranslatorPB.java:558) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:585) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:928) at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2013) at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2009) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:396) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1548) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2007) at org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:354) at org.apache.hive.hcatalog.cli.HCatCli.main(HCatCli.java:138) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.apache.hadoop.util.RunJar.main(RunJar.java:212) Caused by: org.apache.hadoop.security.AccessControlException: Permission denied: user=hrt_qa, access=WRITE, inode=/user/hcat:hcat:hcat:drwxr-xr-x at org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.checkFsPermission(FSPermissionChecker.java:265) at org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.check(FSPermissionChecker.java:251) at org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.check(FSPermissionChecker.java:232) at org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.checkPermission(FSPermissionChecker.java:176) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkPermission(FSNamesystem.java:5497) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkPermission(FSNamesystem.java:5479) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkAncestorAccess(FSNamesystem.java:5453) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.mkdirsInternal(FSNamesystem.java:3596) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.mkdirsInt(FSNamesystem.java:3566) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.mkdirs(FSNamesystem.java:3540) at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.mkdirs(NameNodeRpcServer.java:754) at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.mkdirs(ClientNamenodeProtocolServerSideTranslatorPB.java:558) at
[jira] [Updated] (HIVE-6856) ddl commands fail with permissions issue when running using webhcat in secure Tez cluster
[
https://issues.apache.org/jira/browse/HIVE-6856?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Eugene Koifman updated HIVE-6856:
-
Description:
curl -u : --negotiate -d exec=show tables; -X POST
http://server:50111/templeton/v1/ddl
results in (when Tez is enabled in Secure cluster)
{noformat}
Exception in thread main java.lang.RuntimeException:
org.apache.hadoop.security.AccessControlException: Permission denied:
user=hrt_qa, access=WRITE, inode=/user/hcat:hcat:hcat:drwxr-xr-x
at
org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.checkFsPermission(FSPermissionChecker.java:265)
at
org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.check(FSPermissionChecker.java:251)
at
org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.check(FSPermissionChecker.java:232)
at
org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.checkPermission(FSPermissionChecker.java:176)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkPermission(FSNamesystem.java:5497)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkPermission(FSNamesystem.java:5479)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkAncestorAccess(FSNamesystem.java:5453)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.mkdirsInternal(FSNamesystem.java:3596)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.mkdirsInt(FSNamesystem.java:3566)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.mkdirs(FSNamesystem.java:3540)
at
org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.mkdirs(NameNodeRpcServer.java:754)
at
org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.mkdirs(ClientNamenodeProtocolServerSideTranslatorPB.java:558)
at
org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
at
org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:585)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:928)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2013)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2009)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1548)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2007)
at
org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:354)
at org.apache.hive.hcatalog.cli.HCatCli.main(HCatCli.java:138)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.util.RunJar.main(RunJar.java:212)
Caused by: org.apache.hadoop.security.AccessControlException: Permission
denied: user=hrt_qa, access=WRITE, inode=/user/hcat:hcat:hcat:drwxr-xr-x
at
org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.checkFsPermission(FSPermissionChecker.java:265)
at
org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.check(FSPermissionChecker.java:251)
at
org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.check(FSPermissionChecker.java:232)
at
org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.checkPermission(FSPermissionChecker.java:176)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkPermission(FSNamesystem.java:5497)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkPermission(FSNamesystem.java:5479)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkAncestorAccess(FSNamesystem.java:5453)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.mkdirsInternal(FSNamesystem.java:3596)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.mkdirsInt(FSNamesystem.java:3566)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.mkdirs(FSNamesystem.java:3540)
at
org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.mkdirs(NameNodeRpcServer.java:754)
at
org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.mkdirs(ClientNamenodeProtocolServerSideTranslatorPB.java:558)
at
org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
at
org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:585)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:928)
[jira] [Resolved] (HIVE-2016) alter partition should throw exception if the specified partition does not exist.
[
https://issues.apache.org/jira/browse/HIVE-2016?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chinna Rao Lalam resolved HIVE-2016.
Resolution: Implemented
In trunk It is implemented
alter partition should throw exception if the specified partition does not
exist.
--
Key: HIVE-2016
URL: https://issues.apache.org/jira/browse/HIVE-2016
Project: Hive
Issue Type: Bug
Components: Metastore
Affects Versions: 0.8.0
Environment: Hadoop 0.20.1, hive-0.8.0-SNAPSHOT and SUSE Linux
Enterprise Server 10 SP2 (i586) - Kernel 2.6.16.60-0.21-smp (5).
Reporter: Chinna Rao Lalam
Assignee: Chinna Rao Lalam
To reproduce the issue follow the below steps
{noformat}
set hive.exec.drop.ignorenonexistent=false;
create table page_test(view INT, userid INT, page_url STRING) PARTITIONED
BY(dt STRING, country STRING) STORED AS A TEXTFILE;
LOAD DATA LOCAL INPATH '/home/test.txt' OVERWRITE INTO TABLE page_test
PARTITION(dt='10-10-2010',country='US');
LOAD DATA LOCAL INPATH '/home/test.txt' OVERWRITE INTO TABLE page_test
PARTITION(dt='10-12-2010',country='IN');
{noformat}
{noformat}
ALTER TABLE page_test DROP PARTITION (dt='23-02-2010',country='UK');
{noformat}
This query should throw exception because the requested partition doesn't
exist
This issue related to HIVE-1535
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HIVE-6837) HiveServer2 thrift/http mode binary mode proxy user check fails reporting IP null for client
[
https://issues.apache.org/jira/browse/HIVE-6837?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13962151#comment-13962151
]
Vaibhav Gumashta commented on HIVE-6837:
[~thejas] Thanks for taking a look.
Sure, I'll do that. There's another issue that I noticed caused in
SessionManager#openSession as a result of this:
{code}
public SessionHandle openSession(TProtocolVersion protocol, String username,
String password,
MapString, String sessionConf, boolean withImpersonation, String
delegationToken)
throws HiveSQLException {
HiveSession session;
if (withImpersonation) {
HiveSessionImplwithUGI hiveSessionUgi = new
HiveSessionImplwithUGI(protocol, username, password,
hiveConf, sessionConf, TSetIpAddressProcessor.getUserIpAddress(),
delegationToken);
session = HiveSessionProxy.getProxy(hiveSessionUgi,
hiveSessionUgi.getSessionUgi());
hiveSessionUgi.setProxySession(session);
} else {
session = new HiveSessionImpl(protocol, username, password, hiveConf,
sessionConf,
TSetIpAddressProcessor.getUserIpAddress());
}
session.setSessionManager(this);
session.setOperationManager(operationManager);
session.open();
handleToSession.put(session.getSessionHandle(), session);
try {
executeSessionHooks(session);
} catch (Exception e) {
throw new HiveSQLException(Failed to execute session hooks, e);
}
return session.getSessionHandle();
}
{code}
Notice that if withImpersonation is set to true, we're using
TSetIpAddressProcessor.getUserIpAddress() to get the IP address which is wrong
for a kerberized setup (should use HiveAuthFactory#getIpAddress).
Also, in case of a kerberized setup, we're wrapping the transport in a doAs
(with UGI of the HiveServer2 process) which doesn't make sense to me:
https://github.com/apache/hive/blob/trunk/shims/common-secure/src/main/java/org/apache/hadoop/hive/thrift/HadoopThriftAuthBridge20S.java#L335.
HiveServer2 thrift/http mode binary mode proxy user check fails reporting
IP null for client
--
Key: HIVE-6837
URL: https://issues.apache.org/jira/browse/HIVE-6837
Project: Hive
Issue Type: Bug
Components: HiveServer2
Affects Versions: 0.13.0
Reporter: Dilli Arumugam
Assignee: Vaibhav Gumashta
Fix For: 0.13.0
Attachments: HIVE-6837.1.patch, HIVE-6837.2.patch, HIVE-6837.3.patch,
hive.log
Hive Server running thrift/http with Kerberos security.
Kinited user knox attempting to proxy as sam.
Beeline connection failed reporting error on hive server logs:
Caused by: org.apache.hadoop.security.authorize.AuthorizationException:
Unauthorized connection for super-user: knox from IP null
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Created] (HIVE-6857) Consolidate HiveServer2 threadlocals
Vaibhav Gumashta created HIVE-6857: -- Summary: Consolidate HiveServer2 threadlocals Key: HIVE-6857 URL: https://issues.apache.org/jira/browse/HIVE-6857 Project: Hive Issue Type: Bug Components: HiveServer2 Reporter: Vaibhav Gumashta Assignee: Vaibhav Gumashta Check the discussion here: HIVE-6837 -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HIVE-6856) ddl commands fail with permissions issue when running using webhcat in secure Tez cluster
[
https://issues.apache.org/jira/browse/HIVE-6856?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Eugene Koifman updated HIVE-6856:
-
Description:
curl -u : --negotiate -d exec=show tables; -X POST
http://server:50111/templeton/v1/ddl
results in (when Tez is enabled in Secure cluster)
{noformat}
Exception in thread main java.lang.RuntimeException:
org.apache.hadoop.security.AccessControlException: Permission denied:
user=hrt_qa, access=WRITE, inode=/user/hcat:hcat:hcat:drwxr-xr-x
at
org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.checkFsPermission(FSPermissionChecker.java:265)
at
org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.check(FSPermissionChecker.java:251)
at
org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.check(FSPermissionChecker.java:232)
at
org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.checkPermission(FSPermissionChecker.java:176)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkPermission(FSNamesystem.java:5497)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkPermission(FSNamesystem.java:5479)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkAncestorAccess(FSNamesystem.java:5453)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.mkdirsInternal(FSNamesystem.java:3596)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.mkdirsInt(FSNamesystem.java:3566)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.mkdirs(FSNamesystem.java:3540)
at
org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.mkdirs(NameNodeRpcServer.java:754)
at
org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.mkdirs(ClientNamenodeProtocolServerSideTranslatorPB.java:558)
at
org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
at
org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:585)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:928)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2013)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2009)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1548)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2007)
at
org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:354)
at org.apache.hive.hcatalog.cli.HCatCli.main(HCatCli.java:138)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.util.RunJar.main(RunJar.java:212)
Caused by: org.apache.hadoop.security.AccessControlException: Permission
denied: user=hrt_qa, access=WRITE, inode=/user/hcat:hcat:hcat:drwxr-xr-x
at
org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.checkFsPermission(FSPermissionChecker.java:265)
at
org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.check(FSPermissionChecker.java:251)
at
org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.check(FSPermissionChecker.java:232)
at
org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.checkPermission(FSPermissionChecker.java:176)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkPermission(FSNamesystem.java:5497)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkPermission(FSNamesystem.java:5479)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkAncestorAccess(FSNamesystem.java:5453)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.mkdirsInternal(FSNamesystem.java:3596)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.mkdirsInt(FSNamesystem.java:3566)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.mkdirs(FSNamesystem.java:3540)
at
org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.mkdirs(NameNodeRpcServer.java:754)
at
org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.mkdirs(ClientNamenodeProtocolServerSideTranslatorPB.java:558)
at
org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
at
org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:585)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:928)
[jira] [Updated] (HIVE-6856) ddl commands fail with permissions issue when running using webhcat in secure Tez cluster
[
https://issues.apache.org/jira/browse/HIVE-6856?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Eugene Koifman updated HIVE-6856:
-
Attachment: HIVE-6856.patch
ddl commands fail with permissions issue when running using webhcat in secure
Tez cluster
-
Key: HIVE-6856
URL: https://issues.apache.org/jira/browse/HIVE-6856
Project: Hive
Issue Type: Bug
Components: WebHCat
Affects Versions: 0.13.0
Reporter: Eugene Koifman
Assignee: Eugene Koifman
Attachments: HIVE-6856.patch
curl -u : --negotiate -d exec=show tables; -X POST
http://server:50111/templeton/v1/ddl
results in (when Tez is enabled in Secure cluster)
{noformat}
Exception in thread main java.lang.RuntimeException:
org.apache.hadoop.security.AccessControlException: Permission denied:
user=hrt_qa, access=WRITE, inode=/user/hcat:hcat:hcat:drwxr-xr-x
at
org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.checkFsPermission(FSPermissionChecker.java:265)
at
org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.check(FSPermissionChecker.java:251)
at
org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.check(FSPermissionChecker.java:232)
at
org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.checkPermission(FSPermissionChecker.java:176)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkPermission(FSNamesystem.java:5497)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkPermission(FSNamesystem.java:5479)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkAncestorAccess(FSNamesystem.java:5453)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.mkdirsInternal(FSNamesystem.java:3596)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.mkdirsInt(FSNamesystem.java:3566)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.mkdirs(FSNamesystem.java:3540)
at
org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.mkdirs(NameNodeRpcServer.java:754)
at
org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.mkdirs(ClientNamenodeProtocolServerSideTranslatorPB.java:558)
at
org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
at
org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:585)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:928)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2013)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2009)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1548)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2007)
at
org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:354)
at org.apache.hive.hcatalog.cli.HCatCli.main(HCatCli.java:138)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.util.RunJar.main(RunJar.java:212)
Caused by: org.apache.hadoop.security.AccessControlException: Permission
denied: user=hrt_qa, access=WRITE, inode=/user/hcat:hcat:hcat:drwxr-xr-x
at
org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.checkFsPermission(FSPermissionChecker.java:265)
at
org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.check(FSPermissionChecker.java:251)
at
org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.check(FSPermissionChecker.java:232)
at
org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.checkPermission(FSPermissionChecker.java:176)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkPermission(FSNamesystem.java:5497)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkPermission(FSNamesystem.java:5479)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkAncestorAccess(FSNamesystem.java:5453)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.mkdirsInternal(FSNamesystem.java:3596)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.mkdirsInt(FSNamesystem.java:3566)
at
[jira] [Updated] (HIVE-6856) ddl commands fail with permissions issue when running using webhcat in secure Tez cluster
[
https://issues.apache.org/jira/browse/HIVE-6856?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Eugene Koifman updated HIVE-6856:
-
Status: Patch Available (was: Open)
ddl commands fail with permissions issue when running using webhcat in secure
Tez cluster
-
Key: HIVE-6856
URL: https://issues.apache.org/jira/browse/HIVE-6856
Project: Hive
Issue Type: Bug
Components: WebHCat
Affects Versions: 0.13.0
Reporter: Eugene Koifman
Assignee: Eugene Koifman
Attachments: HIVE-6856.patch
curl -u : --negotiate -d exec=show tables; -X POST
http://server:50111/templeton/v1/ddl
results in (when Tez is enabled in Secure cluster)
{noformat}
Exception in thread main java.lang.RuntimeException:
org.apache.hadoop.security.AccessControlException: Permission denied:
user=hrt_qa, access=WRITE, inode=/user/hcat:hcat:hcat:drwxr-xr-x
at
org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.checkFsPermission(FSPermissionChecker.java:265)
at
org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.check(FSPermissionChecker.java:251)
at
org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.check(FSPermissionChecker.java:232)
at
org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.checkPermission(FSPermissionChecker.java:176)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkPermission(FSNamesystem.java:5497)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkPermission(FSNamesystem.java:5479)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkAncestorAccess(FSNamesystem.java:5453)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.mkdirsInternal(FSNamesystem.java:3596)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.mkdirsInt(FSNamesystem.java:3566)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.mkdirs(FSNamesystem.java:3540)
at
org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.mkdirs(NameNodeRpcServer.java:754)
at
org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.mkdirs(ClientNamenodeProtocolServerSideTranslatorPB.java:558)
at
org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
at
org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:585)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:928)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2013)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2009)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1548)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2007)
at
org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:354)
at org.apache.hive.hcatalog.cli.HCatCli.main(HCatCli.java:138)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.util.RunJar.main(RunJar.java:212)
Caused by: org.apache.hadoop.security.AccessControlException: Permission
denied: user=hrt_qa, access=WRITE, inode=/user/hcat:hcat:hcat:drwxr-xr-x
at
org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.checkFsPermission(FSPermissionChecker.java:265)
at
org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.check(FSPermissionChecker.java:251)
at
org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.check(FSPermissionChecker.java:232)
at
org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.checkPermission(FSPermissionChecker.java:176)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkPermission(FSNamesystem.java:5497)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkPermission(FSNamesystem.java:5479)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkAncestorAccess(FSNamesystem.java:5453)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.mkdirsInternal(FSNamesystem.java:3596)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.mkdirsInt(FSNamesystem.java:3566)
at
Review Request 20096: HIVE-6835: Reading of partitioned Avro data fails if partition schema does not match table schema
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/20096/ --- Review request for hive. Repository: hive-git Description --- The problem occurs when you store the avro.schema.(literal|url) in the SERDEPROPERTIES instead of the TBLPROPERTIES, add a partition, change the table's schema, and then try reading from the old partition. I fixed this problem by passing the table properties to the partition with a table. prefix, and changing the Avro SerDe to always use the table properties when available. Diffs - ql/src/java/org/apache/hadoop/hive/ql/plan/PartitionDesc.java 43cef5c ql/src/test/queries/clientpositive/avro_partitioned.q 068a13c ql/src/test/results/clientpositive/avro_partitioned.q.out 352ec0d serde/src/java/org/apache/hadoop/hive/serde2/avro/AvroSerdeUtils.java 9d58d13 serde/src/test/org/apache/hadoop/hive/serde2/avro/TestAvroSerdeUtils.java 67d5570 Diff: https://reviews.apache.org/r/20096/diff/ Testing --- Added test cases Thanks, Anthony Hsu
[jira] [Updated] (HIVE-6835) Reading of partitioned Avro data fails if partition schema does not match table schema
[
https://issues.apache.org/jira/browse/HIVE-6835?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Anthony Hsu updated HIVE-6835:
--
Attachment: HIVE-6835.1.patch
Uploaded a patch with a fix. Review Board link:
https://reviews.apache.org/r/20096/
Reading of partitioned Avro data fails if partition schema does not match
table schema
--
Key: HIVE-6835
URL: https://issues.apache.org/jira/browse/HIVE-6835
Project: Hive
Issue Type: Bug
Affects Versions: 0.12.0
Reporter: Anthony Hsu
Assignee: Anthony Hsu
Attachments: HIVE-6835.1.patch
To reproduce:
{code}
create table testarray (a arraystring);
load data local inpath '/home/ahsu/test/array.txt' into table testarray;
# create partitioned Avro table with one array column
create table avroarray partitioned by (y string) row format serde
'org.apache.hadoop.hive.serde2.avro.AvroSerDe' with serdeproperties
('avro.schema.literal'='{namespace:test,name:avroarray,type:
record, fields: [ { name:a, type:{type:array,items:string}
} ] }') STORED as INPUTFORMAT
'org.apache.hadoop.hive.ql.io.avro.AvroContainerInputFormat' OUTPUTFORMAT
'org.apache.hadoop.hive.ql.io.avro.AvroContainerOutputFormat';
insert into table avroarray partition(y=1) select * from testarray;
# add an int column with a default value of 0
alter table avroarray set serde
'org.apache.hadoop.hive.serde2.avro.AvroSerDe' with
serdeproperties('avro.schema.literal'='{namespace:test,name:avroarray,type:
record, fields: [ {name:intfield,type:int,default:0},{
name:a, type:{type:array,items:string} } ] }');
# fails with ClassCastException
select * from avroarray;
{code}
The select * fails with:
{code}
Failed with exception java.io.IOException:java.lang.ClassCastException:
org.apache.hadoop.hive.serde2.objectinspector.StandardListObjectInspector
cannot be cast to
org.apache.hadoop.hive.serde2.objectinspector.PrimitiveObjectInspector
{code}
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Assigned] (HIVE-6835) Reading of partitioned Avro data fails if partition schema does not match table schema
[
https://issues.apache.org/jira/browse/HIVE-6835?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Anthony Hsu reassigned HIVE-6835:
-
Assignee: Anthony Hsu
Reading of partitioned Avro data fails if partition schema does not match
table schema
--
Key: HIVE-6835
URL: https://issues.apache.org/jira/browse/HIVE-6835
Project: Hive
Issue Type: Bug
Affects Versions: 0.12.0
Reporter: Anthony Hsu
Assignee: Anthony Hsu
Attachments: HIVE-6835.1.patch
To reproduce:
{code}
create table testarray (a arraystring);
load data local inpath '/home/ahsu/test/array.txt' into table testarray;
# create partitioned Avro table with one array column
create table avroarray partitioned by (y string) row format serde
'org.apache.hadoop.hive.serde2.avro.AvroSerDe' with serdeproperties
('avro.schema.literal'='{namespace:test,name:avroarray,type:
record, fields: [ { name:a, type:{type:array,items:string}
} ] }') STORED as INPUTFORMAT
'org.apache.hadoop.hive.ql.io.avro.AvroContainerInputFormat' OUTPUTFORMAT
'org.apache.hadoop.hive.ql.io.avro.AvroContainerOutputFormat';
insert into table avroarray partition(y=1) select * from testarray;
# add an int column with a default value of 0
alter table avroarray set serde
'org.apache.hadoop.hive.serde2.avro.AvroSerDe' with
serdeproperties('avro.schema.literal'='{namespace:test,name:avroarray,type:
record, fields: [ {name:intfield,type:int,default:0},{
name:a, type:{type:array,items:string} } ] }');
# fails with ClassCastException
select * from avroarray;
{code}
The select * fails with:
{code}
Failed with exception java.io.IOException:java.lang.ClassCastException:
org.apache.hadoop.hive.serde2.objectinspector.StandardListObjectInspector
cannot be cast to
org.apache.hadoop.hive.serde2.objectinspector.PrimitiveObjectInspector
{code}
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Updated] (HIVE-6835) Reading of partitioned Avro data fails if partition schema does not match table schema
[
https://issues.apache.org/jira/browse/HIVE-6835?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Anthony Hsu updated HIVE-6835:
--
Assignee: (was: Anthony Hsu)
Status: Patch Available (was: Open)
Reading of partitioned Avro data fails if partition schema does not match
table schema
--
Key: HIVE-6835
URL: https://issues.apache.org/jira/browse/HIVE-6835
Project: Hive
Issue Type: Bug
Affects Versions: 0.12.0
Reporter: Anthony Hsu
Attachments: HIVE-6835.1.patch
To reproduce:
{code}
create table testarray (a arraystring);
load data local inpath '/home/ahsu/test/array.txt' into table testarray;
# create partitioned Avro table with one array column
create table avroarray partitioned by (y string) row format serde
'org.apache.hadoop.hive.serde2.avro.AvroSerDe' with serdeproperties
('avro.schema.literal'='{namespace:test,name:avroarray,type:
record, fields: [ { name:a, type:{type:array,items:string}
} ] }') STORED as INPUTFORMAT
'org.apache.hadoop.hive.ql.io.avro.AvroContainerInputFormat' OUTPUTFORMAT
'org.apache.hadoop.hive.ql.io.avro.AvroContainerOutputFormat';
insert into table avroarray partition(y=1) select * from testarray;
# add an int column with a default value of 0
alter table avroarray set serde
'org.apache.hadoop.hive.serde2.avro.AvroSerDe' with
serdeproperties('avro.schema.literal'='{namespace:test,name:avroarray,type:
record, fields: [ {name:intfield,type:int,default:0},{
name:a, type:{type:array,items:string} } ] }');
# fails with ClassCastException
select * from avroarray;
{code}
The select * fails with:
{code}
Failed with exception java.io.IOException:java.lang.ClassCastException:
org.apache.hadoop.hive.serde2.objectinspector.StandardListObjectInspector
cannot be cast to
org.apache.hadoop.hive.serde2.objectinspector.PrimitiveObjectInspector
{code}
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HIVE-6856) ddl commands fail with permissions issue when running using webhcat in secure Tez cluster
[
https://issues.apache.org/jira/browse/HIVE-6856?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13962176#comment-13962176
]
Thejas M Nair commented on HIVE-6856:
-
+1
Hcat cli never runs any query on the cluster, so it never needs a runtime
engine. Always using mr as the engine in config works fine.
ddl commands fail with permissions issue when running using webhcat in secure
Tez cluster
-
Key: HIVE-6856
URL: https://issues.apache.org/jira/browse/HIVE-6856
Project: Hive
Issue Type: Bug
Components: WebHCat
Affects Versions: 0.13.0
Reporter: Eugene Koifman
Assignee: Eugene Koifman
Attachments: HIVE-6856.patch
curl -u : --negotiate -d exec=show tables; -X POST
http://server:50111/templeton/v1/ddl
results in (when Tez is enabled in Secure cluster)
{noformat}
Exception in thread main java.lang.RuntimeException:
org.apache.hadoop.security.AccessControlException: Permission denied:
user=hrt_qa, access=WRITE, inode=/user/hcat:hcat:hcat:drwxr-xr-x
at
org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.checkFsPermission(FSPermissionChecker.java:265)
at
org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.check(FSPermissionChecker.java:251)
at
org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.check(FSPermissionChecker.java:232)
at
org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.checkPermission(FSPermissionChecker.java:176)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkPermission(FSNamesystem.java:5497)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkPermission(FSNamesystem.java:5479)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkAncestorAccess(FSNamesystem.java:5453)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.mkdirsInternal(FSNamesystem.java:3596)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.mkdirsInt(FSNamesystem.java:3566)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.mkdirs(FSNamesystem.java:3540)
at
org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.mkdirs(NameNodeRpcServer.java:754)
at
org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.mkdirs(ClientNamenodeProtocolServerSideTranslatorPB.java:558)
at
org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
at
org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:585)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:928)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2013)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2009)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1548)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2007)
at
org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:354)
at org.apache.hive.hcatalog.cli.HCatCli.main(HCatCli.java:138)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.util.RunJar.main(RunJar.java:212)
Caused by: org.apache.hadoop.security.AccessControlException: Permission
denied: user=hrt_qa, access=WRITE, inode=/user/hcat:hcat:hcat:drwxr-xr-x
at
org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.checkFsPermission(FSPermissionChecker.java:265)
at
org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.check(FSPermissionChecker.java:251)
at
org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.check(FSPermissionChecker.java:232)
at
org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.checkPermission(FSPermissionChecker.java:176)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkPermission(FSNamesystem.java:5497)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkPermission(FSNamesystem.java:5479)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkAncestorAccess(FSNamesystem.java:5453)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.mkdirsInternal(FSNamesystem.java:3596)
at
[jira] [Updated] (HIVE-6319) Insert, update, delete functionality needs a compactor
[ https://issues.apache.org/jira/browse/HIVE-6319?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alan Gates updated HIVE-6319: - Attachment: HIVE-6319.patch Attaching new version of the patch with changes as suggested by Ashutosh. I don't think we need to re-run the tests as the changes are very small. Insert, update, delete functionality needs a compactor -- Key: HIVE-6319 URL: https://issues.apache.org/jira/browse/HIVE-6319 Project: Hive Issue Type: Sub-task Reporter: Alan Gates Assignee: Alan Gates Fix For: 0.13.0 Attachments: 6319.wip.patch, HIVE-6319.patch, HIVE-6319.patch, HIVE-6319.patch, HIVE-6319.patch, HiveCompactorDesign.pdf In order to keep the number of delta files from spiraling out of control we need a compactor to collect these delta files together, and eventually rewrite the base file when the deltas get large enough. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Created] (HIVE-6858) Unit tests decimal_udf.q, vectorization_div0.q fail with jdk-7.
Jitendra Nath Pandey created HIVE-6858:
--
Summary: Unit tests decimal_udf.q, vectorization_div0.q fail with
jdk-7.
Key: HIVE-6858
URL: https://issues.apache.org/jira/browse/HIVE-6858
Project: Hive
Issue Type: Bug
Reporter: Jitendra Nath Pandey
Assignee: Jitendra Nath Pandey
Unit tests decimal_udf.q, vectorization_div0.q fail with jdk-7.
{noformat}
-250.06583411.236 1.0 6583411.236 -0.004 -0.0048
---
-250.06583411.236 1.0 6583411.236 -0.0040 -0.0048
{noformat}
Following code reproduces this behavior when run in jdk-7 vs jdk-6. Jdk-7
produces -0.004 while, jdk-6 produces -0.0040.
{code}
public class Main {
public static void main(String[] a) throws Exception {
double val = 0.004;
System.out.println(Value = +val);
}
}
{code}
This happens to be a bug in jdk6, that has been fixed in jdk7.
http://bugs.java.com/bugdatabase/view_bug.do?bug_id=4511638
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Updated] (HIVE-6843) INSTR for UTF-8 returns incorrect position
[ https://issues.apache.org/jira/browse/HIVE-6843?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Szehon Ho updated HIVE-6843: Attachment: HIVE-6843.patch This seems to work, lets see what folks think. Original code was trying to avoid encoding the bytes and just doing byte-counting, but not sure if that is possible when doing unicode char calculations. INSTR for UTF-8 returns incorrect position -- Key: HIVE-6843 URL: https://issues.apache.org/jira/browse/HIVE-6843 Project: Hive Issue Type: Bug Components: UDF Affects Versions: 0.11.0, 0.12.0 Reporter: Clif Kranish Assignee: Szehon Ho Priority: Minor Attachments: HIVE-6843.patch -- This message was sent by Atlassian JIRA (v6.2#6252)
Review Request 20103: HIVE-6843 INSTR for UTF-8 returns incorrect position
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/20103/ --- Review request for hive. Repository: hive-git Description --- Seems the original authors wanted to avoid encoding, but this is not possible if you want to handle Unicode characters. Diffs - ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFUtils.java 7f4a807 ql/src/test/org/apache/hadoop/hive/ql/udf/TestGenericUDFUtils.java d9338a5 Diff: https://reviews.apache.org/r/20103/diff/ Testing --- Adding some unicode test of Cyrillic chars. Thanks, Szehon Ho
[jira] [Updated] (HIVE-6843) INSTR for UTF-8 returns incorrect position
[ https://issues.apache.org/jira/browse/HIVE-6843?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Szehon Ho updated HIVE-6843: Status: Patch Available (was: Open) INSTR for UTF-8 returns incorrect position -- Key: HIVE-6843 URL: https://issues.apache.org/jira/browse/HIVE-6843 Project: Hive Issue Type: Bug Components: UDF Affects Versions: 0.12.0, 0.11.0 Reporter: Clif Kranish Assignee: Szehon Ho Priority: Minor Attachments: HIVE-6843.patch -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HIVE-6837) HiveServer2 thrift/http mode binary mode proxy user check fails reporting IP null for client
[ https://issues.apache.org/jira/browse/HIVE-6837?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13962215#comment-13962215 ] Harish Butani commented on HIVE-6837: - +1 for 0.13 HiveServer2 thrift/http mode binary mode proxy user check fails reporting IP null for client -- Key: HIVE-6837 URL: https://issues.apache.org/jira/browse/HIVE-6837 Project: Hive Issue Type: Bug Components: HiveServer2 Affects Versions: 0.13.0 Reporter: Dilli Arumugam Assignee: Vaibhav Gumashta Fix For: 0.13.0 Attachments: HIVE-6837.1.patch, HIVE-6837.2.patch, HIVE-6837.3.patch, hive.log Hive Server running thrift/http with Kerberos security. Kinited user knox attempting to proxy as sam. Beeline connection failed reporting error on hive server logs: Caused by: org.apache.hadoop.security.authorize.AuthorizationException: Unauthorized connection for super-user: knox from IP null -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HIVE-6739) Hive HBase query fails on Tez due to missing jars and then due to NPE in getSplits
[
https://issues.apache.org/jira/browse/HIVE-6739?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13962236#comment-13962236
]
Hive QA commented on HIVE-6739:
---
{color:red}Overall{color}: -1 at least one tests failed
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12639030/HIVE-6739.04.patch
{color:red}ERROR:{color} -1 due to 1 failed/errored test(s), 5549 tests executed
*Failed tests:*
{noformat}
org.apache.hive.hcatalog.mapreduce.TestHCatMutableDynamicPartitioned.testHCatDynamicPartitionedTable
{noformat}
Test results:
http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/2164/testReport
Console output:
http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/2164/console
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 1 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12639030
Hive HBase query fails on Tez due to missing jars and then due to NPE in
getSplits
--
Key: HIVE-6739
URL: https://issues.apache.org/jira/browse/HIVE-6739
Project: Hive
Issue Type: Bug
Reporter: Sergey Shelukhin
Assignee: Sergey Shelukhin
Fix For: 0.13.0
Attachments: HIVE-6739.01.patch, HIVE-6739.02.patch,
HIVE-6739.03.patch, HIVE-6739.04.patch, HIVE-6739.patch,
HIVE-6739.preliminary.patch
Tez paths in Hive never call configure on the input/output operators, so
(among other things, potentially) requisite files never get added to the job
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Updated] (HIVE-6782) HiveServer2Concurrency issue when running with tez intermittently, throwing org.apache.tez.dag.api.SessionNotRunning: Application not running error
[ https://issues.apache.org/jira/browse/HIVE-6782?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vikram Dixit K updated HIVE-6782: - Status: Open (was: Patch Available) HiveServer2Concurrency issue when running with tez intermittently, throwing org.apache.tez.dag.api.SessionNotRunning: Application not running error - Key: HIVE-6782 URL: https://issues.apache.org/jira/browse/HIVE-6782 Project: Hive Issue Type: Bug Components: Tez Reporter: Vikram Dixit K Assignee: Vikram Dixit K Fix For: 0.13.0, 0.14.0 Attachments: HIVE-6782.1.patch, HIVE-6782.2.patch, HIVE-6782.3.patch, HIVE-6782.4.patch, HIVE-6782.5.patch, HIVE-6782.6.patch, HIVE-6782.7.patch, HIVE-6782.8.patch HiveServer2 concurrency is failing intermittently when using tez, throwing org.apache.tez.dag.api.SessionNotRunning: Application not running error -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HIVE-6782) HiveServer2Concurrency issue when running with tez intermittently, throwing org.apache.tez.dag.api.SessionNotRunning: Application not running error
[ https://issues.apache.org/jira/browse/HIVE-6782?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vikram Dixit K updated HIVE-6782: - Attachment: HIVE-6782.9.patch Fix the case where a tez session is launched without a query. HiveServer2Concurrency issue when running with tez intermittently, throwing org.apache.tez.dag.api.SessionNotRunning: Application not running error - Key: HIVE-6782 URL: https://issues.apache.org/jira/browse/HIVE-6782 Project: Hive Issue Type: Bug Components: Tez Reporter: Vikram Dixit K Assignee: Vikram Dixit K Fix For: 0.13.0, 0.14.0 Attachments: HIVE-6782.1.patch, HIVE-6782.2.patch, HIVE-6782.3.patch, HIVE-6782.4.patch, HIVE-6782.5.patch, HIVE-6782.6.patch, HIVE-6782.7.patch, HIVE-6782.8.patch, HIVE-6782.9.patch HiveServer2 concurrency is failing intermittently when using tez, throwing org.apache.tez.dag.api.SessionNotRunning: Application not running error -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HIVE-6782) HiveServer2Concurrency issue when running with tez intermittently, throwing org.apache.tez.dag.api.SessionNotRunning: Application not running error
[ https://issues.apache.org/jira/browse/HIVE-6782?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vikram Dixit K updated HIVE-6782: - Status: Patch Available (was: Open) HiveServer2Concurrency issue when running with tez intermittently, throwing org.apache.tez.dag.api.SessionNotRunning: Application not running error - Key: HIVE-6782 URL: https://issues.apache.org/jira/browse/HIVE-6782 Project: Hive Issue Type: Bug Components: Tez Reporter: Vikram Dixit K Assignee: Vikram Dixit K Fix For: 0.13.0, 0.14.0 Attachments: HIVE-6782.1.patch, HIVE-6782.2.patch, HIVE-6782.3.patch, HIVE-6782.4.patch, HIVE-6782.5.patch, HIVE-6782.6.patch, HIVE-6782.7.patch, HIVE-6782.8.patch, HIVE-6782.9.patch HiveServer2 concurrency is failing intermittently when using tez, throwing org.apache.tez.dag.api.SessionNotRunning: Application not running error -- This message was sent by Atlassian JIRA (v6.2#6252)
Re: Review Request 20051: HIVE-4904: A little more CP crossing RS boundaries
On April 7, 2014, 6:03 p.m., John Pullokkaran wrote: I took a look at this change; my knowledge of hive code is rather limited. 1. Column Pruner doesn't cross Script operator boundary. Theoretically you could prune above and below the script op separately. 2. It seems column pruner assumes that parent of UDTF is always select; but we haven't formalized this assumption. Other processors should throw exception if it ever come across a child that is UDTF. Theoretically you can push down certain filters below builtin UDTF. We may not be doing that today. 3. In Select Pruner it seems like there is no difference between 'prunedCols' and 'columns'. Thanks John.Here are responses to your points 1. Column Pruner doesn't cross Script operator boundary. The ColumnPrunerWalker explicitly stops at the SelectOp parent of a ScriptOp. This may have been ok when developed; as you point out now it makes sense to continue pruning on the SelectOp ancestors. Can you file a jira for this. 2. The check in ColumnPrunerSelectProc is needed for the LVJoin case, where for the UDTFOp you end up with a empty PrunedList. What I realized was that Navis's fix doesn't cover the LVJoin case. Yes this should be revisited. - Harish --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/20051/#review39706 --- On April 6, 2014, 1:33 a.m., Harish Butani wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/20051/ --- (Updated April 6, 2014, 1:33 a.m.) Review request for hive, Ashutosh Chauhan and Navis Ryu. Bugs: HIVE-4904 https://issues.apache.org/jira/browse/HIVE-4904 Repository: hive-git Description --- Currently, CP context cannot be propagated over RS except for JOIN/EXT. A little more CP is possible. Diffs - ql/src/java/org/apache/hadoop/hive/ql/optimizer/ColumnPruner.java 58a9b59 ql/src/java/org/apache/hadoop/hive/ql/optimizer/ColumnPrunerProcCtx.java db36151 ql/src/java/org/apache/hadoop/hive/ql/optimizer/ColumnPrunerProcFactory.java 0690fb7 ql/src/java/org/apache/hadoop/hive/ql/optimizer/Optimizer.java 3f16dc2 ql/src/java/org/apache/hadoop/hive/ql/optimizer/correlation/CorrelationUtilities.java 94224b3 ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java 3b33dc2 ql/src/test/queries/clientpositive/order_within_subquery.q PRE-CREATION ql/src/test/results/clientpositive/annotate_stats_select.q.out 1e982e6 ql/src/test/results/clientpositive/auto_join18.q.out b8677f4 ql/src/test/results/clientpositive/auto_join27.q.out a576190 ql/src/test/results/clientpositive/auto_join30.q.out 8709198 ql/src/test/results/clientpositive/auto_join31.q.out 1936e45 ql/src/test/results/clientpositive/auto_join32.q.out 05f53e6 ql/src/test/results/clientpositive/auto_sortmerge_join_10.q.out 8882aac ql/src/test/results/clientpositive/count.q.out eb048b6 ql/src/test/results/clientpositive/distinct_stats.q.out f715ea3 ql/src/test/results/clientpositive/groupby2_map.q.out 291f196 ql/src/test/results/clientpositive/groupby2_map_skew.q.out d005b6c ql/src/test/results/clientpositive/groupby3_map.q.out 1dfee08 ql/src/test/results/clientpositive/groupby3_map_skew.q.out 7af59bc ql/src/test/results/clientpositive/groupby_cube1.q.out 92d81f4 ql/src/test/results/clientpositive/groupby_distinct_samekey.q.out b405978 ql/src/test/results/clientpositive/groupby_map_ppr.q.out 27eff75 ql/src/test/results/clientpositive/groupby_multi_insert_common_distinct.q.out ad76252 ql/src/test/results/clientpositive/groupby_multi_single_reducer3.q.out 51a70c4 ql/src/test/results/clientpositive/groupby_position.q.out 727bccb ql/src/test/results/clientpositive/groupby_rollup1.q.out 36bf966 ql/src/test/results/clientpositive/groupby_sort_11.q.out 8ee7571 ql/src/test/results/clientpositive/groupby_sort_8.q.out a27cfaa ql/src/test/results/clientpositive/join18.q.out 7975c79 ql/src/test/results/clientpositive/limit_pushdown.q.out 9c93ada ql/src/test/results/clientpositive/limit_pushdown_negative.q.out 115b171 ql/src/test/results/clientpositive/metadataonly1.q.out 917efdf ql/src/test/results/clientpositive/multi_insert_gby2.q.out ab758cb ql/src/test/results/clientpositive/multi_insert_gby3.q.out 23ccebb ql/src/test/results/clientpositive/multi_insert_lateral_view.q.out 35e70b4 ql/src/test/results/clientpositive/nullgroup.q.out 2ac7dea ql/src/test/results/clientpositive/nullgroup2.q.out cf31dc1 ql/src/test/results/clientpositive/nullgroup4.q.out feae138
[jira] [Updated] (HIVE-6837) HiveServer2 thrift/http mode binary mode proxy user check fails reporting IP null for client
[ https://issues.apache.org/jira/browse/HIVE-6837?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thejas M Nair updated HIVE-6837: Resolution: Fixed Status: Resolved (was: Patch Available) Patch committed to 0.13 branch and trunk. I made a minor edit to apply on 0.13 branch. Thanks for the contribution Vaibhav. Thanks for the review Dilli. HiveServer2 thrift/http mode binary mode proxy user check fails reporting IP null for client -- Key: HIVE-6837 URL: https://issues.apache.org/jira/browse/HIVE-6837 Project: Hive Issue Type: Bug Components: HiveServer2 Affects Versions: 0.13.0 Reporter: Dilli Arumugam Assignee: Vaibhav Gumashta Fix For: 0.13.0 Attachments: HIVE-6837.1.patch, HIVE-6837.2.patch, HIVE-6837.3.patch, hive.log Hive Server running thrift/http with Kerberos security. Kinited user knox attempting to proxy as sam. Beeline connection failed reporting error on hive server logs: Caused by: org.apache.hadoop.security.authorize.AuthorizationException: Unauthorized connection for super-user: knox from IP null -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HIVE-6846) allow safe set commands with sql standard authorization
[ https://issues.apache.org/jira/browse/HIVE-6846?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13962277#comment-13962277 ] Thejas M Nair commented on HIVE-6846: - I will add this to overall sql standard authorization document. I will work on that in a day or two. allow safe set commands with sql standard authorization --- Key: HIVE-6846 URL: https://issues.apache.org/jira/browse/HIVE-6846 Project: Hive Issue Type: Bug Components: Authorization Affects Versions: 0.13.0 Reporter: Thejas M Nair Assignee: Thejas M Nair Fix For: 0.13.0 Attachments: HIVE-6846.1.patch, HIVE-6846.2.patch HIVE-6827 disables all set commands when SQL standard authorization is turned on, but not all set commands are unsafe. We should allow safe set commands. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HIVE-6855) A couple of errors in MySQL db creation script for transaction tables
[ https://issues.apache.org/jira/browse/HIVE-6855?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13962280#comment-13962280 ] Ashutosh Chauhan commented on HIVE-6855: +1 A couple of errors in MySQL db creation script for transaction tables - Key: HIVE-6855 URL: https://issues.apache.org/jira/browse/HIVE-6855 Project: Hive Issue Type: Bug Components: Metastore Affects Versions: 0.13.0 Reporter: Alan Gates Assignee: Alan Gates Attachments: HIVE-6855.patch There are a few small issues in the database creation scripts for mysql. A couple of the tables don't set the engine to InnoDB. None of the tables set default character set to latin1. And the syntax CREATE INDEX...USING HASH doesn't work on older versions of MySQL. Instead the index creation should be done without specifying a method (no USING clause). -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HIVE-6782) HiveServer2Concurrency issue when running with tez intermittently, throwing org.apache.tez.dag.api.SessionNotRunning: Application not running error
[ https://issues.apache.org/jira/browse/HIVE-6782?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13962315#comment-13962315 ] Thejas M Nair commented on HIVE-6782: - +1 to the update as well. HiveServer2Concurrency issue when running with tez intermittently, throwing org.apache.tez.dag.api.SessionNotRunning: Application not running error - Key: HIVE-6782 URL: https://issues.apache.org/jira/browse/HIVE-6782 Project: Hive Issue Type: Bug Components: Tez Reporter: Vikram Dixit K Assignee: Vikram Dixit K Fix For: 0.13.0, 0.14.0 Attachments: HIVE-6782.1.patch, HIVE-6782.2.patch, HIVE-6782.3.patch, HIVE-6782.4.patch, HIVE-6782.5.patch, HIVE-6782.6.patch, HIVE-6782.7.patch, HIVE-6782.8.patch, HIVE-6782.9.patch HiveServer2 concurrency is failing intermittently when using tez, throwing org.apache.tez.dag.api.SessionNotRunning: Application not running error -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HIVE-6846) allow safe set commands with sql standard authorization
[ https://issues.apache.org/jira/browse/HIVE-6846?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13962323#comment-13962323 ] Ashutosh Chauhan commented on HIVE-6846: +1 allow safe set commands with sql standard authorization --- Key: HIVE-6846 URL: https://issues.apache.org/jira/browse/HIVE-6846 Project: Hive Issue Type: Bug Components: Authorization Affects Versions: 0.13.0 Reporter: Thejas M Nair Assignee: Thejas M Nair Fix For: 0.13.0 Attachments: HIVE-6846.1.patch, HIVE-6846.2.patch HIVE-6827 disables all set commands when SQL standard authorization is turned on, but not all set commands are unsafe. We should allow safe set commands. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HIVE-6739) Hive HBase query fails on Tez due to missing jars and then due to NPE in getSplits
[ https://issues.apache.org/jira/browse/HIVE-6739?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HIVE-6739: --- Resolution: Fixed Status: Resolved (was: Patch Available) in trunk and 13 Hive HBase query fails on Tez due to missing jars and then due to NPE in getSplits -- Key: HIVE-6739 URL: https://issues.apache.org/jira/browse/HIVE-6739 Project: Hive Issue Type: Bug Reporter: Sergey Shelukhin Assignee: Sergey Shelukhin Fix For: 0.13.0 Attachments: HIVE-6739.01.patch, HIVE-6739.02.patch, HIVE-6739.03.patch, HIVE-6739.04.patch, HIVE-6739.patch, HIVE-6739.preliminary.patch Tez paths in Hive never call configure on the input/output operators, so (among other things, potentially) requisite files never get added to the job -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HIVE-6134) Merging small files based on file size only works for CTAS queries
[ https://issues.apache.org/jira/browse/HIVE-6134?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13962335#comment-13962335 ] Eric Chu commented on HIVE-6134: Hi [~xuefuz] and [~ashutoshc], it turns out this issues not only affects Hue but also HIVE CLI - in that results won't show up in CLI until more than a minute has passed with timeout error for connection to nodes. I'm trying to make the change myself in GenMRFileSink1.java to support a new property that when it's turned on, Hive will merge files for a regular (i.e., without mvTask), map-only job that uses more than X mappers (another property). I'm wondering if and how we could find out the number of mappers that will be used for that job when we are at that stage of the optimization. I want to set chDir to true when this number is greater than some threshold set via a new property. I notice that currWork.getMapWork().getNumMapTasks() actually returns null. Can you give me some pointers? Merging small files based on file size only works for CTAS queries -- Key: HIVE-6134 URL: https://issues.apache.org/jira/browse/HIVE-6134 Project: Hive Issue Type: Bug Affects Versions: 0.8.0, 0.10.0, 0.11.0, 0.12.0 Reporter: Eric Chu According to the documentation, if we set hive.merge.mapfiles to true, Hive will launch an additional MR job to merge the small output files at the end of a map-only job when the average output file size is smaller than hive.merge.smallfiles.avgsize. Similarly, by setting hive.merge.mapredfiles to true, Hive will merge the output files of a map-reduce job. My expectation is that this is true for all MR queries. However, my observation is that this is only true for CTAS queries. In GenMRFileSink1.java, HIVEMERGEMAPFILES and HIVEMERGEMAPREDFILES are only used if ((ctx.getMvTask() != null) (!ctx.getMvTask().isEmpty())). So, for a regular SELECT query that doesn't have move tasks, these properties are not used. Is my understanding correct and if so, what's the reasoning behind the logic of not supporting this for regular SELECT queries? It seems to me that this should be supported for regular SELECT queries as well. One scenario where this hits us hard is when users try to download the result in HUE, and HUE times out b/c there are thousands of output files. The workaround is to re-run the query as CTAS, but it's a significant time sink. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HIVE-6782) HiveServer2Concurrency issue when running with tez intermittently, throwing org.apache.tez.dag.api.SessionNotRunning: Application not running error
[ https://issues.apache.org/jira/browse/HIVE-6782?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vikram Dixit K updated HIVE-6782: - Attachment: HIVE-6782.10.patch Needed rebase after HIVE-6739. HiveServer2Concurrency issue when running with tez intermittently, throwing org.apache.tez.dag.api.SessionNotRunning: Application not running error - Key: HIVE-6782 URL: https://issues.apache.org/jira/browse/HIVE-6782 Project: Hive Issue Type: Bug Components: Tez Reporter: Vikram Dixit K Assignee: Vikram Dixit K Fix For: 0.13.0, 0.14.0 Attachments: HIVE-6782.1.patch, HIVE-6782.10.patch, HIVE-6782.2.patch, HIVE-6782.3.patch, HIVE-6782.4.patch, HIVE-6782.5.patch, HIVE-6782.6.patch, HIVE-6782.7.patch, HIVE-6782.8.patch, HIVE-6782.9.patch HiveServer2 concurrency is failing intermittently when using tez, throwing org.apache.tez.dag.api.SessionNotRunning: Application not running error -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HIVE-6825) custom jars for Hive query should be uploaded to scratch dir per query; and/or versioned
[ https://issues.apache.org/jira/browse/HIVE-6825?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HIVE-6825: --- Attachment: HIVE-6825.01.patch custom jars for Hive query should be uploaded to scratch dir per query; and/or versioned Key: HIVE-6825 URL: https://issues.apache.org/jira/browse/HIVE-6825 Project: Hive Issue Type: Improvement Reporter: Sergey Shelukhin Assignee: Sergey Shelukhin Fix For: 0.14.0 Attachments: HIVE-6825.01.patch, HIVE-6825.patch Currently the jars are uploaded to either user directory or global, whatever is configured, which is a mess and can cause collisions. We can upload to scratch directory, and/or version. There's a tradeoff between having to upload files every time (for example, for commonly used things like HBase input format) (which is what is done now, into global/user path), and having a mess of one-off custom jars and files, versioned, sitting in .hiveJars. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HIVE-6825) custom jars for Hive query should be uploaded to scratch dir per query; and/or versioned
[ https://issues.apache.org/jira/browse/HIVE-6825?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HIVE-6825: --- Status: Patch Available (was: Open) custom jars for Hive query should be uploaded to scratch dir per query; and/or versioned Key: HIVE-6825 URL: https://issues.apache.org/jira/browse/HIVE-6825 Project: Hive Issue Type: Improvement Reporter: Sergey Shelukhin Assignee: Sergey Shelukhin Fix For: 0.14.0 Attachments: HIVE-6825.01.patch, HIVE-6825.patch Currently the jars are uploaded to either user directory or global, whatever is configured, which is a mess and can cause collisions. We can upload to scratch directory, and/or version. There's a tradeoff between having to upload files every time (for example, for commonly used things like HBase input format) (which is what is done now, into global/user path), and having a mess of one-off custom jars and files, versioned, sitting in .hiveJars. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HIVE-6825) custom jars for Hive query should be uploaded to scratch dir per query; and/or versioned
[ https://issues.apache.org/jira/browse/HIVE-6825?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13962346#comment-13962346 ] Sergey Shelukhin commented on HIVE-6825: [~vikram.dixit] this is the jira custom jars for Hive query should be uploaded to scratch dir per query; and/or versioned Key: HIVE-6825 URL: https://issues.apache.org/jira/browse/HIVE-6825 Project: Hive Issue Type: Improvement Reporter: Sergey Shelukhin Assignee: Sergey Shelukhin Fix For: 0.14.0 Attachments: HIVE-6825.01.patch, HIVE-6825.patch Currently the jars are uploaded to either user directory or global, whatever is configured, which is a mess and can cause collisions. We can upload to scratch directory, and/or version. There's a tradeoff between having to upload files every time (for example, for commonly used things like HBase input format) (which is what is done now, into global/user path), and having a mess of one-off custom jars and files, versioned, sitting in .hiveJars. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HIVE-6846) allow safe set commands with sql standard authorization
[
https://issues.apache.org/jira/browse/HIVE-6846?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13962353#comment-13962353
]
Hive QA commented on HIVE-6846:
---
{color:red}Overall{color}: -1 at least one tests failed
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12639034/HIVE-6846.2.patch
{color:red}ERROR:{color} -1 due to 2 failed/errored test(s), 5552 tests executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.ql.parse.authorization.TestSessionUserName.testSessionConstructorUser
org.apache.hadoop.hive.ql.parse.authorization.TestSessionUserName.testSessionDefaultUser
{noformat}
Test results:
http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/2165/testReport
Console output:
http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/2165/console
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 2 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12639034
allow safe set commands with sql standard authorization
---
Key: HIVE-6846
URL: https://issues.apache.org/jira/browse/HIVE-6846
Project: Hive
Issue Type: Bug
Components: Authorization
Affects Versions: 0.13.0
Reporter: Thejas M Nair
Assignee: Thejas M Nair
Fix For: 0.13.0
Attachments: HIVE-6846.1.patch, HIVE-6846.2.patch
HIVE-6827 disables all set commands when SQL standard authorization is turned
on, but not all set commands are unsafe. We should allow safe set commands.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HIVE-6818) Array out of bounds when ORC is used with ACID and predicate push down
[ https://issues.apache.org/jira/browse/HIVE-6818?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13962359#comment-13962359 ] Owen O'Malley commented on HIVE-6818: - Sergey, My intention is to replace the current xml ast with the serialized SearchArgument. The serialized SearchArgument is much more compact and focused on predicate pushdown. However, in order for that to happen, we need to transition the clients from the old format to the new one. So, yes, the immediate patch only uses it for testing, but it should over time become the mainline path. Array out of bounds when ORC is used with ACID and predicate push down -- Key: HIVE-6818 URL: https://issues.apache.org/jira/browse/HIVE-6818 Project: Hive Issue Type: Bug Components: File Formats Reporter: Owen O'Malley Assignee: Owen O'Malley Priority: Blocker Fix For: 0.13.0 Attachments: HIVE-6818.patch The users gets an ArrayOutOfBoundsException when using ORC, ACID, and predicate push down. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Created] (HIVE-6859) 8
Szehon Ho created HIVE-6859: --- Summary: 8 Key: HIVE-6859 URL: https://issues.apache.org/jira/browse/HIVE-6859 Project: Hive Issue Type: Bug Reporter: Szehon Ho -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Resolved] (HIVE-6859) 8
[ https://issues.apache.org/jira/browse/HIVE-6859?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Szehon Ho resolved HIVE-6859. - Resolution: Invalid Issue created by accident. 8 - Key: HIVE-6859 URL: https://issues.apache.org/jira/browse/HIVE-6859 Project: Hive Issue Type: Bug Reporter: Szehon Ho -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HIVE-6757) Remove deprecated parquet classes from outside of org.apache package
[ https://issues.apache.org/jira/browse/HIVE-6757?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13962367#comment-13962367 ] Owen O'Malley commented on HIVE-6757: - +1 thanks Harish! Remove deprecated parquet classes from outside of org.apache package Key: HIVE-6757 URL: https://issues.apache.org/jira/browse/HIVE-6757 Project: Hive Issue Type: Bug Reporter: Owen O'Malley Assignee: Owen O'Malley Priority: Blocker Fix For: 0.13.0 Attachments: HIVE-6757.2.patch, HIVE-6757.patch, parquet-hive.patch Apache shouldn't release projects with files outside of the org.apache namespace. -- This message was sent by Atlassian JIRA (v6.2#6252)
Review Request 20110: HIVE-6825 custom jars for Hive query should be uploaded to scratch dir per query; and/or versioned
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/20110/ --- Review request for hive and Vikram Dixit Kumaraswamy. Repository: hive-git Description --- See JIRA Diffs - ql/src/java/org/apache/hadoop/hive/ql/exec/tez/DagUtils.java 14d188f ql/src/java/org/apache/hadoop/hive/ql/exec/tez/TezSessionState.java 74940e6 ql/src/java/org/apache/hadoop/hive/ql/exec/tez/TezTask.java c355d5a Diff: https://reviews.apache.org/r/20110/diff/ Testing --- Thanks, Sergey Shelukhin
[jira] [Commented] (HIVE-6825) custom jars for Hive query should be uploaded to scratch dir per query; and/or versioned
[ https://issues.apache.org/jira/browse/HIVE-6825?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13962369#comment-13962369 ] Sergey Shelukhin commented on HIVE-6825: https://reviews.apache.org/r/20110/ custom jars for Hive query should be uploaded to scratch dir per query; and/or versioned Key: HIVE-6825 URL: https://issues.apache.org/jira/browse/HIVE-6825 Project: Hive Issue Type: Improvement Reporter: Sergey Shelukhin Assignee: Sergey Shelukhin Fix For: 0.14.0 Attachments: HIVE-6825.01.patch, HIVE-6825.patch Currently the jars are uploaded to either user directory or global, whatever is configured, which is a mess and can cause collisions. We can upload to scratch directory, and/or version. There's a tradeoff between having to upload files every time (for example, for commonly used things like HBase input format) (which is what is done now, into global/user path), and having a mess of one-off custom jars and files, versioned, sitting in .hiveJars. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HIVE-6850) For FetchOperator, Driver uses the valid transaction list from the previous query
[ https://issues.apache.org/jira/browse/HIVE-6850?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13962417#comment-13962417 ] Harish Butani commented on HIVE-6850: - +1 lgtm +1 for 0.13 For FetchOperator, Driver uses the valid transaction list from the previous query - Key: HIVE-6850 URL: https://issues.apache.org/jira/browse/HIVE-6850 Project: Hive Issue Type: Bug Components: Clients Reporter: Alan Gates Assignee: Owen O'Malley Priority: Blocker Fix For: 0.13.0 Attachments: HIVE-6850.patch The problem is two fold: * FetchTask.initialize, which is called during parsing of the query, converts the HiveConf it is given into a JobConf by copying it. * Driver.recordValidTxns, which runs after parsing, adds the valid transactions to the HiveConf. Thus fetch operators will use the transactions from the previous command. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HIVE-6825) custom jars for Hive query should be uploaded to scratch dir per query; and/or versioned
[ https://issues.apache.org/jira/browse/HIVE-6825?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13962433#comment-13962433 ] Vikram Dixit K commented on HIVE-6825: -- LGTM +1 pending HiveQA test run. custom jars for Hive query should be uploaded to scratch dir per query; and/or versioned Key: HIVE-6825 URL: https://issues.apache.org/jira/browse/HIVE-6825 Project: Hive Issue Type: Improvement Reporter: Sergey Shelukhin Assignee: Sergey Shelukhin Fix For: 0.14.0 Attachments: HIVE-6825.01.patch, HIVE-6825.patch Currently the jars are uploaded to either user directory or global, whatever is configured, which is a mess and can cause collisions. We can upload to scratch directory, and/or version. There's a tradeoff between having to upload files every time (for example, for commonly used things like HBase input format) (which is what is done now, into global/user path), and having a mess of one-off custom jars and files, versioned, sitting in .hiveJars. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HIVE-6858) Unit tests decimal_udf.q, vectorization_div0.q fail with jdk-7.
[
https://issues.apache.org/jira/browse/HIVE-6858?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jitendra Nath Pandey updated HIVE-6858:
---
Status: Patch Available (was: Open)
Unit tests decimal_udf.q, vectorization_div0.q fail with jdk-7.
---
Key: HIVE-6858
URL: https://issues.apache.org/jira/browse/HIVE-6858
Project: Hive
Issue Type: Bug
Reporter: Jitendra Nath Pandey
Assignee: Jitendra Nath Pandey
Attachments: HIVE-6858.1.patch
Unit tests decimal_udf.q, vectorization_div0.q fail with jdk-7.
{noformat}
-250.0 6583411.236 1.0 6583411.236 -0.004 -0.0048
---
-250.0 6583411.236 1.0 6583411.236 -0.0040 -0.0048
{noformat}
Following code reproduces this behavior when run in jdk-7 vs jdk-6. Jdk-7
produces -0.004 while, jdk-6 produces -0.0040.
{code}
public class Main {
public static void main(String[] a) throws Exception {
double val = 0.004;
System.out.println(Value = +val);
}
}
{code}
This happens to be a bug in jdk6, that has been fixed in jdk7.
http://bugs.java.com/bugdatabase/view_bug.do?bug_id=4511638
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Updated] (HIVE-6858) Unit tests decimal_udf.q, vectorization_div0.q fail with jdk-7.
[
https://issues.apache.org/jira/browse/HIVE-6858?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jitendra Nath Pandey updated HIVE-6858:
---
Attachment: HIVE-6858.1.patch
Attached patch modifies the tests so that it doesn't run into the jdk bug.
Unit tests decimal_udf.q, vectorization_div0.q fail with jdk-7.
---
Key: HIVE-6858
URL: https://issues.apache.org/jira/browse/HIVE-6858
Project: Hive
Issue Type: Bug
Reporter: Jitendra Nath Pandey
Assignee: Jitendra Nath Pandey
Attachments: HIVE-6858.1.patch
Unit tests decimal_udf.q, vectorization_div0.q fail with jdk-7.
{noformat}
-250.0 6583411.236 1.0 6583411.236 -0.004 -0.0048
---
-250.0 6583411.236 1.0 6583411.236 -0.0040 -0.0048
{noformat}
Following code reproduces this behavior when run in jdk-7 vs jdk-6. Jdk-7
produces -0.004 while, jdk-6 produces -0.0040.
{code}
public class Main {
public static void main(String[] a) throws Exception {
double val = 0.004;
System.out.println(Value = +val);
}
}
{code}
This happens to be a bug in jdk6, that has been fixed in jdk7.
http://bugs.java.com/bugdatabase/view_bug.do?bug_id=4511638
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Updated] (HIVE-6846) allow safe set commands with sql standard authorization
[ https://issues.apache.org/jira/browse/HIVE-6846?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thejas M Nair updated HIVE-6846: Attachment: HIVE-6846.3.patch HIVE-6846.3.patch - test only changes. I have verified that the tests pass. allow safe set commands with sql standard authorization --- Key: HIVE-6846 URL: https://issues.apache.org/jira/browse/HIVE-6846 Project: Hive Issue Type: Bug Components: Authorization Affects Versions: 0.13.0 Reporter: Thejas M Nair Assignee: Thejas M Nair Fix For: 0.13.0 Attachments: HIVE-6846.1.patch, HIVE-6846.2.patch, HIVE-6846.3.patch HIVE-6827 disables all set commands when SQL standard authorization is turned on, but not all set commands are unsafe. We should allow safe set commands. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HIVE-6821) Fix some non-deterministic tests
[ https://issues.apache.org/jira/browse/HIVE-6821?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashutosh Chauhan updated HIVE-6821: --- Resolution: Fixed Fix Version/s: 0.13.0 Status: Resolved (was: Patch Available) Committed to 0.13 trunk. Thanks, Jason! Fix some non-deterministic tests - Key: HIVE-6821 URL: https://issues.apache.org/jira/browse/HIVE-6821 Project: Hive Issue Type: Bug Components: Tests Reporter: Jason Dere Assignee: Jason Dere Fix For: 0.13.0 Attachments: HIVE-6821.1.patch, HIVE-6821.2.patch, HIVE-6821.3.patch A bunch of qfile tests look like they need an ORDER-BY added to the queries so that the output looks repeatable when testing with hadoop1/hadoop2. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HIVE-6809) Support bulk deleting directories for partition drop with partial spec
[ https://issues.apache.org/jira/browse/HIVE-6809?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Navis updated HIVE-6809: Attachment: HIVE-6809.4.patch.txt Support bulk deleting directories for partition drop with partial spec -- Key: HIVE-6809 URL: https://issues.apache.org/jira/browse/HIVE-6809 Project: Hive Issue Type: Improvement Components: Query Processor Reporter: Navis Assignee: Navis Attachments: HIVE-6809.1.patch.txt, HIVE-6809.2.patch.txt, HIVE-6809.3.patch.txt, HIVE-6809.4.patch.txt In busy hadoop system, dropping many of partitions takes much more time than expected. In hive-0.11.0, removing 1700 partitions by single partial spec took 90 minutes, which is reduced to 3 minutes when deleteData is set false. I couldn't test this in recent hive, which has HIVE-6256 but if the time-taking part is mostly from removing directories, it seemed not helpful to reduce whole processing time. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Created] (HIVE-6860) Issue with FS based stats collection on Tez
Ashutosh Chauhan created HIVE-6860: -- Summary: Issue with FS based stats collection on Tez Key: HIVE-6860 URL: https://issues.apache.org/jira/browse/HIVE-6860 Project: Hive Issue Type: Bug Components: Statistics, Tez Reporter: Ashutosh Chauhan Assignee: Ashutosh Chauhan Statistics from different tasks got overwritten while running on Tez. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HIVE-6860) Issue with FS based stats collection on Tez
[ https://issues.apache.org/jira/browse/HIVE-6860?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashutosh Chauhan updated HIVE-6860: --- Attachment: HIVE-6860.patch Issue with FS based stats collection on Tez --- Key: HIVE-6860 URL: https://issues.apache.org/jira/browse/HIVE-6860 Project: Hive Issue Type: Bug Components: Statistics, Tez Reporter: Ashutosh Chauhan Assignee: Ashutosh Chauhan Attachments: HIVE-6860.patch Statistics from different tasks got overwritten while running on Tez. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HIVE-6860) Issue with FS based stats collection on Tez
[ https://issues.apache.org/jira/browse/HIVE-6860?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashutosh Chauhan updated HIVE-6860: --- Status: Patch Available (was: Open) Issue with FS based stats collection on Tez --- Key: HIVE-6860 URL: https://issues.apache.org/jira/browse/HIVE-6860 Project: Hive Issue Type: Bug Components: Statistics, Tez Reporter: Ashutosh Chauhan Assignee: Ashutosh Chauhan Attachments: HIVE-6860.patch Statistics from different tasks got overwritten while running on Tez. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HIVE-6860) Issue with FS based stats collection on Tez
[ https://issues.apache.org/jira/browse/HIVE-6860?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13962445#comment-13962445 ] Vikram Dixit K commented on HIVE-6860: -- +1 LGTM. Issue with FS based stats collection on Tez --- Key: HIVE-6860 URL: https://issues.apache.org/jira/browse/HIVE-6860 Project: Hive Issue Type: Bug Components: Statistics, Tez Reporter: Ashutosh Chauhan Assignee: Ashutosh Chauhan Attachments: HIVE-6860.patch Statistics from different tasks got overwritten while running on Tez. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Assigned] (HIVE-6648) Permissions are not inherited correctly when tables have multiple partition columns
[
https://issues.apache.org/jira/browse/HIVE-6648?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Szehon Ho reassigned HIVE-6648:
---
Assignee: Szehon Ho
Permissions are not inherited correctly when tables have multiple partition
columns
---
Key: HIVE-6648
URL: https://issues.apache.org/jira/browse/HIVE-6648
Project: Hive
Issue Type: Bug
Affects Versions: 0.12.0
Reporter: Henry Robinson
Assignee: Szehon Ho
{{Warehouse.mkdirs()}} always looks at the immediate parent of the path that
it creates when determining what permissions to inherit. However, it may have
created that parent directory as well, in which case it will have the default
permissions and will not have inherited them.
This is a problem when performing an {{INSERT}} into a table with more than
one partition column. E.g., in an empty table:
{{INSERT INTO TABLE tbl PARTITION(p1=1, p2=2) ... }}
A new subdirectory /p1=1/p2=2 will be created, and with permission
inheritance (per HIVE-2504) enabled, the intention is presumably for both new
directories to inherit the root table dir's permissions. However,
{{mkdirs()}} will only set the permission of the leaf directory (i.e.
/p2=2/), and then only to the permissions of /p1=1/, which was just created.
{code}
public boolean mkdirs(Path f) throws MetaException {
FileSystem fs = null;
try {
fs = getFs(f);
LOG.debug(Creating directory if it doesn't exist: + f);
//Check if the directory already exists. We want to change the
permission
//to that of the parent directory only for newly created directories.
if (this.inheritPerms) {
try {
return fs.getFileStatus(f).isDir();
} catch (FileNotFoundException ignore) {
}
}
boolean success = fs.mkdirs(f);
if (this.inheritPerms success) {
// Set the permission of parent directory.
// HNR: This is the bug - getParent() may refer to a just-created
directory.
fs.setPermission(f, fs.getFileStatus(f.getParent()).getPermission());
}
return success;
} catch (IOException e) {
closeFs(fs);
MetaStoreUtils.logAndThrowMetaException(e);
}
return false;
}
{code}
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Updated] (HIVE-5687) Streaming support in Hive
[ https://issues.apache.org/jira/browse/HIVE-5687?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Roshan Naik updated HIVE-5687: -- Attachment: HIVE-5687.v6.patch Addressing review comments from Alan, Owen and some of Lars. Owen: DDL was used there mostly for convenience and correctness. The other places where API is used, cannot be accomplished via DDL. Streaming support in Hive - Key: HIVE-5687 URL: https://issues.apache.org/jira/browse/HIVE-5687 Project: Hive Issue Type: Sub-task Reporter: Roshan Naik Assignee: Roshan Naik Labels: ACID, Streaming Fix For: 0.13.0 Attachments: 5687-api-spec4.pdf, 5687-draft-api-spec.pdf, 5687-draft-api-spec2.pdf, 5687-draft-api-spec3.pdf, HIVE-5687-unit-test-fix.patch, HIVE-5687.patch, HIVE-5687.v2.patch, HIVE-5687.v3.patch, HIVE-5687.v4.patch, HIVE-5687.v5.patch, HIVE-5687.v6.patch, Hive Streaming Ingest API for v3 patch.pdf, Hive Streaming Ingest API for v4 patch.pdf Implement support for Streaming data into HIVE. - Provide a client streaming API - Transaction support: Clients should be able to periodically commit a batch of records atomically - Immediate visibility: Records should be immediately visible to queries on commit - Should not overload HDFS with too many small files Use Cases: - Streaming logs into HIVE via Flume - Streaming results of computations from Storm -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HIVE-6394) Implement Timestmap in ParquetSerde
[ https://issues.apache.org/jira/browse/HIVE-6394?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13962470#comment-13962470 ] Szehon Ho commented on HIVE-6394: - We upgraded parquet to get the new Int96 libraries, but there is a parquet exception when writing an actual Int96 type, with dictionary encoding on. Filed [https://github.com/Parquet/parquet-mr/issues/350|https://github.com/Parquet/parquet-mr/issues/350] which is being worked on. Will need to wait for the fix + new version of parquet before we can proceed. Implement Timestmap in ParquetSerde --- Key: HIVE-6394 URL: https://issues.apache.org/jira/browse/HIVE-6394 Project: Hive Issue Type: Sub-task Components: Serializers/Deserializers Reporter: Jarek Jarcec Cecho Assignee: Szehon Ho Labels: Parquet This JIRA is to implement timestamp support in Parquet SerDe. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HIVE-6835) Reading of partitioned Avro data fails if partition schema does not match table schema
[
https://issues.apache.org/jira/browse/HIVE-6835?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13962471#comment-13962471
]
Hive QA commented on HIVE-6835:
---
{color:green}Overall{color}: +1 all checks pass
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12639043/HIVE-6835.1.patch
{color:green}SUCCESS:{color} +1 5550 tests passed
Test results:
http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/2167/testReport
Console output:
http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/2167/console
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12639043
Reading of partitioned Avro data fails if partition schema does not match
table schema
--
Key: HIVE-6835
URL: https://issues.apache.org/jira/browse/HIVE-6835
Project: Hive
Issue Type: Bug
Affects Versions: 0.12.0
Reporter: Anthony Hsu
Assignee: Anthony Hsu
Attachments: HIVE-6835.1.patch
To reproduce:
{code}
create table testarray (a arraystring);
load data local inpath '/home/ahsu/test/array.txt' into table testarray;
# create partitioned Avro table with one array column
create table avroarray partitioned by (y string) row format serde
'org.apache.hadoop.hive.serde2.avro.AvroSerDe' with serdeproperties
('avro.schema.literal'='{namespace:test,name:avroarray,type:
record, fields: [ { name:a, type:{type:array,items:string}
} ] }') STORED as INPUTFORMAT
'org.apache.hadoop.hive.ql.io.avro.AvroContainerInputFormat' OUTPUTFORMAT
'org.apache.hadoop.hive.ql.io.avro.AvroContainerOutputFormat';
insert into table avroarray partition(y=1) select * from testarray;
# add an int column with a default value of 0
alter table avroarray set serde
'org.apache.hadoop.hive.serde2.avro.AvroSerDe' with
serdeproperties('avro.schema.literal'='{namespace:test,name:avroarray,type:
record, fields: [ {name:intfield,type:int,default:0},{
name:a, type:{type:array,items:string} } ] }');
# fails with ClassCastException
select * from avroarray;
{code}
The select * fails with:
{code}
Failed with exception java.io.IOException:java.lang.ClassCastException:
org.apache.hadoop.hive.serde2.objectinspector.StandardListObjectInspector
cannot be cast to
org.apache.hadoop.hive.serde2.objectinspector.PrimitiveObjectInspector
{code}
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HIVE-6785) query fails when partitioned table's table level serde is ParquetHiveSerDe and partition level serde is of different SerDe
[ https://issues.apache.org/jira/browse/HIVE-6785?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13962473#comment-13962473 ] Szehon Ho commented on HIVE-6785: - +1 (non-binding) , thanks for adding the q-test and address comments. FYI [~brocknoland] query fails when partitioned table's table level serde is ParquetHiveSerDe and partition level serde is of different SerDe -- Key: HIVE-6785 URL: https://issues.apache.org/jira/browse/HIVE-6785 Project: Hive Issue Type: Bug Components: File Formats, Serializers/Deserializers Affects Versions: 0.13.0 Reporter: Tongjie Chen Fix For: 0.14.0 Attachments: HIVE-6785.1.patch.txt, HIVE-6785.2.patch.txt When a hive table's SerDe is ParquetHiveSerDe, while some partitions are of other SerDe, AND if this table has string column[s], hive generates confusing error message: Failed with exception java.io.IOException:java.lang.ClassCastException: parquet.hive.serde.primitive.ParquetStringInspector cannot be cast to org.apache.hadoop.hive.serde2.objectinspector.primitive.SettableTimestampObjectInspector This is confusing because timestamp is mentioned even if it is not been used by the table. The reason is when there is SerDe difference between table and partition, hive tries to convert objectinspector of two SerDes. ParquetHiveSerDe's object inspector for string type is ParquetStringInspector (newly introduced), neither a subclass of WritableStringObjectInspector nor JavaStringObjectInspector, which ObjectInspectorConverters expect for string category objector inspector. There is no break statement in STRING case statement, hence the following TIMESTAMP case statement is executed, generating confusing error message. see also in the following parquet issue: https://github.com/Parquet/parquet-mr/issues/324 To fix that it is relatively easy, just make ParquetStringInspector subclass of JavaStringObjectInspector instead of AbstractPrimitiveJavaObjectInspector. But because constructor of class JavaStringObjectInspector is package scope instead of public or protected, we would need to move ParquetStringInspector to the same package with JavaStringObjectInspector. Also ArrayWritableObjectInspector's setStructFieldData needs to also accept List data, since the corresponding setStructFieldData and create methods return a list. This is also needed when table SerDe is ParquetHiveSerDe, and partition SerDe is something else. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HIVE-6785) query fails when partitioned table's table level serde is ParquetHiveSerDe and partition level serde is of different SerDe
[ https://issues.apache.org/jira/browse/HIVE-6785?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13962482#comment-13962482 ] Brock Noland commented on HIVE-6785: Hi, LGTM except I see we are using the parquet... class names when creating a table, which are soon to be removed. query fails when partitioned table's table level serde is ParquetHiveSerDe and partition level serde is of different SerDe -- Key: HIVE-6785 URL: https://issues.apache.org/jira/browse/HIVE-6785 Project: Hive Issue Type: Bug Components: File Formats, Serializers/Deserializers Affects Versions: 0.13.0 Reporter: Tongjie Chen Fix For: 0.14.0 Attachments: HIVE-6785.1.patch.txt, HIVE-6785.2.patch.txt When a hive table's SerDe is ParquetHiveSerDe, while some partitions are of other SerDe, AND if this table has string column[s], hive generates confusing error message: Failed with exception java.io.IOException:java.lang.ClassCastException: parquet.hive.serde.primitive.ParquetStringInspector cannot be cast to org.apache.hadoop.hive.serde2.objectinspector.primitive.SettableTimestampObjectInspector This is confusing because timestamp is mentioned even if it is not been used by the table. The reason is when there is SerDe difference between table and partition, hive tries to convert objectinspector of two SerDes. ParquetHiveSerDe's object inspector for string type is ParquetStringInspector (newly introduced), neither a subclass of WritableStringObjectInspector nor JavaStringObjectInspector, which ObjectInspectorConverters expect for string category objector inspector. There is no break statement in STRING case statement, hence the following TIMESTAMP case statement is executed, generating confusing error message. see also in the following parquet issue: https://github.com/Parquet/parquet-mr/issues/324 To fix that it is relatively easy, just make ParquetStringInspector subclass of JavaStringObjectInspector instead of AbstractPrimitiveJavaObjectInspector. But because constructor of class JavaStringObjectInspector is package scope instead of public or protected, we would need to move ParquetStringInspector to the same package with JavaStringObjectInspector. Also ArrayWritableObjectInspector's setStructFieldData needs to also accept List data, since the corresponding setStructFieldData and create methods return a list. This is also needed when table SerDe is ParquetHiveSerDe, and partition SerDe is something else. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HIVE-1608) use sequencefile as the default for storing intermediate results
[ https://issues.apache.org/jira/browse/HIVE-1608?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13962487#comment-13962487 ] Brock Noland commented on HIVE-1608: A couple of notes on this one: 1) Most of the test failures look to be related to the .q.out files being different (referencing TextFile output class not SequenceFile) 2) This change as-is would be backwards incompatible for INSERT OVERWRITE DIRECTORY users. Thus I think we need: 1) Leave the default of TextFile for INSERT OVERWRITE DIRECTORY 2) Update the .q.out files use sequencefile as the default for storing intermediate results Key: HIVE-1608 URL: https://issues.apache.org/jira/browse/HIVE-1608 Project: Hive Issue Type: Bug Components: Query Processor Affects Versions: 0.7.0 Reporter: Namit Jain Assignee: Brock Noland Fix For: 0.14.0 Attachments: HIVE-1608.patch The only argument for having a text file for storing intermediate results seems to be better debuggability. But, tailing a sequence file is possible, and it should be more space efficient -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HIVE-6785) query fails when partitioned table's table level serde is ParquetHiveSerDe and partition level serde is of different SerDe
[ https://issues.apache.org/jira/browse/HIVE-6785?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13962493#comment-13962493 ] Szehon Ho commented on HIVE-6785: - Good catch Brock, I missed that. query fails when partitioned table's table level serde is ParquetHiveSerDe and partition level serde is of different SerDe -- Key: HIVE-6785 URL: https://issues.apache.org/jira/browse/HIVE-6785 Project: Hive Issue Type: Bug Components: File Formats, Serializers/Deserializers Affects Versions: 0.13.0 Reporter: Tongjie Chen Fix For: 0.14.0 Attachments: HIVE-6785.1.patch.txt, HIVE-6785.2.patch.txt When a hive table's SerDe is ParquetHiveSerDe, while some partitions are of other SerDe, AND if this table has string column[s], hive generates confusing error message: Failed with exception java.io.IOException:java.lang.ClassCastException: parquet.hive.serde.primitive.ParquetStringInspector cannot be cast to org.apache.hadoop.hive.serde2.objectinspector.primitive.SettableTimestampObjectInspector This is confusing because timestamp is mentioned even if it is not been used by the table. The reason is when there is SerDe difference between table and partition, hive tries to convert objectinspector of two SerDes. ParquetHiveSerDe's object inspector for string type is ParquetStringInspector (newly introduced), neither a subclass of WritableStringObjectInspector nor JavaStringObjectInspector, which ObjectInspectorConverters expect for string category objector inspector. There is no break statement in STRING case statement, hence the following TIMESTAMP case statement is executed, generating confusing error message. see also in the following parquet issue: https://github.com/Parquet/parquet-mr/issues/324 To fix that it is relatively easy, just make ParquetStringInspector subclass of JavaStringObjectInspector instead of AbstractPrimitiveJavaObjectInspector. But because constructor of class JavaStringObjectInspector is package scope instead of public or protected, we would need to move ParquetStringInspector to the same package with JavaStringObjectInspector. Also ArrayWritableObjectInspector's setStructFieldData needs to also accept List data, since the corresponding setStructFieldData and create methods return a list. This is also needed when table SerDe is ParquetHiveSerDe, and partition SerDe is something else. -- This message was sent by Atlassian JIRA (v6.2#6252)
Re: Review Request 19984: Beeline should accept -i option to Initializing a SQL file
---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/19984/
---
(Updated April 8, 2014, 2:07 a.m.)
Review request for hive.
Changes
---
addressed comment
Bugs: HIVE-6561
https://issues.apache.org/jira/browse/HIVE-6561
Repository: hive-git
Description
---
Hive CLI has -i option. From Hive CLI help:
{code}
...
-i filenameInitialization SQL file
...
{code}
However, Beeline has no such option:
{code}
xzhang@xzlt:~/apa/hive3$
./packaging/target/apache-hive-0.14.0-SNAPSHOT-bin/apache-hive-0.14.0-SNAPSHOT-bin/bin/beeline
-u jdbc:hive2:// -i hive.rc
...
Connected to: Apache Hive (version 0.14.0-SNAPSHOT)
Driver: Hive JDBC (version 0.14.0-SNAPSHOT)
Transaction isolation: TRANSACTION_REPEATABLE_READ
-i (No such file or directory)
Property url is required
Beeline version 0.14.0-SNAPSHOT by Apache Hive
...
{code}
Diffs (updated)
-
beeline/src/java/org/apache/hive/beeline/BeeLine.java 5773109
beeline/src/java/org/apache/hive/beeline/BeeLineOpts.java 44cabdf
beeline/src/java/org/apache/hive/beeline/Commands.java 493f963
beeline/src/main/resources/BeeLine.properties 697c29a
Diff: https://reviews.apache.org/r/19984/diff/
Testing
---
Thanks,
Navis Ryu
[jira] [Updated] (HIVE-6561) Beeline should accept -i option to Initializing a SQL file
[
https://issues.apache.org/jira/browse/HIVE-6561?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Navis updated HIVE-6561:
Attachment: HIVE-6561.2.patch.txt
Beeline should accept -i option to Initializing a SQL file
--
Key: HIVE-6561
URL: https://issues.apache.org/jira/browse/HIVE-6561
Project: Hive
Issue Type: Improvement
Affects Versions: 0.10.0, 0.11.0, 0.12.0
Reporter: Xuefu Zhang
Assignee: Navis
Attachments: HIVE-6561.1.patch.txt, HIVE-6561.2.patch.txt
Hive CLI has -i option. From Hive CLI help:
{code}
...
-i filenameInitialization SQL file
...
{code}
However, Beeline has no such option:
{code}
xzhang@xzlt:~/apa/hive3$
./packaging/target/apache-hive-0.14.0-SNAPSHOT-bin/apache-hive-0.14.0-SNAPSHOT-bin/bin/beeline
-u jdbc:hive2:// -i hive.rc
...
Connected to: Apache Hive (version 0.14.0-SNAPSHOT)
Driver: Hive JDBC (version 0.14.0-SNAPSHOT)
Transaction isolation: TRANSACTION_REPEATABLE_READ
-i (No such file or directory)
Property url is required
Beeline version 0.14.0-SNAPSHOT by Apache Hive
...
{code}
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Created] (HIVE-6861) more hadoop2 only golden files to fix
Jason Dere created HIVE-6861: Summary: more hadoop2 only golden files to fix Key: HIVE-6861 URL: https://issues.apache.org/jira/browse/HIVE-6861 Project: Hive Issue Type: Bug Components: Tests Reporter: Jason Dere Assignee: Jason Dere More hadoop2 golden files to fix due to HIVE-6643, HIVE-6642, HIVE-6808, HIVE-6144. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HIVE-6861) more hadoop2 only golden files to fix
[ https://issues.apache.org/jira/browse/HIVE-6861?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jason Dere updated HIVE-6861: - Status: Patch Available (was: Open) more hadoop2 only golden files to fix - Key: HIVE-6861 URL: https://issues.apache.org/jira/browse/HIVE-6861 Project: Hive Issue Type: Bug Components: Tests Reporter: Jason Dere Assignee: Jason Dere Attachments: HIVE-6861.1.patch More hadoop2 golden files to fix due to HIVE-6643, HIVE-6642, HIVE-6808, HIVE-6144. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HIVE-6861) more hadoop2 only golden files to fix
[ https://issues.apache.org/jira/browse/HIVE-6861?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jason Dere updated HIVE-6861: - Attachment: HIVE-6861.1.patch patch v1. more hadoop2 only golden files to fix - Key: HIVE-6861 URL: https://issues.apache.org/jira/browse/HIVE-6861 Project: Hive Issue Type: Bug Components: Tests Reporter: Jason Dere Assignee: Jason Dere Attachments: HIVE-6861.1.patch More hadoop2 golden files to fix due to HIVE-6643, HIVE-6642, HIVE-6808, HIVE-6144. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HIVE-6319) Insert, update, delete functionality needs a compactor
[
https://issues.apache.org/jira/browse/HIVE-6319?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13962543#comment-13962543
]
Hive QA commented on HIVE-6319:
---
{color:green}Overall{color}: +1 all checks pass
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12639048/HIVE-6319.patch
{color:green}SUCCESS:{color} +1 5591 tests passed
Test results:
http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/2169/testReport
Console output:
http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/2169/console
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12639048
Insert, update, delete functionality needs a compactor
--
Key: HIVE-6319
URL: https://issues.apache.org/jira/browse/HIVE-6319
Project: Hive
Issue Type: Sub-task
Reporter: Alan Gates
Assignee: Alan Gates
Fix For: 0.13.0
Attachments: 6319.wip.patch, HIVE-6319.patch, HIVE-6319.patch,
HIVE-6319.patch, HIVE-6319.patch, HiveCompactorDesign.pdf
In order to keep the number of delta files from spiraling out of control we
need a compactor to collect these delta files together, and eventually
rewrite the base file when the deltas get large enough.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HIVE-6782) HiveServer2Concurrency issue when running with tez intermittently, throwing org.apache.tez.dag.api.SessionNotRunning: Application not running error
[ https://issues.apache.org/jira/browse/HIVE-6782?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13962554#comment-13962554 ] Lefty Leverenz commented on HIVE-6782: -- This adds *hive.localize.resource.wait.interval* and *hive.localize.resource.num.wait.attempts* to HiveConf.java. They need descriptions in hive-default.xml.template or in a release note (since hive-default.xml.template will be generated from the new HiveConf.java after HIVE-6037 gets committed). When the time comes, I'll add them to the post-HIVE-6037 list (in HIVE-6586) and put them in the Configuration Properties wikidoc. HiveServer2Concurrency issue when running with tez intermittently, throwing org.apache.tez.dag.api.SessionNotRunning: Application not running error - Key: HIVE-6782 URL: https://issues.apache.org/jira/browse/HIVE-6782 Project: Hive Issue Type: Bug Components: Tez Reporter: Vikram Dixit K Assignee: Vikram Dixit K Fix For: 0.13.0, 0.14.0 Attachments: HIVE-6782.1.patch, HIVE-6782.10.patch, HIVE-6782.2.patch, HIVE-6782.3.patch, HIVE-6782.4.patch, HIVE-6782.5.patch, HIVE-6782.6.patch, HIVE-6782.7.patch, HIVE-6782.8.patch, HIVE-6782.9.patch HiveServer2 concurrency is failing intermittently when using tez, throwing org.apache.tez.dag.api.SessionNotRunning: Application not running error -- This message was sent by Atlassian JIRA (v6.2#6252)
