[jira] [Commented] (HIVE-6356) Dependency injection in hbase storage handler is broken

[Navis (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-6356?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13901173#comment-13901173
 ] 

Navis commented on HIVE-6356:
-

Ah, you are using hbase-0.96.1, which is applied HBASE-9165. I got it. Hive in 
trunk uses hbase-0.96.0 (addHBaseDependencyJars() does not exist in it).

Should we upgrade hbase to 0.96.1? I prefer to support both versions by 
applying patch3 but I'm good with any way.

> Dependency injection in hbase storage handler is broken
> ---
>
> Key: HIVE-6356
> URL: https://issues.apache.org/jira/browse/HIVE-6356
> Project: Hive
>  Issue Type: Bug
>  Components: HBase Handler
>Reporter: Navis
>Assignee: Navis
>Priority: Minor
> Fix For: 0.13.0
>
> Attachments: HIVE-6356.1.patch.txt, HIVE-6356.2.patch.txt, 
> HIVE-6356.3.patch.txt, HIVE-6356.addendum.00.patch
>
>
> Dependent jars for hbase is not added to tmpjars, which is caused by the 
> change of method signature(TableMapReduceUtil.addDependencyJars).



--
This message was sent by Atlassian JIRA
(v6.1.5#6160)

[jira] [Commented] (HIVE-6411) Support more generic way of using composite key for HBaseHandler

[Brock Noland (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-6411?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13901166#comment-13901166
 ] 

Brock Noland commented on HIVE-6411:


I also don't think the test failures are related. [~xuefuz] also looked at the 
hbase composite key changes IIRC.

> Support more generic way of using composite key for HBaseHandler
> 
>
> Key: HIVE-6411
> URL: https://issues.apache.org/jira/browse/HIVE-6411
> Project: Hive
>  Issue Type: Improvement
>  Components: HBase Handler
>Reporter: Navis
>Assignee: Navis
>Priority: Minor
> Attachments: HIVE-6411.1.patch.txt
>
>
> HIVE-2599 introduced using custom object for the row key. But it forces key 
> objects to extend HBaseCompositeKey, which is again extension of LazyStruct. 
> If user provides proper Object and OI, we can replace internal key and keyOI 
> with those. 
> Initial implementation is based on factory interface.
> {code}
> public interface HBaseKeyFactory {
>   void init(SerDeParameters parameters, Properties properties) throws 
> SerDeException;
>   ObjectInspector createObjectInspector(TypeInfo type) throws SerDeException;
>   LazyObjectBase createObject(ObjectInspector inspector) throws 
> SerDeException;
> }
> {code}



--
This message was sent by Atlassian JIRA
(v6.1.5#6160)

Re: hive precommit tests on bigtop jenkins

[Brock Noland]

Hi,

The pre-commit tests:

1) only test the latest attachment
2) post the attachment id to the JIRA
3) Verify the attachment id has not been tested before running

This means that even if a JIRA is in the queue twice it will only be tested
once.

Below are relevant portions of the script:

curl -s -S --location --retry 3 "${JIRA_ROOT_URL}/jira/browse/${JIRA_NAME}"
> $JIRA_TEXT
...
PATCH_URL=$(grep -o '"/jira/secure/attachment/[0-9]*/[^"]*' $JIRA_TEXT | \
  grep -v -e 'htm[l]*$' | sort | tail -1 | \
  grep -o '/jira/secure/attachment/[0-9]*/[^"]*')
...
# ensure attachment has not already been tested
ATTACHMENT_ID=$(basename $(dirname $PATCH_URL))
if grep -q "ATTACHMENT ID: $ATTACHMENT_ID" $JIRA_TEXT
then
  echo "Attachment $ATTACHMENT_ID is already tested for $JIRA_NAME"
  exit 1
fi





On Fri, Feb 14, 2014 at 12:51 AM, Navis류승우  wrote:

> Recently, precommit test takes more than 1 day (including queue time).
>
> Deduping work queue (currently, HIVE-6403 and HIVE-6418 is queued twice)
> can make this better. Rewriting some test queries simpler (I'm thinking of
> removing order-by clauses just for conforming purpose). Any other ideas?
>
>
> 2014-02-14 6:46 GMT+09:00 Thejas Nair :
>
> > I see a new job now running there. Maybe there is nothing wrong with the
> > infra and builds actually finished (except for the 3 aborted ones).
> > Can't complain about a shorter queue ! :)
> >
> >
> >
> > On Thu, Feb 13, 2014 at 1:30 PM, Thejas Nair 
> > wrote:
> >
> > > Is the jenkins infra used for hive precommit tests under maintenance
>  ? I
> > > see that the long queue has suddenly disappeared. The last few test
> > builds
> > > have been aborted.
> > >
> > > The jenkins used for hive precommit tests -
> > > http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/
> > >
> > > Thanks,
> > > Thejas
> > >
> > >
> > >
> > >
> >
> > --
> > CONFIDENTIALITY NOTICE
> > NOTICE: This message is intended for the use of the individual or entity
> to
> > which it is addressed and may contain information that is confidential,
> > privileged and exempt from disclosure under applicable law. If the reader
> > of this message is not the intended recipient, you are hereby notified
> that
> > any printing, copying, dissemination, distribution, disclosure or
> > forwarding of this communication is strictly prohibited. If you have
> > received this communication in error, please contact the sender
> immediately
> > and delete it from your system. Thank You.
> >
>



-- 
Apache MRUnit - Unit testing MapReduce - http://mrunit.apache.org

[jira] [Commented] (HIVE-6411) Support more generic way of using composite key for HBaseHandler

[Navis (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-6411?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13901164#comment-13901164
 ] 

Navis commented on HIVE-6411:
-

Cannot reproduce fail of TestWriteLock. authorization_revoke_table_priv is 
resolved in other issue. [~brocknoland], [~swarnim], Could you review this? (I 
cannot make review board entry by 502 error)

> Support more generic way of using composite key for HBaseHandler
> 
>
> Key: HIVE-6411
> URL: https://issues.apache.org/jira/browse/HIVE-6411
> Project: Hive
>  Issue Type: Improvement
>  Components: HBase Handler
>Reporter: Navis
>Assignee: Navis
>Priority: Minor
> Attachments: HIVE-6411.1.patch.txt
>
>
> HIVE-2599 introduced using custom object for the row key. But it forces key 
> objects to extend HBaseCompositeKey, which is again extension of LazyStruct. 
> If user provides proper Object and OI, we can replace internal key and keyOI 
> with those. 
> Initial implementation is based on factory interface.
> {code}
> public interface HBaseKeyFactory {
>   void init(SerDeParameters parameters, Properties properties) throws 
> SerDeException;
>   ObjectInspector createObjectInspector(TypeInfo type) throws SerDeException;
>   LazyObjectBase createObject(ObjectInspector inspector) throws 
> SerDeException;
> }
> {code}



--
This message was sent by Atlassian JIRA
(v6.1.5#6160)

Re: hive precommit tests on bigtop jenkins

[Navis류승우]

Recently, precommit test takes more than 1 day (including queue time).

Deduping work queue (currently, HIVE-6403 and HIVE-6418 is queued twice)
can make this better. Rewriting some test queries simpler (I'm thinking of
removing order-by clauses just for conforming purpose). Any other ideas?


2014-02-14 6:46 GMT+09:00 Thejas Nair :

> I see a new job now running there. Maybe there is nothing wrong with the
> infra and builds actually finished (except for the 3 aborted ones).
> Can't complain about a shorter queue ! :)
>
>
>
> On Thu, Feb 13, 2014 at 1:30 PM, Thejas Nair 
> wrote:
>
> > Is the jenkins infra used for hive precommit tests under maintenance  ? I
> > see that the long queue has suddenly disappeared. The last few test
> builds
> > have been aborted.
> >
> > The jenkins used for hive precommit tests -
> > http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/
> >
> > Thanks,
> > Thejas
> >
> >
> >
> >
>
> --
> CONFIDENTIALITY NOTICE
> NOTICE: This message is intended for the use of the individual or entity to
> which it is addressed and may contain information that is confidential,
> privileged and exempt from disclosure under applicable law. If the reader
> of this message is not the intended recipient, you are hereby notified that
> any printing, copying, dissemination, distribution, disclosure or
> forwarding of this communication is strictly prohibited. If you have
> received this communication in error, please contact the sender immediately
> and delete it from your system. Thank You.
>

[jira] [Updated] (HIVE-6037) Synchronize HiveConf with hive-default.xml.template and support show conf

[Navis (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-6037?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Navis updated HIVE-6037:


Attachment: HIVE-6037.9.patch.txt

Rebased to trunk

> Synchronize HiveConf with hive-default.xml.template and support show conf
> -
>
> Key: HIVE-6037
> URL: https://issues.apache.org/jira/browse/HIVE-6037
> Project: Hive
>  Issue Type: Improvement
>  Components: Configuration
>Reporter: Navis
>Assignee: Navis
>Priority: Minor
> Attachments: CHIVE-6037.3.patch.txt, HIVE-6037.1.patch.txt, 
> HIVE-6037.2.patch.txt, HIVE-6037.4.patch.txt, HIVE-6037.5.patch.txt, 
> HIVE-6037.6.patch.txt, HIVE-6037.7.patch.txt, HIVE-6037.8.patch.txt, 
> HIVE-6037.9.patch.txt
>
>
> see HIVE-5879



--
This message was sent by Atlassian JIRA
(v6.1.5#6160)

[jira] [Updated] (HIVE-6037) Synchronize HiveConf with hive-default.xml.template and support show conf

[Navis (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-6037?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Navis updated HIVE-6037:


Attachment: (was: HIVE-6037.9.patch.txt)

> Synchronize HiveConf with hive-default.xml.template and support show conf
> -
>
> Key: HIVE-6037
> URL: https://issues.apache.org/jira/browse/HIVE-6037
> Project: Hive
>  Issue Type: Improvement
>  Components: Configuration
>Reporter: Navis
>Assignee: Navis
>Priority: Minor
> Attachments: CHIVE-6037.3.patch.txt, HIVE-6037.1.patch.txt, 
> HIVE-6037.2.patch.txt, HIVE-6037.4.patch.txt, HIVE-6037.5.patch.txt, 
> HIVE-6037.6.patch.txt, HIVE-6037.7.patch.txt, HIVE-6037.8.patch.txt
>
>
> see HIVE-5879



--
This message was sent by Atlassian JIRA
(v6.1.5#6160)

[jira] [Commented] (HIVE-6203) Privileges of role granted indrectily to user is not applied

[Navis (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-6203?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13901140#comment-13901140
 ] 

Navis commented on HIVE-6203:
-

Yes, it's for minimizing the code difference. If test passes, I can rewrite it 
as you suggested.

> Privileges of role granted indrectily to user is not applied
> 
>
> Key: HIVE-6203
> URL: https://issues.apache.org/jira/browse/HIVE-6203
> Project: Hive
>  Issue Type: Bug
>  Components: Authorization
>Reporter: Navis
>Assignee: Navis
> Attachments: HIVE-6203.1.patch.txt, HIVE-6203.2.patch.txt
>
>
> For example, 
> {noformat}
> create role r1;
> create role r2;
> grant select on table eq to role r1;
> grant role r1 to role r2;
> grant role r2 to user admin;
> select * from eq limit 5;
> {noformat}
> admin -> r2 -> r1 -> SEL on table eq
> but user admin fails to access table eq



--
This message was sent by Atlassian JIRA
(v6.1.5#6160)

[jira] [Updated] (HIVE-6203) Privileges of role granted indrectily to user is not applied

[Navis (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-6203?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Navis updated HIVE-6203:


Attachment: HIVE-6203.2.patch.txt

> Privileges of role granted indrectily to user is not applied
> 
>
> Key: HIVE-6203
> URL: https://issues.apache.org/jira/browse/HIVE-6203
> Project: Hive
>  Issue Type: Bug
>  Components: Authorization
>Reporter: Navis
>Assignee: Navis
> Attachments: HIVE-6203.1.patch.txt, HIVE-6203.2.patch.txt
>
>
> For example, 
> {noformat}
> create role r1;
> create role r2;
> grant select on table eq to role r1;
> grant role r1 to role r2;
> grant role r2 to user admin;
> select * from eq limit 5;
> {noformat}
> admin -> r2 -> r1 -> SEL on table eq
> but user admin fails to access table eq



--
This message was sent by Atlassian JIRA
(v6.1.5#6160)

[jira] [Commented] (HIVE-6403) uncorrelated subquery is failing with auto.convert.join=true

[Navis (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-6403?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13901136#comment-13901136
 ] 

Navis commented on HIVE-6403:
-

[~rhbutani] Took some time to understand multiInsertBigTableCheck() and now got 
it. But seeing auto_join17.q, I've realized that we missed handling map-side 
union cases. Updated in navis2.patch. Could you check that too?

> uncorrelated subquery is failing with auto.convert.join=true
> 
>
> Key: HIVE-6403
> URL: https://issues.apache.org/jira/browse/HIVE-6403
> Project: Hive
>  Issue Type: Bug
>  Components: Query Processor
>Reporter: Navis
>Assignee: Harish Butani
> Attachments: HIVE-6403.1.patch, HIVE-6403.2.patch, navis.patch, 
> navis2.patch
>
>
> Fixing HIVE-5690, I've found query in subquery_multiinsert.q is not working 
> with hive.auto.convert.join=true 
> {noformat}
> set hive.auto.convert.join=true;
> hive> explain
> > from src b 
> > INSERT OVERWRITE TABLE src_4 
> >   select * 
> >   where b.key in 
> >(select a.key 
> > from src a 
> > where b.value = a.value and a.key > '9'
> >) 
> > INSERT OVERWRITE TABLE src_5 
> >   select *  
> >   where b.key not in  ( select key from src s1 where s1.key > '2') 
> >   order by key 
> > ;
> java.lang.IndexOutOfBoundsException: Index: 0, Size: 0
>   at java.util.ArrayList.rangeCheck(ArrayList.java:635)
>   at java.util.ArrayList.get(ArrayList.java:411)
>   at 
> org.apache.hadoop.hive.ql.optimizer.MapJoinProcessor.genMapJoinLocalWork(MapJoinProcessor.java:149)
>   at 
> org.apache.hadoop.hive.ql.optimizer.MapJoinProcessor.genLocalWorkForMapJoin(MapJoinProcessor.java:256)
>   at 
> org.apache.hadoop.hive.ql.optimizer.MapJoinProcessor.genMapJoinOpAndLocalWork(MapJoinProcessor.java:248)
>   at 
> org.apache.hadoop.hive.ql.optimizer.physical.CommonJoinTaskDispatcher.convertTaskToMapJoinTask(CommonJoinTaskDispatcher.java:191)
>   at 
> org.apache.hadoop.hive.ql.optimizer.physical.CommonJoinTaskDispatcher.processCurrentTask(CommonJoinTaskDispatcher.java:481)
>   at 
> org.apache.hadoop.hive.ql.optimizer.physical.AbstractJoinTaskDispatcher.dispatch(AbstractJoinTaskDispatcher.java:182)
>   at 
> org.apache.hadoop.hive.ql.lib.TaskGraphWalker.dispatch(TaskGraphWalker.java:111)
>   at 
> org.apache.hadoop.hive.ql.lib.TaskGraphWalker.walk(TaskGraphWalker.java:194)
>   at 
> org.apache.hadoop.hive.ql.lib.TaskGraphWalker.startWalking(TaskGraphWalker.java:139)
>   at 
> org.apache.hadoop.hive.ql.optimizer.physical.CommonJoinResolver.resolve(CommonJoinResolver.java:79)
>   at 
> org.apache.hadoop.hive.ql.optimizer.physical.PhysicalOptimizer.optimize(PhysicalOptimizer.java:100)
>   at 
> org.apache.hadoop.hive.ql.parse.MapReduceCompiler.optimizeTaskPlan(MapReduceCompiler.java:290)
>   at 
> org.apache.hadoop.hive.ql.parse.TaskCompiler.compile(TaskCompiler.java:216)
>   at 
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeInternal(SemanticAnalyzer.java:9167)
>   at 
> org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:327)
>   at 
> org.apache.hadoop.hive.ql.parse.ExplainSemanticAnalyzer.analyzeInternal(ExplainSemanticAnalyzer.java:64)
>   at 
> org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:327)
>   at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:446)
>   at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:346)
>   at org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:1056)
>   at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1099)
>   at org.apache.hadoop.hive.ql.Driver.run(Driver.java:992)
>   at org.apache.hadoop.hive.ql.Driver.run(Driver.java:982)
>   at 
> org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:268)
>   at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:220)
>   at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:424)
>   at 
> org.apache.hadoop.hive.cli.CliDriver.executeDriver(CliDriver.java:793)
>   at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:687)
>   at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:626)
>   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>   at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
>   at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>   at java.lang.reflect.Method.invoke(Method.java:606)
>   at org.apache.hadoop.util.RunJar.main(RunJar.java:160)
> org.apache.hadoop.hive.ql.parse.SemanticException: Failed to generate new 
> mapJoin operator by exception : Index

[jira] [Updated] (HIVE-6403) uncorrelated subquery is failing with auto.convert.join=true

[Navis (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-6403?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Navis updated HIVE-6403:


Attachment: navis2.patch

> uncorrelated subquery is failing with auto.convert.join=true
> 
>
> Key: HIVE-6403
> URL: https://issues.apache.org/jira/browse/HIVE-6403
> Project: Hive
>  Issue Type: Bug
>  Components: Query Processor
>Reporter: Navis
>Assignee: Harish Butani
> Attachments: HIVE-6403.1.patch, HIVE-6403.2.patch, navis.patch, 
> navis2.patch
>
>
> Fixing HIVE-5690, I've found query in subquery_multiinsert.q is not working 
> with hive.auto.convert.join=true 
> {noformat}
> set hive.auto.convert.join=true;
> hive> explain
> > from src b 
> > INSERT OVERWRITE TABLE src_4 
> >   select * 
> >   where b.key in 
> >(select a.key 
> > from src a 
> > where b.value = a.value and a.key > '9'
> >) 
> > INSERT OVERWRITE TABLE src_5 
> >   select *  
> >   where b.key not in  ( select key from src s1 where s1.key > '2') 
> >   order by key 
> > ;
> java.lang.IndexOutOfBoundsException: Index: 0, Size: 0
>   at java.util.ArrayList.rangeCheck(ArrayList.java:635)
>   at java.util.ArrayList.get(ArrayList.java:411)
>   at 
> org.apache.hadoop.hive.ql.optimizer.MapJoinProcessor.genMapJoinLocalWork(MapJoinProcessor.java:149)
>   at 
> org.apache.hadoop.hive.ql.optimizer.MapJoinProcessor.genLocalWorkForMapJoin(MapJoinProcessor.java:256)
>   at 
> org.apache.hadoop.hive.ql.optimizer.MapJoinProcessor.genMapJoinOpAndLocalWork(MapJoinProcessor.java:248)
>   at 
> org.apache.hadoop.hive.ql.optimizer.physical.CommonJoinTaskDispatcher.convertTaskToMapJoinTask(CommonJoinTaskDispatcher.java:191)
>   at 
> org.apache.hadoop.hive.ql.optimizer.physical.CommonJoinTaskDispatcher.processCurrentTask(CommonJoinTaskDispatcher.java:481)
>   at 
> org.apache.hadoop.hive.ql.optimizer.physical.AbstractJoinTaskDispatcher.dispatch(AbstractJoinTaskDispatcher.java:182)
>   at 
> org.apache.hadoop.hive.ql.lib.TaskGraphWalker.dispatch(TaskGraphWalker.java:111)
>   at 
> org.apache.hadoop.hive.ql.lib.TaskGraphWalker.walk(TaskGraphWalker.java:194)
>   at 
> org.apache.hadoop.hive.ql.lib.TaskGraphWalker.startWalking(TaskGraphWalker.java:139)
>   at 
> org.apache.hadoop.hive.ql.optimizer.physical.CommonJoinResolver.resolve(CommonJoinResolver.java:79)
>   at 
> org.apache.hadoop.hive.ql.optimizer.physical.PhysicalOptimizer.optimize(PhysicalOptimizer.java:100)
>   at 
> org.apache.hadoop.hive.ql.parse.MapReduceCompiler.optimizeTaskPlan(MapReduceCompiler.java:290)
>   at 
> org.apache.hadoop.hive.ql.parse.TaskCompiler.compile(TaskCompiler.java:216)
>   at 
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeInternal(SemanticAnalyzer.java:9167)
>   at 
> org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:327)
>   at 
> org.apache.hadoop.hive.ql.parse.ExplainSemanticAnalyzer.analyzeInternal(ExplainSemanticAnalyzer.java:64)
>   at 
> org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:327)
>   at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:446)
>   at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:346)
>   at org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:1056)
>   at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1099)
>   at org.apache.hadoop.hive.ql.Driver.run(Driver.java:992)
>   at org.apache.hadoop.hive.ql.Driver.run(Driver.java:982)
>   at 
> org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:268)
>   at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:220)
>   at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:424)
>   at 
> org.apache.hadoop.hive.cli.CliDriver.executeDriver(CliDriver.java:793)
>   at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:687)
>   at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:626)
>   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>   at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
>   at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>   at java.lang.reflect.Method.invoke(Method.java:606)
>   at org.apache.hadoop.util.RunJar.main(RunJar.java:160)
> org.apache.hadoop.hive.ql.parse.SemanticException: Failed to generate new 
> mapJoin operator by exception : Index: 0, Size: 0
>   at 
> org.apache.hadoop.hive.ql.optimizer.MapJoinProcessor.genLocalWorkForMapJoin(MapJoinProcessor.java:266)
>   at 
> org.apache.hadoop.hive.ql.optimizer.MapJoinProcessor.genMapJoinOpAndLocalWork(MapJoinProcessor.java:248)
> 

[jira] [Commented] (HIVE-6410) Allow output serializations separators to be set for HDFS path as well.

[Amareshwari Sriramadasu (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-6410?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13901100#comment-13901100
 ] 

Amareshwari Sriramadasu commented on HIVE-6410:
---

[~xuefu.w...@kodak.com], had the patch ready, so uploaded. I don't mind any of 
them being closed as duplicate of other, as long as code gets in.

> Allow output serializations separators to be set for HDFS path as well.
> ---
>
> Key: HIVE-6410
> URL: https://issues.apache.org/jira/browse/HIVE-6410
> Project: Hive
>  Issue Type: Improvement
>  Components: Query Processor
>Reporter: Amareshwari Sriramadasu
>Assignee: Amareshwari Sriramadasu
> Attachments: HIVE-6410.patch
>
>
> HIVE-3682 adds functionality for users to set serialization constants for 
> 'insert overwrite local directory'. The same functionality should be 
> available for hdfs path as well. The workaround suggested is to create a 
> table with required format and insert into the table, which enforces the 
> users to know the schema of the result and create the table ahead. Though 
> that works, it is good to have the functionality for loading into directory 
> as well.
> I'm planning to add the same functionality in 'insert overwrite directory' in 
> this jira.



--
This message was sent by Atlassian JIRA
(v6.1.5#6160)

[jira] [Commented] (HIVE-6250) sql std auth - view authorization should not underlying table. More tests and fixes.

[Ashutosh Chauhan (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-6250?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13901095#comment-13901095
 ] 

Ashutosh Chauhan commented on HIVE-6250:


+1 Nice coverage in test cases. Good work, Thejas! 

> sql std auth - view authorization should not underlying table. More tests and 
> fixes.
> 
>
> Key: HIVE-6250
> URL: https://issues.apache.org/jira/browse/HIVE-6250
> Project: Hive
>  Issue Type: Sub-task
>  Components: Authorization
>Reporter: Thejas M Nair
>Assignee: Thejas M Nair
> Attachments: HIVE-6250.1.patch, HIVE-6250.2.patch
>
>   Original Estimate: 24h
>  Remaining Estimate: 24h
>
> This patch adds more tests for table and view authorization and also fixes a 
> number of issues found during testing -
> - View authorization should happen on only on the view, and not the 
> underlying table (Change in ReadEntity to indicate if it is a direct/indirect 
> dependency)
> - table owner in metadata should be the user as per SessionState 
> authentication provider
> - added utility function for finding the session state authentication 
> provider user
> - authorization should be based on current roles
> - admin user should have all permissions
> - error message improvements



--
This message was sent by Atlassian JIRA
(v6.1.5#6160)

[jira] [Updated] (HIVE-6430) MapJoin hash table has large memory overhead

[Sergey Shelukhin (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-6430?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sergey Shelukhin updated HIVE-6430:
---

Summary: MapJoin hash table has large memory overhead  (was:  HIVE-6429 - 
MapJoin hash table has large memory overhead)

> MapJoin hash table has large memory overhead
> 
>
> Key: HIVE-6430
> URL: https://issues.apache.org/jira/browse/HIVE-6430
> Project: Hive
>  Issue Type: Improvement
>Reporter: Sergey Shelukhin
>
> Right now, in some queries, I see that storing e.g. 4 ints (2 for key and 2 
> for row) can take several hundred bytes, which is ridiculous. I am reducing 
> the size of MJKey and MJRowContainer in other jiras, but in general we don't 
> need to have java hash table there.  We can either use primitive-friendly 
> hashtable like the one from HPPC (Apache-licenced), or some variation, to map 
> primitive keys to single row storage structure without an object per row 
> (similar to vectorization).



--
This message was sent by Atlassian JIRA
(v6.1.5#6160)

[jira] [Created] (HIVE-6430) HIVE-6429 - MapJoin hash table has large memory overhead

[Sergey Shelukhin (JIRA)]

Sergey Shelukhin created HIVE-6430:
--

 Summary:  HIVE-6429 - MapJoin hash table has large memory overhead
 Key: HIVE-6430
 URL: https://issues.apache.org/jira/browse/HIVE-6430
 Project: Hive
  Issue Type: Improvement
Reporter: Sergey Shelukhin


Right now, in some queries, I see that storing e.g. 4 ints (2 for key and 2 for 
row) can take several hundred bytes, which is ridiculous. I am reducing the 
size of MJKey and MJRowContainer in other jiras, but in general we don't need 
to have java hash table there.  We can either use primitive-friendly hashtable 
like the one from HPPC (Apache-licenced), or some variation, to map primitive 
keys to single row storage structure without an object per row (similar to 
vectorization).



--
This message was sent by Atlassian JIRA
(v6.1.5#6160)

[jira] [Created] (HIVE-6429) MapJoinKey has large memory overhead in typical cases

[Sergey Shelukhin (JIRA)]

Sergey Shelukhin created HIVE-6429:
--

 Summary: MapJoinKey has large memory overhead in typical cases
 Key: HIVE-6429
 URL: https://issues.apache.org/jira/browse/HIVE-6429
 Project: Hive
  Issue Type: Improvement
Reporter: Sergey Shelukhin
Assignee: Sergey Shelukhin


The only thing that MJK really needs it hashCode and equals (well, and 
construction), so there's no need to have array of writables in there. Assuming 
all the keys for a table have the same structure, for the common case where 
keys are primitive types, we can store something like a byte array combination 
of keys to reduce the memory usage. Will probably speed up compares too.



--
This message was sent by Atlassian JIRA
(v6.1.5#6160)

[jira] [Updated] (HIVE-6400) Add Apache Rat plugin to pom.xml

[Harish Butani (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-6400?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Harish Butani updated HIVE-6400:


Attachment: HIVE-6400.1.patch

> Add Apache Rat plugin to pom.xml
> 
>
> Key: HIVE-6400
> URL: https://issues.apache.org/jira/browse/HIVE-6400
> Project: Hive
>  Issue Type: Bug
>Reporter: Harish Butani
>Assignee: Harish Butani
> Attachments: HIVE-6400.1.patch
>
>
> So we can generate the Release Audit report. 



--
This message was sent by Atlassian JIRA
(v6.1.5#6160)

[jira] [Updated] (HIVE-6400) Add Apache Rat plugin to pom.xml

[Harish Butani (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-6400?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Harish Butani updated HIVE-6400:


Status: Patch Available  (was: Open)

> Add Apache Rat plugin to pom.xml
> 
>
> Key: HIVE-6400
> URL: https://issues.apache.org/jira/browse/HIVE-6400
> Project: Hive
>  Issue Type: Bug
>Reporter: Harish Butani
>Assignee: Harish Butani
> Attachments: HIVE-6400.1.patch
>
>
> So we can generate the Release Audit report. 



--
This message was sent by Atlassian JIRA
(v6.1.5#6160)

[jira] [Updated] (HIVE-6400) Add Apache Rat plugin to pom.xml

[Harish Butani (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-6400?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Harish Butani updated HIVE-6400:


Attachment: (was: HIVE-6400.1.patch)

> Add Apache Rat plugin to pom.xml
> 
>
> Key: HIVE-6400
> URL: https://issues.apache.org/jira/browse/HIVE-6400
> Project: Hive
>  Issue Type: Bug
>Reporter: Harish Butani
>Assignee: Harish Butani
>
> So we can generate the Release Audit report. 



--
This message was sent by Atlassian JIRA
(v6.1.5#6160)

[jira] [Updated] (HIVE-6400) Add Apache Rat plugin to pom.xml

[Harish Butani (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-6400?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Harish Butani updated HIVE-6400:


Status: Open  (was: Patch Available)

> Add Apache Rat plugin to pom.xml
> 
>
> Key: HIVE-6400
> URL: https://issues.apache.org/jira/browse/HIVE-6400
> Project: Hive
>  Issue Type: Bug
>Reporter: Harish Butani
>Assignee: Harish Butani
>
> So we can generate the Release Audit report. 



--
This message was sent by Atlassian JIRA
(v6.1.5#6160)

[jira] [Updated] (HIVE-6403) uncorrelated subquery is failing with auto.convert.join=true

[Harish Butani (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-6403?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Harish Butani updated HIVE-6403:


Status: Open  (was: Patch Available)

> uncorrelated subquery is failing with auto.convert.join=true
> 
>
> Key: HIVE-6403
> URL: https://issues.apache.org/jira/browse/HIVE-6403
> Project: Hive
>  Issue Type: Bug
>  Components: Query Processor
>Reporter: Navis
>Assignee: Harish Butani
> Attachments: HIVE-6403.1.patch, HIVE-6403.2.patch, navis.patch
>
>
> Fixing HIVE-5690, I've found query in subquery_multiinsert.q is not working 
> with hive.auto.convert.join=true 
> {noformat}
> set hive.auto.convert.join=true;
> hive> explain
> > from src b 
> > INSERT OVERWRITE TABLE src_4 
> >   select * 
> >   where b.key in 
> >(select a.key 
> > from src a 
> > where b.value = a.value and a.key > '9'
> >) 
> > INSERT OVERWRITE TABLE src_5 
> >   select *  
> >   where b.key not in  ( select key from src s1 where s1.key > '2') 
> >   order by key 
> > ;
> java.lang.IndexOutOfBoundsException: Index: 0, Size: 0
>   at java.util.ArrayList.rangeCheck(ArrayList.java:635)
>   at java.util.ArrayList.get(ArrayList.java:411)
>   at 
> org.apache.hadoop.hive.ql.optimizer.MapJoinProcessor.genMapJoinLocalWork(MapJoinProcessor.java:149)
>   at 
> org.apache.hadoop.hive.ql.optimizer.MapJoinProcessor.genLocalWorkForMapJoin(MapJoinProcessor.java:256)
>   at 
> org.apache.hadoop.hive.ql.optimizer.MapJoinProcessor.genMapJoinOpAndLocalWork(MapJoinProcessor.java:248)
>   at 
> org.apache.hadoop.hive.ql.optimizer.physical.CommonJoinTaskDispatcher.convertTaskToMapJoinTask(CommonJoinTaskDispatcher.java:191)
>   at 
> org.apache.hadoop.hive.ql.optimizer.physical.CommonJoinTaskDispatcher.processCurrentTask(CommonJoinTaskDispatcher.java:481)
>   at 
> org.apache.hadoop.hive.ql.optimizer.physical.AbstractJoinTaskDispatcher.dispatch(AbstractJoinTaskDispatcher.java:182)
>   at 
> org.apache.hadoop.hive.ql.lib.TaskGraphWalker.dispatch(TaskGraphWalker.java:111)
>   at 
> org.apache.hadoop.hive.ql.lib.TaskGraphWalker.walk(TaskGraphWalker.java:194)
>   at 
> org.apache.hadoop.hive.ql.lib.TaskGraphWalker.startWalking(TaskGraphWalker.java:139)
>   at 
> org.apache.hadoop.hive.ql.optimizer.physical.CommonJoinResolver.resolve(CommonJoinResolver.java:79)
>   at 
> org.apache.hadoop.hive.ql.optimizer.physical.PhysicalOptimizer.optimize(PhysicalOptimizer.java:100)
>   at 
> org.apache.hadoop.hive.ql.parse.MapReduceCompiler.optimizeTaskPlan(MapReduceCompiler.java:290)
>   at 
> org.apache.hadoop.hive.ql.parse.TaskCompiler.compile(TaskCompiler.java:216)
>   at 
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeInternal(SemanticAnalyzer.java:9167)
>   at 
> org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:327)
>   at 
> org.apache.hadoop.hive.ql.parse.ExplainSemanticAnalyzer.analyzeInternal(ExplainSemanticAnalyzer.java:64)
>   at 
> org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:327)
>   at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:446)
>   at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:346)
>   at org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:1056)
>   at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1099)
>   at org.apache.hadoop.hive.ql.Driver.run(Driver.java:992)
>   at org.apache.hadoop.hive.ql.Driver.run(Driver.java:982)
>   at 
> org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:268)
>   at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:220)
>   at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:424)
>   at 
> org.apache.hadoop.hive.cli.CliDriver.executeDriver(CliDriver.java:793)
>   at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:687)
>   at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:626)
>   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>   at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
>   at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>   at java.lang.reflect.Method.invoke(Method.java:606)
>   at org.apache.hadoop.util.RunJar.main(RunJar.java:160)
> org.apache.hadoop.hive.ql.parse.SemanticException: Failed to generate new 
> mapJoin operator by exception : Index: 0, Size: 0
>   at 
> org.apache.hadoop.hive.ql.optimizer.MapJoinProcessor.genLocalWorkForMapJoin(MapJoinProcessor.java:266)
>   at 
> org.apache.hadoop.hive.ql.optimizer.MapJoinProcessor.genMapJoinOpAndLocalWork(MapJoinProcessor.j

[jira] [Commented] (HIVE-6403) uncorrelated subquery is failing with auto.convert.join=true

[Harish Butani (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-6403?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13901088#comment-13901088
 ] 

Harish Butani commented on HIVE-6403:
-

Resubmitting patch. Ran several of the tests locally, they succeed.
The previous run was aborted.

> uncorrelated subquery is failing with auto.convert.join=true
> 
>
> Key: HIVE-6403
> URL: https://issues.apache.org/jira/browse/HIVE-6403
> Project: Hive
>  Issue Type: Bug
>  Components: Query Processor
>Reporter: Navis
>Assignee: Harish Butani
> Attachments: HIVE-6403.1.patch, HIVE-6403.2.patch, navis.patch
>
>
> Fixing HIVE-5690, I've found query in subquery_multiinsert.q is not working 
> with hive.auto.convert.join=true 
> {noformat}
> set hive.auto.convert.join=true;
> hive> explain
> > from src b 
> > INSERT OVERWRITE TABLE src_4 
> >   select * 
> >   where b.key in 
> >(select a.key 
> > from src a 
> > where b.value = a.value and a.key > '9'
> >) 
> > INSERT OVERWRITE TABLE src_5 
> >   select *  
> >   where b.key not in  ( select key from src s1 where s1.key > '2') 
> >   order by key 
> > ;
> java.lang.IndexOutOfBoundsException: Index: 0, Size: 0
>   at java.util.ArrayList.rangeCheck(ArrayList.java:635)
>   at java.util.ArrayList.get(ArrayList.java:411)
>   at 
> org.apache.hadoop.hive.ql.optimizer.MapJoinProcessor.genMapJoinLocalWork(MapJoinProcessor.java:149)
>   at 
> org.apache.hadoop.hive.ql.optimizer.MapJoinProcessor.genLocalWorkForMapJoin(MapJoinProcessor.java:256)
>   at 
> org.apache.hadoop.hive.ql.optimizer.MapJoinProcessor.genMapJoinOpAndLocalWork(MapJoinProcessor.java:248)
>   at 
> org.apache.hadoop.hive.ql.optimizer.physical.CommonJoinTaskDispatcher.convertTaskToMapJoinTask(CommonJoinTaskDispatcher.java:191)
>   at 
> org.apache.hadoop.hive.ql.optimizer.physical.CommonJoinTaskDispatcher.processCurrentTask(CommonJoinTaskDispatcher.java:481)
>   at 
> org.apache.hadoop.hive.ql.optimizer.physical.AbstractJoinTaskDispatcher.dispatch(AbstractJoinTaskDispatcher.java:182)
>   at 
> org.apache.hadoop.hive.ql.lib.TaskGraphWalker.dispatch(TaskGraphWalker.java:111)
>   at 
> org.apache.hadoop.hive.ql.lib.TaskGraphWalker.walk(TaskGraphWalker.java:194)
>   at 
> org.apache.hadoop.hive.ql.lib.TaskGraphWalker.startWalking(TaskGraphWalker.java:139)
>   at 
> org.apache.hadoop.hive.ql.optimizer.physical.CommonJoinResolver.resolve(CommonJoinResolver.java:79)
>   at 
> org.apache.hadoop.hive.ql.optimizer.physical.PhysicalOptimizer.optimize(PhysicalOptimizer.java:100)
>   at 
> org.apache.hadoop.hive.ql.parse.MapReduceCompiler.optimizeTaskPlan(MapReduceCompiler.java:290)
>   at 
> org.apache.hadoop.hive.ql.parse.TaskCompiler.compile(TaskCompiler.java:216)
>   at 
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeInternal(SemanticAnalyzer.java:9167)
>   at 
> org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:327)
>   at 
> org.apache.hadoop.hive.ql.parse.ExplainSemanticAnalyzer.analyzeInternal(ExplainSemanticAnalyzer.java:64)
>   at 
> org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:327)
>   at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:446)
>   at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:346)
>   at org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:1056)
>   at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1099)
>   at org.apache.hadoop.hive.ql.Driver.run(Driver.java:992)
>   at org.apache.hadoop.hive.ql.Driver.run(Driver.java:982)
>   at 
> org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:268)
>   at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:220)
>   at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:424)
>   at 
> org.apache.hadoop.hive.cli.CliDriver.executeDriver(CliDriver.java:793)
>   at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:687)
>   at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:626)
>   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>   at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
>   at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>   at java.lang.reflect.Method.invoke(Method.java:606)
>   at org.apache.hadoop.util.RunJar.main(RunJar.java:160)
> org.apache.hadoop.hive.ql.parse.SemanticException: Failed to generate new 
> mapJoin operator by exception : Index: 0, Size: 0
>   at 
> org.apache.hadoop.hive.ql.optimizer.MapJoinProcessor.genLocalWorkForMapJoin(MapJoinProcessor.java:2

[jira] [Updated] (HIVE-6403) uncorrelated subquery is failing with auto.convert.join=true

[Harish Butani (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-6403?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Harish Butani updated HIVE-6403:


Status: Patch Available  (was: Open)

> uncorrelated subquery is failing with auto.convert.join=true
> 
>
> Key: HIVE-6403
> URL: https://issues.apache.org/jira/browse/HIVE-6403
> Project: Hive
>  Issue Type: Bug
>  Components: Query Processor
>Reporter: Navis
>Assignee: Harish Butani
> Attachments: HIVE-6403.1.patch, HIVE-6403.2.patch, navis.patch
>
>
> Fixing HIVE-5690, I've found query in subquery_multiinsert.q is not working 
> with hive.auto.convert.join=true 
> {noformat}
> set hive.auto.convert.join=true;
> hive> explain
> > from src b 
> > INSERT OVERWRITE TABLE src_4 
> >   select * 
> >   where b.key in 
> >(select a.key 
> > from src a 
> > where b.value = a.value and a.key > '9'
> >) 
> > INSERT OVERWRITE TABLE src_5 
> >   select *  
> >   where b.key not in  ( select key from src s1 where s1.key > '2') 
> >   order by key 
> > ;
> java.lang.IndexOutOfBoundsException: Index: 0, Size: 0
>   at java.util.ArrayList.rangeCheck(ArrayList.java:635)
>   at java.util.ArrayList.get(ArrayList.java:411)
>   at 
> org.apache.hadoop.hive.ql.optimizer.MapJoinProcessor.genMapJoinLocalWork(MapJoinProcessor.java:149)
>   at 
> org.apache.hadoop.hive.ql.optimizer.MapJoinProcessor.genLocalWorkForMapJoin(MapJoinProcessor.java:256)
>   at 
> org.apache.hadoop.hive.ql.optimizer.MapJoinProcessor.genMapJoinOpAndLocalWork(MapJoinProcessor.java:248)
>   at 
> org.apache.hadoop.hive.ql.optimizer.physical.CommonJoinTaskDispatcher.convertTaskToMapJoinTask(CommonJoinTaskDispatcher.java:191)
>   at 
> org.apache.hadoop.hive.ql.optimizer.physical.CommonJoinTaskDispatcher.processCurrentTask(CommonJoinTaskDispatcher.java:481)
>   at 
> org.apache.hadoop.hive.ql.optimizer.physical.AbstractJoinTaskDispatcher.dispatch(AbstractJoinTaskDispatcher.java:182)
>   at 
> org.apache.hadoop.hive.ql.lib.TaskGraphWalker.dispatch(TaskGraphWalker.java:111)
>   at 
> org.apache.hadoop.hive.ql.lib.TaskGraphWalker.walk(TaskGraphWalker.java:194)
>   at 
> org.apache.hadoop.hive.ql.lib.TaskGraphWalker.startWalking(TaskGraphWalker.java:139)
>   at 
> org.apache.hadoop.hive.ql.optimizer.physical.CommonJoinResolver.resolve(CommonJoinResolver.java:79)
>   at 
> org.apache.hadoop.hive.ql.optimizer.physical.PhysicalOptimizer.optimize(PhysicalOptimizer.java:100)
>   at 
> org.apache.hadoop.hive.ql.parse.MapReduceCompiler.optimizeTaskPlan(MapReduceCompiler.java:290)
>   at 
> org.apache.hadoop.hive.ql.parse.TaskCompiler.compile(TaskCompiler.java:216)
>   at 
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeInternal(SemanticAnalyzer.java:9167)
>   at 
> org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:327)
>   at 
> org.apache.hadoop.hive.ql.parse.ExplainSemanticAnalyzer.analyzeInternal(ExplainSemanticAnalyzer.java:64)
>   at 
> org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:327)
>   at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:446)
>   at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:346)
>   at org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:1056)
>   at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1099)
>   at org.apache.hadoop.hive.ql.Driver.run(Driver.java:992)
>   at org.apache.hadoop.hive.ql.Driver.run(Driver.java:982)
>   at 
> org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:268)
>   at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:220)
>   at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:424)
>   at 
> org.apache.hadoop.hive.cli.CliDriver.executeDriver(CliDriver.java:793)
>   at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:687)
>   at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:626)
>   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>   at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
>   at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>   at java.lang.reflect.Method.invoke(Method.java:606)
>   at org.apache.hadoop.util.RunJar.main(RunJar.java:160)
> org.apache.hadoop.hive.ql.parse.SemanticException: Failed to generate new 
> mapJoin operator by exception : Index: 0, Size: 0
>   at 
> org.apache.hadoop.hive.ql.optimizer.MapJoinProcessor.genLocalWorkForMapJoin(MapJoinProcessor.java:266)
>   at 
> org.apache.hadoop.hive.ql.optimizer.MapJoinProcessor.genMapJoinOpAndLocalWork(MapJoinProcessor.j

[jira] [Updated] (HIVE-6403) uncorrelated subquery is failing with auto.convert.join=true

[Harish Butani (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-6403?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Harish Butani updated HIVE-6403:


Attachment: HIVE-6403.2.patch

> uncorrelated subquery is failing with auto.convert.join=true
> 
>
> Key: HIVE-6403
> URL: https://issues.apache.org/jira/browse/HIVE-6403
> Project: Hive
>  Issue Type: Bug
>  Components: Query Processor
>Reporter: Navis
>Assignee: Harish Butani
> Attachments: HIVE-6403.1.patch, HIVE-6403.2.patch, navis.patch
>
>
> Fixing HIVE-5690, I've found query in subquery_multiinsert.q is not working 
> with hive.auto.convert.join=true 
> {noformat}
> set hive.auto.convert.join=true;
> hive> explain
> > from src b 
> > INSERT OVERWRITE TABLE src_4 
> >   select * 
> >   where b.key in 
> >(select a.key 
> > from src a 
> > where b.value = a.value and a.key > '9'
> >) 
> > INSERT OVERWRITE TABLE src_5 
> >   select *  
> >   where b.key not in  ( select key from src s1 where s1.key > '2') 
> >   order by key 
> > ;
> java.lang.IndexOutOfBoundsException: Index: 0, Size: 0
>   at java.util.ArrayList.rangeCheck(ArrayList.java:635)
>   at java.util.ArrayList.get(ArrayList.java:411)
>   at 
> org.apache.hadoop.hive.ql.optimizer.MapJoinProcessor.genMapJoinLocalWork(MapJoinProcessor.java:149)
>   at 
> org.apache.hadoop.hive.ql.optimizer.MapJoinProcessor.genLocalWorkForMapJoin(MapJoinProcessor.java:256)
>   at 
> org.apache.hadoop.hive.ql.optimizer.MapJoinProcessor.genMapJoinOpAndLocalWork(MapJoinProcessor.java:248)
>   at 
> org.apache.hadoop.hive.ql.optimizer.physical.CommonJoinTaskDispatcher.convertTaskToMapJoinTask(CommonJoinTaskDispatcher.java:191)
>   at 
> org.apache.hadoop.hive.ql.optimizer.physical.CommonJoinTaskDispatcher.processCurrentTask(CommonJoinTaskDispatcher.java:481)
>   at 
> org.apache.hadoop.hive.ql.optimizer.physical.AbstractJoinTaskDispatcher.dispatch(AbstractJoinTaskDispatcher.java:182)
>   at 
> org.apache.hadoop.hive.ql.lib.TaskGraphWalker.dispatch(TaskGraphWalker.java:111)
>   at 
> org.apache.hadoop.hive.ql.lib.TaskGraphWalker.walk(TaskGraphWalker.java:194)
>   at 
> org.apache.hadoop.hive.ql.lib.TaskGraphWalker.startWalking(TaskGraphWalker.java:139)
>   at 
> org.apache.hadoop.hive.ql.optimizer.physical.CommonJoinResolver.resolve(CommonJoinResolver.java:79)
>   at 
> org.apache.hadoop.hive.ql.optimizer.physical.PhysicalOptimizer.optimize(PhysicalOptimizer.java:100)
>   at 
> org.apache.hadoop.hive.ql.parse.MapReduceCompiler.optimizeTaskPlan(MapReduceCompiler.java:290)
>   at 
> org.apache.hadoop.hive.ql.parse.TaskCompiler.compile(TaskCompiler.java:216)
>   at 
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeInternal(SemanticAnalyzer.java:9167)
>   at 
> org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:327)
>   at 
> org.apache.hadoop.hive.ql.parse.ExplainSemanticAnalyzer.analyzeInternal(ExplainSemanticAnalyzer.java:64)
>   at 
> org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:327)
>   at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:446)
>   at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:346)
>   at org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:1056)
>   at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1099)
>   at org.apache.hadoop.hive.ql.Driver.run(Driver.java:992)
>   at org.apache.hadoop.hive.ql.Driver.run(Driver.java:982)
>   at 
> org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:268)
>   at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:220)
>   at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:424)
>   at 
> org.apache.hadoop.hive.cli.CliDriver.executeDriver(CliDriver.java:793)
>   at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:687)
>   at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:626)
>   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>   at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
>   at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>   at java.lang.reflect.Method.invoke(Method.java:606)
>   at org.apache.hadoop.util.RunJar.main(RunJar.java:160)
> org.apache.hadoop.hive.ql.parse.SemanticException: Failed to generate new 
> mapJoin operator by exception : Index: 0, Size: 0
>   at 
> org.apache.hadoop.hive.ql.optimizer.MapJoinProcessor.genLocalWorkForMapJoin(MapJoinProcessor.java:266)
>   at 
> org.apache.hadoop.hive.ql.optimizer.MapJoinProcessor.genMapJoinOpAndLocalWork(MapJoinProcessor.java:248

[jira] [Updated] (HIVE-6403) uncorrelated subquery is failing with auto.convert.join=true

[Harish Butani (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-6403?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Harish Butani updated HIVE-6403:


Attachment: (was: HIVE-6403.2.patch)

> uncorrelated subquery is failing with auto.convert.join=true
> 
>
> Key: HIVE-6403
> URL: https://issues.apache.org/jira/browse/HIVE-6403
> Project: Hive
>  Issue Type: Bug
>  Components: Query Processor
>Reporter: Navis
>Assignee: Harish Butani
> Attachments: HIVE-6403.1.patch, HIVE-6403.2.patch, navis.patch
>
>
> Fixing HIVE-5690, I've found query in subquery_multiinsert.q is not working 
> with hive.auto.convert.join=true 
> {noformat}
> set hive.auto.convert.join=true;
> hive> explain
> > from src b 
> > INSERT OVERWRITE TABLE src_4 
> >   select * 
> >   where b.key in 
> >(select a.key 
> > from src a 
> > where b.value = a.value and a.key > '9'
> >) 
> > INSERT OVERWRITE TABLE src_5 
> >   select *  
> >   where b.key not in  ( select key from src s1 where s1.key > '2') 
> >   order by key 
> > ;
> java.lang.IndexOutOfBoundsException: Index: 0, Size: 0
>   at java.util.ArrayList.rangeCheck(ArrayList.java:635)
>   at java.util.ArrayList.get(ArrayList.java:411)
>   at 
> org.apache.hadoop.hive.ql.optimizer.MapJoinProcessor.genMapJoinLocalWork(MapJoinProcessor.java:149)
>   at 
> org.apache.hadoop.hive.ql.optimizer.MapJoinProcessor.genLocalWorkForMapJoin(MapJoinProcessor.java:256)
>   at 
> org.apache.hadoop.hive.ql.optimizer.MapJoinProcessor.genMapJoinOpAndLocalWork(MapJoinProcessor.java:248)
>   at 
> org.apache.hadoop.hive.ql.optimizer.physical.CommonJoinTaskDispatcher.convertTaskToMapJoinTask(CommonJoinTaskDispatcher.java:191)
>   at 
> org.apache.hadoop.hive.ql.optimizer.physical.CommonJoinTaskDispatcher.processCurrentTask(CommonJoinTaskDispatcher.java:481)
>   at 
> org.apache.hadoop.hive.ql.optimizer.physical.AbstractJoinTaskDispatcher.dispatch(AbstractJoinTaskDispatcher.java:182)
>   at 
> org.apache.hadoop.hive.ql.lib.TaskGraphWalker.dispatch(TaskGraphWalker.java:111)
>   at 
> org.apache.hadoop.hive.ql.lib.TaskGraphWalker.walk(TaskGraphWalker.java:194)
>   at 
> org.apache.hadoop.hive.ql.lib.TaskGraphWalker.startWalking(TaskGraphWalker.java:139)
>   at 
> org.apache.hadoop.hive.ql.optimizer.physical.CommonJoinResolver.resolve(CommonJoinResolver.java:79)
>   at 
> org.apache.hadoop.hive.ql.optimizer.physical.PhysicalOptimizer.optimize(PhysicalOptimizer.java:100)
>   at 
> org.apache.hadoop.hive.ql.parse.MapReduceCompiler.optimizeTaskPlan(MapReduceCompiler.java:290)
>   at 
> org.apache.hadoop.hive.ql.parse.TaskCompiler.compile(TaskCompiler.java:216)
>   at 
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeInternal(SemanticAnalyzer.java:9167)
>   at 
> org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:327)
>   at 
> org.apache.hadoop.hive.ql.parse.ExplainSemanticAnalyzer.analyzeInternal(ExplainSemanticAnalyzer.java:64)
>   at 
> org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:327)
>   at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:446)
>   at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:346)
>   at org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:1056)
>   at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1099)
>   at org.apache.hadoop.hive.ql.Driver.run(Driver.java:992)
>   at org.apache.hadoop.hive.ql.Driver.run(Driver.java:982)
>   at 
> org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:268)
>   at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:220)
>   at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:424)
>   at 
> org.apache.hadoop.hive.cli.CliDriver.executeDriver(CliDriver.java:793)
>   at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:687)
>   at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:626)
>   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>   at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
>   at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>   at java.lang.reflect.Method.invoke(Method.java:606)
>   at org.apache.hadoop.util.RunJar.main(RunJar.java:160)
> org.apache.hadoop.hive.ql.parse.SemanticException: Failed to generate new 
> mapJoin operator by exception : Index: 0, Size: 0
>   at 
> org.apache.hadoop.hive.ql.optimizer.MapJoinProcessor.genLocalWorkForMapJoin(MapJoinProcessor.java:266)
>   at 
> org.apache.hadoop.hive.ql.optimizer.MapJoinProcessor.genMapJoinOpAndLocalWork(MapJoinProcess

[jira] [Updated] (HIVE-6418) MapJoinRowContainer has large memory overhead in typical cases

[Sergey Shelukhin (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-6418?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sergey Shelukhin updated HIVE-6418:
---

Attachment: HIVE-6418.02.patch

tiny fix to serde to make last tez tests pass

> MapJoinRowContainer has large memory overhead in typical cases
> --
>
> Key: HIVE-6418
> URL: https://issues.apache.org/jira/browse/HIVE-6418
> Project: Hive
>  Issue Type: Improvement
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
> Attachments: HIVE-6418.01.patch, HIVE-6418.02.patch, 
> HIVE-6418.WIP.patch, HIVE-6418.patch
>
>




--
This message was sent by Atlassian JIRA
(v6.1.5#6160)

Re: Review Request 18061: HIVE-6418 MapJoinRowContainer has large memory overhead in typical cases

[Sergey Shelukhin]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/18061/
---

(Updated Feb. 14, 2014, 3:38 a.m.)


Review request for hive and Gunther Hagleitner.


Repository: hive-git


Description
---

See JIRA


Diffs
-

  ql/src/java/org/apache/hadoop/hive/ql/exec/CommonJoinOperator.java aacf200 
  ql/src/java/org/apache/hadoop/hive/ql/exec/HashTableSinkOperator.java 24f1229 
  ql/src/java/org/apache/hadoop/hive/ql/exec/JoinOperator.java 2981a3b 
  ql/src/java/org/apache/hadoop/hive/ql/exec/MapJoinOperator.java c0f4cd7 
  ql/src/java/org/apache/hadoop/hive/ql/exec/PTFPartition.java 01d31a7 
  ql/src/java/org/apache/hadoop/hive/ql/exec/SMBMapJoinOperator.java 7a7dc88 
  
ql/src/java/org/apache/hadoop/hive/ql/exec/persistence/AbstractRowContainer.java
 778d016 
  
ql/src/java/org/apache/hadoop/hive/ql/exec/persistence/LazyFlatRowContainer.java
 PRE-CREATION 
  
ql/src/java/org/apache/hadoop/hive/ql/exec/persistence/MapJoinEagerRowContainer.java
 PRE-CREATION 
  
ql/src/java/org/apache/hadoop/hive/ql/exec/persistence/MapJoinRowContainer.java 
b7f66fe 
  
ql/src/java/org/apache/hadoop/hive/ql/exec/persistence/MapJoinTableContainer.java
 9ce0ae6 
  
ql/src/java/org/apache/hadoop/hive/ql/exec/persistence/MapJoinTableContainerSerDe.java
 83ba0f0 
  ql/src/java/org/apache/hadoop/hive/ql/exec/persistence/PTFRowContainer.java 
9b2babc 
  ql/src/java/org/apache/hadoop/hive/ql/exec/persistence/RowContainer.java 
3b0bc2a 
  ql/src/java/org/apache/hadoop/hive/ql/exec/tez/HashTableLoader.java 47f9d21 
  
ql/src/test/org/apache/hadoop/hive/ql/exec/persistence/TestMapJoinEqualityTableContainer.java
 c541ad2 
  
ql/src/test/org/apache/hadoop/hive/ql/exec/persistence/TestMapJoinRowContainer.java
 21de0f5 
  
ql/src/test/org/apache/hadoop/hive/ql/exec/persistence/TestMapJoinTableContainer.java
 61c5741 
  
ql/src/test/org/apache/hadoop/hive/ql/exec/persistence/TestPTFRowContainer.java 
900a574 
  ql/src/test/org/apache/hadoop/hive/ql/exec/persistence/Utilities.java 2cb1ac3 
  serde/src/java/org/apache/hadoop/hive/serde2/lazybinary/LazyBinarySerDe.java 
0324453 
  
serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/ObjectInspectorUtils.java
 64e651c 

Diff: https://reviews.apache.org/r/18061/diff/


Testing
---


Thanks,

Sergey Shelukhin

Re: Review Request 18061: HIVE-6418 MapJoinRowContainer has large memory overhead in typical cases

[Sergey Shelukhin]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/18061/
---

(Updated Feb. 14, 2014, 3:38 a.m.)


Review request for hive and Gunther Hagleitner.


Repository: hive-git


Description
---

See JIRA


Diffs (updated)
-

  ql/src/java/org/apache/hadoop/hive/ql/exec/CommonJoinOperator.java aacf200 
  ql/src/java/org/apache/hadoop/hive/ql/exec/HashTableSinkOperator.java 24f1229 
  ql/src/java/org/apache/hadoop/hive/ql/exec/JoinOperator.java 2981a3b 
  ql/src/java/org/apache/hadoop/hive/ql/exec/MapJoinOperator.java c0f4cd7 
  ql/src/java/org/apache/hadoop/hive/ql/exec/PTFPartition.java 01d31a7 
  ql/src/java/org/apache/hadoop/hive/ql/exec/SMBMapJoinOperator.java 7a7dc88 
  
ql/src/java/org/apache/hadoop/hive/ql/exec/persistence/AbstractRowContainer.java
 778d016 
  
ql/src/java/org/apache/hadoop/hive/ql/exec/persistence/LazyFlatRowContainer.java
 PRE-CREATION 
  
ql/src/java/org/apache/hadoop/hive/ql/exec/persistence/MapJoinEagerRowContainer.java
 PRE-CREATION 
  
ql/src/java/org/apache/hadoop/hive/ql/exec/persistence/MapJoinRowContainer.java 
b7f66fe 
  
ql/src/java/org/apache/hadoop/hive/ql/exec/persistence/MapJoinTableContainer.java
 9ce0ae6 
  
ql/src/java/org/apache/hadoop/hive/ql/exec/persistence/MapJoinTableContainerSerDe.java
 83ba0f0 
  ql/src/java/org/apache/hadoop/hive/ql/exec/persistence/PTFRowContainer.java 
9b2babc 
  ql/src/java/org/apache/hadoop/hive/ql/exec/persistence/RowContainer.java 
3b0bc2a 
  ql/src/java/org/apache/hadoop/hive/ql/exec/tez/HashTableLoader.java 47f9d21 
  
ql/src/test/org/apache/hadoop/hive/ql/exec/persistence/TestMapJoinEqualityTableContainer.java
 c541ad2 
  
ql/src/test/org/apache/hadoop/hive/ql/exec/persistence/TestMapJoinRowContainer.java
 21de0f5 
  
ql/src/test/org/apache/hadoop/hive/ql/exec/persistence/TestMapJoinTableContainer.java
 61c5741 
  
ql/src/test/org/apache/hadoop/hive/ql/exec/persistence/TestPTFRowContainer.java 
900a574 
  ql/src/test/org/apache/hadoop/hive/ql/exec/persistence/Utilities.java 2cb1ac3 
  serde/src/java/org/apache/hadoop/hive/serde2/lazybinary/LazyBinarySerDe.java 
0324453 
  
serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/ObjectInspectorUtils.java
 64e651c 

Diff: https://reviews.apache.org/r/18061/diff/


Testing
---


Thanks,

Sergey Shelukhin

[jira] [Updated] (HIVE-5759) Implement vectorized support for COALESCE conditional expression

[Jitendra Nath Pandey (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-5759?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jitendra Nath Pandey updated HIVE-5759:
---

Status: Patch Available  (was: Open)

> Implement vectorized support for COALESCE conditional expression
> 
>
> Key: HIVE-5759
> URL: https://issues.apache.org/jira/browse/HIVE-5759
> Project: Hive
>  Issue Type: Sub-task
>Reporter: Eric Hanson
>Assignee: Jitendra Nath Pandey
> Attachments: HIVE-5759.1.patch, HIVE-5759.2.patch
>
>
> Implement full, end-to-end support for COALESCE in vectorized mode, including 
> new VectorExpression class(es), VectorizationContext translation to a 
> VectorExpression, and unit tests for these, as well as end-to-end ad hoc 
> testing. An end-to-end .q test is recommended.
> This is lower priority than IF and CASE but it is still a fairly popular 
> expression.



--
This message was sent by Atlassian JIRA
(v6.1.5#6160)

[jira] [Updated] (HIVE-5759) Implement vectorized support for COALESCE conditional expression

[Jitendra Nath Pandey (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-5759?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jitendra Nath Pandey updated HIVE-5759:
---

Status: Open  (was: Patch Available)

> Implement vectorized support for COALESCE conditional expression
> 
>
> Key: HIVE-5759
> URL: https://issues.apache.org/jira/browse/HIVE-5759
> Project: Hive
>  Issue Type: Sub-task
>Reporter: Eric Hanson
>Assignee: Jitendra Nath Pandey
> Attachments: HIVE-5759.1.patch, HIVE-5759.2.patch
>
>
> Implement full, end-to-end support for COALESCE in vectorized mode, including 
> new VectorExpression class(es), VectorizationContext translation to a 
> VectorExpression, and unit tests for these, as well as end-to-end ad hoc 
> testing. An end-to-end .q test is recommended.
> This is lower priority than IF and CASE but it is still a fairly popular 
> expression.



--
This message was sent by Atlassian JIRA
(v6.1.5#6160)

[jira] [Commented] (HIVE-6046) add UDF for converting date time from one presentation to another

[Hive QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-6046?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13901047#comment-13901047
 ] 

Hive QA commented on HIVE-6046:
---



{color:red}Overall{color}: -1 at least one tests failed

Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12628857/Hive-6046.patch

{color:red}ERROR:{color} -1 due to 2 failed/errored test(s), 5098 tests executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_auto_sortmerge_join_16
org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_bucketmapjoin6
{noformat}

Test results: 
http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/1318/testReport
Console output: 
http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/1318/console

Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 2 tests failed
{noformat}

This message is automatically generated.

ATTACHMENT ID: 12628857

> add  UDF for converting date time from one presentation to another
> --
>
> Key: HIVE-6046
> URL: https://issues.apache.org/jira/browse/HIVE-6046
> Project: Hive
>  Issue Type: New Feature
>  Components: UDF
>Affects Versions: 0.13.0
>Reporter: Kostiantyn Kudriavtsev
>Assignee: Kostiantyn Kudriavtsev
> Attachments: Hive-6046.patch
>
>
> it'd be nice to have function for converting datetime to different formats, 
> for example:
> format_date('2013-12-12 00:00:00.0', '-MM-dd HH:mm:ss.S', '/MM/dd')
> There are two signatures to facilitate further using:
> format_date(datetime, fromFormat, toFormat)
> format_date(timestamp, toFormat)
>  



--
This message was sent by Atlassian JIRA
(v6.1.5#6160)

[jira] [Commented] (HIVE-6380) Specify jars/files when creating permanent UDFs

[Jason Dere (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-6380?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13901013#comment-13901013
 ] 

Jason Dere commented on HIVE-6380:
--

The 2 test failures don't seem to be related - both had been showing up in 
precommit tests.  There was a fix for authorization_revoke_table_priv, and 
bucketmapjoin6 is passing locally.

> Specify jars/files when creating permanent UDFs
> ---
>
> Key: HIVE-6380
> URL: https://issues.apache.org/jira/browse/HIVE-6380
> Project: Hive
>  Issue Type: Sub-task
>  Components: UDF
>Reporter: Jason Dere
>Assignee: Jason Dere
> Attachments: HIVE-6380.1.patch
>
>
> Need a way for a permanent UDF to reference jars/files.



--
This message was sent by Atlassian JIRA
(v6.1.5#6160)

[jira] [Updated] (HIVE-5759) Implement vectorized support for COALESCE conditional expression

[Jitendra Nath Pandey (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-5759?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jitendra Nath Pandey updated HIVE-5759:
---

Attachment: HIVE-5759.2.patch

Updated patch addresses review comments.

> Implement vectorized support for COALESCE conditional expression
> 
>
> Key: HIVE-5759
> URL: https://issues.apache.org/jira/browse/HIVE-5759
> Project: Hive
>  Issue Type: Sub-task
>Reporter: Eric Hanson
>Assignee: Jitendra Nath Pandey
> Attachments: HIVE-5759.1.patch, HIVE-5759.2.patch
>
>
> Implement full, end-to-end support for COALESCE in vectorized mode, including 
> new VectorExpression class(es), VectorizationContext translation to a 
> VectorExpression, and unit tests for these, as well as end-to-end ad hoc 
> testing. An end-to-end .q test is recommended.
> This is lower priority than IF and CASE but it is still a fairly popular 
> expression.



--
This message was sent by Atlassian JIRA
(v6.1.5#6160)

Re: Review Request 18025: Implement vectorized support for COALESCE conditional expression

[Jitendra Pandey]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/18025/
---

(Updated Feb. 14, 2014, 1:49 a.m.)


Review request for hive and Eric Hanson.


Bugs: HIVE-5759
https://issues.apache.org/jira/browse/HIVE-5759


Repository: hive-git


Description
---

Implement vectorized support for COALESCE conditional expression


Diffs (updated)
-

  ql/src/java/org/apache/hadoop/hive/ql/exec/vector/BytesColumnVector.java 
f1eef14 
  ql/src/java/org/apache/hadoop/hive/ql/exec/vector/ColumnVector.java 0a8811f 
  ql/src/java/org/apache/hadoop/hive/ql/exec/vector/DecimalColumnVector.java 
d0d8597 
  ql/src/java/org/apache/hadoop/hive/ql/exec/vector/DoubleColumnVector.java 
cb23129 
  ql/src/java/org/apache/hadoop/hive/ql/exec/vector/LongColumnVector.java 
aa05b19 
  ql/src/java/org/apache/hadoop/hive/ql/exec/vector/VectorizationContext.java 
7141d63 
  
ql/src/java/org/apache/hadoop/hive/ql/exec/vector/expressions/VectorCoalesce.java
 PRE-CREATION 
  ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/Vectorizer.java 
21fe8ca 
  ql/src/test/queries/clientpositive/vector_coalesce.q PRE-CREATION 
  ql/src/test/results/clientpositive/vector_coalesce.q.out PRE-CREATION 

Diff: https://reviews.apache.org/r/18025/diff/


Testing
---


Thanks,

Jitendra Pandey

Re: Review Request 18025: Implement vectorized support for COALESCE conditional expression

[Jitendra Pandey]

> On Feb. 13, 2014, 1:30 a.m., Eric Hanson wrote:
> > ql/src/java/org/apache/hadoop/hive/ql/exec/vector/VectorizationContext.java,
> >  line 440
> > 
> >
> > This is the same code block as the previous case. Can you share the 
> > case and change the condition to an OR?
> > 
> > Up to you...

This piece of code will be revisited in HIVE-6416, where I need to add casting 
for many mathematical functions. I will address it there.


> On Feb. 13, 2014, 1:30 a.m., Eric Hanson wrote:
> > ql/src/java/org/apache/hadoop/hive/ql/exec/vector/expressions/VectorCoalesce.java,
> >  line 126
> > 
> >
> > This can have > 1 argument. Please add comment to explain.
> > 
> >

This class should not return a descriptor, because it can take variable number 
of arguments with different data types. 


- Jitendra


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/18025/#review34336
---


On Feb. 14, 2014, 1:49 a.m., Jitendra Pandey wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/18025/
> ---
> 
> (Updated Feb. 14, 2014, 1:49 a.m.)
> 
> 
> Review request for hive and Eric Hanson.
> 
> 
> Bugs: HIVE-5759
> https://issues.apache.org/jira/browse/HIVE-5759
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> Implement vectorized support for COALESCE conditional expression
> 
> 
> Diffs
> -
> 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/vector/BytesColumnVector.java 
> f1eef14 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/vector/ColumnVector.java 0a8811f 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/vector/DecimalColumnVector.java 
> d0d8597 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/vector/DoubleColumnVector.java 
> cb23129 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/vector/LongColumnVector.java 
> aa05b19 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/vector/VectorizationContext.java 
> 7141d63 
>   
> ql/src/java/org/apache/hadoop/hive/ql/exec/vector/expressions/VectorCoalesce.java
>  PRE-CREATION 
>   ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/Vectorizer.java 
> 21fe8ca 
>   ql/src/test/queries/clientpositive/vector_coalesce.q PRE-CREATION 
>   ql/src/test/results/clientpositive/vector_coalesce.q.out PRE-CREATION 
> 
> Diff: https://reviews.apache.org/r/18025/diff/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Jitendra Pandey
> 
>

[jira] [Assigned] (HIVE-6375) Implement CTAS and column rename for parquet

[Szehon Ho (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-6375?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Szehon Ho reassigned HIVE-6375:
---

Assignee: Szehon Ho

> Implement CTAS and column rename for parquet
> 
>
> Key: HIVE-6375
> URL: https://issues.apache.org/jira/browse/HIVE-6375
> Project: Hive
>  Issue Type: Bug
>Reporter: Brock Noland
>Assignee: Szehon Ho
>Priority: Critical
>  Labels: Parquet
>
> More details here:
> https://github.com/Parquet/parquet-mr/issues/272



--
This message was sent by Atlassian JIRA
(v6.1.5#6160)

Re: Review Request 18061: HIVE-6418 MapJoinRowContainer has large memory overhead in typical cases

[Sergey Shelukhin]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/18061/
---

(Updated Feb. 14, 2014, 1:26 a.m.)


Review request for hive and Gunther Hagleitner.


Repository: hive-git


Description
---

See JIRA


Diffs (updated)
-

  ql/src/java/org/apache/hadoop/hive/ql/exec/CommonJoinOperator.java aacf200 
  ql/src/java/org/apache/hadoop/hive/ql/exec/HashTableSinkOperator.java 24f1229 
  ql/src/java/org/apache/hadoop/hive/ql/exec/JoinOperator.java 2981a3b 
  ql/src/java/org/apache/hadoop/hive/ql/exec/MapJoinOperator.java c0f4cd7 
  ql/src/java/org/apache/hadoop/hive/ql/exec/PTFPartition.java 01d31a7 
  ql/src/java/org/apache/hadoop/hive/ql/exec/SMBMapJoinOperator.java 7a7dc88 
  
ql/src/java/org/apache/hadoop/hive/ql/exec/persistence/AbstractRowContainer.java
 778d016 
  
ql/src/java/org/apache/hadoop/hive/ql/exec/persistence/LazyFlatRowContainer.java
 PRE-CREATION 
  
ql/src/java/org/apache/hadoop/hive/ql/exec/persistence/MapJoinEagerRowContainer.java
 PRE-CREATION 
  
ql/src/java/org/apache/hadoop/hive/ql/exec/persistence/MapJoinRowContainer.java 
b7f66fe 
  
ql/src/java/org/apache/hadoop/hive/ql/exec/persistence/MapJoinTableContainer.java
 9ce0ae6 
  
ql/src/java/org/apache/hadoop/hive/ql/exec/persistence/MapJoinTableContainerSerDe.java
 83ba0f0 
  ql/src/java/org/apache/hadoop/hive/ql/exec/persistence/PTFRowContainer.java 
9b2babc 
  ql/src/java/org/apache/hadoop/hive/ql/exec/persistence/RowContainer.java 
3b0bc2a 
  ql/src/java/org/apache/hadoop/hive/ql/exec/tez/HashTableLoader.java 47f9d21 
  
ql/src/test/org/apache/hadoop/hive/ql/exec/persistence/TestMapJoinEqualityTableContainer.java
 c541ad2 
  
ql/src/test/org/apache/hadoop/hive/ql/exec/persistence/TestMapJoinRowContainer.java
 21de0f5 
  
ql/src/test/org/apache/hadoop/hive/ql/exec/persistence/TestMapJoinTableContainer.java
 61c5741 
  
ql/src/test/org/apache/hadoop/hive/ql/exec/persistence/TestPTFRowContainer.java 
900a574 
  ql/src/test/org/apache/hadoop/hive/ql/exec/persistence/Utilities.java 2cb1ac3 
  
serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/ObjectInspectorUtils.java
 64e651c 

Diff: https://reviews.apache.org/r/18061/diff/


Testing
---


Thanks,

Sergey Shelukhin

[jira] [Updated] (HIVE-6418) MapJoinRowContainer has large memory overhead in typical cases

[Sergey Shelukhin (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-6418?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sergey Shelukhin updated HIVE-6418:
---

Attachment: HIVE-6418.01.patch

Added correct handling on 0-length rows (main issue), got rid of some array 
allocations when deserializing, other minor changes and fixes. I have run 
minitez tests and they passed except for mapjoin_mapjoin; with this patch, that 
passes, I am rerunning all tests now.


> MapJoinRowContainer has large memory overhead in typical cases
> --
>
> Key: HIVE-6418
> URL: https://issues.apache.org/jira/browse/HIVE-6418
> Project: Hive
>  Issue Type: Improvement
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
> Attachments: HIVE-6418.01.patch, HIVE-6418.WIP.patch, HIVE-6418.patch
>
>




--
This message was sent by Atlassian JIRA
(v6.1.5#6160)

[jira] [Updated] (HIVE-6420) upgrade script for Hive 13 is missing for Derby

[Sergey Shelukhin (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-6420?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sergey Shelukhin updated HIVE-6420:
---

Priority: Trivial  (was: Minor)

> upgrade script for Hive 13 is missing for Derby
> ---
>
> Key: HIVE-6420
> URL: https://issues.apache.org/jira/browse/HIVE-6420
> Project: Hive
>  Issue Type: Bug
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
>Priority: Trivial
> Attachments: HIVE-6420.patch
>
>
> There's an upgrade script for all DSes but not for Derby. Nothing needs to be 
> done in that script but I'm being told that some tools might break if there's 
> no matching file.



--
This message was sent by Atlassian JIRA
(v6.1.5#6160)

[jira] [Commented] (HIVE-6420) upgrade script for Hive 13 is missing for Derby

[Sergey Shelukhin (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-6420?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13900985#comment-13900985
 ] 

Sergey Shelukhin commented on HIVE-6420:


[~ashutoshc] fyi

> upgrade script for Hive 13 is missing for Derby
> ---
>
> Key: HIVE-6420
> URL: https://issues.apache.org/jira/browse/HIVE-6420
> Project: Hive
>  Issue Type: Bug
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
>Priority: Minor
> Attachments: HIVE-6420.patch
>
>
> There's an upgrade script for all DSes but not for Derby. Nothing needs to be 
> done in that script but I'm being told that some tools might break if there's 
> no matching file.



--
This message was sent by Atlassian JIRA
(v6.1.5#6160)

[jira] [Updated] (HIVE-6420) upgrade script for Hive 13 is missing for Derby

[Sergey Shelukhin (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-6420?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sergey Shelukhin updated HIVE-6420:
---

Attachment: HIVE-6420.patch

> upgrade script for Hive 13 is missing for Derby
> ---
>
> Key: HIVE-6420
> URL: https://issues.apache.org/jira/browse/HIVE-6420
> Project: Hive
>  Issue Type: Bug
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
>Priority: Minor
> Attachments: HIVE-6420.patch
>
>
> There's an upgrade script for all DSes but not for Derby. Nothing needs to be 
> done in that script but I'm being told that some tools might break if there's 
> no matching file.



--
This message was sent by Atlassian JIRA
(v6.1.5#6160)

[jira] [Commented] (HIVE-6379) webhcat is not operational due to Slf4j conflicts

[Eugene Koifman (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-6379?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13900983#comment-13900983
 ] 

Eugene Koifman commented on HIVE-6379:
--

webhcat script uses 'hadoop jar' command.  Hadoop 1.2.1 ships with SLF4J 
version 1.4.3 and adds this library to it's classpath...



> webhcat is not operational due to Slf4j conflicts
> -
>
> Key: HIVE-6379
> URL: https://issues.apache.org/jira/browse/HIVE-6379
> Project: Hive
>  Issue Type: Bug
>  Components: WebHCat
>Affects Versions: 0.12.0
>Reporter: Eugene Koifman
>Assignee: Eugene Koifman
>
> 1. build current trunk: mvn clean install -Pdist,hadoop-1 -DskipTests
> 2. 
> /Users/ekoifman/dev/hive/packaging/target/apache-hive-0.13.0-SNAPSHOT-bin/apache-hive-0.13.0-SNAPSHOT-bin/hcatalog
> 3. ./sbin/webhcat_server.sh start
> 4. http://localhost:50111/templeton/v1/ddl/database?user.name=ekoifman
> webhcat-console-error.log has
> log4j:WARN No appenders could be found for logger 
> (org.apache.hive.hcatalog.templeton.AppConfig).
> log4j:WARN Please initialize the log4j system properly.
> log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more 
> info.
> Exception in thread "main" java.lang.NoSuchMethodError: 
> org.slf4j.helpers.MessageFormatter.arrayFormat(Ljava/lang/String;[Ljava/lang/Object;)Lorg/slf4j/helpers/FormattingTuple;
> at 
> org.eclipse.jetty.util.log.JettyAwareLogger.log(JettyAwareLogger.java:613)
> at 
> org.eclipse.jetty.util.log.JettyAwareLogger.debug(JettyAwareLogger.java:200)
> at org.eclipse.jetty.util.log.Slf4jLog.debug(Slf4jLog.java:94)
> at org.eclipse.jetty.util.component.Container.add(Container.java:201)
> at 
> org.eclipse.jetty.util.component.Container.update(Container.java:164)
> at 
> org.eclipse.jetty.util.component.Container.update(Container.java:106)
> at org.eclipse.jetty.server.Server.setConnectors(Server.java:184)
> at org.eclipse.jetty.server.Server.(Server.java:97)
> at org.apache.hive.hcatalog.templeton.Main.runServer(Main.java:162)
> at org.apache.hive.hcatalog.templeton.Main.run(Main.java:114)
> at org.apache.hive.hcatalog.templeton.Main.main(Main.java:259)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.util.RunJar.main(RunJar.java:160)
> webhcat-console-error.log (END) 



--
This message was sent by Atlassian JIRA
(v6.1.5#6160)

[jira] [Commented] (HIVE-6379) webhcat is not operational due to Slf4j conflicts

[Eugene Koifman (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-6379?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13900969#comment-13900969
 ] 

Eugene Koifman commented on HIVE-6379:
--

using -verbose:class

we get 
[Loaded org.slf4j.Logger from 
file:/Users/ekoifman/dev/hadoop-1.2.1/lib/slf4j-api-1.4.3.jar]
[Loaded org.slf4j.spi.LoggerFactoryBinder from 
file:/Users/ekoifman/dev/hadoop-1.2.1/lib/slf4j-api-1.4.3.jar]
[Loaded org.slf4j.impl.StaticLoggerBinder from 
file:/Users/ekoifman/dev/hadoop-1.2.1/lib/slf4j-log4j12-1.4.3.jar]
[Loaded org.slf4j.ILoggerFactory from 
file:/Users/ekoifman/dev/hadoop-1.2.1/lib/slf4j-api-1.4.3.jar]
[Loaded org.slf4j.impl.Log4jLoggerFactory from 
file:/Users/ekoifman/dev/hadoop-1.2.1/lib/slf4j-log4j12-1.4.3.jar]
[Loaded org.slf4j.LoggerFactory from 
file:/Users/ekoifman/dev/hadoop-1.2.1/lib/slf4j-api-1.4.3.jar]
[Loaded org.slf4j.spi.LocationAwareLogger from 
file:/Users/ekoifman/dev/hadoop-1.2.1/lib/slf4j-api-1.4.3.jar]
[Loaded org.slf4j.helpers.MarkerIgnoringBase from 
file:/Users/ekoifman/dev/hadoop-1.2.1/lib/slf4j-api-1.4.3.jar]
[Loaded org.slf4j.impl.Log4jLoggerAdapter from 
file:/Users/ekoifman/dev/hadoop-1.2.1/lib/slf4j-log4j12-1.4.3.jar]

which explains where the 'wrong' jar comes from

> webhcat is not operational due to Slf4j conflicts
> -
>
> Key: HIVE-6379
> URL: https://issues.apache.org/jira/browse/HIVE-6379
> Project: Hive
>  Issue Type: Bug
>  Components: WebHCat
>Affects Versions: 0.12.0
>Reporter: Eugene Koifman
>Assignee: Eugene Koifman
>
> 1. build current trunk: mvn clean install -Pdist,hadoop-1 -DskipTests
> 2. 
> /Users/ekoifman/dev/hive/packaging/target/apache-hive-0.13.0-SNAPSHOT-bin/apache-hive-0.13.0-SNAPSHOT-bin/hcatalog
> 3. ./sbin/webhcat_server.sh start
> 4. http://localhost:50111/templeton/v1/ddl/database?user.name=ekoifman
> webhcat-console-error.log has
> log4j:WARN No appenders could be found for logger 
> (org.apache.hive.hcatalog.templeton.AppConfig).
> log4j:WARN Please initialize the log4j system properly.
> log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more 
> info.
> Exception in thread "main" java.lang.NoSuchMethodError: 
> org.slf4j.helpers.MessageFormatter.arrayFormat(Ljava/lang/String;[Ljava/lang/Object;)Lorg/slf4j/helpers/FormattingTuple;
> at 
> org.eclipse.jetty.util.log.JettyAwareLogger.log(JettyAwareLogger.java:613)
> at 
> org.eclipse.jetty.util.log.JettyAwareLogger.debug(JettyAwareLogger.java:200)
> at org.eclipse.jetty.util.log.Slf4jLog.debug(Slf4jLog.java:94)
> at org.eclipse.jetty.util.component.Container.add(Container.java:201)
> at 
> org.eclipse.jetty.util.component.Container.update(Container.java:164)
> at 
> org.eclipse.jetty.util.component.Container.update(Container.java:106)
> at org.eclipse.jetty.server.Server.setConnectors(Server.java:184)
> at org.eclipse.jetty.server.Server.(Server.java:97)
> at org.apache.hive.hcatalog.templeton.Main.runServer(Main.java:162)
> at org.apache.hive.hcatalog.templeton.Main.run(Main.java:114)
> at org.apache.hive.hcatalog.templeton.Main.main(Main.java:259)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.util.RunJar.main(RunJar.java:160)
> webhcat-console-error.log (END) 



--
This message was sent by Atlassian JIRA
(v6.1.5#6160)

[jira] [Commented] (HIVE-6415) Disallow transform clause in sql std authorization mode

[Thejas M Nair (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-6415?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13900966#comment-13900966
 ] 

Thejas M Nair commented on HIVE-6415:
-

+1 for updated patch 


> Disallow transform clause in sql std authorization mode
> ---
>
> Key: HIVE-6415
> URL: https://issues.apache.org/jira/browse/HIVE-6415
> Project: Hive
>  Issue Type: Sub-task
>  Components: Authorization
>Reporter: Ashutosh Chauhan
>Assignee: Ashutosh Chauhan
> Attachments: HIVE-6415.1.patch, HIVE-6415.2.patch, HIVE-6415.patch
>
>




--
This message was sent by Atlassian JIRA
(v6.1.5#6160)

[jira] [Updated] (HIVE-5989) Hive metastore authorization check is not threadsafe

[Thejas M Nair (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-5989?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Thejas M Nair updated HIVE-5989:


   Resolution: Fixed
Fix Version/s: 0.13.0
   Status: Resolved  (was: Patch Available)

Patch committed to trunk.
Thanks for the contribution Sushanth!


> Hive metastore authorization check is not threadsafe
> 
>
> Key: HIVE-5989
> URL: https://issues.apache.org/jira/browse/HIVE-5989
> Project: Hive
>  Issue Type: Bug
>  Components: Metastore, Security
>Affects Versions: 0.11.0, 0.12.0, 0.12.1
>Reporter: Sushanth Sowmyan
>Assignee: Sushanth Sowmyan
>Priority: Critical
> Fix For: 0.13.0
>
> Attachments: HIVE-5989.2.patch, HIVE-5989.patch, SleepyAP.patch
>
>
> Metastore-side authorization has a couple of pretty important threadsafety 
> bugs in it:
> a) The HiveMetastoreAuthenticated instantiated by the 
> AuthorizationPreEventListener is static. This is a premature optimization and 
> incorrect, as it will result in Authenticator implementations that store 
> state potentially giving an incorrect result, and this bug very much exists 
> with the DefaultMetastoreAuthenticator.
> b) It assumes HMSHandler.getHiveConf() is itself going to be thread-safe, 
> which it is not. HMSHandler.getConf() is the appropriate thread-safe 
> equivalent.
> The effect of this bug is that if there are two users that are concurrently 
> running jobs on the metastore, we might :
> a) Allow a user to do something they didn't have permission to, because the 
> other person did. (Security hole)
> b) Disallow a user from doing something they should have permission to (More 
> common - annoying and can cause job failures)



--
This message was sent by Atlassian JIRA
(v6.1.5#6160)

[jira] [Updated] (HIVE-6378) HCatClient::createTable() doesn't allow SerDe class to be specified

[Thejas M Nair (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-6378?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Thejas M Nair updated HIVE-6378:


Resolution: Fixed
Status: Resolved  (was: Patch Available)

Patch committed to trunk.
Thanks for the contribution Mithun and Karl !


> HCatClient::createTable() doesn't allow SerDe class to be specified
> ---
>
> Key: HIVE-6378
> URL: https://issues.apache.org/jira/browse/HIVE-6378
> Project: Hive
>  Issue Type: Improvement
>  Components: HCatalog
>Affects Versions: 0.13.0
>Reporter: Karl D. Gierach
>Assignee: Karl D. Gierach
>  Labels: patch
> Fix For: 0.13.0
>
> Attachments: HIVE-6378-1.patch, HIVE-6378.1.patch
>
>   Original Estimate: 4h
>  Time Spent: 4m
>  Remaining Estimate: 3h 56m
>
> Recreating the HCATALOG-641 under HIVE, since HCATALOG was moved into HIVE.
> With respect to HCATALOG-641, a patch was originally provided (but not 
> committed), so this work will consist of simply re-basing the original patch 
> to the current trunk and the latest released version.



--
This message was sent by Atlassian JIRA
(v6.1.5#6160)

[jira] [Commented] (HIVE-6339) Implement new JDK7 schema management APIs in java.sql.Connection

[Navis (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-6339?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13900942#comment-13900942
 ] 

Navis commented on HIVE-6339:
-

lgtm, +1. Let's see the test result.

> Implement new JDK7 schema management APIs in java.sql.Connection 
> -
>
> Key: HIVE-6339
> URL: https://issues.apache.org/jira/browse/HIVE-6339
> Project: Hive
>  Issue Type: Improvement
>  Components: JDBC
>Affects Versions: 0.13.0
>Reporter: Prasad Mujumdar
>Assignee: Prasad Mujumdar
> Attachments: HIVE-6339.1.patch, HIVE-6339.2.patch
>
>
> JDK7 has added a few metadata methods in 
> [java.sql.Conntion|http://docs.oracle.com/javase/7/docs/api/java/sql/Connection.html]
>  
> {noformat}
> getSchema()
> setSchema()
> getCatalog()
> setCatalog()
> {noformat}
> Currently Hive JDBC just has stub implementation for all these methods throws 
> unsupported exception. This needs to be fixed.



--
This message was sent by Atlassian JIRA
(v6.1.5#6160)

[jira] [Commented] (HIVE-6378) HCatClient::createTable() doesn't allow SerDe class to be specified

[Hive QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-6378?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13900923#comment-13900923
 ] 

Hive QA commented on HIVE-6378:
---



{color:green}Overall{color}: +1 all checks pass

Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12628856/HIVE-6378.1.patch

{color:green}SUCCESS:{color} +1 5095 tests passed

Test results: 
http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/1317/testReport
Console output: 
http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/1317/console

Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}

This message is automatically generated.

ATTACHMENT ID: 12628856

> HCatClient::createTable() doesn't allow SerDe class to be specified
> ---
>
> Key: HIVE-6378
> URL: https://issues.apache.org/jira/browse/HIVE-6378
> Project: Hive
>  Issue Type: Improvement
>  Components: HCatalog
>Affects Versions: 0.13.0
>Reporter: Karl D. Gierach
>Assignee: Karl D. Gierach
>  Labels: patch
> Fix For: 0.13.0
>
> Attachments: HIVE-6378-1.patch, HIVE-6378.1.patch
>
>   Original Estimate: 4h
>  Time Spent: 4m
>  Remaining Estimate: 3h 56m
>
> Recreating the HCATALOG-641 under HIVE, since HCATALOG was moved into HIVE.
> With respect to HCATALOG-641, a patch was originally provided (but not 
> committed), so this work will consist of simply re-basing the original patch 
> to the current trunk and the latest released version.



--
This message was sent by Atlassian JIRA
(v6.1.5#6160)

[jira] [Commented] (HIVE-5155) Support secure proxy user access to HiveServer2

[Thejas M Nair (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-5155?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13900906#comment-13900906
 ] 

Thejas M Nair commented on HIVE-5155:
-

Prasad, I also have some comments in review board. One of them is on the lines 
of Owen's comment above.

> Support secure proxy user access to HiveServer2
> ---
>
> Key: HIVE-5155
> URL: https://issues.apache.org/jira/browse/HIVE-5155
> Project: Hive
>  Issue Type: Improvement
>  Components: Authentication, HiveServer2, JDBC
>Affects Versions: 0.12.0
>Reporter: Prasad Mujumdar
>Assignee: Prasad Mujumdar
> Attachments: HIVE-5155-1-nothrift.patch, HIVE-5155-noThrift.2.patch, 
> HIVE-5155-noThrift.4.patch, HIVE-5155-noThrift.5.patch, 
> HIVE-5155-noThrift.6.patch, HIVE-5155.1.patch, HIVE-5155.2.patch, 
> HIVE-5155.3.patch, ProxyAuth.java, ProxyAuth.out, TestKERBEROS_Hive_JDBC.java
>
>
> The HiveServer2 can authenticate a client using via Kerberos and impersonate 
> the connecting user with underlying secure hadoop. This becomes a gateway for 
> a remote client to access secure hadoop cluster. Now this works fine for when 
> the client obtains Kerberos ticket and directly connects to HiveServer2. 
> There's another big use case for middleware tools where the end user wants to 
> access Hive via another server. For example Oozie action or Hue submitting 
> queries or a BI tool server accessing to HiveServer2. In these cases, the 
> third party server doesn't have end user's Kerberos credentials and hence it 
> can't submit queries to HiveServer2 on behalf of the end user.
> This ticket is for enabling proxy access to HiveServer2 for third party tools 
> on behalf of end users. There are two parts of the solution proposed in this 
> ticket:
> 1) Delegation token based connection for Oozie (OOZIE-1457)
> This is the common mechanism for Hadoop ecosystem components. Hive Remote 
> Metastore and HCatalog already support this. This is suitable for tool like 
> Oozie that submits the MR jobs as actions on behalf of its client. Oozie 
> already uses similar mechanism for Metastore/HCatalog access.
> 2) Direct proxy access for privileged hadoop users
> The delegation token implementation can be a challenge for non-hadoop 
> (especially non-java) components. This second part enables a privileged user 
> to directly specify an alternate session user during the connection. If the 
> connecting user has hadoop level privilege to impersonate the requested 
> userid, then HiveServer2 will run the session as that requested user. For 
> example, user Hue is allowed to impersonate user Bob (via core-site.xml proxy 
> user configuration). Then user Hue can connect to HiveServer2 and specify Bob 
> as session user via a session property. HiveServer2 will verify Hue's proxy 
> user privilege and then impersonate user Bob instead of Hue. This will enable 
> any third party tool to impersonate alternate userid without having to 
> implement delegation token connection.



--
This message was sent by Atlassian JIRA
(v6.1.5#6160)

[jira] [Updated] (HIVE-6415) Disallow transform clause in sql std authorization mode

[Ashutosh Chauhan (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-6415?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ashutosh Chauhan updated HIVE-6415:
---

Attachment: HIVE-6415.2.patch

Re-upload for Hive QA to run.

> Disallow transform clause in sql std authorization mode
> ---
>
> Key: HIVE-6415
> URL: https://issues.apache.org/jira/browse/HIVE-6415
> Project: Hive
>  Issue Type: Sub-task
>  Components: Authorization
>Reporter: Ashutosh Chauhan
>Assignee: Ashutosh Chauhan
> Attachments: HIVE-6415.1.patch, HIVE-6415.2.patch, HIVE-6415.patch
>
>




--
This message was sent by Atlassian JIRA
(v6.1.5#6160)

[jira] [Updated] (HIVE-6386) sql std auth - database should have an owner

[Ashutosh Chauhan (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-6386?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ashutosh Chauhan updated HIVE-6386:
---

Status: Patch Available  (was: Open)

> sql std auth - database should have an owner
> 
>
> Key: HIVE-6386
> URL: https://issues.apache.org/jira/browse/HIVE-6386
> Project: Hive
>  Issue Type: Sub-task
>  Components: Authorization, Metastore
>Reporter: Thejas M Nair
>Assignee: Ashutosh Chauhan
> Attachments: HIVE-6386.1.patch, HIVE-6386.2.patch, HIVE-6386.patch
>
>
> Database in metastore does not have owner associated with it. Database owner 
> is needed for sql std authorization rules.



--
This message was sent by Atlassian JIRA
(v6.1.5#6160)

[jira] [Updated] (HIVE-6415) Disallow transform clause in sql std authorization mode

[Ashutosh Chauhan (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-6415?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ashutosh Chauhan updated HIVE-6415:
---

Status: Patch Available  (was: Open)

> Disallow transform clause in sql std authorization mode
> ---
>
> Key: HIVE-6415
> URL: https://issues.apache.org/jira/browse/HIVE-6415
> Project: Hive
>  Issue Type: Sub-task
>  Components: Authorization
>Reporter: Ashutosh Chauhan
>Assignee: Ashutosh Chauhan
> Attachments: HIVE-6415.1.patch, HIVE-6415.2.patch, HIVE-6415.patch
>
>




--
This message was sent by Atlassian JIRA
(v6.1.5#6160)

[jira] [Updated] (HIVE-6415) Disallow transform clause in sql std authorization mode

[Ashutosh Chauhan (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-6415?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ashutosh Chauhan updated HIVE-6415:
---

Status: Open  (was: Patch Available)

> Disallow transform clause in sql std authorization mode
> ---
>
> Key: HIVE-6415
> URL: https://issues.apache.org/jira/browse/HIVE-6415
> Project: Hive
>  Issue Type: Sub-task
>  Components: Authorization
>Reporter: Ashutosh Chauhan
>Assignee: Ashutosh Chauhan
> Attachments: HIVE-6415.1.patch, HIVE-6415.patch
>
>




--
This message was sent by Atlassian JIRA
(v6.1.5#6160)

[jira] [Commented] (HIVE-5155) Support secure proxy user access to HiveServer2

[Prasad Mujumdar (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-5155?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13900893#comment-13900893
 ] 

Prasad Mujumdar commented on HIVE-5155:
---

[~thejas] Thanks! I will rebase the patch and add dock notes.

> Support secure proxy user access to HiveServer2
> ---
>
> Key: HIVE-5155
> URL: https://issues.apache.org/jira/browse/HIVE-5155
> Project: Hive
>  Issue Type: Improvement
>  Components: Authentication, HiveServer2, JDBC
>Affects Versions: 0.12.0
>Reporter: Prasad Mujumdar
>Assignee: Prasad Mujumdar
> Attachments: HIVE-5155-1-nothrift.patch, HIVE-5155-noThrift.2.patch, 
> HIVE-5155-noThrift.4.patch, HIVE-5155-noThrift.5.patch, 
> HIVE-5155-noThrift.6.patch, HIVE-5155.1.patch, HIVE-5155.2.patch, 
> HIVE-5155.3.patch, ProxyAuth.java, ProxyAuth.out, TestKERBEROS_Hive_JDBC.java
>
>
> The HiveServer2 can authenticate a client using via Kerberos and impersonate 
> the connecting user with underlying secure hadoop. This becomes a gateway for 
> a remote client to access secure hadoop cluster. Now this works fine for when 
> the client obtains Kerberos ticket and directly connects to HiveServer2. 
> There's another big use case for middleware tools where the end user wants to 
> access Hive via another server. For example Oozie action or Hue submitting 
> queries or a BI tool server accessing to HiveServer2. In these cases, the 
> third party server doesn't have end user's Kerberos credentials and hence it 
> can't submit queries to HiveServer2 on behalf of the end user.
> This ticket is for enabling proxy access to HiveServer2 for third party tools 
> on behalf of end users. There are two parts of the solution proposed in this 
> ticket:
> 1) Delegation token based connection for Oozie (OOZIE-1457)
> This is the common mechanism for Hadoop ecosystem components. Hive Remote 
> Metastore and HCatalog already support this. This is suitable for tool like 
> Oozie that submits the MR jobs as actions on behalf of its client. Oozie 
> already uses similar mechanism for Metastore/HCatalog access.
> 2) Direct proxy access for privileged hadoop users
> The delegation token implementation can be a challenge for non-hadoop 
> (especially non-java) components. This second part enables a privileged user 
> to directly specify an alternate session user during the connection. If the 
> connecting user has hadoop level privilege to impersonate the requested 
> userid, then HiveServer2 will run the session as that requested user. For 
> example, user Hue is allowed to impersonate user Bob (via core-site.xml proxy 
> user configuration). Then user Hue can connect to HiveServer2 and specify Bob 
> as session user via a session property. HiveServer2 will verify Hue's proxy 
> user privilege and then impersonate user Bob instead of Hue. This will enable 
> any third party tool to impersonate alternate userid without having to 
> implement delegation token connection.



--
This message was sent by Atlassian JIRA
(v6.1.5#6160)

Re: Review Request 18094: HIVE-6339: Implement new JDK7 schema management APIs in java.sql.Connection

[Prasad Mujumdar]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/18094/
---

(Updated Feb. 13, 2014, 11:22 p.m.)


Review request for hive.


Changes
---

The new JDBC APIs are not available in JDK6. Updated the test to invoke those 
via Hive specific interface.


Bugs: HIVE-6339
https://issues.apache.org/jira/browse/HIVE-6339


Repository: hive-git


Description
---

Implement new JDK7 schema management APIs in java.sql.Connection.


Diffs (updated)
-

  itests/hive-unit/src/test/java/org/apache/hive/jdbc/TestJdbcWithMiniHS2.java 
e68a5a9 
  jdbc/src/java/org/apache/hive/jdbc/HiveConnection.java 13fc19b 

Diff: https://reviews.apache.org/r/18094/diff/


Testing
---

Added new test case.


Thanks,

Prasad Mujumdar

[jira] [Updated] (HIVE-6386) sql std auth - database should have an owner

[Ashutosh Chauhan (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-6386?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ashutosh Chauhan updated HIVE-6386:
---

Status: Open  (was: Patch Available)

> sql std auth - database should have an owner
> 
>
> Key: HIVE-6386
> URL: https://issues.apache.org/jira/browse/HIVE-6386
> Project: Hive
>  Issue Type: Sub-task
>  Components: Authorization, Metastore
>Reporter: Thejas M Nair
>Assignee: Ashutosh Chauhan
> Attachments: HIVE-6386.1.patch, HIVE-6386.2.patch, HIVE-6386.patch
>
>
> Database in metastore does not have owner associated with it. Database owner 
> is needed for sql std authorization rules.



--
This message was sent by Atlassian JIRA
(v6.1.5#6160)

[jira] [Commented] (HIVE-6386) sql std auth - database should have an owner

[Ashutosh Chauhan (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-6386?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13900882#comment-13900882
 ] 

Ashutosh Chauhan commented on HIVE-6386:


RB request : https://reviews.apache.org/r/18103/

> sql std auth - database should have an owner
> 
>
> Key: HIVE-6386
> URL: https://issues.apache.org/jira/browse/HIVE-6386
> Project: Hive
>  Issue Type: Sub-task
>  Components: Authorization, Metastore
>Reporter: Thejas M Nair
>Assignee: Ashutosh Chauhan
> Attachments: HIVE-6386.1.patch, HIVE-6386.2.patch, HIVE-6386.patch
>
>
> Database in metastore does not have owner associated with it. Database owner 
> is needed for sql std authorization rules.



--
This message was sent by Atlassian JIRA
(v6.1.5#6160)

[jira] [Updated] (HIVE-6386) sql std auth - database should have an owner

[Ashutosh Chauhan (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-6386?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ashutosh Chauhan updated HIVE-6386:
---

Attachment: HIVE-6386.2.patch

> sql std auth - database should have an owner
> 
>
> Key: HIVE-6386
> URL: https://issues.apache.org/jira/browse/HIVE-6386
> Project: Hive
>  Issue Type: Sub-task
>  Components: Authorization, Metastore
>Reporter: Thejas M Nair
>Assignee: Ashutosh Chauhan
> Attachments: HIVE-6386.1.patch, HIVE-6386.2.patch, HIVE-6386.patch
>
>
> Database in metastore does not have owner associated with it. Database owner 
> is needed for sql std authorization rules.



--
This message was sent by Atlassian JIRA
(v6.1.5#6160)

Review Request 18103: Adds owner for DB.

[Ashutosh Chauhan]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/18103/
---

Review request for hive.


Bugs: HIVE-6386
https://issues.apache.org/jira/browse/HIVE-6386


Repository: hive


Description
---

Adds owner for DB.


Diffs
-

  trunk/common/src/java/org/apache/hive/common/HivePrincipal.java PRE-CREATION 
  
trunk/itests/hive-unit/src/test/java/org/apache/hadoop/hive/metastore/TestHiveMetaStore.java
 1568122 
  trunk/metastore/if/hive_metastore.thrift 1568122 
  trunk/metastore/src/gen/thrift/gen-cpp/hive_metastore_types.h 1568122 
  trunk/metastore/src/gen/thrift/gen-cpp/hive_metastore_types.cpp 1568122 
  
trunk/metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/Database.java
 1568122 
  trunk/metastore/src/gen/thrift/gen-php/metastore/Types.php 1568122 
  trunk/metastore/src/gen/thrift/gen-py/hive_metastore/ttypes.py 1568122 
  trunk/metastore/src/gen/thrift/gen-rb/hive_metastore_types.rb 1568122 
  trunk/metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java 
1568122 
  trunk/metastore/src/java/org/apache/hadoop/hive/metastore/ObjectStore.java 
1568122 
  
trunk/metastore/src/model/org/apache/hadoop/hive/metastore/model/MDatabase.java 
1568122 
  trunk/metastore/src/model/package.jdo 1568122 
  trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/DDLTask.java 1568122 
  
trunk/ql/src/java/org/apache/hadoop/hive/ql/metadata/formatting/JsonMetaDataFormatter.java
 1568122 
  
trunk/ql/src/java/org/apache/hadoop/hive/ql/metadata/formatting/MetaDataFormatter.java
 1568122 
  
trunk/ql/src/java/org/apache/hadoop/hive/ql/metadata/formatting/TextMetaDataFormatter.java
 1568122 
  
trunk/ql/src/java/org/apache/hadoop/hive/ql/security/authorization/AuthorizationUtils.java
 1568122 
  
trunk/ql/src/java/org/apache/hadoop/hive/ql/security/authorization/plugin/HiveAccessController.java
 1568122 
  
trunk/ql/src/java/org/apache/hadoop/hive/ql/security/authorization/plugin/HiveAuthorizer.java
 1568122 
  
trunk/ql/src/java/org/apache/hadoop/hive/ql/security/authorization/plugin/HiveAuthorizerImpl.java
 1568122 
  
trunk/ql/src/java/org/apache/hadoop/hive/ql/security/authorization/plugin/HivePrincipal.java
 1568122 
  
trunk/ql/src/java/org/apache/hadoop/hive/ql/security/authorization/plugin/HivePrivilegeInfo.java
 1568122 
  
trunk/ql/src/java/org/apache/hadoop/hive/ql/security/authorization/plugin/sqlstd/GrantPrivAuthUtils.java
 1568122 
  
trunk/ql/src/java/org/apache/hadoop/hive/ql/security/authorization/plugin/sqlstd/RevokePrivAuthUtils.java
 1568122 
  
trunk/ql/src/java/org/apache/hadoop/hive/ql/security/authorization/plugin/sqlstd/SQLAuthorizationUtils.java
 1568122 
  
trunk/ql/src/java/org/apache/hadoop/hive/ql/security/authorization/plugin/sqlstd/SQLStdHiveAccessController.java
 1568122 
  
trunk/ql/src/java/org/apache/hadoop/hive/ql/security/authorization/plugin/sqlstd/SQLStdHiveAuthorizationValidator.java
 1568122 
  trunk/serde/src/gen/thrift/gen-cpp/serde_constants.h 1568122 
  trunk/serde/src/gen/thrift/gen-cpp/serde_constants.cpp 1568122 
  
trunk/serde/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/serde/test/ThriftTestObj.java
 1568122 
  
trunk/serde/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/serde2/thrift/test/Complex.java
 1568122 
  
trunk/serde/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/serde2/thrift/test/MegaStruct.java
 1568122 
  trunk/serde/src/gen/thrift/gen-php/org/apache/hadoop/hive/serde/Types.php 
1568122 
  trunk/serde/src/gen/thrift/gen-py/org_apache_hadoop_hive_serde/constants.py 
1568122 
  trunk/serde/src/gen/thrift/gen-rb/serde_constants.rb 1568122 

Diff: https://reviews.apache.org/r/18103/diff/


Testing
---

Added new test.


Thanks,

Ashutosh Chauhan

[jira] [Updated] (HIVE-6339) Implement new JDK7 schema management APIs in java.sql.Connection

[Prasad Mujumdar (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-6339?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Prasad Mujumdar updated HIVE-6339:
--

Attachment: HIVE-6339.2.patch

The new JDBC APIs are not available in JDK6. Updated the test to invoke those 
via Hive specific interface.

> Implement new JDK7 schema management APIs in java.sql.Connection 
> -
>
> Key: HIVE-6339
> URL: https://issues.apache.org/jira/browse/HIVE-6339
> Project: Hive
>  Issue Type: Improvement
>  Components: JDBC
>Affects Versions: 0.13.0
>Reporter: Prasad Mujumdar
>Assignee: Prasad Mujumdar
> Attachments: HIVE-6339.1.patch, HIVE-6339.2.patch
>
>
> JDK7 has added a few metadata methods in 
> [java.sql.Conntion|http://docs.oracle.com/javase/7/docs/api/java/sql/Connection.html]
>  
> {noformat}
> getSchema()
> setSchema()
> getCatalog()
> setCatalog()
> {noformat}
> Currently Hive JDBC just has stub implementation for all these methods throws 
> unsupported exception. This needs to be fixed.



--
This message was sent by Atlassian JIRA
(v6.1.5#6160)

[jira] [Updated] (HIVE-6330) Metastore support for permanent UDFs

[Jason Dere (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-6330?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jason Dere updated HIVE-6330:
-

Attachment: HIVE-6330.5.patch

patch v5 - rebasing with trunk since HIVE-6167 is in

> Metastore support for permanent UDFs
> 
>
> Key: HIVE-6330
> URL: https://issues.apache.org/jira/browse/HIVE-6330
> Project: Hive
>  Issue Type: Sub-task
>  Components: UDF
>Reporter: Jason Dere
>Assignee: Jason Dere
> Attachments: HIVE-6330.1.patch, HIVE-6330.2.patch, HIVE-6330.3.patch, 
> HIVE-6330.4.patch, HIVE-6330.5.patch
>
>
> Allow CREATE FUNCTION to add metastore entry for the created function, so 
> that it only needs to be added to Hive once.



--
This message was sent by Atlassian JIRA
(v6.1.5#6160)

[jira] [Commented] (HIVE-6386) sql std auth - database should have an owner

[Thejas M Nair (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-6386?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13900851#comment-13900851
 ] 

Thejas M Nair commented on HIVE-6386:
-

can you create a review board link ?

> sql std auth - database should have an owner
> 
>
> Key: HIVE-6386
> URL: https://issues.apache.org/jira/browse/HIVE-6386
> Project: Hive
>  Issue Type: Sub-task
>  Components: Authorization, Metastore
>Reporter: Thejas M Nair
>Assignee: Ashutosh Chauhan
> Attachments: HIVE-6386.1.patch, HIVE-6386.patch
>
>
> Database in metastore does not have owner associated with it. Database owner 
> is needed for sql std authorization rules.



--
This message was sent by Atlassian JIRA
(v6.1.5#6160)

[jira] [Updated] (HIVE-6428) concat_ws mangles non-ASCII characters

[Nick Dimiduk (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-6428?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Nick Dimiduk updated HIVE-6428:
---

Attachment: ru.txt

> concat_ws mangles non-ASCII characters
> --
>
> Key: HIVE-6428
> URL: https://issues.apache.org/jira/browse/HIVE-6428
> Project: Hive
>  Issue Type: Bug
>  Components: UDF
>Affects Versions: 0.13.0
>Reporter: Nick Dimiduk
>Priority: Critical
> Attachments: ru.txt
>
>
> Marked critical because this results in data loss from using built-in 
> functionality. I think the issue is {{concat_ws}}, though I suppose it could 
> be the VIEW as well.
> Hive is losing the distinction between non-ASCII characters, folding distinct 
> values into the same value. Here are steps to reproduce, and I've attached a 
> small sample containing 3 distinct lines from the larger input file.
> Grab sample data, confirm the number of total records and the number of 
> unique combinations of the first two columns match.
> {noformat}
> $ mkdir /tmp/pagecounts
> $ cd /tmp/pagecounts
> $ wget 
> http://dumps.wikimedia.org/other/pagecounts-raw/2008/2008-10/pagecounts-20081001-00.gz
> $ gzcat pagecounts-20081001-00.gz | cut -d\  -f1,2 | wc -l
>  4268675
> $ gzcat pagecounts-20081001-00.gz | cut -d\  -f1,2 | sort | uniq | wc -l
>  4268675
> {noformat}
> Create hive table over input data.
> {noformat}
> CREATE EXTERNAL TABLE pagecounts (projectcode STRING, pagename STRING, 
> pageviews STRING, bytes STRING)
> ROW FORMAT
>   DELIMITED FIELDS TERMINATED BY ' '
>   LINES TERMINATED BY '\n'
> STORED AS TEXTFILE
> LOCATION '/tmp/pagecounts/';
> {noformat}
> confirm number of unique combinations of the first two columns
> {noformat}
> SELECT count(DISTINCT projectcode, pagename) FROM pagecounts;
> => 4268675
> {noformat}
> Create a view over the raw data, concatenating first two columns. Distinct 
> count does not match.
> {noformat}
> CREATE VIEW pgc_simple (rowkey, pageviews, bytes) AS
> SELECT concat_ws('/', projectcode, pagename), pageviews, bytes
> FROM pagecounts;
> SELECT count(DISTINCT rowkey) FROM pgc_simple;
> => 4268561
> {noformat}
> Perform same "view" from shell. distinct count is retained.
> {noformat}
> $ gzcat pagecounts-20081001-00.gz | awk '{print $1 "/" $2}' | wc -l
>  4268675
> $ gzcat pagecounts-20081001-00.gz | awk '{print $1 "/" $2}' | sort | uniq 
> | wc -l   
> 
>  4268675
> {noformat}
> Look at some data.
> {noformat}
> $ hive -e "SELECT i.* FROM (SELECT rowkey, count(*) AS nbr FROM pgc_simple 
> GROUP BY rowkey) i WHERE i.nbr > 1;" | tail
> OK
> Time taken: 40.493 seconds, Fetched: 37 row(s)
> ru/?�N  2
> ru/�3
> zh/?��  2
> zh/Category:��  2
> zh/Wikipedia:�� 5
> zh/�2
> zh/��   11
> zh/��˹��2
> zh/ 17
> zh/��   15
> {noformat}
> Choose 2nd line of output to inspect on the shell. My locale isn't able to 
> find a character for the codepoints, but sort | uniq identify them as 
> different.
> {noformat}
> $ gzcat pagecounts-20081001-00.gz | egrep '^ru \?. ' | cut -d\  -f1,2 | 
> sort | uniq
> ru ??
> ru ??
> ru ??
> {noformat}
> Print them as C-escape codes. They are indeed distinct.
> {noformat}
> $ cat pagecounts-20081001-00 | egrep '^ru \?. ' | cut -d\  -f1,2 | od -c
> 000r   u   ? 240  \n   r   u   ? 244  \n   r   u   ?
> 020  247  \n
> 022
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.1.5#6160)

[jira] [Created] (HIVE-6428) concat_ws mangles non-ASCII characters

[Nick Dimiduk (JIRA)]

Nick Dimiduk created HIVE-6428:
--

 Summary: concat_ws mangles non-ASCII characters
 Key: HIVE-6428
 URL: https://issues.apache.org/jira/browse/HIVE-6428
 Project: Hive
  Issue Type: Bug
  Components: UDF
Affects Versions: 0.13.0
Reporter: Nick Dimiduk
Priority: Critical
 Attachments: ru.txt

Marked critical because this results in data loss from using built-in 
functionality. I think the issue is {{concat_ws}}, though I suppose it could be 
the VIEW as well.

Hive is losing the distinction between non-ASCII characters, folding distinct 
values into the same value. Here are steps to reproduce, and I've attached a 
small sample containing 3 distinct lines from the larger input file.

Grab sample data, confirm the number of total records and the number of unique 
combinations of the first two columns match.
{noformat}
$ mkdir /tmp/pagecounts
$ cd /tmp/pagecounts
$ wget 
http://dumps.wikimedia.org/other/pagecounts-raw/2008/2008-10/pagecounts-20081001-00.gz
$ gzcat pagecounts-20081001-00.gz | cut -d\  -f1,2 | wc -l
 4268675
$ gzcat pagecounts-20081001-00.gz | cut -d\  -f1,2 | sort | uniq | wc -l
 4268675
{noformat}

Create hive table over input data.
{noformat}
CREATE EXTERNAL TABLE pagecounts (projectcode STRING, pagename STRING, 
pageviews STRING, bytes STRING)
ROW FORMAT
  DELIMITED FIELDS TERMINATED BY ' '
  LINES TERMINATED BY '\n'
STORED AS TEXTFILE
LOCATION '/tmp/pagecounts/';
{noformat}

confirm number of unique combinations of the first two columns
{noformat}
SELECT count(DISTINCT projectcode, pagename) FROM pagecounts;
=> 4268675
{noformat}

Create a view over the raw data, concatenating first two columns. Distinct 
count does not match.
{noformat}
CREATE VIEW pgc_simple (rowkey, pageviews, bytes) AS
SELECT concat_ws('/', projectcode, pagename), pageviews, bytes
FROM pagecounts;

SELECT count(DISTINCT rowkey) FROM pgc_simple;
=> 4268561
{noformat}

Perform same "view" from shell. distinct count is retained.
{noformat}
$ gzcat pagecounts-20081001-00.gz | awk '{print $1 "/" $2}' | wc -l
 4268675
$ gzcat pagecounts-20081001-00.gz | awk '{print $1 "/" $2}' | sort | uniq | 
wc -l   

 4268675
{noformat}

Look at some data.
{noformat}
$ hive -e "SELECT i.* FROM (SELECT rowkey, count(*) AS nbr FROM pgc_simple 
GROUP BY rowkey) i WHERE i.nbr > 1;" | tail
OK
Time taken: 40.493 seconds, Fetched: 37 row(s)
ru/?�N  2
ru/�3
zh/?��  2
zh/Category:��  2
zh/Wikipedia:�� 5
zh/�2
zh/��   11
zh/��˹��2
zh/ 17
zh/��   15
{noformat}

Choose 2nd line of output to inspect on the shell. My locale isn't able to find 
a character for the codepoints, but sort | uniq identify them as different.
{noformat}
$ gzcat pagecounts-20081001-00.gz | egrep '^ru \?. ' | cut -d\  -f1,2 | 
sort | uniq
ru ??
ru ??
ru ??
{noformat}

Print them as C-escape codes. They are indeed distinct.
{noformat}
$ cat pagecounts-20081001-00 | egrep '^ru \?. ' | cut -d\  -f1,2 | od -c
000r   u   ? 240  \n   r   u   ? 244  \n   r   u   ?
020  247  \n
022
{noformat}



--
This message was sent by Atlassian JIRA
(v6.1.5#6160)

[jira] [Updated] (HIVE-6386) sql std auth - database should have an owner

[Ashutosh Chauhan (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-6386?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ashutosh Chauhan updated HIVE-6386:
---

Status: Patch Available  (was: Open)

> sql std auth - database should have an owner
> 
>
> Key: HIVE-6386
> URL: https://issues.apache.org/jira/browse/HIVE-6386
> Project: Hive
>  Issue Type: Sub-task
>  Components: Authorization, Metastore
>Reporter: Thejas M Nair
>Assignee: Ashutosh Chauhan
> Attachments: HIVE-6386.1.patch, HIVE-6386.patch
>
>
> Database in metastore does not have owner associated with it. Database owner 
> is needed for sql std authorization rules.



--
This message was sent by Atlassian JIRA
(v6.1.5#6160)

[jira] [Updated] (HIVE-6386) sql std auth - database should have an owner

[Ashutosh Chauhan (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-6386?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ashutosh Chauhan updated HIVE-6386:
---

Attachment: HIVE-6386.1.patch

> sql std auth - database should have an owner
> 
>
> Key: HIVE-6386
> URL: https://issues.apache.org/jira/browse/HIVE-6386
> Project: Hive
>  Issue Type: Sub-task
>  Components: Authorization, Metastore
>Reporter: Thejas M Nair
>Assignee: Ashutosh Chauhan
> Attachments: HIVE-6386.1.patch, HIVE-6386.patch
>
>
> Database in metastore does not have owner associated with it. Database owner 
> is needed for sql std authorization rules.



--
This message was sent by Atlassian JIRA
(v6.1.5#6160)

[jira] [Updated] (HIVE-6386) sql std auth - database should have an owner

[Ashutosh Chauhan (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-6386?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ashutosh Chauhan updated HIVE-6386:
---

Status: Open  (was: Patch Available)

> sql std auth - database should have an owner
> 
>
> Key: HIVE-6386
> URL: https://issues.apache.org/jira/browse/HIVE-6386
> Project: Hive
>  Issue Type: Sub-task
>  Components: Authorization, Metastore
>Reporter: Thejas M Nair
>Assignee: Ashutosh Chauhan
> Attachments: HIVE-6386.patch
>
>
> Database in metastore does not have owner associated with it. Database owner 
> is needed for sql std authorization rules.



--
This message was sent by Atlassian JIRA
(v6.1.5#6160)

[jira] [Commented] (HIVE-6339) Implement new JDK7 schema management APIs in java.sql.Connection

[Hive QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-6339?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13900815#comment-13900815
 ] 

Hive QA commented on HIVE-6339:
---



{color:red}Overall{color}: -1 no tests executed

Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12628843/HIVE-6339.1.patch

Test results: 
http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/1316/testReport
Console output: 
http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/1316/console

Messages:
{noformat}
 This message was trimmed, see log for full details 
[WARNING] JAR will be empty - no content was marked for inclusion!
[INFO] Building jar: 
/data/hive-ptest/working/apache-svn-trunk-source/itests/hcatalog-unit/target/hive-hcatalog-it-unit-0.13.0-SNAPSHOT.jar
[INFO] 
[INFO] --- maven-jar-plugin:2.2:test-jar (default) @ hive-hcatalog-it-unit ---
[INFO] Building jar: 
/data/hive-ptest/working/apache-svn-trunk-source/itests/hcatalog-unit/target/hive-hcatalog-it-unit-0.13.0-SNAPSHOT-tests.jar
[INFO] 
[INFO] --- maven-install-plugin:2.4:install (default-install) @ 
hive-hcatalog-it-unit ---
[INFO] Installing 
/data/hive-ptest/working/apache-svn-trunk-source/itests/hcatalog-unit/target/hive-hcatalog-it-unit-0.13.0-SNAPSHOT.jar
 to 
/data/hive-ptest/working/maven/org/apache/hive/hive-hcatalog-it-unit/0.13.0-SNAPSHOT/hive-hcatalog-it-unit-0.13.0-SNAPSHOT.jar
[INFO] Installing 
/data/hive-ptest/working/apache-svn-trunk-source/itests/hcatalog-unit/pom.xml 
to 
/data/hive-ptest/working/maven/org/apache/hive/hive-hcatalog-it-unit/0.13.0-SNAPSHOT/hive-hcatalog-it-unit-0.13.0-SNAPSHOT.pom
[INFO] Installing 
/data/hive-ptest/working/apache-svn-trunk-source/itests/hcatalog-unit/target/hive-hcatalog-it-unit-0.13.0-SNAPSHOT-tests.jar
 to 
/data/hive-ptest/working/maven/org/apache/hive/hive-hcatalog-it-unit/0.13.0-SNAPSHOT/hive-hcatalog-it-unit-0.13.0-SNAPSHOT-tests.jar
[INFO] 
[INFO] 
[INFO] Building Hive Integration - Testing Utilities 0.13.0-SNAPSHOT
[INFO] 
[INFO] 
[INFO] --- maven-clean-plugin:2.5:clean (default-clean) @ hive-it-util ---
[INFO] Deleting /data/hive-ptest/working/apache-svn-trunk-source/itests/util 
(includes = [datanucleus.log, derby.log], excludes = [])
[INFO] 
[INFO] --- maven-resources-plugin:2.5:resources (default-resources) @ 
hive-it-util ---
[debug] execute contextualize
[INFO] Using 'UTF-8' encoding to copy filtered resources.
[INFO] skip non existing resourceDirectory 
/data/hive-ptest/working/apache-svn-trunk-source/itests/util/src/main/resources
[INFO] 
[INFO] --- maven-antrun-plugin:1.7:run (define-classpath) @ hive-it-util ---
[INFO] Executing tasks

main:
[INFO] Executed tasks
[INFO] 
[INFO] --- maven-compiler-plugin:3.1:compile (default-compile) @ hive-it-util 
---
[INFO] Compiling 41 source files to 
/data/hive-ptest/working/apache-svn-trunk-source/itests/util/target/classes
[WARNING] Note: Some input files use or override a deprecated API.
[WARNING] Note: Recompile with -Xlint:deprecation for details.
[INFO] 
[INFO] --- maven-resources-plugin:2.5:testResources (default-testResources) @ 
hive-it-util ---
[debug] execute contextualize
[INFO] Using 'UTF-8' encoding to copy filtered resources.
[INFO] skip non existing resourceDirectory 
/data/hive-ptest/working/apache-svn-trunk-source/itests/util/src/test/resources
[INFO] 
[INFO] --- maven-antrun-plugin:1.7:run (setup-test-dirs) @ hive-it-util ---
[INFO] Executing tasks

main:
[mkdir] Created dir: 
/data/hive-ptest/working/apache-svn-trunk-source/itests/util/target/tmp
[mkdir] Created dir: 
/data/hive-ptest/working/apache-svn-trunk-source/itests/util/target/warehouse
[mkdir] Created dir: 
/data/hive-ptest/working/apache-svn-trunk-source/itests/util/target/tmp/conf
 [copy] Copying 5 files to 
/data/hive-ptest/working/apache-svn-trunk-source/itests/util/target/tmp/conf
[INFO] Executed tasks
[INFO] 
[INFO] --- maven-compiler-plugin:3.1:testCompile (default-testCompile) @ 
hive-it-util ---
[INFO] No sources to compile
[INFO] 
[INFO] --- maven-surefire-plugin:2.16:test (default-test) @ hive-it-util ---
[INFO] Tests are skipped.
[INFO] 
[INFO] --- maven-jar-plugin:2.2:jar (default-jar) @ hive-it-util ---
[INFO] Building jar: 
/data/hive-ptest/working/apache-svn-trunk-source/itests/util/target/hive-it-util-0.13.0-SNAPSHOT.jar
[INFO] 
[INFO] --- maven-install-plugin:2.4:install (default-install) @ hive-it-util ---
[INFO] Installing 
/data/hive-ptest/working/apache-svn-trunk-source/itests/util/target/hive-it-util-0.13.0-SNAPSHOT.jar
 to 
/data/hive-ptest/working/maven/org/apache/hive/hive-it-util/0.13.0-SNAPSHOT/hive-it-util-0.13.0-SNAPSHOT.jar
[INFO] Installing 
/data/hive-ptest/working/apache-svn-trun

[jira] [Commented] (HIVE-6403) uncorrelated subquery is failing with auto.convert.join=true

[Hive QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-6403?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13900810#comment-13900810
 ] 

Hive QA commented on HIVE-6403:
---



{color:red}Overall{color}: -1 at least one tests failed

Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12628832/HIVE-6403.2.patch

{color:red}ERROR:{color} -1 due to 22 failed/errored test(s), 5095 tests 
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_join1
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_join11
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_join12
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_join13
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_join15
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_join17
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_join2
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_join22
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_join23
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_join24
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_join27
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_join29
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_join3
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_join32
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_join9
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_sortmerge_join_10
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_join34
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_join_reorder4
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_multiMapJoin2
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_multi_join_union
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_union34
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_vectorized_mapjoin
{noformat}

Test results: 
http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/1315/testReport
Console output: 
http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/1315/console

Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 22 tests failed
{noformat}

This message is automatically generated.

ATTACHMENT ID: 12628832

> uncorrelated subquery is failing with auto.convert.join=true
> 
>
> Key: HIVE-6403
> URL: https://issues.apache.org/jira/browse/HIVE-6403
> Project: Hive
>  Issue Type: Bug
>  Components: Query Processor
>Reporter: Navis
>Assignee: Harish Butani
> Attachments: HIVE-6403.1.patch, HIVE-6403.2.patch, navis.patch
>
>
> Fixing HIVE-5690, I've found query in subquery_multiinsert.q is not working 
> with hive.auto.convert.join=true 
> {noformat}
> set hive.auto.convert.join=true;
> hive> explain
> > from src b 
> > INSERT OVERWRITE TABLE src_4 
> >   select * 
> >   where b.key in 
> >(select a.key 
> > from src a 
> > where b.value = a.value and a.key > '9'
> >) 
> > INSERT OVERWRITE TABLE src_5 
> >   select *  
> >   where b.key not in  ( select key from src s1 where s1.key > '2') 
> >   order by key 
> > ;
> java.lang.IndexOutOfBoundsException: Index: 0, Size: 0
>   at java.util.ArrayList.rangeCheck(ArrayList.java:635)
>   at java.util.ArrayList.get(ArrayList.java:411)
>   at 
> org.apache.hadoop.hive.ql.optimizer.MapJoinProcessor.genMapJoinLocalWork(MapJoinProcessor.java:149)
>   at 
> org.apache.hadoop.hive.ql.optimizer.MapJoinProcessor.genLocalWorkForMapJoin(MapJoinProcessor.java:256)
>   at 
> org.apache.hadoop.hive.ql.optimizer.MapJoinProcessor.genMapJoinOpAndLocalWork(MapJoinProcessor.java:248)
>   at 
> org.apache.hadoop.hive.ql.optimizer.physical.CommonJoinTaskDispatcher.convertTaskToMapJoinTask(CommonJoinTaskDispatcher.java:191)
>   at 
> org.apache.hadoop.hive.ql.optimizer.physical.CommonJoinTaskDispatcher.processCurrentTask(CommonJoinTaskDispatcher.java:481)
>   at 
> org.apache.hadoop.hive.ql.optimizer.physical.AbstractJoinTaskDispatcher.dispatch(AbstractJoinTaskDispatcher.java:182)
>   at 
> org.apache.hadoop.hive.ql.lib.TaskGraphWalker.dispatch(TaskGraphWalker.java:111)
>   at 
> org.apache.hadoop.hive.ql.lib.TaskGraphWalker.walk(TaskGraphWalker.java:194)
>   at 
> org.apache.hadoop.hive.ql.lib.TaskGraphWalker.startWalking(TaskGraphWalker.java:139)
>   at 
> org.apache.hadoop.hive.ql.optimizer.physical.CommonJoinResolver.resolve(CommonJoinResolver.java:79)
>   at 
> org.apache.hadoop.hive.ql.optimizer.physical.PhysicalOptimizer.optim

[jira] [Updated] (HIVE-6372) getDatabaseMajor/Minor version returns wrong values

[Prasad Mujumdar (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-6372?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Prasad Mujumdar updated HIVE-6372:
--

Resolution: Fixed
Status: Resolved  (was: Patch Available)

Patch committed to trunk. Thanks Szehon!

> getDatabaseMajor/Minor version returns wrong values
> ---
>
> Key: HIVE-6372
> URL: https://issues.apache.org/jira/browse/HIVE-6372
> Project: Hive
>  Issue Type: Bug
>  Components: JDBC
>Reporter: Szehon Ho
>Assignee: Szehon Ho
> Attachments: HIVE-6372.patch
>
>
> Currently getDatabaseMajorVersion returns 13, and getDatabaseMinorVersion 
> returns 0.   The index is off by one.



--
This message was sent by Atlassian JIRA
(v6.1.5#6160)

[jira] [Updated] (HIVE-6427) Hive Server2 should reopen Metastore client in case of any Thrift exceptions

[Andrey Stepachev (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-6427?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Andrey Stepachev updated HIVE-6427:
---

Attachment: 0001-HIVE-6427-Hive-Server2-should-reopen-Metastore-clien.patch

> Hive Server2 should reopen Metastore client in case of any Thrift exceptions
> 
>
> Key: HIVE-6427
> URL: https://issues.apache.org/jira/browse/HIVE-6427
> Project: Hive
>  Issue Type: Bug
>  Components: HiveServer2
>Affects Versions: 0.12.0
> Environment: cloudera cdh5 beta2
>Reporter: Andrey Stepachev
>Priority: Critical
> Attachments: 
> 0001-HIVE-6427-Hive-Server2-should-reopen-Metastore-clien.patch
>
>
> In case of metastore restart hive server doesn't reopen connection to 
> metastore. Any command gives broken pipe or similar exceptions.
> http://paste.ubuntu.com/6926215/
> Any subsequent command doesn't reestablish connection and tries to use stale 
> (closed) connection.
> Looks like we shouldn't blindly convert any MetaException to 
> HiveSQLException, but should distinguish between fatal exceptions and logical 
> exceptions.



--
This message was sent by Atlassian JIRA
(v6.1.5#6160)

[jira] [Updated] (HIVE-6427) Hive Server2 should reopen Metastore client in case of any Thrift exceptions

[Andrey Stepachev (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-6427?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Andrey Stepachev updated HIVE-6427:
---

Status: Open  (was: Patch Available)

> Hive Server2 should reopen Metastore client in case of any Thrift exceptions
> 
>
> Key: HIVE-6427
> URL: https://issues.apache.org/jira/browse/HIVE-6427
> Project: Hive
>  Issue Type: Bug
>  Components: HiveServer2
>Affects Versions: 0.12.0
> Environment: cloudera cdh5 beta2
>Reporter: Andrey Stepachev
>Priority: Critical
>
> In case of metastore restart hive server doesn't reopen connection to 
> metastore. Any command gives broken pipe or similar exceptions.
> http://paste.ubuntu.com/6926215/
> Any subsequent command doesn't reestablish connection and tries to use stale 
> (closed) connection.
> Looks like we shouldn't blindly convert any MetaException to 
> HiveSQLException, but should distinguish between fatal exceptions and logical 
> exceptions.



--
This message was sent by Atlassian JIRA
(v6.1.5#6160)

[jira] [Updated] (HIVE-6427) Hive Server2 should reopen Metastore client in case of any Thrift exceptions

[Andrey Stepachev (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-6427?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Andrey Stepachev updated HIVE-6427:
---

Status: Patch Available  (was: Open)

I make a patch against cloudera branch cdh5-0.12.0_5.0.0b2 which somehow solves 
that problem. 
It is copy&paste of RetryingMetaStoreClient initialisation. Also, for properly 
working proxy code I've removed unnecessary exception mangling.

> Hive Server2 should reopen Metastore client in case of any Thrift exceptions
> 
>
> Key: HIVE-6427
> URL: https://issues.apache.org/jira/browse/HIVE-6427
> Project: Hive
>  Issue Type: Bug
>  Components: HiveServer2
>Affects Versions: 0.12.0
> Environment: cloudera cdh5 beta2
>Reporter: Andrey Stepachev
>Priority: Critical
>
> In case of metastore restart hive server doesn't reopen connection to 
> metastore. Any command gives broken pipe or similar exceptions.
> http://paste.ubuntu.com/6926215/
> Any subsequent command doesn't reestablish connection and tries to use stale 
> (closed) connection.
> Looks like we shouldn't blindly convert any MetaException to 
> HiveSQLException, but should distinguish between fatal exceptions and logical 
> exceptions.



--
This message was sent by Atlassian JIRA
(v6.1.5#6160)

Re: hive precommit tests on bigtop jenkins

[Thejas Nair]

I see a new job now running there. Maybe there is nothing wrong with the
infra and builds actually finished (except for the 3 aborted ones).
Can't complain about a shorter queue ! :)



On Thu, Feb 13, 2014 at 1:30 PM, Thejas Nair  wrote:

> Is the jenkins infra used for hive precommit tests under maintenance  ? I
> see that the long queue has suddenly disappeared. The last few test builds
> have been aborted.
>
> The jenkins used for hive precommit tests -
> http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/
>
> Thanks,
> Thejas
>
>
>
>

-- 
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to 
which it is addressed and may contain information that is confidential, 
privileged and exempt from disclosure under applicable law. If the reader 
of this message is not the intended recipient, you are hereby notified that 
any printing, copying, dissemination, distribution, disclosure or 
forwarding of this communication is strictly prohibited. If you have 
received this communication in error, please contact the sender immediately 
and delete it from your system. Thank You.

[jira] [Updated] (HIVE-5504) OrcOutputFormat honors compression properties only from within hive

[Sushanth Sowmyan (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-5504?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sushanth Sowmyan updated HIVE-5504:
---

Release Note: 
Allows OrcOutputFormat.createRecordWriter() to use its table property 
parameters (like "orc.compress") if specified in JobConf as well. This codepath 
is not called from Hive, but will be helpful to those that want to use 
OrcOutputFormat as a generic M/R OutputFormat.

In addition, makes changes to HCatalog to look for certain Orc table 
properties, and if present, copies them out to JobConf before instantiating 
OrcOutputFormat, thus passing the properties back to it. This allows users of 
Orc from outside Hive to have Orc behave as the metadata dictates.
  Status: Patch Available  (was: Open)

> OrcOutputFormat honors  compression  properties only from within hive
> -
>
> Key: HIVE-5504
> URL: https://issues.apache.org/jira/browse/HIVE-5504
> Project: Hive
>  Issue Type: Bug
>  Components: HCatalog
>Affects Versions: 0.12.0, 0.11.0
>Reporter: Venkat Ranganathan
>Assignee: Sushanth Sowmyan
> Attachments: HIVE-5504.patch
>
>
> When we import data into a HCatalog table created with the following storage  
> description
> .. stored as orc tblproperties ("orc.compress"="SNAPPY") 
> the resultant orc file still uses the default zlib compression
> It looks like HCatOutputFormat is ignoring the tblproperties specified.   
> show tblproperties shows that the table indeed has the properties properly 
> saved.
> An insert/select into the table has the resulting orc file honor the tbl 
> property.



--
This message was sent by Atlassian JIRA
(v6.1.5#6160)

hive precommit tests on bigtop jenkins

[Thejas Nair]

Is the jenkins infra used for hive precommit tests under maintenance  ? I
see that the long queue has suddenly disappeared. The last few test builds
have been aborted.

The jenkins used for hive precommit tests -
http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/

Thanks,
Thejas

-- 
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to 
which it is addressed and may contain information that is confidential, 
privileged and exempt from disclosure under applicable law. If the reader 
of this message is not the intended recipient, you are hereby notified that 
any printing, copying, dissemination, distribution, disclosure or 
forwarding of this communication is strictly prohibited. If you have 
received this communication in error, please contact the sender immediately 
and delete it from your system. Thank You.

[jira] [Commented] (HIVE-5504) OrcOutputFormat honors compression properties only from within hive

[Sushanth Sowmyan (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-5504?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13900750#comment-13900750
 ] 

Sushanth Sowmyan commented on HIVE-5504:


Actually, reading further, the defaults are taken care of in the WriterOptions 
constructor, and thus, not a problem for this patch. And the parameters names 
are still in OrcFile, it's only the default values that were moved to HiveConf. 
This patch works as-is, and is compatible with HIVE-5728 and will respect the 
new defaults behaviour as well. Resetting the patch as available.

> OrcOutputFormat honors  compression  properties only from within hive
> -
>
> Key: HIVE-5504
> URL: https://issues.apache.org/jira/browse/HIVE-5504
> Project: Hive
>  Issue Type: Bug
>  Components: HCatalog
>Affects Versions: 0.11.0, 0.12.0
>Reporter: Venkat Ranganathan
>Assignee: Sushanth Sowmyan
> Attachments: HIVE-5504.patch
>
>
> When we import data into a HCatalog table created with the following storage  
> description
> .. stored as orc tblproperties ("orc.compress"="SNAPPY") 
> the resultant orc file still uses the default zlib compression
> It looks like HCatOutputFormat is ignoring the tblproperties specified.   
> show tblproperties shows that the table indeed has the properties properly 
> saved.
> An insert/select into the table has the resulting orc file honor the tbl 
> property.



--
This message was sent by Atlassian JIRA
(v6.1.5#6160)

[jira] [Updated] (HIVE-6046) add UDF for converting date time from one presentation to another

[Kostiantyn Kudriavtsev (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-6046?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kostiantyn Kudriavtsev updated HIVE-6046:
-

Status: Patch Available  (was: Open)

> add  UDF for converting date time from one presentation to another
> --
>
> Key: HIVE-6046
> URL: https://issues.apache.org/jira/browse/HIVE-6046
> Project: Hive
>  Issue Type: New Feature
>  Components: UDF
>Affects Versions: 0.13.0
>Reporter: Kostiantyn Kudriavtsev
>Assignee: Kostiantyn Kudriavtsev
> Attachments: Hive-6046.patch
>
>
> it'd be nice to have function for converting datetime to different formats, 
> for example:
> format_date('2013-12-12 00:00:00.0', '-MM-dd HH:mm:ss.S', '/MM/dd')
> There are two signatures to facilitate further using:
> format_date(datetime, fromFormat, toFormat)
> format_date(timestamp, toFormat)
>  



--
This message was sent by Atlassian JIRA
(v6.1.5#6160)

[jira] [Updated] (HIVE-6046) add UDF for converting date time from one presentation to another

[Kostiantyn Kudriavtsev (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-6046?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kostiantyn Kudriavtsev updated HIVE-6046:
-

Attachment: Hive-6046.patch

> add  UDF for converting date time from one presentation to another
> --
>
> Key: HIVE-6046
> URL: https://issues.apache.org/jira/browse/HIVE-6046
> Project: Hive
>  Issue Type: New Feature
>  Components: UDF
>Affects Versions: 0.13.0
>Reporter: Kostiantyn Kudriavtsev
>Assignee: Kostiantyn Kudriavtsev
> Attachments: Hive-6046.patch
>
>
> it'd be nice to have function for converting datetime to different formats, 
> for example:
> format_date('2013-12-12 00:00:00.0', '-MM-dd HH:mm:ss.S', '/MM/dd')
> There are two signatures to facilitate further using:
> format_date(datetime, fromFormat, toFormat)
> format_date(timestamp, toFormat)
>  



--
This message was sent by Atlassian JIRA
(v6.1.5#6160)

[jira] [Updated] (HIVE-6046) add UDF for converting date time from one presentation to another

[Kostiantyn Kudriavtsev (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-6046?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kostiantyn Kudriavtsev updated HIVE-6046:
-

Status: Open  (was: Patch Available)

> add  UDF for converting date time from one presentation to another
> --
>
> Key: HIVE-6046
> URL: https://issues.apache.org/jira/browse/HIVE-6046
> Project: Hive
>  Issue Type: New Feature
>  Components: UDF
>Affects Versions: 0.13.0
>Reporter: Kostiantyn Kudriavtsev
>Assignee: Kostiantyn Kudriavtsev
>
> it'd be nice to have function for converting datetime to different formats, 
> for example:
> format_date('2013-12-12 00:00:00.0', '-MM-dd HH:mm:ss.S', '/MM/dd')
> There are two signatures to facilitate further using:
> format_date(datetime, fromFormat, toFormat)
> format_date(timestamp, toFormat)
>  



--
This message was sent by Atlassian JIRA
(v6.1.5#6160)

[jira] [Commented] (HIVE-6046) add UDF for converting date time from one presentation to another

[Kostiantyn Kudriavtsev (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-6046?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13900744#comment-13900744
 ] 

Kostiantyn Kudriavtsev commented on HIVE-6046:
--

[~xuefuz] I tried "git diff --no-prefix", but result is not accepted by 
ReviewBoard. It seems there is well known issue: 
https://code.google.com/p/reviewboard/issues/detail?id=1918  So, I attached 
file as regular

> add  UDF for converting date time from one presentation to another
> --
>
> Key: HIVE-6046
> URL: https://issues.apache.org/jira/browse/HIVE-6046
> Project: Hive
>  Issue Type: New Feature
>  Components: UDF
>Affects Versions: 0.13.0
>Reporter: Kostiantyn Kudriavtsev
>Assignee: Kostiantyn Kudriavtsev
>
> it'd be nice to have function for converting datetime to different formats, 
> for example:
> format_date('2013-12-12 00:00:00.0', '-MM-dd HH:mm:ss.S', '/MM/dd')
> There are two signatures to facilitate further using:
> format_date(datetime, fromFormat, toFormat)
> format_date(timestamp, toFormat)
>  



--
This message was sent by Atlassian JIRA
(v6.1.5#6160)

[jira] [Updated] (HIVE-6378) HCatClient::createTable() doesn't allow SerDe class to be specified

[Thejas M Nair (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-6378?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Thejas M Nair updated HIVE-6378:


Attachment: HIVE-6378.1.patch

Attaching patch file in the precommit test friendly format.


> HCatClient::createTable() doesn't allow SerDe class to be specified
> ---
>
> Key: HIVE-6378
> URL: https://issues.apache.org/jira/browse/HIVE-6378
> Project: Hive
>  Issue Type: Improvement
>  Components: HCatalog
>Affects Versions: 0.13.0
>Reporter: Karl D. Gierach
>Assignee: Karl D. Gierach
>  Labels: patch
> Fix For: 0.13.0
>
> Attachments: HIVE-6378-1.patch, HIVE-6378.1.patch
>
>   Original Estimate: 4h
>  Time Spent: 4m
>  Remaining Estimate: 3h 56m
>
> Recreating the HCATALOG-641 under HIVE, since HCATALOG was moved into HIVE.
> With respect to HCATALOG-641, a patch was originally provided (but not 
> committed), so this work will consist of simply re-basing the original patch 
> to the current trunk and the latest released version.



--
This message was sent by Atlassian JIRA
(v6.1.5#6160)

[jira] [Updated] (HIVE-6046) add UDF for converting date time from one presentation to another

[Kostiantyn Kudriavtsev (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-6046?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kostiantyn Kudriavtsev updated HIVE-6046:
-

Attachment: (was: hive-6046.patch)

> add  UDF for converting date time from one presentation to another
> --
>
> Key: HIVE-6046
> URL: https://issues.apache.org/jira/browse/HIVE-6046
> Project: Hive
>  Issue Type: New Feature
>  Components: UDF
>Affects Versions: 0.13.0
>Reporter: Kostiantyn Kudriavtsev
>Assignee: Kostiantyn Kudriavtsev
>
> it'd be nice to have function for converting datetime to different formats, 
> for example:
> format_date('2013-12-12 00:00:00.0', '-MM-dd HH:mm:ss.S', '/MM/dd')
> There are two signatures to facilitate further using:
> format_date(datetime, fromFormat, toFormat)
> format_date(timestamp, toFormat)
>  



--
This message was sent by Atlassian JIRA
(v6.1.5#6160)

[jira] [Updated] (HIVE-5504) OrcOutputFormat honors compression properties only from within hive

[Sushanth Sowmyan (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-5504?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sushanth Sowmyan updated HIVE-5504:
---

Status: Open  (was: Patch Available)

(canceling patch till I can regenerate)

> OrcOutputFormat honors  compression  properties only from within hive
> -
>
> Key: HIVE-5504
> URL: https://issues.apache.org/jira/browse/HIVE-5504
> Project: Hive
>  Issue Type: Bug
>  Components: HCatalog
>Affects Versions: 0.12.0, 0.11.0
>Reporter: Venkat Ranganathan
>Assignee: Sushanth Sowmyan
> Attachments: HIVE-5504.patch
>
>
> When we import data into a HCatalog table created with the following storage  
> description
> .. stored as orc tblproperties ("orc.compress"="SNAPPY") 
> the resultant orc file still uses the default zlib compression
> It looks like HCatOutputFormat is ignoring the tblproperties specified.   
> show tblproperties shows that the table indeed has the properties properly 
> saved.
> An insert/select into the table has the resulting orc file honor the tbl 
> property.



--
This message was sent by Atlassian JIRA
(v6.1.5#6160)

[jira] [Commented] (HIVE-5504) OrcOutputFormat honors compression properties only from within hive

[Sushanth Sowmyan (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-5504?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13900734#comment-13900734
 ] 

Sushanth Sowmyan commented on HIVE-5504:


Yeah, some parameters that were depended upon in OrcFile moved to HiveConf, so 
at the very least, this patch requires regeneration.

One good thing at least is that OrcOutputFormat itself did not change, so that 
is still an easy merge.

The one weird thing is that HIVE-5728 introduces logic to override the default 
behaviour using hive-site.xml based parameters, and removes certain default 
parameters from OrcFile. We need to change this patch to respect that.


> OrcOutputFormat honors  compression  properties only from within hive
> -
>
> Key: HIVE-5504
> URL: https://issues.apache.org/jira/browse/HIVE-5504
> Project: Hive
>  Issue Type: Bug
>  Components: HCatalog
>Affects Versions: 0.11.0, 0.12.0
>Reporter: Venkat Ranganathan
>Assignee: Sushanth Sowmyan
> Attachments: HIVE-5504.patch
>
>
> When we import data into a HCatalog table created with the following storage  
> description
> .. stored as orc tblproperties ("orc.compress"="SNAPPY") 
> the resultant orc file still uses the default zlib compression
> It looks like HCatOutputFormat is ignoring the tblproperties specified.   
> show tblproperties shows that the table indeed has the properties properly 
> saved.
> An insert/select into the table has the resulting orc file honor the tbl 
> property.



--
This message was sent by Atlassian JIRA
(v6.1.5#6160)

[jira] [Commented] (HIVE-6250) sql std auth - view authorization should not underlying table. More tests and fixes.

[Hive QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-6250?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13900723#comment-13900723
 ] 

Hive QA commented on HIVE-6250:
---



{color:green}Overall{color}: +1 all checks pass

Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12628817/HIVE-6250.2.patch

{color:green}SUCCESS:{color} +1 5119 tests passed

Test results: 
http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/1314/testReport
Console output: 
http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/1314/console

Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}

This message is automatically generated.

ATTACHMENT ID: 12628817

> sql std auth - view authorization should not underlying table. More tests and 
> fixes.
> 
>
> Key: HIVE-6250
> URL: https://issues.apache.org/jira/browse/HIVE-6250
> Project: Hive
>  Issue Type: Sub-task
>  Components: Authorization
>Reporter: Thejas M Nair
>Assignee: Thejas M Nair
> Attachments: HIVE-6250.1.patch, HIVE-6250.2.patch
>
>   Original Estimate: 24h
>  Remaining Estimate: 24h
>
> This patch adds more tests for table and view authorization and also fixes a 
> number of issues found during testing -
> - View authorization should happen on only on the view, and not the 
> underlying table (Change in ReadEntity to indicate if it is a direct/indirect 
> dependency)
> - table owner in metadata should be the user as per SessionState 
> authentication provider
> - added utility function for finding the session state authentication 
> provider user
> - authorization should be based on current roles
> - admin user should have all permissions
> - error message improvements



--
This message was sent by Atlassian JIRA
(v6.1.5#6160)

[jira] [Updated] (HIVE-6424) webhcat.jar no longer includes webhcat-lo4j.properties

[Eugene Koifman (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-6424?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Eugene Koifman updated HIVE-6424:
-

Attachment: hive6424.patch

> webhcat.jar no longer includes webhcat-lo4j.properties
> --
>
> Key: HIVE-6424
> URL: https://issues.apache.org/jira/browse/HIVE-6424
> Project: Hive
>  Issue Type: Bug
>  Components: Build Infrastructure, WebHCat
>Affects Versions: 0.13.0
>Reporter: Eugene Koifman
> Attachments: hive6424.patch
>
>
> pre Maven switch, webhcat-log4j.properties and webhcat-default.xml were at 
> the root of hive-webhcat-0.13.0-SNAPSHOT.jar.  They are no longer there.



--
This message was sent by Atlassian JIRA
(v6.1.5#6160)

[jira] [Commented] (HIVE-6379) webhcat is not operational due to Slf4j conflicts

[Eugene Koifman (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-6379?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13900718#comment-13900718
 ] 

Eugene Koifman commented on HIVE-6379:
--

after fixing HIVE-6424, the error changes to:

Exception in thread "main" java.lang.NoSuchMethodError: 
org.slf4j.spi.LocationAwareLogger.log(Lorg/slf4j/Marker;Ljava/lang/String;ILjava/lang/String;[Ljava/lang/Object;Ljava/lang/Throwable;)V
at 
org.slf4j.bridge.SLF4JBridgeHandler.callLocationAwareLogger(SLF4JBridgeHandler.java:224)
at 
org.slf4j.bridge.SLF4JBridgeHandler.publish(SLF4JBridgeHandler.java:301)
at java.util.logging.Logger.log(Logger.java:509)
at java.util.logging.Logger.doLog(Logger.java:531)
at java.util.logging.Logger.log(Logger.java:554)
at 
com.sun.jersey.api.core.PackagesResourceConfig.init(PackagesResourceConfig.java:101)
at 
com.sun.jersey.api.core.PackagesResourceConfig.(PackagesResourceConfig.java:78)
at 
org.apache.hive.hcatalog.templeton.Main.makeJerseyConfig(Main.java:223)
at org.apache.hive.hcatalog.templeton.Main.runServer(Main.java:193)
at org.apache.hive.hcatalog.templeton.Main.run(Main.java:114)
at org.apache.hive.hcatalog.templeton.Main.main(Main.java:259)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.util.RunJar.main(RunJar.java:160)

> webhcat is not operational due to Slf4j conflicts
> -
>
> Key: HIVE-6379
> URL: https://issues.apache.org/jira/browse/HIVE-6379
> Project: Hive
>  Issue Type: Bug
>  Components: WebHCat
>Affects Versions: 0.12.0
>Reporter: Eugene Koifman
>Assignee: Eugene Koifman
>
> 1. build current trunk: mvn clean install -Pdist,hadoop-1 -DskipTests
> 2. 
> /Users/ekoifman/dev/hive/packaging/target/apache-hive-0.13.0-SNAPSHOT-bin/apache-hive-0.13.0-SNAPSHOT-bin/hcatalog
> 3. ./sbin/webhcat_server.sh start
> 4. http://localhost:50111/templeton/v1/ddl/database?user.name=ekoifman
> webhcat-console-error.log has
> log4j:WARN No appenders could be found for logger 
> (org.apache.hive.hcatalog.templeton.AppConfig).
> log4j:WARN Please initialize the log4j system properly.
> log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more 
> info.
> Exception in thread "main" java.lang.NoSuchMethodError: 
> org.slf4j.helpers.MessageFormatter.arrayFormat(Ljava/lang/String;[Ljava/lang/Object;)Lorg/slf4j/helpers/FormattingTuple;
> at 
> org.eclipse.jetty.util.log.JettyAwareLogger.log(JettyAwareLogger.java:613)
> at 
> org.eclipse.jetty.util.log.JettyAwareLogger.debug(JettyAwareLogger.java:200)
> at org.eclipse.jetty.util.log.Slf4jLog.debug(Slf4jLog.java:94)
> at org.eclipse.jetty.util.component.Container.add(Container.java:201)
> at 
> org.eclipse.jetty.util.component.Container.update(Container.java:164)
> at 
> org.eclipse.jetty.util.component.Container.update(Container.java:106)
> at org.eclipse.jetty.server.Server.setConnectors(Server.java:184)
> at org.eclipse.jetty.server.Server.(Server.java:97)
> at org.apache.hive.hcatalog.templeton.Main.runServer(Main.java:162)
> at org.apache.hive.hcatalog.templeton.Main.run(Main.java:114)
> at org.apache.hive.hcatalog.templeton.Main.main(Main.java:259)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.util.RunJar.main(RunJar.java:160)
> webhcat-console-error.log (END) 



--
This message was sent by Atlassian JIRA
(v6.1.5#6160)

[jira] [Updated] (HIVE-6339) Implement new JDK7 schema management APIs in java.sql.Connection

[Prasad Mujumdar (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-6339?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Prasad Mujumdar updated HIVE-6339:
--

Status: Patch Available  (was: Open)

> Implement new JDK7 schema management APIs in java.sql.Connection 
> -
>
> Key: HIVE-6339
> URL: https://issues.apache.org/jira/browse/HIVE-6339
> Project: Hive
>  Issue Type: Improvement
>  Components: JDBC
>Affects Versions: 0.13.0
>Reporter: Prasad Mujumdar
>Assignee: Prasad Mujumdar
> Attachments: HIVE-6339.1.patch
>
>
> JDK7 has added a few metadata methods in 
> [java.sql.Conntion|http://docs.oracle.com/javase/7/docs/api/java/sql/Connection.html]
>  
> {noformat}
> getSchema()
> setSchema()
> getCatalog()
> setCatalog()
> {noformat}
> Currently Hive JDBC just has stub implementation for all these methods throws 
> unsupported exception. This needs to be fixed.



--
This message was sent by Atlassian JIRA
(v6.1.5#6160)

[jira] [Updated] (HIVE-6339) Implement new JDK7 schema management APIs in java.sql.Connection

[Prasad Mujumdar (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-6339?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Prasad Mujumdar updated HIVE-6339:
--

Attachment: HIVE-6339.1.patch

> Implement new JDK7 schema management APIs in java.sql.Connection 
> -
>
> Key: HIVE-6339
> URL: https://issues.apache.org/jira/browse/HIVE-6339
> Project: Hive
>  Issue Type: Improvement
>  Components: JDBC
>Affects Versions: 0.13.0
>Reporter: Prasad Mujumdar
>Assignee: Prasad Mujumdar
> Attachments: HIVE-6339.1.patch
>
>
> JDK7 has added a few metadata methods in 
> [java.sql.Conntion|http://docs.oracle.com/javase/7/docs/api/java/sql/Connection.html]
>  
> {noformat}
> getSchema()
> setSchema()
> getCatalog()
> setCatalog()
> {noformat}
> Currently Hive JDBC just has stub implementation for all these methods throws 
> unsupported exception. This needs to be fixed.



--
This message was sent by Atlassian JIRA
(v6.1.5#6160)

Review Request 18094: HIVE-6339: Implement new JDK7 schema management APIs in java.sql.Connection

[Prasad Mujumdar]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/18094/
---

Review request for hive.


Bugs: HIVE-6339
https://issues.apache.org/jira/browse/HIVE-6339


Repository: hive-git


Description
---

Implement new JDK7 schema management APIs in java.sql.Connection.


Diffs
-

  itests/hive-unit/src/test/java/org/apache/hive/jdbc/TestJdbcWithMiniHS2.java 
e68a5a9 
  jdbc/src/java/org/apache/hive/jdbc/HiveConnection.java 13fc19b 

Diff: https://reviews.apache.org/r/18094/diff/


Testing
---

Added new test case.


Thanks,

Prasad Mujumdar

[jira] [Updated] (HIVE-6339) Implement new JDK7 schema management APIs in java.sql.Connection

[Prasad Mujumdar (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-6339?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Prasad Mujumdar updated HIVE-6339:
--

Summary: Implement new JDK7 schema management APIs in java.sql.Connection   
(was: Implement schema new JDK7 management APIs in java.sql.Connection )

> Implement new JDK7 schema management APIs in java.sql.Connection 
> -
>
> Key: HIVE-6339
> URL: https://issues.apache.org/jira/browse/HIVE-6339
> Project: Hive
>  Issue Type: Improvement
>  Components: JDBC
>Affects Versions: 0.13.0
>Reporter: Prasad Mujumdar
>Assignee: Prasad Mujumdar
>
> JDK7 has added a few metadata methods in 
> [java.sql.Conntion|http://docs.oracle.com/javase/7/docs/api/java/sql/Connection.html]
>  
> {noformat}
> getSchema()
> setSchema()
> getCatalog()
> setCatalog()
> {noformat}
> Currently Hive JDBC just has stub implementation for all these methods throws 
> unsupported exception. This needs to be fixed.



--
This message was sent by Atlassian JIRA
(v6.1.5#6160)

[jira] [Commented] (HIVE-6356) Dependency injection in hbase storage handler is broken

[Nick Dimiduk (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-6356?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13900695#comment-13900695
 ] 

Nick Dimiduk commented on HIVE-6356:


Which version of HBase are you testing against? 
[mapred.TableMapReduceUtil#addDependencyJars(JobConf)|https://github.com/apache/hbase/blob/0.96/hbase-server/src/main/java/org/apache/hadoop/hbase/mapred/TableMapReduceUtil.java#L298-L316]
 calls the [same 
method|https://github.com/apache/hbase/blob/0.96/hbase-server/src/main/java/org/apache/hadoop/hbase/mapreduce/TableMapReduceUtil.java#L586-L612]
 from the mapreduce package, which in turn adds all the hbase jars, plus the 
hbase dependencies you mention.

ClassNotFoundException thrown at the job client or in the job? At the client 
means your local classpath is missing something. In the job means it wasn't 
packaged properly. What's the missing class/jar?

> Dependency injection in hbase storage handler is broken
> ---
>
> Key: HIVE-6356
> URL: https://issues.apache.org/jira/browse/HIVE-6356
> Project: Hive
>  Issue Type: Bug
>  Components: HBase Handler
>Reporter: Navis
>Assignee: Navis
>Priority: Minor
> Fix For: 0.13.0
>
> Attachments: HIVE-6356.1.patch.txt, HIVE-6356.2.patch.txt, 
> HIVE-6356.3.patch.txt, HIVE-6356.addendum.00.patch
>
>
> Dependent jars for hbase is not added to tmpjars, which is caused by the 
> change of method signature(TableMapReduceUtil.addDependencyJars).



--
This message was sent by Atlassian JIRA
(v6.1.5#6160)

[jira] [Commented] (HIVE-6403) uncorrelated subquery is failing with auto.convert.join=true

[Harish Butani (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-6403?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13900676#comment-13900676
 ] 

Harish Butani commented on HIVE-6403:
-

Ok, yes I see what you mean. I incorporated your findAlias changes. and created 
a new patch.
I think the 'multiInsertBigTableCheck' check needs to be still done, because in 
case of multi insert we want the 'source' table to be the only big table 
choice. Do you agree?

> uncorrelated subquery is failing with auto.convert.join=true
> 
>
> Key: HIVE-6403
> URL: https://issues.apache.org/jira/browse/HIVE-6403
> Project: Hive
>  Issue Type: Bug
>  Components: Query Processor
>Reporter: Navis
>Assignee: Harish Butani
> Attachments: HIVE-6403.1.patch, HIVE-6403.2.patch, navis.patch
>
>
> Fixing HIVE-5690, I've found query in subquery_multiinsert.q is not working 
> with hive.auto.convert.join=true 
> {noformat}
> set hive.auto.convert.join=true;
> hive> explain
> > from src b 
> > INSERT OVERWRITE TABLE src_4 
> >   select * 
> >   where b.key in 
> >(select a.key 
> > from src a 
> > where b.value = a.value and a.key > '9'
> >) 
> > INSERT OVERWRITE TABLE src_5 
> >   select *  
> >   where b.key not in  ( select key from src s1 where s1.key > '2') 
> >   order by key 
> > ;
> java.lang.IndexOutOfBoundsException: Index: 0, Size: 0
>   at java.util.ArrayList.rangeCheck(ArrayList.java:635)
>   at java.util.ArrayList.get(ArrayList.java:411)
>   at 
> org.apache.hadoop.hive.ql.optimizer.MapJoinProcessor.genMapJoinLocalWork(MapJoinProcessor.java:149)
>   at 
> org.apache.hadoop.hive.ql.optimizer.MapJoinProcessor.genLocalWorkForMapJoin(MapJoinProcessor.java:256)
>   at 
> org.apache.hadoop.hive.ql.optimizer.MapJoinProcessor.genMapJoinOpAndLocalWork(MapJoinProcessor.java:248)
>   at 
> org.apache.hadoop.hive.ql.optimizer.physical.CommonJoinTaskDispatcher.convertTaskToMapJoinTask(CommonJoinTaskDispatcher.java:191)
>   at 
> org.apache.hadoop.hive.ql.optimizer.physical.CommonJoinTaskDispatcher.processCurrentTask(CommonJoinTaskDispatcher.java:481)
>   at 
> org.apache.hadoop.hive.ql.optimizer.physical.AbstractJoinTaskDispatcher.dispatch(AbstractJoinTaskDispatcher.java:182)
>   at 
> org.apache.hadoop.hive.ql.lib.TaskGraphWalker.dispatch(TaskGraphWalker.java:111)
>   at 
> org.apache.hadoop.hive.ql.lib.TaskGraphWalker.walk(TaskGraphWalker.java:194)
>   at 
> org.apache.hadoop.hive.ql.lib.TaskGraphWalker.startWalking(TaskGraphWalker.java:139)
>   at 
> org.apache.hadoop.hive.ql.optimizer.physical.CommonJoinResolver.resolve(CommonJoinResolver.java:79)
>   at 
> org.apache.hadoop.hive.ql.optimizer.physical.PhysicalOptimizer.optimize(PhysicalOptimizer.java:100)
>   at 
> org.apache.hadoop.hive.ql.parse.MapReduceCompiler.optimizeTaskPlan(MapReduceCompiler.java:290)
>   at 
> org.apache.hadoop.hive.ql.parse.TaskCompiler.compile(TaskCompiler.java:216)
>   at 
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeInternal(SemanticAnalyzer.java:9167)
>   at 
> org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:327)
>   at 
> org.apache.hadoop.hive.ql.parse.ExplainSemanticAnalyzer.analyzeInternal(ExplainSemanticAnalyzer.java:64)
>   at 
> org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:327)
>   at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:446)
>   at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:346)
>   at org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:1056)
>   at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1099)
>   at org.apache.hadoop.hive.ql.Driver.run(Driver.java:992)
>   at org.apache.hadoop.hive.ql.Driver.run(Driver.java:982)
>   at 
> org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:268)
>   at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:220)
>   at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:424)
>   at 
> org.apache.hadoop.hive.cli.CliDriver.executeDriver(CliDriver.java:793)
>   at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:687)
>   at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:626)
>   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>   at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
>   at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>   at java.lang.reflect.Method.invoke(Method.java:606)
>   at org.apache.hadoop.util.RunJar.main(RunJar.java:160)
> org.apache.hadoop.hive.ql.parse.SemanticException: Failed to genera

[jira] [Updated] (HIVE-6403) uncorrelated subquery is failing with auto.convert.join=true

[Harish Butani (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-6403?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Harish Butani updated HIVE-6403:


Attachment: HIVE-6403.2.patch

> uncorrelated subquery is failing with auto.convert.join=true
> 
>
> Key: HIVE-6403
> URL: https://issues.apache.org/jira/browse/HIVE-6403
> Project: Hive
>  Issue Type: Bug
>  Components: Query Processor
>Reporter: Navis
>Assignee: Harish Butani
> Attachments: HIVE-6403.1.patch, HIVE-6403.2.patch, navis.patch
>
>
> Fixing HIVE-5690, I've found query in subquery_multiinsert.q is not working 
> with hive.auto.convert.join=true 
> {noformat}
> set hive.auto.convert.join=true;
> hive> explain
> > from src b 
> > INSERT OVERWRITE TABLE src_4 
> >   select * 
> >   where b.key in 
> >(select a.key 
> > from src a 
> > where b.value = a.value and a.key > '9'
> >) 
> > INSERT OVERWRITE TABLE src_5 
> >   select *  
> >   where b.key not in  ( select key from src s1 where s1.key > '2') 
> >   order by key 
> > ;
> java.lang.IndexOutOfBoundsException: Index: 0, Size: 0
>   at java.util.ArrayList.rangeCheck(ArrayList.java:635)
>   at java.util.ArrayList.get(ArrayList.java:411)
>   at 
> org.apache.hadoop.hive.ql.optimizer.MapJoinProcessor.genMapJoinLocalWork(MapJoinProcessor.java:149)
>   at 
> org.apache.hadoop.hive.ql.optimizer.MapJoinProcessor.genLocalWorkForMapJoin(MapJoinProcessor.java:256)
>   at 
> org.apache.hadoop.hive.ql.optimizer.MapJoinProcessor.genMapJoinOpAndLocalWork(MapJoinProcessor.java:248)
>   at 
> org.apache.hadoop.hive.ql.optimizer.physical.CommonJoinTaskDispatcher.convertTaskToMapJoinTask(CommonJoinTaskDispatcher.java:191)
>   at 
> org.apache.hadoop.hive.ql.optimizer.physical.CommonJoinTaskDispatcher.processCurrentTask(CommonJoinTaskDispatcher.java:481)
>   at 
> org.apache.hadoop.hive.ql.optimizer.physical.AbstractJoinTaskDispatcher.dispatch(AbstractJoinTaskDispatcher.java:182)
>   at 
> org.apache.hadoop.hive.ql.lib.TaskGraphWalker.dispatch(TaskGraphWalker.java:111)
>   at 
> org.apache.hadoop.hive.ql.lib.TaskGraphWalker.walk(TaskGraphWalker.java:194)
>   at 
> org.apache.hadoop.hive.ql.lib.TaskGraphWalker.startWalking(TaskGraphWalker.java:139)
>   at 
> org.apache.hadoop.hive.ql.optimizer.physical.CommonJoinResolver.resolve(CommonJoinResolver.java:79)
>   at 
> org.apache.hadoop.hive.ql.optimizer.physical.PhysicalOptimizer.optimize(PhysicalOptimizer.java:100)
>   at 
> org.apache.hadoop.hive.ql.parse.MapReduceCompiler.optimizeTaskPlan(MapReduceCompiler.java:290)
>   at 
> org.apache.hadoop.hive.ql.parse.TaskCompiler.compile(TaskCompiler.java:216)
>   at 
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeInternal(SemanticAnalyzer.java:9167)
>   at 
> org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:327)
>   at 
> org.apache.hadoop.hive.ql.parse.ExplainSemanticAnalyzer.analyzeInternal(ExplainSemanticAnalyzer.java:64)
>   at 
> org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:327)
>   at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:446)
>   at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:346)
>   at org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:1056)
>   at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1099)
>   at org.apache.hadoop.hive.ql.Driver.run(Driver.java:992)
>   at org.apache.hadoop.hive.ql.Driver.run(Driver.java:982)
>   at 
> org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:268)
>   at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:220)
>   at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:424)
>   at 
> org.apache.hadoop.hive.cli.CliDriver.executeDriver(CliDriver.java:793)
>   at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:687)
>   at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:626)
>   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>   at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
>   at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>   at java.lang.reflect.Method.invoke(Method.java:606)
>   at org.apache.hadoop.util.RunJar.main(RunJar.java:160)
> org.apache.hadoop.hive.ql.parse.SemanticException: Failed to generate new 
> mapJoin operator by exception : Index: 0, Size: 0
>   at 
> org.apache.hadoop.hive.ql.optimizer.MapJoinProcessor.genLocalWorkForMapJoin(MapJoinProcessor.java:266)
>   at 
> org.apache.hadoop.hive.ql.optimizer.MapJoinProcessor.genMapJoinOpAndLocalWork(MapJoinProcessor.java:248

[jira] [Commented] (HIVE-6423) Read Timestamps and Dates from Avro files (stored as Strings in ISO format)

[Hive QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-6423?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13900635#comment-13900635
 ] 

Hive QA commented on HIVE-6423:
---



{color:red}Overall{color}: -1 at least one tests failed

Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12628618/HIVE-6423.patch

{color:red}ERROR:{color} -1 due to 2 failed/errored test(s), 5094 tests executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_bucketizedhiveinputformat
org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_bucketmapjoin6
{noformat}

Test results: 
http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/1311/testReport
Console output: 
http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/1311/console

Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 2 tests failed
{noformat}

This message is automatically generated.

ATTACHMENT ID: 12628618

> Read Timestamps and Dates from Avro files (stored as Strings in ISO format)
> ---
>
> Key: HIVE-6423
> URL: https://issues.apache.org/jira/browse/HIVE-6423
> Project: Hive
>  Issue Type: New Feature
>  Components: HiveServer2
>Affects Versions: 0.12.0
>Reporter: Nick White
> Attachments: HIVE-6423.patch
>
>
> Avro doesn't have native timestamp or date data types. Some tools (e.g. 
> apache Sqoop) work around this by adding the java.sql.Types type number as an 
> "sqlType" attribute to the Avro schema JSON field element. The attached patch 
> makes the Avro SerDe look for an sqlType attribute, and if it is a Hive 
> primitive type that's not an Avro primitive type (i.e. timestamp and date) it 
> returns the appropriate Hive type (the one corresponding to the SQL type).



--
This message was sent by Atlassian JIRA
(v6.1.5#6160)

[jira] [Updated] (HIVE-6250) sql std auth - view authorization should not underlying table. More tests and fixes.

[Thejas M Nair (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-6250?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Thejas M Nair updated HIVE-6250:


Attachment: HIVE-6250.2.patch

 HIVE-6250.2.patch  - patch rebased for trunk



> sql std auth - view authorization should not underlying table. More tests and 
> fixes.
> 
>
> Key: HIVE-6250
> URL: https://issues.apache.org/jira/browse/HIVE-6250
> Project: Hive
>  Issue Type: Sub-task
>  Components: Authorization
>Reporter: Thejas M Nair
>Assignee: Thejas M Nair
> Attachments: HIVE-6250.1.patch, HIVE-6250.2.patch
>
>   Original Estimate: 24h
>  Remaining Estimate: 24h
>
> This patch adds more tests for table and view authorization and also fixes a 
> number of issues found during testing -
> - View authorization should happen on only on the view, and not the 
> underlying table (Change in ReadEntity to indicate if it is a direct/indirect 
> dependency)
> - table owner in metadata should be the user as per SessionState 
> authentication provider
> - added utility function for finding the session state authentication 
> provider user
> - authorization should be based on current roles
> - admin user should have all permissions
> - error message improvements



--
This message was sent by Atlassian JIRA
(v6.1.5#6160)

[jira] [Commented] (HIVE-6383) Newly added tests in TestJdbcDriver2 from HIVE-4395 is not running

[Thejas M Nair (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-6383?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13900618#comment-13900618
 ] 

Thejas M Nair commented on HIVE-6383:
-

+1

> Newly added tests in TestJdbcDriver2 from HIVE-4395 is not running
> --
>
> Key: HIVE-6383
> URL: https://issues.apache.org/jira/browse/HIVE-6383
> Project: Hive
>  Issue Type: Bug
>  Components: JDBC
>Reporter: Navis
>Assignee: Prasad Mujumdar
>Priority: Minor
> Attachments: HIVE-6383.1.patch
>
>
> Newly added tests are not marked with @Test annotation and seemed not 
> running. In my try after adding the annotation, testFetchFirstQuery is 
> failed. [~prasadm] Could you check this?



--
This message was sent by Atlassian JIRA
(v6.1.5#6160)

[jira] [Updated] (HIVE-6109) Support customized location for EXTERNAL tables created by Dynamic Partitioning

[Sushanth Sowmyan (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-6109?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sushanth Sowmyan updated HIVE-6109:
---

Resolution: Fixed
Status: Resolved  (was: Patch Available)

> Support customized location for EXTERNAL tables created by Dynamic 
> Partitioning
> ---
>
> Key: HIVE-6109
> URL: https://issues.apache.org/jira/browse/HIVE-6109
> Project: Hive
>  Issue Type: Improvement
>  Components: HCatalog
>Reporter: Satish Mittal
>Assignee: Satish Mittal
> Fix For: 0.13.0
>
> Attachments: HIVE-6109.1.patch.txt, HIVE-6109.2.patch.txt, 
> HIVE-6109.3.patch.txt, HIVE-6109.pdf
>
>
> Currently when dynamic partitions are created by HCatalog, the underlying 
> directories for the partitions are created in a fixed 'Hive-style' format, 
> i.e. root_dir/key1=value1/key2=value2/ and so on. However in case of 
> external table, user should be able to control the format of directories 
> created for dynamic partitions.



--
This message was sent by Atlassian JIRA
(v6.1.5#6160)

[jira] [Updated] (HIVE-6109) Support customized location for EXTERNAL tables created by Dynamic Partitioning

[Sushanth Sowmyan (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-6109?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sushanth Sowmyan updated HIVE-6109:
---

Fix Version/s: 0.13.0

> Support customized location for EXTERNAL tables created by Dynamic 
> Partitioning
> ---
>
> Key: HIVE-6109
> URL: https://issues.apache.org/jira/browse/HIVE-6109
> Project: Hive
>  Issue Type: Improvement
>  Components: HCatalog
>Reporter: Satish Mittal
>Assignee: Satish Mittal
> Fix For: 0.13.0
>
> Attachments: HIVE-6109.1.patch.txt, HIVE-6109.2.patch.txt, 
> HIVE-6109.3.patch.txt, HIVE-6109.pdf
>
>
> Currently when dynamic partitions are created by HCatalog, the underlying 
> directories for the partitions are created in a fixed 'Hive-style' format, 
> i.e. root_dir/key1=value1/key2=value2/ and so on. However in case of 
> external table, user should be able to control the format of directories 
> created for dynamic partitions.



--
This message was sent by Atlassian JIRA
(v6.1.5#6160)