[jira] [Created] (HIVE-14264) ArrayIndexOutOfBoundsException when cbo is enabled
Amareshwari Sriramadasu created HIVE-14264:
--
Summary: ArrayIndexOutOfBoundsException when cbo is enabled
Key: HIVE-14264
URL: https://issues.apache.org/jira/browse/HIVE-14264
Project: Hive
Issue Type: Bug
Components: Query Processor
Affects Versions: 2.1.0
Reporter: Amareshwari Sriramadasu
We have noticed ArrayIndexOutOfBoundsException for queries with IS NOT NULL
filter. Exception goes away when hive.cbo.enable=false
Here is a stacktrace in our production environment :
{noformat}
Caused by: java.lang.ArrayIndexOutOfBoundsException: -1
at java.util.ArrayList.elementData(ArrayList.java:418) ~[na:1.8.0_72]
at java.util.ArrayList.set(ArrayList.java:446) ~[na:1.8.0_72]
at
org.apache.hadoop.hive.ql.optimizer.physical.MapJoinResolver$LocalMapJoinTaskDispatcher.processCurrentTask(MapJoinResolver.java:173)
~[hive-exec-2.1.2-inm.jar:2.1.2-inm]
at
org.apache.hadoop.hive.ql.optimizer.physical.MapJoinResolver$LocalMapJoinTaskDispatcher.dispatch(MapJoinResolver.java:239)
~[hive-exec-2.1.2-inm.jar:2.1.2-inm]
at
org.apache.hadoop.hive.ql.lib.TaskGraphWalker.dispatch(TaskGraphWalker.java:111)
~[hive-exec-2.1.2-inm.jar:2.1.2-inm]
at
org.apache.hadoop.hive.ql.lib.TaskGraphWalker.walk(TaskGraphWalker.java:180)
~[hive-exec-2.1.2-inm.jar:2.1.2-inm]
at
org.apache.hadoop.hive.ql.lib.TaskGraphWalker.startWalking(TaskGraphWalker.java:125)
~[hive-exec-2.1.2-inm.jar:2.1.2-inm]
at
org.apache.hadoop.hive.ql.optimizer.physical.MapJoinResolver.resolve(MapJoinResolver.java:81)
~[hive-exec-2.1.2-inm.jar:2.1.2-inm]
at
org.apache.hadoop.hive.ql.optimizer.physical.PhysicalOptimizer.optimize(PhysicalOptimizer.java:107)
~[hive-exec-2.1.2-inm.jar:2.1.2-inm]
at
org.apache.hadoop.hive.ql.parse.MapReduceCompiler.optimizeTaskPlan(MapReduceCompiler.java:271)
~[hive-exec-2.1.2-inm.jar:2.1.2-inm]
at
org.apache.hadoop.hive.ql.parse.TaskCompiler.compile(TaskCompiler.java:274)
~[hive-exec-2.1.2-inm.jar:2.1.2-inm]
at
org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeInternal(SemanticAnalyzer.java:10764)
~[hive-exec-2.1.2-inm.jar:2.1.2-inm]
at
org.apache.hadoop.hive.ql.parse.CalcitePlanner.analyzeInternal(CalcitePlanner.java:234)
~[hive-exec-2.1.2-inm.jar:2.1.2-inm]
at
org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:250)
~[hive-exec-2.1.2-inm.jar:2.1.2-inm]
at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:436)
~[hive-exec-2.1.2-inm.jar:2.1.2-inm]
at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:328)
~[hive-exec-2.1.2-inm.jar:2.1.2-inm]
at org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:1156)
~[hive-exec-2.1.2-inm.jar:2.1.2-inm]
at org.apache.hadoop.hive.ql.Driver.compileAndRespond(Driver.java:1143)
~[hive-exec-2.1.2-inm.jar:2.1.2-inm]
at
org.apache.hive.service.cli.operation.SQLOperation.prepare(SQLOperation.java:147)
~[hive-service-2.1.2-inm.jar:2.1.2-inm]
{noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
Re: Review Request 49494: HIVE-14144: Permanent functions are showing up in show functions, but describe says it doesn't exist
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/49494/#review140956 --- Ship it! Ship It! - Amareshwari Sriramadasu On July 1, 2016, 7:22 a.m., Rajat Khandelwal wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/49494/ > --- > > (Updated July 1, 2016, 7:22 a.m.) > > > Review request for hive. > > > Bugs: HIVE-14144 > https://issues.apache.org/jira/browse/HIVE-14144 > > > Repository: hive-git > > > Description > --- > > > Diffs > - > > ql/src/java/org/apache/hadoop/hive/ql/exec/Registry.java > a4584e3eaebc052a6870c543bfbfc9e5d78e771b > ql/src/java/org/apache/hadoop/hive/ql/util/ResourceDownloader.java > 14102b5108164616b4cf2e7f7d22c894321f027e > > Diff: https://reviews.apache.org/r/49494/diff/ > > > Testing > --- > > > Thanks, > > Rajat Khandelwal > >
Re: Review Request 48118: HIVE-13903: getFunctionInfo is downloading jar on every call
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/48118/#review135781 --- Ship it! Ship It! - Amareshwari Sriramadasu On June 1, 2016, 9:44 a.m., Rajat Khandelwal wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/48118/ > --- > > (Updated June 1, 2016, 9:44 a.m.) > > > Review request for hive. > > > Bugs: HIVE-13903 > https://issues.apache.org/jira/browse/HIVE-13903 > > > Repository: hive-git > > > Description > --- > > on queries using permanent udfs, the jar file of the udf is downloaded > multiple times. Each call originating from Registry.getFunctionInfo. This > increases time for the query, especially if that query is just an explain > query. The jar should be downloaded once, and not downloaded again if the udf > class is accessible in the current thread. > > > Diffs > - > > ql/src/java/org/apache/hadoop/hive/ql/exec/Registry.java > 86df74d26ddd60736214c0f9acb835f4253a2c17 > > Diff: https://reviews.apache.org/r/48118/diff/ > > > Testing > --- > > > Thanks, > > Rajat Khandelwal > >
[jira] [Created] (HIVE-13862) org.apache.hadoop.hive.metastore.MetaStoreDirectSql.getNumPartitionsViaSqlFilter falls back to ORM
Amareshwari Sriramadasu created HIVE-13862:
--
Summary:
org.apache.hadoop.hive.metastore.MetaStoreDirectSql.getNumPartitionsViaSqlFilter
falls back to ORM
Key: HIVE-13862
URL: https://issues.apache.org/jira/browse/HIVE-13862
Project: Hive
Issue Type: Bug
Components: Metastore
Reporter: Amareshwari Sriramadasu
Assignee: Rajat Khandelwal
Fix For: 2.1.0
We are seeing following exception and calls fall back to ORM which make it
costly :
{noformat}
WARN org.apache.hadoop.hive.metastore.ObjectStore - Direct SQL failed,
falling back to ORM
java.lang.ClassCastException:
org.datanucleus.store.rdbms.query.ForwardQueryResult cannot be cast to
java.lang.Number
at
org.apache.hadoop.hive.metastore.MetaStoreDirectSql.extractSqlInt(MetaStoreDirectSql.java:892)
~[hive-exec-2.1.2-inm-SNAPSHOT.jar:2.1.2-inm-SNAPSHOT]
at
org.apache.hadoop.hive.metastore.MetaStoreDirectSql.getNumPartitionsViaSqlFilterInternal(MetaStoreDirectSql.java:855)
~[hive-exec-2.1.2-inm-SNAPSHOT.jar:2.1.2-inm-SNAPSHOT]
at
org.apache.hadoop.hive.metastore.MetaStoreDirectSql.getNumPartitionsViaSqlFilter(MetaStoreDirectSql.java:405)
~[hive-exec-2.1.2-inm-SNAPSHOT.jar:2.1.2-inm-SNAPSHOT]
at
org.apache.hadoop.hive.metastore.ObjectStore$5.getSqlResult(ObjectStore.java:2763)
~[hive-exec-2.1.2-inm-SNAPSHOT.jar:2.1.2-inm-SNAPSHOT]
at
org.apache.hadoop.hive.metastore.ObjectStore$5.getSqlResult(ObjectStore.java:2755)
~[hive-exec-2.1.2-inm-SNAPSHOT.jar:2.1.2-inm-SNAPSHOT]
at
org.apache.hadoop.hive.metastore.ObjectStore$GetHelper.run(ObjectStore.java:2606)
~[hive-exec-2.1.2-inm-SNAPSHOT.jar:2.1.2-inm-SNAPSHOT]
at
org.apache.hadoop.hive.metastore.ObjectStore.getNumPartitionsByFilterInternal(ObjectStore.java:2770)
[hive-exec-2.1.2-inm-SNAPSHOT.jar:2.1.2-inm-SNAPSHOT]
at
org.apache.hadoop.hive.metastore.ObjectStore.getNumPartitionsByFilter(ObjectStore.java:2746)
[hive-exec-2.1.2-inm-SNAPSHOT.jar:2.1.2-inm-SNAPSHOT]
{noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
Re: Review Request 45733: HIVE-13421: Propagate job progress in operation status
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/45733/#review129686 --- Ship it! Ship It! - Amareshwari Sriramadasu On April 19, 2016, 8:03 a.m., Rajat Khandelwal wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/45733/ > --- > > (Updated April 19, 2016, 8:03 a.m.) > > > Review request for hive, Amareshwari Sriramadasu and Szehon Ho. > > > Bugs: HIVE-13421 > https://issues.apache.org/jira/browse/HIVE-13421 > > > Repository: hive-git > > > Description > --- > > > Diffs > - > > > itests/hive-unit/src/test/java/org/apache/hive/service/cli/session/TestQueryDisplay.java > 98581e0e09c20b30eafd6aae99068fddf1596d8e > ql/src/java/org/apache/hadoop/hive/ql/Driver.java > 65744ac866ecd1bee885370394cf720631586ed2 > ql/src/java/org/apache/hadoop/hive/ql/QueryDisplay.java > d582bc063fc150002a01d63451ae6632fca29ac1 > ql/src/java/org/apache/hadoop/hive/ql/QueryPlan.java > ef0923d555ba662b4ed30ef45a3d72760cdfad52 > ql/src/java/org/apache/hadoop/hive/ql/exec/ConditionalTask.java > c96c8135a344049e57167559c4d760b876a42ca5 > ql/src/java/org/apache/hadoop/hive/ql/exec/Task.java > 6c677f5bbae024b503594238e59f9fbf6ba283cf > ql/src/java/org/apache/hadoop/hive/ql/exec/mr/ExecDriver.java > d164859219896d88c42a69e56f621cb08012f633 > ql/src/java/org/apache/hadoop/hive/ql/exec/mr/HadoopJobExecHelper.java > 1b296b9986907d983a754f9957f2cbe4f7583ae5 > service/src/test/org/apache/hive/service/cli/CLIServiceTest.java > 698b13d66f100618aab3c3ee2cbf3c3df8477afe > > Diff: https://reviews.apache.org/r/45733/diff/ > > > Testing > --- > > > Thanks, > > Rajat Khandelwal > >
Re: Review Request 45738: HIVE-13415: Decouple Sessions from thrift binary transport
---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/45738/#review128632
---
Ship it!
Changes look good to me.
Carl, Szehon, can you also look at ?
- Amareshwari Sriramadasu
On April 13, 2016, 8:05 a.m., Rajat Khandelwal wrote:
>
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/45738/
> ---
>
> (Updated April 13, 2016, 8:05 a.m.)
>
>
> Review request for hive, Carl Steinbach and Szehon Ho.
>
>
> Bugs: HIVE-13415
> https://issues.apache.org/jira/browse/HIVE-13415
>
>
> Repository: hive-git
>
>
> Description
> ---
>
> Current behaviour is:
>
> * Open a thrift binary transport
> * create a session
> * close the transport
>
> Then the session gets closed. Consequently, all the operations running in the
> session also get killed.
>
> Whereas, if you open an HTTP transport, and close, the enclosing sessions are
> not closed.
>
> This seems like a bad design, having transport and sessions tightly coupled.
> I'd like to fix this.
>
> The issue that introduced it is
> [HIVE-9601|https://github.com/apache/hive/commit/48bea00c48853459af64b4ca9bfdc3e821c4ed82]
> Relevant discussions at
> [here|https://issues.apache.org/jira/browse/HIVE-11485?focusedCommentId=15223546&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15223546],
>
> [here|https://issues.apache.org/jira/browse/HIVE-11485?focusedCommentId=15223827&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15223827]
> and mentioned links on those comments.
>
> Another thing that seems like a slightly bad design is this line of code in
> ThriftBinaryCLIService:
>
> {noformat}
> server.setServerEventHandler(serverEventHandler);
> {noformat}
>
> Whereas serverEventHandler is defined by the base class, with no users except
> one sub-class(ThriftBinaryCLIService), violating the separation of concerns.
>
>
> Diffs
> -
>
> common/src/java/org/apache/hadoop/hive/conf/HiveConf.java
> c7e5b3305f2443a63b1f3210b3f2fdd563cab965
>
> service/src/java/org/apache/hive/service/cli/thrift/ThriftBinaryCLIService.java
> cf575a405f3eed2e1ec470670d5832ec3af9e1c1
> service/src/java/org/apache/hive/service/cli/thrift/ThriftCLIService.java
> be9833d1ac28951cb464716ddab4a7ea9d6b0a93
>
> service/src/test/org/apache/hive/service/cli/TestRetryingThriftCLIServiceClient.java
> 3bd82e614a50c0b5419a926ff00a95dafd4b0ebb
>
> Diff: https://reviews.apache.org/r/45738/diff/
>
>
> Testing
> ---
>
>
> Thanks,
>
> Rajat Khandelwal
>
>
Re: Review Request 44524: HIVE-11484 : Fix ObjectInspector for Char and VarChar
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/44524/#review128339 --- Ship it! Ship It! - Amareshwari Sriramadasu On March 30, 2016, 1:43 p.m., Deepak Barr wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/44524/ > --- > > (Updated March 30, 2016, 1:43 p.m.) > > > Review request for hive and Amareshwari Sriramadasu. > > > Bugs: HIVE-11484 > https://issues.apache.org/jira/browse/HIVE-11484 > > > Repository: hive-git > > > Description > --- > > Fixed ObjectInspectorConverter and Java ObjectInspector for Char and Varchar > > > Diffs > - > > common/src/java/org/apache/hadoop/hive/common/type/HiveBaseChar.java > 3514726 > common/src/java/org/apache/hadoop/hive/common/type/HiveVarchar.java 969d474 > common/src/test/org/apache/hadoop/hive/common/type/TestHiveBaseChar.java > 2848465 > > serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/primitive/JavaHiveCharObjectInspector.java > f429709 > > serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/primitive/JavaHiveVarcharObjectInspector.java > a8e34ff > > serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/primitive/PrimitiveObjectInspectorConverter.java > 5eb41d5 > > serde/src/test/org/apache/hadoop/hive/serde2/objectinspector/TestObjectInspectorConverters.java > dd18517 > > serde/src/test/org/apache/hadoop/hive/serde2/objectinspector/TestStandardObjectInspectors.java > 7d87666 > > Diff: https://reviews.apache.org/r/44524/diff/ > > > Testing > --- > > Yes. > > > Thanks, > > Deepak Barr > >
Re: Review Request 45738: HIVE-13415: Decouple Sessions from thrift binary transport
---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/45738/#review128072
---
service/src/java/org/apache/hive/service/cli/thrift/ThriftCLIService.java
<https://reviews.apache.org/r/45738/#comment191464>
Why is this code movement to other class?
service/src/test/org/apache/hive/service/cli/TestRetryingThriftCLIServiceClient.java
(line 174)
<https://reviews.apache.org/r/45738/#comment191463>
If client transport is closed, how are you able to do closeSession?
Shouldnt you create a new client?
- Amareshwari Sriramadasu
On April 6, 2016, 7:36 a.m., Rajat Khandelwal wrote:
>
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/45738/
> ---
>
> (Updated April 6, 2016, 7:36 a.m.)
>
>
> Review request for hive.
>
>
> Bugs: HIVE-13415
> https://issues.apache.org/jira/browse/HIVE-13415
>
>
> Repository: hive-git
>
>
> Description
> ---
>
> Current behaviour is:
>
> * Open a thrift binary transport
> * create a session
> * close the transport
>
> Then the session gets closed. Consequently, all the operations running in the
> session also get killed.
>
> Whereas, if you open an HTTP transport, and close, the enclosing sessions are
> not closed.
>
> This seems like a bad design, having transport and sessions tightly coupled.
> I'd like to fix this.
>
> The issue that introduced it is
> [HIVE-9601|https://github.com/apache/hive/commit/48bea00c48853459af64b4ca9bfdc3e821c4ed82]
> Relevant discussions at
> [here|https://issues.apache.org/jira/browse/HIVE-11485?focusedCommentId=15223546&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15223546],
>
> [here|https://issues.apache.org/jira/browse/HIVE-11485?focusedCommentId=15223827&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15223827]
> and mentioned links on those comments.
>
> Another thing that seems like a slightly bad design is this line of code in
> ThriftBinaryCLIService:
>
> {noformat}
> server.setServerEventHandler(serverEventHandler);
> {noformat}
>
> Whereas serverEventHandler is defined by the base class, with no users except
> one sub-class(ThriftBinaryCLIService), violating the separation of concerns.
>
>
> Diffs
> -
>
> common/src/java/org/apache/hadoop/hive/conf/HiveConf.java
> 95c5c0efddf9cb91786d5379ecf7c5af50a315ea
>
> service/src/java/org/apache/hive/service/cli/thrift/ThriftBinaryCLIService.java
> cf575a405f3eed2e1ec470670d5832ec3af9e1c1
> service/src/java/org/apache/hive/service/cli/thrift/ThriftCLIService.java
> 0a2a761867ab596e942dbc78eaabc0ef920665a3
>
> service/src/test/org/apache/hive/service/cli/TestRetryingThriftCLIServiceClient.java
> 3bd82e614a50c0b5419a926ff00a95dafd4b0ebb
>
> Diff: https://reviews.apache.org/r/45738/diff/
>
>
> Testing
> ---
>
>
> Thanks,
>
> Rajat Khandelwal
>
>
Re: Review Request 45032: HIVE-13319: HIVE-4570/HIVE-13319: Propagate external handle in task display
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/45032/#review124495 --- Ship it! Ship It! - Amareshwari Sriramadasu On March 20, 2016, 8:44 p.m., Rajat Khandelwal wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/45032/ > --- > > (Updated March 20, 2016, 8:44 p.m.) > > > Review request for hive. > > > Bugs: HIVE-13319 > https://issues.apache.org/jira/browse/HIVE-13319 > > > Repository: hive-git > > > Description > --- > > Currently in Hive Server2, when the query is still executing only the status > is set as STILL_EXECUTING. > > This issue is to give more information to the user such as progress and > running job handles, if possible. > > > Diffs > - > > ql/src/java/org/apache/hadoop/hive/ql/QueryDisplay.java > 467dab66e454d895742e96d4ac5db452fea00551 > service/src/test/org/apache/hive/service/cli/CLIServiceTest.java > e145eb434159d43b90480bad6711f965a82072c5 > > Diff: https://reviews.apache.org/r/45032/diff/ > > > Testing > --- > > > Thanks, > > Rajat Khandelwal > >
Re: Review Request 44783: HIVE-13183: More logs in operation logs
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/44783/#review123610 --- Ship it! Ship It! - Amareshwari Sriramadasu On March 15, 2016, 6:37 a.m., Rajat Khandelwal wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/44783/ > --- > > (Updated March 15, 2016, 6:37 a.m.) > > > Review request for hive. > > > Bugs: HIVE-13183 > https://issues.apache.org/jira/browse/HIVE-13183 > > > Repository: hive-git > > > Description > --- > > > Diffs > - > > common/src/java/org/apache/hive/common/util/StreamPrinter.java > 72638fdb966a63c64e44f2d952ecc49b4b0b5121 > ql/src/java/org/apache/hadoop/hive/ql/exec/TaskRunner.java > f6fd08104f6d5a8c93bca1f5d77468096e823144 > ql/src/java/org/apache/hadoop/hive/ql/exec/mr/MapRedTask.java > 310356c526149088b2c1fc5dc4c1da3cbf166595 > ql/src/java/org/apache/hadoop/hive/ql/exec/mr/MapredLocalTask.java > f5500a4c4ece56fcd756c9c2159169228e542a6e > ql/src/java/org/apache/hadoop/hive/ql/session/OperationLog.java > 2ecdde9799683cd104323af170d8905fb3005b9e > > Diff: https://reviews.apache.org/r/44783/diff/ > > > Testing > --- > > > Thanks, > > Rajat Khandelwal > >
Re: Review Request 44783: HIVE-13183: More logs in operation logs
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/44783/#review123391 --- ql/src/java/org/apache/hadoop/hive/ql/exec/TaskRunner.java (line 106) <https://reviews.apache.org/r/44783/#comment185643> Can you avoid printStackTrace() and change it log ? ql/src/java/org/apache/hadoop/hive/ql/exec/TaskRunner.java (line 108) <https://reviews.apache.org/r/44783/#comment185644> Can we directly log instead of printStackTrace? - Amareshwari Sriramadasu On March 14, 2016, 9:23 a.m., Rajat Khandelwal wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/44783/ > --- > > (Updated March 14, 2016, 9:23 a.m.) > > > Review request for hive. > > > Bugs: HIVE-13183 > https://issues.apache.org/jira/browse/HIVE-13183 > > > Repository: hive-git > > > Description > --- > > > Diffs > - > > common/src/java/org/apache/hive/common/util/StreamPrinter.java > 72638fdb966a63c64e44f2d952ecc49b4b0b5121 > ql/src/java/org/apache/hadoop/hive/ql/exec/TaskRunner.java > f6fd08104f6d5a8c93bca1f5d77468096e823144 > ql/src/java/org/apache/hadoop/hive/ql/exec/mr/MapredLocalTask.java > f5500a4c4ece56fcd756c9c2159169228e542a6e > ql/src/java/org/apache/hadoop/hive/ql/session/OperationLog.java > 2ecdde9799683cd104323af170d8905fb3005b9e > > Diff: https://reviews.apache.org/r/44783/diff/ > > > Testing > --- > > > Thanks, > > Rajat Khandelwal > >
Re: Review Request 44524: HIVE-11484 : Fix ObjectInspector for Char and VarChar
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/44524/#review123098 --- common/src/java/org/apache/hadoop/hive/common/type/HiveBaseChar.java (line 41) <https://reviews.apache.org/r/44524/#comment185243> Can you add unit test? serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/primitive/JavaHiveCharObjectInspector.java (line 53) <https://reviews.apache.org/r/44524/#comment185244> Can you add unit test? serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/primitive/PrimitiveObjectInspectorConverter.java (line 531) <https://reviews.apache.org/r/44524/#comment185245> Can you add unit test? - Amareshwari Sriramadasu On March 8, 2016, 7:17 p.m., Deepak Barr wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/44524/ > --- > > (Updated March 8, 2016, 7:17 p.m.) > > > Review request for hive and Amareshwari Sriramadasu. > > > Bugs: HIVE-11484 > https://issues.apache.org/jira/browse/HIVE-11484 > > > Repository: hive-git > > > Description > --- > > Fixed ObjectInspectorConverter and Java ObjectInspector for Char and Varchar > > > Diffs > - > > common/src/java/org/apache/hadoop/hive/common/type/HiveBaseChar.java > 3514726 > > serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/primitive/JavaHiveCharObjectInspector.java > f429709 > > serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/primitive/JavaHiveVarcharObjectInspector.java > a8e34ff > > serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/primitive/PrimitiveObjectInspectorConverter.java > 5eb41d5 > > Diff: https://reviews.apache.org/r/44524/diff/ > > > Testing > --- > > Yes. > > > Thanks, > > Deepak Barr > >
Re: Review Request 44453: HIVE-4570: More information to user on GetOperationStatus in Hive Server2 when query is still executing
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/44453/#review123095 --- Ship it! Ship It! - Amareshwari Sriramadasu On March 10, 2016, 7:55 a.m., Rajat Khandelwal wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/44453/ > --- > > (Updated March 10, 2016, 7:55 a.m.) > > > Review request for hive, Amareshwari Sriramadasu and Puneet Gupta. > > > Bugs: HIVE-4570 > https://issues.apache.org/jira/browse/HIVE-4570 > > > Repository: hive-git > > > Description > --- > > Currently in Hive Server2, when the query is still executing only the status > is set as STILL_EXECUTING. > > This issue is to give more information to the user such as progress and > running job handles, if possible. > > > Diffs > - > > > itests/hive-unit/src/test/java/org/apache/hive/service/cli/TestEmbeddedThriftBinaryCLIService.java > de66d9efb1cace9d32174e3020920d5e4002dc85 > > itests/hive-unit/src/test/java/org/apache/hive/service/cli/session/TestQueryDisplay.java > 9765b9dbe1ae097c108437c85b11b7bcee8f49da > ql/src/java/org/apache/hadoop/hive/ql/Driver.java > b50c5a2c4e396975f3323b16426445f869ff7b89 > ql/src/java/org/apache/hadoop/hive/ql/QueryDisplay.java > c87c825f95d2ff49bc62f892eb1cea5abf3c65c6 > ql/src/java/org/apache/hadoop/hive/ql/QueryPlan.java > 4933b34710afebde923b548572ffd7f98b7eb728 > ql/src/java/org/apache/hadoop/hive/ql/exec/Task.java > e199e5ee65404d2e9a38d20fa9d2ff72754962e9 > ql/src/java/org/apache/hadoop/hive/ql/exec/mr/ExecDriver.java > ce020a52721b439c4a9a81ee75e9829f2da00221 > ql/src/java/org/apache/hadoop/hive/ql/exec/mr/MapRedTask.java > 5bc3d9e732330af7890d7379334ed42be116ccbe > ql/src/java/org/apache/hadoop/hive/ql/history/HiveHistory.java > 45cd533270a550ae07510fe3974ddceaed30af51 > service-rpc/if/TCLIService.thrift 0aa9d13464bf284a3048092372299efb8e1d6bcc > service-rpc/src/gen/thrift/gen-cpp/TCLIService_types.h > 53642939829ec137db6b5ded46c6c3edef3b9e69 > service-rpc/src/gen/thrift/gen-cpp/TCLIService_types.cpp > e62e8b720e967dac6bbd4f53328a85256a07e022 > > service-rpc/src/gen/thrift/gen-javabean/org/apache/hive/service/rpc/thrift/TGetOperationStatusResp.java > a7a8ebc5c663961e6f4ebdb9df78fa3d53e31d98 > service-rpc/src/gen/thrift/gen-php/Types.php > 76805df4910a31636dccbd119abe91dfbb1568a1 > service-rpc/src/gen/thrift/gen-py/TCLIService/ttypes.py > ef5f5f568ffec92c093bdd0904c154f8ee67606e > service-rpc/src/gen/thrift/gen-rb/t_c_l_i_service_types.rb > f004ec40afd29ab4ed452f65f306cb1af5ff5207 > service/src/jamon/org/apache/hive/tmpl/QueryProfileTmpl.jamon > c51368947ffe5873c727fc7e74aa25e748707370 > service/src/java/org/apache/hive/service/cli/OperationStatus.java > e45b828193daf46e26c1587f7e0b9ea1a43f4930 > service/src/java/org/apache/hive/service/cli/operation/Operation.java > 22f725c948494865e176a8d18543caf462a77c19 > service/src/java/org/apache/hive/service/cli/operation/SQLOperation.java > 100dc6a5ceb10bff1d6591dd4b92062f4d787b98 > service/src/java/org/apache/hive/service/cli/thrift/ThriftCLIService.java > 8dff26467260c365d5869d5b3e3bae15c3acd92d > > service/src/java/org/apache/hive/service/cli/thrift/ThriftCLIServiceClient.java > 5f01165d0dfd131a2599b90a8e1c4d4970650b7a > service/src/test/org/apache/hive/service/cli/CLIServiceTest.java > e78181a15993d99f1cab5a061c08bb21823d2171 > > Diff: https://reviews.apache.org/r/44453/diff/ > > > Testing > --- > > > Thanks, > > Rajat Khandelwal > >
Re: Review Request 44453: HIVE-4570: More information to user on GetOperationStatus in Hive Server2 when query is still executing
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/44453/#review122877 --- - Amareshwari Sriramadasu On March 10, 2016, 7:55 a.m., Rajat Khandelwal wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/44453/ > --- > > (Updated March 10, 2016, 7:55 a.m.) > > > Review request for hive, Amareshwari Sriramadasu and Puneet Gupta. > > > Bugs: HIVE-4570 > https://issues.apache.org/jira/browse/HIVE-4570 > > > Repository: hive-git > > > Description > --- > > Currently in Hive Server2, when the query is still executing only the status > is set as STILL_EXECUTING. > > This issue is to give more information to the user such as progress and > running job handles, if possible. > > > Diffs > - > > > itests/hive-unit/src/test/java/org/apache/hive/service/cli/TestEmbeddedThriftBinaryCLIService.java > de66d9efb1cace9d32174e3020920d5e4002dc85 > > itests/hive-unit/src/test/java/org/apache/hive/service/cli/session/TestQueryDisplay.java > 9765b9dbe1ae097c108437c85b11b7bcee8f49da > ql/src/java/org/apache/hadoop/hive/ql/Driver.java > b50c5a2c4e396975f3323b16426445f869ff7b89 > ql/src/java/org/apache/hadoop/hive/ql/QueryDisplay.java > c87c825f95d2ff49bc62f892eb1cea5abf3c65c6 > ql/src/java/org/apache/hadoop/hive/ql/QueryPlan.java > 4933b34710afebde923b548572ffd7f98b7eb728 > ql/src/java/org/apache/hadoop/hive/ql/exec/Task.java > e199e5ee65404d2e9a38d20fa9d2ff72754962e9 > ql/src/java/org/apache/hadoop/hive/ql/exec/mr/ExecDriver.java > ce020a52721b439c4a9a81ee75e9829f2da00221 > ql/src/java/org/apache/hadoop/hive/ql/exec/mr/MapRedTask.java > 5bc3d9e732330af7890d7379334ed42be116ccbe > ql/src/java/org/apache/hadoop/hive/ql/history/HiveHistory.java > 45cd533270a550ae07510fe3974ddceaed30af51 > service-rpc/if/TCLIService.thrift 0aa9d13464bf284a3048092372299efb8e1d6bcc > service-rpc/src/gen/thrift/gen-cpp/TCLIService_types.h > 53642939829ec137db6b5ded46c6c3edef3b9e69 > service-rpc/src/gen/thrift/gen-cpp/TCLIService_types.cpp > e62e8b720e967dac6bbd4f53328a85256a07e022 > > service-rpc/src/gen/thrift/gen-javabean/org/apache/hive/service/rpc/thrift/TGetOperationStatusResp.java > a7a8ebc5c663961e6f4ebdb9df78fa3d53e31d98 > service-rpc/src/gen/thrift/gen-php/Types.php > 76805df4910a31636dccbd119abe91dfbb1568a1 > service-rpc/src/gen/thrift/gen-py/TCLIService/ttypes.py > ef5f5f568ffec92c093bdd0904c154f8ee67606e > service-rpc/src/gen/thrift/gen-rb/t_c_l_i_service_types.rb > f004ec40afd29ab4ed452f65f306cb1af5ff5207 > service/src/jamon/org/apache/hive/tmpl/QueryProfileTmpl.jamon > c51368947ffe5873c727fc7e74aa25e748707370 > service/src/java/org/apache/hive/service/cli/OperationStatus.java > e45b828193daf46e26c1587f7e0b9ea1a43f4930 > service/src/java/org/apache/hive/service/cli/operation/Operation.java > 22f725c948494865e176a8d18543caf462a77c19 > service/src/java/org/apache/hive/service/cli/operation/SQLOperation.java > 100dc6a5ceb10bff1d6591dd4b92062f4d787b98 > service/src/java/org/apache/hive/service/cli/thrift/ThriftCLIService.java > 8dff26467260c365d5869d5b3e3bae15c3acd92d > > service/src/java/org/apache/hive/service/cli/thrift/ThriftCLIServiceClient.java > 5f01165d0dfd131a2599b90a8e1c4d4970650b7a > service/src/test/org/apache/hive/service/cli/CLIServiceTest.java > e78181a15993d99f1cab5a061c08bb21823d2171 > > Diff: https://reviews.apache.org/r/44453/diff/ > > > Testing > --- > > > Thanks, > > Rajat Khandelwal > >
Re: Review Request 44453: HIVE-4570: More information to user on GetOperationStatus in Hive Server2 when query is still executing
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/44453/#review122876 --- service/src/java/org/apache/hive/service/cli/thrift/ThriftCLIService.java (line 682) <https://reviews.apache.org/r/44453/#comment184963> Is set on start and completion times not required? - Amareshwari Sriramadasu On March 9, 2016, 3:03 p.m., Rajat Khandelwal wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/44453/ > --- > > (Updated March 9, 2016, 3:03 p.m.) > > > Review request for hive, Amareshwari Sriramadasu and Puneet Gupta. > > > Bugs: HIVE-4570 > https://issues.apache.org/jira/browse/HIVE-4570 > > > Repository: hive-git > > > Description > --- > > Currently in Hive Server2, when the query is still executing only the status > is set as STILL_EXECUTING. > > This issue is to give more information to the user such as progress and > running job handles, if possible. > > > Diffs > - > > > itests/hive-unit/src/test/java/org/apache/hive/service/cli/TestEmbeddedThriftBinaryCLIService.java > de66d9efb1cace9d32174e3020920d5e4002dc85 > > itests/hive-unit/src/test/java/org/apache/hive/service/cli/session/TestQueryDisplay.java > 9765b9dbe1ae097c108437c85b11b7bcee8f49da > ql/src/java/org/apache/hadoop/hive/ql/Driver.java > b50c5a2c4e396975f3323b16426445f869ff7b89 > ql/src/java/org/apache/hadoop/hive/ql/QueryDisplay.java > c87c825f95d2ff49bc62f892eb1cea5abf3c65c6 > ql/src/java/org/apache/hadoop/hive/ql/QueryPlan.java > 4933b34710afebde923b548572ffd7f98b7eb728 > ql/src/java/org/apache/hadoop/hive/ql/exec/Task.java > e199e5ee65404d2e9a38d20fa9d2ff72754962e9 > ql/src/java/org/apache/hadoop/hive/ql/exec/mr/ExecDriver.java > ce020a52721b439c4a9a81ee75e9829f2da00221 > ql/src/java/org/apache/hadoop/hive/ql/exec/mr/MapRedTask.java > 5bc3d9e732330af7890d7379334ed42be116ccbe > ql/src/java/org/apache/hadoop/hive/ql/history/HiveHistory.java > 45cd533270a550ae07510fe3974ddceaed30af51 > service-rpc/if/TCLIService.thrift 0aa9d13464bf284a3048092372299efb8e1d6bcc > > service-rpc/src/gen/thrift/gen-javabean/org/apache/hive/service/rpc/thrift/TGetOperationStatusResp.java > a7a8ebc5c663961e6f4ebdb9df78fa3d53e31d98 > service/src/jamon/org/apache/hive/tmpl/QueryProfileTmpl.jamon > c51368947ffe5873c727fc7e74aa25e748707370 > service/src/java/org/apache/hive/service/cli/OperationStatus.java > e45b828193daf46e26c1587f7e0b9ea1a43f4930 > service/src/java/org/apache/hive/service/cli/operation/Operation.java > 22f725c948494865e176a8d18543caf462a77c19 > service/src/java/org/apache/hive/service/cli/operation/SQLOperation.java > 100dc6a5ceb10bff1d6591dd4b92062f4d787b98 > service/src/java/org/apache/hive/service/cli/thrift/ThriftCLIService.java > 8dff26467260c365d5869d5b3e3bae15c3acd92d > > service/src/java/org/apache/hive/service/cli/thrift/ThriftCLIServiceClient.java > 5f01165d0dfd131a2599b90a8e1c4d4970650b7a > service/src/test/org/apache/hive/service/cli/CLIServiceTest.java > e78181a15993d99f1cab5a061c08bb21823d2171 > > Diff: https://reviews.apache.org/r/44453/diff/ > > > Testing > --- > > > Thanks, > > Rajat Khandelwal > >
Re: Review Request 44453: HIVE-4570: More information to user on GetOperationStatus in Hive Server2 when query is still executing
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/44453/#review122476 --- Seems there are merge issues, its showing unnecessary changes. Also, please avoid posting generated thrift files for review, as it becomes hard to review. - Amareshwari Sriramadasu On March 7, 2016, 2:43 p.m., Rajat Khandelwal wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/44453/ > --- > > (Updated March 7, 2016, 2:43 p.m.) > > > Review request for hive, Amareshwari Sriramadasu and Puneet Gupta. > > > Bugs: HIVE-4570 > https://issues.apache.org/jira/browse/HIVE-4570 > > > Repository: hive-git > > > Description > --- > > Currently in Hive Server2, when the query is still executing only the status > is set as STILL_EXECUTING. > > This issue is to give more information to the user such as progress and > running job handles, if possible. > > > Diffs > - > > > itests/hive-unit/src/test/java/org/apache/hive/service/cli/TestEmbeddedThriftBinaryCLIService.java > de66d9efb1cace9d32174e3020920d5e4002dc85 > ql/src/java/org/apache/hadoop/hive/ql/Driver.java > 32531465cba95de6caa1bd292d489c1582efb942 > ql/src/java/org/apache/hadoop/hive/ql/TaskStatus.java PRE-CREATION > ql/src/java/org/apache/hadoop/hive/ql/exec/Task.java > e199e5ee65404d2e9a38d20fa9d2ff72754962e9 > ql/src/java/org/apache/hadoop/hive/ql/exec/mr/ExecDriver.java > b184b4e554cf3f7a61932195b9b840cf13f3594b > ql/src/java/org/apache/hadoop/hive/ql/exec/mr/MapRedTask.java > 5bc3d9e732330af7890d7379334ed42be116ccbe > serde/if/serde.thrift 0b3804deadaa327ab22c9cffef762f1b862200e2 > serde/src/gen/thrift/gen-cpp/serde_constants.h > 6d859284296d108b0dbd6e9fb7bad53f3ce6c913 > serde/src/gen/thrift/gen-cpp/serde_constants.cpp > 75701e2a3db9394c13ef73fc48d3b2e8488d68b4 > > serde/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/serde/serdeConstants.java > 04ed8f556be0cf28f89b0ccd005816a2fb01a555 > serde/src/gen/thrift/gen-php/org/apache/hadoop/hive/serde/Types.php > 0bc6dd7e01280f27750a4ceaa37f8fb9ecfb914a > serde/src/gen/thrift/gen-py/org_apache_hadoop_hive_serde/constants.py > 7939791ed30d5035a84bbfd0af848de9a624926e > serde/src/gen/thrift/gen-rb/serde_constants.rb > d09e3c238a747d9f5f81118f633968baa16589e7 > service-rpc/if/TCLIService.thrift 0aa9d13464bf284a3048092372299efb8e1d6bcc > service-rpc/src/gen/thrift/gen-cpp/TCLIService_types.h > 53642939829ec137db6b5ded46c6c3edef3b9e69 > service-rpc/src/gen/thrift/gen-cpp/TCLIService_types.cpp > e62e8b720e967dac6bbd4f53328a85256a07e022 > > service-rpc/src/gen/thrift/gen-javabean/org/apache/hive/service/rpc/thrift/TGetOperationStatusResp.java > a7a8ebc5c663961e6f4ebdb9df78fa3d53e31d98 > service-rpc/src/gen/thrift/gen-php/Types.php > 76805df4910a31636dccbd119abe91dfbb1568a1 > service-rpc/src/gen/thrift/gen-py/TCLIService/ttypes.py > ef5f5f568ffec92c093bdd0904c154f8ee67606e > service-rpc/src/gen/thrift/gen-rb/t_c_l_i_service_types.rb > f004ec40afd29ab4ed452f65f306cb1af5ff5207 > service/src/gen/thrift/gen-cpp/hive_service_constants.h PRE-CREATION > service/src/gen/thrift/gen-cpp/hive_service_constants.cpp PRE-CREATION > service/src/gen/thrift/gen-cpp/hive_service_types.h PRE-CREATION > service/src/gen/thrift/gen-cpp/hive_service_types.cpp PRE-CREATION > > service/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/service/HiveClusterStatus.java > PRE-CREATION > > service/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/service/HiveServerException.java > PRE-CREATION > > service/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/service/JobTrackerState.java > PRE-CREATION > > service/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/service/ThriftHive.java > PRE-CREATION > service/src/gen/thrift/gen-php/ThriftHive.php PRE-CREATION > service/src/gen/thrift/gen-php/Types.php PRE-CREATION > service/src/gen/thrift/gen-py/__init__.py PRE-CREATION > service/src/gen/thrift/gen-py/hive_service/ThriftHive-remote PRE-CREATION > service/src/gen/thrift/gen-py/hive_service/ThriftHive.py PRE-CREATION > service/src/gen/thrift/gen-py/hive_service/__init__.py PRE-CREATION > service/src/gen/thrift/gen-py/hive_service/constants.py PRE-CREATION > service/src/gen/thrift/gen-py/hive_service/ttypes.py PRE-CREATION > service/src/gen/thrift/gen-rb/hive_service_constants.rb PRE-CREATION >
Re: Review Request 44172: HIVE-11483: Add encoding and decoding for query string config
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/44172/#review121822 --- Ship it! Make sure all occurrences of HiveConf.ConfVars.HIVEQUERYSTRING taken care, by doing a git grep. - Amareshwari Sriramadasu On March 3, 2016, 7:59 a.m., Rajat Khandelwal wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/44172/ > --- > > (Updated March 3, 2016, 7:59 a.m.) > > > Review request for hive and Alan Gates. > > > Bugs: HIVE-11483 > https://issues.apache.org/jira/browse/HIVE-11483 > > > Repository: hive-git > > > Description > --- > > We have seen some queries in production where some of the literals passed in > the query have control characters, which result in exception when query > string is set in the job xml. > > Proposing a solution to encode the query string in configuration and provide > getters decoded string. > > Here is a commit in a forked repo : > https://github.com/InMobi/hive/commit/2faf5761191fa3103a0d779fde584d494ed75bf5 > > Suggestions are welcome on the solution. > > > Diffs > - > > common/src/java/org/apache/hadoop/hive/conf/HiveConf.java > 2723dadd76418f23b6e6358d21304e3312ef1cbd > common/src/test/org/apache/hadoop/hive/conf/TestHiveConf.java > 365d5005d2a1f240b12bb4e9f9a920b3db4d32ba > ql/src/java/org/apache/hadoop/hive/ql/Driver.java > 32531465cba95de6caa1bd292d489c1582efb942 > ql/src/java/org/apache/hadoop/hive/ql/exec/errors/TaskLogProcessor.java > 68123d45c335e833984d3547607dc3ee7d8fd870 > ql/src/java/org/apache/hadoop/hive/ql/exec/mr/ExecDriver.java > b184b4e554cf3f7a61932195b9b840cf13f3594b > ql/src/java/org/apache/hadoop/hive/ql/exec/tez/DagUtils.java > 473dbd61bf538fe08953dcf7994d5821f6bdbacc > ql/src/java/org/apache/hadoop/hive/ql/session/SessionState.java > 70b2bc0cc15cd862cdb1adba0690d0530681ebde > ql/src/test/org/apache/hadoop/hive/ql/exec/errors/TestTaskLogProcessor.java > 67a86a60949a6b70f71593470a6d0ef8c265ee04 > ql/src/test/org/apache/hadoop/hive/ql/hooks/TestHooks.java > 8d27762522f0ac5f59696e019ec52a25e77318db > > Diff: https://reviews.apache.org/r/44172/diff/ > > > Testing > --- > > > Thanks, > > Rajat Khandelwal > >
Re: Review Request 44201: HIVE-13188: Allow users of RetryingThriftClient to close transport
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/44201/#review121820 --- Ship it! Ship It! - Amareshwari Sriramadasu On March 3, 2016, 6:50 a.m., Rajat Khandelwal wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/44201/ > --- > > (Updated March 3, 2016, 6:50 a.m.) > > > Review request for hive. > > > Bugs: HIVE-13188 > https://issues.apache.org/jira/browse/HIVE-13188 > > > Repository: hive-git > > > Description > --- > > RetryingThriftCLIClient opens a TTransport and leaves it open. there should > be a way to close that. > > > Diffs > - > > > service/src/java/org/apache/hive/service/cli/thrift/RetryingThriftCLIServiceClient.java > a81c600236e6e0f53a484806e559303279244616 > > service/src/test/org/apache/hive/service/cli/TestRetryingThriftCLIServiceClient.java > 3798053a4a2e2b26b97b5accf96d73b58c89fb6e > > Diff: https://reviews.apache.org/r/44201/diff/ > > > Testing > --- > > > Thanks, > > Rajat Khandelwal > >
Re: Review Request 44172: HIVE-11483: Add encoding and decoding for query string config
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/44172/#review121807 --- common/src/java/org/apache/hadoop/hive/conf/HiveConf.java (line 139) <https://reviews.apache.org/r/44172/#comment183642> Required any more? common/src/java/org/apache/hadoop/hive/conf/HiveConf.java (line 3389) <https://reviews.apache.org/r/44172/#comment183641> We should have a wrapper for set/getQueryString which will encode with URLencoder. And call callers should change to call this. - Amareshwari Sriramadasu On March 2, 2016, 6:01 a.m., Rajat Khandelwal wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/44172/ > --- > > (Updated March 2, 2016, 6:01 a.m.) > > > Review request for hive and Alan Gates. > > > Bugs: HIVE-11483 > https://issues.apache.org/jira/browse/HIVE-11483 > > > Repository: hive-git > > > Description > --- > > We have seen some queries in production where some of the literals passed in > the query have control characters, which result in exception when query > string is set in the job xml. > > Proposing a solution to encode the query string in configuration and provide > getters decoded string. > > Here is a commit in a forked repo : > https://github.com/InMobi/hive/commit/2faf5761191fa3103a0d779fde584d494ed75bf5 > > Suggestions are welcome on the solution. > > > Diffs > - > > common/src/java/org/apache/hadoop/hive/conf/HiveConf.java > 2723dadd76418f23b6e6358d21304e3312ef1cbd > common/src/test/org/apache/hadoop/hive/conf/TestHiveConf.java > 365d5005d2a1f240b12bb4e9f9a920b3db4d32ba > > Diff: https://reviews.apache.org/r/44172/diff/ > > > Testing > --- > > > Thanks, > > Rajat Khandelwal > >
Re: Review Request 44201: HIVE-13188: Allow users of RetryingThriftClient to close transport
> On March 1, 2016, 8:25 a.m., Amareshwari Sriramadasu wrote: > > service/src/test/org/apache/hive/service/cli/TestRetryingThriftCLIServiceClient.java, > > line 61 > > <https://reviews.apache.org/r/44201/diff/1/?file=1275009#file1275009line61> > > > > Can you update test to call new method introduced? > > Rajat Khandelwal wrote: > Updated. Test is also passing Can you update by calling close Transport() as well ? - Amareshwari --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/44201/#review121407 --- On March 2, 2016, 6:07 a.m., Rajat Khandelwal wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/44201/ > --- > > (Updated March 2, 2016, 6:07 a.m.) > > > Review request for hive. > > > Bugs: HIVE-13188 > https://issues.apache.org/jira/browse/HIVE-13188 > > > Repository: hive-git > > > Description > --- > > RetryingThriftCLIClient opens a TTransport and leaves it open. there should > be a way to close that. > > > Diffs > - > > > service/src/java/org/apache/hive/service/cli/thrift/RetryingThriftCLIServiceClient.java > a81c600236e6e0f53a484806e559303279244616 > > service/src/test/org/apache/hive/service/cli/TestRetryingThriftCLIServiceClient.java > 3798053a4a2e2b26b97b5accf96d73b58c89fb6e > > Diff: https://reviews.apache.org/r/44201/diff/ > > > Testing > --- > > > Thanks, > > Rajat Khandelwal > >
Re: Review Request 18182: HIVE-5370. format_number udf should take user specifed format as argument
> On March 2, 2016, 5:53 p.m., Ashutosh Chauhan wrote: > > ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFFormatNumber.java, > > line 178 > > <https://reviews.apache.org/r/18182/diff/3/?file=1275798#file1275798line178> > > > > I assume you only allow F to be a constant string. If so, this > > evaluation can happen at compile time in initialize(). You can obtain > > value via ConstantStringOI. Yes. Makes sense. Updated. - Amareshwari --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/18182/#review121679 ------- On March 3, 2016, 6:19 a.m., Amareshwari Sriramadasu wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/18182/ > --- > > (Updated March 3, 2016, 6:19 a.m.) > > > Review request for hive, Ashutosh Chauhan, Alan Gates, and Navis Ryu. > > > Bugs: HIVE-5370 > https://issues.apache.org/jira/browse/HIVE-5370 > > > Repository: hive-git > > > Description > --- > > Added the format as optional argument. > > > Diffs > - > > > ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFFormatNumber.java > 71ca8f2 > ql/src/test/queries/clientnegative/udf_format_number_wrong6.q e5b11b9 > ql/src/test/queries/clientpositive/udf_format_number.q 28f087d > ql/src/test/results/clientnegative/udf_format_number_wrong1.q.out c3cb800 > ql/src/test/results/clientnegative/udf_format_number_wrong2.q.out 1fe8a7c > ql/src/test/results/clientnegative/udf_format_number_wrong4.q.out 3953ef1 > ql/src/test/results/clientnegative/udf_format_number_wrong6.q.out d51991f > ql/src/test/results/clientpositive/udf_format_number.q.out 4a2c80d > > Diff: https://reviews.apache.org/r/18182/diff/ > > > Testing > --- > > Unit tested > > > Thanks, > > Amareshwari Sriramadasu > >
Re: Review Request 18182: HIVE-5370. format_number udf should take user specifed format as argument
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/18182/ --- (Updated March 3, 2016, 6:19 a.m.) Review request for hive, Ashutosh Chauhan, Alan Gates, and Navis Ryu. Changes --- Review comments incorporated Bugs: HIVE-5370 https://issues.apache.org/jira/browse/HIVE-5370 Repository: hive-git Description --- Added the format as optional argument. Diffs (updated) - ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFFormatNumber.java 71ca8f2 ql/src/test/queries/clientnegative/udf_format_number_wrong6.q e5b11b9 ql/src/test/queries/clientpositive/udf_format_number.q 28f087d ql/src/test/results/clientnegative/udf_format_number_wrong1.q.out c3cb800 ql/src/test/results/clientnegative/udf_format_number_wrong2.q.out 1fe8a7c ql/src/test/results/clientnegative/udf_format_number_wrong4.q.out 3953ef1 ql/src/test/results/clientnegative/udf_format_number_wrong6.q.out d51991f ql/src/test/results/clientpositive/udf_format_number.q.out 4a2c80d Diff: https://reviews.apache.org/r/18182/diff/ Testing --- Unit tested Thanks, Amareshwari Sriramadasu
Re: Review Request 18182: HIVE-5370. format_number udf should take user specifed format as argument
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/18182/ --- (Updated March 1, 2016, 11:01 p.m.) Review request for hive, Ashutosh Chauhan, Alan Gates, and Navis Ryu. Changes --- Patch updated to master Bugs: HIVE-5370 https://issues.apache.org/jira/browse/HIVE-5370 Repository: hive-git Description (updated) --- Added the format as optional argument. Diffs (updated) - ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFFormatNumber.java 71ca8f2 ql/src/test/queries/clientnegative/udf_format_number_wrong6.q e5b11b9 ql/src/test/queries/clientpositive/udf_format_number.q 28f087d ql/src/test/results/clientnegative/udf_format_number_wrong1.q.out c3cb800 ql/src/test/results/clientnegative/udf_format_number_wrong2.q.out 1fe8a7c ql/src/test/results/clientnegative/udf_format_number_wrong4.q.out 3953ef1 ql/src/test/results/clientnegative/udf_format_number_wrong6.q.out d51991f ql/src/test/results/clientpositive/udf_format_number.q.out 4a2c80d Diff: https://reviews.apache.org/r/18182/diff/ Testing --- Unit tested Thanks, Amareshwari Sriramadasu
Re: Review Request 44172: HIVE-11483: Add encoding and decoding for query string config
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/44172/#review121409 --- common/src/java/org/apache/hadoop/hive/conf/HiveConf.java (line 138) <https://reviews.apache.org/r/44172/#comment183085> I see adding a factory is giving the flexibility of using encoder for any param. But intercepting every get call with decoder can cause performance issues, especially with classes like HiveConf, which are used heavily. Can we have some numbers around the performance overhead of intercepting vs direct getter. - Amareshwari Sriramadasu On March 1, 2016, 7:39 a.m., Rajat Khandelwal wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/44172/ > --- > > (Updated March 1, 2016, 7:39 a.m.) > > > Review request for hive and Alan Gates. > > > Bugs: HIVE-11483 > https://issues.apache.org/jira/browse/HIVE-11483 > > > Repository: hive-git > > > Description > --- > > We have seen some queries in production where some of the literals passed in > the query have control characters, which result in exception when query > string is set in the job xml. > > Proposing a solution to encode the query string in configuration and provide > getters decoded string. > > Here is a commit in a forked repo : > https://github.com/InMobi/hive/commit/2faf5761191fa3103a0d779fde584d494ed75bf5 > > Suggestions are welcome on the solution. > > > Diffs > - > > common/src/java/org/apache/hadoop/hive/conf/HiveConf.java > 2723dadd76418f23b6e6358d21304e3312ef1cbd > common/src/test/org/apache/hadoop/hive/conf/TestHiveConf.java > 365d5005d2a1f240b12bb4e9f9a920b3db4d32ba > > Diff: https://reviews.apache.org/r/44172/diff/ > > > Testing > --- > > > Thanks, > > Rajat Khandelwal > >
Re: Review Request 44201: HIVE-13188: Allow users of RetryingThriftClient to close transport
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/44201/#review121407 --- service/src/test/org/apache/hive/service/cli/TestRetryingThriftCLIServiceClient.java (line 61) <https://reviews.apache.org/r/44201/#comment183083> Can you update test to call new method introduced? - Amareshwari Sriramadasu On March 1, 2016, 7:18 a.m., Rajat Khandelwal wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/44201/ > --- > > (Updated March 1, 2016, 7:18 a.m.) > > > Review request for hive. > > > Bugs: HIVE-13188 > https://issues.apache.org/jira/browse/HIVE-13188 > > > Repository: hive-git > > > Description > --- > > RetryingThriftCLIClient opens a TTransport and leaves it open. there should > be a way to close that. > > > Diffs > - > > > service/src/java/org/apache/hive/service/cli/thrift/RetryingThriftCLIServiceClient.java > a81c600236e6e0f53a484806e559303279244616 > > service/src/test/org/apache/hive/service/cli/TestRetryingThriftCLIServiceClient.java > 3798053a4a2e2b26b97b5accf96d73b58c89fb6e > > Diff: https://reviews.apache.org/r/44201/diff/ > > > Testing > --- > > > Thanks, > > Rajat Khandelwal > >
Re: Review Request 44146: HIVE-13179: Allow custom HiveConf to be passed to Authentication Providers
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/44146/#review121215 --- Ship it! +1 - Amareshwari Sriramadasu On Feb. 28, 2016, 5:20 p.m., Rajat Khandelwal wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/44146/ > --- > > (Updated Feb. 28, 2016, 5:20 p.m.) > > > Review request for hive. > > > Bugs: HIVE-13179 > https://issues.apache.org/jira/browse/HIVE-13179 > > > Repository: hive-git > > > Description > --- > > Right now if I want to create an ldap auth provider, I have to create a > hive-site.xml, set endpoints and other relevant properties there, then > instantiate `LdapAuthenticationProviderImpl`, since inside the constructor, a > new HiveConf is constructed. > > A better and more reusable design would be to ask for the conf in the > constructor itself. That will allow an external user to create a HiveConf, > set all relevant properties and instantiate `LdapAuthenticationProviderImpl` > with that conf. > > > Diffs > - > > > service/src/java/org/apache/hive/service/auth/AuthenticationProviderFactory.java > 4b95503 > > service/src/java/org/apache/hive/service/auth/CustomAuthenticationProviderImpl.java > 3dc0aa8 > > service/src/java/org/apache/hive/service/auth/LdapAuthenticationProviderImpl.java > 1d4aba2 > > service/src/java/org/apache/hive/service/auth/PamAuthenticationProviderImpl.java > fd58081 > > service/src/test/org/apache/hive/service/auth/TestLdapAtnProviderWithMiniDS.java > e5cee37 > > service/src/test/org/apache/hive/service/auth/TestLdapAuthenticationProviderImpl.java > 5e91a0e > > Diff: https://reviews.apache.org/r/44146/diff/ > > > Testing > --- > > > Thanks, > > Rajat Khandelwal > >
Re: Review Request 43408: HIVE-13034 : Add jdeb plugin to build debian
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/43408/#review120482 --- Ship it! Can you update testing done? - Amareshwari Sriramadasu On Feb. 24, 2016, 9:22 a.m., Arshad Matin wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/43408/ > --- > > (Updated Feb. 24, 2016, 9:22 a.m.) > > > Review request for hive. > > > Bugs: HIVE-13034 > https://issues.apache.org/jira/browse/HIVE-13034 > > > Repository: hive-git > > > Description > --- > > HIVE-13034 : Add jdeb plugin to build debian > > > Diffs > - > > packaging/pom.xml e7feb62 > packaging/src/deb/control/control PRE-CREATION > > Diff: https://reviews.apache.org/r/43408/diff/ > > > Testing > --- > > > Thanks, > > Arshad Matin > >
Re: Review Request 43597: HIVE-13064: Serde properties are not working while expecting output in a directory
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/43597/#review119866 --- Ship it! Ship It! - Amareshwari Sriramadasu On Feb. 19, 2016, 9:21 a.m., Rajat Khandelwal wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/43597/ > --- > > (Updated Feb. 19, 2016, 9:21 a.m.) > > > Review request for hive and Amareshwari Sriramadasu. > > > Bugs: HIVE-13064 > https://issues.apache.org/jira/browse/HIVE-13064 > > > Repository: hive-git > > > Description > --- > > Added handling of serde properties. > > > Diffs > - > > ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java 8a06582 > ql/src/java/org/apache/hadoop/hive/ql/plan/PlanUtils.java 04d26f3 > ql/src/test/queries/clientpositive/insert_overwrite_directory.q c3d419b > ql/src/test/results/clientpositive/insert_overwrite_directory.q.out e5d2455 > > Diff: https://reviews.apache.org/r/43597/diff/ > > > Testing > --- > > > Thanks, > > Rajat Khandelwal > >
Re: Review Request 43597: HIVE-13064: Serde properties are not working while expecting output in a directory
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/43597/#review119298 --- ql/src/test/results/clientpositive/insert_overwrite_directory.q.out (line 1204) <https://reviews.apache.org/r/43597/#comment180628> Can you add colelction.delim (not a typo property has typo) also to the test? and update output accordingly? ql/src/test/results/clientpositive/insert_overwrite_directory.q.out (line 1241) <https://reviews.apache.org/r/43597/#comment180629> Can you add colelction.delim and mapkey.delim also and update output accordingly? - Amareshwari Sriramadasu On Feb. 16, 2016, 11:30 a.m., Rajat Khandelwal wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/43597/ > --- > > (Updated Feb. 16, 2016, 11:30 a.m.) > > > Review request for hive and Amareshwari Sriramadasu. > > > Bugs: HIVE-13064 > https://issues.apache.org/jira/browse/HIVE-13064 > > > Repository: hive-git > > > Description > --- > > Added handling of serde properties. > > > Diffs > - > > ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java 8a06582 > ql/src/java/org/apache/hadoop/hive/ql/plan/PlanUtils.java 04d26f3 > ql/src/test/results/clientpositive/insert_overwrite_directory.q.out e5d2455 > > Diff: https://reviews.apache.org/r/43597/diff/ > > > Testing > --- > > > Thanks, > > Rajat Khandelwal > >
Re: Review Request 42134: More information to user on GetOperationStatus in Hive Server2 when query is still executing
---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/42134/#review118997
---
service/src/java/org/apache/hive/service/cli/thrift/ThriftCLIService.java (line
686)
<https://reviews.apache.org/r/42134/#comment180294>
Shouldnt set on task status also be called?
service/src/test/org/apache/hive/service/cli/CLIServiceTest.java (line 675)
<https://reviews.apache.org/r/42134/#comment180296>
can you put the values you are seeing in the testing done and add test
assert here?
service/src/test/org/apache/hive/service/cli/CLIServiceTest.java (line 678)
<https://reviews.apache.org/r/42134/#comment180295>
Shouldnt this fail instead of system out?
- Amareshwari Sriramadasu
On Feb. 11, 2016, 12:35 p.m., Akshay Goyal wrote:
>
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/42134/
> ---
>
> (Updated Feb. 11, 2016, 12:35 p.m.)
>
>
> Review request for hive and Amareshwari Sriramadasu.
>
>
> Bugs: HIVE-4570
> https://issues.apache.org/jira/browse/HIVE-4570
>
>
> Repository: hive-git
>
>
> Description
> ---
>
> Driver maintains list of running and runnable tasks although that info is not
> exposed outside. It's kept locally in the driver's execute method. We can add
> Driver.getTaskStatuses() to return status on all tasks (both running and
> runnable). Similarly, start and completion times for operations.
>
> Proposed changes are :
>
> struct TGetOperationStatusResp {
> 1: required TStatus status
> 2: optional TOperationState operationState
>
> // If operationState is ERROR_STATE, then the following fields may be set
> // sqlState as defined in the ISO/IEF CLI specification
> 3: optional string sqlState
>
> // Internal error code
> 4: optional i32 errorCode
>
> // Error message
> 5: optional string errorMessage
>
> // List of statuses of sub tasks
> 6: optional string taskStatus
>
> // When was the operation started
> 7: optional i64 operationStarted
> // When was the operation completed
> 8: optional i64 operationCompleted
>
> }
>
>
> Diffs
> -
>
> ql/src/java/org/apache/hadoop/hive/ql/Driver.java 7147a9a
> ql/src/java/org/apache/hadoop/hive/ql/TaskStatus.java PRE-CREATION
> ql/src/java/org/apache/hadoop/hive/ql/exec/Task.java 40c89cb
> ql/src/java/org/apache/hadoop/hive/ql/exec/mr/ExecDriver.java b184b4e
> service-rpc/if/TCLIService.thrift 0aa9d13
> service/src/java/org/apache/hive/service/cli/OperationStatus.java e45b828
>
> service/src/java/org/apache/hive/service/cli/operation/GetCatalogsOperation.java
> 8868ec1
>
> service/src/java/org/apache/hive/service/cli/operation/GetColumnsOperation.java
> 35b6c52
>
> service/src/java/org/apache/hive/service/cli/operation/GetFunctionsOperation.java
> 8db2e62
>
> service/src/java/org/apache/hive/service/cli/operation/GetSchemasOperation.java
> d6f6280
>
> service/src/java/org/apache/hive/service/cli/operation/GetTableTypesOperation.java
> a09b39a
>
> service/src/java/org/apache/hive/service/cli/operation/GetTablesOperation.java
> 740b851
>
> service/src/java/org/apache/hive/service/cli/operation/GetTypeInfoOperation.java
> 2a0fec2
>
> service/src/java/org/apache/hive/service/cli/operation/HiveCommandOperation.java
> f5a9771
> service/src/java/org/apache/hive/service/cli/operation/Operation.java
> 22f725c
> service/src/java/org/apache/hive/service/cli/operation/SQLOperation.java
> 100dc6a
> service/src/java/org/apache/hive/service/cli/thrift/ThriftCLIService.java
> d9a9e3b
>
> service/src/java/org/apache/hive/service/cli/thrift/ThriftCLIServiceClient.java
> 5f01165
> service/src/test/org/apache/hive/service/cli/CLIServiceTest.java e78181a
>
> Diff: https://reviews.apache.org/r/42134/diff/
>
>
> Testing
> ---
>
>
> Thanks,
>
> Akshay Goyal
>
>
Re: Review Request 43408: HIVE-13034 : Add jdeb plugin to build debian
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/43408/#review118848 --- packaging/pom.xml (line 92) <https://reviews.apache.org/r/43408/#comment180154> Can we move to latest verison 1.5 ? - Amareshwari Sriramadasu On Feb. 10, 2016, 6:34 a.m., Arshad Matin wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/43408/ > --- > > (Updated Feb. 10, 2016, 6:34 a.m.) > > > Review request for hive. > > > Repository: hive-git > > > Description > --- > > HIVE-13034 : Add jdeb plugin to build debian > > > Diffs > - > > packaging/pom.xml e7feb62 > packaging/src/deb/control/control PRE-CREATION > > Diff: https://reviews.apache.org/r/43408/diff/ > > > Testing > --- > > > Thanks, > > Arshad Matin > >
Re: Review Request 43408: HIVE-13034 : Add jdeb plugin to build debian
---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/43408/#review118623
---
packaging/pom.xml (line 101)
<https://reviews.apache.org/r/43408/#comment179917>
Should be ${project.version} ?
packaging/pom.xml (line 108)
<https://reviews.apache.org/r/43408/#comment179916>
Directory should /usr/local/hive/bin/ instead of top level hive/
packaging/src/deb/control/control (line 20)
<https://reviews.apache.org/r/43408/#comment179915>
Should we make it java7 ?
packaging/src/deb/control/control (line 21)
<https://reviews.apache.org/r/43408/#comment179914>
Should be dev@hive.apache.org
- Amareshwari Sriramadasu
On Feb. 10, 2016, 6:34 a.m., Arshad Matin wrote:
>
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/43408/
> ---
>
> (Updated Feb. 10, 2016, 6:34 a.m.)
>
>
> Review request for hive.
>
>
> Repository: hive-git
>
>
> Description
> ---
>
> HIVE-13034 : Add jdeb plugin to build debian
>
>
> Diffs
> -
>
> packaging/pom.xml e7feb62
> packaging/src/deb/control/control PRE-CREATION
>
> Diff: https://reviews.apache.org/r/43408/diff/
>
>
> Testing
> ---
>
>
> Thanks,
>
> Arshad Matin
>
>
Re: Review Request 41928: HIVE-12777: Add capability to restore session
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/41928/#review116943 --- Ship it! Ship It! - Amareshwari Sriramadasu On Jan. 29, 2016, 7:24 a.m., Rajat Khandelwal wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/41928/ > --- > > (Updated Jan. 29, 2016, 7:24 a.m.) > > > Review request for hive. > > > Bugs: HIVE-12777 > https://issues.apache.org/jira/browse/HIVE-12777 > > > Repository: hive-git > > > Description > --- > > Extensions using Hive session handles should be able to restore the hive > session from the handle. > > Apache Lens depends on a fork of hive and that fork has such a capability. > > Relevant commit: > https://github.com/InMobi/hive/commit/931fe9116161a18952c082c14223ad6745fefe00#diff-0acb35f7cab7492f522b0c40ce3ce1be > > > Diffs > - > > service/src/java/org/apache/hive/service/cli/CLIService.java > 0d5ae1e491bdcf98a12f5eef7b35f23c7b9dfc00 > service/src/java/org/apache/hive/service/cli/SessionHandle.java > 4e4f0211de09408168c8da669626fb49f3db364c > service/src/java/org/apache/hive/service/cli/session/HiveSessionImpl.java > 3c5700bd114f6b792df10df23e57dad9b9e689f6 > > service/src/java/org/apache/hive/service/cli/session/HiveSessionImplwithUGI.java > c56d97afc969ea57810e9b3222b20e086e847d67 > service/src/java/org/apache/hive/service/cli/session/SessionManager.java > de18f1d93502704e7a8fbb8d31d93bd770c4d232 > service/src/test/org/apache/hive/service/cli/CLIServiceRestoreTest.java > PRE-CREATION > > Diff: https://reviews.apache.org/r/41928/diff/ > > > Testing > --- > > > Thanks, > > Rajat Khandelwal > >
Re: Review Request 41928: HIVE-12777: Add capability to restore session
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/41928/#review116920 --- service/src/java/org/apache/hive/service/cli/CLIService.java (line 207) <https://reviews.apache.org/r/41928/#comment178004> Can you rename method to createSessionWithHandle as suggested? service/src/java/org/apache/hive/service/cli/session/SessionManager.java (line 283) <https://reviews.apache.org/r/41928/#comment178006> Can we make it package private? service/src/java/org/apache/hive/service/cli/session/SessionManager.java (line 356) <https://reviews.apache.org/r/41928/#comment178005> Is this method required? Can CLIService directly call createSession ? - Amareshwari Sriramadasu On Jan. 28, 2016, 7:18 a.m., Rajat Khandelwal wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/41928/ > --- > > (Updated Jan. 28, 2016, 7:18 a.m.) > > > Review request for hive. > > > Bugs: HIVE-12777 > https://issues.apache.org/jira/browse/HIVE-12777 > > > Repository: hive-git > > > Description > --- > > Extensions using Hive session handles should be able to restore the hive > session from the handle. > > Apache Lens depends on a fork of hive and that fork has such a capability. > > Relevant commit: > https://github.com/InMobi/hive/commit/931fe9116161a18952c082c14223ad6745fefe00#diff-0acb35f7cab7492f522b0c40ce3ce1be > > > Diffs > - > > service/src/java/org/apache/hive/service/cli/CLIService.java > 0d5ae1e491bdcf98a12f5eef7b35f23c7b9dfc00 > service/src/java/org/apache/hive/service/cli/SessionHandle.java > 4e4f0211de09408168c8da669626fb49f3db364c > service/src/java/org/apache/hive/service/cli/session/HiveSessionImpl.java > 3c5700bd114f6b792df10df23e57dad9b9e689f6 > > service/src/java/org/apache/hive/service/cli/session/HiveSessionImplwithUGI.java > c56d97afc969ea57810e9b3222b20e086e847d67 > service/src/java/org/apache/hive/service/cli/session/SessionManager.java > de18f1d93502704e7a8fbb8d31d93bd770c4d232 > service/src/test/org/apache/hive/service/cli/CLIServiceRestoreTest.java > PRE-CREATION > > Diff: https://reviews.apache.org/r/41928/diff/ > > > Testing > --- > > > Thanks, > > Rajat Khandelwal > >
Re: Review Request 41928: HIVE-12777: Add capability to restore session
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/41928/#review116580 --- service/src/java/org/apache/hive/service/cli/CLIService.java (line 207) <https://reviews.apache.org/r/41928/#comment177606> Shall we remove the return from the method, which is not adding any? service/src/java/org/apache/hive/service/cli/session/SessionManager.java (line 311) <https://reviews.apache.org/r/41928/#comment177605> Why is the ipAddress variable changed? service/src/test/org/apache/hive/service/cli/CLIServiceRestoreTest.java (line 36) <https://reviews.apache.org/r/41928/#comment177604> Do a stop() after the test? - Amareshwari Sriramadasu On Jan. 21, 2016, 1:39 p.m., Rajat Khandelwal wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/41928/ > --- > > (Updated Jan. 21, 2016, 1:39 p.m.) > > > Review request for hive. > > > Bugs: HIVE-12777 > https://issues.apache.org/jira/browse/HIVE-12777 > > > Repository: hive-git > > > Description > --- > > Extensions using Hive session handles should be able to restore the hive > session from the handle. > > Apache Lens depends on a fork of hive and that fork has such a capability. > > Relevant commit: > https://github.com/InMobi/hive/commit/931fe9116161a18952c082c14223ad6745fefe00#diff-0acb35f7cab7492f522b0c40ce3ce1be > > > Diffs > - > > service/src/java/org/apache/hive/service/cli/CLIService.java > 0d5ae1e491bdcf98a12f5eef7b35f23c7b9dfc00 > service/src/java/org/apache/hive/service/cli/SessionHandle.java > 4e4f0211de09408168c8da669626fb49f3db364c > service/src/java/org/apache/hive/service/cli/session/HiveSessionImpl.java > 3c5700bd114f6b792df10df23e57dad9b9e689f6 > > service/src/java/org/apache/hive/service/cli/session/HiveSessionImplwithUGI.java > c56d97afc969ea57810e9b3222b20e086e847d67 > service/src/java/org/apache/hive/service/cli/session/SessionManager.java > de18f1d93502704e7a8fbb8d31d93bd770c4d232 > service/src/test/org/apache/hive/service/cli/CLIServiceRestoreTest.java > PRE-CREATION > > Diff: https://reviews.apache.org/r/41928/diff/ > > > Testing > --- > > > Thanks, > > Rajat Khandelwal > >
Re: Review Request 42134: More information to user on GetOperationStatus in Hive Server2 when query is still executing
---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/42134/#review113961
---
service/src/java/org/apache/hive/service/cli/OperationStatus.java (line 45)
<https://reviews.apache.org/r/42134/#comment174742>
Can you add test for validating task status values ?
- Amareshwari Sriramadasu
On Jan. 12, 2016, 8:45 a.m., Akshay Goyal wrote:
>
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/42134/
> ---
>
> (Updated Jan. 12, 2016, 8:45 a.m.)
>
>
> Review request for hive and Amareshwari Sriramadasu.
>
>
> Bugs: HIVE-4570
> https://issues.apache.org/jira/browse/HIVE-4570
>
>
> Repository: hive-git
>
>
> Description
> ---
>
> Driver maintains list of running and runnable tasks although that info is not
> exposed outside. It's kept locally in the driver's execute method. We can add
> Driver.getTaskStatuses() to return status on all tasks (both running and
> runnable). Similarly, start and completion times for operations.
>
> Proposed changes are :
>
> struct TGetOperationStatusResp {
> 1: required TStatus status
> 2: optional TOperationState operationState
>
> // If operationState is ERROR_STATE, then the following fields may be set
> // sqlState as defined in the ISO/IEF CLI specification
> 3: optional string sqlState
>
> // Internal error code
> 4: optional i32 errorCode
>
> // Error message
> 5: optional string errorMessage
>
> // List of statuses of sub tasks
> 6: optional string taskStatus
>
> // When was the operation started
> 7: optional i64 operationStarted
> // When was the operation completed
> 8: optional i64 operationCompleted
>
> }
>
>
> Diffs
> -
>
> ql/src/java/org/apache/hadoop/hive/ql/Driver.java 29e6315
> ql/src/java/org/apache/hadoop/hive/ql/TaskStatus.java PRE-CREATION
> ql/src/java/org/apache/hadoop/hive/ql/exec/Task.java 0eab63e
> ql/src/java/org/apache/hadoop/hive/ql/exec/mr/ExecDriver.java ab7fd93
> service/if/TCLIService.thrift baf583f
> service/src/java/org/apache/hive/service/cli/OperationStatus.java e45b828
>
> service/src/java/org/apache/hive/service/cli/operation/GetCatalogsOperation.java
> 8868ec1
>
> service/src/java/org/apache/hive/service/cli/operation/GetColumnsOperation.java
> 8ecdc2e
>
> service/src/java/org/apache/hive/service/cli/operation/GetFunctionsOperation.java
> 6df1e8a
>
> service/src/java/org/apache/hive/service/cli/operation/GetSchemasOperation.java
> e56686a
>
> service/src/java/org/apache/hive/service/cli/operation/GetTableTypesOperation.java
> a09b39a
>
> service/src/java/org/apache/hive/service/cli/operation/GetTablesOperation.java
> 740b851
>
> service/src/java/org/apache/hive/service/cli/operation/GetTypeInfoOperation.java
> 2a0fec2
>
> service/src/java/org/apache/hive/service/cli/operation/HiveCommandOperation.java
> 04dc6e3
> service/src/java/org/apache/hive/service/cli/operation/Operation.java
> d2b3f9c
> service/src/java/org/apache/hive/service/cli/operation/SQLOperation.java
> 2eaab4a
>
> service/src/java/org/apache/hive/service/cli/thrift/ThriftCLIServiceClient.java
> 1af4539
> service/src/test/org/apache/hive/service/cli/CLIServiceTest.java e78181a
>
> Diff: https://reviews.apache.org/r/42134/diff/
>
>
> Testing
> ---
>
>
> Thanks,
>
> Akshay Goyal
>
>
Re: Review Request 41928: HIVE-12777: Add capability to restore session
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/41928/#review113730 --- service/src/test/org/apache/hive/service/cli/CLIServiceTest.java (line 94) <https://reviews.apache.org/r/41928/#comment174548> Can you do some operations on restored session? - Amareshwari Sriramadasu On Jan. 11, 2016, 10:22 a.m., Rajat Khandelwal wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/41928/ > --- > > (Updated Jan. 11, 2016, 10:22 a.m.) > > > Review request for hive. > > > Bugs: HIVE-12777 > https://issues.apache.org/jira/browse/HIVE-12777 > > > Repository: hive-git > > > Description > --- > > Extensions using Hive session handles should be able to restore the hive > session from the handle. > > Apache Lens depends on a fork of hive and that fork has such a capability. > > Relevant commit: > https://github.com/InMobi/hive/commit/931fe9116161a18952c082c14223ad6745fefe00#diff-0acb35f7cab7492f522b0c40ce3ce1be > > > Diffs > - > > > itests/hive-unit/src/test/java/org/apache/hive/service/cli/TestEmbeddedThriftBinaryCLIService.java > de66d9efb1cace9d32174e3020920d5e4002dc85 > service/if/TCLIService.thrift baf583fb3ecd4a0ab248540ffb5db57100028968 > service/src/gen/thrift/gen-cpp/TCLIService.h > 2ea80c76e4e8618c5535f8270f20c70c8b35f765 > service/src/gen/thrift/gen-cpp/TCLIService.cpp > 7e2c028dc5e038e40d4dfb5e06a80789b4eabe3b > service/src/gen/thrift/gen-cpp/TCLIService_server.skeleton.cpp > 988bb4c11ddb717f585e0ba2fb4773ec5fff77e6 > service/src/gen/thrift/gen-cpp/TCLIService_types.h > b078c99db2f436e1ac32514b8f873c150952900b > service/src/gen/thrift/gen-cpp/TCLIService_types.cpp > b85237994f8019bd58aa1c8fdbef3aabce9940e2 > > service/src/gen/thrift/gen-javabean/org/apache/hive/service/cli/thrift/TCLIService.java > ded848f2b81bf698a819732a2e9ea4bc67fd2a26 > > service/src/gen/thrift/gen-javabean/org/apache/hive/service/cli/thrift/TExecuteStatementReq.java > 30cf243f541304a2ac05c775c912d542520ebe05 > > service/src/gen/thrift/gen-javabean/org/apache/hive/service/cli/thrift/TGetTablesReq.java > 4f2273214ab43b1116008183ff42a520769c6710 > > service/src/gen/thrift/gen-javabean/org/apache/hive/service/cli/thrift/TRestoreSessionReq.java > PRE-CREATION > > service/src/gen/thrift/gen-javabean/org/apache/hive/service/cli/thrift/TRestoreSessionResp.java > PRE-CREATION > service/src/gen/thrift/gen-php/TCLIService.php > eba62f12de0899280b9961dd82330bb149d40758 > service/src/gen/thrift/gen-py/TCLIService/TCLIService-remote > 56f5c5dc36d44627a2cd571c2bc1fcd1a79e62f2 > service/src/gen/thrift/gen-py/TCLIService/TCLIService.py > ad2d71d720376c76632d9e3a147f227b3ff67c17 > service/src/gen/thrift/gen-py/TCLIService/ttypes.py > ef5f5f568ffec92c093bdd0904c154f8ee67606e > service/src/gen/thrift/gen-rb/t_c_l_i_service.rb > 7d7f7a78a17cab50b18fa77805022bf4ca0197d9 > service/src/gen/thrift/gen-rb/t_c_l_i_service_types.rb > f004ec40afd29ab4ed452f65f306cb1af5ff5207 > service/src/java/org/apache/hive/service/cli/CLIService.java > adc98098437b2d0429dd2af40d8eaae323fbf3f0 > service/src/java/org/apache/hive/service/cli/CLIServiceClient.java > 3155c238ff688bfea16b0aaeea950599bb659b5b > service/src/java/org/apache/hive/service/cli/EmbeddedCLIServiceClient.java > 9cad5be198c063115a8e90c67b1c2fd910ca8bc6 > service/src/java/org/apache/hive/service/cli/ICLIService.java > c9cc1f4da56f1cd10f6348ea2b9e17e203b87664 > service/src/java/org/apache/hive/service/cli/SessionHandle.java > 52e0ad4834d8b44d5038bb5c1a851eb702cd38ca > service/src/java/org/apache/hive/service/cli/session/HiveSessionImpl.java > 50c912ee606bce73f447fba69ce0f57d8c7fc6c9 > service/src/java/org/apache/hive/service/cli/session/SessionManager.java > 637cdca3bcea9d320f397de032fd78d59ce6afef > > service/src/java/org/apache/hive/service/cli/thrift/RetryingThriftCLIServiceClient.java > 529eaa45d29d22e0e80ea2c08a27b84f2909eda3 > service/src/java/org/apache/hive/service/cli/thrift/ThriftCLIService.java > 8434965e87366e86d27685c5ca70ee904446bae9 > > service/src/java/org/apache/hive/service/cli/thrift/ThriftCLIServiceClient.java > 1af45398b895cd7616c5627d318422e14b81e734 > service/src/test/org/apache/hive/service/cli/CLIServiceTest.java > e78181a15993d99f1cab5a061c08bb21823d2171 > > Diff: https://reviews.apache.org/r/41928/diff/ > > > Testing > --- > > > Thanks, > > Rajat Khandelwal > >
Re: Review Request 41924: HIVE - 11485 : Session close should not close async SQL operations
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/41924/#review113729 --- ql/src/java/org/apache/hadoop/hive/ql/session/SessionState.java <https://reviews.apache.org/r/41924/#comment174547> If drop is not happening here, when will it happen? - Amareshwari Sriramadasu On Jan. 5, 2016, 10:23 a.m., Deepak Barr wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/41924/ > --- > > (Updated Jan. 5, 2016, 10:23 a.m.) > > > Review request for hive and Amareshwari Sriramadasu. > > > Bugs: HIVE-11485 > https://issues.apache.org/jira/browse/HIVE-11485 > > > Repository: hive-git > > > Description > --- > > Removed 3 actions on closing of a hive session : > 1. Closing of operations running on the session. > 2. Dropping of the session paths. > 3. FileSystem.closeAllForUGI() for the session. > > > Diffs > - > > ql/src/java/org/apache/hadoop/hive/ql/session/SessionState.java 5f15557 > service/src/java/org/apache/hive/service/cli/session/HiveSessionImpl.java > 50c912e > > service/src/java/org/apache/hive/service/cli/session/HiveSessionImplwithUGI.java > 441db7c > > Diff: https://reviews.apache.org/r/41924/diff/ > > > Testing > --- > > > Thanks, > > Deepak Barr > >
Re: Review Request 41928: HIVE-12777: Add capability to restore session
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/41928/#review113180 --- service/src/java/org/apache/hive/service/cli/session/SessionManager.java (line 297) <https://reviews.apache.org/r/41928/#comment173676> Dont we need changes here? service/src/test/org/apache/hive/service/cli/CLIServiceTest.java (line 91) <https://reviews.apache.org/r/41928/#comment173677> Restoring a closed session? I dont think this should be allowed. The test should wrt service restart and restore session. - Amareshwari Sriramadasu On Jan. 6, 2016, 6:59 a.m., Rajat Khandelwal wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/41928/ > --- > > (Updated Jan. 6, 2016, 6:59 a.m.) > > > Review request for hive. > > > Bugs: HIVE-12777 > https://issues.apache.org/jira/browse/HIVE-12777 > > > Repository: hive-git > > > Description > --- > > Extensions using Hive session handles should be able to restore the hive > session from the handle. > > Apache Lens depends on a fork of hive and that fork has such a capability. > > Relevant commit: > https://github.com/InMobi/hive/commit/931fe9116161a18952c082c14223ad6745fefe00#diff-0acb35f7cab7492f522b0c40ce3ce1be > > > Diffs > - > > service/if/TCLIService.thrift baf583fb3ecd4a0ab248540ffb5db57100028968 > service/src/java/org/apache/hive/service/cli/CLIService.java > adc98098437b2d0429dd2af40d8eaae323fbf3f0 > service/src/java/org/apache/hive/service/cli/CLIServiceClient.java > 3155c238ff688bfea16b0aaeea950599bb659b5b > service/src/java/org/apache/hive/service/cli/EmbeddedCLIServiceClient.java > 9cad5be198c063115a8e90c67b1c2fd910ca8bc6 > service/src/java/org/apache/hive/service/cli/ICLIService.java > c9cc1f4da56f1cd10f6348ea2b9e17e203b87664 > service/src/java/org/apache/hive/service/cli/SessionHandle.java > 52e0ad4834d8b44d5038bb5c1a851eb702cd38ca > service/src/java/org/apache/hive/service/cli/session/HiveSessionImpl.java > 50c912ee606bce73f447fba69ce0f57d8c7fc6c9 > service/src/java/org/apache/hive/service/cli/session/SessionManager.java > 637cdca3bcea9d320f397de032fd78d59ce6afef > > service/src/java/org/apache/hive/service/cli/thrift/RetryingThriftCLIServiceClient.java > 529eaa45d29d22e0e80ea2c08a27b84f2909eda3 > service/src/java/org/apache/hive/service/cli/thrift/ThriftCLIService.java > 8434965e87366e86d27685c5ca70ee904446bae9 > > service/src/java/org/apache/hive/service/cli/thrift/ThriftCLIServiceClient.java > 1af45398b895cd7616c5627d318422e14b81e734 > service/src/test/org/apache/hive/service/cli/CLIServiceTest.java > e78181a15993d99f1cab5a061c08bb21823d2171 > > Diff: https://reviews.apache.org/r/41928/diff/ > > > Testing > --- > > > Thanks, > > Rajat Khandelwal > >
Re: Review Request 41928: HIVE-12777: Add capability to restore session
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/41928/#review113021 --- Can you include code only with thrift changes and no generated code? Will be eacy for review. For patch on jira, include generated code as well. - Amareshwari Sriramadasu On Jan. 6, 2016, 6:39 a.m., Rajat Khandelwal wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/41928/ > --- > > (Updated Jan. 6, 2016, 6:39 a.m.) > > > Review request for hive. > > > Bugs: HIVE-12777 > https://issues.apache.org/jira/browse/HIVE-12777 > > > Repository: hive-git > > > Description > --- > > Extensions using Hive session handles should be able to restore the hive > session from the handle. > > Apache Lens depends on a fork of hive and that fork has such a capability. > > Relevant commit: > https://github.com/InMobi/hive/commit/931fe9116161a18952c082c14223ad6745fefe00#diff-0acb35f7cab7492f522b0c40ce3ce1be > > > Diffs > - > > service/if/TCLIService.thrift baf583fb3ecd4a0ab248540ffb5db57100028968 > service/src/gen/thrift/gen-cpp/TCLIService.h > 2ea80c76e4e8618c5535f8270f20c70c8b35f765 > service/src/gen/thrift/gen-cpp/TCLIService.cpp > 7e2c028dc5e038e40d4dfb5e06a80789b4eabe3b > service/src/gen/thrift/gen-cpp/TCLIService_server.skeleton.cpp > 988bb4c11ddb717f585e0ba2fb4773ec5fff77e6 > service/src/gen/thrift/gen-cpp/TCLIService_types.h > b078c99db2f436e1ac32514b8f873c150952900b > service/src/gen/thrift/gen-cpp/TCLIService_types.cpp > b85237994f8019bd58aa1c8fdbef3aabce9940e2 > > service/src/gen/thrift/gen-javabean/org/apache/hive/service/cli/thrift/TCLIService.java > ded848f2b81bf698a819732a2e9ea4bc67fd2a26 > > service/src/gen/thrift/gen-javabean/org/apache/hive/service/cli/thrift/TExecuteStatementReq.java > 30cf243f541304a2ac05c775c912d542520ebe05 > > service/src/gen/thrift/gen-javabean/org/apache/hive/service/cli/thrift/TGetTablesReq.java > 4f2273214ab43b1116008183ff42a520769c6710 > > service/src/gen/thrift/gen-javabean/org/apache/hive/service/cli/thrift/TRestoreSessionReq.java > PRE-CREATION > > service/src/gen/thrift/gen-javabean/org/apache/hive/service/cli/thrift/TRestoreSessionResp.java > PRE-CREATION > service/src/gen/thrift/gen-php/TCLIService.php > eba62f12de0899280b9961dd82330bb149d40758 > service/src/gen/thrift/gen-py/TCLIService/TCLIService-remote > 56f5c5dc36d44627a2cd571c2bc1fcd1a79e62f2 > service/src/gen/thrift/gen-py/TCLIService/TCLIService.py > ad2d71d720376c76632d9e3a147f227b3ff67c17 > service/src/gen/thrift/gen-py/TCLIService/ttypes.py > ef5f5f568ffec92c093bdd0904c154f8ee67606e > service/src/gen/thrift/gen-rb/t_c_l_i_service.rb > 7d7f7a78a17cab50b18fa77805022bf4ca0197d9 > service/src/gen/thrift/gen-rb/t_c_l_i_service_types.rb > f004ec40afd29ab4ed452f65f306cb1af5ff5207 > service/src/java/org/apache/hive/service/cli/CLIService.java > adc98098437b2d0429dd2af40d8eaae323fbf3f0 > service/src/java/org/apache/hive/service/cli/CLIServiceClient.java > 3155c238ff688bfea16b0aaeea950599bb659b5b > service/src/java/org/apache/hive/service/cli/EmbeddedCLIServiceClient.java > 9cad5be198c063115a8e90c67b1c2fd910ca8bc6 > service/src/java/org/apache/hive/service/cli/ICLIService.java > c9cc1f4da56f1cd10f6348ea2b9e17e203b87664 > service/src/java/org/apache/hive/service/cli/SessionHandle.java > 52e0ad4834d8b44d5038bb5c1a851eb702cd38ca > service/src/java/org/apache/hive/service/cli/session/HiveSessionImpl.java > 50c912ee606bce73f447fba69ce0f57d8c7fc6c9 > service/src/java/org/apache/hive/service/cli/session/SessionManager.java > 637cdca3bcea9d320f397de032fd78d59ce6afef > > service/src/java/org/apache/hive/service/cli/thrift/RetryingThriftCLIServiceClient.java > 529eaa45d29d22e0e80ea2c08a27b84f2909eda3 > service/src/java/org/apache/hive/service/cli/thrift/ThriftCLIService.java > 8434965e87366e86d27685c5ca70ee904446bae9 > > service/src/java/org/apache/hive/service/cli/thrift/ThriftCLIServiceClient.java > 1af45398b895cd7616c5627d318422e14b81e734 > service/src/test/org/apache/hive/service/cli/CLIServiceTest.java > e78181a15993d99f1cab5a061c08bb21823d2171 > > Diff: https://reviews.apache.org/r/41928/diff/ > > > Testing > --- > > > Thanks, > > Rajat Khandelwal > >
Re: Review Request 41928: HIVE-12777: Add capability to restore session
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/41928/#review113017 --- service/src/java/org/apache/hive/service/cli/CLIService.java (line 48) <https://reviews.apache.org/r/41928/#comment173513> Avoid of the move of import statements service/src/java/org/apache/hive/service/cli/CLIService.java (line 206) <https://reviews.apache.org/r/41928/#comment173514> Comment does not make sense. Please add proper javadoc Please add unit tests - Amareshwari Sriramadasu On Jan. 5, 2016, 3:12 p.m., Rajat Khandelwal wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/41928/ > --- > > (Updated Jan. 5, 2016, 3:12 p.m.) > > > Review request for hive. > > > Bugs: HIVE-12777 > https://issues.apache.org/jira/browse/HIVE-12777 > > > Repository: hive-git > > > Description > --- > > Extensions using Hive session handles should be able to restore the hive > session from the handle. > > Apache Lens depends on a fork of hive and that fork has such a capability. > > Relevant commit: > https://github.com/InMobi/hive/commit/931fe9116161a18952c082c14223ad6745fefe00#diff-0acb35f7cab7492f522b0c40ce3ce1be > > > Diffs > - > > service/src/java/org/apache/hive/service/cli/CLIService.java > adc98098437b2d0429dd2af40d8eaae323fbf3f0 > service/src/java/org/apache/hive/service/cli/SessionHandle.java > 52e0ad4834d8b44d5038bb5c1a851eb702cd38ca > service/src/java/org/apache/hive/service/cli/session/HiveSessionImpl.java > 50c912ee606bce73f447fba69ce0f57d8c7fc6c9 > service/src/java/org/apache/hive/service/cli/session/SessionManager.java > 637cdca3bcea9d320f397de032fd78d59ce6afef > > Diff: https://reviews.apache.org/r/41928/diff/ > > > Testing > --- > > > Thanks, > > Rajat Khandelwal > >
Re: Review Request 41924: HIVE - 11485 : Session close should not close async SQL operations
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/41924/#review113013 --- Please include a unit test. - Amareshwari Sriramadasu On Jan. 5, 2016, 10:23 a.m., Deepak Barr wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/41924/ > --- > > (Updated Jan. 5, 2016, 10:23 a.m.) > > > Review request for hive and Amareshwari Sriramadasu. > > > Bugs: HIVE-11485 > https://issues.apache.org/jira/browse/HIVE-11485 > > > Repository: hive-git > > > Description > --- > > Removed 3 actions on closing of a hive session : > 1. Closing of operations running on the session. > 2. Dropping of the session paths. > 3. FileSystem.closeAllForUGI() for the session. > > > Diffs > - > > ql/src/java/org/apache/hadoop/hive/ql/session/SessionState.java 5f15557 > service/src/java/org/apache/hive/service/cli/session/HiveSessionImpl.java > 50c912e > > service/src/java/org/apache/hive/service/cli/session/HiveSessionImplwithUGI.java > 441db7c > > Diff: https://reviews.apache.org/r/41924/diff/ > > > Testing > --- > > > Thanks, > > Deepak Barr > >
Re: Review Request 41931: HIVE-12776: Add code for parsing any stand-alone HQL expression
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/41931/#review112988 --- Ship it! ql/src/test/org/apache/hadoop/hive/ql/parse/TestParseDriver.java (line 101) <https://reviews.apache.org/r/41931/#comment173467> Please avoid System.out. - Amareshwari Sriramadasu On Jan. 5, 2016, 3:10 p.m., Rajat Khandelwal wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/41931/ > --- > > (Updated Jan. 5, 2016, 3:10 p.m.) > > > Review request for hive. > > > Bugs: HIVE-12776 > https://issues.apache.org/jira/browse/HIVE-12776 > > > Repository: hive-git > > > Description > --- > > Extensions that use Hive QL as their standard language, will benefit from > this. > > Apache Lens uses HQL as its language of choice. To support that, it depends > on a fork of Hive, which has such code. I'm planning to port that to Apache > Hive. > > Relevant commit: > https://github.com/InMobi/hive/commit/7caea9ed1d269c1cd1d1326cb39c1db7e0bf2bba#diff-fb3acd67881ceb02e83c2e42cf70beef > > > Diffs > - > > ql/src/java/org/apache/hadoop/hive/ql/parse/ParseDriver.java > 90a9a4e6bbefdb4679bc76e21661ba4658a1df3c > ql/src/test/org/apache/hadoop/hive/ql/parse/TestParseDriver.java > PRE-CREATION > > Diff: https://reviews.apache.org/r/41931/diff/ > > > Testing > --- > > > Thanks, > > Rajat Khandelwal > >
Re: Review Request 37925: HIVE-11487: Add getNumPartitionsByFilter api in metastore api
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/37925/#review110941 --- Ship it! Sergey Shelukhin, Can you also have a look at the changes ? - Amareshwari Sriramadasu On Dec. 16, 2015, 12:18 p.m., Akshay Goyal wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/37925/ > --- > > (Updated Dec. 16, 2015, 12:18 p.m.) > > > Review request for hive, Amareshwari Sriramadasu and Sergey Shelukhin. > > > Bugs: HIVE-11487 > https://issues.apache.org/jira/browse/HIVE-11487 > > > Repository: hive-git > > > Description > --- > > Adding api for getting number of partitions for a filter. In order to be more > optimal when only interested in the number. getAllPartitions will construct > all the partition object which can be time consuming and not required. > > > Diffs > - > > metastore/if/hive_metastore.thrift bb754f1 > metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java > 0940fd7 > > metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStoreClient.java > c5e7a5f > metastore/src/java/org/apache/hadoop/hive/metastore/IMetaStoreClient.java > aa96f77 > metastore/src/java/org/apache/hadoop/hive/metastore/MetaStoreDirectSql.java > 36b316a > metastore/src/java/org/apache/hadoop/hive/metastore/ObjectStore.java > abfe2b8 > metastore/src/java/org/apache/hadoop/hive/metastore/RawStore.java e118a3b > metastore/src/java/org/apache/hadoop/hive/metastore/hbase/HBaseStore.java > b9509ab > > metastore/src/test/org/apache/hadoop/hive/metastore/DummyRawStoreControlledCommit.java > c1156b3 > > metastore/src/test/org/apache/hadoop/hive/metastore/DummyRawStoreForJdoConnection.java > bf20e99 > metastore/src/test/org/apache/hadoop/hive/metastore/TestObjectStore.java > 9089d1c > ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java c682df2 > > Diff: https://reviews.apache.org/r/37925/diff/ > > > Testing > --- > > > Thanks, > > Akshay Goyal > >
Re: Review Request 37925: HIVE-11487: Add getNumPartitionsByFilter api in metastore api
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/37925/#review103209 --- Overall changes look fine to me. Sergey, Can you also look at the changes updated? metastore/src/test/org/apache/hadoop/hive/metastore/TestObjectStore.java (line 178) <https://reviews.apache.org/r/37925/#comment161190> Can you one more test passing filter ? - Amareshwari Sriramadasu On Sept. 14, 2015, 8:05 a.m., Akshay Goyal wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/37925/ > --- > > (Updated Sept. 14, 2015, 8:05 a.m.) > > > Review request for hive and Amareshwari Sriramadasu. > > > Bugs: HIVE-11487 > https://issues.apache.org/jira/browse/HIVE-11487 > > > Repository: hive-git > > > Description > --- > > Adding api for getting number of partitions for a filter. In order to be more > optimal when only interested in the number. getAllPartitions will construct > all the partition object which can be time consuming and not required. > > > Diffs > - > > metastore/if/hive_metastore.thrift d651195e0477ee607657fec13b68ff3184ddbebf > metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java > 1840e76cc567e95e1942d912b8ab0db516d63a3b > > metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStoreClient.java > 482f27851c0732604cdb04e9a0b981757643210b > metastore/src/java/org/apache/hadoop/hive/metastore/IMetaStoreClient.java > 5fde6d34bff7fd091b3ca36472d929269660c9b9 > metastore/src/java/org/apache/hadoop/hive/metastore/MetaStoreDirectSql.java > 1f89b7c83299d8345722ae21267151a11676c332 > metastore/src/java/org/apache/hadoop/hive/metastore/ObjectStore.java > 4d6bfccbd1164fd22ed1c4592c3f227d8182936f > metastore/src/java/org/apache/hadoop/hive/metastore/RawStore.java > 7c85eea21375f349f88e918329030364acca2771 > > metastore/src/test/org/apache/hadoop/hive/metastore/DummyRawStoreControlledCommit.java > f184c568b5eb5b16f239c6128f1ad20bd83957cb > > metastore/src/test/org/apache/hadoop/hive/metastore/DummyRawStoreForJdoConnection.java > 83fb4bb900a51101d8c411d199357c8ddd29486e > metastore/src/test/org/apache/hadoop/hive/metastore/TestObjectStore.java > a4f9f6cee2c7aeb1235a814fdb06e75d37809424 > ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java > c78e8f474f928c77a45ff29134054d43184f0324 > > Diff: https://reviews.apache.org/r/37925/diff/ > > > Testing > --- > > > Thanks, > > Akshay Goyal > >
Re: Review Request 37925: HIVE-11487: Add getNumPartitionsByFilter api in metastore api
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/37925/#review97617 --- metastore/if/hive_metastore.thrift (line 990) <https://reviews.apache.org/r/37925/#comment153633> Is max_parts required in this api? metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStoreClient.java (line 828) <https://reviews.apache.org/r/37925/#comment153635> Can you avoid doing indentation changes in code that is not touched ? - Amareshwari Sriramadasu On Sept. 2, 2015, 11:14 a.m., Akshay Goyal wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/37925/ > --- > > (Updated Sept. 2, 2015, 11:14 a.m.) > > > Review request for hive and Amareshwari Sriramadasu. > > > Bugs: HIVE-11487 > https://issues.apache.org/jira/browse/HIVE-11487 > > > Repository: hive-git > > > Description > --- > > Adding api for getting number of partitions for a filter. In order to be more > optimal when only interested in the number. getAllPartitions will construct > all the partition object which can be time consuming and not required. > > > Diffs > - > > metastore/if/hive_metastore.thrift d651195e0477ee607657fec13b68ff3184ddbebf > metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java > 1840e76cc567e95e1942d912b8ab0db516d63a3b > > metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStoreClient.java > 482f27851c0732604cdb04e9a0b981757643210b > metastore/src/java/org/apache/hadoop/hive/metastore/IMetaStoreClient.java > 5fde6d34bff7fd091b3ca36472d929269660c9b9 > metastore/src/java/org/apache/hadoop/hive/metastore/MetaStoreDirectSql.java > 522fcc275f42a3b75af9f4bee9bf1ab3cd70364b > metastore/src/java/org/apache/hadoop/hive/metastore/ObjectStore.java > d165fc82386eafa5200068f5c56a9e98788627aa > metastore/src/java/org/apache/hadoop/hive/metastore/RawStore.java > 7c85eea21375f349f88e918329030364acca2771 > > metastore/src/test/org/apache/hadoop/hive/metastore/DummyRawStoreControlledCommit.java > f184c568b5eb5b16f239c6128f1ad20bd83957cb > > metastore/src/test/org/apache/hadoop/hive/metastore/DummyRawStoreForJdoConnection.java > 83fb4bb900a51101d8c411d199357c8ddd29486e > ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java > 396c07044ec69d8b74e38fb156333b6eb35b9f4d > > Diff: https://reviews.apache.org/r/37925/diff/ > > > Testing > --- > > > Thanks, > > Akshay Goyal > >
Re: Review Request 37925: HIVE-11487: Add getNumPartitionsByFilter api in metastore api
> On Sept. 2, 2015, 7:36 p.m., Sergey Shelukhin wrote: > > metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStoreClient.java, > > line 1323 > > <https://reviews.apache.org/r/37925/diff/3/?file=1062211#file1062211line1323> > > > > should this be uncommented? :) +1 > On Sept. 2, 2015, 7:36 p.m., Sergey Shelukhin wrote: > > metastore/src/java/org/apache/hadoop/hive/metastore/MetaStoreDirectSql.java, > > line 854 > > <https://reviews.apache.org/r/37925/diff/3/?file=1062213#file1062213line854> > > > > btw, why not use COUNT in the query in the first place I agree with Sergey, we should try count() instead of getting partition ids and doing a size on that. - Amareshwari --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/37925/#review97519 --- On Sept. 2, 2015, 11:14 a.m., Akshay Goyal wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/37925/ > ------- > > (Updated Sept. 2, 2015, 11:14 a.m.) > > > Review request for hive and Amareshwari Sriramadasu. > > > Bugs: HIVE-11487 > https://issues.apache.org/jira/browse/HIVE-11487 > > > Repository: hive-git > > > Description > --- > > Adding api for getting number of partitions for a filter. In order to be more > optimal when only interested in the number. getAllPartitions will construct > all the partition object which can be time consuming and not required. > > > Diffs > - > > metastore/if/hive_metastore.thrift d651195e0477ee607657fec13b68ff3184ddbebf > metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java > 1840e76cc567e95e1942d912b8ab0db516d63a3b > > metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStoreClient.java > 482f27851c0732604cdb04e9a0b981757643210b > metastore/src/java/org/apache/hadoop/hive/metastore/IMetaStoreClient.java > 5fde6d34bff7fd091b3ca36472d929269660c9b9 > metastore/src/java/org/apache/hadoop/hive/metastore/MetaStoreDirectSql.java > 522fcc275f42a3b75af9f4bee9bf1ab3cd70364b > metastore/src/java/org/apache/hadoop/hive/metastore/ObjectStore.java > d165fc82386eafa5200068f5c56a9e98788627aa > metastore/src/java/org/apache/hadoop/hive/metastore/RawStore.java > 7c85eea21375f349f88e918329030364acca2771 > > metastore/src/test/org/apache/hadoop/hive/metastore/DummyRawStoreControlledCommit.java > f184c568b5eb5b16f239c6128f1ad20bd83957cb > > metastore/src/test/org/apache/hadoop/hive/metastore/DummyRawStoreForJdoConnection.java > 83fb4bb900a51101d8c411d199357c8ddd29486e > ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java > 396c07044ec69d8b74e38fb156333b6eb35b9f4d > > Diff: https://reviews.apache.org/r/37925/diff/ > > > Testing > --- > > > Thanks, > > Akshay Goyal > >
Re: Review Request 37935: HIVE-11482: Add retrying thrift client for HiveServer2
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/37935/#review97616 --- Ship it! Ship It! - Amareshwari Sriramadasu On Sept. 2, 2015, 9:17 a.m., Akshay Goyal wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/37935/ > --- > > (Updated Sept. 2, 2015, 9:17 a.m.) > > > Review request for hive and Amareshwari Sriramadasu. > > > Bugs: HIVE-11482 > https://issues.apache.org/jira/browse/HIVE-11482 > > > Repository: hive-git > > > Description > --- > > Similar to RetryingMetaStoreClient, this improvement request is to add a > retrying thrift client for HiveServer2 to do retries in case of thrift > exceptions. > > > Diffs > - > > common/src/java/org/apache/hadoop/hive/conf/HiveConf.java > 8a00079631b657acd79877202e9819ff282dfdd6 > > service/src/java/org/apache/hive/service/cli/thrift/RetryingThriftCLIServiceClient.java > PRE-CREATION > > service/src/test/org/apache/hive/service/cli/TestRetryingThriftCLIServiceClient.java > PRE-CREATION > > Diff: https://reviews.apache.org/r/37935/diff/ > > > Testing > --- > > Added test case TestRetryingThriftCLIServiceClient to check whether retries > are happening in case of exceptions and also whether connections are going > through if everything is fine. > > > Thanks, > > Akshay Goyal > >
Re: Review Request 37925: HIVE-11487: Add getNumPartitionsByFilter api in metastore api
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/37925/#review97407 --- Akshay, it is hard to review with generated code put here. Can you update review request without any generated code? - Amareshwari Sriramadasu On Aug. 30, 2015, 8:41 p.m., Akshay Goyal wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/37925/ > --- > > (Updated Aug. 30, 2015, 8:41 p.m.) > > > Review request for hive and Amareshwari Sriramadasu. > > > Bugs: HIVE-11487 > https://issues.apache.org/jira/browse/HIVE-11487 > > > Repository: hive-git > > > Description > --- > > Adding api for getting number of partitions for a filter. In order to be more > optimal when only interested in the number. getAllPartitions will construct > all the partition object which can be time consuming and not required. > > > Diffs > - > > metastore/if/hive_metastore.thrift d651195e0477ee607657fec13b68ff3184ddbebf > metastore/src/gen/thrift/gen-cpp/ThriftHiveMetastore.h > 520c6e3be0f0e329da9136738db4c591eac671c1 > metastore/src/gen/thrift/gen-cpp/ThriftHiveMetastore.cpp > 0354fe18111d5e4a360f90a45e67b7a11b7bd0e4 > metastore/src/gen/thrift/gen-cpp/ThriftHiveMetastore_server.skeleton.cpp > 0afcf8ce7f250f99d0ebd6637db41faf10ddd95e > > metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/AbortTxnRequest.java > 37e5bf1a29006efcfc5620210f2613ecff1268bd > > metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/AddDynamicPartitions.java > 9c78c49d383fa00980b1354f81d9f46965919467 > > metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/AddPartitionsRequest.java > fcfaaf3a5cc630f499d70db05b4b3f8877d1091c > > metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/AddPartitionsResult.java > 9022019e901177581e96a9f0ca8d4ace6b84468f > > metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/AggrStats.java > 917cec0aee1afdacb3155d7beef05686689ab70b > > metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/AlreadyExistsException.java > d7a317b5e9f7d0d665c463d9e22c2700769abb8c > > metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/BinaryColumnStatsData.java > 00b312d85466d55aedb78bc5015f8d8b88d3447e > > metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/BooleanColumnStatsData.java > a0f3ab82a92b2fab66cc464411700b9f92675334 > > metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/CheckLockRequest.java > 82e30316d9545e9f44bfce93e453e078368f791e > > metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/ColumnStatistics.java > 510dace2a9b886aad53a4a45ce197d1f4a35a695 > > metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/ColumnStatisticsDesc.java > cf967b6e25ca66a079980b34087b4e89635d0210 > > metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/ColumnStatisticsObj.java > 3c2b1238aca7bb7e801267e772701e2d3c5b3044 > > metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/CommitTxnRequest.java > 5e8cd04f4923dd86548771725c047bf1bc29e2f9 > > metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/CompactionRequest.java > e9088e061f2e2e2fcf8bd706b12b0477cb4b2442 > > metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/ConfigValSecurityException.java > 000670aaf1527df7227e334f63ed7f162f4fe9ec > > metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/CurrentNotificationEventId.java > d55d87444e4ae4075ae892cd609d69baf35c011c > > metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/Database.java > 56b72819f6a3c1747a15c4400039c32db38d3433 > > metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/Date.java > 0b406d682fb5f877b021d39d8f263da348924912 > > metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/DateColumnStatsData.java > 7a3d4ed18e095bb8a49bc9d620693f3d53ce2e60 > > metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/Decimal.java > 9215ce95deef33361d5b852121d938369b239d5c > > metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/DecimalColumnStatsData.java > e
Re: Review Request 37935: HIVE-11482: Add retrying thrift client for HiveServer2
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/37935/#review97272 --- common/src/java/org/apache/hadoop/hive/conf/HiveConf.java (line 2026) <https://reviews.apache.org/r/37935/#comment153106> Change description as "Number of retries upon failure of Thrift HiveServer2 calls" common/src/java/org/apache/hadoop/hive/conf/HiveConf.java (line 2028) <https://reviews.apache.org/r/37935/#comment153105> Change description as " Number of retries while opening a connection to HiveServe2" common/src/java/org/apache/hadoop/hive/conf/HiveConf.java (line 2029) <https://reviews.apache.org/r/37935/#comment153107> Specify a TimeValidator similar to param METASTORE_CLIENT_CONNECT_RETRY_DELAY. Also, change description to "Number of seconds for the HiveServer2 thrift client to wait between consecutive connection attempts; also specifies the time to wait between retrying thrift calls upon failures" - Amareshwari Sriramadasu On Aug. 30, 2015, 8:48 p.m., Akshay Goyal wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/37935/ > --- > > (Updated Aug. 30, 2015, 8:48 p.m.) > > > Review request for hive and Amareshwari Sriramadasu. > > > Bugs: HIVE-11482 > https://issues.apache.org/jira/browse/HIVE-11482 > > > Repository: hive-git > > > Description > --- > > Similar to RetryingMetaStoreClient, this improvement request is to add a > retrying thrift client for HiveServer2 to do retries in case of thrift > exceptions. > > > Diffs > - > > common/src/java/org/apache/hadoop/hive/conf/HiveConf.java > 8a00079631b657acd79877202e9819ff282dfdd6 > > service/src/java/org/apache/hive/service/cli/thrift/RetryingThriftCLIServiceClient.java > PRE-CREATION > > service/src/test/org/apache/hive/service/cli/TestRetryingThriftCLIServiceClient.java > PRE-CREATION > > Diff: https://reviews.apache.org/r/37935/diff/ > > > Testing > --- > > Added test case TestRetryingThriftCLIServiceClient to check whether retries > are happening in case of exceptions and also whether connections are going > through if everything is fine. > > > Thanks, > > Akshay Goyal > >
Re: Review Request 33890: HIVE-10435: Make HiveSession implementation pluggable through configuration
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/33890/#review94910 --- Ship it! Ship It! - Amareshwari Sriramadasu On Aug. 11, 2015, 7:18 a.m., Akshay Goyal wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/33890/ > --- > > (Updated Aug. 11, 2015, 7:18 a.m.) > > > Review request for hive and Amareshwari Sriramadasu. > > > Bugs: HIVE-10435 > https://issues.apache.org/jira/browse/HIVE-10435 > > > Repository: hive-git > > > Description > --- > > SessionManager in CLIService creates and keeps track of HiveSession. > Right now, it creates HiveSessionImpl which is one implementation of > HiveSession. This improvement request is to make it pluggable through a > configuration so that other implementations can be passed. > > > Diffs > - > > common/src/java/org/apache/hadoop/hive/conf/HiveConf.java fe4eb9f > service/src/java/org/apache/hive/service/cli/session/SessionManager.java > c46bbce > > service/src/test/org/apache/hive/service/cli/session/TestPluggableHiveSessionImpl.java > PRE-CREATION > > Diff: https://reviews.apache.org/r/33890/diff/ > > > Testing > --- > > Newly added unit test passed. > > > Thanks, > > Akshay Goyal > >
[jira] [Created] (HIVE-11487) Add getNumPartitionsByFilter api in metastore api
Amareshwari Sriramadasu created HIVE-11487: -- Summary: Add getNumPartitionsByFilter api in metastore api Key: HIVE-11487 URL: https://issues.apache.org/jira/browse/HIVE-11487 Project: Hive Issue Type: Improvement Components: Metastore Reporter: Amareshwari Sriramadasu Adding api for getting number of partitions for a filter will be more optimal when we are only interested in the number. getAllPartitions will construct all the partition object which can be time consuming and not required. Here is a commit we pushed in a forked repo in our organization - https://github.com/inmobi/hive/commit/68b3534d3e6c4d978132043cec668798ed53e444. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HIVE-11486) Hive should log exceptions for better debuggability with full trace
Amareshwari Sriramadasu created HIVE-11486: -- Summary: Hive should log exceptions for better debuggability with full trace Key: HIVE-11486 URL: https://issues.apache.org/jira/browse/HIVE-11486 Project: Hive Issue Type: Improvement Components: Diagnosability Reporter: Amareshwari Sriramadasu For ex : https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java#L2638 https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/exec/mr/MapRedTask.java#L315 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HIVE-11485) Session close should not close async SQL operations
Amareshwari Sriramadasu created HIVE-11485: -- Summary: Session close should not close async SQL operations Key: HIVE-11485 URL: https://issues.apache.org/jira/browse/HIVE-11485 Project: Hive Issue Type: Improvement Components: HiveServer2 Reporter: Amareshwari Sriramadasu Right now, session close on HiveServer closes all operations. But, queries running are actually available across sessions and they are not tied to a session (expect the launch - which requires configuration and resources). And it allows getting the status of the query across sessions. But session close of the session ( on which operation is launched) closes all the operations as well. So, we should avoid closing all operations upon closing a session. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HIVE-11484) Fix ObjectInspector for Char and VarChar
Amareshwari Sriramadasu created HIVE-11484: -- Summary: Fix ObjectInspector for Char and VarChar Key: HIVE-11484 URL: https://issues.apache.org/jira/browse/HIVE-11484 Project: Hive Issue Type: Bug Components: Serializers/Deserializers Reporter: Amareshwari Sriramadasu The creation of HiveChar and Varchar is not happening through ObjectInspector. Here is fix we pushed internally : https://github.com/InMobi/hive/commit/fe95c7850e7130448209141155f28b25d3504216 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HIVE-11483) Add encoding and decoding for query string config
Amareshwari Sriramadasu created HIVE-11483: -- Summary: Add encoding and decoding for query string config Key: HIVE-11483 URL: https://issues.apache.org/jira/browse/HIVE-11483 Project: Hive Issue Type: Improvement Components: Query Processor Reporter: Amareshwari Sriramadasu We have seen some queries in production where some of the literals passed in the query have control characters, which result in exception when query string is set in the job xml. Proposing a solution to encode the query string in configuration and provide getters decoded string. Here is a commit in a forked repo : https://github.com/InMobi/hive/commit/2faf5761191fa3103a0d779fde584d494ed75bf5 Suggestions are welcome on the solution. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HIVE-11482) Add retrying thrift client for HiveServer2
Amareshwari Sriramadasu created HIVE-11482: -- Summary: Add retrying thrift client for HiveServer2 Key: HIVE-11482 URL: https://issues.apache.org/jira/browse/HIVE-11482 Project: Hive Issue Type: Improvement Components: HiveServer2 Reporter: Amareshwari Sriramadasu Similar to https://github.com/apache/hive/blob/master/metastore/src/java/org/apache/hadoop/hive/metastore/RetryingMetaStoreClient.java, this improvement request is to add a retrying thrift client for HiveServer2 to do retries upon thrift exceptions. Here are few commits done on a forked branch that can be picked - https://github.com/InMobi/hive/commit/7fb957fb9c2b6000d37c53294e256460010cb6b7 https://github.com/InMobi/hive/commit/11e4b330f051c3f58927a276d562446761c9cd6d https://github.com/InMobi/hive/commit/241386fd870373a9253dca0bcbdd4ea7e665406c -- This message was sent by Atlassian JIRA (v6.3.4#6332)
Re: Review Request 36939: HIVE-11376: CombineHiveInputFormat is falling back to HiveInputFormat in case codecs are found for one of the input files
---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/36939/#review93869
---
Ship it!
Ship It!
- Amareshwari Sriramadasu
On July 30, 2015, 1:19 p.m., Rajat Khandelwal wrote:
>
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/36939/
> ---
>
> (Updated July 30, 2015, 1:19 p.m.)
>
>
> Review request for hive.
>
>
> Bugs: HIVE-11376
> https://issues.apache.org/jira/browse/HIVE-11376
>
>
> Repository: hive-git
>
>
> Description
> ---
>
> https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/io/CombineHiveInputFormat.java#L379
>
> This is the exact code snippet:
>
> {noformat}
>
> / Since there is no easy way of knowing whether MAPREDUCE-1597 is present in
> the tree or not,
> // we use a configuration variable for the same
> if (this.mrwork != null && !this.mrwork.getHadoopSupportsSplittable()) {
> // The following code should be removed, once
> // https://issues.apache.org/jira/browse/MAPREDUCE-1597 is fixed.
> // Hadoop does not handle non-splittable files correctly for
> CombineFileInputFormat,
> // so don't use CombineFileInputFormat for non-splittable files
>
> //ie, dont't combine if inputformat is a TextInputFormat and has
> compression turned on
>
> {noformat}
>
>
> Diffs
> -
>
> common/src/java/org/apache/hadoop/hive/conf/HiveConf.java
> 33b67dd7b0fde41f81f8d86ea8c83d29c631e3d7
> ql/src/java/org/apache/hadoop/hive/ql/io/CombineHiveInputFormat.java
> 1de7e4073f5eea4c7be8423a7ebe6a89cb51d9f1
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/GenMapRedUtils.java
> 693d8c7e9f956999b2da33593d780e37ddf2b3b8
> ql/src/java/org/apache/hadoop/hive/ql/plan/MapWork.java
> 3217df27bb5731a1dcd5db1ae17c5bdff2e3fbfc
> ql/src/java/org/apache/hadoop/hive/ql/plan/PlanUtils.java
> 76926e79729d9ea4823de0ffc9b1e5bac6364842
>
> Diff: https://reviews.apache.org/r/36939/diff/
>
>
> Testing
> ---
>
>
> Thanks,
>
> Rajat Khandelwal
>
>
Re: Review Request 33890: HIVE-10435: Make HiveSession implementation pluggable through configuration
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/33890/#review82789 --- common/src/java/org/apache/hadoop/hive/conf/HiveConf.java <https://reviews.apache.org/r/33890/#comment133607> Can you add the new properties in ./build/dist/conf/hive-default.xml.template with description ? service/src/java/org/apache/hive/service/cli/session/SessionManager.java <https://reviews.apache.org/r/33890/#comment133603> Can we pass all the params passed for HiveSessionImplwithUGI here as well ? service/src/java/org/apache/hive/service/cli/session/SessionManager.java <https://reviews.apache.org/r/33890/#comment133604> Same as above, Can we all params passed for HiveSessionImpl here as well? It is fine if implementations want to ignore some, but passing less will be a gap. service/src/test/org/apache/hive/service/cli/session/TestPluggableHiveSessionImpl.java <https://reviews.apache.org/r/33890/#comment133606> Can you get the session object from service and assert it is an instance of TestHiveSessionImpl ? service/src/test/org/apache/hive/service/cli/session/TestPluggableHiveSessionImpl.java <https://reviews.apache.org/r/33890/#comment133605> I dont see the constructor added with params expected, are you sure the test is passing fine? - Amareshwari Sriramadasu On May 6, 2015, 12:29 p.m., Akshay Goyal wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/33890/ > --- > > (Updated May 6, 2015, 12:29 p.m.) > > > Review request for hive and Amareshwari Sriramadasu. > > > Bugs: HIVE-10435 > https://issues.apache.org/jira/browse/HIVE-10435 > > > Repository: hive-git > > > Description > --- > > SessionManager in CLIService creates and keeps track of HiveSession. > Right now, it creates HiveSessionImpl which is one implementation of > HiveSession. This improvement request is to make it pluggable through a > configuration so that other implementations can be passed. > > > Diffs > - > > common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 69fda45 > service/src/java/org/apache/hive/service/cli/session/SessionManager.java > 36a30b1 > > service/src/test/org/apache/hive/service/cli/session/TestPluggableHiveSessionImpl.java > PRE-CREATION > > Diff: https://reviews.apache.org/r/33890/diff/ > > > Testing > --- > > Newly added unit test passed. > > > Thanks, > > Akshay Goyal > >
Re: Review Request 33211: [HIVE-10288] : permanent UDFs are not working after restarting the server
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/33211/#review81667 --- trunk/ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java <https://reviews.apache.org/r/33211/#comment132094> I did not understand how the changes are related to restart? We have not faced in our setup. - Amareshwari Sriramadasu On April 15, 2015, 6:04 a.m., Chinna Lalam wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/33211/ > --- > > (Updated April 15, 2015, 6:04 a.m.) > > > Review request for hive, Jason Dere and Navis Ryu. > > > Repository: hive > > > Description > --- > > While reloading the permanent functions we need to add udf type > (genericUDF/genericUDTF/genericUDAFResolver). Currently this information is > not there, so it is throwing NullpointerException. > We have download the related jars when using this functions. > > > Diffs > - > > trunk/ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java 1673646 > > Diff: https://reviews.apache.org/r/33211/diff/ > > > Testing > --- > > Related tests are pass. > > > Thanks, > > Chinna Lalam > >
Re: Review Request 33259: [HIVE-10277] : Unable to process Comment line '--'
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/33259/#review81666 --- Ship it! Ship It! - Amareshwari Sriramadasu On April 16, 2015, 12:31 p.m., Chinna Lalam wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/33259/ > --- > > (Updated April 16, 2015, 12:31 p.m.) > > > Review request for hive and Szehon Ho. > > > Repository: hive > > > Description > --- > > commented line are skipped before processing the command. > > > Diffs > - > > trunk/cli/src/java/org/apache/hadoop/hive/cli/CliDriver.java 1673646 > > Diff: https://reviews.apache.org/r/33259/diff/ > > > Testing > --- > > Related tests are passed. > > > Thanks, > > Chinna Lalam > >
[jira] [Created] (HIVE-10435) Make HiveSession implementation pluggable through configuration
Amareshwari Sriramadasu created HIVE-10435: -- Summary: Make HiveSession implementation pluggable through configuration Key: HIVE-10435 URL: https://issues.apache.org/jira/browse/HIVE-10435 Project: Hive Issue Type: Improvement Components: HiveServer2 Reporter: Amareshwari Sriramadasu SessionManager in CLIService creates and keeps track of HiveSession. Right now, it creates HiveSessionImpl which is one implementation of HiveSession. This improvement request is to make it pluggable through a configuration sothat other implementations can be passed. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-9324) Reduce side joins failing with IOException from RowContainer.nextBlock
[
https://issues.apache.org/jira/browse/HIVE-9324?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14270779#comment-14270779
]
Amareshwari Sriramadasu commented on HIVE-9324:
---
After doing some code walkthrough, here is what i found,
On JoinOperator, whenever any key as more values than BLOCKSIZE(hardcoded to
25000), it spills the values to a file on disk, and spill uses SequenceFile
format.
Here is the table description for spill (from
org.apache.hadoop.hive.ql.exec.JoinUtil.java)
{noformat}
TableDesc tblDesc = new TableDesc(
SequenceFileInputFormat.class, HiveSequenceFileOutputFormat.class,
Utilities.makeProperties(
org.apache.hadoop.hive.serde.serdeConstants.SERIALIZATION_FORMAT, ""
+ Utilities.ctrlaCode,
org.apache.hadoop.hive.serde.serdeConstants.LIST_COLUMNS, colNames
.toString(),
org.apache.hadoop.hive.serde.serdeConstants.LIST_COLUMN_TYPES,
colTypes.toString(),
serdeConstants.SERIALIZATION_LIB,LazyBinarySerDe.class.getName()));
spillTableDesc[tag] = tblDesc;
{noformat}
>From the exception:
{noformat}
Caused by: java.io.IOException:
org.apache.hadoop.hive.ql.io.RCFile$KeyBuffer@42610e8 read 1 bytes, should read
27264
at org.apache.hadoop.io.SequenceFile$Reader.next(SequenceFile.java:2435)
at
org.apache.hadoop.mapred.SequenceFileRecordReader.next(SequenceFileRecordReader.java:76)
at
org.apache.hadoop.hive.ql.exec.persistence.RowContainer.nextBlock(RowContainer.java:360)
... 13 more
{noformat}
I see that the value in SequenceFile is RCFile$KeyBuffer, dont know why. Also
couldnt figure out the reason why the reading went wrong.
Following is the code snippet from SequenceFile.java for the exception we are
hitting :
{noformat}
2417 public synchronized Object next(Object key) throws IOException {
2418 if (key != null && key.getClass() != getKeyClass()) {
2419 throw new IOException("wrong key class: "+key.getClass().getName()
2420 +" is not "+keyClass);
2421 }
2422
2423 if (!blockCompressed) {
2424 outBuf.reset();
2425
2426 keyLength = next(outBuf);
2427 if (keyLength < 0)
2428 return null;
2429
2430 valBuffer.reset(outBuf.getData(), outBuf.getLength());
2431
2432 key = deserializeKey(key);
2433 valBuffer.mark(0);
2434 if (valBuffer.getPosition() != keyLength)
2435 throw new IOException(key + " read " + valBuffer.getPosition()
2436 + " bytes, should read " + keyLength);
{noformat}
> Reduce side joins failing with IOException from RowContainer.nextBlock
> --
>
> Key: HIVE-9324
> URL: https://issues.apache.org/jira/browse/HIVE-9324
> Project: Hive
> Issue Type: Bug
> Components: Query Processor
>Affects Versions: 0.13.1
>Reporter: Amareshwari Sriramadasu
>
> We are seeing some reduce side join mapreduce jobs failing with following
> exception :
> {noformat}
> 2014-12-14 16:58:51,296 ERROR
> org.apache.hadoop.hive.ql.exec.persistence.RowContainer:
> org.apache.hadoop.hive.ql.io.RCFile$KeyBuffer@42610e8 read 1 bytes, should
> read 27264
> java.io.IOException: org.apache.hadoop.hive.ql.io.RCFile$KeyBuffer@42610e8
> read 1 bytes, should read 27264
> at org.apache.hadoop.io.SequenceFile$Reader.next(SequenceFile.java:2435)
> at
> org.apache.hadoop.mapred.SequenceFileRecordReader.next(SequenceFileRecordReader.java:76)
> at
> org.apache.hadoop.hive.ql.exec.persistence.RowContainer.nextBlock(RowContainer.java:360)
> at
> org.apache.hadoop.hive.ql.exec.persistence.RowContainer.first(RowContainer.java:230)
> at
> org.apache.hadoop.hive.ql.exec.persistence.RowContainer.first(RowContainer.java:74)
> at
> org.apache.hadoop.hive.ql.exec.CommonJoinOperator.genUniqueJoinObject(CommonJoinOperator.java:644)
> at
> org.apache.hadoop.hive.ql.exec.CommonJoinOperator.checkAndGenObject(CommonJoinOperator.java:758)
> at
> org.apache.hadoop.hive.ql.exec.JoinOperator.endGroup(JoinOperator.java:256)
> at
> org.apache.hadoop.hive.ql.exec.mr.ExecReducer.reduce(ExecReducer.java:216)
> at
> org.apache.hadoop.mapred.ReduceTask.runOldReducer(ReduceTask.java:506)
> at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:447)
> at org.apache.hadoop.mapred.Child$4.run(Child.java:268)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:4
[jira] [Comment Edited] (HIVE-9324) Reduce side joins failing with IOException from RowContainer.nextBlock
[
https://issues.apache.org/jira/browse/HIVE-9324?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14270588#comment-14270588
]
Amareshwari Sriramadasu edited comment on HIVE-9324 at 1/9/15 5:54 AM:
---
More task log :
{noformat}
2014-12-14 16:58:03,905 INFO org.apache.hadoop.hive.ql.exec.mr.ObjectCache:
Ignoring retrieval request: __REDUCE_PLAN__
2014-12-14 16:58:03,945 INFO org.apache.hadoop.hive.ql.log.PerfLogger:
2014-12-14 16:58:03,945 INFO org.apache.hadoop.hive.ql.exec.Utilities:
Deserializing ReduceWork via kryo
2014-12-14 16:58:04,987 INFO org.apache.hadoop.hive.ql.log.PerfLogger:
2014-12-14 16:58:04,988 INFO org.apache.hadoop.hive.ql.exec.mr.ObjectCache:
Ignoring cache key: __REDUCE_PLAN__
2014-12-14 16:58:05,327 INFO ExecReducer:
Id =0
Id =1
<\Children>
Id = 0 null<\Parent>
<\FS>
<\Children>
<\Parent>
<\JOIN>
2014-12-14 16:58:05,327 INFO org.apache.hadoop.hive.ql.exec.JoinOperator:
Initializing Self 0 JOIN
2014-12-14 16:58:05,377 INFO org.apache.hadoop.hive.ql.exec.CommonJoinOperator:
JOIN
struct<_col23:string,_col65:double,_col99:double,_col237:double,_col240:double,_col250:string,_col367:int>
totalsz = 7
2014-12-14 16:58:05,377 INFO org.apache.hadoop.hive.ql.exec.JoinOperator:
Operator 0 JOIN initialized
2014-12-14 16:58:05,377 INFO org.apache.hadoop.hive.ql.exec.JoinOperator:
Initializing children of 0 JOIN
2014-12-14 16:58:05,377 INFO org.apache.hadoop.hive.ql.exec.FileSinkOperator:
Initializing child 1 FS
2014-12-14 16:58:05,377 INFO org.apache.hadoop.hive.ql.exec.FileSinkOperator:
Initializing Self 1 FS
2014-12-14 16:58:05,394 INFO org.apache.hadoop.hive.ql.exec.FileSinkOperator:
Operator 1 FS initialized
2014-12-14 16:58:05,394 INFO org.apache.hadoop.hive.ql.exec.FileSinkOperator:
Initialization Done 1 FS
2014-12-14 16:58:05,395 INFO org.apache.hadoop.hive.ql.exec.JoinOperator:
Initialization Done 0 JOIN
2014-12-14 16:58:05,401 INFO ExecReducer: ExecReducer: processing 1 rows: used
memory = 242598168
2014-12-14 16:58:05,406 INFO ExecReducer: ExecReducer: processing 10 rows: used
memory = 242759392
2014-12-14 16:58:05,437 INFO ExecReducer: ExecReducer: processing 100 rows:
used memory = 242759392
2014-12-14 16:58:05,657 INFO ExecReducer: ExecReducer: processing 1000 rows:
used memory = 243653240
2014-12-14 16:58:06,976 INFO ExecReducer: ExecReducer: processing 1 rows:
used memory = 247197944
2014-12-14 16:58:07,646 INFO ExecReducer: ExecReducer: processing 10 rows:
used memory = 277801256
2014-12-14 16:58:11,511 INFO ExecReducer: ExecReducer: processing 100 rows:
used memory = 283150744
2014-12-14 16:58:14,993 INFO ExecReducer: ExecReducer: processing 200 rows:
used memory = 293036992
2014-12-14 16:58:18,497 INFO ExecReducer: ExecReducer: processing 300 rows:
used memory = 311449488
2014-12-14 16:58:20,815 INFO ExecReducer: ExecReducer: processing 400 rows:
used memory = 285251752
2014-12-14 16:58:26,460 INFO ExecReducer: ExecReducer: processing 500 rows:
used memory = 328223864
2014-12-14 16:58:29,412 INFO ExecReducer: ExecReducer: processing 600 rows:
used memory = 263175576
2014-12-14 16:58:31,331 INFO ExecReducer: ExecReducer: processing 700 rows:
used memory = 282021320
2014-12-14 16:58:35,099 INFO ExecReducer: ExecReducer: processing 800 rows:
used memory = 299301184
2014-12-14 16:58:37,981 INFO ExecReducer: ExecReducer: processing 900 rows:
used memory = 306925648
2014-12-14 16:58:40,506 INFO ExecReducer: ExecReducer: processing 1000
rows: used memory = 307407920
2014-12-14 16:58:42,242 INFO ExecReducer: ExecReducer: processing 1100
rows: used memory = 304664048
2014-12-14 16:58:46,142 INFO ExecReducer: ExecReducer: processing 1200
rows: used memory = 298347024
2014-12-14 16:58:48,549 INFO org.apache.hadoop.hive.ql.exec.CommonJoinOperator:
table 0 has 1000 rows for join key [003b9de7876541c2bcce9029ff0d3873]
2014-12-14 16:58:48,622 INFO org.apache.hadoop.hive.ql.exec.CommonJoinOperator:
table 0 has 2000 rows for join key [003b9de7876541c2bcce9029ff0d3873]
2014-12-14 16:58:48,677 INFO org.apache.hadoop.hive.ql.exec.CommonJoinOperator:
table 0 has 4000 rows for join key [003b9de7876541c2bcce9029ff0d3873]
2014-12-14 16:58:48,679 INFO org.apache.hadoop.hive.ql.exec.FileSinkOperator:
Final Path: FS
hdfs://test-machine:8020/tmp/hive-dataqa/hive_2014-12-14_16-49-14_996_1630664550753106415-32/_tmp.-mr-10002/00_0
2014-12-14 16:58:48,680 INFO org.apache.hadoop.hive.ql.exec.FileSinkOperator:
Writing to temp file: FS
hdfs://test-machine:8020/tmp/hive-dataqa/hive_2014-12-14_16-49-14_996_1630664550753106415-32/_task_tmp.-mr-10002/_tmp.00_0
2014-12-14 16:58:48,680 INFO org.apache.hadoop.hive.ql.exec.FileSinkOperator:
New Final Path: FS
hdfs://test-machine:8020/tmp/hive-dataqa/hive_2014-12-14_16-4
[jira] [Commented] (HIVE-9324) Reduce side joins failing with IOException from RowContainer.nextBlock
[
https://issues.apache.org/jira/browse/HIVE-9324?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14270588#comment-14270588
]
Amareshwari Sriramadasu commented on HIVE-9324:
---
More task log :
{noformat}
2014-12-14 16:58:03,905 INFO org.apache.hadoop.hive.ql.exec.mr.ObjectCache:
Ignoring retrieval request: __REDUCE_PLAN__
2014-12-14 16:58:03,945 INFO org.apache.hadoop.hive.ql.log.PerfLogger:
2014-12-14 16:58:03,945 INFO org.apache.hadoop.hive.ql.exec.Utilities:
Deserializing ReduceWork via kryo
2014-12-14 16:58:04,987 INFO org.apache.hadoop.hive.ql.log.PerfLogger:
2014-12-14 16:58:04,988 INFO org.apache.hadoop.hive.ql.exec.mr.ObjectCache:
Ignoring cache key: __REDUCE_PLAN__
2014-12-14 16:58:05,327 INFO ExecReducer:
Id =0
Id =1
<\Children>
Id = 0 null<\Parent>
<\FS>
<\Children>
<\Parent>
<\JOIN>
2014-12-14 16:58:05,327 INFO org.apache.hadoop.hive.ql.exec.JoinOperator:
Initializing Self 0 JOIN
2014-12-14 16:58:05,377 INFO org.apache.hadoop.hive.ql.exec.CommonJoinOperator:
JOIN
struct<_col23:string,_col65:double,_col99:double,_col237:double,_col240:double,_col250:string,_col367:int>
totalsz = 7
2014-12-14 16:58:05,377 INFO org.apache.hadoop.hive.ql.exec.JoinOperator:
Operator 0 JOIN initialized
2014-12-14 16:58:05,377 INFO org.apache.hadoop.hive.ql.exec.JoinOperator:
Initializing children of 0 JOIN
2014-12-14 16:58:05,377 INFO org.apache.hadoop.hive.ql.exec.FileSinkOperator:
Initializing child 1 FS
2014-12-14 16:58:05,377 INFO org.apache.hadoop.hive.ql.exec.FileSinkOperator:
Initializing Self 1 FS
2014-12-14 16:58:05,394 INFO org.apache.hadoop.hive.ql.exec.FileSinkOperator:
Operator 1 FS initialized
2014-12-14 16:58:05,394 INFO org.apache.hadoop.hive.ql.exec.FileSinkOperator:
Initialization Done 1 FS
2014-12-14 16:58:05,395 INFO org.apache.hadoop.hive.ql.exec.JoinOperator:
Initialization Done 0 JOIN
2014-12-14 16:58:05,401 INFO ExecReducer: ExecReducer: processing 1 rows: used
memory = 242598168
2014-12-14 16:58:05,406 INFO ExecReducer: ExecReducer: processing 10 rows: used
memory = 242759392
2014-12-14 16:58:05,437 INFO ExecReducer: ExecReducer: processing 100 rows:
used memory = 242759392
2014-12-14 16:58:05,657 INFO ExecReducer: ExecReducer: processing 1000 rows:
used memory = 243653240
2014-12-14 16:58:06,976 INFO ExecReducer: ExecReducer: processing 1 rows:
used memory = 247197944
2014-12-14 16:58:07,646 INFO ExecReducer: ExecReducer: processing 10 rows:
used memory = 277801256
2014-12-14 16:58:11,511 INFO ExecReducer: ExecReducer: processing 100 rows:
used memory = 283150744
2014-12-14 16:58:14,993 INFO ExecReducer: ExecReducer: processing 200 rows:
used memory = 293036992
2014-12-14 16:58:18,497 INFO ExecReducer: ExecReducer: processing 300 rows:
used memory = 311449488
2014-12-14 16:58:20,815 INFO ExecReducer: ExecReducer: processing 400 rows:
used memory = 285251752
2014-12-14 16:58:26,460 INFO ExecReducer: ExecReducer: processing 500 rows:
used memory = 328223864
2014-12-14 16:58:29,412 INFO ExecReducer: ExecReducer: processing 600 rows:
used memory = 263175576
2014-12-14 16:58:31,331 INFO ExecReducer: ExecReducer: processing 700 rows:
used memory = 282021320
2014-12-14 16:58:35,099 INFO ExecReducer: ExecReducer: processing 800 rows:
used memory = 299301184
2014-12-14 16:58:37,981 INFO ExecReducer: ExecReducer: processing 900 rows:
used memory = 306925648
2014-12-14 16:58:40,506 INFO ExecReducer: ExecReducer: processing 1000
rows: used memory = 307407920
2014-12-14 16:58:42,242 INFO ExecReducer: ExecReducer: processing 1100
rows: used memory = 304664048
2014-12-14 16:58:46,142 INFO ExecReducer: ExecReducer: processing 1200
rows: used memory = 298347024
2014-12-14 16:58:48,549 INFO org.apache.hadoop.hive.ql.exec.CommonJoinOperator:
table 0 has 1000 rows for join key [003b9de7876541c2bcce9029ff0d3873]
2014-12-14 16:58:48,622 INFO org.apache.hadoop.hive.ql.exec.CommonJoinOperator:
table 0 has 2000 rows for join key [003b9de7876541c2bcce9029ff0d3873]
2014-12-14 16:58:48,677 INFO org.apache.hadoop.hive.ql.exec.CommonJoinOperator:
table 0 has 4000 rows for join key [003b9de7876541c2bcce9029ff0d3873]
2014-12-14 16:58:48,679 INFO org.apache.hadoop.hive.ql.exec.FileSinkOperator:
Final Path: FS
hdfs://data-grill300-null.arshad.ev1.inmobi.com:8020/tmp/hive-dataqa/hive_2014-12-14_16-49-14_996_1630664550753106415-32/_tmp.-mr-10002/00_0
2014-12-14 16:58:48,680 INFO org.apache.hadoop.hive.ql.exec.FileSinkOperator:
Writing to temp file: FS
hdfs://data-grill300-null.arshad.ev1.inmobi.com:8020/tmp/hive-dataqa/hive_2014-12-14_16-49-14_996_1630664550753106415-32/_task_tmp.-mr-10002/_tmp.00_0
2014-12-14 16:58:48,680 INFO org.apache.hadoop.hive.ql.exec.FileSinkOperator:
New Final Path: FS
hdfs://data-grill300-null.arshad.ev1.inmobi.
[jira] [Created] (HIVE-9324) Reduce side joins failing with IOException from RowContainer.nextBlock
Amareshwari Sriramadasu created HIVE-9324:
-
Summary: Reduce side joins failing with IOException from
RowContainer.nextBlock
Key: HIVE-9324
URL: https://issues.apache.org/jira/browse/HIVE-9324
Project: Hive
Issue Type: Bug
Components: Query Processor
Affects Versions: 0.13.1
Reporter: Amareshwari Sriramadasu
We are seeing some reduce side join mapreduce jobs failing with following
exception :
{noformat}
2014-12-14 16:58:51,296 ERROR
org.apache.hadoop.hive.ql.exec.persistence.RowContainer:
org.apache.hadoop.hive.ql.io.RCFile$KeyBuffer@42610e8 read 1 bytes, should read
27264
java.io.IOException: org.apache.hadoop.hive.ql.io.RCFile$KeyBuffer@42610e8 read
1 bytes, should read 27264
at org.apache.hadoop.io.SequenceFile$Reader.next(SequenceFile.java:2435)
at
org.apache.hadoop.mapred.SequenceFileRecordReader.next(SequenceFileRecordReader.java:76)
at
org.apache.hadoop.hive.ql.exec.persistence.RowContainer.nextBlock(RowContainer.java:360)
at
org.apache.hadoop.hive.ql.exec.persistence.RowContainer.first(RowContainer.java:230)
at
org.apache.hadoop.hive.ql.exec.persistence.RowContainer.first(RowContainer.java:74)
at
org.apache.hadoop.hive.ql.exec.CommonJoinOperator.genUniqueJoinObject(CommonJoinOperator.java:644)
at
org.apache.hadoop.hive.ql.exec.CommonJoinOperator.checkAndGenObject(CommonJoinOperator.java:758)
at
org.apache.hadoop.hive.ql.exec.JoinOperator.endGroup(JoinOperator.java:256)
at
org.apache.hadoop.hive.ql.exec.mr.ExecReducer.reduce(ExecReducer.java:216)
at
org.apache.hadoop.mapred.ReduceTask.runOldReducer(ReduceTask.java:506)
at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:447)
at org.apache.hadoop.mapred.Child$4.run(Child.java:268)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:416)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1408)
at org.apache.hadoop.mapred.Child.main(Child.java:262)
2014-12-14 16:58:51,334 FATAL ExecReducer:
org.apache.hadoop.hive.ql.metadata.HiveException:
org.apache.hadoop.hive.ql.metadata.HiveException: java.io.IOException:
org.apache.hadoop.hive.ql.io.RCFile$KeyBuffer@42610e8 read 1 bytes, should read
27264
at
org.apache.hadoop.hive.ql.exec.persistence.RowContainer.first(RowContainer.java:237)
at
org.apache.hadoop.hive.ql.exec.persistence.RowContainer.first(RowContainer.java:74)
at
org.apache.hadoop.hive.ql.exec.CommonJoinOperator.genUniqueJoinObject(CommonJoinOperator.java:644)
at
org.apache.hadoop.hive.ql.exec.CommonJoinOperator.checkAndGenObject(CommonJoinOperator.java:758)
at
org.apache.hadoop.hive.ql.exec.JoinOperator.endGroup(JoinOperator.java:256)
at
org.apache.hadoop.hive.ql.exec.mr.ExecReducer.reduce(ExecReducer.java:216)
at
org.apache.hadoop.mapred.ReduceTask.runOldReducer(ReduceTask.java:506)
at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:447)
at org.apache.hadoop.mapred.Child$4.run(Child.java:268)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:416)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1408)
at org.apache.hadoop.mapred.Child.main(Child.java:262)
Caused by: org.apache.hadoop.hive.ql.metadata.HiveException:
java.io.IOException: org.apache.hadoop.hive.ql.io.RCFile$KeyBuffer@42610e8 read
1 bytes, should read 27264
at
org.apache.hadoop.hive.ql.exec.persistence.RowContainer.nextBlock(RowContainer.java:385)
at
org.apache.hadoop.hive.ql.exec.persistence.RowContainer.first(RowContainer.java:230)
... 12 more
Caused by: java.io.IOException:
org.apache.hadoop.hive.ql.io.RCFile$KeyBuffer@42610e8 read 1 bytes, should read
27264
at org.apache.hadoop.io.SequenceFile$Reader.next(SequenceFile.java:2435)
at
org.apache.hadoop.mapred.SequenceFileRecordReader.next(SequenceFileRecordReader.java:76)
at
org.apache.hadoop.hive.ql.exec.persistence.RowContainer.nextBlock(RowContainer.java:360)
... 13 more
{noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-4115) Introduce cube abstraction in hive
[ https://issues.apache.org/jira/browse/HIVE-4115?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Amareshwari Sriramadasu updated HIVE-4115: -- Resolution: Won't Fix Status: Resolved (was: Patch Available) This effort has been incubated into apache here - http://incubator.apache.org/projects/lens.html > Introduce cube abstraction in hive > -- > > Key: HIVE-4115 > URL: https://issues.apache.org/jira/browse/HIVE-4115 > Project: Hive > Issue Type: New Feature > Reporter: Amareshwari Sriramadasu > Assignee: Amareshwari Sriramadasu > Attachments: HIVE-4115.D10689.1.patch, HIVE-4115.D10689.2.patch, > HIVE-4115.D10689.3.patch, HIVE-4115.D10689.4.patch, cube-design-2.pdf, > cube-design.docx > > > We would like to define a cube abstraction so that user can query at cube > layer and do not know anything about storage and rollups. > Will describe the model more in following comments. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Comment Edited] (HIVE-7936) Support for handling Thrift Union types
[ https://issues.apache.org/jira/browse/HIVE-7936?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14133946#comment-14133946 ] Amareshwari Sriramadasu edited comment on HIVE-7936 at 9/15/14 2:37 PM: Sorry.. missed committing binary file changes. Merged it now in a different commit was (Author: amareshwari): Sorry.. missed commenting binary file changes. Merging it now. > Support for handling Thrift Union types > > > Key: HIVE-7936 > URL: https://issues.apache.org/jira/browse/HIVE-7936 > Project: Hive > Issue Type: Bug > Components: Serializers/Deserializers >Affects Versions: 0.13.1 >Reporter: Suma Shivaprasad >Assignee: Suma Shivaprasad > Fix For: 0.14.0 > > Attachments: HIVE-7936.1.patch, HIVE-7936.2.patch, HIVE-7936.patch, > complex.seq > > > Currently hive does not support thrift unions through ThriftDeserializer. > Need to add support for the same -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-7936) Support for handling Thrift Union types
[ https://issues.apache.org/jira/browse/HIVE-7936?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14133946#comment-14133946 ] Amareshwari Sriramadasu commented on HIVE-7936: --- Sorry.. missed commenting binary file changes. Merging it now. > Support for handling Thrift Union types > > > Key: HIVE-7936 > URL: https://issues.apache.org/jira/browse/HIVE-7936 > Project: Hive > Issue Type: Bug > Components: Serializers/Deserializers >Affects Versions: 0.13.1 >Reporter: Suma Shivaprasad >Assignee: Suma Shivaprasad > Fix For: 0.14.0 > > Attachments: HIVE-7936.1.patch, HIVE-7936.2.patch, HIVE-7936.patch, > complex.seq > > > Currently hive does not support thrift unions through ThriftDeserializer. > Need to add support for the same -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-7936) Support for handling Thrift Union types
[ https://issues.apache.org/jira/browse/HIVE-7936?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Amareshwari Sriramadasu updated HIVE-7936: -- Resolution: Fixed Release Note: Support Thrift union type as hive union type Hadoop Flags: Reviewed Status: Resolved (was: Patch Available) I just committed this. Thanks Suma! > Support for handling Thrift Union types > > > Key: HIVE-7936 > URL: https://issues.apache.org/jira/browse/HIVE-7936 > Project: Hive > Issue Type: Bug > Components: Serializers/Deserializers >Affects Versions: 0.13.1 >Reporter: Suma Shivaprasad >Assignee: Suma Shivaprasad > Fix For: 0.14.0 > > Attachments: HIVE-7936.1.patch, HIVE-7936.2.patch, HIVE-7936.patch, > complex.seq > > > Currently hive does not support thrift unions through ThriftDeserializer. > Need to add support for the same -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-7936) Support for handling Thrift Union types
[ https://issues.apache.org/jira/browse/HIVE-7936?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14133617#comment-14133617 ] Amareshwari Sriramadasu commented on HIVE-7936: --- +1 Patch looks fine. Can you update the test output here? > Support for handling Thrift Union types > > > Key: HIVE-7936 > URL: https://issues.apache.org/jira/browse/HIVE-7936 > Project: Hive > Issue Type: Bug > Components: Serializers/Deserializers >Affects Versions: 0.13.1 >Reporter: Suma Shivaprasad >Assignee: Suma Shivaprasad > Fix For: 0.14.0 > > Attachments: HIVE-7936.1.patch, HIVE-7936.2.patch, HIVE-7936.patch, > complex.seq > > > Currently hive does not support thrift unions through ThriftDeserializer. > Need to add support for the same -- This message was sent by Atlassian JIRA (v6.3.4#6332)
Re: Review Request 25492: HIVE-7936 - Thrift Union support
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/25492/#review53314 --- Ship it! Ship It! - Amareshwari Sriramadasu On Sept. 11, 2014, 4:26 p.m., Suma Shivaprasad wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/25492/ > --- > > (Updated Sept. 11, 2014, 4:26 p.m.) > > > Review request for hive, Amareshwari Sriramadasu and Ashutosh Chauhan. > > > Bugs: HIVE-7936 > https://issues.apache.org/jira/browse/HIVE-7936 > > > Repository: hive-git > > > Description > --- > > ThriftDeserializer currently does not support UNION types > > > Diffs > - > > contrib/src/test/results/clientpositive/udf_example_arraymapstruct.q.out > e876cdd > data/files/complex.seq c27d5c09b1da881d8fd6fb2aaa1f5d169d1de3ae > ql/src/test/queries/clientpositive/input_lazyserde.q 69c0d04 > ql/src/test/results/clientnegative/describe_xpath1.q.out d81c96e > ql/src/test/results/clientnegative/describe_xpath2.q.out 2bd0f06 > ql/src/test/results/clientpositive/case_sensitivity.q.out 8684557 > ql/src/test/results/clientpositive/columnarserde_create_shortcut.q.out > 4805836 > ql/src/test/results/clientpositive/input17.q.out 8fff21b > ql/src/test/results/clientpositive/input5.q.out 7524ca7 > ql/src/test/results/clientpositive/input_columnarserde.q.out 13cfb7f > ql/src/test/results/clientpositive/input_dynamicserde.q.out ebcf1d8 > ql/src/test/results/clientpositive/input_lazyserde.q.out 0f685f2 > ql/src/test/results/clientpositive/input_testxpath.q.out 3f4b96e > ql/src/test/results/clientpositive/input_testxpath2.q.out af1e999 > ql/src/test/results/clientpositive/input_testxpath3.q.out b31b2f3 > ql/src/test/results/clientpositive/input_testxpath4.q.out 3dca8bf > ql/src/test/results/clientpositive/inputddl8.q.out fc13356 > ql/src/test/results/clientpositive/join_thrift.q.out e1588c5 > ql/src/test/results/clientpositive/udf_case_thrift.q.out 0fc8e84 > ql/src/test/results/clientpositive/udf_coalesce.q.out 0d32476 > ql/src/test/results/clientpositive/udf_isnull_isnotnull.q.out 1f600b4 > ql/src/test/results/clientpositive/udf_size.q.out d7a4fa2 > ql/src/test/results/clientpositive/union21.q.out 0e47ff4 > ql/src/test/results/compiler/plan/case_sensitivity.q.xml c624418 > ql/src/test/results/compiler/plan/input5.q.xml a881da3 > ql/src/test/results/compiler/plan/input_testxpath.q.xml a3dacdf > ql/src/test/results/compiler/plan/input_testxpath2.q.xml 7fc3a49 > serde/if/test/complex.thrift 308b64c > serde/src/gen/thrift/gen-cpp/complex_types.h 17991d4 > serde/src/gen/thrift/gen-cpp/complex_types.cpp 9526d3d > > serde/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/serde2/thrift/test/Complex.java > e36a792 > > serde/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/serde2/thrift/test/PropValueUnion.java > PRE-CREATION > serde/src/gen/thrift/gen-py/complex/ttypes.py 7283e4c > serde/src/gen/thrift/gen-rb/complex_types.rb 5527096 > > serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/ObjectInspectorFactory.java > 9a226b3 > > serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/ReflectionStructObjectInspector.java > ee5b0d0 > > serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/ThriftObjectInspectorUtils.java > PRE-CREATION > > serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/ThriftUnionObjectInspector.java > PRE-CREATION > > serde/src/test/org/apache/hadoop/hive/serde2/objectinspector/TestObjectInspectorUtils.java > a18f4a7 > > serde/src/test/org/apache/hadoop/hive/serde2/objectinspector/TestThriftObjectInspectors.java > 5f692fb > > serde/src/test/org/apache/hadoop/hive/serde2/thrift_test/CreateSequenceFile.java > 7269cd0 > > Diff: https://reviews.apache.org/r/25492/diff/ > > > Testing > --- > > input_lazyserde.q > > > Thanks, > > Suma Shivaprasad > >
Re: Review Request 25473: Thrift Set type not working with Hive
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/25473/#review53313 --- Ship it! - Amareshwari Sriramadasu On Sept. 10, 2014, 7:03 a.m., Satish Mittal wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/25473/ > --- > > (Updated Sept. 10, 2014, 7:03 a.m.) > > > Review request for hive, Amareshwari Sriramadasu, Ashutosh Chauhan, and Navis > Ryu. > > > Bugs: HIVE-7892 > https://issues.apache.org/jira/browse/HIVE-7892 > > > Repository: hive-git > > > Description > --- > > Thrift supports List, Map and Struct complex types, which get mapped to > Array, Map and Struct complex types in Hive respectively. However thrift Set > type doesn't get mapped to any Hive type, and hence doesn't work with > ThriftDeserializer serde. > > > Diffs > - > > ql/src/test/results/beelinepositive/convert_enum_to_string.q.out 24acdcd > ql/src/test/results/clientpositive/convert_enum_to_string.q.out a1ef04f > serde/if/test/complex.thrift 308b64c > > serde/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/serde2/thrift/test/SetIntString.java > PRE-CREATION > > serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/ObjectInspectorFactory.java > 9a226b3 > > serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/StandardListObjectInspector.java > 6eb8803 > > serde/src/test/org/apache/hadoop/hive/serde2/objectinspector/TestThriftObjectInspectors.java > 5f692fb > > Diff: https://reviews.apache.org/r/25473/diff/ > > > Testing > --- > > 1) Added Unit test along with the fix. > 2) Manually tested by creating a table with ThriftDeserializer serde and > having thrift set columns: >a) described the table >b) issued query to select the set column > > > Thanks, > > Satish Mittal > >
[jira] [Updated] (HIVE-7892) Thrift Set type not working with Hive
[
https://issues.apache.org/jira/browse/HIVE-7892?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Amareshwari Sriramadasu updated HIVE-7892:
--
Resolution: Fixed
Fix Version/s: 0.14.0
Release Note: Maps thrift's set type to hive's array type.
Hadoop Flags: Reviewed
Status: Resolved (was: Patch Available)
I just committed this. Thanks Satish !
> Thrift Set type not working with Hive
> -
>
> Key: HIVE-7892
> URL: https://issues.apache.org/jira/browse/HIVE-7892
> Project: Hive
> Issue Type: Bug
> Components: Serializers/Deserializers
>Reporter: Satish Mittal
>Assignee: Satish Mittal
> Fix For: 0.14.0
>
> Attachments: HIVE-7892.1.patch, HIVE-7892.patch.txt
>
>
> Thrift supports List, Map and Struct complex types, which get mapped to
> Array, Map and Struct complex types in Hive respectively. However thrift Set
> type doesn't seem to be working.
> Here is an example thrift struct:
> {noformat}
> namespace java sample.thrift
> struct setrow {
> 1: required set ids,
> 2: required string name,
> }
> {noformat}
> A Hive table is created with ROW FORMAT SERDE
> 'org.apache.hadoop.hive.serde2.thrift.ThriftDeserializer' WITH
> SERDEPROPERTIES ('serialization.class'='sample.thrift.setrow',
> 'serialization.format'='org.apache.thrift.protocol.TBinaryProtocol').
> Describing the table shows:
> {noformat}
> hive> describe settable;
> OK
> ids struct<>from deserializer
> namestringfrom deserializer
> {noformat}
> Issuing a select query on set column throws SemanticException:
> {noformat}
> hive> select ids from settable;
> FAILED: SemanticException java.lang.IllegalArgumentException: Error: name
> expected at the position 7 of 'struct<>' but '>' is found.
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-7936) Support for handling Thrift Union types
[ https://issues.apache.org/jira/browse/HIVE-7936?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14128313#comment-14128313 ] Amareshwari Sriramadasu commented on HIVE-7936: --- The code changes look fine. Put a few comments on the review board. Since the patch involves a binary file change, i think jenkins wont be able to apply the patch. Can you run the tests on a local machine and update the result here? > Support for handling Thrift Union types > > > Key: HIVE-7936 > URL: https://issues.apache.org/jira/browse/HIVE-7936 > Project: Hive > Issue Type: Bug > Components: Serializers/Deserializers >Affects Versions: 0.13.1 >Reporter: Suma Shivaprasad >Assignee: Suma Shivaprasad > Fix For: 0.14.0 > > Attachments: HIVE-7936.1.patch, HIVE-7936.patch, complex.seq > > > Currently hive does not support thrift unions through ThriftDeserializer. > Need to add support for the same -- This message was sent by Atlassian JIRA (v6.3.4#6332)
Re: Review Request 25492: HIVE-7936 - Thrift Union support
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/25492/#review52844 --- serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/ThriftObjectInspectorUtils.java <https://reviews.apache.org/r/25492/#comment92014> Remove this method if not used - Amareshwari Sriramadasu On Sept. 10, 2014, 5:27 a.m., Suma Shivaprasad wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/25492/ > --- > > (Updated Sept. 10, 2014, 5:27 a.m.) > > > Review request for hive, Amareshwari Sriramadasu and Ashutosh Chauhan. > > > Bugs: HIVE-7936 > https://issues.apache.org/jira/browse/HIVE-7936 > > > Repository: hive-git > > > Description > --- > > ThriftDeserializer currently does not support UNION types > > > Diffs > - > > contrib/src/test/results/clientpositive/udf_example_arraymapstruct.q.out > e876cdd > data/files/complex.seq c27d5c09b1da881d8fd6fb2aaa1f5d169d1de3ae > ql/src/test/queries/clientpositive/input_lazyserde.q 69c0d04 > ql/src/test/results/clientnegative/describe_xpath1.q.out d81c96e > ql/src/test/results/clientnegative/describe_xpath2.q.out 2bd0f06 > ql/src/test/results/clientpositive/case_sensitivity.q.out 8684557 > ql/src/test/results/clientpositive/columnarserde_create_shortcut.q.out > 4805836 > ql/src/test/results/clientpositive/input17.q.out 8fff21b > ql/src/test/results/clientpositive/input5.q.out 7524ca7 > ql/src/test/results/clientpositive/input_columnarserde.q.out 13cfb7f > ql/src/test/results/clientpositive/input_dynamicserde.q.out ebcf1d8 > ql/src/test/results/clientpositive/input_lazyserde.q.out 0f685f2 > ql/src/test/results/clientpositive/input_testxpath.q.out 3f4b96e > ql/src/test/results/clientpositive/input_testxpath2.q.out af1e999 > ql/src/test/results/clientpositive/input_testxpath3.q.out b31b2f3 > ql/src/test/results/clientpositive/input_testxpath4.q.out 3dca8bf > ql/src/test/results/clientpositive/inputddl8.q.out fc13356 > ql/src/test/results/clientpositive/join_thrift.q.out e1588c5 > ql/src/test/results/clientpositive/udf_case_thrift.q.out 0fc8e84 > ql/src/test/results/clientpositive/udf_coalesce.q.out 0d32476 > ql/src/test/results/clientpositive/udf_isnull_isnotnull.q.out 1f600b4 > ql/src/test/results/clientpositive/udf_size.q.out d7a4fa2 > ql/src/test/results/clientpositive/union21.q.out 0e47ff4 > serde/if/test/complex.thrift 308b64c > serde/src/gen/thrift/gen-cpp/complex_types.h 17991d4 > serde/src/gen/thrift/gen-cpp/complex_types.cpp 9526d3d > > serde/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/serde2/thrift/test/Complex.java > e36a792 > > serde/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/serde2/thrift/test/PropValueUnion.java > PRE-CREATION > serde/src/gen/thrift/gen-py/complex/ttypes.py 7283e4c > serde/src/gen/thrift/gen-rb/complex_types.rb 5527096 > > serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/ObjectInspectorFactory.java > 9a226b3 > > serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/ReflectionStructObjectInspector.java > ee5b0d0 > > serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/ThriftObjectInspectorUtils.java > PRE-CREATION > > serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/ThriftUnionObjectInspector.java > PRE-CREATION > > serde/src/test/org/apache/hadoop/hive/serde2/objectinspector/TestObjectInspectorUtils.java > a18f4a7 > > serde/src/test/org/apache/hadoop/hive/serde2/objectinspector/TestThriftObjectInspectors.java > 5f692fb > > serde/src/test/org/apache/hadoop/hive/serde2/thrift_test/CreateSequenceFile.java > 7269cd0 > > Diff: https://reviews.apache.org/r/25492/diff/ > > > Testing > --- > > input_lazyserde.q > > > Thanks, > > Suma Shivaprasad > >
[jira] [Commented] (HIVE-7892) Thrift Set type not working with Hive
[
https://issues.apache.org/jira/browse/HIVE-7892?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14128138#comment-14128138
]
Amareshwari Sriramadasu commented on HIVE-7892:
---
Code changes look fine. Can you update the test output for
convert_enum_to_string.q and upload the patch?
> Thrift Set type not working with Hive
> -
>
> Key: HIVE-7892
> URL: https://issues.apache.org/jira/browse/HIVE-7892
> Project: Hive
> Issue Type: Bug
> Components: Serializers/Deserializers
>Reporter: Satish Mittal
>Assignee: Satish Mittal
> Attachments: HIVE-7892.patch.txt
>
>
> Thrift supports List, Map and Struct complex types, which get mapped to
> Array, Map and Struct complex types in Hive respectively. However thrift Set
> type doesn't seem to be working.
> Here is an example thrift struct:
> {noformat}
> namespace java sample.thrift
> struct setrow {
> 1: required set ids,
> 2: required string name,
> }
> {noformat}
> A Hive table is created with ROW FORMAT SERDE
> 'org.apache.hadoop.hive.serde2.thrift.ThriftDeserializer' WITH
> SERDEPROPERTIES ('serialization.class'='sample.thrift.setrow',
> 'serialization.format'='org.apache.thrift.protocol.TBinaryProtocol').
> Describing the table shows:
> {noformat}
> hive> describe settable;
> OK
> ids struct<>from deserializer
> namestringfrom deserializer
> {noformat}
> Issuing a select query on set column throws SemanticException:
> {noformat}
> hive> select ids from settable;
> FAILED: SemanticException java.lang.IllegalArgumentException: Error: name
> expected at the position 7 of 'struct<>' but '>' is found.
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-7694) SMB join on tables differing by number of sorted by columns with same join prefix fails
[
https://issues.apache.org/jira/browse/HIVE-7694?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14128116#comment-14128116
]
Amareshwari Sriramadasu commented on HIVE-7694:
---
I just committed this. Thanks Suma!
> SMB join on tables differing by number of sorted by columns with same join
> prefix fails
> ---
>
> Key: HIVE-7694
> URL: https://issues.apache.org/jira/browse/HIVE-7694
> Project: Hive
> Issue Type: Bug
> Components: Query Processor
>Affects Versions: 0.13.1
>Reporter: Suma Shivaprasad
>Assignee: Suma Shivaprasad
> Fix For: 0.14.0
>
> Attachments: HIVE-7694.1.patch, HIVE-7694.2.patch, HIVE-7694.patch
>
>
> For eg: If two tables T1 sorted by (a, b, c) clustered by a and T2 sorted by
> (a) and clustered by (a) are joined, the following exception is seen
> {noformat}
> 14/08/11 09:09:38 ERROR ql.Driver: FAILED: IndexOutOfBoundsException Index:
> 1, Size: 1
> java.lang.IndexOutOfBoundsException: Index: 1, Size: 1
> at java.util.ArrayList.RangeCheck(ArrayList.java:547)
> at java.util.ArrayList.get(ArrayList.java:322)
> at
> org.apache.hadoop.hive.ql.optimizer.AbstractSMBJoinProc.checkSortColsAndJoinCols(AbstractSMBJoinProc.java:378)
> at
> org.apache.hadoop.hive.ql.optimizer.AbstractSMBJoinProc.isEligibleForBucketSortMergeJoin(AbstractSMBJoinProc.java:352)
> at
> org.apache.hadoop.hive.ql.optimizer.AbstractSMBJoinProc.canConvertBucketMapJoinToSMBJoin(AbstractSMBJoinProc.java:119)
> at
> org.apache.hadoop.hive.ql.optimizer.SortedMergeBucketMapjoinProc.process(SortedMergeBucketMapjoinProc.java:51)
> at
> org.apache.hadoop.hive.ql.lib.DefaultRuleDispatcher.dispatch(DefaultRuleDispatcher.java:90)
> at
> org.apache.hadoop.hive.ql.lib.DefaultGraphWalker.dispatchAndReturn(DefaultGraphWalker.java:94)
> at
> org.apache.hadoop.hive.ql.lib.DefaultGraphWalker.dispatch(DefaultGraphWalker.java:78)
> at
> org.apache.hadoop.hive.ql.lib.DefaultGraphWalker.walk(DefaultGraphWalker.java:132)
> at
> org.apache.hadoop.hive.ql.lib.DefaultGraphWalker.startWalking(DefaultGraphWalker.java:109)
> at
> org.apache.hadoop.hive.ql.optimizer.SortedMergeBucketMapJoinOptimizer.transform(SortedMergeBucketMapJoinOptimizer.java:109)
> at org.apache.hadoop.hive.ql.optimizer.Optimizer.optimize(Optimizer.java:146)
> at
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeInternal(SemanticAnalyzer.java:9305)
> at
> org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:327)
> at
> org.apache.hadoop.hive.ql.parse.ExplainSemanticAnalyzer.analyzeInternal(ExplainSemanticAnalyzer.java:64)
> at
> org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:327)
> at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:393)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-7694) SMB join on tables differing by number of sorted by columns with same join prefix fails
[
https://issues.apache.org/jira/browse/HIVE-7694?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Amareshwari Sriramadasu updated HIVE-7694:
--
Release Note: SMB join on tables differing by number of sorted by columns
with same join prefix (was: I just committed this. Thanks Suma!)
> SMB join on tables differing by number of sorted by columns with same join
> prefix fails
> ---
>
> Key: HIVE-7694
> URL: https://issues.apache.org/jira/browse/HIVE-7694
> Project: Hive
> Issue Type: Bug
> Components: Query Processor
>Affects Versions: 0.13.1
>Reporter: Suma Shivaprasad
>Assignee: Suma Shivaprasad
> Fix For: 0.14.0
>
> Attachments: HIVE-7694.1.patch, HIVE-7694.2.patch, HIVE-7694.patch
>
>
> For eg: If two tables T1 sorted by (a, b, c) clustered by a and T2 sorted by
> (a) and clustered by (a) are joined, the following exception is seen
> {noformat}
> 14/08/11 09:09:38 ERROR ql.Driver: FAILED: IndexOutOfBoundsException Index:
> 1, Size: 1
> java.lang.IndexOutOfBoundsException: Index: 1, Size: 1
> at java.util.ArrayList.RangeCheck(ArrayList.java:547)
> at java.util.ArrayList.get(ArrayList.java:322)
> at
> org.apache.hadoop.hive.ql.optimizer.AbstractSMBJoinProc.checkSortColsAndJoinCols(AbstractSMBJoinProc.java:378)
> at
> org.apache.hadoop.hive.ql.optimizer.AbstractSMBJoinProc.isEligibleForBucketSortMergeJoin(AbstractSMBJoinProc.java:352)
> at
> org.apache.hadoop.hive.ql.optimizer.AbstractSMBJoinProc.canConvertBucketMapJoinToSMBJoin(AbstractSMBJoinProc.java:119)
> at
> org.apache.hadoop.hive.ql.optimizer.SortedMergeBucketMapjoinProc.process(SortedMergeBucketMapjoinProc.java:51)
> at
> org.apache.hadoop.hive.ql.lib.DefaultRuleDispatcher.dispatch(DefaultRuleDispatcher.java:90)
> at
> org.apache.hadoop.hive.ql.lib.DefaultGraphWalker.dispatchAndReturn(DefaultGraphWalker.java:94)
> at
> org.apache.hadoop.hive.ql.lib.DefaultGraphWalker.dispatch(DefaultGraphWalker.java:78)
> at
> org.apache.hadoop.hive.ql.lib.DefaultGraphWalker.walk(DefaultGraphWalker.java:132)
> at
> org.apache.hadoop.hive.ql.lib.DefaultGraphWalker.startWalking(DefaultGraphWalker.java:109)
> at
> org.apache.hadoop.hive.ql.optimizer.SortedMergeBucketMapJoinOptimizer.transform(SortedMergeBucketMapJoinOptimizer.java:109)
> at org.apache.hadoop.hive.ql.optimizer.Optimizer.optimize(Optimizer.java:146)
> at
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeInternal(SemanticAnalyzer.java:9305)
> at
> org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:327)
> at
> org.apache.hadoop.hive.ql.parse.ExplainSemanticAnalyzer.analyzeInternal(ExplainSemanticAnalyzer.java:64)
> at
> org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:327)
> at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:393)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-7694) SMB join on tables differing by number of sorted by columns with same join prefix fails
[
https://issues.apache.org/jira/browse/HIVE-7694?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Amareshwari Sriramadasu updated HIVE-7694:
--
Resolution: Fixed
Release Note: I just committed this. Thanks Suma!
Hadoop Flags: Reviewed
Status: Resolved (was: Patch Available)
> SMB join on tables differing by number of sorted by columns with same join
> prefix fails
> ---
>
> Key: HIVE-7694
> URL: https://issues.apache.org/jira/browse/HIVE-7694
> Project: Hive
> Issue Type: Bug
> Components: Query Processor
>Affects Versions: 0.13.1
>Reporter: Suma Shivaprasad
>Assignee: Suma Shivaprasad
> Fix For: 0.14.0
>
> Attachments: HIVE-7694.1.patch, HIVE-7694.2.patch, HIVE-7694.patch
>
>
> For eg: If two tables T1 sorted by (a, b, c) clustered by a and T2 sorted by
> (a) and clustered by (a) are joined, the following exception is seen
> {noformat}
> 14/08/11 09:09:38 ERROR ql.Driver: FAILED: IndexOutOfBoundsException Index:
> 1, Size: 1
> java.lang.IndexOutOfBoundsException: Index: 1, Size: 1
> at java.util.ArrayList.RangeCheck(ArrayList.java:547)
> at java.util.ArrayList.get(ArrayList.java:322)
> at
> org.apache.hadoop.hive.ql.optimizer.AbstractSMBJoinProc.checkSortColsAndJoinCols(AbstractSMBJoinProc.java:378)
> at
> org.apache.hadoop.hive.ql.optimizer.AbstractSMBJoinProc.isEligibleForBucketSortMergeJoin(AbstractSMBJoinProc.java:352)
> at
> org.apache.hadoop.hive.ql.optimizer.AbstractSMBJoinProc.canConvertBucketMapJoinToSMBJoin(AbstractSMBJoinProc.java:119)
> at
> org.apache.hadoop.hive.ql.optimizer.SortedMergeBucketMapjoinProc.process(SortedMergeBucketMapjoinProc.java:51)
> at
> org.apache.hadoop.hive.ql.lib.DefaultRuleDispatcher.dispatch(DefaultRuleDispatcher.java:90)
> at
> org.apache.hadoop.hive.ql.lib.DefaultGraphWalker.dispatchAndReturn(DefaultGraphWalker.java:94)
> at
> org.apache.hadoop.hive.ql.lib.DefaultGraphWalker.dispatch(DefaultGraphWalker.java:78)
> at
> org.apache.hadoop.hive.ql.lib.DefaultGraphWalker.walk(DefaultGraphWalker.java:132)
> at
> org.apache.hadoop.hive.ql.lib.DefaultGraphWalker.startWalking(DefaultGraphWalker.java:109)
> at
> org.apache.hadoop.hive.ql.optimizer.SortedMergeBucketMapJoinOptimizer.transform(SortedMergeBucketMapJoinOptimizer.java:109)
> at org.apache.hadoop.hive.ql.optimizer.Optimizer.optimize(Optimizer.java:146)
> at
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeInternal(SemanticAnalyzer.java:9305)
> at
> org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:327)
> at
> org.apache.hadoop.hive.ql.parse.ExplainSemanticAnalyzer.analyzeInternal(ExplainSemanticAnalyzer.java:64)
> at
> org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:327)
> at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:393)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
Re: Review Request 24630: HIVE-7694 - SMB joins on tables differing by number of sorted by columns but same sort prefix and join keys fail
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/24630/#review52695 --- Ship it! Ship It! - Amareshwari Sriramadasu On Sept. 8, 2014, 5:25 p.m., Suma Shivaprasad wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/24630/ > --- > > (Updated Sept. 8, 2014, 5:25 p.m.) > > > Review request for hive, Amareshwari Sriramadasu, Brock Noland, Gunther > Hagleitner, and Navis Ryu. > > > Bugs: HIVE-7694 > https://issues.apache.org/jira/browse/HIVE-7694 > > > Repository: hive-git > > > Description > --- > > For eg: If two tables T1 sorted by (a, b, c) clustered by a and T2 sorted by > (a) and clustered by (a) are joined, an exception is seen as reported in > https://issues.apache.org/jira/browse/HIVE-7694 > > > Diffs > - > > ql/src/java/org/apache/hadoop/hive/ql/optimizer/AbstractSMBJoinProc.java > 0b7b1a3 > ql/src/test/queries/clientpositive/sort_merge_join_desc_8.q PRE-CREATION > ql/src/test/results/clientpositive/sort_merge_join_desc_8.q.out > PRE-CREATION > > Diff: https://reviews.apache.org/r/24630/diff/ > > > Testing > --- > > sort_merge_join_desc_8.q added for testing the above cases > > > Thanks, > > Suma Shivaprasad > >
[jira] [Commented] (HIVE-7694) SMB join on tables differing by number of sorted by columns with same join prefix fails
[
https://issues.apache.org/jira/browse/HIVE-7694?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14126785#comment-14126785
]
Amareshwari Sriramadasu commented on HIVE-7694:
---
+1 Code changes look fine to me.
[~suma.shivaprasad], Can you rebase the patch? Also run tests once again as the
last test build was having test failures.
Make sure failed tests are not failing on your local machine before submitting
again
> SMB join on tables differing by number of sorted by columns with same join
> prefix fails
> ---
>
> Key: HIVE-7694
> URL: https://issues.apache.org/jira/browse/HIVE-7694
> Project: Hive
> Issue Type: Bug
> Components: Query Processor
>Affects Versions: 0.13.1
>Reporter: Suma Shivaprasad
>Assignee: Suma Shivaprasad
> Fix For: 0.14.0
>
> Attachments: HIVE-7694.1.patch, HIVE-7694.patch
>
>
> For eg: If two tables T1 sorted by (a, b, c) clustered by a and T2 sorted by
> (a) and clustered by (a) are joined, the following exception is seen
> {noformat}
> 14/08/11 09:09:38 ERROR ql.Driver: FAILED: IndexOutOfBoundsException Index:
> 1, Size: 1
> java.lang.IndexOutOfBoundsException: Index: 1, Size: 1
> at java.util.ArrayList.RangeCheck(ArrayList.java:547)
> at java.util.ArrayList.get(ArrayList.java:322)
> at
> org.apache.hadoop.hive.ql.optimizer.AbstractSMBJoinProc.checkSortColsAndJoinCols(AbstractSMBJoinProc.java:378)
> at
> org.apache.hadoop.hive.ql.optimizer.AbstractSMBJoinProc.isEligibleForBucketSortMergeJoin(AbstractSMBJoinProc.java:352)
> at
> org.apache.hadoop.hive.ql.optimizer.AbstractSMBJoinProc.canConvertBucketMapJoinToSMBJoin(AbstractSMBJoinProc.java:119)
> at
> org.apache.hadoop.hive.ql.optimizer.SortedMergeBucketMapjoinProc.process(SortedMergeBucketMapjoinProc.java:51)
> at
> org.apache.hadoop.hive.ql.lib.DefaultRuleDispatcher.dispatch(DefaultRuleDispatcher.java:90)
> at
> org.apache.hadoop.hive.ql.lib.DefaultGraphWalker.dispatchAndReturn(DefaultGraphWalker.java:94)
> at
> org.apache.hadoop.hive.ql.lib.DefaultGraphWalker.dispatch(DefaultGraphWalker.java:78)
> at
> org.apache.hadoop.hive.ql.lib.DefaultGraphWalker.walk(DefaultGraphWalker.java:132)
> at
> org.apache.hadoop.hive.ql.lib.DefaultGraphWalker.startWalking(DefaultGraphWalker.java:109)
> at
> org.apache.hadoop.hive.ql.optimizer.SortedMergeBucketMapJoinOptimizer.transform(SortedMergeBucketMapJoinOptimizer.java:109)
> at org.apache.hadoop.hive.ql.optimizer.Optimizer.optimize(Optimizer.java:146)
> at
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeInternal(SemanticAnalyzer.java:9305)
> at
> org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:327)
> at
> org.apache.hadoop.hive.ql.parse.ExplainSemanticAnalyzer.analyzeInternal(ExplainSemanticAnalyzer.java:64)
> at
> org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:327)
> at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:393)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-2390) Expand support for union types
[
https://issues.apache.org/jira/browse/HIVE-2390?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Amareshwari Sriramadasu updated HIVE-2390:
--
Resolution: Fixed
Release Note: Adds UnionType support in LazyBinarySerde
Hadoop Flags: Reviewed
Status: Resolved (was: Patch Available)
I just committed this. Thanks Suma!
> Expand support for union types
> --
>
> Key: HIVE-2390
> URL: https://issues.apache.org/jira/browse/HIVE-2390
> Project: Hive
> Issue Type: Bug
>Affects Versions: 0.13.1
>Reporter: Jakob Homan
>Assignee: Suma Shivaprasad
> Labels: uniontype
> Fix For: 0.14.0
>
> Attachments: HIVE-2390.1.patch, HIVE-2390.patch
>
>
> When the union type was introduced, full support for it wasn't provided. For
> instance, when working with a union that gets passed to LazyBinarySerde:
> {noformat}Caused by: java.lang.RuntimeException: Unrecognized type: UNION
> at
> org.apache.hadoop.hive.serde2.lazybinary.LazyBinarySerDe.serialize(LazyBinarySerDe.java:468)
> at
> org.apache.hadoop.hive.serde2.lazybinary.LazyBinarySerDe.serializeStruct(LazyBinarySerDe.java:230)
> at
> org.apache.hadoop.hive.serde2.lazybinary.LazyBinarySerDe.serialize(LazyBinarySerDe.java:184)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-2390) Expand support for union types
[
https://issues.apache.org/jira/browse/HIVE-2390?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14125210#comment-14125210
]
Amareshwari Sriramadasu commented on HIVE-2390:
---
+1 Changes look fine to me.
[~suma.shivaprasad], Test failure seems unrelated to me. Can you look into and
confirm?
> Expand support for union types
> --
>
> Key: HIVE-2390
> URL: https://issues.apache.org/jira/browse/HIVE-2390
> Project: Hive
> Issue Type: Bug
>Affects Versions: 0.13.1
>Reporter: Jakob Homan
>Assignee: Suma Shivaprasad
> Labels: uniontype
> Fix For: 0.14.0
>
> Attachments: HIVE-2390.1.patch, HIVE-2390.patch
>
>
> When the union type was introduced, full support for it wasn't provided. For
> instance, when working with a union that gets passed to LazyBinarySerde:
> {noformat}Caused by: java.lang.RuntimeException: Unrecognized type: UNION
> at
> org.apache.hadoop.hive.serde2.lazybinary.LazyBinarySerDe.serialize(LazyBinarySerDe.java:468)
> at
> org.apache.hadoop.hive.serde2.lazybinary.LazyBinarySerDe.serializeStruct(LazyBinarySerDe.java:230)
> at
> org.apache.hadoop.hive.serde2.lazybinary.LazyBinarySerDe.serialize(LazyBinarySerDe.java:184)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-7629) Problem in SMB Joins between two Parquet tables
[
https://issues.apache.org/jira/browse/HIVE-7629?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Amareshwari Sriramadasu updated HIVE-7629:
--
Assignee: Suma Shivaprasad
> Problem in SMB Joins between two Parquet tables
> ---
>
> Key: HIVE-7629
> URL: https://issues.apache.org/jira/browse/HIVE-7629
> Project: Hive
> Issue Type: Bug
> Components: Serializers/Deserializers
>Affects Versions: 0.13.0
>Reporter: Suma Shivaprasad
>Assignee: Suma Shivaprasad
> Labels: Parquet
> Fix For: 0.14.0
>
> Attachments: HIVE-7629.1.patch, HIVE-7629.patch
>
>
> The issue is clearly seen when two bucketed and sorted parquet tables with
> different number of columns are involved in the join . The following
> exception is seen
> {noformat}
> Caused by: java.lang.IndexOutOfBoundsException: Index: 2, Size: 2
> at java.util.ArrayList.rangeCheck(ArrayList.java:635)
> at java.util.ArrayList.get(ArrayList.java:411)
> at
> org.apache.hadoop.hive.ql.io.parquet.read.DataWritableReadSupport.init(DataWritableReadSupport.java:101)
> at
> org.apache.hadoop.hive.ql.io.parquet.read.ParquetRecordReaderWrapper.getSplit(ParquetRecordReaderWrapper.java:204)
> at
> org.apache.hadoop.hive.ql.io.parquet.read.ParquetRecordReaderWrapper.(ParquetRecordReaderWrapper.java:79)
> at
> org.apache.hadoop.hive.ql.io.parquet.read.ParquetRecordReaderWrapper.(ParquetRecordReaderWrapper.java:66)
> at
> org.apache.hadoop.hive.ql.io.parquet.MapredParquetInputFormat.getRecordReader(MapredParquetInputFormat.java:51)
> at
> org.apache.hadoop.hive.ql.io.CombineHiveRecordReader.(CombineHiveRecordReader.java:65)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Updated] (HIVE-7694) SMB join on tables differing by number of sorted by columns with same join prefix fails
[
https://issues.apache.org/jira/browse/HIVE-7694?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Amareshwari Sriramadasu updated HIVE-7694:
--
Assignee: Suma Shivaprasad
> SMB join on tables differing by number of sorted by columns with same join
> prefix fails
> ---
>
> Key: HIVE-7694
> URL: https://issues.apache.org/jira/browse/HIVE-7694
> Project: Hive
> Issue Type: Bug
> Components: Query Processor
>Affects Versions: 0.13.1
>Reporter: Suma Shivaprasad
>Assignee: Suma Shivaprasad
> Fix For: 0.14.0
>
> Attachments: HIVE-7694.1.patch, HIVE-7694.patch
>
>
> For eg: If two tables T1 sorted by (a, b, c) clustered by a and T2 sorted by
> (a) and clustered by (a) are joined, the following exception is seen
> {noformat}
> 14/08/11 09:09:38 ERROR ql.Driver: FAILED: IndexOutOfBoundsException Index:
> 1, Size: 1
> java.lang.IndexOutOfBoundsException: Index: 1, Size: 1
> at java.util.ArrayList.RangeCheck(ArrayList.java:547)
> at java.util.ArrayList.get(ArrayList.java:322)
> at
> org.apache.hadoop.hive.ql.optimizer.AbstractSMBJoinProc.checkSortColsAndJoinCols(AbstractSMBJoinProc.java:378)
> at
> org.apache.hadoop.hive.ql.optimizer.AbstractSMBJoinProc.isEligibleForBucketSortMergeJoin(AbstractSMBJoinProc.java:352)
> at
> org.apache.hadoop.hive.ql.optimizer.AbstractSMBJoinProc.canConvertBucketMapJoinToSMBJoin(AbstractSMBJoinProc.java:119)
> at
> org.apache.hadoop.hive.ql.optimizer.SortedMergeBucketMapjoinProc.process(SortedMergeBucketMapjoinProc.java:51)
> at
> org.apache.hadoop.hive.ql.lib.DefaultRuleDispatcher.dispatch(DefaultRuleDispatcher.java:90)
> at
> org.apache.hadoop.hive.ql.lib.DefaultGraphWalker.dispatchAndReturn(DefaultGraphWalker.java:94)
> at
> org.apache.hadoop.hive.ql.lib.DefaultGraphWalker.dispatch(DefaultGraphWalker.java:78)
> at
> org.apache.hadoop.hive.ql.lib.DefaultGraphWalker.walk(DefaultGraphWalker.java:132)
> at
> org.apache.hadoop.hive.ql.lib.DefaultGraphWalker.startWalking(DefaultGraphWalker.java:109)
> at
> org.apache.hadoop.hive.ql.optimizer.SortedMergeBucketMapJoinOptimizer.transform(SortedMergeBucketMapJoinOptimizer.java:109)
> at org.apache.hadoop.hive.ql.optimizer.Optimizer.optimize(Optimizer.java:146)
> at
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeInternal(SemanticAnalyzer.java:9305)
> at
> org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:327)
> at
> org.apache.hadoop.hive.ql.parse.ExplainSemanticAnalyzer.analyzeInternal(ExplainSemanticAnalyzer.java:64)
> at
> org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:327)
> at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:393)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Updated] (HIVE-5733) Publish hive-exec artifact without all the dependencies
[
https://issues.apache.org/jira/browse/HIVE-5733?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Amareshwari Sriramadasu updated HIVE-5733:
--
Status: Open (was: Patch Available)
There are problems with uploaded patch, canceling
> Publish hive-exec artifact without all the dependencies
> ---
>
> Key: HIVE-5733
> URL: https://issues.apache.org/jira/browse/HIVE-5733
> Project: Hive
> Issue Type: Bug
>Affects Versions: 0.12.0
>Reporter: Jarek Jarcec Cecho
> Assignee: Amareshwari Sriramadasu
> Attachments: HIVE-5733.1.patch
>
>
> Currently the artifact {{hive-exec}} that is available in
> [maven|http://search.maven.org/remotecontent?filepath=org/apache/hive/hive-exec/0.12.0/hive-exec-0.12.0.jar]
> is shading all the dependencies (= the jar contains all Hive's
> dependencies). As other projects that are depending on Hive might be use
> slightly different version of the dependencies, it can easily happens that
> Hive's shaded version will be used instead which leads to very time consuming
> debugging of what is happening (for example SQOOP-1198).
> Would it be feasible publish {{hive-exec}} jar that will be build without
> shading any dependency? For example
> [avro-tools|http://search.maven.org/#artifactdetails%7Corg.apache.avro%7Cavro-tools%7C1.7.5%7Cjar]
> is having classifier "nodeps" that represents artifact without any
> dependencies.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Updated] (HIVE-5733) Publish hive-exec artifact without all the dependencies
[
https://issues.apache.org/jira/browse/HIVE-5733?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Amareshwari Sriramadasu updated HIVE-5733:
--
Attachment: (was: HIVE-5733.1.patch)
> Publish hive-exec artifact without all the dependencies
> ---
>
> Key: HIVE-5733
> URL: https://issues.apache.org/jira/browse/HIVE-5733
> Project: Hive
> Issue Type: Bug
>Affects Versions: 0.12.0
>Reporter: Jarek Jarcec Cecho
> Assignee: Amareshwari Sriramadasu
> Attachments: HIVE-5733.1.patch
>
>
> Currently the artifact {{hive-exec}} that is available in
> [maven|http://search.maven.org/remotecontent?filepath=org/apache/hive/hive-exec/0.12.0/hive-exec-0.12.0.jar]
> is shading all the dependencies (= the jar contains all Hive's
> dependencies). As other projects that are depending on Hive might be use
> slightly different version of the dependencies, it can easily happens that
> Hive's shaded version will be used instead which leads to very time consuming
> debugging of what is happening (for example SQOOP-1198).
> Would it be feasible publish {{hive-exec}} jar that will be build without
> shading any dependency? For example
> [avro-tools|http://search.maven.org/#artifactdetails%7Corg.apache.avro%7Cavro-tools%7C1.7.5%7Cjar]
> is having classifier "nodeps" that represents artifact without any
> dependencies.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Updated] (HIVE-5733) Publish hive-exec artifact without all the dependencies
[
https://issues.apache.org/jira/browse/HIVE-5733?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Amareshwari Sriramadasu updated HIVE-5733:
--
Attachment: HIVE-5733.1.patch
As per the documentation of maven-shade-plugin the plugin will replace the
project's main artifact with the shaded artifact. Giving a different classifier
like nodep also was not helpful. As per the doc, the shaded artifact can be
given a different name. Doc -
http://maven.apache.org/plugins/maven-shade-plugin/examples/attached-artifact.html
Attaching the patch which generates hive-exec-.jar with no
dependencies and hive-exec-.-withdep.jar as the shaded jar.
> Publish hive-exec artifact without all the dependencies
> ---
>
> Key: HIVE-5733
> URL: https://issues.apache.org/jira/browse/HIVE-5733
> Project: Hive
> Issue Type: Bug
>Affects Versions: 0.12.0
>Reporter: Jarek Jarcec Cecho
> Assignee: Amareshwari Sriramadasu
> Attachments: HIVE-5733.1.patch
>
>
> Currently the artifact {{hive-exec}} that is available in
> [maven|http://search.maven.org/remotecontent?filepath=org/apache/hive/hive-exec/0.12.0/hive-exec-0.12.0.jar]
> is shading all the dependencies (= the jar contains all Hive's
> dependencies). As other projects that are depending on Hive might be use
> slightly different version of the dependencies, it can easily happens that
> Hive's shaded version will be used instead which leads to very time consuming
> debugging of what is happening (for example SQOOP-1198).
> Would it be feasible publish {{hive-exec}} jar that will be build without
> shading any dependency? For example
> [avro-tools|http://search.maven.org/#artifactdetails%7Corg.apache.avro%7Cavro-tools%7C1.7.5%7Cjar]
> is having classifier "nodeps" that represents artifact without any
> dependencies.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Updated] (HIVE-5733) Publish hive-exec artifact without all the dependencies
[
https://issues.apache.org/jira/browse/HIVE-5733?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Amareshwari Sriramadasu updated HIVE-5733:
--
Attachment: HIVE-5733.1.patch
> Publish hive-exec artifact without all the dependencies
> ---
>
> Key: HIVE-5733
> URL: https://issues.apache.org/jira/browse/HIVE-5733
> Project: Hive
> Issue Type: Bug
>Affects Versions: 0.12.0
>Reporter: Jarek Jarcec Cecho
> Assignee: Amareshwari Sriramadasu
> Attachments: HIVE-5733.1.patch
>
>
> Currently the artifact {{hive-exec}} that is available in
> [maven|http://search.maven.org/remotecontent?filepath=org/apache/hive/hive-exec/0.12.0/hive-exec-0.12.0.jar]
> is shading all the dependencies (= the jar contains all Hive's
> dependencies). As other projects that are depending on Hive might be use
> slightly different version of the dependencies, it can easily happens that
> Hive's shaded version will be used instead which leads to very time consuming
> debugging of what is happening (for example SQOOP-1198).
> Would it be feasible publish {{hive-exec}} jar that will be build without
> shading any dependency? For example
> [avro-tools|http://search.maven.org/#artifactdetails%7Corg.apache.avro%7Cavro-tools%7C1.7.5%7Cjar]
> is having classifier "nodeps" that represents artifact without any
> dependencies.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Updated] (HIVE-5733) Publish hive-exec artifact without all the dependencies
[
https://issues.apache.org/jira/browse/HIVE-5733?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Amareshwari Sriramadasu updated HIVE-5733:
--
Status: Patch Available (was: Open)
> Publish hive-exec artifact without all the dependencies
> ---
>
> Key: HIVE-5733
> URL: https://issues.apache.org/jira/browse/HIVE-5733
> Project: Hive
> Issue Type: Bug
>Affects Versions: 0.12.0
>Reporter: Jarek Jarcec Cecho
> Assignee: Amareshwari Sriramadasu
> Attachments: HIVE-5733.1.patch
>
>
> Currently the artifact {{hive-exec}} that is available in
> [maven|http://search.maven.org/remotecontent?filepath=org/apache/hive/hive-exec/0.12.0/hive-exec-0.12.0.jar]
> is shading all the dependencies (= the jar contains all Hive's
> dependencies). As other projects that are depending on Hive might be use
> slightly different version of the dependencies, it can easily happens that
> Hive's shaded version will be used instead which leads to very time consuming
> debugging of what is happening (for example SQOOP-1198).
> Would it be feasible publish {{hive-exec}} jar that will be build without
> shading any dependency? For example
> [avro-tools|http://search.maven.org/#artifactdetails%7Corg.apache.avro%7Cavro-tools%7C1.7.5%7Cjar]
> is having classifier "nodeps" that represents artifact without any
> dependencies.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
Review Request 21479: HIVE-5733 Publish hive-exec artifact without all the dependencies
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/21479/ --- Review request for hive, Ashutosh Chauhan and Navis Ryu. Bugs: HIVE-5733 https://issues.apache.org/jira/browse/HIVE-5733 Repository: hive-git Description --- Generates hive-exec-.jar with no dependencies and hive-exec-.-withdep.jar as the shaded jar. Diffs - packaging/pom.xml 118037b ql/pom.xml 71daa26 Diff: https://reviews.apache.org/r/21479/diff/ Testing --- Thanks, Amareshwari Sriramadasu
[jira] [Assigned] (HIVE-5733) Publish hive-exec artifact without all the dependencies
[
https://issues.apache.org/jira/browse/HIVE-5733?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Amareshwari Sriramadasu reassigned HIVE-5733:
-
Assignee: Amareshwari Sriramadasu
> Publish hive-exec artifact without all the dependencies
> ---
>
> Key: HIVE-5733
> URL: https://issues.apache.org/jira/browse/HIVE-5733
> Project: Hive
> Issue Type: Bug
>Affects Versions: 0.12.0
>Reporter: Jarek Jarcec Cecho
> Assignee: Amareshwari Sriramadasu
>
> Currently the artifact {{hive-exec}} that is available in
> [maven|http://search.maven.org/remotecontent?filepath=org/apache/hive/hive-exec/0.12.0/hive-exec-0.12.0.jar]
> is shading all the dependencies (= the jar contains all Hive's
> dependencies). As other projects that are depending on Hive might be use
> slightly different version of the dependencies, it can easily happens that
> Hive's shaded version will be used instead which leads to very time consuming
> debugging of what is happening (for example SQOOP-1198).
> Would it be feasible publish {{hive-exec}} jar that will be build without
> shading any dependency? For example
> [avro-tools|http://search.maven.org/#artifactdetails%7Corg.apache.avro%7Cavro-tools%7C1.7.5%7Cjar]
> is having classifier "nodeps" that represents artifact without any
> dependencies.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HIVE-5733) Publish hive-exec artifact without all the dependencies
[
https://issues.apache.org/jira/browse/HIVE-5733?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13997276#comment-13997276
]
Amareshwari Sriramadasu commented on HIVE-5733:
---
+1 This is much required.
I agree it has become difficult to depend on hive exec jar, because of ql
module shading all the dependencies.
I will try to put a patch.
> Publish hive-exec artifact without all the dependencies
> ---
>
> Key: HIVE-5733
> URL: https://issues.apache.org/jira/browse/HIVE-5733
> Project: Hive
> Issue Type: Bug
>Affects Versions: 0.12.0
>Reporter: Jarek Jarcec Cecho
>
> Currently the artifact {{hive-exec}} that is available in
> [maven|http://search.maven.org/remotecontent?filepath=org/apache/hive/hive-exec/0.12.0/hive-exec-0.12.0.jar]
> is shading all the dependencies (= the jar contains all Hive's
> dependencies). As other projects that are depending on Hive might be use
> slightly different version of the dependencies, it can easily happens that
> Hive's shaded version will be used instead which leads to very time consuming
> debugging of what is happening (for example SQOOP-1198).
> Would it be feasible publish {{hive-exec}} jar that will be build without
> shading any dependency? For example
> [avro-tools|http://search.maven.org/#artifactdetails%7Corg.apache.avro%7Cavro-tools%7C1.7.5%7Cjar]
> is having classifier "nodeps" that represents artifact without any
> dependencies.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Resolved] (HIVE-6953) All CompactorTest failing with Table/View 'NEXT_TXN_ID' does not exist
[
https://issues.apache.org/jira/browse/HIVE-6953?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Amareshwari Sriramadasu resolved HIVE-6953.
---
Resolution: Duplicate
The test failures were because of HIVE-6877. After applying the patch from
HIVE-6877 into branch-0.13, all the tests are passing
> All CompactorTest failing with Table/View 'NEXT_TXN_ID' does not exist
> --
>
> Key: HIVE-6953
> URL: https://issues.apache.org/jira/browse/HIVE-6953
> Project: Hive
> Issue Type: Bug
> Components: Query Processor
>Affects Versions: 0.13.0
> Reporter: Amareshwari Sriramadasu
>Assignee: Alan Gates
> Attachments:
> TEST-org.apache.hadoop.hive.ql.txn.compactor.TestCleaner.xml,
> TEST-org.apache.hadoop.hive.ql.txn.compactor.TestInitiator.xml,
> TEST-org.apache.hadoop.hive.ql.txn.compactor.TestWorker.xml, nohup.out.gz
>
>
> When I'm running all tests through the command 'mvn clean install
> -Phadoop-1', all CompactorTest classes TestInitiator, TestWorker, TestCleaner
> fail with following exception :
> {noformat}
> org.apache.hadoop.hive.metastore.api.MetaException: Unable to select from
> transaction database java.sql.SQLSyntaxErrorException: Table/View
> 'NEXT_TXN_ID' does not exist.
> at
> org.apache.derby.impl.jdbc.SQLExceptionFactory40.getSQLException(Unknown
> Source)
> at org.apache.derby.impl.jdbc.Util.generateCsSQLException(Unknown
> Source)
> at
> org.apache.derby.impl.jdbc.TransactionResourceImpl.wrapInSQLException(Unknown
> Source)
> at
> org.apache.derby.impl.jdbc.TransactionResourceImpl.handleException(Unknown
> Source)
>
> Caused by: java.sql.SQLException: Table/View 'NEXT_TXN_ID' does not exist.
> at
> org.apache.derby.impl.jdbc.SQLExceptionFactory.getSQLException(Unknown Source)
> at
> org.apache.derby.impl.jdbc.SQLExceptionFactory40.wrapArgsForTransportAcrossDRDA(Unknown
> Source)
> {noformat}
> This is happening on branch-0.13. Has anyone faced this problem?
> [~owen.omalley] or someone else help me solve this. Do i have to set anything?
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Comment Edited] (HIVE-6953) All CompactorTest failing with Table/View 'NEXT_TXN_ID' does not exist
[
https://issues.apache.org/jira/browse/HIVE-6953?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13979312#comment-13979312
]
Amareshwari Sriramadasu edited comment on HIVE-6953 at 4/24/14 5:47 AM:
Also, there are some tests failing randomly because they fail to create path in
/user/hive/warehouse
For ex: org.apache.hadoop.hive.ql.io.TestSymlinkTextInputFormat failed with
following errors
{noformat}
log4j:ERROR Could not read configuration file from URL
[file://ql/target/tmp/conf/hive-log4j.properties].
java.io.FileNotFoundException:
//ql/target/tmp/conf/hive-log4j.properties (No such file or
directory)
FAILED: Execution Error, return code 1 from
org.apache.hadoop.hive.ql.exec.DDLTask.
MetaException(message:file:/user/hive/warehouse/text_symlink_text is not a
directory or unable to create one)
{noformat}
So, most probably something is getting cleaned-up. But I could not find where
to start.
Can someone help me to find the root cause? Where can i start to look at it.
was (Author: amareshwari):
Also, there are some tests failing randomly because they fail to create path in
/user/hive/warehouse
For ex: org.apache.hadoop.hive.ql.io.TestSymlinkTextInputFormat failed with
following errors
{noformat}
log4j:ERROR Could not read configuration file from URL
[file://ql/target/tmp/conf/hive-log4j.properties].
java.io.FileNotFoundException:
//ql/target/tmp/conf/hive-log4j.properties (No such file or
directory)
FAILED: Execution Error, return code 1 from
org.apache.hadoop.hive.ql.exec.DDLTask.
MetaException(message:file:/user/hive/warehouse/text_symlink_text is not a
directory or unable to create one)
{noformat}
So, most probably something is getting cleaned-up. But I could find where to
start.
Can someone help me to find the root cause? Where can i start to look at it.
> All CompactorTest failing with Table/View 'NEXT_TXN_ID' does not exist
> --
>
> Key: HIVE-6953
> URL: https://issues.apache.org/jira/browse/HIVE-6953
> Project: Hive
> Issue Type: Bug
> Components: Query Processor
>Affects Versions: 0.13.0
> Reporter: Amareshwari Sriramadasu
>Assignee: Alan Gates
> Attachments:
> TEST-org.apache.hadoop.hive.ql.txn.compactor.TestCleaner.xml,
> TEST-org.apache.hadoop.hive.ql.txn.compactor.TestInitiator.xml,
> TEST-org.apache.hadoop.hive.ql.txn.compactor.TestWorker.xml, nohup.out.gz
>
>
> When I'm running all tests through the command 'mvn clean install
> -Phadoop-1', all CompactorTest classes TestInitiator, TestWorker, TestCleaner
> fail with following exception :
> {noformat}
> org.apache.hadoop.hive.metastore.api.MetaException: Unable to select from
> transaction database java.sql.SQLSyntaxErrorException: Table/View
> 'NEXT_TXN_ID' does not exist.
> at
> org.apache.derby.impl.jdbc.SQLExceptionFactory40.getSQLException(Unknown
> Source)
> at org.apache.derby.impl.jdbc.Util.generateCsSQLException(Unknown
> Source)
> at
> org.apache.derby.impl.jdbc.TransactionResourceImpl.wrapInSQLException(Unknown
> Source)
> at
> org.apache.derby.impl.jdbc.TransactionResourceImpl.handleException(Unknown
> Source)
>
> Caused by: java.sql.SQLException: Table/View 'NEXT_TXN_ID' does not exist.
> at
> org.apache.derby.impl.jdbc.SQLExceptionFactory.getSQLException(Unknown Source)
> at
> org.apache.derby.impl.jdbc.SQLExceptionFactory40.wrapArgsForTransportAcrossDRDA(Unknown
> Source)
> {noformat}
> This is happening on branch-0.13. Has anyone faced this problem?
> [~owen.omalley] or someone else help me solve this. Do i have to set anything?
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HIVE-6953) All CompactorTest failing with Table/View 'NEXT_TXN_ID' does not exist
[
https://issues.apache.org/jira/browse/HIVE-6953?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13979312#comment-13979312
]
Amareshwari Sriramadasu commented on HIVE-6953:
---
Also, there are some tests failing randomly because they fail to create path in
/user/hive/warehouse
For ex: org.apache.hadoop.hive.ql.io.TestSymlinkTextInputFormat failed with
following errors
{noformat}
log4j:ERROR Could not read configuration file from URL
[file://ql/target/tmp/conf/hive-log4j.properties].
java.io.FileNotFoundException:
//ql/target/tmp/conf/hive-log4j.properties (No such file or
directory)
FAILED: Execution Error, return code 1 from
org.apache.hadoop.hive.ql.exec.DDLTask.
MetaException(message:file:/user/hive/warehouse/text_symlink_text is not a
directory or unable to create one)
{noformat}
So, most probably something is getting cleaned-up. But I could find where to
start.
Can someone help me to find the root cause? Where can i start to look at it.
> All CompactorTest failing with Table/View 'NEXT_TXN_ID' does not exist
> --
>
> Key: HIVE-6953
> URL: https://issues.apache.org/jira/browse/HIVE-6953
> Project: Hive
> Issue Type: Bug
> Components: Query Processor
>Affects Versions: 0.13.0
> Reporter: Amareshwari Sriramadasu
>Assignee: Alan Gates
> Attachments:
> TEST-org.apache.hadoop.hive.ql.txn.compactor.TestCleaner.xml,
> TEST-org.apache.hadoop.hive.ql.txn.compactor.TestInitiator.xml,
> TEST-org.apache.hadoop.hive.ql.txn.compactor.TestWorker.xml, nohup.out.gz
>
>
> When I'm running all tests through the command 'mvn clean install
> -Phadoop-1', all CompactorTest classes TestInitiator, TestWorker, TestCleaner
> fail with following exception :
> {noformat}
> org.apache.hadoop.hive.metastore.api.MetaException: Unable to select from
> transaction database java.sql.SQLSyntaxErrorException: Table/View
> 'NEXT_TXN_ID' does not exist.
> at
> org.apache.derby.impl.jdbc.SQLExceptionFactory40.getSQLException(Unknown
> Source)
> at org.apache.derby.impl.jdbc.Util.generateCsSQLException(Unknown
> Source)
> at
> org.apache.derby.impl.jdbc.TransactionResourceImpl.wrapInSQLException(Unknown
> Source)
> at
> org.apache.derby.impl.jdbc.TransactionResourceImpl.handleException(Unknown
> Source)
>
> Caused by: java.sql.SQLException: Table/View 'NEXT_TXN_ID' does not exist.
> at
> org.apache.derby.impl.jdbc.SQLExceptionFactory.getSQLException(Unknown Source)
> at
> org.apache.derby.impl.jdbc.SQLExceptionFactory40.wrapArgsForTransportAcrossDRDA(Unknown
> Source)
> {noformat}
> This is happening on branch-0.13. Has anyone faced this problem?
> [~owen.omalley] or someone else help me solve this. Do i have to set anything?
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Updated] (HIVE-6953) All CompactorTest failing with Table/View 'NEXT_TXN_ID' does not exist
[
https://issues.apache.org/jira/browse/HIVE-6953?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Amareshwari Sriramadasu updated HIVE-6953:
--
Attachment: TEST-org.apache.hadoop.hive.ql.txn.compactor.TestInitiator.xml
Attaching TestInitiator.xml
> All CompactorTest failing with Table/View 'NEXT_TXN_ID' does not exist
> --
>
> Key: HIVE-6953
> URL: https://issues.apache.org/jira/browse/HIVE-6953
> Project: Hive
> Issue Type: Bug
> Components: Query Processor
>Affects Versions: 0.13.0
> Reporter: Amareshwari Sriramadasu
>Assignee: Alan Gates
> Attachments:
> TEST-org.apache.hadoop.hive.ql.txn.compactor.TestInitiator.xml, nohup.out.gz
>
>
> When I'm running all tests through the command 'mvn clean install
> -Phadoop-1', all CompactorTest classes TestInitiator, TestWorker, TestCleaner
> fail with following exception :
> {noformat}
> org.apache.hadoop.hive.metastore.api.MetaException: Unable to select from
> transaction database java.sql.SQLSyntaxErrorException: Table/View
> 'NEXT_TXN_ID' does not exist.
> at
> org.apache.derby.impl.jdbc.SQLExceptionFactory40.getSQLException(Unknown
> Source)
> at org.apache.derby.impl.jdbc.Util.generateCsSQLException(Unknown
> Source)
> at
> org.apache.derby.impl.jdbc.TransactionResourceImpl.wrapInSQLException(Unknown
> Source)
> at
> org.apache.derby.impl.jdbc.TransactionResourceImpl.handleException(Unknown
> Source)
>
> Caused by: java.sql.SQLException: Table/View 'NEXT_TXN_ID' does not exist.
> at
> org.apache.derby.impl.jdbc.SQLExceptionFactory.getSQLException(Unknown Source)
> at
> org.apache.derby.impl.jdbc.SQLExceptionFactory40.wrapArgsForTransportAcrossDRDA(Unknown
> Source)
> {noformat}
> This is happening on branch-0.13. Has anyone faced this problem?
> [~owen.omalley] or someone else help me solve this. Do i have to set anything?
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HIVE-6953) All CompactorTest failing with Table/View 'NEXT_TXN_ID' does not exist
[
https://issues.apache.org/jira/browse/HIVE-6953?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13977849#comment-13977849
]
Amareshwari Sriramadasu commented on HIVE-6953:
---
The machine on which I'm running is a Linux machine. Same thing happens on Mac
as well.
{noformat}
uname -a
Linux 2.6.32-21-server #32-Ubuntu SMP Fri Apr 16 09:17:34 UTC 2010
x86_64 GNU/Linux
$ java -version
java version "1.6.0_26"
Java(TM) SE Runtime Environment (build 1.6.0_26-b03)
Java HotSpot(TM) 64-Bit Server VM (build 20.1-b02, mixed mode)
{noformat}
> All CompactorTest failing with Table/View 'NEXT_TXN_ID' does not exist
> --
>
> Key: HIVE-6953
> URL: https://issues.apache.org/jira/browse/HIVE-6953
> Project: Hive
> Issue Type: Bug
> Components: Query Processor
>Affects Versions: 0.13.0
>Reporter: Amareshwari Sriramadasu
>Assignee: Alan Gates
> Attachments: nohup.out.gz
>
>
> When I'm running all tests through the command 'mvn clean install
> -Phadoop-1', all CompactorTest classes TestInitiator, TestWorker, TestCleaner
> fail with following exception :
> {noformat}
> org.apache.hadoop.hive.metastore.api.MetaException: Unable to select from
> transaction database java.sql.SQLSyntaxErrorException: Table/View
> 'NEXT_TXN_ID' does not exist.
> at
> org.apache.derby.impl.jdbc.SQLExceptionFactory40.getSQLException(Unknown
> Source)
> at org.apache.derby.impl.jdbc.Util.generateCsSQLException(Unknown
> Source)
> at
> org.apache.derby.impl.jdbc.TransactionResourceImpl.wrapInSQLException(Unknown
> Source)
> at
> org.apache.derby.impl.jdbc.TransactionResourceImpl.handleException(Unknown
> Source)
>
> Caused by: java.sql.SQLException: Table/View 'NEXT_TXN_ID' does not exist.
> at
> org.apache.derby.impl.jdbc.SQLExceptionFactory.getSQLException(Unknown Source)
> at
> org.apache.derby.impl.jdbc.SQLExceptionFactory40.wrapArgsForTransportAcrossDRDA(Unknown
> Source)
> {noformat}
> This is happening on branch-0.13. Has anyone faced this problem?
> [~owen.omalley] or someone else help me solve this. Do i have to set anything?
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HIVE-6953) All CompactorTest failing with Table/View 'NEXT_TXN_ID' does not exist
[
https://issues.apache.org/jira/browse/HIVE-6953?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13977847#comment-13977847
]
Amareshwari Sriramadasu commented on HIVE-6953:
---
Thanks [~alangates] and [~rhbutani] for trying. The tests are passing when i
run them individually. When all the tests are run together, they are failing.
Here is what i have done :
{noformat}
git clone https://github.com/apache/hive apache-hive
git checkout branch-0.13
nohup mvn clean install -Phadoop-1 &
{noformat}
Attaching nohup output for reference.
bq. Is there anything in your logs indicating it tried to create the tables and
failed?
Will check and update.
bq. Are you doing anything in your build to turn off the hive.in.test config
value?
No.
I'm thinking the test db or conf is getting cleaned up by some-other means,
when all the tests are run together.
> All CompactorTest failing with Table/View 'NEXT_TXN_ID' does not exist
> --
>
> Key: HIVE-6953
> URL: https://issues.apache.org/jira/browse/HIVE-6953
> Project: Hive
> Issue Type: Bug
> Components: Query Processor
> Affects Versions: 0.13.0
>Reporter: Amareshwari Sriramadasu
>Assignee: Alan Gates
> Attachments: nohup.out.gz
>
>
> When I'm running all tests through the command 'mvn clean install
> -Phadoop-1', all CompactorTest classes TestInitiator, TestWorker, TestCleaner
> fail with following exception :
> {noformat}
> org.apache.hadoop.hive.metastore.api.MetaException: Unable to select from
> transaction database java.sql.SQLSyntaxErrorException: Table/View
> 'NEXT_TXN_ID' does not exist.
> at
> org.apache.derby.impl.jdbc.SQLExceptionFactory40.getSQLException(Unknown
> Source)
> at org.apache.derby.impl.jdbc.Util.generateCsSQLException(Unknown
> Source)
> at
> org.apache.derby.impl.jdbc.TransactionResourceImpl.wrapInSQLException(Unknown
> Source)
> at
> org.apache.derby.impl.jdbc.TransactionResourceImpl.handleException(Unknown
> Source)
>
> Caused by: java.sql.SQLException: Table/View 'NEXT_TXN_ID' does not exist.
> at
> org.apache.derby.impl.jdbc.SQLExceptionFactory.getSQLException(Unknown Source)
> at
> org.apache.derby.impl.jdbc.SQLExceptionFactory40.wrapArgsForTransportAcrossDRDA(Unknown
> Source)
> {noformat}
> This is happening on branch-0.13. Has anyone faced this problem?
> [~owen.omalley] or someone else help me solve this. Do i have to set anything?
--
This message was sent by Atlassian JIRA
(v6.2#6252)
