[jira] [Commented] (HIVE-15434) Add UDF to allow interrogation of uniontype values
[
https://issues.apache.org/jira/browse/HIVE-15434?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15825798#comment-15825798
]
Amareshwari Sriramadasu commented on HIVE-15434:
I had a look at the patch. Seems the new UDF is all new code and not touching
any of existing code. And there shouldn't be any issue checking this code in.
+1 from my side. UDF code also look fine.
[~hagleitn] or [~ashutoshc] Can you help have a look at new UDF if it has
taken care of practices in UDF writing?
> Add UDF to allow interrogation of uniontype values
> --
>
> Key: HIVE-15434
> URL: https://issues.apache.org/jira/browse/HIVE-15434
> Project: Hive
> Issue Type: New Feature
> Components: UDF
>Affects Versions: 2.1.1
>Reporter: David Maughan
>Assignee: David Maughan
> Attachments: HIVE-15434.01.patch, HIVE-15434.02.patch
>
>
> h2. Overview
> As stated in the documention:
> {quote}
> UNIONTYPE support is incomplete The UNIONTYPE datatype was introduced in Hive
> 0.7.0 (HIVE-537), but full support for this type in Hive remains incomplete.
> Queries that reference UNIONTYPE fields in JOIN (HIVE-2508), WHERE, and GROUP
> BY clauses will fail, and Hive does not define syntax to extract the tag or
> value fields of a UNIONTYPE. This means that UNIONTYPEs are effectively
> look-at-only.
> {quote}
> It is essential to have a usable uniontype. Until full support is added to
> Hive users should at least have the ability to inspect and extract values for
> further comparison or transformation.
> h2. Proposal
> I propose to add a GenericUDF that has 2 modes of operation. Consider the
> following schema and data that contains a union:
> Schema:
> {code}
> struct>
> {code}
> Query:
> {code}
> hive> select field1 from thing;
> {0:0}
> {1:"one"}
> {code}
> h4. Explode to Struct
> This method will recursively convert all unions within the type to structs
> with fields named {{tag_n}}, {{n}} being the tag number. Only the {{tag_*}}
> field that matches the tag of the union will be populated with the value. In
> the case above the schema of field1 will be converted to:
> {code}
> struct
> {code}
> {code}
> hive> select extract_union(field1) from thing;
> {"tag_0":0,"tag_1":null}
> {"tag_0":null,"tag_1":one}
> {code}
> {code}
> hive> select extract_union(field1).tag_0 from thing;
> 0
> null
> {code}
> h4. Extract the specified tag
> This method will simply extract the value of the specified tag. If the tag
> number matches then the value is returned, if it does not, then null is
> returned.
> {code}
> hive> select extract_union(field1, 0) from thing;
> 0
> null
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-14424) CLIRestoreTest failing in branch 2.1
[
https://issues.apache.org/jira/browse/HIVE-14424?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15407477#comment-15407477
]
Amareshwari Sriramadasu commented on HIVE-14424:
[~prongs], Seems you need to overwrite the configuration in CLIRestoreTest,
instead of changing in conf.xml
> CLIRestoreTest failing in branch 2.1
>
>
> Key: HIVE-14424
> URL: https://issues.apache.org/jira/browse/HIVE-14424
> Project: Hive
> Issue Type: Bug
>Reporter: Rajat Khandelwal
>Assignee: Rajat Khandelwal
> Fix For: 1.3.0, 2.2.0, 2.1.1
>
> Attachments: HIVE-14424.1.patch, HIVE-14424.patch
>
>
> {noformat}
> java.lang.RuntimeException: Error applying authorization policy on hive
> configuration: org.apache.hadoop.hive.ql.metadata.HiveException:
> java.lang.ClassNotFoundException:
> org.apache.hadoop.hive.ql.security.authorization.plugin.sqlstd.SQLStdHiveAuthorizerFactoryForTest
> at org.apache.hive.service.cli.CLIService.init(CLIService.java:113)
> at
> org.apache.hive.service.cli.CLIServiceRestoreTest.getService(CLIServiceRestoreTest.java:48)
> at
> org.apache.hive.service.cli.CLIServiceRestoreTest.(CLIServiceRestoreTest.java:28)
> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
> at
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
> at
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
> at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
> at
> org.junit.runners.BlockJUnit4ClassRunner.createTest(BlockJUnit4ClassRunner.java:195)
> at
> org.junit.runners.BlockJUnit4ClassRunner$1.runReflectiveCall(BlockJUnit4ClassRunner.java:244)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.BlockJUnit4ClassRunner.methodBlock(BlockJUnit4ClassRunner.java:241)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:70)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:50)
> at org.junit.runners.ParentRunner$3.run(ParentRunner.java:238)
> at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:63)
> at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:236)
> at org.junit.runners.ParentRunner.access$000(ParentRunner.java:53)
> at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:229)
> at org.junit.runners.ParentRunner.run(ParentRunner.java:309)
> at org.junit.runner.JUnitCore.run(JUnitCore.java:160)
> at
> com.intellij.junit4.JUnit4IdeaTestRunner.startRunnerWithArgs(JUnit4IdeaTestRunner.java:69)
> at
> com.intellij.rt.execution.junit.JUnitStarter.prepareStreamsAndStart(JUnitStarter.java:234)
> at
> com.intellij.rt.execution.junit.JUnitStarter.main(JUnitStarter.java:74)
> Caused by: java.lang.RuntimeException:
> org.apache.hadoop.hive.ql.metadata.HiveException:
> java.lang.ClassNotFoundException:
> org.apache.hadoop.hive.ql.security.authorization.plugin.sqlstd.SQLStdHiveAuthorizerFactoryForTest
> at
> org.apache.hadoop.hive.ql.session.SessionState.setupAuth(SessionState.java:836)
> at
> org.apache.hadoop.hive.ql.session.SessionState.applyAuthorizationPolicy(SessionState.java:1602)
> at
> org.apache.hive.service.cli.CLIService.applyAuthorizationConfigPolicy(CLIService.java:126)
> at org.apache.hive.service.cli.CLIService.init(CLIService.java:110)
> ... 22 more
> Caused by: org.apache.hadoop.hive.ql.metadata.HiveException:
> java.lang.ClassNotFoundException:
> org.apache.hadoop.hive.ql.security.authorization.plugin.sqlstd.SQLStdHiveAuthorizerFactoryForTest
> at
> org.apache.hadoop.hive.ql.metadata.HiveUtils.getAuthorizeProviderManager(HiveUtils.java:385)
> at
> org.apache.hadoop.hive.ql.session.SessionState.setupAuth(SessionState.java:812)
> ... 25 more
> Caused by: java.lang.ClassNotFoundException:
> org.apache.hadoop.hive.ql.security.authorization.plugin.sqlstd.SQLStdHiveAuthorizerFactoryForTest
> at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
> at java.lang.Class.forName0(Native Method)
> at java.lang.Class.forName(Class.java:348)
> at
> org.apache.hadoop.hive.ql.metadata.HiveUtils.getAuthorizeProviderManager(HiveUtils.java:375)
> ... 26 more
> {noformat}
> But is caused by HIVE-14221. Code changes are here:
> https://github.com/apache/hive/commit/de5ae86ee70d9396d5cefc499507b5f31fecc916
> So the issue is that, in this patch, everywhere the
[jira] [Updated] (HIVE-14335) TaskDisplay's return value is not getting deserialized properly
[ https://issues.apache.org/jira/browse/HIVE-14335?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Amareshwari Sriramadasu updated HIVE-14335: --- Fix Version/s: 2.1.1 Committed to branch-2.1 as well. > TaskDisplay's return value is not getting deserialized properly > --- > > Key: HIVE-14335 > URL: https://issues.apache.org/jira/browse/HIVE-14335 > Project: Hive > Issue Type: Bug >Reporter: Rajat Khandelwal >Assignee: Rajat Khandelwal > Fix For: 2.2.0, 2.1.1 > > Attachments: HIVE-14335.01.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-14335) TaskDisplay's return value is not getting deserialized properly
[ https://issues.apache.org/jira/browse/HIVE-14335?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15399185#comment-15399185 ] Amareshwari Sriramadasu commented on HIVE-14335: If there are no objections, I would like to commit to branch-2.1 cc [~szehon] [~jcamachorodriguez] > TaskDisplay's return value is not getting deserialized properly > --- > > Key: HIVE-14335 > URL: https://issues.apache.org/jira/browse/HIVE-14335 > Project: Hive > Issue Type: Bug >Reporter: Rajat Khandelwal >Assignee: Rajat Khandelwal > Fix For: 2.2.0 > > Attachments: HIVE-14335.01.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-14264) ArrayIndexOutOfBoundsException when cbo is enabled
[
https://issues.apache.org/jira/browse/HIVE-14264?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15396993#comment-15396993
]
Amareshwari Sriramadasu commented on HIVE-14264:
[~gszadovszky] We are running queries on hive server on version 2.1 and using
MR execution engine. The tables are partitioned and the issue mostly happens
when col is not projected but it is only in filter clause with 'IS NOT NULL'
filter. But we have seen failures in some cases when column is projected as
well.
> ArrayIndexOutOfBoundsException when cbo is enabled
> ---
>
> Key: HIVE-14264
> URL: https://issues.apache.org/jira/browse/HIVE-14264
> Project: Hive
> Issue Type: Bug
> Components: Query Processor
>Affects Versions: 2.1.0
>Reporter: Amareshwari Sriramadasu
>Assignee: Gabor Szadovszky
>
> We have noticed ArrayIndexOutOfBoundsException for queries with IS NOT NULL
> filter. Exception goes away when hive.cbo.enable=false
> Here is a stacktrace in our production environment :
> {noformat}
> Caused by: java.lang.ArrayIndexOutOfBoundsException: -1
> at java.util.ArrayList.elementData(ArrayList.java:418) ~[na:1.8.0_72]
> at java.util.ArrayList.set(ArrayList.java:446) ~[na:1.8.0_72]
> at
> org.apache.hadoop.hive.ql.optimizer.physical.MapJoinResolver$LocalMapJoinTaskDispatcher.processCurrentTask(MapJoinResolver.java:173)
> ~[hive-exec-2.1.2-inm.jar:2.1.2-inm]
> at
> org.apache.hadoop.hive.ql.optimizer.physical.MapJoinResolver$LocalMapJoinTaskDispatcher.dispatch(MapJoinResolver.java:239)
> ~[hive-exec-2.1.2-inm.jar:2.1.2-inm]
> at
> org.apache.hadoop.hive.ql.lib.TaskGraphWalker.dispatch(TaskGraphWalker.java:111)
> ~[hive-exec-2.1.2-inm.jar:2.1.2-inm]
> at
> org.apache.hadoop.hive.ql.lib.TaskGraphWalker.walk(TaskGraphWalker.java:180)
> ~[hive-exec-2.1.2-inm.jar:2.1.2-inm]
> at
> org.apache.hadoop.hive.ql.lib.TaskGraphWalker.startWalking(TaskGraphWalker.java:125)
> ~[hive-exec-2.1.2-inm.jar:2.1.2-inm]
> at
> org.apache.hadoop.hive.ql.optimizer.physical.MapJoinResolver.resolve(MapJoinResolver.java:81)
> ~[hive-exec-2.1.2-inm.jar:2.1.2-inm]
> at
> org.apache.hadoop.hive.ql.optimizer.physical.PhysicalOptimizer.optimize(PhysicalOptimizer.java:107)
> ~[hive-exec-2.1.2-inm.jar:2.1.2-inm]
> at
> org.apache.hadoop.hive.ql.parse.MapReduceCompiler.optimizeTaskPlan(MapReduceCompiler.java:271)
> ~[hive-exec-2.1.2-inm.jar:2.1.2-inm]
> at
> org.apache.hadoop.hive.ql.parse.TaskCompiler.compile(TaskCompiler.java:274)
> ~[hive-exec-2.1.2-inm.jar:2.1.2-inm]
> at
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeInternal(SemanticAnalyzer.java:10764)
> ~[hive-exec-2.1.2-inm.jar:2.1.2-inm]
> at
> org.apache.hadoop.hive.ql.parse.CalcitePlanner.analyzeInternal(CalcitePlanner.java:234)
> ~[hive-exec-2.1.2-inm.jar:2.1.2-inm]
> at
> org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:250)
> ~[hive-exec-2.1.2-inm.jar:2.1.2-inm]
> at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:436)
> ~[hive-exec-2.1.2-inm.jar:2.1.2-inm]
> at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:328)
> ~[hive-exec-2.1.2-inm.jar:2.1.2-inm]
> at org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:1156)
> ~[hive-exec-2.1.2-inm.jar:2.1.2-inm]
> at
> org.apache.hadoop.hive.ql.Driver.compileAndRespond(Driver.java:1143)
> ~[hive-exec-2.1.2-inm.jar:2.1.2-inm]
> at
> org.apache.hive.service.cli.operation.SQLOperation.prepare(SQLOperation.java:147)
> ~[hive-service-2.1.2-inm.jar:2.1.2-inm]
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-14335) TaskDisplay's return value is not getting deserialized properly
[ https://issues.apache.org/jira/browse/HIVE-14335?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15396884#comment-15396884 ] Amareshwari Sriramadasu commented on HIVE-14335: [~szehon], Can we commit this to branch-2.1 as well? Thanks > TaskDisplay's return value is not getting deserialized properly > --- > > Key: HIVE-14335 > URL: https://issues.apache.org/jira/browse/HIVE-14335 > Project: Hive > Issue Type: Bug >Reporter: Rajat Khandelwal >Assignee: Rajat Khandelwal > Fix For: 2.2.0 > > Attachments: HIVE-14335.01.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-14264) ArrayIndexOutOfBoundsException when cbo is enabled
[
https://issues.apache.org/jira/browse/HIVE-14264?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15395626#comment-15395626
]
Amareshwari Sriramadasu commented on HIVE-14264:
[~gszadovszky], Thanks for looking into this. We were also not able reproduce
in our local setup, query was only failing in our production with above
stacktrace. By disabling cbo, we were able to move forward.
> ArrayIndexOutOfBoundsException when cbo is enabled
> ---
>
> Key: HIVE-14264
> URL: https://issues.apache.org/jira/browse/HIVE-14264
> Project: Hive
> Issue Type: Bug
> Components: Query Processor
>Affects Versions: 2.1.0
>Reporter: Amareshwari Sriramadasu
>Assignee: Gabor Szadovszky
>
> We have noticed ArrayIndexOutOfBoundsException for queries with IS NOT NULL
> filter. Exception goes away when hive.cbo.enable=false
> Here is a stacktrace in our production environment :
> {noformat}
> Caused by: java.lang.ArrayIndexOutOfBoundsException: -1
> at java.util.ArrayList.elementData(ArrayList.java:418) ~[na:1.8.0_72]
> at java.util.ArrayList.set(ArrayList.java:446) ~[na:1.8.0_72]
> at
> org.apache.hadoop.hive.ql.optimizer.physical.MapJoinResolver$LocalMapJoinTaskDispatcher.processCurrentTask(MapJoinResolver.java:173)
> ~[hive-exec-2.1.2-inm.jar:2.1.2-inm]
> at
> org.apache.hadoop.hive.ql.optimizer.physical.MapJoinResolver$LocalMapJoinTaskDispatcher.dispatch(MapJoinResolver.java:239)
> ~[hive-exec-2.1.2-inm.jar:2.1.2-inm]
> at
> org.apache.hadoop.hive.ql.lib.TaskGraphWalker.dispatch(TaskGraphWalker.java:111)
> ~[hive-exec-2.1.2-inm.jar:2.1.2-inm]
> at
> org.apache.hadoop.hive.ql.lib.TaskGraphWalker.walk(TaskGraphWalker.java:180)
> ~[hive-exec-2.1.2-inm.jar:2.1.2-inm]
> at
> org.apache.hadoop.hive.ql.lib.TaskGraphWalker.startWalking(TaskGraphWalker.java:125)
> ~[hive-exec-2.1.2-inm.jar:2.1.2-inm]
> at
> org.apache.hadoop.hive.ql.optimizer.physical.MapJoinResolver.resolve(MapJoinResolver.java:81)
> ~[hive-exec-2.1.2-inm.jar:2.1.2-inm]
> at
> org.apache.hadoop.hive.ql.optimizer.physical.PhysicalOptimizer.optimize(PhysicalOptimizer.java:107)
> ~[hive-exec-2.1.2-inm.jar:2.1.2-inm]
> at
> org.apache.hadoop.hive.ql.parse.MapReduceCompiler.optimizeTaskPlan(MapReduceCompiler.java:271)
> ~[hive-exec-2.1.2-inm.jar:2.1.2-inm]
> at
> org.apache.hadoop.hive.ql.parse.TaskCompiler.compile(TaskCompiler.java:274)
> ~[hive-exec-2.1.2-inm.jar:2.1.2-inm]
> at
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeInternal(SemanticAnalyzer.java:10764)
> ~[hive-exec-2.1.2-inm.jar:2.1.2-inm]
> at
> org.apache.hadoop.hive.ql.parse.CalcitePlanner.analyzeInternal(CalcitePlanner.java:234)
> ~[hive-exec-2.1.2-inm.jar:2.1.2-inm]
> at
> org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:250)
> ~[hive-exec-2.1.2-inm.jar:2.1.2-inm]
> at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:436)
> ~[hive-exec-2.1.2-inm.jar:2.1.2-inm]
> at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:328)
> ~[hive-exec-2.1.2-inm.jar:2.1.2-inm]
> at org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:1156)
> ~[hive-exec-2.1.2-inm.jar:2.1.2-inm]
> at
> org.apache.hadoop.hive.ql.Driver.compileAndRespond(Driver.java:1143)
> ~[hive-exec-2.1.2-inm.jar:2.1.2-inm]
> at
> org.apache.hive.service.cli.operation.SQLOperation.prepare(SQLOperation.java:147)
> ~[hive-service-2.1.2-inm.jar:2.1.2-inm]
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-14144) Permanent functions are showing up in show functions, but describe says it doesn't exist
[ https://issues.apache.org/jira/browse/HIVE-14144?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Amareshwari Sriramadasu updated HIVE-14144: --- Resolution: Fixed Hadoop Flags: Reviewed Fix Version/s: 2.1.1 Status: Resolved (was: Patch Available) Committed to both master and branch-2.1. Thanks [~prongs] > Permanent functions are showing up in show functions, but describe says it > doesn't exist > > > Key: HIVE-14144 > URL: https://issues.apache.org/jira/browse/HIVE-14144 > Project: Hive > Issue Type: Bug >Affects Versions: 2.1.0 >Reporter: Rajat Khandelwal >Assignee: Rajat Khandelwal > Fix For: 2.1.1 > > Attachments: HIVE-14144.01-branch-2.1.patch, HIVE-14144.01.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-14144) Permanent functions are showing up in show functions, but describe says it doesn't exist
[ https://issues.apache.org/jira/browse/HIVE-14144?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Amareshwari Sriramadasu updated HIVE-14144: --- Status: Patch Available (was: Open) > Permanent functions are showing up in show functions, but describe says it > doesn't exist > > > Key: HIVE-14144 > URL: https://issues.apache.org/jira/browse/HIVE-14144 > Project: Hive > Issue Type: Bug >Affects Versions: 2.1.0 >Reporter: Rajat Khandelwal >Assignee: Rajat Khandelwal > Attachments: HIVE-14144.01.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-14144) Permanent functions are showing up in show functions, but describe says it doesn't exist
[ https://issues.apache.org/jira/browse/HIVE-14144?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Amareshwari Sriramadasu updated HIVE-14144: --- Status: Open (was: Patch Available) Cancelling to re-trigger pre-commit build > Permanent functions are showing up in show functions, but describe says it > doesn't exist > > > Key: HIVE-14144 > URL: https://issues.apache.org/jira/browse/HIVE-14144 > Project: Hive > Issue Type: Bug >Affects Versions: 2.1.0 >Reporter: Rajat Khandelwal >Assignee: Rajat Khandelwal > Attachments: HIVE-14144.01.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-14144) Permanent functions are showing up in show functions, but describe says it doesn't exist
[ https://issues.apache.org/jira/browse/HIVE-14144?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Amareshwari Sriramadasu updated HIVE-14144: --- Target Version/s: 2.1.1 > Permanent functions are showing up in show functions, but describe says it > doesn't exist > > > Key: HIVE-14144 > URL: https://issues.apache.org/jira/browse/HIVE-14144 > Project: Hive > Issue Type: Bug >Affects Versions: 2.1.0 >Reporter: Rajat Khandelwal >Assignee: Rajat Khandelwal > Attachments: HIVE-14144.01.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13903) getFunctionInfo is downloading jar on every call
[ https://issues.apache.org/jira/browse/HIVE-13903?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Amareshwari Sriramadasu updated HIVE-13903: --- Resolution: Fixed Fix Version/s: 2.1.1 Status: Resolved (was: Patch Available) Committed. Thanks [~prongs]. Thanks [~jcamachorodriguez] for review. > getFunctionInfo is downloading jar on every call > > > Key: HIVE-13903 > URL: https://issues.apache.org/jira/browse/HIVE-13903 > Project: Hive > Issue Type: Bug >Reporter: Rajat Khandelwal >Assignee: Rajat Khandelwal > Fix For: 2.1.1 > > Attachments: HIVE-13903.01.patch, HIVE-13903.01.patch, > HIVE-13903.02.patch > > > on queries using permanent udfs, the jar file of the udf is downloaded > multiple times. Each call originating from Registry.getFunctionInfo. This > increases time for the query, especially if that query is just an explain > query. The jar should be downloaded once, and not downloaded again if the udf > class is accessible in the current thread. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13903) getFunctionInfo is downloading jar on every call
[ https://issues.apache.org/jira/browse/HIVE-13903?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15326805#comment-15326805 ] Amareshwari Sriramadasu commented on HIVE-13903: [~sershe] and [~jcamachorodriguez], Can you review the changes put by [~prongs] ? The changes look fine to me. > getFunctionInfo is downloading jar on every call > > > Key: HIVE-13903 > URL: https://issues.apache.org/jira/browse/HIVE-13903 > Project: Hive > Issue Type: Bug >Reporter: Rajat Khandelwal >Assignee: Rajat Khandelwal > Attachments: HIVE-13903.01.patch, HIVE-13903.01.patch, > HIVE-13903.02.patch > > > on queries using permanent udfs, the jar file of the udf is downloaded > multiple times. Each call originating from Registry.getFunctionInfo. This > increases time for the query, especially if that query is just an explain > query. The jar should be downloaded once, and not downloaded again if the udf > class is accessible in the current thread. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13903) getFunctionInfo is downloading jar on every call
[ https://issues.apache.org/jira/browse/HIVE-13903?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Amareshwari Sriramadasu updated HIVE-13903: --- Resolution: Fixed Hadoop Flags: Reviewed Fix Version/s: 2.1.0 Status: Resolved (was: Patch Available) Committed to master and branch-2.1 cc [~jcamachorodriguez] > getFunctionInfo is downloading jar on every call > > > Key: HIVE-13903 > URL: https://issues.apache.org/jira/browse/HIVE-13903 > Project: Hive > Issue Type: Bug >Reporter: Rajat Khandelwal >Assignee: Rajat Khandelwal > Fix For: 2.1.0 > > Attachments: HIVE-13903.01.patch > > > on queries using permanent udfs, the jar file of the udf is downloaded > multiple times. Each call originating from Registry.getFunctionInfo. This > increases time for the query, especially if that query is just an explain > query. The jar should be downloaded once, and not downloaded again if the udf > class is accessible in the current thread. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13903) getFunctionInfo is downloading jar on every call
[ https://issues.apache.org/jira/browse/HIVE-13903?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15313975#comment-15313975 ] Amareshwari Sriramadasu commented on HIVE-13903: [~sershe], HIVE-6672 (which is marked in the code comment) was adding jars under a check, but later changes has removed the check. so, putting it back looks fine to me. If no objections, would like to commit this. Thanks > getFunctionInfo is downloading jar on every call > > > Key: HIVE-13903 > URL: https://issues.apache.org/jira/browse/HIVE-13903 > Project: Hive > Issue Type: Bug >Reporter: Rajat Khandelwal >Assignee: Rajat Khandelwal > Attachments: HIVE-13903.01.patch > > > on queries using permanent udfs, the jar file of the udf is downloaded > multiple times. Each call originating from Registry.getFunctionInfo. This > increases time for the query, especially if that query is just an explain > query. The jar should be downloaded once, and not downloaded again if the udf > class is accessible in the current thread. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13862) org.apache.hadoop.hive.metastore.MetaStoreDirectSql.getNumPartitionsViaSqlFilter falls back to ORM
[
https://issues.apache.org/jira/browse/HIVE-13862?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Amareshwari Sriramadasu updated HIVE-13862:
---
Target Version/s: 2.1.1 (was: 2.1.0)
Fix Version/s: (was: 2.1.0)
2.1.1
I see release-2.1.0-rc1 tag is already created. [~jcamachorodriguez], Changed
fix version to 2.1.1 - Please change if there is another RC for 2.1.0. Thanks
> org.apache.hadoop.hive.metastore.MetaStoreDirectSql.getNumPartitionsViaSqlFilter
> falls back to ORM
> ---
>
> Key: HIVE-13862
> URL: https://issues.apache.org/jira/browse/HIVE-13862
> Project: Hive
> Issue Type: Bug
> Components: Metastore
>Reporter: Amareshwari Sriramadasu
>Assignee: Rajat Khandelwal
> Fix For: 2.1.1
>
> Attachments: HIVE-13862.1.patch, HIVE-13862.patch
>
>
> We are seeing following exception and calls fall back to ORM which make it
> costly :
> {noformat}
> WARN org.apache.hadoop.hive.metastore.ObjectStore - Direct SQL failed,
> falling back to ORM
> java.lang.ClassCastException:
> org.datanucleus.store.rdbms.query.ForwardQueryResult cannot be cast to
> java.lang.Number
> at
> org.apache.hadoop.hive.metastore.MetaStoreDirectSql.extractSqlInt(MetaStoreDirectSql.java:892)
> ~[hive-exec-2.1.2-inm-SNAPSHOT.jar:2.1.2-inm-SNAPSHOT]
> at
> org.apache.hadoop.hive.metastore.MetaStoreDirectSql.getNumPartitionsViaSqlFilterInternal(MetaStoreDirectSql.java:855)
> ~[hive-exec-2.1.2-inm-SNAPSHOT.jar:2.1.2-inm-SNAPSHOT]
> at
> org.apache.hadoop.hive.metastore.MetaStoreDirectSql.getNumPartitionsViaSqlFilter(MetaStoreDirectSql.java:405)
> ~[hive-exec-2.1.2-inm-SNAPSHOT.jar:2.1.2-inm-SNAPSHOT]
> at
> org.apache.hadoop.hive.metastore.ObjectStore$5.getSqlResult(ObjectStore.java:2763)
> ~[hive-exec-2.1.2-inm-SNAPSHOT.jar:2.1.2-inm-SNAPSHOT]
> at
> org.apache.hadoop.hive.metastore.ObjectStore$5.getSqlResult(ObjectStore.java:2755)
> ~[hive-exec-2.1.2-inm-SNAPSHOT.jar:2.1.2-inm-SNAPSHOT]
> at
> org.apache.hadoop.hive.metastore.ObjectStore$GetHelper.run(ObjectStore.java:2606)
> ~[hive-exec-2.1.2-inm-SNAPSHOT.jar:2.1.2-inm-SNAPSHOT]
> at
> org.apache.hadoop.hive.metastore.ObjectStore.getNumPartitionsByFilterInternal(ObjectStore.java:2770)
> [hive-exec-2.1.2-inm-SNAPSHOT.jar:2.1.2-inm-SNAPSHOT]
> at
> org.apache.hadoop.hive.metastore.ObjectStore.getNumPartitionsByFilter(ObjectStore.java:2746)
> [hive-exec-2.1.2-inm-SNAPSHOT.jar:2.1.2-inm-SNAPSHOT]
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-13862) org.apache.hadoop.hive.metastore.MetaStoreDirectSql.getNumPartitionsViaSqlFilter falls back to ORM
[
https://issues.apache.org/jira/browse/HIVE-13862?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Amareshwari Sriramadasu updated HIVE-13862:
---
Resolution: Fixed
Hadoop Flags: Reviewed
Fix Version/s: 2.1.0
Target Version/s: 2.1.0 (was: 2.2.0)
Status: Resolved (was: Patch Available)
Committed. Thanks [~prongs].
[~jcamachorodriguez], Have merged in master and branch-2.1, Please include if
you are putting up one more RC, if not please change the fix version to 2.1.1.
Thanks!
> org.apache.hadoop.hive.metastore.MetaStoreDirectSql.getNumPartitionsViaSqlFilter
> falls back to ORM
> ---
>
> Key: HIVE-13862
> URL: https://issues.apache.org/jira/browse/HIVE-13862
> Project: Hive
> Issue Type: Bug
> Components: Metastore
>Reporter: Amareshwari Sriramadasu
>Assignee: Rajat Khandelwal
> Fix For: 2.1.0
>
> Attachments: HIVE-13862.1.patch, HIVE-13862.patch
>
>
> We are seeing following exception and calls fall back to ORM which make it
> costly :
> {noformat}
> WARN org.apache.hadoop.hive.metastore.ObjectStore - Direct SQL failed,
> falling back to ORM
> java.lang.ClassCastException:
> org.datanucleus.store.rdbms.query.ForwardQueryResult cannot be cast to
> java.lang.Number
> at
> org.apache.hadoop.hive.metastore.MetaStoreDirectSql.extractSqlInt(MetaStoreDirectSql.java:892)
> ~[hive-exec-2.1.2-inm-SNAPSHOT.jar:2.1.2-inm-SNAPSHOT]
> at
> org.apache.hadoop.hive.metastore.MetaStoreDirectSql.getNumPartitionsViaSqlFilterInternal(MetaStoreDirectSql.java:855)
> ~[hive-exec-2.1.2-inm-SNAPSHOT.jar:2.1.2-inm-SNAPSHOT]
> at
> org.apache.hadoop.hive.metastore.MetaStoreDirectSql.getNumPartitionsViaSqlFilter(MetaStoreDirectSql.java:405)
> ~[hive-exec-2.1.2-inm-SNAPSHOT.jar:2.1.2-inm-SNAPSHOT]
> at
> org.apache.hadoop.hive.metastore.ObjectStore$5.getSqlResult(ObjectStore.java:2763)
> ~[hive-exec-2.1.2-inm-SNAPSHOT.jar:2.1.2-inm-SNAPSHOT]
> at
> org.apache.hadoop.hive.metastore.ObjectStore$5.getSqlResult(ObjectStore.java:2755)
> ~[hive-exec-2.1.2-inm-SNAPSHOT.jar:2.1.2-inm-SNAPSHOT]
> at
> org.apache.hadoop.hive.metastore.ObjectStore$GetHelper.run(ObjectStore.java:2606)
> ~[hive-exec-2.1.2-inm-SNAPSHOT.jar:2.1.2-inm-SNAPSHOT]
> at
> org.apache.hadoop.hive.metastore.ObjectStore.getNumPartitionsByFilterInternal(ObjectStore.java:2770)
> [hive-exec-2.1.2-inm-SNAPSHOT.jar:2.1.2-inm-SNAPSHOT]
> at
> org.apache.hadoop.hive.metastore.ObjectStore.getNumPartitionsByFilter(ObjectStore.java:2746)
> [hive-exec-2.1.2-inm-SNAPSHOT.jar:2.1.2-inm-SNAPSHOT]
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-13862) org.apache.hadoop.hive.metastore.MetaStoreDirectSql.getNumPartitionsViaSqlFilter falls back to ORM
[
https://issues.apache.org/jira/browse/HIVE-13862?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15309243#comment-15309243
]
Amareshwari Sriramadasu commented on HIVE-13862:
+1 for the patch. Waiting for Hive QA builds to happen. But no response from
Hive QA for last 3 days.
cc [~jcamachorodriguez]
> org.apache.hadoop.hive.metastore.MetaStoreDirectSql.getNumPartitionsViaSqlFilter
> falls back to ORM
> ---
>
> Key: HIVE-13862
> URL: https://issues.apache.org/jira/browse/HIVE-13862
> Project: Hive
> Issue Type: Bug
> Components: Metastore
>Reporter: Amareshwari Sriramadasu
>Assignee: Rajat Khandelwal
> Fix For: 2.1.0
>
> Attachments: HIVE-13862.1.patch, HIVE-13862.patch
>
>
> We are seeing following exception and calls fall back to ORM which make it
> costly :
> {noformat}
> WARN org.apache.hadoop.hive.metastore.ObjectStore - Direct SQL failed,
> falling back to ORM
> java.lang.ClassCastException:
> org.datanucleus.store.rdbms.query.ForwardQueryResult cannot be cast to
> java.lang.Number
> at
> org.apache.hadoop.hive.metastore.MetaStoreDirectSql.extractSqlInt(MetaStoreDirectSql.java:892)
> ~[hive-exec-2.1.2-inm-SNAPSHOT.jar:2.1.2-inm-SNAPSHOT]
> at
> org.apache.hadoop.hive.metastore.MetaStoreDirectSql.getNumPartitionsViaSqlFilterInternal(MetaStoreDirectSql.java:855)
> ~[hive-exec-2.1.2-inm-SNAPSHOT.jar:2.1.2-inm-SNAPSHOT]
> at
> org.apache.hadoop.hive.metastore.MetaStoreDirectSql.getNumPartitionsViaSqlFilter(MetaStoreDirectSql.java:405)
> ~[hive-exec-2.1.2-inm-SNAPSHOT.jar:2.1.2-inm-SNAPSHOT]
> at
> org.apache.hadoop.hive.metastore.ObjectStore$5.getSqlResult(ObjectStore.java:2763)
> ~[hive-exec-2.1.2-inm-SNAPSHOT.jar:2.1.2-inm-SNAPSHOT]
> at
> org.apache.hadoop.hive.metastore.ObjectStore$5.getSqlResult(ObjectStore.java:2755)
> ~[hive-exec-2.1.2-inm-SNAPSHOT.jar:2.1.2-inm-SNAPSHOT]
> at
> org.apache.hadoop.hive.metastore.ObjectStore$GetHelper.run(ObjectStore.java:2606)
> ~[hive-exec-2.1.2-inm-SNAPSHOT.jar:2.1.2-inm-SNAPSHOT]
> at
> org.apache.hadoop.hive.metastore.ObjectStore.getNumPartitionsByFilterInternal(ObjectStore.java:2770)
> [hive-exec-2.1.2-inm-SNAPSHOT.jar:2.1.2-inm-SNAPSHOT]
> at
> org.apache.hadoop.hive.metastore.ObjectStore.getNumPartitionsByFilter(ObjectStore.java:2746)
> [hive-exec-2.1.2-inm-SNAPSHOT.jar:2.1.2-inm-SNAPSHOT]
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-13885) Hive session close is not resetting thread name
[ https://issues.apache.org/jira/browse/HIVE-13885?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Amareshwari Sriramadasu updated HIVE-13885: --- Target Version/s: 2.1.0 Fix Version/s: 2.1.0 > Hive session close is not resetting thread name > --- > > Key: HIVE-13885 > URL: https://issues.apache.org/jira/browse/HIVE-13885 > Project: Hive > Issue Type: Bug >Reporter: Rajat Khandelwal >Assignee: Rajat Khandelwal > Fix For: 2.1.0 > > Attachments: HIVE-13885.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13862) org.apache.hadoop.hive.metastore.MetaStoreDirectSql.getNumPartitionsViaSqlFilter falls back to ORM
[
https://issues.apache.org/jira/browse/HIVE-13862?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15303484#comment-15303484
]
Amareshwari Sriramadasu commented on HIVE-13862:
Yeah.. seems it was always falling to back to ORM - and never worked with
directsql earlier with HIVE-11487. I dont think we have a way to test whether
api is answered from directsql vs orm in unit tests.
btw, we deployed the above fix in our production environment, and it is working
fine.
bq. IIRC some methods use a call on the query object that forces a single
result, that may be a better option here.
Didnt find any. Can you give more pointers?
> org.apache.hadoop.hive.metastore.MetaStoreDirectSql.getNumPartitionsViaSqlFilter
> falls back to ORM
> ---
>
> Key: HIVE-13862
> URL: https://issues.apache.org/jira/browse/HIVE-13862
> Project: Hive
> Issue Type: Bug
> Components: Metastore
>Reporter: Amareshwari Sriramadasu
>Assignee: Rajat Khandelwal
> Fix For: 2.1.0
>
> Attachments: HIVE-13862.patch
>
>
> We are seeing following exception and calls fall back to ORM which make it
> costly :
> {noformat}
> WARN org.apache.hadoop.hive.metastore.ObjectStore - Direct SQL failed,
> falling back to ORM
> java.lang.ClassCastException:
> org.datanucleus.store.rdbms.query.ForwardQueryResult cannot be cast to
> java.lang.Number
> at
> org.apache.hadoop.hive.metastore.MetaStoreDirectSql.extractSqlInt(MetaStoreDirectSql.java:892)
> ~[hive-exec-2.1.2-inm-SNAPSHOT.jar:2.1.2-inm-SNAPSHOT]
> at
> org.apache.hadoop.hive.metastore.MetaStoreDirectSql.getNumPartitionsViaSqlFilterInternal(MetaStoreDirectSql.java:855)
> ~[hive-exec-2.1.2-inm-SNAPSHOT.jar:2.1.2-inm-SNAPSHOT]
> at
> org.apache.hadoop.hive.metastore.MetaStoreDirectSql.getNumPartitionsViaSqlFilter(MetaStoreDirectSql.java:405)
> ~[hive-exec-2.1.2-inm-SNAPSHOT.jar:2.1.2-inm-SNAPSHOT]
> at
> org.apache.hadoop.hive.metastore.ObjectStore$5.getSqlResult(ObjectStore.java:2763)
> ~[hive-exec-2.1.2-inm-SNAPSHOT.jar:2.1.2-inm-SNAPSHOT]
> at
> org.apache.hadoop.hive.metastore.ObjectStore$5.getSqlResult(ObjectStore.java:2755)
> ~[hive-exec-2.1.2-inm-SNAPSHOT.jar:2.1.2-inm-SNAPSHOT]
> at
> org.apache.hadoop.hive.metastore.ObjectStore$GetHelper.run(ObjectStore.java:2606)
> ~[hive-exec-2.1.2-inm-SNAPSHOT.jar:2.1.2-inm-SNAPSHOT]
> at
> org.apache.hadoop.hive.metastore.ObjectStore.getNumPartitionsByFilterInternal(ObjectStore.java:2770)
> [hive-exec-2.1.2-inm-SNAPSHOT.jar:2.1.2-inm-SNAPSHOT]
> at
> org.apache.hadoop.hive.metastore.ObjectStore.getNumPartitionsByFilter(ObjectStore.java:2746)
> [hive-exec-2.1.2-inm-SNAPSHOT.jar:2.1.2-inm-SNAPSHOT]
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-13862) org.apache.hadoop.hive.metastore.MetaStoreDirectSql.getNumPartitionsViaSqlFilter falls back to ORM
[
https://issues.apache.org/jira/browse/HIVE-13862?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Amareshwari Sriramadasu updated HIVE-13862:
---
Status: Patch Available (was: Open)
> org.apache.hadoop.hive.metastore.MetaStoreDirectSql.getNumPartitionsViaSqlFilter
> falls back to ORM
> ---
>
> Key: HIVE-13862
> URL: https://issues.apache.org/jira/browse/HIVE-13862
> Project: Hive
> Issue Type: Bug
> Components: Metastore
>Reporter: Amareshwari Sriramadasu
>Assignee: Rajat Khandelwal
> Fix For: 2.1.0
>
> Attachments: HIVE-13862.patch
>
>
> We are seeing following exception and calls fall back to ORM which make it
> costly :
> {noformat}
> WARN org.apache.hadoop.hive.metastore.ObjectStore - Direct SQL failed,
> falling back to ORM
> java.lang.ClassCastException:
> org.datanucleus.store.rdbms.query.ForwardQueryResult cannot be cast to
> java.lang.Number
> at
> org.apache.hadoop.hive.metastore.MetaStoreDirectSql.extractSqlInt(MetaStoreDirectSql.java:892)
> ~[hive-exec-2.1.2-inm-SNAPSHOT.jar:2.1.2-inm-SNAPSHOT]
> at
> org.apache.hadoop.hive.metastore.MetaStoreDirectSql.getNumPartitionsViaSqlFilterInternal(MetaStoreDirectSql.java:855)
> ~[hive-exec-2.1.2-inm-SNAPSHOT.jar:2.1.2-inm-SNAPSHOT]
> at
> org.apache.hadoop.hive.metastore.MetaStoreDirectSql.getNumPartitionsViaSqlFilter(MetaStoreDirectSql.java:405)
> ~[hive-exec-2.1.2-inm-SNAPSHOT.jar:2.1.2-inm-SNAPSHOT]
> at
> org.apache.hadoop.hive.metastore.ObjectStore$5.getSqlResult(ObjectStore.java:2763)
> ~[hive-exec-2.1.2-inm-SNAPSHOT.jar:2.1.2-inm-SNAPSHOT]
> at
> org.apache.hadoop.hive.metastore.ObjectStore$5.getSqlResult(ObjectStore.java:2755)
> ~[hive-exec-2.1.2-inm-SNAPSHOT.jar:2.1.2-inm-SNAPSHOT]
> at
> org.apache.hadoop.hive.metastore.ObjectStore$GetHelper.run(ObjectStore.java:2606)
> ~[hive-exec-2.1.2-inm-SNAPSHOT.jar:2.1.2-inm-SNAPSHOT]
> at
> org.apache.hadoop.hive.metastore.ObjectStore.getNumPartitionsByFilterInternal(ObjectStore.java:2770)
> [hive-exec-2.1.2-inm-SNAPSHOT.jar:2.1.2-inm-SNAPSHOT]
> at
> org.apache.hadoop.hive.metastore.ObjectStore.getNumPartitionsByFilter(ObjectStore.java:2746)
> [hive-exec-2.1.2-inm-SNAPSHOT.jar:2.1.2-inm-SNAPSHOT]
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-13862) org.apache.hadoop.hive.metastore.MetaStoreDirectSql.getNumPartitionsViaSqlFilter falls back to ORM
[
https://issues.apache.org/jira/browse/HIVE-13862?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15302044#comment-15302044
]
Amareshwari Sriramadasu commented on HIVE-13862:
[~jcamachorodriguez], We would like to get this in 2.1.0 release.
> org.apache.hadoop.hive.metastore.MetaStoreDirectSql.getNumPartitionsViaSqlFilter
> falls back to ORM
> ---
>
> Key: HIVE-13862
> URL: https://issues.apache.org/jira/browse/HIVE-13862
> Project: Hive
> Issue Type: Bug
> Components: Metastore
>Reporter: Amareshwari Sriramadasu
>Assignee: Rajat Khandelwal
> Fix For: 2.1.0
>
>
> We are seeing following exception and calls fall back to ORM which make it
> costly :
> {noformat}
> WARN org.apache.hadoop.hive.metastore.ObjectStore - Direct SQL failed,
> falling back to ORM
> java.lang.ClassCastException:
> org.datanucleus.store.rdbms.query.ForwardQueryResult cannot be cast to
> java.lang.Number
> at
> org.apache.hadoop.hive.metastore.MetaStoreDirectSql.extractSqlInt(MetaStoreDirectSql.java:892)
> ~[hive-exec-2.1.2-inm-SNAPSHOT.jar:2.1.2-inm-SNAPSHOT]
> at
> org.apache.hadoop.hive.metastore.MetaStoreDirectSql.getNumPartitionsViaSqlFilterInternal(MetaStoreDirectSql.java:855)
> ~[hive-exec-2.1.2-inm-SNAPSHOT.jar:2.1.2-inm-SNAPSHOT]
> at
> org.apache.hadoop.hive.metastore.MetaStoreDirectSql.getNumPartitionsViaSqlFilter(MetaStoreDirectSql.java:405)
> ~[hive-exec-2.1.2-inm-SNAPSHOT.jar:2.1.2-inm-SNAPSHOT]
> at
> org.apache.hadoop.hive.metastore.ObjectStore$5.getSqlResult(ObjectStore.java:2763)
> ~[hive-exec-2.1.2-inm-SNAPSHOT.jar:2.1.2-inm-SNAPSHOT]
> at
> org.apache.hadoop.hive.metastore.ObjectStore$5.getSqlResult(ObjectStore.java:2755)
> ~[hive-exec-2.1.2-inm-SNAPSHOT.jar:2.1.2-inm-SNAPSHOT]
> at
> org.apache.hadoop.hive.metastore.ObjectStore$GetHelper.run(ObjectStore.java:2606)
> ~[hive-exec-2.1.2-inm-SNAPSHOT.jar:2.1.2-inm-SNAPSHOT]
> at
> org.apache.hadoop.hive.metastore.ObjectStore.getNumPartitionsByFilterInternal(ObjectStore.java:2770)
> [hive-exec-2.1.2-inm-SNAPSHOT.jar:2.1.2-inm-SNAPSHOT]
> at
> org.apache.hadoop.hive.metastore.ObjectStore.getNumPartitionsByFilter(ObjectStore.java:2746)
> [hive-exec-2.1.2-inm-SNAPSHOT.jar:2.1.2-inm-SNAPSHOT]
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-13727) Getting error Failed rule: 'orderByClause clusterByClause distributeByClause sortByClause limitClause can only be applied to the whole union.' in subquery
[
https://issues.apache.org/jira/browse/HIVE-13727?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Amareshwari Sriramadasu updated HIVE-13727:
---
Fix Version/s: 2.1.0
> Getting error Failed rule: 'orderByClause clusterByClause distributeByClause
> sortByClause limitClause can only be applied to the whole union.' in subquery
> ---
>
> Key: HIVE-13727
> URL: https://issues.apache.org/jira/browse/HIVE-13727
> Project: Hive
> Issue Type: Bug
>Reporter: Rajat Khandelwal
> Fix For: 2.1.0
>
>
> The error comes in the following query:
> {noformat}
> SELECT *
> FROM
> (SELECT *
>FROM srcpart a
>WHERE a.ds = '2008-04-08'
> AND a.hr = '11'
>ORDER BY a.key LIMIT 5
>UNION ALL
>SELECT *
>FROM srcpart b
>WHERE b.ds = '2008-04-08'
> AND b.hr = '14'
>ORDER BY b.key LIMIT 5) subq
> ORDER BY KEY LIMIT 5
> {noformat}
> But the following query works:
> {noformat}
> SELECT *
> FROM
> (SELECT *
>FROM
> (SELECT *
> FROM srcpart a
> WHERE a.ds = '2008-04-08'
> AND a.hr = '11'
> ORDER BY a.key LIMIT 5) pa
>UNION ALL SELECT *
>FROM
> (SELECT *
> FROM srcpart b
> WHERE b.ds = '2008-04-08'
> AND b.hr = '14'
> ORDER BY b.key LIMIT 5) pb) subq
> ORDER BY KEY LIMIT 5
> {noformat}
> The queries are logically identical, the query that's rejected has dummy
> select * clauses around the sub-queries.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-13421) Propagate job progress in operation status
[ https://issues.apache.org/jira/browse/HIVE-13421?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Amareshwari Sriramadasu updated HIVE-13421: --- Resolution: Fixed Hadoop Flags: Reviewed Status: Resolved (was: Patch Available) Committed. Thanks [~prongs] > Propagate job progress in operation status > -- > > Key: HIVE-13421 > URL: https://issues.apache.org/jira/browse/HIVE-13421 > Project: Hive > Issue Type: Improvement >Reporter: Rajat Khandelwal >Assignee: Rajat Khandelwal > Fix For: 2.1.0 > > Attachments: HIVE-13421.01.patch, HIVE-13421.02.patch, > HIVE-13421.03.patch, HIVE-13421.04.patch, HIVE-13421.05.patch, > HIVE-13421.06.patch, HIVE-13421.07.patch, HIVE-13421.08.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13421) Propagate job progress in operation status
[ https://issues.apache.org/jira/browse/HIVE-13421?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15265097#comment-15265097 ] Amareshwari Sriramadasu commented on HIVE-13421: I see there is no difference between HIVE-13421.04.patch and HIVE-13421.08.patch - except that it is updated to master. Checked all the failed tests - most of them have the more age than 1 and the failed ones with age 1 are not related to the patch - they look intermittent failures. Going ahead and committing patch HIVE-13421.08.patch > Propagate job progress in operation status > -- > > Key: HIVE-13421 > URL: https://issues.apache.org/jira/browse/HIVE-13421 > Project: Hive > Issue Type: Improvement >Reporter: Rajat Khandelwal >Assignee: Rajat Khandelwal > Fix For: 2.1.0 > > Attachments: HIVE-13421.01.patch, HIVE-13421.02.patch, > HIVE-13421.03.patch, HIVE-13421.04.patch, HIVE-13421.05.patch, > HIVE-13421.06.patch, HIVE-13421.07.patch, HIVE-13421.08.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13421) Propagate job progress in operation status
[ https://issues.apache.org/jira/browse/HIVE-13421?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Amareshwari Sriramadasu updated HIVE-13421: --- Status: Patch Available (was: Open) Resubmitting for jenkins to pick > Propagate job progress in operation status > -- > > Key: HIVE-13421 > URL: https://issues.apache.org/jira/browse/HIVE-13421 > Project: Hive > Issue Type: Improvement >Reporter: Rajat Khandelwal >Assignee: Rajat Khandelwal > Fix For: 2.1.0 > > Attachments: HIVE-13421.01.patch, HIVE-13421.02.patch, > HIVE-13421.03.patch, HIVE-13421.04.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13421) Propagate job progress in operation status
[ https://issues.apache.org/jira/browse/HIVE-13421?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Amareshwari Sriramadasu updated HIVE-13421: --- Fix Version/s: 2.1.0 Status: Open (was: Patch Available) > Propagate job progress in operation status > -- > > Key: HIVE-13421 > URL: https://issues.apache.org/jira/browse/HIVE-13421 > Project: Hive > Issue Type: Improvement >Reporter: Rajat Khandelwal >Assignee: Rajat Khandelwal > Fix For: 2.1.0 > > Attachments: HIVE-13421.01.patch, HIVE-13421.02.patch, > HIVE-13421.03.patch, HIVE-13421.04.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13421) Propagate job progress in operation status
[ https://issues.apache.org/jira/browse/HIVE-13421?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15249269#comment-15249269 ] Amareshwari Sriramadasu commented on HIVE-13421: +1 https://issues.apache.org/jira/secure/attachment/12799456/HIVE-13421.04.patch looks good. > Propagate job progress in operation status > -- > > Key: HIVE-13421 > URL: https://issues.apache.org/jira/browse/HIVE-13421 > Project: Hive > Issue Type: Improvement >Reporter: Rajat Khandelwal >Assignee: Rajat Khandelwal > Attachments: HIVE-13421.01.patch, HIVE-13421.02.patch, > HIVE-13421.03.patch, HIVE-13421.04.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13500) Launching big queries fails with Out of Memory Exception
[
https://issues.apache.org/jira/browse/HIVE-13500?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Amareshwari Sriramadasu updated HIVE-13500:
---
Resolution: Fixed
Hadoop Flags: Reviewed
Fix Version/s: 2.1.0
Status: Resolved (was: Patch Available)
Committed. Thanks [~prongs]
> Launching big queries fails with Out of Memory Exception
>
>
> Key: HIVE-13500
> URL: https://issues.apache.org/jira/browse/HIVE-13500
> Project: Hive
> Issue Type: Bug
>Reporter: Rajat Khandelwal
>Assignee: Rajat Khandelwal
> Fix For: 2.1.0
>
> Attachments: HIVE-13500.patch
>
>
> There is a code snipped in Driver class in compile method:
> {noformat}
> if (conf.getBoolVar(ConfVars.HIVE_LOG_EXPLAIN_OUTPUT) ||
>conf.isWebUiQueryInfoCacheEnabled()) {
> String explainOutput = getExplainOutput(sem, plan, tree);
> if (explainOutput != null) {
> if (conf.getBoolVar(ConfVars.HIVE_LOG_EXPLAIN_OUTPUT)) {
> LOG.info("EXPLAIN output for queryid " + queryId + " : "
> + explainOutput);
> }
> if (conf.isWebUiQueryInfoCacheEnabled()) {
> queryDisplay.setExplainPlan(explainOutput);
> }
> }
> }
> {noformat}
> This is the stack trace
> {noformat}
> org.apache.hive.service.cli.HiveSQLException: Error running query:
> java.lang.OutOfMemoryError
> at
> org.apache.hive.service.cli.operation.SQLOperation.prepare(SQLOperation.java:178)
> ~[hive-service-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT]
> at
> org.apache.hive.service.cli.operation.SQLOperation.runInternal(SQLOperation.java:216)
> ~[hive-service-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT]
> at
> org.apache.hive.service.cli.operation.Operation.run(Operation.java:325)
> ~[hive-service-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT]
> at
> org.apache.hive.service.cli.session.HiveSessionImpl.executeStatementInternal(HiveSessionImpl.java:456)
> ~[hive-service-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT]
> at
> org.apache.hive.service.cli.session.HiveSessionImpl.executeStatementAsync(HiveSessionImpl.java:433)
> ~[hive-service-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT]
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> ~[?:1.8.0_72]
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> ~[?:1.8.0_72]
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> ~[?:1.8.0_72]
> at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_72]
> at
> org.apache.hive.service.cli.session.HiveSessionProxy.invoke(HiveSessionProxy.java:78)
> ~[hive-service-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT]
> at
> org.apache.hive.service.cli.session.HiveSessionProxy.access$000(HiveSessionProxy.java:36)
> ~[hive-service-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT]
> at
> org.apache.hive.service.cli.session.HiveSessionProxy$1.run(HiveSessionProxy.java:63)
> ~[hive-service-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT]
> at java.security.AccessController.doPrivileged(Native Method)
> ~[?:1.8.0_72]
> at javax.security.auth.Subject.doAs(Subject.java:422) ~[?:1.8.0_72]
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1628)
> ~[hadoop-common-2.6.0.2.2.4.2-2.jar:?]
> at
> org.apache.hive.service.cli.session.HiveSessionProxy.invoke(HiveSessionProxy.java:59)
> ~[hive-service-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT]
> at com.sun.proxy.$Proxy33.executeStatementAsync(Unknown Source) ~[?:?]
> at
> org.apache.hive.service.cli.CLIService.executeStatementAsync(CLIService.java:272)
> ~[hive-service-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT]
> at
> org.apache.hive.service.cli.thrift.ThriftCLIService.ExecuteStatement(ThriftCLIService.java:554)
> [hive-service-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT]
> at
> org.apache.hive.service.rpc.thrift.TCLIService$Processor$ExecuteStatement.getResult(TCLIService.java:1317)
> [hive-service-rpc-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT]
> at
> org.apache.hive.service.rpc.thrift.TCLIService$Processor$ExecuteStatement.getResult(TCLIService.java:1302)
> [hive-service-rpc-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT]
> at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
> [hive-exec-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT]
> at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39)
> [hive-exec-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT]
> at
> org.apache.hive.service.auth.TSetIpAddressProcessor.process(TSetIpAddressProcessor.java:56)
> [hive-service-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT]
> at
> org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:286)
> [hive-
[jira] [Updated] (HIVE-13415) Decouple Sessions from thrift binary transport
[
https://issues.apache.org/jira/browse/HIVE-13415?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Amareshwari Sriramadasu updated HIVE-13415:
---
Resolution: Fixed
Hadoop Flags: Reviewed
Status: Resolved (was: Patch Available)
Committed. Thanks [~prongs]
> Decouple Sessions from thrift binary transport
> --
>
> Key: HIVE-13415
> URL: https://issues.apache.org/jira/browse/HIVE-13415
> Project: Hive
> Issue Type: Bug
> Components: HiveServer2
>Affects Versions: 1.2.0
>Reporter: Rajat Khandelwal
>Assignee: Rajat Khandelwal
> Fix For: 2.1.0
>
> Attachments: HIVE-13415.01.patch, HIVE-13415.04.patch,
> HIVE-13415.05.patch
>
>
> Current behaviour is:
> * Open a thrift binary transport
> * create a session
> * close the transport
> Then the session gets closed. Consequently, all the operations running in the
> session also get killed.
> Whereas, if you open an HTTP transport, and close, the enclosing sessions are
> not closed.
> This seems like a bad design, having transport and sessions tightly coupled.
> I'd like to fix this.
> The issue that introduced it is
> [HIVE-9601|https://github.com/apache/hive/commit/48bea00c48853459af64b4ca9bfdc3e821c4ed82]
> Relevant discussions at
> [here|https://issues.apache.org/jira/browse/HIVE-11485?focusedCommentId=15223546&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15223546],
>
> [here|https://issues.apache.org/jira/browse/HIVE-11485?focusedCommentId=15223827&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15223827]
> and mentioned links on those comments.
> Another thing that seems like a slightly bad design is this line of code in
> ThriftBinaryCLIService:
> {noformat}
> server.setServerEventHandler(serverEventHandler);
> {noformat}
> Whereas serverEventHandler is defined by the base class, with no users except
> one sub-class(ThriftBinaryCLIService), violating the separation of concerns.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-13415) Decouple Sessions from thrift binary transport
[
https://issues.apache.org/jira/browse/HIVE-13415?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Amareshwari Sriramadasu updated HIVE-13415:
---
Affects Version/s: (was: 2.0.0)
1.2.0
Fix Version/s: 2.1.0
Component/s: HiveServer2
> Decouple Sessions from thrift binary transport
> --
>
> Key: HIVE-13415
> URL: https://issues.apache.org/jira/browse/HIVE-13415
> Project: Hive
> Issue Type: Bug
> Components: HiveServer2
>Affects Versions: 1.2.0
>Reporter: Rajat Khandelwal
>Assignee: Rajat Khandelwal
> Fix For: 2.1.0
>
> Attachments: HIVE-13415.01.patch, HIVE-13415.04.patch,
> HIVE-13415.05.patch
>
>
> Current behaviour is:
> * Open a thrift binary transport
> * create a session
> * close the transport
> Then the session gets closed. Consequently, all the operations running in the
> session also get killed.
> Whereas, if you open an HTTP transport, and close, the enclosing sessions are
> not closed.
> This seems like a bad design, having transport and sessions tightly coupled.
> I'd like to fix this.
> The issue that introduced it is
> [HIVE-9601|https://github.com/apache/hive/commit/48bea00c48853459af64b4ca9bfdc3e821c4ed82]
> Relevant discussions at
> [here|https://issues.apache.org/jira/browse/HIVE-11485?focusedCommentId=15223546&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15223546],
>
> [here|https://issues.apache.org/jira/browse/HIVE-11485?focusedCommentId=15223827&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15223827]
> and mentioned links on those comments.
> Another thing that seems like a slightly bad design is this line of code in
> ThriftBinaryCLIService:
> {noformat}
> server.setServerEventHandler(serverEventHandler);
> {noformat}
> Whereas serverEventHandler is defined by the base class, with no users except
> one sub-class(ThriftBinaryCLIService), violating the separation of concerns.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-13500) Launching big queries fails with Out of Memory Exception
[
https://issues.apache.org/jira/browse/HIVE-13500?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15240790#comment-15240790
]
Amareshwari Sriramadasu commented on HIVE-13500:
+1 for
https://issues.apache.org/jira/secure/attachment/12798655/HIVE-13500.patch
> Launching big queries fails with Out of Memory Exception
>
>
> Key: HIVE-13500
> URL: https://issues.apache.org/jira/browse/HIVE-13500
> Project: Hive
> Issue Type: Bug
>Reporter: Rajat Khandelwal
>Assignee: Rajat Khandelwal
> Attachments: HIVE-13500.patch
>
>
> There is a code snipped in Driver class in compile method:
> {noformat}
> if (conf.getBoolVar(ConfVars.HIVE_LOG_EXPLAIN_OUTPUT) ||
>conf.isWebUiQueryInfoCacheEnabled()) {
> String explainOutput = getExplainOutput(sem, plan, tree);
> if (explainOutput != null) {
> if (conf.getBoolVar(ConfVars.HIVE_LOG_EXPLAIN_OUTPUT)) {
> LOG.info("EXPLAIN output for queryid " + queryId + " : "
> + explainOutput);
> }
> if (conf.isWebUiQueryInfoCacheEnabled()) {
> queryDisplay.setExplainPlan(explainOutput);
> }
> }
> }
> {noformat}
> This is the stack trace
> {noformat}
> org.apache.hive.service.cli.HiveSQLException: Error running query:
> java.lang.OutOfMemoryError
> at
> org.apache.hive.service.cli.operation.SQLOperation.prepare(SQLOperation.java:178)
> ~[hive-service-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT]
> at
> org.apache.hive.service.cli.operation.SQLOperation.runInternal(SQLOperation.java:216)
> ~[hive-service-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT]
> at
> org.apache.hive.service.cli.operation.Operation.run(Operation.java:325)
> ~[hive-service-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT]
> at
> org.apache.hive.service.cli.session.HiveSessionImpl.executeStatementInternal(HiveSessionImpl.java:456)
> ~[hive-service-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT]
> at
> org.apache.hive.service.cli.session.HiveSessionImpl.executeStatementAsync(HiveSessionImpl.java:433)
> ~[hive-service-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT]
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> ~[?:1.8.0_72]
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> ~[?:1.8.0_72]
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> ~[?:1.8.0_72]
> at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_72]
> at
> org.apache.hive.service.cli.session.HiveSessionProxy.invoke(HiveSessionProxy.java:78)
> ~[hive-service-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT]
> at
> org.apache.hive.service.cli.session.HiveSessionProxy.access$000(HiveSessionProxy.java:36)
> ~[hive-service-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT]
> at
> org.apache.hive.service.cli.session.HiveSessionProxy$1.run(HiveSessionProxy.java:63)
> ~[hive-service-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT]
> at java.security.AccessController.doPrivileged(Native Method)
> ~[?:1.8.0_72]
> at javax.security.auth.Subject.doAs(Subject.java:422) ~[?:1.8.0_72]
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1628)
> ~[hadoop-common-2.6.0.2.2.4.2-2.jar:?]
> at
> org.apache.hive.service.cli.session.HiveSessionProxy.invoke(HiveSessionProxy.java:59)
> ~[hive-service-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT]
> at com.sun.proxy.$Proxy33.executeStatementAsync(Unknown Source) ~[?:?]
> at
> org.apache.hive.service.cli.CLIService.executeStatementAsync(CLIService.java:272)
> ~[hive-service-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT]
> at
> org.apache.hive.service.cli.thrift.ThriftCLIService.ExecuteStatement(ThriftCLIService.java:554)
> [hive-service-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT]
> at
> org.apache.hive.service.rpc.thrift.TCLIService$Processor$ExecuteStatement.getResult(TCLIService.java:1317)
> [hive-service-rpc-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT]
> at
> org.apache.hive.service.rpc.thrift.TCLIService$Processor$ExecuteStatement.getResult(TCLIService.java:1302)
> [hive-service-rpc-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT]
> at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
> [hive-exec-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT]
> at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39)
> [hive-exec-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT]
> at
> org.apache.hive.service.auth.TSetIpAddressProcessor.process(TSetIpAddressProcessor.java:56)
> [hive-service-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT]
> at
> org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:286)
> [hive-exec-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT]
>
[jira] [Commented] (HIVE-13415) Decouple Sessions from thrift binary transport
[
https://issues.apache.org/jira/browse/HIVE-13415?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15240469#comment-15240469
]
Amareshwari Sriramadasu commented on HIVE-13415:
+1 for
https://issues.apache.org/jira/secure/attachment/12798463/HIVE-13415.05.patch
> Decouple Sessions from thrift binary transport
> --
>
> Key: HIVE-13415
> URL: https://issues.apache.org/jira/browse/HIVE-13415
> Project: Hive
> Issue Type: Bug
>Affects Versions: 2.0.0
>Reporter: Rajat Khandelwal
>Assignee: Rajat Khandelwal
> Attachments: HIVE-13415.01.patch, HIVE-13415.04.patch,
> HIVE-13415.05.patch
>
>
> Current behaviour is:
> * Open a thrift binary transport
> * create a session
> * close the transport
> Then the session gets closed. Consequently, all the operations running in the
> session also get killed.
> Whereas, if you open an HTTP transport, and close, the enclosing sessions are
> not closed.
> This seems like a bad design, having transport and sessions tightly coupled.
> I'd like to fix this.
> The issue that introduced it is
> [HIVE-9601|https://github.com/apache/hive/commit/48bea00c48853459af64b4ca9bfdc3e821c4ed82]
> Relevant discussions at
> [here|https://issues.apache.org/jira/browse/HIVE-11485?focusedCommentId=15223546&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15223546],
>
> [here|https://issues.apache.org/jira/browse/HIVE-11485?focusedCommentId=15223827&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15223827]
> and mentioned links on those comments.
> Another thing that seems like a slightly bad design is this line of code in
> ThriftBinaryCLIService:
> {noformat}
> server.setServerEventHandler(serverEventHandler);
> {noformat}
> Whereas serverEventHandler is defined by the base class, with no users except
> one sub-class(ThriftBinaryCLIService), violating the separation of concerns.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-13500) Launching big queries fails with Out of Memory Exception
[
https://issues.apache.org/jira/browse/HIVE-13500?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15240424#comment-15240424
]
Amareshwari Sriramadasu commented on HIVE-13500:
We should check only for (conf.getBoolVar(ConfVars.HIVE_LOG_EXPLAIN_OUTPUT) for
calling explain. The OR clause with web ui enabled should be removed.
> Launching big queries fails with Out of Memory Exception
>
>
> Key: HIVE-13500
> URL: https://issues.apache.org/jira/browse/HIVE-13500
> Project: Hive
> Issue Type: Bug
>Reporter: Rajat Khandelwal
>Assignee: Rajat Khandelwal
>
> There is a code snipped in Driver class in compile method:
> {noformat}
> if (conf.getBoolVar(ConfVars.HIVE_LOG_EXPLAIN_OUTPUT) ||
>conf.isWebUiQueryInfoCacheEnabled()) {
> String explainOutput = getExplainOutput(sem, plan, tree);
> if (explainOutput != null) {
> if (conf.getBoolVar(ConfVars.HIVE_LOG_EXPLAIN_OUTPUT)) {
> LOG.info("EXPLAIN output for queryid " + queryId + " : "
> + explainOutput);
> }
> if (conf.isWebUiQueryInfoCacheEnabled()) {
> queryDisplay.setExplainPlan(explainOutput);
> }
> }
> }
> {noformat}
> This is the stack trace
> {noformat}
> org.apache.hive.service.cli.HiveSQLException: Error running query:
> java.lang.OutOfMemoryError
> at
> org.apache.hive.service.cli.operation.SQLOperation.prepare(SQLOperation.java:178)
> ~[hive-service-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT]
> at
> org.apache.hive.service.cli.operation.SQLOperation.runInternal(SQLOperation.java:216)
> ~[hive-service-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT]
> at
> org.apache.hive.service.cli.operation.Operation.run(Operation.java:325)
> ~[hive-service-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT]
> at
> org.apache.hive.service.cli.session.HiveSessionImpl.executeStatementInternal(HiveSessionImpl.java:456)
> ~[hive-service-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT]
> at
> org.apache.hive.service.cli.session.HiveSessionImpl.executeStatementAsync(HiveSessionImpl.java:433)
> ~[hive-service-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT]
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> ~[?:1.8.0_72]
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> ~[?:1.8.0_72]
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> ~[?:1.8.0_72]
> at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_72]
> at
> org.apache.hive.service.cli.session.HiveSessionProxy.invoke(HiveSessionProxy.java:78)
> ~[hive-service-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT]
> at
> org.apache.hive.service.cli.session.HiveSessionProxy.access$000(HiveSessionProxy.java:36)
> ~[hive-service-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT]
> at
> org.apache.hive.service.cli.session.HiveSessionProxy$1.run(HiveSessionProxy.java:63)
> ~[hive-service-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT]
> at java.security.AccessController.doPrivileged(Native Method)
> ~[?:1.8.0_72]
> at javax.security.auth.Subject.doAs(Subject.java:422) ~[?:1.8.0_72]
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1628)
> ~[hadoop-common-2.6.0.2.2.4.2-2.jar:?]
> at
> org.apache.hive.service.cli.session.HiveSessionProxy.invoke(HiveSessionProxy.java:59)
> ~[hive-service-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT]
> at com.sun.proxy.$Proxy33.executeStatementAsync(Unknown Source) ~[?:?]
> at
> org.apache.hive.service.cli.CLIService.executeStatementAsync(CLIService.java:272)
> ~[hive-service-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT]
> at
> org.apache.hive.service.cli.thrift.ThriftCLIService.ExecuteStatement(ThriftCLIService.java:554)
> [hive-service-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT]
> at
> org.apache.hive.service.rpc.thrift.TCLIService$Processor$ExecuteStatement.getResult(TCLIService.java:1317)
> [hive-service-rpc-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT]
> at
> org.apache.hive.service.rpc.thrift.TCLIService$Processor$ExecuteStatement.getResult(TCLIService.java:1302)
> [hive-service-rpc-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT]
> at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
> [hive-exec-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT]
> at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39)
> [hive-exec-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT]
> at
> org.apache.hive.service.auth.TSetIpAddressProcessor.process(TSetIpAddressProcessor.java:56)
> [hive-service-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT]
> at
> org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:286)
> [hive-exec-2.1.0-SNAPSHOT.jar
[jira] [Commented] (HIVE-13500) Launching big queries fails with Out of Memory Exception
[
https://issues.apache.org/jira/browse/HIVE-13500?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15240421#comment-15240421
]
Amareshwari Sriramadasu commented on HIVE-13500:
But current code is calling explain and storing it in the string, which is
causing OOM - even though HIVE_LOG_EXPLAIN_OUTPUT is false.
> Launching big queries fails with Out of Memory Exception
>
>
> Key: HIVE-13500
> URL: https://issues.apache.org/jira/browse/HIVE-13500
> Project: Hive
> Issue Type: Bug
>Reporter: Rajat Khandelwal
>Assignee: Rajat Khandelwal
>
> There is a code snipped in Driver class in compile method:
> {noformat}
> if (conf.getBoolVar(ConfVars.HIVE_LOG_EXPLAIN_OUTPUT) ||
>conf.isWebUiQueryInfoCacheEnabled()) {
> String explainOutput = getExplainOutput(sem, plan, tree);
> if (explainOutput != null) {
> if (conf.getBoolVar(ConfVars.HIVE_LOG_EXPLAIN_OUTPUT)) {
> LOG.info("EXPLAIN output for queryid " + queryId + " : "
> + explainOutput);
> }
> if (conf.isWebUiQueryInfoCacheEnabled()) {
> queryDisplay.setExplainPlan(explainOutput);
> }
> }
> }
> {noformat}
> This is the stack trace
> {noformat}
> org.apache.hive.service.cli.HiveSQLException: Error running query:
> java.lang.OutOfMemoryError
> at
> org.apache.hive.service.cli.operation.SQLOperation.prepare(SQLOperation.java:178)
> ~[hive-service-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT]
> at
> org.apache.hive.service.cli.operation.SQLOperation.runInternal(SQLOperation.java:216)
> ~[hive-service-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT]
> at
> org.apache.hive.service.cli.operation.Operation.run(Operation.java:325)
> ~[hive-service-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT]
> at
> org.apache.hive.service.cli.session.HiveSessionImpl.executeStatementInternal(HiveSessionImpl.java:456)
> ~[hive-service-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT]
> at
> org.apache.hive.service.cli.session.HiveSessionImpl.executeStatementAsync(HiveSessionImpl.java:433)
> ~[hive-service-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT]
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> ~[?:1.8.0_72]
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> ~[?:1.8.0_72]
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> ~[?:1.8.0_72]
> at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_72]
> at
> org.apache.hive.service.cli.session.HiveSessionProxy.invoke(HiveSessionProxy.java:78)
> ~[hive-service-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT]
> at
> org.apache.hive.service.cli.session.HiveSessionProxy.access$000(HiveSessionProxy.java:36)
> ~[hive-service-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT]
> at
> org.apache.hive.service.cli.session.HiveSessionProxy$1.run(HiveSessionProxy.java:63)
> ~[hive-service-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT]
> at java.security.AccessController.doPrivileged(Native Method)
> ~[?:1.8.0_72]
> at javax.security.auth.Subject.doAs(Subject.java:422) ~[?:1.8.0_72]
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1628)
> ~[hadoop-common-2.6.0.2.2.4.2-2.jar:?]
> at
> org.apache.hive.service.cli.session.HiveSessionProxy.invoke(HiveSessionProxy.java:59)
> ~[hive-service-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT]
> at com.sun.proxy.$Proxy33.executeStatementAsync(Unknown Source) ~[?:?]
> at
> org.apache.hive.service.cli.CLIService.executeStatementAsync(CLIService.java:272)
> ~[hive-service-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT]
> at
> org.apache.hive.service.cli.thrift.ThriftCLIService.ExecuteStatement(ThriftCLIService.java:554)
> [hive-service-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT]
> at
> org.apache.hive.service.rpc.thrift.TCLIService$Processor$ExecuteStatement.getResult(TCLIService.java:1317)
> [hive-service-rpc-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT]
> at
> org.apache.hive.service.rpc.thrift.TCLIService$Processor$ExecuteStatement.getResult(TCLIService.java:1302)
> [hive-service-rpc-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT]
> at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
> [hive-exec-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT]
> at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39)
> [hive-exec-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT]
> at
> org.apache.hive.service.auth.TSetIpAddressProcessor.process(TSetIpAddressProcessor.java:56)
> [hive-service-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT]
> at
> org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:286)
> [hive-exec-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT
[jira] [Commented] (HIVE-13500) Launching big queries fails with Out of Memory Exception
[
https://issues.apache.org/jira/browse/HIVE-13500?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15240390#comment-15240390
]
Amareshwari Sriramadasu commented on HIVE-13500:
[~szehon], Above configs are making the web UI to be off as well? Can WebUI
have explain output as optional with HIVE_LOG_EXPLAIN_OUTPUT alone?
Basically code change will be && clause instead of ||
if (conf.getBoolVar(ConfVars.HIVE_LOG_EXPLAIN_OUTPUT) &&
conf.isWebUiQueryInfoCacheEnabled()) {
}
> Launching big queries fails with Out of Memory Exception
>
>
> Key: HIVE-13500
> URL: https://issues.apache.org/jira/browse/HIVE-13500
> Project: Hive
> Issue Type: Bug
>Reporter: Rajat Khandelwal
>Assignee: Rajat Khandelwal
>
> There is a code snipped in Driver class in compile method:
> {noformat}
> if (conf.getBoolVar(ConfVars.HIVE_LOG_EXPLAIN_OUTPUT) ||
>conf.isWebUiQueryInfoCacheEnabled()) {
> String explainOutput = getExplainOutput(sem, plan, tree);
> if (explainOutput != null) {
> if (conf.getBoolVar(ConfVars.HIVE_LOG_EXPLAIN_OUTPUT)) {
> LOG.info("EXPLAIN output for queryid " + queryId + " : "
> + explainOutput);
> }
> if (conf.isWebUiQueryInfoCacheEnabled()) {
> queryDisplay.setExplainPlan(explainOutput);
> }
> }
> }
> {noformat}
> This is the stack trace
> {noformat}
> org.apache.hive.service.cli.HiveSQLException: Error running query:
> java.lang.OutOfMemoryError
> at
> org.apache.hive.service.cli.operation.SQLOperation.prepare(SQLOperation.java:178)
> ~[hive-service-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT]
> at
> org.apache.hive.service.cli.operation.SQLOperation.runInternal(SQLOperation.java:216)
> ~[hive-service-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT]
> at
> org.apache.hive.service.cli.operation.Operation.run(Operation.java:325)
> ~[hive-service-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT]
> at
> org.apache.hive.service.cli.session.HiveSessionImpl.executeStatementInternal(HiveSessionImpl.java:456)
> ~[hive-service-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT]
> at
> org.apache.hive.service.cli.session.HiveSessionImpl.executeStatementAsync(HiveSessionImpl.java:433)
> ~[hive-service-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT]
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> ~[?:1.8.0_72]
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> ~[?:1.8.0_72]
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> ~[?:1.8.0_72]
> at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_72]
> at
> org.apache.hive.service.cli.session.HiveSessionProxy.invoke(HiveSessionProxy.java:78)
> ~[hive-service-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT]
> at
> org.apache.hive.service.cli.session.HiveSessionProxy.access$000(HiveSessionProxy.java:36)
> ~[hive-service-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT]
> at
> org.apache.hive.service.cli.session.HiveSessionProxy$1.run(HiveSessionProxy.java:63)
> ~[hive-service-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT]
> at java.security.AccessController.doPrivileged(Native Method)
> ~[?:1.8.0_72]
> at javax.security.auth.Subject.doAs(Subject.java:422) ~[?:1.8.0_72]
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1628)
> ~[hadoop-common-2.6.0.2.2.4.2-2.jar:?]
> at
> org.apache.hive.service.cli.session.HiveSessionProxy.invoke(HiveSessionProxy.java:59)
> ~[hive-service-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT]
> at com.sun.proxy.$Proxy33.executeStatementAsync(Unknown Source) ~[?:?]
> at
> org.apache.hive.service.cli.CLIService.executeStatementAsync(CLIService.java:272)
> ~[hive-service-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT]
> at
> org.apache.hive.service.cli.thrift.ThriftCLIService.ExecuteStatement(ThriftCLIService.java:554)
> [hive-service-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT]
> at
> org.apache.hive.service.rpc.thrift.TCLIService$Processor$ExecuteStatement.getResult(TCLIService.java:1317)
> [hive-service-rpc-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT]
> at
> org.apache.hive.service.rpc.thrift.TCLIService$Processor$ExecuteStatement.getResult(TCLIService.java:1302)
> [hive-service-rpc-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT]
> at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
> [hive-exec-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT]
> at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39)
> [hive-exec-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT]
> at
> org.apache.hive.service.auth.TSetIpAddressProcessor.process(TSetIpAddressProcessor.java:56)
> [hive-service-2.1.0-SNAPSHOT
[jira] [Updated] (HIVE-11484) Fix ObjectInspector for Char and VarChar
[ https://issues.apache.org/jira/browse/HIVE-11484?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Amareshwari Sriramadasu updated HIVE-11484: --- Resolution: Fixed Hadoop Flags: Reviewed Fix Version/s: 2.1.0 Status: Resolved (was: Patch Available) Committed. Thanks [~deepak.barr] > Fix ObjectInspector for Char and VarChar > > > Key: HIVE-11484 > URL: https://issues.apache.org/jira/browse/HIVE-11484 > Project: Hive > Issue Type: Bug > Components: Serializers/Deserializers >Reporter: Amareshwari Sriramadasu >Assignee: Deepak Barr > Fix For: 2.1.0 > > Attachments: HIVE-11484.01.patch, HIVE-11484.02.patch > > > The creation of HiveChar and Varchar is not happening through ObjectInspector. > Here is fix we pushed internally : > https://github.com/InMobi/hive/commit/fe95c7850e7130448209141155f28b25d3504216 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13183) More logs in operation logs
[ https://issues.apache.org/jira/browse/HIVE-13183?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Amareshwari Sriramadasu updated HIVE-13183: --- Resolution: Fixed Status: Resolved (was: Patch Available) Committed. Thanks [~prongs] > More logs in operation logs > --- > > Key: HIVE-13183 > URL: https://issues.apache.org/jira/browse/HIVE-13183 > Project: Hive > Issue Type: Improvement >Reporter: Rajat Khandelwal >Assignee: Rajat Khandelwal > Fix For: 2.1.0 > > Attachments: HIVE-13183.02.patch, HIVE-13183.03.patch, > HIVE-13183.04.patch, HIVE-13183.05.patch, HIVE-13183.06.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-11484) Fix ObjectInspector for Char and VarChar
[ https://issues.apache.org/jira/browse/HIVE-11484?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15236610#comment-15236610 ] Amareshwari Sriramadasu commented on HIVE-11484: +1 for patch https://issues.apache.org/jira/secure/attachment/12796075/HIVE-11484.02.patch > Fix ObjectInspector for Char and VarChar > > > Key: HIVE-11484 > URL: https://issues.apache.org/jira/browse/HIVE-11484 > Project: Hive > Issue Type: Bug > Components: Serializers/Deserializers >Reporter: Amareshwari Sriramadasu >Assignee: Deepak Barr > Attachments: HIVE-11484.01.patch, HIVE-11484.02.patch > > > The creation of HiveChar and Varchar is not happening through ObjectInspector. > Here is fix we pushed internally : > https://github.com/InMobi/hive/commit/fe95c7850e7130448209141155f28b25d3504216 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13183) More logs in operation logs
[ https://issues.apache.org/jira/browse/HIVE-13183?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15234562#comment-15234562 ] Amareshwari Sriramadasu commented on HIVE-13183: +1 for latest patch - https://issues.apache.org/jira/secure/attachment/12796822/HIVE-13183.06.patch > More logs in operation logs > --- > > Key: HIVE-13183 > URL: https://issues.apache.org/jira/browse/HIVE-13183 > Project: Hive > Issue Type: Improvement >Reporter: Rajat Khandelwal >Assignee: Rajat Khandelwal > Fix For: 2.1.0 > > Attachments: HIVE-13183.02.patch, HIVE-13183.03.patch, > HIVE-13183.04.patch, HIVE-13183.05.patch, HIVE-13183.06.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13415) Decouple Sessions from thrift binary transport
[
https://issues.apache.org/jira/browse/HIVE-13415?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15234531#comment-15234531
]
Amareshwari Sriramadasu commented on HIVE-13415:
[~szehon], Not sure which one you looked at, but latest version on review board
seems fine.
> Decouple Sessions from thrift binary transport
> --
>
> Key: HIVE-13415
> URL: https://issues.apache.org/jira/browse/HIVE-13415
> Project: Hive
> Issue Type: Bug
>Affects Versions: 2.0.0
>Reporter: Rajat Khandelwal
>Assignee: Rajat Khandelwal
> Attachments: HIVE-13415.01.patch
>
>
> Current behaviour is:
> * Open a thrift binary transport
> * create a session
> * close the transport
> Then the session gets closed. Consequently, all the operations running in the
> session also get killed.
> Whereas, if you open an HTTP transport, and close, the enclosing sessions are
> not closed.
> This seems like a bad design, having transport and sessions tightly coupled.
> I'd like to fix this.
> The issue that introduced it is
> [HIVE-9601|https://github.com/apache/hive/commit/48bea00c48853459af64b4ca9bfdc3e821c4ed82]
> Relevant discussions at
> [here|https://issues.apache.org/jira/browse/HIVE-11485?focusedCommentId=15223546&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15223546],
>
> [here|https://issues.apache.org/jira/browse/HIVE-11485?focusedCommentId=15223827&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15223827]
> and mentioned links on those comments.
> Another thing that seems like a slightly bad design is this line of code in
> ThriftBinaryCLIService:
> {noformat}
> server.setServerEventHandler(serverEventHandler);
> {noformat}
> Whereas serverEventHandler is defined by the base class, with no users except
> one sub-class(ThriftBinaryCLIService), violating the separation of concerns.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-11485) Session close should not close async SQL operations
[ https://issues.apache.org/jira/browse/HIVE-11485?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Amareshwari Sriramadasu updated HIVE-11485: --- Status: Open (was: Patch Available) Cancelling patch as it needs more work on consensus on the design. > Session close should not close async SQL operations > --- > > Key: HIVE-11485 > URL: https://issues.apache.org/jira/browse/HIVE-11485 > Project: Hive > Issue Type: Improvement > Components: HiveServer2 >Reporter: Amareshwari Sriramadasu >Assignee: Deepak Barr > Attachments: HIVE-11485.master.patch > > > Right now, session close on HiveServer closes all operations. But, queries > running are actually available across sessions and they are not tied to a > session (expect the launch - which requires configuration and resources). And > it allows getting the status of the query across sessions. > But session close of the session ( on which operation is launched) closes all > the operations as well. > So, we should avoid closing all operations upon closing a session. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-11485) Session close should not close async SQL operations
[ https://issues.apache.org/jira/browse/HIVE-11485?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15217703#comment-15217703 ] Amareshwari Sriramadasu commented on HIVE-11485: After HIVE-9601, we are seeing the issue is prominent. Whenever the thrift connection closes, the session gets closed and causes the running query to fail. Can we have a consensus on this and make progress? > Session close should not close async SQL operations > --- > > Key: HIVE-11485 > URL: https://issues.apache.org/jira/browse/HIVE-11485 > Project: Hive > Issue Type: Improvement > Components: HiveServer2 >Reporter: Amareshwari Sriramadasu >Assignee: Deepak Barr > Attachments: HIVE-11485.master.patch > > > Right now, session close on HiveServer closes all operations. But, queries > running are actually available across sessions and they are not tied to a > session (expect the launch - which requires configuration and resources). And > it allows getting the status of the query across sessions. > But session close of the session ( on which operation is launched) closes all > the operations as well. > So, we should avoid closing all operations upon closing a session. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13319) Propagate external handles in task display
[ https://issues.apache.org/jira/browse/HIVE-13319?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Amareshwari Sriramadasu updated HIVE-13319: --- Resolution: Fixed Hadoop Flags: Reviewed Fix Version/s: 2.1.0 Status: Resolved (was: Patch Available) Committed. Thanks [~prongs] > Propagate external handles in task display > -- > > Key: HIVE-13319 > URL: https://issues.apache.org/jira/browse/HIVE-13319 > Project: Hive > Issue Type: Improvement >Reporter: Rajat Khandelwal >Assignee: Rajat Khandelwal > Fix For: 2.1.0 > > Attachments: HIVE-13319.02.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13319) Propagate external handles in task display
[ https://issues.apache.org/jira/browse/HIVE-13319?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15203718#comment-15203718 ] Amareshwari Sriramadasu commented on HIVE-13319: +1 for https://issues.apache.org/jira/secure/attachment/12794429/HIVE-13319.02.patch > Propagate external handles in task display > -- > > Key: HIVE-13319 > URL: https://issues.apache.org/jira/browse/HIVE-13319 > Project: Hive > Issue Type: Improvement >Reporter: Rajat Khandelwal >Assignee: Rajat Khandelwal > Attachments: HIVE-13319.02.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13183) More logs in operation logs
[ https://issues.apache.org/jira/browse/HIVE-13183?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15198728#comment-15198728 ] Amareshwari Sriramadasu commented on HIVE-13183: +1 for https://issues.apache.org/jira/secure/attachment/12793835/HIVE-13183.05.patch > More logs in operation logs > --- > > Key: HIVE-13183 > URL: https://issues.apache.org/jira/browse/HIVE-13183 > Project: Hive > Issue Type: Improvement >Reporter: Rajat Khandelwal >Assignee: Rajat Khandelwal > Attachments: HIVE-13183.02.patch, HIVE-13183.03.patch, > HIVE-13183.04.patch, HIVE-13183.05.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13183) More logs in operation logs
[ https://issues.apache.org/jira/browse/HIVE-13183?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Amareshwari Sriramadasu updated HIVE-13183: --- Resolution: Fixed Hadoop Flags: Reviewed Fix Version/s: 2.1.0 Status: Resolved (was: Patch Available) > More logs in operation logs > --- > > Key: HIVE-13183 > URL: https://issues.apache.org/jira/browse/HIVE-13183 > Project: Hive > Issue Type: Improvement >Reporter: Rajat Khandelwal >Assignee: Rajat Khandelwal > Fix For: 2.1.0 > > Attachments: HIVE-13183.02.patch, HIVE-13183.03.patch, > HIVE-13183.04.patch, HIVE-13183.05.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13183) More logs in operation logs
[ https://issues.apache.org/jira/browse/HIVE-13183?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15201081#comment-15201081 ] Amareshwari Sriramadasu commented on HIVE-13183: Committed. Thanks [~prongs] > More logs in operation logs > --- > > Key: HIVE-13183 > URL: https://issues.apache.org/jira/browse/HIVE-13183 > Project: Hive > Issue Type: Improvement >Reporter: Rajat Khandelwal >Assignee: Rajat Khandelwal > Fix For: 2.1.0 > > Attachments: HIVE-13183.02.patch, HIVE-13183.03.patch, > HIVE-13183.04.patch, HIVE-13183.05.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Reopened] (HIVE-13183) More logs in operation logs
[ https://issues.apache.org/jira/browse/HIVE-13183?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Amareshwari Sriramadasu reopened HIVE-13183: Reverted the commit. Thanks [~sershe] for noting the broken test. I though the failure was unrelated. > More logs in operation logs > --- > > Key: HIVE-13183 > URL: https://issues.apache.org/jira/browse/HIVE-13183 > Project: Hive > Issue Type: Improvement >Reporter: Rajat Khandelwal >Assignee: Rajat Khandelwal > Fix For: 2.1.0 > > Attachments: HIVE-13183.02.patch, HIVE-13183.03.patch, > HIVE-13183.04.patch, HIVE-13183.05.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-4570) More information to user on GetOperationStatus in Hive Server2 when query is still executing
[ https://issues.apache.org/jira/browse/HIVE-4570?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15192841#comment-15192841 ] Amareshwari Sriramadasu commented on HIVE-4570: --- Thanks [~akshaygoyal] and [~jaideepdhok] for your contributions on this issue. > More information to user on GetOperationStatus in Hive Server2 when query is > still executing > > > Key: HIVE-4570 > URL: https://issues.apache.org/jira/browse/HIVE-4570 > Project: Hive > Issue Type: Improvement > Components: HiveServer2 >Reporter: Amareshwari Sriramadasu >Assignee: Rajat Khandelwal > Fix For: 2.1.0 > > Attachments: HIVE-4570.01.patch, HIVE-4570.02.patch, > HIVE-4570.03.patch, HIVE-4570.03.patch, HIVE-4570.04.patch, > HIVE-4570.04.patch, HIVE-4570.06.patch, HIVE-4570.07.patch > > > Currently in Hive Server2, when the query is still executing only the status > is set as STILL_EXECUTING. > This issue is to give more information to the user such as progress and > running job handles, if possible. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-4570) More information to user on GetOperationStatus in Hive Server2 when query is still executing
[ https://issues.apache.org/jira/browse/HIVE-4570?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Amareshwari Sriramadasu updated HIVE-4570: -- Resolution: Fixed Hadoop Flags: Reviewed Fix Version/s: 2.1.0 Status: Resolved (was: Patch Available) Committed. Thanks [~prongs] > More information to user on GetOperationStatus in Hive Server2 when query is > still executing > > > Key: HIVE-4570 > URL: https://issues.apache.org/jira/browse/HIVE-4570 > Project: Hive > Issue Type: Improvement > Components: HiveServer2 >Reporter: Amareshwari Sriramadasu >Assignee: Rajat Khandelwal > Fix For: 2.1.0 > > Attachments: HIVE-4570.01.patch, HIVE-4570.02.patch, > HIVE-4570.03.patch, HIVE-4570.03.patch, HIVE-4570.04.patch, > HIVE-4570.04.patch, HIVE-4570.06.patch, HIVE-4570.07.patch > > > Currently in Hive Server2, when the query is still executing only the status > is set as STILL_EXECUTING. > This issue is to give more information to the user such as progress and > running job handles, if possible. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-4570) More information to user on GetOperationStatus in Hive Server2 when query is still executing
[ https://issues.apache.org/jira/browse/HIVE-4570?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15192829#comment-15192829 ] Amareshwari Sriramadasu commented on HIVE-4570: --- +1 > More information to user on GetOperationStatus in Hive Server2 when query is > still executing > > > Key: HIVE-4570 > URL: https://issues.apache.org/jira/browse/HIVE-4570 > Project: Hive > Issue Type: Improvement > Components: HiveServer2 >Reporter: Amareshwari Sriramadasu >Assignee: Rajat Khandelwal > Attachments: HIVE-4570.01.patch, HIVE-4570.02.patch, > HIVE-4570.03.patch, HIVE-4570.03.patch, HIVE-4570.04.patch, > HIVE-4570.04.patch, HIVE-4570.06.patch, HIVE-4570.07.patch > > > Currently in Hive Server2, when the query is still executing only the status > is set as STILL_EXECUTING. > This issue is to give more information to the user such as progress and > running job handles, if possible. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-11483) Add encoding and decoding for query string config
[ https://issues.apache.org/jira/browse/HIVE-11483?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Amareshwari Sriramadasu updated HIVE-11483: --- Resolution: Fixed Hadoop Flags: Reviewed Fix Version/s: 2.1.0 Status: Resolved (was: Patch Available) Committed. Thanks [~prongs] > Add encoding and decoding for query string config > - > > Key: HIVE-11483 > URL: https://issues.apache.org/jira/browse/HIVE-11483 > Project: Hive > Issue Type: Improvement > Components: Query Processor >Reporter: Amareshwari Sriramadasu >Assignee: Rajat Khandelwal > Fix For: 2.1.0 > > Attachments: HIVE-11483.01.patch, HIVE-11483.02.patch, > HIVE-11483.03.patch, HIVE-11483.04.patch, HIVE-11483.05.patch > > > We have seen some queries in production where some of the literals passed in > the query have control characters, which result in exception when query > string is set in the job xml. > Proposing a solution to encode the query string in configuration and provide > getters decoded string. > Here is a commit in a forked repo : > https://github.com/InMobi/hive/commit/2faf5761191fa3103a0d779fde584d494ed75bf5 > Suggestions are welcome on the solution. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13188) Allow users of RetryingThriftClient to close transport
[ https://issues.apache.org/jira/browse/HIVE-13188?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Amareshwari Sriramadasu updated HIVE-13188: --- Resolution: Fixed Hadoop Flags: Reviewed Fix Version/s: 2.1.0 Status: Resolved (was: Patch Available) Committed. Thanks [~prongs] > Allow users of RetryingThriftClient to close transport > -- > > Key: HIVE-13188 > URL: https://issues.apache.org/jira/browse/HIVE-13188 > Project: Hive > Issue Type: Task >Reporter: Rajat Khandelwal >Assignee: Rajat Khandelwal > Fix For: 2.1.0 > > Attachments: HIVE-13188.02.patch, HIVE-13188.03.patch > > > RetryingThriftCLIClient opens a TTransport and leaves it open. there should > be a way to close that. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-5370) format_number udf should take user specifed format as argument
[ https://issues.apache.org/jira/browse/HIVE-5370?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Amareshwari Sriramadasu updated HIVE-5370: -- Resolution: Fixed Hadoop Flags: Reviewed Status: Resolved (was: Patch Available) Committed. > format_number udf should take user specifed format as argument > -- > > Key: HIVE-5370 > URL: https://issues.apache.org/jira/browse/HIVE-5370 > Project: Hive > Issue Type: Improvement > Components: UDF >Reporter: Amareshwari Sriramadasu >Assignee: Amareshwari Sriramadasu >Priority: Minor > Fix For: 2.1.0 > > Attachments: D13185.1.patch, D13185.2.patch, HIVE-5370.2.patch, > HIVE-5370.3.patch, HIVE-5370.patch, HIVE-5370.patch > > > Currently, format_number udf formats the number to #,###,###.##, but it > should also take a user specified format as optional input. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13179) Allow custom HiveConf to be passed to Authentication Providers
[ https://issues.apache.org/jira/browse/HIVE-13179?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Amareshwari Sriramadasu updated HIVE-13179: --- Resolution: Fixed Hadoop Flags: Reviewed Fix Version/s: 2.1.0 Status: Resolved (was: Patch Available) Committed. Thanks [~prongs] > Allow custom HiveConf to be passed to Authentication Providers > -- > > Key: HIVE-13179 > URL: https://issues.apache.org/jira/browse/HIVE-13179 > Project: Hive > Issue Type: Improvement >Reporter: Rajat Khandelwal >Assignee: Rajat Khandelwal > Fix For: 2.1.0 > > Attachments: HIVE-13179.05.patch, HIVE-13179.1.patch, > HIVE-13179.patch, HIVE-13179.patch > > > Right now if I want to create an ldap auth provider, I have to create a > hive-site.xml, set endpoints and other relevant properties there, then > instantiate `LdapAuthenticationProviderImpl`, since inside the constructor, a > new HiveConf is constructed. > A better and more reusable design would be to ask for the conf in the > constructor itself. That will allow an external user to create a HiveConf, > set all relevant properties and instantiate `LdapAuthenticationProviderImpl` > with that conf. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13188) Allow users of RetryingThriftClient to close transport
[ https://issues.apache.org/jira/browse/HIVE-13188?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15179283#comment-15179283 ] Amareshwari Sriramadasu commented on HIVE-13188: +1 for https://issues.apache.org/jira/secure/attachment/12791124/HIVE-13188.03.patch > Allow users of RetryingThriftClient to close transport > -- > > Key: HIVE-13188 > URL: https://issues.apache.org/jira/browse/HIVE-13188 > Project: Hive > Issue Type: Task >Reporter: Rajat Khandelwal >Assignee: Rajat Khandelwal > Attachments: HIVE-13188.02.patch, HIVE-13188.03.patch > > > RetryingThriftCLIClient opens a TTransport and leaves it open. there should > be a way to close that. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13179) Allow custom HiveConf to be passed to Authentication Providers
[ https://issues.apache.org/jira/browse/HIVE-13179?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15179281#comment-15179281 ] Amareshwari Sriramadasu commented on HIVE-13179: +1 for https://issues.apache.org/jira/secure/attachment/12791127/HIVE-13179.05.patch > Allow custom HiveConf to be passed to Authentication Providers > -- > > Key: HIVE-13179 > URL: https://issues.apache.org/jira/browse/HIVE-13179 > Project: Hive > Issue Type: Improvement >Reporter: Rajat Khandelwal >Assignee: Rajat Khandelwal > Attachments: HIVE-13179.05.patch, HIVE-13179.1.patch, > HIVE-13179.patch, HIVE-13179.patch > > > Right now if I want to create an ldap auth provider, I have to create a > hive-site.xml, set endpoints and other relevant properties there, then > instantiate `LdapAuthenticationProviderImpl`, since inside the constructor, a > new HiveConf is constructed. > A better and more reusable design would be to ask for the conf in the > constructor itself. That will allow an external user to create a HiveConf, > set all relevant properties and instantiate `LdapAuthenticationProviderImpl` > with that conf. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-11483) Add encoding and decoding for query string config
[ https://issues.apache.org/jira/browse/HIVE-11483?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15179279#comment-15179279 ] Amareshwari Sriramadasu commented on HIVE-11483: +1 > Add encoding and decoding for query string config > - > > Key: HIVE-11483 > URL: https://issues.apache.org/jira/browse/HIVE-11483 > Project: Hive > Issue Type: Improvement > Components: Query Processor >Reporter: Amareshwari Sriramadasu >Assignee: Rajat Khandelwal > Attachments: HIVE-11483.01.patch, HIVE-11483.02.patch, > HIVE-11483.03.patch, HIVE-11483.04.patch > > > We have seen some queries in production where some of the literals passed in > the query have control characters, which result in exception when query > string is set in the job xml. > Proposing a solution to encode the query string in configuration and provide > getters decoded string. > Here is a commit in a forked repo : > https://github.com/InMobi/hive/commit/2faf5761191fa3103a0d779fde584d494ed75bf5 > Suggestions are welcome on the solution. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-5370) format_number udf should take user specifed format as argument
[ https://issues.apache.org/jira/browse/HIVE-5370?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15177511#comment-15177511 ] Amareshwari Sriramadasu commented on HIVE-5370: --- Test failures say " java.lang.NoClassDefFoundError: Could not initialize class org.apache.hadoop.hive.cli.TestMiniTezCliDriver". Not related to the patch. > format_number udf should take user specifed format as argument > -- > > Key: HIVE-5370 > URL: https://issues.apache.org/jira/browse/HIVE-5370 > Project: Hive > Issue Type: Improvement > Components: UDF >Reporter: Amareshwari Sriramadasu >Assignee: Amareshwari Sriramadasu >Priority: Minor > Fix For: 2.1.0 > > Attachments: D13185.1.patch, D13185.2.patch, HIVE-5370.2.patch, > HIVE-5370.3.patch, HIVE-5370.patch, HIVE-5370.patch > > > Currently, format_number udf formats the number to #,###,###.##, but it > should also take a user specified format as optional input. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-5370) format_number udf should take user specifed format as argument
[ https://issues.apache.org/jira/browse/HIVE-5370?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Amareshwari Sriramadasu updated HIVE-5370: -- Status: Patch Available (was: Open) > format_number udf should take user specifed format as argument > -- > > Key: HIVE-5370 > URL: https://issues.apache.org/jira/browse/HIVE-5370 > Project: Hive > Issue Type: Improvement > Components: UDF >Reporter: Amareshwari Sriramadasu >Assignee: Amareshwari Sriramadasu >Priority: Minor > Fix For: 2.1.0 > > Attachments: D13185.1.patch, D13185.2.patch, HIVE-5370.2.patch, > HIVE-5370.3.patch, HIVE-5370.patch, HIVE-5370.patch > > > Currently, format_number udf formats the number to #,###,###.##, but it > should also take a user specified format as optional input. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-5370) format_number udf should take user specifed format as argument
[ https://issues.apache.org/jira/browse/HIVE-5370?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Amareshwari Sriramadasu updated HIVE-5370: -- Attachment: HIVE-5370.3.patch Updated patch with review comments incorporated > format_number udf should take user specifed format as argument > -- > > Key: HIVE-5370 > URL: https://issues.apache.org/jira/browse/HIVE-5370 > Project: Hive > Issue Type: Improvement > Components: UDF >Reporter: Amareshwari Sriramadasu >Assignee: Amareshwari Sriramadasu >Priority: Minor > Fix For: 2.1.0 > > Attachments: D13185.1.patch, D13185.2.patch, HIVE-5370.2.patch, > HIVE-5370.3.patch, HIVE-5370.patch, HIVE-5370.patch > > > Currently, format_number udf formats the number to #,###,###.##, but it > should also take a user specified format as optional input. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-5370) format_number udf should take user specifed format as argument
[ https://issues.apache.org/jira/browse/HIVE-5370?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Amareshwari Sriramadasu updated HIVE-5370: -- Status: Open (was: Patch Available) > format_number udf should take user specifed format as argument > -- > > Key: HIVE-5370 > URL: https://issues.apache.org/jira/browse/HIVE-5370 > Project: Hive > Issue Type: Improvement > Components: UDF >Reporter: Amareshwari Sriramadasu >Assignee: Amareshwari Sriramadasu >Priority: Minor > Fix For: 2.1.0 > > Attachments: D13185.1.patch, D13185.2.patch, HIVE-5370.2.patch, > HIVE-5370.patch, HIVE-5370.patch > > > Currently, format_number udf formats the number to #,###,###.##, but it > should also take a user specified format as optional input. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13179) Allow custom HiveConf to be passed to Authentication Providers
[ https://issues.apache.org/jira/browse/HIVE-13179?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15175556#comment-15175556 ] Amareshwari Sriramadasu commented on HIVE-13179: [~prongs], Can you check test failures TestHiveMetaStorePartitionSpecs and org.apache.hive.jdbc.TestSSL? Seems first time failures on the build. > Allow custom HiveConf to be passed to Authentication Providers > -- > > Key: HIVE-13179 > URL: https://issues.apache.org/jira/browse/HIVE-13179 > Project: Hive > Issue Type: Improvement >Reporter: Rajat Khandelwal >Assignee: Rajat Khandelwal > Attachments: HIVE-13179.1.patch, HIVE-13179.patch, HIVE-13179.patch > > > Right now if I want to create an ldap auth provider, I have to create a > hive-site.xml, set endpoints and other relevant properties there, then > instantiate `LdapAuthenticationProviderImpl`, since inside the constructor, a > new HiveConf is constructed. > A better and more reusable design would be to ask for the conf in the > constructor itself. That will allow an external user to create a HiveConf, > set all relevant properties and instantiate `LdapAuthenticationProviderImpl` > with that conf. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (HIVE-2270) Hive should not call methods from org.apache.hadoop.mapred.JobTracker
[ https://issues.apache.org/jira/browse/HIVE-2270?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Amareshwari Sriramadasu resolved HIVE-2270. --- Resolution: Not A Problem Assignee: (was: Amareshwari Sriramadasu) > Hive should not call methods from org.apache.hadoop.mapred.JobTracker > - > > Key: HIVE-2270 > URL: https://issues.apache.org/jira/browse/HIVE-2270 > Project: Hive > Issue Type: Bug >Reporter: Amareshwari Sriramadasu > > Currently Hive calls getAddress() and getState() api from > org.apache.hadoop.mapred.JobTracker. These should be replaced with > client-side ones if any, since the class is private(annotated) to hadoop. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-5370) format_number udf should take user specifed format as argument
[ https://issues.apache.org/jira/browse/HIVE-5370?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Amareshwari Sriramadasu updated HIVE-5370: -- Fix Version/s: 2.1.0 Status: Patch Available (was: Open) > format_number udf should take user specifed format as argument > -- > > Key: HIVE-5370 > URL: https://issues.apache.org/jira/browse/HIVE-5370 > Project: Hive > Issue Type: Improvement > Components: UDF >Reporter: Amareshwari Sriramadasu >Assignee: Amareshwari Sriramadasu >Priority: Minor > Fix For: 2.1.0 > > Attachments: D13185.1.patch, D13185.2.patch, HIVE-5370.2.patch, > HIVE-5370.patch, HIVE-5370.patch > > > Currently, format_number udf formats the number to #,###,###.##, but it > should also take a user specified format as optional input. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-5370) format_number udf should take user specifed format as argument
[ https://issues.apache.org/jira/browse/HIVE-5370?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Amareshwari Sriramadasu updated HIVE-5370: -- Attachment: HIVE-5370.2.patch Updated patch to master and verified all affected tests > format_number udf should take user specifed format as argument > -- > > Key: HIVE-5370 > URL: https://issues.apache.org/jira/browse/HIVE-5370 > Project: Hive > Issue Type: Improvement > Components: UDF >Reporter: Amareshwari Sriramadasu >Assignee: Amareshwari Sriramadasu >Priority: Minor > Attachments: D13185.1.patch, D13185.2.patch, HIVE-5370.2.patch, > HIVE-5370.patch, HIVE-5370.patch > > > Currently, format_number udf formats the number to #,###,###.##, but it > should also take a user specified format as optional input. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-5370) format_number udf should take user specifed format as argument
[ https://issues.apache.org/jira/browse/HIVE-5370?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Amareshwari Sriramadasu updated HIVE-5370: -- Target Version/s: 2.1.0 Status: Open (was: Patch Available) > format_number udf should take user specifed format as argument > -- > > Key: HIVE-5370 > URL: https://issues.apache.org/jira/browse/HIVE-5370 > Project: Hive > Issue Type: Improvement > Components: UDF >Reporter: Amareshwari Sriramadasu >Assignee: Amareshwari Sriramadasu >Priority: Minor > Attachments: D13185.1.patch, D13185.2.patch, HIVE-5370.patch, > HIVE-5370.patch > > > Currently, format_number udf formats the number to #,###,###.##, but it > should also take a user specified format as optional input. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-5370) format_number udf should take user specifed format as argument
[ https://issues.apache.org/jira/browse/HIVE-5370?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15171826#comment-15171826 ] Amareshwari Sriramadasu commented on HIVE-5370: --- Can anyone review this patch and provide feedback? The jira is PATCH AVAILABLE from more than one year. > format_number udf should take user specifed format as argument > -- > > Key: HIVE-5370 > URL: https://issues.apache.org/jira/browse/HIVE-5370 > Project: Hive > Issue Type: Improvement > Components: UDF >Reporter: Amareshwari Sriramadasu >Assignee: Amareshwari Sriramadasu >Priority: Minor > Attachments: D13185.1.patch, D13185.2.patch, HIVE-5370.patch, > HIVE-5370.patch > > > Currently, format_number udf formats the number to #,###,###.##, but it > should also take a user specified format as optional input. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13034) Add jdeb plugin to build debian
[ https://issues.apache.org/jira/browse/HIVE-13034?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Amareshwari Sriramadasu updated HIVE-13034: --- Resolution: Fixed Status: Resolved (was: Patch Available) Committed the incremental patch. Thanks [~arshadmatin20]! > Add jdeb plugin to build debian > --- > > Key: HIVE-13034 > URL: https://issues.apache.org/jira/browse/HIVE-13034 > Project: Hive > Issue Type: Improvement > Components: Build Infrastructure >Affects Versions: 2.1.0 >Reporter: Arshad Matin >Assignee: Arshad Matin > Fix For: 2.1.0 > > Attachments: HIVE-13034.1.patch, HIVE-13034.patch > > > It would be nice to also generate a debian as a part of build. This can be > done by adding jdeb plugin to dist profile. > NO PRECOMMIT TESTS -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13034) Add jdeb plugin to build debian
[ https://issues.apache.org/jira/browse/HIVE-13034?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15167103#comment-15167103 ] Amareshwari Sriramadasu commented on HIVE-13034: +1 for permission fix through https://issues.apache.org/jira/secure/attachment/12789906/HIVE-13034.1.patch > Add jdeb plugin to build debian > --- > > Key: HIVE-13034 > URL: https://issues.apache.org/jira/browse/HIVE-13034 > Project: Hive > Issue Type: Improvement > Components: Build Infrastructure >Affects Versions: 2.1.0 >Reporter: Arshad Matin >Assignee: Arshad Matin > Fix For: 2.1.0 > > Attachments: HIVE-13034.1.patch, HIVE-13034.patch > > > It would be nice to also generate a debian as a part of build. This can be > done by adding jdeb plugin to dist profile. > NO PRECOMMIT TESTS -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13034) Add jdeb plugin to build debian
[ https://issues.apache.org/jira/browse/HIVE-13034?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15162796#comment-15162796 ] Amareshwari Sriramadasu commented on HIVE-13034: +1 for https://issues.apache.org/jira/secure/attachment/12789531/HIVE-13034.patch > Add jdeb plugin to build debian > --- > > Key: HIVE-13034 > URL: https://issues.apache.org/jira/browse/HIVE-13034 > Project: Hive > Issue Type: Improvement >Reporter: Arshad Matin >Assignee: Arshad Matin > Attachments: HIVE-13034.patch > > > It would be nice to also generate a debian as a part of build. This can be > done by adding jdeb plugin to dist profile. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13064) Serde properties are not working while expecting output in a directory
[ https://issues.apache.org/jira/browse/HIVE-13064?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Amareshwari Sriramadasu updated HIVE-13064: --- Resolution: Fixed Hadoop Flags: Reviewed Fix Version/s: 2.1.0 Status: Resolved (was: Patch Available) > Serde properties are not working while expecting output in a directory > -- > > Key: HIVE-13064 > URL: https://issues.apache.org/jira/browse/HIVE-13064 > Project: Hive > Issue Type: Bug >Reporter: Rajat Khandelwal >Assignee: Rajat Khandelwal > Fix For: 2.1.0 > > Attachments: HIVE-13064.2.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13064) Serde properties are not working while expecting output in a directory
[ https://issues.apache.org/jira/browse/HIVE-13064?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15158335#comment-15158335 ] Amareshwari Sriramadasu commented on HIVE-13064: +1 for the patch https://issues.apache.org/jira/secure/attachment/12788672/HIVE-13064.2.patch. The test failures are not related to the patch. > Serde properties are not working while expecting output in a directory > -- > > Key: HIVE-13064 > URL: https://issues.apache.org/jira/browse/HIVE-13064 > Project: Hive > Issue Type: Bug >Reporter: Rajat Khandelwal >Assignee: Rajat Khandelwal > Attachments: HIVE-13064.2.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (HIVE-12973) Create Debian in Hive packaging module
[ https://issues.apache.org/jira/browse/HIVE-12973?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Amareshwari Sriramadasu resolved HIVE-12973. Resolution: Duplicate > Create Debian in Hive packaging module > -- > > Key: HIVE-12973 > URL: https://issues.apache.org/jira/browse/HIVE-12973 > Project: Hive > Issue Type: New Feature >Reporter: Rajat Khandelwal >Assignee: Rajat Khandelwal > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-12777) Add capability to restore session in CLIService and SessionManager
[ https://issues.apache.org/jira/browse/HIVE-12777?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15125923#comment-15125923 ] Amareshwari Sriramadasu commented on HIVE-12777: Not required > Add capability to restore session in CLIService and SessionManager > -- > > Key: HIVE-12777 > URL: https://issues.apache.org/jira/browse/HIVE-12777 > Project: Hive > Issue Type: Improvement >Reporter: Rajat Khandelwal >Assignee: Rajat Khandelwal > Fix For: 2.1.0 > > Attachments: HIVE-12777.04.patch, HIVE-12777.08.patch, > HIVE-12777.09.patch, HIVE-12777.11.patch, HIVE-12777.12.patch, > HIVE-12777.13.patch, HIVE-12777.15.patch, HIVE-12777.16.patch, > HIVE-12777.17.patch, HIVE-12777.18.patch, HIVE-12777.19.patch, > HIVE-12777.21.patch > > > Extensions using CLIService session handles should be able to restore the > session from the handle. > Apache Lens depends on a fork of hive and that fork has such a capability. > Functionality added: Restoring a session. A session opened once is lost once > the cli service is re-created. There may be some operation going on in that > session at the time the service is restarted. It's useful to be able to > restore a previously existing session. > Have added code in CLIService for that effect. Have also added a test class. > Relevant commit: > https://github.com/InMobi/hive/commit/931fe9116161a18952c082c14223ad6745fefe00#diff-0acb35f7cab7492f522b0c40ce3ce1be -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-12777) Add capability to restore session in CLIService and SessionManager
[ https://issues.apache.org/jira/browse/HIVE-12777?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15123413#comment-15123413 ] Amareshwari Sriramadasu commented on HIVE-12777: The patch https://issues.apache.org/jira/secure/attachment/12785113/HIVE-12777.21.patch looks fine to me. +1 > Add capability to restore session in CLIService and SessionManager > -- > > Key: HIVE-12777 > URL: https://issues.apache.org/jira/browse/HIVE-12777 > Project: Hive > Issue Type: Improvement >Reporter: Rajat Khandelwal >Assignee: Rajat Khandelwal > Attachments: HIVE-12777.04.patch, HIVE-12777.08.patch, > HIVE-12777.09.patch, HIVE-12777.11.patch, HIVE-12777.12.patch, > HIVE-12777.13.patch, HIVE-12777.15.patch, HIVE-12777.16.patch, > HIVE-12777.17.patch, HIVE-12777.18.patch, HIVE-12777.19.patch, > HIVE-12777.21.patch > > > Extensions using CLIService session handles should be able to restore the > session from the handle. > Apache Lens depends on a fork of hive and that fork has such a capability. > Functionality added: Restoring a session. A session opened once is lost once > the cli service is re-created. There may be some operation going on in that > session at the time the service is restarted. It's useful to be able to > restore a previously existing session. > Have added code in CLIService for that effect. Have also added a test class. > Relevant commit: > https://github.com/InMobi/hive/commit/931fe9116161a18952c082c14223ad6745fefe00#diff-0acb35f7cab7492f522b0c40ce3ce1be -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-12777) Add capability to restore session in CLIService and SessionManager
[ https://issues.apache.org/jira/browse/HIVE-12777?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15119153#comment-15119153 ] Amareshwari Sriramadasu commented on HIVE-12777: [~prongs], Posted some comments on review board. Also check if any of the failure reported by pre-commit build are related to the patch. Run them in master make sure. [~xuefuz], Agree with you : restoreSession on Hive Server crashes is more work and this jira is not doing that and It is doing a simple restore of the session handle. Let us know if you want any more to be added in this jira. This simple change would unblock us to move to hive master, and help move forward. > Add capability to restore session in CLIService and SessionManager > -- > > Key: HIVE-12777 > URL: https://issues.apache.org/jira/browse/HIVE-12777 > Project: Hive > Issue Type: Improvement >Reporter: Rajat Khandelwal >Assignee: Rajat Khandelwal > Attachments: HIVE-12777.04.patch, HIVE-12777.08.patch, > HIVE-12777.09.patch, HIVE-12777.11.patch, HIVE-12777.12.patch, > HIVE-12777.13.patch, HIVE-12777.15.patch, HIVE-12777.16.patch, > HIVE-12777.17.patch > > > Extensions using CLIService session handles should be able to restore the > session from the handle. > Apache Lens depends on a fork of hive and that fork has such a capability. > Functionality added: Restoring a session. A session opened once is lost once > the cli service is re-created. There may be some operation going on in that > session at the time the service is restarted. It's useful to be able to > restore a previously existing session. > Have added code in CLIService for that effect. Have also added a test class. > Relevant commit: > https://github.com/InMobi/hive/commit/931fe9116161a18952c082c14223ad6745fefe00#diff-0acb35f7cab7492f522b0c40ce3ce1be -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-12776) Add parse utility method for parsing any stand-alone HQL expression
[ https://issues.apache.org/jira/browse/HIVE-12776?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15093767#comment-15093767 ] Amareshwari Sriramadasu commented on HIVE-12776: [~leftylev], AFAIK, not required. > Add parse utility method for parsing any stand-alone HQL expression > --- > > Key: HIVE-12776 > URL: https://issues.apache.org/jira/browse/HIVE-12776 > Project: Hive > Issue Type: Improvement >Reporter: Rajat Khandelwal >Assignee: Rajat Khandelwal > Fix For: 2.1.0 > > Attachments: HIVE-12776.01.patch, HIVE-12776.02.patch > > > Extensions that use Hive QL as their standard language, will benefit from > this. > Apache Lens uses HQL as its language of choice. To support that, it depends > on a fork of Hive, which has such code. I'm planning to port that to Apache > Hive. > Relevant commit: > https://github.com/InMobi/hive/commit/7caea9ed1d269c1cd1d1326cb39c1db7e0bf2bba#diff-fb3acd67881ceb02e83c2e42cf70beef -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-11485) Session close should not close async SQL operations
[ https://issues.apache.org/jira/browse/HIVE-11485?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15093318#comment-15093318 ] Amareshwari Sriramadasu commented on HIVE-11485: bq. Users may not have any idea of asyc execution, such as those of Hue. If some users are running only sync execution, that should be perfectly fine. bq. It's way easy to leave those operations behind, consuming HS2 and cluster resources. My bad : I was under impression that all operations (except async executions) are closed immediately by Hive itself - seeing this - https://github.com/apache/hive/blob/master/service/src/java/org/apache/hive/service/cli/session/HiveSessionImpl.java#L443 - close is happening only in case of exceptions. Then it is a problem if session close is not clearing operations. bq. Not to mention that they don't even have a way to cancel unwanted running operations. Not sure what do you mean by that. cancel operation api already exists. But, sync executions return only after operation finishes. bq. Nothing prevents a user session from running days or as long as the operation takes. Sure. whats the story from Hive if the client crashes ? bq. Async execution was introduced as a problematic feature. It was a mistake in my opinion. I disagree here. For long running jobs, if the execute is returning only after it finishes, there is no way for clients to know the status or cancel the query. And if client crashes after launching the query, the query would launch MR job on cluster and finish to completion, but client wont be aware of it and there is no way to consume the result as well. [~xuefuz], Can you please share what is problematic in async execution feature, what problem do you see in the design? We should correct them if there are any problems. > Session close should not close async SQL operations > --- > > Key: HIVE-11485 > URL: https://issues.apache.org/jira/browse/HIVE-11485 > Project: Hive > Issue Type: Improvement > Components: HiveServer2 >Reporter: Amareshwari Sriramadasu >Assignee: Deepak Barr > Attachments: HIVE-11485.master.patch > > > Right now, session close on HiveServer closes all operations. But, queries > running are actually available across sessions and they are not tied to a > session (expect the launch - which requires configuration and resources). And > it allows getting the status of the query across sessions. > But session close of the session ( on which operation is launched) closes all > the operations as well. > So, we should avoid closing all operations upon closing a session. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-12776) Add parse utility method for parsing any stand-alone HQL expression
[ https://issues.apache.org/jira/browse/HIVE-12776?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15088681#comment-15088681 ] Amareshwari Sriramadasu commented on HIVE-12776: +1 for the patch. Test failures seem unrelated and they are all failing in other builds as well. > Add parse utility method for parsing any stand-alone HQL expression > --- > > Key: HIVE-12776 > URL: https://issues.apache.org/jira/browse/HIVE-12776 > Project: Hive > Issue Type: Improvement >Reporter: Rajat Khandelwal >Assignee: Rajat Khandelwal > Attachments: HIVE-12776.01.patch, HIVE-12776.02.patch > > > Extensions that use Hive QL as their standard language, will benefit from > this. > Apache Lens uses HQL as its language of choice. To support that, it depends > on a fork of Hive, which has such code. I'm planning to port that to Apache > Hive. > Relevant commit: > https://github.com/InMobi/hive/commit/7caea9ed1d269c1cd1d1326cb39c1db7e0bf2bba#diff-fb3acd67881ceb02e83c2e42cf70beef -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-12776) Add parse utility method for parsing any stand-alone HQL expression
[ https://issues.apache.org/jira/browse/HIVE-12776?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15088677#comment-15088677 ] Amareshwari Sriramadasu commented on HIVE-12776: [~alangates] and [~thejas], Agree with both of you. It should be fine. > Add parse utility method for parsing any stand-alone HQL expression > --- > > Key: HIVE-12776 > URL: https://issues.apache.org/jira/browse/HIVE-12776 > Project: Hive > Issue Type: Improvement >Reporter: Rajat Khandelwal >Assignee: Rajat Khandelwal > Attachments: HIVE-12776.01.patch, HIVE-12776.02.patch > > > Extensions that use Hive QL as their standard language, will benefit from > this. > Apache Lens uses HQL as its language of choice. To support that, it depends > on a fork of Hive, which has such code. I'm planning to port that to Apache > Hive. > Relevant commit: > https://github.com/InMobi/hive/commit/7caea9ed1d269c1cd1d1326cb39c1db7e0bf2bba#diff-fb3acd67881ceb02e83c2e42cf70beef -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-11485) Session close should not close async SQL operations
[ https://issues.apache.org/jira/browse/HIVE-11485?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15086790#comment-15086790 ] Amareshwari Sriramadasu commented on HIVE-11485: [~xuefuz], Please go through if the following makes sense. Asynchronous queries are meant to be for long running queries. They can take time from few minutes to hours to days depending on cluster load and data being queried. I agree one would need a session to launch an asynchronous query. But clients can get status of asynchronous queries from a different session as well and fetch results from different session. And the ask here is if launched session closed/expired by server, the query should not fail. Causing it to fail results in wastage of resources. And query is killed because of a session expiry by server, it is unfair to the user. bq. Leaving these operations behind after session is closed creates hazard in managing the operations and depletes server resources. We should enforce clients should call close operation for asynchronous operations. All other sync operations are closed immediately by HiveSession right now. The only operations closed upon session close are async operations, which is unfair in my opinion. If we still see orphan operations, we should add an expiry and close the operations. Thoughts? > Session close should not close async SQL operations > --- > > Key: HIVE-11485 > URL: https://issues.apache.org/jira/browse/HIVE-11485 > Project: Hive > Issue Type: Improvement > Components: HiveServer2 >Reporter: Amareshwari Sriramadasu >Assignee: Deepak Barr > Attachments: HIVE-11485.master.patch > > > Right now, session close on HiveServer closes all operations. But, queries > running are actually available across sessions and they are not tied to a > session (expect the launch - which requires configuration and resources). And > it allows getting the status of the query across sessions. > But session close of the session ( on which operation is launched) closes all > the operations as well. > So, we should avoid closing all operations upon closing a session. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-12776) Add parse utility method for parsing any stand-alone HQL expression
[ https://issues.apache.org/jira/browse/HIVE-12776?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Amareshwari Sriramadasu updated HIVE-12776: --- Summary: Add parse utility method for parsing any stand-alone HQL expression (was: Add code for parsing any stand-alone HQL expression) > Add parse utility method for parsing any stand-alone HQL expression > --- > > Key: HIVE-12776 > URL: https://issues.apache.org/jira/browse/HIVE-12776 > Project: Hive > Issue Type: Improvement >Reporter: Rajat Khandelwal >Assignee: Rajat Khandelwal > Attachments: HIVE-12776.01.patch, HIVE-12776.02.patch > > > Extensions that use Hive QL as their standard language, will benefit from > this. > Apache Lens uses HQL as its language of choice. To support that, it depends > on a fork of Hive, which has such code. I'm planning to port that to Apache > Hive. > Relevant commit: > https://github.com/InMobi/hive/commit/7caea9ed1d269c1cd1d1326cb39c1db7e0bf2bba#diff-fb3acd67881ceb02e83c2e42cf70beef -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-12776) Add code for parsing any stand-alone HQL expression
[ https://issues.apache.org/jira/browse/HIVE-12776?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15086783#comment-15086783 ] Amareshwari Sriramadasu commented on HIVE-12776: [~alangates], Let me try to answer your question. bq. what expectations are you putting on the format of Hive's AST? No expectations. Right now there is no way to look at AST of an expression. For ex: case when dim.x=0 then m1 else m2 end. You can get AST only for a query. The patch is adding utility method to get AST corresponding to an expression and not changing any of the existing parsing or any grammar. I did not understand why this would happen - "Without that though anyone who calls this method may see different AST's depending on the version of Hive they are working with". > Add code for parsing any stand-alone HQL expression > --- > > Key: HIVE-12776 > URL: https://issues.apache.org/jira/browse/HIVE-12776 > Project: Hive > Issue Type: Improvement >Reporter: Rajat Khandelwal >Assignee: Rajat Khandelwal > Attachments: HIVE-12776.01.patch, HIVE-12776.02.patch > > > Extensions that use Hive QL as their standard language, will benefit from > this. > Apache Lens uses HQL as its language of choice. To support that, it depends > on a fork of Hive, which has such code. I'm planning to port that to Apache > Hive. > Relevant commit: > https://github.com/InMobi/hive/commit/7caea9ed1d269c1cd1d1326cb39c1db7e0bf2bba#diff-fb3acd67881ceb02e83c2e42cf70beef -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Reopened] (HIVE-11487) Add getNumPartitionsByFilter api in metastore api
[ https://issues.apache.org/jira/browse/HIVE-11487?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Amareshwari Sriramadasu reopened HIVE-11487: Oops. I committed and just realised the generated code is with Autogenerated by Thrift Compiler (0.9.2). HIVE-11591 has changed the generated code to be with 0.9.3 and uses undated annotations [~akshaygoyal], please re-generate with 0.9.3 > Add getNumPartitionsByFilter api in metastore api > - > > Key: HIVE-11487 > URL: https://issues.apache.org/jira/browse/HIVE-11487 > Project: Hive > Issue Type: Improvement > Components: Metastore >Reporter: Amareshwari Sriramadasu >Assignee: Akshay Goyal > Fix For: 2.1.0 > > Attachments: HIVE-11487.01.patch, HIVE-11487.02.patch, > HIVE-11487.03.patch, HIVE-11487.04.patch, HIVE-11487.05.patch > > > Adding api for getting number of partitions for a filter will be more optimal > when we are only interested in the number. getAllPartitions will construct > all the partition object which can be time consuming and not required. > Here is a commit we pushed in a forked repo in our organization - > https://github.com/inmobi/hive/commit/68b3534d3e6c4d978132043cec668798ed53e444. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-11487) Add getNumPartitionsByFilter api in metastore api
[ https://issues.apache.org/jira/browse/HIVE-11487?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15063403#comment-15063403 ] Amareshwari Sriramadasu commented on HIVE-11487: +1 for the patch. Test failures are not related to the patch and they are being seen in other builds and master branch as well. > Add getNumPartitionsByFilter api in metastore api > - > > Key: HIVE-11487 > URL: https://issues.apache.org/jira/browse/HIVE-11487 > Project: Hive > Issue Type: Improvement > Components: Metastore >Reporter: Amareshwari Sriramadasu >Assignee: Akshay Goyal > Attachments: HIVE-11487.01.patch, HIVE-11487.02.patch, > HIVE-11487.03.patch, HIVE-11487.04.patch > > > Adding api for getting number of partitions for a filter will be more optimal > when we are only interested in the number. getAllPartitions will construct > all the partition object which can be time consuming and not required. > Here is a commit we pushed in a forked repo in our organization - > https://github.com/inmobi/hive/commit/68b3534d3e6c4d978132043cec668798ed53e444. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-8376) Umbrella Jira for HiveServer2 dynamic service discovery
[ https://issues.apache.org/jira/browse/HIVE-8376?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15052221#comment-15052221 ] Amareshwari Sriramadasu commented on HIVE-8376: --- [~vgumashta], I see all sub tasks of this are resolved. Is anything pending for this jira? Can this be marked resolved? If so, can you resolve it with proper fix version? > Umbrella Jira for HiveServer2 dynamic service discovery > --- > > Key: HIVE-8376 > URL: https://issues.apache.org/jira/browse/HIVE-8376 > Project: Hive > Issue Type: New Feature > Components: HiveServer2, JDBC >Affects Versions: 0.14.0 >Reporter: Vaibhav Gumashta >Assignee: Vaibhav Gumashta > Labels: TODOC14 > Attachments: HiveServer2DynamicServiceDiscovery.pdf > > > Creating an ☂ Jira for documentation purpose. I'll add a detailed doc for the > implementation & usage here. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-11482) Add retrying thrift client for HiveServer2
[ https://issues.apache.org/jira/browse/HIVE-11482?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14728547#comment-14728547 ] Amareshwari Sriramadasu commented on HIVE-11482: +1 for latest patch > Add retrying thrift client for HiveServer2 > -- > > Key: HIVE-11482 > URL: https://issues.apache.org/jira/browse/HIVE-11482 > Project: Hive > Issue Type: Improvement > Components: HiveServer2 >Reporter: Amareshwari Sriramadasu >Assignee: Akshay Goyal > Attachments: HIVE-11482.02.patch > > > Similar to > https://github.com/apache/hive/blob/master/metastore/src/java/org/apache/hadoop/hive/metastore/RetryingMetaStoreClient.java, > this improvement request is to add a retrying thrift client for HiveServer2 > to do retries upon thrift exceptions. > Here are few commits done on a forked branch that can be picked - > https://github.com/InMobi/hive/commit/7fb957fb9c2b6000d37c53294e256460010cb6b7 > https://github.com/InMobi/hive/commit/11e4b330f051c3f58927a276d562446761c9cd6d > https://github.com/InMobi/hive/commit/241386fd870373a9253dca0bcbdd4ea7e665406c -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-11485) Session close should not close async SQL operations
[ https://issues.apache.org/jira/browse/HIVE-11485?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14716180#comment-14716180 ] Amareshwari Sriramadasu commented on HIVE-11485: Also, session close should not result in running queries to fail. We have seen a scenario here - https://issues.apache.org/jira/browse/LENS-760?focusedCommentId=14716176&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14716176 > Session close should not close async SQL operations > --- > > Key: HIVE-11485 > URL: https://issues.apache.org/jira/browse/HIVE-11485 > Project: Hive > Issue Type: Improvement > Components: HiveServer2 >Reporter: Amareshwari Sriramadasu >Assignee: Deepak Barr > > Right now, session close on HiveServer closes all operations. But, queries > running are actually available across sessions and they are not tied to a > session (expect the launch - which requires configuration and resources). And > it allows getting the status of the query across sessions. > But session close of the session ( on which operation is launched) closes all > the operations as well. > So, we should avoid closing all operations upon closing a session. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-10276) Implement date_format(timestamp, fmt) UDF
[ https://issues.apache.org/jira/browse/HIVE-10276?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14696883#comment-14696883 ] Amareshwari Sriramadasu commented on HIVE-10276: The implementation done here does not look SQL compliant. For ex: D is day of year in SimpleDateFormat (https://docs.oracle.com/javase/7/docs/api/java/text/SimpleDateFormat.html), but is month of the year on SQL : https://dev.mysql.com/doc/refman/5.5/en/date-and-time-functions.html#function_date-format > Implement date_format(timestamp, fmt) UDF > - > > Key: HIVE-10276 > URL: https://issues.apache.org/jira/browse/HIVE-10276 > Project: Hive > Issue Type: Improvement > Components: UDF >Reporter: Alexander Pivovarov >Assignee: Alexander Pivovarov > Fix For: 1.2.0 > > Attachments: HIVE-10276.01.patch > > > date_format(date/timestamp/string, fmt) converts a date/timestamp/string to a > value of String in the format specified by the java date format fmt. > Supported formats listed here: > https://docs.oracle.com/javase/7/docs/api/java/text/SimpleDateFormat.html -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-11483) Add encoding and decoding for query string config
[ https://issues.apache.org/jira/browse/HIVE-11483?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14694685#comment-14694685 ] Amareshwari Sriramadasu commented on HIVE-11483: bq. but does any current Hive behavior depend on this? Or are you worried about external tools depending on it? The only current behavior is for logging, which we can fix by calling HiveConf.getDecodedString(). If there are no external tools consuming (i hope they are not), I think adding one config to say whether to encode or not - is unnecessary. But not to break any, I'm fine adding one more config. +1 for URL encoding then, as it is reasonably human readable. > Add encoding and decoding for query string config > - > > Key: HIVE-11483 > URL: https://issues.apache.org/jira/browse/HIVE-11483 > Project: Hive > Issue Type: Improvement > Components: Query Processor >Reporter: Amareshwari Sriramadasu >Assignee: Akshay Goyal > > We have seen some queries in production where some of the literals passed in > the query have control characters, which result in exception when query > string is set in the job xml. > Proposing a solution to encode the query string in configuration and provide > getters decoded string. > Here is a commit in a forked repo : > https://github.com/InMobi/hive/commit/2faf5761191fa3103a0d779fde584d494ed75bf5 > Suggestions are welcome on the solution. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-11483) Add encoding and decoding for query string config
[ https://issues.apache.org/jira/browse/HIVE-11483?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14693112#comment-14693112 ] Amareshwari Sriramadasu commented on HIVE-11483: Thanks [~alangates] for you comments. bq. Do we need the query string in the config at all? What on the backend reads it? Right now, the usage in hive is to set the query on the conf(which goes in xml), and is used for logging the query in PRE and POST exec hooks. bq. One issue with this is it makes the query string not human readable. Agree. But there are options to use online base64 decoders if user really want to. We can look at other options for encoding instead of Base64 like URL encoding. bq. If we want to preserve this should we instead just mask the control characters (ie change actual ctrl-A to ^A or something) How do we distinguish between actual ^A with control-A ? And how to decode them back. > Add encoding and decoding for query string config > - > > Key: HIVE-11483 > URL: https://issues.apache.org/jira/browse/HIVE-11483 > Project: Hive > Issue Type: Improvement > Components: Query Processor >Reporter: Amareshwari Sriramadasu > > We have seen some queries in production where some of the literals passed in > the query have control characters, which result in exception when query > string is set in the job xml. > Proposing a solution to encode the query string in configuration and provide > getters decoded string. > Here is a commit in a forked repo : > https://github.com/InMobi/hive/commit/2faf5761191fa3103a0d779fde584d494ed75bf5 > Suggestions are welcome on the solution. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-10435) Make HiveSession implementation pluggable through configuration
[ https://issues.apache.org/jira/browse/HIVE-10435?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14681687#comment-14681687 ] Amareshwari Sriramadasu commented on HIVE-10435: +1 for the latest patch, pending test results. > Make HiveSession implementation pluggable through configuration > --- > > Key: HIVE-10435 > URL: https://issues.apache.org/jira/browse/HIVE-10435 > Project: Hive > Issue Type: Improvement > Components: HiveServer2 >Reporter: Amareshwari Sriramadasu >Assignee: Akshay Goyal > Attachments: HIVE-10435.1.patch, HIVE-10435.2.patch > > > SessionManager in CLIService creates and keeps track of HiveSession. > Right now, it creates HiveSessionImpl which is one implementation of > HiveSession. This improvement request is to make it pluggable through a > configuration sothat other implementations can be passed. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-11507) Allow Custom Authentication Provider to take HiveConf Argument
[
https://issues.apache.org/jira/browse/HIVE-11507?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14680075#comment-14680075
]
Amareshwari Sriramadasu commented on HIVE-11507:
I think only change required is :
- CustomAuthenticationProviderImpl () {
-HiveConf conf = new HiveConf();
+ CustomAuthenticationProviderImpl(HiveConf conf) throws
AuthenticationException {
The other changes to do removing ReflectionUtils.newInstance are required,
because conf is getting passed to ReflectionUtils.
> Allow Custom Authentication Provider to take HiveConf Argument
> --
>
> Key: HIVE-11507
> URL: https://issues.apache.org/jira/browse/HIVE-11507
> Project: Hive
> Issue Type: Improvement
>Reporter: Rajat Khandelwal
>Assignee: Rajat Khandelwal
> Attachments: HIVE-11507.01.patch
>
>
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-11278) Partition.setOutputFormatClass should not do toString for Class object
[
https://issues.apache.org/jira/browse/HIVE-11278?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14679966#comment-14679966
]
Amareshwari Sriramadasu commented on HIVE-11278:
+1 for the patch
> Partition.setOutputFormatClass should not do toString for Class object
> ---
>
> Key: HIVE-11278
> URL: https://issues.apache.org/jira/browse/HIVE-11278
> Project: Hive
> Issue Type: Bug
>Reporter: Rajat Khandelwal
>Assignee: Rajat Khandelwal
> Attachments: HIVE-11278.01.patch
>
>
> https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/metadata/Partition.java#L286
> inside setInputFormatClass, we're doing:
> {noformat}
> public void setInputFormatClass(Class
> inputFormatClass) {
> this.inputFormatClass = inputFormatClass;
> tPartition.getSd().setInputFormat(inputFormatClass.getName());
> }
> {noformat}
> But inside setOutputFormatClass, we're doing toString for class, instead of
> getName().
> {noformat}
> public void setOutputFormatClass(Class
> outputFormatClass) {
> this.outputFormatClass = outputFormatClass;
> tPartition.getSd().setOutputFormat(HiveFileFormatUtils
> .getOutputFormatSubstitute(outputFormatClass).toString());
> }
> {noformat}
> Difference is that, for Class A.class, toString is "class A.class", getName
> is "A.class". So Class.forName(cls.getName()) succeeds, but
> Class.forName(cls.toString()) is not valid.
> So if you get a partition, set outputformat, and make an alter call, then get
> the partition again and make a getOutputFormatClass call on that object, it
> throws a ClassNotFoundException on
> https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/metadata/Partition.java#L316,
> because it's basically calling Class.forName("class a.b.c.ClassName.class")
> which is wrong!
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-11484) Fix ObjectInspector for Char and VarChar
[ https://issues.apache.org/jira/browse/HIVE-11484?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14659891#comment-14659891 ] Amareshwari Sriramadasu commented on HIVE-11484: Here is another commit - https://github.com/InMobi/hive/commit/d7b1916da379b5a310639d479604786b05499cb2 > Fix ObjectInspector for Char and VarChar > > > Key: HIVE-11484 > URL: https://issues.apache.org/jira/browse/HIVE-11484 > Project: Hive > Issue Type: Bug > Components: Serializers/Deserializers >Reporter: Amareshwari Sriramadasu > > The creation of HiveChar and Varchar is not happening through ObjectInspector. > Here is fix we pushed internally : > https://github.com/InMobi/hive/commit/fe95c7850e7130448209141155f28b25d3504216 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-4570) More information to user on GetOperationStatus in Hive Server2 when query is still executing
[
https://issues.apache.org/jira/browse/HIVE-4570?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14659856#comment-14659856
]
Amareshwari Sriramadasu commented on HIVE-4570:
---
Here is the modified thrift structure proposed :
{noformat}
struct TGetOperationStatusResp {
1: required TStatus status
2: optional TOperationState operationState
// List of statuses of sub tasks
3: optional string taskStatus
// If operationState is ERROR_STATE, then the following fields may be set
// sqlState as defined in the ISO/IEF CLI specification
4: optional string sqlState
// Internal error code
5: optional i32 errorCode
// Error message
6: optional string errorMessage
// When was the operation started
7: optional i64 operationStarted
// When was the operation completed
8: optional i64 operationCompleted
}
{noformat}
Here is few commits in forked repo in our organization, which can be picked up
as patch here :
https://github.com/inmobi/hive/commit/8eb3fd4a799157b1634876490c19061e257e83fd
https://github.com/InMobi/hive/commit/99475a9ed0dc840dd5445dcf100cd7abf322afc1
https://github.com/InMobi/hive/commit/85bf27311baaa4f83d928a39b44a1a182671b66f
> More information to user on GetOperationStatus in Hive Server2 when query is
> still executing
>
>
> Key: HIVE-4570
> URL: https://issues.apache.org/jira/browse/HIVE-4570
> Project: Hive
> Issue Type: Improvement
> Components: HiveServer2
>Affects Versions: 0.11.0
>Reporter: Amareshwari Sriramadasu
>
> Currently in Hive Server2, when the query is still executing only the status
> is set as STILL_EXECUTING.
> This issue is to give more information to the user such as progress and
> running job handles, if possible.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Reopened] (HIVE-4570) More information to user on GetOperationStatus in Hive Server2 when query is still executing
[ https://issues.apache.org/jira/browse/HIVE-4570?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Amareshwari Sriramadasu reopened HIVE-4570: --- Assignee: (was: Jaideep Dhok) Reopening as the functionality to view task/stage progress is not available yet. > More information to user on GetOperationStatus in Hive Server2 when query is > still executing > > > Key: HIVE-4570 > URL: https://issues.apache.org/jira/browse/HIVE-4570 > Project: Hive > Issue Type: Improvement > Components: HiveServer2 >Affects Versions: 0.11.0 >Reporter: Amareshwari Sriramadasu > > Currently in Hive Server2, when the query is still executing only the status > is set as STILL_EXECUTING. > This issue is to give more information to the user such as progress and > running job handles, if possible. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-11376) CombineHiveInputFormat is falling back to HiveInputFormat in case codecs are found for one of the input files
[
https://issues.apache.org/jira/browse/HIVE-11376?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14651480#comment-14651480
]
Amareshwari Sriramadasu commented on HIVE-11376:
+1 for the patch, pending test results.
> CombineHiveInputFormat is falling back to HiveInputFormat in case codecs are
> found for one of the input files
> -
>
> Key: HIVE-11376
> URL: https://issues.apache.org/jira/browse/HIVE-11376
> Project: Hive
> Issue Type: Bug
>Reporter: Rajat Khandelwal
>Assignee: Rajat Khandelwal
> Attachments: HIVE-11376_02.patch
>
>
> https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/io/CombineHiveInputFormat.java#L379
> This is the exact code snippet:
> {noformat}
> / Since there is no easy way of knowing whether MAPREDUCE-1597 is present in
> the tree or not,
> // we use a configuration variable for the same
> if (this.mrwork != null && !this.mrwork.getHadoopSupportsSplittable()) {
> // The following code should be removed, once
> // https://issues.apache.org/jira/browse/MAPREDUCE-1597 is fixed.
> // Hadoop does not handle non-splittable files correctly for
> CombineFileInputFormat,
> // so don't use CombineFileInputFormat for non-splittable files
> //ie, dont't combine if inputformat is a TextInputFormat and has
> compression turned on
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
