[jira] [Updated] (HIVE-21457) Perf optimizations in ORC split-generation
[ https://issues.apache.org/jira/browse/HIVE-21457?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Prasanth Jayachandran updated HIVE-21457: - Attachment: HIVE-21457.1.patch > Perf optimizations in ORC split-generation > -- > > Key: HIVE-21457 > URL: https://issues.apache.org/jira/browse/HIVE-21457 > Project: Hive > Issue Type: Improvement >Affects Versions: 4.0.0 >Reporter: Prasanth Jayachandran >Assignee: Prasanth Jayachandran >Priority: Minor > Attachments: HIVE-21457.1.patch > > > Minor split generation optimizations > * Reuse vectorization checks > * Reuse isAcid checks > * Reuse filesystem objects > * Improved logging (log at top-level instead of inside the thread pool) -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-21457) Perf optimizations in ORC split-generation
[ https://issues.apache.org/jira/browse/HIVE-21457?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Prasanth Jayachandran updated HIVE-21457: - Summary: Perf optimizations in ORC split-generation (was: Perf optimizations in split-generation) > Perf optimizations in ORC split-generation > -- > > Key: HIVE-21457 > URL: https://issues.apache.org/jira/browse/HIVE-21457 > Project: Hive > Issue Type: Improvement >Affects Versions: 4.0.0 >Reporter: Prasanth Jayachandran >Assignee: Prasanth Jayachandran >Priority: Minor > > Minor split generation optimizations > * Reuse vectorization checks > * Reuse isAcid checks > * Reuse filesystem objects > * Improved logging (log at top-level instead of inside the thread pool) -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (HIVE-21457) Perf optimizations in split-generation
[ https://issues.apache.org/jira/browse/HIVE-21457?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Prasanth Jayachandran reassigned HIVE-21457: > Perf optimizations in split-generation > -- > > Key: HIVE-21457 > URL: https://issues.apache.org/jira/browse/HIVE-21457 > Project: Hive > Issue Type: Improvement >Affects Versions: 4.0.0 >Reporter: Prasanth Jayachandran >Assignee: Prasanth Jayachandran >Priority: Minor > > Minor split generation optimizations > * Reuse vectorization checks > * Reuse isAcid checks > * Reuse filesystem objects > * Improved logging (log at top-level instead of inside the thread pool) -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-21369) LLAP: Logging is expensive in encoded reader path
[
https://issues.apache.org/jira/browse/HIVE-21369?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16793181#comment-16793181
]
Prasanth Jayachandran commented on HIVE-21369:
--
I am not sure if parametrized logging will help, but it is safer to change
LOG.isInfoEnabled() to LOG.isDebugEnabled().
> LLAP: Logging is expensive in encoded reader path
> -
>
> Key: HIVE-21369
> URL: https://issues.apache.org/jira/browse/HIVE-21369
> Project: Hive
> Issue Type: Bug
> Components: Logging
>Affects Versions: 4.0.0, 3.2.0
>Reporter: Prasanth Jayachandran
>Assignee: Nita Dembla
>Priority: Major
> Attachments: HIVE-21369.patch
>
>
> There should be no INFO logging in EncodedReaderImpl. Stringifying of disk
> ranges is expensive in core read path.
> {code:java}
> 2019-03-01T17:55:56.322852142Z 2019-03-01T17:55:56,306 INFO
> [IO-Elevator-Thread-3
> (hive_20190301175546_a279f33c-4f2b-4cd5-8695-57bc8b042a61)]
> encoded.EncodedReaderImpl: Disk ranges after cache (found everything true;
> file [-3693547618692831801, 1551190876000, 1047660824], base offset
> 792920167): [{start: 887940 end: 1003508 cache buffer: 0x5165f83d(1)},
> {start: 1003508 end: 1119078 cache buffer: 0xb63cac3(1)}, {start: 1119078
> end: 1234745 cache buffer: 0x41a724fa(1)}, {start: 1234745 end: 1350261 cache
> buffer: 0x2f71bc38(1)}, {start: 1350261 end: 1465752 cache buffer:
> 0x2c38e1bb(1)}, {start: 1465752 end: 1581231 cache buffer: 0x5827982(1)},
> {start: 1581231 end: 1696885 cache buffer: 0x75a6773c(1)}, {start: 1696885
> end: 1812492 cache buffer: 0x2ed060f9(1)},{start: 1812492 end: 1928086 cache
> buffer: 0x20b2c8aa(1)}, {start: 1928086 end: 2043588 cache buffer:
> 0x6559aacb(1)}, {start: 2043588 end: 2159089 cache buffer: 0x569c85e1(1)},
> {start: 2159089 end: 2274725 cache buffer: 0x25a88dd0(1)}, {start: 2274725
> end: 2390228 cache buffer: 0x738b7e87(1)}, {start: 2390228 end: 2505715 cache
> buffer: 0x26edafa0(1)}, {start: 2505715 end: 2621322 cache buffer:
> 0x69db7752(1)}, {start: 2621322 end: 2736844 cache b{code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-21390) BI split strategy does not work for blob stores
[ https://issues.apache.org/jira/browse/HIVE-21390?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Prasanth Jayachandran updated HIVE-21390: - Attachment: HIVE-21390.4.patch > BI split strategy does not work for blob stores > --- > > Key: HIVE-21390 > URL: https://issues.apache.org/jira/browse/HIVE-21390 > Project: Hive > Issue Type: Bug >Affects Versions: 4.0.0, 3.2.0 >Reporter: Prasanth Jayachandran >Assignee: Prasanth Jayachandran >Priority: Major > Attachments: HIVE-21390.1.patch, HIVE-21390.2.patch, > HIVE-21390.3.patch, HIVE-21390.4.patch > > > BI split strategy cuts the split at block boundaries however there are no > block boundaries in blob storage so we end up with 1 split for BI split > strategy. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-20656) Sensible defaults: Map aggregation memory configs are too aggressive
[
https://issues.apache.org/jira/browse/HIVE-20656?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Prasanth Jayachandran updated HIVE-20656:
-
Resolution: Fixed
Fix Version/s: 4.0.0
Status: Resolved (was: Patch Available)
Committed to master. Thanks for the review!
> Sensible defaults: Map aggregation memory configs are too aggressive
>
>
> Key: HIVE-20656
> URL: https://issues.apache.org/jira/browse/HIVE-20656
> Project: Hive
> Issue Type: Bug
>Affects Versions: 4.0.0, 3.2.0
>Reporter: Prasanth Jayachandran
>Assignee: Prasanth Jayachandran
>Priority: Major
> Fix For: 4.0.0
>
> Attachments: HIVE-20656.1.patch, HIVE-20656.2.patch,
> HIVE-20656.3.patch
>
>
> The defaults for the following configs seems to be too aggressive. In java
> this can easily lead to several full GC pauses whose memory cannot be
> reclaimed.
> {code:java}
> HIVEMAPAGGRHASHMEMORY("hive.map.aggr.hash.percentmemory", (float) 0.99,
> "Portion of total memory to be used by map-side group aggregation hash
> table"),
> HIVEMAPAGGRMEMORYTHRESHOLD("hive.map.aggr.hash.force.flush.memory.threshold",
> (float) 0.9,
> "The max memory to be used by map-side group aggregation hash table.\n" +
> "If the memory usage is higher than this number, force to flush
> data"),{code}
>
> We can be little bit conservative for these configs to avoid getting into GC
> pause.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-21339) LLAP: Cache hit also initializes an FS object
[
https://issues.apache.org/jira/browse/HIVE-21339?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Prasanth Jayachandran updated HIVE-21339:
-
Attachment: HIVE-21339.5.patch
> LLAP: Cache hit also initializes an FS object
> --
>
> Key: HIVE-21339
> URL: https://issues.apache.org/jira/browse/HIVE-21339
> Project: Hive
> Issue Type: Bug
> Components: llap
>Affects Versions: 4.0.0
>Reporter: Gopal V
>Assignee: Prasanth Jayachandran
>Priority: Major
> Attachments: HIVE-21339.1.patch, HIVE-21339.2.patch,
> HIVE-21339.3.patch, HIVE-21339.4.patch, HIVE-21339.5.patch,
> llap-cache-fs-get.png, llap-query7-cached.svg
>
>
> https://github.com/apache/hive/blob/master/llap-server/src/java/org/apache/hadoop/hive/llap/io/encoded/OrcEncodedDataReader.java#L214
> {code}
> // 1. Get file metadata from cache, or create the reader and read it.
> // Don't cache the filesystem object for now; Tez closes it and FS cache
> will fix all that
> fs = split.getPath().getFileSystem(jobConf);
> fileKey = determineFileId(fs, split,
> HiveConf.getBoolVar(daemonConf,
> ConfVars.LLAP_CACHE_ALLOW_SYNTHETIC_FILEID),
> HiveConf.getBoolVar(daemonConf,
> ConfVars.LLAP_CACHE_DEFAULT_FS_FILE_ID),
> !HiveConf.getBoolVar(daemonConf, ConfVars.LLAP_IO_USE_FILEID_PATH)
> );
> {code}
> !llap-cache-fs-get.png!
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-21339) LLAP: Cache hit also initializes an FS object
[
https://issues.apache.org/jira/browse/HIVE-21339?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16788452#comment-16788452
]
Prasanth Jayachandran commented on HIVE-21339:
--
{code:java}
Caused by: org.apache.hive.com.esotericsoftware.kryo.KryoException:
java.lang.IllegalArgumentException: Unable to create serializer
"org.apache.hive.com.esotericsoftware.kryo.serializers.FieldSerializer" for
class: org.apache.hadoop.hive.ql.io.TeradataBinaryFileOutputFormat
Serialization trace:
outputFileFormatClass (org.apache.hadoop.hive.ql.plan.TableDesc)
tableInfo (org.apache.hadoop.hive.ql.plan.FileSinkDesc)
conf (org.apache.hadoop.hive.ql.exec.FileSinkOperator)
childOperators (org.apache.hadoop.hive.ql.exec.SelectOperator)
parentOperators (org.apache.hadoop.hive.ql.exec.SelectOperator)
parentOperators (org.apache.hadoop.hive.ql.exec.GroupByOperator)
parentOperators (org.apache.hadoop.hive.ql.exec.ReduceSinkOperator)
parentOperators (org.apache.hadoop.hive.ql.exec.GroupByOperator)
reducer (org.apache.hadoop.hive.ql.plan.ReduceWork)
at
org.apache.hive.com.esotericsoftware.kryo.serializers.ObjectField.write(ObjectField.java:101)
at
org.apache.hive.com.esotericsoftware.kryo.serializers.FieldSerializer.write(FieldSerializer.java:518)
at
org.apache.hive.com.esotericsoftware.kryo.Kryo.writeObject(Kryo.java:552)
at
org.apache.hive.com.esotericsoftware.kryo.serializers.ObjectField.write(ObjectField.java:80)
at
org.apache.hive.com.esotericsoftware.kryo.serializers.FieldSerializer.write(FieldSerializer.java:518)
at
org.apache.hive.com.esotericsoftware.kryo.Kryo.writeObject(Kryo.java:552)
at
org.apache.hive.com.esotericsoftware.kryo.serializers.ObjectField.write(ObjectField.java:80)
at
org.apache.hive.com.esotericsoftware.kryo.serializers.FieldSerializer.write(FieldSerializer.java:518)
at
org.apache.hive.com.esotericsoftware.kryo.Kryo.writeClassAndObject(Kryo.java:628)
at
org.apache.hive.com.esotericsoftware.kryo.serializers.CollectionSerializer.write(CollectionSerializer.java:100)
at
org.apache.hive.com.esotericsoftware.kryo.serializers.CollectionSerializer.write(CollectionSerializer.java:40)
at
org.apache.hive.com.esotericsoftware.kryo.Kryo.writeObject(Kryo.java:552)
at
org.apache.hive.com.esotericsoftware.kryo.serializers.ObjectField.write(ObjectField.java:80)
at
org.apache.hive.com.esotericsoftware.kryo.serializers.FieldSerializer.write(FieldSerializer.java:518)
at
org.apache.hive.com.esotericsoftware.kryo.Kryo.writeClassAndObject(Kryo.java:628)
at
org.apache.hive.com.esotericsoftware.kryo.serializers.CollectionSerializer.write(CollectionSerializer.java:100)
at
org.apache.hive.com.esotericsoftware.kryo.serializers.CollectionSerializer.write(CollectionSerializer.java:40)
at
org.apache.hive.com.esotericsoftware.kryo.Kryo.writeObject(Kryo.java:552)
at
org.apache.hive.com.esotericsoftware.kryo.serializers.ObjectField.write(ObjectField.java:80)
at
org.apache.hive.com.esotericsoftware.kryo.serializers.FieldSerializer.write(FieldSerializer.java:518)
at
org.apache.hive.com.esotericsoftware.kryo.Kryo.writeClassAndObject(Kryo.java:628)
at
org.apache.hive.com.esotericsoftware.kryo.serializers.CollectionSerializer.write(CollectionSerializer.java:100)
at
org.apache.hive.com.esotericsoftware.kryo.serializers.CollectionSerializer.write(CollectionSerializer.java:40)
at
org.apache.hive.com.esotericsoftware.kryo.Kryo.writeObject(Kryo.java:552)
at
org.apache.hive.com.esotericsoftware.kryo.serializers.ObjectField.write(ObjectField.java:80)
at
org.apache.hive.com.esotericsoftware.kryo.serializers.FieldSerializer.write(FieldSerializer.java:518)
at
org.apache.hive.com.esotericsoftware.kryo.Kryo.writeClassAndObject(Kryo.java:628)
at
org.apache.hive.com.esotericsoftware.kryo.serializers.CollectionSerializer.write(CollectionSerializer.java:100)
at
org.apache.hive.com.esotericsoftware.kryo.serializers.CollectionSerializer.write(CollectionSerializer.java:40)
at
org.apache.hive.com.esotericsoftware.kryo.Kryo.writeObject(Kryo.java:552)
at
org.apache.hive.com.esotericsoftware.kryo.serializers.ObjectField.write(ObjectField.java:80)
at
org.apache.hive.com.esotericsoftware.kryo.serializers.FieldSerializer.write(FieldSerializer.java:518)
at
org.apache.hive.com.esotericsoftware.kryo.Kryo.writeClassAndObject(Kryo.java:628)
at
org.apache.hive.com.esotericsoftware.kryo.serializers.CollectionSerializer.write(CollectionSerializer.java:100)
at
org.apache.hive.com.esotericsoftware.kryo.serializers.CollectionSerializer.write(CollectionSerializer.java:40)
at
org.apache.hive.com.esotericsoftware.kryo.Kryo.writeObject(Kryo.java:552)
at
org.apache.
[jira] [Updated] (HIVE-21415) Parallel build is failing, trying to download incorrect hadoop-hdfs-client version
[
https://issues.apache.org/jira/browse/HIVE-21415?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Prasanth Jayachandran updated HIVE-21415:
-
Resolution: Fixed
Fix Version/s: 4.0.0
Status: Resolved (was: Patch Available)
Test failure is unrelated. Patch is benign as this touches only few pom files
for parallel build issue. Committed to master.
> Parallel build is failing, trying to download incorrect hadoop-hdfs-client
> version
> --
>
> Key: HIVE-21415
> URL: https://issues.apache.org/jira/browse/HIVE-21415
> Project: Hive
> Issue Type: Bug
>Affects Versions: 4.0.0
>Reporter: Prasanth Jayachandran
>Assignee: Prasanth Jayachandran
>Priority: Major
> Fix For: 4.0.0
>
> Attachments: HIVE-21415.1.patch
>
>
> Running the following build command
> {code:java}
> mvn clean install -Pdist -DskipTests -Dpackaging.minimizeJar=false -T 1C
> -DskipShade -Dremoteresources.skip=true -Dmaven.javadoc.skip=true{code}
> fails with the following exception for 3 modules (hplql, kryo-registrator,
> packaging)
> {code:java}
> [ERROR] Failed to execute goal on project hive-packaging: Could not resolve
> dependencies for project org.apache.hive:hive-packaging:pom:4.0.0-SNAPSHOT:
> Failure to find org.apache.hadoop:hadoop-hdfs-client:jar:2.7.3 in
> http://www.datanucleus.org/downloads/maven2 was cached in the local
> repository, resolution will not be reattempted until the update interval of
> datanucleus has elapsed or updates are forced -> [Help 1]
> [ERROR]
> [ERROR] To see the full stack trace of the errors, re-run Maven with the -e
> switch.
> [ERROR] Re-run Maven using the -X switch to enable full debug logging.
> [ERROR]
> [ERROR] For more information about the errors and possible solutions, please
> read the following articles:
> [ERROR] [Help 1]
> http://cwiki.apache.org/confluence/display/MAVEN/DependencyResolutionException
> [ERROR]
> [ERROR] After correcting the problems, you can resume the build with the
> command
> [ERROR] mvn -rf :hive-packaging{code}
>
> It is trying to download 2.7.3 version but hadoop.version refers to 3.1.0
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-21390) BI split strategy does not work for blob stores
[ https://issues.apache.org/jira/browse/HIVE-21390?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16788378#comment-16788378 ] Prasanth Jayachandran commented on HIVE-21390: -- My bad. Looking at the failure I ran the TestStreaming in top level streaming module instead of hcatalog streaming module. Will fix it. > BI split strategy does not work for blob stores > --- > > Key: HIVE-21390 > URL: https://issues.apache.org/jira/browse/HIVE-21390 > Project: Hive > Issue Type: Bug >Affects Versions: 4.0.0, 3.2.0 >Reporter: Prasanth Jayachandran >Assignee: Prasanth Jayachandran >Priority: Major > Attachments: HIVE-21390.1.patch, HIVE-21390.2.patch, > HIVE-21390.3.patch > > > BI split strategy cuts the split at block boundaries however there are no > block boundaries in blob storage so we end up with 1 split for BI split > strategy. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20656) Sensible defaults: Map aggregation memory configs are too aggressive
[
https://issues.apache.org/jira/browse/HIVE-20656?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16788348#comment-16788348
]
Prasanth Jayachandran commented on HIVE-20656:
--
Tried running both failures locally and it doesn't seem to fail. Not sure if it
is already fixed in master. Giving it another try.
> Sensible defaults: Map aggregation memory configs are too aggressive
>
>
> Key: HIVE-20656
> URL: https://issues.apache.org/jira/browse/HIVE-20656
> Project: Hive
> Issue Type: Bug
>Affects Versions: 4.0.0, 3.2.0
>Reporter: Prasanth Jayachandran
>Assignee: Prasanth Jayachandran
>Priority: Major
> Attachments: HIVE-20656.1.patch, HIVE-20656.2.patch,
> HIVE-20656.3.patch
>
>
> The defaults for the following configs seems to be too aggressive. In java
> this can easily lead to several full GC pauses whose memory cannot be
> reclaimed.
> {code:java}
> HIVEMAPAGGRHASHMEMORY("hive.map.aggr.hash.percentmemory", (float) 0.99,
> "Portion of total memory to be used by map-side group aggregation hash
> table"),
> HIVEMAPAGGRMEMORYTHRESHOLD("hive.map.aggr.hash.force.flush.memory.threshold",
> (float) 0.9,
> "The max memory to be used by map-side group aggregation hash table.\n" +
> "If the memory usage is higher than this number, force to flush
> data"),{code}
>
> We can be little bit conservative for these configs to avoid getting into GC
> pause.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-20656) Sensible defaults: Map aggregation memory configs are too aggressive
[
https://issues.apache.org/jira/browse/HIVE-20656?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Prasanth Jayachandran updated HIVE-20656:
-
Attachment: HIVE-20656.3.patch
> Sensible defaults: Map aggregation memory configs are too aggressive
>
>
> Key: HIVE-20656
> URL: https://issues.apache.org/jira/browse/HIVE-20656
> Project: Hive
> Issue Type: Bug
>Affects Versions: 4.0.0, 3.2.0
>Reporter: Prasanth Jayachandran
>Assignee: Prasanth Jayachandran
>Priority: Major
> Attachments: HIVE-20656.1.patch, HIVE-20656.2.patch,
> HIVE-20656.3.patch
>
>
> The defaults for the following configs seems to be too aggressive. In java
> this can easily lead to several full GC pauses whose memory cannot be
> reclaimed.
> {code:java}
> HIVEMAPAGGRHASHMEMORY("hive.map.aggr.hash.percentmemory", (float) 0.99,
> "Portion of total memory to be used by map-side group aggregation hash
> table"),
> HIVEMAPAGGRMEMORYTHRESHOLD("hive.map.aggr.hash.force.flush.memory.threshold",
> (float) 0.9,
> "The max memory to be used by map-side group aggregation hash table.\n" +
> "If the memory usage is higher than this number, force to flush
> data"),{code}
>
> We can be little bit conservative for these configs to avoid getting into GC
> pause.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Assigned] (HIVE-21415) Parallel build is failing, trying to download incorrect hadoop-hdfs-client version
[
https://issues.apache.org/jira/browse/HIVE-21415?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Prasanth Jayachandran reassigned HIVE-21415:
> Parallel build is failing, trying to download incorrect hadoop-hdfs-client
> version
> --
>
> Key: HIVE-21415
> URL: https://issues.apache.org/jira/browse/HIVE-21415
> Project: Hive
> Issue Type: Bug
>Affects Versions: 4.0.0
>Reporter: Prasanth Jayachandran
>Assignee: Prasanth Jayachandran
>Priority: Major
>
> Running the following build command
> {code:java}
> mvn clean install -Pdist -DskipTests -Dpackaging.minimizeJar=false -T 1C
> -DskipShade -Dremoteresources.skip=true -Dmaven.javadoc.skip=true{code}
> fails with the following exception for 3 modules (hplql, kryo-registrator,
> packaging)
> {code:java}
> [ERROR] Failed to execute goal on project hive-packaging: Could not resolve
> dependencies for project org.apache.hive:hive-packaging:pom:4.0.0-SNAPSHOT:
> Failure to find org.apache.hadoop:hadoop-hdfs-client:jar:2.7.3 in
> http://www.datanucleus.org/downloads/maven2 was cached in the local
> repository, resolution will not be reattempted until the update interval of
> datanucleus has elapsed or updates are forced -> [Help 1]
> [ERROR]
> [ERROR] To see the full stack trace of the errors, re-run Maven with the -e
> switch.
> [ERROR] Re-run Maven using the -X switch to enable full debug logging.
> [ERROR]
> [ERROR] For more information about the errors and possible solutions, please
> read the following articles:
> [ERROR] [Help 1]
> http://cwiki.apache.org/confluence/display/MAVEN/DependencyResolutionException
> [ERROR]
> [ERROR] After correcting the problems, you can resume the build with the
> command
> [ERROR] mvn -rf :hive-packaging{code}
>
> It is trying to download 2.7.3 version but hadoop.version refers to 3.1.0
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-21415) Parallel build is failing, trying to download incorrect hadoop-hdfs-client version
[
https://issues.apache.org/jira/browse/HIVE-21415?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16788345#comment-16788345
]
Prasanth Jayachandran commented on HIVE-21415:
--
[~vgarg] could you please review? Explicitly specifying version seems to be
working.
> Parallel build is failing, trying to download incorrect hadoop-hdfs-client
> version
> --
>
> Key: HIVE-21415
> URL: https://issues.apache.org/jira/browse/HIVE-21415
> Project: Hive
> Issue Type: Bug
>Affects Versions: 4.0.0
>Reporter: Prasanth Jayachandran
>Assignee: Prasanth Jayachandran
>Priority: Major
> Attachments: HIVE-21415.1.patch
>
>
> Running the following build command
> {code:java}
> mvn clean install -Pdist -DskipTests -Dpackaging.minimizeJar=false -T 1C
> -DskipShade -Dremoteresources.skip=true -Dmaven.javadoc.skip=true{code}
> fails with the following exception for 3 modules (hplql, kryo-registrator,
> packaging)
> {code:java}
> [ERROR] Failed to execute goal on project hive-packaging: Could not resolve
> dependencies for project org.apache.hive:hive-packaging:pom:4.0.0-SNAPSHOT:
> Failure to find org.apache.hadoop:hadoop-hdfs-client:jar:2.7.3 in
> http://www.datanucleus.org/downloads/maven2 was cached in the local
> repository, resolution will not be reattempted until the update interval of
> datanucleus has elapsed or updates are forced -> [Help 1]
> [ERROR]
> [ERROR] To see the full stack trace of the errors, re-run Maven with the -e
> switch.
> [ERROR] Re-run Maven using the -X switch to enable full debug logging.
> [ERROR]
> [ERROR] For more information about the errors and possible solutions, please
> read the following articles:
> [ERROR] [Help 1]
> http://cwiki.apache.org/confluence/display/MAVEN/DependencyResolutionException
> [ERROR]
> [ERROR] After correcting the problems, you can resume the build with the
> command
> [ERROR] mvn -rf :hive-packaging{code}
>
> It is trying to download 2.7.3 version but hadoop.version refers to 3.1.0
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-21415) Parallel build is failing, trying to download incorrect hadoop-hdfs-client version
[
https://issues.apache.org/jira/browse/HIVE-21415?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Prasanth Jayachandran updated HIVE-21415:
-
Attachment: HIVE-21415.1.patch
> Parallel build is failing, trying to download incorrect hadoop-hdfs-client
> version
> --
>
> Key: HIVE-21415
> URL: https://issues.apache.org/jira/browse/HIVE-21415
> Project: Hive
> Issue Type: Bug
>Affects Versions: 4.0.0
>Reporter: Prasanth Jayachandran
>Assignee: Prasanth Jayachandran
>Priority: Major
> Attachments: HIVE-21415.1.patch
>
>
> Running the following build command
> {code:java}
> mvn clean install -Pdist -DskipTests -Dpackaging.minimizeJar=false -T 1C
> -DskipShade -Dremoteresources.skip=true -Dmaven.javadoc.skip=true{code}
> fails with the following exception for 3 modules (hplql, kryo-registrator,
> packaging)
> {code:java}
> [ERROR] Failed to execute goal on project hive-packaging: Could not resolve
> dependencies for project org.apache.hive:hive-packaging:pom:4.0.0-SNAPSHOT:
> Failure to find org.apache.hadoop:hadoop-hdfs-client:jar:2.7.3 in
> http://www.datanucleus.org/downloads/maven2 was cached in the local
> repository, resolution will not be reattempted until the update interval of
> datanucleus has elapsed or updates are forced -> [Help 1]
> [ERROR]
> [ERROR] To see the full stack trace of the errors, re-run Maven with the -e
> switch.
> [ERROR] Re-run Maven using the -X switch to enable full debug logging.
> [ERROR]
> [ERROR] For more information about the errors and possible solutions, please
> read the following articles:
> [ERROR] [Help 1]
> http://cwiki.apache.org/confluence/display/MAVEN/DependencyResolutionException
> [ERROR]
> [ERROR] After correcting the problems, you can resume the build with the
> command
> [ERROR] mvn -rf :hive-packaging{code}
>
> It is trying to download 2.7.3 version but hadoop.version refers to 3.1.0
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-21415) Parallel build is failing, trying to download incorrect hadoop-hdfs-client version
[
https://issues.apache.org/jira/browse/HIVE-21415?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Prasanth Jayachandran updated HIVE-21415:
-
Status: Patch Available (was: Open)
> Parallel build is failing, trying to download incorrect hadoop-hdfs-client
> version
> --
>
> Key: HIVE-21415
> URL: https://issues.apache.org/jira/browse/HIVE-21415
> Project: Hive
> Issue Type: Bug
>Affects Versions: 4.0.0
>Reporter: Prasanth Jayachandran
>Assignee: Prasanth Jayachandran
>Priority: Major
> Attachments: HIVE-21415.1.patch
>
>
> Running the following build command
> {code:java}
> mvn clean install -Pdist -DskipTests -Dpackaging.minimizeJar=false -T 1C
> -DskipShade -Dremoteresources.skip=true -Dmaven.javadoc.skip=true{code}
> fails with the following exception for 3 modules (hplql, kryo-registrator,
> packaging)
> {code:java}
> [ERROR] Failed to execute goal on project hive-packaging: Could not resolve
> dependencies for project org.apache.hive:hive-packaging:pom:4.0.0-SNAPSHOT:
> Failure to find org.apache.hadoop:hadoop-hdfs-client:jar:2.7.3 in
> http://www.datanucleus.org/downloads/maven2 was cached in the local
> repository, resolution will not be reattempted until the update interval of
> datanucleus has elapsed or updates are forced -> [Help 1]
> [ERROR]
> [ERROR] To see the full stack trace of the errors, re-run Maven with the -e
> switch.
> [ERROR] Re-run Maven using the -X switch to enable full debug logging.
> [ERROR]
> [ERROR] For more information about the errors and possible solutions, please
> read the following articles:
> [ERROR] [Help 1]
> http://cwiki.apache.org/confluence/display/MAVEN/DependencyResolutionException
> [ERROR]
> [ERROR] After correcting the problems, you can resume the build with the
> command
> [ERROR] mvn -rf :hive-packaging{code}
>
> It is trying to download 2.7.3 version but hadoop.version refers to 3.1.0
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-21339) LLAP: Cache hit also initializes an FS object
[
https://issues.apache.org/jira/browse/HIVE-21339?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16788327#comment-16788327
]
Prasanth Jayachandran commented on HIVE-21339:
--
The failure looks unrelated. Another try.
> LLAP: Cache hit also initializes an FS object
> --
>
> Key: HIVE-21339
> URL: https://issues.apache.org/jira/browse/HIVE-21339
> Project: Hive
> Issue Type: Bug
> Components: llap
>Affects Versions: 4.0.0
>Reporter: Gopal V
>Assignee: Prasanth Jayachandran
>Priority: Major
> Attachments: HIVE-21339.1.patch, HIVE-21339.2.patch,
> HIVE-21339.3.patch, HIVE-21339.4.patch, llap-cache-fs-get.png,
> llap-query7-cached.svg
>
>
> https://github.com/apache/hive/blob/master/llap-server/src/java/org/apache/hadoop/hive/llap/io/encoded/OrcEncodedDataReader.java#L214
> {code}
> // 1. Get file metadata from cache, or create the reader and read it.
> // Don't cache the filesystem object for now; Tez closes it and FS cache
> will fix all that
> fs = split.getPath().getFileSystem(jobConf);
> fileKey = determineFileId(fs, split,
> HiveConf.getBoolVar(daemonConf,
> ConfVars.LLAP_CACHE_ALLOW_SYNTHETIC_FILEID),
> HiveConf.getBoolVar(daemonConf,
> ConfVars.LLAP_CACHE_DEFAULT_FS_FILE_ID),
> !HiveConf.getBoolVar(daemonConf, ConfVars.LLAP_IO_USE_FILEID_PATH)
> );
> {code}
> !llap-cache-fs-get.png!
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-21339) LLAP: Cache hit also initializes an FS object
[
https://issues.apache.org/jira/browse/HIVE-21339?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Prasanth Jayachandran updated HIVE-21339:
-
Attachment: HIVE-21339.4.patch
> LLAP: Cache hit also initializes an FS object
> --
>
> Key: HIVE-21339
> URL: https://issues.apache.org/jira/browse/HIVE-21339
> Project: Hive
> Issue Type: Bug
> Components: llap
>Affects Versions: 4.0.0
>Reporter: Gopal V
>Assignee: Prasanth Jayachandran
>Priority: Major
> Attachments: HIVE-21339.1.patch, HIVE-21339.2.patch,
> HIVE-21339.3.patch, HIVE-21339.4.patch, llap-cache-fs-get.png,
> llap-query7-cached.svg
>
>
> https://github.com/apache/hive/blob/master/llap-server/src/java/org/apache/hadoop/hive/llap/io/encoded/OrcEncodedDataReader.java#L214
> {code}
> // 1. Get file metadata from cache, or create the reader and read it.
> // Don't cache the filesystem object for now; Tez closes it and FS cache
> will fix all that
> fs = split.getPath().getFileSystem(jobConf);
> fileKey = determineFileId(fs, split,
> HiveConf.getBoolVar(daemonConf,
> ConfVars.LLAP_CACHE_ALLOW_SYNTHETIC_FILEID),
> HiveConf.getBoolVar(daemonConf,
> ConfVars.LLAP_CACHE_DEFAULT_FS_FILE_ID),
> !HiveConf.getBoolVar(daemonConf, ConfVars.LLAP_IO_USE_FILEID_PATH)
> );
> {code}
> !llap-cache-fs-get.png!
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-21390) BI split strategy does not work for blob stores
[ https://issues.apache.org/jira/browse/HIVE-21390?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16788322#comment-16788322 ] Prasanth Jayachandran commented on HIVE-21390: -- Fixes test failure > BI split strategy does not work for blob stores > --- > > Key: HIVE-21390 > URL: https://issues.apache.org/jira/browse/HIVE-21390 > Project: Hive > Issue Type: Bug >Affects Versions: 4.0.0, 3.2.0 >Reporter: Prasanth Jayachandran >Assignee: Prasanth Jayachandran >Priority: Major > Attachments: HIVE-21390.1.patch, HIVE-21390.2.patch, > HIVE-21390.3.patch > > > BI split strategy cuts the split at block boundaries however there are no > block boundaries in blob storage so we end up with 1 split for BI split > strategy. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-21390) BI split strategy does not work for blob stores
[ https://issues.apache.org/jira/browse/HIVE-21390?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Prasanth Jayachandran updated HIVE-21390: - Attachment: HIVE-21390.3.patch > BI split strategy does not work for blob stores > --- > > Key: HIVE-21390 > URL: https://issues.apache.org/jira/browse/HIVE-21390 > Project: Hive > Issue Type: Bug >Affects Versions: 4.0.0, 3.2.0 >Reporter: Prasanth Jayachandran >Assignee: Prasanth Jayachandran >Priority: Major > Attachments: HIVE-21390.1.patch, HIVE-21390.2.patch, > HIVE-21390.3.patch > > > BI split strategy cuts the split at block boundaries however there are no > block boundaries in blob storage so we end up with 1 split for BI split > strategy. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-21390) BI split strategy does not work for blob stores
[ https://issues.apache.org/jira/browse/HIVE-21390?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Prasanth Jayachandran updated HIVE-21390: - Attachment: (was: HIVE-21390.2.patch) > BI split strategy does not work for blob stores > --- > > Key: HIVE-21390 > URL: https://issues.apache.org/jira/browse/HIVE-21390 > Project: Hive > Issue Type: Bug >Affects Versions: 4.0.0, 3.2.0 >Reporter: Prasanth Jayachandran >Assignee: Prasanth Jayachandran >Priority: Major > Attachments: HIVE-21390.1.patch, HIVE-21390.2.patch, > HIVE-21390.3.patch > > > BI split strategy cuts the split at block boundaries however there are no > block boundaries in blob storage so we end up with 1 split for BI split > strategy. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-21390) BI split strategy does not work for blob stores
[ https://issues.apache.org/jira/browse/HIVE-21390?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Prasanth Jayachandran updated HIVE-21390: - Attachment: HIVE-21390.2.patch > BI split strategy does not work for blob stores > --- > > Key: HIVE-21390 > URL: https://issues.apache.org/jira/browse/HIVE-21390 > Project: Hive > Issue Type: Bug >Affects Versions: 4.0.0, 3.2.0 >Reporter: Prasanth Jayachandran >Assignee: Prasanth Jayachandran >Priority: Major > Attachments: HIVE-21390.1.patch, HIVE-21390.2.patch, > HIVE-21390.3.patch > > > BI split strategy cuts the split at block boundaries however there are no > block boundaries in blob storage so we end up with 1 split for BI split > strategy. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-21400) Vectorization: LazyBinarySerializeWrite allocates Field() within the loop
[
https://issues.apache.org/jira/browse/HIVE-21400?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16787261#comment-16787261
]
Prasanth Jayachandran commented on HIVE-21400:
--
lgtm, +1
> Vectorization: LazyBinarySerializeWrite allocates Field() within the loop
> -
>
> Key: HIVE-21400
> URL: https://issues.apache.org/jira/browse/HIVE-21400
> Project: Hive
> Issue Type: Bug
> Components: Vectorization
>Affects Versions: 4.0.0
>Reporter: Gopal V
>Assignee: Gopal V
>Priority: Major
> Attachments: HIVE-21400.1.patch
>
>
> GC thrash from an unexpected source in ReduceSinkOperator.
> {code}
> org.apache.hadoop.hive.serde2.lazybinary.fast.LazyBinarySerializeWrite.resetWithoutOutput(LazyBinarySerializeWrite.java:136)
> at
> org.apache.hadoop.hive.serde2.lazybinary.fast.LazyBinarySerializeWrite.reset(LazyBinarySerializeWrite.java:132)
> at
> org.apache.hadoop.hive.ql.exec.vector.reducesink.VectorReduceSinkUniformHashOperator.process(VectorReduceSinkUniformHashOperator.java:180)
> {code}
> GC space is getting thrashed by the
> {code}
> root = new Field(STRUCT);
> {code}
> for every row.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Comment Edited] (HIVE-21380) No documented command to list queries in a Hive Workload resource pool
[ https://issues.apache.org/jira/browse/HIVE-21380?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16786144#comment-16786144 ] Prasanth Jayachandran edited comment on HIVE-21380 at 3/6/19 9:25 PM: -- There is a mx bean that is exposed via metrics endpoint that prints the sessions running in the pool. It should be easy to enhance that to add session.getQueryId() here [https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/exec/tez/WorkloadManager.java#L753] was (Author: prasanth_j): The is mx bean that is exposed via metrics endpoint that prints the sessions running in the pool. It should be easy to enhance that to add session.getQueryId() here [https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/exec/tez/WorkloadManager.java#L753] > No documented command to list queries in a Hive Workload resource pool > -- > > Key: HIVE-21380 > URL: https://issues.apache.org/jira/browse/HIVE-21380 > Project: Hive > Issue Type: Bug > Components: Hive >Affects Versions: 3.1.0 > Environment: Kerberos, Ranger >Reporter: Michael DeGuzis >Priority: Major > > We have searched all over > https://docs.hortonworks.com/HDPDocuments/HDP3/HDP-3.1.0/hive-workload/content/hive_workload_management_entity_data_in_sys.html > and even the Apache hive code base. How do you monitor what queries are > running in what resource pool??? -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-21380) No documented command to list queries in a Hive Workload resource pool
[ https://issues.apache.org/jira/browse/HIVE-21380?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16786144#comment-16786144 ] Prasanth Jayachandran commented on HIVE-21380: -- The is mx bean that is exposed via metrics endpoint that prints the sessions running in the pool. It should be easy to enhance that to add session.getQueryId() here [https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/exec/tez/WorkloadManager.java#L753] > No documented command to list queries in a Hive Workload resource pool > -- > > Key: HIVE-21380 > URL: https://issues.apache.org/jira/browse/HIVE-21380 > Project: Hive > Issue Type: Bug > Components: Hive >Affects Versions: 3.1.0 > Environment: Kerberos, Ranger >Reporter: Michael DeGuzis >Priority: Major > > We have searched all over > https://docs.hortonworks.com/HDPDocuments/HDP3/HDP-3.1.0/hive-workload/content/hive_workload_management_entity_data_in_sys.html > and even the Apache hive code base. How do you monitor what queries are > running in what resource pool??? -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-21391) LLAP: Pool of column vector buffers can cause memory pressure
[ https://issues.apache.org/jira/browse/HIVE-21391?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16786124#comment-16786124 ] Prasanth Jayachandran commented on HIVE-21391: -- [~gopalv] can you please review this patch? The strong ref fixed object pool of CVBs is made to weak ref. > LLAP: Pool of column vector buffers can cause memory pressure > - > > Key: HIVE-21391 > URL: https://issues.apache.org/jira/browse/HIVE-21391 > Project: Hive > Issue Type: Bug > Components: llap >Affects Versions: 4.0.0, 3.2.0 >Reporter: Prasanth Jayachandran >Assignee: Prasanth Jayachandran >Priority: Major > Attachments: HIVE-21391.1.patch > > > Where there are too many columns (in the order of 100s), with decimal, string > types the column vector pool of buffers created here > [https://github.com/apache/hive/blob/master/llap-server/src/java/org/apache/hadoop/hive/llap/io/decode/EncodedDataConsumer.java#L59] > can cause memory pressure. > Example: > 128 (poolSize) * 300 (numCols) * 1024 (batchSize) * 80 (decimalSize) ~= 3GB > The pool size keeps increasing when there is slow consumer but fast llap io > (SSDs) leading to GC pressure when all LLAP io threads read splits from same > table. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-21391) LLAP: Pool of column vector buffers can cause memory pressure
[ https://issues.apache.org/jira/browse/HIVE-21391?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Prasanth Jayachandran updated HIVE-21391: - Attachment: HIVE-21391.1.patch > LLAP: Pool of column vector buffers can cause memory pressure > - > > Key: HIVE-21391 > URL: https://issues.apache.org/jira/browse/HIVE-21391 > Project: Hive > Issue Type: Bug > Components: llap >Affects Versions: 4.0.0, 3.2.0 >Reporter: Prasanth Jayachandran >Assignee: Prasanth Jayachandran >Priority: Major > Attachments: HIVE-21391.1.patch > > > Where there are too many columns (in the order of 100s), with decimal, string > types the column vector pool of buffers created here > [https://github.com/apache/hive/blob/master/llap-server/src/java/org/apache/hadoop/hive/llap/io/decode/EncodedDataConsumer.java#L59] > can cause memory pressure. > Example: > 128 (poolSize) * 300 (numCols) * 1024 (batchSize) * 80 (decimalSize) ~= 3GB > The pool size keeps increasing when there is slow consumer but fast llap io > (SSDs) leading to GC pressure when all LLAP io threads read splits from same > table. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-21391) LLAP: Pool of column vector buffers can cause memory pressure
[ https://issues.apache.org/jira/browse/HIVE-21391?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Prasanth Jayachandran updated HIVE-21391: - Status: Patch Available (was: Open) > LLAP: Pool of column vector buffers can cause memory pressure > - > > Key: HIVE-21391 > URL: https://issues.apache.org/jira/browse/HIVE-21391 > Project: Hive > Issue Type: Bug > Components: llap >Affects Versions: 4.0.0, 3.2.0 >Reporter: Prasanth Jayachandran >Assignee: Prasanth Jayachandran >Priority: Major > Attachments: HIVE-21391.1.patch > > > Where there are too many columns (in the order of 100s), with decimal, string > types the column vector pool of buffers created here > [https://github.com/apache/hive/blob/master/llap-server/src/java/org/apache/hadoop/hive/llap/io/decode/EncodedDataConsumer.java#L59] > can cause memory pressure. > Example: > 128 (poolSize) * 300 (numCols) * 1024 (batchSize) * 80 (decimalSize) ~= 3GB > The pool size keeps increasing when there is slow consumer but fast llap io > (SSDs) leading to GC pressure when all LLAP io threads read splits from same > table. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-20656) Sensible defaults: Map aggregation memory configs are too aggressive
[
https://issues.apache.org/jira/browse/HIVE-20656?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Prasanth Jayachandran updated HIVE-20656:
-
Attachment: HIVE-20656.2.patch
> Sensible defaults: Map aggregation memory configs are too aggressive
>
>
> Key: HIVE-20656
> URL: https://issues.apache.org/jira/browse/HIVE-20656
> Project: Hive
> Issue Type: Bug
>Affects Versions: 4.0.0, 3.2.0
>Reporter: Prasanth Jayachandran
>Assignee: Prasanth Jayachandran
>Priority: Major
> Attachments: HIVE-20656.1.patch, HIVE-20656.2.patch
>
>
> The defaults for the following configs seems to be too aggressive. In java
> this can easily lead to several full GC pauses whose memory cannot be
> reclaimed.
> {code:java}
> HIVEMAPAGGRHASHMEMORY("hive.map.aggr.hash.percentmemory", (float) 0.99,
> "Portion of total memory to be used by map-side group aggregation hash
> table"),
> HIVEMAPAGGRMEMORYTHRESHOLD("hive.map.aggr.hash.force.flush.memory.threshold",
> (float) 0.9,
> "The max memory to be used by map-side group aggregation hash table.\n" +
> "If the memory usage is higher than this number, force to flush
> data"),{code}
>
> We can be little bit conservative for these configs to avoid getting into GC
> pause.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-20656) Sensible defaults: Map aggregation memory configs are too aggressive
[
https://issues.apache.org/jira/browse/HIVE-20656?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16785157#comment-16785157
]
Prasanth Jayachandran commented on HIVE-20656:
--
Golden files updated.
> Sensible defaults: Map aggregation memory configs are too aggressive
>
>
> Key: HIVE-20656
> URL: https://issues.apache.org/jira/browse/HIVE-20656
> Project: Hive
> Issue Type: Bug
>Affects Versions: 4.0.0, 3.2.0
>Reporter: Prasanth Jayachandran
>Assignee: Prasanth Jayachandran
>Priority: Major
> Attachments: HIVE-20656.1.patch, HIVE-20656.2.patch
>
>
> The defaults for the following configs seems to be too aggressive. In java
> this can easily lead to several full GC pauses whose memory cannot be
> reclaimed.
> {code:java}
> HIVEMAPAGGRHASHMEMORY("hive.map.aggr.hash.percentmemory", (float) 0.99,
> "Portion of total memory to be used by map-side group aggregation hash
> table"),
> HIVEMAPAGGRMEMORYTHRESHOLD("hive.map.aggr.hash.force.flush.memory.threshold",
> (float) 0.9,
> "The max memory to be used by map-side group aggregation hash table.\n" +
> "If the memory usage is higher than this number, force to flush
> data"),{code}
>
> We can be little bit conservative for these configs to avoid getting into GC
> pause.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-21339) LLAP: Cache hit also initializes an FS object
[
https://issues.apache.org/jira/browse/HIVE-21339?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Prasanth Jayachandran updated HIVE-21339:
-
Attachment: HIVE-21339.3.patch
> LLAP: Cache hit also initializes an FS object
> --
>
> Key: HIVE-21339
> URL: https://issues.apache.org/jira/browse/HIVE-21339
> Project: Hive
> Issue Type: Bug
> Components: llap
>Affects Versions: 4.0.0
>Reporter: Gopal V
>Assignee: Prasanth Jayachandran
>Priority: Major
> Attachments: HIVE-21339.1.patch, HIVE-21339.2.patch,
> HIVE-21339.3.patch, llap-cache-fs-get.png, llap-query7-cached.svg
>
>
> https://github.com/apache/hive/blob/master/llap-server/src/java/org/apache/hadoop/hive/llap/io/encoded/OrcEncodedDataReader.java#L214
> {code}
> // 1. Get file metadata from cache, or create the reader and read it.
> // Don't cache the filesystem object for now; Tez closes it and FS cache
> will fix all that
> fs = split.getPath().getFileSystem(jobConf);
> fileKey = determineFileId(fs, split,
> HiveConf.getBoolVar(daemonConf,
> ConfVars.LLAP_CACHE_ALLOW_SYNTHETIC_FILEID),
> HiveConf.getBoolVar(daemonConf,
> ConfVars.LLAP_CACHE_DEFAULT_FS_FILE_ID),
> !HiveConf.getBoolVar(daemonConf, ConfVars.LLAP_IO_USE_FILEID_PATH)
> );
> {code}
> !llap-cache-fs-get.png!
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-21339) LLAP: Cache hit also initializes an FS object
[
https://issues.apache.org/jira/browse/HIVE-21339?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16785147#comment-16785147
]
Prasanth Jayachandran commented on HIVE-21339:
--
Looks like precommit tried to apply the svg file. Reuploading the same patch.
> LLAP: Cache hit also initializes an FS object
> --
>
> Key: HIVE-21339
> URL: https://issues.apache.org/jira/browse/HIVE-21339
> Project: Hive
> Issue Type: Bug
> Components: llap
>Affects Versions: 4.0.0
>Reporter: Gopal V
>Assignee: Prasanth Jayachandran
>Priority: Major
> Attachments: HIVE-21339.1.patch, HIVE-21339.2.patch,
> HIVE-21339.3.patch, llap-cache-fs-get.png, llap-query7-cached.svg
>
>
> https://github.com/apache/hive/blob/master/llap-server/src/java/org/apache/hadoop/hive/llap/io/encoded/OrcEncodedDataReader.java#L214
> {code}
> // 1. Get file metadata from cache, or create the reader and read it.
> // Don't cache the filesystem object for now; Tez closes it and FS cache
> will fix all that
> fs = split.getPath().getFileSystem(jobConf);
> fileKey = determineFileId(fs, split,
> HiveConf.getBoolVar(daemonConf,
> ConfVars.LLAP_CACHE_ALLOW_SYNTHETIC_FILEID),
> HiveConf.getBoolVar(daemonConf,
> ConfVars.LLAP_CACHE_DEFAULT_FS_FILE_ID),
> !HiveConf.getBoolVar(daemonConf, ConfVars.LLAP_IO_USE_FILEID_PATH)
> );
> {code}
> !llap-cache-fs-get.png!
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-21390) BI split strategy does not work for blob stores
[ https://issues.apache.org/jira/browse/HIVE-21390?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Prasanth Jayachandran updated HIVE-21390: - Attachment: HIVE-21390.2.patch > BI split strategy does not work for blob stores > --- > > Key: HIVE-21390 > URL: https://issues.apache.org/jira/browse/HIVE-21390 > Project: Hive > Issue Type: Bug >Affects Versions: 4.0.0, 3.2.0 >Reporter: Prasanth Jayachandran >Assignee: Prasanth Jayachandran >Priority: Major > Attachments: HIVE-21390.1.patch, HIVE-21390.2.patch > > > BI split strategy cuts the split at block boundaries however there are no > block boundaries in blob storage so we end up with 1 split for BI split > strategy. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-21390) BI split strategy does not work for blob stores
[ https://issues.apache.org/jira/browse/HIVE-21390?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16784929#comment-16784929 ] Prasanth Jayachandran commented on HIVE-21390: -- [~gopalv] can you please take a look? > BI split strategy does not work for blob stores > --- > > Key: HIVE-21390 > URL: https://issues.apache.org/jira/browse/HIVE-21390 > Project: Hive > Issue Type: Bug >Affects Versions: 4.0.0, 3.2.0 >Reporter: Prasanth Jayachandran >Assignee: Prasanth Jayachandran >Priority: Major > Attachments: HIVE-21390.1.patch > > > BI split strategy cuts the split at block boundaries however there are no > block boundaries in blob storage so we end up with 1 split for BI split > strategy. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-21390) BI split strategy does not work for blob stores
[ https://issues.apache.org/jira/browse/HIVE-21390?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Prasanth Jayachandran updated HIVE-21390: - Status: Patch Available (was: Open) > BI split strategy does not work for blob stores > --- > > Key: HIVE-21390 > URL: https://issues.apache.org/jira/browse/HIVE-21390 > Project: Hive > Issue Type: Bug >Affects Versions: 4.0.0, 3.2.0 >Reporter: Prasanth Jayachandran >Assignee: Prasanth Jayachandran >Priority: Major > Attachments: HIVE-21390.1.patch > > > BI split strategy cuts the split at block boundaries however there are no > block boundaries in blob storage so we end up with 1 split for BI split > strategy. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-21390) BI split strategy does not work for blob stores
[ https://issues.apache.org/jira/browse/HIVE-21390?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Prasanth Jayachandran updated HIVE-21390: - Attachment: HIVE-21390.1.patch > BI split strategy does not work for blob stores > --- > > Key: HIVE-21390 > URL: https://issues.apache.org/jira/browse/HIVE-21390 > Project: Hive > Issue Type: Bug >Affects Versions: 4.0.0, 3.2.0 >Reporter: Prasanth Jayachandran >Assignee: Prasanth Jayachandran >Priority: Major > Attachments: HIVE-21390.1.patch > > > BI split strategy cuts the split at block boundaries however there are no > block boundaries in blob storage so we end up with 1 split for BI split > strategy. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-21390) BI split strategy does not work for blob stores
[ https://issues.apache.org/jira/browse/HIVE-21390?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Prasanth Jayachandran updated HIVE-21390: - Summary: BI split strategy does not work for blob stores (was: BI split strategy does work for blob stores) > BI split strategy does not work for blob stores > --- > > Key: HIVE-21390 > URL: https://issues.apache.org/jira/browse/HIVE-21390 > Project: Hive > Issue Type: Bug >Affects Versions: 4.0.0, 3.2.0 >Reporter: Prasanth Jayachandran >Assignee: Prasanth Jayachandran >Priority: Major > > BI split strategy cuts the split at block boundaries however there are no > block boundaries in blob storage so we end up with 1 split for BI split > strategy. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (HIVE-21391) LLAP: Pool of column vector buffers can cause memory pressure
[ https://issues.apache.org/jira/browse/HIVE-21391?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Prasanth Jayachandran reassigned HIVE-21391: Assignee: Prasanth Jayachandran > LLAP: Pool of column vector buffers can cause memory pressure > - > > Key: HIVE-21391 > URL: https://issues.apache.org/jira/browse/HIVE-21391 > Project: Hive > Issue Type: Bug > Components: llap >Affects Versions: 4.0.0, 3.2.0 >Reporter: Prasanth Jayachandran >Assignee: Prasanth Jayachandran >Priority: Major > > Where there are too many columns (in the order of 100s), with decimal, string > types the column vector pool of buffers created here > [https://github.com/apache/hive/blob/master/llap-server/src/java/org/apache/hadoop/hive/llap/io/decode/EncodedDataConsumer.java#L59] > can cause memory pressure. > Example: > 128 (poolSize) * 300 (numCols) * 1024 (batchSize) * 80 (decimalSize) ~= 3GB > The pool size keeps increasing when there is slow consumer but fast llap io > (SSDs) leading to GC pressure when all LLAP io threads read splits from same > table. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (HIVE-21390) BI split strategy does work for blob stores
[ https://issues.apache.org/jira/browse/HIVE-21390?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Prasanth Jayachandran reassigned HIVE-21390: > BI split strategy does work for blob stores > --- > > Key: HIVE-21390 > URL: https://issues.apache.org/jira/browse/HIVE-21390 > Project: Hive > Issue Type: Bug >Affects Versions: 4.0.0, 3.2.0 >Reporter: Prasanth Jayachandran >Assignee: Prasanth Jayachandran >Priority: Major > > BI split strategy cuts the split at block boundaries however there are no > block boundaries in blob storage so we end up with 1 split for BI split > strategy. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (HIVE-21369) LLAP: Logging is expensive in encoded reader path
[
https://issues.apache.org/jira/browse/HIVE-21369?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Prasanth Jayachandran reassigned HIVE-21369:
> LLAP: Logging is expensive in encoded reader path
> -
>
> Key: HIVE-21369
> URL: https://issues.apache.org/jira/browse/HIVE-21369
> Project: Hive
> Issue Type: Bug
> Components: Logging
>Affects Versions: 4.0.0, 3.2.0
>Reporter: Prasanth Jayachandran
>Assignee: Nita Dembla
>Priority: Major
>
> There should be no INFO logging in EncodedReaderImpl. Stringifying of disk
> ranges is expensive in core read path.
> {code:java}
> 2019-03-01T17:55:56.322852142Z 2019-03-01T17:55:56,306 INFO
> [IO-Elevator-Thread-3
> (hive_20190301175546_a279f33c-4f2b-4cd5-8695-57bc8b042a61)]
> encoded.EncodedReaderImpl: Disk ranges after cache (found everything true;
> file [-3693547618692831801, 1551190876000, 1047660824], base offset
> 792920167): [{start: 887940 end: 1003508 cache buffer: 0x5165f83d(1)},
> {start: 1003508 end: 1119078 cache buffer: 0xb63cac3(1)}, {start: 1119078
> end: 1234745 cache buffer: 0x41a724fa(1)}, {start: 1234745 end: 1350261 cache
> buffer: 0x2f71bc38(1)}, {start: 1350261 end: 1465752 cache buffer:
> 0x2c38e1bb(1)}, {start: 1465752 end: 1581231 cache buffer: 0x5827982(1)},
> {start: 1581231 end: 1696885 cache buffer: 0x75a6773c(1)}, {start: 1696885
> end: 1812492 cache buffer: 0x2ed060f9(1)},{start: 1812492 end: 1928086 cache
> buffer: 0x20b2c8aa(1)}, {start: 1928086 end: 2043588 cache buffer:
> 0x6559aacb(1)}, {start: 2043588 end: 2159089 cache buffer: 0x569c85e1(1)},
> {start: 2159089 end: 2274725 cache buffer: 0x25a88dd0(1)}, {start: 2274725
> end: 2390228 cache buffer: 0x738b7e87(1)}, {start: 2390228 end: 2505715 cache
> buffer: 0x26edafa0(1)}, {start: 2505715 end: 2621322 cache buffer:
> 0x69db7752(1)}, {start: 2621322 end: 2736844 cache b{code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-21339) LLAP: Cache hit also initializes an FS object
[
https://issues.apache.org/jira/browse/HIVE-21339?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Prasanth Jayachandran updated HIVE-21339:
-
Attachment: HIVE-21339.2.patch
> LLAP: Cache hit also initializes an FS object
> --
>
> Key: HIVE-21339
> URL: https://issues.apache.org/jira/browse/HIVE-21339
> Project: Hive
> Issue Type: Bug
> Components: llap
>Affects Versions: 4.0.0
>Reporter: Gopal V
>Assignee: Prasanth Jayachandran
>Priority: Major
> Attachments: HIVE-21339.1.patch, HIVE-21339.2.patch,
> llap-cache-fs-get.png
>
>
> https://github.com/apache/hive/blob/master/llap-server/src/java/org/apache/hadoop/hive/llap/io/encoded/OrcEncodedDataReader.java#L214
> {code}
> // 1. Get file metadata from cache, or create the reader and read it.
> // Don't cache the filesystem object for now; Tez closes it and FS cache
> will fix all that
> fs = split.getPath().getFileSystem(jobConf);
> fileKey = determineFileId(fs, split,
> HiveConf.getBoolVar(daemonConf,
> ConfVars.LLAP_CACHE_ALLOW_SYNTHETIC_FILEID),
> HiveConf.getBoolVar(daemonConf,
> ConfVars.LLAP_CACHE_DEFAULT_FS_FILE_ID),
> !HiveConf.getBoolVar(daemonConf, ConfVars.LLAP_IO_USE_FILEID_PATH)
> );
> {code}
> !llap-cache-fs-get.png!
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-21339) LLAP: Cache hit also initializes an FS object
[
https://issues.apache.org/jira/browse/HIVE-21339?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Prasanth Jayachandran updated HIVE-21339:
-
Attachment: HIVE-21339.1.patch
> LLAP: Cache hit also initializes an FS object
> --
>
> Key: HIVE-21339
> URL: https://issues.apache.org/jira/browse/HIVE-21339
> Project: Hive
> Issue Type: Bug
> Components: llap
>Affects Versions: 4.0.0
>Reporter: Gopal V
>Assignee: Prasanth Jayachandran
>Priority: Major
> Attachments: HIVE-21339.1.patch, llap-cache-fs-get.png
>
>
> https://github.com/apache/hive/blob/master/llap-server/src/java/org/apache/hadoop/hive/llap/io/encoded/OrcEncodedDataReader.java#L214
> {code}
> // 1. Get file metadata from cache, or create the reader and read it.
> // Don't cache the filesystem object for now; Tez closes it and FS cache
> will fix all that
> fs = split.getPath().getFileSystem(jobConf);
> fileKey = determineFileId(fs, split,
> HiveConf.getBoolVar(daemonConf,
> ConfVars.LLAP_CACHE_ALLOW_SYNTHETIC_FILEID),
> HiveConf.getBoolVar(daemonConf,
> ConfVars.LLAP_CACHE_DEFAULT_FS_FILE_ID),
> !HiveConf.getBoolVar(daemonConf, ConfVars.LLAP_IO_USE_FILEID_PATH)
> );
> {code}
> !llap-cache-fs-get.png!
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-21339) LLAP: Cache hit also initializes an FS object
[
https://issues.apache.org/jira/browse/HIVE-21339?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Prasanth Jayachandran updated HIVE-21339:
-
Status: Patch Available (was: Open)
> LLAP: Cache hit also initializes an FS object
> --
>
> Key: HIVE-21339
> URL: https://issues.apache.org/jira/browse/HIVE-21339
> Project: Hive
> Issue Type: Bug
> Components: llap
>Affects Versions: 4.0.0
>Reporter: Gopal V
>Assignee: Prasanth Jayachandran
>Priority: Major
> Attachments: HIVE-21339.1.patch, llap-cache-fs-get.png
>
>
> https://github.com/apache/hive/blob/master/llap-server/src/java/org/apache/hadoop/hive/llap/io/encoded/OrcEncodedDataReader.java#L214
> {code}
> // 1. Get file metadata from cache, or create the reader and read it.
> // Don't cache the filesystem object for now; Tez closes it and FS cache
> will fix all that
> fs = split.getPath().getFileSystem(jobConf);
> fileKey = determineFileId(fs, split,
> HiveConf.getBoolVar(daemonConf,
> ConfVars.LLAP_CACHE_ALLOW_SYNTHETIC_FILEID),
> HiveConf.getBoolVar(daemonConf,
> ConfVars.LLAP_CACHE_DEFAULT_FS_FILE_ID),
> !HiveConf.getBoolVar(daemonConf, ConfVars.LLAP_IO_USE_FILEID_PATH)
> );
> {code}
> !llap-cache-fs-get.png!
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-21339) LLAP: Cache hit also initializes an FS object
[
https://issues.apache.org/jira/browse/HIVE-21339?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Prasanth Jayachandran updated HIVE-21339:
-
Attachment: (was: HIVE-21339.1.patch)
> LLAP: Cache hit also initializes an FS object
> --
>
> Key: HIVE-21339
> URL: https://issues.apache.org/jira/browse/HIVE-21339
> Project: Hive
> Issue Type: Bug
> Components: llap
>Affects Versions: 4.0.0
>Reporter: Gopal V
>Assignee: Prasanth Jayachandran
>Priority: Major
> Attachments: HIVE-21339.1.patch, llap-cache-fs-get.png
>
>
> https://github.com/apache/hive/blob/master/llap-server/src/java/org/apache/hadoop/hive/llap/io/encoded/OrcEncodedDataReader.java#L214
> {code}
> // 1. Get file metadata from cache, or create the reader and read it.
> // Don't cache the filesystem object for now; Tez closes it and FS cache
> will fix all that
> fs = split.getPath().getFileSystem(jobConf);
> fileKey = determineFileId(fs, split,
> HiveConf.getBoolVar(daemonConf,
> ConfVars.LLAP_CACHE_ALLOW_SYNTHETIC_FILEID),
> HiveConf.getBoolVar(daemonConf,
> ConfVars.LLAP_CACHE_DEFAULT_FS_FILE_ID),
> !HiveConf.getBoolVar(daemonConf, ConfVars.LLAP_IO_USE_FILEID_PATH)
> );
> {code}
> !llap-cache-fs-get.png!
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-21339) LLAP: Cache hit also initializes an FS object
[
https://issues.apache.org/jira/browse/HIVE-21339?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Prasanth Jayachandran updated HIVE-21339:
-
Attachment: HIVE-21339.1.patch
> LLAP: Cache hit also initializes an FS object
> --
>
> Key: HIVE-21339
> URL: https://issues.apache.org/jira/browse/HIVE-21339
> Project: Hive
> Issue Type: Bug
> Components: llap
>Affects Versions: 4.0.0
>Reporter: Gopal V
>Assignee: Prasanth Jayachandran
>Priority: Major
> Attachments: HIVE-21339.1.patch, llap-cache-fs-get.png
>
>
> https://github.com/apache/hive/blob/master/llap-server/src/java/org/apache/hadoop/hive/llap/io/encoded/OrcEncodedDataReader.java#L214
> {code}
> // 1. Get file metadata from cache, or create the reader and read it.
> // Don't cache the filesystem object for now; Tez closes it and FS cache
> will fix all that
> fs = split.getPath().getFileSystem(jobConf);
> fileKey = determineFileId(fs, split,
> HiveConf.getBoolVar(daemonConf,
> ConfVars.LLAP_CACHE_ALLOW_SYNTHETIC_FILEID),
> HiveConf.getBoolVar(daemonConf,
> ConfVars.LLAP_CACHE_DEFAULT_FS_FILE_ID),
> !HiveConf.getBoolVar(daemonConf, ConfVars.LLAP_IO_USE_FILEID_PATH)
> );
> {code}
> !llap-cache-fs-get.png!
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-21339) LLAP: Cache hit also initializes an FS object
[
https://issues.apache.org/jira/browse/HIVE-21339?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16779932#comment-16779932
]
Prasanth Jayachandran commented on HIVE-21339:
--
[~gopalv] can you please review?
> LLAP: Cache hit also initializes an FS object
> --
>
> Key: HIVE-21339
> URL: https://issues.apache.org/jira/browse/HIVE-21339
> Project: Hive
> Issue Type: Bug
> Components: llap
>Affects Versions: 4.0.0
>Reporter: Gopal V
>Assignee: Prasanth Jayachandran
>Priority: Major
> Attachments: HIVE-21339.1.patch, llap-cache-fs-get.png
>
>
> https://github.com/apache/hive/blob/master/llap-server/src/java/org/apache/hadoop/hive/llap/io/encoded/OrcEncodedDataReader.java#L214
> {code}
> // 1. Get file metadata from cache, or create the reader and read it.
> // Don't cache the filesystem object for now; Tez closes it and FS cache
> will fix all that
> fs = split.getPath().getFileSystem(jobConf);
> fileKey = determineFileId(fs, split,
> HiveConf.getBoolVar(daemonConf,
> ConfVars.LLAP_CACHE_ALLOW_SYNTHETIC_FILEID),
> HiveConf.getBoolVar(daemonConf,
> ConfVars.LLAP_CACHE_DEFAULT_FS_FILE_ID),
> !HiveConf.getBoolVar(daemonConf, ConfVars.LLAP_IO_USE_FILEID_PATH)
> );
> {code}
> !llap-cache-fs-get.png!
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Assigned] (HIVE-21339) LLAP: Cache hit also initializes an FS object
[
https://issues.apache.org/jira/browse/HIVE-21339?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Prasanth Jayachandran reassigned HIVE-21339:
Assignee: Prasanth Jayachandran
> LLAP: Cache hit also initializes an FS object
> --
>
> Key: HIVE-21339
> URL: https://issues.apache.org/jira/browse/HIVE-21339
> Project: Hive
> Issue Type: Bug
> Components: llap
>Affects Versions: 4.0.0
>Reporter: Gopal V
>Assignee: Prasanth Jayachandran
>Priority: Major
> Attachments: llap-cache-fs-get.png
>
>
> https://github.com/apache/hive/blob/master/llap-server/src/java/org/apache/hadoop/hive/llap/io/encoded/OrcEncodedDataReader.java#L214
> {code}
> // 1. Get file metadata from cache, or create the reader and read it.
> // Don't cache the filesystem object for now; Tez closes it and FS cache
> will fix all that
> fs = split.getPath().getFileSystem(jobConf);
> fileKey = determineFileId(fs, split,
> HiveConf.getBoolVar(daemonConf,
> ConfVars.LLAP_CACHE_ALLOW_SYNTHETIC_FILEID),
> HiveConf.getBoolVar(daemonConf,
> ConfVars.LLAP_CACHE_DEFAULT_FS_FILE_ID),
> !HiveConf.getBoolVar(daemonConf, ConfVars.LLAP_IO_USE_FILEID_PATH)
> );
> {code}
> !llap-cache-fs-get.png!
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Assigned] (HIVE-20656) Map aggregation memory configs are too aggressive
[
https://issues.apache.org/jira/browse/HIVE-20656?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Prasanth Jayachandran reassigned HIVE-20656:
Assignee: Prasanth Jayachandran
> Map aggregation memory configs are too aggressive
> -
>
> Key: HIVE-20656
> URL: https://issues.apache.org/jira/browse/HIVE-20656
> Project: Hive
> Issue Type: Bug
>Affects Versions: 4.0.0, 3.2.0
>Reporter: Prasanth Jayachandran
>Assignee: Prasanth Jayachandran
>Priority: Major
> Attachments: HIVE-20656.1.patch
>
>
> The defaults for the following configs seems to be too aggressive. In java
> this can easily lead to several full GC pauses whose memory cannot be
> reclaimed.
> {code:java}
> HIVEMAPAGGRHASHMEMORY("hive.map.aggr.hash.percentmemory", (float) 0.99,
> "Portion of total memory to be used by map-side group aggregation hash
> table"),
> HIVEMAPAGGRMEMORYTHRESHOLD("hive.map.aggr.hash.force.flush.memory.threshold",
> (float) 0.9,
> "The max memory to be used by map-side group aggregation hash table.\n" +
> "If the memory usage is higher than this number, force to flush
> data"),{code}
>
> We can be little bit conservative for these configs to avoid getting into GC
> pause.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-20656) Map aggregation memory configs are too aggressive
[
https://issues.apache.org/jira/browse/HIVE-20656?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Prasanth Jayachandran updated HIVE-20656:
-
Attachment: HIVE-20656.1.patch
> Map aggregation memory configs are too aggressive
> -
>
> Key: HIVE-20656
> URL: https://issues.apache.org/jira/browse/HIVE-20656
> Project: Hive
> Issue Type: Bug
>Affects Versions: 4.0.0, 3.2.0
>Reporter: Prasanth Jayachandran
>Priority: Major
> Attachments: HIVE-20656.1.patch
>
>
> The defaults for the following configs seems to be too aggressive. In java
> this can easily lead to several full GC pauses whose memory cannot be
> reclaimed.
> {code:java}
> HIVEMAPAGGRHASHMEMORY("hive.map.aggr.hash.percentmemory", (float) 0.99,
> "Portion of total memory to be used by map-side group aggregation hash
> table"),
> HIVEMAPAGGRMEMORYTHRESHOLD("hive.map.aggr.hash.force.flush.memory.threshold",
> (float) 0.9,
> "The max memory to be used by map-side group aggregation hash table.\n" +
> "If the memory usage is higher than this number, force to flush
> data"),{code}
>
> We can be little bit conservative for these configs to avoid getting into GC
> pause.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-20656) Map aggregation memory configs are too aggressive
[
https://issues.apache.org/jira/browse/HIVE-20656?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Prasanth Jayachandran updated HIVE-20656:
-
Status: Patch Available (was: Open)
> Map aggregation memory configs are too aggressive
> -
>
> Key: HIVE-20656
> URL: https://issues.apache.org/jira/browse/HIVE-20656
> Project: Hive
> Issue Type: Bug
>Affects Versions: 4.0.0, 3.2.0
>Reporter: Prasanth Jayachandran
>Assignee: Prasanth Jayachandran
>Priority: Major
> Attachments: HIVE-20656.1.patch
>
>
> The defaults for the following configs seems to be too aggressive. In java
> this can easily lead to several full GC pauses whose memory cannot be
> reclaimed.
> {code:java}
> HIVEMAPAGGRHASHMEMORY("hive.map.aggr.hash.percentmemory", (float) 0.99,
> "Portion of total memory to be used by map-side group aggregation hash
> table"),
> HIVEMAPAGGRMEMORYTHRESHOLD("hive.map.aggr.hash.force.flush.memory.threshold",
> (float) 0.9,
> "The max memory to be used by map-side group aggregation hash table.\n" +
> "If the memory usage is higher than this number, force to flush
> data"),{code}
>
> We can be little bit conservative for these configs to avoid getting into GC
> pause.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-20854) Sensible Defaults: Hive's Zookeeper heartbeat interval is 20 minutes, change to 2
[
https://issues.apache.org/jira/browse/HIVE-20854?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16778946#comment-16778946
]
Prasanth Jayachandran commented on HIVE-20854:
--
+1
there is another prod outage today related to stale entries in zk for LLAP.
Longer term we could have separate configs for zk lock manager and LLAP.
> Sensible Defaults: Hive's Zookeeper heartbeat interval is 20 minutes, change
> to 2
> -
>
> Key: HIVE-20854
> URL: https://issues.apache.org/jira/browse/HIVE-20854
> Project: Hive
> Issue Type: Bug
>Reporter: Gopal V
>Assignee: Gopal V
>Priority: Major
> Attachments: HIVE-20854.1.patch
>
>
> {code}
> HIVE_ZOOKEEPER_SESSION_TIMEOUT("hive.zookeeper.session.timeout",
> "120ms",
> new TimeValidator(TimeUnit.MILLISECONDS),
> "ZooKeeper client's session timeout (in milliseconds). The client is
> disconnected, and as a result, all locks released, \n" +
> "if a heartbeat is not sent in the timeout."),
> {code}
> That's 1,200,000ms which is too long for all practical purposes - a 20 minute
> outage in case a node has a failure is too long.
> That is too long for the JDBC load-balancing, LLAP failure tolerance and the
> lock manager expiry.
> Change to 2 minutes, as a sensible default
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-21254) Pre-upgrade tool should handle exceptions and skip db/tables
[ https://issues.apache.org/jira/browse/HIVE-21254?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Prasanth Jayachandran updated HIVE-21254: - Resolution: Fixed Fix Version/s: 4.0.0 Status: Resolved (was: Patch Available) Committed to master. Thanks all for the reviews! > Pre-upgrade tool should handle exceptions and skip db/tables > > > Key: HIVE-21254 > URL: https://issues.apache.org/jira/browse/HIVE-21254 > Project: Hive > Issue Type: Bug > Components: Transactions >Affects Versions: 4.0.0 >Reporter: Prasanth Jayachandran >Assignee: Prasanth Jayachandran >Priority: Major > Fix For: 4.0.0 > > Attachments: HIVE-21254.1.patch, HIVE-21254.2.patch, > HIVE-21254.3.patch, HIVE-21254.4.patch, HIVE-21254.5.patch, > HIVE-21254.6.patch, HIVE-21254.7.patch, HIVE-21254.8.patch, HIVE-21254.9.patch > > > When exceptions like AccessControlException is thrown, pre-upgrade tool > fails. If hive user does not have read access to database or tables (some > external tables denies read access to hive), pre-upgrade tool should just > assume they are external tables and move on without failing pre-upgrade > process. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-21254) Pre-upgrade tool should handle exceptions and skip db/tables
[ https://issues.apache.org/jira/browse/HIVE-21254?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16772268#comment-16772268 ] Prasanth Jayachandran commented on HIVE-21254: -- Another try for clean run > Pre-upgrade tool should handle exceptions and skip db/tables > > > Key: HIVE-21254 > URL: https://issues.apache.org/jira/browse/HIVE-21254 > Project: Hive > Issue Type: Bug > Components: Transactions >Affects Versions: 4.0.0 >Reporter: Prasanth Jayachandran >Assignee: Prasanth Jayachandran >Priority: Major > Attachments: HIVE-21254.1.patch, HIVE-21254.2.patch, > HIVE-21254.3.patch, HIVE-21254.4.patch, HIVE-21254.5.patch, > HIVE-21254.6.patch, HIVE-21254.7.patch, HIVE-21254.8.patch, HIVE-21254.9.patch > > > When exceptions like AccessControlException is thrown, pre-upgrade tool > fails. If hive user does not have read access to database or tables (some > external tables denies read access to hive), pre-upgrade tool should just > assume they are external tables and move on without failing pre-upgrade > process. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-21254) Pre-upgrade tool should handle exceptions and skip db/tables
[ https://issues.apache.org/jira/browse/HIVE-21254?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Prasanth Jayachandran updated HIVE-21254: - Attachment: HIVE-21254.9.patch > Pre-upgrade tool should handle exceptions and skip db/tables > > > Key: HIVE-21254 > URL: https://issues.apache.org/jira/browse/HIVE-21254 > Project: Hive > Issue Type: Bug > Components: Transactions >Affects Versions: 4.0.0 >Reporter: Prasanth Jayachandran >Assignee: Prasanth Jayachandran >Priority: Major > Attachments: HIVE-21254.1.patch, HIVE-21254.2.patch, > HIVE-21254.3.patch, HIVE-21254.4.patch, HIVE-21254.5.patch, > HIVE-21254.6.patch, HIVE-21254.7.patch, HIVE-21254.8.patch, HIVE-21254.9.patch > > > When exceptions like AccessControlException is thrown, pre-upgrade tool > fails. If hive user does not have read access to database or tables (some > external tables denies read access to hive), pre-upgrade tool should just > assume they are external tables and move on without failing pre-upgrade > process. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-21254) Pre-upgrade tool should handle exceptions and skip db/tables
[ https://issues.apache.org/jira/browse/HIVE-21254?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Prasanth Jayachandran updated HIVE-21254: - Attachment: HIVE-21254.8.patch > Pre-upgrade tool should handle exceptions and skip db/tables > > > Key: HIVE-21254 > URL: https://issues.apache.org/jira/browse/HIVE-21254 > Project: Hive > Issue Type: Bug > Components: Transactions >Affects Versions: 4.0.0 >Reporter: Prasanth Jayachandran >Assignee: Prasanth Jayachandran >Priority: Major > Attachments: HIVE-21254.1.patch, HIVE-21254.2.patch, > HIVE-21254.3.patch, HIVE-21254.4.patch, HIVE-21254.5.patch, > HIVE-21254.6.patch, HIVE-21254.7.patch, HIVE-21254.8.patch > > > When exceptions like AccessControlException is thrown, pre-upgrade tool > fails. If hive user does not have read access to database or tables (some > external tables denies read access to hive), pre-upgrade tool should just > assume they are external tables and move on without failing pre-upgrade > process. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-21254) Pre-upgrade tool should handle exceptions and skip db/tables
[ https://issues.apache.org/jira/browse/HIVE-21254?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16769763#comment-16769763 ] Prasanth Jayachandran commented on HIVE-21254: -- Another unrelated failure. > Pre-upgrade tool should handle exceptions and skip db/tables > > > Key: HIVE-21254 > URL: https://issues.apache.org/jira/browse/HIVE-21254 > Project: Hive > Issue Type: Bug > Components: Transactions >Affects Versions: 4.0.0 >Reporter: Prasanth Jayachandran >Assignee: Prasanth Jayachandran >Priority: Major > Attachments: HIVE-21254.1.patch, HIVE-21254.2.patch, > HIVE-21254.3.patch, HIVE-21254.4.patch, HIVE-21254.5.patch, > HIVE-21254.6.patch, HIVE-21254.7.patch, HIVE-21254.8.patch > > > When exceptions like AccessControlException is thrown, pre-upgrade tool > fails. If hive user does not have read access to database or tables (some > external tables denies read access to hive), pre-upgrade tool should just > assume they are external tables and move on without failing pre-upgrade > process. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-21254) Pre-upgrade tool should handle exceptions and skip db/tables
[ https://issues.apache.org/jira/browse/HIVE-21254?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16768767#comment-16768767 ] Prasanth Jayachandran commented on HIVE-21254: -- Test failures unrelated. Giving it another try. > Pre-upgrade tool should handle exceptions and skip db/tables > > > Key: HIVE-21254 > URL: https://issues.apache.org/jira/browse/HIVE-21254 > Project: Hive > Issue Type: Bug > Components: Transactions >Affects Versions: 4.0.0 >Reporter: Prasanth Jayachandran >Assignee: Prasanth Jayachandran >Priority: Major > Attachments: HIVE-21254.1.patch, HIVE-21254.2.patch, > HIVE-21254.3.patch, HIVE-21254.4.patch, HIVE-21254.5.patch, > HIVE-21254.6.patch, HIVE-21254.7.patch > > > When exceptions like AccessControlException is thrown, pre-upgrade tool > fails. If hive user does not have read access to database or tables (some > external tables denies read access to hive), pre-upgrade tool should just > assume they are external tables and move on without failing pre-upgrade > process. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-21254) Pre-upgrade tool should handle exceptions and skip db/tables
[ https://issues.apache.org/jira/browse/HIVE-21254?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Prasanth Jayachandran updated HIVE-21254: - Attachment: HIVE-21254.7.patch > Pre-upgrade tool should handle exceptions and skip db/tables > > > Key: HIVE-21254 > URL: https://issues.apache.org/jira/browse/HIVE-21254 > Project: Hive > Issue Type: Bug > Components: Transactions >Affects Versions: 4.0.0 >Reporter: Prasanth Jayachandran >Assignee: Prasanth Jayachandran >Priority: Major > Attachments: HIVE-21254.1.patch, HIVE-21254.2.patch, > HIVE-21254.3.patch, HIVE-21254.4.patch, HIVE-21254.5.patch, > HIVE-21254.6.patch, HIVE-21254.7.patch > > > When exceptions like AccessControlException is thrown, pre-upgrade tool > fails. If hive user does not have read access to database or tables (some > external tables denies read access to hive), pre-upgrade tool should just > assume they are external tables and move on without failing pre-upgrade > process. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-21254) Pre-upgrade tool should handle exceptions and skip db/tables
[ https://issues.apache.org/jira/browse/HIVE-21254?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16768613#comment-16768613 ] Prasanth Jayachandran commented on HIVE-21254: -- no idea why applying .5 patch for failed. Rebased patch. > Pre-upgrade tool should handle exceptions and skip db/tables > > > Key: HIVE-21254 > URL: https://issues.apache.org/jira/browse/HIVE-21254 > Project: Hive > Issue Type: Bug > Components: Transactions >Affects Versions: 4.0.0 >Reporter: Prasanth Jayachandran >Assignee: Prasanth Jayachandran >Priority: Major > Attachments: HIVE-21254.1.patch, HIVE-21254.2.patch, > HIVE-21254.3.patch, HIVE-21254.4.patch, HIVE-21254.5.patch, HIVE-21254.6.patch > > > When exceptions like AccessControlException is thrown, pre-upgrade tool > fails. If hive user does not have read access to database or tables (some > external tables denies read access to hive), pre-upgrade tool should just > assume they are external tables and move on without failing pre-upgrade > process. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-21254) Pre-upgrade tool should handle exceptions and skip db/tables
[ https://issues.apache.org/jira/browse/HIVE-21254?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Prasanth Jayachandran updated HIVE-21254: - Attachment: HIVE-21254.6.patch > Pre-upgrade tool should handle exceptions and skip db/tables > > > Key: HIVE-21254 > URL: https://issues.apache.org/jira/browse/HIVE-21254 > Project: Hive > Issue Type: Bug > Components: Transactions >Affects Versions: 4.0.0 >Reporter: Prasanth Jayachandran >Assignee: Prasanth Jayachandran >Priority: Major > Attachments: HIVE-21254.1.patch, HIVE-21254.2.patch, > HIVE-21254.3.patch, HIVE-21254.4.patch, HIVE-21254.5.patch, HIVE-21254.6.patch > > > When exceptions like AccessControlException is thrown, pre-upgrade tool > fails. If hive user does not have read access to database or tables (some > external tables denies read access to hive), pre-upgrade tool should just > assume they are external tables and move on without failing pre-upgrade > process. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-21254) Pre-upgrade tool should handle exceptions and skip db/tables
[ https://issues.apache.org/jira/browse/HIVE-21254?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16767731#comment-16767731 ] Prasanth Jayachandran commented on HIVE-21254: -- Updated patch to fail with workaround suggestion in case access control exception. > Pre-upgrade tool should handle exceptions and skip db/tables > > > Key: HIVE-21254 > URL: https://issues.apache.org/jira/browse/HIVE-21254 > Project: Hive > Issue Type: Bug > Components: Transactions >Affects Versions: 4.0.0 >Reporter: Prasanth Jayachandran >Assignee: Prasanth Jayachandran >Priority: Major > Attachments: HIVE-21254.1.patch, HIVE-21254.2.patch, > HIVE-21254.3.patch, HIVE-21254.4.patch, HIVE-21254.5.patch > > > When exceptions like AccessControlException is thrown, pre-upgrade tool > fails. If hive user does not have read access to database or tables (some > external tables denies read access to hive), pre-upgrade tool should just > assume they are external tables and move on without failing pre-upgrade > process. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-21254) Pre-upgrade tool should handle exceptions and skip db/tables
[ https://issues.apache.org/jira/browse/HIVE-21254?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Prasanth Jayachandran updated HIVE-21254: - Attachment: HIVE-21254.5.patch > Pre-upgrade tool should handle exceptions and skip db/tables > > > Key: HIVE-21254 > URL: https://issues.apache.org/jira/browse/HIVE-21254 > Project: Hive > Issue Type: Bug > Components: Transactions >Affects Versions: 4.0.0 >Reporter: Prasanth Jayachandran >Assignee: Prasanth Jayachandran >Priority: Major > Attachments: HIVE-21254.1.patch, HIVE-21254.2.patch, > HIVE-21254.3.patch, HIVE-21254.4.patch, HIVE-21254.5.patch > > > When exceptions like AccessControlException is thrown, pre-upgrade tool > fails. If hive user does not have read access to database or tables (some > external tables denies read access to hive), pre-upgrade tool should just > assume they are external tables and move on without failing pre-upgrade > process. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-21254) Pre-upgrade tool should handle exceptions and skip db/tables
[ https://issues.apache.org/jira/browse/HIVE-21254?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16767581#comment-16767581 ] Prasanth Jayachandran commented on HIVE-21254: -- Yup. Just discussed with [~ashutoshc] about the same. Detect ACL Exception and explicitly throw with property to allow read access and retry. Will update the patch. > Pre-upgrade tool should handle exceptions and skip db/tables > > > Key: HIVE-21254 > URL: https://issues.apache.org/jira/browse/HIVE-21254 > Project: Hive > Issue Type: Bug > Components: Transactions >Affects Versions: 4.0.0 >Reporter: Prasanth Jayachandran >Assignee: Prasanth Jayachandran >Priority: Major > Attachments: HIVE-21254.1.patch, HIVE-21254.2.patch, > HIVE-21254.3.patch, HIVE-21254.4.patch > > > When exceptions like AccessControlException is thrown, pre-upgrade tool > fails. If hive user does not have read access to database or tables (some > external tables denies read access to hive), pre-upgrade tool should just > assume they are external tables and move on without failing pre-upgrade > process. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-21254) Pre-upgrade tool should handle exceptions and skip db/tables
[ https://issues.apache.org/jira/browse/HIVE-21254?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16767559#comment-16767559 ] Prasanth Jayachandran commented on HIVE-21254: -- Not sure if we can do that since it could be a server side config. > Pre-upgrade tool should handle exceptions and skip db/tables > > > Key: HIVE-21254 > URL: https://issues.apache.org/jira/browse/HIVE-21254 > Project: Hive > Issue Type: Bug > Components: Transactions >Affects Versions: 4.0.0 >Reporter: Prasanth Jayachandran >Assignee: Prasanth Jayachandran >Priority: Major > Attachments: HIVE-21254.1.patch, HIVE-21254.2.patch, > HIVE-21254.3.patch, HIVE-21254.4.patch > > > When exceptions like AccessControlException is thrown, pre-upgrade tool > fails. If hive user does not have read access to database or tables (some > external tables denies read access to hive), pre-upgrade tool should just > assume they are external tables and move on without failing pre-upgrade > process. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-21254) Pre-upgrade tool should handle exceptions and skip db/tables
[ https://issues.apache.org/jira/browse/HIVE-21254?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16767555#comment-16767555 ] Prasanth Jayachandran commented on HIVE-21254: -- Seems like there is flag to allow read access on db and tables. hive.security.metastore.authorization.auth.reads Should we set this to false in pre-upgrade tool explicitly to get read access? > Pre-upgrade tool should handle exceptions and skip db/tables > > > Key: HIVE-21254 > URL: https://issues.apache.org/jira/browse/HIVE-21254 > Project: Hive > Issue Type: Bug > Components: Transactions >Affects Versions: 4.0.0 >Reporter: Prasanth Jayachandran >Assignee: Prasanth Jayachandran >Priority: Major > Attachments: HIVE-21254.1.patch, HIVE-21254.2.patch, > HIVE-21254.3.patch, HIVE-21254.4.patch > > > When exceptions like AccessControlException is thrown, pre-upgrade tool > fails. If hive user does not have read access to database or tables (some > external tables denies read access to hive), pre-upgrade tool should just > assume they are external tables and move on without failing pre-upgrade > process. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-21254) Pre-upgrade tool should handle exceptions and skip db/tables
[ https://issues.apache.org/jira/browse/HIVE-21254?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16767544#comment-16767544 ] Prasanth Jayachandran commented on HIVE-21254: -- Same is applicable for pre-upgrade tool as well. If getTable() in pre-upgrade tool fails with ACL exception then what are the options? This patch is skipping such cases. > Pre-upgrade tool should handle exceptions and skip db/tables > > > Key: HIVE-21254 > URL: https://issues.apache.org/jira/browse/HIVE-21254 > Project: Hive > Issue Type: Bug > Components: Transactions >Affects Versions: 4.0.0 >Reporter: Prasanth Jayachandran >Assignee: Prasanth Jayachandran >Priority: Major > Attachments: HIVE-21254.1.patch, HIVE-21254.2.patch, > HIVE-21254.3.patch, HIVE-21254.4.patch > > > When exceptions like AccessControlException is thrown, pre-upgrade tool > fails. If hive user does not have read access to database or tables (some > external tables denies read access to hive), pre-upgrade tool should just > assume they are external tables and move on without failing pre-upgrade > process. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Comment Edited] (HIVE-21254) Pre-upgrade tool should handle exceptions and skip db/tables
[ https://issues.apache.org/jira/browse/HIVE-21254?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16767537#comment-16767537 ] Prasanth Jayachandran edited comment on HIVE-21254 at 2/13/19 7:57 PM: --- Also how does compactor know the table user, if it cannot do getTable() to get location and find user? (getTable() throws ACL exception if hive user does not have read access to end user location) was (Author: prasanth_j): Also how does compactor know the table user if it cannot do getTable() to get location and find user? getTable() throws ACL exception if hive user does not have read access to end user location. > Pre-upgrade tool should handle exceptions and skip db/tables > > > Key: HIVE-21254 > URL: https://issues.apache.org/jira/browse/HIVE-21254 > Project: Hive > Issue Type: Bug > Components: Transactions >Affects Versions: 4.0.0 >Reporter: Prasanth Jayachandran >Assignee: Prasanth Jayachandran >Priority: Major > Attachments: HIVE-21254.1.patch, HIVE-21254.2.patch, > HIVE-21254.3.patch, HIVE-21254.4.patch > > > When exceptions like AccessControlException is thrown, pre-upgrade tool > fails. If hive user does not have read access to database or tables (some > external tables denies read access to hive), pre-upgrade tool should just > assume they are external tables and move on without failing pre-upgrade > process. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-21254) Pre-upgrade tool should handle exceptions and skip db/tables
[ https://issues.apache.org/jira/browse/HIVE-21254?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16767537#comment-16767537 ] Prasanth Jayachandran commented on HIVE-21254: -- Also how does compactor know the table user if it cannot do getTable() to get location and find user? getTable() throws ACL exception if hive user does not have read access to end user location. > Pre-upgrade tool should handle exceptions and skip db/tables > > > Key: HIVE-21254 > URL: https://issues.apache.org/jira/browse/HIVE-21254 > Project: Hive > Issue Type: Bug > Components: Transactions >Affects Versions: 4.0.0 >Reporter: Prasanth Jayachandran >Assignee: Prasanth Jayachandran >Priority: Major > Attachments: HIVE-21254.1.patch, HIVE-21254.2.patch, > HIVE-21254.3.patch, HIVE-21254.4.patch > > > When exceptions like AccessControlException is thrown, pre-upgrade tool > fails. If hive user does not have read access to database or tables (some > external tables denies read access to hive), pre-upgrade tool should just > assume they are external tables and move on without failing pre-upgrade > process. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-21254) Pre-upgrade tool should handle exceptions and skip db/tables
[ https://issues.apache.org/jira/browse/HIVE-21254?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16767533#comment-16767533 ] Prasanth Jayachandran commented on HIVE-21254: -- What does compactor do in such cases (ACL exception)? does it switch to table owner user and retry compaction? > Pre-upgrade tool should handle exceptions and skip db/tables > > > Key: HIVE-21254 > URL: https://issues.apache.org/jira/browse/HIVE-21254 > Project: Hive > Issue Type: Bug > Components: Transactions >Affects Versions: 4.0.0 >Reporter: Prasanth Jayachandran >Assignee: Prasanth Jayachandran >Priority: Major > Attachments: HIVE-21254.1.patch, HIVE-21254.2.patch, > HIVE-21254.3.patch, HIVE-21254.4.patch > > > When exceptions like AccessControlException is thrown, pre-upgrade tool > fails. If hive user does not have read access to database or tables (some > external tables denies read access to hive), pre-upgrade tool should just > assume they are external tables and move on without failing pre-upgrade > process. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-21254) Pre-upgrade tool should handle exceptions and skip db/tables
[ https://issues.apache.org/jira/browse/HIVE-21254?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16766867#comment-16766867 ] Prasanth Jayachandran commented on HIVE-21254: -- Doesn't look like related test failure. Retrying. > Pre-upgrade tool should handle exceptions and skip db/tables > > > Key: HIVE-21254 > URL: https://issues.apache.org/jira/browse/HIVE-21254 > Project: Hive > Issue Type: Bug > Components: Transactions >Affects Versions: 4.0.0 >Reporter: Prasanth Jayachandran >Assignee: Prasanth Jayachandran >Priority: Major > Attachments: HIVE-21254.1.patch, HIVE-21254.2.patch, > HIVE-21254.3.patch, HIVE-21254.4.patch > > > When exceptions like AccessControlException is thrown, pre-upgrade tool > fails. If hive user does not have read access to database or tables (some > external tables denies read access to hive), pre-upgrade tool should just > assume they are external tables and move on without failing pre-upgrade > process. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-21254) Pre-upgrade tool should handle exceptions and skip db/tables
[ https://issues.apache.org/jira/browse/HIVE-21254?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Prasanth Jayachandran updated HIVE-21254: - Attachment: HIVE-21254.4.patch > Pre-upgrade tool should handle exceptions and skip db/tables > > > Key: HIVE-21254 > URL: https://issues.apache.org/jira/browse/HIVE-21254 > Project: Hive > Issue Type: Bug > Components: Transactions >Affects Versions: 4.0.0 >Reporter: Prasanth Jayachandran >Assignee: Prasanth Jayachandran >Priority: Major > Attachments: HIVE-21254.1.patch, HIVE-21254.2.patch, > HIVE-21254.3.patch, HIVE-21254.4.patch > > > When exceptions like AccessControlException is thrown, pre-upgrade tool > fails. If hive user does not have read access to database or tables (some > external tables denies read access to hive), pre-upgrade tool should just > assume they are external tables and move on without failing pre-upgrade > process. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-21254) Pre-upgrade tool should handle exceptions and skip db/tables
[ https://issues.apache.org/jira/browse/HIVE-21254?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=1678#comment-1678 ] Prasanth Jayachandran commented on HIVE-21254: -- Added fix for test cleanup issue. > Pre-upgrade tool should handle exceptions and skip db/tables > > > Key: HIVE-21254 > URL: https://issues.apache.org/jira/browse/HIVE-21254 > Project: Hive > Issue Type: Bug > Components: Transactions >Affects Versions: 4.0.0 >Reporter: Prasanth Jayachandran >Assignee: Prasanth Jayachandran >Priority: Major > Attachments: HIVE-21254.1.patch, HIVE-21254.2.patch, > HIVE-21254.3.patch > > > When exceptions like AccessControlException is thrown, pre-upgrade tool > fails. If hive user does not have read access to database or tables (some > external tables denies read access to hive), pre-upgrade tool should just > assume they are external tables and move on without failing pre-upgrade > process. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-21254) Pre-upgrade tool should handle exceptions and skip db/tables
[ https://issues.apache.org/jira/browse/HIVE-21254?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Prasanth Jayachandran updated HIVE-21254: - Attachment: HIVE-21254.3.patch > Pre-upgrade tool should handle exceptions and skip db/tables > > > Key: HIVE-21254 > URL: https://issues.apache.org/jira/browse/HIVE-21254 > Project: Hive > Issue Type: Bug > Components: Transactions >Affects Versions: 4.0.0 >Reporter: Prasanth Jayachandran >Assignee: Prasanth Jayachandran >Priority: Major > Attachments: HIVE-21254.1.patch, HIVE-21254.2.patch, > HIVE-21254.3.patch > > > When exceptions like AccessControlException is thrown, pre-upgrade tool > fails. If hive user does not have read access to database or tables (some > external tables denies read access to hive), pre-upgrade tool should just > assume they are external tables and move on without failing pre-upgrade > process. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-21254) Pre-upgrade tool should handle exceptions and skip db/tables
[ https://issues.apache.org/jira/browse/HIVE-21254?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Prasanth Jayachandran updated HIVE-21254: - Attachment: HIVE-21254.2.patch > Pre-upgrade tool should handle exceptions and skip db/tables > > > Key: HIVE-21254 > URL: https://issues.apache.org/jira/browse/HIVE-21254 > Project: Hive > Issue Type: Bug > Components: Transactions >Affects Versions: 4.0.0 >Reporter: Prasanth Jayachandran >Assignee: Prasanth Jayachandran >Priority: Major > Attachments: HIVE-21254.1.patch, HIVE-21254.2.patch > > > When exceptions like AccessControlException is thrown, pre-upgrade tool > fails. If hive user does not have read access to database or tables (some > external tables denies read access to hive), pre-upgrade tool should just > assume they are external tables and move on without failing pre-upgrade > process. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-21254) Pre-upgrade tool should handle exceptions and skip db/tables
[ https://issues.apache.org/jira/browse/HIVE-21254?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16766630#comment-16766630 ] Prasanth Jayachandran commented on HIVE-21254: -- Handled ACL exception with a warning to run manual compaction per offline discussion with [~jdere] > Pre-upgrade tool should handle exceptions and skip db/tables > > > Key: HIVE-21254 > URL: https://issues.apache.org/jira/browse/HIVE-21254 > Project: Hive > Issue Type: Bug > Components: Transactions >Affects Versions: 4.0.0 >Reporter: Prasanth Jayachandran >Assignee: Prasanth Jayachandran >Priority: Major > Attachments: HIVE-21254.1.patch, HIVE-21254.2.patch > > > When exceptions like AccessControlException is thrown, pre-upgrade tool > fails. If hive user does not have read access to database or tables (some > external tables denies read access to hive), pre-upgrade tool should just > assume they are external tables and move on without failing pre-upgrade > process. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-21254) Pre-upgrade tool should handle exceptions and skip db/tables
[ https://issues.apache.org/jira/browse/HIVE-21254?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16766602#comment-16766602 ] Prasanth Jayachandran commented on HIVE-21254: -- Make sense. Will update patch to handle only AccessControlException. > Pre-upgrade tool should handle exceptions and skip db/tables > > > Key: HIVE-21254 > URL: https://issues.apache.org/jira/browse/HIVE-21254 > Project: Hive > Issue Type: Bug > Components: Transactions >Affects Versions: 4.0.0 >Reporter: Prasanth Jayachandran >Assignee: Prasanth Jayachandran >Priority: Major > Attachments: HIVE-21254.1.patch > > > When exceptions like AccessControlException is thrown, pre-upgrade tool > fails. If hive user does not have read access to database or tables (some > external tables denies read access to hive), pre-upgrade tool should just > assume they are external tables and move on without failing pre-upgrade > process. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-21254) Pre-upgrade tool should handle exceptions and skip db/tables
[ https://issues.apache.org/jira/browse/HIVE-21254?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16766595#comment-16766595 ] Prasanth Jayachandran commented on HIVE-21254: -- Attaching RB link. [~ekoifman]/ [~jdere] can someone please review this patch? > Pre-upgrade tool should handle exceptions and skip db/tables > > > Key: HIVE-21254 > URL: https://issues.apache.org/jira/browse/HIVE-21254 > Project: Hive > Issue Type: Bug > Components: Transactions >Affects Versions: 4.0.0 >Reporter: Prasanth Jayachandran >Assignee: Prasanth Jayachandran >Priority: Major > Attachments: HIVE-21254.1.patch > > > When exceptions like AccessControlException is thrown, pre-upgrade tool > fails. If hive user does not have read access to database or tables (some > external tables denies read access to hive), pre-upgrade tool should just > assume they are external tables and move on without failing pre-upgrade > process. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-21254) Pre-upgrade tool should handle exceptions and skip db/tables
[ https://issues.apache.org/jira/browse/HIVE-21254?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Prasanth Jayachandran updated HIVE-21254: - Status: Patch Available (was: Open) > Pre-upgrade tool should handle exceptions and skip db/tables > > > Key: HIVE-21254 > URL: https://issues.apache.org/jira/browse/HIVE-21254 > Project: Hive > Issue Type: Bug > Components: Transactions >Affects Versions: 4.0.0 >Reporter: Prasanth Jayachandran >Assignee: Prasanth Jayachandran >Priority: Major > Attachments: HIVE-21254.1.patch > > > When exceptions like AccessControlException is thrown, pre-upgrade tool > fails. If hive user does not have read access to database or tables (some > external tables denies read access to hive), pre-upgrade tool should just > assume they are external tables and move on without failing pre-upgrade > process. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-21254) Pre-upgrade tool should handle exceptions and skip db/tables
[ https://issues.apache.org/jira/browse/HIVE-21254?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Prasanth Jayachandran updated HIVE-21254: - Attachment: HIVE-21254.1.patch > Pre-upgrade tool should handle exceptions and skip db/tables > > > Key: HIVE-21254 > URL: https://issues.apache.org/jira/browse/HIVE-21254 > Project: Hive > Issue Type: Bug > Components: Transactions >Affects Versions: 4.0.0 >Reporter: Prasanth Jayachandran >Assignee: Prasanth Jayachandran >Priority: Major > Attachments: HIVE-21254.1.patch > > > When exceptions like AccessControlException is thrown, pre-upgrade tool > fails. If hive user does not have read access to database or tables (some > external tables denies read access to hive), pre-upgrade tool should just > assume they are external tables and move on without failing pre-upgrade > process. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (HIVE-21254) Pre-upgrade tool should handle exceptions and skip db/tables
[ https://issues.apache.org/jira/browse/HIVE-21254?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Prasanth Jayachandran reassigned HIVE-21254: > Pre-upgrade tool should handle exceptions and skip db/tables > > > Key: HIVE-21254 > URL: https://issues.apache.org/jira/browse/HIVE-21254 > Project: Hive > Issue Type: Bug > Components: Transactions >Affects Versions: 4.0.0 >Reporter: Prasanth Jayachandran >Assignee: Prasanth Jayachandran >Priority: Major > > When exceptions like AccessControlException is thrown, pre-upgrade tool > fails. If hive user does not have read access to database or tables (some > external tables denies read access to hive), pre-upgrade tool should just > assume they are external tables and move on without failing pre-upgrade > process. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-21212) LLAP: shuffle port config uses internal configuration
[ https://issues.apache.org/jira/browse/HIVE-21212?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Prasanth Jayachandran updated HIVE-21212: - Resolution: Fixed Fix Version/s: 4.0.0 Status: Resolved (was: Patch Available) Committed to master. Thanks for the review! > LLAP: shuffle port config uses internal configuration > - > > Key: HIVE-21212 > URL: https://issues.apache.org/jira/browse/HIVE-21212 > Project: Hive > Issue Type: Bug > Components: llap >Affects Versions: 4.0.0 >Reporter: Prasanth Jayachandran >Assignee: Prasanth Jayachandran >Priority: Major > Fix For: 4.0.0 > > Attachments: HIVE-21212.1.patch > > > LlapDaemon main() reads daemon configuration but for shuffle port it reads > internal config instead of hive.llap.daemon.yarn.shuffle.port > [https://github.com/apache/hive/blob/c8eb03affa2533f4827cf6497e7c9873bc9520a7/llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/LlapDaemon.java#L535] > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-21103) PartitionManagementTask should not modify DN configs to avoid closing persistence manager
[ https://issues.apache.org/jira/browse/HIVE-21103?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Prasanth Jayachandran updated HIVE-21103: - Resolution: Fixed Fix Version/s: 4.0.0 Status: Resolved (was: Patch Available) Test failure is fixed in master already (result of missed binary file during commit). Committed to master. Thanks for the review! > PartitionManagementTask should not modify DN configs to avoid closing > persistence manager > - > > Key: HIVE-21103 > URL: https://issues.apache.org/jira/browse/HIVE-21103 > Project: Hive > Issue Type: Bug > Components: Standalone Metastore >Affects Versions: 4.0.0 >Reporter: Prasanth Jayachandran >Assignee: Prasanth Jayachandran >Priority: Critical > Fix For: 4.0.0 > > Attachments: HIVE-21103.1.patch, HIVE-21103.2.patch, > HIVE-21103.3.patch > > > HIVE-20707 added automatic partition management which uses thread pools to > run parallel msck repair. It also modifies datanucleus connection pool size > to avoid explosion of connections to backend database. But object store > closes the persistence manager when it detects a change in datanuclues or jdo > configs. So when PartitionManagementTask is running and when HS2 tries to > connect to metastore HS2 will get persistence manager close exception. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-20841) LLAP: Make dynamic ports configurable
[ https://issues.apache.org/jira/browse/HIVE-20841?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Prasanth Jayachandran updated HIVE-20841: - Resolution: Fixed Status: Resolved (was: Patch Available) Test failure is already fixed in master (result of missed binary file in commit). Committed to master and branch-3. Thanks for the review! > LLAP: Make dynamic ports configurable > - > > Key: HIVE-20841 > URL: https://issues.apache.org/jira/browse/HIVE-20841 > Project: Hive > Issue Type: Bug > Components: llap >Affects Versions: 4.0.0, 3.2.0 >Reporter: Prasanth Jayachandran >Assignee: Prasanth Jayachandran >Priority: Major > Fix For: 4.0.0, 3.2.0 > > Attachments: HIVE-20841.1.patch, HIVE-20841.2.patch > > > Some ports in llap -> tez interaction code uses dynamic ports, provide an > option to make them configurable to facilitate adding them to iptable rules > in some environment. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-21222) ACID: When there are no delete deltas skip finding min max keys
[ https://issues.apache.org/jira/browse/HIVE-21222?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Prasanth Jayachandran updated HIVE-21222: - Resolution: Fixed Fix Version/s: 4.0.0 Target Version/s: 4.0.0 (was: 4.0.0, 3.2.0) Status: Resolved (was: Patch Available) Committed to master. Thanks for the review!. The test failure is already fixed in master (result of a missed binary file). > ACID: When there are no delete deltas skip finding min max keys > --- > > Key: HIVE-21222 > URL: https://issues.apache.org/jira/browse/HIVE-21222 > Project: Hive > Issue Type: Bug > Components: Transactions >Affects Versions: 4.0.0, 3.2.0 >Reporter: Prasanth Jayachandran >Assignee: Prasanth Jayachandran >Priority: Major > Fix For: 4.0.0 > > Attachments: HIVE-21222.1.patch, HIVE-21222.2.patch > > > We create an orc reader in VectorizedOrcAcidRowBatchReader.findMinMaxKeys > (which will read 16K footer) even for cases where delete deltas does not > exist. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-21235) LLAP: make the name of log4j2 properties file configurable
[ https://issues.apache.org/jira/browse/HIVE-21235?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Prasanth Jayachandran updated HIVE-21235: - Resolution: Fixed Fix Version/s: 3.2.0 4.0.0 Status: Resolved (was: Patch Available) Committed to branch-3 and master. Thanks for the review! > LLAP: make the name of log4j2 properties file configurable > -- > > Key: HIVE-21235 > URL: https://issues.apache.org/jira/browse/HIVE-21235 > Project: Hive > Issue Type: Bug > Components: llap >Affects Versions: 4.0.0 >Reporter: Prasanth Jayachandran >Assignee: Prasanth Jayachandran >Priority: Major > Fix For: 4.0.0, 3.2.0 > > Attachments: HIVE-21235.1.patch > > > For llap daemon, the name of llap-daemon-log4j2.properties is fixed. If a > conf dir and jar contain the same filename, it will mess up log4j2 > initialization. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-21223) CachedStore returns null partition when partition does not exist
[
https://issues.apache.org/jira/browse/HIVE-21223?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Prasanth Jayachandran updated HIVE-21223:
-
Resolution: Fixed
Fix Version/s: 3.2.0
4.0.0
Status: Resolved (was: Patch Available)
Committed to branch-3 and master. Thanks for the review!
> CachedStore returns null partition when partition does not exist
>
>
> Key: HIVE-21223
> URL: https://issues.apache.org/jira/browse/HIVE-21223
> Project: Hive
> Issue Type: Bug
> Components: Metastore
>Affects Versions: 4.0.0, 3.2.0
>Reporter: Prasanth Jayachandran
>Assignee: Prasanth Jayachandran
>Priority: Major
> Fix For: 4.0.0, 3.2.0
>
> Attachments: HIVE-21223.1.patch, HIVE-21223.2.patch
>
>
> CachedStore can return null partition for getPartitionWithAuth() when
> partition does not exist. null value serialization in thrift will break the
> connection. Instead if partition does not exist it should throw
> NoSuchObjectException.
> Clients will see this exception
> {code:java}
> org.apache.thrift.TApplicationException: get_partition_with_auth failed:
> unknown result
> at
> org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.recv_get_partition_with_auth(ThriftHiveMetastore.java:3017)
> ~[hive-exec-3.1.0.3.0.100.0-266.jar:3.1.0.3.0.100.0-266]
> at
> org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.get_partition_with_auth(ThriftHiveMetastore.java:2990)
> ~[hive-exec-3.1.0.3.0.100.0-266.jar:3.1.0.3.0.100.0-266]
> at
> org.apache.hadoop.hive.metastore.HiveMetaStoreClient.getPartitionWithAuthInfo(HiveMetaStoreClient.java:1679)
> ~[hive-exec-3.1.0.3.0.100.0-266.jar:3.1.0.3.0.100.0-266]
> at
> org.apache.hadoop.hive.metastore.HiveMetaStoreClient.getPartitionWithAuthInfo(HiveMetaStoreClient.java:1671)
> ~[hive-exec-3.1.0.3.0.100.0-266.jar:3.1.0.3.0.100.0-266]
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0_181]
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> ~[?:1.8.0_181]
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> ~[?:1.8.0_181]
> at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_181]
> at
> org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.invoke(RetryingMetaStoreClient.java:212)
> ~[hive-exec-3.1.0.3.0.100.0-266.jar:3.1.0.3.0.100.0-266]
> at com.sun.proxy.$Proxy36.getPartitionWithAuthInfo(Unknown Source) ~[?:?]
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0_181]
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> ~[?:1.8.0_181]
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> ~[?:1.8.0_181]
> at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_181]
> at
> org.apache.hadoop.hive.metastore.HiveMetaStoreClient$SynchronizedHandler.invoke(HiveMetaStoreClient.java:2976)
> ~[hive-exec-3.1.0.3.0.100.0-266.jar:3.1.0.3.0.100.0-266]
> at com.sun.proxy.$Proxy36.getPartitionWithAuthInfo(Unknown Source) ~[?:?]
> at
> org.apache.hadoop.hive.metastore.SynchronizedMetaStoreClient.getPartitionWithAuthInfo(SynchronizedMetaStoreClient.java:101)
> ~[hive-exec-3.1.0.3.0.100.0-266.jar:3.1.0.3.0.100.0-266]
> at org.apache.hadoop.hive.ql.metadata.Hive.getPartition(Hive.java:2870)
> ~[hive-exec-3.1.0.3.0.100.0-266.jar:3.1.0.3.0.100.0-266]
> at org.apache.hadoop.hive.ql.metadata.Hive.getPartition(Hive.java:2835)
> ~[hive-exec-3.1.0.3.0.100.0-266.jar:3.1.0.3.0.100.0-266]
> at org.apache.hadoop.hive.ql.metadata.Hive.loadPartition(Hive.java:1950)
> ~[hive-exec-3.1.0.3.0.100.0-266.jar:3.1.0.3.0.100.0-266]
> at org.apache.hadoop.hive.ql.metadata.Hive$4.call(Hive.java:2490)
> ~[hive-exec-3.1.0.3.0.100.0-266.jar:3.1.0.3.0.100.0-266]
> at org.apache.hadoop.hive.ql.metadata.Hive$4.call(Hive.java:2481)
> ~[hive-exec-3.1.0.3.0.100.0-266.jar:3.1.0.3.0.100.0-266]
> at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[?:1.8.0_181]
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> ~[?:1.8.0_181]
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> ~[?:1.8.0_181]
> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_181]{code}
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-21244) NPE in Hive Proto Logger
[ https://issues.apache.org/jira/browse/HIVE-21244?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Prasanth Jayachandran updated HIVE-21244: - Attachment: HIVE-21244.2.patch > NPE in Hive Proto Logger > > > Key: HIVE-21244 > URL: https://issues.apache.org/jira/browse/HIVE-21244 > Project: Hive > Issue Type: Bug >Affects Versions: 4.0.0 >Reporter: Prasanth Jayachandran >Assignee: Prasanth Jayachandran >Priority: Major > Attachments: HIVE-21244.1.patch, HIVE-21244.2.patch > > > [https://github.com/apache/hive/blob/4ddc9de90b6de032d77709c9631ab787cef225d5/ql/src/java/org/apache/hadoop/hive/ql/hooks/HiveProtoLoggingHook.java#L308] > can cause NPE. There is no uncaught exception handler for this thread. This > NPE can silently fail and drop the event. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-21244) NPE in Hive Proto Logger
[ https://issues.apache.org/jira/browse/HIVE-21244?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16765777#comment-16765777 ] Prasanth Jayachandran commented on HIVE-21244: -- Updated the comment > NPE in Hive Proto Logger > > > Key: HIVE-21244 > URL: https://issues.apache.org/jira/browse/HIVE-21244 > Project: Hive > Issue Type: Bug >Affects Versions: 4.0.0 >Reporter: Prasanth Jayachandran >Assignee: Prasanth Jayachandran >Priority: Major > Attachments: HIVE-21244.1.patch, HIVE-21244.2.patch > > > [https://github.com/apache/hive/blob/4ddc9de90b6de032d77709c9631ab787cef225d5/ql/src/java/org/apache/hadoop/hive/ql/hooks/HiveProtoLoggingHook.java#L308] > can cause NPE. There is no uncaught exception handler for this thread. This > NPE can silently fail and drop the event. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-21223) CachedStore returns null partition when partition does not exist
[
https://issues.apache.org/jira/browse/HIVE-21223?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Prasanth Jayachandran updated HIVE-21223:
-
Attachment: HIVE-21223.2.patch
> CachedStore returns null partition when partition does not exist
>
>
> Key: HIVE-21223
> URL: https://issues.apache.org/jira/browse/HIVE-21223
> Project: Hive
> Issue Type: Bug
> Components: Metastore
>Affects Versions: 4.0.0, 3.2.0
>Reporter: Prasanth Jayachandran
>Assignee: Prasanth Jayachandran
>Priority: Major
> Attachments: HIVE-21223.1.patch, HIVE-21223.2.patch
>
>
> CachedStore can return null partition for getPartitionWithAuth() when
> partition does not exist. null value serialization in thrift will break the
> connection. Instead if partition does not exist it should throw
> NoSuchObjectException.
> Clients will see this exception
> {code:java}
> org.apache.thrift.TApplicationException: get_partition_with_auth failed:
> unknown result
> at
> org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.recv_get_partition_with_auth(ThriftHiveMetastore.java:3017)
> ~[hive-exec-3.1.0.3.0.100.0-266.jar:3.1.0.3.0.100.0-266]
> at
> org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.get_partition_with_auth(ThriftHiveMetastore.java:2990)
> ~[hive-exec-3.1.0.3.0.100.0-266.jar:3.1.0.3.0.100.0-266]
> at
> org.apache.hadoop.hive.metastore.HiveMetaStoreClient.getPartitionWithAuthInfo(HiveMetaStoreClient.java:1679)
> ~[hive-exec-3.1.0.3.0.100.0-266.jar:3.1.0.3.0.100.0-266]
> at
> org.apache.hadoop.hive.metastore.HiveMetaStoreClient.getPartitionWithAuthInfo(HiveMetaStoreClient.java:1671)
> ~[hive-exec-3.1.0.3.0.100.0-266.jar:3.1.0.3.0.100.0-266]
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0_181]
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> ~[?:1.8.0_181]
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> ~[?:1.8.0_181]
> at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_181]
> at
> org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.invoke(RetryingMetaStoreClient.java:212)
> ~[hive-exec-3.1.0.3.0.100.0-266.jar:3.1.0.3.0.100.0-266]
> at com.sun.proxy.$Proxy36.getPartitionWithAuthInfo(Unknown Source) ~[?:?]
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0_181]
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> ~[?:1.8.0_181]
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> ~[?:1.8.0_181]
> at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_181]
> at
> org.apache.hadoop.hive.metastore.HiveMetaStoreClient$SynchronizedHandler.invoke(HiveMetaStoreClient.java:2976)
> ~[hive-exec-3.1.0.3.0.100.0-266.jar:3.1.0.3.0.100.0-266]
> at com.sun.proxy.$Proxy36.getPartitionWithAuthInfo(Unknown Source) ~[?:?]
> at
> org.apache.hadoop.hive.metastore.SynchronizedMetaStoreClient.getPartitionWithAuthInfo(SynchronizedMetaStoreClient.java:101)
> ~[hive-exec-3.1.0.3.0.100.0-266.jar:3.1.0.3.0.100.0-266]
> at org.apache.hadoop.hive.ql.metadata.Hive.getPartition(Hive.java:2870)
> ~[hive-exec-3.1.0.3.0.100.0-266.jar:3.1.0.3.0.100.0-266]
> at org.apache.hadoop.hive.ql.metadata.Hive.getPartition(Hive.java:2835)
> ~[hive-exec-3.1.0.3.0.100.0-266.jar:3.1.0.3.0.100.0-266]
> at org.apache.hadoop.hive.ql.metadata.Hive.loadPartition(Hive.java:1950)
> ~[hive-exec-3.1.0.3.0.100.0-266.jar:3.1.0.3.0.100.0-266]
> at org.apache.hadoop.hive.ql.metadata.Hive$4.call(Hive.java:2490)
> ~[hive-exec-3.1.0.3.0.100.0-266.jar:3.1.0.3.0.100.0-266]
> at org.apache.hadoop.hive.ql.metadata.Hive$4.call(Hive.java:2481)
> ~[hive-exec-3.1.0.3.0.100.0-266.jar:3.1.0.3.0.100.0-266]
> at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[?:1.8.0_181]
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> ~[?:1.8.0_181]
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> ~[?:1.8.0_181]
> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_181]{code}
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-21244) NPE in Hive Proto Logger
[ https://issues.apache.org/jira/browse/HIVE-21244?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16765366#comment-16765366 ] Prasanth Jayachandran commented on HIVE-21244: -- [~harishjp]/[~anishek] I tested this patch on cluster and it seems to work correctly. Can you please review? > NPE in Hive Proto Logger > > > Key: HIVE-21244 > URL: https://issues.apache.org/jira/browse/HIVE-21244 > Project: Hive > Issue Type: Bug >Affects Versions: 4.0.0 >Reporter: Prasanth Jayachandran >Assignee: Prasanth Jayachandran >Priority: Major > Attachments: HIVE-21244.1.patch > > > [https://github.com/apache/hive/blob/4ddc9de90b6de032d77709c9631ab787cef225d5/ql/src/java/org/apache/hadoop/hive/ql/hooks/HiveProtoLoggingHook.java#L308] > can cause NPE. There is no uncaught exception handler for this thread. This > NPE can silently fail and drop the event. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-21244) NPE in Hive Proto Logger
[ https://issues.apache.org/jira/browse/HIVE-21244?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Prasanth Jayachandran updated HIVE-21244: - Attachment: HIVE-21244.1.patch > NPE in Hive Proto Logger > > > Key: HIVE-21244 > URL: https://issues.apache.org/jira/browse/HIVE-21244 > Project: Hive > Issue Type: Bug >Affects Versions: 4.0.0 >Reporter: Prasanth Jayachandran >Assignee: Prasanth Jayachandran >Priority: Major > Attachments: HIVE-21244.1.patch > > > [https://github.com/apache/hive/blob/4ddc9de90b6de032d77709c9631ab787cef225d5/ql/src/java/org/apache/hadoop/hive/ql/hooks/HiveProtoLoggingHook.java#L308] > can cause NPE. There is no uncaught exception handler for this thread. This > NPE can silently fail and drop the event. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-21244) NPE in Hive Proto Logger
[ https://issues.apache.org/jira/browse/HIVE-21244?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Prasanth Jayachandran updated HIVE-21244: - Status: Patch Available (was: Open) > NPE in Hive Proto Logger > > > Key: HIVE-21244 > URL: https://issues.apache.org/jira/browse/HIVE-21244 > Project: Hive > Issue Type: Bug >Affects Versions: 4.0.0 >Reporter: Prasanth Jayachandran >Assignee: Prasanth Jayachandran >Priority: Major > Attachments: HIVE-21244.1.patch > > > [https://github.com/apache/hive/blob/4ddc9de90b6de032d77709c9631ab787cef225d5/ql/src/java/org/apache/hadoop/hive/ql/hooks/HiveProtoLoggingHook.java#L308] > can cause NPE. There is no uncaught exception handler for this thread. This > NPE can silently fail and drop the event. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (HIVE-21244) NPE in Hive Proto Logger
[ https://issues.apache.org/jira/browse/HIVE-21244?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Prasanth Jayachandran reassigned HIVE-21244: > NPE in Hive Proto Logger > > > Key: HIVE-21244 > URL: https://issues.apache.org/jira/browse/HIVE-21244 > Project: Hive > Issue Type: Bug >Affects Versions: 4.0.0 >Reporter: Prasanth Jayachandran >Assignee: Prasanth Jayachandran >Priority: Major > > [https://github.com/apache/hive/blob/4ddc9de90b6de032d77709c9631ab787cef225d5/ql/src/java/org/apache/hadoop/hive/ql/hooks/HiveProtoLoggingHook.java#L308] > can cause NPE. There is no uncaught exception handler for this thread. This > NPE can silently fail and drop the event. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-21223) CachedStore returns null partition when partition does not exist
[
https://issues.apache.org/jira/browse/HIVE-21223?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16764752#comment-16764752
]
Prasanth Jayachandran commented on HIVE-21223:
--
[~vgumashta] can you please review this small patch?
> CachedStore returns null partition when partition does not exist
>
>
> Key: HIVE-21223
> URL: https://issues.apache.org/jira/browse/HIVE-21223
> Project: Hive
> Issue Type: Bug
> Components: Metastore
>Affects Versions: 4.0.0, 3.2.0
>Reporter: Prasanth Jayachandran
>Assignee: Prasanth Jayachandran
>Priority: Major
> Attachments: HIVE-21223.1.patch
>
>
> CachedStore can return null partition for getPartitionWithAuth() when
> partition does not exist. null value serialization in thrift will break the
> connection. Instead if partition does not exist it should throw
> NoSuchObjectException.
> Clients will see this exception
> {code:java}
> org.apache.thrift.TApplicationException: get_partition_with_auth failed:
> unknown result
> at
> org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.recv_get_partition_with_auth(ThriftHiveMetastore.java:3017)
> ~[hive-exec-3.1.0.3.0.100.0-266.jar:3.1.0.3.0.100.0-266]
> at
> org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.get_partition_with_auth(ThriftHiveMetastore.java:2990)
> ~[hive-exec-3.1.0.3.0.100.0-266.jar:3.1.0.3.0.100.0-266]
> at
> org.apache.hadoop.hive.metastore.HiveMetaStoreClient.getPartitionWithAuthInfo(HiveMetaStoreClient.java:1679)
> ~[hive-exec-3.1.0.3.0.100.0-266.jar:3.1.0.3.0.100.0-266]
> at
> org.apache.hadoop.hive.metastore.HiveMetaStoreClient.getPartitionWithAuthInfo(HiveMetaStoreClient.java:1671)
> ~[hive-exec-3.1.0.3.0.100.0-266.jar:3.1.0.3.0.100.0-266]
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0_181]
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> ~[?:1.8.0_181]
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> ~[?:1.8.0_181]
> at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_181]
> at
> org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.invoke(RetryingMetaStoreClient.java:212)
> ~[hive-exec-3.1.0.3.0.100.0-266.jar:3.1.0.3.0.100.0-266]
> at com.sun.proxy.$Proxy36.getPartitionWithAuthInfo(Unknown Source) ~[?:?]
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0_181]
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> ~[?:1.8.0_181]
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> ~[?:1.8.0_181]
> at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_181]
> at
> org.apache.hadoop.hive.metastore.HiveMetaStoreClient$SynchronizedHandler.invoke(HiveMetaStoreClient.java:2976)
> ~[hive-exec-3.1.0.3.0.100.0-266.jar:3.1.0.3.0.100.0-266]
> at com.sun.proxy.$Proxy36.getPartitionWithAuthInfo(Unknown Source) ~[?:?]
> at
> org.apache.hadoop.hive.metastore.SynchronizedMetaStoreClient.getPartitionWithAuthInfo(SynchronizedMetaStoreClient.java:101)
> ~[hive-exec-3.1.0.3.0.100.0-266.jar:3.1.0.3.0.100.0-266]
> at org.apache.hadoop.hive.ql.metadata.Hive.getPartition(Hive.java:2870)
> ~[hive-exec-3.1.0.3.0.100.0-266.jar:3.1.0.3.0.100.0-266]
> at org.apache.hadoop.hive.ql.metadata.Hive.getPartition(Hive.java:2835)
> ~[hive-exec-3.1.0.3.0.100.0-266.jar:3.1.0.3.0.100.0-266]
> at org.apache.hadoop.hive.ql.metadata.Hive.loadPartition(Hive.java:1950)
> ~[hive-exec-3.1.0.3.0.100.0-266.jar:3.1.0.3.0.100.0-266]
> at org.apache.hadoop.hive.ql.metadata.Hive$4.call(Hive.java:2490)
> ~[hive-exec-3.1.0.3.0.100.0-266.jar:3.1.0.3.0.100.0-266]
> at org.apache.hadoop.hive.ql.metadata.Hive$4.call(Hive.java:2481)
> ~[hive-exec-3.1.0.3.0.100.0-266.jar:3.1.0.3.0.100.0-266]
> at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[?:1.8.0_181]
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> ~[?:1.8.0_181]
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> ~[?:1.8.0_181]
> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_181]{code}
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-21223) CachedStore returns null partition when partition does not exist
[
https://issues.apache.org/jira/browse/HIVE-21223?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Prasanth Jayachandran updated HIVE-21223:
-
Attachment: HIVE-21223.1.patch
> CachedStore returns null partition when partition does not exist
>
>
> Key: HIVE-21223
> URL: https://issues.apache.org/jira/browse/HIVE-21223
> Project: Hive
> Issue Type: Bug
> Components: Metastore
>Affects Versions: 4.0.0, 3.2.0
>Reporter: Prasanth Jayachandran
>Priority: Major
> Attachments: HIVE-21223.1.patch
>
>
> CachedStore can return null partition for getPartitionWithAuth() when
> partition does not exist. null value serialization in thrift will break the
> connection. Instead if partition does not exist it should throw
> NoSuchObjectException.
> Clients will see this exception
> {code:java}
> org.apache.thrift.TApplicationException: get_partition_with_auth failed:
> unknown result
> at
> org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.recv_get_partition_with_auth(ThriftHiveMetastore.java:3017)
> ~[hive-exec-3.1.0.3.0.100.0-266.jar:3.1.0.3.0.100.0-266]
> at
> org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.get_partition_with_auth(ThriftHiveMetastore.java:2990)
> ~[hive-exec-3.1.0.3.0.100.0-266.jar:3.1.0.3.0.100.0-266]
> at
> org.apache.hadoop.hive.metastore.HiveMetaStoreClient.getPartitionWithAuthInfo(HiveMetaStoreClient.java:1679)
> ~[hive-exec-3.1.0.3.0.100.0-266.jar:3.1.0.3.0.100.0-266]
> at
> org.apache.hadoop.hive.metastore.HiveMetaStoreClient.getPartitionWithAuthInfo(HiveMetaStoreClient.java:1671)
> ~[hive-exec-3.1.0.3.0.100.0-266.jar:3.1.0.3.0.100.0-266]
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0_181]
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> ~[?:1.8.0_181]
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> ~[?:1.8.0_181]
> at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_181]
> at
> org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.invoke(RetryingMetaStoreClient.java:212)
> ~[hive-exec-3.1.0.3.0.100.0-266.jar:3.1.0.3.0.100.0-266]
> at com.sun.proxy.$Proxy36.getPartitionWithAuthInfo(Unknown Source) ~[?:?]
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0_181]
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> ~[?:1.8.0_181]
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> ~[?:1.8.0_181]
> at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_181]
> at
> org.apache.hadoop.hive.metastore.HiveMetaStoreClient$SynchronizedHandler.invoke(HiveMetaStoreClient.java:2976)
> ~[hive-exec-3.1.0.3.0.100.0-266.jar:3.1.0.3.0.100.0-266]
> at com.sun.proxy.$Proxy36.getPartitionWithAuthInfo(Unknown Source) ~[?:?]
> at
> org.apache.hadoop.hive.metastore.SynchronizedMetaStoreClient.getPartitionWithAuthInfo(SynchronizedMetaStoreClient.java:101)
> ~[hive-exec-3.1.0.3.0.100.0-266.jar:3.1.0.3.0.100.0-266]
> at org.apache.hadoop.hive.ql.metadata.Hive.getPartition(Hive.java:2870)
> ~[hive-exec-3.1.0.3.0.100.0-266.jar:3.1.0.3.0.100.0-266]
> at org.apache.hadoop.hive.ql.metadata.Hive.getPartition(Hive.java:2835)
> ~[hive-exec-3.1.0.3.0.100.0-266.jar:3.1.0.3.0.100.0-266]
> at org.apache.hadoop.hive.ql.metadata.Hive.loadPartition(Hive.java:1950)
> ~[hive-exec-3.1.0.3.0.100.0-266.jar:3.1.0.3.0.100.0-266]
> at org.apache.hadoop.hive.ql.metadata.Hive$4.call(Hive.java:2490)
> ~[hive-exec-3.1.0.3.0.100.0-266.jar:3.1.0.3.0.100.0-266]
> at org.apache.hadoop.hive.ql.metadata.Hive$4.call(Hive.java:2481)
> ~[hive-exec-3.1.0.3.0.100.0-266.jar:3.1.0.3.0.100.0-266]
> at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[?:1.8.0_181]
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> ~[?:1.8.0_181]
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> ~[?:1.8.0_181]
> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_181]{code}
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Assigned] (HIVE-21223) CachedStore returns null partition when partition does not exist
[
https://issues.apache.org/jira/browse/HIVE-21223?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Prasanth Jayachandran reassigned HIVE-21223:
Assignee: Prasanth Jayachandran
> CachedStore returns null partition when partition does not exist
>
>
> Key: HIVE-21223
> URL: https://issues.apache.org/jira/browse/HIVE-21223
> Project: Hive
> Issue Type: Bug
> Components: Metastore
>Affects Versions: 4.0.0, 3.2.0
>Reporter: Prasanth Jayachandran
>Assignee: Prasanth Jayachandran
>Priority: Major
> Attachments: HIVE-21223.1.patch
>
>
> CachedStore can return null partition for getPartitionWithAuth() when
> partition does not exist. null value serialization in thrift will break the
> connection. Instead if partition does not exist it should throw
> NoSuchObjectException.
> Clients will see this exception
> {code:java}
> org.apache.thrift.TApplicationException: get_partition_with_auth failed:
> unknown result
> at
> org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.recv_get_partition_with_auth(ThriftHiveMetastore.java:3017)
> ~[hive-exec-3.1.0.3.0.100.0-266.jar:3.1.0.3.0.100.0-266]
> at
> org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.get_partition_with_auth(ThriftHiveMetastore.java:2990)
> ~[hive-exec-3.1.0.3.0.100.0-266.jar:3.1.0.3.0.100.0-266]
> at
> org.apache.hadoop.hive.metastore.HiveMetaStoreClient.getPartitionWithAuthInfo(HiveMetaStoreClient.java:1679)
> ~[hive-exec-3.1.0.3.0.100.0-266.jar:3.1.0.3.0.100.0-266]
> at
> org.apache.hadoop.hive.metastore.HiveMetaStoreClient.getPartitionWithAuthInfo(HiveMetaStoreClient.java:1671)
> ~[hive-exec-3.1.0.3.0.100.0-266.jar:3.1.0.3.0.100.0-266]
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0_181]
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> ~[?:1.8.0_181]
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> ~[?:1.8.0_181]
> at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_181]
> at
> org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.invoke(RetryingMetaStoreClient.java:212)
> ~[hive-exec-3.1.0.3.0.100.0-266.jar:3.1.0.3.0.100.0-266]
> at com.sun.proxy.$Proxy36.getPartitionWithAuthInfo(Unknown Source) ~[?:?]
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0_181]
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> ~[?:1.8.0_181]
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> ~[?:1.8.0_181]
> at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_181]
> at
> org.apache.hadoop.hive.metastore.HiveMetaStoreClient$SynchronizedHandler.invoke(HiveMetaStoreClient.java:2976)
> ~[hive-exec-3.1.0.3.0.100.0-266.jar:3.1.0.3.0.100.0-266]
> at com.sun.proxy.$Proxy36.getPartitionWithAuthInfo(Unknown Source) ~[?:?]
> at
> org.apache.hadoop.hive.metastore.SynchronizedMetaStoreClient.getPartitionWithAuthInfo(SynchronizedMetaStoreClient.java:101)
> ~[hive-exec-3.1.0.3.0.100.0-266.jar:3.1.0.3.0.100.0-266]
> at org.apache.hadoop.hive.ql.metadata.Hive.getPartition(Hive.java:2870)
> ~[hive-exec-3.1.0.3.0.100.0-266.jar:3.1.0.3.0.100.0-266]
> at org.apache.hadoop.hive.ql.metadata.Hive.getPartition(Hive.java:2835)
> ~[hive-exec-3.1.0.3.0.100.0-266.jar:3.1.0.3.0.100.0-266]
> at org.apache.hadoop.hive.ql.metadata.Hive.loadPartition(Hive.java:1950)
> ~[hive-exec-3.1.0.3.0.100.0-266.jar:3.1.0.3.0.100.0-266]
> at org.apache.hadoop.hive.ql.metadata.Hive$4.call(Hive.java:2490)
> ~[hive-exec-3.1.0.3.0.100.0-266.jar:3.1.0.3.0.100.0-266]
> at org.apache.hadoop.hive.ql.metadata.Hive$4.call(Hive.java:2481)
> ~[hive-exec-3.1.0.3.0.100.0-266.jar:3.1.0.3.0.100.0-266]
> at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[?:1.8.0_181]
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> ~[?:1.8.0_181]
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> ~[?:1.8.0_181]
> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_181]{code}
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-21223) CachedStore returns null partition when partition does not exist
[
https://issues.apache.org/jira/browse/HIVE-21223?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Prasanth Jayachandran updated HIVE-21223:
-
Status: Patch Available (was: Open)
> CachedStore returns null partition when partition does not exist
>
>
> Key: HIVE-21223
> URL: https://issues.apache.org/jira/browse/HIVE-21223
> Project: Hive
> Issue Type: Bug
> Components: Metastore
>Affects Versions: 4.0.0, 3.2.0
>Reporter: Prasanth Jayachandran
>Priority: Major
> Attachments: HIVE-21223.1.patch
>
>
> CachedStore can return null partition for getPartitionWithAuth() when
> partition does not exist. null value serialization in thrift will break the
> connection. Instead if partition does not exist it should throw
> NoSuchObjectException.
> Clients will see this exception
> {code:java}
> org.apache.thrift.TApplicationException: get_partition_with_auth failed:
> unknown result
> at
> org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.recv_get_partition_with_auth(ThriftHiveMetastore.java:3017)
> ~[hive-exec-3.1.0.3.0.100.0-266.jar:3.1.0.3.0.100.0-266]
> at
> org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.get_partition_with_auth(ThriftHiveMetastore.java:2990)
> ~[hive-exec-3.1.0.3.0.100.0-266.jar:3.1.0.3.0.100.0-266]
> at
> org.apache.hadoop.hive.metastore.HiveMetaStoreClient.getPartitionWithAuthInfo(HiveMetaStoreClient.java:1679)
> ~[hive-exec-3.1.0.3.0.100.0-266.jar:3.1.0.3.0.100.0-266]
> at
> org.apache.hadoop.hive.metastore.HiveMetaStoreClient.getPartitionWithAuthInfo(HiveMetaStoreClient.java:1671)
> ~[hive-exec-3.1.0.3.0.100.0-266.jar:3.1.0.3.0.100.0-266]
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0_181]
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> ~[?:1.8.0_181]
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> ~[?:1.8.0_181]
> at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_181]
> at
> org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.invoke(RetryingMetaStoreClient.java:212)
> ~[hive-exec-3.1.0.3.0.100.0-266.jar:3.1.0.3.0.100.0-266]
> at com.sun.proxy.$Proxy36.getPartitionWithAuthInfo(Unknown Source) ~[?:?]
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0_181]
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> ~[?:1.8.0_181]
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> ~[?:1.8.0_181]
> at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_181]
> at
> org.apache.hadoop.hive.metastore.HiveMetaStoreClient$SynchronizedHandler.invoke(HiveMetaStoreClient.java:2976)
> ~[hive-exec-3.1.0.3.0.100.0-266.jar:3.1.0.3.0.100.0-266]
> at com.sun.proxy.$Proxy36.getPartitionWithAuthInfo(Unknown Source) ~[?:?]
> at
> org.apache.hadoop.hive.metastore.SynchronizedMetaStoreClient.getPartitionWithAuthInfo(SynchronizedMetaStoreClient.java:101)
> ~[hive-exec-3.1.0.3.0.100.0-266.jar:3.1.0.3.0.100.0-266]
> at org.apache.hadoop.hive.ql.metadata.Hive.getPartition(Hive.java:2870)
> ~[hive-exec-3.1.0.3.0.100.0-266.jar:3.1.0.3.0.100.0-266]
> at org.apache.hadoop.hive.ql.metadata.Hive.getPartition(Hive.java:2835)
> ~[hive-exec-3.1.0.3.0.100.0-266.jar:3.1.0.3.0.100.0-266]
> at org.apache.hadoop.hive.ql.metadata.Hive.loadPartition(Hive.java:1950)
> ~[hive-exec-3.1.0.3.0.100.0-266.jar:3.1.0.3.0.100.0-266]
> at org.apache.hadoop.hive.ql.metadata.Hive$4.call(Hive.java:2490)
> ~[hive-exec-3.1.0.3.0.100.0-266.jar:3.1.0.3.0.100.0-266]
> at org.apache.hadoop.hive.ql.metadata.Hive$4.call(Hive.java:2481)
> ~[hive-exec-3.1.0.3.0.100.0-266.jar:3.1.0.3.0.100.0-266]
> at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[?:1.8.0_181]
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> ~[?:1.8.0_181]
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> ~[?:1.8.0_181]
> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_181]{code}
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-21235) LLAP: make the name of log4j2 properties file configurable
[ https://issues.apache.org/jira/browse/HIVE-21235?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Prasanth Jayachandran updated HIVE-21235: - Status: Patch Available (was: Open) > LLAP: make the name of log4j2 properties file configurable > -- > > Key: HIVE-21235 > URL: https://issues.apache.org/jira/browse/HIVE-21235 > Project: Hive > Issue Type: Bug > Components: llap >Affects Versions: 4.0.0 >Reporter: Prasanth Jayachandran >Assignee: Prasanth Jayachandran >Priority: Major > Attachments: HIVE-21235.1.patch > > > For llap daemon, the name of llap-daemon-log4j2.properties is fixed. If a > conf dir and jar contain the same filename, it will mess up log4j2 > initialization. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-21235) LLAP: make the name of log4j2 properties file configurable
[ https://issues.apache.org/jira/browse/HIVE-21235?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Prasanth Jayachandran updated HIVE-21235: - Attachment: HIVE-21235.1.patch > LLAP: make the name of log4j2 properties file configurable > -- > > Key: HIVE-21235 > URL: https://issues.apache.org/jira/browse/HIVE-21235 > Project: Hive > Issue Type: Bug > Components: llap >Affects Versions: 4.0.0 >Reporter: Prasanth Jayachandran >Assignee: Prasanth Jayachandran >Priority: Major > Attachments: HIVE-21235.1.patch > > > For llap daemon, the name of llap-daemon-log4j2.properties is fixed. If a > conf dir and jar contain the same filename, it will mess up log4j2 > initialization. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (HIVE-21235) LLAP: make the name of log4j2 properties file configurable
[ https://issues.apache.org/jira/browse/HIVE-21235?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Prasanth Jayachandran reassigned HIVE-21235: > LLAP: make the name of log4j2 properties file configurable > -- > > Key: HIVE-21235 > URL: https://issues.apache.org/jira/browse/HIVE-21235 > Project: Hive > Issue Type: Bug > Components: llap >Affects Versions: 4.0.0 >Reporter: Prasanth Jayachandran >Assignee: Prasanth Jayachandran >Priority: Major > > For llap daemon, the name of llap-daemon-log4j2.properties is fixed. If a > conf dir and jar contain the same filename, it will mess up log4j2 > initialization. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
