[jira] [Commented] (HIVE-13086) LLAP: Programmatically initialize log4j2 to print out the properties location
[
https://issues.apache.org/jira/browse/HIVE-13086?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15153843#comment-15153843
]
Gopal V commented on HIVE-13086:
LGTM - +1, tests pending.
{code}
2016-02-19T02:17:27,792 INFO [main[]]: impl.LlapDaemon
(LlapDaemon.java:initializeLogging(219)) - LLAP daemon logging initialized from
file:/grid/4/hadoop/yarn/local/usercache/gopal/appcache/application_1455662455106_0226/container_e14_1455662455106_0226_01_02/app/install/conf/llap-daemon-log4j2.properties
in 22 ms
{code}
> LLAP: Programmatically initialize log4j2 to print out the properties location
> -
>
> Key: HIVE-13086
> URL: https://issues.apache.org/jira/browse/HIVE-13086
> Project: Hive
> Issue Type: Bug
>Affects Versions: 2.1.0
>Reporter: Prasanth Jayachandran
>Assignee: Prasanth Jayachandran
> Attachments: HIVE-13086.1.patch, HIVE-13086.2.patch
>
>
> In some cases, llap daemon gets initialized with different log4j2.properties
> than the expected llap-daemon-log4j2.properties. It will be easier if
> programmatically configure log4j2 so that we can print out the location of
> properties file that is used for initialization.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-13093) hive metastore does not exit on start failure
[ https://issues.apache.org/jira/browse/HIVE-13093?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15153833#comment-15153833 ] Thejas M Nair commented on HIVE-13093: -- [~daijy] [~alangates] Can you please review ? > hive metastore does not exit on start failure > - > > Key: HIVE-13093 > URL: https://issues.apache.org/jira/browse/HIVE-13093 > Project: Hive > Issue Type: Bug > Components: Metastore >Affects Versions: 0.13.1, 1.0.0, 1.1.1, 1.2.1 >Reporter: Thejas M Nair >Assignee: Thejas M Nair > Attachments: HIVE-13093.1.patch > > > If metastore startup fails for some reason, such as not being able to access > the database, it fails to exit. Instead the process continues to be up in a > bad state. > This is happening because of a non daemon thread. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13093) hive metastore does not exit on start failure
[ https://issues.apache.org/jira/browse/HIVE-13093?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thejas M Nair updated HIVE-13093: - Status: Patch Available (was: Open) > hive metastore does not exit on start failure > - > > Key: HIVE-13093 > URL: https://issues.apache.org/jira/browse/HIVE-13093 > Project: Hive > Issue Type: Bug > Components: Metastore >Affects Versions: 1.2.1, 1.1.1, 1.0.0, 0.13.1 >Reporter: Thejas M Nair >Assignee: Thejas M Nair > Attachments: HIVE-13093.1.patch > > > If metastore startup fails for some reason, such as not being able to access > the database, it fails to exit. Instead the process continues to be up in a > bad state. > This is happening because of a non daemon thread. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13093) hive metastore does not exit on start failure
[ https://issues.apache.org/jira/browse/HIVE-13093?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thejas M Nair updated HIVE-13093: - Attachment: HIVE-13093.1.patch > hive metastore does not exit on start failure > - > > Key: HIVE-13093 > URL: https://issues.apache.org/jira/browse/HIVE-13093 > Project: Hive > Issue Type: Bug > Components: Metastore >Affects Versions: 0.13.1, 1.0.0, 1.1.1, 1.2.1 >Reporter: Thejas M Nair >Assignee: Thejas M Nair > Attachments: HIVE-13093.1.patch > > > If metastore startup fails for some reason, such as not being able to access > the database, it fails to exit. Instead the process continues to be up in a > bad state. > This is happening because of a non daemon thread. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13093) hive metastore does not exit on start failure
[
https://issues.apache.org/jira/browse/HIVE-13093?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15153828#comment-15153828
]
Thejas M Nair commented on HIVE-13093:
--
The jstack of such a metastore is like this. Note the thread with
HiveMetaStore.java in stack trace -
{code}
"Attach Listener" #18 daemon prio=9 os_prio=0 tid=0x7fa749d4f000 nid=0x7e7a
runnable [0x]
java.lang.Thread.State: RUNNABLE
"DestroyJavaVM" #17 prio=5 os_prio=0 tid=0x7fa74801b000 nid=0x7c1c waiting
on condition [0x]
java.lang.Thread.State: RUNNABLE
"org.apache.hadoop.hive.common.JvmPauseMonitor$Monitor@77c43175" #16 daemon
prio=5 os_prio=0 tid=0x7fa73400e000 nid=0x7c3f waiting on condition
[0x7fa732d7b000]
java.lang.Thread.State: TIMED_WAITING (sleeping)
at java.lang.Thread.sleep(Native Method)
at
org.apache.hadoop.hive.common.JvmPauseMonitor$Monitor.run(JvmPauseMonitor.java:180)
at java.lang.Thread.run(Thread.java:745)
"Thread-5" #15 prio=5 os_prio=0 tid=0x7fa749e9d800 nid=0x7c3e waiting on
condition [0x7fa73328f000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0xde718150> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
at
org.apache.hadoop.hive.metastore.HiveMetaStore$5.run(HiveMetaStore.java:6155)
"AsyncLogger-1" #9 daemon prio=5 os_prio=0 tid=0x7fa749c77000 nid=0x7c27
waiting on condition [0x7fa7335ee000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0xde7896e8> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
at
com.lmax.disruptor.BlockingWaitStrategy.waitFor(BlockingWaitStrategy.java:45)
at
com.lmax.disruptor.ProcessingSequenceBarrier.waitFor(ProcessingSequenceBarrier.java:55)
at
com.lmax.disruptor.BatchEventProcessor.run(BatchEventProcessor.java:123)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
"Service Thread" #7 daemon prio=9 os_prio=0 tid=0x7fa7480ce000 nid=0x7c25
runnable [0x]
java.lang.Thread.State: RUNNABLE
"C1 CompilerThread1" #6 daemon prio=9 os_prio=0 tid=0x7fa7480be800
nid=0x7c24 waiting on condition [0x]
java.lang.Thread.State: RUNNABLE
"C2 CompilerThread0" #5 daemon prio=9 os_prio=0 tid=0x7fa7480bb800
nid=0x7c23 waiting on condition [0x]
java.lang.Thread.State: RUNNABLE
"Signal Dispatcher" #4 daemon prio=9 os_prio=0 tid=0x7fa7480b9000
nid=0x7c22 runnable [0x]
java.lang.Thread.State: RUNNABLE
"Finalizer" #3 daemon prio=8 os_prio=0 tid=0x7fa74808c000 nid=0x7c21 in
Object.wait() [0x7fa74caea000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:143)
- locked <0xdec7a100> (a java.lang.ref.ReferenceQueue$Lock)
at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:164)
at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:209)
"Reference Handler" #2 daemon prio=10 os_prio=0 tid=0x7fa74808a000
nid=0x7c20 in Object.wait() [0x7fa74cbeb000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
at java.lang.Object.wait(Object.java:502)
at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:157)
- locked <0xdec7a140> (a java.lang.ref.Reference$Lock)
"VM Thread" os_prio=0 tid=0x7fa748084800 nid=0x7c1f runnable
"GC task thread#0 (ParallelGC)" os_prio=0 tid=0x7fa74803 nid=0x7c1d
runnable
"GC task thread#1 (ParallelGC)" os_prio=0 tid=0x7fa748032000 nid=0x7c1e
runnable
"VM Periodic Task Thread" os_prio=0 tid=0x7fa7480d0800 nid=0x7c26 waiting
on condition
JNI global references: 314
{code}
> hive metastore does not exit on start failure
> -
>
> Key: HIVE-13093
> URL: https://issues.apache.org/jira/browse/HIVE-13093
> Project: Hive
> Issue Type: Bug
> Components: Metastore
>Affects Versions: 0.13.1, 1.0.0, 1.1.1,
[jira] [Updated] (HIVE-13077) LLAP: Scrub daemon-site.xml from client configs
[
https://issues.apache.org/jira/browse/HIVE-13077?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Gopal V updated HIVE-13077:
---
Resolution: Fixed
Fix Version/s: 2.1.0
Status: Resolved (was: Patch Available)
> LLAP: Scrub daemon-site.xml from client configs
> ---
>
> Key: HIVE-13077
> URL: https://issues.apache.org/jira/browse/HIVE-13077
> Project: Hive
> Issue Type: Bug
> Components: llap, Tez
>Affects Versions: 2.1.0
>Reporter: Gopal V
>Assignee: Gopal V
> Fix For: 2.1.0
>
> Attachments: HIVE-13077.1.patch
>
>
> {code}
> if (llapMode) {
> // add configs for llap-daemon-site.xml + localize llap jars
> // they cannot be referred to directly as it would be a circular
> depedency
> conf.addResource("llap-daemon-site.xml");
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-13079) LLAP: Allow reading log4j properties from default JAR resources
[ https://issues.apache.org/jira/browse/HIVE-13079?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gopal V updated HIVE-13079: --- Resolution: Fixed Fix Version/s: 2.1.0 Status: Resolved (was: Patch Available) > LLAP: Allow reading log4j properties from default JAR resources > --- > > Key: HIVE-13079 > URL: https://issues.apache.org/jira/browse/HIVE-13079 > Project: Hive > Issue Type: Bug > Components: llap >Affects Versions: 2.1.0 >Reporter: Gopal V >Assignee: Gopal V > Fix For: 2.1.0 > > Attachments: HIVE-13079.1.patch > > > If the log4j2 configuration is not overriden by the user, the Slider pkg > creation fails since the config is generated from a URL. > Allow for the .properties file to be created from default JAR resources if > user provides no overrides. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13082) Enable constant propagation optimization in query with left semi join
[
https://issues.apache.org/jira/browse/HIVE-13082?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15153798#comment-15153798
]
Hive QA commented on HIVE-13082:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12788387/HIVE-13082.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 14 failed/errored test(s), 9814 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_partition_coltype_literals
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_semijoin2
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_semijoin4
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_vector_mapjoin_reduce
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_constprog_partitioner
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_explainuser_1
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_stats_only_null
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_vector_mapjoin_reduce
org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_constprog_partitioner
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_authorization_uri_import
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_stats_only_null
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_vector_mapjoin_reduce
org.apache.hadoop.hive.ql.TestTxnCommands2.testInitiatorWithMultipleFailedCompactions
org.apache.hive.jdbc.TestSSL.testSSLVersion
{noformat}
Test results:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/7025/testReport
Console output:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/7025/console
Test logs:
http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-7025/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 14 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12788387 - PreCommit-HIVE-TRUNK-Build
> Enable constant propagation optimization in query with left semi join
> -
>
> Key: HIVE-13082
> URL: https://issues.apache.org/jira/browse/HIVE-13082
> Project: Hive
> Issue Type: Bug
> Components: Query Processor
>Affects Versions: 2.0.0
>Reporter: Chaoyu Tang
>Assignee: Chaoyu Tang
> Attachments: HIVE-13082.patch
>
>
> Currently constant folding is only allowed for inner or unique join, I think
> it is also applicable and allowed for left semi join. Otherwise the query
> like following having multiple joins with left semi joins will fail:
> {code}
> select table1.id, table1.val, table2.val2 from table1 inner join table2 on
> table1.val = 't1val01' and table1.id = table2.id left semi join table3 on
> table1.dimid = table3.id;
> {code}
> with errors:
> {code}
> java.lang.Exception: java.lang.RuntimeException: Error in configuring object
> at
> org.apache.hadoop.mapred.LocalJobRunner$Job.runTasks(LocalJobRunner.java:462)
> ~[hadoop-mapreduce-client-common-2.6.0.jar:?]
> at
> org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:522)
> [hadoop-mapreduce-client-common-2.6.0.jar:?]
> Caused by: java.lang.RuntimeException: Error in configuring object
> at
> org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:109)
> ~[hadoop-common-2.6.0.jar:?]
> at
> org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:75)
> ~[hadoop-common-2.6.0.jar:?]
> at
> org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:133)
> ~[hadoop-common-2.6.0.jar:?]
> at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:446)
> ~[hadoop-mapreduce-client-core-2.6.0.jar:?]
> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:343)
> ~[hadoop-mapreduce-client-core-2.6.0.jar:?]
> at
> org.apache.hadoop.mapred.LocalJobRunner$Job$MapTaskRunnable.run(LocalJobRunner.java:243)
> ~[hadoop-mapreduce-client-common-2.6.0.jar:?]
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
> ~[?:1.7.0_45]
> at java.util.concurrent.FutureTask.run(FutureTask.java:262)
> ~[?:1.7.0_45]
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> ~[?:1.7.0_45]
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> ~[?:1.7.0_45]
> at java.lang.Thread.run(Thread.java:744) ~[?:1.7.0_45]
> ...
> Caused by:
[jira] [Commented] (HIVE-13077) LLAP: Scrub daemon-site.xml from client configs
[
https://issues.apache.org/jira/browse/HIVE-13077?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15153726#comment-15153726
]
Hive QA commented on HIVE-13077:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12788363/HIVE-13077.1.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 8 failed/errored test(s), 9796 tests executed
*Failed tests:*
{noformat}
TestSparkCliDriver-timestamp_lazy.q-bucketsortoptimize_insert_4.q-date_udf.q-and-12-more
- did not produce a TEST-*.xml file
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_ivyDownload
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_partition_coltype_literals
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_stats_only_null
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_authorization_uri_import
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_stats_only_null
org.apache.hive.hcatalog.hbase.TestPigHBaseStorageHandler.org.apache.hive.hcatalog.hbase.TestPigHBaseStorageHandler
org.apache.hive.jdbc.TestSSL.testSSLVersion
{noformat}
Test results:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/7024/testReport
Console output:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/7024/console
Test logs:
http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-7024/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 8 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12788363 - PreCommit-HIVE-TRUNK-Build
> LLAP: Scrub daemon-site.xml from client configs
> ---
>
> Key: HIVE-13077
> URL: https://issues.apache.org/jira/browse/HIVE-13077
> Project: Hive
> Issue Type: Bug
> Components: llap, Tez
>Affects Versions: 2.1.0
>Reporter: Gopal V
>Assignee: Gopal V
> Attachments: HIVE-13077.1.patch
>
>
> {code}
> if (llapMode) {
> // add configs for llap-daemon-site.xml + localize llap jars
> // they cannot be referred to directly as it would be a circular
> depedency
> conf.addResource("llap-daemon-site.xml");
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Assigned] (HIVE-13091) Beeline logs the username and password to the history file for connect commands
[ https://issues.apache.org/jira/browse/HIVE-13091?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chinna Rao Lalam reassigned HIVE-13091: --- Assignee: Chinna Rao Lalam > Beeline logs the username and password to the history file for connect > commands > --- > > Key: HIVE-13091 > URL: https://issues.apache.org/jira/browse/HIVE-13091 > Project: Hive > Issue Type: Bug >Affects Versions: 1.2.0 >Reporter: Ravi Prakash >Assignee: Chinna Rao Lalam > > [~farisa] and [~tthompso] found that the beeline client also logs the > username and password from a connect command into the beeline history file > (Usually found at ~/.beeline/history). We should not be logging the password > anywhere. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13083) Writing HiveDecimal to ORC can wrongly suppress present stream
[
https://issues.apache.org/jira/browse/HIVE-13083?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Prasanth Jayachandran updated HIVE-13083:
-
Attachment: HIVE-13083-branch-1.patch
> Writing HiveDecimal to ORC can wrongly suppress present stream
> --
>
> Key: HIVE-13083
> URL: https://issues.apache.org/jira/browse/HIVE-13083
> Project: Hive
> Issue Type: Bug
>Affects Versions: 0.13.0, 0.14.0, 1.0.0, 1.2.0, 1.1.0, 1.3.0, 2.0.0, 2.1.0
>Reporter: Yi Zhang
>Assignee: Prasanth Jayachandran
> Attachments: HIVE-13083-branch-1.patch, HIVE-13083.1.patch,
> HIVE-13083.2.patch
>
>
> HIVE-3976 can cause ORC file to be unreadable. The changes introduced in
> HIVE-3976 for DecimalTreeWriter can create null values after updating the
> isPresent stream.
> https://github.com/apache/hive/blob/branch-0.13/ql/src/java/org/apache/hadoop/hive/ql/io/orc/WriterImpl.java#L1337
> As result of the above return statement, isPresent stream state can become
> wrong. The isPresent stream thinks all values are non-null and hence
> suppressed. But the data stream will be of 0 length. When reading such files
> we will get the following exception
> {code}
> Caused by: java.io.EOFException: Reading BigInteger past EOF from compressed
> stream Stream for column 3 kind DATA position: 0 length: 0 range: 0 offset: 0
> limit: 0
> at
> org.apache.hadoop.hive.ql.io.orc.SerializationUtils.readBigInteger(SerializationUtils.java:176)
> at
> org.apache.hadoop.hive.ql.io.orc.TreeReaderFactory$DecimalTreeReader.next(TreeReaderFactory.java:1264)
> at
> org.apache.hadoop.hive.ql.io.orc.TreeReaderFactory$StructTreeReader.next(TreeReaderFactory.java:2004)
> at
> org.apache.hadoop.hive.ql.io.orc.RecordReaderImpl.next(RecordReaderImpl.java:1039)
> ... 24 more
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-13083) Writing HiveDecimal to ORC can wrongly suppress present stream
[
https://issues.apache.org/jira/browse/HIVE-13083?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Prasanth Jayachandran updated HIVE-13083:
-
Attachment: (was: HIVE-13083-branch-1.patch)
> Writing HiveDecimal to ORC can wrongly suppress present stream
> --
>
> Key: HIVE-13083
> URL: https://issues.apache.org/jira/browse/HIVE-13083
> Project: Hive
> Issue Type: Bug
>Affects Versions: 0.13.0, 0.14.0, 1.0.0, 1.2.0, 1.1.0, 1.3.0, 2.0.0, 2.1.0
>Reporter: Yi Zhang
>Assignee: Prasanth Jayachandran
> Attachments: HIVE-13083-branch-1.patch, HIVE-13083.1.patch,
> HIVE-13083.2.patch
>
>
> HIVE-3976 can cause ORC file to be unreadable. The changes introduced in
> HIVE-3976 for DecimalTreeWriter can create null values after updating the
> isPresent stream.
> https://github.com/apache/hive/blob/branch-0.13/ql/src/java/org/apache/hadoop/hive/ql/io/orc/WriterImpl.java#L1337
> As result of the above return statement, isPresent stream state can become
> wrong. The isPresent stream thinks all values are non-null and hence
> suppressed. But the data stream will be of 0 length. When reading such files
> we will get the following exception
> {code}
> Caused by: java.io.EOFException: Reading BigInteger past EOF from compressed
> stream Stream for column 3 kind DATA position: 0 length: 0 range: 0 offset: 0
> limit: 0
> at
> org.apache.hadoop.hive.ql.io.orc.SerializationUtils.readBigInteger(SerializationUtils.java:176)
> at
> org.apache.hadoop.hive.ql.io.orc.TreeReaderFactory$DecimalTreeReader.next(TreeReaderFactory.java:1264)
> at
> org.apache.hadoop.hive.ql.io.orc.TreeReaderFactory$StructTreeReader.next(TreeReaderFactory.java:2004)
> at
> org.apache.hadoop.hive.ql.io.orc.RecordReaderImpl.next(RecordReaderImpl.java:1039)
> ... 24 more
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-13092) Vectorized java.lang.ClassCastException: org.apache.hadoop.hive.serde2.typeinfo.ListTypeInfo cannot be cast to org.apache.hadoop.hive.serde2.typeinfo.PrimitiveTypeInfo
[
https://issues.apache.org/jira/browse/HIVE-13092?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Matt McCline updated HIVE-13092:
Attachment: HIVE-13092.01.patch
> Vectorized java.lang.ClassCastException:
> org.apache.hadoop.hive.serde2.typeinfo.ListTypeInfo cannot be cast to
> org.apache.hadoop.hive.serde2.typeinfo.PrimitiveTypeInfo
> ---
>
> Key: HIVE-13092
> URL: https://issues.apache.org/jira/browse/HIVE-13092
> Project: Hive
> Issue Type: Bug
> Components: Hive
>Affects Versions: 1.3.0
>Reporter: Matt McCline
>Assignee: Matt McCline
>Priority: Critical
> Fix For: 1.3.0
>
> Attachments: HIVE-13092.01.patch
>
>
> When table being queried has a COMPLEX data type(s) but those columns are not
> referenced and they are not excluded by the include columns, then we get this
> runtime exception:
> {code}
> Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime
> Error while processing row
> at
> org.apache.hadoop.hive.ql.exec.vector.VectorMapOperator.process(VectorMapOperator.java:52)
> at
> org.apache.hadoop.hive.ql.exec.tez.MapRecordSource.processRow(MapRecordSource.java:83)
> ... 17 more
> Caused by: java.lang.ClassCastException:
> org.apache.hadoop.hive.serde2.typeinfo.ListTypeInfo cannot be cast to
> org.apache.hadoop.hive.serde2.typeinfo.PrimitiveTypeInfo
> at
> org.apache.hadoop.hive.ql.exec.vector.VectorExtractRow.init(VectorExtractRow.java:678)
> at
> org.apache.hadoop.hive.ql.exec.vector.VectorReduceSinkOperator.process(VectorReduceSinkOperator.java:82)
> at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:838)
> at
> org.apache.hadoop.hive.ql.exec.vector.VectorFilterOperator.process(VectorFilterOperator.java:117)
> at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:838)
> at
> org.apache.hadoop.hive.ql.exec.TableScanOperator.process(TableScanOperator.java:97)
> at
> org.apache.hadoop.hive.ql.exec.MapOperator$MapOpCtx.forward(MapOperator.java:164)
> at
> org.apache.hadoop.hive.ql.exec.vector.VectorMapOperator.process(VectorMapOperator.java:45)
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Comment Edited] (HIVE-13083) Writing HiveDecimal to ORC can wrongly suppress present stream
[
https://issues.apache.org/jira/browse/HIVE-13083?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15153673#comment-15153673
]
Prasanth Jayachandran edited comment on HIVE-13083 at 2/19/16 4:01 AM:
---
Updated some golden files after writer version change
was (Author: prasanth_j):
Updated some golden files are writer version change
> Writing HiveDecimal to ORC can wrongly suppress present stream
> --
>
> Key: HIVE-13083
> URL: https://issues.apache.org/jira/browse/HIVE-13083
> Project: Hive
> Issue Type: Bug
>Affects Versions: 0.13.0, 0.14.0, 1.0.0, 1.2.0, 1.1.0, 1.3.0, 2.0.0, 2.1.0
>Reporter: Yi Zhang
>Assignee: Prasanth Jayachandran
> Attachments: HIVE-13083-branch-1.patch, HIVE-13083.1.patch,
> HIVE-13083.2.patch, HIVE-13083.3.patch
>
>
> HIVE-3976 can cause ORC file to be unreadable. The changes introduced in
> HIVE-3976 for DecimalTreeWriter can create null values after updating the
> isPresent stream.
> https://github.com/apache/hive/blob/branch-0.13/ql/src/java/org/apache/hadoop/hive/ql/io/orc/WriterImpl.java#L1337
> As result of the above return statement, isPresent stream state can become
> wrong. The isPresent stream thinks all values are non-null and hence
> suppressed. But the data stream will be of 0 length. When reading such files
> we will get the following exception
> {code}
> Caused by: java.io.EOFException: Reading BigInteger past EOF from compressed
> stream Stream for column 3 kind DATA position: 0 length: 0 range: 0 offset: 0
> limit: 0
> at
> org.apache.hadoop.hive.ql.io.orc.SerializationUtils.readBigInteger(SerializationUtils.java:176)
> at
> org.apache.hadoop.hive.ql.io.orc.TreeReaderFactory$DecimalTreeReader.next(TreeReaderFactory.java:1264)
> at
> org.apache.hadoop.hive.ql.io.orc.TreeReaderFactory$StructTreeReader.next(TreeReaderFactory.java:2004)
> at
> org.apache.hadoop.hive.ql.io.orc.RecordReaderImpl.next(RecordReaderImpl.java:1039)
> ... 24 more
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-13083) Writing HiveDecimal to ORC can wrongly suppress present stream
[
https://issues.apache.org/jira/browse/HIVE-13083?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Prasanth Jayachandran updated HIVE-13083:
-
Attachment: HIVE-13083.3.patch
Updated some golden files are writer version change
> Writing HiveDecimal to ORC can wrongly suppress present stream
> --
>
> Key: HIVE-13083
> URL: https://issues.apache.org/jira/browse/HIVE-13083
> Project: Hive
> Issue Type: Bug
>Affects Versions: 0.13.0, 0.14.0, 1.0.0, 1.2.0, 1.1.0, 1.3.0, 2.0.0, 2.1.0
>Reporter: Yi Zhang
>Assignee: Prasanth Jayachandran
> Attachments: HIVE-13083-branch-1.patch, HIVE-13083.1.patch,
> HIVE-13083.2.patch, HIVE-13083.3.patch
>
>
> HIVE-3976 can cause ORC file to be unreadable. The changes introduced in
> HIVE-3976 for DecimalTreeWriter can create null values after updating the
> isPresent stream.
> https://github.com/apache/hive/blob/branch-0.13/ql/src/java/org/apache/hadoop/hive/ql/io/orc/WriterImpl.java#L1337
> As result of the above return statement, isPresent stream state can become
> wrong. The isPresent stream thinks all values are non-null and hence
> suppressed. But the data stream will be of 0 length. When reading such files
> we will get the following exception
> {code}
> Caused by: java.io.EOFException: Reading BigInteger past EOF from compressed
> stream Stream for column 3 kind DATA position: 0 length: 0 range: 0 offset: 0
> limit: 0
> at
> org.apache.hadoop.hive.ql.io.orc.SerializationUtils.readBigInteger(SerializationUtils.java:176)
> at
> org.apache.hadoop.hive.ql.io.orc.TreeReaderFactory$DecimalTreeReader.next(TreeReaderFactory.java:1264)
> at
> org.apache.hadoop.hive.ql.io.orc.TreeReaderFactory$StructTreeReader.next(TreeReaderFactory.java:2004)
> at
> org.apache.hadoop.hive.ql.io.orc.RecordReaderImpl.next(RecordReaderImpl.java:1039)
> ... 24 more
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Comment Edited] (HIVE-13075) Metastore shuts down when no delegation token is found in ZooKeeper
[
https://issues.apache.org/jira/browse/HIVE-13075?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15153626#comment-15153626
]
Thejas M Nair edited comment on HIVE-13075 at 2/19/16 3:03 AM:
---
[~wikp] Thanks for reporting this and the diagnosis!
I should have checked for this first. Lets track it in the new jira with the
patch.
was (Author: thejas):
[~wikp] Thanks for reporting this and the diagnosis!
I should have checked this first. Lets track it in the new jira with the patch.
> Metastore shuts down when no delegation token is found in ZooKeeper
> ---
>
> Key: HIVE-13075
> URL: https://issues.apache.org/jira/browse/HIVE-13075
> Project: Hive
> Issue Type: Bug
>Affects Versions: 1.3.0, 1.2.1
>Reporter: Piotr Wikieł
>Priority: Minor
>
> {{ZooKeeperTokenStore}} looks [as
> follows|https://github.com/apache/hive/blob/branch-1.2/shims/common/src/main/java/org/apache/hadoop/hive/thrift/ZooKeeperTokenStore.java#L397]:
> {code:java}
> @Override
> public DelegationTokenInformation getToken(DelegationTokenIdentifier
> tokenIdentifier) {
> byte[] tokenBytes = zkGetData(getTokenPath(tokenIdentifier));
> try {
> return
> HiveDelegationTokenSupport.decodeDelegationTokenInformation(tokenBytes);
> } catch (Exception ex) {
> throw new TokenStoreException("Failed to decode token", ex);
> }
> }
> {code}
> which is slightly different from [DBTokenStore
> implementation|https://github.com/apache/hive/blob/branch-1.2/shims/common/src/main/java/org/apache/hadoop/hive/thrift/DBTokenStore.java#L85]
> that is protected against {{tokenBytes==null}} because nullable
> {{tokenBytes}} causes NPE to be thrown in
> [HiveDelegationTokenSupport#decodeDelegationTokenInformation|https://github.com/apache/hive/blob/branch-1.2/shims/common/src/main/java/org/apache/hadoop/security/token/delegation/HiveDelegationTokenSupport.java#L51]
> Furthermore, NPE thrown here causes
> [TokenStoreDelegationTokenSecretManager.ExpiredTokenRemover|https://github.com/apache/hive/blob/branch-1.2/shims/common/src/main/java/org/apache/hadoop/hive/thrift/TokenStoreDelegationTokenSecretManager.java#L333]
> to catch it and exits MetaStore.
> {{null}} from
> {{[zkGetData()|https://github.com/apache/hive/blob/branch-1.2/shims/common/src/main/java/org/apache/hadoop/hive/thrift/ZooKeeperTokenStore.java#L284]}}
> is possible during ZooKeeper failure or (and that was our case) when another
> metastore instance removes tokens during {{ExpiredTokenRemover}} run. There
> were two solutions of this problem:
> * distributed lock in ZooKeeper acquired during one metastore instance's
> ExpiredTokenRemover run,
> * simple null check
> I think null check is sufficient if it is in {{DBTokenStore}}.
> Patch will be attached.
> Sorry for an edit but I think worth mentioning is a fact that possible
> workaround for this issue is setting
> {{hive.cluster.delegation.key.update-interval}},
> {{hive.cluster.delegation.token.renew-interval}} and
> {{hive.cluster.delegation.token.max-lifetime}} to one year as described
> [here|https://community.cloudera.com/t5/Web-UI-Hue-Beeswax/Potential-misconfiguration-detected-Hue-Hive-Editor-HiveServer2/m-p/26117/highlight/true#M763].
> But in my opinion it is not an engineer-way of doing things ;)
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-13075) Metastore shuts down when no delegation token is found in ZooKeeper
[
https://issues.apache.org/jira/browse/HIVE-13075?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Thejas M Nair updated HIVE-13075:
-
Resolution: Duplicate
Status: Resolved (was: Patch Available)
> Metastore shuts down when no delegation token is found in ZooKeeper
> ---
>
> Key: HIVE-13075
> URL: https://issues.apache.org/jira/browse/HIVE-13075
> Project: Hive
> Issue Type: Bug
>Affects Versions: 1.3.0, 1.2.1
>Reporter: Piotr Wikieł
>Priority: Minor
>
> {{ZooKeeperTokenStore}} looks [as
> follows|https://github.com/apache/hive/blob/branch-1.2/shims/common/src/main/java/org/apache/hadoop/hive/thrift/ZooKeeperTokenStore.java#L397]:
> {code:java}
> @Override
> public DelegationTokenInformation getToken(DelegationTokenIdentifier
> tokenIdentifier) {
> byte[] tokenBytes = zkGetData(getTokenPath(tokenIdentifier));
> try {
> return
> HiveDelegationTokenSupport.decodeDelegationTokenInformation(tokenBytes);
> } catch (Exception ex) {
> throw new TokenStoreException("Failed to decode token", ex);
> }
> }
> {code}
> which is slightly different from [DBTokenStore
> implementation|https://github.com/apache/hive/blob/branch-1.2/shims/common/src/main/java/org/apache/hadoop/hive/thrift/DBTokenStore.java#L85]
> that is protected against {{tokenBytes==null}} because nullable
> {{tokenBytes}} causes NPE to be thrown in
> [HiveDelegationTokenSupport#decodeDelegationTokenInformation|https://github.com/apache/hive/blob/branch-1.2/shims/common/src/main/java/org/apache/hadoop/security/token/delegation/HiveDelegationTokenSupport.java#L51]
> Furthermore, NPE thrown here causes
> [TokenStoreDelegationTokenSecretManager.ExpiredTokenRemover|https://github.com/apache/hive/blob/branch-1.2/shims/common/src/main/java/org/apache/hadoop/hive/thrift/TokenStoreDelegationTokenSecretManager.java#L333]
> to catch it and exits MetaStore.
> {{null}} from
> {{[zkGetData()|https://github.com/apache/hive/blob/branch-1.2/shims/common/src/main/java/org/apache/hadoop/hive/thrift/ZooKeeperTokenStore.java#L284]}}
> is possible during ZooKeeper failure or (and that was our case) when another
> metastore instance removes tokens during {{ExpiredTokenRemover}} run. There
> were two solutions of this problem:
> * distributed lock in ZooKeeper acquired during one metastore instance's
> ExpiredTokenRemover run,
> * simple null check
> I think null check is sufficient if it is in {{DBTokenStore}}.
> Patch will be attached.
> Sorry for an edit but I think worth mentioning is a fact that possible
> workaround for this issue is setting
> {{hive.cluster.delegation.key.update-interval}},
> {{hive.cluster.delegation.token.renew-interval}} and
> {{hive.cluster.delegation.token.max-lifetime}} to one year as described
> [here|https://community.cloudera.com/t5/Web-UI-Hue-Beeswax/Potential-misconfiguration-detected-Hue-Hive-Editor-HiveServer2/m-p/26117/highlight/true#M763].
> But in my opinion it is not an engineer-way of doing things ;)
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-13090) Hive metastore crashes on NPE with ZooKeeperTokenStore
[
https://issues.apache.org/jira/browse/HIVE-13090?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15153623#comment-15153623
]
Thejas M Nair commented on HIVE-13090:
--
Yes, same as HIVE-13075 , linking it.
> Hive metastore crashes on NPE with ZooKeeperTokenStore
> --
>
> Key: HIVE-13090
> URL: https://issues.apache.org/jira/browse/HIVE-13090
> Project: Hive
> Issue Type: Bug
> Components: Metastore, Security
>Affects Versions: 1.0.0, 1.1.1, 1.2.1
>Reporter: Thejas M Nair
>Assignee: Thejas M Nair
> Attachments: HIVE-13090.1.patch
>
>
> Observed that hive metastore shutdown with NPE from ZookeeperTokenStore.
> {code}
> INFO [pool-5-thread-192]: metastore.HiveMetaStore
> (HiveMetaStore.java:logInfo(714)) - 191: Metastore shutdown complete.
> INFO [pool-5-thread-192]: HiveMetaStore.audit
> (HiveMetaStore.java:logAuditEvent(340)) - ugi=cvdpqap ip=/19.1.2.129
> cmd=Metastore shutdown complete.
> ERROR [Thread[Thread-6,5,main]]:
> thrift.TokenStoreDelegationTokenSecretManager
> (TokenStoreDelegationTokenSecretManager.java:run(331)) - ExpiredTokenRemover
> thread received unexpected exception.
> org.apache.hadoop.hive.thrift.DelegationTokenStore$TokenStoreException:
> Failed to decode token
> org.apache.hadoop.hive.thrift.DelegationTokenStore$TokenStoreException:
> Failed to decode token
> at
> org.apache.hadoop.hive.thrift.ZooKeeperTokenStore.getToken(ZooKeeperTokenStore.java:401)
> at
> org.apache.hadoop.hive.thrift.TokenStoreDelegationTokenSecretManager.removeExpiredTokens(TokenStoreDelegationTokenSecretManager.java:256)
> at
> org.apache.hadoop.hive.thrift.TokenStoreDelegationTokenSecretManager$ExpiredTokenRemover.run(TokenStoreDelegationTokenSecretManager.java:319)
> at java.lang.Thread.run(Thread.java:744)
> Caused by: java.lang.NullPointerException
> at java.io.ByteArrayInputStream.(ByteArrayInputStream.java:106)
> at
> org.apache.hadoop.security.token.delegation.HiveDelegationTokenSupport.decodeDelegationTokenInformation(HiveDelegationTokenSupport.java:53)
> at
> org.apache.hadoop.hive.thrift.ZooKeeperTokenStore.getToken(ZooKeeperTokenStore.java:399)
> ... 3 more
> INFO [Thread-3]: metastore.HiveMetaStore (HiveMetaStore.java:run(5639)) -
> Shutting down hive metastore.
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-13083) Writing HiveDecimal to ORC can wrongly suppress present stream
[
https://issues.apache.org/jira/browse/HIVE-13083?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Prasanth Jayachandran updated HIVE-13083:
-
Attachment: HIVE-13083.2.patch
Addressed [~gopalv]'s review comments. Also bumped up the writer version
> Writing HiveDecimal to ORC can wrongly suppress present stream
> --
>
> Key: HIVE-13083
> URL: https://issues.apache.org/jira/browse/HIVE-13083
> Project: Hive
> Issue Type: Bug
>Affects Versions: 0.13.0, 0.14.0, 1.0.0, 1.2.0, 1.1.0, 1.3.0, 2.0.0, 2.1.0
>Reporter: Yi Zhang
>Assignee: Prasanth Jayachandran
> Attachments: HIVE-13083-branch-1.patch, HIVE-13083.1.patch,
> HIVE-13083.2.patch
>
>
> HIVE-3976 can cause ORC file to be unreadable. The changes introduced in
> HIVE-3976 for DecimalTreeWriter can create null values after updating the
> isPresent stream.
> https://github.com/apache/hive/blob/branch-0.13/ql/src/java/org/apache/hadoop/hive/ql/io/orc/WriterImpl.java#L1337
> As result of the above return statement, isPresent stream state can become
> wrong. The isPresent stream thinks all values are non-null and hence
> suppressed. But the data stream will be of 0 length. When reading such files
> we will get the following exception
> {code}
> Caused by: java.io.EOFException: Reading BigInteger past EOF from compressed
> stream Stream for column 3 kind DATA position: 0 length: 0 range: 0 offset: 0
> limit: 0
> at
> org.apache.hadoop.hive.ql.io.orc.SerializationUtils.readBigInteger(SerializationUtils.java:176)
> at
> org.apache.hadoop.hive.ql.io.orc.TreeReaderFactory$DecimalTreeReader.next(TreeReaderFactory.java:1264)
> at
> org.apache.hadoop.hive.ql.io.orc.TreeReaderFactory$StructTreeReader.next(TreeReaderFactory.java:2004)
> at
> org.apache.hadoop.hive.ql.io.orc.RecordReaderImpl.next(RecordReaderImpl.java:1039)
> ... 24 more
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-4095) Add exchange partition in Hive
[ https://issues.apache.org/jira/browse/HIVE-4095?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15153595#comment-15153595 ] Andrew Sears commented on HIVE-4095: Updated the docs. > Add exchange partition in Hive > -- > > Key: HIVE-4095 > URL: https://issues.apache.org/jira/browse/HIVE-4095 > Project: Hive > Issue Type: New Feature > Components: Query Processor >Reporter: Namit Jain >Assignee: Dheeraj Kumar Singh > Fix For: 0.12.0 > > Attachments: HIVE-4095.D10155.1.patch, HIVE-4095.D10155.2.patch, > HIVE-4095.D10347.1.patch, HIVE-4095.part11.patch.txt, > HIVE-4095.part12.patch.txt, hive.4095.1.patch, hive.4095.refresh.patch, > hive.4095.svn.thrift.patch, hive.4095.svn.thrift.patch.refresh > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13040) Handle empty bucket creations more efficiently
[ https://issues.apache.org/jira/browse/HIVE-13040?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashutosh Chauhan updated HIVE-13040: Status: Open (was: Patch Available) > Handle empty bucket creations more efficiently > --- > > Key: HIVE-13040 > URL: https://issues.apache.org/jira/browse/HIVE-13040 > Project: Hive > Issue Type: Improvement > Components: Query Processor >Affects Versions: 2.0.0, 1.1.0, 1.2.0, 1.0.0 >Reporter: Ashutosh Chauhan >Assignee: Ashutosh Chauhan > Attachments: HIVE-13040.2.patch, HIVE-13040.3.patch, > HIVE-13040.4.patch, HIVE-13040.5.patch, HIVE-13040.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13040) Handle empty bucket creations more efficiently
[ https://issues.apache.org/jira/browse/HIVE-13040?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashutosh Chauhan updated HIVE-13040: Status: Patch Available (was: Open) > Handle empty bucket creations more efficiently > --- > > Key: HIVE-13040 > URL: https://issues.apache.org/jira/browse/HIVE-13040 > Project: Hive > Issue Type: Improvement > Components: Query Processor >Affects Versions: 2.0.0, 1.1.0, 1.2.0, 1.0.0 >Reporter: Ashutosh Chauhan >Assignee: Ashutosh Chauhan > Attachments: HIVE-13040.2.patch, HIVE-13040.3.patch, > HIVE-13040.4.patch, HIVE-13040.5.patch, HIVE-13040.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13040) Handle empty bucket creations more efficiently
[ https://issues.apache.org/jira/browse/HIVE-13040?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashutosh Chauhan updated HIVE-13040: Attachment: HIVE-13040.5.patch > Handle empty bucket creations more efficiently > --- > > Key: HIVE-13040 > URL: https://issues.apache.org/jira/browse/HIVE-13040 > Project: Hive > Issue Type: Improvement > Components: Query Processor >Affects Versions: 1.0.0, 1.2.0, 1.1.0, 2.0.0 >Reporter: Ashutosh Chauhan >Assignee: Ashutosh Chauhan > Attachments: HIVE-13040.2.patch, HIVE-13040.3.patch, > HIVE-13040.4.patch, HIVE-13040.5.patch, HIVE-13040.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13079) LLAP: Allow reading log4j properties from default JAR resources
[
https://issues.apache.org/jira/browse/HIVE-13079?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15153534#comment-15153534
]
Hive QA commented on HIVE-13079:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12788359/HIVE-13079.1.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 6 failed/errored test(s), 9798 tests executed
*Failed tests:*
{noformat}
TestSparkCliDriver-timestamp_lazy.q-bucketsortoptimize_insert_4.q-date_udf.q-and-12-more
- did not produce a TEST-*.xml file
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_partition_coltype_literals
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_stats_only_null
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_authorization_uri_import
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_stats_only_null
org.apache.hive.jdbc.TestSSL.testSSLVersion
{noformat}
Test results:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/7023/testReport
Console output:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/7023/console
Test logs:
http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-7023/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 6 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12788359 - PreCommit-HIVE-TRUNK-Build
> LLAP: Allow reading log4j properties from default JAR resources
> ---
>
> Key: HIVE-13079
> URL: https://issues.apache.org/jira/browse/HIVE-13079
> Project: Hive
> Issue Type: Bug
> Components: llap
>Affects Versions: 2.1.0
>Reporter: Gopal V
>Assignee: Gopal V
> Attachments: HIVE-13079.1.patch
>
>
> If the log4j2 configuration is not overriden by the user, the Slider pkg
> creation fails since the config is generated from a URL.
> Allow for the .properties file to be created from default JAR resources if
> user provides no overrides.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-12856) LLAP: update (add/remove) the UDFs available in LLAP when they are changed (refresh periodically)
[ https://issues.apache.org/jira/browse/HIVE-12856?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15153471#comment-15153471 ] Gunther Hagleitner commented on HIVE-12856: --- If you want to validate that HS2/metastore allows it, use signature? > LLAP: update (add/remove) the UDFs available in LLAP when they are changed > (refresh periodically) > - > > Key: HIVE-12856 > URL: https://issues.apache.org/jira/browse/HIVE-12856 > Project: Hive > Issue Type: Sub-task >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > Attachments: HIVE-12856.01.nogen.patch, HIVE-12856.01.patch, > HIVE-12856.02.nogen.patch, HIVE-12856.02.patch, HIVE-12856.nogen.patch, > HIVE-12856.patch > > > I don't think re-querying the functions is going to scale, and the sessions > obviously cannot notify all LLAP clusters of every change. We should add > global versioning to metastore functions to track changes, and then possibly > add a notification mechanism, potentially thru ZK to avoid overloading the > metastore itself. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-12856) LLAP: update (add/remove) the UDFs available in LLAP when they are changed (refresh periodically)
[ https://issues.apache.org/jira/browse/HIVE-12856?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15153452#comment-15153452 ] Sergey Shelukhin commented on HIVE-12856: - That defeats the purpose of not allowing UDFs in LLAP, if I understand it correctly - if every fragment can tell you what's in the whitelist then why have the whitelist at all. > LLAP: update (add/remove) the UDFs available in LLAP when they are changed > (refresh periodically) > - > > Key: HIVE-12856 > URL: https://issues.apache.org/jira/browse/HIVE-12856 > Project: Hive > Issue Type: Sub-task >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > Attachments: HIVE-12856.01.nogen.patch, HIVE-12856.01.patch, > HIVE-12856.02.nogen.patch, HIVE-12856.02.patch, HIVE-12856.nogen.patch, > HIVE-12856.patch > > > I don't think re-querying the functions is going to scale, and the sessions > obviously cannot notify all LLAP clusters of every change. We should add > global versioning to metastore functions to track changes, and then possibly > add a notification mechanism, potentially thru ZK to avoid overloading the > metastore itself. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-12856) LLAP: update (add/remove) the UDFs available in LLAP when they are changed (refresh periodically)
[ https://issues.apache.org/jira/browse/HIVE-12856?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15153414#comment-15153414 ] Gunther Hagleitner commented on HIVE-12856: --- I've just glanced at this - is there a writeup somewhere on how this will work? Looks like you're polling metastore from llap and that will force you to have a metastore server somewhere. Why can't you push the info into the daemon with the query fragment? > LLAP: update (add/remove) the UDFs available in LLAP when they are changed > (refresh periodically) > - > > Key: HIVE-12856 > URL: https://issues.apache.org/jira/browse/HIVE-12856 > Project: Hive > Issue Type: Sub-task >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > Attachments: HIVE-12856.01.nogen.patch, HIVE-12856.01.patch, > HIVE-12856.02.nogen.patch, HIVE-12856.02.patch, HIVE-12856.nogen.patch, > HIVE-12856.patch > > > I don't think re-querying the functions is going to scale, and the sessions > obviously cannot notify all LLAP clusters of every change. We should add > global versioning to metastore functions to track changes, and then possibly > add a notification mechanism, potentially thru ZK to avoid overloading the > metastore itself. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-12857) LLAP: modify the decider to allow using LLAP with whitelisted UDFs
[ https://issues.apache.org/jira/browse/HIVE-12857?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15153403#comment-15153403 ] Gunther Hagleitner commented on HIVE-12857: --- +1 > LLAP: modify the decider to allow using LLAP with whitelisted UDFs > -- > > Key: HIVE-12857 > URL: https://issues.apache.org/jira/browse/HIVE-12857 > Project: Hive > Issue Type: Sub-task >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > Attachments: HIVE-12857.01.patch, HIVE-12857.02.patch, > HIVE-12857.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13090) Hive metastore crashes on NPE with ZooKeeperTokenStore
[
https://issues.apache.org/jira/browse/HIVE-13090?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15153263#comment-15153263
]
Ashutosh Chauhan commented on HIVE-13090:
-
+1 is this same as described on HIVE-13075 ? If so, we can close that one as
dupe.
> Hive metastore crashes on NPE with ZooKeeperTokenStore
> --
>
> Key: HIVE-13090
> URL: https://issues.apache.org/jira/browse/HIVE-13090
> Project: Hive
> Issue Type: Bug
> Components: Metastore, Security
>Affects Versions: 1.0.0, 1.1.1, 1.2.1
>Reporter: Thejas M Nair
>Assignee: Thejas M Nair
> Attachments: HIVE-13090.1.patch
>
>
> Observed that hive metastore shutdown with NPE from ZookeeperTokenStore.
> {code}
> INFO [pool-5-thread-192]: metastore.HiveMetaStore
> (HiveMetaStore.java:logInfo(714)) - 191: Metastore shutdown complete.
> INFO [pool-5-thread-192]: HiveMetaStore.audit
> (HiveMetaStore.java:logAuditEvent(340)) - ugi=cvdpqap ip=/19.1.2.129
> cmd=Metastore shutdown complete.
> ERROR [Thread[Thread-6,5,main]]:
> thrift.TokenStoreDelegationTokenSecretManager
> (TokenStoreDelegationTokenSecretManager.java:run(331)) - ExpiredTokenRemover
> thread received unexpected exception.
> org.apache.hadoop.hive.thrift.DelegationTokenStore$TokenStoreException:
> Failed to decode token
> org.apache.hadoop.hive.thrift.DelegationTokenStore$TokenStoreException:
> Failed to decode token
> at
> org.apache.hadoop.hive.thrift.ZooKeeperTokenStore.getToken(ZooKeeperTokenStore.java:401)
> at
> org.apache.hadoop.hive.thrift.TokenStoreDelegationTokenSecretManager.removeExpiredTokens(TokenStoreDelegationTokenSecretManager.java:256)
> at
> org.apache.hadoop.hive.thrift.TokenStoreDelegationTokenSecretManager$ExpiredTokenRemover.run(TokenStoreDelegationTokenSecretManager.java:319)
> at java.lang.Thread.run(Thread.java:744)
> Caused by: java.lang.NullPointerException
> at java.io.ByteArrayInputStream.(ByteArrayInputStream.java:106)
> at
> org.apache.hadoop.security.token.delegation.HiveDelegationTokenSupport.decodeDelegationTokenInformation(HiveDelegationTokenSupport.java:53)
> at
> org.apache.hadoop.hive.thrift.ZooKeeperTokenStore.getToken(ZooKeeperTokenStore.java:399)
> ... 3 more
> INFO [Thread-3]: metastore.HiveMetaStore (HiveMetaStore.java:run(5639)) -
> Shutting down hive metastore.
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Resolved] (HIVE-13022) Header in output file
[ https://issues.apache.org/jira/browse/HIVE-13022?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] raju puli resolved HIVE-13022. -- Resolution: Fixed > Header in output file > -- > > Key: HIVE-13022 > URL: https://issues.apache.org/jira/browse/HIVE-13022 > Project: Hive > Issue Type: Bug > Environment: Hdoop 2.1.0, hive >Reporter: raju puli > Original Estimate: 12h > Remaining Estimate: 12h > > When i'm printing my out put with the header, each field name is coming with > _u1. , i need fix this one as soon as possible my project is on production > stage it need to upload to the web, it will make lot of conflicts. > for example my field names are looks like this > field1 field2 field3 field4 > but in out put it's printing like this > _u1.field1 _u1.field2 _u1.field3 _u1.field4 > my script looks like this > select * from( > ) > union all > select * from ( > ); > i'm running this script with using TEZ. > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13090) Hive metastore crashes on NPE with ZooKeeperTokenStore
[
https://issues.apache.org/jira/browse/HIVE-13090?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Thejas M Nair updated HIVE-13090:
-
Status: Patch Available (was: Open)
> Hive metastore crashes on NPE with ZooKeeperTokenStore
> --
>
> Key: HIVE-13090
> URL: https://issues.apache.org/jira/browse/HIVE-13090
> Project: Hive
> Issue Type: Bug
> Components: Metastore, Security
>Affects Versions: 1.2.1, 1.1.1, 1.0.0
>Reporter: Thejas M Nair
>Assignee: Thejas M Nair
> Attachments: HIVE-13090.1.patch
>
>
> Observed that hive metastore shutdown with NPE from ZookeeperTokenStore.
> {code}
> INFO [pool-5-thread-192]: metastore.HiveMetaStore
> (HiveMetaStore.java:logInfo(714)) - 191: Metastore shutdown complete.
> INFO [pool-5-thread-192]: HiveMetaStore.audit
> (HiveMetaStore.java:logAuditEvent(340)) - ugi=cvdpqap ip=/19.1.2.129
> cmd=Metastore shutdown complete.
> ERROR [Thread[Thread-6,5,main]]:
> thrift.TokenStoreDelegationTokenSecretManager
> (TokenStoreDelegationTokenSecretManager.java:run(331)) - ExpiredTokenRemover
> thread received unexpected exception.
> org.apache.hadoop.hive.thrift.DelegationTokenStore$TokenStoreException:
> Failed to decode token
> org.apache.hadoop.hive.thrift.DelegationTokenStore$TokenStoreException:
> Failed to decode token
> at
> org.apache.hadoop.hive.thrift.ZooKeeperTokenStore.getToken(ZooKeeperTokenStore.java:401)
> at
> org.apache.hadoop.hive.thrift.TokenStoreDelegationTokenSecretManager.removeExpiredTokens(TokenStoreDelegationTokenSecretManager.java:256)
> at
> org.apache.hadoop.hive.thrift.TokenStoreDelegationTokenSecretManager$ExpiredTokenRemover.run(TokenStoreDelegationTokenSecretManager.java:319)
> at java.lang.Thread.run(Thread.java:744)
> Caused by: java.lang.NullPointerException
> at java.io.ByteArrayInputStream.(ByteArrayInputStream.java:106)
> at
> org.apache.hadoop.security.token.delegation.HiveDelegationTokenSupport.decodeDelegationTokenInformation(HiveDelegationTokenSupport.java:53)
> at
> org.apache.hadoop.hive.thrift.ZooKeeperTokenStore.getToken(ZooKeeperTokenStore.java:399)
> ... 3 more
> INFO [Thread-3]: metastore.HiveMetaStore (HiveMetaStore.java:run(5639)) -
> Shutting down hive metastore.
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-13022) Header in output file
[ https://issues.apache.org/jira/browse/HIVE-13022?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15153245#comment-15153245 ] raju puli commented on HIVE-13022: -- Thanks Minu... > Header in output file > -- > > Key: HIVE-13022 > URL: https://issues.apache.org/jira/browse/HIVE-13022 > Project: Hive > Issue Type: Bug > Environment: Hdoop 2.1.0, hive >Reporter: raju puli > Original Estimate: 12h > Remaining Estimate: 12h > > When i'm printing my out put with the header, each field name is coming with > _u1. , i need fix this one as soon as possible my project is on production > stage it need to upload to the web, it will make lot of conflicts. > for example my field names are looks like this > field1 field2 field3 field4 > but in out put it's printing like this > _u1.field1 _u1.field2 _u1.field3 _u1.field4 > my script looks like this > select * from( > ) > union all > select * from ( > ); > i'm running this script with using TEZ. > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13090) Hive metastore crashes on NPE with ZooKeeperTokenStore
[
https://issues.apache.org/jira/browse/HIVE-13090?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15153244#comment-15153244
]
Thejas M Nair commented on HIVE-13090:
--
[~ashutoshc]
Can you please review ?
> Hive metastore crashes on NPE with ZooKeeperTokenStore
> --
>
> Key: HIVE-13090
> URL: https://issues.apache.org/jira/browse/HIVE-13090
> Project: Hive
> Issue Type: Bug
> Components: Metastore, Security
>Affects Versions: 1.0.0, 1.1.1, 1.2.1
>Reporter: Thejas M Nair
>Assignee: Thejas M Nair
> Attachments: HIVE-13090.1.patch
>
>
> Observed that hive metastore shutdown with NPE from ZookeeperTokenStore.
> {code}
> INFO [pool-5-thread-192]: metastore.HiveMetaStore
> (HiveMetaStore.java:logInfo(714)) - 191: Metastore shutdown complete.
> INFO [pool-5-thread-192]: HiveMetaStore.audit
> (HiveMetaStore.java:logAuditEvent(340)) - ugi=cvdpqap ip=/19.1.2.129
> cmd=Metastore shutdown complete.
> ERROR [Thread[Thread-6,5,main]]:
> thrift.TokenStoreDelegationTokenSecretManager
> (TokenStoreDelegationTokenSecretManager.java:run(331)) - ExpiredTokenRemover
> thread received unexpected exception.
> org.apache.hadoop.hive.thrift.DelegationTokenStore$TokenStoreException:
> Failed to decode token
> org.apache.hadoop.hive.thrift.DelegationTokenStore$TokenStoreException:
> Failed to decode token
> at
> org.apache.hadoop.hive.thrift.ZooKeeperTokenStore.getToken(ZooKeeperTokenStore.java:401)
> at
> org.apache.hadoop.hive.thrift.TokenStoreDelegationTokenSecretManager.removeExpiredTokens(TokenStoreDelegationTokenSecretManager.java:256)
> at
> org.apache.hadoop.hive.thrift.TokenStoreDelegationTokenSecretManager$ExpiredTokenRemover.run(TokenStoreDelegationTokenSecretManager.java:319)
> at java.lang.Thread.run(Thread.java:744)
> Caused by: java.lang.NullPointerException
> at java.io.ByteArrayInputStream.(ByteArrayInputStream.java:106)
> at
> org.apache.hadoop.security.token.delegation.HiveDelegationTokenSupport.decodeDelegationTokenInformation(HiveDelegationTokenSupport.java:53)
> at
> org.apache.hadoop.hive.thrift.ZooKeeperTokenStore.getToken(ZooKeeperTokenStore.java:399)
> ... 3 more
> INFO [Thread-3]: metastore.HiveMetaStore (HiveMetaStore.java:run(5639)) -
> Shutting down hive metastore.
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-13090) Hive metastore crashes on NPE with ZooKeeperTokenStore
[
https://issues.apache.org/jira/browse/HIVE-13090?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Thejas M Nair updated HIVE-13090:
-
Attachment: HIVE-13090.1.patch
HIVE-13090.1.patch -
1. Change to check for null instead of NPE
2. Remove the Runtime.getRuntime().exit on error, and just log the error.
> Hive metastore crashes on NPE with ZooKeeperTokenStore
> --
>
> Key: HIVE-13090
> URL: https://issues.apache.org/jira/browse/HIVE-13090
> Project: Hive
> Issue Type: Bug
> Components: Metastore, Security
>Affects Versions: 1.0.0, 1.1.1, 1.2.1
>Reporter: Thejas M Nair
>Assignee: Thejas M Nair
> Attachments: HIVE-13090.1.patch
>
>
> Observed that hive metastore shutdown with NPE from ZookeeperTokenStore.
> {code}
> INFO [pool-5-thread-192]: metastore.HiveMetaStore
> (HiveMetaStore.java:logInfo(714)) - 191: Metastore shutdown complete.
> INFO [pool-5-thread-192]: HiveMetaStore.audit
> (HiveMetaStore.java:logAuditEvent(340)) - ugi=cvdpqap ip=/19.1.2.129
> cmd=Metastore shutdown complete.
> ERROR [Thread[Thread-6,5,main]]:
> thrift.TokenStoreDelegationTokenSecretManager
> (TokenStoreDelegationTokenSecretManager.java:run(331)) - ExpiredTokenRemover
> thread received unexpected exception.
> org.apache.hadoop.hive.thrift.DelegationTokenStore$TokenStoreException:
> Failed to decode token
> org.apache.hadoop.hive.thrift.DelegationTokenStore$TokenStoreException:
> Failed to decode token
> at
> org.apache.hadoop.hive.thrift.ZooKeeperTokenStore.getToken(ZooKeeperTokenStore.java:401)
> at
> org.apache.hadoop.hive.thrift.TokenStoreDelegationTokenSecretManager.removeExpiredTokens(TokenStoreDelegationTokenSecretManager.java:256)
> at
> org.apache.hadoop.hive.thrift.TokenStoreDelegationTokenSecretManager$ExpiredTokenRemover.run(TokenStoreDelegationTokenSecretManager.java:319)
> at java.lang.Thread.run(Thread.java:744)
> Caused by: java.lang.NullPointerException
> at java.io.ByteArrayInputStream.(ByteArrayInputStream.java:106)
> at
> org.apache.hadoop.security.token.delegation.HiveDelegationTokenSupport.decodeDelegationTokenInformation(HiveDelegationTokenSupport.java:53)
> at
> org.apache.hadoop.hive.thrift.ZooKeeperTokenStore.getToken(ZooKeeperTokenStore.java:399)
> ... 3 more
> INFO [Thread-3]: metastore.HiveMetaStore (HiveMetaStore.java:run(5639)) -
> Shutting down hive metastore.
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-13045) move guava dependency back to 14 after HIVE-12952
[ https://issues.apache.org/jira/browse/HIVE-13045?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15153222#comment-15153222 ] Xuefu Zhang commented on HIVE-13045: +1 > move guava dependency back to 14 after HIVE-12952 > - > > Key: HIVE-13045 > URL: https://issues.apache.org/jira/browse/HIVE-13045 > Project: Hive > Issue Type: Bug >Reporter: Mohit Sabharwal >Assignee: Mohit Sabharwal > Attachments: HIVE-13045.patch, HIVE-13045.patch > > > HIVE-12952 removed usage of EvictingQueue, so we don't need to up dependency > to guava 15 at this point - avoid version related conflicts with clients if > we can avoid it. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13051) Deadline class has numerous issues
[ https://issues.apache.org/jira/browse/HIVE-13051?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15153221#comment-15153221 ] Prasanth Jayachandran commented on HIVE-13051: -- LGTM, +1 > Deadline class has numerous issues > -- > > Key: HIVE-13051 > URL: https://issues.apache.org/jira/browse/HIVE-13051 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > Attachments: HIVE-13501.patch > > > currentTimeMillis is not a correct way to measure intervals of time; it can > easily be adjusted e.g. by ntpd. System.nanoTime should be used. > It's also unsafe for failure cases, and doesn't appear to update from config > updates correctly. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (HIVE-13087) LLAP: Print STW pause time and useful application time
[
https://issues.apache.org/jira/browse/HIVE-13087?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Prasanth Jayachandran resolved HIVE-13087.
--
Resolution: Fixed
Fix Version/s: 2.1.0
Committed to master
> LLAP: Print STW pause time and useful application time
> --
>
> Key: HIVE-13087
> URL: https://issues.apache.org/jira/browse/HIVE-13087
> Project: Hive
> Issue Type: Bug
>Affects Versions: 2.1.0
>Reporter: Prasanth Jayachandran
>Assignee: Prasanth Jayachandran
> Fix For: 2.1.0
>
> Attachments: HIVE-13087.1.patch
>
>
> The daemons currently prints GC details. It will be useful to print the total
> useful time application spent and the total time for which application
> threads are stopped.
> Need to add
> {code}
> -XX:+PrintGCApplicationStoppedTime -XX:+PrintGCApplicationConcurrentTime
> {code}
> to get something like
> {code}
> Application time: 0.3440086 seconds
> Total time for which application threads were stopped: 0.0620105 seconds
> Application time: 0.2100691 seconds
> Total time for which application threads were stopped: 0.0890223 seconds
> {code}
> Reference:
> https://plumbr.eu/blog/performance-blog/logging-stop-the-world-pauses-in-jvm
> NO PRECOMMIT TESTS
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-13087) LLAP: Print STW pause time and useful application time
[
https://issues.apache.org/jira/browse/HIVE-13087?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15153211#comment-15153211
]
Gopal V commented on HIVE-13087:
+1 - LGTM.
> LLAP: Print STW pause time and useful application time
> --
>
> Key: HIVE-13087
> URL: https://issues.apache.org/jira/browse/HIVE-13087
> Project: Hive
> Issue Type: Bug
>Affects Versions: 2.1.0
>Reporter: Prasanth Jayachandran
>Assignee: Prasanth Jayachandran
> Attachments: HIVE-13087.1.patch
>
>
> The daemons currently prints GC details. It will be useful to print the total
> useful time application spent and the total time for which application
> threads are stopped.
> Need to add
> {code}
> -XX:+PrintGCApplicationStoppedTime -XX:+PrintGCApplicationConcurrentTime
> {code}
> to get something like
> {code}
> Application time: 0.3440086 seconds
> Total time for which application threads were stopped: 0.0620105 seconds
> Application time: 0.2100691 seconds
> Total time for which application threads were stopped: 0.0890223 seconds
> {code}
> Reference:
> https://plumbr.eu/blog/performance-blog/logging-stop-the-world-pauses-in-jvm
> NO PRECOMMIT TESTS
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-13087) LLAP: Print STW pause time and useful application time
[
https://issues.apache.org/jira/browse/HIVE-13087?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Gopal V updated HIVE-13087:
---
Description:
The daemons currently prints GC details. It will be useful to print the total
useful time application spent and the total time for which application threads
are stopped.
Need to add
{code}
-XX:+PrintGCApplicationStoppedTime -XX:+PrintGCApplicationConcurrentTime
{code}
to get something like
{code}
Application time: 0.3440086 seconds
Total time for which application threads were stopped: 0.0620105 seconds
Application time: 0.2100691 seconds
Total time for which application threads were stopped: 0.0890223 seconds
{code}
Reference:
https://plumbr.eu/blog/performance-blog/logging-stop-the-world-pauses-in-jvm
NO PRECOMMIT TESTS
was:
The daemons currently prints GC details. It will be useful to print the total
useful time application spent and the total time for which application threads
are stopped.
Need to add
{code}
-XX:+PrintGCApplicationStoppedTime -XX:+PrintGCApplicationConcurrentTime
{code}
to get something like
{code}
Application time: 0.3440086 seconds
Total time for which application threads were stopped: 0.0620105 seconds
Application time: 0.2100691 seconds
Total time for which application threads were stopped: 0.0890223 seconds
{code}
Reference:
https://plumbr.eu/blog/performance-blog/logging-stop-the-world-pauses-in-jvm
> LLAP: Print STW pause time and useful application time
> --
>
> Key: HIVE-13087
> URL: https://issues.apache.org/jira/browse/HIVE-13087
> Project: Hive
> Issue Type: Bug
>Affects Versions: 2.1.0
>Reporter: Prasanth Jayachandran
>Assignee: Prasanth Jayachandran
> Attachments: HIVE-13087.1.patch
>
>
> The daemons currently prints GC details. It will be useful to print the total
> useful time application spent and the total time for which application
> threads are stopped.
> Need to add
> {code}
> -XX:+PrintGCApplicationStoppedTime -XX:+PrintGCApplicationConcurrentTime
> {code}
> to get something like
> {code}
> Application time: 0.3440086 seconds
> Total time for which application threads were stopped: 0.0620105 seconds
> Application time: 0.2100691 seconds
> Total time for which application threads were stopped: 0.0890223 seconds
> {code}
> Reference:
> https://plumbr.eu/blog/performance-blog/logging-stop-the-world-pauses-in-jvm
> NO PRECOMMIT TESTS
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-9013) Hive set command exposes metastore db password
[
https://issues.apache.org/jira/browse/HIVE-9013?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15153208#comment-15153208
]
Ravi Prakash commented on HIVE-9013:
Thanks for this fix folks! We also found that when we use the {{!connect}}
command with a password, the password gets logged to ~/.beeline/history. I've
filed https://issues.apache.org/jira/browse/HIVE-13091 . Could you please
suggest how to fix that leak over on that JIRA?
> Hive set command exposes metastore db password
> --
>
> Key: HIVE-9013
> URL: https://issues.apache.org/jira/browse/HIVE-9013
> Project: Hive
> Issue Type: Bug
>Affects Versions: 0.13.1
>Reporter: Binglin Chang
>Assignee: Binglin Chang
> Labels: TODOC1.2, TODOC1.3
> Fix For: 1.3.0, 2.0.0, 1.2.2
>
> Attachments: HIVE-9013.1.patch, HIVE-9013.2.patch, HIVE-9013.3.patch,
> HIVE-9013.4.patch, HIVE-9013.5.patch, HIVE-9013.5.patch,
> HIVE-9013.5.patch-branch1, HIVE-9013.5.patch-branch1.2
>
>
> When auth is enabled, we still need set command to set some variables(e.g.
> mapreduce.job.queuename), but set command alone also list all
> information(including vars in restrict list), this exposes like
> "javax.jdo.option.ConnectionPassword"
> I think conf var in the restrict list should also excluded from dump vars
> command.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-12873) Saving, sharing and securing Hive visualizations
[ https://issues.apache.org/jira/browse/HIVE-12873?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thejas M Nair updated HIVE-12873: - Assignee: (was: Thejas M Nair) > Saving, sharing and securing Hive visualizations > > > Key: HIVE-12873 > URL: https://issues.apache.org/jira/browse/HIVE-12873 > Project: Hive > Issue Type: Improvement > Components: Authentication, Authorization, Hive, Logging, Statistics > Environment: HDP 2.3.2 on Hortonworks Sandbox, Windows OS, Hive > Visualization in general. >Reporter: Bidyut Bhattacharjee > Labels: Enhancement, Visualization, hive > Original Estimate: 2,880h > Remaining Estimate: 2,880h > > We can create charts from Hive tables using the Visualization tab on Hive > Query Editor->Visualization, on the HortonWorks Hadoop Ambari Web Interface. > The 'data explorer' tab and the 'advanced visualization' allows us to create > slices of the data > Currently, the visualizations/reports created under the 'Visualization' tab > cannot be saved. > I would like to request the following features to be added to Hive > Visualization. In my opinion the following features will boost product > adoption as reporting is a critical component of data-centric environment. > 1. Ability to SAVE the HIVE visualizations > Users should be allowed to save the visualizations or reports created through > the 'data explorer' and 'Advanced visualization' features. > 2. Ability to SHARE the saved HIVE visualizations or reports > After the visualizations or reports saved users should be able to share the > reports with other users through URL access. > 3. Securing the saved visualizations/reports > It is important to secure the saved reports. > The following reporting 'roles' may be considered for controlling access to > the reports and to manage the reports: > (3.1) Report Administrator: The 'Report Administrator' administers(backs-up > reporting metadata, restores reports and metadata, will do performance > tuning, provide access rights, creates reporting roles, maintains and manages > reporting system wide configurations etc) of the overall reporting > environment. > > (3.2) Report Publisher: The 'Report Publisher' role is created by the > 'Report Administrator'. The publisher would be responsible for creating > reports or visualizations for the 'Report Browsers'. The report publishers > would be able to alter the report metadata of reports they have access to. > (3.3) Report Browser : The 'Report Administrator' and the 'Report Publisher' > should be allowed to create the 'Report Browser' role. The 'Report Browser' > browses or consumes the reports. > All these roles should be available by default (template) , custom roles may > be derived from the these roles. > 4. Emailing reports > Report Publishers or Report Administrators should have the ability to > schedule automated notification the report consumers that the 'reports' have > been updated after scheduled data refresh executed successfully, this can be > done by simply emailing a hyperlink of the specific reports or by sending an > SMS (Short Messaging Service) to a mobile device. Alternatively, 'Report > Browsers' should be able to 'Subscribe' to the reports to be delivered to > them at a scheduled interval ( hourly, daily, weekly, monthly, quarterly or > yearly) > 5. Exporting reports > Once the reports are created or saved the users should be able to export the > reports to one of the following formats: > 5.1 PDF (Portable Document Format) > 5.2 Microsoft Excel ( .xls or .xlsx extension) > 5.3 LibreOffice or OpenOffice (.ODS) format. > 6. Dashboards > Dashboards contain multiple reports. Once the visualizations are saved, users > should be able to create dashboards using the saved reports. Example: Let's > assume that we are able to save two separate visualizations or reports. > Report 'Monthly Sales' consists of 'sales by month' and the report 'Sales by > state' consists of 'Sales by each state'. Users should be able to create a > sales dashboard 'Sales Dashboard' comprising of both the reports. The > ability to display separate visualizations into one dashboard is very useful > as it allows re-usability, effective for decision making, saves time and > improves analyst productivity. > A 'Dashboard' tab may be added to the Hive->Query Window web interface. > 7. Report execution statistics > There should be a repository to log every report execution. The log > table/file should store the unique execution identifier, date and time the > report or the dashboard was executed, duration for rending the report in > seconds, the user who executed the report, error message if any, report > path, calling application name etc. > 8. Automated report documentation > Most reports
[jira] [Commented] (HIVE-12873) Saving, sharing and securing Hive visualizations
[ https://issues.apache.org/jira/browse/HIVE-12873?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15153190#comment-15153190 ] Thejas M Nair commented on HIVE-12873: -- FYI [~cartershanklin] [~rchoudhary] > Saving, sharing and securing Hive visualizations > > > Key: HIVE-12873 > URL: https://issues.apache.org/jira/browse/HIVE-12873 > Project: Hive > Issue Type: Improvement > Components: Authentication, Authorization, Hive, Logging, Statistics > Environment: HDP 2.3.2 on Hortonworks Sandbox, Windows OS, Hive > Visualization in general. >Reporter: Bidyut Bhattacharjee > Labels: Enhancement, Visualization, hive > Original Estimate: 2,880h > Remaining Estimate: 2,880h > > We can create charts from Hive tables using the Visualization tab on Hive > Query Editor->Visualization, on the HortonWorks Hadoop Ambari Web Interface. > The 'data explorer' tab and the 'advanced visualization' allows us to create > slices of the data > Currently, the visualizations/reports created under the 'Visualization' tab > cannot be saved. > I would like to request the following features to be added to Hive > Visualization. In my opinion the following features will boost product > adoption as reporting is a critical component of data-centric environment. > 1. Ability to SAVE the HIVE visualizations > Users should be allowed to save the visualizations or reports created through > the 'data explorer' and 'Advanced visualization' features. > 2. Ability to SHARE the saved HIVE visualizations or reports > After the visualizations or reports saved users should be able to share the > reports with other users through URL access. > 3. Securing the saved visualizations/reports > It is important to secure the saved reports. > The following reporting 'roles' may be considered for controlling access to > the reports and to manage the reports: > (3.1) Report Administrator: The 'Report Administrator' administers(backs-up > reporting metadata, restores reports and metadata, will do performance > tuning, provide access rights, creates reporting roles, maintains and manages > reporting system wide configurations etc) of the overall reporting > environment. > > (3.2) Report Publisher: The 'Report Publisher' role is created by the > 'Report Administrator'. The publisher would be responsible for creating > reports or visualizations for the 'Report Browsers'. The report publishers > would be able to alter the report metadata of reports they have access to. > (3.3) Report Browser : The 'Report Administrator' and the 'Report Publisher' > should be allowed to create the 'Report Browser' role. The 'Report Browser' > browses or consumes the reports. > All these roles should be available by default (template) , custom roles may > be derived from the these roles. > 4. Emailing reports > Report Publishers or Report Administrators should have the ability to > schedule automated notification the report consumers that the 'reports' have > been updated after scheduled data refresh executed successfully, this can be > done by simply emailing a hyperlink of the specific reports or by sending an > SMS (Short Messaging Service) to a mobile device. Alternatively, 'Report > Browsers' should be able to 'Subscribe' to the reports to be delivered to > them at a scheduled interval ( hourly, daily, weekly, monthly, quarterly or > yearly) > 5. Exporting reports > Once the reports are created or saved the users should be able to export the > reports to one of the following formats: > 5.1 PDF (Portable Document Format) > 5.2 Microsoft Excel ( .xls or .xlsx extension) > 5.3 LibreOffice or OpenOffice (.ODS) format. > 6. Dashboards > Dashboards contain multiple reports. Once the visualizations are saved, users > should be able to create dashboards using the saved reports. Example: Let's > assume that we are able to save two separate visualizations or reports. > Report 'Monthly Sales' consists of 'sales by month' and the report 'Sales by > state' consists of 'Sales by each state'. Users should be able to create a > sales dashboard 'Sales Dashboard' comprising of both the reports. The > ability to display separate visualizations into one dashboard is very useful > as it allows re-usability, effective for decision making, saves time and > improves analyst productivity. > A 'Dashboard' tab may be added to the Hive->Query Window web interface. > 7. Report execution statistics > There should be a repository to log every report execution. The log > table/file should store the unique execution identifier, date and time the > report or the dashboard was executed, duration for rending the report in > seconds, the user who executed the report, error message if any, report > path, calling application name etc. > 8.
[jira] [Commented] (HIVE-13086) LLAP: Programmatically initialize log4j2 to print out the properties location
[ https://issues.apache.org/jira/browse/HIVE-13086?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15153186#comment-15153186 ] Prasanth Jayachandran commented on HIVE-13086: -- LlapDaemon is invoked from other place (MiniLlap) and main(). So added it at top of ctor. > LLAP: Programmatically initialize log4j2 to print out the properties location > - > > Key: HIVE-13086 > URL: https://issues.apache.org/jira/browse/HIVE-13086 > Project: Hive > Issue Type: Bug >Affects Versions: 2.1.0 >Reporter: Prasanth Jayachandran >Assignee: Prasanth Jayachandran > Attachments: HIVE-13086.1.patch, HIVE-13086.2.patch > > > In some cases, llap daemon gets initialized with different log4j2.properties > than the expected llap-daemon-log4j2.properties. It will be easier if > programmatically configure log4j2 so that we can print out the location of > properties file that is used for initialization. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-12730) MetadataUpdater: provide a mechanism to edit the basic statistics of a table (or a partition)
[ https://issues.apache.org/jira/browse/HIVE-12730?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15153184#comment-15153184 ] Ashutosh Chauhan commented on HIVE-12730: - +1 > MetadataUpdater: provide a mechanism to edit the basic statistics of a table > (or a partition) > - > > Key: HIVE-12730 > URL: https://issues.apache.org/jira/browse/HIVE-12730 > Project: Hive > Issue Type: New Feature >Reporter: Pengcheng Xiong >Assignee: Pengcheng Xiong > Attachments: HIVE-12730.01.patch, HIVE-12730.02.patch, > HIVE-12730.03.patch, HIVE-12730.04.patch, HIVE-12730.05.patch, > HIVE-12730.06.patch, HIVE-12730.07.patch, HIVE-12730.08.patch > > > We would like to provide a way for developers/users to modify the numRows and > dataSize for a table/partition. Right now although they are part of the table > properties, they will be set to -1 when the task is not coming from a > statsTask. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13086) LLAP: Programmatically initialize log4j2 to print out the properties location
[ https://issues.apache.org/jira/browse/HIVE-13086?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Prasanth Jayachandran updated HIVE-13086: - Attachment: (was: HIVE-13086.2.patch) > LLAP: Programmatically initialize log4j2 to print out the properties location > - > > Key: HIVE-13086 > URL: https://issues.apache.org/jira/browse/HIVE-13086 > Project: Hive > Issue Type: Bug >Affects Versions: 2.1.0 >Reporter: Prasanth Jayachandran >Assignee: Prasanth Jayachandran > Attachments: HIVE-13086.1.patch, HIVE-13086.2.patch > > > In some cases, llap daemon gets initialized with different log4j2.properties > than the expected llap-daemon-log4j2.properties. It will be easier if > programmatically configure log4j2 so that we can print out the location of > properties file that is used for initialization. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13086) LLAP: Programmatically initialize log4j2 to print out the properties location
[ https://issues.apache.org/jira/browse/HIVE-13086?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Prasanth Jayachandran updated HIVE-13086: - Attachment: HIVE-13086.2.patch > LLAP: Programmatically initialize log4j2 to print out the properties location > - > > Key: HIVE-13086 > URL: https://issues.apache.org/jira/browse/HIVE-13086 > Project: Hive > Issue Type: Bug >Affects Versions: 2.1.0 >Reporter: Prasanth Jayachandran >Assignee: Prasanth Jayachandran > Attachments: HIVE-13086.1.patch, HIVE-13086.2.patch > > > In some cases, llap daemon gets initialized with different log4j2.properties > than the expected llap-daemon-log4j2.properties. It will be easier if > programmatically configure log4j2 so that we can print out the location of > properties file that is used for initialization. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13086) LLAP: Programmatically initialize log4j2 to print out the properties location
[ https://issues.apache.org/jira/browse/HIVE-13086?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15153179#comment-15153179 ] Prasanth Jayachandran commented on HIVE-13086: -- This is not related to SLF4J's runtime binding. This is just to avoid erroneous autoconfiguration of logging for llap. [~gopalv]'s comment https://issues.apache.org/jira/browse/HIVE-13027?focusedCommentId=15150141=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15150141 suspects logging initialized from some other properties files. If for some reason logging is initialized with wrong properties file this initialization (reconfiguration) will make sure llap always uses llap-daemon-log4j2.properties file. Also it prints which file it picked up from classpath. > LLAP: Programmatically initialize log4j2 to print out the properties location > - > > Key: HIVE-13086 > URL: https://issues.apache.org/jira/browse/HIVE-13086 > Project: Hive > Issue Type: Bug >Affects Versions: 2.1.0 >Reporter: Prasanth Jayachandran >Assignee: Prasanth Jayachandran > Attachments: HIVE-13086.1.patch, HIVE-13086.2.patch > > > In some cases, llap daemon gets initialized with different log4j2.properties > than the expected llap-daemon-log4j2.properties. It will be easier if > programmatically configure log4j2 so that we can print out the location of > properties file that is used for initialization. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13052) Allow tests to start MiniHS2 for manual testing
[ https://issues.apache.org/jira/browse/HIVE-13052?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jason Dere updated HIVE-13052: -- Resolution: Fixed Fix Version/s: 2.1.0 Status: Resolved (was: Patch Available) Committed to master > Allow tests to start MiniHS2 for manual testing > --- > > Key: HIVE-13052 > URL: https://issues.apache.org/jira/browse/HIVE-13052 > Project: Hive > Issue Type: Bug > Components: Testing Infrastructure >Reporter: Jason Dere >Assignee: Jason Dere > Fix For: 2.1.0 > > Attachments: HIVE-13052.1.patch, HIVE-13052.2.patch, > HIVE-13052.3.patch > > > Useful for some manual testing -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13086) LLAP: Programmatically initialize log4j2 to print out the properties location
[ https://issues.apache.org/jira/browse/HIVE-13086?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Prasanth Jayachandran updated HIVE-13086: - Attachment: HIVE-13086.2.patch Moved the logging initialization to top. > LLAP: Programmatically initialize log4j2 to print out the properties location > - > > Key: HIVE-13086 > URL: https://issues.apache.org/jira/browse/HIVE-13086 > Project: Hive > Issue Type: Bug >Affects Versions: 2.1.0 >Reporter: Prasanth Jayachandran >Assignee: Prasanth Jayachandran > Attachments: HIVE-13086.1.patch, HIVE-13086.2.patch > > > In some cases, llap daemon gets initialized with different log4j2.properties > than the expected llap-daemon-log4j2.properties. It will be easier if > programmatically configure log4j2 so that we can print out the location of > properties file that is used for initialization. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13087) LLAP: Print STW pause time and useful application time
[
https://issues.apache.org/jira/browse/HIVE-13087?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Prasanth Jayachandran updated HIVE-13087:
-
Attachment: HIVE-13087.1.patch
> LLAP: Print STW pause time and useful application time
> --
>
> Key: HIVE-13087
> URL: https://issues.apache.org/jira/browse/HIVE-13087
> Project: Hive
> Issue Type: Bug
>Affects Versions: 2.1.0
>Reporter: Prasanth Jayachandran
>Assignee: Prasanth Jayachandran
> Attachments: HIVE-13087.1.patch
>
>
> The daemons currently prints GC details. It will be useful to print the total
> useful time application spent and the total time for which application
> threads are stopped.
> Need to add
> {code}
> -XX:+PrintGCApplicationStoppedTime -XX:+PrintGCApplicationConcurrentTime
> {code}
> to get something like
> {code}
> Application time: 0.3440086 seconds
> Total time for which application threads were stopped: 0.0620105 seconds
> Application time: 0.2100691 seconds
> Total time for which application threads were stopped: 0.0890223 seconds
> {code}
> Reference:
> https://plumbr.eu/blog/performance-blog/logging-stop-the-world-pauses-in-jvm
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-13052) Allow tests to start MiniHS2 for manual testing
[ https://issues.apache.org/jira/browse/HIVE-13052?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jason Dere updated HIVE-13052: -- Attachment: HIVE-13052.3.patch Rebasing patch to master > Allow tests to start MiniHS2 for manual testing > --- > > Key: HIVE-13052 > URL: https://issues.apache.org/jira/browse/HIVE-13052 > Project: Hive > Issue Type: Bug > Components: Testing Infrastructure >Reporter: Jason Dere >Assignee: Jason Dere > Attachments: HIVE-13052.1.patch, HIVE-13052.2.patch, > HIVE-13052.3.patch > > > Useful for some manual testing -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13052) Allow tests to start MiniHS2 for manual testing
[ https://issues.apache.org/jira/browse/HIVE-13052?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15153127#comment-15153127 ] Jason Dere commented on HIVE-13052: --- Tested the failures which were not previously occurring (TestCliDriver.testCliDriver_index_auto_mult_tables, TestHiveMetaStorePartitionSpecs), these tests were passing when I run locally. I'm going to commit this patch. > Allow tests to start MiniHS2 for manual testing > --- > > Key: HIVE-13052 > URL: https://issues.apache.org/jira/browse/HIVE-13052 > Project: Hive > Issue Type: Bug > Components: Testing Infrastructure >Reporter: Jason Dere >Assignee: Jason Dere > Attachments: HIVE-13052.1.patch, HIVE-13052.2.patch > > > Useful for some manual testing -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-10632) Make sure TXN_COMPONENTS gets cleaned up if table is dropped before compaction.
[ https://issues.apache.org/jira/browse/HIVE-10632?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15153090#comment-15153090 ] Wei Zheng commented on HIVE-10632: -- Test failures are not related. [~alangates] Can you take a look at this one? Thanks. > Make sure TXN_COMPONENTS gets cleaned up if table is dropped before > compaction. > --- > > Key: HIVE-10632 > URL: https://issues.apache.org/jira/browse/HIVE-10632 > Project: Hive > Issue Type: Bug > Components: Metastore, Transactions >Affects Versions: 1.0.0 >Reporter: Eugene Koifman >Assignee: Wei Zheng >Priority: Critical > Attachments: HIVE-10632.1.patch, HIVE-10632.2.patch, > HIVE-10632.3.patch, HIVE-10632.4.patch, HIVE-10632.5.patch > > > The compaction process will clean up entries in TXNS, > COMPLETED_TXN_COMPONENTS, TXN_COMPONENTS. If the table/partition is dropped > before compaction is complete there will be data left in these tables. Need > to investigate if there are other situations where this may happen and > address it. > see HIVE-10595 for additional info -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13089) Rounding in Stats for equality expressions
[ https://issues.apache.org/jira/browse/HIVE-13089?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jesus Camacho Rodriguez updated HIVE-13089: --- Attachment: HIVE-13089.patch > Rounding in Stats for equality expressions > -- > > Key: HIVE-13089 > URL: https://issues.apache.org/jira/browse/HIVE-13089 > Project: Hive > Issue Type: Bug > Components: Statistics >Affects Versions: 2.1.0 >Reporter: Jesus Camacho Rodriguez >Assignee: Jesus Camacho Rodriguez > Attachments: HIVE-13089.patch > > > Currently we divide numRows(long) by countDistinct(long), thus ignoring the > decimals. We should do proper rounding. > This is specially useful for equality expressions over columns whose values > are unique. As NDV estimates allow for a certain error, if countDistinct > > numRows, we end up with 0 rows in the estimate for the expression. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Work started] (HIVE-13089) Rounding in Stats for equality expressions
[ https://issues.apache.org/jira/browse/HIVE-13089?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on HIVE-13089 started by Jesus Camacho Rodriguez. -- > Rounding in Stats for equality expressions > -- > > Key: HIVE-13089 > URL: https://issues.apache.org/jira/browse/HIVE-13089 > Project: Hive > Issue Type: Bug > Components: Statistics >Affects Versions: 2.1.0 >Reporter: Jesus Camacho Rodriguez >Assignee: Jesus Camacho Rodriguez > Attachments: HIVE-13089.patch > > > Currently we divide numRows(long) by countDistinct(long), thus ignoring the > decimals. We should do proper rounding. > This is specially useful for equality expressions over columns whose values > are unique. As NDV estimates allow for a certain error, if countDistinct > > numRows, we end up with 0 rows in the estimate for the expression. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13089) Rounding in Stats for equality expressions
[ https://issues.apache.org/jira/browse/HIVE-13089?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jesus Camacho Rodriguez updated HIVE-13089: --- Status: Patch Available (was: In Progress) > Rounding in Stats for equality expressions > -- > > Key: HIVE-13089 > URL: https://issues.apache.org/jira/browse/HIVE-13089 > Project: Hive > Issue Type: Bug > Components: Statistics >Affects Versions: 2.1.0 >Reporter: Jesus Camacho Rodriguez >Assignee: Jesus Camacho Rodriguez > Attachments: HIVE-13089.patch > > > Currently we divide numRows(long) by countDistinct(long), thus ignoring the > decimals. We should do proper rounding. > This is specially useful for equality expressions over columns whose values > are unique. As NDV estimates allow for a certain error, if countDistinct > > numRows, we end up with 0 rows in the estimate for the expression. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13055) Add unit tests for HIVE-11512
[ https://issues.apache.org/jira/browse/HIVE-13055?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15153001#comment-15153001 ] Aihua Xu commented on HIVE-13055: - Pushed to branch-1 as well. > Add unit tests for HIVE-11512 > - > > Key: HIVE-13055 > URL: https://issues.apache.org/jira/browse/HIVE-13055 > Project: Hive > Issue Type: Test > Components: HiveServer2 >Affects Versions: 1.3.0 >Reporter: Naveen Gangam >Assignee: Naveen Gangam > Fix For: 1.3.0, 2.1.0 > > Attachments: HIVE-13055.patch > > > HIVE-11866 adds the ability to test LDAP authenticator with a LDAP Server. We > need more test coverage for feature in HIVE-11512. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13055) Add unit tests for HIVE-11512
[ https://issues.apache.org/jira/browse/HIVE-13055?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aihua Xu updated HIVE-13055: Fix Version/s: 1.3.0 > Add unit tests for HIVE-11512 > - > > Key: HIVE-13055 > URL: https://issues.apache.org/jira/browse/HIVE-13055 > Project: Hive > Issue Type: Test > Components: HiveServer2 >Affects Versions: 1.3.0 >Reporter: Naveen Gangam >Assignee: Naveen Gangam > Fix For: 1.3.0, 2.1.0 > > Attachments: HIVE-13055.patch > > > HIVE-11866 adds the ability to test LDAP authenticator with a LDAP Server. We > need more test coverage for feature in HIVE-11512. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13088) Display correct error message on QTest failure commands
[ https://issues.apache.org/jira/browse/HIVE-13088?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergio Peña updated HIVE-13088: --- Status: Patch Available (was: In Progress) > Display correct error message on QTest failure commands > --- > > Key: HIVE-13088 > URL: https://issues.apache.org/jira/browse/HIVE-13088 > Project: Hive > Issue Type: Improvement > Components: Testing Infrastructure >Affects Versions: 2.0.0, 1.2.1 >Reporter: Sergio Peña >Assignee: Sergio Peña >Priority: Minor > Attachments: HIVE-13088.1.patch > > > Everytime a QTest command fails, a generic error message is displayed on > console and surefire-output files. This does not help to understand the error > at all. > We should display the correct error message that Hive throws to troubleshoot > the issue faster. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13088) Display correct error message on QTest failure commands
[
https://issues.apache.org/jira/browse/HIVE-13088?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15152992#comment-15152992
]
Sergio Peña commented on HIVE-13088:
This patch will display an error message on console and the surefire output
files like this:
{code}
Running org.apache.hadoop.hive.cli.TestCliDriver
Tests run: 1, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 89.254 sec <<<
FAILURE! - in org.apache.hadoop.hive.cli.TestCliDriver
testCliDriver_analyze_tbl_part(org.apache.hadoop.hive.cli.TestCliDriver) Time
elapsed: 16.314 sec <<< FAILURE!
java.lang.AssertionError: Unexpected exception java.lang.AssertionError:
"Invalid partitioning key/value specified in ANALYZE statement"
command=[ANALYZE TABLE src_stat_string_part partition (partitionName="p'1")
COMPUTE STATISTICS for columns key, value] errorCode=[30007] debugHint=[See
./ql/target/tmp/log/hive.log or ./itests/qtest/target/tmp/log/hive.log, or
check ./ql/target/surefire-reports or ./itests/qtest/target/surefire-reports/
for specific test cases logs.]
at org.junit.Assert.fail(Assert.java:88)
at org.apache.hadoop.hive.ql.QTestUtil.failed(QTestUtil.java:2006)
{code}
The stacktrace is still the same generic error, but that is because we use
Assert.fail() to make the test case fail, and this prints such stack trace.
> Display correct error message on QTest failure commands
> ---
>
> Key: HIVE-13088
> URL: https://issues.apache.org/jira/browse/HIVE-13088
> Project: Hive
> Issue Type: Improvement
> Components: Testing Infrastructure
>Affects Versions: 1.2.1, 2.0.0
>Reporter: Sergio Peña
>Assignee: Sergio Peña
>Priority: Minor
> Attachments: HIVE-13088.1.patch
>
>
> Everytime a QTest command fails, a generic error message is displayed on
> console and surefire-output files. This does not help to understand the error
> at all.
> We should display the correct error message that Hive throws to troubleshoot
> the issue faster.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-13088) Display correct error message on QTest failure commands
[ https://issues.apache.org/jira/browse/HIVE-13088?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergio Peña updated HIVE-13088: --- Attachment: HIVE-13088.1.patch > Display correct error message on QTest failure commands > --- > > Key: HIVE-13088 > URL: https://issues.apache.org/jira/browse/HIVE-13088 > Project: Hive > Issue Type: Improvement > Components: Testing Infrastructure >Affects Versions: 1.2.1, 2.0.0 >Reporter: Sergio Peña >Assignee: Sergio Peña >Priority: Minor > Attachments: HIVE-13088.1.patch > > > Everytime a QTest command fails, a generic error message is displayed on > console and surefire-output files. This does not help to understand the error > at all. > We should display the correct error message that Hive throws to troubleshoot > the issue faster. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13088) Display correct error message on QTest failure commands
[ https://issues.apache.org/jira/browse/HIVE-13088?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergio Peña updated HIVE-13088: --- Attachment: (was: HIVE-13088.1.patch) > Display correct error message on QTest failure commands > --- > > Key: HIVE-13088 > URL: https://issues.apache.org/jira/browse/HIVE-13088 > Project: Hive > Issue Type: Improvement > Components: Testing Infrastructure >Affects Versions: 1.2.1, 2.0.0 >Reporter: Sergio Peña >Assignee: Sergio Peña >Priority: Minor > > Everytime a QTest command fails, a generic error message is displayed on > console and surefire-output files. This does not help to understand the error > at all. > We should display the correct error message that Hive throws to troubleshoot > the issue faster. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13088) Display correct error message on QTest failure commands
[ https://issues.apache.org/jira/browse/HIVE-13088?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergio Peña updated HIVE-13088: --- Attachment: HIVE-13088.1.patch > Display correct error message on QTest failure commands > --- > > Key: HIVE-13088 > URL: https://issues.apache.org/jira/browse/HIVE-13088 > Project: Hive > Issue Type: Improvement > Components: Testing Infrastructure >Affects Versions: 1.2.1, 2.0.0 >Reporter: Sergio Peña >Assignee: Sergio Peña >Priority: Minor > Attachments: HIVE-13088.1.patch > > > Everytime a QTest command fails, a generic error message is displayed on > console and surefire-output files. This does not help to understand the error > at all. > We should display the correct error message that Hive throws to troubleshoot > the issue faster. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Work started] (HIVE-13088) Display correct error message on QTest failure commands
[ https://issues.apache.org/jira/browse/HIVE-13088?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on HIVE-13088 started by Sergio Peña. -- > Display correct error message on QTest failure commands > --- > > Key: HIVE-13088 > URL: https://issues.apache.org/jira/browse/HIVE-13088 > Project: Hive > Issue Type: Improvement > Components: Testing Infrastructure >Affects Versions: 1.2.1, 2.0.0 >Reporter: Sergio Peña >Assignee: Sergio Peña >Priority: Minor > > Everytime a QTest command fails, a generic error message is displayed on > console and surefire-output files. This does not help to understand the error > at all. > We should display the correct error message that Hive throws to troubleshoot > the issue faster. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13088) Display correct error message on QTest failure commands
[ https://issues.apache.org/jira/browse/HIVE-13088?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergio Peña updated HIVE-13088: --- Component/s: (was: Hive) Testing Infrastructure > Display correct error message on QTest failure commands > --- > > Key: HIVE-13088 > URL: https://issues.apache.org/jira/browse/HIVE-13088 > Project: Hive > Issue Type: Improvement > Components: Testing Infrastructure >Affects Versions: 1.2.1, 2.0.0 >Reporter: Sergio Peña >Assignee: Sergio Peña >Priority: Minor > > Everytime a QTest command fails, a generic error message is displayed on > console and surefire-output files. This does not help to understand the error > at all. > We should display the correct error message that Hive throws to troubleshoot > the issue faster. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-10632) Make sure TXN_COMPONENTS gets cleaned up if table is dropped before compaction.
[
https://issues.apache.org/jira/browse/HIVE-10632?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15152974#comment-15152974
]
Hive QA commented on HIVE-10632:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12788348/HIVE-10632.5.patch
{color:green}SUCCESS:{color} +1 due to 2 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 5 failed/errored test(s), 9814 tests executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_partition_coltype_literals
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_stats_only_null
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_authorization_uri_import
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_stats_only_null
org.apache.hive.jdbc.TestSSL.testSSLVersion
{noformat}
Test results:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/7021/testReport
Console output:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/7021/console
Test logs:
http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-7021/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 5 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12788348 - PreCommit-HIVE-TRUNK-Build
> Make sure TXN_COMPONENTS gets cleaned up if table is dropped before
> compaction.
> ---
>
> Key: HIVE-10632
> URL: https://issues.apache.org/jira/browse/HIVE-10632
> Project: Hive
> Issue Type: Bug
> Components: Metastore, Transactions
>Affects Versions: 1.0.0
>Reporter: Eugene Koifman
>Assignee: Wei Zheng
>Priority: Critical
> Attachments: HIVE-10632.1.patch, HIVE-10632.2.patch,
> HIVE-10632.3.patch, HIVE-10632.4.patch, HIVE-10632.5.patch
>
>
> The compaction process will clean up entries in TXNS,
> COMPLETED_TXN_COMPONENTS, TXN_COMPONENTS. If the table/partition is dropped
> before compaction is complete there will be data left in these tables. Need
> to investigate if there are other situations where this may happen and
> address it.
> see HIVE-10595 for additional info
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-13086) LLAP: Programmatically initialize log4j2 to print out the properties location
[ https://issues.apache.org/jira/browse/HIVE-13086?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15152973#comment-15152973 ] Prasanth Jayachandran commented on HIVE-13086: -- [~gopalv]/[~sershe] Can someone take a look? > LLAP: Programmatically initialize log4j2 to print out the properties location > - > > Key: HIVE-13086 > URL: https://issues.apache.org/jira/browse/HIVE-13086 > Project: Hive > Issue Type: Bug >Affects Versions: 2.1.0 >Reporter: Prasanth Jayachandran >Assignee: Prasanth Jayachandran > Attachments: HIVE-13086.1.patch > > > In some cases, llap daemon gets initialized with different log4j2.properties > than the expected llap-daemon-log4j2.properties. It will be easier if > programmatically configure log4j2 so that we can print out the location of > properties file that is used for initialization. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13086) LLAP: Programmatically initialize log4j2 to print out the properties location
[ https://issues.apache.org/jira/browse/HIVE-13086?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Prasanth Jayachandran updated HIVE-13086: - Status: Patch Available (was: Open) > LLAP: Programmatically initialize log4j2 to print out the properties location > - > > Key: HIVE-13086 > URL: https://issues.apache.org/jira/browse/HIVE-13086 > Project: Hive > Issue Type: Bug >Affects Versions: 2.1.0 >Reporter: Prasanth Jayachandran > Attachments: HIVE-13086.1.patch > > > In some cases, llap daemon gets initialized with different log4j2.properties > than the expected llap-daemon-log4j2.properties. It will be easier if > programmatically configure log4j2 so that we can print out the location of > properties file that is used for initialization. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13086) LLAP: Programmatically initialize log4j2 to print out the properties location
[ https://issues.apache.org/jira/browse/HIVE-13086?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Prasanth Jayachandran updated HIVE-13086: - Attachment: HIVE-13086.1.patch > LLAP: Programmatically initialize log4j2 to print out the properties location > - > > Key: HIVE-13086 > URL: https://issues.apache.org/jira/browse/HIVE-13086 > Project: Hive > Issue Type: Bug >Affects Versions: 2.1.0 >Reporter: Prasanth Jayachandran > Attachments: HIVE-13086.1.patch > > > In some cases, llap daemon gets initialized with different log4j2.properties > than the expected llap-daemon-log4j2.properties. It will be easier if > programmatically configure log4j2 so that we can print out the location of > properties file that is used for initialization. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Assigned] (HIVE-13086) LLAP: Programmatically initialize log4j2 to print out the properties location
[ https://issues.apache.org/jira/browse/HIVE-13086?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Prasanth Jayachandran reassigned HIVE-13086: Assignee: Prasanth Jayachandran > LLAP: Programmatically initialize log4j2 to print out the properties location > - > > Key: HIVE-13086 > URL: https://issues.apache.org/jira/browse/HIVE-13086 > Project: Hive > Issue Type: Bug >Affects Versions: 2.1.0 >Reporter: Prasanth Jayachandran >Assignee: Prasanth Jayachandran > Attachments: HIVE-13086.1.patch > > > In some cases, llap daemon gets initialized with different log4j2.properties > than the expected llap-daemon-log4j2.properties. It will be easier if > programmatically configure log4j2 so that we can print out the location of > properties file that is used for initialization. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13083) Writing HiveDecimal to ORC can wrongly suppress present stream
[
https://issues.apache.org/jira/browse/HIVE-13083?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15152966#comment-15152966
]
Prasanth Jayachandran commented on HIVE-13083:
--
[~gopalv] I think this would need bump in file version because of writer
changes.
> Writing HiveDecimal to ORC can wrongly suppress present stream
> --
>
> Key: HIVE-13083
> URL: https://issues.apache.org/jira/browse/HIVE-13083
> Project: Hive
> Issue Type: Bug
>Affects Versions: 0.13.0, 0.14.0, 1.0.0, 1.2.0, 1.1.0, 1.3.0, 2.0.0, 2.1.0
>Reporter: Yi Zhang
>Assignee: Prasanth Jayachandran
> Attachments: HIVE-13083-branch-1.patch, HIVE-13083.1.patch
>
>
> HIVE-3976 can cause ORC file to be unreadable. The changes introduced in
> HIVE-3976 for DecimalTreeWriter can create null values after updating the
> isPresent stream.
> https://github.com/apache/hive/blob/branch-0.13/ql/src/java/org/apache/hadoop/hive/ql/io/orc/WriterImpl.java#L1337
> As result of the above return statement, isPresent stream state can become
> wrong. The isPresent stream thinks all values are non-null and hence
> suppressed. But the data stream will be of 0 length. When reading such files
> we will get the following exception
> {code}
> Caused by: java.io.EOFException: Reading BigInteger past EOF from compressed
> stream Stream for column 3 kind DATA position: 0 length: 0 range: 0 offset: 0
> limit: 0
> at
> org.apache.hadoop.hive.ql.io.orc.SerializationUtils.readBigInteger(SerializationUtils.java:176)
> at
> org.apache.hadoop.hive.ql.io.orc.TreeReaderFactory$DecimalTreeReader.next(TreeReaderFactory.java:1264)
> at
> org.apache.hadoop.hive.ql.io.orc.TreeReaderFactory$StructTreeReader.next(TreeReaderFactory.java:2004)
> at
> org.apache.hadoop.hive.ql.io.orc.RecordReaderImpl.next(RecordReaderImpl.java:1039)
> ... 24 more
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-11720) Allow HiveServer2 to set custom http request/response header size
[ https://issues.apache.org/jira/browse/HIVE-11720?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15152928#comment-15152928 ] Kevin Vasko commented on HIVE-11720: Is there a workaround for this for older versions? > Allow HiveServer2 to set custom http request/response header size > - > > Key: HIVE-11720 > URL: https://issues.apache.org/jira/browse/HIVE-11720 > Project: Hive > Issue Type: Improvement > Components: HiveServer2 >Reporter: Vaibhav Gumashta >Assignee: Vaibhav Gumashta > Labels: TODOC1.3 > Fix For: 1.3.0, 2.0.0 > > Attachments: HIVE-11720.1.patch, HIVE-11720.2.patch, > HIVE-11720.3.patch, HIVE-11720.4.patch, HIVE-11720.4.patch > > > In HTTP transport mode, authentication information is sent over as part of > HTTP headers. Sometimes (observed when Kerberos is used) the default buffer > size for the headers is not enough, resulting in an HTTP 413 FULL head error. > We can expose those as customizable params. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13070) Precommit HMS tests should run in addition to precommit normal tests, not instead of
[ https://issues.apache.org/jira/browse/HIVE-13070?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15152898#comment-15152898 ] Ashutosh Chauhan commented on HIVE-13070: - Changes look good to me. [~szehon] Would you also like to take a look? Do we need to restart test master for having these changes take effect? > Precommit HMS tests should run in addition to precommit normal tests, not > instead of > > > Key: HIVE-13070 > URL: https://issues.apache.org/jira/browse/HIVE-13070 > Project: Hive > Issue Type: Bug > Components: Testing Infrastructure >Affects Versions: 2.1.0 >Reporter: Jesus Camacho Rodriguez >Assignee: Jesus Camacho Rodriguez > Attachments: HIVE-13070.patch > > > When a certain patch makes changes in the metastore upgrade scripts folder, > precommit HMS tests are triggered. The problem is that precommit HMS marks > the patch as tested, thus normal precommit tests are never triggered. > I hit the issue while testing HIVE-12994. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-12988) Improve dynamic partition loading IV
[ https://issues.apache.org/jira/browse/HIVE-12988?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15152879#comment-15152879 ] Ashutosh Chauhan commented on HIVE-12988: - [~prasanth_j] Can you review this one when you get a chance? > Improve dynamic partition loading IV > > > Key: HIVE-12988 > URL: https://issues.apache.org/jira/browse/HIVE-12988 > Project: Hive > Issue Type: Improvement > Components: Query Processor >Affects Versions: 1.2.0, 2.0.0 >Reporter: Ashutosh Chauhan >Assignee: Ashutosh Chauhan > Attachments: HIVE-12988.2.patch, HIVE-12988.2.patch, > HIVE-12988.3.patch, HIVE-12988.4.patch, HIVE-12988.patch > > > Parallelize copyFiles() -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Assigned] (HIVE-13087) LLAP: Print STW pause time and useful application time
[
https://issues.apache.org/jira/browse/HIVE-13087?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Prasanth Jayachandran reassigned HIVE-13087:
Assignee: Prasanth Jayachandran
> LLAP: Print STW pause time and useful application time
> --
>
> Key: HIVE-13087
> URL: https://issues.apache.org/jira/browse/HIVE-13087
> Project: Hive
> Issue Type: Bug
>Affects Versions: 2.1.0
>Reporter: Prasanth Jayachandran
>Assignee: Prasanth Jayachandran
>
> The daemons currently prints GC details. It will be useful to print the total
> useful time application spent and the total time for which application
> threads are stopped.
> Need to add
> {code}
> -XX:+PrintGCApplicationStoppedTime -XX:+PrintGCApplicationConcurrentTime
> {code}
> to get something like
> {code}
> Application time: 0.3440086 seconds
> Total time for which application threads were stopped: 0.0620105 seconds
> Application time: 0.2100691 seconds
> Total time for which application threads were stopped: 0.0890223 seconds
> {code}
> Reference:
> https://plumbr.eu/blog/performance-blog/logging-stop-the-world-pauses-in-jvm
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-13065) Hive throws NPE when writing map type data to a HBase backed table
[
https://issues.apache.org/jira/browse/HIVE-13065?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yongzhi Chen updated HIVE-13065:
Resolution: Fixed
Fix Version/s: 2.1.0
1.3.0
Status: Resolved (was: Patch Available)
Committed to master and branch-1
Thanks [~aihuaxu] for reviewing the code.
> Hive throws NPE when writing map type data to a HBase backed table
> --
>
> Key: HIVE-13065
> URL: https://issues.apache.org/jira/browse/HIVE-13065
> Project: Hive
> Issue Type: Bug
> Components: HBase Handler
>Affects Versions: 1.1.0, 2.0.0
>Reporter: Yongzhi Chen
>Assignee: Yongzhi Chen
> Fix For: 1.3.0, 2.1.0
>
> Attachments: HIVE-13065.1.patch
>
>
> Hive throws NPE when writing data to a HBase backed table with below
> conditions:
> # There is a map type column
> # The map type column has NULL in its values
> Below are the reproduce steps:
> *1) Create a HBase backed Hive table*
> {code:sql}
> create table hbase_test (id bigint, data map)
> stored by 'org.apache.hadoop.hive.hbase.HBaseStorageHandler'
> with serdeproperties ("hbase.columns.mapping" = ":key,cf:map_col")
> tblproperties ("hbase.table.name" = "hive_test");
> {code}
> *2) insert data into above table*
> {code:sql}
> insert overwrite table hbase_test select 1 as id, map('abcd', null) as data
> from src limit 1;
> {code}
> The mapreduce job for insert query fails. Error messages are as below:
> {noformat}
> 2016-02-15 02:26:33,225 WARN [main] org.apache.hadoop.mapred.YarnChild:
> Exception running child : java.lang.RuntimeException:
> org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime Error while
> processing row (tag=0) {"key":{},"value":{"_col0":1,"_col1":{"abcd":null}}}
> at
> org.apache.hadoop.hive.ql.exec.mr.ExecReducer.reduce(ExecReducer.java:265)
> at
> org.apache.hadoop.mapred.ReduceTask.runOldReducer(ReduceTask.java:444)
> at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:392)
> at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:163)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:415)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1671)
> at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158)
> Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime
> Error while processing row (tag=0)
> {"key":{},"value":{"_col0":1,"_col1":{"abcd":null}}}
> at
> org.apache.hadoop.hive.ql.exec.mr.ExecReducer.reduce(ExecReducer.java:253)
> ... 7 more
> Caused by: org.apache.hadoop.hive.ql.metadata.HiveException:
> org.apache.hadoop.hive.serde2.SerDeException: java.lang.NullPointerException
> at
> org.apache.hadoop.hive.ql.exec.FileSinkOperator.processOp(FileSinkOperator.java:731)
> at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:815)
> at
> org.apache.hadoop.hive.ql.exec.SelectOperator.processOp(SelectOperator.java:84)
> at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:815)
> at
> org.apache.hadoop.hive.ql.exec.LimitOperator.processOp(LimitOperator.java:51)
> at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:815)
> at
> org.apache.hadoop.hive.ql.exec.SelectOperator.processOp(SelectOperator.java:84)
> at
> org.apache.hadoop.hive.ql.exec.mr.ExecReducer.reduce(ExecReducer.java:244)
> ... 7 more
> Caused by: org.apache.hadoop.hive.serde2.SerDeException:
> java.lang.NullPointerException
> at
> org.apache.hadoop.hive.hbase.HBaseSerDe.serialize(HBaseSerDe.java:286)
> at
> org.apache.hadoop.hive.ql.exec.FileSinkOperator.processOp(FileSinkOperator.java:666)
> ... 14 more
> Caused by: java.lang.NullPointerException
> at
> org.apache.hadoop.hive.serde2.lazy.LazyUtils.writePrimitiveUTF8(LazyUtils.java:221)
> at
> org.apache.hadoop.hive.hbase.HBaseRowSerializer.serialize(HBaseRowSerializer.java:236)
> at
> org.apache.hadoop.hive.hbase.HBaseRowSerializer.serialize(HBaseRowSerializer.java:275)
> at
> org.apache.hadoop.hive.hbase.HBaseRowSerializer.serialize(HBaseRowSerializer.java:222)
> at
> org.apache.hadoop.hive.hbase.HBaseRowSerializer.serializeField(HBaseRowSerializer.java:194)
> at
> org.apache.hadoop.hive.hbase.HBaseRowSerializer.serialize(HBaseRowSerializer.java:118)
> at
> org.apache.hadoop.hive.hbase.HBaseSerDe.serialize(HBaseSerDe.java:282)
> ... 15 more
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-12856) LLAP: update (add/remove) the UDFs available in LLAP when they are changed (refresh periodically)
[ https://issues.apache.org/jira/browse/HIVE-12856?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15152842#comment-15152842 ] Sergey Shelukhin commented on HIVE-12856: - [~vikram.dixit] [~prasanth_j] can you take a look? > LLAP: update (add/remove) the UDFs available in LLAP when they are changed > (refresh periodically) > - > > Key: HIVE-12856 > URL: https://issues.apache.org/jira/browse/HIVE-12856 > Project: Hive > Issue Type: Sub-task >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > Attachments: HIVE-12856.01.nogen.patch, HIVE-12856.01.patch, > HIVE-12856.02.nogen.patch, HIVE-12856.02.patch, HIVE-12856.nogen.patch, > HIVE-12856.patch > > > I don't think re-querying the functions is going to scale, and the sessions > obviously cannot notify all LLAP clusters of every change. We should add > global versioning to metastore functions to track changes, and then possibly > add a notification mechanism, potentially thru ZK to avoid overloading the > metastore itself. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Comment Edited] (HIVE-12856) LLAP: update (add/remove) the UDFs available in LLAP when they are changed (refresh periodically)
[ https://issues.apache.org/jira/browse/HIVE-12856?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15152842#comment-15152842 ] Sergey Shelukhin edited comment on HIVE-12856 at 2/18/16 6:52 PM: -- [~vikram.dixit] [~prasanth_j] can you take a look at LLAP side of things? was (Author: sershe): [~vikram.dixit] [~prasanth_j] can you take a look? > LLAP: update (add/remove) the UDFs available in LLAP when they are changed > (refresh periodically) > - > > Key: HIVE-12856 > URL: https://issues.apache.org/jira/browse/HIVE-12856 > Project: Hive > Issue Type: Sub-task >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > Attachments: HIVE-12856.01.nogen.patch, HIVE-12856.01.patch, > HIVE-12856.02.nogen.patch, HIVE-12856.02.patch, HIVE-12856.nogen.patch, > HIVE-12856.patch > > > I don't think re-querying the functions is going to scale, and the sessions > obviously cannot notify all LLAP clusters of every change. We should add > global versioning to metastore functions to track changes, and then possibly > add a notification mechanism, potentially thru ZK to avoid overloading the > metastore itself. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13083) Writing HiveDecimal to ORC can wrongly suppress present stream
[
https://issues.apache.org/jira/browse/HIVE-13083?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15152805#comment-15152805
]
Sergey Shelukhin commented on HIVE-13083:
-
Ok for 2.0.1
> Writing HiveDecimal to ORC can wrongly suppress present stream
> --
>
> Key: HIVE-13083
> URL: https://issues.apache.org/jira/browse/HIVE-13083
> Project: Hive
> Issue Type: Bug
>Affects Versions: 0.13.0, 0.14.0, 1.0.0, 1.2.0, 1.1.0, 1.3.0, 2.0.0, 2.1.0
>Reporter: Yi Zhang
>Assignee: Prasanth Jayachandran
> Attachments: HIVE-13083-branch-1.patch, HIVE-13083.1.patch
>
>
> HIVE-3976 can cause ORC file to be unreadable. The changes introduced in
> HIVE-3976 for DecimalTreeWriter can create null values after updating the
> isPresent stream.
> https://github.com/apache/hive/blob/branch-0.13/ql/src/java/org/apache/hadoop/hive/ql/io/orc/WriterImpl.java#L1337
> As result of the above return statement, isPresent stream state can become
> wrong. The isPresent stream thinks all values are non-null and hence
> suppressed. But the data stream will be of 0 length. When reading such files
> we will get the following exception
> {code}
> Caused by: java.io.EOFException: Reading BigInteger past EOF from compressed
> stream Stream for column 3 kind DATA position: 0 length: 0 range: 0 offset: 0
> limit: 0
> at
> org.apache.hadoop.hive.ql.io.orc.SerializationUtils.readBigInteger(SerializationUtils.java:176)
> at
> org.apache.hadoop.hive.ql.io.orc.TreeReaderFactory$DecimalTreeReader.next(TreeReaderFactory.java:1264)
> at
> org.apache.hadoop.hive.ql.io.orc.TreeReaderFactory$StructTreeReader.next(TreeReaderFactory.java:2004)
> at
> org.apache.hadoop.hive.ql.io.orc.RecordReaderImpl.next(RecordReaderImpl.java:1039)
> ... 24 more
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-12808) Logical PPD: Push filter clauses through PTF(Windowing) into TS
[
https://issues.apache.org/jira/browse/HIVE-12808?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15152749#comment-15152749
]
Hive QA commented on HIVE-12808:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12788344/HIVE-12808.04.patch
{color:green}SUCCESS:{color} +1 due to 2 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 10 failed/errored test(s), 9815 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_partition_coltype_literals
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_subquery_in
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_subquery_unqualcolumnrefs
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_stats_only_null
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_subquery_in
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_authorization_uri_import
org.apache.hadoop.hive.cli.TestPerfCliDriver.testPerfCliDriver_query70
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_stats_only_null
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_subquery_in
org.apache.hive.jdbc.TestSSL.testSSLVersion
{noformat}
Test results:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/7020/testReport
Console output:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/7020/console
Test logs:
http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-7020/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 10 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12788344 - PreCommit-HIVE-TRUNK-Build
> Logical PPD: Push filter clauses through PTF(Windowing) into TS
> ---
>
> Key: HIVE-12808
> URL: https://issues.apache.org/jira/browse/HIVE-12808
> Project: Hive
> Issue Type: Bug
> Components: Logical Optimizer
>Affects Versions: 1.2.1, 2.0.0
>Reporter: Gopal V
>Assignee: Laljo John Pullokkaran
> Attachments: HIVE-12808.01.patch, HIVE-12808.02.patch,
> HIVE-12808.03.patch, HIVE-12808.04.patch
>
>
> Simplified repro case of [HCC
> #8880|https://community.hortonworks.com/questions/8880/hive-on-tez-pushdown-predicate-doesnt-work-in-part.html],
> with the slow query showing the push-down miss.
> And the manually rewritten query to indicate the expected one.
> Part of the problem could be the window range not being split apart for PPD,
> but the FIL is not pushed down even if the rownum filter is removed.
> {code}
> create temporary table positions (regionid string, id bigint, deviceid
> string, ts string);
> insert into positions values('1d6a0be1-6366-4692-9597-ebd5cd0f01d1',
> 1422792010, '6c5d1a30-2331-448b-a726-a380d6b3a432', '2016-01-01'),
> ('1d6a0be1-6366-4692-9597-ebd5cd0f01d1', 1422792010,
> '6c5d1a30-2331-448b-a726-a380d6b3a432', '2016-01-01'),
> ('1d6a0be1-6366-4692-9597-ebd5cd0f01d1', 1422792010,
> '6c5d1a30-2331-448b-a726-a380d6b3a432', '2016-01-02'),
> ('1d6a0be1-6366-4692-9597-ebd5cd0f01d1', 1422792010,
> '6c5d1a30-2331-448b-a726-a380d6b3a432', '2016-01-02');
> -- slow query
> explain
> WITH t1 AS
> (
> SELECT *,
> Row_number() over ( PARTITION BY regionid, id, deviceid
> ORDER BY ts DESC) AS rownos
> FROM positions ),
> latestposition as (
>SELECT *
>FROM t1
>WHERE rownos = 1)
> SELECT *
> FROM latestposition
> WHERE regionid='1d6a0be1-6366-4692-9597-ebd5cd0f01d1'
> ANDid=1422792010
> ANDdeviceid='6c5d1a30-2331-448b-a726-a380d6b3a432';
> -- fast query
> explain
> WITH t1 AS
> (
> SELECT *,
> Row_number() over ( PARTITION BY regionid, id, deviceid
> ORDER BY ts DESC) AS rownos
> FROM positions
> WHERE regionid='1d6a0be1-6366-4692-9597-ebd5cd0f01d1'
> ANDid=1422792010
> ANDdeviceid='6c5d1a30-2331-448b-a726-a380d6b3a432'
> ),latestposition as (
>SELECT *
>FROM t1
>WHERE rownos = 1)
> SELECT *
> FROM latestposition
> ;
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-12856) LLAP: update (add/remove) the UDFs available in LLAP when they are changed (refresh periodically)
[ https://issues.apache.org/jira/browse/HIVE-12856?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15152741#comment-15152741 ] Alan Gates commented on HIVE-12856: --- +1 on the metastore changes. I'll let others more familiar with LLAP review those portions. > LLAP: update (add/remove) the UDFs available in LLAP when they are changed > (refresh periodically) > - > > Key: HIVE-12856 > URL: https://issues.apache.org/jira/browse/HIVE-12856 > Project: Hive > Issue Type: Sub-task >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > Attachments: HIVE-12856.01.nogen.patch, HIVE-12856.01.patch, > HIVE-12856.02.nogen.patch, HIVE-12856.02.patch, HIVE-12856.nogen.patch, > HIVE-12856.patch > > > I don't think re-querying the functions is going to scale, and the sessions > obviously cannot notify all LLAP clusters of every change. We should add > global versioning to metastore functions to track changes, and then possibly > add a notification mechanism, potentially thru ZK to avoid overloading the > metastore itself. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-12927) HBase metastore: sequences should be one per row, not all in one row
[
https://issues.apache.org/jira/browse/HIVE-12927?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Alan Gates updated HIVE-12927:
--
Resolution: Fixed
Fix Version/s: 2.1.0
Status: Resolved (was: Patch Available)
Patch committed. Thanks Sergey for the review.
> HBase metastore: sequences should be one per row, not all in one row
>
>
> Key: HIVE-12927
> URL: https://issues.apache.org/jira/browse/HIVE-12927
> Project: Hive
> Issue Type: Bug
>Affects Versions: 2.0.0
>Reporter: Sergey Shelukhin
>Assignee: Alan Gates
>Priority: Critical
> Fix For: 2.1.0
>
> Attachments: HIVE-12927.2.patch, HIVE-12927.patch
>
>
> {noformat}
> long getNextSequence(byte[] sequence) throws IOException {
> {noformat}
> Is not safe in presence of any concurrency. It should use HBase increment API.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-12165) wrong result when hive.optimize.sampling.orderby=true with some aggregate functions
[
https://issues.apache.org/jira/browse/HIVE-12165?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15152560#comment-15152560
]
Hive QA commented on HIVE-12165:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12788323/HIVE-12165.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 7 failed/errored test(s), 9790 tests executed
*Failed tests:*
{noformat}
TestMiniTezCliDriver-vector_distinct_2.q-load_dyn_part2.q-bucket3.q-and-12-more
- did not produce a TEST-*.xml file
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_partition_coltype_literals
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_stats_only_null
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_authorization_uri_import
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_stats_only_null
org.apache.hive.jdbc.TestJdbcWithMiniHS2.testAddJarDataNucleusUnCaching
org.apache.hive.jdbc.TestSSL.testSSLVersion
{noformat}
Test results:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/7019/testReport
Console output:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/7019/console
Test logs:
http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-7019/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 7 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12788323 - PreCommit-HIVE-TRUNK-Build
> wrong result when hive.optimize.sampling.orderby=true with some aggregate
> functions
> ---
>
> Key: HIVE-12165
> URL: https://issues.apache.org/jira/browse/HIVE-12165
> Project: Hive
> Issue Type: Bug
>Affects Versions: 2.1.0
> Environment: hortonworks 2.3
>Reporter: ErwanMAS
>Assignee: Aihua Xu
>Priority: Critical
> Attachments: HIVE-12165.patch
>
>
> This simple query give wrong result , when , i use the parallel order .
> {noformat}
> select count(*) , count(distinct dummyint ) , min(dummyint),max(dummyint)
> from foobar_1M ;
> {noformat}
> Current wrong result :
> {noformat}
> c0c1 c2 c3
> 32740 32740 0 163695
> 113172113172 163700 729555
> 54088 54088 729560 95
> {noformat}
> Right result :
> {noformat}
> c0c1 c2 c3
> 100 100 0 99
> {noformat}
> The sql script for my test
> {noformat}
> drop table foobar_1 ;
> create table foobar_1 ( dummyint int , dummystr string ) ;
> insert into table foobar_1 select count(*),'dummy 0' from foobar_1 ;
> drop table foobar_1M ;
> create table foobar_1M ( dummyint bigint , dummystr string ) ;
> insert overwrite table foobar_1M
>select val_int , concat('dummy ',val_int) from
> ( select ((d_1*10)+d_2)*10+d_3)*10+d_4)*10+d_5)*10+d_6) as
> val_int from foobar_1
> lateral view outer explode(split("0,1,2,3,4,5,6,7,8,9",","))
> tbl_1 as d_1
> lateral view outer explode(split("0,1,2,3,4,5,6,7,8,9",","))
> tbl_2 as d_2
> lateral view outer explode(split("0,1,2,3,4,5,6,7,8,9",","))
> tbl_3 as d_3
> lateral view outer explode(split("0,1,2,3,4,5,6,7,8,9",","))
> tbl_4 as d_4
> lateral view outer explode(split("0,1,2,3,4,5,6,7,8,9",","))
> tbl_5 as d_5
> lateral view outer explode(split("0,1,2,3,4,5,6,7,8,9",","))
> tbl_6 as d_6 ) as f ;
> set hive.optimize.sampling.orderby.number=1;
> set hive.optimize.sampling.orderby.percent=0.1f;
> set mapreduce.job.reduces=3 ;
> set hive.optimize.sampling.orderby=false;
> select count(*) , count(distinct dummyint ) , min(dummyint),max(dummyint)
> from foobar_1M ;
> set hive.optimize.sampling.orderby=true;
> select count(*) , count(distinct dummyint ) , min(dummyint),max(dummyint)
> from foobar_1M ;
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-13048) Rogue SQL statement in an upgrade SQL file for oracle.
[ https://issues.apache.org/jira/browse/HIVE-13048?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15152542#comment-15152542 ] Naveen Gangam commented on HIVE-13048: -- Thank you Sergey!!! > Rogue SQL statement in an upgrade SQL file for oracle. > -- > > Key: HIVE-13048 > URL: https://issues.apache.org/jira/browse/HIVE-13048 > Project: Hive > Issue Type: Bug >Reporter: Naveen Gangam >Assignee: Naveen Gangam > Fix For: 2.1.0 > > Attachments: HIVE-13048.patch > > > metastore/scripts/upgrade/oracle/033-HIVE-12892.oracle.sql has > VERSION_COMMENT VARCHAR(255) NOT NULL > CREATE TABLE CHANGE_VERSION ( > CHANGE_VERSION_ID NUMBER NOT NULL, > VERSION NUMBER NOT NULL, > TOPIC VARCHAR(255) NOT NULL > ); > ALTER TABLE CHANGE_VERSION ADD CONSTRAINT CHANGE_VERSION_PK PRIMARY KEY > (CHANGE_VERSION_ID); > CREATE UNIQUE INDEX UNIQUE_CHANGE_VERSION ON CHANGE_VERSION (TOPIC); > The first line appears to be a typo and should not really be there. I noticed > that in the METASTORE-Test precommit builds. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13055) Add unit tests for HIVE-11512
[ https://issues.apache.org/jira/browse/HIVE-13055?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15152536#comment-15152536 ] Naveen Gangam commented on HIVE-13055: -- Thanks Aihua. > Add unit tests for HIVE-11512 > - > > Key: HIVE-13055 > URL: https://issues.apache.org/jira/browse/HIVE-13055 > Project: Hive > Issue Type: Test > Components: HiveServer2 >Affects Versions: 1.3.0 >Reporter: Naveen Gangam >Assignee: Naveen Gangam > Fix For: 2.1.0 > > Attachments: HIVE-13055.patch > > > HIVE-11866 adds the ability to test LDAP authenticator with a LDAP Server. We > need more test coverage for feature in HIVE-11512. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13039) BETWEEN predicate is not functioning correctly with predicate pushdown on Parquet table
[ https://issues.apache.org/jira/browse/HIVE-13039?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15152510#comment-15152510 ] Yongzhi Chen commented on HIVE-13039: - Yes, only for branch-1 > BETWEEN predicate is not functioning correctly with predicate pushdown on > Parquet table > --- > > Key: HIVE-13039 > URL: https://issues.apache.org/jira/browse/HIVE-13039 > Project: Hive > Issue Type: Bug > Components: Physical Optimizer >Affects Versions: 1.2.1, 2.0.0 >Reporter: Yongzhi Chen >Assignee: Yongzhi Chen > Fix For: 1.3.0, 2.1.0 > > Attachments: HIVE-13039.1.branch1.txt, HIVE-13039.1.patch, > HIVE-13039.2.branch-1.txt, HIVE-13039.2.patch, HIVE-13039.3.patch > > > BETWEEN becomes exclusive in parquet table when predicate pushdown is on (as > it is by default in newer Hive versions). To reproduce(in a cluster, not > local setup): > CREATE TABLE parquet_tbl( > key int, > ldate string) > PARTITIONED BY ( > lyear string ) > ROW FORMAT SERDE > 'org.apache.hadoop.hive.ql.io.parquet.serde.ParquetHiveSerDe' > STORED AS INPUTFORMAT > 'org.apache.hadoop.hive.ql.io.parquet.MapredParquetInputFormat' > OUTPUTFORMAT > 'org.apache.hadoop.hive.ql.io.parquet.MapredParquetOutputFormat'; > insert overwrite table parquet_tbl partition (lyear='2016') select > 1, > '2016-02-03' from src limit 1; > set hive.optimize.ppd.storage = true; > set hive.optimize.ppd = true; > select * from parquet_tbl where ldate between '2016-02-03' and '2016-02-03'; > No row will be returned in a cluster. > But if you turn off hive.optimize.ppd, one row will be returned. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13050) The row count is not correct after changing partition location to point to another partition location
[
https://issues.apache.org/jira/browse/HIVE-13050?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15152476#comment-15152476
]
Aihua Xu commented on HIVE-13050:
-
Seems we assume one partition for each location when we submit a MR job. Change
it to improvement rather than bug.
> The row count is not correct after changing partition location to point to
> another partition location
> -
>
> Key: HIVE-13050
> URL: https://issues.apache.org/jira/browse/HIVE-13050
> Project: Hive
> Issue Type: Improvement
> Components: Query Processor
>Affects Versions: 2.1.0
>Reporter: Aihua Xu
>Assignee: Aihua Xu
>
> {noformat}
> CREATE TABLE test (s STRING) PARTITIONED BY (p SMALLINT) location
> 'data/test';
> INSERT INTO test PARTITION (`p`=1) VALUES ("v1");
> INSERT INTO test PARTITION (`p`=2) VALUES ("v2");
> ALTER TABLE test PARTITION (`p`=2) SET LOCATION '/data/test/p=1';
> {noformat}
> {{select * from test;}} shows 2 rows while {{SELECT count(*) FROM test;}}
> shows 1.
> That is inconsistent.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-13050) The row count is not correct after changing partition location to point to another partition location
[
https://issues.apache.org/jira/browse/HIVE-13050?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Aihua Xu updated HIVE-13050:
Issue Type: Improvement (was: Bug)
> The row count is not correct after changing partition location to point to
> another partition location
> -
>
> Key: HIVE-13050
> URL: https://issues.apache.org/jira/browse/HIVE-13050
> Project: Hive
> Issue Type: Improvement
> Components: Query Processor
>Affects Versions: 2.1.0
>Reporter: Aihua Xu
>Assignee: Aihua Xu
>
> {noformat}
> CREATE TABLE test (s STRING) PARTITIONED BY (p SMALLINT) location
> 'data/test';
> INSERT INTO test PARTITION (`p`=1) VALUES ("v1");
> INSERT INTO test PARTITION (`p`=2) VALUES ("v2");
> ALTER TABLE test PARTITION (`p`=2) SET LOCATION '/data/test/p=1';
> {noformat}
> {{select * from test;}} shows 2 rows while {{SELECT count(*) FROM test;}}
> shows 1.
> That is inconsistent.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-13055) Add unit tests for HIVE-11512
[ https://issues.apache.org/jira/browse/HIVE-13055?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aihua Xu updated HIVE-13055: Resolution: Fixed Fix Version/s: 2.1.0 Status: Resolved (was: Patch Available) Pushed to master. Thanks Naveen for your contribution. > Add unit tests for HIVE-11512 > - > > Key: HIVE-13055 > URL: https://issues.apache.org/jira/browse/HIVE-13055 > Project: Hive > Issue Type: Test > Components: HiveServer2 >Affects Versions: 1.3.0 >Reporter: Naveen Gangam >Assignee: Naveen Gangam > Fix For: 2.1.0 > > Attachments: HIVE-13055.patch > > > HIVE-11866 adds the ability to test LDAP authenticator with a LDAP Server. We > need more test coverage for feature in HIVE-11512. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (HIVE-4413) Parse Exception : character '@' not supported while granting privileges to user in a Secure Cluster through hive client.
[ https://issues.apache.org/jira/browse/HIVE-4413?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bolke de Bruin resolved HIVE-4413. -- Resolution: Not A Problem Apologies, quoting works fine with ` > Parse Exception : character '@' not supported while granting privileges to > user in a Secure Cluster through hive client. > > > Key: HIVE-4413 > URL: https://issues.apache.org/jira/browse/HIVE-4413 > Project: Hive > Issue Type: Bug > Components: CLI >Affects Versions: 0.10.0, 0.14.0, 1.2.0, 1.2.1, 2.0.0 >Reporter: Navin Madathil >Assignee: Bolke de Bruin > Labels: cli, hive > > While running through hive CLI , hive grant command throws a parseException > '@' not supported. But in a secure cluster ( Kerberos ) the username is > appended with the realmname seperated by the character '@'.Without giving the > full username the permissions are not granted to the intended user. > "grant all on table tablename to user user@REALM" -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-12077) MSCK Repair table should fix partitions in batches
[ https://issues.apache.org/jira/browse/HIVE-12077?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chinna Rao Lalam updated HIVE-12077: Attachment: HIVE-12077.1.patch > MSCK Repair table should fix partitions in batches > --- > > Key: HIVE-12077 > URL: https://issues.apache.org/jira/browse/HIVE-12077 > Project: Hive > Issue Type: Bug > Components: Hive >Reporter: Ryan P > Attachments: HIVE-12077.1.patch > > > If a user attempts to run MSCK REPAIR TABLE on a directory with a large > number of untracked partitions HMS will OOME. I suspect this is because it > attempts to do one large bulk load in an effort to save time. Ultimately this > can lead to a collection so large in size that HMS eventually hits an Out of > Memory Exception. > Instead I suggest that Hive include a configurable batch size that HMS can > use to break up the load. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Assigned] (HIVE-12077) MSCK Repair table should fix partitions in batches
[ https://issues.apache.org/jira/browse/HIVE-12077?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chinna Rao Lalam reassigned HIVE-12077: --- Assignee: Chinna Rao Lalam > MSCK Repair table should fix partitions in batches > --- > > Key: HIVE-12077 > URL: https://issues.apache.org/jira/browse/HIVE-12077 > Project: Hive > Issue Type: Bug > Components: Hive >Reporter: Ryan P >Assignee: Chinna Rao Lalam > Attachments: HIVE-12077.1.patch > > > If a user attempts to run MSCK REPAIR TABLE on a directory with a large > number of untracked partitions HMS will OOME. I suspect this is because it > attempts to do one large bulk load in an effort to save time. Ultimately this > can lead to a collection so large in size that HMS eventually hits an Out of > Memory Exception. > Instead I suggest that Hive include a configurable batch size that HMS can > use to break up the load. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-335) External Tables should have the option to be marked Read Only
[ https://issues.apache.org/jira/browse/HIVE-335?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15152337#comment-15152337 ] Matt Pollock commented on HIVE-335: --- My organization has lots of data producing users/accounts and not all data producers are hive users. It would be extremely valuable to be able to create a read-only external table to avoid data copy and enable flexible query on data the consumer did not produce but has read access to. > External Tables should have the option to be marked Read Only > - > > Key: HIVE-335 > URL: https://issues.apache.org/jira/browse/HIVE-335 > Project: Hive > Issue Type: Improvement > Components: Metastore, Query Processor >Reporter: Richard Lee > > When creating an External Table, it'd be awesome to have the option of NOT > allowing writes to it (disallow any INSERTs or if hive ever allows UPDATEs). > Adding and Dropping Partitions should still be allowed. > This will enable hive to "play well" with external data stores other than > hdfs where data should be non-maleable. > I'd recomend the following syntax, which applies ONLY to external tables: > CREATE EXTERNAL [READONLY] TABLE ... -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13040) Handle empty bucket creations more efficiently
[
https://issues.apache.org/jira/browse/HIVE-13040?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15152311#comment-15152311
]
Hive QA commented on HIVE-13040:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12788298/HIVE-13040.4.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 39 failed/errored test(s), 9730 tests
executed
*Failed tests:*
{noformat}
TestMiniTezCliDriver-vector_decimal_2.q-schema_evol_text_fetchwork_table.q-vector_null_projection.q-and-12-more
- did not produce a TEST-*.xml file
TestSparkCliDriver-groupby3_map.q-sample2.q-auto_join14.q-and-12-more - did not
produce a TEST-*.xml file
TestSparkCliDriver-groupby_map_ppr_multi_distinct.q-table_access_keys_stats.q-groupby4_noskew.q-and-12-more
- did not produce a TEST-*.xml file
TestSparkCliDriver-join_rc.q-insert1.q-vectorized_rcfile_columnar.q-and-12-more
- did not produce a TEST-*.xml file
TestSparkCliDriver-ppd_join4.q-join9.q-ppd_join3.q-and-12-more - did not
produce a TEST-*.xml file
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_cbo_const
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_decimal_precision
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_groupby_neg_float
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_groupby_sort_11
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_lateral_view_explode2
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_limit_partition_metadataonly
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_metadataOnlyOptimizer
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_metadataonly1
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_min_structvalue
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_partInit
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_partition_date
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_partition_date2
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_partition_timestamp
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_partition_timestamp2
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_partition_varchar1
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_unionDistinct_1
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_vector_decimal_precision
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_vector_null_projection
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_vector_outer_join1
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_vector_outer_join4
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_smb_cache
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_union_fast_stats
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_authorization_uri_import
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_lateral_view_explode2
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_limit_partition_metadataonly
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_stats_only_null
org.apache.hadoop.hive.ql.TestTxnCommands2.testNonAcidToAcidConversionAndMajorCompaction
org.apache.hadoop.hive.ql.io.TestAcidUtils.testObsoleteOriginals
org.apache.hadoop.hive.ql.io.TestAcidUtils.testOriginal
org.apache.hadoop.hive.ql.io.TestAcidUtils.testOriginalDeltas
org.apache.hadoop.hive.ql.io.orc.TestInputOutputFormat.testEtlCombinedStrategy
org.apache.hadoop.hive.ql.io.orc.TestInputOutputFormat.testFileGenerator
org.apache.hadoop.hive.ql.io.orc.TestInputOutputFormat.testSplitGenFailure
org.apache.hive.jdbc.TestSSL.testSSLVersion
{noformat}
Test results:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/7018/testReport
Console output:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/7018/console
Test logs:
http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-7018/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 39 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12788298 - PreCommit-HIVE-TRUNK-Build
> Handle empty bucket creations more efficiently
> ---
>
> Key: HIVE-13040
> URL: https://issues.apache.org/jira/browse/HIVE-13040
> Project: Hive
> Issue Type: Improvement
> Components: Query Processor
>Affects Versions: 1.0.0, 1.2.0, 1.1.0, 2.0.0
>Reporter: Ashutosh Chauhan
>Assignee: Ashutosh Chauhan
> Attachments: HIVE-13040.2.patch, HIVE-13040.3.patch,
> HIVE-13040.4.patch, HIVE-13040.patch
[jira] [Updated] (HIVE-4413) Parse Exception : character '@' not supported while granting privileges to user in a Secure Cluster through hive client.
[ https://issues.apache.org/jira/browse/HIVE-4413?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bolke de Bruin updated HIVE-4413: - Affects Version/s: 0.14.0 1.2.0 1.2.1 2.0.0 > Parse Exception : character '@' not supported while granting privileges to > user in a Secure Cluster through hive client. > > > Key: HIVE-4413 > URL: https://issues.apache.org/jira/browse/HIVE-4413 > Project: Hive > Issue Type: Bug > Components: CLI >Affects Versions: 0.10.0, 0.14.0, 1.2.0, 1.2.1, 2.0.0 >Reporter: Navin Madathil >Assignee: Bolke de Bruin > Labels: cli, hive > > While running through hive CLI , hive grant command throws a parseException > '@' not supported. But in a secure cluster ( Kerberos ) the username is > appended with the realmname seperated by the character '@'.Without giving the > full username the permissions are not granted to the intended user. > "grant all on table tablename to user user@REALM" -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13066) Hive on Spark gives incorrect results when speculation is on
[ https://issues.apache.org/jira/browse/HIVE-13066?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15152252#comment-15152252 ] Rui Li commented on HIVE-13066: --- I'm not able to reproduce the issue. But I tried to make the task fail if the output file already exists. And speculative task fails as expected, which means it's possible that speculative tasks write to the same file as original tasks and corrupt the results. Therefore I think we should append the real attempt ID for each task, if that doesn't cause too much trouble. Meanwhile we still need a way to verify the root cause of the issue. Any ideas? > Hive on Spark gives incorrect results when speculation is on > > > Key: HIVE-13066 > URL: https://issues.apache.org/jira/browse/HIVE-13066 > Project: Hive > Issue Type: Bug > Components: Spark >Reporter: Rui Li >Assignee: Rui Li > > The issue is reported by users. One possible reason is that we always append > 0 as the attempt ID for each task so that hive won't be able to distinguish > between speculative tasks and original ones. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Reopened] (HIVE-4413) Parse Exception : character '@' not supported while granting privileges to user in a Secure Cluster through hive client.
[ https://issues.apache.org/jira/browse/HIVE-4413?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bolke de Bruin reopened HIVE-4413: -- Assignee: Bolke de Bruin HADOOP-12751 is aiming to remove the limitation on characters within a username, eg. '@' will be allowed in a username. This is important for interoperability across different trusted domains which create usernames with their own realm included. The proper fix for this issue should be to allow these characters in a username, possibly by escaping them or quoted. grant all on table tablename to user bolke\@ad.domain grant all on table tablename to user "bolke@ad.domain" > Parse Exception : character '@' not supported while granting privileges to > user in a Secure Cluster through hive client. > > > Key: HIVE-4413 > URL: https://issues.apache.org/jira/browse/HIVE-4413 > Project: Hive > Issue Type: Bug > Components: CLI >Affects Versions: 0.10.0 >Reporter: Navin Madathil >Assignee: Bolke de Bruin > Labels: cli, hive > > While running through hive CLI , hive grant command throws a parseException > '@' not supported. But in a secure cluster ( Kerberos ) the username is > appended with the realmname seperated by the character '@'.Without giving the > full username the permissions are not granted to the intended user. > "grant all on table tablename to user user@REALM" -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13074) NPE with HS2 Web UI when SQLOperationDisplay is null
[
https://issues.apache.org/jira/browse/HIVE-13074?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15152112#comment-15152112
]
Hive QA commented on HIVE-13074:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12788245/HIVE-13074.1.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 5 failed/errored test(s), 9805 tests executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_partition_coltype_literals
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_stats_only_null
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_authorization_uri_import
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_stats_only_null
org.apache.hive.jdbc.TestSSL.testSSLVersion
{noformat}
Test results:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/7017/testReport
Console output:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/7017/console
Test logs:
http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-7017/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 5 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12788245 - PreCommit-HIVE-TRUNK-Build
> NPE with HS2 Web UI when SQLOperationDisplay is null
>
>
> Key: HIVE-13074
> URL: https://issues.apache.org/jira/browse/HIVE-13074
> Project: Hive
> Issue Type: Bug
> Components: HiveServer2
>Reporter: Shinichi Yamashita
>Assignee: Shinichi Yamashita
> Attachments: HIVE-13074.1.patch
>
>
> In current master, I executed the following command,
> {code}
> 0: jdbc:hive2://XXX:1/> set hive.execution.engine;
> {code}
> Next, I accessed Web page of HS2. And the following messages were displayed.
> {code}
> HTTP ERROR 500
> Problem accessing /hiveserver2.jsp. Reason:
> Server Error
> Caused by:
> java.lang.NullPointerException
> at
> org.apache.hive.generated.hiveserver2.hiveserver2_jsp._jspService(hiveserver2_jsp.java:131)
> at org.apache.jasper.runtime.HttpJspBase.service(HttpJspBase.java:98)
> at javax.servlet.http.HttpServlet.service(HttpServlet.java:848)
> at
> org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:565)
> at
> org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:479)
> (snip)
> {code}
> I think {{SQLOperationDisplay}} needs the handle when being null.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-13085) Need an API / configuration parameter to find out the authenticated user from beeline
[ https://issues.apache.org/jira/browse/HIVE-13085?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Oleksiy Sayankin updated HIVE-13085: Fix Version/s: 2.0.0 > Need an API / configuration parameter to find out the authenticated user > from beeline > -- > > Key: HIVE-13085 > URL: https://issues.apache.org/jira/browse/HIVE-13085 > Project: Hive > Issue Type: Improvement >Reporter: Oleksiy Sayankin >Assignee: Oleksiy Sayankin > Fix For: 2.0.0 > > Attachments: HIVE-13085.1.patch > > > This request has come from the customer who wanted an API / configuration > parameter to identify the authenticated user from the beeline. This is > similar to the request done in the thread below: > https://community.hortonworks.com/questions/2620/hadoop-environment-variable-or-configuration-varia.html > But this would not be a feasible option for the requestor. So here the > general ask is once the user is logged in to the beeline, they wanted to > identify who this user is and then use this information to enforce the ACLs > on the tables through the customer's custom code. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13085) Need an API / configuration parameter to find out the authenticated user from beeline
[
https://issues.apache.org/jira/browse/HIVE-13085?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15152103#comment-15152103
]
Oleksiy Sayankin commented on HIVE-13085:
-
*Added {{!username}} command to beeline CLI.*
1. If there is no connection to database
{noformat}
beeline> !username
{noformat}
returns nothing.
2 If user successfully connected to HiveServer2 {{!username}} returns logged in
user:
{noformat}
beeline> !connect jdbc:hive2://localhost:1
Connecting to jdbc:hive2://localhost:1
Enter username for jdbc:hive2://localhost:1: testuser2
Enter password for jdbc:hive2://localhost:1: *
Connected to: Apache Hive
Driver: Hive JDBC
Transaction isolation: TRANSACTION_REPEATABLE_READ
0: jdbc:hive2://localhost:1> !username
testuser2
{noformat}
3. If there is two connections, {{!username}} returns logged in user for
current connection (#0 in our case):
{noformat}
0: jdbc:hive2://localhost:1> !list
2 active connections:
#0 open jdbc:hive2://localhost:1
#1 open jdbc:hive2://localhost:1
0: jdbc:hive2://localhost:1> !username
test
{noformat}
4. If we change current connection, {{!username}} returns new value:
{noformat}
0: jdbc:hive2://localhost:1> !go 1
1: jdbc:hive2://localhost:1> !username
testuser1
{noformat}
5. If there are two consoles at the same cluster node with different sets of
connections, then the results of {{!username}} is independent each other
console, that is we have two {{Beeline.class}} instances and each one has its
own current connection.
> Need an API / configuration parameter to find out the authenticated user
> from beeline
> --
>
> Key: HIVE-13085
> URL: https://issues.apache.org/jira/browse/HIVE-13085
> Project: Hive
> Issue Type: Improvement
>Reporter: Oleksiy Sayankin
>Assignee: Oleksiy Sayankin
> Attachments: HIVE-13085.1.patch
>
>
> This request has come from the customer who wanted an API / configuration
> parameter to identify the authenticated user from the beeline. This is
> similar to the request done in the thread below:
> https://community.hortonworks.com/questions/2620/hadoop-environment-variable-or-configuration-varia.html
> But this would not be a feasible option for the requestor. So here the
> general ask is once the user is logged in to the beeline, they wanted to
> identify who this user is and then use this information to enforce the ACLs
> on the tables through the customer's custom code.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
