[jira] [Updated] (HIVE-14113) Create function failed but function in show function list
[
https://issues.apache.org/jira/browse/HIVE-14113?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Rajat Khandelwal updated HIVE-14113:
Assignee: Navis (was: niklaus xiao)
> Create function failed but function in show function list
> -
>
> Key: HIVE-14113
> URL: https://issues.apache.org/jira/browse/HIVE-14113
> Project: Hive
> Issue Type: Bug
> Components: UDF
>Affects Versions: 1.2.0
>Reporter: niklaus xiao
>Assignee: Navis
> Fix For: 1.3.0
>
> Attachments: HIVE-14113.1.patch
>
>
> 1. create function with invalid hdfs path, /udf/udf-test.jar does not exists
> {quote}
> create function my_lower as 'com.tang.UDFLower' using jar
> 'hdfs:///udf/udf-test.jar';
> {quote}
> Failed with following exception:
> {quote}
> 0: jdbc:hive2://189.39.151.44:1/> create function my_lower as
> 'com.tang.UDFLower' using jar 'hdfs:///udf/udf-test.jar';
> INFO : converting to local hdfs:///udf/udf-test.jar
> ERROR : Failed to read external resource hdfs:///udf/udf-test.jar
> java.lang.RuntimeException: Failed to read external resource
> hdfs:///udf/udf-test.jar
> at
> org.apache.hadoop.hive.ql.session.SessionState.downloadResource(SessionState.java:1384)
> at
> org.apache.hadoop.hive.ql.session.SessionState.resolveAndDownload(SessionState.java:1340)
> at
> org.apache.hadoop.hive.ql.session.SessionState.add_resources(SessionState.java:1264)
> at
> org.apache.hadoop.hive.ql.session.SessionState.add_resources(SessionState.java:1250)
> at
> org.apache.hadoop.hive.ql.exec.FunctionTask.addFunctionResources(FunctionTask.java:306)
> at
> org.apache.hadoop.hive.ql.exec.Registry.registerToSessionRegistry(Registry.java:466)
> at
> org.apache.hadoop.hive.ql.exec.Registry.registerPermanentFunction(Registry.java:206)
> at
> org.apache.hadoop.hive.ql.exec.FunctionRegistry.registerPermanentFunction(FunctionRegistry.java:1551)
> at
> org.apache.hadoop.hive.ql.exec.FunctionTask.createPermanentFunction(FunctionTask.java:136)
> at
> org.apache.hadoop.hive.ql.exec.FunctionTask.execute(FunctionTask.java:75)
> at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:158)
> at
> org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:101)
> at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:1965)
> at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1723)
> at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1475)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1283)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1278)
> at
> org.apache.hive.service.cli.operation.SQLOperation.runQuery(SQLOperation.java:167)
> at
> org.apache.hive.service.cli.operation.SQLOperation.access$200(SQLOperation.java:75)
> at
> org.apache.hive.service.cli.operation.SQLOperation$1$1.run(SQLOperation.java:245)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1711)
> at
> org.apache.hive.service.cli.operation.SQLOperation$1.run(SQLOperation.java:258)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: java.io.FileNotFoundException: File does not exist:
> hdfs:/udf/udf-test.jar
> at
> org.apache.hadoop.hdfs.DistributedFileSystem$25.doCall(DistributedFileSystem.java:1391)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem$25.doCall(DistributedFileSystem.java:1383)
> at
> org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1383)
> at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:340)
> at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:292)
> at org.apache.hadoop.fs.FileSystem.copyToLocalFile(FileSystem.java:2034)
> at org.apache.hadoop.fs.FileSystem.copyToLocalFile(FileSystem.java:2003)
> at org.apache.hadoop.fs.FileSystem.copyToLocalFile(FileSystem.java:1979)
> at
> org.apache.hadoop.hive.ql.session.SessionState.downloadResource(SessionState.java:1370)
> ... 28 more
> ERROR : Failed to register default.my_lower using class com.tang.UDFLower
> Error: Error while processing statement: FAILED: Execution Error, return code
> 1 from or
[jira] [Updated] (HIVE-14113) Create function failed but function in show function list
[
https://issues.apache.org/jira/browse/HIVE-14113?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
niklaus xiao updated HIVE-14113:
Description:
1. create function with invalid hdfs path, /udf/udf-test.jar does not exists
{quote}
create function my_lower as 'com.tang.UDFLower' using jar
'hdfs:///udf/udf-test.jar';
{quote}
Failed with following exception:
{quote}
0: jdbc:hive2://189.39.151.44:1/> create function my_lower as
'com.tang.UDFLower' using jar 'hdfs:///udf/udf-test.jar';
INFO : converting to local hdfs:///udf/udf-test.jar
ERROR : Failed to read external resource hdfs:///udf/udf-test.jar
java.lang.RuntimeException: Failed to read external resource
hdfs:///udf/udf-test.jar
at
org.apache.hadoop.hive.ql.session.SessionState.downloadResource(SessionState.java:1384)
at
org.apache.hadoop.hive.ql.session.SessionState.resolveAndDownload(SessionState.java:1340)
at
org.apache.hadoop.hive.ql.session.SessionState.add_resources(SessionState.java:1264)
at
org.apache.hadoop.hive.ql.session.SessionState.add_resources(SessionState.java:1250)
at
org.apache.hadoop.hive.ql.exec.FunctionTask.addFunctionResources(FunctionTask.java:306)
at
org.apache.hadoop.hive.ql.exec.Registry.registerToSessionRegistry(Registry.java:466)
at
org.apache.hadoop.hive.ql.exec.Registry.registerPermanentFunction(Registry.java:206)
at
org.apache.hadoop.hive.ql.exec.FunctionRegistry.registerPermanentFunction(FunctionRegistry.java:1551)
at
org.apache.hadoop.hive.ql.exec.FunctionTask.createPermanentFunction(FunctionTask.java:136)
at
org.apache.hadoop.hive.ql.exec.FunctionTask.execute(FunctionTask.java:75)
at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:158)
at
org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:101)
at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:1965)
at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1723)
at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1475)
at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1283)
at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1278)
at
org.apache.hive.service.cli.operation.SQLOperation.runQuery(SQLOperation.java:167)
at
org.apache.hive.service.cli.operation.SQLOperation.access$200(SQLOperation.java:75)
at
org.apache.hive.service.cli.operation.SQLOperation$1$1.run(SQLOperation.java:245)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1711)
at
org.apache.hive.service.cli.operation.SQLOperation$1.run(SQLOperation.java:258)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.io.FileNotFoundException: File does not exist:
hdfs:/udf/udf-test.jar
at
org.apache.hadoop.hdfs.DistributedFileSystem$25.doCall(DistributedFileSystem.java:1391)
at
org.apache.hadoop.hdfs.DistributedFileSystem$25.doCall(DistributedFileSystem.java:1383)
at
org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
at
org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1383)
at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:340)
at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:292)
at org.apache.hadoop.fs.FileSystem.copyToLocalFile(FileSystem.java:2034)
at org.apache.hadoop.fs.FileSystem.copyToLocalFile(FileSystem.java:2003)
at org.apache.hadoop.fs.FileSystem.copyToLocalFile(FileSystem.java:1979)
at
org.apache.hadoop.hive.ql.session.SessionState.downloadResource(SessionState.java:1370)
... 28 more
ERROR : Failed to register default.my_lower using class com.tang.UDFLower
Error: Error while processing statement: FAILED: Execution Error, return code 1
from org.apache.hadoop.hive.ql.exec.FunctionTask (state=08S01,code=1)
{quote}
2. Execute show functions, the failed function my_lower is in the function list
{quote}
0: jdbc:hive2://189.39.151.44:21066/> show functions;
+-+--+
|tab_name |
+-+--+
| day |
| dayofmonth |
| decode |
| default.my_lower |
| degrees |
| dense_rank |
0: jdbc:hive2://189.39.151.44:1/> select my_lower(name) from stu;
Error: Error while compiling statement: FAILED: Sem
[jira] [Updated] (HIVE-14115) Custom FetchFormatter is not supported
[
https://issues.apache.org/jira/browse/HIVE-14115?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ryu Kobayashi updated HIVE-14115:
-
Status: Patch Available (was: Open)
> Custom FetchFormatter is not supported
> --
>
> Key: HIVE-14115
> URL: https://issues.apache.org/jira/browse/HIVE-14115
> Project: Hive
> Issue Type: Bug
>Affects Versions: 2.1.0
>Reporter: Ryu Kobayashi
>Assignee: Ryu Kobayashi
>Priority: Minor
> Attachments: HIVE-14115.01.patch
>
>
> The following code is supported only FetchFormatter of ThriftFormatter and
> DefaultFetchFormatter. It can not be used Custom FetchFormatter.
> {code}
> if (SessionState.get().isHiveServerQuery()) {
>
> conf.set(SerDeUtils.LIST_SINK_OUTPUT_FORMATTER,ThriftFormatter.class.getName());
> } else {
> conf.set(SerDeUtils.LIST_SINK_OUTPUT_FORMATTER,
> DefaultFetchFormatter.class.getName());
> }
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-14115) Custom FetchFormatter is not supported
[
https://issues.apache.org/jira/browse/HIVE-14115?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ryu Kobayashi updated HIVE-14115:
-
Attachment: HIVE-14115.01.patch
> Custom FetchFormatter is not supported
> --
>
> Key: HIVE-14115
> URL: https://issues.apache.org/jira/browse/HIVE-14115
> Project: Hive
> Issue Type: Bug
>Affects Versions: 2.1.0
>Reporter: Ryu Kobayashi
>Assignee: Ryu Kobayashi
>Priority: Minor
> Attachments: HIVE-14115.01.patch
>
>
> The following code is supported only FetchFormatter of ThriftFormatter and
> DefaultFetchFormatter. It can not be used Custom FetchFormatter.
> {code}
> if (SessionState.get().isHiveServerQuery()) {
>
> conf.set(SerDeUtils.LIST_SINK_OUTPUT_FORMATTER,ThriftFormatter.class.getName());
> } else {
> conf.set(SerDeUtils.LIST_SINK_OUTPUT_FORMATTER,
> DefaultFetchFormatter.class.getName());
> }
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-13560) Adding Omid as connection manager for HBase Metastore
[
https://issues.apache.org/jira/browse/HIVE-13560?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15352380#comment-15352380
]
Hive QA commented on HIVE-13560:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12813962/HIVE-13560.6.patch
{color:green}SUCCESS:{color} +1 due to 7 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 84 failed/errored test(s), 9850 tests
executed
*Failed tests:*
{noformat}
TestBeelineArgParsing - did not produce a TEST-*.xml file
TestGenMapRedUtilsUsePartitionColumnsPositive - did not produce a TEST-*.xml
file
TestHCatDynamicPartitioned - did not produce a TEST-*.xml file
TestHCatExternalDynamicPartitioned - did not produce a TEST-*.xml file
TestHCatExternalNonPartitioned - did not produce a TEST-*.xml file
TestHCatExternalPartitioned - did not produce a TEST-*.xml file
TestHCatLoader - did not produce a TEST-*.xml file
TestHCatLoaderComplexSchema - did not produce a TEST-*.xml file
TestHCatLoaderEncryption - did not produce a TEST-*.xml file
TestHCatMutableDynamicPartitioned - did not produce a TEST-*.xml file
TestHCatMutableNonPartitioned - did not produce a TEST-*.xml file
TestHCatMutablePartitioned - did not produce a TEST-*.xml file
TestHCatNonPartitioned - did not produce a TEST-*.xml file
TestHCatPartitioned - did not produce a TEST-*.xml file
TestHCatStorer - did not produce a TEST-*.xml file
TestHCatStorerMulti - did not produce a TEST-*.xml file
TestOrcFile - did not produce a TEST-*.xml file
TestPositivePartitionPrunerCompactExpr - did not produce a TEST-*.xml file
TestSessionManagerMetrics - did not produce a TEST-*.xml file
TestSessionState - did not produce a TEST-*.xml file
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_list_bucket_dml_12
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_stats_list_bucket
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_vector_complex_all
on
testCliDriver_vector_complex_all(org.apache.hadoop.hive.cli.TestMiniLlapCliDriver)
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_vector_complex_join
on
testCliDriver_vector_complex_join(org.apache.hadoop.hive.cli.TestMiniLlapCliDriver)
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_auto_sortmerge_join_16
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_bucket4
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_bucket5
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_bucket6
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_bucketizedhiveinputformat
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_bucketmapjoin6
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_bucketmapjoin7
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_constprog_partitioner
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_constprog_semijoin
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_disable_merge_for_bucketing
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_empty_dir_in_table
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_external_table_with_space_in_location_path
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_file_with_header_footer
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_gen_udf_example_add10
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_import_exported_table
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_index_bitmap3
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_index_bitmap_auto
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_infer_bucket_sort_bucketed_table
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_infer_bucket_sort_map_operators
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_infer_bucket_sort_merge
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_infer_bucket_sort_num_buckets
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_infer_bucket_sort_reducers_power_two
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_input16_cc
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_insert_overwrite_directory2
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_leftsemijoin_mr
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_list_bucket_dml_10
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_load_fs2
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_load_hdfs_file_with_space_in_the_name
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_orc_merge1
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver
[jira] [Updated] (HIVE-14111) better concurrency handling for TezSessionState - part I
[ https://issues.apache.org/jira/browse/HIVE-14111?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Siddharth Seth updated HIVE-14111: -- Attachment: sessionPoolNotes.txt > better concurrency handling for TezSessionState - part I > > > Key: HIVE-14111 > URL: https://issues.apache.org/jira/browse/HIVE-14111 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > Attachments: HIVE-14111.patch, sessionPoolNotes.txt > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-14111) better concurrency handling for TezSessionState - part I
[ https://issues.apache.org/jira/browse/HIVE-14111?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Siddharth Seth updated HIVE-14111: -- Target Version/s: 2.2.0, 2.1.1 (was: 2.2.0) > better concurrency handling for TezSessionState - part I > > > Key: HIVE-14111 > URL: https://issues.apache.org/jira/browse/HIVE-14111 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > Attachments: HIVE-14111.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-14111) better concurrency handling for TezSessionState - part I
[
https://issues.apache.org/jira/browse/HIVE-14111?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15352351#comment-15352351
]
Siddharth Seth commented on HIVE-14111:
---
FileSinkOperator, Operation, MetadataOperation, SQLOperation changes carried
forward from some other patch ?
The try { } finally {} in TezTask needs to deal with one special condition. An
attempt to submit() dag can result in session.destroy being invoked - in which
case I believe the session is lost again, and will never be added back to the
queue.
Also - while going through this code, I noticed some other interesting bits.
Will post that as a diff - let me know what you think. Definitely seems like a
part2 will be required, and unit tests.
> better concurrency handling for TezSessionState - part I
>
>
> Key: HIVE-14111
> URL: https://issues.apache.org/jira/browse/HIVE-14111
> Project: Hive
> Issue Type: Bug
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
> Attachments: HIVE-14111.patch
>
>
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-10100) Warning "yarn jar" instead of "hadoop jar" in hadoop 2.7.0
[
https://issues.apache.org/jira/browse/HIVE-10100?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15352322#comment-15352322
]
Siddharth Seth commented on HIVE-10100:
---
Why is this critical? - due to the confusion caused by the error message
printed.
It may be worth asking for a way to suppress this specific message rather than
trying to change Hive code all over the place.
'yarn jar metastore' <- what does this mean ?
> Warning "yarn jar" instead of "hadoop jar" in hadoop 2.7.0
> --
>
> Key: HIVE-10100
> URL: https://issues.apache.org/jira/browse/HIVE-10100
> Project: Hive
> Issue Type: Bug
>Affects Versions: 1.3.0, 2.2.0
>Reporter: Gunther Hagleitner
>Assignee: Prasanth Jayachandran
>Priority: Critical
> Attachments: HIVE-10100.1.patch, HIVE-10100.2.patch, yarn_bin.patch
>
>
> HADOOP-11257 adds a warning to stdout
> {noformat}
> WARNING: Use "yarn jar" to launch YARN applications.
> {noformat}
> which will cause issues if untreated with folks that programatically parse
> stdout for query results (i.e.: CLI, silent mode, etc).
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-10100) Warning "yarn jar" instead of "hadoop jar" in hadoop 2.7.0
[
https://issues.apache.org/jira/browse/HIVE-10100?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15352319#comment-15352319
]
Siddharth Seth commented on HIVE-10100:
---
hadoop jar from within Java code (instead of using the programatic API) ? - I'm
sure there's a good reason to do this :)
> Warning "yarn jar" instead of "hadoop jar" in hadoop 2.7.0
> --
>
> Key: HIVE-10100
> URL: https://issues.apache.org/jira/browse/HIVE-10100
> Project: Hive
> Issue Type: Bug
>Affects Versions: 1.3.0, 2.2.0
>Reporter: Gunther Hagleitner
>Assignee: Prasanth Jayachandran
>Priority: Critical
> Attachments: HIVE-10100.1.patch, HIVE-10100.2.patch, yarn_bin.patch
>
>
> HADOOP-11257 adds a warning to stdout
> {noformat}
> WARNING: Use "yarn jar" to launch YARN applications.
> {noformat}
> which will cause issues if untreated with folks that programatically parse
> stdout for query results (i.e.: CLI, silent mode, etc).
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-14114) Ensure RecordWriter in streaming API is using the same UserGroupInformation as StreamingConnection
[ https://issues.apache.org/jira/browse/HIVE-14114?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eugene Koifman updated HIVE-14114: -- Status: Patch Available (was: Open) > Ensure RecordWriter in streaming API is using the same UserGroupInformation > as StreamingConnection > -- > > Key: HIVE-14114 > URL: https://issues.apache.org/jira/browse/HIVE-14114 > Project: Hive > Issue Type: Bug > Components: Transactions >Affects Versions: 1.0.0 >Reporter: Eugene Koifman >Assignee: Eugene Koifman > Attachments: HIVE-14114.patch > > > currently both DelimitedInputWriter and StrictJsonWriter perform some > Metastore access operations but without using UGI created by the caller for > Metastore operations made by matching StreamingConnection & TransactionBatch -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-14114) Ensure RecordWriter in streaming API is using the same UserGroupInformation as StreamingConnection
[ https://issues.apache.org/jira/browse/HIVE-14114?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eugene Koifman updated HIVE-14114: -- Attachment: HIVE-14114.patch > Ensure RecordWriter in streaming API is using the same UserGroupInformation > as StreamingConnection > -- > > Key: HIVE-14114 > URL: https://issues.apache.org/jira/browse/HIVE-14114 > Project: Hive > Issue Type: Bug > Components: Transactions >Affects Versions: 1.0.0 >Reporter: Eugene Koifman >Assignee: Eugene Koifman > Attachments: HIVE-14114.patch > > > currently both DelimitedInputWriter and StrictJsonWriter perform some > Metastore access operations but without using UGI created by the caller for > Metastore operations made by matching StreamingConnection & TransactionBatch -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-14114) Ensure RecordWriter in streaming API is using the same UserGroupInformation as StreamingConnection
[ https://issues.apache.org/jira/browse/HIVE-14114?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eugene Koifman updated HIVE-14114: -- Component/s: Transactions > Ensure RecordWriter in streaming API is using the same UserGroupInformation > as StreamingConnection > -- > > Key: HIVE-14114 > URL: https://issues.apache.org/jira/browse/HIVE-14114 > Project: Hive > Issue Type: Bug > Components: Transactions >Affects Versions: 1.0.0 >Reporter: Eugene Koifman >Assignee: Eugene Koifman > > currently both DelimitedInputWriter and StrictJsonWriter perform some > Metastore access operations but without using UGI created by the caller for > Metastore operations made by matching StreamingConnection & TransactionBatch -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-14101) Adding type/event notification/version/constraints to hbase metastore
[ https://issues.apache.org/jira/browse/HIVE-14101?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15352286#comment-15352286 ] Daniel Dai commented on HIVE-14101: --- I don't find either. It is included in the initial dump and I cannot find any trace why it get introduced. If so, I don't bother implement it either. > Adding type/event notification/version/constraints to hbase metastore > - > > Key: HIVE-14101 > URL: https://issues.apache.org/jira/browse/HIVE-14101 > Project: Hive > Issue Type: Improvement > Components: HBase Metastore >Reporter: Daniel Dai >Assignee: Daniel Dai > > type/event notification/version/constraints are missing in hbase metastore, > we need to add the missing piece. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13884) Disallow queries fetching more than a configured number of partitions in PartitionPruner
[
https://issues.apache.org/jira/browse/HIVE-13884?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15352278#comment-15352278
]
Hive QA commented on HIVE-13884:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12813827/HIVE-13884.8.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 4 failed/errored test(s), 10273 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_schema_evol_orc_nonvec_mapwork_part
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_stats_list_bucket
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_vector_complex_all
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_vector_complex_join
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-MASTER-Build/280/testReport
Console output:
https://builds.apache.org/job/PreCommit-HIVE-MASTER-Build/280/console
Test logs:
http://ec2-50-18-27-0.us-west-1.compute.amazonaws.com/logs/PreCommit-HIVE-MASTER-Build-280/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 4 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12813827 - PreCommit-HIVE-MASTER-Build
> Disallow queries fetching more than a configured number of partitions in
> PartitionPruner
>
>
> Key: HIVE-13884
> URL: https://issues.apache.org/jira/browse/HIVE-13884
> Project: Hive
> Issue Type: Improvement
>Reporter: Mohit Sabharwal
>Assignee: Sergio Peña
> Attachments: HIVE-13884.1.patch, HIVE-13884.2.patch,
> HIVE-13884.3.patch, HIVE-13884.4.patch, HIVE-13884.5.patch,
> HIVE-13884.6.patch, HIVE-13884.7.patch, HIVE-13884.8.patch
>
>
> Currently the PartitionPruner requests either all partitions or partitions
> based on filter expression. In either scenarios, if the number of partitions
> accessed is large there can be significant memory pressure at the HMS server
> end.
> We already have a config {{hive.limit.query.max.table.partition}} that
> enforces limits on number of partitions that may be scanned per operator. But
> this check happens after the PartitionPruner has already fetched all
> partitions.
> We should add an option at PartitionPruner level to disallow queries that
> attempt to access number of partitions beyond a configurable limit.
> Note that {{hive.mapred.mode=strict}} disallow queries without a partition
> filter in PartitionPruner, but this check accepts any query with a pruning
> condition, even if partitions fetched are large. In multi-tenant
> environments, admins could use more control w.r.t. number of partitions
> allowed based on HMS memory capacity.
> One option is to have PartitionPruner first fetch the partition names
> (instead of partition specs) and throw an exception if number of partitions
> exceeds the configured value. Otherwise, fetch the partition specs.
> Looks like the existing {{listPartitionNames}} call could be used if extended
> to take partition filter expressions like {{getPartitionsByExpr}} call does.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-14113) Create function failed but function in show function list
[
https://issues.apache.org/jira/browse/HIVE-14113?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15352272#comment-15352272
]
niklaus xiao commented on HIVE-14113:
-
[~navis] Could you take a look please, since you are the original author of
this code.
> Create function failed but function in show function list
> -
>
> Key: HIVE-14113
> URL: https://issues.apache.org/jira/browse/HIVE-14113
> Project: Hive
> Issue Type: Bug
> Components: UDF
>Affects Versions: 1.2.0
>Reporter: niklaus xiao
>Assignee: niklaus xiao
> Fix For: 1.3.0
>
> Attachments: HIVE-14113.1.patch
>
>
> 1. create function with invalid hdfs path, /udf/udf-test.jar does not exists
> {quote}
> create function my_lower as 'com.tang.UDFLower' using jar
> 'hdfs:///udf/udf-test.jar';
> {quote}
> Failed with following exception:
> {quote}
> 0: jdbc:hive2://189.39.151.44:1/> create function my_lower as
> 'com.tang.UDFLower' using jar 'hdfs:///udf/udf-test.jar';
> INFO : converting to local hdfs:///udf/udf-test.jar
> ERROR : Failed to read external resource hdfs:///udf/udf-test.jar
> java.lang.RuntimeException: Failed to read external resource
> hdfs:///udf/udf-test.jar
> at
> org.apache.hadoop.hive.ql.session.SessionState.downloadResource(SessionState.java:1384)
> at
> org.apache.hadoop.hive.ql.session.SessionState.resolveAndDownload(SessionState.java:1340)
> at
> org.apache.hadoop.hive.ql.session.SessionState.add_resources(SessionState.java:1264)
> at
> org.apache.hadoop.hive.ql.session.SessionState.add_resources(SessionState.java:1250)
> at
> org.apache.hadoop.hive.ql.exec.FunctionTask.addFunctionResources(FunctionTask.java:306)
> at
> org.apache.hadoop.hive.ql.exec.Registry.registerToSessionRegistry(Registry.java:466)
> at
> org.apache.hadoop.hive.ql.exec.Registry.registerPermanentFunction(Registry.java:206)
> at
> org.apache.hadoop.hive.ql.exec.FunctionRegistry.registerPermanentFunction(FunctionRegistry.java:1551)
> at
> org.apache.hadoop.hive.ql.exec.FunctionTask.createPermanentFunction(FunctionTask.java:136)
> at
> org.apache.hadoop.hive.ql.exec.FunctionTask.execute(FunctionTask.java:75)
> at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:158)
> at
> org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:101)
> at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:1965)
> at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1723)
> at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1475)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1283)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1278)
> at
> org.apache.hive.service.cli.operation.SQLOperation.runQuery(SQLOperation.java:167)
> at
> org.apache.hive.service.cli.operation.SQLOperation.access$200(SQLOperation.java:75)
> at
> org.apache.hive.service.cli.operation.SQLOperation$1$1.run(SQLOperation.java:245)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1711)
> at
> org.apache.hive.service.cli.operation.SQLOperation$1.run(SQLOperation.java:258)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: java.io.FileNotFoundException: File does not exist:
> hdfs:/udf/udf-test.jar
> at
> org.apache.hadoop.hdfs.DistributedFileSystem$25.doCall(DistributedFileSystem.java:1391)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem$25.doCall(DistributedFileSystem.java:1383)
> at
> org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1383)
> at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:340)
> at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:292)
> at org.apache.hadoop.fs.FileSystem.copyToLocalFile(FileSystem.java:2034)
> at org.apache.hadoop.fs.FileSystem.copyToLocalFile(FileSystem.java:2003)
> at org.apache.hadoop.fs.FileSystem.copyToLocalFile(FileSystem.java:1979)
> at
> org.apache.hadoop.hive.ql.session.SessionState.downloadResource(SessionState.java:1370)
> ... 28 more
> ERROR : Failed to register default.my_lower using class com.ta
[jira] [Updated] (HIVE-14113) Create function failed but function in show function list
[
https://issues.apache.org/jira/browse/HIVE-14113?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
niklaus xiao updated HIVE-14113:
Attachment: HIVE-14113.1.patch
> Create function failed but function in show function list
> -
>
> Key: HIVE-14113
> URL: https://issues.apache.org/jira/browse/HIVE-14113
> Project: Hive
> Issue Type: Bug
> Components: UDF
>Affects Versions: 1.2.0
>Reporter: niklaus xiao
>Assignee: niklaus xiao
> Fix For: 1.3.0
>
> Attachments: HIVE-14113.1.patch
>
>
> 1. create function with invalid hdfs path, /udf/udf-test.jar does not exists
> {quote}
> create function my_lower as 'com.tang.UDFLower' using jar
> 'hdfs:///udf/udf-test.jar';
> {quote}
> Failed with following exception:
> {quote}
> 0: jdbc:hive2://189.39.151.44:1/> create function my_lower as
> 'com.tang.UDFLower' using jar 'hdfs:///udf/udf-test.jar';
> INFO : converting to local hdfs:///udf/udf-test.jar
> ERROR : Failed to read external resource hdfs:///udf/udf-test.jar
> java.lang.RuntimeException: Failed to read external resource
> hdfs:///udf/udf-test.jar
> at
> org.apache.hadoop.hive.ql.session.SessionState.downloadResource(SessionState.java:1384)
> at
> org.apache.hadoop.hive.ql.session.SessionState.resolveAndDownload(SessionState.java:1340)
> at
> org.apache.hadoop.hive.ql.session.SessionState.add_resources(SessionState.java:1264)
> at
> org.apache.hadoop.hive.ql.session.SessionState.add_resources(SessionState.java:1250)
> at
> org.apache.hadoop.hive.ql.exec.FunctionTask.addFunctionResources(FunctionTask.java:306)
> at
> org.apache.hadoop.hive.ql.exec.Registry.registerToSessionRegistry(Registry.java:466)
> at
> org.apache.hadoop.hive.ql.exec.Registry.registerPermanentFunction(Registry.java:206)
> at
> org.apache.hadoop.hive.ql.exec.FunctionRegistry.registerPermanentFunction(FunctionRegistry.java:1551)
> at
> org.apache.hadoop.hive.ql.exec.FunctionTask.createPermanentFunction(FunctionTask.java:136)
> at
> org.apache.hadoop.hive.ql.exec.FunctionTask.execute(FunctionTask.java:75)
> at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:158)
> at
> org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:101)
> at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:1965)
> at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1723)
> at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1475)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1283)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1278)
> at
> org.apache.hive.service.cli.operation.SQLOperation.runQuery(SQLOperation.java:167)
> at
> org.apache.hive.service.cli.operation.SQLOperation.access$200(SQLOperation.java:75)
> at
> org.apache.hive.service.cli.operation.SQLOperation$1$1.run(SQLOperation.java:245)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1711)
> at
> org.apache.hive.service.cli.operation.SQLOperation$1.run(SQLOperation.java:258)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: java.io.FileNotFoundException: File does not exist:
> hdfs:/udf/udf-test.jar
> at
> org.apache.hadoop.hdfs.DistributedFileSystem$25.doCall(DistributedFileSystem.java:1391)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem$25.doCall(DistributedFileSystem.java:1383)
> at
> org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1383)
> at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:340)
> at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:292)
> at org.apache.hadoop.fs.FileSystem.copyToLocalFile(FileSystem.java:2034)
> at org.apache.hadoop.fs.FileSystem.copyToLocalFile(FileSystem.java:2003)
> at org.apache.hadoop.fs.FileSystem.copyToLocalFile(FileSystem.java:1979)
> at
> org.apache.hadoop.hive.ql.session.SessionState.downloadResource(SessionState.java:1370)
> ... 28 more
> ERROR : Failed to register default.my_lower using class com.tang.UDFLower
> Error: Error while processing statement: FAILED: Execution Error, return code
> 1 from org.apach
[jira] [Updated] (HIVE-14113) Create function failed but function in show function list
[
https://issues.apache.org/jira/browse/HIVE-14113?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
niklaus xiao updated HIVE-14113:
Fix Version/s: 1.3.0
Status: Patch Available (was: Open)
> Create function failed but function in show function list
> -
>
> Key: HIVE-14113
> URL: https://issues.apache.org/jira/browse/HIVE-14113
> Project: Hive
> Issue Type: Bug
> Components: UDF
>Affects Versions: 1.2.0
>Reporter: niklaus xiao
>Assignee: niklaus xiao
> Fix For: 1.3.0
>
> Attachments: HIVE-14113.1.patch
>
>
> 1. create function with invalid hdfs path, /udf/udf-test.jar does not exists
> {quote}
> create function my_lower as 'com.tang.UDFLower' using jar
> 'hdfs:///udf/udf-test.jar';
> {quote}
> Failed with following exception:
> {quote}
> 0: jdbc:hive2://189.39.151.44:1/> create function my_lower as
> 'com.tang.UDFLower' using jar 'hdfs:///udf/udf-test.jar';
> INFO : converting to local hdfs:///udf/udf-test.jar
> ERROR : Failed to read external resource hdfs:///udf/udf-test.jar
> java.lang.RuntimeException: Failed to read external resource
> hdfs:///udf/udf-test.jar
> at
> org.apache.hadoop.hive.ql.session.SessionState.downloadResource(SessionState.java:1384)
> at
> org.apache.hadoop.hive.ql.session.SessionState.resolveAndDownload(SessionState.java:1340)
> at
> org.apache.hadoop.hive.ql.session.SessionState.add_resources(SessionState.java:1264)
> at
> org.apache.hadoop.hive.ql.session.SessionState.add_resources(SessionState.java:1250)
> at
> org.apache.hadoop.hive.ql.exec.FunctionTask.addFunctionResources(FunctionTask.java:306)
> at
> org.apache.hadoop.hive.ql.exec.Registry.registerToSessionRegistry(Registry.java:466)
> at
> org.apache.hadoop.hive.ql.exec.Registry.registerPermanentFunction(Registry.java:206)
> at
> org.apache.hadoop.hive.ql.exec.FunctionRegistry.registerPermanentFunction(FunctionRegistry.java:1551)
> at
> org.apache.hadoop.hive.ql.exec.FunctionTask.createPermanentFunction(FunctionTask.java:136)
> at
> org.apache.hadoop.hive.ql.exec.FunctionTask.execute(FunctionTask.java:75)
> at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:158)
> at
> org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:101)
> at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:1965)
> at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1723)
> at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1475)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1283)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1278)
> at
> org.apache.hive.service.cli.operation.SQLOperation.runQuery(SQLOperation.java:167)
> at
> org.apache.hive.service.cli.operation.SQLOperation.access$200(SQLOperation.java:75)
> at
> org.apache.hive.service.cli.operation.SQLOperation$1$1.run(SQLOperation.java:245)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1711)
> at
> org.apache.hive.service.cli.operation.SQLOperation$1.run(SQLOperation.java:258)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: java.io.FileNotFoundException: File does not exist:
> hdfs:/udf/udf-test.jar
> at
> org.apache.hadoop.hdfs.DistributedFileSystem$25.doCall(DistributedFileSystem.java:1391)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem$25.doCall(DistributedFileSystem.java:1383)
> at
> org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1383)
> at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:340)
> at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:292)
> at org.apache.hadoop.fs.FileSystem.copyToLocalFile(FileSystem.java:2034)
> at org.apache.hadoop.fs.FileSystem.copyToLocalFile(FileSystem.java:2003)
> at org.apache.hadoop.fs.FileSystem.copyToLocalFile(FileSystem.java:1979)
> at
> org.apache.hadoop.hive.ql.session.SessionState.downloadResource(SessionState.java:1370)
> ... 28 more
> ERROR : Failed to register default.my_lower using class com.tang.UDFLower
> Error: Error while processing statement: FAILED: Execution
[jira] [Updated] (HIVE-14113) Create function failed but function in show function list
[
https://issues.apache.org/jira/browse/HIVE-14113?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
niklaus xiao updated HIVE-14113:
Description:
1. create function with invalid hdfs path, /udf/udf-test.jar does not exists
{quote}
create function my_lower as 'com.tang.UDFLower' using jar
'hdfs:///udf/udf-test.jar';
{quote}
Failed with following exception:
{quote}
0: jdbc:hive2://189.39.151.44:1/> create function my_lower as
'com.tang.UDFLower' using jar 'hdfs:///udf/udf-test.jar';
INFO : converting to local hdfs:///udf/udf-test.jar
ERROR : Failed to read external resource hdfs:///udf/udf-test.jar
java.lang.RuntimeException: Failed to read external resource
hdfs:///udf/udf-test.jar
at
org.apache.hadoop.hive.ql.session.SessionState.downloadResource(SessionState.java:1384)
at
org.apache.hadoop.hive.ql.session.SessionState.resolveAndDownload(SessionState.java:1340)
at
org.apache.hadoop.hive.ql.session.SessionState.add_resources(SessionState.java:1264)
at
org.apache.hadoop.hive.ql.session.SessionState.add_resources(SessionState.java:1250)
at
org.apache.hadoop.hive.ql.exec.FunctionTask.addFunctionResources(FunctionTask.java:306)
at
org.apache.hadoop.hive.ql.exec.Registry.registerToSessionRegistry(Registry.java:466)
at
org.apache.hadoop.hive.ql.exec.Registry.registerPermanentFunction(Registry.java:206)
at
org.apache.hadoop.hive.ql.exec.FunctionRegistry.registerPermanentFunction(FunctionRegistry.java:1551)
at
org.apache.hadoop.hive.ql.exec.FunctionTask.createPermanentFunction(FunctionTask.java:136)
at
org.apache.hadoop.hive.ql.exec.FunctionTask.execute(FunctionTask.java:75)
at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:158)
at
org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:101)
at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:1965)
at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1723)
at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1475)
at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1283)
at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1278)
at
org.apache.hive.service.cli.operation.SQLOperation.runQuery(SQLOperation.java:167)
at
org.apache.hive.service.cli.operation.SQLOperation.access$200(SQLOperation.java:75)
at
org.apache.hive.service.cli.operation.SQLOperation$1$1.run(SQLOperation.java:245)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1711)
at
org.apache.hive.service.cli.operation.SQLOperation$1.run(SQLOperation.java:258)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.io.FileNotFoundException: File does not exist:
hdfs:/udf/udf-test.jar
at
org.apache.hadoop.hdfs.DistributedFileSystem$25.doCall(DistributedFileSystem.java:1391)
at
org.apache.hadoop.hdfs.DistributedFileSystem$25.doCall(DistributedFileSystem.java:1383)
at
org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
at
org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1383)
at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:340)
at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:292)
at org.apache.hadoop.fs.FileSystem.copyToLocalFile(FileSystem.java:2034)
at org.apache.hadoop.fs.FileSystem.copyToLocalFile(FileSystem.java:2003)
at org.apache.hadoop.fs.FileSystem.copyToLocalFile(FileSystem.java:1979)
at
org.apache.hadoop.hive.ql.session.SessionState.downloadResource(SessionState.java:1370)
... 28 more
ERROR : Failed to register default.my_lower using class com.tang.UDFLower
Error: Error while processing statement: FAILED: Execution Error, return code 1
from org.apache.hadoop.hive.ql.exec.FunctionTask (state=08S01,code=1)
{quote}
2. Execute show functions, the failed function my_lower is in the function list
{quote}
0: jdbc:hive2://189.39.151.44:21066/> show functions;
+-+--+
|tab_name |
+-+--+
| day |
| dayofmonth |
| decode |
| default.my_lower1 |
| degrees |
| dense_rank |
0: jdbc:hive2://189.39.151.44:1/> select my_lower(name) from stu;
Error: Error while compiling statement: FAILED: Se
[jira] [Updated] (HIVE-10100) Warning "yarn jar" instead of "hadoop jar" in hadoop 2.7.0
[
https://issues.apache.org/jira/browse/HIVE-10100?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Gunther Hagleitner updated HIVE-10100:
--
Attachment: yarn_bin.patch
[~prasanth_j] there's other places we use "hadoop jar". We use it in local
tasks and map joins etc. See yarn_bin.patch. I haven't tested the changes, but
seems to me we should change that too, no?
> Warning "yarn jar" instead of "hadoop jar" in hadoop 2.7.0
> --
>
> Key: HIVE-10100
> URL: https://issues.apache.org/jira/browse/HIVE-10100
> Project: Hive
> Issue Type: Bug
>Affects Versions: 1.3.0, 2.2.0
>Reporter: Gunther Hagleitner
>Assignee: Prasanth Jayachandran
>Priority: Critical
> Attachments: HIVE-10100.1.patch, HIVE-10100.2.patch, yarn_bin.patch
>
>
> HADOOP-11257 adds a warning to stdout
> {noformat}
> WARNING: Use "yarn jar" to launch YARN applications.
> {noformat}
> which will cause issues if untreated with folks that programatically parse
> stdout for query results (i.e.: CLI, silent mode, etc).
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-13901) Hivemetastore add partitions can be slow depending on filesystems
[
https://issues.apache.org/jira/browse/HIVE-13901?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Rajesh Balamohan updated HIVE-13901:
Attachment: HIVE-13901.7.patch
Checked with recent master build with the patch. Here are the results,
org.apache.hadoop.hive.metastore.TestRemoteHiveMetaStore &
org.apache.hadoop.hive.metastore.TestSetUGIOnOnlyServer anyways fail without
the patch as well. Renaming patch as 0.7 for jenkins
{noformat}
---
T E S T S
---
---
T E S T S
---
Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=512m;
support was removed in 8.0
Running org.apache.hadoop.hive.metastore.TestEmbeddedHiveMetaStore
Tests run: 34, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 90.21 sec - in
org.apache.hadoop.hive.metastore.TestEmbeddedHiveMetaStore
Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=512m;
support was removed in 8.0
Running org.apache.hadoop.hive.metastore.TestRemoteHiveMetaStore
Tests run: 34, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 88.612 sec <<<
FAILURE! - in org.apache.hadoop.hive.metastore.TestRemoteHiveMetaStore
testTransactionalValidation(org.apache.hadoop.hive.metastore.TestRemoteHiveMetaStore)
Time elapsed: 0.112 sec <<< ERROR!
org.apache.hadoop.hive.metastore.api.AlreadyExistsException: Table acidTable
already exists
at
org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$create_table_with_environment_context_result$create_table_with_environment_context_resultStandardScheme.read(ThriftHiveMetastore.java:41480)
at
org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$create_table_with_environment_context_result$create_table_with_environment_context_resultStandardScheme.read(ThriftHiveMetastore.java:41466)
at
org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$create_table_with_environment_context_result.read(ThriftHiveMetastore.java:41392)
at org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:86)
at
org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.recv_create_table_with_environment_context(ThriftHiveMetastore.java:1183)
at
org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.create_table_with_environment_context(ThriftHiveMetastore.java:1169)
at
org.apache.hadoop.hive.metastore.HiveMetaStoreClient.create_table_with_environment_context(HiveMetaStoreClient.java:2325)
at
org.apache.hadoop.hive.metastore.HiveMetaStoreClient.createTable(HiveMetaStoreClient.java:738)
at
org.apache.hadoop.hive.metastore.HiveMetaStoreClient.createTable(HiveMetaStoreClient.java:726)
at
org.apache.hadoop.hive.metastore.TestHiveMetaStore.createTable(TestHiveMetaStore.java:2967)
at
org.apache.hadoop.hive.metastore.TestHiveMetaStore.testTransactionalValidation(TestHiveMetaStore.java:2897)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:483)
at junit.framework.TestCase.runTest(TestCase.java:176)
at junit.framework.TestCase.runBare(TestCase.java:141)
at junit.framework.TestResult$1.protect(TestResult.java:122)
at junit.framework.TestResult.runProtected(TestResult.java:142)
at junit.framework.TestResult.run(TestResult.java:125)
at junit.framework.TestCase.run(TestCase.java:129)
at junit.framework.TestSuite.runTest(TestSuite.java:255)
at junit.framework.TestSuite.run(TestSuite.java:250)
at
org.junit.internal.runners.JUnit38ClassRunner.run(JUnit38ClassRunner.java:84)
at
org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:264)
at
org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:153)
at
org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:124)
at
org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:200)
at
org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:153)
at
org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:103)
Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=512m;
support was removed in 8.0
Running org.apache.hadoop.hive.metastore.TestSetUGIOnOnlyServer
Tests run: 34, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 89.748 sec <<<
FAILURE! - in org.apache.hadoop.hive.metastore.TestSetUGIOnOnlyServer
testTransactionalValidation(org.apache.h
[jira] [Commented] (HIVE-13997) Insert overwrite directory doesn't overwrite existing files
[
https://issues.apache.org/jira/browse/HIVE-13997?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15352229#comment-15352229
]
Rui Li commented on HIVE-13997:
---
Thanks for the review Ashutosh :)
> Insert overwrite directory doesn't overwrite existing files
> ---
>
> Key: HIVE-13997
> URL: https://issues.apache.org/jira/browse/HIVE-13997
> Project: Hive
> Issue Type: Bug
> Components: Query Processor
>Affects Versions: 2.1.0
>Reporter: Rui Li
>Assignee: Rui Li
> Fix For: 2.2.0, 2.1.1
>
> Attachments: HIVE-13997.1.patch, HIVE-13997.2.patch,
> HIVE-13997.2.patch
>
>
> Can be easily reproduced by running {{INSERT OVERWRITE DIRECTORY}} to the
> same dir twice.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Comment Edited] (HIVE-13945) Decimal value is displayed as rounded when selecting where clause with that decimal value.
[
https://issues.apache.org/jira/browse/HIVE-13945?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15352227#comment-15352227
]
Sergey Shelukhin edited comment on HIVE-13945 at 6/28/16 2:28 AM:
--
First batch of out file updates.
I didn't look at TestBeeLineWithArgs which is broken.
Added new test decimal_divide; it appears that precision for int/decimal
division is chosen such that the results are rounded from what it was with
double. Need to look into that.
vector_struct_in and orc_ppd_basic are tests that have suspicious diffs, need
to investigate further.
Another interesting effect - where previously float-op-literal arithmetic would
be double (being float-op-double), now it's float-op-decimal, which resolves to
float result; that can affect precision.
was (Author: sershe):
First batch of out file updates.
I didn't look at TestBeeLineWithArgs which is broken.
Added new test decimal_divide; it appears that precision for int/decimal
division is chosen such that the results are rounded from what it was with
double. Need to look into that.
vector_struct_in and orc_ppd_basic are tests that have suspicious diffs, need
to investigate further.
TestBeeLineWithArgs straight up fails, need to look.
Another interesting effect - where previously float-op-literal arithmetic would
be double (being float-op-double), now it's float-op-decimal, which resolves to
float result; that can affect precision.
> Decimal value is displayed as rounded when selecting where clause with that
> decimal value.
> --
>
> Key: HIVE-13945
> URL: https://issues.apache.org/jira/browse/HIVE-13945
> Project: Hive
> Issue Type: Bug
>Affects Versions: 2.1.0
>Reporter: Takahiko Saito
>Assignee: Sergey Shelukhin
>Priority: Critical
> Attachments: HIVE-13945.01.patch, HIVE-13945.02.patch,
> HIVE-13945.patch
>
>
> Create a table withe a column of decimal type(38,18) and insert
> '4327269606205.029297'. Then select with that value displays its rounded
> value, which is 4327269606205.029300
> {noformat}
> 0: jdbc:hive2://os-r7-mvjkcu-hiveserver2-11-4> drop table if exists test;
> No rows affected (0.229 seconds)
> 0: jdbc:hive2://os-r7-mvjkcu-hiveserver2-11-4>
> 0: jdbc:hive2://os-r7-mvjkcu-hiveserver2-11-4> create table test (dc
> decimal(38,18));
> No rows affected (0.125 seconds)
> 0: jdbc:hive2://os-r7-mvjkcu-hiveserver2-11-4>
> 0: jdbc:hive2://os-r7-mvjkcu-hiveserver2-11-4> insert into table test values
> (4327269606205.029297);
> No rows affected (2.372 seconds)
> 0: jdbc:hive2://os-r7-mvjkcu-hiveserver2-11-4>
> 0: jdbc:hive2://os-r7-mvjkcu-hiveserver2-11-4> select * from test;
> +---+--+
> | test.dc |
> +---+--+
> | 4327269606205.029297 |
> +---+--+
> 1 row selected (0.123 seconds)
> 0: jdbc:hive2://os-r7-mvjkcu-hiveserver2-11-4>
> 0: jdbc:hive2://os-r7-mvjkcu-hiveserver2-11-4> select * from test where dc =
> 4327269606205.029297;
> +---+--+
> | test.dc |
> +---+--+
> | 4327269606205.029300 |
> +---+--+
> 1 row selected (0.109 seconds)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-13945) Decimal value is displayed as rounded when selecting where clause with that decimal value.
[
https://issues.apache.org/jira/browse/HIVE-13945?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sergey Shelukhin updated HIVE-13945:
Attachment: HIVE-13945.02.patch
First batch of out file updates.
I didn't look at TestBeeLineWithArgs which is broken.
Added new test decimal_divide; it appears that precision for int/decimal
division is chosen such that the results are rounded from what it was with
double. Need to look into that.
vector_struct_in and orc_ppd_basic are tests that have suspicious diffs, need
to investigate further.
TestBeeLineWithArgs straight up fails, need to look.
Another interesting effect - where previously float-op-literal arithmetic would
be double (being float-op-double), now it's float-op-decimal, which resolves to
float result; that can affect precision.
> Decimal value is displayed as rounded when selecting where clause with that
> decimal value.
> --
>
> Key: HIVE-13945
> URL: https://issues.apache.org/jira/browse/HIVE-13945
> Project: Hive
> Issue Type: Bug
>Affects Versions: 2.1.0
>Reporter: Takahiko Saito
>Assignee: Sergey Shelukhin
>Priority: Critical
> Attachments: HIVE-13945.01.patch, HIVE-13945.02.patch,
> HIVE-13945.patch
>
>
> Create a table withe a column of decimal type(38,18) and insert
> '4327269606205.029297'. Then select with that value displays its rounded
> value, which is 4327269606205.029300
> {noformat}
> 0: jdbc:hive2://os-r7-mvjkcu-hiveserver2-11-4> drop table if exists test;
> No rows affected (0.229 seconds)
> 0: jdbc:hive2://os-r7-mvjkcu-hiveserver2-11-4>
> 0: jdbc:hive2://os-r7-mvjkcu-hiveserver2-11-4> create table test (dc
> decimal(38,18));
> No rows affected (0.125 seconds)
> 0: jdbc:hive2://os-r7-mvjkcu-hiveserver2-11-4>
> 0: jdbc:hive2://os-r7-mvjkcu-hiveserver2-11-4> insert into table test values
> (4327269606205.029297);
> No rows affected (2.372 seconds)
> 0: jdbc:hive2://os-r7-mvjkcu-hiveserver2-11-4>
> 0: jdbc:hive2://os-r7-mvjkcu-hiveserver2-11-4> select * from test;
> +---+--+
> | test.dc |
> +---+--+
> | 4327269606205.029297 |
> +---+--+
> 1 row selected (0.123 seconds)
> 0: jdbc:hive2://os-r7-mvjkcu-hiveserver2-11-4>
> 0: jdbc:hive2://os-r7-mvjkcu-hiveserver2-11-4> select * from test where dc =
> 4327269606205.029297;
> +---+--+
> | test.dc |
> +---+--+
> | 4327269606205.029300 |
> +---+--+
> 1 row selected (0.109 seconds)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-14089) complex type support in LLAP IO is broken
[
https://issues.apache.org/jira/browse/HIVE-14089?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sergey Shelukhin updated HIVE-14089:
Attachment: HIVE-14089.WIP.patch
Partial patch that fixes a couple issues and idenitifies the next one where
context is not created for a certain column that has no data streams, the
assumption that for every column there would be a context at corresponding
position is broken.
Probably easy to fix depending on what 0 data streams mean - nothing to be read
for that column?
Then perhaps something else will break.
I also modified the test. So far it fails.
> complex type support in LLAP IO is broken
> --
>
> Key: HIVE-14089
> URL: https://issues.apache.org/jira/browse/HIVE-14089
> Project: Hive
> Issue Type: Bug
>Affects Versions: 2.2.0
>Reporter: Prasanth Jayachandran
>Assignee: Sergey Shelukhin
> Attachments: HIVE-14089.WIP.patch
>
>
> HIVE-13617 is causing MiniLlapCliDriver following test failures
> {code}
> org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_vector_complex_all
> org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_vector_complex_join
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-14089) complex type support in LLAP IO is broken
[
https://issues.apache.org/jira/browse/HIVE-14089?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sergey Shelukhin updated HIVE-14089:
Assignee: Prasanth Jayachandran (was: Sergey Shelukhin)
> complex type support in LLAP IO is broken
> --
>
> Key: HIVE-14089
> URL: https://issues.apache.org/jira/browse/HIVE-14089
> Project: Hive
> Issue Type: Bug
>Affects Versions: 2.2.0
>Reporter: Prasanth Jayachandran
>Assignee: Prasanth Jayachandran
> Attachments: HIVE-14089.WIP.patch
>
>
> HIVE-13617 is causing MiniLlapCliDriver following test failures
> {code}
> org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_vector_complex_all
> org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_vector_complex_join
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-14112) Join a HBase mapped big table shouldn't convert to MapJoin
[
https://issues.apache.org/jira/browse/HIVE-14112?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yuming Wang updated HIVE-14112:
---
Attachment: HIVE-14112.1.patch
> Join a HBase mapped big table shouldn't convert to MapJoin
> --
>
> Key: HIVE-14112
> URL: https://issues.apache.org/jira/browse/HIVE-14112
> Project: Hive
> Issue Type: Bug
> Components: StorageHandler
>Affects Versions: 1.2.0, 1.1.0
>Reporter: Yuming Wang
>Assignee: Yuming Wang
>Priority: Minor
> Attachments: HIVE-14112.1.patch
>
>
> Two tables, _hbasetable_risk_control_defense_idx_uid_ is HBase mapped table:
> {noformat}
> [root@dev01 ~]# hadoop fs -du -s -h
> /hbase/data/tandem/hbase-table-risk-control-defense-idx-uid
> 3.0 G 9.0 G /hbase/data/tandem/hbase-table-risk-control-defense-idx-uid
> [root@dev01 ~]# hadoop fs -du -s -h /user/hive/warehouse/openapi_invoke_base
> 6.6 G 19.7 G /user/hive/warehouse/openapi_invoke_base
> {noformat}
> The smallest table is 3.0G, is greater than
> _hive.mapjoin.smalltable.filesize_ and
> _hive.auto.convert.join.noconditionaltask.size_. When join these tables, Hive
> auto convert it to mapjoin:
> {noformat}
> hive> select count(*) from hbasetable_risk_control_defense_idx_uid t1 join
> openapi_invoke_base t2 on (t1.key=t2.merchantid);
> Query ID = root_2016062809_9f9d3f25-857b-412c-8a75-3d9228bd5ee5
> Total jobs = 1
> Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=512M;
> support was removed in 8.0
> Execution log at:
> /tmp/root/root_2016062809_9f9d3f25-857b-412c-8a75-3d9228bd5ee5.log
> 2016-06-28 09:22:10 Starting to launch local task to process map join;
> maximum memory = 1908932608
> {noformat}
> the root cause is hive use
> _/user/hive/warehouse/hbasetable_risk_control_defense_idx_uid_ as it
> location, but it empty. so hive auto convert it to mapjoin.
> My opinion is set right location when mapping HBase table.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-14073) update config whiltelist for sql std authorization
[ https://issues.apache.org/jira/browse/HIVE-14073?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thejas M Nair updated HIVE-14073: - Resolution: Fixed Fix Version/s: 2.1.1 2.2.0 1.3.0 Status: Resolved (was: Patch Available) > update config whiltelist for sql std authorization > --- > > Key: HIVE-14073 > URL: https://issues.apache.org/jira/browse/HIVE-14073 > Project: Hive > Issue Type: Bug > Components: Security, SQLStandardAuthorization >Affects Versions: 2.1.0 >Reporter: Thejas M Nair >Assignee: Thejas M Nair > Fix For: 1.3.0, 2.2.0, 2.1.1 > > Attachments: HIVE-14073-branch-1.1.patch, > HIVE-14073-branch-1.2.patch, HIVE-14073.1.patch, HIVE-14073.2.patch > > > New configs that should go in security whitelist have been added. Whitelist > needs updating. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-14073) update config whiltelist for sql std authorization
[ https://issues.apache.org/jira/browse/HIVE-14073?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15352208#comment-15352208 ] Thejas M Nair commented on HIVE-14073: -- Verified that TestMetaStoreInitListener was a test flakiness issue. The other test failures are consistently happening in builds. > update config whiltelist for sql std authorization > --- > > Key: HIVE-14073 > URL: https://issues.apache.org/jira/browse/HIVE-14073 > Project: Hive > Issue Type: Bug > Components: Security, SQLStandardAuthorization >Affects Versions: 2.1.0 >Reporter: Thejas M Nair >Assignee: Thejas M Nair > Attachments: HIVE-14073-branch-1.1.patch, > HIVE-14073-branch-1.2.patch, HIVE-14073.1.patch, HIVE-14073.2.patch > > > New configs that should go in security whitelist have been added. Whitelist > needs updating. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-14037) java.lang.ClassNotFoundException for the jar in hive.reloadable.aux.jars.path in mapreduce
[
https://issues.apache.org/jira/browse/HIVE-14037?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15352199#comment-15352199
]
Ferdinand Xu commented on HIVE-14037:
-
Thanks [~aihuaxu] for your patch. I left some comments in RB.
> java.lang.ClassNotFoundException for the jar in hive.reloadable.aux.jars.path
> in mapreduce
> ---
>
> Key: HIVE-14037
> URL: https://issues.apache.org/jira/browse/HIVE-14037
> Project: Hive
> Issue Type: Bug
>Reporter: Aihua Xu
>Assignee: Aihua Xu
> Attachments: HIVE-14037.1.patch, HIVE-14037.2.patch,
> HIVE-14037.3.patch
>
>
> The jars in hive.reloadable.aux.jars.path seem to be available in HS2 process
> while they are not available in the Mapper or Reducer nodes which will throw
> the following exception.
> {noformat}
> org.apache.hadoop.hive.ql.exec.ExprNodeGenericFuncEvaluator.isDeterministic(ExprNodeGenericFuncEvaluator.java:152)
> at
> org.apache.hadoop.hive.ql.exec.ExprNodeEvaluatorFactory.iterate(ExprNodeEvaluatorFactory.java:97)
> at
> org.apache.hadoop.hive.ql.exec.ExprNodeEvaluatorFactory.toCachedEvals(ExprNodeEvaluatorFactory.java:71)
> at
> org.apache.hadoop.hive.ql.exec.SelectOperator.initializeOp(SelectOperator.java:59)
> at org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:385)
> at org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:469)
> at
> org.apache.hadoop.hive.ql.exec.Operator.initializeChildren(Operator.java:425)
> at
> org.apache.hadoop.hive.ql.exec.TableScanOperator.initializeOp(TableScanOperator.java:193)
> at org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:385)
> at
> org.apache.hadoop.hive.ql.exec.MapOperator.initializeOp(MapOperator.java:431)
> at org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:385)
> at
> org.apache.hadoop.hive.ql.exec.mr.ExecMapper.configure(ExecMapper.java:126)
> ... 22 more
> Caused by: java.lang.ClassNotFoundException: test.UDF
> at java.net.URLClassLoader$1.run(URLClassLoader.java:366)
> at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
> at java.security.AccessController.doPrivileged(Native Method)
> at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
> at java.lang.Class.forName0(Native Method)
> at java.lang.Class.forName(Class.java:270)
> at
> org.apache.hadoop.hive.ql.udf.generic.GenericUDFBridge.getUdfClass(GenericUDFBridge.java:132)
> ... 36 more
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-14013) Describe table doesn't show unicode properly
[
https://issues.apache.org/jira/browse/HIVE-14013?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15352179#comment-15352179
]
Hive QA commented on HIVE-14013:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12813823/HIVE-14013.4.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 4 failed/errored test(s), 10274 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_list_bucket_dml_13
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_stats_list_bucket
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_vector_complex_all
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_vector_complex_join
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-MASTER-Build/279/testReport
Console output:
https://builds.apache.org/job/PreCommit-HIVE-MASTER-Build/279/console
Test logs:
http://ec2-50-18-27-0.us-west-1.compute.amazonaws.com/logs/PreCommit-HIVE-MASTER-Build-279/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 4 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12813823 - PreCommit-HIVE-MASTER-Build
> Describe table doesn't show unicode properly
>
>
> Key: HIVE-14013
> URL: https://issues.apache.org/jira/browse/HIVE-14013
> Project: Hive
> Issue Type: Bug
> Components: Query Planning
>Affects Versions: 2.2.0
>Reporter: Aihua Xu
>Assignee: Aihua Xu
> Attachments: HIVE-14013.1.patch, HIVE-14013.2.patch,
> HIVE-14013.3.patch, HIVE-14013.4.patch
>
>
> Describe table output will show comments incorrectly rather than the unicode
> itself.
> {noformat}
> hive> desc formatted t1;
> # Detailed Table Information
> Table Type: MANAGED_TABLE
> Table Parameters:
> COLUMN_STATS_ACCURATE {\"BASIC_STATS\":\"true\"}
> comment \u8868\u4E2D\u6587\u6D4B\u8BD5
> numFiles0
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-10100) Warning "yarn jar" instead of "hadoop jar" in hadoop 2.7.0
[
https://issues.apache.org/jira/browse/HIVE-10100?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15352141#comment-15352141
]
Siddharth Seth commented on HIVE-10100:
---
YARN_USER_CLASSPATH needs to be set in a couple more scripts - beeline, hplsql,
hwi.
Other than that, looks good to me.
Don't really know why hadoop jar was deprecated in favor of yarn jar, and what
the difference between the two is.
> Warning "yarn jar" instead of "hadoop jar" in hadoop 2.7.0
> --
>
> Key: HIVE-10100
> URL: https://issues.apache.org/jira/browse/HIVE-10100
> Project: Hive
> Issue Type: Bug
>Affects Versions: 1.3.0, 2.2.0
>Reporter: Gunther Hagleitner
>Assignee: Prasanth Jayachandran
>Priority: Critical
> Attachments: HIVE-10100.1.patch, HIVE-10100.2.patch
>
>
> HADOOP-11257 adds a warning to stdout
> {noformat}
> WARNING: Use "yarn jar" to launch YARN applications.
> {noformat}
> which will cause issues if untreated with folks that programatically parse
> stdout for query results (i.e.: CLI, silent mode, etc).
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-14027) NULL values produced by left outer join do not behave as NULL
[
https://issues.apache.org/jira/browse/HIVE-14027?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jesus Camacho Rodriguez updated HIVE-14027:
---
Affects Version/s: 2.2.0
> NULL values produced by left outer join do not behave as NULL
> -
>
> Key: HIVE-14027
> URL: https://issues.apache.org/jira/browse/HIVE-14027
> Project: Hive
> Issue Type: Bug
> Components: Query Processor
>Affects Versions: 1.2.1, 2.2.0
>Reporter: Vaibhav Gumashta
>Assignee: Vaibhav Gumashta
>
> Consider the following setup:
> {code}
> create table tbl (n bigint, t string);
> insert into tbl values (1, 'one');
> insert into tbl values(2, 'two');
> select a.n, a.t, isnull(b.n), isnull(b.t) from (select * from tbl where n =
> 1) a left outer join (select * from tbl where 1 = 2) b on a.n = b.n;
> 1onefalsetrue
> {code}
> The query should return true for isnull(b.n).
> I've tested by inserting a row with null value for the bigint column into
> tbl, and isnull returns true in that case.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-11402) HS2 - disallow parallel query execution within a single Session
[ https://issues.apache.org/jira/browse/HIVE-11402?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HIVE-11402: Attachment: HIVE-11402.01.patch > HS2 - disallow parallel query execution within a single Session > --- > > Key: HIVE-11402 > URL: https://issues.apache.org/jira/browse/HIVE-11402 > Project: Hive > Issue Type: Bug > Components: HiveServer2 >Reporter: Thejas M Nair >Assignee: Sergey Shelukhin > Attachments: HIVE-11402.01.patch, HIVE-11402.patch > > > HiveServer2 currently allows concurrent queries to be run in a single > session. However, every HS2 session has an associated SessionState object, > and the use of SessionState in many places assumes that only one thread is > using it, ie it is not thread safe. > There are many places where SesssionState thread safety needs to be > addressed, and until then we should serialize all query execution for a > single HS2 session. -This problem can become more visible with HIVE-4239 now > allowing parallel query compilation.- > Note that running queries in parallel for single session is not > straightforward with jdbc, you need to spawn another thread as the > Statement.execute calls are blocking. I believe ODBC has non blocking query > execution API, and Hue is another well known application that shares sessions > for all queries that a user runs. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-14101) Adding type/event notification/version/constraints to hbase metastore
[ https://issues.apache.org/jira/browse/HIVE-14101?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15352105#comment-15352105 ] Alan Gates commented on HIVE-14101: --- Is type used anywhere? I couldn't find where it was. > Adding type/event notification/version/constraints to hbase metastore > - > > Key: HIVE-14101 > URL: https://issues.apache.org/jira/browse/HIVE-14101 > Project: Hive > Issue Type: Improvement > Components: HBase Metastore >Reporter: Daniel Dai >Assignee: Daniel Dai > > type/event notification/version/constraints are missing in hbase metastore, > we need to add the missing piece. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-14108) Add missing objects in hbaseimport
[ https://issues.apache.org/jira/browse/HIVE-14108?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15352096#comment-15352096 ] Alan Gates commented on HIVE-14108: --- I fully agree we should support privs and versions. I couldn't find anywhere type was actually used, so I didn't bother with it. My take was there's no need to copy column stats, as they can be re-constructed. But I'm open to other opinions here. > Add missing objects in hbaseimport > -- > > Key: HIVE-14108 > URL: https://issues.apache.org/jira/browse/HIVE-14108 > Project: Hive > Issue Type: Improvement > Components: HBase Metastore >Reporter: Daniel Dai >Assignee: Daniel Dai > > The following objects are not imported with hbaseimport: > privs (table/partition/column) > column stats > type/constraint/version -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-14093) LLAP output format connection should wait for all writes to finish before closing channel
[ https://issues.apache.org/jira/browse/HIVE-14093?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15352092#comment-15352092 ] Prasanth Jayachandran commented on HIVE-14093: -- nit: Can you initialize the maxPendingWrites with some static constant? Also the default 2 seems low? Other than that looks good to me, +1 > LLAP output format connection should wait for all writes to finish before > closing channel > - > > Key: HIVE-14093 > URL: https://issues.apache.org/jira/browse/HIVE-14093 > Project: Hive > Issue Type: Sub-task > Components: llap >Reporter: Jason Dere >Assignee: Jason Dere > Attachments: HIVE-14093.1.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-14078) LLAP input split should get task attempt number from conf if available
[ https://issues.apache.org/jira/browse/HIVE-14078?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jason Dere updated HIVE-14078: -- Attachment: HIVE-14078.2.patch Add debug line to note when the attempt number is being set via conf. > LLAP input split should get task attempt number from conf if available > -- > > Key: HIVE-14078 > URL: https://issues.apache.org/jira/browse/HIVE-14078 > Project: Hive > Issue Type: Sub-task > Components: llap >Reporter: Jason Dere >Assignee: Jason Dere > Attachments: HIVE-14078.1.patch, HIVE-14078.2.patch > > > Currently the attempt number is hard-coded to 0. If the split is being > fetched as part of a hadoop job we can get the task attempt ID from the conf > if it has been set, and use the attempt number from that. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-11402) HS2 - disallow parallel query execution within a single Session
[ https://issues.apache.org/jira/browse/HIVE-11402?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HIVE-11402: Status: Patch Available (was: Open) > HS2 - disallow parallel query execution within a single Session > --- > > Key: HIVE-11402 > URL: https://issues.apache.org/jira/browse/HIVE-11402 > Project: Hive > Issue Type: Bug > Components: HiveServer2 >Reporter: Thejas M Nair >Assignee: Sergey Shelukhin > Attachments: HIVE-11402.patch > > > HiveServer2 currently allows concurrent queries to be run in a single > session. However, every HS2 session has an associated SessionState object, > and the use of SessionState in many places assumes that only one thread is > using it, ie it is not thread safe. > There are many places where SesssionState thread safety needs to be > addressed, and until then we should serialize all query execution for a > single HS2 session. -This problem can become more visible with HIVE-4239 now > allowing parallel query compilation.- > Note that running queries in parallel for single session is not > straightforward with jdbc, you need to spawn another thread as the > Statement.execute calls are blocking. I believe ODBC has non blocking query > execution API, and Hue is another well known application that shares sessions > for all queries that a user runs. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13995) Hive generates inefficient metastore queries for TPCDS tables with 1800+ partitions leading to higher compile time
[
https://issues.apache.org/jira/browse/HIVE-13995?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15352062#comment-15352062
]
Hive QA commented on HIVE-13995:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12812961/HIVE-13995.1.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 523 failed/errored test(s), 10273 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_acid_table_stats
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_alter_partition_coltype
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_analyze_table_null_partition
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_annotate_stats_part
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_autoColumnStats_1
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_autoColumnStats_2
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_autoColumnStats_8
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_join14
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_join19
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_join19_inclause
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_join32
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_join9
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_sortmerge_join_1
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_sortmerge_join_11
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_sortmerge_join_12
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_sortmerge_join_2
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_sortmerge_join_3
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_sortmerge_join_4
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_sortmerge_join_7
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_sortmerge_join_8
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_avro_schema_evolution_native
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_bucket3
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_bucket_groupby
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_bucket_map_join_spark1
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_bucket_map_join_spark2
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_bucket_map_join_spark3
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_bucketcontext_1
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_bucketcontext_2
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_bucketcontext_3
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_bucketcontext_4
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_bucketcontext_6
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_bucketcontext_7
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_bucketcontext_8
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_bucketmapjoin1
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_bucketmapjoin10
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_bucketmapjoin11
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_bucketmapjoin12
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_bucketmapjoin13
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_bucketmapjoin2
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_bucketmapjoin3
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_bucketmapjoin5
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_bucketmapjoin8
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_bucketmapjoin9
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_bucketmapjoin_negative
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_bucketmapjoin_negative2
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_bucketsortoptimize_insert_1
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_bucketsortoptimize_insert_2
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_bucketsortoptimize_insert_3
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_bucketsortoptimize_insert_4
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_bucketsortoptimize_insert_5
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_bucketsortoptimize_insert_6
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_bucketsortoptimize_insert_7
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_bucketsortoptimize_insert_8
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_cbo_const
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_cbo_input26
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_cbo_rp_join0
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_cbo_rp_outer_join_ppr
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_cbo_union_view
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_columnStatsUpdateForStatsOptimizer_1
org.apache.hadoop.hive.cli.TestCliDr
[jira] [Updated] (HIVE-13369) AcidUtils.getAcidState() is not paying attention toValidTxnList when choosing the "best" base file
[ https://issues.apache.org/jira/browse/HIVE-13369?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei Zheng updated HIVE-13369: - Status: Patch Available (was: Open) > AcidUtils.getAcidState() is not paying attention toValidTxnList when choosing > the "best" base file > -- > > Key: HIVE-13369 > URL: https://issues.apache.org/jira/browse/HIVE-13369 > Project: Hive > Issue Type: Bug > Components: Transactions >Affects Versions: 1.0.0 >Reporter: Eugene Koifman >Assignee: Wei Zheng >Priority: Blocker > Attachments: HIVE-13369.1.patch > > > The JavaDoc on getAcidState() reads, in part: > "Note that because major compactions don't >preserve the history, we can't use a base directory that includes a >transaction id that we must exclude." > which is correct but there is nothing in the code that does this. > And if we detect a situation where txn X must be excluded but and there are > deltas that contain X, we'll have to aborted the txn. This can't > (reasonably) happen with auto commit mode, but with multi statement txns it's > possible. > Suppose some long running txn starts and lock in snapshot at 17 (HWM). An > hour later it decides to access some partition for which all txns < 20 (for > example) have already been compacted (i.e. GC'd). -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13369) AcidUtils.getAcidState() is not paying attention toValidTxnList when choosing the "best" base file
[ https://issues.apache.org/jira/browse/HIVE-13369?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei Zheng updated HIVE-13369: - Attachment: HIVE-13369.1.patch Patch 1 is about the first part in the problem description. It also reduce the frequency of AcidOpenTxnsCounterService logging. > AcidUtils.getAcidState() is not paying attention toValidTxnList when choosing > the "best" base file > -- > > Key: HIVE-13369 > URL: https://issues.apache.org/jira/browse/HIVE-13369 > Project: Hive > Issue Type: Bug > Components: Transactions >Affects Versions: 1.0.0 >Reporter: Eugene Koifman >Assignee: Wei Zheng >Priority: Blocker > Attachments: HIVE-13369.1.patch > > > The JavaDoc on getAcidState() reads, in part: > "Note that because major compactions don't >preserve the history, we can't use a base directory that includes a >transaction id that we must exclude." > which is correct but there is nothing in the code that does this. > And if we detect a situation where txn X must be excluded but and there are > deltas that contain X, we'll have to aborted the txn. This can't > (reasonably) happen with auto commit mode, but with multi statement txns it's > possible. > Suppose some long running txn starts and lock in snapshot at 17 (HWM). An > hour later it decides to access some partition for which all txns < 20 (for > example) have already been compacted (i.e. GC'd). -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-11402) HS2 - disallow parallel query execution within a single Session
[ https://issues.apache.org/jira/browse/HIVE-11402?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HIVE-11402: Attachment: HIVE-11402.patch Simple patch that adds a semaphore to acquire/release methods, configurable and on by default (parallel operations are off, rather). [~thejas] can you take a look? Esp. wrt what is an operation and what isn't, e.g. should fetching results be allowed to run in parallel with other things? I will post an RB shortly > HS2 - disallow parallel query execution within a single Session > --- > > Key: HIVE-11402 > URL: https://issues.apache.org/jira/browse/HIVE-11402 > Project: Hive > Issue Type: Bug > Components: HiveServer2 >Reporter: Thejas M Nair >Assignee: Sergey Shelukhin > Attachments: HIVE-11402.patch > > > HiveServer2 currently allows concurrent queries to be run in a single > session. However, every HS2 session has an associated SessionState object, > and the use of SessionState in many places assumes that only one thread is > using it, ie it is not thread safe. > There are many places where SesssionState thread safety needs to be > addressed, and until then we should serialize all query execution for a > single HS2 session. -This problem can become more visible with HIVE-4239 now > allowing parallel query compilation.- > Note that running queries in parallel for single session is not > straightforward with jdbc, you need to spawn another thread as the > Statement.execute calls are blocking. I believe ODBC has non blocking query > execution API, and Hue is another well known application that shares sessions > for all queries that a user runs. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-14093) LLAP output format connection should wait for all writes to finish before closing channel
[ https://issues.apache.org/jira/browse/HIVE-14093?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15352015#comment-15352015 ] Jason Dere commented on HIVE-14093: --- The new failure looks like a recurring failure, it already has a Jira associated with it (HIVE-14088) > LLAP output format connection should wait for all writes to finish before > closing channel > - > > Key: HIVE-14093 > URL: https://issues.apache.org/jira/browse/HIVE-14093 > Project: Hive > Issue Type: Sub-task > Components: llap >Reporter: Jason Dere >Assignee: Jason Dere > Attachments: HIVE-14093.1.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Comment Edited] (HIVE-14091) some errors are not propagated to LLAP external clients
[
https://issues.apache.org/jira/browse/HIVE-14091?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15352006#comment-15352006
]
Siddharth Seth edited comment on HIVE-14091 at 6/27/16 11:06 PM:
-
The main change here is to close the socket in case of an exception, correct?
and hope that this causes the InputStream read to return immediately - since
the interrupt does not work. Afaik - this is best effort - and there's a
comment in the patch which indicates the same.
This will cause any reads on the InputStream to fail - likely with a
ClosedChannelException (or equivalent). Do we need to handle this in a specific
manner in the reader code - at least to indicate the kind of error so that
debugging is easier.
Mostly unrelated to this jira.
{code}
case ERROR:
throw new IOException("Received reader event error: " +
event.getMessage());
default:
throw new IOException("Got reader event type " +
event.getEventType() + ", expected error event");
{code}
This gets rid of the original exception. Would be worth propagating the
exception further up, or at least logging it.
I don't think the addition of taskFailed on the Responder is required. This
will be invoked in any case when the Umbilical heartbeat implementation invokes
responder.heartbeat. (adding the method implies the error being sent twice to
the responder)
Should the socket also be cleaned up during ReaderBase.close()
was (Author: sseth):
The main change here is to close the socket in case of an exception, correct?
and hope that this causes the InputStream read to return immediately - since
the interrupt does not work. Afaik - this is best effort - and there's a
comment in the patch which indicates the same.
This will cause any reads on the InputStream to fail - likely with a
ClosedChannelException (or equivalent). Do we need to handle this in a specific
manner in the reader code - at least to indicate the kind of error so that
debugging is easier.
Mostly unrelated to this jira.
{code}
case ERROR:
throw new IOException("Received reader event error: " +
event.getMessage());
default:
throw new IOException("Got reader event type " +
event.getEventType() + ", expected error event");
{code}
This gets rid of the original exception. Would be worth propagating the
exception further up, or at least logging it.
I don't think the addition of taskFailed on the Responder is required. This
will be invoked in any case when the Umbilical heartbeat implementation invokes
responder.heartbeat. (adding the method implies the error being sent twice to
the responder)
Should the socket also be cleaned up during ReaderBase.close()
Kind of related to the patch.
{code}
> some errors are not propagated to LLAP external clients
> ---
>
> Key: HIVE-14091
> URL: https://issues.apache.org/jira/browse/HIVE-14091
> Project: Hive
> Issue Type: Bug
>Reporter: Jason Dere
>Assignee: Sergey Shelukhin
> Attachments: HIVE-14091.patch
>
>
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-14111) better concurrency handling for TezSessionState - part I
[ https://issues.apache.org/jira/browse/HIVE-14111?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HIVE-14111: Attachment: (was: HIVE-14111.patch) > better concurrency handling for TezSessionState - part I > > > Key: HIVE-14111 > URL: https://issues.apache.org/jira/browse/HIVE-14111 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > Attachments: HIVE-14111.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-14091) some errors are not propagated to LLAP external clients
[
https://issues.apache.org/jira/browse/HIVE-14091?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15352006#comment-15352006
]
Siddharth Seth commented on HIVE-14091:
---
The main change here is to close the socket in case of an exception, correct?
and hope that this causes the InputStream read to return immediately - since
the interrupt does not work. Afaik - this is best effort - and there's a
comment in the patch which indicates the same.
This will cause any reads on the InputStream to fail - likely with a
ClosedChannelException (or equivalent). Do we need to handle this in a specific
manner in the reader code - at least to indicate the kind of error so that
debugging is easier.
Mostly unrelated to this jira.
{code}
case ERROR:
throw new IOException("Received reader event error: " +
event.getMessage());
default:
throw new IOException("Got reader event type " +
event.getEventType() + ", expected error event");
{code}
This gets rid of the original exception. Would be worth propagating the
exception further up, or at least logging it.
I don't think the addition of taskFailed on the Responder is required. This
will be invoked in any case when the Umbilical heartbeat implementation invokes
responder.heartbeat. (adding the method implies the error being sent twice to
the responder)
Should the socket also be cleaned up during ReaderBase.close()
Kind of related to the patch.
{code}
> some errors are not propagated to LLAP external clients
> ---
>
> Key: HIVE-14091
> URL: https://issues.apache.org/jira/browse/HIVE-14091
> Project: Hive
> Issue Type: Bug
>Reporter: Jason Dere
>Assignee: Sergey Shelukhin
> Attachments: HIVE-14091.patch
>
>
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-14111) better concurrency handling for TezSessionState - part I
[ https://issues.apache.org/jira/browse/HIVE-14111?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HIVE-14111: Attachment: HIVE-14111.patch > better concurrency handling for TezSessionState - part I > > > Key: HIVE-14111 > URL: https://issues.apache.org/jira/browse/HIVE-14111 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > Attachments: HIVE-14111.patch, HIVE-14111.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Assigned] (HIVE-11402) HS2 - disallow parallel query execution within a single Session
[ https://issues.apache.org/jira/browse/HIVE-11402?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin reassigned HIVE-11402: --- Assignee: Sergey Shelukhin > HS2 - disallow parallel query execution within a single Session > --- > > Key: HIVE-11402 > URL: https://issues.apache.org/jira/browse/HIVE-11402 > Project: Hive > Issue Type: Bug > Components: HiveServer2 >Reporter: Thejas M Nair >Assignee: Sergey Shelukhin > > HiveServer2 currently allows concurrent queries to be run in a single > session. However, every HS2 session has an associated SessionState object, > and the use of SessionState in many places assumes that only one thread is > using it, ie it is not thread safe. > There are many places where SesssionState thread safety needs to be > addressed, and until then we should serialize all query execution for a > single HS2 session. -This problem can become more visible with HIVE-4239 now > allowing parallel query compilation.- > Note that running queries in parallel for single session is not > straightforward with jdbc, you need to spawn another thread as the > Statement.execute calls are blocking. I believe ODBC has non blocking query > execution API, and Hue is another well known application that shares sessions > for all queries that a user runs. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-14111) better concurrency handling for TezSessionState - part I
[ https://issues.apache.org/jira/browse/HIVE-14111?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HIVE-14111: Target Version/s: 2.2.0 Status: Patch Available (was: Open) > better concurrency handling for TezSessionState - part I > > > Key: HIVE-14111 > URL: https://issues.apache.org/jira/browse/HIVE-14111 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > Attachments: HIVE-14111.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-14111) better concurrency handling for TezSessionState - part I
[ https://issues.apache.org/jira/browse/HIVE-14111?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15351988#comment-15351988 ] Sergey Shelukhin commented on HIVE-14111: - [~sseth] fyi > better concurrency handling for TezSessionState - part I > > > Key: HIVE-14111 > URL: https://issues.apache.org/jira/browse/HIVE-14111 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > Attachments: HIVE-14111.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-14111) better concurrency handling for TezSessionState - part I
[ https://issues.apache.org/jira/browse/HIVE-14111?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HIVE-14111: Attachment: HIVE-14111.patch Based on potential errors from the stricter checks, there may be part 2 > better concurrency handling for TezSessionState - part I > > > Key: HIVE-14111 > URL: https://issues.apache.org/jira/browse/HIVE-14111 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > Attachments: HIVE-14111.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-14072) QueryIds reused across different queries
[ https://issues.apache.org/jira/browse/HIVE-14072?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15351970#comment-15351970 ] Siddharth Seth commented on HIVE-14072: --- [~sershe] - the changes seem reasonable to me - resetting the queryState between each run. Tested it locally on TestMiniLlapCluster - and the duplicate IDs for different queries are gone. (Duplicates exist for queries which are split across multiple jobs, but that should be ok). That said, I'd be far more comfortable if someone who knows this code better takes a look at it. cc [~ashutoshc] > QueryIds reused across different queries > > > Key: HIVE-14072 > URL: https://issues.apache.org/jira/browse/HIVE-14072 > Project: Hive > Issue Type: Bug >Reporter: Siddharth Seth >Assignee: Sergey Shelukhin >Priority: Critical > Attachments: HIVE-14072.patch > > > While testing HIVE-14023, and running TestMiniLlapCluster - query ids were > re-uesd for the entire init scripts. 30+ different queries - same queryId, > new Tez dag submission, for different queries. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-10100) Warning "yarn jar" instead of "hadoop jar" in hadoop 2.7.0
[
https://issues.apache.org/jira/browse/HIVE-10100?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Prasanth Jayachandran updated HIVE-10100:
-
Attachment: HIVE-10100.2.patch
> Warning "yarn jar" instead of "hadoop jar" in hadoop 2.7.0
> --
>
> Key: HIVE-10100
> URL: https://issues.apache.org/jira/browse/HIVE-10100
> Project: Hive
> Issue Type: Bug
>Affects Versions: 1.3.0, 2.2.0
>Reporter: Gunther Hagleitner
>Assignee: Prasanth Jayachandran
>Priority: Critical
> Attachments: HIVE-10100.1.patch, HIVE-10100.2.patch
>
>
> HADOOP-11257 adds a warning to stdout
> {noformat}
> WARNING: Use "yarn jar" to launch YARN applications.
> {noformat}
> which will cause issues if untreated with folks that programatically parse
> stdout for query results (i.e.: CLI, silent mode, etc).
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-10100) Warning "yarn jar" instead of "hadoop jar" in hadoop 2.7.0
[
https://issues.apache.org/jira/browse/HIVE-10100?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15351931#comment-15351931
]
Prasanth Jayachandran commented on HIVE-10100:
--
bq.For HADOOP_CLASSPATH_FIRST and YARN_CLASSPATH_FIRST - should we use the
value set in HADOOP_CLASSPATH_FIRST or the one used in YARN_CLASSPATH_FIRST ?
I think we need this to be always true for proper initialization of log4j2 and
jline.
> Warning "yarn jar" instead of "hadoop jar" in hadoop 2.7.0
> --
>
> Key: HIVE-10100
> URL: https://issues.apache.org/jira/browse/HIVE-10100
> Project: Hive
> Issue Type: Bug
>Affects Versions: 1.3.0, 2.2.0
>Reporter: Gunther Hagleitner
>Assignee: Prasanth Jayachandran
>Priority: Critical
> Attachments: HIVE-10100.1.patch
>
>
> HADOOP-11257 adds a warning to stdout
> {noformat}
> WARNING: Use "yarn jar" to launch YARN applications.
> {noformat}
> which will cause issues if untreated with folks that programatically parse
> stdout for query results (i.e.: CLI, silent mode, etc).
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-10100) Warning "yarn jar" instead of "hadoop jar" in hadoop 2.7.0
[
https://issues.apache.org/jira/browse/HIVE-10100?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15351929#comment-15351929
]
Prasanth Jayachandran commented on HIVE-10100:
--
bq. Why is this snippet used in some files, but not others ?
Some scripts explicitly modifies the variables after hive script sets it up to
append service specific info. Example: beeline seems to be doing some ordering
of classpath and metastore appends its own java opts which might not be
required otherwise.
I can't find YARN_CLASSPATH in yarn script. I think after some version
YARN_CLASSPATH is renamed to YARN_USER_CLASSPATH. So I added HADOOP_CLASSPATH
to YARN_USER_CLASSPATH in the next patch.
Yes. Hive script is slow for 2 reasons
1) Automatic initialization of logging (when trying to run hadoop or hbase
commands entire classpath is searched to look for log4j2.properties file)
2) hbase mapredcp takes time
There is some info here HIVE-12497. Regardless we should bring down the time.
> Warning "yarn jar" instead of "hadoop jar" in hadoop 2.7.0
> --
>
> Key: HIVE-10100
> URL: https://issues.apache.org/jira/browse/HIVE-10100
> Project: Hive
> Issue Type: Bug
>Affects Versions: 1.3.0, 2.2.0
>Reporter: Gunther Hagleitner
>Assignee: Prasanth Jayachandran
>Priority: Critical
> Attachments: HIVE-10100.1.patch
>
>
> HADOOP-11257 adds a warning to stdout
> {noformat}
> WARNING: Use "yarn jar" to launch YARN applications.
> {noformat}
> which will cause issues if untreated with folks that programatically parse
> stdout for query results (i.e.: CLI, silent mode, etc).
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-13560) Adding Omid as connection manager for HBase Metastore
[ https://issues.apache.org/jira/browse/HIVE-13560?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daniel Dai updated HIVE-13560: -- Attachment: HIVE-13560.6.patch Trigger precommit test as Omid 0.8.2.0 released. > Adding Omid as connection manager for HBase Metastore > - > > Key: HIVE-13560 > URL: https://issues.apache.org/jira/browse/HIVE-13560 > Project: Hive > Issue Type: Improvement > Components: HBase Metastore >Reporter: Daniel Dai >Assignee: Daniel Dai > Attachments: HIVE-13560.1.patch, HIVE-13560.2.patch, > HIVE-13560.3.patch, HIVE-13560.4.patch, HIVE-13560.5.patch, HIVE-13560.6.patch > > > Adding Omid as a transaction manager to HBase Metastore. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-14073) update config whiltelist for sql std authorization
[
https://issues.apache.org/jira/browse/HIVE-14073?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15351898#comment-15351898
]
Hive QA commented on HIVE-14073:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12813802/HIVE-14073.2.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 5 failed/errored test(s), 10273 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_list_bucket_dml_13
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_stats_list_bucket
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_vector_complex_all
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_vector_complex_join
org.apache.hadoop.hive.metastore.TestMetaStoreInitListener.testMetaStoreInitListener
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-MASTER-Build/277/testReport
Console output:
https://builds.apache.org/job/PreCommit-HIVE-MASTER-Build/277/console
Test logs:
http://ec2-50-18-27-0.us-west-1.compute.amazonaws.com/logs/PreCommit-HIVE-MASTER-Build-277/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 5 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12813802 - PreCommit-HIVE-MASTER-Build
> update config whiltelist for sql std authorization
> ---
>
> Key: HIVE-14073
> URL: https://issues.apache.org/jira/browse/HIVE-14073
> Project: Hive
> Issue Type: Bug
> Components: Security, SQLStandardAuthorization
>Affects Versions: 2.1.0
>Reporter: Thejas M Nair
>Assignee: Thejas M Nair
> Attachments: HIVE-14073-branch-1.1.patch,
> HIVE-14073-branch-1.2.patch, HIVE-14073.1.patch, HIVE-14073.2.patch
>
>
> New configs that should go in security whitelist have been added. Whitelist
> needs updating.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-14073) update config whiltelist for sql std authorization
[ https://issues.apache.org/jira/browse/HIVE-14073?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15351896#comment-15351896 ] Sushanth Sowmyan commented on HIVE-14073: - +1. > update config whiltelist for sql std authorization > --- > > Key: HIVE-14073 > URL: https://issues.apache.org/jira/browse/HIVE-14073 > Project: Hive > Issue Type: Bug > Components: Security, SQLStandardAuthorization >Affects Versions: 2.1.0 >Reporter: Thejas M Nair >Assignee: Thejas M Nair > Attachments: HIVE-14073-branch-1.1.patch, > HIVE-14073-branch-1.2.patch, HIVE-14073.1.patch, HIVE-14073.2.patch > > > New configs that should go in security whitelist have been added. Whitelist > needs updating. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-14110) Implement a better ObjectCache in hbase metastore
[ https://issues.apache.org/jira/browse/HIVE-14110?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daniel Dai updated HIVE-14110: -- Summary: Implement a better ObjectCache in hbase metastore (was: Implement a better ObjectStore in hbase metastore) > Implement a better ObjectCache in hbase metastore > - > > Key: HIVE-14110 > URL: https://issues.apache.org/jira/browse/HIVE-14110 > Project: Hive > Issue Type: Improvement > Components: HBase Metastore >Reporter: Daniel Dai >Assignee: Daniel Dai > > The ObjectCache in hbase metastore is very naive and we need to enhance it to > a decent one. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-14110) Implement a better ObjectStore in hbase metastore
[ https://issues.apache.org/jira/browse/HIVE-14110?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daniel Dai updated HIVE-14110: -- Description: The ObjectCache in hbase metastore is very naive and we need to enhance it to a decent one. (was: The ObjectStore in hbase metastore is very naive and we need to enhance it to a decent one.) > Implement a better ObjectStore in hbase metastore > - > > Key: HIVE-14110 > URL: https://issues.apache.org/jira/browse/HIVE-14110 > Project: Hive > Issue Type: Improvement > Components: HBase Metastore >Reporter: Daniel Dai >Assignee: Daniel Dai > > The ObjectCache in hbase metastore is very naive and we need to enhance it to > a decent one. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-14109) query execuction throws NPE when hive.exec.submitviachild is set to true
[
https://issues.apache.org/jira/browse/HIVE-14109?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Aihua Xu updated HIVE-14109:
Description:
If we set hive.exec.submitviachild to true and execute select count(*) from
src, the following exception is thrown.
Seems queryState is not initialized when ExecDriver is called from main() in
ExecDriver.
{noformat}
java.lang.NullPointerException
at
org.apache.hadoop.hive.ql.exec.mr.HadoopJobExecHelper.progress(HadoopJobExecHelper.java:262)
at
org.apache.hadoop.hive.ql.exec.mr.HadoopJobExecHelper.progress(HadoopJobExecHelper.java:555)
at
org.apache.hadoop.hive.ql.exec.mr.ExecDriver.execute(ExecDriver.java:436)
at
org.apache.hadoop.hive.ql.exec.mr.ExecDriver.main(ExecDriver.java:756)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.apache.hadoop.util.RunJar.run(RunJar.java:221)
at org.apache.hadoop.util.RunJar.main(RunJar.java:136)
{noformat}
was:
If we set hive.exec.submitviachild to true and execute select coun(*) from src,
the following exception is thrown.
Seems queryState is not initialized when ExecDriver is called from main() in
ExecDriver.
{noformat}
java.lang.NullPointerException
at
org.apache.hadoop.hive.ql.exec.mr.HadoopJobExecHelper.progress(HadoopJobExecHelper.java:262)
at
org.apache.hadoop.hive.ql.exec.mr.HadoopJobExecHelper.progress(HadoopJobExecHelper.java:555)
at
org.apache.hadoop.hive.ql.exec.mr.ExecDriver.execute(ExecDriver.java:436)
at
org.apache.hadoop.hive.ql.exec.mr.ExecDriver.main(ExecDriver.java:756)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.apache.hadoop.util.RunJar.run(RunJar.java:221)
at org.apache.hadoop.util.RunJar.main(RunJar.java:136)
{noformat}
> query execuction throws NPE when hive.exec.submitviachild is set to true
>
>
> Key: HIVE-14109
> URL: https://issues.apache.org/jira/browse/HIVE-14109
> Project: Hive
> Issue Type: Bug
> Components: Query Processor
>Affects Versions: 2.2.0
>Reporter: Aihua Xu
>Assignee: Aihua Xu
>
> If we set hive.exec.submitviachild to true and execute select count(*) from
> src, the following exception is thrown.
> Seems queryState is not initialized when ExecDriver is called from main() in
> ExecDriver.
> {noformat}
> java.lang.NullPointerException
> at
> org.apache.hadoop.hive.ql.exec.mr.HadoopJobExecHelper.progress(HadoopJobExecHelper.java:262)
> at
> org.apache.hadoop.hive.ql.exec.mr.HadoopJobExecHelper.progress(HadoopJobExecHelper.java:555)
> at
> org.apache.hadoop.hive.ql.exec.mr.ExecDriver.execute(ExecDriver.java:436)
> at
> org.apache.hadoop.hive.ql.exec.mr.ExecDriver.main(ExecDriver.java:756)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at org.apache.hadoop.util.RunJar.run(RunJar.java:221)
> at org.apache.hadoop.util.RunJar.main(RunJar.java:136)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-14102) LLAP Improvements to per executor memory computation in daemons
[ https://issues.apache.org/jira/browse/HIVE-14102?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Siddharth Seth updated HIVE-14102: -- Labels: llap (was: ) > LLAP Improvements to per executor memory computation in daemons > --- > > Key: HIVE-14102 > URL: https://issues.apache.org/jira/browse/HIVE-14102 > Project: Hive > Issue Type: Improvement >Reporter: Siddharth Seth > Labels: llap > > Currently set to 80% of maxAvailableMemory - irrespective of container size. > This can end up with a lot of wastage. > Also - the 80% / overhead may already be accounted for in YARN container > sizing calculations. This should not be double counted. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-14013) Describe table doesn't show unicode properly
[
https://issues.apache.org/jira/browse/HIVE-14013?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15351762#comment-15351762
]
Yongzhi Chen commented on HIVE-14013:
-
patch-4 LGTM, +1 pending on tests
> Describe table doesn't show unicode properly
>
>
> Key: HIVE-14013
> URL: https://issues.apache.org/jira/browse/HIVE-14013
> Project: Hive
> Issue Type: Bug
> Components: Query Planning
>Affects Versions: 2.2.0
>Reporter: Aihua Xu
>Assignee: Aihua Xu
> Attachments: HIVE-14013.1.patch, HIVE-14013.2.patch,
> HIVE-14013.3.patch, HIVE-14013.4.patch
>
>
> Describe table output will show comments incorrectly rather than the unicode
> itself.
> {noformat}
> hive> desc formatted t1;
> # Detailed Table Information
> Table Type: MANAGED_TABLE
> Table Parameters:
> COLUMN_STATS_ACCURATE {\"BASIC_STATS\":\"true\"}
> comment \u8868\u4E2D\u6587\u6D4B\u8BD5
> numFiles0
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-13884) Disallow queries fetching more than a configured number of partitions in PartitionPruner
[
https://issues.apache.org/jira/browse/HIVE-13884?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sergio Peña updated HIVE-13884:
---
Attachment: HIVE-13884.8.patch
Re-attach patch as tests were not executed.
> Disallow queries fetching more than a configured number of partitions in
> PartitionPruner
>
>
> Key: HIVE-13884
> URL: https://issues.apache.org/jira/browse/HIVE-13884
> Project: Hive
> Issue Type: Improvement
>Reporter: Mohit Sabharwal
>Assignee: Sergio Peña
> Attachments: HIVE-13884.1.patch, HIVE-13884.2.patch,
> HIVE-13884.3.patch, HIVE-13884.4.patch, HIVE-13884.5.patch,
> HIVE-13884.6.patch, HIVE-13884.7.patch, HIVE-13884.8.patch
>
>
> Currently the PartitionPruner requests either all partitions or partitions
> based on filter expression. In either scenarios, if the number of partitions
> accessed is large there can be significant memory pressure at the HMS server
> end.
> We already have a config {{hive.limit.query.max.table.partition}} that
> enforces limits on number of partitions that may be scanned per operator. But
> this check happens after the PartitionPruner has already fetched all
> partitions.
> We should add an option at PartitionPruner level to disallow queries that
> attempt to access number of partitions beyond a configurable limit.
> Note that {{hive.mapred.mode=strict}} disallow queries without a partition
> filter in PartitionPruner, but this check accepts any query with a pruning
> condition, even if partitions fetched are large. In multi-tenant
> environments, admins could use more control w.r.t. number of partitions
> allowed based on HMS memory capacity.
> One option is to have PartitionPruner first fetch the partition names
> (instead of partition specs) and throw an exception if number of partitions
> exceeds the configured value. Otherwise, fetch the partition specs.
> Looks like the existing {{listPartitionNames}} call could be used if extended
> to take partition filter expressions like {{getPartitionsByExpr}} call does.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-14078) LLAP input split should get task attempt number from conf if available
[ https://issues.apache.org/jira/browse/HIVE-14078?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15351746#comment-15351746 ] Siddharth Seth commented on HIVE-14078: --- Looks good. May want to add a debug line on where the attempt number is coming from. I believe Tez will be setting this MR string as well. > LLAP input split should get task attempt number from conf if available > -- > > Key: HIVE-14078 > URL: https://issues.apache.org/jira/browse/HIVE-14078 > Project: Hive > Issue Type: Sub-task > Components: llap >Reporter: Jason Dere >Assignee: Jason Dere > Attachments: HIVE-14078.1.patch > > > Currently the attempt number is hard-coded to 0. If the split is being > fetched as part of a hadoop job we can get the task attempt ID from the conf > if it has been set, and use the attempt number from that. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-14013) Describe table doesn't show unicode properly
[
https://issues.apache.org/jira/browse/HIVE-14013?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Aihua Xu updated HIVE-14013:
Attachment: HIVE-14013.4.patch
Patch-4: address comments. Use the similar way as common.lang3 does. Included
the lib of common.lang3 in pom.xml.
> Describe table doesn't show unicode properly
>
>
> Key: HIVE-14013
> URL: https://issues.apache.org/jira/browse/HIVE-14013
> Project: Hive
> Issue Type: Bug
> Components: Query Planning
>Affects Versions: 2.2.0
>Reporter: Aihua Xu
>Assignee: Aihua Xu
> Attachments: HIVE-14013.1.patch, HIVE-14013.2.patch,
> HIVE-14013.3.patch, HIVE-14013.4.patch
>
>
> Describe table output will show comments incorrectly rather than the unicode
> itself.
> {noformat}
> hive> desc formatted t1;
> # Detailed Table Information
> Table Type: MANAGED_TABLE
> Table Parameters:
> COLUMN_STATS_ACCURATE {\"BASIC_STATS\":\"true\"}
> comment \u8868\u4E2D\u6587\u6D4B\u8BD5
> numFiles0
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-14085) Allow type widening primitive conversion on hive/parquet tables
[
https://issues.apache.org/jira/browse/HIVE-14085?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sergio Peña updated HIVE-14085:
---
Resolution: Fixed
Fix Version/s: 2.2.0
Status: Resolved (was: Patch Available)
Thanks [~vihangk1] I committed this to 2.2
> Allow type widening primitive conversion on hive/parquet tables
> ---
>
> Key: HIVE-14085
> URL: https://issues.apache.org/jira/browse/HIVE-14085
> Project: Hive
> Issue Type: Improvement
> Components: File Formats
>Reporter: Vihang Karajgaonkar
>Assignee: Vihang Karajgaonkar
> Fix For: 2.2.0
>
> Attachments: HIVE-14085.1.patch
>
>
> There is a JIRA ticket on upstream that brought this usability improvement in
> Hive to support auto type widening for Parquet tables. See
> https://issues.apache.org/jira/browse/HIVE-12080
> This improvement is very useful for users who have schema evolution on their
> tables. For example, a Hive table with a "bigint" can read parquet files with
> "int32" and "int64" types.
> The patch only supports widening conversions from int->bigint and
> float->double. We should support more types to allow users read their changed
> parquet schema.
> Here's a list of widening conversions we should support:
> {code}
> tinyint -> smallint,int,bigint,float,double
> smallint -> int,bigint,float,double
> int -> bigint,float,double
> bigint -> float,double
> float -> double
> double -> --
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-14085) Allow type widening primitive conversion on hive/parquet tables
[
https://issues.apache.org/jira/browse/HIVE-14085?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15351677#comment-15351677
]
Sergio Peña commented on HIVE-14085:
Patch looks good, and tests are not related. The one that fails is a flaky
issue detected in older tests.
+1
> Allow type widening primitive conversion on hive/parquet tables
> ---
>
> Key: HIVE-14085
> URL: https://issues.apache.org/jira/browse/HIVE-14085
> Project: Hive
> Issue Type: Improvement
> Components: File Formats
>Reporter: Vihang Karajgaonkar
>Assignee: Vihang Karajgaonkar
> Attachments: HIVE-14085.1.patch
>
>
> There is a JIRA ticket on upstream that brought this usability improvement in
> Hive to support auto type widening for Parquet tables. See
> https://issues.apache.org/jira/browse/HIVE-12080
> This improvement is very useful for users who have schema evolution on their
> tables. For example, a Hive table with a "bigint" can read parquet files with
> "int32" and "int64" types.
> The patch only supports widening conversions from int->bigint and
> float->double. We should support more types to allow users read their changed
> parquet schema.
> Here's a list of widening conversions we should support:
> {code}
> tinyint -> smallint,int,bigint,float,double
> smallint -> int,bigint,float,double
> int -> bigint,float,double
> bigint -> float,double
> float -> double
> double -> --
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-14090) JDOExceptions thrown by the Metastore have their full stack trace returned to clients
[
https://issues.apache.org/jira/browse/HIVE-14090?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15351672#comment-15351672
]
Hive QA commented on HIVE-14090:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12813112/HIVE-14090.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 6 failed/errored test(s), 10273 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_list_bucket_dml_12
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_list_bucket_dml_13
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_stats_list_bucket
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_vector_complex_all
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_vector_complex_join
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_create_with_constraints_duplicate_name
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-MASTER-Build/276/testReport
Console output:
https://builds.apache.org/job/PreCommit-HIVE-MASTER-Build/276/console
Test logs:
http://ec2-50-18-27-0.us-west-1.compute.amazonaws.com/logs/PreCommit-HIVE-MASTER-Build-276/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 6 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12813112 - PreCommit-HIVE-MASTER-Build
> JDOExceptions thrown by the Metastore have their full stack trace returned to
> clients
> -
>
> Key: HIVE-14090
> URL: https://issues.apache.org/jira/browse/HIVE-14090
> Project: Hive
> Issue Type: Bug
>Affects Versions: 1.1.0, 2.1.0
>Reporter: Sahil Takiar
>Assignee: Sahil Takiar
> Attachments: HIVE-14090.patch
>
>
> When user try to create any database or table with a name longer than 128
> characters:
> {code}
> create database
> test_longname_looonglooonglooonglooonglooonglooonglooonglooonglooonglooonglooongNametableFAIL;
> {code}
> It dumps the full exception stack-trace in a non-user-friendly message. The
> lends to relatively negative user-experience for Beeline users who hit this
> exception, they are generally not interested in the full stack-trace.
> The formatted stack-trace is below:
> {code}
> Error while processing statement: FAILED: Execution Error, return code 1 from
> org.apache.hadoop.hive.ql.exec.DDLTask.
> MetaException(message:javax.jdo.JDOFatalUserException: Attempt to store value
> "test_longname_looonglooonglooonglooonglooonglooonglooonglooonglooonglooonglooongnametablefail2"
> in column "`NAME`" that has maximum length of 128. Please correct your data!
> at
> org.datanucleus.api.jdo.NucleusJDOHelper.getJDOExceptionForNucleusException(NucleusJDOHelper.java:528)
> at
> org.datanucleus.api.jdo.JDOPersistenceManager.jdoMakePersistent(JDOPersistenceManager.java:732)
> at
> org.datanucleus.api.jdo.JDOPersistenceManager.makePersistent(JDOPersistenceManager.java:752)
> at
> org.apache.hadoop.hive.metastore.ObjectStore.createDatabase(ObjectStore.java:569)
> at sun.reflect.GeneratedMethodAccessor31.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at
> org.apache.hadoop.hive.metastore.RawStoreProxy.invoke(RawStoreProxy.java:114)
> at com.sun.proxy.$Proxy10.createDatabase(Unknown Source)
> at
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.create_database_core(HiveMetaStore.java:923)
> at
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.create_database(HiveMetaStore.java:962)
> at sun.reflect.GeneratedMethodAccessor30.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at
> org.apache.hadoop.hive.metastore.RetryingHMSHandler.invokeInternal(RetryingHMSHandler.java:138)
> at
> org.apache.hadoop.hive.metastore.RetryingHMSHandler.invoke(RetryingHMSHandler.java:99)
> at com.sun.proxy.$Proxy12.create_database(Unknown Source)
> at
> org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$create_database.getResult(ThriftHiveMetastore.java:8863)
> at
> org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$create_database.getResult(ThriftHiveMetastore.java:8847)
> at org.apache.thrift.ProcessFunction.process(ProcessFunc
[jira] [Commented] (HIVE-13945) Decimal value is displayed as rounded when selecting where clause with that decimal value.
[ https://issues.apache.org/jira/browse/HIVE-13945?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15351615#comment-15351615 ] Sergey Shelukhin commented on HIVE-13945: - Hello I am HiveQA! https://builds.apache.org/job/PreCommit-HIVE-MASTER-Build/250/#showFailuresLink org.apache.hive.beeline.TestBeeLineWithArgs.testDSVOutputWithDoubleQuotes org.apache.hive.beeline.TestBeeLineWithArgs.testCSV2OutputWithDoubleQuotes org.apache.hive.beeline.TestBeeLineWithArgs.testTSVOutputWithDoubleQuotes org.apache.hive.beeline.TestBeeLineWithArgs.testTSV2OutputWithDoubleQuotes org.apache.hive.beeline.TestBeeLineWithArgs.testDSVOutput org.apache.hive.beeline.TestBeeLineWithArgs.testCSVOutputWithDoubleQuotes org.apache.hive.beeline.TestBeeLineWithArgs.testTSVOutput org.apache.hive.beeline.TestBeeLineWithArgs.testTSV2Output org.apache.hive.beeline.TestBeeLineWithArgs.testCSVOutput org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_udf_if org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_udf_round org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_decimal_6 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_vector_decimal_expressions org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_udf_java_method org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_vectorization_8 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_lineage3 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_udf_reflect org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_infer_const_type org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_input49 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_udf_union org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_vectorization_12 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_annotate_stats_select org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_list_bucket_dml_13 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_vectorization_short_regress org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_udf_printf org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_udf_least org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_vectorization_13 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_vectorization_16 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_vectorization_14 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_vectorization_1 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_vectorization_div0 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_vectorization_0 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_vector_decimal_6 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_orc_predicate_pushdown org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_view_cast org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_vector_decimal_math_funcs org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_vectorization_9 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_cbo_rp_udaf_percentile_approx_23 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_udaf_percentile_approx_23 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_parquet_predicate_pushdown org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_udf_round_3 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_udf_coalesce org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_vectorization_17 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_windowing_expressions org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_vector_between_in org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_udf_when org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_cast1 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_udf_sort_array org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_literal_double org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_udf_case org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_vector_struct_in org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_vectorization_15 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_temp_table_windowing_expressions org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_multi_insert_gby2 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_udf_abs org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_udf6 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_udf_bround org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_vectorization_3 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_udf_format_number org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_metadata_only_queries org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_vectorized_math_funcs org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_udf4
[jira] [Commented] (HIVE-14013) Describe table doesn't show unicode properly
[
https://issues.apache.org/jira/browse/HIVE-14013?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15351613#comment-15351613
]
Yongzhi Chen commented on HIVE-14013:
-
Should you do something similar to what it does in ?
{noformat}
/**
* Translator object for escaping Java.
*
* While {@link #escapeJava(String)} is the expected method of use, this
* object allows the Java escaping functionality to be used
* as the foundation for a custom translator.
*
* @since 3.0
*/
public static final CharSequenceTranslator ESCAPE_JAVA =
new LookupTranslator(
new String[][] {
{"\"", "\\\""},
{"\\", ""},
}).with(
new LookupTranslator(EntityArrays.JAVA_CTRL_CHARS_ESCAPE())
).with(
UnicodeEscaper.outsideOf(32, 0x7f)
);
{noformat}
Just remove last .with(..) ?
> Describe table doesn't show unicode properly
>
>
> Key: HIVE-14013
> URL: https://issues.apache.org/jira/browse/HIVE-14013
> Project: Hive
> Issue Type: Bug
> Components: Query Planning
>Affects Versions: 2.2.0
>Reporter: Aihua Xu
>Assignee: Aihua Xu
> Attachments: HIVE-14013.1.patch, HIVE-14013.2.patch,
> HIVE-14013.3.patch
>
>
> Describe table output will show comments incorrectly rather than the unicode
> itself.
> {noformat}
> hive> desc formatted t1;
> # Detailed Table Information
> Table Type: MANAGED_TABLE
> Table Parameters:
> COLUMN_STATS_ACCURATE {\"BASIC_STATS\":\"true\"}
> comment \u8868\u4E2D\u6587\u6D4B\u8BD5
> numFiles0
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-14073) update config whiltelist for sql std authorization
[ https://issues.apache.org/jira/browse/HIVE-14073?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15351582#comment-15351582 ] Sergey Shelukhin commented on HIVE-14073: - hive.llap.allow.permanent.fns should not be settable by user iirc. It's about packaging right? > update config whiltelist for sql std authorization > --- > > Key: HIVE-14073 > URL: https://issues.apache.org/jira/browse/HIVE-14073 > Project: Hive > Issue Type: Bug > Components: Security, SQLStandardAuthorization >Affects Versions: 2.1.0 >Reporter: Thejas M Nair >Assignee: Thejas M Nair > Attachments: HIVE-14073-branch-1.1.patch, > HIVE-14073-branch-1.2.patch, HIVE-14073.1.patch, HIVE-14073.2.patch > > > New configs that should go in security whitelist have been added. Whitelist > needs updating. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-14073) update config whiltelist for sql std authorization
[ https://issues.apache.org/jira/browse/HIVE-14073?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15351572#comment-15351572 ] Siddharth Seth commented on HIVE-14073: --- [~sershe] - Is "hive.llap.allow.permanent.fns" configurable by users ? There's a bunch of other settings which apply to the AM. However, there's no consistent way for these to actually get used, since they require a restart of the AM at the moment. It's better to leave them as not-configurable at the moment. e.g. LLAP_TASK_SCHEDULER_NODE_REENABLE_MIN_TIMEOUT_MS, LLAP_TASK_SCHEDULER_NODE_REENABLE_MAX_TIMEOUT_MS, LLAP_TASK_SCHEDULER_NODE_DISABLE_BACK_OFF_FACTOR, LLAP_TASK_SCHEDULER_NUM_SCHEDULABLE_TASKS_PER_NODE, LLAP_TASK_SCHEDULER_LOCALITY_DELAY Would be good to separate the client configs into a separate list at least, if not renaming them / moving them to other files. There's a jira open for this - but not done yet. > update config whiltelist for sql std authorization > --- > > Key: HIVE-14073 > URL: https://issues.apache.org/jira/browse/HIVE-14073 > Project: Hive > Issue Type: Bug > Components: Security, SQLStandardAuthorization >Affects Versions: 2.1.0 >Reporter: Thejas M Nair >Assignee: Thejas M Nair > Attachments: HIVE-14073-branch-1.1.patch, > HIVE-14073-branch-1.2.patch, HIVE-14073.1.patch, HIVE-14073.2.patch > > > New configs that should go in security whitelist have been added. Whitelist > needs updating. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-14085) Allow type widening primitive conversion on hive/parquet tables
[
https://issues.apache.org/jira/browse/HIVE-14085?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15351534#comment-15351534
]
Vihang Karajgaonkar commented on HIVE-14085:
All the tests except for TestMiniSparkOnYarnCliDriver for index_bitmap3.q are
failing even without this change in the latest code base in master branch.
index_bitmap3.q with TestMiniSparkOnYarnCliDriver is working fine locally so
its failure doesn't seem to be related to this change.
> Allow type widening primitive conversion on hive/parquet tables
> ---
>
> Key: HIVE-14085
> URL: https://issues.apache.org/jira/browse/HIVE-14085
> Project: Hive
> Issue Type: Improvement
> Components: File Formats
>Reporter: Vihang Karajgaonkar
>Assignee: Vihang Karajgaonkar
> Attachments: HIVE-14085.1.patch
>
>
> There is a JIRA ticket on upstream that brought this usability improvement in
> Hive to support auto type widening for Parquet tables. See
> https://issues.apache.org/jira/browse/HIVE-12080
> This improvement is very useful for users who have schema evolution on their
> tables. For example, a Hive table with a "bigint" can read parquet files with
> "int32" and "int64" types.
> The patch only supports widening conversions from int->bigint and
> float->double. We should support more types to allow users read their changed
> parquet schema.
> Here's a list of widening conversions we should support:
> {code}
> tinyint -> smallint,int,bigint,float,double
> smallint -> int,bigint,float,double
> int -> bigint,float,double
> bigint -> float,double
> float -> double
> double -> --
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-14092) Kryo exception when deserializing VectorFileSinkOperator
[
https://issues.apache.org/jira/browse/HIVE-14092?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Prasanth Jayachandran updated HIVE-14092:
-
Resolution: Fixed
Fix Version/s: 2.1.1
2.2.0
Status: Resolved (was: Patch Available)
Committed to branch-2.1 and master
> Kryo exception when deserializing VectorFileSinkOperator
>
>
> Key: HIVE-14092
> URL: https://issues.apache.org/jira/browse/HIVE-14092
> Project: Hive
> Issue Type: Bug
> Components: Serializers/Deserializers
>Affects Versions: 2.2.0
>Reporter: Prasanth Jayachandran
>Assignee: Prasanth Jayachandran
> Fix For: 2.2.0, 2.1.1
>
> Attachments: HIVE-14092.1.patch
>
>
> Following exception is thrown for queries using VectorFileSinkOperator
> {code}
> Caused by: java.lang.IllegalArgumentException: Unable to create serializer
> "org.apache.hive.com.esotericsoftware.kryo.serializers.FieldSerializer" for
> class: org.apache.hadoop.hive.ql.exec.vector.VectorFileSinkOperator
> at
> org.apache.hive.com.esotericsoftware.kryo.factories.ReflectionSerializerFactory.makeSerializer(ReflectionSerializerFactory.java:67)
> at
> org.apache.hive.com.esotericsoftware.kryo.factories.ReflectionSerializerFactory.makeSerializer(ReflectionSerializerFactory.java:45)
> at
> org.apache.hive.com.esotericsoftware.kryo.Kryo.newDefaultSerializer(Kryo.java:380)
> at
> org.apache.hive.com.esotericsoftware.kryo.Kryo.getDefaultSerializer(Kryo.java:364)
> at
> org.apache.hive.com.esotericsoftware.kryo.util.DefaultClassResolver.registerImplicit(DefaultClassResolver.java:74)
> at
> org.apache.hive.com.esotericsoftware.kryo.Kryo.getRegistration(Kryo.java:490)
> at
> org.apache.hive.com.esotericsoftware.kryo.util.DefaultClassResolver.readName(DefaultClassResolver.java:166)
> at
> org.apache.hive.com.esotericsoftware.kryo.util.DefaultClassResolver.readClass(DefaultClassResolver.java:133)
> at
> org.apache.hive.com.esotericsoftware.kryo.Kryo.readClass(Kryo.java:670)
> at
> org.apache.hadoop.hive.ql.exec.SerializationUtilities$KryoWithHooks.readClass(SerializationUtilities.java:180)
> at
> org.apache.hive.com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:781)
> at
> org.apache.hadoop.hive.ql.exec.SerializationUtilities$KryoWithHooks.readClassAndObject(SerializationUtilities.java:175)
> at
> org.apache.hive.com.esotericsoftware.kryo.serializers.CollectionSerializer.read(CollectionSerializer.java:134)
> at
> org.apache.hive.com.esotericsoftware.kryo.serializers.CollectionSerializer.read(CollectionSerializer.java:40)
> at
> org.apache.hive.com.esotericsoftware.kryo.Kryo.readObject(Kryo.java:708)
> at
> org.apache.hadoop.hive.ql.exec.SerializationUtilities$KryoWithHooks.readObject(SerializationUtilities.java:213)
> at
> org.apache.hive.com.esotericsoftware.kryo.serializers.ObjectField.read(ObjectField.java:125)
> ... 46 more
> Caused by: java.lang.reflect.InvocationTargetException
> at sun.reflect.GeneratedConstructorAccessor6.newInstance(Unknown Source)
> at
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
> at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
> at
> org.apache.hive.com.esotericsoftware.kryo.factories.ReflectionSerializerFactory.makeSerializer(ReflectionSerializerFactory.java:54)
> ... 62 more
> Caused by: java.lang.StackOverflowError
> at java.util.HashMap.hash(HashMap.java:338)
> at java.util.HashMap.get(HashMap.java:556)
> at
> org.apache.hive.com.esotericsoftware.kryo.Generics.getConcreteClass(Generics.java:61)
> at
> org.apache.hive.com.esotericsoftware.kryo.Generics.getConcreteClass(Generics.java:62)
> at
> org.apache.hive.com.esotericsoftware.kryo.Generics.getConcreteClass(Generics.java:62)
> at
> org.apache.hive.com.esotericsoftware.kryo.Generics.getConcreteClass(Generics.java:62)
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Assigned] (HIVE-14053) Hive should report that primary keys can't be null.
[
https://issues.apache.org/jira/browse/HIVE-14053?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hari Sankar Sivarama Subramaniyan reassigned HIVE-14053:
Assignee: Hari Sankar Sivarama Subramaniyan
> Hive should report that primary keys can't be null.
> ---
>
> Key: HIVE-14053
> URL: https://issues.apache.org/jira/browse/HIVE-14053
> Project: Hive
> Issue Type: Bug
>Reporter: Carter Shanklin
>Assignee: Hari Sankar Sivarama Subramaniyan
>Priority: Minor
>
> HIVE-13076 introduces "rely novalidate" primary and foreign keys to Hive.
> With the right driver in place, tools like Tableau can do join elimination
> and queries can run much faster.
> Some gaps remain, currently getAttributes() in HiveDatabaseMetaData doesn't
> work quite right for keys. In particular, primary keys by definition are not
> null and the metadata should reflect this for improved join elimination.
> In this example that uses the TPC-H schema and its constraints, we sum
> l_extendedprice and group by l_shipmode. This query should not use more than
> just the lineitem table.
> With all the constraints in place, Tableau generates this query:
> {code}
> SELECT `lineitem`.`l_shipmode` AS `l_shipmode`,
> SUM(`lineitem`.`l_extendedprice`) AS `sum_l_extendedprice_ok`
> FROM `tpch_bin_flat_orc_2`.`lineitem` `lineitem`
> JOIN `tpch_bin_flat_orc_2`.`orders` `orders` ON (`lineitem`.`l_orderkey` =
> `orders`.`o_orderkey`)
> JOIN `tpch_bin_flat_orc_2`.`customer` `customer` ON (`orders`.`o_custkey` =
> `customer`.`c_custkey`)
> JOIN `tpch_bin_flat_orc_2`.`nation` `nation` ON (`customer`.`c_nationkey` =
> `nation`.`n_nationkey`)
> WHERE NOT (`lineitem`.`l_partkey` IS NULL)) AND (NOT
> (`lineitem`.`l_suppkey` IS NULL))) AND ((NOT (`lineitem`.`l_partkey` IS
> NULL)) AND (NOT (`lineitem`.`l_suppkey` IS NULL AND (NOT
> (`nation`.`n_regionkey` IS NULL)))
> {code}
> Since these are the primary keys the denormalization and the where condition
> is unnecessary and this sort of query can be a lot faster by just accessing
> the lineitem table.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-13995) Hive generates inefficient metastore queries for TPCDS tables with 1800+ partitions leading to higher compile time
[
https://issues.apache.org/jira/browse/HIVE-13995?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hari Sankar Sivarama Subramaniyan updated HIVE-13995:
-
Status: Patch Available (was: Open)
> Hive generates inefficient metastore queries for TPCDS tables with 1800+
> partitions leading to higher compile time
> --
>
> Key: HIVE-13995
> URL: https://issues.apache.org/jira/browse/HIVE-13995
> Project: Hive
> Issue Type: Bug
> Components: Hive
>Affects Versions: 2.2.0
>Reporter: Nita Dembla
>Assignee: Hari Sankar Sivarama Subramaniyan
> Attachments: HIVE-13995.1.patch
>
>
> TPCDS fact tables (store_sales, catalog_sales) have 1800+ partitions and when
> the query does not a filter on the partition column, metastore queries
> generated have a large IN clause listing all the partition names. Most RDBMS
> systems have issues optimizing large IN clause and even when a good index
> plan is chosen , comparing to 1800+ string values will not lead to best
> execution time.
> When all partitions are chosen, not specifying the partition list and having
> filters only on table and column name will generate the same result set as
> long as there are no concurrent modifications to partition list of the hive
> table (adding/dropping partitions).
> For eg: For TPCDS query18, the metastore query gathering partition column
> statistics runs in 0.5 secs in Mysql. Following is output from mysql log
> {noformat}
> -- Query_time: 0.482063 Lock_time: 0.003037 Rows_sent: 1836 Rows_examined:
> 18360
> select count("COLUMN_NAME") from "PART_COL_STATS"
> where "DB_NAME" = 'tpcds_bin_partitioned_orc_3' and "TABLE_NAME" =
> 'catalog_sales'
> and "COLUMN_NAME" in
> ('cs_bill_customer_sk','cs_bill_cdemo_sk','cs_item_sk','cs_quantity','cs_list_price','cs_sales_price','cs_coupon_amt','cs_net_profit')
> and "PARTITION_NAME" in
> ('cs_sold_date_sk=2450815','cs_sold_date_sk=2450816','cs_sold_date_sk=2450817','cs_sold_date_sk=2450818','cs_sold_date_sk=2450819','cs_sold_date_sk=2450820','cs_sold_date_sk=2450821','cs_sold_date_sk=2450822','cs_sold_date_sk=2450823','cs_sold_date_sk=2450824','cs_sold_date_sk=2450825','cs_sold_date_sk=2450826','cs_sold_date_sk=2450827','cs_sold_date_sk=2450828','cs_sold_date_sk=2450829','cs_sold_date_sk=2450830','cs_sold_date_sk=2450831','cs_sold_date_sk=2450832','cs_sold_date_sk=2450833','cs_sold_date_sk=2450834','cs_sold_date_sk=2450835','cs_sold_date_sk=2450836','cs_sold_date_sk=2450837','cs_sold_date_sk=2450838','cs_sold_date_sk=2450839','cs_sold_date_sk=2450840','cs_sold_date_sk=2450841','cs_sold_date_sk=2450842','cs_sold_date_sk=2450843','cs_sold_date_sk=2450844','cs_sold_date_sk=2450845','cs_sold_date_sk=2450846','cs_sold_date_sk=2450847','cs_sold_date_sk=2450848','cs_sold_date_sk=2450849','cs_sold_date_sk=2450850','cs_sold_date_sk=2450851','cs_sold_date_sk=2450852','cs_sold_date_sk=2450853','cs_sold_date_sk=2450854','cs_sold_date_sk=2450855','cs_sold_date_sk=2450856',...,'cs_sold_date_sk=2452654')
> group by "PARTITION_NAME";
> {noformat}
> Functionally equivalent query runs in 0.1 seconds
> {noformat}
> --Query_time: 0.121296 Lock_time: 0.000156 Rows_sent: 1836 Rows_examined:
> 18360
> select count("COLUMN_NAME") from "PART_COL_STATS"
> where "DB_NAME" = 'tpcds_bin_partitioned_orc_3' and "TABLE_NAME" =
> 'catalog_sales' and "COLUMN_NAME" in
> ('cs_bill_customer_sk','cs_bill_cdemo_sk','cs_item_sk','cs_quantity','cs_list_price','cs_sales_price','cs_coupon_amt','cs_net_profit')
> group by "PARTITION_NAME";
> {noformat}
> If removing the partition list seems drastic, its also possible to simply
> list the range since hive gets a ordered list of partition names. This
> performs equally well as earlier query
> {noformat}
> # Query_time: 0.143874 Lock_time: 0.000154 Rows_sent: 1836 Rows_examined:
> 18360
> SET timestamp=1464014881;
> select count("COLUMN_NAME") from "PART_COL_STATS" where "DB_NAME" =
> 'tpcds_bin_partitioned_orc_3' and "TABLE_NAME" = 'catalog_sales' and
> "COLUMN_NAME" in
> ('cs_bill_customer_sk','cs_bill_cdemo_sk','cs_item_sk','cs_quantity','cs_list_price','cs_sales_price','cs_coupon_amt','cs_net_profit')
> and "PARTITION_NAME" >= 'cs_sold_date_sk=2450815' and "PARTITION_NAME" <=
> 'cs_sold_date_sk=2452654'
> group by "PARTITION_NAME";
> {noformat}
> Another thing to check is the IN clause of column names. Columns in
> projection list of hive query are mentioned here. Not sure if statistics of
> these columns are required for hive query optimization.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-14073) update config whiltelist for sql std authorization
[ https://issues.apache.org/jira/browse/HIVE-14073?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thejas M Nair updated HIVE-14073: - Attachment: HIVE-14073.2.patch > update config whiltelist for sql std authorization > --- > > Key: HIVE-14073 > URL: https://issues.apache.org/jira/browse/HIVE-14073 > Project: Hive > Issue Type: Bug > Components: Security, SQLStandardAuthorization >Affects Versions: 2.1.0 >Reporter: Thejas M Nair >Assignee: Thejas M Nair > Attachments: HIVE-14073-branch-1.1.patch, > HIVE-14073-branch-1.2.patch, HIVE-14073.1.patch, HIVE-14073.2.patch > > > New configs that should go in security whitelist have been added. Whitelist > needs updating. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-14073) update config whiltelist for sql std authorization
[ https://issues.apache.org/jira/browse/HIVE-14073?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thejas M Nair updated HIVE-14073: - Attachment: HIVE-14073-branch-1.2.patch OOZIE-2355 sets oozie.* params to enable tracking of hive jobs. Adding regex to allow those params to be set. > update config whiltelist for sql std authorization > --- > > Key: HIVE-14073 > URL: https://issues.apache.org/jira/browse/HIVE-14073 > Project: Hive > Issue Type: Bug > Components: Security, SQLStandardAuthorization >Affects Versions: 2.1.0 >Reporter: Thejas M Nair >Assignee: Thejas M Nair > Attachments: HIVE-14073-branch-1.1.patch, > HIVE-14073-branch-1.2.patch, HIVE-14073.1.patch > > > New configs that should go in security whitelist have been added. Whitelist > needs updating. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13803) More aggressive inference of transitive predicates for inner joins
[
https://issues.apache.org/jira/browse/HIVE-13803?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15351376#comment-15351376
]
Jesus Camacho Rodriguez commented on HIVE-13803:
I just did. I pointed out the changes that I made in the code that I brought
from Calcite so it is easier for you to review it.
> More aggressive inference of transitive predicates for inner joins
> --
>
> Key: HIVE-13803
> URL: https://issues.apache.org/jira/browse/HIVE-13803
> Project: Hive
> Issue Type: Sub-task
> Components: CBO
>Affects Versions: 2.1.0
>Reporter: Jesus Camacho Rodriguez
>Assignee: Jesus Camacho Rodriguez
> Attachments: HIVE-13803.01.patch, HIVE-13803.02.patch,
> HIVE-13803.patch
>
>
> Follow-up of HIVE-13068.
> Currently for inner joins, we do not infer transitive predicates that do not
> reference any of the columns of the input. These predicates can be evaluated
> statically and can be useful to quickly discard intermediary results.
> Ex. ql/src/test/results/clientpositive/constprog3.q.out
> {noformat}
> explain
> select table1.id, table1.val, table1.val1
> from table1 inner join table3
> on table1.dimid = table3.id and table3.id = 1 where table1.dimid <> 1
> {noformat}
> Current plan:
> {noformat}
> STAGE DEPENDENCIES:
> Stage-1 is a root stage
> Stage-0 depends on stages: Stage-1
> STAGE PLANS:
> Stage: Stage-1
> Map Reduce
> Map Operator Tree:
> TableScan
> alias: table1
> Statistics: Num rows: 1 Data size: 0 Basic stats: PARTIAL Column
> stats: NONE
> Filter Operator
> predicate: false (type: boolean)
> Statistics: Num rows: 1 Data size: 0 Basic stats: PARTIAL
> Column stats: NONE
> Select Operator
> expressions: id (type: int), val (type: int), val1 (type: int)
> outputColumnNames: _col0, _col1, _col2
> Statistics: Num rows: 1 Data size: 0 Basic stats: PARTIAL
> Column stats: NONE
> Reduce Output Operator
> sort order:
> Statistics: Num rows: 1 Data size: 0 Basic stats: PARTIAL
> Column stats: NONE
> value expressions: _col0 (type: int), _col1 (type: int),
> _col2 (type: int)
> TableScan
> alias: table3
> Statistics: Num rows: 1 Data size: 0 Basic stats: PARTIAL Column
> stats: NONE
> Filter Operator
> predicate: (id = 1) (type: boolean)
> Statistics: Num rows: 1 Data size: 0 Basic stats: PARTIAL
> Column stats: NONE
> Select Operator
> Statistics: Num rows: 1 Data size: 0 Basic stats: PARTIAL
> Column stats: NONE
> Reduce Output Operator
> sort order:
> Statistics: Num rows: 1 Data size: 0 Basic stats: PARTIAL
> Column stats: NONE
> Reduce Operator Tree:
> Join Operator
> condition map:
>Inner Join 0 to 1
> keys:
> 0
> 1
> outputColumnNames: _col0, _col1, _col2
> Statistics: Num rows: 1 Data size: 0 Basic stats: PARTIAL Column
> stats: NONE
> File Output Operator
> compressed: false
> Statistics: Num rows: 1 Data size: 0 Basic stats: PARTIAL Column
> stats: NONE
> table:
> input format: org.apache.hadoop.mapred.SequenceFileInputFormat
> output format:
> org.apache.hadoop.hive.ql.io.HiveSequenceFileOutputFormat
> serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe
> Stage: Stage-0
> Fetch Operator
> limit: -1
> Processor Tree:
> ListSink
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-13803) More aggressive inference of transitive predicates for inner joins
[
https://issues.apache.org/jira/browse/HIVE-13803?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15351371#comment-15351371
]
Ashutosh Chauhan commented on HIVE-13803:
-
Can you create a RB for this?
> More aggressive inference of transitive predicates for inner joins
> --
>
> Key: HIVE-13803
> URL: https://issues.apache.org/jira/browse/HIVE-13803
> Project: Hive
> Issue Type: Sub-task
> Components: CBO
>Affects Versions: 2.1.0
>Reporter: Jesus Camacho Rodriguez
>Assignee: Jesus Camacho Rodriguez
> Attachments: HIVE-13803.01.patch, HIVE-13803.02.patch,
> HIVE-13803.patch
>
>
> Follow-up of HIVE-13068.
> Currently for inner joins, we do not infer transitive predicates that do not
> reference any of the columns of the input. These predicates can be evaluated
> statically and can be useful to quickly discard intermediary results.
> Ex. ql/src/test/results/clientpositive/constprog3.q.out
> {noformat}
> explain
> select table1.id, table1.val, table1.val1
> from table1 inner join table3
> on table1.dimid = table3.id and table3.id = 1 where table1.dimid <> 1
> {noformat}
> Current plan:
> {noformat}
> STAGE DEPENDENCIES:
> Stage-1 is a root stage
> Stage-0 depends on stages: Stage-1
> STAGE PLANS:
> Stage: Stage-1
> Map Reduce
> Map Operator Tree:
> TableScan
> alias: table1
> Statistics: Num rows: 1 Data size: 0 Basic stats: PARTIAL Column
> stats: NONE
> Filter Operator
> predicate: false (type: boolean)
> Statistics: Num rows: 1 Data size: 0 Basic stats: PARTIAL
> Column stats: NONE
> Select Operator
> expressions: id (type: int), val (type: int), val1 (type: int)
> outputColumnNames: _col0, _col1, _col2
> Statistics: Num rows: 1 Data size: 0 Basic stats: PARTIAL
> Column stats: NONE
> Reduce Output Operator
> sort order:
> Statistics: Num rows: 1 Data size: 0 Basic stats: PARTIAL
> Column stats: NONE
> value expressions: _col0 (type: int), _col1 (type: int),
> _col2 (type: int)
> TableScan
> alias: table3
> Statistics: Num rows: 1 Data size: 0 Basic stats: PARTIAL Column
> stats: NONE
> Filter Operator
> predicate: (id = 1) (type: boolean)
> Statistics: Num rows: 1 Data size: 0 Basic stats: PARTIAL
> Column stats: NONE
> Select Operator
> Statistics: Num rows: 1 Data size: 0 Basic stats: PARTIAL
> Column stats: NONE
> Reduce Output Operator
> sort order:
> Statistics: Num rows: 1 Data size: 0 Basic stats: PARTIAL
> Column stats: NONE
> Reduce Operator Tree:
> Join Operator
> condition map:
>Inner Join 0 to 1
> keys:
> 0
> 1
> outputColumnNames: _col0, _col1, _col2
> Statistics: Num rows: 1 Data size: 0 Basic stats: PARTIAL Column
> stats: NONE
> File Output Operator
> compressed: false
> Statistics: Num rows: 1 Data size: 0 Basic stats: PARTIAL Column
> stats: NONE
> table:
> input format: org.apache.hadoop.mapred.SequenceFileInputFormat
> output format:
> org.apache.hadoop.hive.ql.io.HiveSequenceFileOutputFormat
> serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe
> Stage: Stage-0
> Fetch Operator
> limit: -1
> Processor Tree:
> ListSink
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-14090) JDOExceptions thrown by the Metastore have their full stack trace returned to clients
[
https://issues.apache.org/jira/browse/HIVE-14090?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sahil Takiar updated HIVE-14090:
Status: Patch Available (was: In Progress)
> JDOExceptions thrown by the Metastore have their full stack trace returned to
> clients
> -
>
> Key: HIVE-14090
> URL: https://issues.apache.org/jira/browse/HIVE-14090
> Project: Hive
> Issue Type: Bug
>Affects Versions: 2.1.0, 1.1.0
>Reporter: Sahil Takiar
>Assignee: Sahil Takiar
> Attachments: HIVE-14090.patch
>
>
> When user try to create any database or table with a name longer than 128
> characters:
> {code}
> create database
> test_longname_looonglooonglooonglooonglooonglooonglooonglooonglooonglooonglooongNametableFAIL;
> {code}
> It dumps the full exception stack-trace in a non-user-friendly message. The
> lends to relatively negative user-experience for Beeline users who hit this
> exception, they are generally not interested in the full stack-trace.
> The formatted stack-trace is below:
> {code}
> Error while processing statement: FAILED: Execution Error, return code 1 from
> org.apache.hadoop.hive.ql.exec.DDLTask.
> MetaException(message:javax.jdo.JDOFatalUserException: Attempt to store value
> "test_longname_looonglooonglooonglooonglooonglooonglooonglooonglooonglooonglooongnametablefail2"
> in column "`NAME`" that has maximum length of 128. Please correct your data!
> at
> org.datanucleus.api.jdo.NucleusJDOHelper.getJDOExceptionForNucleusException(NucleusJDOHelper.java:528)
> at
> org.datanucleus.api.jdo.JDOPersistenceManager.jdoMakePersistent(JDOPersistenceManager.java:732)
> at
> org.datanucleus.api.jdo.JDOPersistenceManager.makePersistent(JDOPersistenceManager.java:752)
> at
> org.apache.hadoop.hive.metastore.ObjectStore.createDatabase(ObjectStore.java:569)
> at sun.reflect.GeneratedMethodAccessor31.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at
> org.apache.hadoop.hive.metastore.RawStoreProxy.invoke(RawStoreProxy.java:114)
> at com.sun.proxy.$Proxy10.createDatabase(Unknown Source)
> at
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.create_database_core(HiveMetaStore.java:923)
> at
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.create_database(HiveMetaStore.java:962)
> at sun.reflect.GeneratedMethodAccessor30.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at
> org.apache.hadoop.hive.metastore.RetryingHMSHandler.invokeInternal(RetryingHMSHandler.java:138)
> at
> org.apache.hadoop.hive.metastore.RetryingHMSHandler.invoke(RetryingHMSHandler.java:99)
> at com.sun.proxy.$Proxy12.create_database(Unknown Source)
> at
> org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$create_database.getResult(ThriftHiveMetastore.java:8863)
> at
> org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$create_database.getResult(ThriftHiveMetastore.java:8847)
> at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
> at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39)
> at
> org.apache.hadoop.hive.thrift.HadoopThriftAuthBridge$Server$TUGIAssumingProcessor$1.run(HadoopThriftAuthBridge.java:707)
> at
> org.apache.hadoop.hive.thrift.HadoopThriftAuthBridge$Server$TUGIAssumingProcessor$1.run(HadoopThriftAuthBridge.java:702)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:415)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1693)
> at
> org.apache.hadoop.hive.thrift.HadoopThriftAuthBridge$Server$TUGIAssumingProcessor.process(HadoopThriftAuthBridge.java:702)
> at
> org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:286)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:745) NestedThrowablesStackTrace: Attempt
> to store value
> "test_longname_looonglooonglooonglooonglooonglooonglooonglooonglooonglooonglooongnametablefail2"
> in column "`NAME`" that has maximum length of 128. Please correct your data!
> org.datanucleus.exceptions.NucleusUserException: Attempt to store value
> "test_longname_looonglooonglooonglooonglooonglooonglooonglooonglooonglooonglooongnametablefail2"
> in column
[jira] [Commented] (HIVE-13803) More aggressive inference of transitive predicates for inner joins
[
https://issues.apache.org/jira/browse/HIVE-13803?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15351313#comment-15351313
]
Jesus Camacho Rodriguez commented on HIVE-13803:
[~ashutoshc], fails are unrelated, could you review? Thanks
> More aggressive inference of transitive predicates for inner joins
> --
>
> Key: HIVE-13803
> URL: https://issues.apache.org/jira/browse/HIVE-13803
> Project: Hive
> Issue Type: Sub-task
> Components: CBO
>Affects Versions: 2.1.0
>Reporter: Jesus Camacho Rodriguez
>Assignee: Jesus Camacho Rodriguez
> Attachments: HIVE-13803.01.patch, HIVE-13803.02.patch,
> HIVE-13803.patch
>
>
> Follow-up of HIVE-13068.
> Currently for inner joins, we do not infer transitive predicates that do not
> reference any of the columns of the input. These predicates can be evaluated
> statically and can be useful to quickly discard intermediary results.
> Ex. ql/src/test/results/clientpositive/constprog3.q.out
> {noformat}
> explain
> select table1.id, table1.val, table1.val1
> from table1 inner join table3
> on table1.dimid = table3.id and table3.id = 1 where table1.dimid <> 1
> {noformat}
> Current plan:
> {noformat}
> STAGE DEPENDENCIES:
> Stage-1 is a root stage
> Stage-0 depends on stages: Stage-1
> STAGE PLANS:
> Stage: Stage-1
> Map Reduce
> Map Operator Tree:
> TableScan
> alias: table1
> Statistics: Num rows: 1 Data size: 0 Basic stats: PARTIAL Column
> stats: NONE
> Filter Operator
> predicate: false (type: boolean)
> Statistics: Num rows: 1 Data size: 0 Basic stats: PARTIAL
> Column stats: NONE
> Select Operator
> expressions: id (type: int), val (type: int), val1 (type: int)
> outputColumnNames: _col0, _col1, _col2
> Statistics: Num rows: 1 Data size: 0 Basic stats: PARTIAL
> Column stats: NONE
> Reduce Output Operator
> sort order:
> Statistics: Num rows: 1 Data size: 0 Basic stats: PARTIAL
> Column stats: NONE
> value expressions: _col0 (type: int), _col1 (type: int),
> _col2 (type: int)
> TableScan
> alias: table3
> Statistics: Num rows: 1 Data size: 0 Basic stats: PARTIAL Column
> stats: NONE
> Filter Operator
> predicate: (id = 1) (type: boolean)
> Statistics: Num rows: 1 Data size: 0 Basic stats: PARTIAL
> Column stats: NONE
> Select Operator
> Statistics: Num rows: 1 Data size: 0 Basic stats: PARTIAL
> Column stats: NONE
> Reduce Output Operator
> sort order:
> Statistics: Num rows: 1 Data size: 0 Basic stats: PARTIAL
> Column stats: NONE
> Reduce Operator Tree:
> Join Operator
> condition map:
>Inner Join 0 to 1
> keys:
> 0
> 1
> outputColumnNames: _col0, _col1, _col2
> Statistics: Num rows: 1 Data size: 0 Basic stats: PARTIAL Column
> stats: NONE
> File Output Operator
> compressed: false
> Statistics: Num rows: 1 Data size: 0 Basic stats: PARTIAL Column
> stats: NONE
> table:
> input format: org.apache.hadoop.mapred.SequenceFileInputFormat
> output format:
> org.apache.hadoop.hive.ql.io.HiveSequenceFileOutputFormat
> serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe
> Stage: Stage-0
> Fetch Operator
> limit: -1
> Processor Tree:
> ListSink
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-14098) Logging task properties, and environment variables might contain passwords
[ https://issues.apache.org/jira/browse/HIVE-14098?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergio Peña updated HIVE-14098: --- Resolution: Fixed Fix Version/s: 2.2.0 Status: Resolved (was: Patch Available) Thanks [~pvary]. I committed this to 2.2. > Logging task properties, and environment variables might contain passwords > -- > > Key: HIVE-14098 > URL: https://issues.apache.org/jira/browse/HIVE-14098 > Project: Hive > Issue Type: Bug > Components: HiveServer2, Logging, Spark >Reporter: Peter Vary >Assignee: Peter Vary > Fix For: 2.2.0 > > Attachments: HIVE-14098.2.patch, HIVE-14098.patch > > > Hive MapredLocalTask Can Print Environment Passwords, like > -Djavax.net.ssl.trustStorePassword. > The same could happen, when logging spark properties -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13997) Insert overwrite directory doesn't overwrite existing files
[
https://issues.apache.org/jira/browse/HIVE-13997?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ashutosh Chauhan updated HIVE-13997:
Affects Version/s: 2.1.0
> Insert overwrite directory doesn't overwrite existing files
> ---
>
> Key: HIVE-13997
> URL: https://issues.apache.org/jira/browse/HIVE-13997
> Project: Hive
> Issue Type: Bug
> Components: Query Processor
>Affects Versions: 2.1.0
>Reporter: Rui Li
>Assignee: Rui Li
> Fix For: 2.2.0, 2.1.1
>
> Attachments: HIVE-13997.1.patch, HIVE-13997.2.patch,
> HIVE-13997.2.patch
>
>
> Can be easily reproduced by running {{INSERT OVERWRITE DIRECTORY}} to the
> same dir twice.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-13997) Insert overwrite directory doesn't overwrite existing files
[
https://issues.apache.org/jira/browse/HIVE-13997?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ashutosh Chauhan updated HIVE-13997:
Component/s: Query Processor
> Insert overwrite directory doesn't overwrite existing files
> ---
>
> Key: HIVE-13997
> URL: https://issues.apache.org/jira/browse/HIVE-13997
> Project: Hive
> Issue Type: Bug
> Components: Query Processor
>Affects Versions: 2.1.0
>Reporter: Rui Li
>Assignee: Rui Li
> Fix For: 2.2.0, 2.1.1
>
> Attachments: HIVE-13997.1.patch, HIVE-13997.2.patch,
> HIVE-13997.2.patch
>
>
> Can be easily reproduced by running {{INSERT OVERWRITE DIRECTORY}} to the
> same dir twice.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-13997) Insert overwrite directory doesn't overwrite existing files
[
https://issues.apache.org/jira/browse/HIVE-13997?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ashutosh Chauhan updated HIVE-13997:
Resolution: Fixed
Fix Version/s: 2.1.1
2.2.0
Status: Resolved (was: Patch Available)
Pushed to master & branch-2.1 Thanks, Rui!
> Insert overwrite directory doesn't overwrite existing files
> ---
>
> Key: HIVE-13997
> URL: https://issues.apache.org/jira/browse/HIVE-13997
> Project: Hive
> Issue Type: Bug
> Components: Query Processor
>Affects Versions: 2.1.0
>Reporter: Rui Li
>Assignee: Rui Li
> Fix For: 2.2.0, 2.1.1
>
> Attachments: HIVE-13997.1.patch, HIVE-13997.2.patch,
> HIVE-13997.2.patch
>
>
> Can be easily reproduced by running {{INSERT OVERWRITE DIRECTORY}} to the
> same dir twice.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-13991) Union All on view fail with no valid permission on underneath table
[
https://issues.apache.org/jira/browse/HIVE-13991?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yongzhi Chen updated HIVE-13991:
Resolution: Fixed
Fix Version/s: 2.1.1
2.2.0
Status: Resolved (was: Patch Available)
Thanks Aihua for the review, committed to master and branch-2.1
> Union All on view fail with no valid permission on underneath table
> ---
>
> Key: HIVE-13991
> URL: https://issues.apache.org/jira/browse/HIVE-13991
> Project: Hive
> Issue Type: Bug
> Components: Query Planning
>Reporter: Yongzhi Chen
>Assignee: Yongzhi Chen
> Fix For: 2.2.0, 2.1.1
>
> Attachments: HIVE-13991.1.patch, HIVE-13991.2.patch
>
>
> When sentry is enabled.
> create view V as select * from T;
> When the user has read permission on view V, but does not have read
> permission on table T,
> select * from V union all select * from V
> failed with:
> {noformat}
> 0: jdbc:hive2://> select * from s07view union all select * from
> s07view limit 1;
> Error: Error while compiling statement: FAILED: SemanticException No valid
> privileges
> Required privileges for this query:
> Server=server1->Db=default->Table=sample_07->action=select;
> (state=42000,code=4)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-14037) java.lang.ClassNotFoundException for the jar in hive.reloadable.aux.jars.path in mapreduce
[
https://issues.apache.org/jira/browse/HIVE-14037?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15351254#comment-15351254
]
Hive QA commented on HIVE-14037:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12813656/HIVE-14037.3.patch
{color:green}SUCCESS:{color} +1 due to 2 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 5 failed/errored test(s), 10271 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_list_bucket_dml_13
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_stats_list_bucket
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_vector_complex_all
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_vector_complex_join
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_index_bitmap3
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-MASTER-Build/275/testReport
Console output:
https://builds.apache.org/job/PreCommit-HIVE-MASTER-Build/275/console
Test logs:
http://ec2-50-18-27-0.us-west-1.compute.amazonaws.com/logs/PreCommit-HIVE-MASTER-Build-275/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 5 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12813656 - PreCommit-HIVE-MASTER-Build
> java.lang.ClassNotFoundException for the jar in hive.reloadable.aux.jars.path
> in mapreduce
> ---
>
> Key: HIVE-14037
> URL: https://issues.apache.org/jira/browse/HIVE-14037
> Project: Hive
> Issue Type: Bug
>Reporter: Aihua Xu
>Assignee: Aihua Xu
> Attachments: HIVE-14037.1.patch, HIVE-14037.2.patch,
> HIVE-14037.3.patch
>
>
> The jars in hive.reloadable.aux.jars.path seem to be available in HS2 process
> while they are not available in the Mapper or Reducer nodes which will throw
> the following exception.
> {noformat}
> org.apache.hadoop.hive.ql.exec.ExprNodeGenericFuncEvaluator.isDeterministic(ExprNodeGenericFuncEvaluator.java:152)
> at
> org.apache.hadoop.hive.ql.exec.ExprNodeEvaluatorFactory.iterate(ExprNodeEvaluatorFactory.java:97)
> at
> org.apache.hadoop.hive.ql.exec.ExprNodeEvaluatorFactory.toCachedEvals(ExprNodeEvaluatorFactory.java:71)
> at
> org.apache.hadoop.hive.ql.exec.SelectOperator.initializeOp(SelectOperator.java:59)
> at org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:385)
> at org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:469)
> at
> org.apache.hadoop.hive.ql.exec.Operator.initializeChildren(Operator.java:425)
> at
> org.apache.hadoop.hive.ql.exec.TableScanOperator.initializeOp(TableScanOperator.java:193)
> at org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:385)
> at
> org.apache.hadoop.hive.ql.exec.MapOperator.initializeOp(MapOperator.java:431)
> at org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:385)
> at
> org.apache.hadoop.hive.ql.exec.mr.ExecMapper.configure(ExecMapper.java:126)
> ... 22 more
> Caused by: java.lang.ClassNotFoundException: test.UDF
> at java.net.URLClassLoader$1.run(URLClassLoader.java:366)
> at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
> at java.security.AccessController.doPrivileged(Native Method)
> at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
> at java.lang.Class.forName0(Native Method)
> at java.lang.Class.forName(Class.java:270)
> at
> org.apache.hadoop.hive.ql.udf.generic.GenericUDFBridge.getUdfClass(GenericUDFBridge.java:132)
> ... 36 more
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-13945) Decimal value is displayed as rounded when selecting where clause with that decimal value.
[
https://issues.apache.org/jira/browse/HIVE-13945?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15351246#comment-15351246
]
Ashutosh Chauhan commented on HIVE-13945:
-
Patch LGTM. But likely needs golden file updates.
> Decimal value is displayed as rounded when selecting where clause with that
> decimal value.
> --
>
> Key: HIVE-13945
> URL: https://issues.apache.org/jira/browse/HIVE-13945
> Project: Hive
> Issue Type: Bug
>Affects Versions: 2.1.0
>Reporter: Takahiko Saito
>Assignee: Sergey Shelukhin
>Priority: Critical
> Attachments: HIVE-13945.01.patch, HIVE-13945.patch
>
>
> Create a table withe a column of decimal type(38,18) and insert
> '4327269606205.029297'. Then select with that value displays its rounded
> value, which is 4327269606205.029300
> {noformat}
> 0: jdbc:hive2://os-r7-mvjkcu-hiveserver2-11-4> drop table if exists test;
> No rows affected (0.229 seconds)
> 0: jdbc:hive2://os-r7-mvjkcu-hiveserver2-11-4>
> 0: jdbc:hive2://os-r7-mvjkcu-hiveserver2-11-4> create table test (dc
> decimal(38,18));
> No rows affected (0.125 seconds)
> 0: jdbc:hive2://os-r7-mvjkcu-hiveserver2-11-4>
> 0: jdbc:hive2://os-r7-mvjkcu-hiveserver2-11-4> insert into table test values
> (4327269606205.029297);
> No rows affected (2.372 seconds)
> 0: jdbc:hive2://os-r7-mvjkcu-hiveserver2-11-4>
> 0: jdbc:hive2://os-r7-mvjkcu-hiveserver2-11-4> select * from test;
> +---+--+
> | test.dc |
> +---+--+
> | 4327269606205.029297 |
> +---+--+
> 1 row selected (0.123 seconds)
> 0: jdbc:hive2://os-r7-mvjkcu-hiveserver2-11-4>
> 0: jdbc:hive2://os-r7-mvjkcu-hiveserver2-11-4> select * from test where dc =
> 4327269606205.029297;
> +---+--+
> | test.dc |
> +---+--+
> | 4327269606205.029300 |
> +---+--+
> 1 row selected (0.109 seconds)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-14098) Logging task properties, and environment variables might contain passwords
[ https://issues.apache.org/jira/browse/HIVE-14098?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15351154#comment-15351154 ] Sergio Peña commented on HIVE-14098: The patch looks good. +1 > Logging task properties, and environment variables might contain passwords > -- > > Key: HIVE-14098 > URL: https://issues.apache.org/jira/browse/HIVE-14098 > Project: Hive > Issue Type: Bug > Components: HiveServer2, Logging, Spark >Reporter: Peter Vary >Assignee: Peter Vary > Attachments: HIVE-14098.2.patch, HIVE-14098.patch > > > Hive MapredLocalTask Can Print Environment Passwords, like > -Djavax.net.ssl.trustStorePassword. > The same could happen, when logging spark properties -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-14098) Logging task properties, and environment variables might contain passwords
[ https://issues.apache.org/jira/browse/HIVE-14098?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15351007#comment-15351007 ] Peter Vary commented on HIVE-14098: --- Checked the results, and I think these are not related > Logging task properties, and environment variables might contain passwords > -- > > Key: HIVE-14098 > URL: https://issues.apache.org/jira/browse/HIVE-14098 > Project: Hive > Issue Type: Bug > Components: HiveServer2, Logging, Spark >Reporter: Peter Vary >Assignee: Peter Vary > Attachments: HIVE-14098.2.patch, HIVE-14098.patch > > > Hive MapredLocalTask Can Print Environment Passwords, like > -Djavax.net.ssl.trustStorePassword. > The same could happen, when logging spark properties -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-14098) Logging task properties, and environment variables might contain passwords
[
https://issues.apache.org/jira/browse/HIVE-14098?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15351004#comment-15351004
]
Hive QA commented on HIVE-14098:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12813642/HIVE-14098.2.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 6 failed/errored test(s), 10271 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_list_bucket_dml_12
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_stats_list_bucket
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_vector_complex_all
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_vector_complex_join
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_index_bitmap3
org.apache.hadoop.hive.llap.tezplugins.TestLlapTaskSchedulerService.testDelayedLocalityNodeCommErrorImmediateAllocation
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-MASTER-Build/274/testReport
Console output:
https://builds.apache.org/job/PreCommit-HIVE-MASTER-Build/274/console
Test logs:
http://ec2-50-18-27-0.us-west-1.compute.amazonaws.com/logs/PreCommit-HIVE-MASTER-Build-274/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 6 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12813642 - PreCommit-HIVE-MASTER-Build
> Logging task properties, and environment variables might contain passwords
> --
>
> Key: HIVE-14098
> URL: https://issues.apache.org/jira/browse/HIVE-14098
> Project: Hive
> Issue Type: Bug
> Components: HiveServer2, Logging, Spark
>Reporter: Peter Vary
>Assignee: Peter Vary
> Attachments: HIVE-14098.2.patch, HIVE-14098.patch
>
>
> Hive MapredLocalTask Can Print Environment Passwords, like
> -Djavax.net.ssl.trustStorePassword.
> The same could happen, when logging spark properties
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-14100) current_user() returns invalid information
[ https://issues.apache.org/jira/browse/HIVE-14100?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15350995#comment-15350995 ] Peter Vary commented on HIVE-14100: --- The description states (current_user() - Returns current user name | SessionState *UserFromAuthenticator*), but when using HadoopDefaultAuthentication, it returns the unix username running the hiveserver2 process. [~apivovarov] What was the original intention of the current_user()? Should it return the user used for the background process authentication (UserFromAuthenticator), or the user used to authenticate the connection (SessionState.get().getUserName())? Thanks, Peter > current_user() returns invalid information > -- > > Key: HIVE-14100 > URL: https://issues.apache.org/jira/browse/HIVE-14100 > Project: Hive > Issue Type: Bug > Components: Authentication, Beeline >Reporter: Peter Vary >Priority: Minor > > Using HadoopDeaultAuthenticator the current_user() returns the username of > the unix user running hiveservice2. > Using SessionStateAuthenticator the current_user returns the username which > is provided when the connection started. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-14037) java.lang.ClassNotFoundException for the jar in hive.reloadable.aux.jars.path in mapreduce
[
https://issues.apache.org/jira/browse/HIVE-14037?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15350968#comment-15350968
]
Aihua Xu commented on HIVE-14037:
-
BTW: the test cases are not related comparing to the first test result.
> java.lang.ClassNotFoundException for the jar in hive.reloadable.aux.jars.path
> in mapreduce
> ---
>
> Key: HIVE-14037
> URL: https://issues.apache.org/jira/browse/HIVE-14037
> Project: Hive
> Issue Type: Bug
>Reporter: Aihua Xu
>Assignee: Aihua Xu
> Attachments: HIVE-14037.1.patch, HIVE-14037.2.patch,
> HIVE-14037.3.patch
>
>
> The jars in hive.reloadable.aux.jars.path seem to be available in HS2 process
> while they are not available in the Mapper or Reducer nodes which will throw
> the following exception.
> {noformat}
> org.apache.hadoop.hive.ql.exec.ExprNodeGenericFuncEvaluator.isDeterministic(ExprNodeGenericFuncEvaluator.java:152)
> at
> org.apache.hadoop.hive.ql.exec.ExprNodeEvaluatorFactory.iterate(ExprNodeEvaluatorFactory.java:97)
> at
> org.apache.hadoop.hive.ql.exec.ExprNodeEvaluatorFactory.toCachedEvals(ExprNodeEvaluatorFactory.java:71)
> at
> org.apache.hadoop.hive.ql.exec.SelectOperator.initializeOp(SelectOperator.java:59)
> at org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:385)
> at org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:469)
> at
> org.apache.hadoop.hive.ql.exec.Operator.initializeChildren(Operator.java:425)
> at
> org.apache.hadoop.hive.ql.exec.TableScanOperator.initializeOp(TableScanOperator.java:193)
> at org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:385)
> at
> org.apache.hadoop.hive.ql.exec.MapOperator.initializeOp(MapOperator.java:431)
> at org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:385)
> at
> org.apache.hadoop.hive.ql.exec.mr.ExecMapper.configure(ExecMapper.java:126)
> ... 22 more
> Caused by: java.lang.ClassNotFoundException: test.UDF
> at java.net.URLClassLoader$1.run(URLClassLoader.java:366)
> at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
> at java.security.AccessController.doPrivileged(Native Method)
> at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
> at java.lang.Class.forName0(Native Method)
> at java.lang.Class.forName(Class.java:270)
> at
> org.apache.hadoop.hive.ql.udf.generic.GenericUDFBridge.getUdfClass(GenericUDFBridge.java:132)
> ... 36 more
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-14037) java.lang.ClassNotFoundException for the jar in hive.reloadable.aux.jars.path in mapreduce
[
https://issues.apache.org/jira/browse/HIVE-14037?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Aihua Xu updated HIVE-14037:
Attachment: HIVE-14037.3.patch
Patch-3: minor change. Updates the incorrect comments and remove the
unnecessary spaces.
> java.lang.ClassNotFoundException for the jar in hive.reloadable.aux.jars.path
> in mapreduce
> ---
>
> Key: HIVE-14037
> URL: https://issues.apache.org/jira/browse/HIVE-14037
> Project: Hive
> Issue Type: Bug
>Reporter: Aihua Xu
>Assignee: Aihua Xu
> Attachments: HIVE-14037.1.patch, HIVE-14037.2.patch,
> HIVE-14037.3.patch
>
>
> The jars in hive.reloadable.aux.jars.path seem to be available in HS2 process
> while they are not available in the Mapper or Reducer nodes which will throw
> the following exception.
> {noformat}
> org.apache.hadoop.hive.ql.exec.ExprNodeGenericFuncEvaluator.isDeterministic(ExprNodeGenericFuncEvaluator.java:152)
> at
> org.apache.hadoop.hive.ql.exec.ExprNodeEvaluatorFactory.iterate(ExprNodeEvaluatorFactory.java:97)
> at
> org.apache.hadoop.hive.ql.exec.ExprNodeEvaluatorFactory.toCachedEvals(ExprNodeEvaluatorFactory.java:71)
> at
> org.apache.hadoop.hive.ql.exec.SelectOperator.initializeOp(SelectOperator.java:59)
> at org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:385)
> at org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:469)
> at
> org.apache.hadoop.hive.ql.exec.Operator.initializeChildren(Operator.java:425)
> at
> org.apache.hadoop.hive.ql.exec.TableScanOperator.initializeOp(TableScanOperator.java:193)
> at org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:385)
> at
> org.apache.hadoop.hive.ql.exec.MapOperator.initializeOp(MapOperator.java:431)
> at org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:385)
> at
> org.apache.hadoop.hive.ql.exec.mr.ExecMapper.configure(ExecMapper.java:126)
> ... 22 more
> Caused by: java.lang.ClassNotFoundException: test.UDF
> at java.net.URLClassLoader$1.run(URLClassLoader.java:366)
> at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
> at java.security.AccessController.doPrivileged(Native Method)
> at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
> at java.lang.Class.forName0(Native Method)
> at java.lang.Class.forName(Class.java:270)
> at
> org.apache.hadoop.hive.ql.udf.generic.GenericUDFBridge.getUdfClass(GenericUDFBridge.java:132)
> ... 36 more
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-14098) Logging task properties, and environment variables might contain passwords
[ https://issues.apache.org/jira/browse/HIVE-14098?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15350851#comment-15350851 ] Peter Vary commented on HIVE-14098: --- [~spena] please review https://reviews.apache.org/r/49259/ > Logging task properties, and environment variables might contain passwords > -- > > Key: HIVE-14098 > URL: https://issues.apache.org/jira/browse/HIVE-14098 > Project: Hive > Issue Type: Bug > Components: HiveServer2, Logging, Spark >Reporter: Peter Vary >Assignee: Peter Vary > Attachments: HIVE-14098.2.patch, HIVE-14098.patch > > > Hive MapredLocalTask Can Print Environment Passwords, like > -Djavax.net.ssl.trustStorePassword. > The same could happen, when logging spark properties -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-14098) Logging task properties, and environment variables might contain passwords
[ https://issues.apache.org/jira/browse/HIVE-14098?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Peter Vary updated HIVE-14098: -- Attachment: HIVE-14098.2.patch Git generated format > Logging task properties, and environment variables might contain passwords > -- > > Key: HIVE-14098 > URL: https://issues.apache.org/jira/browse/HIVE-14098 > Project: Hive > Issue Type: Bug > Components: HiveServer2, Logging, Spark >Reporter: Peter Vary >Assignee: Peter Vary > Attachments: HIVE-14098.2.patch, HIVE-14098.patch > > > Hive MapredLocalTask Can Print Environment Passwords, like > -Djavax.net.ssl.trustStorePassword. > The same could happen, when logging spark properties -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-14098) Logging task properties, and environment variables might contain passwords
[ https://issues.apache.org/jira/browse/HIVE-14098?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15350814#comment-15350814 ] Peter Vary commented on HIVE-14098: --- Checked the results, and I think these are not related > Logging task properties, and environment variables might contain passwords > -- > > Key: HIVE-14098 > URL: https://issues.apache.org/jira/browse/HIVE-14098 > Project: Hive > Issue Type: Bug > Components: HiveServer2, Logging, Spark >Reporter: Peter Vary >Assignee: Peter Vary > Attachments: HIVE-14098.patch > > > Hive MapredLocalTask Can Print Environment Passwords, like > -Djavax.net.ssl.trustStorePassword. > The same could happen, when logging spark properties -- This message was sent by Atlassian JIRA (v6.3.4#6332)
