[jira] [Commented] (HIVE-4881) hive local mode: java.io.FileNotFoundException: emptyFile
[
https://issues.apache.org/jira/browse/HIVE-4881?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15184573#comment-15184573
]
Athira Krishnankutty commented on HIVE-4881:
Local mode redirect to local dir path of the local file system
> hive local mode: java.io.FileNotFoundException: emptyFile
> -
>
> Key: HIVE-4881
> URL: https://issues.apache.org/jira/browse/HIVE-4881
> Project: Hive
> Issue Type: Bug
> Environment: hive 0.9.0+158-1.cdh4.1.3.p0.23~squeeze-cdh4.1.3
>Reporter: Bartosz Cisek
>Assignee: Athira Krishnankutty
>Priority: Critical
>
> Our hive jobs fail due to strange error pasted below. Strace showed that
> process created this file, accessed it a few times and then it throwed
> exception that it couldn't find file it just accessed. In next step it
> unliked it. Yay.
> Very similar problem was reported [in already closed
> task|https://issues.apache.org/jira/browse/HIVE-1633?focusedCommentId=13598983=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13598983]
> or left unresolved on [mailing
> lists|http://mail-archives.apache.org/mod_mbox/hive-user/201307.mbox/%3c94f02eb368b740ebbcd94df4d5d1d...@amxpr03mb054.eurprd03.prod.outlook.com%3E].
> I'll be happy to provide required additional details.
> {code:title=Stack trace}
> 2013-07-18 12:49:46,109 ERROR security.UserGroupInformation
> (UserGroupInformation.java:doAs(1335)) - PriviledgedActionException
> as:username (auth:SIMPLE) cause:java.io.FileNotFoundException: File does not
> exist:
> /tmp/username/hive_2013-07-18_12-49-45_218_605775464480014480/-mr-1/1/emptyFile
> 2013-07-18 12:49:46,113 ERROR exec.ExecDriver
> (SessionState.java:printError(403)) - Job Submission failed with exception
> 'java.io.FileNotFoundException(File does not exist:
> /tmp/username/hive_2013-07-18_12-49-45_218_605775464480014480/-mr-1/1/emptyFile)'
> java.io.FileNotFoundException: File does not exist:
> /tmp/username/hive_2013-07-18_12-49-45_218_605775464480014480/-mr-1/1/emptyFile
> at
> org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:787)
> at
> org.apache.hadoop.mapred.lib.CombineFileInputFormat$OneFileInfo.(CombineFileInputFormat.java:462)
> at
> org.apache.hadoop.mapred.lib.CombineFileInputFormat.getMoreSplits(CombineFileInputFormat.java:256)
> at
> org.apache.hadoop.mapred.lib.CombineFileInputFormat.getSplits(CombineFileInputFormat.java:212)
> at
> org.apache.hadoop.hive.shims.HadoopShimsSecure$CombineFileInputFormatShim.getSplits(HadoopShimsSecure.java:392)
> at
> org.apache.hadoop.hive.shims.HadoopShimsSecure$CombineFileInputFormatShim.getSplits(HadoopShimsSecure.java:358)
> at
> org.apache.hadoop.hive.ql.io.CombineHiveInputFormat.getSplits(CombineHiveInputFormat.java:387)
> at
> org.apache.hadoop.mapred.JobClient.writeOldSplits(JobClient.java:1040)
> at org.apache.hadoop.mapred.JobClient.writeSplits(JobClient.java:1032)
> at org.apache.hadoop.mapred.JobClient.access$600(JobClient.java:172)
> at org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:942)
> at org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:895)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:396)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1332)
> at
> org.apache.hadoop.mapred.JobClient.submitJobInternal(JobClient.java:895)
> at org.apache.hadoop.mapred.JobClient.submitJob(JobClient.java:869)
> at
> org.apache.hadoop.hive.ql.exec.ExecDriver.execute(ExecDriver.java:435)
> at org.apache.hadoop.hive.ql.exec.ExecDriver.main(ExecDriver.java:677)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.util.RunJar.main(RunJar.java:208)
> {code}
> {code:title=strace with grep emptyFile}
> 7385 14:48:02.808096
> stat("/tmp/username/hive_2013-07-18_14-48-00_700_8005967322498387476/-mr-1/1/emptyFile",
> {st_mode=S_IFREG|0755, st_size=0, ...}) = 0
> 7385 14:48:02.808201
> stat("/tmp/username/hive_2013-07-18_14-48-00_700_8005967322498387476/-mr-1/1/emptyFile",
> {st_mode=S_IFREG|0755, st_size=0, ...}) = 0
> 7385 14:48:02.808277
> stat("/tmp/username/hive_2013-07-18_14-48-00_700_8005967322498387476/-mr-1/1/emptyFile",
> {st_mode=S_IFREG|0755, st_size=0, ...}) = 0
> 7385 14:48:02.808348
>
[jira] [Resolved] (HIVE-4881) hive local mode: java.io.FileNotFoundException: emptyFile
[
https://issues.apache.org/jira/browse/HIVE-4881?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Athira Krishnankutty resolved HIVE-4881.
Resolution: Fixed
> hive local mode: java.io.FileNotFoundException: emptyFile
> -
>
> Key: HIVE-4881
> URL: https://issues.apache.org/jira/browse/HIVE-4881
> Project: Hive
> Issue Type: Bug
> Environment: hive 0.9.0+158-1.cdh4.1.3.p0.23~squeeze-cdh4.1.3
>Reporter: Bartosz Cisek
>Assignee: Athira Krishnankutty
>Priority: Critical
>
> Our hive jobs fail due to strange error pasted below. Strace showed that
> process created this file, accessed it a few times and then it throwed
> exception that it couldn't find file it just accessed. In next step it
> unliked it. Yay.
> Very similar problem was reported [in already closed
> task|https://issues.apache.org/jira/browse/HIVE-1633?focusedCommentId=13598983=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13598983]
> or left unresolved on [mailing
> lists|http://mail-archives.apache.org/mod_mbox/hive-user/201307.mbox/%3c94f02eb368b740ebbcd94df4d5d1d...@amxpr03mb054.eurprd03.prod.outlook.com%3E].
> I'll be happy to provide required additional details.
> {code:title=Stack trace}
> 2013-07-18 12:49:46,109 ERROR security.UserGroupInformation
> (UserGroupInformation.java:doAs(1335)) - PriviledgedActionException
> as:username (auth:SIMPLE) cause:java.io.FileNotFoundException: File does not
> exist:
> /tmp/username/hive_2013-07-18_12-49-45_218_605775464480014480/-mr-1/1/emptyFile
> 2013-07-18 12:49:46,113 ERROR exec.ExecDriver
> (SessionState.java:printError(403)) - Job Submission failed with exception
> 'java.io.FileNotFoundException(File does not exist:
> /tmp/username/hive_2013-07-18_12-49-45_218_605775464480014480/-mr-1/1/emptyFile)'
> java.io.FileNotFoundException: File does not exist:
> /tmp/username/hive_2013-07-18_12-49-45_218_605775464480014480/-mr-1/1/emptyFile
> at
> org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:787)
> at
> org.apache.hadoop.mapred.lib.CombineFileInputFormat$OneFileInfo.(CombineFileInputFormat.java:462)
> at
> org.apache.hadoop.mapred.lib.CombineFileInputFormat.getMoreSplits(CombineFileInputFormat.java:256)
> at
> org.apache.hadoop.mapred.lib.CombineFileInputFormat.getSplits(CombineFileInputFormat.java:212)
> at
> org.apache.hadoop.hive.shims.HadoopShimsSecure$CombineFileInputFormatShim.getSplits(HadoopShimsSecure.java:392)
> at
> org.apache.hadoop.hive.shims.HadoopShimsSecure$CombineFileInputFormatShim.getSplits(HadoopShimsSecure.java:358)
> at
> org.apache.hadoop.hive.ql.io.CombineHiveInputFormat.getSplits(CombineHiveInputFormat.java:387)
> at
> org.apache.hadoop.mapred.JobClient.writeOldSplits(JobClient.java:1040)
> at org.apache.hadoop.mapred.JobClient.writeSplits(JobClient.java:1032)
> at org.apache.hadoop.mapred.JobClient.access$600(JobClient.java:172)
> at org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:942)
> at org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:895)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:396)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1332)
> at
> org.apache.hadoop.mapred.JobClient.submitJobInternal(JobClient.java:895)
> at org.apache.hadoop.mapred.JobClient.submitJob(JobClient.java:869)
> at
> org.apache.hadoop.hive.ql.exec.ExecDriver.execute(ExecDriver.java:435)
> at org.apache.hadoop.hive.ql.exec.ExecDriver.main(ExecDriver.java:677)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.util.RunJar.main(RunJar.java:208)
> {code}
> {code:title=strace with grep emptyFile}
> 7385 14:48:02.808096
> stat("/tmp/username/hive_2013-07-18_14-48-00_700_8005967322498387476/-mr-1/1/emptyFile",
> {st_mode=S_IFREG|0755, st_size=0, ...}) = 0
> 7385 14:48:02.808201
> stat("/tmp/username/hive_2013-07-18_14-48-00_700_8005967322498387476/-mr-1/1/emptyFile",
> {st_mode=S_IFREG|0755, st_size=0, ...}) = 0
> 7385 14:48:02.808277
> stat("/tmp/username/hive_2013-07-18_14-48-00_700_8005967322498387476/-mr-1/1/emptyFile",
> {st_mode=S_IFREG|0755, st_size=0, ...}) = 0
> 7385 14:48:02.808348
>
[jira] [Assigned] (HIVE-4881) hive local mode: java.io.FileNotFoundException: emptyFile
[
https://issues.apache.org/jira/browse/HIVE-4881?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Athira Krishnankutty reassigned HIVE-4881:
--
Assignee: Athira Krishnankutty (was: Prasad Mujumdar)
> hive local mode: java.io.FileNotFoundException: emptyFile
> -
>
> Key: HIVE-4881
> URL: https://issues.apache.org/jira/browse/HIVE-4881
> Project: Hive
> Issue Type: Bug
> Environment: hive 0.9.0+158-1.cdh4.1.3.p0.23~squeeze-cdh4.1.3
>Reporter: Bartosz Cisek
>Assignee: Athira Krishnankutty
>Priority: Critical
>
> Our hive jobs fail due to strange error pasted below. Strace showed that
> process created this file, accessed it a few times and then it throwed
> exception that it couldn't find file it just accessed. In next step it
> unliked it. Yay.
> Very similar problem was reported [in already closed
> task|https://issues.apache.org/jira/browse/HIVE-1633?focusedCommentId=13598983=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13598983]
> or left unresolved on [mailing
> lists|http://mail-archives.apache.org/mod_mbox/hive-user/201307.mbox/%3c94f02eb368b740ebbcd94df4d5d1d...@amxpr03mb054.eurprd03.prod.outlook.com%3E].
> I'll be happy to provide required additional details.
> {code:title=Stack trace}
> 2013-07-18 12:49:46,109 ERROR security.UserGroupInformation
> (UserGroupInformation.java:doAs(1335)) - PriviledgedActionException
> as:username (auth:SIMPLE) cause:java.io.FileNotFoundException: File does not
> exist:
> /tmp/username/hive_2013-07-18_12-49-45_218_605775464480014480/-mr-1/1/emptyFile
> 2013-07-18 12:49:46,113 ERROR exec.ExecDriver
> (SessionState.java:printError(403)) - Job Submission failed with exception
> 'java.io.FileNotFoundException(File does not exist:
> /tmp/username/hive_2013-07-18_12-49-45_218_605775464480014480/-mr-1/1/emptyFile)'
> java.io.FileNotFoundException: File does not exist:
> /tmp/username/hive_2013-07-18_12-49-45_218_605775464480014480/-mr-1/1/emptyFile
> at
> org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:787)
> at
> org.apache.hadoop.mapred.lib.CombineFileInputFormat$OneFileInfo.(CombineFileInputFormat.java:462)
> at
> org.apache.hadoop.mapred.lib.CombineFileInputFormat.getMoreSplits(CombineFileInputFormat.java:256)
> at
> org.apache.hadoop.mapred.lib.CombineFileInputFormat.getSplits(CombineFileInputFormat.java:212)
> at
> org.apache.hadoop.hive.shims.HadoopShimsSecure$CombineFileInputFormatShim.getSplits(HadoopShimsSecure.java:392)
> at
> org.apache.hadoop.hive.shims.HadoopShimsSecure$CombineFileInputFormatShim.getSplits(HadoopShimsSecure.java:358)
> at
> org.apache.hadoop.hive.ql.io.CombineHiveInputFormat.getSplits(CombineHiveInputFormat.java:387)
> at
> org.apache.hadoop.mapred.JobClient.writeOldSplits(JobClient.java:1040)
> at org.apache.hadoop.mapred.JobClient.writeSplits(JobClient.java:1032)
> at org.apache.hadoop.mapred.JobClient.access$600(JobClient.java:172)
> at org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:942)
> at org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:895)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:396)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1332)
> at
> org.apache.hadoop.mapred.JobClient.submitJobInternal(JobClient.java:895)
> at org.apache.hadoop.mapred.JobClient.submitJob(JobClient.java:869)
> at
> org.apache.hadoop.hive.ql.exec.ExecDriver.execute(ExecDriver.java:435)
> at org.apache.hadoop.hive.ql.exec.ExecDriver.main(ExecDriver.java:677)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.util.RunJar.main(RunJar.java:208)
> {code}
> {code:title=strace with grep emptyFile}
> 7385 14:48:02.808096
> stat("/tmp/username/hive_2013-07-18_14-48-00_700_8005967322498387476/-mr-1/1/emptyFile",
> {st_mode=S_IFREG|0755, st_size=0, ...}) = 0
> 7385 14:48:02.808201
> stat("/tmp/username/hive_2013-07-18_14-48-00_700_8005967322498387476/-mr-1/1/emptyFile",
> {st_mode=S_IFREG|0755, st_size=0, ...}) = 0
> 7385 14:48:02.808277
> stat("/tmp/username/hive_2013-07-18_14-48-00_700_8005967322498387476/-mr-1/1/emptyFile",
> {st_mode=S_IFREG|0755, st_size=0, ...}) = 0
> 7385 14:48:02.808348
>
[jira] [Commented] (HIVE-13198) Authorization issues with cascading views
[
https://issues.apache.org/jira/browse/HIVE-13198?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15184433#comment-15184433
]
Hive QA commented on HIVE-13198:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12791604/HIVE-13198.01.patch
{color:green}SUCCESS:{color} +1 due to 12 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 18 failed/errored test(s), 9803 tests
executed
*Failed tests:*
{noformat}
TestSparkCliDriver-groupby3_map.q-sample2.q-auto_join14.q-and-12-more - did not
produce a TEST-*.xml file
TestSparkCliDriver-groupby_map_ppr_multi_distinct.q-table_access_keys_stats.q-groupby4_noskew.q-and-12-more
- did not produce a TEST-*.xml file
TestSparkCliDriver-join_rc.q-insert1.q-vectorized_rcfile_columnar.q-and-12-more
- did not produce a TEST-*.xml file
TestSparkCliDriver-ppd_join4.q-join9.q-ppd_join3.q-and-12-more - did not
produce a TEST-*.xml file
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_cbo_rp_views
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_cbo_views
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_create_view
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_explain_dependency
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_serde_user_properties
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_special_character_in_tabnames_1
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_subquery_views
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_view_cast
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_cbo_views
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_explainuser_1
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_authorization_uri_import
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_create_or_replace_view7
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_recursive_view
org.apache.hive.jdbc.TestSSL.testSSLVersion
{noformat}
Test results:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/7189/testReport
Console output:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/7189/console
Test logs:
http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-7189/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 18 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12791604 - PreCommit-HIVE-TRUNK-Build
> Authorization issues with cascading views
> -
>
> Key: HIVE-13198
> URL: https://issues.apache.org/jira/browse/HIVE-13198
> Project: Hive
> Issue Type: Sub-task
>Reporter: Pengcheng Xiong
>Assignee: Pengcheng Xiong
> Fix For: 2.1.0
>
> Attachments: HIVE-13198.01.patch
>
>
> Here is a use case. They have a base table t1, from which they create a view
> v1. They further create a view v2 from v1 by applying a filter. User has
> access to only view v2, not view v1 or table t1. When user tries to access
> v2, they are denied access.
> Steps to recreate:
> There is a base table t1 that exists in the default database with primary key
> id and some employee data (name, ssn etc)
> Create view v1 - “create view v1 as select * from default.t1;”
> Created v2 - “create view v2 as select * from v1 where id =1;”
> Permissions provided for user to select all columns from view v2. When user
> runs select * from v2, hive throws an error “user does not have permissions
> to select view v1".
> Apparently Hive is converting the query to underlying views.
> SELECT * FROM v2 LIMIT 100
> To
> select `v1`.`id`, `v1`.`name`, `v1`.`ssn`, `v1`.`join_date`, `v1`.`location`
> from `hr`.`v1` where `v1`.`id`=1
> Hive should only check for permissions for the view being run in the query,
> not any parent views. (This is consistent with ORACLE).
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Resolved] (HIVE-13225) LLAP: Stripe metadata cache holds unreachable keys
[
https://issues.apache.org/jira/browse/HIVE-13225?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Gopal V resolved HIVE-13225.
Resolution: Invalid
Cache issue seems to be entirely related to the linked patch - back to
inspecting that patch for cache hit-rate metrics.
> LLAP: Stripe metadata cache holds unreachable keys
> ---
>
> Key: HIVE-13225
> URL: https://issues.apache.org/jira/browse/HIVE-13225
> Project: Hive
> Issue Type: Bug
> Components: llap
>Affects Versions: 2.0.0, 2.1.0
>Reporter: Gopal V
>Assignee: Gopal V
>
> The Hash inspection reported that there were some unreachable keys in the
> metadata hashmap.
> {code}
> for (...) {
> if (hasFileId && metadataCache != null) {
> stripeKey.stripeIx = stripeIx;
> value = metadataCache.getStripeMetadata(stripeKey);
> }
> if (value == null || !value.hasAllIndexes(globalInc)) {
> if (hasFileId && metadataCache != null) {
> value = metadataCache.putStripeMetadata(value);
> ...
> }
> {code}
> Means that the hashCode of the key changes after the put if there are > 1
> stripes in the file.
> {code}
> public OrcStripeMetadata putStripeMetadata(OrcStripeMetadata metaData) {
>
> OrcStripeMetadata val = stripeMetadata.putIfAbsent(metaData.getKey(),
> metaData);
> {code}
> needs to make a copy of the Key, if it needs to preserve hash consistency.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-12988) Improve dynamic partition loading IV
[ https://issues.apache.org/jira/browse/HIVE-12988?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashutosh Chauhan updated HIVE-12988: Status: Patch Available (was: Open) > Improve dynamic partition loading IV > > > Key: HIVE-12988 > URL: https://issues.apache.org/jira/browse/HIVE-12988 > Project: Hive > Issue Type: Improvement > Components: Query Processor >Affects Versions: 2.0.0, 1.2.0 >Reporter: Ashutosh Chauhan >Assignee: Ashutosh Chauhan > Attachments: HIVE-12988.2.patch, HIVE-12988.2.patch, > HIVE-12988.3.patch, HIVE-12988.4.patch, HIVE-12988.5.patch, > HIVE-12988.6.patch, HIVE-12988.7.patch, HIVE-12988.patch > > > Parallelize copyFiles() -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-12988) Improve dynamic partition loading IV
[ https://issues.apache.org/jira/browse/HIVE-12988?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashutosh Chauhan updated HIVE-12988: Attachment: (was: HIVE-12988.7.patch) > Improve dynamic partition loading IV > > > Key: HIVE-12988 > URL: https://issues.apache.org/jira/browse/HIVE-12988 > Project: Hive > Issue Type: Improvement > Components: Query Processor >Affects Versions: 1.2.0, 2.0.0 >Reporter: Ashutosh Chauhan >Assignee: Ashutosh Chauhan > Attachments: HIVE-12988.2.patch, HIVE-12988.2.patch, > HIVE-12988.3.patch, HIVE-12988.4.patch, HIVE-12988.5.patch, > HIVE-12988.6.patch, HIVE-12988.7.patch, HIVE-12988.patch > > > Parallelize copyFiles() -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-12988) Improve dynamic partition loading IV
[ https://issues.apache.org/jira/browse/HIVE-12988?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashutosh Chauhan updated HIVE-12988: Status: Open (was: Patch Available) > Improve dynamic partition loading IV > > > Key: HIVE-12988 > URL: https://issues.apache.org/jira/browse/HIVE-12988 > Project: Hive > Issue Type: Improvement > Components: Query Processor >Affects Versions: 2.0.0, 1.2.0 >Reporter: Ashutosh Chauhan >Assignee: Ashutosh Chauhan > Attachments: HIVE-12988.2.patch, HIVE-12988.2.patch, > HIVE-12988.3.patch, HIVE-12988.4.patch, HIVE-12988.5.patch, > HIVE-12988.6.patch, HIVE-12988.7.patch, HIVE-12988.patch > > > Parallelize copyFiles() -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-12988) Improve dynamic partition loading IV
[ https://issues.apache.org/jira/browse/HIVE-12988?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashutosh Chauhan updated HIVE-12988: Attachment: HIVE-12988.7.patch > Improve dynamic partition loading IV > > > Key: HIVE-12988 > URL: https://issues.apache.org/jira/browse/HIVE-12988 > Project: Hive > Issue Type: Improvement > Components: Query Processor >Affects Versions: 1.2.0, 2.0.0 >Reporter: Ashutosh Chauhan >Assignee: Ashutosh Chauhan > Attachments: HIVE-12988.2.patch, HIVE-12988.2.patch, > HIVE-12988.3.patch, HIVE-12988.4.patch, HIVE-12988.5.patch, > HIVE-12988.6.patch, HIVE-12988.7.patch, HIVE-12988.patch > > > Parallelize copyFiles() -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-12988) Improve dynamic partition loading IV
[ https://issues.apache.org/jira/browse/HIVE-12988?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashutosh Chauhan updated HIVE-12988: Status: Open (was: Patch Available) > Improve dynamic partition loading IV > > > Key: HIVE-12988 > URL: https://issues.apache.org/jira/browse/HIVE-12988 > Project: Hive > Issue Type: Improvement > Components: Query Processor >Affects Versions: 2.0.0, 1.2.0 >Reporter: Ashutosh Chauhan >Assignee: Ashutosh Chauhan > Attachments: HIVE-12988.2.patch, HIVE-12988.2.patch, > HIVE-12988.3.patch, HIVE-12988.4.patch, HIVE-12988.5.patch, > HIVE-12988.6.patch, HIVE-12988.7.patch, HIVE-12988.patch > > > Parallelize copyFiles() -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-12988) Improve dynamic partition loading IV
[ https://issues.apache.org/jira/browse/HIVE-12988?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashutosh Chauhan updated HIVE-12988: Attachment: (was: HIVE-12988.7.patch) > Improve dynamic partition loading IV > > > Key: HIVE-12988 > URL: https://issues.apache.org/jira/browse/HIVE-12988 > Project: Hive > Issue Type: Improvement > Components: Query Processor >Affects Versions: 1.2.0, 2.0.0 >Reporter: Ashutosh Chauhan >Assignee: Ashutosh Chauhan > Attachments: HIVE-12988.2.patch, HIVE-12988.2.patch, > HIVE-12988.3.patch, HIVE-12988.4.patch, HIVE-12988.5.patch, > HIVE-12988.6.patch, HIVE-12988.7.patch, HIVE-12988.patch > > > Parallelize copyFiles() -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-12988) Improve dynamic partition loading IV
[ https://issues.apache.org/jira/browse/HIVE-12988?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashutosh Chauhan updated HIVE-12988: Status: Patch Available (was: Open) > Improve dynamic partition loading IV > > > Key: HIVE-12988 > URL: https://issues.apache.org/jira/browse/HIVE-12988 > Project: Hive > Issue Type: Improvement > Components: Query Processor >Affects Versions: 2.0.0, 1.2.0 >Reporter: Ashutosh Chauhan >Assignee: Ashutosh Chauhan > Attachments: HIVE-12988.2.patch, HIVE-12988.2.patch, > HIVE-12988.3.patch, HIVE-12988.4.patch, HIVE-12988.5.patch, > HIVE-12988.6.patch, HIVE-12988.7.patch, HIVE-12988.patch > > > Parallelize copyFiles() -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-12988) Improve dynamic partition loading IV
[ https://issues.apache.org/jira/browse/HIVE-12988?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashutosh Chauhan updated HIVE-12988: Attachment: HIVE-12988.7.patch > Improve dynamic partition loading IV > > > Key: HIVE-12988 > URL: https://issues.apache.org/jira/browse/HIVE-12988 > Project: Hive > Issue Type: Improvement > Components: Query Processor >Affects Versions: 1.2.0, 2.0.0 >Reporter: Ashutosh Chauhan >Assignee: Ashutosh Chauhan > Attachments: HIVE-12988.2.patch, HIVE-12988.2.patch, > HIVE-12988.3.patch, HIVE-12988.4.patch, HIVE-12988.5.patch, > HIVE-12988.6.patch, HIVE-12988.7.patch, HIVE-12988.patch > > > Parallelize copyFiles() -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Assigned] (HIVE-13225) LLAP: Stripe metadata cache holds unreachable keys
[
https://issues.apache.org/jira/browse/HIVE-13225?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Gopal V reassigned HIVE-13225:
--
Assignee: Gopal V (was: Sergey Shelukhin)
> LLAP: Stripe metadata cache holds unreachable keys
> ---
>
> Key: HIVE-13225
> URL: https://issues.apache.org/jira/browse/HIVE-13225
> Project: Hive
> Issue Type: Bug
> Components: llap
>Affects Versions: 2.0.0, 2.1.0
>Reporter: Gopal V
>Assignee: Gopal V
>
> The Hash inspection reported that there were some unreachable keys in the
> metadata hashmap.
> {code}
> for (...) {
> if (hasFileId && metadataCache != null) {
> stripeKey.stripeIx = stripeIx;
> value = metadataCache.getStripeMetadata(stripeKey);
> }
> if (value == null || !value.hasAllIndexes(globalInc)) {
> if (hasFileId && metadataCache != null) {
> value = metadataCache.putStripeMetadata(value);
> ...
> }
> {code}
> Means that the hashCode of the key changes after the put if there are > 1
> stripes in the file.
> {code}
> public OrcStripeMetadata putStripeMetadata(OrcStripeMetadata metaData) {
>
> OrcStripeMetadata val = stripeMetadata.putIfAbsent(metaData.getKey(),
> metaData);
> {code}
> needs to make a copy of the Key, if it needs to preserve hash consistency.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-13224) tt
[ https://issues.apache.org/jira/browse/HIVE-13224?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] yb updated HIVE-13224: -- Description: i test a table for about 60 data, when I use limit then not any problem if no limit then log message as comments version:2.0.0 tez 0.8.2 > tt > -- > > Key: HIVE-13224 > URL: https://issues.apache.org/jira/browse/HIVE-13224 > Project: Hive > Issue Type: Bug >Reporter: yb > > i test a table for about 60 data, when I use limit then not any > problem if no limit then log message as comments version:2.0.0 tez 0.8.2 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13225) LLAP: Stripe metadata cache holds unreachable keys
[
https://issues.apache.org/jira/browse/HIVE-13225?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Gopal V updated HIVE-13225:
---
Affects Version/s: 2.1.0
2.0.0
> LLAP: Stripe metadata cache holds unreachable keys
> ---
>
> Key: HIVE-13225
> URL: https://issues.apache.org/jira/browse/HIVE-13225
> Project: Hive
> Issue Type: Bug
> Components: llap
>Affects Versions: 2.0.0, 2.1.0
>Reporter: Gopal V
>Assignee: Sergey Shelukhin
>
> The Hash inspection reported that there were some unreachable keys in the
> metadata hashmap.
> {code}
> for (...) {
> if (hasFileId && metadataCache != null) {
> stripeKey.stripeIx = stripeIx;
> value = metadataCache.getStripeMetadata(stripeKey);
> }
> if (value == null || !value.hasAllIndexes(globalInc)) {
> if (hasFileId && metadataCache != null) {
> value = metadataCache.putStripeMetadata(value);
> ...
> }
> {code}
> Means that the hashCode of the key changes after the put if there are > 1
> stripes in the file.
> {code}
> public OrcStripeMetadata putStripeMetadata(OrcStripeMetadata metaData) {
>
> OrcStripeMetadata val = stripeMetadata.putIfAbsent(metaData.getKey(),
> metaData);
> {code}
> needs to make a copy of the Key, if it needs to preserve hash consistency.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-13225) LLAP: Stripe metadata cache holds unreachable keys
[
https://issues.apache.org/jira/browse/HIVE-13225?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Gopal V updated HIVE-13225:
---
Component/s: llap
> LLAP: Stripe metadata cache holds unreachable keys
> ---
>
> Key: HIVE-13225
> URL: https://issues.apache.org/jira/browse/HIVE-13225
> Project: Hive
> Issue Type: Bug
> Components: llap
>Affects Versions: 2.0.0, 2.1.0
>Reporter: Gopal V
>Assignee: Sergey Shelukhin
>
> The Hash inspection reported that there were some unreachable keys in the
> metadata hashmap.
> {code}
> for (...) {
> if (hasFileId && metadataCache != null) {
> stripeKey.stripeIx = stripeIx;
> value = metadataCache.getStripeMetadata(stripeKey);
> }
> if (value == null || !value.hasAllIndexes(globalInc)) {
> if (hasFileId && metadataCache != null) {
> value = metadataCache.putStripeMetadata(value);
> ...
> }
> {code}
> Means that the hashCode of the key changes after the put if there are > 1
> stripes in the file.
> {code}
> public OrcStripeMetadata putStripeMetadata(OrcStripeMetadata metaData) {
>
> OrcStripeMetadata val = stripeMetadata.putIfAbsent(metaData.getKey(),
> metaData);
> {code}
> needs to make a copy of the Key, if it needs to preserve hash consistency.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-13224) tt
[ https://issues.apache.org/jira/browse/HIVE-13224?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15184321#comment-15184321 ] yb commented on HIVE-13224: --- java.sql.SQLException: Error while processing statement: FAILED: Execution Error, return code 2 from org.apache.hadoop.hive.ql.exec.tez.TezTask. Vertex failed, vertexName=Map 1, vertexId=vertex_145396034_0235_1_00, diagnostics=[Task failed, taskId=task_145396034_0235_1_00_20, diagnostics=[TaskAttempt 0 failed, info=[Error: Failure while running task: attempt_145396034_0235_1_00_20_0:org.apache.hadoop.hive.ql.metadata.HiveException: java.lang.RuntimeException: Failed to load plan: hdfs://cluster1/tmp/hive/hive/9fccf1c1-dfc5-4952-9a79-acda54f8882e/hive_2016-03-08_11-13-34_799_1789527048117920518-12/hive/_tez_scratch_dir/eced05a1-ca0c-41ac-89ff-e767fd6fde2c/map.xml: org.apache.hive.com.esotericsoftware.kryo.KryoException: Encountered unregistered class ID: 21 Serialization trace: outputFileFormatClass (org.apache.hadoop.hive.ql.plan.PartitionDesc) aliasToPartnInfo (org.apache.hadoop.hive.ql.plan.MapWork) at org.apache.hadoop.hive.ql.exec.tez.ObjectCache.retrieve(ObjectCache.java:82) at org.apache.hadoop.hive.ql.exec.tez.MapRecordProcessor.(MapRecordProcessor.java:105) at org.apache.hadoop.hive.ql.exec.tez.TezProcessor.run(TezProcessor.java:132) at org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:351) at org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:71) at org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:59) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:415) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1628) at org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:59) at org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:36) at org.apache.tez.common.CallableWithNdc.call(CallableWithNdc.java:36) at java.util.concurrent.FutureTask.run(FutureTask.java:262) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745) Caused by: java.lang.RuntimeException: Failed to load plan: hdfs://cluster1/tmp/hive/hive/9fccf1c1-dfc5-4952-9a79-acda54f8882e/hive_2016-03-08_11-13-34_799_1789527048117920518-12/hive/_tez_scratch_dir/eced05a1-ca0c-41ac-89ff-e767fd6fde2c/map.xml: org.apache.hive.com.esotericsoftware.kryo.KryoException: Encountered unregistered class ID: 21 Serialization trace: outputFileFormatClass (org.apache.hadoop.hive.ql.plan.PartitionDesc) aliasToPartnInfo (org.apache.hadoop.hive.ql.plan.MapWork) at org.apache.hadoop.hive.ql.exec.Utilities.getBaseWork(Utilities.java:462) at org.apache.hadoop.hive.ql.exec.Utilities.getMapWork(Utilities.java:301) at org.apache.hadoop.hive.ql.exec.tez.MapRecordProcessor$1.call(MapRecordProcessor.java:108) at org.apache.hadoop.hive.ql.exec.tez.ObjectCache.retrieve(ObjectCache.java:75) ... 15 more Caused by: org.apache.hive.com.esotericsoftware.kryo.KryoException: Encountered unregistered class ID: 21 Serialization trace: outputFileFormatClass (org.apache.hadoop.hive.ql.plan.PartitionDesc) aliasToPartnInfo (org.apache.hadoop.hive.ql.plan.MapWork) at org.apache.hive.com.esotericsoftware.kryo.util.DefaultClassResolver.readClass(DefaultClassResolver.java:119) at org.apache.hive.com.esotericsoftware.kryo.Kryo.readClass(Kryo.java:656) at org.apache.hive.com.esotericsoftware.kryo.serializers.DefaultSerializers$ClassSerializer.read(DefaultSerializers.java:238) at org.apache.hive.com.esotericsoftware.kryo.serializers.DefaultSerializers$ClassSerializer.read(DefaultSerializers.java:226) at org.apache.hive.com.esotericsoftware.kryo.Kryo.readObjectOrNull(Kryo.java:745) at org.apache.hive.com.esotericsoftware.kryo.serializers.ObjectField.read(ObjectField.java:113) at org.apache.hive.com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:507) at org.apache.hive.com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:776) at org.apache.hive.com.esotericsoftware.kryo.serializers.MapSerializer.read(MapSerializer.java:139) at org.apache.hive.com.esotericsoftware.kryo.serializers.MapSerializer.read(MapSerializer.java:17) at org.apache.hive.com.esotericsoftware.kryo.Kryo.readObject(Kryo.java:694) at org.apache.hive.com.esotericsoftware.kryo.serializers.ObjectField.read(ObjectField.java:106) at
[jira] [Updated] (HIVE-12988) Improve dynamic partition loading IV
[ https://issues.apache.org/jira/browse/HIVE-12988?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashutosh Chauhan updated HIVE-12988: Status: Patch Available (was: Open) > Improve dynamic partition loading IV > > > Key: HIVE-12988 > URL: https://issues.apache.org/jira/browse/HIVE-12988 > Project: Hive > Issue Type: Improvement > Components: Query Processor >Affects Versions: 2.0.0, 1.2.0 >Reporter: Ashutosh Chauhan >Assignee: Ashutosh Chauhan > Attachments: HIVE-12988.2.patch, HIVE-12988.2.patch, > HIVE-12988.3.patch, HIVE-12988.4.patch, HIVE-12988.5.patch, > HIVE-12988.6.patch, HIVE-12988.7.patch, HIVE-12988.patch > > > Parallelize copyFiles() -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-12988) Improve dynamic partition loading IV
[ https://issues.apache.org/jira/browse/HIVE-12988?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashutosh Chauhan updated HIVE-12988: Attachment: HIVE-12988.7.patch > Improve dynamic partition loading IV > > > Key: HIVE-12988 > URL: https://issues.apache.org/jira/browse/HIVE-12988 > Project: Hive > Issue Type: Improvement > Components: Query Processor >Affects Versions: 1.2.0, 2.0.0 >Reporter: Ashutosh Chauhan >Assignee: Ashutosh Chauhan > Attachments: HIVE-12988.2.patch, HIVE-12988.2.patch, > HIVE-12988.3.patch, HIVE-12988.4.patch, HIVE-12988.5.patch, > HIVE-12988.6.patch, HIVE-12988.7.patch, HIVE-12988.patch > > > Parallelize copyFiles() -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (HIVE-6247) select count(distinct) should be MRR in Tez
[
https://issues.apache.org/jira/browse/HIVE-6247?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Gopal V resolved HIVE-6247.
---
Resolution: Not A Problem
> select count(distinct) should be MRR in Tez
> ---
>
> Key: HIVE-6247
> URL: https://issues.apache.org/jira/browse/HIVE-6247
> Project: Hive
> Issue Type: Bug
> Components: Tez
>Affects Versions: 0.13.0
>Reporter: Gopal V
>Assignee: Gunther Hagleitner
>
> The MR query plan for "select count(distinct) " fires off multiple reducers,
> with a local work task to perform final aggregation.
> The Tez version fires off exactly 1 reducer for the entire data-set which
> chokes and dies/slows down massively.
> To reproduce on a TPC-DS database (meaningless query)
> {code}
> select count(distinct ss_net_profit) from store_sales ss join store s on
> ss.ss_store_sk = s.s_store_sk;
> {code}
> This spins up Map 1, Map 2 (for the dim table + fact table) & Reducer 1 which
> is always "0/1".
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-12995) LLAP: Synthetic file ids need collision checks
[ https://issues.apache.org/jira/browse/HIVE-12995?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15184257#comment-15184257 ] Gopal V commented on HIVE-12995: LGTM - +1, pending minor comments left on RB (s/fileId/fileKey/ pretty much) > LLAP: Synthetic file ids need collision checks > -- > > Key: HIVE-12995 > URL: https://issues.apache.org/jira/browse/HIVE-12995 > Project: Hive > Issue Type: Bug > Components: llap >Affects Versions: 2.1.0 >Reporter: Gopal V >Assignee: Sergey Shelukhin > Attachments: HIVE-12995.01.patch, HIVE-12995.patch > > > LLAP synthetic file ids do not have any way of checking whether a collision > occurs other than a data-error. > Synthetic file-ids have only been used with unit tests so far - but they will > be needed to add cache mechanisms to non-HDFS filesystems. > In case of Synthetic file-ids, it is recommended that we track the full-tuple > (path, mtime, len) in the cache so that a cache-hit for the synthetic file-id > can be compared against the parameters & only accepted if those match. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13178) Enhance ORC Schema Evolution to handle more standard data type conversions
[
https://issues.apache.org/jira/browse/HIVE-13178?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15184243#comment-15184243
]
Hive QA commented on HIVE-13178:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12791841/HIVE-13178.03.patch
{color:green}SUCCESS:{color} +1 due to 35 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 27 failed/errored test(s), 9804 tests
executed
*Failed tests:*
{noformat}
TestSparkCliDriver-groupby3_map.q-sample2.q-auto_join14.q-and-12-more - did not
produce a TEST-*.xml file
TestSparkCliDriver-groupby_map_ppr_multi_distinct.q-table_access_keys_stats.q-groupby4_noskew.q-and-12-more
- did not produce a TEST-*.xml file
TestSparkCliDriver-join_rc.q-insert1.q-vectorized_rcfile_columnar.q-and-12-more
- did not produce a TEST-*.xml file
TestSparkCliDriver-ppd_join4.q-join9.q-ppd_join3.q-and-12-more - did not
produce a TEST-*.xml file
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_schema_evol_orc_acid_mapwork_part
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_schema_evol_orc_acidvec_mapwork_part
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_schema_evol_orc_acidvec_mapwork_table
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.org.apache.hadoop.hive.cli.TestMiniTezCliDriver
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_cbo_semijoin
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_cbo_subq_exists
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_cte_mat_3
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_cte_mat_5
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_delete_where_partitioned
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_schema_evol_orc_acid_mapwork_part
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_schema_evol_orc_acidvec_mapwork_part
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_schema_evol_orc_acidvec_mapwork_table
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_schema_evol_orc_vec_mapwork_table_objonly
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_tez_dml
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_tez_union
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_update_where_partitioned
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_vector_non_string_partition
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_vector_outer_join6
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_vector_partitioned_date_time
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_vector_reduce_groupby_decimal
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_vectorization_8
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_authorization_uri_import
org.apache.hive.jdbc.TestSSL.testSSLVersion
{noformat}
Test results:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/7188/testReport
Console output:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/7188/console
Test logs:
http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-7188/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 27 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12791841 - PreCommit-HIVE-TRUNK-Build
> Enhance ORC Schema Evolution to handle more standard data type conversions
> --
>
> Key: HIVE-13178
> URL: https://issues.apache.org/jira/browse/HIVE-13178
> Project: Hive
> Issue Type: Bug
> Components: Hive, ORC
>Reporter: Matt McCline
>Assignee: Matt McCline
>Priority: Critical
> Attachments: HIVE-13178.01.patch, HIVE-13178.02.patch,
> HIVE-13178.03.patch
>
>
> Currently, SHORT -> INT -> BIGINT is supported.
> Handle ORC data type conversions permitted by Implicit conversion allowed by
> TypeIntoUtils.implicitConvertible method.
>* STRING_GROUP -> DOUBLE
>* STRING_GROUP -> DECIMAL
>* DATE_GROUP -> STRING
>* NUMERIC_GROUP -> STRING
>* STRING_GROUP -> STRING_GROUP
>*
>* // Upward from "lower" type to "higher" numeric type:
>* BYTE -> SHORT -> INT -> BIGINT -> FLOAT -> DOUBLE -> DECIMAL
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-13221) expose metastore APIs from HS2
[ https://issues.apache.org/jira/browse/HIVE-13221?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HIVE-13221: Description: I was always wondering why we don't do that, for the people who run HS2 and also need metastore due to it being used externally; they don't need to run a standalone metastore. (was: I was always wondering why we don't do that, for people who run HS2 and also need metastore due to external use; they don't need to run a standalone metastore.) > expose metastore APIs from HS2 > -- > > Key: HIVE-13221 > URL: https://issues.apache.org/jira/browse/HIVE-13221 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > Attachments: HIVE-13221.patch > > > I was always wondering why we don't do that, for the people who run HS2 and > also need metastore due to it being used externally; they don't need to run a > standalone metastore. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13221) expose metastore APIs from HS2
[ https://issues.apache.org/jira/browse/HIVE-13221?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HIVE-13221: Status: Patch Available (was: In Progress) > expose metastore APIs from HS2 > -- > > Key: HIVE-13221 > URL: https://issues.apache.org/jira/browse/HIVE-13221 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > Attachments: HIVE-13221.patch > > > I was always wondering why we don't do that, for people who run HS2 and also > need metastore due to external use; they don't need to run a standalone > metastore. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-12049) Provide an option to write serialized thrift objects in final tasks
[
https://issues.apache.org/jira/browse/HIVE-12049?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Rohit Dholakia updated HIVE-12049:
--
Attachment: HIVE-12049.12.patch
> Provide an option to write serialized thrift objects in final tasks
> ---
>
> Key: HIVE-12049
> URL: https://issues.apache.org/jira/browse/HIVE-12049
> Project: Hive
> Issue Type: Sub-task
> Components: HiveServer2
>Reporter: Rohit Dholakia
>Assignee: Rohit Dholakia
> Attachments: HIVE-12049.1.patch, HIVE-12049.11.patch,
> HIVE-12049.12.patch, HIVE-12049.2.patch, HIVE-12049.3.patch,
> HIVE-12049.4.patch, HIVE-12049.5.patch, HIVE-12049.6.patch,
> HIVE-12049.7.patch, HIVE-12049.9.patch
>
>
> For each fetch request to HiveServer2, we pay the penalty of deserializing
> the row objects and translating them into a different representation suitable
> for the RPC transfer. In a moderate to high concurrency scenarios, this can
> result in significant CPU and memory wastage. By having each task write the
> appropriate thrift objects to the output files, HiveServer2 can simply stream
> a batch of rows on the wire without incurring any of the additional cost of
> deserialization and translation.
> This can be implemented by writing a new SerDe, which the FileSinkOperator
> can use to write thrift formatted row batches to the output file. Using the
> pluggable property of the {{hive.query.result.fileformat}}, we can set it to
> use SequenceFile and write a batch of thrift formatted rows as a value blob.
> The FetchTask can now simply read the blob and send it over the wire. On the
> client side, the *DBC driver can read the blob and since it is already
> formatted in the way it expects, it can continue building the ResultSet the
> way it does in the current implementation.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-13221) expose metastore APIs from HS2
[ https://issues.apache.org/jira/browse/HIVE-13221?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HIVE-13221: Attachment: HIVE-13221.patch Simple patch. Calls the horrible static init method; also cleans up some init. > expose metastore APIs from HS2 > -- > > Key: HIVE-13221 > URL: https://issues.apache.org/jira/browse/HIVE-13221 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > Attachments: HIVE-13221.patch > > > I was always wondering why we don't do that, for people who run HS2 and also > need metastore due to external use; they don't need to run a standalone > metastore. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-12049) Provide an option to write serialized thrift objects in final tasks
[
https://issues.apache.org/jira/browse/HIVE-12049?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Rohit Dholakia updated HIVE-12049:
--
Status: Open (was: Patch Available)
> Provide an option to write serialized thrift objects in final tasks
> ---
>
> Key: HIVE-12049
> URL: https://issues.apache.org/jira/browse/HIVE-12049
> Project: Hive
> Issue Type: Sub-task
> Components: HiveServer2
>Reporter: Rohit Dholakia
>Assignee: Rohit Dholakia
> Attachments: HIVE-12049.1.patch, HIVE-12049.11.patch,
> HIVE-12049.2.patch, HIVE-12049.3.patch, HIVE-12049.4.patch,
> HIVE-12049.5.patch, HIVE-12049.6.patch, HIVE-12049.7.patch, HIVE-12049.9.patch
>
>
> For each fetch request to HiveServer2, we pay the penalty of deserializing
> the row objects and translating them into a different representation suitable
> for the RPC transfer. In a moderate to high concurrency scenarios, this can
> result in significant CPU and memory wastage. By having each task write the
> appropriate thrift objects to the output files, HiveServer2 can simply stream
> a batch of rows on the wire without incurring any of the additional cost of
> deserialization and translation.
> This can be implemented by writing a new SerDe, which the FileSinkOperator
> can use to write thrift formatted row batches to the output file. Using the
> pluggable property of the {{hive.query.result.fileformat}}, we can set it to
> use SequenceFile and write a batch of thrift formatted rows as a value blob.
> The FetchTask can now simply read the blob and send it over the wire. On the
> client side, the *DBC driver can read the blob and since it is already
> formatted in the way it expects, it can continue building the ResultSet the
> way it does in the current implementation.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-13221) expose metastore APIs from HS2
[ https://issues.apache.org/jira/browse/HIVE-13221?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HIVE-13221: Description: I was always wondering why we don't do that, for people who run HS2 and also need metastore due to external use; they don't need to run a standalone metastore. (was: I was always wondering why we don't do that, for people who run HS2 and don't need to run a separate metastore.) > expose metastore APIs from HS2 > -- > > Key: HIVE-13221 > URL: https://issues.apache.org/jira/browse/HIVE-13221 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > > I was always wondering why we don't do that, for people who run HS2 and also > need metastore due to external use; they don't need to run a standalone > metastore. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-12049) Provide an option to write serialized thrift objects in final tasks
[
https://issues.apache.org/jira/browse/HIVE-12049?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Rohit Dholakia updated HIVE-12049:
--
Status: Patch Available (was: Open)
> Provide an option to write serialized thrift objects in final tasks
> ---
>
> Key: HIVE-12049
> URL: https://issues.apache.org/jira/browse/HIVE-12049
> Project: Hive
> Issue Type: Sub-task
> Components: HiveServer2
>Reporter: Rohit Dholakia
>Assignee: Rohit Dholakia
> Attachments: HIVE-12049.1.patch, HIVE-12049.11.patch,
> HIVE-12049.2.patch, HIVE-12049.3.patch, HIVE-12049.4.patch,
> HIVE-12049.5.patch, HIVE-12049.6.patch, HIVE-12049.7.patch, HIVE-12049.9.patch
>
>
> For each fetch request to HiveServer2, we pay the penalty of deserializing
> the row objects and translating them into a different representation suitable
> for the RPC transfer. In a moderate to high concurrency scenarios, this can
> result in significant CPU and memory wastage. By having each task write the
> appropriate thrift objects to the output files, HiveServer2 can simply stream
> a batch of rows on the wire without incurring any of the additional cost of
> deserialization and translation.
> This can be implemented by writing a new SerDe, which the FileSinkOperator
> can use to write thrift formatted row batches to the output file. Using the
> pluggable property of the {{hive.query.result.fileformat}}, we can set it to
> use SequenceFile and write a batch of thrift formatted rows as a value blob.
> The FetchTask can now simply read the blob and send it over the wire. On the
> client side, the *DBC driver can read the blob and since it is already
> formatted in the way it expects, it can continue building the ResultSet the
> way it does in the current implementation.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-13221) expose metastore APIs from HS2
[ https://issues.apache.org/jira/browse/HIVE-13221?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HIVE-13221: Description: (was: Need to be able to use them w/o standalone metastore. Luckily, HIVE-11675 encapsulates all metastore interactions into a 50-line class, so it should be trivially easy to use HS2 APIs instead. Or so I hope, that being HS2, after all.) > expose metastore APIs from HS2 > -- > > Key: HIVE-13221 > URL: https://issues.apache.org/jira/browse/HIVE-13221 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13221) expose metastore APIs from HS2
[ https://issues.apache.org/jira/browse/HIVE-13221?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HIVE-13221: Description: I was always wondering why we don't do that, for people who run HS2 and don't need to run a separate metastore. > expose metastore APIs from HS2 > -- > > Key: HIVE-13221 > URL: https://issues.apache.org/jira/browse/HIVE-13221 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > > I was always wondering why we don't do that, for people who run HS2 and don't > need to run a separate metastore. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-12925) make sure metastore footer cache doesn't get all functions
[ https://issues.apache.org/jira/browse/HIVE-12925?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HIVE-12925: Description: See HIVE-12918, also some other JIRA that added an option to not pre-load all functions. was: See HIVE-12918, also some other JIRA that added an option to not pre-load all functions. NO PRECOMMIT TESTS > make sure metastore footer cache doesn't get all functions > -- > > Key: HIVE-12925 > URL: https://issues.apache.org/jira/browse/HIVE-12925 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > Attachments: HIVE-12925.patch > > > See HIVE-12918, also some other JIRA that added an option to not pre-load all > functions. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-12925) make sure metastore footer cache doesn't get all functions
[ https://issues.apache.org/jira/browse/HIVE-12925?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HIVE-12925: Status: Open (was: Patch Available) > make sure metastore footer cache doesn't get all functions > -- > > Key: HIVE-12925 > URL: https://issues.apache.org/jira/browse/HIVE-12925 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > Attachments: HIVE-12925.patch > > > See HIVE-12918, also some other JIRA that added an option to not pre-load all > functions. > NO PRECOMMIT TESTS -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-12925) make sure metastore footer cache doesn't get all functions
[ https://issues.apache.org/jira/browse/HIVE-12925?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HIVE-12925: Summary: make sure metastore footer cache doesn't get all functions (was: make sure metastore footer cache usage never creates embedded metastore, and doesn't get all functions) > make sure metastore footer cache doesn't get all functions > -- > > Key: HIVE-12925 > URL: https://issues.apache.org/jira/browse/HIVE-12925 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > Attachments: HIVE-12925.patch > > > See HIVE-12918, also some other JIRA that added an option to not pre-load all > functions. > NO PRECOMMIT TESTS -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13211) normalize Hive.get overloads to go thru one path
[ https://issues.apache.org/jira/browse/HIVE-13211?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HIVE-13211: Attachment: HIVE-13211.01.patch Fixing an NPE > normalize Hive.get overloads to go thru one path > > > Key: HIVE-13211 > URL: https://issues.apache.org/jira/browse/HIVE-13211 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > Attachments: HIVE-13211.01.patch, HIVE-13211.patch > > > There are multiple subtly different paths in Hive.get(...) methods. Some > close the old db on refresh, some don't. Some check if the client is > compatible with config, some don't. Also there were some parameters (don't > register functions, disallow embedded metastore) that were added recently. > Need to make this stuff go thru one path. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Work started] (HIVE-13221) expose metadata cache APIs from HS2
[ https://issues.apache.org/jira/browse/HIVE-13221?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on HIVE-13221 started by Sergey Shelukhin. --- > expose metadata cache APIs from HS2 > --- > > Key: HIVE-13221 > URL: https://issues.apache.org/jira/browse/HIVE-13221 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > > Need to be able to use them w/o standalone metastore. Luckily, HIVE-11675 > encapsulates all metastore interactions into a 50-line class, so it should be > trivially easy to use HS2 APIs instead. Or so I hope, that being HS2, after > all. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13223) HoS may hang for queries that run on 0 splits
[ https://issues.apache.org/jira/browse/HIVE-13223?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashutosh Chauhan updated HIVE-13223: Status: Patch Available (was: Open) > HoS may hang for queries that run on 0 splits > --- > > Key: HIVE-13223 > URL: https://issues.apache.org/jira/browse/HIVE-13223 > Project: Hive > Issue Type: Bug > Components: Spark >Affects Versions: 2.1.0 >Reporter: Ashutosh Chauhan >Assignee: Ashutosh Chauhan > Attachments: HIVE-13223.patch > > > Can be seen on all timed out tests after HIVE-13040 went in -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13223) HoS may hang for queries that run on 0 splits
[ https://issues.apache.org/jira/browse/HIVE-13223?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashutosh Chauhan updated HIVE-13223: Attachment: HIVE-13223.patch > HoS may hang for queries that run on 0 splits > --- > > Key: HIVE-13223 > URL: https://issues.apache.org/jira/browse/HIVE-13223 > Project: Hive > Issue Type: Bug > Components: Spark >Affects Versions: 2.1.0 >Reporter: Ashutosh Chauhan >Assignee: Ashutosh Chauhan > Attachments: HIVE-13223.patch > > > Can be seen on all timed out tests after HIVE-13040 went in -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13216) ORC Reader will leave file open until GC when opening a malformed ORC file
[
https://issues.apache.org/jira/browse/HIVE-13216?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15184064#comment-15184064
]
Prasanth Jayachandran commented on HIVE-13216:
--
lgmt, +1.
> ORC Reader will leave file open until GC when opening a malformed ORC file
> --
>
> Key: HIVE-13216
> URL: https://issues.apache.org/jira/browse/HIVE-13216
> Project: Hive
> Issue Type: Bug
> Components: File Formats, ORC
>Affects Versions: 1.2.0, 1.2.1, 2.0.0
>Reporter: Yuxing Yao
>Assignee: Sergey Shelukhin
>Priority: Minor
> Attachments: HIVE-13216.patch
>
>
> In ORC extractMetaInfoFromFooter method of ReaderImpl.java:
> A new input stream is open without try-catch-finally to enforce closing.
> Once the footer parse has some exception, the stream close will miss.
> Until GC happen to close the stream.
> private static FileMetaInfo extractMetaInfoFromFooter(FileSystem fs,
> Path path,
> long maxFileLength
> ) throws IOException {
> FSDataInputStream file = fs.open(path);
> ...
> file.close();
> return new FileMetaInfo(
> ps.getCompression().toString(),
> (int) ps.getCompressionBlockSize(),
> (int) ps.getMetadataLength(),
> buffer,
> ps.getVersionList(),
> writerVersion
> );
> }
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-12995) LLAP: Synthetic file ids need collision checks
[ https://issues.apache.org/jira/browse/HIVE-12995?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15184024#comment-15184024 ] Sergey Shelukhin commented on HIVE-12995: - Test failures are unrelated. > LLAP: Synthetic file ids need collision checks > -- > > Key: HIVE-12995 > URL: https://issues.apache.org/jira/browse/HIVE-12995 > Project: Hive > Issue Type: Bug > Components: llap >Affects Versions: 2.1.0 >Reporter: Gopal V >Assignee: Sergey Shelukhin > Attachments: HIVE-12995.01.patch, HIVE-12995.patch > > > LLAP synthetic file ids do not have any way of checking whether a collision > occurs other than a data-error. > Synthetic file-ids have only been used with unit tests so far - but they will > be needed to add cache mechanisms to non-HDFS filesystems. > In case of Synthetic file-ids, it is recommended that we track the full-tuple > (path, mtime, len) in the cache so that a cache-hit for the synthetic file-id > can be compared against the parameters & only accepted if those match. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13204) Vectorization: Add ChainedCheckerFactory for LIKE
[
https://issues.apache.org/jira/browse/HIVE-13204?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15184019#comment-15184019
]
Gopal V commented on HIVE-13204:
The 2 failed Hbase tests had unrelated mini-cluster start issues
{code}
Caused by: java.io.FileNotFoundException: File
file:/tmp/hadoop-yarn/staging/history/done does not exist
at
org.apache.hadoop.fs.RawLocalFileSystem.listStatus(RawLocalFileSystem.java:376)
at
org.apache.hadoop.fs.DelegateToFileSystem.listStatus(DelegateToFileSystem.java:149)
at org.apache.hadoop.fs.ChecksumFs.listStatus(ChecksumFs.java:516)
at
org.apache.hadoop.fs.AbstractFileSystem$1.(AbstractFileSystem.java:870)
at
org.apache.hadoop.fs.AbstractFileSystem.listStatusIterator(AbstractFileSystem.java:868)
at org.apache.hadoop.fs.FileContext$21.next(FileContext.java:1445)
at org.apache.hadoop.fs.FileContext$21.next(FileContext.java:1440)
at org.apache.hadoop.fs.FSLinkResolver.resolve(FSLinkResolver.java:90)
at org.apache.hadoop.fs.FileContext.listStatus(FileContext.java:1440)
at
org.apache.hadoop.mapreduce.v2.jobhistory.JobHistoryUtils.localGlobber(JobHistoryUtils.java:456)
at
org.apache.hadoop.mapreduce.v2.jobhistory.JobHistoryUtils.localGlobber(JobHistoryUtils.java:444)
at
org.apache.hadoop.mapreduce.v2.jobhistory.JobHistoryUtils.localGlobber(JobHistoryUtils.java:439)
at
org.apache.hadoop.mapreduce.v2.hs.HistoryFileManager.findTimestampedDirectories(HistoryFileManager.java:762)
at
org.apache.hadoop.mapreduce.v2.hs.HistoryFileManager.initExisting(HistoryFileManager.java:665)
at
org.apache.hadoop.mapreduce.v2.hs.JobHistory.serviceInit(JobHistory.java:96)
at
org.apache.hadoop.service.AbstractService.init(AbstractService.java:163)
at
org.apache.hadoop.service.CompositeService.serviceInit(CompositeService.java:107)
at
org.apache.hadoop.mapreduce.v2.hs.JobHistoryServer.serviceInit(JobHistoryServer.java:142)
at
org.apache.hadoop.service.AbstractService.init(AbstractService.java:163)
at
org.apache.hadoop.mapreduce.v2.MiniMRYarnCluster$JobHistoryServerWrapper.serviceStart(MiniMRYarnCluster.java:210)
at
org.apache.hadoop.service.AbstractService.start(AbstractService.java:193)
at
org.apache.hadoop.service.CompositeService.serviceStart(CompositeService.java:120)
at
org.apache.hadoop.service.AbstractService.start(AbstractService.java:193)
at
org.apache.hadoop.mapred.MiniMRClientClusterFactory.create(MiniMRClientClusterFactory.java:80)
at org.apache.hadoop.mapred.MiniMRCluster.(MiniMRCluster.java:187)
at org.apache.hadoop.mapred.MiniMRCluster.(MiniMRCluster.java:175)
at org.apache.hadoop.mapred.MiniMRCluster.(MiniMRCluster.java:167)
at
org.apache.hive.hcatalog.hbase.ManyMiniCluster.setupMRCluster(ManyMiniCluster.java:218)
at
org.apache.hive.hcatalog.hbase.ManyMiniCluster.start(ManyMiniCluster.java:104)
at
org.apache.hive.hcatalog.hbase.SkeletonHBaseTest$Context.start(SkeletonHBaseTest.java:186)
at
org.apache.hive.hcatalog.hbase.SkeletonHBaseTest.setup(SkeletonHBaseTest.java:96
{code}
> Vectorization: Add ChainedCheckerFactory for LIKE
> --
>
> Key: HIVE-13204
> URL: https://issues.apache.org/jira/browse/HIVE-13204
> Project: Hive
> Issue Type: Improvement
> Components: Vectorization
>Affects Versions: 2.1.0
>Reporter: Gopal V
>Assignee: Gopal V
> Attachments: HIVE-13204.1.patch, HIVE-13204.2.patch,
> HIVE-13204.3.patch, HIVE-13204.4.patch, regex-cpu-like.png
>
>
> Currently, Vectorization runs through a UTF-8 decode to produce a String &
> then check for mildly complex patterns like "http://%.exe; using a Regex.
> Since this pattern doesn't need any backtracking patterns, using a full
> fledged Regex is too expensive.
> !regex-cpu-like.png!
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-13204) Vectorization: Add ChainedCheckerFactory for LIKE
[
https://issues.apache.org/jira/browse/HIVE-13204?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15183987#comment-15183987
]
Hive QA commented on HIVE-13204:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12791583/HIVE-13204.3.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 7 failed/errored test(s), 9785 tests executed
*Failed tests:*
{noformat}
TestSparkCliDriver-groupby3_map.q-sample2.q-auto_join14.q-and-12-more - did not
produce a TEST-*.xml file
TestSparkCliDriver-groupby_map_ppr_multi_distinct.q-table_access_keys_stats.q-groupby4_noskew.q-and-12-more
- did not produce a TEST-*.xml file
TestSparkCliDriver-join_rc.q-insert1.q-vectorized_rcfile_columnar.q-and-12-more
- did not produce a TEST-*.xml file
TestSparkCliDriver-ppd_join4.q-join9.q-ppd_join3.q-and-12-more - did not
produce a TEST-*.xml file
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_authorization_uri_import
org.apache.hive.hcatalog.hbase.TestPigHBaseStorageHandler.org.apache.hive.hcatalog.hbase.TestPigHBaseStorageHandler
org.apache.hive.jdbc.TestSSL.testSSLVersion
{noformat}
Test results:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/7187/testReport
Console output:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/7187/console
Test logs:
http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-7187/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 7 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12791583 - PreCommit-HIVE-TRUNK-Build
> Vectorization: Add ChainedCheckerFactory for LIKE
> --
>
> Key: HIVE-13204
> URL: https://issues.apache.org/jira/browse/HIVE-13204
> Project: Hive
> Issue Type: Improvement
> Components: Vectorization
>Affects Versions: 2.1.0
>Reporter: Gopal V
>Assignee: Gopal V
> Attachments: HIVE-13204.1.patch, HIVE-13204.2.patch,
> HIVE-13204.3.patch, HIVE-13204.4.patch, regex-cpu-like.png
>
>
> Currently, Vectorization runs through a UTF-8 decode to produce a String &
> then check for mildly complex patterns like "http://%.exe; using a Regex.
> Since this pattern doesn't need any backtracking patterns, using a full
> fledged Regex is too expensive.
> !regex-cpu-like.png!
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-13216) ORC Reader will leave file open until GC when opening a malformed ORC file
[
https://issues.apache.org/jira/browse/HIVE-13216?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sergey Shelukhin updated HIVE-13216:
Attachment: HIVE-13216.patch
This patch adds try-finally to the method. Most of the rest of the changes are
just whitespace. [~prasanth_j] can you take a look?
> ORC Reader will leave file open until GC when opening a malformed ORC file
> --
>
> Key: HIVE-13216
> URL: https://issues.apache.org/jira/browse/HIVE-13216
> Project: Hive
> Issue Type: Bug
> Components: File Formats, ORC
>Affects Versions: 1.2.0, 1.2.1, 2.0.0
>Reporter: Yuxing Yao
>Assignee: Sergey Shelukhin
>Priority: Minor
> Attachments: HIVE-13216.patch
>
>
> In ORC extractMetaInfoFromFooter method of ReaderImpl.java:
> A new input stream is open without try-catch-finally to enforce closing.
> Once the footer parse has some exception, the stream close will miss.
> Until GC happen to close the stream.
> private static FileMetaInfo extractMetaInfoFromFooter(FileSystem fs,
> Path path,
> long maxFileLength
> ) throws IOException {
> FSDataInputStream file = fs.open(path);
> ...
> file.close();
> return new FileMetaInfo(
> ps.getCompression().toString(),
> (int) ps.getCompressionBlockSize(),
> (int) ps.getMetadataLength(),
> buffer,
> ps.getVersionList(),
> writerVersion
> );
> }
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-13216) ORC Reader will leave file open until GC when opening a malformed ORC file
[
https://issues.apache.org/jira/browse/HIVE-13216?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sergey Shelukhin updated HIVE-13216:
Affects Version/s: 2.0.0
Status: Patch Available (was: Open)
> ORC Reader will leave file open until GC when opening a malformed ORC file
> --
>
> Key: HIVE-13216
> URL: https://issues.apache.org/jira/browse/HIVE-13216
> Project: Hive
> Issue Type: Bug
> Components: File Formats, ORC
>Affects Versions: 2.0.0, 1.2.1, 1.2.0
>Reporter: Yuxing Yao
>Assignee: Sergey Shelukhin
>Priority: Minor
> Attachments: HIVE-13216.patch
>
>
> In ORC extractMetaInfoFromFooter method of ReaderImpl.java:
> A new input stream is open without try-catch-finally to enforce closing.
> Once the footer parse has some exception, the stream close will miss.
> Until GC happen to close the stream.
> private static FileMetaInfo extractMetaInfoFromFooter(FileSystem fs,
> Path path,
> long maxFileLength
> ) throws IOException {
> FSDataInputStream file = fs.open(path);
> ...
> file.close();
> return new FileMetaInfo(
> ps.getCompression().toString(),
> (int) ps.getCompressionBlockSize(),
> (int) ps.getMetadataLength(),
> buffer,
> ps.getVersionList(),
> writerVersion
> );
> }
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-13204) Vectorization: Add ChainedCheckerFactory for LIKE
[ https://issues.apache.org/jira/browse/HIVE-13204?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gopal V updated HIVE-13204: --- Attachment: HIVE-13204.4.patch Fixed existing test-cases with @Test markers > Vectorization: Add ChainedCheckerFactory for LIKE > -- > > Key: HIVE-13204 > URL: https://issues.apache.org/jira/browse/HIVE-13204 > Project: Hive > Issue Type: Improvement > Components: Vectorization >Affects Versions: 2.1.0 >Reporter: Gopal V >Assignee: Gopal V > Attachments: HIVE-13204.1.patch, HIVE-13204.2.patch, > HIVE-13204.3.patch, HIVE-13204.4.patch, regex-cpu-like.png > > > Currently, Vectorization runs through a UTF-8 decode to produce a String & > then check for mildly complex patterns like "http://%.exe; using a Regex. > Since this pattern doesn't need any backtracking patterns, using a full > fledged Regex is too expensive. > !regex-cpu-like.png! -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13218) LLAP: better configs part 1
[ https://issues.apache.org/jira/browse/HIVE-13218?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15183952#comment-15183952 ] Gopal V commented on HIVE-13218: LGTM - +1. > LLAP: better configs part 1 > --- > > Key: HIVE-13218 > URL: https://issues.apache.org/jira/browse/HIVE-13218 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > Attachments: HIVE-13218.patch > > > 1) IO threads need to be settable when creating the package, and should be > equal to the number of executors by default. > 2) uber should be disabled in "all" mode as it's slower than running in LLAP. > Maybe others. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Assigned] (HIVE-13216) ORC Reader will leave file open until GC when opening a malformed ORC file
[
https://issues.apache.org/jira/browse/HIVE-13216?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sergey Shelukhin reassigned HIVE-13216:
---
Assignee: Sergey Shelukhin
> ORC Reader will leave file open until GC when opening a malformed ORC file
> --
>
> Key: HIVE-13216
> URL: https://issues.apache.org/jira/browse/HIVE-13216
> Project: Hive
> Issue Type: Bug
> Components: File Formats, ORC
>Affects Versions: 1.2.0, 1.2.1
>Reporter: Yuxing Yao
>Assignee: Sergey Shelukhin
>Priority: Minor
>
> In ORC extractMetaInfoFromFooter method of ReaderImpl.java:
> A new input stream is open without try-catch-finally to enforce closing.
> Once the footer parse has some exception, the stream close will miss.
> Until GC happen to close the stream.
> private static FileMetaInfo extractMetaInfoFromFooter(FileSystem fs,
> Path path,
> long maxFileLength
> ) throws IOException {
> FSDataInputStream file = fs.open(path);
> ...
> file.close();
> return new FileMetaInfo(
> ps.getCompression().toString(),
> (int) ps.getCompressionBlockSize(),
> (int) ps.getMetadataLength(),
> buffer,
> ps.getVersionList(),
> writerVersion
> );
> }
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-13218) LLAP: better configs part 1
[ https://issues.apache.org/jira/browse/HIVE-13218?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HIVE-13218: Status: Patch Available (was: Open) [~gopalv] fyi > LLAP: better configs part 1 > --- > > Key: HIVE-13218 > URL: https://issues.apache.org/jira/browse/HIVE-13218 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > Attachments: HIVE-13218.patch > > > 1) IO threads need to be settable when creating the package, and should be > equal to the number of executors by default. > 2) uber should be disabled in "all" mode as it's slower than running in LLAP. > Maybe others. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13222) Move rc-file-v0.rc used on TestRCFile.java to src/test/resources
[
https://issues.apache.org/jira/browse/HIVE-13222?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15183900#comment-15183900
]
Szehon Ho commented on HIVE-13222:
--
+1
> Move rc-file-v0.rc used on TestRCFile.java to src/test/resources
>
>
> Key: HIVE-13222
> URL: https://issues.apache.org/jira/browse/HIVE-13222
> Project: Hive
> Issue Type: Improvement
>Affects Versions: 2.0.0
>Reporter: Sergio Peña
>Assignee: Sergio Peña
>Priority: Minor
> Attachments: HIVE-13222.1.patch
>
>
> The file 'rc-file-v0.rc' is used by TestRCFile to validate RC files. This
> file is on {{src/test/data}}, and it should be good to move it to
> {{src/test/resources}} so the file is packaged into the test-sources.jar when
> -Psources is invoked.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-13222) Move rc-file-v0.rc used on TestRCFile.java to src/test/resources
[
https://issues.apache.org/jira/browse/HIVE-13222?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sergio Peña updated HIVE-13222:
---
Status: Patch Available (was: Open)
> Move rc-file-v0.rc used on TestRCFile.java to src/test/resources
>
>
> Key: HIVE-13222
> URL: https://issues.apache.org/jira/browse/HIVE-13222
> Project: Hive
> Issue Type: Improvement
>Affects Versions: 2.0.0
>Reporter: Sergio Peña
>Assignee: Sergio Peña
>Priority: Minor
> Attachments: HIVE-13222.1.patch
>
>
> The file 'rc-file-v0.rc' is used by TestRCFile to validate RC files. This
> file is on {{src/test/data}}, and it should be good to move it to
> {{src/test/resources}} so the file is packaged into the test-sources.jar when
> -Psources is invoked.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-13222) Move rc-file-v0.rc used on TestRCFile.java to src/test/resources
[
https://issues.apache.org/jira/browse/HIVE-13222?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sergio Peña updated HIVE-13222:
---
Attachment: HIVE-13222.1.patch
> Move rc-file-v0.rc used on TestRCFile.java to src/test/resources
>
>
> Key: HIVE-13222
> URL: https://issues.apache.org/jira/browse/HIVE-13222
> Project: Hive
> Issue Type: Improvement
>Affects Versions: 2.0.0
>Reporter: Sergio Peña
>Assignee: Sergio Peña
>Priority: Minor
> Attachments: HIVE-13222.1.patch
>
>
> The file 'rc-file-v0.rc' is used by TestRCFile to validate RC files. This
> file is on {{src/test/data}}, and it should be good to move it to
> {{src/test/resources}} so the file is packaged into the test-sources.jar when
> -Psources is invoked.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-13206) Create a test-sources.jar when -Psources profile is invoked
[ https://issues.apache.org/jira/browse/HIVE-13206?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15183889#comment-15183889 ] Szehon Ho commented on HIVE-13206: -- Got it, +1 > Create a test-sources.jar when -Psources profile is invoked > --- > > Key: HIVE-13206 > URL: https://issues.apache.org/jira/browse/HIVE-13206 > Project: Hive > Issue Type: Improvement >Affects Versions: 2.0.0 >Reporter: Sergio Peña >Assignee: Sergio Peña >Priority: Minor > Attachments: HIVE-13206.1.patch, HIVE-13206.2.patch > > > It'd be nice to attach a test-sources jar alongside the others as part of the > build, to provide test resources. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13206) Create a test-sources.jar when -Psources profile is invoked
[
https://issues.apache.org/jira/browse/HIVE-13206?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15183873#comment-15183873
]
Sergio Peña commented on HIVE-13206:
The tests-sources.jar are created only when {{-Psources}} is invoked. And the
building time when adding such profile is only 6 seconds more.
These test-jars are sometimes used on testing platforms where only the .jars
are used, and not the full source directory. So, we need to pack the test
sources in such jars to be able to execute the unit test.
> Create a test-sources.jar when -Psources profile is invoked
> ---
>
> Key: HIVE-13206
> URL: https://issues.apache.org/jira/browse/HIVE-13206
> Project: Hive
> Issue Type: Improvement
>Affects Versions: 2.0.0
>Reporter: Sergio Peña
>Assignee: Sergio Peña
>Priority: Minor
> Attachments: HIVE-13206.1.patch, HIVE-13206.2.patch
>
>
> It'd be nice to attach a test-sources jar alongside the others as part of the
> build, to provide test resources.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-7532) allow disabling direct sql per query with external metastore
[ https://issues.apache.org/jira/browse/HIVE-7532?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15183855#comment-15183855 ] Aihua Xu commented on HIVE-7532: Thanks for explaining. Yeah. I also noticed that. It works differently than local mode. > allow disabling direct sql per query with external metastore > > > Key: HIVE-7532 > URL: https://issues.apache.org/jira/browse/HIVE-7532 > Project: Hive > Issue Type: Improvement >Reporter: Sergey Shelukhin >Assignee: Navis > Fix For: 0.14.0 > > Attachments: HIVE-7532.1.patch.txt, HIVE-7532.2.nogen, > HIVE-7532.2.patch.txt, HIVE-7532.3.patch.txt, HIVE-7532.4.patch.txt, > HIVE-7532.5.patch.txt, HIVE-7532.6.patch.txt > > > Currently with external metastore, direct sql can only be disabled via > metastore config globally. Perhaps it makes sense to have the ability to > propagate the setting per query from client to override the metastore > setting, e.g. if one particular query causes it to fail. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13206) Create a test-sources.jar when -Psources profile is invoked
[ https://issues.apache.org/jira/browse/HIVE-13206?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15183843#comment-15183843 ] Szehon Ho commented on HIVE-13206: -- Cool thanks. Do you see if it increase build-time a lot? And I'm not sure the use case, but you need all these test-jars? > Create a test-sources.jar when -Psources profile is invoked > --- > > Key: HIVE-13206 > URL: https://issues.apache.org/jira/browse/HIVE-13206 > Project: Hive > Issue Type: Improvement >Affects Versions: 2.0.0 >Reporter: Sergio Peña >Assignee: Sergio Peña >Priority: Minor > Attachments: HIVE-13206.1.patch, HIVE-13206.2.patch > > > It'd be nice to attach a test-sources jar alongside the others as part of the > build, to provide test resources. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13218) LLAP: better configs part 1
[ https://issues.apache.org/jira/browse/HIVE-13218?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HIVE-13218: Attachment: HIVE-13218.patch > LLAP: better configs part 1 > --- > > Key: HIVE-13218 > URL: https://issues.apache.org/jira/browse/HIVE-13218 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > Attachments: HIVE-13218.patch > > > 1) IO threads need to be settable when creating the package, and should be > equal to the number of executors by default. > 2) uber should be disabled in "all" mode as it's slower than running in LLAP. > Maybe others. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13178) Enhance ORC Schema Evolution to handle more standard data type conversions
[ https://issues.apache.org/jira/browse/HIVE-13178?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matt McCline updated HIVE-13178: Attachment: (was: HIVE-13178.03.patch) > Enhance ORC Schema Evolution to handle more standard data type conversions > -- > > Key: HIVE-13178 > URL: https://issues.apache.org/jira/browse/HIVE-13178 > Project: Hive > Issue Type: Bug > Components: Hive, ORC >Reporter: Matt McCline >Assignee: Matt McCline >Priority: Critical > Attachments: HIVE-13178.01.patch, HIVE-13178.02.patch > > > Currently, SHORT -> INT -> BIGINT is supported. > Handle ORC data type conversions permitted by Implicit conversion allowed by > TypeIntoUtils.implicitConvertible method. >* STRING_GROUP -> DOUBLE >* STRING_GROUP -> DECIMAL >* DATE_GROUP -> STRING >* NUMERIC_GROUP -> STRING >* STRING_GROUP -> STRING_GROUP >* >* // Upward from "lower" type to "higher" numeric type: >* BYTE -> SHORT -> INT -> BIGINT -> FLOAT -> DOUBLE -> DECIMAL -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13206) Create a test-sources.jar when -Psources profile is invoked
[ https://issues.apache.org/jira/browse/HIVE-13206?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergio Peña updated HIVE-13206: --- Attachment: HIVE-13206.2.patch [~szehon] I moved the changes to each of the pom.xml found on components with tests. > Create a test-sources.jar when -Psources profile is invoked > --- > > Key: HIVE-13206 > URL: https://issues.apache.org/jira/browse/HIVE-13206 > Project: Hive > Issue Type: Improvement >Affects Versions: 2.0.0 >Reporter: Sergio Peña >Assignee: Sergio Peña >Priority: Minor > Attachments: HIVE-13206.1.patch, HIVE-13206.2.patch > > > It'd be nice to attach a test-sources jar alongside the others as part of the > build, to provide test resources. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13178) Enhance ORC Schema Evolution to handle more standard data type conversions
[ https://issues.apache.org/jira/browse/HIVE-13178?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matt McCline updated HIVE-13178: Attachment: HIVE-13178.03.patch > Enhance ORC Schema Evolution to handle more standard data type conversions > -- > > Key: HIVE-13178 > URL: https://issues.apache.org/jira/browse/HIVE-13178 > Project: Hive > Issue Type: Bug > Components: Hive, ORC >Reporter: Matt McCline >Assignee: Matt McCline >Priority: Critical > Attachments: HIVE-13178.01.patch, HIVE-13178.02.patch, > HIVE-13178.03.patch > > > Currently, SHORT -> INT -> BIGINT is supported. > Handle ORC data type conversions permitted by Implicit conversion allowed by > TypeIntoUtils.implicitConvertible method. >* STRING_GROUP -> DOUBLE >* STRING_GROUP -> DECIMAL >* DATE_GROUP -> STRING >* NUMERIC_GROUP -> STRING >* STRING_GROUP -> STRING_GROUP >* >* // Upward from "lower" type to "higher" numeric type: >* BYTE -> SHORT -> INT -> BIGINT -> FLOAT -> DOUBLE -> DECIMAL -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13204) Vectorization: Add ChainedCheckerFactory for LIKE
[ https://issues.apache.org/jira/browse/HIVE-13204?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15183801#comment-15183801 ] Gopal V commented on HIVE-13204: The existing tests for UDFLike don't actually run because they're missing @Test annotations. Fixing those tests in the next pass. > Vectorization: Add ChainedCheckerFactory for LIKE > -- > > Key: HIVE-13204 > URL: https://issues.apache.org/jira/browse/HIVE-13204 > Project: Hive > Issue Type: Improvement > Components: Vectorization >Affects Versions: 2.1.0 >Reporter: Gopal V >Assignee: Gopal V > Attachments: HIVE-13204.1.patch, HIVE-13204.2.patch, > HIVE-13204.3.patch, regex-cpu-like.png > > > Currently, Vectorization runs through a UTF-8 decode to produce a String & > then check for mildly complex patterns like "http://%.exe; using a Regex. > Since this pattern doesn't need any backtracking patterns, using a full > fledged Regex is too expensive. > !regex-cpu-like.png! -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-11675) make use of file footer PPD API in ETL strategy or separate strategy
[ https://issues.apache.org/jira/browse/HIVE-11675?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15183798#comment-15183798 ] Prasanth Jayachandran commented on HIVE-11675: -- new changes look good to me, +1. Pending another test run > make use of file footer PPD API in ETL strategy or separate strategy > > > Key: HIVE-11675 > URL: https://issues.apache.org/jira/browse/HIVE-11675 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > Attachments: HIVE-11675.01.patch, HIVE-11675.02.patch, > HIVE-11675.03.patch, HIVE-11675.04.patch, HIVE-11675.05.patch, > HIVE-11675.06.patch, HIVE-11675.07.patch, HIVE-11675.08.patch, > HIVE-11675.09.patch, HIVE-11675.10.patch, HIVE-11675.11.patch, > HIVE-11675.patch, HIVE-11675.premature.opti.patch > > > Need to take a look at the best flow. It won't be much different if we do > filtering metastore call for each partition. So perhaps we'd need the custom > sync point/batching after all. > Or we can make it opportunistic and not fetch any footers unless it can be > pushed down to metastore or fetched from local cache, that way the only slow > threaded op is directory listings -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (HIVE-9820) LLAP: Use a share-nothing scoreboard /status implementation
[ https://issues.apache.org/jira/browse/HIVE-9820?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gopal V resolved HIVE-9820. --- Resolution: Not A Problem This issue is now stale - closing the issue as "Not a Problem". > LLAP: Use a share-nothing scoreboard /status implementation > --- > > Key: HIVE-9820 > URL: https://issues.apache.org/jira/browse/HIVE-9820 > Project: Hive > Issue Type: Sub-task > Components: llap >Affects Versions: llap >Reporter: Gopal V >Assignee: Yohei Abe > > To prevent thread-conflicts in executor information, the Apache HTTP servers > use a share-nothing data structure known as a scoreboard. > This is read by various systems like mod_status to read out the current state > of executors available for PHP (and similar mod_* engines). > The /status output is traditionally periodically read by the load-balancers > to route requests away from busy machines. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-7532) allow disabling direct sql per query with external metastore
[ https://issues.apache.org/jira/browse/HIVE-7532?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15183765#comment-15183765 ] Sergey Shelukhin commented on HIVE-7532: I think the main problem was propagating it to metastore. By default, there is no such option if the metastore is external. It's possible that there's a bug if the connection is reopened. > allow disabling direct sql per query with external metastore > > > Key: HIVE-7532 > URL: https://issues.apache.org/jira/browse/HIVE-7532 > Project: Hive > Issue Type: Improvement >Reporter: Sergey Shelukhin >Assignee: Navis > Fix For: 0.14.0 > > Attachments: HIVE-7532.1.patch.txt, HIVE-7532.2.nogen, > HIVE-7532.2.patch.txt, HIVE-7532.3.patch.txt, HIVE-7532.4.patch.txt, > HIVE-7532.5.patch.txt, HIVE-7532.6.patch.txt > > > Currently with external metastore, direct sql can only be disabled via > metastore config globally. Perhaps it makes sense to have the ability to > propagate the setting per query from client to override the metastore > setting, e.g. if one particular query causes it to fail. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-11675) make use of file footer PPD API in ETL strategy or separate strategy
[ https://issues.apache.org/jira/browse/HIVE-11675?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HIVE-11675: Attachment: (was: HIVE-11675.11.patch) > make use of file footer PPD API in ETL strategy or separate strategy > > > Key: HIVE-11675 > URL: https://issues.apache.org/jira/browse/HIVE-11675 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > Attachments: HIVE-11675.01.patch, HIVE-11675.02.patch, > HIVE-11675.03.patch, HIVE-11675.04.patch, HIVE-11675.05.patch, > HIVE-11675.06.patch, HIVE-11675.07.patch, HIVE-11675.08.patch, > HIVE-11675.09.patch, HIVE-11675.10.patch, HIVE-11675.11.patch, > HIVE-11675.patch, HIVE-11675.premature.opti.patch > > > Need to take a look at the best flow. It won't be much different if we do > filtering metastore call for each partition. So perhaps we'd need the custom > sync point/batching after all. > Or we can make it opportunistic and not fetch any footers unless it can be > pushed down to metastore or fetched from local cache, that way the only slow > threaded op is directory listings -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-11675) make use of file footer PPD API in ETL strategy or separate strategy
[ https://issues.apache.org/jira/browse/HIVE-11675?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HIVE-11675: Attachment: HIVE-11675.11.patch > make use of file footer PPD API in ETL strategy or separate strategy > > > Key: HIVE-11675 > URL: https://issues.apache.org/jira/browse/HIVE-11675 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > Attachments: HIVE-11675.01.patch, HIVE-11675.02.patch, > HIVE-11675.03.patch, HIVE-11675.04.patch, HIVE-11675.05.patch, > HIVE-11675.06.patch, HIVE-11675.07.patch, HIVE-11675.08.patch, > HIVE-11675.09.patch, HIVE-11675.10.patch, HIVE-11675.11.patch, > HIVE-11675.patch, HIVE-11675.premature.opti.patch > > > Need to take a look at the best flow. It won't be much different if we do > filtering metastore call for each partition. So perhaps we'd need the custom > sync point/batching after all. > Or we can make it opportunistic and not fetch any footers unless it can be > pushed down to metastore or fetched from local cache, that way the only slow > threaded op is directory listings -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-11675) make use of file footer PPD API in ETL strategy or separate strategy
[ https://issues.apache.org/jira/browse/HIVE-11675?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HIVE-11675: Attachment: HIVE-11675.11.patch > make use of file footer PPD API in ETL strategy or separate strategy > > > Key: HIVE-11675 > URL: https://issues.apache.org/jira/browse/HIVE-11675 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > Attachments: HIVE-11675.01.patch, HIVE-11675.02.patch, > HIVE-11675.03.patch, HIVE-11675.04.patch, HIVE-11675.05.patch, > HIVE-11675.06.patch, HIVE-11675.07.patch, HIVE-11675.08.patch, > HIVE-11675.09.patch, HIVE-11675.10.patch, HIVE-11675.11.patch, > HIVE-11675.patch, HIVE-11675.premature.opti.patch > > > Need to take a look at the best flow. It won't be much different if we do > filtering metastore call for each partition. So perhaps we'd need the custom > sync point/batching after all. > Or we can make it opportunistic and not fetch any footers unless it can be > pushed down to metastore or fetched from local cache, that way the only slow > threaded op is directory listings -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-11675) make use of file footer PPD API in ETL strategy or separate strategy
[ https://issues.apache.org/jira/browse/HIVE-11675?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HIVE-11675: Attachment: HIVE-11675.premature.opti.patch Removing some premature optimization and attaching the patch to re-add it if desired. > make use of file footer PPD API in ETL strategy or separate strategy > > > Key: HIVE-11675 > URL: https://issues.apache.org/jira/browse/HIVE-11675 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > Attachments: HIVE-11675.01.patch, HIVE-11675.02.patch, > HIVE-11675.03.patch, HIVE-11675.04.patch, HIVE-11675.05.patch, > HIVE-11675.06.patch, HIVE-11675.07.patch, HIVE-11675.08.patch, > HIVE-11675.09.patch, HIVE-11675.10.patch, HIVE-11675.patch, > HIVE-11675.premature.opti.patch > > > Need to take a look at the best flow. It won't be much different if we do > filtering metastore call for each partition. So perhaps we'd need the custom > sync point/batching after all. > Or we can make it opportunistic and not fetch any footers unless it can be > pushed down to metastore or fetched from local cache, that way the only slow > threaded op is directory listings -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13206) Create a test-sources.jar when -Psources profile is invoked
[ https://issues.apache.org/jira/browse/HIVE-13206?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15183745#comment-15183745 ] Sergio Peña commented on HIVE-13206: You're right. I saw that there are some JAR files created that have just JAR information in it without any other file. > Create a test-sources.jar when -Psources profile is invoked > --- > > Key: HIVE-13206 > URL: https://issues.apache.org/jira/browse/HIVE-13206 > Project: Hive > Issue Type: Improvement >Affects Versions: 2.0.0 >Reporter: Sergio Peña >Assignee: Sergio Peña >Priority: Minor > Attachments: HIVE-13206.1.patch > > > It'd be nice to attach a test-sources jar alongside the others as part of the > build, to provide test resources. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13083) Writing HiveDecimal to ORC can wrongly suppress present stream
[
https://issues.apache.org/jira/browse/HIVE-13083?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15183720#comment-15183720
]
Prasanth Jayachandran commented on HIVE-13083:
--
Committed patch to master, branch-2.0 and branch-1
> Writing HiveDecimal to ORC can wrongly suppress present stream
> --
>
> Key: HIVE-13083
> URL: https://issues.apache.org/jira/browse/HIVE-13083
> Project: Hive
> Issue Type: Bug
>Affects Versions: 0.13.0, 0.14.0, 1.0.0, 1.2.0, 1.1.0, 1.3.0, 2.0.0, 2.1.0
>Reporter: Yi Zhang
>Assignee: Prasanth Jayachandran
> Fix For: 1.3.0, 2.1.0, 2.0.1
>
> Attachments: HIVE-13083-branch-1.patch, HIVE-13083.1.patch,
> HIVE-13083.2.patch, HIVE-13083.3.patch, HIVE-13083.4.patch,
> HIVE-13083.4.patch, HIVE-13083.5.patch
>
>
> HIVE-3976 can cause ORC file to be unreadable. The changes introduced in
> HIVE-3976 for DecimalTreeWriter can create null values after updating the
> isPresent stream.
> https://github.com/apache/hive/blob/branch-0.13/ql/src/java/org/apache/hadoop/hive/ql/io/orc/WriterImpl.java#L1337
> As result of the above return statement, isPresent stream state can become
> wrong. The isPresent stream thinks all values are non-null and hence
> suppressed. But the data stream will be of 0 length. When reading such files
> we will get the following exception
> {code}
> Caused by: java.io.EOFException: Reading BigInteger past EOF from compressed
> stream Stream for column 3 kind DATA position: 0 length: 0 range: 0 offset: 0
> limit: 0
> at
> org.apache.hadoop.hive.ql.io.orc.SerializationUtils.readBigInteger(SerializationUtils.java:176)
> at
> org.apache.hadoop.hive.ql.io.orc.TreeReaderFactory$DecimalTreeReader.next(TreeReaderFactory.java:1264)
> at
> org.apache.hadoop.hive.ql.io.orc.TreeReaderFactory$StructTreeReader.next(TreeReaderFactory.java:2004)
> at
> org.apache.hadoop.hive.ql.io.orc.RecordReaderImpl.next(RecordReaderImpl.java:1039)
> ... 24 more
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-13083) Writing HiveDecimal to ORC can wrongly suppress present stream
[
https://issues.apache.org/jira/browse/HIVE-13083?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Prasanth Jayachandran updated HIVE-13083:
-
Fix Version/s: 2.0.1
> Writing HiveDecimal to ORC can wrongly suppress present stream
> --
>
> Key: HIVE-13083
> URL: https://issues.apache.org/jira/browse/HIVE-13083
> Project: Hive
> Issue Type: Bug
>Affects Versions: 0.13.0, 0.14.0, 1.0.0, 1.2.0, 1.1.0, 1.3.0, 2.0.0, 2.1.0
>Reporter: Yi Zhang
>Assignee: Prasanth Jayachandran
> Fix For: 1.3.0, 2.1.0, 2.0.1
>
> Attachments: HIVE-13083-branch-1.patch, HIVE-13083.1.patch,
> HIVE-13083.2.patch, HIVE-13083.3.patch, HIVE-13083.4.patch,
> HIVE-13083.4.patch, HIVE-13083.5.patch
>
>
> HIVE-3976 can cause ORC file to be unreadable. The changes introduced in
> HIVE-3976 for DecimalTreeWriter can create null values after updating the
> isPresent stream.
> https://github.com/apache/hive/blob/branch-0.13/ql/src/java/org/apache/hadoop/hive/ql/io/orc/WriterImpl.java#L1337
> As result of the above return statement, isPresent stream state can become
> wrong. The isPresent stream thinks all values are non-null and hence
> suppressed. But the data stream will be of 0 length. When reading such files
> we will get the following exception
> {code}
> Caused by: java.io.EOFException: Reading BigInteger past EOF from compressed
> stream Stream for column 3 kind DATA position: 0 length: 0 range: 0 offset: 0
> limit: 0
> at
> org.apache.hadoop.hive.ql.io.orc.SerializationUtils.readBigInteger(SerializationUtils.java:176)
> at
> org.apache.hadoop.hive.ql.io.orc.TreeReaderFactory$DecimalTreeReader.next(TreeReaderFactory.java:1264)
> at
> org.apache.hadoop.hive.ql.io.orc.TreeReaderFactory$StructTreeReader.next(TreeReaderFactory.java:2004)
> at
> org.apache.hadoop.hive.ql.io.orc.RecordReaderImpl.next(RecordReaderImpl.java:1039)
> ... 24 more
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-13083) Writing HiveDecimal to ORC can wrongly suppress present stream
[
https://issues.apache.org/jira/browse/HIVE-13083?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Prasanth Jayachandran updated HIVE-13083:
-
Attachment: HIVE-13083.5.patch
HIVE-13083-branch-1.patch
Latest master and branch-1 patches that got committed.
> Writing HiveDecimal to ORC can wrongly suppress present stream
> --
>
> Key: HIVE-13083
> URL: https://issues.apache.org/jira/browse/HIVE-13083
> Project: Hive
> Issue Type: Bug
>Affects Versions: 0.13.0, 0.14.0, 1.0.0, 1.2.0, 1.1.0, 1.3.0, 2.0.0, 2.1.0
>Reporter: Yi Zhang
>Assignee: Prasanth Jayachandran
> Fix For: 1.3.0, 2.1.0
>
> Attachments: HIVE-13083-branch-1.patch, HIVE-13083.1.patch,
> HIVE-13083.2.patch, HIVE-13083.3.patch, HIVE-13083.4.patch,
> HIVE-13083.4.patch, HIVE-13083.5.patch
>
>
> HIVE-3976 can cause ORC file to be unreadable. The changes introduced in
> HIVE-3976 for DecimalTreeWriter can create null values after updating the
> isPresent stream.
> https://github.com/apache/hive/blob/branch-0.13/ql/src/java/org/apache/hadoop/hive/ql/io/orc/WriterImpl.java#L1337
> As result of the above return statement, isPresent stream state can become
> wrong. The isPresent stream thinks all values are non-null and hence
> suppressed. But the data stream will be of 0 length. When reading such files
> we will get the following exception
> {code}
> Caused by: java.io.EOFException: Reading BigInteger past EOF from compressed
> stream Stream for column 3 kind DATA position: 0 length: 0 range: 0 offset: 0
> limit: 0
> at
> org.apache.hadoop.hive.ql.io.orc.SerializationUtils.readBigInteger(SerializationUtils.java:176)
> at
> org.apache.hadoop.hive.ql.io.orc.TreeReaderFactory$DecimalTreeReader.next(TreeReaderFactory.java:1264)
> at
> org.apache.hadoop.hive.ql.io.orc.TreeReaderFactory$StructTreeReader.next(TreeReaderFactory.java:2004)
> at
> org.apache.hadoop.hive.ql.io.orc.RecordReaderImpl.next(RecordReaderImpl.java:1039)
> ... 24 more
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-13083) Writing HiveDecimal to ORC can wrongly suppress present stream
[
https://issues.apache.org/jira/browse/HIVE-13083?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Prasanth Jayachandran updated HIVE-13083:
-
Attachment: (was: HIVE-13083-branch-1.patch)
> Writing HiveDecimal to ORC can wrongly suppress present stream
> --
>
> Key: HIVE-13083
> URL: https://issues.apache.org/jira/browse/HIVE-13083
> Project: Hive
> Issue Type: Bug
>Affects Versions: 0.13.0, 0.14.0, 1.0.0, 1.2.0, 1.1.0, 1.3.0, 2.0.0, 2.1.0
>Reporter: Yi Zhang
>Assignee: Prasanth Jayachandran
> Attachments: HIVE-13083.1.patch, HIVE-13083.2.patch,
> HIVE-13083.3.patch, HIVE-13083.4.patch, HIVE-13083.4.patch
>
>
> HIVE-3976 can cause ORC file to be unreadable. The changes introduced in
> HIVE-3976 for DecimalTreeWriter can create null values after updating the
> isPresent stream.
> https://github.com/apache/hive/blob/branch-0.13/ql/src/java/org/apache/hadoop/hive/ql/io/orc/WriterImpl.java#L1337
> As result of the above return statement, isPresent stream state can become
> wrong. The isPresent stream thinks all values are non-null and hence
> suppressed. But the data stream will be of 0 length. When reading such files
> we will get the following exception
> {code}
> Caused by: java.io.EOFException: Reading BigInteger past EOF from compressed
> stream Stream for column 3 kind DATA position: 0 length: 0 range: 0 offset: 0
> limit: 0
> at
> org.apache.hadoop.hive.ql.io.orc.SerializationUtils.readBigInteger(SerializationUtils.java:176)
> at
> org.apache.hadoop.hive.ql.io.orc.TreeReaderFactory$DecimalTreeReader.next(TreeReaderFactory.java:1264)
> at
> org.apache.hadoop.hive.ql.io.orc.TreeReaderFactory$StructTreeReader.next(TreeReaderFactory.java:2004)
> at
> org.apache.hadoop.hive.ql.io.orc.RecordReaderImpl.next(RecordReaderImpl.java:1039)
> ... 24 more
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-12558) LLAP: output QueryFragmentCounters somewhere
[ https://issues.apache.org/jira/browse/HIVE-12558?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Prasanth Jayachandran updated HIVE-12558: - Status: Open (was: Patch Available) > LLAP: output QueryFragmentCounters somewhere > > > Key: HIVE-12558 > URL: https://issues.apache.org/jira/browse/HIVE-12558 > Project: Hive > Issue Type: Bug > Components: llap >Reporter: Sergey Shelukhin >Assignee: Prasanth Jayachandran > Attachments: HIVE-12558.1.patch, HIVE-12558.2.patch, > HIVE-12558.3.patch, HIVE-12558.3.patch, HIVE-12558.wip.patch, > sample-output.png, sample-output2.png > > > Right now, LLAP logs counters for every fragment; most of them are IO related > and could be very useful, they also include table names so that things like > cache hit ratio, etc., could be calculated for every table. > We need to output them to some metrics system (preserving the breakdown by > table, possibly also adding query ID or even stage) so that they'd be usable > without grep/sed/awk. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-12558) LLAP: output QueryFragmentCounters somewhere
[ https://issues.apache.org/jira/browse/HIVE-12558?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Prasanth Jayachandran updated HIVE-12558: - Status: Patch Available (was: Open) > LLAP: output QueryFragmentCounters somewhere > > > Key: HIVE-12558 > URL: https://issues.apache.org/jira/browse/HIVE-12558 > Project: Hive > Issue Type: Bug > Components: llap >Reporter: Sergey Shelukhin >Assignee: Prasanth Jayachandran > Attachments: HIVE-12558.1.patch, HIVE-12558.2.patch, > HIVE-12558.3.patch, HIVE-12558.3.patch, HIVE-12558.wip.patch, > sample-output.png, sample-output2.png > > > Right now, LLAP logs counters for every fragment; most of them are IO related > and could be very useful, they also include table names so that things like > cache hit ratio, etc., could be calculated for every table. > We need to output them to some metrics system (preserving the breakdown by > table, possibly also adding query ID or even stage) so that they'd be usable > without grep/sed/awk. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-12558) LLAP: output QueryFragmentCounters somewhere
[ https://issues.apache.org/jira/browse/HIVE-12558?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Prasanth Jayachandran updated HIVE-12558: - Attachment: HIVE-12558.3.patch > LLAP: output QueryFragmentCounters somewhere > > > Key: HIVE-12558 > URL: https://issues.apache.org/jira/browse/HIVE-12558 > Project: Hive > Issue Type: Bug > Components: llap >Reporter: Sergey Shelukhin >Assignee: Prasanth Jayachandran > Attachments: HIVE-12558.1.patch, HIVE-12558.2.patch, > HIVE-12558.3.patch, HIVE-12558.3.patch, HIVE-12558.wip.patch, > sample-output.png, sample-output2.png > > > Right now, LLAP logs counters for every fragment; most of them are IO related > and could be very useful, they also include table names so that things like > cache hit ratio, etc., could be calculated for every table. > We need to output them to some metrics system (preserving the breakdown by > table, possibly also adding query ID or even stage) so that they'd be usable > without grep/sed/awk. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-12558) LLAP: output QueryFragmentCounters somewhere
[ https://issues.apache.org/jira/browse/HIVE-12558?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15183675#comment-15183675 ] Prasanth Jayachandran commented on HIVE-12558: -- I don't know why it lost precommit spot in the queue. Reuploading again. > LLAP: output QueryFragmentCounters somewhere > > > Key: HIVE-12558 > URL: https://issues.apache.org/jira/browse/HIVE-12558 > Project: Hive > Issue Type: Bug > Components: llap >Reporter: Sergey Shelukhin >Assignee: Prasanth Jayachandran > Attachments: HIVE-12558.1.patch, HIVE-12558.2.patch, > HIVE-12558.3.patch, HIVE-12558.3.patch, HIVE-12558.wip.patch, > sample-output.png, sample-output2.png > > > Right now, LLAP logs counters for every fragment; most of them are IO related > and could be very useful, they also include table names so that things like > cache hit ratio, etc., could be calculated for every table. > We need to output them to some metrics system (preserving the breakdown by > table, possibly also adding query ID or even stage) so that they'd be usable > without grep/sed/awk. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-12875) Verify sem.getInputs() and sem.getOutputs()
[ https://issues.apache.org/jira/browse/HIVE-12875?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15183652#comment-15183652 ] Steve Scaffidi commented on HIVE-12875: --- Just adding some info so other people don't have to go through all the effort I did to find this JIRA. Hopefully it will make it easier to find when you google "CVE-2015-7521" * The bug this issue fixes was reported as CVE-2015-7521 which I found out about at the following: ** hive-user mailing list: http://mail-archives.apache.org/mod_mbox/hive-user/201601.mbox/%3c20160128205008.2154f18...@minotaur.apache.org%3E ** bugtraq: http://seclists.org/bugtraq/2016/Jan/157 ** redhat cve page: https://access.redhat.com/security/cve/cve-2015-7521 ** http://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2015-7521 ** https://web.nvd.nist.gov/view/vuln/detail?vulnId=CVE-2015-7521 http://seclists.org/bugtraq/2016/Jan/157 * A workaround jar and source are available here: http://apache.org/dist/hive/hive-parent-auth-hook/ * The git commit of this fix is here: https://git-wip-us.apache.org/repos/asf?p=hive.git;a=commit;h=98f933f269e6b528ef84912b3d701ca3272ec04b > Verify sem.getInputs() and sem.getOutputs() > --- > > Key: HIVE-12875 > URL: https://issues.apache.org/jira/browse/HIVE-12875 > Project: Hive > Issue Type: Bug >Reporter: Sushanth Sowmyan >Assignee: Sushanth Sowmyan > Fix For: 1.3.0, 2.0.0, 1.0.2, 1.2.2, 1.1.2, 2.1.0 > > Attachments: HIVE-12875.patch > > > For every partition entity object present in sem.getInputs() and > sem.getOutputs(), we must verify the appropriate Table in the list of > Entities. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-11675) make use of file footer PPD API in ETL strategy or separate strategy
[ https://issues.apache.org/jira/browse/HIVE-11675?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HIVE-11675: Attachment: HIVE-11675.10.patch This patch needs a small update, I'll take a queue spot for now ;) > make use of file footer PPD API in ETL strategy or separate strategy > > > Key: HIVE-11675 > URL: https://issues.apache.org/jira/browse/HIVE-11675 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > Attachments: HIVE-11675.01.patch, HIVE-11675.02.patch, > HIVE-11675.03.patch, HIVE-11675.04.patch, HIVE-11675.05.patch, > HIVE-11675.06.patch, HIVE-11675.07.patch, HIVE-11675.08.patch, > HIVE-11675.09.patch, HIVE-11675.10.patch, HIVE-11675.patch > > > Need to take a look at the best flow. It won't be much different if we do > filtering metastore call for each partition. So perhaps we'd need the custom > sync point/batching after all. > Or we can make it opportunistic and not fetch any footers unless it can be > pushed down to metastore or fetched from local cache, that way the only slow > threaded op is directory listings -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13213) make DbLockManger work for non-acid resources
[ https://issues.apache.org/jira/browse/HIVE-13213?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15183615#comment-15183615 ] Eugene Koifman commented on HIVE-13213: --- That is the behavior documented previously in https://cwiki.apache.org/confluence/display/Hive/Locking. W/o this, if users enable ACID but continue to run their existing workflows with non-acid resources they end up with with different semantics than before and they have no way to get those semantics back except to disable acid. > make DbLockManger work for non-acid resources > - > > Key: HIVE-13213 > URL: https://issues.apache.org/jira/browse/HIVE-13213 > Project: Hive > Issue Type: Bug > Components: Transactions >Affects Versions: 1.0.0 >Reporter: Eugene Koifman >Assignee: Eugene Koifman > Attachments: HIVE-13213.patch > > > for example, > insert into T values(...) > if T is an ACID table we acquire Read lock > but for non-acid table it should acquire Exclusive lock -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13213) make DbLockManger work for non-acid resources
[ https://issues.apache.org/jira/browse/HIVE-13213?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15183595#comment-15183595 ] Alan Gates commented on HIVE-13213: --- Well, not necessarily. As it stands now non-ACID readers get read committed consistency. So you're arguing that both acid and non-acid tables should have the same consistency model. I need to think about that. It's not obvious to me whether that should be true or not. > make DbLockManger work for non-acid resources > - > > Key: HIVE-13213 > URL: https://issues.apache.org/jira/browse/HIVE-13213 > Project: Hive > Issue Type: Bug > Components: Transactions >Affects Versions: 1.0.0 >Reporter: Eugene Koifman >Assignee: Eugene Koifman > Attachments: HIVE-13213.patch > > > for example, > insert into T values(...) > if T is an ACID table we acquire Read lock > but for non-acid table it should acquire Exclusive lock -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13083) Writing HiveDecimal to ORC can wrongly suppress present stream
[
https://issues.apache.org/jira/browse/HIVE-13083?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15183590#comment-15183590
]
Prasanth Jayachandran commented on HIVE-13083:
--
Thanks! I will update golden files for TestJsonFileDump on commit.
> Writing HiveDecimal to ORC can wrongly suppress present stream
> --
>
> Key: HIVE-13083
> URL: https://issues.apache.org/jira/browse/HIVE-13083
> Project: Hive
> Issue Type: Bug
>Affects Versions: 0.13.0, 0.14.0, 1.0.0, 1.2.0, 1.1.0, 1.3.0, 2.0.0, 2.1.0
>Reporter: Yi Zhang
>Assignee: Prasanth Jayachandran
> Attachments: HIVE-13083-branch-1.patch, HIVE-13083.1.patch,
> HIVE-13083.2.patch, HIVE-13083.3.patch, HIVE-13083.4.patch, HIVE-13083.4.patch
>
>
> HIVE-3976 can cause ORC file to be unreadable. The changes introduced in
> HIVE-3976 for DecimalTreeWriter can create null values after updating the
> isPresent stream.
> https://github.com/apache/hive/blob/branch-0.13/ql/src/java/org/apache/hadoop/hive/ql/io/orc/WriterImpl.java#L1337
> As result of the above return statement, isPresent stream state can become
> wrong. The isPresent stream thinks all values are non-null and hence
> suppressed. But the data stream will be of 0 length. When reading such files
> we will get the following exception
> {code}
> Caused by: java.io.EOFException: Reading BigInteger past EOF from compressed
> stream Stream for column 3 kind DATA position: 0 length: 0 range: 0 offset: 0
> limit: 0
> at
> org.apache.hadoop.hive.ql.io.orc.SerializationUtils.readBigInteger(SerializationUtils.java:176)
> at
> org.apache.hadoop.hive.ql.io.orc.TreeReaderFactory$DecimalTreeReader.next(TreeReaderFactory.java:1264)
> at
> org.apache.hadoop.hive.ql.io.orc.TreeReaderFactory$StructTreeReader.next(TreeReaderFactory.java:2004)
> at
> org.apache.hadoop.hive.ql.io.orc.RecordReaderImpl.next(RecordReaderImpl.java:1039)
> ... 24 more
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-13160) HS2 unable to load UDFs on startup when HMS is not ready
[
https://issues.apache.org/jira/browse/HIVE-13160?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15183552#comment-15183552
]
Aihua Xu commented on HIVE-13160:
-
Sure. Do you know how we load all the functions for such scenario?
> HS2 unable to load UDFs on startup when HMS is not ready
>
>
> Key: HIVE-13160
> URL: https://issues.apache.org/jira/browse/HIVE-13160
> Project: Hive
> Issue Type: Bug
> Components: HiveServer2
>Affects Versions: 1.2.1
>Reporter: Eric Lin
>Assignee: Aihua Xu
> Fix For: 2.1.0
>
> Attachments: HIVE-13160.1.patch, HIVE-13160.2.patch
>
>
> The error looks like this:
> {code}
> 2016-02-18 14:43:54,251 INFO hive.metastore: [main]: Trying to connect to
> metastore with URI thrift://host-10-17-81-201.coe.cloudera.com:9083
> 2016-02-18 14:48:54,692 WARN hive.metastore: [main]: Failed to connect to
> the MetaStore Server...
> 2016-02-18 14:48:54,692 INFO hive.metastore: [main]: Waiting 1 seconds
> before next connection attempt.
> 2016-02-18 14:48:55,692 INFO hive.metastore: [main]: Trying to connect to
> metastore with URI thrift://host-10-17-81-201.coe.cloudera.com:9083
> 2016-02-18 14:53:55,800 WARN hive.metastore: [main]: Failed to connect to
> the MetaStore Server...
> 2016-02-18 14:53:55,800 INFO hive.metastore: [main]: Waiting 1 seconds
> before next connection attempt.
> 2016-02-18 14:53:56,801 INFO hive.metastore: [main]: Trying to connect to
> metastore with URI thrift://host-10-17-81-201.coe.cloudera.com:9083
> 2016-02-18 14:58:56,967 WARN hive.metastore: [main]: Failed to connect to
> the MetaStore Server...
> 2016-02-18 14:58:56,967 INFO hive.metastore: [main]: Waiting 1 seconds
> before next connection attempt.
> 2016-02-18 14:58:57,994 WARN hive.ql.metadata.Hive: [main]: Failed to
> register all functions.
> java.lang.RuntimeException: Unable to instantiate
> org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient
> at
> org.apache.hadoop.hive.metastore.MetaStoreUtils.newInstance(MetaStoreUtils.java:1492)
> at
> org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.(RetryingMetaStoreClient.java:64)
> at
> org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.getProxy(RetryingMetaStoreClient.java:74)
> at
> org.apache.hadoop.hive.ql.metadata.Hive.createMetaStoreClient(Hive.java:2915)
> ...
> 016-02-18 14:58:57,997 INFO hive.metastore: [main]: Trying to connect to
> metastore with URI thrift://host-10-17-81-201.coe.cloudera.com:9083
> 2016-02-18 15:03:58,094 WARN hive.metastore: [main]: Failed to connect to
> the MetaStore Server...
> 2016-02-18 15:03:58,095 INFO hive.metastore: [main]: Waiting 1 seconds
> before next connection attempt.
> 2016-02-18 15:03:59,095 INFO hive.metastore: [main]: Trying to connect to
> metastore with URI thrift://host-10-17-81-201.coe.cloudera.com:9083
> 2016-02-18 15:08:59,203 WARN hive.metastore: [main]: Failed to connect to
> the MetaStore Server...
> 2016-02-18 15:08:59,203 INFO hive.metastore: [main]: Waiting 1 seconds
> before next connection attempt.
> 2016-02-18 15:09:00,203 INFO hive.metastore: [main]: Trying to connect to
> metastore with URI thrift://host-10-17-81-201.coe.cloudera.com:9083
> 2016-02-18 15:14:00,304 WARN hive.metastore: [main]: Failed to connect to
> the MetaStore Server...
> 2016-02-18 15:14:00,304 INFO hive.metastore: [main]: Waiting 1 seconds
> before next connection attempt.
> 2016-02-18 15:14:01,306 INFO org.apache.hive.service.server.HiveServer2:
> [main]: Shutting down HiveServer2
> 2016-02-18 15:14:01,308 INFO org.apache.hive.service.server.HiveServer2:
> [main]: Exception caught when calling stop of HiveServer2 before retrying
> start
> java.lang.NullPointerException
> at
> org.apache.hive.service.server.HiveServer2.stop(HiveServer2.java:283)
> at
> org.apache.hive.service.server.HiveServer2.startHiveServer2(HiveServer2.java:351)
> at
> org.apache.hive.service.server.HiveServer2.access$400(HiveServer2.java:69)
> at
> org.apache.hive.service.server.HiveServer2$StartOptionExecutor.execute(HiveServer2.java:545)
> {code}
> And then none of the functions will be available for use as HS2 does not
> re-register them after HMS is up and ready.
> This is not desired behaviour, we shouldn't allow HS2 to be in a servicing
> state if function list is not ready. Or, maybe instead of initialize the
> function list when HS2 starts, try to load the function list when each Hive
> session is created. Of course we can have a cache of function list somewhere
> for better performance, but we would better decouple it from class Hive.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-13213) make DbLockManger work for non-acid resources
[ https://issues.apache.org/jira/browse/HIVE-13213?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15183549#comment-15183549 ] Eugene Koifman commented on HIVE-13213: --- Because unless you have MVCC architecture in your data so that readers can lock in a snapshot, any modification must block readers. > make DbLockManger work for non-acid resources > - > > Key: HIVE-13213 > URL: https://issues.apache.org/jira/browse/HIVE-13213 > Project: Hive > Issue Type: Bug > Components: Transactions >Affects Versions: 1.0.0 >Reporter: Eugene Koifman >Assignee: Eugene Koifman > Attachments: HIVE-13213.patch > > > for example, > insert into T values(...) > if T is an ACID table we acquire Read lock > but for non-acid table it should acquire Exclusive lock -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-7532) allow disabling direct sql per query with external metastore
[ https://issues.apache.org/jira/browse/HIVE-7532?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15183543#comment-15183543 ] Aihua Xu commented on HIVE-7532: >From my understanding on the code, the setting will be applied on one >metastore connection. Depends on if the following query will get a new >connection or not, such setting will not or will be applied. It's not session >wise setting. If we want a session wise setting, we can just use property >override for that session, right? Let me know if I misunderstand the usage. > allow disabling direct sql per query with external metastore > > > Key: HIVE-7532 > URL: https://issues.apache.org/jira/browse/HIVE-7532 > Project: Hive > Issue Type: Improvement >Reporter: Sergey Shelukhin >Assignee: Navis > Fix For: 0.14.0 > > Attachments: HIVE-7532.1.patch.txt, HIVE-7532.2.nogen, > HIVE-7532.2.patch.txt, HIVE-7532.3.patch.txt, HIVE-7532.4.patch.txt, > HIVE-7532.5.patch.txt, HIVE-7532.6.patch.txt > > > Currently with external metastore, direct sql can only be disabled via > metastore config globally. Perhaps it makes sense to have the ability to > propagate the setting per query from client to override the metastore > setting, e.g. if one particular query causes it to fail. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13210) Revert changes in HIVE-12994 related to metastore
[
https://issues.apache.org/jira/browse/HIVE-13210?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15183516#comment-15183516
]
Hive QA commented on HIVE-13210:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12791556/HIVE-13210.patch
{color:green}SUCCESS:{color} +1 due to 4 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 34 failed/errored test(s), 9755 tests
executed
*Failed tests:*
{noformat}
TestMiniTezCliDriver-auto_join30.q-vector_data_types.q-filter_join_breaktask.q-and-12-more
- did not produce a TEST-*.xml file
TestSparkCliDriver-groupby3_map.q-sample2.q-auto_join14.q-and-12-more - did not
produce a TEST-*.xml file
TestSparkCliDriver-groupby_map_ppr_multi_distinct.q-table_access_keys_stats.q-groupby4_noskew.q-and-12-more
- did not produce a TEST-*.xml file
TestSparkCliDriver-join_rc.q-insert1.q-vectorized_rcfile_columnar.q-and-12-more
- did not produce a TEST-*.xml file
TestSparkCliDriver-ppd_join4.q-join9.q-ppd_join3.q-and-12-more - did not
produce a TEST-*.xml file
TestSparkCliDriver-timestamp_lazy.q-bucketsortoptimize_insert_4.q-date_udf.q-and-12-more
- did not produce a TEST-*.xml file
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_alter_numbuckets_partitioned_table2_h23
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_alter_numbuckets_partitioned_table_h23
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_alter_partition_clusterby_sortby
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_alter_table_not_sorted
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_authorization_index
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_dynpart_sort_opt_vectorization
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_dynpart_sort_optimization
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_index_skewtable
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_infer_bucket_sort
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_infer_bucket_sort_convert_join
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_infer_bucket_sort_grouping_operators
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_infer_bucket_sort_multi_insert
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_orc_analyze
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_partition_coltype_literals
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_show_create_table_alter
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_bucket5
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_infer_bucket_sort_bucketed_table
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_dynpart_sort_opt_vectorization
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_dynpart_sort_optimization
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_orc_analyze
org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_bucket5
org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_infer_bucket_sort_bucketed_table
org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_infer_bucket_sort_dyn_part
org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_infer_bucket_sort_map_operators
org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_infer_bucket_sort_merge
org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_infer_bucket_sort_reducers_power_two
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_authorization_uri_import
org.apache.hive.jdbc.TestSSL.testSSLVersion
{noformat}
Test results:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/7185/testReport
Console output:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/7185/console
Test logs:
http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-7185/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 34 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12791556 - PreCommit-HIVE-TRUNK-Build
> Revert changes in HIVE-12994 related to metastore
> -
>
> Key: HIVE-13210
> URL: https://issues.apache.org/jira/browse/HIVE-13210
> Project: Hive
> Issue Type: Bug
>Affects Versions: 2.1.0
>Reporter: Jesus Camacho Rodriguez
>Assignee: Jesus Camacho Rodriguez
> Attachments: HIVE-13210.patch
>
>
> As we do not control what is written in the physical layer and thus we cannot
> ensure NULLS ORDER (and even if we did, currently we do not take advantage of
> it), it seems exposing the NULLS
[jira] [Commented] (HIVE-13213) make DbLockManger work for non-acid resources
[ https://issues.apache.org/jira/browse/HIVE-13213?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15183506#comment-15183506 ] Alan Gates commented on HIVE-13213: --- Why is it that insert should get an exclusive lock for non-acid tables? Two inserts still can't interfere with each other in any way. Insert overwrite should certainly get an xlock, but I believe it already does. > make DbLockManger work for non-acid resources > - > > Key: HIVE-13213 > URL: https://issues.apache.org/jira/browse/HIVE-13213 > Project: Hive > Issue Type: Bug > Components: Transactions >Affects Versions: 1.0.0 >Reporter: Eugene Koifman >Assignee: Eugene Koifman > Attachments: HIVE-13213.patch > > > for example, > insert into T values(...) > if T is an ACID table we acquire Read lock > but for non-acid table it should acquire Exclusive lock -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13171) Add unit test for hs2 webui
[ https://issues.apache.org/jira/browse/HIVE-13171?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Szehon Ho updated HIVE-13171: - Resolution: Fixed Status: Resolved (was: Patch Available) Committed to master, thanks Aihua for review! > Add unit test for hs2 webui > --- > > Key: HIVE-13171 > URL: https://issues.apache.org/jira/browse/HIVE-13171 > Project: Hive > Issue Type: Sub-task > Components: HiveServer2 >Reporter: Szehon Ho >Assignee: Szehon Ho > Attachments: HIVE-13171.2.patch, HIVE-13171.3.patch, HIVE-13171.patch > > > With more complex changes going into webui, it is hard to manually verify all > the kinds of cases. > With HIVE-12952, HS2 webui now uses jamon, which should be more unit-testable > than plain old jsp. We can perhaps add unit test for the jamon servlets, or > test the new OperationDisplay classes queried by the servlets. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13156) Allow specifying the name of the queue in which llap will run
[ https://issues.apache.org/jira/browse/HIVE-13156?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Siddharth Seth updated HIVE-13156: -- Resolution: Fixed Fix Version/s: 2.1.0 Status: Resolved (was: Patch Available) > Allow specifying the name of the queue in which llap will run > - > > Key: HIVE-13156 > URL: https://issues.apache.org/jira/browse/HIVE-13156 > Project: Hive > Issue Type: Improvement >Reporter: Siddharth Seth >Assignee: Siddharth Seth > Fix For: 2.1.0 > > Attachments: HIVE-13156.1.patch, HIVE-13156.2.patch, HIVE-13156.3.txt > > > llap service driver should accept a parameter for the llap queue name. > cc [~gopalv] -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13156) Allow specifying the name of the queue in which llap will run
[ https://issues.apache.org/jira/browse/HIVE-13156?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15183461#comment-15183461 ] Siddharth Seth commented on HIVE-13156: --- The test failures are not related. Committing latest patch. > Allow specifying the name of the queue in which llap will run > - > > Key: HIVE-13156 > URL: https://issues.apache.org/jira/browse/HIVE-13156 > Project: Hive > Issue Type: Improvement >Reporter: Siddharth Seth >Assignee: Siddharth Seth > Attachments: HIVE-13156.1.patch, HIVE-13156.2.patch, HIVE-13156.3.txt > > > llap service driver should accept a parameter for the llap queue name. > cc [~gopalv] -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13156) Allow specifying the name of the queue in which llap will run
[ https://issues.apache.org/jira/browse/HIVE-13156?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Siddharth Seth updated HIVE-13156: -- Attachment: HIVE-13156.3.txt [~leftylev] - wrapped the description into two lines, to keep it within the 100 character per line limit. I don't want to change formatting all over the file. I don't even think there's a defined standard on this indentation. > Allow specifying the name of the queue in which llap will run > - > > Key: HIVE-13156 > URL: https://issues.apache.org/jira/browse/HIVE-13156 > Project: Hive > Issue Type: Improvement >Reporter: Siddharth Seth >Assignee: Siddharth Seth > Attachments: HIVE-13156.1.patch, HIVE-13156.2.patch, HIVE-13156.3.txt > > > llap service driver should accept a parameter for the llap queue name. > cc [~gopalv] -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13217) Replication for HoS mapjoin small file needs to respect dfs.replication.max
[ https://issues.apache.org/jira/browse/HIVE-13217?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Szehon Ho updated HIVE-13217: - Assignee: (was: Xuefu Zhang) > Replication for HoS mapjoin small file needs to respect dfs.replication.max > --- > > Key: HIVE-13217 > URL: https://issues.apache.org/jira/browse/HIVE-13217 > Project: Hive > Issue Type: Bug > Components: Spark >Affects Versions: 1.2.1, 2.0.0 >Reporter: Szehon Ho >Priority: Minor > > Currently Hive on Spark Mapjoin replicates small table file to a hard-coded > value of 10. See SparkHashTableSinkOperator.MIN_REPLICATION. > When dfs.replication.max is less than 10, HoS query fails. This constant > should cap at dfs.replication.max. > Normally dfs.replication.max seems set at 512. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-7532) allow disabling direct sql per query with external metastore
[ https://issues.apache.org/jira/browse/HIVE-7532?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15183443#comment-15183443 ] Sergey Shelukhin commented on HIVE-7532: I don't recall, I think this JIRA is supposed to basically propagate certain 'set' settings to metastore. So it would enable/disable it for the queries (session, I guess?) on metastore side. > allow disabling direct sql per query with external metastore > > > Key: HIVE-7532 > URL: https://issues.apache.org/jira/browse/HIVE-7532 > Project: Hive > Issue Type: Improvement >Reporter: Sergey Shelukhin >Assignee: Navis > Fix For: 0.14.0 > > Attachments: HIVE-7532.1.patch.txt, HIVE-7532.2.nogen, > HIVE-7532.2.patch.txt, HIVE-7532.3.patch.txt, HIVE-7532.4.patch.txt, > HIVE-7532.5.patch.txt, HIVE-7532.6.patch.txt > > > Currently with external metastore, direct sql can only be disabled via > metastore config globally. Perhaps it makes sense to have the ability to > propagate the setting per query from client to override the metastore > setting, e.g. if one particular query causes it to fail. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13160) HS2 unable to load UDFs on startup when HMS is not ready
[
https://issues.apache.org/jira/browse/HIVE-13160?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15183441#comment-15183441
]
Sergey Shelukhin commented on HIVE-13160:
-
Well, backward compat is also important for HS2 (and cli)
> HS2 unable to load UDFs on startup when HMS is not ready
>
>
> Key: HIVE-13160
> URL: https://issues.apache.org/jira/browse/HIVE-13160
> Project: Hive
> Issue Type: Bug
> Components: HiveServer2
>Affects Versions: 1.2.1
>Reporter: Eric Lin
>Assignee: Aihua Xu
> Fix For: 2.1.0
>
> Attachments: HIVE-13160.1.patch, HIVE-13160.2.patch
>
>
> The error looks like this:
> {code}
> 2016-02-18 14:43:54,251 INFO hive.metastore: [main]: Trying to connect to
> metastore with URI thrift://host-10-17-81-201.coe.cloudera.com:9083
> 2016-02-18 14:48:54,692 WARN hive.metastore: [main]: Failed to connect to
> the MetaStore Server...
> 2016-02-18 14:48:54,692 INFO hive.metastore: [main]: Waiting 1 seconds
> before next connection attempt.
> 2016-02-18 14:48:55,692 INFO hive.metastore: [main]: Trying to connect to
> metastore with URI thrift://host-10-17-81-201.coe.cloudera.com:9083
> 2016-02-18 14:53:55,800 WARN hive.metastore: [main]: Failed to connect to
> the MetaStore Server...
> 2016-02-18 14:53:55,800 INFO hive.metastore: [main]: Waiting 1 seconds
> before next connection attempt.
> 2016-02-18 14:53:56,801 INFO hive.metastore: [main]: Trying to connect to
> metastore with URI thrift://host-10-17-81-201.coe.cloudera.com:9083
> 2016-02-18 14:58:56,967 WARN hive.metastore: [main]: Failed to connect to
> the MetaStore Server...
> 2016-02-18 14:58:56,967 INFO hive.metastore: [main]: Waiting 1 seconds
> before next connection attempt.
> 2016-02-18 14:58:57,994 WARN hive.ql.metadata.Hive: [main]: Failed to
> register all functions.
> java.lang.RuntimeException: Unable to instantiate
> org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient
> at
> org.apache.hadoop.hive.metastore.MetaStoreUtils.newInstance(MetaStoreUtils.java:1492)
> at
> org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.(RetryingMetaStoreClient.java:64)
> at
> org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.getProxy(RetryingMetaStoreClient.java:74)
> at
> org.apache.hadoop.hive.ql.metadata.Hive.createMetaStoreClient(Hive.java:2915)
> ...
> 016-02-18 14:58:57,997 INFO hive.metastore: [main]: Trying to connect to
> metastore with URI thrift://host-10-17-81-201.coe.cloudera.com:9083
> 2016-02-18 15:03:58,094 WARN hive.metastore: [main]: Failed to connect to
> the MetaStore Server...
> 2016-02-18 15:03:58,095 INFO hive.metastore: [main]: Waiting 1 seconds
> before next connection attempt.
> 2016-02-18 15:03:59,095 INFO hive.metastore: [main]: Trying to connect to
> metastore with URI thrift://host-10-17-81-201.coe.cloudera.com:9083
> 2016-02-18 15:08:59,203 WARN hive.metastore: [main]: Failed to connect to
> the MetaStore Server...
> 2016-02-18 15:08:59,203 INFO hive.metastore: [main]: Waiting 1 seconds
> before next connection attempt.
> 2016-02-18 15:09:00,203 INFO hive.metastore: [main]: Trying to connect to
> metastore with URI thrift://host-10-17-81-201.coe.cloudera.com:9083
> 2016-02-18 15:14:00,304 WARN hive.metastore: [main]: Failed to connect to
> the MetaStore Server...
> 2016-02-18 15:14:00,304 INFO hive.metastore: [main]: Waiting 1 seconds
> before next connection attempt.
> 2016-02-18 15:14:01,306 INFO org.apache.hive.service.server.HiveServer2:
> [main]: Shutting down HiveServer2
> 2016-02-18 15:14:01,308 INFO org.apache.hive.service.server.HiveServer2:
> [main]: Exception caught when calling stop of HiveServer2 before retrying
> start
> java.lang.NullPointerException
> at
> org.apache.hive.service.server.HiveServer2.stop(HiveServer2.java:283)
> at
> org.apache.hive.service.server.HiveServer2.startHiveServer2(HiveServer2.java:351)
> at
> org.apache.hive.service.server.HiveServer2.access$400(HiveServer2.java:69)
> at
> org.apache.hive.service.server.HiveServer2$StartOptionExecutor.execute(HiveServer2.java:545)
> {code}
> And then none of the functions will be available for use as HS2 does not
> re-register them after HMS is up and ready.
> This is not desired behaviour, we shouldn't allow HS2 to be in a servicing
> state if function list is not ready. Or, maybe instead of initialize the
> function list when HS2 starts, try to load the function list when each Hive
> session is created. Of course we can have a cache of function list somewhere
> for better performance, but we would better decouple it from class Hive.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-12270) Add DBTokenStore support to HS2 delegation token
[ https://issues.apache.org/jira/browse/HIVE-12270?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15183408#comment-15183408 ] Szehon Ho commented on HIVE-12270: -- +1 > Add DBTokenStore support to HS2 delegation token > > > Key: HIVE-12270 > URL: https://issues.apache.org/jira/browse/HIVE-12270 > Project: Hive > Issue Type: New Feature >Reporter: Chaoyu Tang >Assignee: Chaoyu Tang > Attachments: HIVE-12270.1.nothrift.patch, HIVE-12270.1.patch, > HIVE-12270.2.patch, HIVE-12270.3.nothrift.patch, HIVE-12270.3.patch, > HIVE-12270.nothrift.patch > > > DBTokenStore was initially introduced by HIVE-3255 in Hive-0.12 and it is > mainly for HMS delegation token. Later in Hive-0.13, the HS2 delegation token > support was introduced by HIVE-5155 but it used MemoryTokenStore as token > store. That the HIVE-9622 uses the shared RawStore (or HMSHandler) to access > the token/keys information in HMS DB directly from HS2 seems not the right > approach to support DBTokenStore in HS2. I think we should use > HiveMetaStoreClient in HS2 instead. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13206) Create a test-sources.jar when -Psources profile is invoked
[ https://issues.apache.org/jira/browse/HIVE-13206?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15183396#comment-15183396 ] Szehon Ho commented on HIVE-13206: -- Goal makes sense to me.. just quick question, would it make more sense to target specific module(s) for test-jar? I guess this will do it for all modules, even those without tests? > Create a test-sources.jar when -Psources profile is invoked > --- > > Key: HIVE-13206 > URL: https://issues.apache.org/jira/browse/HIVE-13206 > Project: Hive > Issue Type: Improvement >Affects Versions: 2.0.0 >Reporter: Sergio Peña >Assignee: Sergio Peña >Priority: Minor > Attachments: HIVE-13206.1.patch > > > It'd be nice to attach a test-sources jar alongside the others as part of the > build, to provide test resources. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-10485) Create md5 UDF
[
https://issues.apache.org/jira/browse/HIVE-10485?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15183384#comment-15183384
]
Duryea Lee commented on HIVE-10485:
---
I am getting the same error that Russell Pierce received October 2015. What
should I be doing that I am not.
> Create md5 UDF
> --
>
> Key: HIVE-10485
> URL: https://issues.apache.org/jira/browse/HIVE-10485
> Project: Hive
> Issue Type: Task
> Components: UDF
>Reporter: Alexander Pivovarov
>Assignee: Alexander Pivovarov
> Fix For: 1.3.0, 2.0.0
>
> Attachments: HIVE-10485.1.patch, HIVE-10485.2.patch,
> HIVE-10485.3.patch
>
>
> MD5(str)
> Calculates an MD5 128-bit checksum for the string. The value is returned as a
> string of 32 hex digits, or NULL if the argument was NULL. The return value
> can, for example, be used as a hash key.
> Example:
> {code}
> SELECT MD5('udf_md5');
> 'ce62ef0d2d27dc37b6d488b92f4b24fd'
> {code}
> online md5 generator: http://www.md5.cz/
> MySQL has md5 function:
> https://dev.mysql.com/doc/refman/5.5/en/encryption-functions.html#function_md5
> PostgreSQL also has md5 function:
> http://www.postgresql.org/docs/9.1/static/functions-string.html
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-13213) make DbLockManger work for non-acid resources
[ https://issues.apache.org/jira/browse/HIVE-13213?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15183361#comment-15183361 ] Eugene Koifman commented on HIVE-13213: --- [~alangates] could you review please > make DbLockManger work for non-acid resources > - > > Key: HIVE-13213 > URL: https://issues.apache.org/jira/browse/HIVE-13213 > Project: Hive > Issue Type: Bug > Components: Transactions >Affects Versions: 1.0.0 >Reporter: Eugene Koifman >Assignee: Eugene Koifman > Attachments: HIVE-13213.patch > > > for example, > insert into T values(...) > if T is an ACID table we acquire Read lock > but for non-acid table it should acquire Exclusive lock -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13201) Compaction shouldn't be allowed on non-ACID table
[ https://issues.apache.org/jira/browse/HIVE-13201?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15183336#comment-15183336 ] Eugene Koifman commented on HIVE-13201: --- Seems to me like a test bug... > Compaction shouldn't be allowed on non-ACID table > - > > Key: HIVE-13201 > URL: https://issues.apache.org/jira/browse/HIVE-13201 > Project: Hive > Issue Type: Bug > Components: Transactions >Affects Versions: 2.0.0 >Reporter: Wei Zheng >Assignee: Wei Zheng > Attachments: HIVE-13201.1.patch > > > Looks like compaction is allowed on non-ACID table, although that's of no > sense and does nothing. Moreover the compaction request will be enqueued into > COMPACTION_QUEUE metastore table, which brings unnecessary overhead. > We should prevent compaction commands being allowed on non-ACID tables. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13209) metastore get_delegation_token fails with null ip address
[
https://issues.apache.org/jira/browse/HIVE-13209?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Thejas M Nair updated HIVE-13209:
-
Resolution: Fixed
Status: Resolved (was: Patch Available)
Patch committed to master. Thanks for the review [~sushanth]!
> metastore get_delegation_token fails with null ip address
> -
>
> Key: HIVE-13209
> URL: https://issues.apache.org/jira/browse/HIVE-13209
> Project: Hive
> Issue Type: Bug
> Components: Metastore
>Affects Versions: 2.1.0
>Reporter: Aswathy Chellammal Sreekumar
>Assignee: Thejas M Nair
> Fix For: 2.1.0
>
> Attachments: HIVE-13209.1.patch, HIVE-13209.2.patch
>
>
> After changes in HIVE-13169, metastore get_delegation_token fails with null
> ip address.
> {code}
> 2016-03-03 07:45:31,055 ERROR [pool-6-thread-22]:
> metastore.RetryingHMSHandler (RetryingHMSHandler.java:invoke(159)) -
> MetaException(message:Unauthorized connection for super-user:
> HTTP/ from IP null)
> at
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.get_delegation_token(HiveMetaStore.java:5290)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at
> org.apache.hadoop.hive.metastore.RetryingHMSHandler.invoke(RetryingHMSHandler.java:107)
> at com.sun.proxy.$Proxy16.get_delegation_token(Unknown Source)
> at
> org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$get_delegation_token.getResult(ThriftHiveMetastore.java:11492)
> at
> org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$get_delegation_token.getResult(ThriftHiveMetastore.java:11476)
> at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
> at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39)
> at
> org.apache.hadoop.hive.thrift.HadoopThriftAuthBridge$Server$TUGIAssumingProcessor$1.run(HadoopThriftAuthBridge.java:551)
> at
> org.apache.hadoop.hive.thrift.HadoopThriftAuthBridge$Server$TUGIAssumingProcessor$1.run(HadoopThriftAuthBridge.java:546)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:415)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)
> at
> org.apache.hadoop.hive.thrift.HadoopThriftAuthBridge$Server$TUGIAssumingProcessor.process(HadoopThriftAuthBridge.java:546)
> at
> org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:285)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:745)
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
