[jira] [Updated] (HIVE-14804) HPLSQL multiple db connection does not switch back to Hive
[ https://issues.apache.org/jira/browse/HIVE-14804?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Fei Hui updated HIVE-14804: --- Attachment: HIVE-14804.2-branch-2.1.patch HIVE-14804.2-branch-2.0.patch update patch remove 'No newline at end of file' > HPLSQL multiple db connection does not switch back to Hive > -- > > Key: HIVE-14804 > URL: https://issues.apache.org/jira/browse/HIVE-14804 > Project: Hive > Issue Type: Bug > Components: hpl/sql >Reporter: Dmitry Kozlov >Assignee: Fei Hui >Priority: Blocker > Attachments: HIVE-14804.1-branch-2.0.patch, > HIVE-14804.1-branch-2.1.patch, HIVE-14804.2-branch-2.0.patch, > HIVE-14804.2-branch-2.1.patch > > > I have a problem with a multi database connection. I have 3 environments that > I would like to connect in my HPLSQL code Hive, DB2 and MySql. As soon as I > map any table either from DB2 or MySQL my code stops to recognize Hive > tables. Actually it starts to think that it is a table from the same database > (DB2 or MySql) that was mapped the last. It means your example > http://www.hplsql.org/map-object works only one way from Hive to MySQL and it > is not possible to go back to Hive. > Here is a simple piece of code. > declare cnt int; > begin > /* > PRINT 'Start MySQL'; > MAP OBJECT tbls TO hive.TBLS AT mysqlconn; > select count(*) > into cnt > from tbls; > PRINT cnt; > PRINT 'Start Db2'; > MAP OBJECT exch TO DBDEV2.TEST_EXCHANGE AT db2conn; > select count(1) > into cnt > from exch; > PRINT cnt;*/ > PRINT 'Check Hive'; > SELECT count(1) > into cnt > FROM dev.test_sqoop; > PRINT cnt; > end; > It has three blocks. One select from MySQL, second from DB2 and third from > Hive ORC table. > When first two blocks are commented then block 3 works. See below > Check Hive > 16/09/20 18:08:08 INFO jdbc.Utils: Supplied authorities: localhost:1 > 16/09/20 18:08:08 INFO jdbc.Utils: Resolved authority: localhost:1 > 16/09/20 18:08:08 INFO jdbc.HiveConnection: Will try to open client transport > with JDBC Uri: jdbc:hive2://localhost:1 > Open connection: jdbc:hive2://localhost:1 (497 ms) > Starting query > Query executed successfully (177 ms) > 82 > When I try to uncomment any of those blocks then block 3 stops working. For > example, if I uncomment block 1 I get this output. It is now assumes that > dev.test_sqoop is a MySQL table. Contrarily to your example > Start MySQL > Open connection: jdbc:mysql://10.11.12.144:3306/hive (489 ms) > Starting query > Query executed successfully (4 ms) > 539 > Check Hive > Starting query > Unhandled exception in HPL/SQL > com.mysql.jdbc.exceptions.jdbc4.MySQLSyntaxErrorException: Table > 'dev.test_sqoop' doesn't exist > If I comment the second block then it starts to assume that dev.test_sqoop is > a DB2 table. See below. So switch between DB2 and MySQL is working, however, > the hive table is still not working > Start MySQL > Open connection: jdbc:mysql://10.11.12.144:3306/hive (485 ms) > Starting query > Query executed successfully (5 ms) > 539 > Start Db2 > Open connection: jdbc:db2://10.11.12.141:5/WM (227 ms) > Starting query > Query executed successfully (48 ms) > 0 > Check Hive > Starting query > Unhandled exception in HPL/SQL > com.ibm.db2.jcc.am.SqlSyntaxErrorException: DB2 SQL Error: SQLCODE=-204, > SQLSTATE=42704, SQLERRMC=DEV.TEST_SQOOP, DRIVER=4.16.53 > Could you, please, provide your feedback on this finding. In addition, I > would like to check if it would be possible to insert into a DB2 table > records that were selected from a Hive with one statement as soon as DB2 > table is properly mapped. Please, explain. > Looking forward to hearing from you soon. > Regards, > Dmitry Kozlov > Daisy Intelligence -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-15314) ThriftCLIService should LOG.error rather than LOG.warn when Exception occurs
[ https://issues.apache.org/jira/browse/HIVE-15314?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Fei Hui updated HIVE-15314: --- Component/s: Logging > ThriftCLIService should LOG.error rather than LOG.warn when Exception occurs > > > Key: HIVE-15314 > URL: https://issues.apache.org/jira/browse/HIVE-15314 > Project: Hive > Issue Type: Bug > Components: CLI, Logging >Affects Versions: 2.2.0 >Reporter: Fei Hui >Assignee: Fei Hui > Attachments: HIVE-15314.1.patch > > > When catch exception, critical error occurs, > and the message in log is ''Error executing statement", "Error getting type > info", etc. > So we should use LOG.error, and it will remind users. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-15314) ThriftCLIService should LOG.error rather than LOG.warn when Exception occurs
[ https://issues.apache.org/jira/browse/HIVE-15314?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15707969#comment-15707969 ] Fei Hui commented on HIVE-15314: hi [~prasanth_j] This is an logging problem, i see you are lead of logging. could you please give suggestions ? > ThriftCLIService should LOG.error rather than LOG.warn when Exception occurs > > > Key: HIVE-15314 > URL: https://issues.apache.org/jira/browse/HIVE-15314 > Project: Hive > Issue Type: Bug > Components: CLI, Logging >Affects Versions: 2.2.0 >Reporter: Fei Hui >Assignee: Fei Hui > Attachments: HIVE-15314.1.patch > > > When catch exception, critical error occurs, > and the message in log is ''Error executing statement", "Error getting type > info", etc. > So we should use LOG.error, and it will remind users. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-15314) ThriftCLIService should LOG.error rather than LOG.warn when Exception occurs
[ https://issues.apache.org/jira/browse/HIVE-15314?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15707991#comment-15707991 ] Prasanth Jayachandran commented on HIVE-15314: -- I guess the intention here is to log and set error status in thrift response. If my understanding is correct, these exceptions are logged just for debugging purpose and are usually handled/acted upon by the clients. In case of error response, the client will throw HiveException that will contain the stacktrace received from ThriftCLIService. [~thejas] I guess that was your initial intention, right? > ThriftCLIService should LOG.error rather than LOG.warn when Exception occurs > > > Key: HIVE-15314 > URL: https://issues.apache.org/jira/browse/HIVE-15314 > Project: Hive > Issue Type: Bug > Components: CLI, Logging >Affects Versions: 2.2.0 >Reporter: Fei Hui >Assignee: Fei Hui > Attachments: HIVE-15314.1.patch > > > When catch exception, critical error occurs, > and the message in log is ''Error executing statement", "Error getting type > info", etc. > So we should use LOG.error, and it will remind users. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-15314) ThriftCLIService should LOG.error rather than LOG.warn when Exception occurs
[ https://issues.apache.org/jira/browse/HIVE-15314?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15708041#comment-15708041 ] Fei Hui commented on HIVE-15314: when queries failed, i checked the log /tmp/hadoop/hive.log and found that 2016-11-30T11:52:49,401 WARN [909b45b3-deed-4a39-a54d-864e97b057b3909b45b3-deed-4a39-a54d-864e97b057b3b6b20a18-cef5-4b3b-ae09-65ab5f8c880eb6b20a18-cef5-4b3b-ae09-65ab5f8c880ec671ec17-c7a4-4811-a9f8-68bb4bbcb9a5c671ec17-c7a4-4811-a9f8-68bb4bbcb9a555adbcc6-c1fa-4646-9c83-92a1f3c21b4155adbcc6-c1fa-4646-9c83-92a1f3c21b41HiveServer2-Handler-Pool: Thread-42]: thrift.ThriftCLIService (ThriftCLIService.java:ExecuteStatement(515)) - Error executing statement: Here is 'WARN', but messages contains 'Error executing statement'. So i'm confused. If it's 'ERROR',maybe i understand it's serious error That's point thanks [~prasanth_j]'s reply > ThriftCLIService should LOG.error rather than LOG.warn when Exception occurs > > > Key: HIVE-15314 > URL: https://issues.apache.org/jira/browse/HIVE-15314 > Project: Hive > Issue Type: Bug > Components: CLI, Logging >Affects Versions: 2.2.0 >Reporter: Fei Hui >Assignee: Fei Hui > Attachments: HIVE-15314.1.patch > > > When catch exception, critical error occurs, > and the message in log is ''Error executing statement", "Error getting type > info", etc. > So we should use LOG.error, and it will remind users. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-14804) HPLSQL multiple db connection does not switch back to Hive
[
https://issues.apache.org/jira/browse/HIVE-14804?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15708084#comment-15708084
]
Hive QA commented on HIVE-14804:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12841001/HIVE-14804.1-branch-2.1.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 10 failed/errored test(s), 10474 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_acid_globallimit
(batchId=51)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_list_bucket_dml_12
(batchId=87)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_order_null (batchId=35)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_stats_list_bucket
(batchId=118)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_union_fast_stats
(batchId=89)
org.apache.hive.beeline.TestBeelineArgParsing.testAddLocalJarWithoutAddDriverClazz[0]
(batchId=286)
org.apache.hive.beeline.TestBeelineArgParsing.testAddLocalJar[0] (batchId=286)
org.apache.hive.beeline.TestBeelineArgParsing.testAddLocalJar[1] (batchId=286)
org.apache.hive.hcatalog.hbase.TestPigHBaseStorageHandler.org.apache.hive.hcatalog.hbase.TestPigHBaseStorageHandler
(batchId=514)
org.apache.hive.jdbc.TestJdbcWithMiniLlap.testLlapInputFormatEndToEnd
(batchId=487)
{noformat}
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/2339/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/2339/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-2339/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 10 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12841001 - PreCommit-HIVE-Build
> HPLSQL multiple db connection does not switch back to Hive
> --
>
> Key: HIVE-14804
> URL: https://issues.apache.org/jira/browse/HIVE-14804
> Project: Hive
> Issue Type: Bug
> Components: hpl/sql
>Reporter: Dmitry Kozlov
>Assignee: Fei Hui
>Priority: Blocker
> Attachments: HIVE-14804.1-branch-2.0.patch,
> HIVE-14804.1-branch-2.1.patch, HIVE-14804.2-branch-2.0.patch,
> HIVE-14804.2-branch-2.1.patch
>
>
> I have a problem with a multi database connection. I have 3 environments that
> I would like to connect in my HPLSQL code Hive, DB2 and MySql. As soon as I
> map any table either from DB2 or MySQL my code stops to recognize Hive
> tables. Actually it starts to think that it is a table from the same database
> (DB2 or MySql) that was mapped the last. It means your example
> http://www.hplsql.org/map-object works only one way from Hive to MySQL and it
> is not possible to go back to Hive.
> Here is a simple piece of code.
> declare cnt int;
> begin
> /*
> PRINT 'Start MySQL';
> MAP OBJECT tbls TO hive.TBLS AT mysqlconn;
> select count(*)
> into cnt
> from tbls;
> PRINT cnt;
> PRINT 'Start Db2';
> MAP OBJECT exch TO DBDEV2.TEST_EXCHANGE AT db2conn;
> select count(1)
> into cnt
> from exch;
> PRINT cnt;*/
> PRINT 'Check Hive';
> SELECT count(1)
> into cnt
> FROM dev.test_sqoop;
> PRINT cnt;
> end;
> It has three blocks. One select from MySQL, second from DB2 and third from
> Hive ORC table.
> When first two blocks are commented then block 3 works. See below
> Check Hive
> 16/09/20 18:08:08 INFO jdbc.Utils: Supplied authorities: localhost:1
> 16/09/20 18:08:08 INFO jdbc.Utils: Resolved authority: localhost:1
> 16/09/20 18:08:08 INFO jdbc.HiveConnection: Will try to open client transport
> with JDBC Uri: jdbc:hive2://localhost:1
> Open connection: jdbc:hive2://localhost:1 (497 ms)
> Starting query
> Query executed successfully (177 ms)
> 82
> When I try to uncomment any of those blocks then block 3 stops working. For
> example, if I uncomment block 1 I get this output. It is now assumes that
> dev.test_sqoop is a MySQL table. Contrarily to your example
> Start MySQL
> Open connection: jdbc:mysql://10.11.12.144:3306/hive (489 ms)
> Starting query
> Query executed successfully (4 ms)
> 539
> Check Hive
> Starting query
> Unhandled exception in HPL/SQL
> com.mysql.jdbc.exceptions.jdbc4.MySQLSyntaxErrorException: Table
> 'dev.test_sqoop' doesn't exist
> If I comment the second block then it starts to assume that dev.test_sqoop is
> a DB2 table. See below. So switch between DB2 and MySQL is working, however,
> the hive table is still not working
> Start MySQL
> Open connection: jdbc:mysql://10.11.12.144:3306/hive (485 ms)
> Starting query
> Query e
[jira] [Commented] (HIVE-15276) CLIs spell "substitution" as "subsitution" and "auxiliary" as "auxillary"
[ https://issues.apache.org/jira/browse/HIVE-15276?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15708108#comment-15708108 ] Lefty Leverenz commented on HIVE-15276: --- Thanks for the "Hive" fixes. Although this doesn't really need to be documented in the wiki, it would be nice to update the output of "hive -H" from 0.9.0 (with misspelling) to 2.2.0: * [Hive CLI -- Hive Command Line Options | https://cwiki.apache.org/confluence/display/Hive/LanguageManual+Cli#LanguageManualCli-HiveCommandLineOptions] > CLIs spell "substitution" as "subsitution" and "auxiliary" as "auxillary" > - > > Key: HIVE-15276 > URL: https://issues.apache.org/jira/browse/HIVE-15276 > Project: Hive > Issue Type: Bug > Components: CLI >Affects Versions: 1.1.0 >Reporter: Grant Sohn >Assignee: Grant Sohn >Priority: Trivial > Fix For: 2.2.0 > > Attachments: HIVE-15276.1.patch, HIVE-15276.2.patch, > HIVE-15276.3.patch, HIVE-15276.4.patch, HIVE-15276.5.patch > > > Found some obvious spelling typos in the CLI help. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-14804) HPLSQL multiple db connection does not switch back to Hive
[ https://issues.apache.org/jira/browse/HIVE-14804?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15708109#comment-15708109 ] Fei Hui commented on HIVE-14804: hi [~dmtolpeko] upload the patch for branch-2.0 & branch-2.1 to fix this problem could you please give suggestions and review it ? failed cases are not related > HPLSQL multiple db connection does not switch back to Hive > -- > > Key: HIVE-14804 > URL: https://issues.apache.org/jira/browse/HIVE-14804 > Project: Hive > Issue Type: Bug > Components: hpl/sql >Reporter: Dmitry Kozlov >Assignee: Fei Hui >Priority: Blocker > Attachments: HIVE-14804.1-branch-2.0.patch, > HIVE-14804.1-branch-2.1.patch, HIVE-14804.2-branch-2.0.patch, > HIVE-14804.2-branch-2.1.patch > > > I have a problem with a multi database connection. I have 3 environments that > I would like to connect in my HPLSQL code Hive, DB2 and MySql. As soon as I > map any table either from DB2 or MySQL my code stops to recognize Hive > tables. Actually it starts to think that it is a table from the same database > (DB2 or MySql) that was mapped the last. It means your example > http://www.hplsql.org/map-object works only one way from Hive to MySQL and it > is not possible to go back to Hive. > Here is a simple piece of code. > declare cnt int; > begin > /* > PRINT 'Start MySQL'; > MAP OBJECT tbls TO hive.TBLS AT mysqlconn; > select count(*) > into cnt > from tbls; > PRINT cnt; > PRINT 'Start Db2'; > MAP OBJECT exch TO DBDEV2.TEST_EXCHANGE AT db2conn; > select count(1) > into cnt > from exch; > PRINT cnt;*/ > PRINT 'Check Hive'; > SELECT count(1) > into cnt > FROM dev.test_sqoop; > PRINT cnt; > end; > It has three blocks. One select from MySQL, second from DB2 and third from > Hive ORC table. > When first two blocks are commented then block 3 works. See below > Check Hive > 16/09/20 18:08:08 INFO jdbc.Utils: Supplied authorities: localhost:1 > 16/09/20 18:08:08 INFO jdbc.Utils: Resolved authority: localhost:1 > 16/09/20 18:08:08 INFO jdbc.HiveConnection: Will try to open client transport > with JDBC Uri: jdbc:hive2://localhost:1 > Open connection: jdbc:hive2://localhost:1 (497 ms) > Starting query > Query executed successfully (177 ms) > 82 > When I try to uncomment any of those blocks then block 3 stops working. For > example, if I uncomment block 1 I get this output. It is now assumes that > dev.test_sqoop is a MySQL table. Contrarily to your example > Start MySQL > Open connection: jdbc:mysql://10.11.12.144:3306/hive (489 ms) > Starting query > Query executed successfully (4 ms) > 539 > Check Hive > Starting query > Unhandled exception in HPL/SQL > com.mysql.jdbc.exceptions.jdbc4.MySQLSyntaxErrorException: Table > 'dev.test_sqoop' doesn't exist > If I comment the second block then it starts to assume that dev.test_sqoop is > a DB2 table. See below. So switch between DB2 and MySQL is working, however, > the hive table is still not working > Start MySQL > Open connection: jdbc:mysql://10.11.12.144:3306/hive (485 ms) > Starting query > Query executed successfully (5 ms) > 539 > Start Db2 > Open connection: jdbc:db2://10.11.12.141:5/WM (227 ms) > Starting query > Query executed successfully (48 ms) > 0 > Check Hive > Starting query > Unhandled exception in HPL/SQL > com.ibm.db2.jcc.am.SqlSyntaxErrorException: DB2 SQL Error: SQLCODE=-204, > SQLSTATE=42704, SQLERRMC=DEV.TEST_SQOOP, DRIVER=4.16.53 > Could you, please, provide your feedback on this finding. In addition, I > would like to check if it would be possible to insert into a DB2 table > records that were selected from a Hive with one statement as soon as DB2 > table is properly mapped. Please, explain. > Looking forward to hearing from you soon. > Regards, > Dmitry Kozlov > Daisy Intelligence -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-15115) Flaky test: TestMiniLlapLocalCliDriver.testCliDriver[union_fast_stats]
[
https://issues.apache.org/jira/browse/HIVE-15115?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Barna Zsombor Klara updated HIVE-15115:
---
Attachment: HIVE-15115.1.patch
> Flaky test: TestMiniLlapLocalCliDriver.testCliDriver[union_fast_stats]
> --
>
> Key: HIVE-15115
> URL: https://issues.apache.org/jira/browse/HIVE-15115
> Project: Hive
> Issue Type: Sub-task
>Reporter: Barna Zsombor Klara
>Assignee: Barna Zsombor Klara
> Fix For: 2.2.0
>
> Attachments: HIVE-15115.1.patch, HIVE-15115.patch
>
>
> This test was identified as flaky before, it seems it turned flaky again.
> Earlier Jira:
> [HIVE-14976|https://issues.apache.org/jira/browse/HIVE-14976]
> New flaky runs:
> https://builds.apache.org/job/PreCommit-HIVE-Build/1931/testReport
> https://builds.apache.org/job/PreCommit-HIVE-Build/1930/testReport
> {code}
> 516c516
> < totalSize 3220
> ---
> > totalSize 3224
> 569c569
> < totalSize 3220
> ---
> > totalSize 3224
> 634c634
> < totalSize 4577
> ---
> > totalSize 4581
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Resolved] (HIVE-15315) Query compilation: PartitionDesc can reduce the number of table metadata calls
[ https://issues.apache.org/jira/browse/HIVE-15315?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rajesh Balamohan resolved HIVE-15315. - Resolution: Invalid > Query compilation: PartitionDesc can reduce the number of table metadata calls > -- > > Key: HIVE-15315 > URL: https://issues.apache.org/jira/browse/HIVE-15315 > Project: Hive > Issue Type: Improvement > Components: Metastore >Reporter: Rajesh Balamohan >Assignee: Rajesh Balamohan >Priority: Minor > > https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/plan/PartitionDesc.java#L88 > When compiling the query with partition dataset, partitiondesc ends getting > table metadata from metastore lot many times. This can be optimized to reduce > the number of hits to the database. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-14804) HPLSQL multiple db connection does not switch back to Hive
[
https://issues.apache.org/jira/browse/HIVE-14804?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15708299#comment-15708299
]
Hive QA commented on HIVE-14804:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12841006/HIVE-14804.2-branch-2.1.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 10 failed/errored test(s), 10476 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_acid_globallimit
(batchId=51)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_list_bucket_dml_12
(batchId=87)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_list_bucket_dml_13
(batchId=44)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_order_null (batchId=35)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_stats_list_bucket
(batchId=118)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_union_fast_stats
(batchId=89)
org.apache.hive.beeline.TestBeelineArgParsing.testAddLocalJarWithoutAddDriverClazz[0]
(batchId=286)
org.apache.hive.beeline.TestBeelineArgParsing.testAddLocalJar[0] (batchId=286)
org.apache.hive.beeline.TestBeelineArgParsing.testAddLocalJar[1] (batchId=286)
org.apache.hive.jdbc.TestJdbcWithMiniLlap.testLlapInputFormatEndToEnd
(batchId=487)
{noformat}
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/2340/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/2340/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-2340/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 10 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12841006 - PreCommit-HIVE-Build
> HPLSQL multiple db connection does not switch back to Hive
> --
>
> Key: HIVE-14804
> URL: https://issues.apache.org/jira/browse/HIVE-14804
> Project: Hive
> Issue Type: Bug
> Components: hpl/sql
>Reporter: Dmitry Kozlov
>Assignee: Fei Hui
>Priority: Blocker
> Attachments: HIVE-14804.1-branch-2.0.patch,
> HIVE-14804.1-branch-2.1.patch, HIVE-14804.2-branch-2.0.patch,
> HIVE-14804.2-branch-2.1.patch
>
>
> I have a problem with a multi database connection. I have 3 environments that
> I would like to connect in my HPLSQL code Hive, DB2 and MySql. As soon as I
> map any table either from DB2 or MySQL my code stops to recognize Hive
> tables. Actually it starts to think that it is a table from the same database
> (DB2 or MySql) that was mapped the last. It means your example
> http://www.hplsql.org/map-object works only one way from Hive to MySQL and it
> is not possible to go back to Hive.
> Here is a simple piece of code.
> declare cnt int;
> begin
> /*
> PRINT 'Start MySQL';
> MAP OBJECT tbls TO hive.TBLS AT mysqlconn;
> select count(*)
> into cnt
> from tbls;
> PRINT cnt;
> PRINT 'Start Db2';
> MAP OBJECT exch TO DBDEV2.TEST_EXCHANGE AT db2conn;
> select count(1)
> into cnt
> from exch;
> PRINT cnt;*/
> PRINT 'Check Hive';
> SELECT count(1)
> into cnt
> FROM dev.test_sqoop;
> PRINT cnt;
> end;
> It has three blocks. One select from MySQL, second from DB2 and third from
> Hive ORC table.
> When first two blocks are commented then block 3 works. See below
> Check Hive
> 16/09/20 18:08:08 INFO jdbc.Utils: Supplied authorities: localhost:1
> 16/09/20 18:08:08 INFO jdbc.Utils: Resolved authority: localhost:1
> 16/09/20 18:08:08 INFO jdbc.HiveConnection: Will try to open client transport
> with JDBC Uri: jdbc:hive2://localhost:1
> Open connection: jdbc:hive2://localhost:1 (497 ms)
> Starting query
> Query executed successfully (177 ms)
> 82
> When I try to uncomment any of those blocks then block 3 stops working. For
> example, if I uncomment block 1 I get this output. It is now assumes that
> dev.test_sqoop is a MySQL table. Contrarily to your example
> Start MySQL
> Open connection: jdbc:mysql://10.11.12.144:3306/hive (489 ms)
> Starting query
> Query executed successfully (4 ms)
> 539
> Check Hive
> Starting query
> Unhandled exception in HPL/SQL
> com.mysql.jdbc.exceptions.jdbc4.MySQLSyntaxErrorException: Table
> 'dev.test_sqoop' doesn't exist
> If I comment the second block then it starts to assume that dev.test_sqoop is
> a DB2 table. See below. So switch between DB2 and MySQL is working, however,
> the hive table is still not working
> Start MySQL
> Open connection: jdbc:mysql://10.11.12.144:3306/hive (485 ms)
> Starting query
> Query executed successfully (5 ms)
> 539
> Start D
[jira] [Commented] (HIVE-15227) Optimize join + gby into semijoin
[ https://issues.apache.org/jira/browse/HIVE-15227?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15708332#comment-15708332 ] Jesus Camacho Rodriguez commented on HIVE-15227: LGTM, +1 Is there a Calcite issue to make SemiJoinRule use a custom RelBuilder? Could you create it/link it to this issue? We should try to include it in new Calcite release. You might want to run locally _auto_sortmerge_join_2.q_ to double-check that it is not related to the patch. > Optimize join + gby into semijoin > - > > Key: HIVE-15227 > URL: https://issues.apache.org/jira/browse/HIVE-15227 > Project: Hive > Issue Type: Bug > Components: Logical Optimizer >Reporter: Ashutosh Chauhan >Assignee: Ashutosh Chauhan > Attachments: HIVE-15227.2.patch, HIVE-15227.3.patch, > HIVE-15227.4.patch, HIVE-15227.patch > > > Calcite has a rule which can do this transformation. Lets take advantage of > this since Hive has native Left semi join operator. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-15115) Flaky test: TestMiniLlapLocalCliDriver.testCliDriver[union_fast_stats]
[
https://issues.apache.org/jira/browse/HIVE-15115?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15708340#comment-15708340
]
Barna Zsombor Klara commented on HIVE-15115:
I can confirm that the timezones were indeed different (PST and CET) and I
think that the ordering of the rows in the test is not guaranteed either. What
is strange is that it seems the tests are now not failing on Jenkins even
though there was no change to either the q nor the qout file. On my end the OSX
machine with the CET timezone was running the test successfully and I would be
surprised if Jenkins ran the test in that timezone. I will try make the row
ordering deterministic by adding order by clauses and see if I can get the test
to run successfully on the machine with the PST timezone.
> Flaky test: TestMiniLlapLocalCliDriver.testCliDriver[union_fast_stats]
> --
>
> Key: HIVE-15115
> URL: https://issues.apache.org/jira/browse/HIVE-15115
> Project: Hive
> Issue Type: Sub-task
>Reporter: Barna Zsombor Klara
>Assignee: Barna Zsombor Klara
> Fix For: 2.2.0
>
> Attachments: HIVE-15115.1.patch, HIVE-15115.patch
>
>
> This test was identified as flaky before, it seems it turned flaky again.
> Earlier Jira:
> [HIVE-14976|https://issues.apache.org/jira/browse/HIVE-14976]
> New flaky runs:
> https://builds.apache.org/job/PreCommit-HIVE-Build/1931/testReport
> https://builds.apache.org/job/PreCommit-HIVE-Build/1930/testReport
> {code}
> 516c516
> < totalSize 3220
> ---
> > totalSize 3224
> 569c569
> < totalSize 3220
> ---
> > totalSize 3224
> 634c634
> < totalSize 4577
> ---
> > totalSize 4581
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-15316) CTAS STORED AS AVRO: AvroTypeException Found default.record_0, expecting union
[
https://issues.apache.org/jira/browse/HIVE-15316?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
David Maughan updated HIVE-15316:
-
Description:
There's an issue when querying a table that has been created as Avro via CTAS
when the target struct is at least 2 struct-levels deep. It can be replicated
with the following steps:
{code}
CREATE TABLE a
STORED AS AVRO
AS
SELECT named_struct('c', named_struct('d', 1)) as b;
SELECT b FROM a;
org.apache.avro.AvroTypeException: Found default.record_0, expecting union
{code}
The reason for this is that during table creation, the Avro schema is generated
from the Hive columns in {{AvroSerDe}} and then passed through the Avro Schema
Parser: {{new Schema.Parser().parse(schema.toString())}}. For the above
example, this creates the below schema in the Avro file. Note that the lowest
level struct, {{record_0}} has {{"namespace": "default"}}.
{code}
{
"type": "record",
"name": "a",
"namespace": "default",
"fields": [
{
"name": "b",
"type": [
"null",
{
"type": "record",
"name": "record_1",
"namespace": "",
"doc": "struct>",
"fields": [
{
"name": "c",
"type": [
"null",
{
"type": "record",
"name": "record_0",
"namespace": "default",
"doc": "struct",
"fields": [
{
"name": "d",
"type": [ "null", "int" ],
"doc": "int",
"default": null
}
]
}
],
"doc": "struct",
"default": null
}
]
}
],
"default": null
}
]
}
{code}
On a subsequent select query, the Avro schema is again generated from the Hive
columns. However, this time it is not passed through the Avro Schema Parser and
the {{namespace}} attribute is not present in {{record_0}}. The actual Error
message _"Found default.record_0, expecting union"_ is slightly misleading.
Although it is a expecting a union, it is specifically expecting a null or a
record named {{record_0}} but it finds {{default.record_0}}.
I believe this is a bug in Avro. I'm not sure whether correct behaviour is to
cascade the namespace down or not but it is definitely an inconsistency between
creating a schema via the builders and parser. I've created
[AVRO-1965|https://issues.apache.org/jira/browse/AVRO-1965] for this. However,
I believe that defensively passing the schema through the Avro Schema Parser on
a select query would fix this issue in Hive without an Avro fix and version
bump in Hive.
was:
There's an issue when querying a table that has been created as Avro via CTAS
when the target struct is at least 2 struct-levels deep. It can be replicated
with the following steps:
{code}
CREATE TABLE a
STORED AS AVRO
AS
SELECT named_struct('c', named_struct('d', 1)) as b;
SELECT b FROM a;
org.apache.avro.AvroTypeException: Found default.record_0, expecting union
{code}
The reason for this is that during table creation, the Avro schema is generated
from the Hive columns in {{AvroSerDe}} and then passed through the Avro Schema
Parser: {{new Schema.Parser().parse(schema.toString())}}. For the above
example, this creates the below schema in the Avro file. Note that the lowest
level struct, {{record_0}} has {{"namespace": "default"}}.
{code}
{
"type": "record",
"name": "a",
"namespace": "default",
"fields": [
{
"name": "b",
"type": [
"null",
{
"type": "record",
"name": "record_1",
"namespace": "",
"doc": "struct>",
"fields": [
{
"name": "c",
"type": [
"null",
{
"type": "record",
"name": "record_0",
"namespace": "default",
"doc": "struct",
"fields": [
{
"name": "d",
"type": [ "null", "int" ],
"doc": "int",
"default": null
}
]
}
],
"doc": "struct",
"default": null
}
]
}
],
"default": null
}
]
}
{code}
On a subsequent select query, the Avro schema is again generated from the Hive
columns. However, this time it is not passed through the Avro Schema Parser and
the {{namespace}} attribute is not present in {{record_0}}. The actual Error
message _"Found default.record_0, expecting union"_ is slightly misleading.
Although it is a expected a union, it is specifically expect
[jira] [Updated] (HIVE-15316) CTAS STORED AS AVRO: AvroTypeException Found default.record_0, expecting union
[
https://issues.apache.org/jira/browse/HIVE-15316?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
David Maughan updated HIVE-15316:
-
Description:
There's an issue when querying a table that has been created as Avro via CTAS
when the target struct is at least 2 struct-levels deep. It can be replicated
with the following steps:
{code}
CREATE TABLE a
STORED AS AVRO
AS
SELECT named_struct('c', named_struct('d', 1)) as b;
SELECT b FROM a;
org.apache.avro.AvroTypeException: Found default.record_0, expecting union
{code}
The reason for this is that during table creation, the Avro schema is generated
from the Hive columns in {{AvroSerDe}} and then passed through the Avro Schema
Parser: {{new Schema.Parser().parse(schema.toString())}}. For the above
example, this creates the below schema in the Avro file. Note that the lowest
level struct, {{record_0}} has {{"namespace": "default"}}.
{code}
{
"type": "record",
"name": "a",
"namespace": "default",
"fields": [
{
"name": "b",
"type": [
"null",
{
"type": "record",
"name": "record_1",
"namespace": "",
"doc": "struct>",
"fields": [
{
"name": "c",
"type": [
"null",
{
"type": "record",
"name": "record_0",
"namespace": "default",
"doc": "struct",
"fields": [
{
"name": "d",
"type": [ "null", "int" ],
"doc": "int",
"default": null
}
]
}
],
"doc": "struct",
"default": null
}
]
}
],
"default": null
}
]
}
{code}
On a subsequent select query, the Avro schema is again generated from the Hive
columns. However, this time it is not passed through the Avro Schema Parser and
the {{namespace}} attribute is not present in {{record_0}}. The actual Error
message _"Found default.record_0, expecting union"_ is slightly misleading.
Although it is expecting a union, it is specifically expecting a null or a
record named {{record_0}} but it finds {{default.record_0}}.
I believe this is a bug in Avro. I'm not sure whether correct behaviour is to
cascade the namespace down or not but it is definitely an inconsistency between
creating a schema via the builders and parser. I've created
[AVRO-1965|https://issues.apache.org/jira/browse/AVRO-1965] for this. However,
I believe that defensively passing the schema through the Avro Schema Parser on
a select query would fix this issue in Hive without an Avro fix and version
bump in Hive.
was:
There's an issue when querying a table that has been created as Avro via CTAS
when the target struct is at least 2 struct-levels deep. It can be replicated
with the following steps:
{code}
CREATE TABLE a
STORED AS AVRO
AS
SELECT named_struct('c', named_struct('d', 1)) as b;
SELECT b FROM a;
org.apache.avro.AvroTypeException: Found default.record_0, expecting union
{code}
The reason for this is that during table creation, the Avro schema is generated
from the Hive columns in {{AvroSerDe}} and then passed through the Avro Schema
Parser: {{new Schema.Parser().parse(schema.toString())}}. For the above
example, this creates the below schema in the Avro file. Note that the lowest
level struct, {{record_0}} has {{"namespace": "default"}}.
{code}
{
"type": "record",
"name": "a",
"namespace": "default",
"fields": [
{
"name": "b",
"type": [
"null",
{
"type": "record",
"name": "record_1",
"namespace": "",
"doc": "struct>",
"fields": [
{
"name": "c",
"type": [
"null",
{
"type": "record",
"name": "record_0",
"namespace": "default",
"doc": "struct",
"fields": [
{
"name": "d",
"type": [ "null", "int" ],
"doc": "int",
"default": null
}
]
}
],
"doc": "struct",
"default": null
}
]
}
],
"default": null
}
]
}
{code}
On a subsequent select query, the Avro schema is again generated from the Hive
columns. However, this time it is not passed through the Avro Schema Parser and
the {{namespace}} attribute is not present in {{record_0}}. The actual Error
message _"Found default.record_0, expecting union"_ is slightly misleading.
Although it is a expecting a union, it is specifically expecti
[jira] [Updated] (HIVE-15316) CTAS STORED AS AVRO: AvroTypeException Found default.record_0, expecting union
[
https://issues.apache.org/jira/browse/HIVE-15316?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
David Maughan updated HIVE-15316:
-
Description:
There's an issue when querying a table that has been created as Avro via CTAS
when the target struct is at least 2 struct-levels deep. It can be replicated
with the following steps:
{code}
CREATE TABLE a
STORED AS AVRO
AS
SELECT named_struct('c', named_struct('d', 1)) as b;
SELECT b FROM a;
org.apache.avro.AvroTypeException: Found default.record_0, expecting union
{code}
The reason for this is that during table creation, the Avro schema is generated
from the Hive columns in {{AvroSerDe}} and then passed through the Avro Schema
Parser: {{new Schema.Parser().parse(schema.toString())}}. For the above
example, this creates the below schema in the Avro file. Note that the lowest
level struct, {{record_0}} has {{"namespace": "default"}}.
{code}
{
"type": "record",
"name": "a",
"namespace": "default",
"fields": [
{
"name": "b",
"type": [
"null",
{
"type": "record",
"name": "record_1",
"namespace": "",
"doc": "struct>",
"fields": [
{
"name": "c",
"type": [
"null",
{
"type": "record",
"name": "record_0",
"namespace": "default",
"doc": "struct",
"fields": [
{
"name": "d",
"type": [ "null", "int" ],
"doc": "int",
"default": null
}
]
}
],
"doc": "struct",
"default": null
}
]
}
],
"default": null
}
]
}
{code}
On a subsequent select query, the Avro schema is again generated from the Hive
columns. However, this time it is not passed through the Avro Schema Parser and
the {{namespace}} attribute is not present in {{record_0}}. The actual Error
message _"Found default.record_0, expecting union"_ is slightly misleading.
Although it is expecting a union, it is specifically expecting a null or a
record named {{record_0}} but it finds {{default.record_0}}.
I believe this is a bug in Avro. I'm not sure whether the correct behaviour is
to cascade the namespace down or not but it is definitely an inconsistency
between creating a schema via the builders and parser. I've created
[AVRO-1965|https://issues.apache.org/jira/browse/AVRO-1965] for this. However,
I believe that defensively passing the schema through the Avro Schema Parser on
a select query would fix this issue in Hive without an Avro fix and version
bump in Hive.
was:
There's an issue when querying a table that has been created as Avro via CTAS
when the target struct is at least 2 struct-levels deep. It can be replicated
with the following steps:
{code}
CREATE TABLE a
STORED AS AVRO
AS
SELECT named_struct('c', named_struct('d', 1)) as b;
SELECT b FROM a;
org.apache.avro.AvroTypeException: Found default.record_0, expecting union
{code}
The reason for this is that during table creation, the Avro schema is generated
from the Hive columns in {{AvroSerDe}} and then passed through the Avro Schema
Parser: {{new Schema.Parser().parse(schema.toString())}}. For the above
example, this creates the below schema in the Avro file. Note that the lowest
level struct, {{record_0}} has {{"namespace": "default"}}.
{code}
{
"type": "record",
"name": "a",
"namespace": "default",
"fields": [
{
"name": "b",
"type": [
"null",
{
"type": "record",
"name": "record_1",
"namespace": "",
"doc": "struct>",
"fields": [
{
"name": "c",
"type": [
"null",
{
"type": "record",
"name": "record_0",
"namespace": "default",
"doc": "struct",
"fields": [
{
"name": "d",
"type": [ "null", "int" ],
"doc": "int",
"default": null
}
]
}
],
"doc": "struct",
"default": null
}
]
}
],
"default": null
}
]
}
{code}
On a subsequent select query, the Avro schema is again generated from the Hive
columns. However, this time it is not passed through the Avro Schema Parser and
the {{namespace}} attribute is not present in {{record_0}}. The actual Error
message _"Found default.record_0, expecting union"_ is slightly misleading.
Although it is expecting a union, it is specifically expec
[jira] [Updated] (HIVE-15291) Comparison of timestamp fails if only date part is provided.
[
https://issues.apache.org/jira/browse/HIVE-15291?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Dhiraj Kumar updated HIVE-15291:
Attachment: HIVE-15291.4.patch
> Comparison of timestamp fails if only date part is provided.
> -
>
> Key: HIVE-15291
> URL: https://issues.apache.org/jira/browse/HIVE-15291
> Project: Hive
> Issue Type: Bug
> Components: Hive, UDF
>Affects Versions: 2.1.0
>Reporter: Dhiraj Kumar
>Assignee: Dhiraj Kumar
> Attachments: HIVE-15291.1.patch, HIVE-15291.2.patch,
> HIVE-15291.3.patch, HIVE-15291.4.patch
>
>
> Summary : If a query needs to compare two timestamp with one timestamp
> provided in "-MM-DD" format, skipping the time part, it returns incorrect
> result.
> Steps to reproduce :
> 1. Start a hive-cli.
> 2. Fire up the query -> select cast("2016-12-31 12:00:00" as timestamp) >
> "2016-12-30";
> 3. Expected result : true
> 4. Actual result : NULL
> Detailed description :
> If two primitives of different type needs to compared, a common comparator
> type is chosen. Prior to 2.1, Common type Text was chosen to compare
> Timestamp type and Text type.
> In version 2.1, Common type Timestamp is chosen to compare Timestamp type and
> Text type. This leads to converting Text type (-MM-DD) into
> java.sql.Timestamp which throws exception saying the input is not in proper
> format. The exception is suppressed and a null is returned.
> Code below from org.apache.hadoop.hive.ql.exec.FunctionRegistry
> {code:java}
> if (pgA == PrimitiveGrouping.STRING_GROUP && pgB ==
> PrimitiveGrouping.DATE_GROUP) {
> return b;
> }
> // date/timestamp is higher precedence than String_GROUP
> if (pgB == PrimitiveGrouping.STRING_GROUP && pgA ==
> PrimitiveGrouping.DATE_GROUP) {
> return a;
> }
> {code}
> The bug was introduced in
> [HIVE-13381|https://issues.apache.org/jira/browse/HIVE-13381]
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-15115) Flaky test: TestMiniLlapLocalCliDriver.testCliDriver[union_fast_stats]
[
https://issues.apache.org/jira/browse/HIVE-15115?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15708404#comment-15708404
]
Hive QA commented on HIVE-15115:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12841018/HIVE-15115.1.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 8 failed/errored test(s), 10748 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[sample2] (batchId=5)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[sample4] (batchId=15)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[sample6] (batchId=60)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[sample7] (batchId=59)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[sample9] (batchId=38)
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[transform_ppr2]
(batchId=133)
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver[explainanalyze_2]
(batchId=91)
org.apache.hadoop.hive.metastore.TestMetaStoreEndFunctionListener.testEndFunctionListener
(batchId=191)
{noformat}
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/2341/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/2341/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-2341/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 8 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12841018 - PreCommit-HIVE-Build
> Flaky test: TestMiniLlapLocalCliDriver.testCliDriver[union_fast_stats]
> --
>
> Key: HIVE-15115
> URL: https://issues.apache.org/jira/browse/HIVE-15115
> Project: Hive
> Issue Type: Sub-task
>Reporter: Barna Zsombor Klara
>Assignee: Barna Zsombor Klara
> Fix For: 2.2.0
>
> Attachments: HIVE-15115.1.patch, HIVE-15115.patch
>
>
> This test was identified as flaky before, it seems it turned flaky again.
> Earlier Jira:
> [HIVE-14976|https://issues.apache.org/jira/browse/HIVE-14976]
> New flaky runs:
> https://builds.apache.org/job/PreCommit-HIVE-Build/1931/testReport
> https://builds.apache.org/job/PreCommit-HIVE-Build/1930/testReport
> {code}
> 516c516
> < totalSize 3220
> ---
> > totalSize 3224
> 569c569
> < totalSize 3220
> ---
> > totalSize 3224
> 634c634
> < totalSize 4577
> ---
> > totalSize 4581
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-15291) Comparison of timestamp fails if only date part is provided.
[
https://issues.apache.org/jira/browse/HIVE-15291?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15708409#comment-15708409
]
Dhiraj Kumar commented on HIVE-15291:
-
Added the .q test.
> Comparison of timestamp fails if only date part is provided.
> -
>
> Key: HIVE-15291
> URL: https://issues.apache.org/jira/browse/HIVE-15291
> Project: Hive
> Issue Type: Bug
> Components: Hive, UDF
>Affects Versions: 2.1.0
>Reporter: Dhiraj Kumar
>Assignee: Dhiraj Kumar
> Attachments: HIVE-15291.1.patch, HIVE-15291.2.patch,
> HIVE-15291.3.patch, HIVE-15291.4.patch
>
>
> Summary : If a query needs to compare two timestamp with one timestamp
> provided in "-MM-DD" format, skipping the time part, it returns incorrect
> result.
> Steps to reproduce :
> 1. Start a hive-cli.
> 2. Fire up the query -> select cast("2016-12-31 12:00:00" as timestamp) >
> "2016-12-30";
> 3. Expected result : true
> 4. Actual result : NULL
> Detailed description :
> If two primitives of different type needs to compared, a common comparator
> type is chosen. Prior to 2.1, Common type Text was chosen to compare
> Timestamp type and Text type.
> In version 2.1, Common type Timestamp is chosen to compare Timestamp type and
> Text type. This leads to converting Text type (-MM-DD) into
> java.sql.Timestamp which throws exception saying the input is not in proper
> format. The exception is suppressed and a null is returned.
> Code below from org.apache.hadoop.hive.ql.exec.FunctionRegistry
> {code:java}
> if (pgA == PrimitiveGrouping.STRING_GROUP && pgB ==
> PrimitiveGrouping.DATE_GROUP) {
> return b;
> }
> // date/timestamp is higher precedence than String_GROUP
> if (pgB == PrimitiveGrouping.STRING_GROUP && pgA ==
> PrimitiveGrouping.DATE_GROUP) {
> return a;
> }
> {code}
> The bug was introduced in
> [HIVE-13381|https://issues.apache.org/jira/browse/HIVE-13381]
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Issue Comment Deleted] (HIVE-15291) Comparison of timestamp fails if only date part is provided.
[
https://issues.apache.org/jira/browse/HIVE-15291?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Dhiraj Kumar updated HIVE-15291:
Comment: was deleted
(was: Added the .q test.)
> Comparison of timestamp fails if only date part is provided.
> -
>
> Key: HIVE-15291
> URL: https://issues.apache.org/jira/browse/HIVE-15291
> Project: Hive
> Issue Type: Bug
> Components: Hive, UDF
>Affects Versions: 2.1.0
>Reporter: Dhiraj Kumar
>Assignee: Dhiraj Kumar
> Attachments: HIVE-15291.1.patch, HIVE-15291.2.patch,
> HIVE-15291.3.patch, HIVE-15291.4.patch
>
>
> Summary : If a query needs to compare two timestamp with one timestamp
> provided in "-MM-DD" format, skipping the time part, it returns incorrect
> result.
> Steps to reproduce :
> 1. Start a hive-cli.
> 2. Fire up the query -> select cast("2016-12-31 12:00:00" as timestamp) >
> "2016-12-30";
> 3. Expected result : true
> 4. Actual result : NULL
> Detailed description :
> If two primitives of different type needs to compared, a common comparator
> type is chosen. Prior to 2.1, Common type Text was chosen to compare
> Timestamp type and Text type.
> In version 2.1, Common type Timestamp is chosen to compare Timestamp type and
> Text type. This leads to converting Text type (-MM-DD) into
> java.sql.Timestamp which throws exception saying the input is not in proper
> format. The exception is suppressed and a null is returned.
> Code below from org.apache.hadoop.hive.ql.exec.FunctionRegistry
> {code:java}
> if (pgA == PrimitiveGrouping.STRING_GROUP && pgB ==
> PrimitiveGrouping.DATE_GROUP) {
> return b;
> }
> // date/timestamp is higher precedence than String_GROUP
> if (pgB == PrimitiveGrouping.STRING_GROUP && pgA ==
> PrimitiveGrouping.DATE_GROUP) {
> return a;
> }
> {code}
> The bug was introduced in
> [HIVE-13381|https://issues.apache.org/jira/browse/HIVE-13381]
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-15066) push down projection is mismatched in HiveInputFormat
[ https://issues.apache.org/jira/browse/HIVE-15066?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15708408#comment-15708408 ] Jeongdae Kim commented on HIVE-15066: - [~prasanth_j] Do you remember anything about [~sershe]'s comments? Could you please review when you get a chance? > push down projection is mismatched in HiveInputFormat > - > > Key: HIVE-15066 > URL: https://issues.apache.org/jira/browse/HIVE-15066 > Project: Hive > Issue Type: Bug > Components: File Formats >Affects Versions: 2.1.0 >Reporter: Jeongdae Kim >Assignee: Jeongdae Kim > Attachments: HIVE-15066.1.patch > > > if HiveInputFormat.getSplits() is called with more than one table, > inputFormatClass.getSplits() for the first table will be called with read > column information for the second table. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-15291) Comparison of timestamp fails if only date part is provided.
[
https://issues.apache.org/jira/browse/HIVE-15291?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15708407#comment-15708407
]
Dhiraj Kumar commented on HIVE-15291:
-
Added the .q test.
> Comparison of timestamp fails if only date part is provided.
> -
>
> Key: HIVE-15291
> URL: https://issues.apache.org/jira/browse/HIVE-15291
> Project: Hive
> Issue Type: Bug
> Components: Hive, UDF
>Affects Versions: 2.1.0
>Reporter: Dhiraj Kumar
>Assignee: Dhiraj Kumar
> Attachments: HIVE-15291.1.patch, HIVE-15291.2.patch,
> HIVE-15291.3.patch, HIVE-15291.4.patch
>
>
> Summary : If a query needs to compare two timestamp with one timestamp
> provided in "-MM-DD" format, skipping the time part, it returns incorrect
> result.
> Steps to reproduce :
> 1. Start a hive-cli.
> 2. Fire up the query -> select cast("2016-12-31 12:00:00" as timestamp) >
> "2016-12-30";
> 3. Expected result : true
> 4. Actual result : NULL
> Detailed description :
> If two primitives of different type needs to compared, a common comparator
> type is chosen. Prior to 2.1, Common type Text was chosen to compare
> Timestamp type and Text type.
> In version 2.1, Common type Timestamp is chosen to compare Timestamp type and
> Text type. This leads to converting Text type (-MM-DD) into
> java.sql.Timestamp which throws exception saying the input is not in proper
> format. The exception is suppressed and a null is returned.
> Code below from org.apache.hadoop.hive.ql.exec.FunctionRegistry
> {code:java}
> if (pgA == PrimitiveGrouping.STRING_GROUP && pgB ==
> PrimitiveGrouping.DATE_GROUP) {
> return b;
> }
> // date/timestamp is higher precedence than String_GROUP
> if (pgB == PrimitiveGrouping.STRING_GROUP && pgA ==
> PrimitiveGrouping.DATE_GROUP) {
> return a;
> }
> {code}
> The bug was introduced in
> [HIVE-13381|https://issues.apache.org/jira/browse/HIVE-13381]
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-15115) Flaky test: TestMiniLlapLocalCliDriver.testCliDriver[union_fast_stats]
[
https://issues.apache.org/jira/browse/HIVE-15115?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Barna Zsombor Klara updated HIVE-15115:
---
Component/s: Tests
> Flaky test: TestMiniLlapLocalCliDriver.testCliDriver[union_fast_stats]
> --
>
> Key: HIVE-15115
> URL: https://issues.apache.org/jira/browse/HIVE-15115
> Project: Hive
> Issue Type: Sub-task
> Components: Tests
>Reporter: Barna Zsombor Klara
>Assignee: Barna Zsombor Klara
> Fix For: 2.2.0
>
> Attachments: HIVE-15115.1.patch, HIVE-15115.patch
>
>
> This test was identified as flaky before, it seems it turned flaky again.
> Earlier Jira:
> [HIVE-14976|https://issues.apache.org/jira/browse/HIVE-14976]
> New flaky runs:
> https://builds.apache.org/job/PreCommit-HIVE-Build/1931/testReport
> https://builds.apache.org/job/PreCommit-HIVE-Build/1930/testReport
> {code}
> 516c516
> < totalSize 3220
> ---
> > totalSize 3224
> 569c569
> < totalSize 3220
> ---
> > totalSize 3224
> 634c634
> < totalSize 4577
> ---
> > totalSize 4581
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-15291) Comparison of timestamp fails if only date part is provided.
[
https://issues.apache.org/jira/browse/HIVE-15291?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15708531#comment-15708531
]
Hive QA commented on HIVE-15291:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12841028/HIVE-15291.4.patch
{color:green}SUCCESS:{color} +1 due to 2 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 8 failed/errored test(s), 10750 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[sample2] (batchId=5)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[sample4] (batchId=15)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[sample6] (batchId=60)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[sample7] (batchId=59)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[sample9] (batchId=38)
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[transform_ppr2]
(batchId=134)
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver[explainanalyze_1]
(batchId=91)
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver[explainanalyze_3]
(batchId=91)
{noformat}
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/2342/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/2342/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-2342/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 8 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12841028 - PreCommit-HIVE-Build
> Comparison of timestamp fails if only date part is provided.
> -
>
> Key: HIVE-15291
> URL: https://issues.apache.org/jira/browse/HIVE-15291
> Project: Hive
> Issue Type: Bug
> Components: Hive, UDF
>Affects Versions: 2.1.0
>Reporter: Dhiraj Kumar
>Assignee: Dhiraj Kumar
> Attachments: HIVE-15291.1.patch, HIVE-15291.2.patch,
> HIVE-15291.3.patch, HIVE-15291.4.patch
>
>
> Summary : If a query needs to compare two timestamp with one timestamp
> provided in "-MM-DD" format, skipping the time part, it returns incorrect
> result.
> Steps to reproduce :
> 1. Start a hive-cli.
> 2. Fire up the query -> select cast("2016-12-31 12:00:00" as timestamp) >
> "2016-12-30";
> 3. Expected result : true
> 4. Actual result : NULL
> Detailed description :
> If two primitives of different type needs to compared, a common comparator
> type is chosen. Prior to 2.1, Common type Text was chosen to compare
> Timestamp type and Text type.
> In version 2.1, Common type Timestamp is chosen to compare Timestamp type and
> Text type. This leads to converting Text type (-MM-DD) into
> java.sql.Timestamp which throws exception saying the input is not in proper
> format. The exception is suppressed and a null is returned.
> Code below from org.apache.hadoop.hive.ql.exec.FunctionRegistry
> {code:java}
> if (pgA == PrimitiveGrouping.STRING_GROUP && pgB ==
> PrimitiveGrouping.DATE_GROUP) {
> return b;
> }
> // date/timestamp is higher precedence than String_GROUP
> if (pgB == PrimitiveGrouping.STRING_GROUP && pgA ==
> PrimitiveGrouping.DATE_GROUP) {
> return a;
> }
> {code}
> The bug was introduced in
> [HIVE-13381|https://issues.apache.org/jira/browse/HIVE-13381]
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-15319) Beeline is not validating Kerberos realm
[ https://issues.apache.org/jira/browse/HIVE-15319?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matyas Orhidi updated HIVE-15319: - Description: Having "hive.server2.authentication.kerberos.principal" property set as "hive/somehost@SOME.REALM" [1] in HS2 - When connecting to the service using beeline, seemingly the realm part of the service principal in the JDBC URL is not validated - You can connect to HS2 using any realm e.g. principal=hive/somehost@ANYOTHER.REALM [2] [1] hive.server2.authentication.kerberos.principal hive/somehost@SOME.REALM [2] 'jdbc:hive2://somehost:1/default;principal=hive/somehost@ANYOTHER.REALM' was: Having "hive.server2.authentication.kerberos.principal" property set as "hive/somehost@SOME.REALM" [1] in HS2 - When connecting to the service using beeline, seemingly the realm part of the service principal in the JDBC URL is not validated - You can connect to HS2 using any realm e.g. principal=hive/somehost@ANYOTHER.REALM [2] *** [1] hive.server2.authentication.kerberos.principal hive/somehost@SOME.REALM [2] 'jdbc:hive2://somehost:1/default;principal=hive/somehost@ANYOTHER.REALM' > Beeline is not validating Kerberos realm > > > Key: HIVE-15319 > URL: https://issues.apache.org/jira/browse/HIVE-15319 > Project: Hive > Issue Type: Bug >Reporter: Matyas Orhidi > > Having "hive.server2.authentication.kerberos.principal" property set as > "hive/somehost@SOME.REALM" [1] in HS2 > - When connecting to the service using beeline, seemingly the realm part of > the service principal in the JDBC URL is not validated > - You can connect to HS2 using any realm e.g. > principal=hive/somehost@ANYOTHER.REALM [2] > [1] > hive.server2.authentication.kerberos.principal > hive/somehost@SOME.REALM > > [2] > 'jdbc:hive2://somehost:1/default;principal=hive/somehost@ANYOTHER.REALM' -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-15319) Beeline is not validating Kerberos realm
[ https://issues.apache.org/jira/browse/HIVE-15319?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matyas Orhidi updated HIVE-15319: - Summary: Beeline is not validating Kerberos realm (was: Beeline is not validating Kerberos Realm) > Beeline is not validating Kerberos realm > > > Key: HIVE-15319 > URL: https://issues.apache.org/jira/browse/HIVE-15319 > Project: Hive > Issue Type: Bug >Reporter: Matyas Orhidi > > Having "hive.server2.authentication.kerberos.principal" property set as > "hive/somehost@SOME.REALM" [1] in HS2 > - When connecting to the service using beeline, seemingly the realm part of > the service principal in the JDBC URL is not validated > - You can connect to HS2 using any realm e.g. > principal=hive/somehost@ANYOTHER.REALM [2] > *** > [1] > hive.server2.authentication.kerberos.principal > hive/somehost@SOME.REALM > > [2] > 'jdbc:hive2://somehost:1/default;principal=hive/somehost@ANYOTHER.REALM' -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-15285) err info for itests mvn building is not correct
[ https://issues.apache.org/jira/browse/HIVE-15285?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15708953#comment-15708953 ] Sergio Peña commented on HIVE-15285: I think the 'amp;' from the this line is incorrect: echo "download spark error, please retry" >&2 > err info for itests mvn building is not correct > --- > > Key: HIVE-15285 > URL: https://issues.apache.org/jira/browse/HIVE-15285 > Project: Hive > Issue Type: Bug > Components: Tests >Affects Versions: 2.1.0, 2.2.0 >Reporter: Fei Hui >Assignee: Fei Hui > Attachments: HIVE-15285.1.patch > > > When i build itests, i found some err info > ''' > [exec] cp: cannot stat > `./target/../../..//data/conf/spark/log4j2.properties': No such file or > directory > ''' > But i found that the real reason is spark down load error. The msg above > confuse users ， it is not root cause -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-15302) Relax the requirement that HoS needs Spark built w/o Hive
[ https://issues.apache.org/jira/browse/HIVE-15302?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15709208#comment-15709208 ] Marcelo Vanzin commented on HIVE-15302: --- bq. I plan to find the needed jars from the Spark installed in the cluster That's kind of what I meant. Wouldn't it be better to just directly depend on the parts of Spark that Hive needs, package those with Hive, and not have to depend on any cluster deployment of Spark? Then the user doesn't need to care about a separate Spark installation when he wants to run HoS. > Relax the requirement that HoS needs Spark built w/o Hive > - > > Key: HIVE-15302 > URL: https://issues.apache.org/jira/browse/HIVE-15302 > Project: Hive > Issue Type: Improvement >Reporter: Rui Li >Assignee: Rui Li > > This requirement becomes more and more unacceptable as SparkSQL becomes > widely adopted. Let's use this JIRA to find out how we can relax the > limitation. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-15291) Comparison of timestamp fails if only date part is provided.
[
https://issues.apache.org/jira/browse/HIVE-15291?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15709324#comment-15709324
]
Jason Dere commented on HIVE-15291:
---
+1
cc [~ashutoshc]
> Comparison of timestamp fails if only date part is provided.
> -
>
> Key: HIVE-15291
> URL: https://issues.apache.org/jira/browse/HIVE-15291
> Project: Hive
> Issue Type: Bug
> Components: Hive, UDF
>Affects Versions: 2.1.0
>Reporter: Dhiraj Kumar
>Assignee: Dhiraj Kumar
> Attachments: HIVE-15291.1.patch, HIVE-15291.2.patch,
> HIVE-15291.3.patch, HIVE-15291.4.patch
>
>
> Summary : If a query needs to compare two timestamp with one timestamp
> provided in "-MM-DD" format, skipping the time part, it returns incorrect
> result.
> Steps to reproduce :
> 1. Start a hive-cli.
> 2. Fire up the query -> select cast("2016-12-31 12:00:00" as timestamp) >
> "2016-12-30";
> 3. Expected result : true
> 4. Actual result : NULL
> Detailed description :
> If two primitives of different type needs to compared, a common comparator
> type is chosen. Prior to 2.1, Common type Text was chosen to compare
> Timestamp type and Text type.
> In version 2.1, Common type Timestamp is chosen to compare Timestamp type and
> Text type. This leads to converting Text type (-MM-DD) into
> java.sql.Timestamp which throws exception saying the input is not in proper
> format. The exception is suppressed and a null is returned.
> Code below from org.apache.hadoop.hive.ql.exec.FunctionRegistry
> {code:java}
> if (pgA == PrimitiveGrouping.STRING_GROUP && pgB ==
> PrimitiveGrouping.DATE_GROUP) {
> return b;
> }
> // date/timestamp is higher precedence than String_GROUP
> if (pgB == PrimitiveGrouping.STRING_GROUP && pgA ==
> PrimitiveGrouping.DATE_GROUP) {
> return a;
> }
> {code}
> The bug was introduced in
> [HIVE-13381|https://issues.apache.org/jira/browse/HIVE-13381]
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-15311) Analyze column stats should skip non-primitive column types
[ https://issues.apache.org/jira/browse/HIVE-15311?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Pengcheng Xiong updated HIVE-15311: --- Fix Version/s: 2.2.0 > Analyze column stats should skip non-primitive column types > --- > > Key: HIVE-15311 > URL: https://issues.apache.org/jira/browse/HIVE-15311 > Project: Hive > Issue Type: Bug >Affects Versions: 1.2.0, 2.0.0, 2.1.0 >Reporter: Pengcheng Xiong >Assignee: Pengcheng Xiong > Fix For: 2.2.0 > > Attachments: HIVE-15311.01.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-15311) Analyze column stats should skip non-primitive column types
[ https://issues.apache.org/jira/browse/HIVE-15311?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Pengcheng Xiong updated HIVE-15311: --- Affects Version/s: 1.2.0 2.0.0 2.1.0 > Analyze column stats should skip non-primitive column types > --- > > Key: HIVE-15311 > URL: https://issues.apache.org/jira/browse/HIVE-15311 > Project: Hive > Issue Type: Bug >Affects Versions: 1.2.0, 2.0.0, 2.1.0 >Reporter: Pengcheng Xiong >Assignee: Pengcheng Xiong > Fix For: 2.2.0 > > Attachments: HIVE-15311.01.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-15311) Analyze column stats should skip non-primitive column types
[ https://issues.apache.org/jira/browse/HIVE-15311?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Pengcheng Xiong updated HIVE-15311: --- Description: after this patch, when you compute column stats, it will skip the non-primitive column types and give you warning on the console. > Analyze column stats should skip non-primitive column types > --- > > Key: HIVE-15311 > URL: https://issues.apache.org/jira/browse/HIVE-15311 > Project: Hive > Issue Type: Bug >Affects Versions: 1.2.0, 2.0.0, 2.1.0 >Reporter: Pengcheng Xiong >Assignee: Pengcheng Xiong > Fix For: 2.2.0 > > Attachments: HIVE-15311.01.patch > > > after this patch, when you compute column stats, it will skip the > non-primitive column types and give you warning on the console. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-15311) Analyze column stats should skip non-primitive column types
[ https://issues.apache.org/jira/browse/HIVE-15311?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Pengcheng Xiong updated HIVE-15311: --- Resolution: Fixed Status: Resolved (was: Patch Available) pushed to master. thanks [~ashutoshc] for the review. > Analyze column stats should skip non-primitive column types > --- > > Key: HIVE-15311 > URL: https://issues.apache.org/jira/browse/HIVE-15311 > Project: Hive > Issue Type: Bug >Affects Versions: 1.2.0, 2.0.0, 2.1.0 >Reporter: Pengcheng Xiong >Assignee: Pengcheng Xiong > Fix For: 2.2.0 > > Attachments: HIVE-15311.01.patch > > > after this patch, when you compute column stats, it will skip the > non-primitive column types and give you warning on the console. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-15311) Analyze column stats should skip non-primitive column types
[ https://issues.apache.org/jira/browse/HIVE-15311?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15709426#comment-15709426 ] Pengcheng Xiong commented on HIVE-15311: btw, I updated golden file for columnstats_tbllvl_complex_type and the rest q tests are unrelated. > Analyze column stats should skip non-primitive column types > --- > > Key: HIVE-15311 > URL: https://issues.apache.org/jira/browse/HIVE-15311 > Project: Hive > Issue Type: Bug >Affects Versions: 1.2.0, 2.0.0, 2.1.0 >Reporter: Pengcheng Xiong >Assignee: Pengcheng Xiong > Fix For: 2.2.0 > > Attachments: HIVE-15311.01.patch > > > after this patch, when you compute column stats, it will skip the > non-primitive column types and give you warning on the console. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-14582) Add trunc(numeric) udf
[ https://issues.apache.org/jira/browse/HIVE-14582?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chinna Rao Lalam updated HIVE-14582: Status: Open (was: Patch Available) > Add trunc(numeric) udf > -- > > Key: HIVE-14582 > URL: https://issues.apache.org/jira/browse/HIVE-14582 > Project: Hive > Issue Type: Sub-task > Components: SQL >Reporter: Ashutosh Chauhan >Assignee: Chinna Rao Lalam > Attachments: HIVE-14582.1.patch, HIVE-14582.2.patch, > HIVE-14582.3.patch, HIVE-14582.patch > > > https://docs.oracle.com/cd/B19306_01/server.102/b14200/functions200.htm -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-14582) Add trunc(numeric) udf
[ https://issues.apache.org/jira/browse/HIVE-14582?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chinna Rao Lalam updated HIVE-14582: Attachment: HIVE-14582.4.patch > Add trunc(numeric) udf > -- > > Key: HIVE-14582 > URL: https://issues.apache.org/jira/browse/HIVE-14582 > Project: Hive > Issue Type: Sub-task > Components: SQL >Reporter: Ashutosh Chauhan >Assignee: Chinna Rao Lalam > Attachments: HIVE-14582.1.patch, HIVE-14582.2.patch, > HIVE-14582.3.patch, HIVE-14582.4.patch, HIVE-14582.patch > > > https://docs.oracle.com/cd/B19306_01/server.102/b14200/functions200.htm -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-15202) Concurrent compactions for the same partition may generate malformed folder structure
[
https://issues.apache.org/jira/browse/HIVE-15202?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15709444#comment-15709444
]
Wei Zheng commented on HIVE-15202:
--
Patch 3 looks good. +1
> Concurrent compactions for the same partition may generate malformed folder
> structure
> -
>
> Key: HIVE-15202
> URL: https://issues.apache.org/jira/browse/HIVE-15202
> Project: Hive
> Issue Type: Bug
> Components: Transactions
>Reporter: Rui Li
>Assignee: Eugene Koifman
> Attachments: HIVE-15202.01.patch, HIVE-15202.02.patch,
> HIVE-15202.03.patch
>
>
> If two compactions run concurrently on a single partition, it may generate
> folder structure like this: (nested base dir)
> {noformat}
> drwxr-xr-x - root supergroup 0 2016-11-14 22:23
> /user/hive/warehouse/test/z=1/base_007/base_007
> -rw-r--r-- 3 root supergroup201 2016-11-14 21:46
> /user/hive/warehouse/test/z=1/base_007/bucket_0
> -rw-r--r-- 3 root supergroup611 2016-11-14 21:46
> /user/hive/warehouse/test/z=1/base_007/bucket_1
> -rw-r--r-- 3 root supergroup614 2016-11-14 21:46
> /user/hive/warehouse/test/z=1/base_007/bucket_2
> -rw-r--r-- 3 root supergroup621 2016-11-14 21:46
> /user/hive/warehouse/test/z=1/base_007/bucket_3
> -rw-r--r-- 3 root supergroup621 2016-11-14 21:46
> /user/hive/warehouse/test/z=1/base_007/bucket_4
> -rw-r--r-- 3 root supergroup201 2016-11-14 21:46
> /user/hive/warehouse/test/z=1/base_007/bucket_5
> -rw-r--r-- 3 root supergroup201 2016-11-14 21:46
> /user/hive/warehouse/test/z=1/base_007/bucket_6
> -rw-r--r-- 3 root supergroup201 2016-11-14 21:46
> /user/hive/warehouse/test/z=1/base_007/bucket_7
> -rw-r--r-- 3 root supergroup201 2016-11-14 21:46
> /user/hive/warehouse/test/z=1/base_007/bucket_8
> -rw-r--r-- 3 root supergroup201 2016-11-14 21:46
> /user/hive/warehouse/test/z=1/base_007/bucket_9
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-15312) reduce logging in certain places
[ https://issues.apache.org/jira/browse/HIVE-15312?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HIVE-15312: Attachment: HIVE-15312.01.patch A file was missing. > reduce logging in certain places > > > Key: HIVE-15312 > URL: https://issues.apache.org/jira/browse/HIVE-15312 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > Attachments: HIVE-15312.01.patch, HIVE-15312.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-15242) LLAP: Act on Node update notifications from registry, fix isAlive checks
[ https://issues.apache.org/jira/browse/HIVE-15242?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HIVE-15242: Attachment: HIVE-15242.03.patch > LLAP: Act on Node update notifications from registry, fix isAlive checks > > > Key: HIVE-15242 > URL: https://issues.apache.org/jira/browse/HIVE-15242 > Project: Hive > Issue Type: Task >Reporter: Siddharth Seth >Assignee: Sergey Shelukhin > Attachments: HIVE-15242.01.patch, HIVE-15242.02.patch, > HIVE-15242.03.patch > > > isAlive checks are currently completely broken, since the ZK registry does > not update existing ServiceIsntances. Instead it creates new instances each > time. > This causes non-existant nodes to be used for scheduling in case of node > failures. > Also, act on the notifications sent by the registry about nodes going down, > or new nodes being added. (new nodes already handled) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Assigned] (HIVE-15242) LLAP: Act on Node update notifications from registry, fix isAlive checks
[ https://issues.apache.org/jira/browse/HIVE-15242?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin reassigned HIVE-15242: --- Assignee: Sergey Shelukhin (was: Siddharth Seth) > LLAP: Act on Node update notifications from registry, fix isAlive checks > > > Key: HIVE-15242 > URL: https://issues.apache.org/jira/browse/HIVE-15242 > Project: Hive > Issue Type: Task >Reporter: Siddharth Seth >Assignee: Sergey Shelukhin > Attachments: HIVE-15242.01.patch, HIVE-15242.02.patch, > HIVE-15242.03.patch > > > isAlive checks are currently completely broken, since the ZK registry does > not update existing ServiceIsntances. Instead it creates new instances each > time. > This causes non-existant nodes to be used for scheduling in case of node > failures. > Also, act on the notifications sent by the registry about nodes going down, > or new nodes being added. (new nodes already handled) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-15202) Concurrent compactions for the same partition may generate malformed folder structure
[
https://issues.apache.org/jira/browse/HIVE-15202?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15709447#comment-15709447
]
Eugene Koifman commented on HIVE-15202:
---
It's not. The cleaner can only remove files that are obsolete, i.e. there is
another (better/wider) delta/base that includes the same data. No reader
(including compaction) reads obsolete data
> Concurrent compactions for the same partition may generate malformed folder
> structure
> -
>
> Key: HIVE-15202
> URL: https://issues.apache.org/jira/browse/HIVE-15202
> Project: Hive
> Issue Type: Bug
> Components: Transactions
>Reporter: Rui Li
>Assignee: Eugene Koifman
> Attachments: HIVE-15202.01.patch, HIVE-15202.02.patch,
> HIVE-15202.03.patch
>
>
> If two compactions run concurrently on a single partition, it may generate
> folder structure like this: (nested base dir)
> {noformat}
> drwxr-xr-x - root supergroup 0 2016-11-14 22:23
> /user/hive/warehouse/test/z=1/base_007/base_007
> -rw-r--r-- 3 root supergroup201 2016-11-14 21:46
> /user/hive/warehouse/test/z=1/base_007/bucket_0
> -rw-r--r-- 3 root supergroup611 2016-11-14 21:46
> /user/hive/warehouse/test/z=1/base_007/bucket_1
> -rw-r--r-- 3 root supergroup614 2016-11-14 21:46
> /user/hive/warehouse/test/z=1/base_007/bucket_2
> -rw-r--r-- 3 root supergroup621 2016-11-14 21:46
> /user/hive/warehouse/test/z=1/base_007/bucket_3
> -rw-r--r-- 3 root supergroup621 2016-11-14 21:46
> /user/hive/warehouse/test/z=1/base_007/bucket_4
> -rw-r--r-- 3 root supergroup201 2016-11-14 21:46
> /user/hive/warehouse/test/z=1/base_007/bucket_5
> -rw-r--r-- 3 root supergroup201 2016-11-14 21:46
> /user/hive/warehouse/test/z=1/base_007/bucket_6
> -rw-r--r-- 3 root supergroup201 2016-11-14 21:46
> /user/hive/warehouse/test/z=1/base_007/bucket_7
> -rw-r--r-- 3 root supergroup201 2016-11-14 21:46
> /user/hive/warehouse/test/z=1/base_007/bucket_8
> -rw-r--r-- 3 root supergroup201 2016-11-14 21:46
> /user/hive/warehouse/test/z=1/base_007/bucket_9
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-14582) Add trunc(numeric) udf
[ https://issues.apache.org/jira/browse/HIVE-14582?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chinna Rao Lalam updated HIVE-14582: Status: Patch Available (was: Open) Hi [~ashutoshc], Thanks for the review. I have referred round() and bround() functions, which accepts scale argument as constant. Now trunc() function is updated to accept scale argument as constant or non-constant. Please take a look at patch. > Add trunc(numeric) udf > -- > > Key: HIVE-14582 > URL: https://issues.apache.org/jira/browse/HIVE-14582 > Project: Hive > Issue Type: Sub-task > Components: SQL >Reporter: Ashutosh Chauhan >Assignee: Chinna Rao Lalam > Attachments: HIVE-14582.1.patch, HIVE-14582.2.patch, > HIVE-14582.3.patch, HIVE-14582.4.patch, HIVE-14582.patch > > > https://docs.oracle.com/cd/B19306_01/server.102/b14200/functions200.htm -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-15321) Change to read as long for HiveConf.ConfVars.METASTORESERVERMAXMESSAGESIZE
[
https://issues.apache.org/jira/browse/HIVE-15321?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Aihua Xu updated HIVE-15321:
Attachment: HIVE-15321.1.patch
> Change to read as long for HiveConf.ConfVars.METASTORESERVERMAXMESSAGESIZE
> --
>
> Key: HIVE-15321
> URL: https://issues.apache.org/jira/browse/HIVE-15321
> Project: Hive
> Issue Type: Improvement
> Components: Metastore
>Affects Versions: 1.2.0, 1.1.0
>Reporter: Aihua Xu
>Assignee: Aihua Xu
> Attachments: HIVE-15321.1.patch
>
>
> Follow up on HIVE-11240 which tries to change the type from int to long,
> while we are still read with {{conf.getIntVar()}}.
> Seems we should use {{conf.getLongVar()}}.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-15115) Flaky test: TestMiniLlapLocalCliDriver.testCliDriver[union_fast_stats]
[
https://issues.apache.org/jira/browse/HIVE-15115?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15709505#comment-15709505
]
Prasanth Jayachandran commented on HIVE-15115:
--
I guess the tests are not failing because it is getting instances from the same
region (PST). If it gets instances from different regions then this can fail.
May be we should build a better resiliency for these file size related
differences in qtestutil. Another option would be add a new marker "TIMEZONE
SENSITIVE TEST" to the q file and mask total file size during result comparison
that have the marker?
> Flaky test: TestMiniLlapLocalCliDriver.testCliDriver[union_fast_stats]
> --
>
> Key: HIVE-15115
> URL: https://issues.apache.org/jira/browse/HIVE-15115
> Project: Hive
> Issue Type: Sub-task
> Components: Tests
>Reporter: Barna Zsombor Klara
>Assignee: Barna Zsombor Klara
> Fix For: 2.2.0
>
> Attachments: HIVE-15115.1.patch, HIVE-15115.patch
>
>
> This test was identified as flaky before, it seems it turned flaky again.
> Earlier Jira:
> [HIVE-14976|https://issues.apache.org/jira/browse/HIVE-14976]
> New flaky runs:
> https://builds.apache.org/job/PreCommit-HIVE-Build/1931/testReport
> https://builds.apache.org/job/PreCommit-HIVE-Build/1930/testReport
> {code}
> 516c516
> < totalSize 3220
> ---
> > totalSize 3224
> 569c569
> < totalSize 3220
> ---
> > totalSize 3224
> 634c634
> < totalSize 4577
> ---
> > totalSize 4581
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-15321) Change to read as long for HiveConf.ConfVars.METASTORESERVERMAXMESSAGESIZE
[
https://issues.apache.org/jira/browse/HIVE-15321?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Aihua Xu updated HIVE-15321:
Status: Patch Available (was: Open)
> Change to read as long for HiveConf.ConfVars.METASTORESERVERMAXMESSAGESIZE
> --
>
> Key: HIVE-15321
> URL: https://issues.apache.org/jira/browse/HIVE-15321
> Project: Hive
> Issue Type: Improvement
> Components: Metastore
>Affects Versions: 1.1.0, 1.2.0
>Reporter: Aihua Xu
>Assignee: Aihua Xu
> Attachments: HIVE-15321.1.patch
>
>
> Follow up on HIVE-11240 which tries to change the type from int to long,
> while we are still read with {{conf.getIntVar()}}.
> Seems we should use {{conf.getLongVar()}}.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-11345) Fix formatting of Show Compations/Transactions/Locks
[ https://issues.apache.org/jira/browse/HIVE-11345?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eugene Koifman updated HIVE-11345: -- Assignee: Wei Zheng > Fix formatting of Show Compations/Transactions/Locks > > > Key: HIVE-11345 > URL: https://issues.apache.org/jira/browse/HIVE-11345 > Project: Hive > Issue Type: Bug > Components: CLI, Transactions >Affects Versions: 1.0.0 >Reporter: Eugene Koifman >Assignee: Wei Zheng > > all the columns of the output are variable length (in each row, based on > data) - makes it really difficult to read -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-11345) Fix formatting of Show Compations/Transactions/Locks
[ https://issues.apache.org/jira/browse/HIVE-11345?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eugene Koifman updated HIVE-11345: -- Target Version/s: 2.2.0 > Fix formatting of Show Compations/Transactions/Locks > > > Key: HIVE-11345 > URL: https://issues.apache.org/jira/browse/HIVE-11345 > Project: Hive > Issue Type: Bug > Components: CLI, Transactions >Affects Versions: 1.0.0 >Reporter: Eugene Koifman >Assignee: Wei Zheng > > all the columns of the output are variable length (in each row, based on > data) - makes it really difficult to read -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-15308) Create ACID table failed intermittently: due to Postgres (SQLState=25P02, ErrorCode=0)
[
https://issues.apache.org/jira/browse/HIVE-15308?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15709535#comment-15709535
]
Wei Zheng commented on HIVE-15308:
--
+1
> Create ACID table failed intermittently: due to Postgres (SQLState=25P02,
> ErrorCode=0)
> --
>
> Key: HIVE-15308
> URL: https://issues.apache.org/jira/browse/HIVE-15308
> Project: Hive
> Issue Type: Bug
> Components: Transactions
>Affects Versions: 2.1.0
>Reporter: Eugene Koifman
>Assignee: Eugene Koifman
> Attachments: HIVE-15308.01.patch
>
>
> if 2 concurrent calls to MutexApi.acquireLock() happen with the same "key"
> value and there is no row in AUX_TABLE for that value yet (i.e. both are
> attempting to insert it) Postgres kills the txn which gets the Duplicate Key
> error and no more statements can be executed on this txn.
> (This is different from the way most DBs behave).
> {noformat}
> Caused by: MetaException(message:Unable to lock 'CheckLock' due to: ERROR:
> current transaction is aborted, commands ignored until end of transaction
> block (SQLState=25P02, ErrorCode=0); org.postgresql.util.PSQLException:
> ERROR: current transaction is aborted, commands ignored until end of
> transaction block
> at
> org.postgresql.core.v3.QueryExecutorImpl.receiveErrorResponse(QueryExecutorImpl.java:2198)
> at
> org.postgresql.core.v3.QueryExecutorImpl.processResults(QueryExecutorImpl.java:1927)
> at
> org.postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:255)
> at
> org.postgresql.jdbc2.AbstractJdbc2Statement.execute(AbstractJdbc2Statement.java:561)
> at
> org.postgresql.jdbc2.AbstractJdbc2Statement.executeWithFlags(AbstractJdbc2Statement.java:405)
> at
> org.postgresql.jdbc2.AbstractJdbc2Statement.executeQuery(AbstractJdbc2Statement.java:285)
> at
> com.jolbox.bonecp.StatementHandle.executeQuery(StatementHandle.java:464)
> at
> org.apache.hadoop.hive.metastore.txn.TxnHandler.acquireLock(TxnHandler.java:3250)
> at
> org.apache.hadoop.hive.metastore.txn.TxnHandler.checkLock(TxnHandler.java:2319)
> at
> org.apache.hadoop.hive.metastore.txn.TxnHandler.checkLockWithRetry(TxnHandler.java:1022)
> at
> org.apache.hadoop.hive.metastore.txn.TxnHandler.lock(TxnHandler.java:794)
> at
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.lock(HiveMetaStore.java:5941)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.apache.hadoop.hive.metastore.RetryingHMSHandler.invokeInternal(RetryingHMSHandler.java:140)
> at
> org.apache.hadoop.hive.metastore.RetryingHMSHandler.invoke(RetryingHMSHandler.java:99)
> at com.sun.proxy.$Proxy30.lock(Unknown Source)
> at
> org.apache.hadoop.hive.metastore.HiveMetaStoreClient.lock(HiveMetaStoreClient.java:2109)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.invoke(RetryingMetaStoreClient.java:154)
> at com.sun.proxy.$Proxy31.lock(Unknown Source)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.apache.hadoop.hive.metastore.HiveMetaStoreClient$SynchronizedHandler.invoke(HiveMetaStoreClient.java:2259)
> at com.sun.proxy.$Proxy31.lock(Unknown Source)
> at
> org.apache.hadoop.hive.ql.lockmgr.DbTxnManager$SynchronizedMetaStoreClient.lock(DbTxnManager.java:745)
> at
> org.apache.hadoop.hive.ql.lockmgr.DbLockManager.lock(DbLockManager.java:103)
> at
> org.apache.hadoop.hive.ql.lockmgr.DbTxnManager.acquireLocks(DbTxnManager.java:341)
> at
> org.apache.hadoop.hive.ql.lockmgr.DbTxnManager.acquireLocksWithHeartbeatDelay(DbTxnManager.java:357)
> at
> org.apache.hadoop.hive.ql.lockmgr.DbTxnManager.acquireLocks(DbTxnManager.java:167)
> at
> org.apache.hadoop.hive.ql.Driver.acquireLocksAndOpenTxn(Driver.java:985)
> at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:132
[jira] [Commented] (HIVE-15321) Change to read as long for HiveConf.ConfVars.METASTORESERVERMAXMESSAGESIZE
[
https://issues.apache.org/jira/browse/HIVE-15321?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15709538#comment-15709538
]
Yongzhi Chen commented on HIVE-15321:
-
LGTM +1
> Change to read as long for HiveConf.ConfVars.METASTORESERVERMAXMESSAGESIZE
> --
>
> Key: HIVE-15321
> URL: https://issues.apache.org/jira/browse/HIVE-15321
> Project: Hive
> Issue Type: Improvement
> Components: Metastore
>Affects Versions: 1.2.0, 1.1.0
>Reporter: Aihua Xu
>Assignee: Aihua Xu
> Attachments: HIVE-15321.1.patch
>
>
> Follow up on HIVE-11240 which tries to change the type from int to long,
> while we are still read with {{conf.getIntVar()}}.
> Seems we should use {{conf.getLongVar()}}.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-15321) Change to read as long for HiveConf.ConfVars.METASTORESERVERMAXMESSAGESIZE
[
https://issues.apache.org/jira/browse/HIVE-15321?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15709512#comment-15709512
]
Aihua Xu commented on HIVE-15321:
-
Patch-1: change to conf.getLongVar() so we can handle larger range of message
size.
> Change to read as long for HiveConf.ConfVars.METASTORESERVERMAXMESSAGESIZE
> --
>
> Key: HIVE-15321
> URL: https://issues.apache.org/jira/browse/HIVE-15321
> Project: Hive
> Issue Type: Improvement
> Components: Metastore
>Affects Versions: 1.2.0, 1.1.0
>Reporter: Aihua Xu
>Assignee: Aihua Xu
> Attachments: HIVE-15321.1.patch
>
>
> Follow up on HIVE-11240 which tries to change the type from int to long,
> while we are still read with {{conf.getIntVar()}}.
> Seems we should use {{conf.getLongVar()}}.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-14582) Add trunc(numeric) udf
[ https://issues.apache.org/jira/browse/HIVE-14582?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15709581#comment-15709581 ] Ashutosh Chauhan commented on HIVE-14582: - +1 > Add trunc(numeric) udf > -- > > Key: HIVE-14582 > URL: https://issues.apache.org/jira/browse/HIVE-14582 > Project: Hive > Issue Type: Sub-task > Components: SQL >Reporter: Ashutosh Chauhan >Assignee: Chinna Rao Lalam > Attachments: HIVE-14582.1.patch, HIVE-14582.2.patch, > HIVE-14582.3.patch, HIVE-14582.4.patch, HIVE-14582.patch > > > https://docs.oracle.com/cd/B19306_01/server.102/b14200/functions200.htm -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-15314) ThriftCLIService should LOG.error rather than LOG.warn when Exception occurs
[ https://issues.apache.org/jira/browse/HIVE-15314?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15709623#comment-15709623 ] Thejas M Nair commented on HIVE-15314: -- I believe this is set to warn because it is not an HS2 code error (which is like a wrapper around the Driver code in most cases), but an error while running the query. In these cases, the underlying driver code usually logs it at error level. Also, the client is sent the error as well. cc [~prasadm] [~vgumashta] > ThriftCLIService should LOG.error rather than LOG.warn when Exception occurs > > > Key: HIVE-15314 > URL: https://issues.apache.org/jira/browse/HIVE-15314 > Project: Hive > Issue Type: Bug > Components: CLI, Logging >Affects Versions: 2.2.0 >Reporter: Fei Hui >Assignee: Fei Hui > Attachments: HIVE-15314.1.patch > > > When catch exception, critical error occurs, > and the message in log is ''Error executing statement", "Error getting type > info", etc. > So we should use LOG.error, and it will remind users. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-14582) Add trunc(numeric) udf
[
https://issues.apache.org/jira/browse/HIVE-14582?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15709653#comment-15709653
]
Hive QA commented on HIVE-14582:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12841094/HIVE-14582.4.patch
{color:green}SUCCESS:{color} +1 due to 2 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 7 failed/errored test(s), 10752 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.org.apache.hadoop.hive.cli.TestCliDriver
(batchId=50)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[sample2] (batchId=5)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[sample4] (batchId=15)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[sample6] (batchId=60)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[sample7] (batchId=60)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[sample9] (batchId=38)
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[transform_ppr2]
(batchId=134)
{noformat}
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/2343/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/2343/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-2343/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 7 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12841094 - PreCommit-HIVE-Build
> Add trunc(numeric) udf
> --
>
> Key: HIVE-14582
> URL: https://issues.apache.org/jira/browse/HIVE-14582
> Project: Hive
> Issue Type: Sub-task
> Components: SQL
>Reporter: Ashutosh Chauhan
>Assignee: Chinna Rao Lalam
> Attachments: HIVE-14582.1.patch, HIVE-14582.2.patch,
> HIVE-14582.3.patch, HIVE-14582.4.patch, HIVE-14582.patch
>
>
> https://docs.oracle.com/cd/B19306_01/server.102/b14200/functions200.htm
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-15321) Change to read as long for HiveConf.ConfVars.METASTORESERVERMAXMESSAGESIZE
[
https://issues.apache.org/jira/browse/HIVE-15321?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Aihua Xu updated HIVE-15321:
Status: In Progress (was: Patch Available)
> Change to read as long for HiveConf.ConfVars.METASTORESERVERMAXMESSAGESIZE
> --
>
> Key: HIVE-15321
> URL: https://issues.apache.org/jira/browse/HIVE-15321
> Project: Hive
> Issue Type: Improvement
> Components: Metastore
>Affects Versions: 1.1.0, 1.2.0
>Reporter: Aihua Xu
>Assignee: Aihua Xu
> Attachments: HIVE-15321.1.patch, HIVE-15321.2.patch
>
>
> Follow up on HIVE-11240 which tries to change the type from int to long,
> while we are still read with {{conf.getIntVar()}}.
> Seems we should use {{conf.getLongVar()}}.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-15321) Change to read as long for HiveConf.ConfVars.METASTORESERVERMAXMESSAGESIZE
[
https://issues.apache.org/jira/browse/HIVE-15321?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Aihua Xu updated HIVE-15321:
Status: Patch Available (was: In Progress)
> Change to read as long for HiveConf.ConfVars.METASTORESERVERMAXMESSAGESIZE
> --
>
> Key: HIVE-15321
> URL: https://issues.apache.org/jira/browse/HIVE-15321
> Project: Hive
> Issue Type: Improvement
> Components: Metastore
>Affects Versions: 1.1.0, 1.2.0
>Reporter: Aihua Xu
>Assignee: Aihua Xu
> Attachments: HIVE-15321.1.patch, HIVE-15321.2.patch
>
>
> Follow up on HIVE-11240 which tries to change the type from int to long,
> while we are still read with {{conf.getIntVar()}}.
> Seems we should use {{conf.getLongVar()}}.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-15321) Change to read as long for HiveConf.ConfVars.METASTORESERVERMAXMESSAGESIZE
[
https://issues.apache.org/jira/browse/HIVE-15321?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Aihua Xu updated HIVE-15321:
Attachment: HIVE-15321.2.patch
> Change to read as long for HiveConf.ConfVars.METASTORESERVERMAXMESSAGESIZE
> --
>
> Key: HIVE-15321
> URL: https://issues.apache.org/jira/browse/HIVE-15321
> Project: Hive
> Issue Type: Improvement
> Components: Metastore
>Affects Versions: 1.2.0, 1.1.0
>Reporter: Aihua Xu
>Assignee: Aihua Xu
> Attachments: HIVE-15321.1.patch, HIVE-15321.2.patch
>
>
> Follow up on HIVE-11240 which tries to change the type from int to long,
> while we are still read with {{conf.getIntVar()}}.
> Seems we should use {{conf.getLongVar()}}.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-14848) S3 creds added to a hidden list by HIVE-14588 are not working on MR jobs
[
https://issues.apache.org/jira/browse/HIVE-14848?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15709721#comment-15709721
]
Thomas Poepping commented on HIVE-14848:
[~spena] suggest you also change the call to stripHiddenConfigurations() in
ql/exec/tez/DagUtils.createConfiguration(), because this is currently broken
for running hive-blobstore suite against Tez.
> S3 creds added to a hidden list by HIVE-14588 are not working on MR jobs
>
>
> Key: HIVE-14848
> URL: https://issues.apache.org/jira/browse/HIVE-14848
> Project: Hive
> Issue Type: Bug
> Components: Hive
>Affects Versions: 2.2.0
>Reporter: Sergio Peña
>Assignee: Sergio Peña
> Attachments: HIVE-14848.1.patch, HIVE-14848.1.patch
>
>
> When S3 credentials are included in hive-site.xml, then MR jobs that need to
> read
> data from S3 cannot use them because S3 values are stripped from the Job
> configuration
> object before submitting the MR job.
> {noformat}
> @Override
> public void initialize(HiveConf conf, QueryPlan queryPlan, DriverContext
> driverContext) {
> ...
> conf.stripHiddenConfigurations(job);
> this.jobExecHelper = new HadoopJobExecHelper(job, console, this, this);
> }
> {noformat}
> A nice to have (available on hadoop 2.9.0) is an MR
> {{mapreduce.job.redacted-properties}} that can be used to hide this list on
> the MR side (such as history server UI) to allow MR run the job without
> issues.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-14848) S3 creds added to a hidden list by HIVE-14588 are not working on MR jobs
[
https://issues.apache.org/jira/browse/HIVE-14848?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sergio Peña updated HIVE-14848:
---
Target Version/s: 2.2.0
> S3 creds added to a hidden list by HIVE-14588 are not working on MR jobs
>
>
> Key: HIVE-14848
> URL: https://issues.apache.org/jira/browse/HIVE-14848
> Project: Hive
> Issue Type: Bug
> Components: Hive
>Affects Versions: 2.2.0
>Reporter: Sergio Peña
>Assignee: Sergio Peña
> Attachments: HIVE-14848.1.patch, HIVE-14848.1.patch
>
>
> When S3 credentials are included in hive-site.xml, then MR jobs that need to
> read
> data from S3 cannot use them because S3 values are stripped from the Job
> configuration
> object before submitting the MR job.
> {noformat}
> @Override
> public void initialize(HiveConf conf, QueryPlan queryPlan, DriverContext
> driverContext) {
> ...
> conf.stripHiddenConfigurations(job);
> this.jobExecHelper = new HadoopJobExecHelper(job, console, this, this);
> }
> {noformat}
> A nice to have (available on hadoop 2.9.0) is an MR
> {{mapreduce.job.redacted-properties}} that can be used to hide this list on
> the MR side (such as history server UI) to allow MR run the job without
> issues.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-14848) S3 creds added to a hidden list by HIVE-14588 are not working on MR jobs
[
https://issues.apache.org/jira/browse/HIVE-14848?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sergio Peña updated HIVE-14848:
---
Description:
When S3 credentials are included in hive-site.xml, then MR jobs that need to
read
data from S3 cannot use them because S3 values are stripped from the Job
configuration
object before submitting the MR job.
{noformat}
@Override
public void initialize(HiveConf conf, QueryPlan queryPlan, DriverContext
driverContext) {
...
conf.stripHiddenConfigurations(job);
this.jobExecHelper = new HadoopJobExecHelper(job, console, this, this);
}
{noformat}
A nice to have (available on hadoop 2.9.0) is an MR
{{mapreduce.job.redacted-properties}} that can be used to hide this list on the
MR side (such as history server UI) to allow MR run the job without issues.
UPDATE:
Change the call to stripHiddenConfigurations() in
ql/exec/tez/DagUtils.createConfiguration(), because this is currently broken
for running hive-blobstore suite against Tez
was:
When S3 credentials are included in hive-site.xml, then MR jobs that need to
read
data from S3 cannot use them because S3 values are stripped from the Job
configuration
object before submitting the MR job.
{noformat}
@Override
public void initialize(HiveConf conf, QueryPlan queryPlan, DriverContext
driverContext) {
...
conf.stripHiddenConfigurations(job);
this.jobExecHelper = new HadoopJobExecHelper(job, console, this, this);
}
{noformat}
A nice to have (available on hadoop 2.9.0) is an MR
{{mapreduce.job.redacted-properties}} that can be used to hide this list on the
MR side (such as history server UI) to allow MR run the job without issues.
> S3 creds added to a hidden list by HIVE-14588 are not working on MR jobs
>
>
> Key: HIVE-14848
> URL: https://issues.apache.org/jira/browse/HIVE-14848
> Project: Hive
> Issue Type: Bug
> Components: Hive
>Affects Versions: 2.2.0
>Reporter: Sergio Peña
>Assignee: Sergio Peña
> Attachments: HIVE-14848.1.patch, HIVE-14848.1.patch
>
>
> When S3 credentials are included in hive-site.xml, then MR jobs that need to
> read
> data from S3 cannot use them because S3 values are stripped from the Job
> configuration
> object before submitting the MR job.
> {noformat}
> @Override
> public void initialize(HiveConf conf, QueryPlan queryPlan, DriverContext
> driverContext) {
> ...
> conf.stripHiddenConfigurations(job);
> this.jobExecHelper = new HadoopJobExecHelper(job, console, this, this);
> }
> {noformat}
> A nice to have (available on hadoop 2.9.0) is an MR
> {{mapreduce.job.redacted-properties}} that can be used to hide this list on
> the MR side (such as history server UI) to allow MR run the job without
> issues.
> UPDATE:
> Change the call to stripHiddenConfigurations() in
> ql/exec/tez/DagUtils.createConfiguration(), because this is currently broken
> for running hive-blobstore suite against Tez
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-14848) S3 creds added to a hidden list by HIVE-14588 are not working on MR jobs
[
https://issues.apache.org/jira/browse/HIVE-14848?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15709802#comment-15709802
]
Sergio Peña commented on HIVE-14848:
Thanks [~poeppt].
> S3 creds added to a hidden list by HIVE-14588 are not working on MR jobs
>
>
> Key: HIVE-14848
> URL: https://issues.apache.org/jira/browse/HIVE-14848
> Project: Hive
> Issue Type: Bug
> Components: Hive
>Affects Versions: 2.2.0
>Reporter: Sergio Peña
>Assignee: Sergio Peña
> Attachments: HIVE-14848.1.patch, HIVE-14848.1.patch
>
>
> When S3 credentials are included in hive-site.xml, then MR jobs that need to
> read
> data from S3 cannot use them because S3 values are stripped from the Job
> configuration
> object before submitting the MR job.
> {noformat}
> @Override
> public void initialize(HiveConf conf, QueryPlan queryPlan, DriverContext
> driverContext) {
> ...
> conf.stripHiddenConfigurations(job);
> this.jobExecHelper = new HadoopJobExecHelper(job, console, this, this);
> }
> {noformat}
> A nice to have (available on hadoop 2.9.0) is an MR
> {{mapreduce.job.redacted-properties}} that can be used to hide this list on
> the MR side (such as history server UI) to allow MR run the job without
> issues.
> UPDATE:
> Change the call to stripHiddenConfigurations() in
> ql/exec/tez/DagUtils.createConfiguration(), because this is currently broken
> for running hive-blobstore suite against Tez
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-15321) Change to read as long for HiveConf.ConfVars.METASTORESERVERMAXMESSAGESIZE
[
https://issues.apache.org/jira/browse/HIVE-15321?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15709808#comment-15709808
]
Hive QA commented on HIVE-15321:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12841123/HIVE-15321.2.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 9 failed/errored test(s), 10735 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[sample2] (batchId=5)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[sample4] (batchId=15)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[sample6] (batchId=60)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[sample7] (batchId=59)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[sample9] (batchId=38)
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[orc_ppd_schema_evol_3a]
(batchId=134)
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[transform_ppr2]
(batchId=134)
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver[explainanalyze_4]
(batchId=92)
org.apache.hadoop.hive.cli.TestSparkCliDriver.org.apache.hadoop.hive.cli.TestSparkCliDriver
(batchId=96)
{noformat}
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/2347/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/2347/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-2347/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 9 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12841123 - PreCommit-HIVE-Build
> Change to read as long for HiveConf.ConfVars.METASTORESERVERMAXMESSAGESIZE
> --
>
> Key: HIVE-15321
> URL: https://issues.apache.org/jira/browse/HIVE-15321
> Project: Hive
> Issue Type: Improvement
> Components: Metastore
>Affects Versions: 1.2.0, 1.1.0
>Reporter: Aihua Xu
>Assignee: Aihua Xu
> Attachments: HIVE-15321.1.patch, HIVE-15321.2.patch
>
>
> Follow up on HIVE-11240 which tries to change the type from int to long,
> while we are still read with {{conf.getIntVar()}}.
> Seems we should use {{conf.getLongVar()}}.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-15227) Optimize join + gby into semijoin
[ https://issues.apache.org/jira/browse/HIVE-15227?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashutosh Chauhan updated HIVE-15227: Resolution: Fixed Fix Version/s: 2.2.0 Status: Resolved (was: Patch Available) > Optimize join + gby into semijoin > - > > Key: HIVE-15227 > URL: https://issues.apache.org/jira/browse/HIVE-15227 > Project: Hive > Issue Type: Bug > Components: Logical Optimizer >Reporter: Ashutosh Chauhan >Assignee: Ashutosh Chauhan > Fix For: 2.2.0 > > Attachments: HIVE-15227.2.patch, HIVE-15227.3.patch, > HIVE-15227.4.patch, HIVE-15227.patch > > > Calcite has a rule which can do this transformation. Lets take advantage of > this since Hive has native Left semi join operator. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-15227) Optimize join + gby into semijoin
[
https://issues.apache.org/jira/browse/HIVE-15227?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15709843#comment-15709843
]
Ashutosh Chauhan commented on HIVE-15227:
-
We can use CALCITE-1494 for changes in this rule. _auto_sortmerge_join_2.q_ is
a flaky test. It fails intermittently even locally. Its a poorly written test
since it depends on memory of system.
By the way, I also added the flag {{hive.enable.semijoin.conversion}} for the
rule.
Pushed to master.
> Optimize join + gby into semijoin
> -
>
> Key: HIVE-15227
> URL: https://issues.apache.org/jira/browse/HIVE-15227
> Project: Hive
> Issue Type: Bug
> Components: Logical Optimizer
>Reporter: Ashutosh Chauhan
>Assignee: Ashutosh Chauhan
> Fix For: 2.2.0
>
> Attachments: HIVE-15227.2.patch, HIVE-15227.3.patch,
> HIVE-15227.4.patch, HIVE-15227.patch
>
>
> Calcite has a rule which can do this transformation. Lets take advantage of
> this since Hive has native Left semi join operator.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Comment Edited] (HIVE-15227) Optimize join + gby into semijoin
[
https://issues.apache.org/jira/browse/HIVE-15227?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15709843#comment-15709843
]
Ashutosh Chauhan edited comment on HIVE-15227 at 11/30/16 9:33 PM:
---
We can use CALCITE-1495 for changes in this rule. _auto_sortmerge_join_2.q_ is
a flaky test. It fails intermittently even locally. Its a poorly written test
since it depends on memory of system.
By the way, I also added the flag {{hive.enable.semijoin.conversion}} for the
rule.
Pushed to master.
was (Author: ashutoshc):
We can use CALCITE-1494 for changes in this rule. _auto_sortmerge_join_2.q_ is
a flaky test. It fails intermittently even locally. Its a poorly written test
since it depends on memory of system.
By the way, I also added the flag {{hive.enable.semijoin.conversion}} for the
rule.
Pushed to master.
> Optimize join + gby into semijoin
> -
>
> Key: HIVE-15227
> URL: https://issues.apache.org/jira/browse/HIVE-15227
> Project: Hive
> Issue Type: Bug
> Components: Logical Optimizer
>Reporter: Ashutosh Chauhan
>Assignee: Ashutosh Chauhan
> Fix For: 2.2.0
>
> Attachments: HIVE-15227.2.patch, HIVE-15227.3.patch,
> HIVE-15227.4.patch, HIVE-15227.patch
>
>
> Calcite has a rule which can do this transformation. Lets take advantage of
> this since Hive has native Left semi join operator.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-15309) Miscellaneous logging clean up
[
https://issues.apache.org/jira/browse/HIVE-15309?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15709872#comment-15709872
]
Wei Zheng commented on HIVE-15309:
--
+1
Maybe it's better to add the check in OrcAcidUtils before "Path lengths =
getSideFile(deltaFile);"?
> Miscellaneous logging clean up
> --
>
> Key: HIVE-15309
> URL: https://issues.apache.org/jira/browse/HIVE-15309
> Project: Hive
> Issue Type: Bug
> Components: Transactions
>Reporter: Eugene Koifman
>Assignee: Eugene Koifman
> Attachments: HIVE-15309.01.patch
>
>
> OrcAcidUtils.getLastFlushLength() should check for file existence first.
> Currently causes unnecessary/confusing logging:
> {noformat}
> org.apache.hadoop.ipc.RemoteException(java.io.FileNotFoundException): File
> does not exist: /domains/adl/rrslog/data_history/rrslog/r\
> rslog/hot/server_date=2016-08-19/delta_0005913_0005913/bucket_00023_flush_length
> at
> org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:71)
> at
> org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:61)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocationsInt(FSNamesystem.java:1860)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:1831)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:1744)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.getBlockLocations(NameNodeRpcServer.java:693)
> at
> org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.getBlockLocations(ClientNamenodeProtocolSe\
> rverSideTranslatorPB.java:373)
> at
> org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientName\
> nodeProtocolProtos.java)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:640)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:982)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2313)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2309)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1724)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2307)
> at org.apache.hadoop.ipc.Client.getRpcResponse(Client.java:1552)
> at org.apache.hadoop.ipc.Client.call(Client.java:1496)
> at org.apache.hadoop.ipc.Client.call(Client.java:1396)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:233)
> at com.sun.proxy.$Proxy14.getBlockLocations(Unknown Source)
> at
> org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.getBlockLocations(ClientNamenodeProtocolTranslatorPB\
> .java:270)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:497)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:278)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:194)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:176)
> at com.sun.proxy.$Proxy15.getBlockLocations(Unknown Source)
> at
> org.apache.hadoop.hdfs.DFSClient.callGetBlockLocations(DFSClient.java:1236)
> at
> org.apache.hadoop.hdfs.DFSClient.getLocatedBlocks(DFSClient.java:1223)
> at
> org.apache.hadoop.hdfs.DFSClient.getLocatedBlocks(DFSClient.java:1211)
> at
> org.apache.hadoop.hdfs.DFSInputStream.fetchLocatedBlocksAndGetLastBlockLength(DFSInputStream.java:309)
> at
> org.apache.hadoop.hdfs.DFSInputStream.openInfo(DFSInputStream.java:274)
> at
> org.apache.hadoop.hdfs.DFSInputStream.(DFSInputStream.java:266)
> at org.apache.hadoop.hdfs.DFSClient.open(DFSClient.java:1536)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem$4.doCall(DistributedFileSystem.java:330)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem$4.doCall(DistributedFileSystem.java:326)
> at
> org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem.open(DistributedFileSystem.java:326)
>
[jira] [Updated] (HIVE-15057) Support other types of operators (other than SELECT)
[ https://issues.apache.org/jira/browse/HIVE-15057?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chao Sun updated HIVE-15057: Attachment: (was: HIVE-15057.wip.patch) > Support other types of operators (other than SELECT) > > > Key: HIVE-15057 > URL: https://issues.apache.org/jira/browse/HIVE-15057 > Project: Hive > Issue Type: Sub-task > Components: Logical Optimizer, Physical Optimizer >Reporter: Chao Sun >Assignee: Chao Sun > Attachments: HIVE-15057.1.patch > > > Currently only SELECT operators are supported for nested column pruning. We > should add support for other types of operators so the optimization can work > for complex queries. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-15057) Support other types of operators (other than SELECT)
[ https://issues.apache.org/jira/browse/HIVE-15057?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chao Sun updated HIVE-15057: Attachment: HIVE-15057.1.patch > Support other types of operators (other than SELECT) > > > Key: HIVE-15057 > URL: https://issues.apache.org/jira/browse/HIVE-15057 > Project: Hive > Issue Type: Sub-task > Components: Logical Optimizer, Physical Optimizer >Reporter: Chao Sun >Assignee: Chao Sun > Attachments: HIVE-15057.1.patch > > > Currently only SELECT operators are supported for nested column pruning. We > should add support for other types of operators so the optimization can work > for complex queries. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-15309) Miscellaneous logging clean up
[
https://issues.apache.org/jira/browse/HIVE-15309?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15709916#comment-15709916
]
Eugene Koifman commented on HIVE-15309:
---
doh! no, because it should be checking existence of "lengths" file
> Miscellaneous logging clean up
> --
>
> Key: HIVE-15309
> URL: https://issues.apache.org/jira/browse/HIVE-15309
> Project: Hive
> Issue Type: Bug
> Components: Transactions
>Reporter: Eugene Koifman
>Assignee: Eugene Koifman
> Attachments: HIVE-15309.01.patch
>
>
> OrcAcidUtils.getLastFlushLength() should check for file existence first.
> Currently causes unnecessary/confusing logging:
> {noformat}
> org.apache.hadoop.ipc.RemoteException(java.io.FileNotFoundException): File
> does not exist: /domains/adl/rrslog/data_history/rrslog/r\
> rslog/hot/server_date=2016-08-19/delta_0005913_0005913/bucket_00023_flush_length
> at
> org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:71)
> at
> org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:61)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocationsInt(FSNamesystem.java:1860)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:1831)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:1744)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.getBlockLocations(NameNodeRpcServer.java:693)
> at
> org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.getBlockLocations(ClientNamenodeProtocolSe\
> rverSideTranslatorPB.java:373)
> at
> org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientName\
> nodeProtocolProtos.java)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:640)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:982)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2313)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2309)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1724)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2307)
> at org.apache.hadoop.ipc.Client.getRpcResponse(Client.java:1552)
> at org.apache.hadoop.ipc.Client.call(Client.java:1496)
> at org.apache.hadoop.ipc.Client.call(Client.java:1396)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:233)
> at com.sun.proxy.$Proxy14.getBlockLocations(Unknown Source)
> at
> org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.getBlockLocations(ClientNamenodeProtocolTranslatorPB\
> .java:270)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:497)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:278)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:194)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:176)
> at com.sun.proxy.$Proxy15.getBlockLocations(Unknown Source)
> at
> org.apache.hadoop.hdfs.DFSClient.callGetBlockLocations(DFSClient.java:1236)
> at
> org.apache.hadoop.hdfs.DFSClient.getLocatedBlocks(DFSClient.java:1223)
> at
> org.apache.hadoop.hdfs.DFSClient.getLocatedBlocks(DFSClient.java:1211)
> at
> org.apache.hadoop.hdfs.DFSInputStream.fetchLocatedBlocksAndGetLastBlockLength(DFSInputStream.java:309)
> at
> org.apache.hadoop.hdfs.DFSInputStream.openInfo(DFSInputStream.java:274)
> at
> org.apache.hadoop.hdfs.DFSInputStream.(DFSInputStream.java:266)
> at org.apache.hadoop.hdfs.DFSClient.open(DFSClient.java:1536)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem$4.doCall(DistributedFileSystem.java:330)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem$4.doCall(DistributedFileSystem.java:326)
> at
> org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem.open(DistributedFileSystem.java:326)
> at org.apache.hadoop.fs.F
[jira] [Updated] (HIVE-15309) Miscellaneous logging clean up
[
https://issues.apache.org/jira/browse/HIVE-15309?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Eugene Koifman updated HIVE-15309:
--
Attachment: HIVE-15309.02.patch
> Miscellaneous logging clean up
> --
>
> Key: HIVE-15309
> URL: https://issues.apache.org/jira/browse/HIVE-15309
> Project: Hive
> Issue Type: Bug
> Components: Transactions
>Reporter: Eugene Koifman
>Assignee: Eugene Koifman
> Attachments: HIVE-15309.01.patch, HIVE-15309.02.patch
>
>
> OrcAcidUtils.getLastFlushLength() should check for file existence first.
> Currently causes unnecessary/confusing logging:
> {noformat}
> org.apache.hadoop.ipc.RemoteException(java.io.FileNotFoundException): File
> does not exist: /domains/adl/rrslog/data_history/rrslog/r\
> rslog/hot/server_date=2016-08-19/delta_0005913_0005913/bucket_00023_flush_length
> at
> org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:71)
> at
> org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:61)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocationsInt(FSNamesystem.java:1860)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:1831)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:1744)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.getBlockLocations(NameNodeRpcServer.java:693)
> at
> org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.getBlockLocations(ClientNamenodeProtocolSe\
> rverSideTranslatorPB.java:373)
> at
> org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientName\
> nodeProtocolProtos.java)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:640)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:982)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2313)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2309)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1724)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2307)
> at org.apache.hadoop.ipc.Client.getRpcResponse(Client.java:1552)
> at org.apache.hadoop.ipc.Client.call(Client.java:1496)
> at org.apache.hadoop.ipc.Client.call(Client.java:1396)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:233)
> at com.sun.proxy.$Proxy14.getBlockLocations(Unknown Source)
> at
> org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.getBlockLocations(ClientNamenodeProtocolTranslatorPB\
> .java:270)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:497)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:278)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:194)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:176)
> at com.sun.proxy.$Proxy15.getBlockLocations(Unknown Source)
> at
> org.apache.hadoop.hdfs.DFSClient.callGetBlockLocations(DFSClient.java:1236)
> at
> org.apache.hadoop.hdfs.DFSClient.getLocatedBlocks(DFSClient.java:1223)
> at
> org.apache.hadoop.hdfs.DFSClient.getLocatedBlocks(DFSClient.java:1211)
> at
> org.apache.hadoop.hdfs.DFSInputStream.fetchLocatedBlocksAndGetLastBlockLength(DFSInputStream.java:309)
> at
> org.apache.hadoop.hdfs.DFSInputStream.openInfo(DFSInputStream.java:274)
> at
> org.apache.hadoop.hdfs.DFSInputStream.(DFSInputStream.java:266)
> at org.apache.hadoop.hdfs.DFSClient.open(DFSClient.java:1536)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem$4.doCall(DistributedFileSystem.java:330)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem$4.doCall(DistributedFileSystem.java:326)
> at
> org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem.open(DistributedFileSystem.java:326)
> at org.apache.hadoop.fs.FileSystem.open(FileSystem.java:782)
> at
> org.apache.hadoop
[jira] [Commented] (HIVE-15309) Miscellaneous logging clean up
[
https://issues.apache.org/jira/browse/HIVE-15309?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15709971#comment-15709971
]
Wei Zheng commented on HIVE-15309:
--
+1 for patch 2
> Miscellaneous logging clean up
> --
>
> Key: HIVE-15309
> URL: https://issues.apache.org/jira/browse/HIVE-15309
> Project: Hive
> Issue Type: Bug
> Components: Transactions
>Reporter: Eugene Koifman
>Assignee: Eugene Koifman
> Attachments: HIVE-15309.01.patch, HIVE-15309.02.patch
>
>
> OrcAcidUtils.getLastFlushLength() should check for file existence first.
> Currently causes unnecessary/confusing logging:
> {noformat}
> org.apache.hadoop.ipc.RemoteException(java.io.FileNotFoundException): File
> does not exist: /domains/adl/rrslog/data_history/rrslog/r\
> rslog/hot/server_date=2016-08-19/delta_0005913_0005913/bucket_00023_flush_length
> at
> org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:71)
> at
> org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:61)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocationsInt(FSNamesystem.java:1860)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:1831)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:1744)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.getBlockLocations(NameNodeRpcServer.java:693)
> at
> org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.getBlockLocations(ClientNamenodeProtocolSe\
> rverSideTranslatorPB.java:373)
> at
> org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientName\
> nodeProtocolProtos.java)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:640)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:982)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2313)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2309)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1724)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2307)
> at org.apache.hadoop.ipc.Client.getRpcResponse(Client.java:1552)
> at org.apache.hadoop.ipc.Client.call(Client.java:1496)
> at org.apache.hadoop.ipc.Client.call(Client.java:1396)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:233)
> at com.sun.proxy.$Proxy14.getBlockLocations(Unknown Source)
> at
> org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.getBlockLocations(ClientNamenodeProtocolTranslatorPB\
> .java:270)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:497)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:278)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:194)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:176)
> at com.sun.proxy.$Proxy15.getBlockLocations(Unknown Source)
> at
> org.apache.hadoop.hdfs.DFSClient.callGetBlockLocations(DFSClient.java:1236)
> at
> org.apache.hadoop.hdfs.DFSClient.getLocatedBlocks(DFSClient.java:1223)
> at
> org.apache.hadoop.hdfs.DFSClient.getLocatedBlocks(DFSClient.java:1211)
> at
> org.apache.hadoop.hdfs.DFSInputStream.fetchLocatedBlocksAndGetLastBlockLength(DFSInputStream.java:309)
> at
> org.apache.hadoop.hdfs.DFSInputStream.openInfo(DFSInputStream.java:274)
> at
> org.apache.hadoop.hdfs.DFSInputStream.(DFSInputStream.java:266)
> at org.apache.hadoop.hdfs.DFSClient.open(DFSClient.java:1536)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem$4.doCall(DistributedFileSystem.java:330)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem$4.doCall(DistributedFileSystem.java:326)
> at
> org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem.open(DistributedFileSystem.java:326)
> at org.apache.hadoop.fs.FileSystem.open(FileSystem.java:782)
>
[jira] [Resolved] (HIVE-15318) Query "insert into table values()" creates the tmp table under the current database
[
https://issues.apache.org/jira/browse/HIVE-15318?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Aihua Xu resolved HIVE-15318.
-
Resolution: Duplicate
Assignee: (was: Aihua Xu)
> Query "insert into table values()" creates the tmp table under the current
> database
> ---
>
> Key: HIVE-15318
> URL: https://issues.apache.org/jira/browse/HIVE-15318
> Project: Hive
> Issue Type: Bug
> Components: Query Planning
>Affects Versions: 2.2.0
>Reporter: Aihua Xu
>
> The current implementation of "insert into db1.table1 values()" creates a tmp
> table under the current database while table1 may not be under current
> database.
> e.g.,
> {noformat}
> use default;
> create database db1;
> create table db1.table1(x int);
> insert into db1.table1 values(3);
> {noformat}
> It will create the tmp table under default database. Now if authorization is
> turned on and the current user only has access to db1 but not default
> database, then it will cause access issue.
> We may need to rethink the approach for the implementation.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-14582) Add trunc(numeric) udf
[ https://issues.apache.org/jira/browse/HIVE-14582?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashutosh Chauhan updated HIVE-14582: Resolution: Fixed Fix Version/s: 2.2.0 Status: Resolved (was: Patch Available) Pushed to master. Thanks, [~chinnalalam] > Add trunc(numeric) udf > -- > > Key: HIVE-14582 > URL: https://issues.apache.org/jira/browse/HIVE-14582 > Project: Hive > Issue Type: Sub-task > Components: SQL >Reporter: Ashutosh Chauhan >Assignee: Chinna Rao Lalam > Fix For: 2.2.0 > > Attachments: HIVE-14582.1.patch, HIVE-14582.2.patch, > HIVE-14582.3.patch, HIVE-14582.4.patch, HIVE-14582.patch > > > https://docs.oracle.com/cd/B19306_01/server.102/b14200/functions200.htm -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-15057) Support other types of operators (other than SELECT)
[
https://issues.apache.org/jira/browse/HIVE-15057?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15710132#comment-15710132
]
Hive QA commented on HIVE-15057:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12841141/HIVE-15057.1.patch
{color:green}SUCCESS:{color} +1 due to 3 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 16 failed/errored test(s), 10753 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.org.apache.hadoop.hive.cli.TestCliDriver
(batchId=50)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[index_auto_mult_tables_compact]
(batchId=32)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[nested_column_pruning]
(batchId=31)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[sample2] (batchId=5)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[sample4] (batchId=15)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[sample6] (batchId=60)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[sample7] (batchId=60)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[sample9] (batchId=38)
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[transform_ppr2]
(batchId=134)
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver[explainanalyze_1]
(batchId=91)
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver[explainanalyze_2]
(batchId=92)
org.apache.hadoop.hive.ql.io.parquet.read.TestDataWritableReadSupport.testGetProjectedSchema1
(batchId=247)
org.apache.hadoop.hive.ql.io.parquet.read.TestDataWritableReadSupport.testGetProjectedSchema2
(batchId=247)
org.apache.hadoop.hive.ql.io.parquet.read.TestDataWritableReadSupport.testGetProjectedSchema3
(batchId=247)
org.apache.hadoop.hive.ql.io.parquet.read.TestDataWritableReadSupport.testGetProjectedSchema4
(batchId=247)
org.apache.hadoop.hive.ql.io.parquet.read.TestDataWritableReadSupport.testGetProjectedSchema5
(batchId=247)
{noformat}
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/2348/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/2348/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-2348/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 16 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12841141 - PreCommit-HIVE-Build
> Support other types of operators (other than SELECT)
>
>
> Key: HIVE-15057
> URL: https://issues.apache.org/jira/browse/HIVE-15057
> Project: Hive
> Issue Type: Sub-task
> Components: Logical Optimizer, Physical Optimizer
>Reporter: Chao Sun
>Assignee: Chao Sun
> Attachments: HIVE-15057.1.patch
>
>
> Currently only SELECT operators are supported for nested column pruning. We
> should add support for other types of operators so the optimization can work
> for complex queries.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-15296) AM may lose task failures and not reschedule when scheduling to LLAP
[
https://issues.apache.org/jira/browse/HIVE-15296?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sergey Shelukhin updated HIVE-15296:
Attachment: HIVE-15296.patch
Not 100% sure of the root cause.
1) The difference in heartbeat mechanisms before and after the task is running
is actually irrelevant, since it is only present on LLAP side.
2) The tracking and killing itself is done by Tez AM. Unless there's a bug in
Tez, which I cannot see in the code, it should work.
The problem must be in the part that connects these together. Upon examination,
I wonder if having heartbeats from the node (host+port) blindly heartbeat all
the tasks that AM *thinks* are on the node is the culrpit? The restarted LLAP
can heartbeat in time.
This patch propagates unique daemonId usage to node heartbeats (and submit task
response, to know the original unique node ID for the task in the first place).
I left some TODO comments to improve the handling for this, but for now I'd
rather not rewrite EntityTracker entirely to use unique ID everywhere.
[~sseth] [~prasanth_j] can you take a look?
> AM may lose task failures and not reschedule when scheduling to LLAP
>
>
> Key: HIVE-15296
> URL: https://issues.apache.org/jira/browse/HIVE-15296
> Project: Hive
> Issue Type: Bug
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
> Attachments: HIVE-15296.patch
>
>
> First attempt and failure detection:
> {noformat}
> 2016-11-18 20:20:01,980 [INFO] [TaskSchedulerEventHandlerThread]
> |tezplugins.LlapTaskSchedulerService|: Received allocateRequest.
> task=attempt_1478967587833_2622_1_06_31_0, priority=65,
> capability=, hosts=[3n01]
> 2016-11-18 20:20:01,982 [INFO] [LlapScheduler]
> |tezplugins.LlapTaskSchedulerService|: Assigned task
> TaskInfo{task=attempt_1478967587833_2622_1_06_31_0, priority=65,
> startTime=0, containerId=null, assignedInstance=null, uniqueId=55,
> localityDelayTimeout=9223372036854775807} to container
> container_1_2622_01_56 on node=DynamicServiceInstance
> [alive=true, host=3n01:15001 with resources=,
> shufflePort=15551, servicesAddress=http://3n01:15002, mgmtPort=15004]
> 2016-11-18 20:20:01,982 [INFO] [LlapScheduler]
> |tezplugins.LlapTaskSchedulerService|: ScheduleResult for Task:
> TaskInfo{task=attempt_1478967587833_2622_1_06_31_0, priority=65,
> startTime=10550817928, containerId=container_1_2622_01_56,
> assignedInstance=DynamicServiceInstance [alive=true, host=3n01:15001 with
> resources=, shufflePort=15551,
> servicesAddress=http://3n01:15002, mgmtPort=15004], uniqueId=55,
> localityDelayTimeout=9223372036854775807} = SCHEDULED
> 2016-11-18 20:20:03,427 [INFO] [Dispatcher thread {Central}]
> |impl.TaskAttemptImpl|: TaskAttempt:
> [attempt_1478967587833_2622_1_06_31_0] started. Is using containerId:
> [container_1_2622_01_56] on NM: [3n01:15001]
> 2016-11-18 20:20:03,427 [INFO] [Dispatcher thread {Central}]
> |history.HistoryEventHandler|:
> [HISTORY][DAG:dag_1478967587833_2622_1][Event:TASK_ATTEMPT_STARTED]:
> vertexName=Map 1, taskAttemptId=attempt_1478967587833_2622_1_06_31_0,
> startTime=1479500403427, containerId=container_1_2622_01_56,
> nodeId=3n01:15001
> 2016-11-18 20:20:03,430 [INFO] [TaskCommunicator # 1]
> |tezplugins.LlapTaskCommunicator|: Successfully launched task:
> attempt_1478967587833_2622_1_06_31_0
> 2016-11-18 20:20:03,434 [INFO] [IPC Server handler 11 on 43092]
> |impl.TaskImpl|: TaskAttempt:attempt_1478967587833_2622_1_06_31_0 sent
> events: (0-1).
> 2016-11-18 20:20:03,434 [INFO] [IPC Server handler 11 on 43092]
> |impl.VertexImpl|: Sending attempt_1478967587833_2622_1_06_31_0 24 events
> [0,24) total 24 vertex_1478967587833_2622_1_06 [Map 1]
> 2016-11-18 20:25:43,249 [INFO] [Dispatcher thread {Central}]
> |history.HistoryEventHandler|:
> [HISTORY][DAG:dag_1478967587833_2622_1][Event:TASK_ATTEMPT_FINISHED]:
> vertexName=Map 1, taskAttemptId=attempt_1478967587833_2622_1_06_31_0,
> creationTime=1479500401929, allocationTime=1479500403426,
> startTime=1479500403427, finishTime=1479500743249, timeTaken=339822,
> status=FAILED, taskFailureType=NON_FATAL, errorEnum=TASK_HEARTBEAT_ERROR,
> diagnostics=AttemptID:attempt_1478967587833_2622_1_06_31_0 Timed out
> after 300 secs, nodeHttpAddress=http://3n01:15002, counters=Counters: 1,
> org.apache.tez.common.counters.DAGCounter, DATA_LOCAL_TASKS=1
> 2016-11-18 20:25:43,255 [INFO] [TaskSchedulerEventHandlerThread]
> |tezplugins.LlapTaskSchedulerService|: Processing de-allocate request for
> task=attempt_1478967587833_2622_1_06_31_0, state=ASSIGNED, endReason=OTHER
> 2016-11-18 20:25:43,259 [INFO] [Dispatcher
[jira] [Updated] (HIVE-15296) AM may lose task failures and not reschedule when scheduling to LLAP
[
https://issues.apache.org/jira/browse/HIVE-15296?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sergey Shelukhin updated HIVE-15296:
Status: Patch Available (was: Open)
> AM may lose task failures and not reschedule when scheduling to LLAP
>
>
> Key: HIVE-15296
> URL: https://issues.apache.org/jira/browse/HIVE-15296
> Project: Hive
> Issue Type: Bug
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
> Attachments: HIVE-15296.patch
>
>
> First attempt and failure detection:
> {noformat}
> 2016-11-18 20:20:01,980 [INFO] [TaskSchedulerEventHandlerThread]
> |tezplugins.LlapTaskSchedulerService|: Received allocateRequest.
> task=attempt_1478967587833_2622_1_06_31_0, priority=65,
> capability=, hosts=[3n01]
> 2016-11-18 20:20:01,982 [INFO] [LlapScheduler]
> |tezplugins.LlapTaskSchedulerService|: Assigned task
> TaskInfo{task=attempt_1478967587833_2622_1_06_31_0, priority=65,
> startTime=0, containerId=null, assignedInstance=null, uniqueId=55,
> localityDelayTimeout=9223372036854775807} to container
> container_1_2622_01_56 on node=DynamicServiceInstance
> [alive=true, host=3n01:15001 with resources=,
> shufflePort=15551, servicesAddress=http://3n01:15002, mgmtPort=15004]
> 2016-11-18 20:20:01,982 [INFO] [LlapScheduler]
> |tezplugins.LlapTaskSchedulerService|: ScheduleResult for Task:
> TaskInfo{task=attempt_1478967587833_2622_1_06_31_0, priority=65,
> startTime=10550817928, containerId=container_1_2622_01_56,
> assignedInstance=DynamicServiceInstance [alive=true, host=3n01:15001 with
> resources=, shufflePort=15551,
> servicesAddress=http://3n01:15002, mgmtPort=15004], uniqueId=55,
> localityDelayTimeout=9223372036854775807} = SCHEDULED
> 2016-11-18 20:20:03,427 [INFO] [Dispatcher thread {Central}]
> |impl.TaskAttemptImpl|: TaskAttempt:
> [attempt_1478967587833_2622_1_06_31_0] started. Is using containerId:
> [container_1_2622_01_56] on NM: [3n01:15001]
> 2016-11-18 20:20:03,427 [INFO] [Dispatcher thread {Central}]
> |history.HistoryEventHandler|:
> [HISTORY][DAG:dag_1478967587833_2622_1][Event:TASK_ATTEMPT_STARTED]:
> vertexName=Map 1, taskAttemptId=attempt_1478967587833_2622_1_06_31_0,
> startTime=1479500403427, containerId=container_1_2622_01_56,
> nodeId=3n01:15001
> 2016-11-18 20:20:03,430 [INFO] [TaskCommunicator # 1]
> |tezplugins.LlapTaskCommunicator|: Successfully launched task:
> attempt_1478967587833_2622_1_06_31_0
> 2016-11-18 20:20:03,434 [INFO] [IPC Server handler 11 on 43092]
> |impl.TaskImpl|: TaskAttempt:attempt_1478967587833_2622_1_06_31_0 sent
> events: (0-1).
> 2016-11-18 20:20:03,434 [INFO] [IPC Server handler 11 on 43092]
> |impl.VertexImpl|: Sending attempt_1478967587833_2622_1_06_31_0 24 events
> [0,24) total 24 vertex_1478967587833_2622_1_06 [Map 1]
> 2016-11-18 20:25:43,249 [INFO] [Dispatcher thread {Central}]
> |history.HistoryEventHandler|:
> [HISTORY][DAG:dag_1478967587833_2622_1][Event:TASK_ATTEMPT_FINISHED]:
> vertexName=Map 1, taskAttemptId=attempt_1478967587833_2622_1_06_31_0,
> creationTime=1479500401929, allocationTime=1479500403426,
> startTime=1479500403427, finishTime=1479500743249, timeTaken=339822,
> status=FAILED, taskFailureType=NON_FATAL, errorEnum=TASK_HEARTBEAT_ERROR,
> diagnostics=AttemptID:attempt_1478967587833_2622_1_06_31_0 Timed out
> after 300 secs, nodeHttpAddress=http://3n01:15002, counters=Counters: 1,
> org.apache.tez.common.counters.DAGCounter, DATA_LOCAL_TASKS=1
> 2016-11-18 20:25:43,255 [INFO] [TaskSchedulerEventHandlerThread]
> |tezplugins.LlapTaskSchedulerService|: Processing de-allocate request for
> task=attempt_1478967587833_2622_1_06_31_0, state=ASSIGNED, endReason=OTHER
> 2016-11-18 20:25:43,259 [INFO] [Dispatcher thread {Central}]
> |node.AMNodeImpl|: Attempt failed on node: 3n01:15001 TA:
> attempt_1478967587833_2622_1_06_31_0 failed: true container:
> container_1_2622_01_56 numFailedTAs: 7
> 2016-11-18 20:25:43,262 [INFO] [Dispatcher thread {Central}]
> |impl.VertexImpl|: Source task attempt completed for vertex:
> vertex_1478967587833_2622_1_07 [Reducer 2] attempt:
> attempt_1478967587833_2622_1_06_31_0 with state: FAILED vertexState:
> RUNNING
> {noformat}
> Second attempt:
> {noformat}
> 2016-11-18 20:25:43,267 [INFO] [TaskSchedulerEventHandlerThread]
> |tezplugins.LlapTaskSchedulerService|: Received allocateRequest.
> task=attempt_1478967587833_2622_1_06_31_1, priority=64,
> capability=, hosts=null
> 2016-11-18 20:25:43,297 [INFO] [LlapScheduler]
> |tezplugins.LlapTaskSchedulerService|: ScheduleResult for Task:
> TaskInfo{task=attempt_1478967587833_26
[jira] [Updated] (HIVE-15057) Support other types of operators (other than SELECT)
[ https://issues.apache.org/jira/browse/HIVE-15057?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chao Sun updated HIVE-15057: Attachment: HIVE-15057.2.patch > Support other types of operators (other than SELECT) > > > Key: HIVE-15057 > URL: https://issues.apache.org/jira/browse/HIVE-15057 > Project: Hive > Issue Type: Sub-task > Components: Logical Optimizer, Physical Optimizer >Reporter: Chao Sun >Assignee: Chao Sun > Attachments: HIVE-15057.1.patch, HIVE-15057.2.patch > > > Currently only SELECT operators are supported for nested column pruning. We > should add support for other types of operators so the optimization can work > for complex queries. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-15309) Miscellaneous logging clean up
[
https://issues.apache.org/jira/browse/HIVE-15309?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15710247#comment-15710247
]
Hive QA commented on HIVE-15309:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12841148/HIVE-15309.02.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 209 failed/errored test(s), 10752 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.org.apache.hadoop.hive.cli.TestCliDriver
(batchId=50)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[acid_join] (batchId=14)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[acid_mapjoin]
(batchId=10)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[acid_table_stats]
(batchId=48)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[acid_vectorization]
(batchId=58)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[acid_vectorization_partition]
(batchId=65)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[acid_vectorization_project]
(batchId=18)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[autoColumnStats_4]
(batchId=11)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[delete_all_non_partitioned]
(batchId=26)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[delete_all_partitioned]
(batchId=26)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[delete_orig_table]
(batchId=36)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[delete_tmp_table]
(batchId=46)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[delete_where_no_match]
(batchId=26)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[delete_where_non_partitioned]
(batchId=35)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[delete_where_partitioned]
(batchId=36)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[delete_whole_partition]
(batchId=9)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[insert_acid_dynamic_partition]
(batchId=18)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[insert_nonacid_from_acid]
(batchId=66)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[insert_orig_table]
(batchId=56)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[insert_update_delete]
(batchId=76)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[insert_values_dynamic_partitioned]
(batchId=68)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[insert_values_non_partitioned]
(batchId=18)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[insert_values_orig_table]
(batchId=51)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[insert_values_orig_table_use_metadata]
(batchId=57)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[insert_values_partitioned]
(batchId=68)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[insert_values_tmp_table]
(batchId=4)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[llap_acid] (batchId=72)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[orc_ppd_exception]
(batchId=30)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[sample2] (batchId=5)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[sample4] (batchId=15)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[sample6] (batchId=60)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[sample7] (batchId=60)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[sample9] (batchId=38)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[transform_acid]
(batchId=18)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[update_after_multiple_inserts]
(batchId=61)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[update_after_multiple_inserts_special_characters]
(batchId=65)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[update_all_non_partitioned]
(batchId=7)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[update_all_partitioned]
(batchId=46)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[update_all_types]
(batchId=16)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[update_orig_table]
(batchId=55)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[update_tmp_table]
(batchId=32)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[update_two_cols]
(batchId=19)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[update_where_no_match]
(batchId=18)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[update_where_non_partitioned]
(batchId=14)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[update_where_partitioned]
(batchId=56)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[vector_acid3]
(batchId=25)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[vector_join_part_col_char]
(batchId=21)
org.apache.hadoop.hive.cli.TestEncryptedHDFSCliDriver.testCliDriver[encryption_insert_partition_dynamic]
(batchId=155)
org.apache.hadoop.hive.cli.Te
[jira] [Commented] (HIVE-15276) CLIs spell "substitution" as "subsitution" and "auxiliary" as "auxillary"
[ https://issues.apache.org/jira/browse/HIVE-15276?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15710255#comment-15710255 ] Grant Sohn commented on HIVE-15276: --- I tried but don't have rights to change the wiki. Do I need to go through http://www.apache.org/dev/cmsref.html#non-committer to submit the change? Thanks. > CLIs spell "substitution" as "subsitution" and "auxiliary" as "auxillary" > - > > Key: HIVE-15276 > URL: https://issues.apache.org/jira/browse/HIVE-15276 > Project: Hive > Issue Type: Bug > Components: CLI >Affects Versions: 1.1.0 >Reporter: Grant Sohn >Assignee: Grant Sohn >Priority: Trivial > Fix For: 2.2.0 > > Attachments: HIVE-15276.1.patch, HIVE-15276.2.patch, > HIVE-15276.3.patch, HIVE-15276.4.patch, HIVE-15276.5.patch > > > Found some obvious spelling typos in the CLI help. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-15309) Miscellaneous logging clean up
[
https://issues.apache.org/jira/browse/HIVE-15309?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Eugene Koifman updated HIVE-15309:
--
Attachment: HIVE-15309.03.patch
> Miscellaneous logging clean up
> --
>
> Key: HIVE-15309
> URL: https://issues.apache.org/jira/browse/HIVE-15309
> Project: Hive
> Issue Type: Bug
> Components: Transactions
>Reporter: Eugene Koifman
>Assignee: Eugene Koifman
> Attachments: HIVE-15309.01.patch, HIVE-15309.02.patch,
> HIVE-15309.03.patch
>
>
> OrcAcidUtils.getLastFlushLength() should check for file existence first.
> Currently causes unnecessary/confusing logging:
> {noformat}
> org.apache.hadoop.ipc.RemoteException(java.io.FileNotFoundException): File
> does not exist: /domains/adl/rrslog/data_history/rrslog/r\
> rslog/hot/server_date=2016-08-19/delta_0005913_0005913/bucket_00023_flush_length
> at
> org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:71)
> at
> org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:61)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocationsInt(FSNamesystem.java:1860)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:1831)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:1744)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.getBlockLocations(NameNodeRpcServer.java:693)
> at
> org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.getBlockLocations(ClientNamenodeProtocolSe\
> rverSideTranslatorPB.java:373)
> at
> org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientName\
> nodeProtocolProtos.java)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:640)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:982)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2313)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2309)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1724)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2307)
> at org.apache.hadoop.ipc.Client.getRpcResponse(Client.java:1552)
> at org.apache.hadoop.ipc.Client.call(Client.java:1496)
> at org.apache.hadoop.ipc.Client.call(Client.java:1396)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:233)
> at com.sun.proxy.$Proxy14.getBlockLocations(Unknown Source)
> at
> org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.getBlockLocations(ClientNamenodeProtocolTranslatorPB\
> .java:270)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:497)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:278)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:194)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:176)
> at com.sun.proxy.$Proxy15.getBlockLocations(Unknown Source)
> at
> org.apache.hadoop.hdfs.DFSClient.callGetBlockLocations(DFSClient.java:1236)
> at
> org.apache.hadoop.hdfs.DFSClient.getLocatedBlocks(DFSClient.java:1223)
> at
> org.apache.hadoop.hdfs.DFSClient.getLocatedBlocks(DFSClient.java:1211)
> at
> org.apache.hadoop.hdfs.DFSInputStream.fetchLocatedBlocksAndGetLastBlockLength(DFSInputStream.java:309)
> at
> org.apache.hadoop.hdfs.DFSInputStream.openInfo(DFSInputStream.java:274)
> at
> org.apache.hadoop.hdfs.DFSInputStream.(DFSInputStream.java:266)
> at org.apache.hadoop.hdfs.DFSClient.open(DFSClient.java:1536)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem$4.doCall(DistributedFileSystem.java:330)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem$4.doCall(DistributedFileSystem.java:326)
> at
> org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem.open(DistributedFileSystem.java:326)
> at org.apache.hadoop.fs.FileSystem.open(FileSystem.java:782)
>
[jira] [Comment Edited] (HIVE-15202) Concurrent compactions for the same partition may generate malformed folder structure
[
https://issues.apache.org/jira/browse/HIVE-15202?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15709447#comment-15709447
]
Eugene Koifman edited comment on HIVE-15202 at 12/1/16 12:23 AM:
-
It's not. The cleaner can only remove files that are obsolete, i.e. there is
another (better/wider) delta/base that includes the same data. No reader
(including compaction) reads obsolete data.
AcidUtils.getAcidState() encapsulates this
was (Author: ekoifman):
It's not. The cleaner can only remove files that are obsolete, i.e. there is
another (better/wider) delta/base that includes the same data. No reader
(including compaction) reads obsolete data
> Concurrent compactions for the same partition may generate malformed folder
> structure
> -
>
> Key: HIVE-15202
> URL: https://issues.apache.org/jira/browse/HIVE-15202
> Project: Hive
> Issue Type: Bug
> Components: Transactions
>Reporter: Rui Li
>Assignee: Eugene Koifman
> Attachments: HIVE-15202.01.patch, HIVE-15202.02.patch,
> HIVE-15202.03.patch
>
>
> If two compactions run concurrently on a single partition, it may generate
> folder structure like this: (nested base dir)
> {noformat}
> drwxr-xr-x - root supergroup 0 2016-11-14 22:23
> /user/hive/warehouse/test/z=1/base_007/base_007
> -rw-r--r-- 3 root supergroup201 2016-11-14 21:46
> /user/hive/warehouse/test/z=1/base_007/bucket_0
> -rw-r--r-- 3 root supergroup611 2016-11-14 21:46
> /user/hive/warehouse/test/z=1/base_007/bucket_1
> -rw-r--r-- 3 root supergroup614 2016-11-14 21:46
> /user/hive/warehouse/test/z=1/base_007/bucket_2
> -rw-r--r-- 3 root supergroup621 2016-11-14 21:46
> /user/hive/warehouse/test/z=1/base_007/bucket_3
> -rw-r--r-- 3 root supergroup621 2016-11-14 21:46
> /user/hive/warehouse/test/z=1/base_007/bucket_4
> -rw-r--r-- 3 root supergroup201 2016-11-14 21:46
> /user/hive/warehouse/test/z=1/base_007/bucket_5
> -rw-r--r-- 3 root supergroup201 2016-11-14 21:46
> /user/hive/warehouse/test/z=1/base_007/bucket_6
> -rw-r--r-- 3 root supergroup201 2016-11-14 21:46
> /user/hive/warehouse/test/z=1/base_007/bucket_7
> -rw-r--r-- 3 root supergroup201 2016-11-14 21:46
> /user/hive/warehouse/test/z=1/base_007/bucket_8
> -rw-r--r-- 3 root supergroup201 2016-11-14 21:46
> /user/hive/warehouse/test/z=1/base_007/bucket_9
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-15321) Change to read as long for HiveConf.ConfVars.METASTORESERVERMAXMESSAGESIZE
[
https://issues.apache.org/jira/browse/HIVE-15321?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15710368#comment-15710368
]
Aihua Xu commented on HIVE-15321:
-
The tests are not related.
> Change to read as long for HiveConf.ConfVars.METASTORESERVERMAXMESSAGESIZE
> --
>
> Key: HIVE-15321
> URL: https://issues.apache.org/jira/browse/HIVE-15321
> Project: Hive
> Issue Type: Improvement
> Components: Metastore
>Affects Versions: 1.2.0, 1.1.0
>Reporter: Aihua Xu
>Assignee: Aihua Xu
> Attachments: HIVE-15321.1.patch, HIVE-15321.2.patch
>
>
> Follow up on HIVE-11240 which tries to change the type from int to long,
> while we are still read with {{conf.getIntVar()}}.
> Seems we should use {{conf.getLongVar()}}.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-15309) Miscellaneous logging clean up
[
https://issues.apache.org/jira/browse/HIVE-15309?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15710379#comment-15710379
]
Wei Zheng commented on HIVE-15309:
--
+1 on patch 3
> Miscellaneous logging clean up
> --
>
> Key: HIVE-15309
> URL: https://issues.apache.org/jira/browse/HIVE-15309
> Project: Hive
> Issue Type: Bug
> Components: Transactions
>Reporter: Eugene Koifman
>Assignee: Eugene Koifman
> Attachments: HIVE-15309.01.patch, HIVE-15309.02.patch,
> HIVE-15309.03.patch
>
>
> OrcAcidUtils.getLastFlushLength() should check for file existence first.
> Currently causes unnecessary/confusing logging:
> {noformat}
> org.apache.hadoop.ipc.RemoteException(java.io.FileNotFoundException): File
> does not exist: /domains/adl/rrslog/data_history/rrslog/r\
> rslog/hot/server_date=2016-08-19/delta_0005913_0005913/bucket_00023_flush_length
> at
> org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:71)
> at
> org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:61)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocationsInt(FSNamesystem.java:1860)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:1831)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:1744)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.getBlockLocations(NameNodeRpcServer.java:693)
> at
> org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.getBlockLocations(ClientNamenodeProtocolSe\
> rverSideTranslatorPB.java:373)
> at
> org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientName\
> nodeProtocolProtos.java)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:640)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:982)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2313)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2309)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1724)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2307)
> at org.apache.hadoop.ipc.Client.getRpcResponse(Client.java:1552)
> at org.apache.hadoop.ipc.Client.call(Client.java:1496)
> at org.apache.hadoop.ipc.Client.call(Client.java:1396)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:233)
> at com.sun.proxy.$Proxy14.getBlockLocations(Unknown Source)
> at
> org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.getBlockLocations(ClientNamenodeProtocolTranslatorPB\
> .java:270)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:497)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:278)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:194)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:176)
> at com.sun.proxy.$Proxy15.getBlockLocations(Unknown Source)
> at
> org.apache.hadoop.hdfs.DFSClient.callGetBlockLocations(DFSClient.java:1236)
> at
> org.apache.hadoop.hdfs.DFSClient.getLocatedBlocks(DFSClient.java:1223)
> at
> org.apache.hadoop.hdfs.DFSClient.getLocatedBlocks(DFSClient.java:1211)
> at
> org.apache.hadoop.hdfs.DFSInputStream.fetchLocatedBlocksAndGetLastBlockLength(DFSInputStream.java:309)
> at
> org.apache.hadoop.hdfs.DFSInputStream.openInfo(DFSInputStream.java:274)
> at
> org.apache.hadoop.hdfs.DFSInputStream.(DFSInputStream.java:266)
> at org.apache.hadoop.hdfs.DFSClient.open(DFSClient.java:1536)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem$4.doCall(DistributedFileSystem.java:330)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem$4.doCall(DistributedFileSystem.java:326)
> at
> org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem.open(DistributedFileSystem.java:326)
> at org.apache.hadoop.fs.FileSystem.open(File
[jira] [Commented] (HIVE-15057) Support other types of operators (other than SELECT)
[
https://issues.apache.org/jira/browse/HIVE-15057?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15710407#comment-15710407
]
Hive QA commented on HIVE-15057:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12841173/HIVE-15057.2.patch
{color:green}SUCCESS:{color} +1 due to 3 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 24 failed/errored test(s), 10740 tests
executed
*Failed tests:*
{noformat}
TestSparkCliDriver - did not produce a TEST-*.xml file (likely timed out)
(batchId=111)
[vectorization_16.q,load_dyn_part5.q,join_casesensitive.q,transform_ppr2.q,join23.q,groupby7_map_skew.q,ppd_outer_join5.q,create_merge_compressed.q,louter_join_ppr.q,sample9.q,smb_mapjoin_16.q,vectorization_not.q,having.q,ppd_outer_join1.q,union_remove_12.q]
org.apache.hadoop.hive.cli.TestCliDriver.org.apache.hadoop.hive.cli.TestCliDriver
(batchId=50)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[sample2] (batchId=5)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[sample4] (batchId=15)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[sample6] (batchId=60)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[sample7] (batchId=60)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[sample9] (batchId=38)
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[transform_ppr2]
(batchId=134)
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver[explainanalyze_2]
(batchId=92)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[auto_join28]
(batchId=123)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[auto_join4]
(batchId=123)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[bucketmapjoin11]
(batchId=123)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[groupby1_map]
(batchId=123)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[join17]
(batchId=123)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[join34]
(batchId=123)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[join_nullsafe]
(batchId=123)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[load_dyn_part11]
(batchId=123)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[mergejoins_mixed]
(batchId=123)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[nullgroup]
(batchId=123)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[skewjoinopt11]
(batchId=123)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[sort] (batchId=123)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[stats8]
(batchId=123)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[table_access_keys_stats]
(batchId=123)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[union17]
(batchId=123)
{noformat}
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/2350/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/2350/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-2350/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 24 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12841173 - PreCommit-HIVE-Build
> Support other types of operators (other than SELECT)
>
>
> Key: HIVE-15057
> URL: https://issues.apache.org/jira/browse/HIVE-15057
> Project: Hive
> Issue Type: Sub-task
> Components: Logical Optimizer, Physical Optimizer
>Reporter: Chao Sun
>Assignee: Chao Sun
> Attachments: HIVE-15057.1.patch, HIVE-15057.2.patch
>
>
> Currently only SELECT operators are supported for nested column pruning. We
> should add support for other types of operators so the optimization can work
> for complex queries.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-15066) push down projection is mismatched in HiveInputFormat
[ https://issues.apache.org/jira/browse/HIVE-15066?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15710427#comment-15710427 ] Prasanth Jayachandran commented on HIVE-15066: -- HIVE-7052 have more context to it. Appending values to config object is very CPU intensive and increases split computation time. > push down projection is mismatched in HiveInputFormat > - > > Key: HIVE-15066 > URL: https://issues.apache.org/jira/browse/HIVE-15066 > Project: Hive > Issue Type: Bug > Components: File Formats >Affects Versions: 2.1.0 >Reporter: Jeongdae Kim >Assignee: Jeongdae Kim > Attachments: HIVE-15066.1.patch > > > if HiveInputFormat.getSplits() is called with more than one table, > inputFormatClass.getSplits() for the first table will be called with read > column information for the second table. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-15322) Skipping "hbase mapredcp" in hive script for certain services
[ https://issues.apache.org/jira/browse/HIVE-15322?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daniel Dai updated HIVE-15322: -- Attachment: HIVE-15322.1.patch > Skipping "hbase mapredcp" in hive script for certain services > - > > Key: HIVE-15322 > URL: https://issues.apache.org/jira/browse/HIVE-15322 > Project: Hive > Issue Type: Improvement >Reporter: Daniel Dai >Assignee: Daniel Dai > Attachments: HIVE-15322.1.patch > > > "hbase mapredcp" is intended to append hbase classpath to hive. However, the > command can take some time when the system is heavy loaded. In some extreme > cases, we saw ~20s delay due to it. For certain commands, such as > "schemaTool", hbase classpath is certainly useless, and we can safely skip > invoking it. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-15322) Skipping "hbase mapredcp" in hive script for certain services
[ https://issues.apache.org/jira/browse/HIVE-15322?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daniel Dai updated HIVE-15322: -- Status: Patch Available (was: Open) > Skipping "hbase mapredcp" in hive script for certain services > - > > Key: HIVE-15322 > URL: https://issues.apache.org/jira/browse/HIVE-15322 > Project: Hive > Issue Type: Improvement >Reporter: Daniel Dai >Assignee: Daniel Dai > Attachments: HIVE-15322.1.patch > > > "hbase mapredcp" is intended to append hbase classpath to hive. However, the > command can take some time when the system is heavy loaded. In some extreme > cases, we saw ~20s delay due to it. For certain commands, such as > "schemaTool", hbase classpath is certainly useless, and we can safely skip > invoking it. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-15239) hive on spark combine equivalentwork get wrong result because of tablescan operation compare
[
https://issues.apache.org/jira/browse/HIVE-15239?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15710435#comment-15710435
]
Xuefu Zhang commented on HIVE-15239:
Sorry for the delay.
Re: my point #1, I was referring to this:
{code}
Set> firstRootOperators = first.getAllRootOperators();
Set> secondRootOperators = second.getAllRootOperators();
if (firstRootOperators.size() != secondRootOperators.size()) {
return false;
}
// need to check paths and partition desc for MapWorks
if (first instanceof MapWork && !compareMapWork((MapWork) first,
(MapWork) second)) {
return false;
}
{code}
I think it's better to be like the following in order to logical unit of code
together.
{code}
// need to check paths and partition desc for MapWorks
if (first instanceof MapWork && !compareMapWork((MapWork) first,
(MapWork) second)) {
return false;
}
Set> firstRootOperators = first.getAllRootOperators();
Set> secondRootOperators = second.getAllRootOperators();
if (firstRootOperators.size() != secondRootOperators.size()) {
return false;
}
{code}
As to exhaustive check, your fix will solve the problem describe here. I would
even believe there is a possibility that there are two two mapwork that works
on different partitions of the same table, such as in case of union.
Overall, I feel more testing is needed for this feature. Of course this goes
beyond the scope of this JIRA.
> hive on spark combine equivalentwork get wrong result because of tablescan
> operation compare
> -
>
> Key: HIVE-15239
> URL: https://issues.apache.org/jira/browse/HIVE-15239
> Project: Hive
> Issue Type: Bug
> Components: Spark
>Affects Versions: 1.2.0, 2.1.0
>Reporter: wangwenli
>Assignee: Rui Li
> Attachments: HIVE-15239.1.patch, HIVE-15239.2.patch
>
>
> env: hive on spark engine
> reproduce step:
> {code}
> create table a1(KEHHAO string, START_DT string) partitioned by (END_DT
> string);
> create table a2(KEHHAO string, START_DT string) partitioned by (END_DT
> string);
> alter table a1 add partition(END_DT='20161020');
> alter table a1 add partition(END_DT='20161021');
> insert into table a1 partition(END_DT='20161020')
> values('2000721360','20161001');
> SELECT T1.KEHHAO,COUNT(1) FROM (
> SELECT KEHHAO FROM a1 T
> WHERE T.KEHHAO = '2000721360' AND '20161018' BETWEEN T.START_DT AND
> T.END_DT-1
> UNION ALL
> SELECT KEHHAO FROM a2 T
> WHERE T.KEHHAO = '2000721360' AND '20161018' BETWEEN T.START_DT AND
> T.END_DT-1
> ) T1
> GROUP BY T1.KEHHAO
> HAVING COUNT(1)>1;
> +-+--+--+
> | t1.kehhao | _c1 |
> +-+--+--+
> | 2000721360 | 2|
> +-+--+--+
> {code}
> the result should be none record
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Comment Edited] (HIVE-15239) hive on spark combine equivalentwork get wrong result because of tablescan operation compare
[
https://issues.apache.org/jira/browse/HIVE-15239?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15710435#comment-15710435
]
Xuefu Zhang edited comment on HIVE-15239 at 12/1/16 1:12 AM:
-
Sorry for the delay.
Re: my point #1, I was referring to this:
{code}
Set> firstRootOperators = first.getAllRootOperators();
Set> secondRootOperators = second.getAllRootOperators();
if (firstRootOperators.size() != secondRootOperators.size()) {
return false;
}
// need to check paths and partition desc for MapWorks
if (first instanceof MapWork && !compareMapWork((MapWork) first,
(MapWork) second)) {
return false;
}
{code}
I think it's better to be like the following in order to put logical unit of
code (for operator check) together.
{code}
// need to check paths and partition desc for MapWorks
if (first instanceof MapWork && !compareMapWork((MapWork) first,
(MapWork) second)) {
return false;
}
Set> firstRootOperators = first.getAllRootOperators();
Set> secondRootOperators = second.getAllRootOperators();
if (firstRootOperators.size() != secondRootOperators.size()) {
return false;
}
{code}
As to exhaustive check, your fix will solve the problem describe here. I would
even believe there is a possibility that there are two two mapwork that works
on different partitions of the same table, such as in case of union.
Overall, I feel more testing is needed for this feature. Of course this goes
beyond the scope of this JIRA.
was (Author: xuefuz):
Sorry for the delay.
Re: my point #1, I was referring to this:
{code}
Set> firstRootOperators = first.getAllRootOperators();
Set> secondRootOperators = second.getAllRootOperators();
if (firstRootOperators.size() != secondRootOperators.size()) {
return false;
}
// need to check paths and partition desc for MapWorks
if (first instanceof MapWork && !compareMapWork((MapWork) first,
(MapWork) second)) {
return false;
}
{code}
I think it's better to be like the following in order to logical unit of code
together.
{code}
// need to check paths and partition desc for MapWorks
if (first instanceof MapWork && !compareMapWork((MapWork) first,
(MapWork) second)) {
return false;
}
Set> firstRootOperators = first.getAllRootOperators();
Set> secondRootOperators = second.getAllRootOperators();
if (firstRootOperators.size() != secondRootOperators.size()) {
return false;
}
{code}
As to exhaustive check, your fix will solve the problem describe here. I would
even believe there is a possibility that there are two two mapwork that works
on different partitions of the same table, such as in case of union.
Overall, I feel more testing is needed for this feature. Of course this goes
beyond the scope of this JIRA.
> hive on spark combine equivalentwork get wrong result because of tablescan
> operation compare
> -
>
> Key: HIVE-15239
> URL: https://issues.apache.org/jira/browse/HIVE-15239
> Project: Hive
> Issue Type: Bug
> Components: Spark
>Affects Versions: 1.2.0, 2.1.0
>Reporter: wangwenli
>Assignee: Rui Li
> Attachments: HIVE-15239.1.patch, HIVE-15239.2.patch
>
>
> env: hive on spark engine
> reproduce step:
> {code}
> create table a1(KEHHAO string, START_DT string) partitioned by (END_DT
> string);
> create table a2(KEHHAO string, START_DT string) partitioned by (END_DT
> string);
> alter table a1 add partition(END_DT='20161020');
> alter table a1 add partition(END_DT='20161021');
> insert into table a1 partition(END_DT='20161020')
> values('2000721360','20161001');
> SELECT T1.KEHHAO,COUNT(1) FROM (
> SELECT KEHHAO FROM a1 T
> WHERE T.KEHHAO = '2000721360' AND '20161018' BETWEEN T.START_DT AND
> T.END_DT-1
> UNION ALL
> SELECT KEHHAO FROM a2 T
> WHERE T.KEHHAO = '2000721360' AND '20161018' BETWEEN T.START_DT AND
> T.END_DT-1
> ) T1
> GROUP BY T1.KEHHAO
> HAVING COUNT(1)>1;
> +-+--+--+
> | t1.kehhao | _c1 |
> +-+--+--+
> | 2000721360 | 2|
> +-+--+--+
> {code}
> the result should be none record
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-15322) Skipping "hbase mapredcp" in hive script for certain services
[ https://issues.apache.org/jira/browse/HIVE-15322?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15710438#comment-15710438 ] Prasanth Jayachandran commented on HIVE-15322: -- [~sershe] Remember you had created jira/patch for this? > Skipping "hbase mapredcp" in hive script for certain services > - > > Key: HIVE-15322 > URL: https://issues.apache.org/jira/browse/HIVE-15322 > Project: Hive > Issue Type: Improvement >Reporter: Daniel Dai >Assignee: Daniel Dai > Attachments: HIVE-15322.1.patch > > > "hbase mapredcp" is intended to append hbase classpath to hive. However, the > command can take some time when the system is heavy loaded. In some extreme > cases, we saw ~20s delay due to it. For certain commands, such as > "schemaTool", hbase classpath is certainly useless, and we can safely skip > invoking it. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-15322) Skipping "hbase mapredcp" in hive script for certain services
[ https://issues.apache.org/jira/browse/HIVE-15322?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15710443#comment-15710443 ] Prasanth Jayachandran commented on HIVE-15322: -- We might have to add llap services as well. llapstatus doesn't need hbase jars. llap service also doesn't seem to be including hbase jars. May be [~sershe] can confirm? > Skipping "hbase mapredcp" in hive script for certain services > - > > Key: HIVE-15322 > URL: https://issues.apache.org/jira/browse/HIVE-15322 > Project: Hive > Issue Type: Improvement >Reporter: Daniel Dai >Assignee: Daniel Dai > Attachments: HIVE-15322.1.patch > > > "hbase mapredcp" is intended to append hbase classpath to hive. However, the > command can take some time when the system is heavy loaded. In some extreme > cases, we saw ~20s delay due to it. For certain commands, such as > "schemaTool", hbase classpath is certainly useless, and we can safely skip > invoking it. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-15276) CLIs spell "substitution" as "subsitution" and "auxiliary" as "auxillary"
[ https://issues.apache.org/jira/browse/HIVE-15276?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15710445#comment-15710445 ] Grant Sohn commented on HIVE-15276: --- Never mind. Found it. https://cwiki.apache.org/confluence/display/Hive/How+to+edit+the+website > CLIs spell "substitution" as "subsitution" and "auxiliary" as "auxillary" > - > > Key: HIVE-15276 > URL: https://issues.apache.org/jira/browse/HIVE-15276 > Project: Hive > Issue Type: Bug > Components: CLI >Affects Versions: 1.1.0 >Reporter: Grant Sohn >Assignee: Grant Sohn >Priority: Trivial > Fix For: 2.2.0 > > Attachments: HIVE-15276.1.patch, HIVE-15276.2.patch, > HIVE-15276.3.patch, HIVE-15276.4.patch, HIVE-15276.5.patch > > > Found some obvious spelling typos in the CLI help. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-15202) Concurrent compactions for the same partition may generate malformed folder structure
[
https://issues.apache.org/jira/browse/HIVE-15202?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15710450#comment-15710450
]
Rui Li commented on HIVE-15202:
---
I see. Thanks for the explanations Eugene :)
> Concurrent compactions for the same partition may generate malformed folder
> structure
> -
>
> Key: HIVE-15202
> URL: https://issues.apache.org/jira/browse/HIVE-15202
> Project: Hive
> Issue Type: Bug
> Components: Transactions
>Reporter: Rui Li
>Assignee: Eugene Koifman
> Attachments: HIVE-15202.01.patch, HIVE-15202.02.patch,
> HIVE-15202.03.patch
>
>
> If two compactions run concurrently on a single partition, it may generate
> folder structure like this: (nested base dir)
> {noformat}
> drwxr-xr-x - root supergroup 0 2016-11-14 22:23
> /user/hive/warehouse/test/z=1/base_007/base_007
> -rw-r--r-- 3 root supergroup201 2016-11-14 21:46
> /user/hive/warehouse/test/z=1/base_007/bucket_0
> -rw-r--r-- 3 root supergroup611 2016-11-14 21:46
> /user/hive/warehouse/test/z=1/base_007/bucket_1
> -rw-r--r-- 3 root supergroup614 2016-11-14 21:46
> /user/hive/warehouse/test/z=1/base_007/bucket_2
> -rw-r--r-- 3 root supergroup621 2016-11-14 21:46
> /user/hive/warehouse/test/z=1/base_007/bucket_3
> -rw-r--r-- 3 root supergroup621 2016-11-14 21:46
> /user/hive/warehouse/test/z=1/base_007/bucket_4
> -rw-r--r-- 3 root supergroup201 2016-11-14 21:46
> /user/hive/warehouse/test/z=1/base_007/bucket_5
> -rw-r--r-- 3 root supergroup201 2016-11-14 21:46
> /user/hive/warehouse/test/z=1/base_007/bucket_6
> -rw-r--r-- 3 root supergroup201 2016-11-14 21:46
> /user/hive/warehouse/test/z=1/base_007/bucket_7
> -rw-r--r-- 3 root supergroup201 2016-11-14 21:46
> /user/hive/warehouse/test/z=1/base_007/bucket_8
> -rw-r--r-- 3 root supergroup201 2016-11-14 21:46
> /user/hive/warehouse/test/z=1/base_007/bucket_9
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-15313) Add export spark.yarn.archive or spark.yarn.jars variable in Hive on Spark document
[
https://issues.apache.org/jira/browse/HIVE-15313?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15710449#comment-15710449
]
Ferdinand Xu commented on HIVE-15313:
-
I'd like to update Hive WIKI about configuring spark.yarn.jars configuration
for better performance for queries at small data scale. [~xuefuz] [~lirui], any
suggestions about this?
> Add export spark.yarn.archive or spark.yarn.jars variable in Hive on Spark
> document
> ---
>
> Key: HIVE-15313
> URL: https://issues.apache.org/jira/browse/HIVE-15313
> Project: Hive
> Issue Type: Bug
>Reporter: liyunzhang_intel
>Priority: Minor
> Attachments: performance.improvement.after.set.spark.yarn.archive.PNG
>
>
> According to
> [wiki|https://cwiki.apache.org/confluence/display/Hive/Hive+on+Spark%3A+Getting+Started],
> run queries in HOS16 and HOS20 in yarn mode.
> Following table shows the difference in query time between HOS16 and HOS20.
> ||Version||Total time||Time for Jobs||Time for preparing jobs||
> |Spark16|51|39|12|
> |Spark20|54|40|14|
> HOS20 spends more time(2 secs) on preparing jobs than HOS16. After reviewing
> the source code of spark, found that following point causes this:
> code:[Client#distribute|https://github.com/apache/spark/blob/master/yarn/src/main/scala/org/apache/spark/deploy/yarn/Client.scala#L546],
> In spark20, if spark cannot find spark.yarn.archive and spark.yarn.jars in
> spark configuration file, it will first copy all jars in $SPARK_HOME/jars to
> a tmp directory and upload the tmp directory to distribute cache. Comparing
> [spark16|https://github.com/apache/spark/blob/branch-1.6/yarn/src/main/scala/org/apache/spark/deploy/yarn/Client.scala#L1145],
>
> In spark16, it searches spark-assembly*.jar and upload it to distribute cache.
> In spark20, it spends 2 more seconds to copy all jars in $SPARK_HOME/jar to a
> tmp directory if we don't set "spark.yarn.archive" or "spark.yarn.jars".
> We can accelerate the startup of hive on spark 20 by settintg
> "spark.yarn.archive" or "spark.yarn.jars":
> set "spark.yarn.archive":
> {code}
> zip spark-archive.zip $SPARK_HOME/jars/*
> $ hadoop fs -copyFromLocal spark-archive.zip
> $ echo "spark.yarn.archive=hdfs:///xxx:8020/spark-archive.zip" >>
> conf/spark-defaults.conf
> {code}
> set "spark.yarn.jars":
> {code}
> $ hadoop fs mkdir spark-2.0.0-bin-hadoop
> $hadoop fs -copyFromLocal $SPARK_HOME/jars/* spark-2.0.0-bin-hadoop
> $ echo "spark.yarn.jars=hdfs:///xxx:8020/spark-2.0.0-bin-hadoop/*" >>
> conf/spark-defaults.conf
> {code}
> Suggest to add this part in wiki.
> performance.improvement.after.set.spark.yarn.archive.PNG shows the detail
> performance impovement after setting spark.yarn.archive in small queries.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-15322) Skipping "hbase mapredcp" in hive script for certain services
[
https://issues.apache.org/jira/browse/HIVE-15322?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15710453#comment-15710453
]
Hive QA commented on HIVE-15322:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12841188/HIVE-15322.1.patch
{color:red}ERROR:{color} -1 due to build exiting with an error
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/2353/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/2353/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-2353/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Tests exited with: NonZeroExitCodeException
Command 'bash /data/hiveptest/working/scratch/source-prep.sh' failed with exit
status 1 and output '+ date '+%Y-%m-%d %T.%3N'
2016-12-01 01:19:20.878
+ [[ -n /usr/lib/jvm/java-8-openjdk-amd64 ]]
+ export JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
+ JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
+ export
PATH=/usr/lib/jvm/java-8-openjdk-amd64/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games
+
PATH=/usr/lib/jvm/java-8-openjdk-amd64/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games
+ export 'ANT_OPTS=-Xmx1g -XX:MaxPermSize=256m '
+ ANT_OPTS='-Xmx1g -XX:MaxPermSize=256m '
+ export 'MAVEN_OPTS=-Xmx1g '
+ MAVEN_OPTS='-Xmx1g '
+ cd /data/hiveptest/working/
+ tee /data/hiveptest/logs/PreCommit-HIVE-Build-2353/source-prep.txt
+ [[ false == \t\r\u\e ]]
+ mkdir -p maven ivy
+ [[ git = \s\v\n ]]
+ [[ git = \g\i\t ]]
+ [[ -z master ]]
+ [[ -d apache-github-source-source ]]
+ [[ ! -d apache-github-source-source/.git ]]
+ [[ ! -d apache-github-source-source ]]
+ date '+%Y-%m-%d %T.%3N'
2016-12-01 01:19:20.881
+ cd apache-github-source-source
+ git fetch origin
>From https://github.com/apache/hive
76b311f..54b3abf master -> origin/master
+ git reset --hard HEAD
HEAD is now at 76b311f HIVE-14582 : Add trunc(numeric) udf (Chinna Rao Llam via
Ashutosh Chauhan)
+ git clean -f -d
Removing ql/src/test/queries/clientpositive/nested_column_pruning.q
Removing ql/src/test/results/clientpositive/nested_column_pruning.q.out
+ git checkout master
Already on 'master'
Your branch is behind 'origin/master' by 3 commits, and can be fast-forwarded.
(use "git pull" to update your local branch)
+ git reset --hard origin/master
HEAD is now at 54b3abf HIVE-15309 Miscellaneous logging clean up (Eugene
Koifman, reviewed by Wei Zheng)
+ git merge --ff-only origin/master
Already up-to-date.
+ date '+%Y-%m-%d %T.%3N'
2016-12-01 01:19:22.377
+ patchCommandPath=/data/hiveptest/working/scratch/smart-apply-patch.sh
+ patchFilePath=/data/hiveptest/working/scratch/build.patch
+ [[ -f /data/hiveptest/working/scratch/build.patch ]]
+ chmod +x /data/hiveptest/working/scratch/smart-apply-patch.sh
+ /data/hiveptest/working/scratch/smart-apply-patch.sh
/data/hiveptest/working/scratch/build.patch
error: a/bin/hive: No such file or directory
The patch does not appear to apply with p0, p1, or p2
+ exit 1
'
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12841188 - PreCommit-HIVE-Build
> Skipping "hbase mapredcp" in hive script for certain services
> -
>
> Key: HIVE-15322
> URL: https://issues.apache.org/jira/browse/HIVE-15322
> Project: Hive
> Issue Type: Improvement
>Reporter: Daniel Dai
>Assignee: Daniel Dai
> Attachments: HIVE-15322.1.patch
>
>
> "hbase mapredcp" is intended to append hbase classpath to hive. However, the
> command can take some time when the system is heavy loaded. In some extreme
> cases, we saw ~20s delay due to it. For certain commands, such as
> "schemaTool", hbase classpath is certainly useless, and we can safely skip
> invoking it.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-15202) Concurrent compactions for the same partition may generate malformed folder structure
[
https://issues.apache.org/jira/browse/HIVE-15202?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Eugene Koifman updated HIVE-15202:
--
Resolution: Fixed
Fix Version/s: 2.2.0
Status: Resolved (was: Patch Available)
Committed to master
Thanks Wei for the review
> Concurrent compactions for the same partition may generate malformed folder
> structure
> -
>
> Key: HIVE-15202
> URL: https://issues.apache.org/jira/browse/HIVE-15202
> Project: Hive
> Issue Type: Bug
> Components: Transactions
>Reporter: Rui Li
>Assignee: Eugene Koifman
> Fix For: 2.2.0
>
> Attachments: HIVE-15202.01.patch, HIVE-15202.02.patch,
> HIVE-15202.03.patch
>
>
> If two compactions run concurrently on a single partition, it may generate
> folder structure like this: (nested base dir)
> {noformat}
> drwxr-xr-x - root supergroup 0 2016-11-14 22:23
> /user/hive/warehouse/test/z=1/base_007/base_007
> -rw-r--r-- 3 root supergroup201 2016-11-14 21:46
> /user/hive/warehouse/test/z=1/base_007/bucket_0
> -rw-r--r-- 3 root supergroup611 2016-11-14 21:46
> /user/hive/warehouse/test/z=1/base_007/bucket_1
> -rw-r--r-- 3 root supergroup614 2016-11-14 21:46
> /user/hive/warehouse/test/z=1/base_007/bucket_2
> -rw-r--r-- 3 root supergroup621 2016-11-14 21:46
> /user/hive/warehouse/test/z=1/base_007/bucket_3
> -rw-r--r-- 3 root supergroup621 2016-11-14 21:46
> /user/hive/warehouse/test/z=1/base_007/bucket_4
> -rw-r--r-- 3 root supergroup201 2016-11-14 21:46
> /user/hive/warehouse/test/z=1/base_007/bucket_5
> -rw-r--r-- 3 root supergroup201 2016-11-14 21:46
> /user/hive/warehouse/test/z=1/base_007/bucket_6
> -rw-r--r-- 3 root supergroup201 2016-11-14 21:46
> /user/hive/warehouse/test/z=1/base_007/bucket_7
> -rw-r--r-- 3 root supergroup201 2016-11-14 21:46
> /user/hive/warehouse/test/z=1/base_007/bucket_8
> -rw-r--r-- 3 root supergroup201 2016-11-14 21:46
> /user/hive/warehouse/test/z=1/base_007/bucket_9
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-14980) Minor compaction when triggered simultaniously on the same table/partition deletes data
[ https://issues.apache.org/jira/browse/HIVE-14980?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15710467#comment-15710467 ] Eugene Koifman commented on HIVE-14980: --- This should now be fixed via HIVE-15202 > Minor compaction when triggered simultaniously on the same table/partition > deletes data > --- > > Key: HIVE-14980 > URL: https://issues.apache.org/jira/browse/HIVE-14980 > Project: Hive > Issue Type: Bug > Components: Metastore, Transactions >Affects Versions: 2.1.0 >Reporter: Mahipal Jupalli >Assignee: Mahipal Jupalli >Priority: Critical > Original Estimate: 96h > Remaining Estimate: 96h > > I have two tables (TABLEA, TABLEB). If I manually trigger compaction after > each INSERT into TABLEB from TABLEA, compactions are triggered on random > metastore asynchronously and are stepping on each other which is causing the > data to be deleted. > Example here: > TABLEA - has 10k rows. > insert into mj.tableb select * from mj.tablea; > alter table mj.tableb compact 'MINOR'; > insert into mj.tableb select * from mj.tablea; > alter table mj.tableb compact 'MINOR'; > Once all the compactions are complete, I should ideally see 20k rows in > TABLEB. But I see only 10k rows (Only the rows INSERTED before the last > compaction persist, the old rows are deleted. I believe the old delta files > are deleted). > To further confirm the bug, if I do only one compaction after two inserts, I > see 20k rows in TABLEB. > Proposed Fix: > I have identified the bug in the code, it requires an additional check in the > org.apache.hadoop.hive.ql.txn.compactor.Worker class to check for any active > compactions on the table/partition. I will 'share the details of the fix once > I test it. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-15309) Miscellaneous logging clean up
[
https://issues.apache.org/jira/browse/HIVE-15309?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Eugene Koifman updated HIVE-15309:
--
Resolution: Fixed
Fix Version/s: 2.2.0
Status: Resolved (was: Patch Available)
Committed to master
Thanks Wei for the review
> Miscellaneous logging clean up
> --
>
> Key: HIVE-15309
> URL: https://issues.apache.org/jira/browse/HIVE-15309
> Project: Hive
> Issue Type: Bug
> Components: Transactions
>Reporter: Eugene Koifman
>Assignee: Eugene Koifman
> Fix For: 2.2.0
>
> Attachments: HIVE-15309.01.patch, HIVE-15309.02.patch,
> HIVE-15309.03.patch
>
>
> OrcAcidUtils.getLastFlushLength() should check for file existence first.
> Currently causes unnecessary/confusing logging:
> {noformat}
> org.apache.hadoop.ipc.RemoteException(java.io.FileNotFoundException): File
> does not exist: /domains/adl/rrslog/data_history/rrslog/r\
> rslog/hot/server_date=2016-08-19/delta_0005913_0005913/bucket_00023_flush_length
> at
> org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:71)
> at
> org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:61)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocationsInt(FSNamesystem.java:1860)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:1831)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:1744)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.getBlockLocations(NameNodeRpcServer.java:693)
> at
> org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.getBlockLocations(ClientNamenodeProtocolSe\
> rverSideTranslatorPB.java:373)
> at
> org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientName\
> nodeProtocolProtos.java)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:640)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:982)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2313)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2309)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1724)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2307)
> at org.apache.hadoop.ipc.Client.getRpcResponse(Client.java:1552)
> at org.apache.hadoop.ipc.Client.call(Client.java:1496)
> at org.apache.hadoop.ipc.Client.call(Client.java:1396)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:233)
> at com.sun.proxy.$Proxy14.getBlockLocations(Unknown Source)
> at
> org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.getBlockLocations(ClientNamenodeProtocolTranslatorPB\
> .java:270)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:497)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:278)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:194)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:176)
> at com.sun.proxy.$Proxy15.getBlockLocations(Unknown Source)
> at
> org.apache.hadoop.hdfs.DFSClient.callGetBlockLocations(DFSClient.java:1236)
> at
> org.apache.hadoop.hdfs.DFSClient.getLocatedBlocks(DFSClient.java:1223)
> at
> org.apache.hadoop.hdfs.DFSClient.getLocatedBlocks(DFSClient.java:1211)
> at
> org.apache.hadoop.hdfs.DFSInputStream.fetchLocatedBlocksAndGetLastBlockLength(DFSInputStream.java:309)
> at
> org.apache.hadoop.hdfs.DFSInputStream.openInfo(DFSInputStream.java:274)
> at
> org.apache.hadoop.hdfs.DFSInputStream.(DFSInputStream.java:266)
> at org.apache.hadoop.hdfs.DFSClient.open(DFSClient.java:1536)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem$4.doCall(DistributedFileSystem.java:330)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem$4.doCall(DistributedFileSystem.java:326)
> at
> org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
> at
> org.apache.hadoop.
[jira] [Updated] (HIVE-15308) Create ACID table failed intermittently: due to Postgres (SQLState=25P02, ErrorCode=0)
[
https://issues.apache.org/jira/browse/HIVE-15308?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Eugene Koifman updated HIVE-15308:
--
Resolution: Fixed
Fix Version/s: 2.2.0
Status: Resolved (was: Patch Available)
Committed to master
Thanks Wei for the review
> Create ACID table failed intermittently: due to Postgres (SQLState=25P02,
> ErrorCode=0)
> --
>
> Key: HIVE-15308
> URL: https://issues.apache.org/jira/browse/HIVE-15308
> Project: Hive
> Issue Type: Bug
> Components: Transactions
>Affects Versions: 2.1.0
>Reporter: Eugene Koifman
>Assignee: Eugene Koifman
> Fix For: 2.2.0
>
> Attachments: HIVE-15308.01.patch
>
>
> if 2 concurrent calls to MutexApi.acquireLock() happen with the same "key"
> value and there is no row in AUX_TABLE for that value yet (i.e. both are
> attempting to insert it) Postgres kills the txn which gets the Duplicate Key
> error and no more statements can be executed on this txn.
> (This is different from the way most DBs behave).
> {noformat}
> Caused by: MetaException(message:Unable to lock 'CheckLock' due to: ERROR:
> current transaction is aborted, commands ignored until end of transaction
> block (SQLState=25P02, ErrorCode=0); org.postgresql.util.PSQLException:
> ERROR: current transaction is aborted, commands ignored until end of
> transaction block
> at
> org.postgresql.core.v3.QueryExecutorImpl.receiveErrorResponse(QueryExecutorImpl.java:2198)
> at
> org.postgresql.core.v3.QueryExecutorImpl.processResults(QueryExecutorImpl.java:1927)
> at
> org.postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:255)
> at
> org.postgresql.jdbc2.AbstractJdbc2Statement.execute(AbstractJdbc2Statement.java:561)
> at
> org.postgresql.jdbc2.AbstractJdbc2Statement.executeWithFlags(AbstractJdbc2Statement.java:405)
> at
> org.postgresql.jdbc2.AbstractJdbc2Statement.executeQuery(AbstractJdbc2Statement.java:285)
> at
> com.jolbox.bonecp.StatementHandle.executeQuery(StatementHandle.java:464)
> at
> org.apache.hadoop.hive.metastore.txn.TxnHandler.acquireLock(TxnHandler.java:3250)
> at
> org.apache.hadoop.hive.metastore.txn.TxnHandler.checkLock(TxnHandler.java:2319)
> at
> org.apache.hadoop.hive.metastore.txn.TxnHandler.checkLockWithRetry(TxnHandler.java:1022)
> at
> org.apache.hadoop.hive.metastore.txn.TxnHandler.lock(TxnHandler.java:794)
> at
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.lock(HiveMetaStore.java:5941)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.apache.hadoop.hive.metastore.RetryingHMSHandler.invokeInternal(RetryingHMSHandler.java:140)
> at
> org.apache.hadoop.hive.metastore.RetryingHMSHandler.invoke(RetryingHMSHandler.java:99)
> at com.sun.proxy.$Proxy30.lock(Unknown Source)
> at
> org.apache.hadoop.hive.metastore.HiveMetaStoreClient.lock(HiveMetaStoreClient.java:2109)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.invoke(RetryingMetaStoreClient.java:154)
> at com.sun.proxy.$Proxy31.lock(Unknown Source)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.apache.hadoop.hive.metastore.HiveMetaStoreClient$SynchronizedHandler.invoke(HiveMetaStoreClient.java:2259)
> at com.sun.proxy.$Proxy31.lock(Unknown Source)
> at
> org.apache.hadoop.hive.ql.lockmgr.DbTxnManager$SynchronizedMetaStoreClient.lock(DbTxnManager.java:745)
> at
> org.apache.hadoop.hive.ql.lockmgr.DbLockManager.lock(DbLockManager.java:103)
> at
> org.apache.hadoop.hive.ql.lockmgr.DbTxnManager.acquireLocks(DbTxnManager.java:341)
> at
> org.apache.hadoop.hive.ql.lockmgr.DbTxnManager.acquireLocksWithHeartbeatDelay(DbTxnManager.java:357)
> at
> org.apache.hadoop.hive.ql.lockmgr.DbTxnManager.acquireLocks(DbTxnManager.java:167)
> at
> org.apache.hado
[jira] [Commented] (HIVE-15302) Relax the requirement that HoS needs Spark built w/o Hive
[ https://issues.apache.org/jira/browse/HIVE-15302?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15710478#comment-15710478 ] Rui Li commented on HIVE-15302: --- We don't only depend on Spark jars, but also the scripts like spark-submit. Not sure how to package them into Hive. [~xuefuz], what do you think about Marcelo's suggestions? > Relax the requirement that HoS needs Spark built w/o Hive > - > > Key: HIVE-15302 > URL: https://issues.apache.org/jira/browse/HIVE-15302 > Project: Hive > Issue Type: Improvement >Reporter: Rui Li >Assignee: Rui Li > > This requirement becomes more and more unacceptable as SparkSQL becomes > widely adopted. Let's use this JIRA to find out how we can relax the > limitation. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-15322) Skipping "hbase mapredcp" in hive script for certain services
[ https://issues.apache.org/jira/browse/HIVE-15322?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15710491#comment-15710491 ] Sergey Shelukhin commented on HIVE-15322: - This is HIVE-15108, although it needs to be specified explicitly right now > Skipping "hbase mapredcp" in hive script for certain services > - > > Key: HIVE-15322 > URL: https://issues.apache.org/jira/browse/HIVE-15322 > Project: Hive > Issue Type: Improvement >Reporter: Daniel Dai >Assignee: Daniel Dai > Attachments: HIVE-15322.1.patch > > > "hbase mapredcp" is intended to append hbase classpath to hive. However, the > command can take some time when the system is heavy loaded. In some extreme > cases, we saw ~20s delay due to it. For certain commands, such as > "schemaTool", hbase classpath is certainly useless, and we can safely skip > invoking it. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
