[jira] [Commented] (HIVE-17830) dbnotification fails to work with rdbms other than postgres
[ https://issues.apache.org/jira/browse/HIVE-17830?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16559258#comment-16559258 ] ASF GitHub Bot commented on HIVE-17830: --- Github user anishek closed the pull request at: https://github.com/apache/hive/pull/263 > dbnotification fails to work with rdbms other than postgres > --- > > Key: HIVE-17830 > URL: https://issues.apache.org/jira/browse/HIVE-17830 > Project: Hive > Issue Type: Bug > Components: HiveServer2 >Reporter: anishek >Assignee: Daniel Dai >Priority: Critical > Labels: pull-request-available > Fix For: 3.0.0 > > Attachments: HIVE-17830.0.patch, HIVE-17830.1.patch > > > as part of HIVE-17721 we had changed the direct sql to acquire the lock for > postgres as > {code} > select "NEXT_EVENT_ID" from "NOTIFICATION_SEQUENCE" for update; > {code} > however this breaks other databases and we have to use different sql > statements for different databases > for postgres use > {code} > select "NEXT_EVENT_ID" from "NOTIFICATION_SEQUENCE" for update; > {code} > for SQLServer > {code} > select "NEXT_EVENT_ID" from "NOTIFICATION_SEQUENCE" with (updlock); > {code} > for other databases > {code} > select NEXT_EVENT_ID from NOTIFICATION_SEQUENCE for update; > {code} -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-17830) dbnotification fails to work with rdbms other than postgres
[ https://issues.apache.org/jira/browse/HIVE-17830?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16295893#comment-16295893 ] Daniel Dai commented on HIVE-17830: --- Created HIVE-18299 for continuation work. > dbnotification fails to work with rdbms other than postgres > --- > > Key: HIVE-17830 > URL: https://issues.apache.org/jira/browse/HIVE-17830 > Project: Hive > Issue Type: Bug > Components: HiveServer2 >Reporter: anishek >Assignee: Daniel Dai >Priority: Critical > Fix For: 3.0.0 > > Attachments: HIVE-17830.0.patch, HIVE-17830.1.patch > > > as part of HIVE-17721 we had changed the direct sql to acquire the lock for > postgres as > {code} > select "NEXT_EVENT_ID" from "NOTIFICATION_SEQUENCE" for update; > {code} > however this breaks other databases and we have to use different sql > statements for different databases > for postgres use > {code} > select "NEXT_EVENT_ID" from "NOTIFICATION_SEQUENCE" for update; > {code} > for SQLServer > {code} > select "NEXT_EVENT_ID" from "NOTIFICATION_SEQUENCE" with (updlock); > {code} > for other databases > {code} > select NEXT_EVENT_ID from NOTIFICATION_SEQUENCE for update; > {code} -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HIVE-17830) dbnotification fails to work with rdbms other than postgres
[ https://issues.apache.org/jira/browse/HIVE-17830?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16214731#comment-16214731 ] Daniel Dai commented on HIVE-17830: --- Ok so "SET @@session.sql_mode=ANSI_QUOTES" will be required, right? Last time I read the code, it seems prepareTxn will be invoked every time we created a new ObjectStore. However, I must miss somewhere as otherwise, we will never hit the sql syntax error. > dbnotification fails to work with rdbms other than postgres > --- > > Key: HIVE-17830 > URL: https://issues.apache.org/jira/browse/HIVE-17830 > Project: Hive > Issue Type: Bug > Components: HiveServer2 >Reporter: anishek >Assignee: Daniel Dai >Priority: Critical > Fix For: 3.0.0 > > Attachments: HIVE-17830.0.patch, HIVE-17830.1.patch > > > as part of HIVE-17721 we had changed the direct sql to acquire the lock for > postgres as > {code} > select "NEXT_EVENT_ID" from "NOTIFICATION_SEQUENCE" for update; > {code} > however this breaks other databases and we have to use different sql > statements for different databases > for postgres use > {code} > select "NEXT_EVENT_ID" from "NOTIFICATION_SEQUENCE" for update; > {code} > for SQLServer > {code} > select "NEXT_EVENT_ID" from "NOTIFICATION_SEQUENCE" with (updlock); > {code} > for other databases > {code} > select NEXT_EVENT_ID from NOTIFICATION_SEQUENCE for update; > {code} -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HIVE-17830) dbnotification fails to work with rdbms other than postgres
[ https://issues.apache.org/jira/browse/HIVE-17830?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16214723#comment-16214723 ] anishek commented on HIVE-17830: Thanks [~daijy] for this patch. A Quick question. on looking at the code which sets the ANSI_QUOTE its in *MetastoreDirectSql.java* {code} public void prepareTxn() throws MetaException { if (dbType != DatabaseProduct.MYSQL) return; try { assert pm.currentTransaction().isActive(); // must be inside tx together with queries executeNoResult("SET @@session.sql_mode=ANSI_QUOTES"); } catch (SQLException sqlEx) { throw new MetaException("Error setting ansi quotes: " + sqlEx.getMessage()); } } {code} here we are setting the sql_mode only for the *session* and not *global*. I just ran the below on a mysql server without modifying the sql_mode {code} mysql> select "NEXT_EVENT_ID" from NOTIFICATION_SEQUENCE; +---+ | NEXT_EVENT_ID | +---+ | NEXT_EVENT_ID | +---+ 1 row in set (0.00 sec) {code} since we use connection pooling depending on which connection is used to execute the above statement we will get different results, wont we. May be i am missing something here. cc [~thejas] > dbnotification fails to work with rdbms other than postgres > --- > > Key: HIVE-17830 > URL: https://issues.apache.org/jira/browse/HIVE-17830 > Project: Hive > Issue Type: Bug > Components: HiveServer2 >Reporter: anishek >Assignee: Daniel Dai >Priority: Critical > Fix For: 3.0.0 > > Attachments: HIVE-17830.0.patch, HIVE-17830.1.patch > > > as part of HIVE-17721 we had changed the direct sql to acquire the lock for > postgres as > {code} > select "NEXT_EVENT_ID" from "NOTIFICATION_SEQUENCE" for update; > {code} > however this breaks other databases and we have to use different sql > statements for different databases > for postgres use > {code} > select "NEXT_EVENT_ID" from "NOTIFICATION_SEQUENCE" for update; > {code} > for SQLServer > {code} > select "NEXT_EVENT_ID" from "NOTIFICATION_SEQUENCE" with (updlock); > {code} > for other databases > {code} > select NEXT_EVENT_ID from NOTIFICATION_SEQUENCE for update; > {code} -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HIVE-17830) dbnotification fails to work with rdbms other than postgres
[ https://issues.apache.org/jira/browse/HIVE-17830?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16210928#comment-16210928 ] Hive QA commented on HIVE-17830: Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12892944/HIVE-17830.1.patch {color:red}ERROR:{color} -1 due to no test(s) being added or modified. {color:red}ERROR:{color} -1 due to 8 failed/errored test(s), 11302 tests executed *Failed tests:* {noformat} org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[optimize_nullscan] (batchId=163) org.apache.hadoop.hive.cli.TestSparkPerfCliDriver.testCliDriver[query16] (batchId=243) org.apache.hadoop.hive.cli.TestSparkPerfCliDriver.testCliDriver[query94] (batchId=243) org.apache.hadoop.hive.cli.TestTezPerfCliDriver.testCliDriver[query14] (batchId=241) org.apache.hadoop.hive.cli.TestTezPerfCliDriver.testCliDriver[query16] (batchId=241) org.apache.hadoop.hive.cli.TestTezPerfCliDriver.testCliDriver[query94] (batchId=241) org.apache.hadoop.hive.cli.control.TestDanglingQOuts.checkDanglingQOut (batchId=204) org.apache.hive.jdbc.TestTriggersTezSessionPoolManager.testTriggerHighShuffleBytes (batchId=228) {noformat} Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/7377/testReport Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/7377/console Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-7377/ Messages: {noformat} Executing org.apache.hive.ptest.execution.TestCheckPhase Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase Tests exited with: TestsFailedException: 8 tests failed {noformat} This message is automatically generated. ATTACHMENT ID: 12892944 - PreCommit-HIVE-Build > dbnotification fails to work with rdbms other than postgres > --- > > Key: HIVE-17830 > URL: https://issues.apache.org/jira/browse/HIVE-17830 > Project: Hive > Issue Type: Bug > Components: HiveServer2 >Reporter: anishek >Assignee: Daniel Dai >Priority: Critical > Fix For: 3.0.0 > > Attachments: HIVE-17830.0.patch, HIVE-17830.1.patch > > > as part of HIVE-17721 we had changed the direct sql to acquire the lock for > postgres as > {code} > select "NEXT_EVENT_ID" from "NOTIFICATION_SEQUENCE" for update; > {code} > however this breaks other databases and we have to use different sql > statements for different databases > for postgres use > {code} > select "NEXT_EVENT_ID" from "NOTIFICATION_SEQUENCE" for update; > {code} > for SQLServer > {code} > select "NEXT_EVENT_ID" from "NOTIFICATION_SEQUENCE" with (updlock); > {code} > for other databases > {code} > select NEXT_EVENT_ID from NOTIFICATION_SEQUENCE for update; > {code} -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HIVE-17830) dbnotification fails to work with rdbms other than postgres
[ https://issues.apache.org/jira/browse/HIVE-17830?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16210377#comment-16210377 ] Thejas M Nair commented on HIVE-17830: -- +1 > dbnotification fails to work with rdbms other than postgres > --- > > Key: HIVE-17830 > URL: https://issues.apache.org/jira/browse/HIVE-17830 > Project: Hive > Issue Type: Bug > Components: HiveServer2 >Reporter: anishek >Assignee: Daniel Dai >Priority: Critical > Fix For: 3.0.0 > > Attachments: HIVE-17830.0.patch, HIVE-17830.1.patch > > > as part of HIVE-17721 we had changed the direct sql to acquire the lock for > postgres as > {code} > select "NEXT_EVENT_ID" from "NOTIFICATION_SEQUENCE" for update; > {code} > however this breaks other databases and we have to use different sql > statements for different databases > for postgres use > {code} > select "NEXT_EVENT_ID" from "NOTIFICATION_SEQUENCE" for update; > {code} > for SQLServer > {code} > select "NEXT_EVENT_ID" from "NOTIFICATION_SEQUENCE" with (updlock); > {code} > for other databases > {code} > select NEXT_EVENT_ID from NOTIFICATION_SEQUENCE for update; > {code} -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HIVE-17830) dbnotification fails to work with rdbms other than postgres
[ https://issues.apache.org/jira/browse/HIVE-17830?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16209945#comment-16209945 ] Daniel Dai commented on HIVE-17830: --- We shall use "SET @@session.sql_mode=ANSI_QUOTES" to handle quotes, that's how MetaStoreDirectSQL works. > dbnotification fails to work with rdbms other than postgres > --- > > Key: HIVE-17830 > URL: https://issues.apache.org/jira/browse/HIVE-17830 > Project: Hive > Issue Type: Bug > Components: HiveServer2 >Reporter: anishek >Assignee: anishek >Priority: Critical > Fix For: 3.0.0 > > Attachments: HIVE-17830.0.patch > > > as part of HIVE-17721 we had changed the direct sql to acquire the lock for > postgres as > {code} > select "NEXT_EVENT_ID" from "NOTIFICATION_SEQUENCE" for update; > {code} > however this breaks other databases and we have to use different sql > statements for different databases > for postgres use > {code} > select "NEXT_EVENT_ID" from "NOTIFICATION_SEQUENCE" for update; > {code} > for SQLServer > {code} > select "NEXT_EVENT_ID" from "NOTIFICATION_SEQUENCE" with (updlock); > {code} > for other databases > {code} > select NEXT_EVENT_ID from NOTIFICATION_SEQUENCE for update; > {code} -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HIVE-17830) dbnotification fails to work with rdbms other than postgres
[ https://issues.apache.org/jira/browse/HIVE-17830?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16209700#comment-16209700 ] Alexander Kolbasov commented on HIVE-17830: --- [~spena] [~kkalyan] FYI > dbnotification fails to work with rdbms other than postgres > --- > > Key: HIVE-17830 > URL: https://issues.apache.org/jira/browse/HIVE-17830 > Project: Hive > Issue Type: Bug > Components: HiveServer2 >Reporter: anishek >Assignee: anishek >Priority: Critical > Fix For: 3.0.0 > > Attachments: HIVE-17830.0.patch > > > as part of HIVE-17721 we had changed the direct sql to acquire the lock for > postgres as > {code} > select "NEXT_EVENT_ID" from "NOTIFICATION_SEQUENCE" for update; > {code} > however this breaks other databases and we have to use different sql > statements for different databases > for postgres use > {code} > select "NEXT_EVENT_ID" from "NOTIFICATION_SEQUENCE" for update; > {code} > for SQLServer > {code} > select "NEXT_EVENT_ID" from "NOTIFICATION_SEQUENCE" with (updlock); > {code} > for other databases > {code} > select NEXT_EVENT_ID from NOTIFICATION_SEQUENCE for update; > {code} -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HIVE-17830) dbnotification fails to work with rdbms other than postgres
[ https://issues.apache.org/jira/browse/HIVE-17830?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16209639#comment-16209639 ] Thejas M Nair commented on HIVE-17830: -- [~anishek] Looking at metastore direct sql code, it looks like select can be common across dbs - metastoredirect sql does seem to use quoted column names, but not quoted table name - eg - MetaStoreDirectSQL.java - {code} String queryTextDbSelector= "select " + "\"DB_ID\", \"NAME\", \"DB_LOCATION_URI\", \"DESC\", " + "\"OWNER_NAME\", \"OWNER_TYPE\" " + "FROM "+ DBS +" where \"NAME\" = ? "; {code} > dbnotification fails to work with rdbms other than postgres > --- > > Key: HIVE-17830 > URL: https://issues.apache.org/jira/browse/HIVE-17830 > Project: Hive > Issue Type: Bug > Components: HiveServer2 >Reporter: anishek >Assignee: anishek >Priority: Critical > Fix For: 3.0.0 > > Attachments: HIVE-17830.0.patch > > > as part of HIVE-17721 we had changed the direct sql to acquire the lock for > postgres as > {code} > select "NEXT_EVENT_ID" from "NOTIFICATION_SEQUENCE" for update; > {code} > however this breaks other databases and we have to use different sql > statements for different databases > for postgres use > {code} > select "NEXT_EVENT_ID" from "NOTIFICATION_SEQUENCE" for update; > {code} > for SQLServer > {code} > select "NEXT_EVENT_ID" from "NOTIFICATION_SEQUENCE" with (updlock); > {code} > for other databases > {code} > select NEXT_EVENT_ID from NOTIFICATION_SEQUENCE for update; > {code} -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HIVE-17830) dbnotification fails to work with rdbms other than postgres
[ https://issues.apache.org/jira/browse/HIVE-17830?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16209213#comment-16209213 ] Hive QA commented on HIVE-17830: Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12892753/HIVE-17830.0.patch {color:red}ERROR:{color} -1 due to no test(s) being added or modified. {color:red}ERROR:{color} -1 due to 16 failed/errored test(s), 11277 tests executed *Failed tests:* {noformat} org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[optimize_nullscan] (batchId=163) org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver[explainanalyze_2] (batchId=101) org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[subquery_multi] (batchId=110) org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[subquery_notin] (batchId=133) org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[subquery_scalar] (batchId=119) org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[subquery_select] (batchId=119) org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[subquery_views] (batchId=108) org.apache.hadoop.hive.cli.TestSparkPerfCliDriver.testCliDriver[query16] (batchId=243) org.apache.hadoop.hive.cli.TestSparkPerfCliDriver.testCliDriver[query94] (batchId=243) org.apache.hadoop.hive.cli.TestTezPerfCliDriver.testCliDriver[query14] (batchId=241) org.apache.hadoop.hive.cli.TestTezPerfCliDriver.testCliDriver[query16] (batchId=241) org.apache.hadoop.hive.cli.TestTezPerfCliDriver.testCliDriver[query23] (batchId=241) org.apache.hadoop.hive.cli.TestTezPerfCliDriver.testCliDriver[query94] (batchId=241) org.apache.hadoop.hive.cli.control.TestDanglingQOuts.checkDanglingQOut (batchId=204) org.apache.hive.jdbc.TestTriggersTezSessionPoolManager.testTriggerHighShuffleBytes (batchId=229) org.apache.hive.jdbc.TestTriggersWorkloadManager.testTriggerHighShuffleBytes (batchId=229) {noformat} Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/7362/testReport Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/7362/console Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-7362/ Messages: {noformat} Executing org.apache.hive.ptest.execution.TestCheckPhase Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase Tests exited with: TestsFailedException: 16 tests failed {noformat} This message is automatically generated. ATTACHMENT ID: 12892753 - PreCommit-HIVE-Build > dbnotification fails to work with rdbms other than postgres > --- > > Key: HIVE-17830 > URL: https://issues.apache.org/jira/browse/HIVE-17830 > Project: Hive > Issue Type: Bug > Components: HiveServer2 >Reporter: anishek >Assignee: anishek >Priority: Critical > Fix For: 3.0.0 > > Attachments: HIVE-17830.0.patch > > > as part of HIVE-17721 we had changed the direct sql to acquire the lock for > postgres as > {code} > select "NEXT_EVENT_ID" from "NOTIFICATION_SEQUENCE" for update; > {code} > however this breaks other databases and we have to use different sql > statements for different databases > for postgres use > {code} > select "NEXT_EVENT_ID" from "NOTIFICATION_SEQUENCE" for update; > {code} > for SQLServer > {code} > select "NEXT_EVENT_ID" from "NOTIFICATION_SEQUENCE" with (updlock); > {code} > for other databases > {code} > select NEXT_EVENT_ID from NOTIFICATION_SEQUENCE for update; > {code} -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HIVE-17830) dbnotification fails to work with rdbms other than postgres
[ https://issues.apache.org/jira/browse/HIVE-17830?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16209007#comment-16209007 ] anishek commented on HIVE-17830: [~thejas] can you please review. > dbnotification fails to work with rdbms other than postgres > --- > > Key: HIVE-17830 > URL: https://issues.apache.org/jira/browse/HIVE-17830 > Project: Hive > Issue Type: Bug > Components: HiveServer2 >Reporter: anishek >Assignee: anishek >Priority: Critical > Fix For: 3.0.0 > > Attachments: HIVE-17830.0.patch > > > as part of HIVE-17721 we had changed the direct sql to acquire the lock for > postgres as > {code} > select "NEXT_EVENT_ID" from "NOTIFICATION_SEQUENCE" for update; > {code} > however this breaks other databases and we have to use different sql > statements for different databases > for postgres use > {code} > select "NEXT_EVENT_ID" from "NOTIFICATION_SEQUENCE" for update; > {code} > for SQLServer > {code} > select "NEXT_EVENT_ID" from "NOTIFICATION_SEQUENCE" with (updlock); > {code} > for other databases > {code} > select NEXT_EVENT_ID from NOTIFICATION_SEQUENCE for update; > {code} -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HIVE-17830) dbnotification fails to work with rdbms other than postgres
[ https://issues.apache.org/jira/browse/HIVE-17830?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16209004#comment-16209004 ] ASF GitHub Bot commented on HIVE-17830: --- GitHub user anishek opened a pull request: https://github.com/apache/hive/pull/263 HIVE-17830: dbnotification fails to work with rdbms other than postgres You can merge this pull request into a Git repository by running: $ git pull https://github.com/anishek/hive HIVE-17830 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/hive/pull/263.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #263 commit 03e1e2d58f3884ef88949e38976058794899c895 Author: Anishek Agarwal Date: 2017-10-18T08:53:47Z HIVE-17830: dbnotification fails to work with rdbms other than postgres > dbnotification fails to work with rdbms other than postgres > --- > > Key: HIVE-17830 > URL: https://issues.apache.org/jira/browse/HIVE-17830 > Project: Hive > Issue Type: Bug > Components: HiveServer2 >Reporter: anishek >Assignee: anishek >Priority: Critical > Fix For: 3.0.0 > > Attachments: HIVE-17830.0.patch > > > as part of HIVE-17721 we had changed the direct sql to acquire the lock for > postgres as > {code} > select "NEXT_EVENT_ID" from "NOTIFICATION_SEQUENCE" for update; > {code} > however this breaks other databases and we have to use different sql > statements for different databases > for postgres use > {code} > select "NEXT_EVENT_ID" from "NOTIFICATION_SEQUENCE" for update; > {code} > for SQLServer > {code} > select "NEXT_EVENT_ID" from "NOTIFICATION_SEQUENCE"with (updlock); > {code} > for other databases > {code} > select NEXT_EVENT_ID from NOTIFICATION_SEQUENCE for update; > {code} -- This message was sent by Atlassian JIRA (v6.4.14#64029)