[jira] [Updated] (HIVE-16449) BeeLineDriver should handle query result sorting
[ https://issues.apache.org/jira/browse/HIVE-16449?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Peter Vary updated HIVE-16449: -- Attachment: HIVE-16449.05.patch Addressed review comments: - Created ConvertedOutputFile.java, and moved converter specific code there - Minimized the changes in OutputFile > BeeLineDriver should handle query result sorting > > > Key: HIVE-16449 > URL: https://issues.apache.org/jira/browse/HIVE-16449 > Project: Hive > Issue Type: Improvement > Components: Testing Infrastructure >Affects Versions: 3.0.0 >Reporter: Peter Vary >Assignee: Peter Vary > Attachments: HIVE-16449.02.patch, HIVE-16449.03.patch, > HIVE-16449.04.patch, HIVE-16449.05.patch, HIVE-16449.patch > > > The CLI driver supports the following features: > -- SORT_QUERY_RESULTS > -- HASH_QUERY_RESULTS > -- SORT_AND_HASH_QUERY_RESULTS > BeeLineDriver should find a way to support these -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HIVE-16449) BeeLineDriver should handle query result sorting
[ https://issues.apache.org/jira/browse/HIVE-16449?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Peter Vary updated HIVE-16449: -- Status: Patch Available (was: Open) Let's see the PreCommit test results > BeeLineDriver should handle query result sorting > > > Key: HIVE-16449 > URL: https://issues.apache.org/jira/browse/HIVE-16449 > Project: Hive > Issue Type: Improvement > Components: Testing Infrastructure >Affects Versions: 3.0.0 >Reporter: Peter Vary >Assignee: Peter Vary > Attachments: HIVE-16449.patch > > > The CLI driver supports the following features: > -- SORT_BEFORE_DIFF > -- SORT_QUERY_RESULTS > -- HASH_QUERY_RESULTS > -- SORT_AND_HASH_QUERY_RESULTS > BeeLineDriver should find a way to support these -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HIVE-16449) BeeLineDriver should handle query result sorting
[ https://issues.apache.org/jira/browse/HIVE-16449?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Peter Vary updated HIVE-16449: -- Attachment: HIVE-16449.patch Added the possibility to the OutputFile, to use the existing converters on the BeeLine results > BeeLineDriver should handle query result sorting > > > Key: HIVE-16449 > URL: https://issues.apache.org/jira/browse/HIVE-16449 > Project: Hive > Issue Type: Improvement > Components: Testing Infrastructure >Affects Versions: 3.0.0 >Reporter: Peter Vary >Assignee: Peter Vary > Attachments: HIVE-16449.patch > > > The CLI driver supports the following features: > -- SORT_BEFORE_DIFF > -- SORT_QUERY_RESULTS > -- HASH_QUERY_RESULTS > -- SORT_AND_HASH_QUERY_RESULTS > BeeLineDriver should find a way to support these -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HIVE-16449) BeeLineDriver should handle query result sorting
[ https://issues.apache.org/jira/browse/HIVE-16449?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Peter Vary updated HIVE-16449: -- Description: The CLI driver supports the following features: -- SORT_QUERY_RESULTS -- HASH_QUERY_RESULTS -- SORT_AND_HASH_QUERY_RESULTS BeeLineDriver should find a way to support these was: The CLI driver supports the following features: -- SORT_BEFORE_DIFF -- SORT_QUERY_RESULTS -- HASH_QUERY_RESULTS -- SORT_AND_HASH_QUERY_RESULTS BeeLineDriver should find a way to support these > BeeLineDriver should handle query result sorting > > > Key: HIVE-16449 > URL: https://issues.apache.org/jira/browse/HIVE-16449 > Project: Hive > Issue Type: Improvement > Components: Testing Infrastructure >Affects Versions: 3.0.0 >Reporter: Peter Vary >Assignee: Peter Vary > Attachments: HIVE-16449.02.patch, HIVE-16449.patch > > > The CLI driver supports the following features: > -- SORT_QUERY_RESULTS > -- HASH_QUERY_RESULTS > -- SORT_AND_HASH_QUERY_RESULTS > BeeLineDriver should find a way to support these -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HIVE-16449) BeeLineDriver should handle query result sorting
[ https://issues.apache.org/jira/browse/HIVE-16449?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Peter Vary updated HIVE-16449: -- Attachment: HIVE-16449.02.patch Removed – SORT_BEFORE_DIFF, since it is implemented differently > BeeLineDriver should handle query result sorting > > > Key: HIVE-16449 > URL: https://issues.apache.org/jira/browse/HIVE-16449 > Project: Hive > Issue Type: Improvement > Components: Testing Infrastructure >Affects Versions: 3.0.0 >Reporter: Peter Vary >Assignee: Peter Vary > Attachments: HIVE-16449.02.patch, HIVE-16449.patch > > > The CLI driver supports the following features: > -- SORT_BEFORE_DIFF > -- SORT_QUERY_RESULTS > -- HASH_QUERY_RESULTS > -- SORT_AND_HASH_QUERY_RESULTS > BeeLineDriver should find a way to support these -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HIVE-16146) If possible find a better way to filter the TestBeeLineDriver output
[ https://issues.apache.org/jira/browse/HIVE-16146?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15968956#comment-15968956 ] Peter Vary commented on HIVE-16146: --- Not related failures: - HIVE-15535 - Flaky test : TestHS2HttpServer.testContextRootUrlRewrite - HIVE-15776 - Flaky test: TestMiniLlapLocalCliDriver vector_if_expr - HIVE-15289 - Flaky test: TestSparkCliDriver.org.apache.hadoop.hive.cli.TestSparkCliDriver (setup) Maybe related: - TestBeeLineDriver.testCliDriver[smb_mapjoin_11] Will check the last one, but I do not think it is related to this test. For the first glance it seems to me a more general race condition instead. > If possible find a better way to filter the TestBeeLineDriver output > > > Key: HIVE-16146 > URL: https://issues.apache.org/jira/browse/HIVE-16146 > Project: Hive > Issue Type: Improvement > Components: Testing Infrastructure >Affects Versions: 2.2.0 >Reporter: Peter Vary >Assignee: Peter Vary > Attachments: HIVE-16146.02.patch, HIVE-16146.03.patch, > HIVE-16146.04.patch, HIVE-16146.05.patch, HIVE-16146.06.patch, > HIVE-16146.patch > > > Currently we apply a blacklist to filter the output of the BeeLine Qtest runs. > It might be a good idea to go thorough of the possibilities and find a better > way, if possible. > I think our main goal could be for the TestBeeLineDriver test output to match > the TestCliDriver output of the came query file. Or if it is not possible, > then at least a similar one > CC: [~vihangk1] -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Assigned] (HIVE-16449) BeeLineDriver should handle query result sorting
[ https://issues.apache.org/jira/browse/HIVE-16449?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Peter Vary reassigned HIVE-16449: - > BeeLineDriver should handle query result sorting > > > Key: HIVE-16449 > URL: https://issues.apache.org/jira/browse/HIVE-16449 > Project: Hive > Issue Type: Improvement > Components: Testing Infrastructure >Affects Versions: 3.0.0 >Reporter: Peter Vary >Assignee: Peter Vary > > The CLI driver supports the following features: > -- SORT_BEFORE_DIFF > -- SORT_QUERY_RESULTS > -- HASH_QUERY_RESULTS > -- SORT_AND_HASH_QUERY_RESULTS > BeeLineDriver should find a way to support these -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HIVE-16146) If possible find a better way to filter the TestBeeLineDriver output
[ https://issues.apache.org/jira/browse/HIVE-16146?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Peter Vary updated HIVE-16146: -- Attachment: HIVE-16146.06.patch Rebased the patch. Since after the rebase the TestBeeLineDriver runs more query files, more replace patterns are needed: - The output contains the original queries, so we should rewrite (default.src->src) for the output file as well - More static regex for hiding the file locations, and serdes to match the CliDriver output - More specific regex for hiding the database change in the explain results Followup jira (HIVE-16449) is created to handle -- SORT_BEFORE_DIFF. Otherwise the query output files are the same as the CliDriver version output > If possible find a better way to filter the TestBeeLineDriver output > > > Key: HIVE-16146 > URL: https://issues.apache.org/jira/browse/HIVE-16146 > Project: Hive > Issue Type: Improvement > Components: Testing Infrastructure >Affects Versions: 2.2.0 >Reporter: Peter Vary >Assignee: Peter Vary > Attachments: HIVE-16146.02.patch, HIVE-16146.03.patch, > HIVE-16146.04.patch, HIVE-16146.05.patch, HIVE-16146.06.patch, > HIVE-16146.patch > > > Currently we apply a blacklist to filter the output of the BeeLine Qtest runs. > It might be a good idea to go thorough of the possibilities and find a better > way, if possible. > I think our main goal could be for the TestBeeLineDriver test output to match > the TestCliDriver output of the came query file. Or if it is not possible, > then at least a similar one > CC: [~vihangk1] -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HIVE-16416) Service: move constants out from HiveAuthFactory
[ https://issues.apache.org/jira/browse/HIVE-16416?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15968706#comment-15968706 ] Peter Vary commented on HIVE-16416: --- +1 LGTM (non-binding) > Service: move constants out from HiveAuthFactory > > > Key: HIVE-16416 > URL: https://issues.apache.org/jira/browse/HIVE-16416 > Project: Hive > Issue Type: Sub-task > Components: Server Infrastructure >Reporter: Zoltan Haindrich >Assignee: Zoltan Haindrich > Attachments: HIVE-16416.1.patch > > > It took me a while to notice that there are only some constants which are > keep pulling in this class :) > it contains a tricky dependency to the whole ql module; but in client mode > that part is totally unused - moving the constants out from it, enables the > client to operate without the factory. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HIVE-16355) Service: embedded mode should only be available if service is loaded onto the classpath
[
https://issues.apache.org/jira/browse/HIVE-16355?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15968710#comment-15968710
]
Peter Vary commented on HIVE-16355:
---
Hi [~kgyrtkirk],
How complicated would be to use serviceloader as stated in the comment? I would
prefer that solution above using reflection.
Thanks,
Peter
> Service: embedded mode should only be available if service is loaded onto the
> classpath
> ---
>
> Key: HIVE-16355
> URL: https://issues.apache.org/jira/browse/HIVE-16355
> Project: Hive
> Issue Type: Sub-task
> Components: Metastore, Server Infrastructure
>Reporter: Zoltan Haindrich
>Assignee: Zoltan Haindrich
> Attachments: HIVE-16355.1.patch, HIVE-16355.2.patch,
> HIVE-16355.2.patch
>
>
> I would like to relax the hard reference to
> {{EmbeddedThriftBinaryCLIService}} to be only used in case {{service}} module
> is loaded onto the classpath.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Resolved] (HIVE-16320) Flaky test: TestBeeLineDriver.testCliDriver[escape_comments]
[ https://issues.apache.org/jira/browse/HIVE-16320?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Peter Vary resolved HIVE-16320. --- Resolution: Done Fix Version/s: 3.0.0 Is should be solved by now... > Flaky test: TestBeeLineDriver.testCliDriver[escape_comments] > > > Key: HIVE-16320 > URL: https://issues.apache.org/jira/browse/HIVE-16320 > Project: Hive > Issue Type: Sub-task >Reporter: Sahil Takiar >Assignee: Peter Vary > Fix For: 3.0.0 > > -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HIVE-16146) If possible find a better way to filter the TestBeeLineDriver output
[ https://issues.apache.org/jira/browse/HIVE-16146?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15969075#comment-15969075 ] Peter Vary commented on HIVE-16146: --- Checked the TestBeeLineDriver.testCliDriver[smb_mapjoin_11] error. It is caused by this: - HIVE-16451- Race condition between HiveStatement.getQueryLog and HiveStatement.runAsyncOnServer So the errors are unrelated > If possible find a better way to filter the TestBeeLineDriver output > > > Key: HIVE-16146 > URL: https://issues.apache.org/jira/browse/HIVE-16146 > Project: Hive > Issue Type: Improvement > Components: Testing Infrastructure >Affects Versions: 2.2.0 >Reporter: Peter Vary >Assignee: Peter Vary > Attachments: HIVE-16146.02.patch, HIVE-16146.03.patch, > HIVE-16146.04.patch, HIVE-16146.05.patch, HIVE-16146.06.patch, > HIVE-16146.patch > > > Currently we apply a blacklist to filter the output of the BeeLine Qtest runs. > It might be a good idea to go thorough of the possibilities and find a better > way, if possible. > I think our main goal could be for the TestBeeLineDriver test output to match > the TestCliDriver output of the came query file. Or if it is not possible, > then at least a similar one > CC: [~vihangk1] -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HIVE-16451) Race condition between HiveStatement.getQueryLog and HiveStatement.runAsyncOnServer
[
https://issues.apache.org/jira/browse/HIVE-16451?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15969081#comment-15969081
]
Peter Vary commented on HIVE-16451:
---
Confirmed by adding sleep between the two method, like this:
{code:title=HiveStatement.runAsyncOnServer}
private void runAsyncOnServer(String sql) throws SQLException {
checkConnection("execute");
closeClientOperation();
try {
Thread.currentThread().sleep(8000L);
} catch (Exception e) {
e.printStackTrace();
}
initFlags();
[..]
}
{code}
This reproduced the errors.
> Race condition between HiveStatement.getQueryLog and
> HiveStatement.runAsyncOnServer
> ---
>
> Key: HIVE-16451
> URL: https://issues.apache.org/jira/browse/HIVE-16451
> Project: Hive
> Issue Type: Bug
> Components: Beeline
>Affects Versions: 3.0.0
>Reporter: Peter Vary
>Assignee: Peter Vary
>
> During the BeeLineDriver testing I have met the following race condition:
> - Run the query asynchronously through BeeLine
> - Querying the logs in the BeeLine
> In the following code:
> {code:title=HiveStatement.runAsyncOnServer}
> private void runAsyncOnServer(String sql) throws SQLException {
> checkConnection("execute");
> closeClientOperation();
> initFlags();
> [..]
> }
> {code}
> {code:title=HiveStatement.getQueryLog}
> public List getQueryLog(boolean incremental, int fetchSize)
> throws SQLException, ClosedOrCancelledStatementException {
> [..]
> try {
> if (stmtHandle != null) {
> [..]
> } else {
> if (isQueryClosed) {
> throw new ClosedOrCancelledStatementException("Method getQueryLog()
> failed. The " +
> "statement has been closed or cancelled.");
> } else {
> return logs;
> }
> }
> } catch (SQLException e) {
> [..]
> }
> [..]
> }
> {code}
> The runAsyncOnServer {{closeClientOperation}} sets {{isQueryClosed}} flag to
> true:
> {code:title=HiveStatement.closeClientOperation}
> void closeClientOperation() throws SQLException {
> [..]
> isQueryClosed = true;
> isExecuteStatementFailed = false;
> stmtHandle = null;
> }
> {code}
> The {{initFlags}} sets it to false:
> {code}
> private void initFlags() {
> isCancelled = false;
> isQueryClosed = false;
> isLogBeingGenerated = true;
> isExecuteStatementFailed = false;
> isOperationComplete = false;
> }
> {code}
> If the {{getQueryLog}} is called after the {{closeClientOperation}}, but
> before the {{initFlags}}, then we will have a following warning if verbose
> mode is set to true in BeeLine:
> {code}
> Warning: org.apache.hive.jdbc.ClosedOrCancelledStatementException: Method
> getQueryLog() failed. The statement has been closed or cancelled.
> (state=,code=0)
> {code}
> This caused this fail:
> https://builds.apache.org/job/PreCommit-HIVE-Build/4691/testReport/org.apache.hadoop.hive.cli/TestBeeLineDriver/testCliDriver_smb_mapjoin_11_/
> {code}
> Error Message
> Client result comparison failed with error code = 1 while executing
> fname=smb_mapjoin_11
> 16a17
> > Warning: org.apache.hive.jdbc.ClosedOrCancelledStatementException: Method
> > getQueryLog() failed. The statement has been closed or cancelled.
> > (state=,code=0)
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Assigned] (HIVE-16451) Race condition between HiveStatement.getQueryLog and HiveStatement.runAsyncOnServer
[
https://issues.apache.org/jira/browse/HIVE-16451?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Peter Vary reassigned HIVE-16451:
-
> Race condition between HiveStatement.getQueryLog and
> HiveStatement.runAsyncOnServer
> ---
>
> Key: HIVE-16451
> URL: https://issues.apache.org/jira/browse/HIVE-16451
> Project: Hive
> Issue Type: Bug
> Components: Beeline
>Affects Versions: 3.0.0
>Reporter: Peter Vary
>Assignee: Peter Vary
>
> During the BeeLineDriver testing I have met the following race condition:
> - Run the query asynchronously through BeeLine
> - Querying the logs in the BeeLine
> In the following code:
> {code:title=HiveStatement.runAsyncOnServer}
> private void runAsyncOnServer(String sql) throws SQLException {
> checkConnection("execute");
> closeClientOperation();
> initFlags();
> [..]
> }
> {code}
> {code:title=HiveStatement.getQueryLog}
> public List getQueryLog(boolean incremental, int fetchSize)
> throws SQLException, ClosedOrCancelledStatementException {
> [..]
> try {
> if (stmtHandle != null) {
> [..]
> } else {
> if (isQueryClosed) {
> throw new ClosedOrCancelledStatementException("Method getQueryLog()
> failed. The " +
> "statement has been closed or cancelled.");
> } else {
> return logs;
> }
> }
> } catch (SQLException e) {
> [..]
> }
> [..]
> }
> {code}
> The runAsyncOnServer {{closeClientOperation}} sets {{isQueryClosed}} flag to
> true:
> {code:title=HiveStatement.closeClientOperation}
> void closeClientOperation() throws SQLException {
> [..]
> isQueryClosed = true;
> isExecuteStatementFailed = false;
> stmtHandle = null;
> }
> {code}
> The {{initFlags}} sets it to false:
> {code}
> private void initFlags() {
> isCancelled = false;
> isQueryClosed = false;
> isLogBeingGenerated = true;
> isExecuteStatementFailed = false;
> isOperationComplete = false;
> }
> {code}
> If the {{getQueryLog}} is called after the {{closeClientOperation}}, but
> before the {{initFlags}}, then we will have a following warning if verbose
> mode is set to true in BeeLine:
> {code}
> Warning: org.apache.hive.jdbc.ClosedOrCancelledStatementException: Method
> getQueryLog() failed. The statement has been closed or cancelled.
> (state=,code=0)
> {code}
> This caused this fail:
> https://builds.apache.org/job/PreCommit-HIVE-Build/4691/testReport/org.apache.hadoop.hive.cli/TestBeeLineDriver/testCliDriver_smb_mapjoin_11_/
> {code}
> Error Message
> Client result comparison failed with error code = 1 while executing
> fname=smb_mapjoin_11
> 16a17
> > Warning: org.apache.hive.jdbc.ClosedOrCancelledStatementException: Method
> > getQueryLog() failed. The statement has been closed or cancelled.
> > (state=,code=0)
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Assigned] (HIVE-16487) Serious Zookeeper exception is logged when a race condition happens
[
https://issues.apache.org/jira/browse/HIVE-16487?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Peter Vary reassigned HIVE-16487:
-
> Serious Zookeeper exception is logged when a race condition happens
> ---
>
> Key: HIVE-16487
> URL: https://issues.apache.org/jira/browse/HIVE-16487
> Project: Hive
> Issue Type: Bug
> Components: Locking
>Affects Versions: 3.0.0
>Reporter: Peter Vary
>Assignee: Peter Vary
>
> A customer started to see this in the logs, but happily everything was
> working as intended:
> {code}
> 2017-03-30 12:01:59,446 ERROR ZooKeeperHiveLockManager:
> [HiveServer2-Background-Pool: Thread-620]: Serious Zookeeper exception:
> org.apache.zookeeper.KeeperException$NoNodeException: KeeperErrorCode =
> NoNode for /hive_zookeeper_namespace//LOCK-SHARED-
> {code}
> This was happening, because a race condition between the lock releasing, and
> lock acquiring. The thread releasing the lock removes the parent ZK node just
> after the thread acquiring the lock made sure, that the parent node exists.
> Since this can happen without any real problem, I plan to add NODEEXISTS, and
> NONODE as a transient ZooKeeper exception, so the users are not confused.
> Also, the original author of ZooKeeperHiveLockManager maybe planned to handle
> different ZooKeeperExceptions differently, and the code is hard to
> understand. See the {{continue}} and the {{break}}. The {{break}} only breaks
> the switch, and not the loop which IMHO is not intuitive:
> {code}
> do {
> try {
> [..]
> ret = lockPrimitive(key, mode, keepAlive, parentCreated,
> } catch (Exception e1) {
> if (e1 instanceof KeeperException) {
> KeeperException e = (KeeperException) e1;
> switch (e.code()) {
> case CONNECTIONLOSS:
> case OPERATIONTIMEOUT:
> LOG.debug("Possibly transient ZooKeeper exception: ", e);
> continue;
> default:
> LOG.error("Serious Zookeeper exception: ", e);
> break;
> }
> }
> [..]
> }
> } while (tryNum < numRetriesForLock);
> {code}
> If we do not want to try again in case of a "Serious Zookeeper exception:",
> then we should add a label to the do loop, and break it in the switch.
> If we do want to try regardless of the type of the ZK exception, then we
> should just change the {{continue;}} to {{break;}} and move the lines part of
> the code which did not run in case of {{continue}} to the {{default}} switch,
> so it is easier to understand the code.
> Any suggestions or ideas [~ctang.ma] or [~szehon]?
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HIVE-16451) Race condition between HiveStatement.getQueryLog and HiveStatement.runAsyncOnServer
[
https://issues.apache.org/jira/browse/HIVE-16451?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15977107#comment-15977107
]
Peter Vary commented on HIVE-16451:
---
[~vihangk1], [~ychena] could you please take a look?
Thanks,
Peter
> Race condition between HiveStatement.getQueryLog and
> HiveStatement.runAsyncOnServer
> ---
>
> Key: HIVE-16451
> URL: https://issues.apache.org/jira/browse/HIVE-16451
> Project: Hive
> Issue Type: Bug
> Components: Beeline, JDBC
>Affects Versions: 3.0.0
>Reporter: Peter Vary
>Assignee: Peter Vary
> Attachments: HIVE-16451.02.patch, HIVE-16451.03.patch,
> HIVE-16451.patch
>
>
> During the BeeLineDriver testing I have met the following race condition:
> - Run the query asynchronously through BeeLine
> - Querying the logs in the BeeLine
> In the following code:
> {code:title=HiveStatement.runAsyncOnServer}
> private void runAsyncOnServer(String sql) throws SQLException {
> checkConnection("execute");
> closeClientOperation();
> initFlags();
> [..]
> }
> {code}
> {code:title=HiveStatement.getQueryLog}
> public List getQueryLog(boolean incremental, int fetchSize)
> throws SQLException, ClosedOrCancelledStatementException {
> [..]
> try {
> if (stmtHandle != null) {
> [..]
> } else {
> if (isQueryClosed) {
> throw new ClosedOrCancelledStatementException("Method getQueryLog()
> failed. The " +
> "statement has been closed or cancelled.");
> } else {
> return logs;
> }
> }
> } catch (SQLException e) {
> [..]
> }
> [..]
> }
> {code}
> The runAsyncOnServer {{closeClientOperation}} sets {{isQueryClosed}} flag to
> true:
> {code:title=HiveStatement.closeClientOperation}
> void closeClientOperation() throws SQLException {
> [..]
> isQueryClosed = true;
> isExecuteStatementFailed = false;
> stmtHandle = null;
> }
> {code}
> The {{initFlags}} sets it to false:
> {code}
> private void initFlags() {
> isCancelled = false;
> isQueryClosed = false;
> isLogBeingGenerated = true;
> isExecuteStatementFailed = false;
> isOperationComplete = false;
> }
> {code}
> If the {{getQueryLog}} is called after the {{closeClientOperation}}, but
> before the {{initFlags}}, then we will have a following warning if verbose
> mode is set to true in BeeLine:
> {code}
> Warning: org.apache.hive.jdbc.ClosedOrCancelledStatementException: Method
> getQueryLog() failed. The statement has been closed or cancelled.
> (state=,code=0)
> {code}
> This caused this fail:
> https://builds.apache.org/job/PreCommit-HIVE-Build/4691/testReport/org.apache.hadoop.hive.cli/TestBeeLineDriver/testCliDriver_smb_mapjoin_11_/
> {code}
> Error Message
> Client result comparison failed with error code = 1 while executing
> fname=smb_mapjoin_11
> 16a17
> > Warning: org.apache.hive.jdbc.ClosedOrCancelledStatementException: Method
> > getQueryLog() failed. The statement has been closed or cancelled.
> > (state=,code=0)
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Updated] (HIVE-16487) Serious Zookeeper exception is logged when a race condition happens
[
https://issues.apache.org/jira/browse/HIVE-16487?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Peter Vary updated HIVE-16487:
--
Status: Patch Available (was: Open)
> Serious Zookeeper exception is logged when a race condition happens
> ---
>
> Key: HIVE-16487
> URL: https://issues.apache.org/jira/browse/HIVE-16487
> Project: Hive
> Issue Type: Bug
> Components: Locking
>Affects Versions: 3.0.0
>Reporter: Peter Vary
>Assignee: Peter Vary
> Attachments: HIVE-16487.patch
>
>
> A customer started to see this in the logs, but happily everything was
> working as intended:
> {code}
> 2017-03-30 12:01:59,446 ERROR ZooKeeperHiveLockManager:
> [HiveServer2-Background-Pool: Thread-620]: Serious Zookeeper exception:
> org.apache.zookeeper.KeeperException$NoNodeException: KeeperErrorCode =
> NoNode for /hive_zookeeper_namespace//LOCK-SHARED-
> {code}
> This was happening, because a race condition between the lock releasing, and
> lock acquiring. The thread releasing the lock removes the parent ZK node just
> after the thread acquiring the lock made sure, that the parent node exists.
> Since this can happen without any real problem, I plan to add NODEEXISTS, and
> NONODE as a transient ZooKeeper exception, so the users are not confused.
> Also, the original author of ZooKeeperHiveLockManager maybe planned to handle
> different ZooKeeperExceptions differently, and the code is hard to
> understand. See the {{continue}} and the {{break}}. The {{break}} only breaks
> the switch, and not the loop which IMHO is not intuitive:
> {code}
> do {
> try {
> [..]
> ret = lockPrimitive(key, mode, keepAlive, parentCreated,
> } catch (Exception e1) {
> if (e1 instanceof KeeperException) {
> KeeperException e = (KeeperException) e1;
> switch (e.code()) {
> case CONNECTIONLOSS:
> case OPERATIONTIMEOUT:
> LOG.debug("Possibly transient ZooKeeper exception: ", e);
> continue;
> default:
> LOG.error("Serious Zookeeper exception: ", e);
> break;
> }
> }
> [..]
> }
> } while (tryNum < numRetriesForLock);
> {code}
> If we do not want to try again in case of a "Serious Zookeeper exception:",
> then we should add a label to the do loop, and break it in the switch.
> If we do want to try regardless of the type of the ZK exception, then we
> should just change the {{continue;}} to {{break;}} and move the lines part of
> the code which did not run in case of {{continue}} to the {{default}} switch,
> so it is easier to understand the code.
> Any suggestions or ideas [~ctang.ma] or [~szehon]?
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Updated] (HIVE-16487) Serious Zookeeper exception is logged when a race condition happens
[
https://issues.apache.org/jira/browse/HIVE-16487?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Peter Vary updated HIVE-16487:
--
Attachment: HIVE-16487.patch
The proposed patch:
- Keeping the retying behavior
- NONODE, and NODEEXISTS are not Serious Zookeeper exceptions
- Moved the exit check into the switch
> Serious Zookeeper exception is logged when a race condition happens
> ---
>
> Key: HIVE-16487
> URL: https://issues.apache.org/jira/browse/HIVE-16487
> Project: Hive
> Issue Type: Bug
> Components: Locking
>Affects Versions: 3.0.0
>Reporter: Peter Vary
>Assignee: Peter Vary
> Attachments: HIVE-16487.patch
>
>
> A customer started to see this in the logs, but happily everything was
> working as intended:
> {code}
> 2017-03-30 12:01:59,446 ERROR ZooKeeperHiveLockManager:
> [HiveServer2-Background-Pool: Thread-620]: Serious Zookeeper exception:
> org.apache.zookeeper.KeeperException$NoNodeException: KeeperErrorCode =
> NoNode for /hive_zookeeper_namespace//LOCK-SHARED-
> {code}
> This was happening, because a race condition between the lock releasing, and
> lock acquiring. The thread releasing the lock removes the parent ZK node just
> after the thread acquiring the lock made sure, that the parent node exists.
> Since this can happen without any real problem, I plan to add NODEEXISTS, and
> NONODE as a transient ZooKeeper exception, so the users are not confused.
> Also, the original author of ZooKeeperHiveLockManager maybe planned to handle
> different ZooKeeperExceptions differently, and the code is hard to
> understand. See the {{continue}} and the {{break}}. The {{break}} only breaks
> the switch, and not the loop which IMHO is not intuitive:
> {code}
> do {
> try {
> [..]
> ret = lockPrimitive(key, mode, keepAlive, parentCreated,
> } catch (Exception e1) {
> if (e1 instanceof KeeperException) {
> KeeperException e = (KeeperException) e1;
> switch (e.code()) {
> case CONNECTIONLOSS:
> case OPERATIONTIMEOUT:
> LOG.debug("Possibly transient ZooKeeper exception: ", e);
> continue;
> default:
> LOG.error("Serious Zookeeper exception: ", e);
> break;
> }
> }
> [..]
> }
> } while (tryNum < numRetriesForLock);
> {code}
> If we do not want to try again in case of a "Serious Zookeeper exception:",
> then we should add a label to the do loop, and break it in the switch.
> If we do want to try regardless of the type of the ZK exception, then we
> should just change the {{continue;}} to {{break;}} and move the lines part of
> the code which did not run in case of {{continue}} to the {{default}} switch,
> so it is easier to understand the code.
> Any suggestions or ideas [~ctang.ma] or [~szehon]?
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Updated] (HIVE-16451) Race condition between HiveStatement.getQueryLog and HiveStatement.runAsyncOnServer
[
https://issues.apache.org/jira/browse/HIVE-16451?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Peter Vary updated HIVE-16451:
--
Status: Patch Available (was: Open)
> Race condition between HiveStatement.getQueryLog and
> HiveStatement.runAsyncOnServer
> ---
>
> Key: HIVE-16451
> URL: https://issues.apache.org/jira/browse/HIVE-16451
> Project: Hive
> Issue Type: Bug
> Components: Beeline
>Affects Versions: 3.0.0
>Reporter: Peter Vary
>Assignee: Peter Vary
> Attachments: HIVE-16451.patch
>
>
> During the BeeLineDriver testing I have met the following race condition:
> - Run the query asynchronously through BeeLine
> - Querying the logs in the BeeLine
> In the following code:
> {code:title=HiveStatement.runAsyncOnServer}
> private void runAsyncOnServer(String sql) throws SQLException {
> checkConnection("execute");
> closeClientOperation();
> initFlags();
> [..]
> }
> {code}
> {code:title=HiveStatement.getQueryLog}
> public List getQueryLog(boolean incremental, int fetchSize)
> throws SQLException, ClosedOrCancelledStatementException {
> [..]
> try {
> if (stmtHandle != null) {
> [..]
> } else {
> if (isQueryClosed) {
> throw new ClosedOrCancelledStatementException("Method getQueryLog()
> failed. The " +
> "statement has been closed or cancelled.");
> } else {
> return logs;
> }
> }
> } catch (SQLException e) {
> [..]
> }
> [..]
> }
> {code}
> The runAsyncOnServer {{closeClientOperation}} sets {{isQueryClosed}} flag to
> true:
> {code:title=HiveStatement.closeClientOperation}
> void closeClientOperation() throws SQLException {
> [..]
> isQueryClosed = true;
> isExecuteStatementFailed = false;
> stmtHandle = null;
> }
> {code}
> The {{initFlags}} sets it to false:
> {code}
> private void initFlags() {
> isCancelled = false;
> isQueryClosed = false;
> isLogBeingGenerated = true;
> isExecuteStatementFailed = false;
> isOperationComplete = false;
> }
> {code}
> If the {{getQueryLog}} is called after the {{closeClientOperation}}, but
> before the {{initFlags}}, then we will have a following warning if verbose
> mode is set to true in BeeLine:
> {code}
> Warning: org.apache.hive.jdbc.ClosedOrCancelledStatementException: Method
> getQueryLog() failed. The statement has been closed or cancelled.
> (state=,code=0)
> {code}
> This caused this fail:
> https://builds.apache.org/job/PreCommit-HIVE-Build/4691/testReport/org.apache.hadoop.hive.cli/TestBeeLineDriver/testCliDriver_smb_mapjoin_11_/
> {code}
> Error Message
> Client result comparison failed with error code = 1 while executing
> fname=smb_mapjoin_11
> 16a17
> > Warning: org.apache.hive.jdbc.ClosedOrCancelledStatementException: Method
> > getQueryLog() failed. The statement has been closed or cancelled.
> > (state=,code=0)
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Updated] (HIVE-16451) Race condition between HiveStatement.getQueryLog and HiveStatement.runAsyncOnServer
[
https://issues.apache.org/jira/browse/HIVE-16451?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Peter Vary updated HIVE-16451:
--
Attachment: HIVE-16451.patch
A patch to solve the current problem.
Create a new method {{closeStatementIfNeeded}}, which only closes the
statement, but does not touch the flags.
Using this method should solve the problem, since the flags are not
flip-flopping any more.
> Race condition between HiveStatement.getQueryLog and
> HiveStatement.runAsyncOnServer
> ---
>
> Key: HIVE-16451
> URL: https://issues.apache.org/jira/browse/HIVE-16451
> Project: Hive
> Issue Type: Bug
> Components: Beeline
>Affects Versions: 3.0.0
>Reporter: Peter Vary
>Assignee: Peter Vary
> Attachments: HIVE-16451.patch
>
>
> During the BeeLineDriver testing I have met the following race condition:
> - Run the query asynchronously through BeeLine
> - Querying the logs in the BeeLine
> In the following code:
> {code:title=HiveStatement.runAsyncOnServer}
> private void runAsyncOnServer(String sql) throws SQLException {
> checkConnection("execute");
> closeClientOperation();
> initFlags();
> [..]
> }
> {code}
> {code:title=HiveStatement.getQueryLog}
> public List getQueryLog(boolean incremental, int fetchSize)
> throws SQLException, ClosedOrCancelledStatementException {
> [..]
> try {
> if (stmtHandle != null) {
> [..]
> } else {
> if (isQueryClosed) {
> throw new ClosedOrCancelledStatementException("Method getQueryLog()
> failed. The " +
> "statement has been closed or cancelled.");
> } else {
> return logs;
> }
> }
> } catch (SQLException e) {
> [..]
> }
> [..]
> }
> {code}
> The runAsyncOnServer {{closeClientOperation}} sets {{isQueryClosed}} flag to
> true:
> {code:title=HiveStatement.closeClientOperation}
> void closeClientOperation() throws SQLException {
> [..]
> isQueryClosed = true;
> isExecuteStatementFailed = false;
> stmtHandle = null;
> }
> {code}
> The {{initFlags}} sets it to false:
> {code}
> private void initFlags() {
> isCancelled = false;
> isQueryClosed = false;
> isLogBeingGenerated = true;
> isExecuteStatementFailed = false;
> isOperationComplete = false;
> }
> {code}
> If the {{getQueryLog}} is called after the {{closeClientOperation}}, but
> before the {{initFlags}}, then we will have a following warning if verbose
> mode is set to true in BeeLine:
> {code}
> Warning: org.apache.hive.jdbc.ClosedOrCancelledStatementException: Method
> getQueryLog() failed. The statement has been closed or cancelled.
> (state=,code=0)
> {code}
> This caused this fail:
> https://builds.apache.org/job/PreCommit-HIVE-Build/4691/testReport/org.apache.hadoop.hive.cli/TestBeeLineDriver/testCliDriver_smb_mapjoin_11_/
> {code}
> Error Message
> Client result comparison failed with error code = 1 while executing
> fname=smb_mapjoin_11
> 16a17
> > Warning: org.apache.hive.jdbc.ClosedOrCancelledStatementException: Method
> > getQueryLog() failed. The statement has been closed or cancelled.
> > (state=,code=0)
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Updated] (HIVE-16449) BeeLineDriver should handle query result sorting
[ https://issues.apache.org/jira/browse/HIVE-16449?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Peter Vary updated HIVE-16449: -- Attachment: HIVE-16449.03.patch Added null check, which caused test failures. > BeeLineDriver should handle query result sorting > > > Key: HIVE-16449 > URL: https://issues.apache.org/jira/browse/HIVE-16449 > Project: Hive > Issue Type: Improvement > Components: Testing Infrastructure >Affects Versions: 3.0.0 >Reporter: Peter Vary >Assignee: Peter Vary > Attachments: HIVE-16449.02.patch, HIVE-16449.03.patch, > HIVE-16449.patch > > > The CLI driver supports the following features: > -- SORT_QUERY_RESULTS > -- HASH_QUERY_RESULTS > -- SORT_AND_HASH_QUERY_RESULTS > BeeLineDriver should find a way to support these -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HIVE-16467) Flaky test: TestCliDriver.testCliDriver[vector_order_null]
[
https://issues.apache.org/jira/browse/HIVE-16467?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Peter Vary updated HIVE-16467:
--
Attachment: HIVE-16467.03.patch
There were test infra problems, so the tests did not run
> Flaky test: TestCliDriver.testCliDriver[vector_order_null]
> --
>
> Key: HIVE-16467
> URL: https://issues.apache.org/jira/browse/HIVE-16467
> Project: Hive
> Issue Type: Sub-task
>Reporter: Peter Vary
>Assignee: Peter Vary
> Attachments: HIVE-16467.02.patch, HIVE-16467.03.patch,
> HIVE-16467.patch
>
>
> Seems to me, that we have to add a {{-- SORT_QUERY_RESULTS}} to the query
> test.
> The error message is
> {code}
> Client Execution succeeded but contained differences (error code = 1) after
> executing vector_order_null.q
> 153a154
> > 2 B
> 156d156
> < 2 B
> 247a248
> > 2 B
> 250d250
> < 2 B
> 531a532
> > 2 B
> 534d534
> < 2 B
> 626a627
> > 2 B
> 629d629
> < 2 B
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Assigned] (HIVE-16468) BeeLineDriver should be able to run tests against an externally created cluster
[ https://issues.apache.org/jira/browse/HIVE-16468?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Peter Vary reassigned HIVE-16468: - > BeeLineDriver should be able to run tests against an externally created > cluster > --- > > Key: HIVE-16468 > URL: https://issues.apache.org/jira/browse/HIVE-16468 > Project: Hive > Issue Type: Bug > Components: Testing Infrastructure >Reporter: Peter Vary >Assignee: Peter Vary > > It should be possible to run the query tests against an externally created > cluster using the BeeLineDriver, and the query results. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HIVE-16468) BeeLineDriver should be able to run tests against an externally created cluster
[ https://issues.apache.org/jira/browse/HIVE-16468?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Peter Vary updated HIVE-16468: -- Description: It should be possible to run the query tests against an externally created cluster using the BeeLineDriver, and the query files and results. (was: It should be possible to run the query tests against an externally created cluster using the BeeLineDriver, and the query results.) > BeeLineDriver should be able to run tests against an externally created > cluster > --- > > Key: HIVE-16468 > URL: https://issues.apache.org/jira/browse/HIVE-16468 > Project: Hive > Issue Type: Bug > Components: Testing Infrastructure >Reporter: Peter Vary >Assignee: Peter Vary > > It should be possible to run the query tests against an externally created > cluster using the BeeLineDriver, and the query files and results. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HIVE-16467) Flaky test: TestCliDriver.testCliDriver[vector_order_null]
[
https://issues.apache.org/jira/browse/HIVE-16467?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Peter Vary updated HIVE-16467:
--
Attachment: HIVE-16467.02.patch
Cleaned up test files...
> Flaky test: TestCliDriver.testCliDriver[vector_order_null]
> --
>
> Key: HIVE-16467
> URL: https://issues.apache.org/jira/browse/HIVE-16467
> Project: Hive
> Issue Type: Sub-task
>Reporter: Peter Vary
>Assignee: Peter Vary
> Attachments: HIVE-16467.02.patch, HIVE-16467.patch
>
>
> Seems to me, that we have to add a {{-- SORT_QUERY_RESULTS}} to the query
> test.
> The error message is
> {code}
> Client Execution succeeded but contained differences (error code = 1) after
> executing vector_order_null.q
> 153a154
> > 2 B
> 156d156
> < 2 B
> 247a248
> > 2 B
> 250d250
> < 2 B
> 531a532
> > 2 B
> 534d534
> < 2 B
> 626a627
> > 2 B
> 629d629
> < 2 B
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Updated] (HIVE-16451) Race condition between HiveStatement.getQueryLog and HiveStatement.runAsyncOnServer
[
https://issues.apache.org/jira/browse/HIVE-16451?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Peter Vary updated HIVE-16451:
--
Component/s: JDBC
> Race condition between HiveStatement.getQueryLog and
> HiveStatement.runAsyncOnServer
> ---
>
> Key: HIVE-16451
> URL: https://issues.apache.org/jira/browse/HIVE-16451
> Project: Hive
> Issue Type: Bug
> Components: Beeline, JDBC
>Affects Versions: 3.0.0
>Reporter: Peter Vary
>Assignee: Peter Vary
> Attachments: HIVE-16451.02.patch, HIVE-16451.03.patch,
> HIVE-16451.patch
>
>
> During the BeeLineDriver testing I have met the following race condition:
> - Run the query asynchronously through BeeLine
> - Querying the logs in the BeeLine
> In the following code:
> {code:title=HiveStatement.runAsyncOnServer}
> private void runAsyncOnServer(String sql) throws SQLException {
> checkConnection("execute");
> closeClientOperation();
> initFlags();
> [..]
> }
> {code}
> {code:title=HiveStatement.getQueryLog}
> public List getQueryLog(boolean incremental, int fetchSize)
> throws SQLException, ClosedOrCancelledStatementException {
> [..]
> try {
> if (stmtHandle != null) {
> [..]
> } else {
> if (isQueryClosed) {
> throw new ClosedOrCancelledStatementException("Method getQueryLog()
> failed. The " +
> "statement has been closed or cancelled.");
> } else {
> return logs;
> }
> }
> } catch (SQLException e) {
> [..]
> }
> [..]
> }
> {code}
> The runAsyncOnServer {{closeClientOperation}} sets {{isQueryClosed}} flag to
> true:
> {code:title=HiveStatement.closeClientOperation}
> void closeClientOperation() throws SQLException {
> [..]
> isQueryClosed = true;
> isExecuteStatementFailed = false;
> stmtHandle = null;
> }
> {code}
> The {{initFlags}} sets it to false:
> {code}
> private void initFlags() {
> isCancelled = false;
> isQueryClosed = false;
> isLogBeingGenerated = true;
> isExecuteStatementFailed = false;
> isOperationComplete = false;
> }
> {code}
> If the {{getQueryLog}} is called after the {{closeClientOperation}}, but
> before the {{initFlags}}, then we will have a following warning if verbose
> mode is set to true in BeeLine:
> {code}
> Warning: org.apache.hive.jdbc.ClosedOrCancelledStatementException: Method
> getQueryLog() failed. The statement has been closed or cancelled.
> (state=,code=0)
> {code}
> This caused this fail:
> https://builds.apache.org/job/PreCommit-HIVE-Build/4691/testReport/org.apache.hadoop.hive.cli/TestBeeLineDriver/testCliDriver_smb_mapjoin_11_/
> {code}
> Error Message
> Client result comparison failed with error code = 1 while executing
> fname=smb_mapjoin_11
> 16a17
> > Warning: org.apache.hive.jdbc.ClosedOrCancelledStatementException: Method
> > getQueryLog() failed. The statement has been closed or cancelled.
> > (state=,code=0)
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HIVE-16345) BeeLineDriver should be able to run qtest files which are using default database tables
[ https://issues.apache.org/jira/browse/HIVE-16345?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15972613#comment-15972613 ] Peter Vary commented on HIVE-16345: --- Thanks [~ychena] for the review and the commit! > BeeLineDriver should be able to run qtest files which are using default > database tables > --- > > Key: HIVE-16345 > URL: https://issues.apache.org/jira/browse/HIVE-16345 > Project: Hive > Issue Type: Improvement > Components: Testing Infrastructure >Reporter: Peter Vary >Assignee: Peter Vary > Fix For: 3.0.0 > > Attachments: HIVE-16345.2.patch, HIVE-16345.3.patch, HIVE-16345.patch > > > It would be good to be able to run the default clientpositive tests. > Currently we can not do that, since we start with a specific database. We > should filter the query input and replace the table references -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HIVE-16467) Flaky test: TestCliDriver.testCliDriver[vector_order_null]
[
https://issues.apache.org/jira/browse/HIVE-16467?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15973179#comment-15973179
]
Peter Vary commented on HIVE-16467:
---
This failed test is a known flaky, see:
- HIVE-15776 - Flaky test: TestMiniLlapLocalCliDriver vector_if_expr
Any comments [~mmccline]?
Thanks,
Peter
> Flaky test: TestCliDriver.testCliDriver[vector_order_null]
> --
>
> Key: HIVE-16467
> URL: https://issues.apache.org/jira/browse/HIVE-16467
> Project: Hive
> Issue Type: Sub-task
>Reporter: Peter Vary
>Assignee: Peter Vary
> Attachments: HIVE-16467.02.patch, HIVE-16467.03.patch,
> HIVE-16467.patch
>
>
> Seems to me, that we have to add a {{-- SORT_QUERY_RESULTS}} to the query
> test.
> The error message is
> {code}
> Client Execution succeeded but contained differences (error code = 1) after
> executing vector_order_null.q
> 153a154
> > 2 B
> 156d156
> < 2 B
> 247a248
> > 2 B
> 250d250
> < 2 B
> 531a532
> > 2 B
> 534d534
> < 2 B
> 626a627
> > 2 B
> 629d629
> < 2 B
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HIVE-15751) Make it possible to run findbugs for itest modules as well
[ https://issues.apache.org/jira/browse/HIVE-15751?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15903405#comment-15903405 ] Peter Vary commented on HIVE-15751: --- Thanks for the review, and the commit! > Make it possible to run findbugs for itest modules as well > -- > > Key: HIVE-15751 > URL: https://issues.apache.org/jira/browse/HIVE-15751 > Project: Hive > Issue Type: Sub-task > Components: Testing Infrastructure >Affects Versions: 2.2.0 >Reporter: Peter Vary >Assignee: Peter Vary >Priority: Minor > Fix For: 2.2.0 > > Attachments: HIVE-15751.2.patch, HIVE-15751.patch > > > Remove relative paths from the findbugs configuration, so it could be run for > every module. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HIVE-16146) If possible find a better way to filter the TestBeeLineDriver output
[
https://issues.apache.org/jira/browse/HIVE-16146?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15901471#comment-15901471
]
Peter Vary commented on HIVE-16146:
---
Tested:
- silent mode removes every message (PREHOOK, POSTHOOK) as well.
- We might get away with running with {{verbose=false}}, since it is used
mostly to log messages which are already removed from the result, but there are
only a limited number of these
> If possible find a better way to filter the TestBeeLineDriver output
>
>
> Key: HIVE-16146
> URL: https://issues.apache.org/jira/browse/HIVE-16146
> Project: Hive
> Issue Type: Improvement
> Components: Testing Infrastructure
>Affects Versions: 2.2.0
>Reporter: Peter Vary
>Assignee: Peter Vary
>
> Currently we apply a blacklist to filter the output of the BeeLine Qtest runs.
> It might be a good idea to go thorough of the possibilities and find a better
> way, if possible.
> I think our main goal could be for the TestBeeLineDriver test output to match
> the TestCliDriver output of the came query file. Or if it is not possible,
> then at least a similar one
> CC: [~vihangk1]
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HIVE-16146) If possible find a better way to filter the TestBeeLineDriver output
[ https://issues.apache.org/jira/browse/HIVE-16146?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15901399#comment-15901399 ] Peter Vary commented on HIVE-16146: --- I can imagine 2 different approach: - Blacklist - Keeping every line, but removing the blacklisted ones - Whitelist - Keeping only specific lines I prefer the blacklist solution, because the other approach could hide unintended changes easily. I think we might find a better logging level to use for the qtests. Currently we using the following settings: - verbose - print debug messages, stack traces in case of an exception - shownestederrs - print the nested exception - not dependent on verbose :) - showwarnings - needs verbose on - prints out any warnings that exist for the current connection. We might produce the same results as the TestCliDriver qtest if we use the silent mode. I have to check that. > If possible find a better way to filter the TestBeeLineDriver output > > > Key: HIVE-16146 > URL: https://issues.apache.org/jira/browse/HIVE-16146 > Project: Hive > Issue Type: Improvement > Components: Testing Infrastructure >Affects Versions: 2.2.0 >Reporter: Peter Vary >Assignee: Peter Vary > > Currently we apply a blacklist to filter the output of the BeeLine Qtest runs. > It might be a good idea to go thorough of the possibilities and find a better > way, if possible. > I think our main goal could be for the TestBeeLineDriver test output to match > the TestCliDriver output of the came query file. Or if it is not possible, > then at least a similar one > CC: [~vihangk1] -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HIVE-16127) Separate database initialization from actual query run in TestBeeLineDriver
[ https://issues.apache.org/jira/browse/HIVE-16127?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Peter Vary updated HIVE-16127: -- Attachment: HIVE-16127.03.patch Addressed RB comments, and rebased > Separate database initialization from actual query run in TestBeeLineDriver > --- > > Key: HIVE-16127 > URL: https://issues.apache.org/jira/browse/HIVE-16127 > Project: Hive > Issue Type: Improvement > Components: Testing Infrastructure >Affects Versions: 2.2.0 >Reporter: Peter Vary >Assignee: Peter Vary >Priority: Minor > Attachments: HIVE-16127.01.patch, HIVE-16127.02.patch, > HIVE-16127.03.patch > > > Improve the TestBeeLineDriver, so when running multiple tests, then reuse the > default database for multiple runs. This helps to keep the runtimes in check. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HIVE-16127) Separate database initialization from actual query run in TestBeeLineDriver
[ https://issues.apache.org/jira/browse/HIVE-16127?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Peter Vary updated HIVE-16127: -- Attachment: HIVE-16127.02.patch Addressed review comments > Separate database initialization from actual query run in TestBeeLineDriver > --- > > Key: HIVE-16127 > URL: https://issues.apache.org/jira/browse/HIVE-16127 > Project: Hive > Issue Type: Improvement > Components: Testing Infrastructure >Affects Versions: 2.2.0 >Reporter: Peter Vary >Assignee: Peter Vary >Priority: Minor > Attachments: HIVE-16127.01.patch, HIVE-16127.02.patch > > > Improve the TestBeeLineDriver, so when running multiple tests, then reuse the > default database for multiple runs. This helps to keep the runtimes in check. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HIVE-16127) Separate database initialization from actual query run in TestBeeLineDriver
[ https://issues.apache.org/jira/browse/HIVE-16127?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Peter Vary updated HIVE-16127: -- Status: Patch Available (was: Open) > Separate database initialization from actual query run in TestBeeLineDriver > --- > > Key: HIVE-16127 > URL: https://issues.apache.org/jira/browse/HIVE-16127 > Project: Hive > Issue Type: Improvement > Components: Testing Infrastructure >Affects Versions: 2.2.0 >Reporter: Peter Vary >Assignee: Peter Vary >Priority: Minor > Attachments: HIVE-16127.01.patch, HIVE-16127.02.patch > > > Improve the TestBeeLineDriver, so when running multiple tests, then reuse the > default database for multiple runs. This helps to keep the runtimes in check. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Assigned] (HIVE-16152) TestBeeLineDriver logging improvements
[ https://issues.apache.org/jira/browse/HIVE-16152?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Peter Vary reassigned HIVE-16152: - > TestBeeLineDriver logging improvements > -- > > Key: HIVE-16152 > URL: https://issues.apache.org/jira/browse/HIVE-16152 > Project: Hive > Issue Type: Improvement > Components: Testing Infrastructure >Affects Versions: 2.2.0 >Reporter: Peter Vary >Assignee: Peter Vary > > During the review of HIVE-16127 we agreed, that it would be great to have > improved logging and error messages during the TestBeeLineDriver run. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HIVE-16127) Separate database initialization from actual query run in TestBeeLineDriver
[ https://issues.apache.org/jira/browse/HIVE-16127?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15903108#comment-15903108 ] Peter Vary commented on HIVE-16127: --- A green run! :) Failures are not related :D > Separate database initialization from actual query run in TestBeeLineDriver > --- > > Key: HIVE-16127 > URL: https://issues.apache.org/jira/browse/HIVE-16127 > Project: Hive > Issue Type: Improvement > Components: Testing Infrastructure >Affects Versions: 2.2.0 >Reporter: Peter Vary >Assignee: Peter Vary >Priority: Minor > Attachments: HIVE-16127.01.patch, HIVE-16127.02.patch, > HIVE-16127.03.patch > > > Improve the TestBeeLineDriver, so when running multiple tests, then reuse the > default database for multiple runs. This helps to keep the runtimes in check. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HIVE-16152) TestBeeLineDriver logging improvements
[
https://issues.apache.org/jira/browse/HIVE-16152?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15924243#comment-15924243
]
Peter Vary commented on HIVE-16152:
---
The changed output:
- diff problem:
{code}
Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=256M;
support was removed in 8.0
Running org.apache.hadoop.hive.cli.TestBeeLineDriver
Tests run: 2, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 13.999 sec <<<
FAILURE! - in org.apache.hadoop.hive.cli.TestBeeLineDriver
testCliDriver[drop_with_concurrency](org.apache.hadoop.hive.cli.TestBeeLineDriver)
Time elapsed: 2.223 sec <<< FAILURE!
java.lang.AssertionError: Client result comparison failed with error code = 1
while executing fname=drop_with_concurrency
3c3
< Change!!! No rows affected
---
> No rows affected
{code}
- Missing out file:
{code}
testCliDriver[escape_comments](org.apache.hadoop.hive.cli.TestBeeLineDriver)
Time elapsed: 2.703 sec <<< ERROR!
java.lang.Exception: Exception running or analyzing the results of the query
file: org.apache.hive.beeline.qfile.QFile@4df7d9ee
The following files can help you identifying the problem:
- Query file:
/Users/petervary/dev/upstream/hive/ql/src/test/queries/clientpositive/escape_comments.q
- Raw output file:
/Users/petervary/dev/upstream/hive/itests/qtest/target/qfile-results/beelinepositive/escape_comments.q.out.raw
- Filtered output file:
/Users/petervary/dev/upstream/hive/itests/qtest/target/qfile-results/beelinepositive/escape_comments.q.out
- Expected output file:
/Users/petervary/dev/upstream/hive/ql/src/test/results/clientpositive/beeline/escape_comments.q.out
- Client log file:
/Users/petervary/dev/upstream/hive/itests/qtest/target/qfile-results/beelinepositive/escape_comments.q.beeline
- Client log files before the test:
/Users/petervary/dev/upstream/hive/itests/qtest/target/qfile-results/beelinepositive/escape_comments.q.out.before
- Client log files after the test:
/Users/petervary/dev/upstream/hive/itests/qtest/target/qfile-results/beelinepositive/escape_comments.q.out.after
- Hiveserver2 log file: ./itests/qtest/target/tmp/hive.log
at
org.apache.hadoop.hive.cli.control.CoreBeeLineDriver.runTest(CoreBeeLineDriver.java:173)
at
org.apache.hadoop.hive.cli.control.CoreBeeLineDriver.runTest(CoreBeeLineDriver.java:188)
at
org.apache.hadoop.hive.cli.control.CliAdapter.runTest(CliAdapter.java:104)
at
org.apache.hadoop.hive.cli.TestBeeLineDriver.testCliDriver(TestBeeLineDriver.java:59)
[..]
Caused by: java.io.IOException: Expected results file does not exist:
/Users/petervary/dev/upstream/hive/ql/src/test/results/clientpositive/beeline/escape_comments.q.out
at org.apache.hive.beeline.qfile.QFile.compareResults(QFile.java:111)
at
org.apache.hadoop.hive.cli.control.CoreBeeLineDriver.runTest(CoreBeeLineDriver.java:156)
... 39 more
{code}
- Query file problem:
{code}
testCliDriver[drop_with_concurrency](org.apache.hadoop.hive.cli.TestBeeLineDriver)
Time elapsed: 0.522 sec <<< ERROR!
java.sql.SQLException: Error executing SQL command: !run
/Users/petervary/dev/upstream/hive/ql/src/test/queries/clientpositive/drop_with_concurrency.q
at
org.apache.hive.beeline.qfile.QFileBeeLineClient.execute(QFileBeeLineClient.java:64)
at
org.apache.hive.beeline.qfile.QFileBeeLineClient.execute(QFileBeeLineClient.java:93)
at
org.apache.hadoop.hive.cli.control.CoreBeeLineDriver.runTest(CoreBeeLineDriver.java:144)
{code}
- Init file problem:
{code}
org.apache.hadoop.hive.cli.TestBeeLineDriver Time elapsed: 8.093 sec <<<
ERROR!
java.sql.SQLException: Error running infra script:
/Users/petervary/dev/upstream/hive/data/scripts/q_test_init_src.sql
Check the following logs for details:
-
/Users/petervary/dev/upstream/hive/itests/qtest/target/qfile-results/beelinepositive/init.beeline
-
/Users/petervary/dev/upstream/hive/itests/qtest/target/qfile-results/beelinepositive/init.raw
at
org.apache.hadoop.hive.cli.control.CoreBeeLineDriver.runInfraScript(CoreBeeLineDriver.java:121)
at
org.apache.hadoop.hive.cli.control.CoreBeeLineDriver.beforeClass(CoreBeeLineDriver.java:105)
at
org.apache.hadoop.hive.cli.control.CliAdapter$1$1.evaluate(CliAdapter.java:71)
at org.junit.rules.RunRules.evaluate(RunRules.java:20)
at org.junit.runners.ParentRunner.run(ParentRunner.java:309)
at
org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:283)
at
org.apache.maven.surefire.junit4.JUnit4Provider.executeWithRerun(JUnit4Provider.java:173)
at
org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:153)
at
org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:128)
at
org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:203)
at
[jira] [Updated] (HIVE-16152) TestBeeLineDriver logging improvements
[ https://issues.apache.org/jira/browse/HIVE-16152?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Peter Vary updated HIVE-16152: -- Attachment: HIVE-16152.patch First version. Please review [~kgyrtkirk], [~vihangk1], [~zsombor.klara], [~ngangam]. I am open for any suggestions. There is a degradation compared to the QTestUtil, HiveCli version. I was not able to provide the failed sql command in the exception. With this method the failed command is most probably the "run scriptfile.q" one. The actual failed command can be found in the scriptfile.q.beeline file. > TestBeeLineDriver logging improvements > -- > > Key: HIVE-16152 > URL: https://issues.apache.org/jira/browse/HIVE-16152 > Project: Hive > Issue Type: Improvement > Components: Testing Infrastructure >Affects Versions: 2.2.0 >Reporter: Peter Vary >Assignee: Peter Vary > Attachments: HIVE-16152.patch > > > During the review of HIVE-16127 we agreed, that it would be great to have > improved logging and error messages during the TestBeeLineDriver run. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HIVE-16152) TestBeeLineDriver logging improvements
[ https://issues.apache.org/jira/browse/HIVE-16152?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Peter Vary updated HIVE-16152: -- Status: Patch Available (was: Open) > TestBeeLineDriver logging improvements > -- > > Key: HIVE-16152 > URL: https://issues.apache.org/jira/browse/HIVE-16152 > Project: Hive > Issue Type: Improvement > Components: Testing Infrastructure >Affects Versions: 2.2.0 >Reporter: Peter Vary >Assignee: Peter Vary > Attachments: HIVE-16152.patch > > > During the review of HIVE-16127 we agreed, that it would be great to have > improved logging and error messages during the TestBeeLineDriver run. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HIVE-16152) TestBeeLineDriver logging improvements
[ https://issues.apache.org/jira/browse/HIVE-16152?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15907121#comment-15907121 ] Peter Vary commented on HIVE-16152: --- I would prefer to use the result class created in the patch > TestBeeLineDriver logging improvements > -- > > Key: HIVE-16152 > URL: https://issues.apache.org/jira/browse/HIVE-16152 > Project: Hive > Issue Type: Improvement > Components: Testing Infrastructure >Affects Versions: 2.2.0 >Reporter: Peter Vary >Assignee: Peter Vary > > During the review of HIVE-16127 we agreed, that it would be great to have > improved logging and error messages during the TestBeeLineDriver run. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Assigned] (HIVE-16146) If possible find a better way to filter the TestBeeLineDriver output
[ https://issues.apache.org/jira/browse/HIVE-16146?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Peter Vary reassigned HIVE-16146: - > If possible find a better way to filter the TestBeeLineDriver output > > > Key: HIVE-16146 > URL: https://issues.apache.org/jira/browse/HIVE-16146 > Project: Hive > Issue Type: Improvement > Components: Testing Infrastructure >Affects Versions: 2.2.0 >Reporter: Peter Vary >Assignee: Peter Vary > > Currently we apply a blacklist to filter the output of the BeeLine Qtest runs. > It might be a good idea to go thorough of the possibilities and find a better > way, if possible. > I think our main goal could be for the TestBeeLineDriver test output to match > the TestCliDriver output of the came query file. Or if it is not possible, > then at least a similar one > CC: [~vihangk1] -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HIVE-16127) Separate database initialization from actual query run in TestBeeLineDriver
[ https://issues.apache.org/jira/browse/HIVE-16127?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15922954#comment-15922954 ] Peter Vary commented on HIVE-16127: --- And of course thanks [~vihangk1] and [~zsombor.klara] for the review > Separate database initialization from actual query run in TestBeeLineDriver > --- > > Key: HIVE-16127 > URL: https://issues.apache.org/jira/browse/HIVE-16127 > Project: Hive > Issue Type: Improvement > Components: Testing Infrastructure >Affects Versions: 2.2.0 >Reporter: Peter Vary >Assignee: Peter Vary >Priority: Minor > Fix For: 2.2.0 > > Attachments: HIVE-16127.01.patch, HIVE-16127.02.patch, > HIVE-16127.03.patch > > > Improve the TestBeeLineDriver, so when running multiple tests, then reuse the > default database for multiple runs. This helps to keep the runtimes in check. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HIVE-15616) Improve contents of qfile test output
[ https://issues.apache.org/jira/browse/HIVE-15616?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15922873#comment-15922873 ] Peter Vary commented on HIVE-15616: --- There is a missing file in the commit (QTestProcessExecResult) Probably not p0? > Improve contents of qfile test output > - > > Key: HIVE-15616 > URL: https://issues.apache.org/jira/browse/HIVE-15616 > Project: Hive > Issue Type: Improvement > Components: Tests >Affects Versions: 2.1.1 >Reporter: Barna Zsombor Klara >Assignee: Barna Zsombor Klara >Priority: Minor > Fix For: 2.2.0 > > Attachments: HIVE-15616.1.patch, HIVE-15616.2.patch, > HIVE-15616.3.patch, HIVE-15616.4.patch, HIVE-15616.patch > > > The current output of the failed qtests has a less than ideal signal to noise > ratio. > We have duplicated stack traces and messages between the error message/stack > trace/error out. > For diff errors the actual difference is missing from the error message and > can be found only in the standard out. > I would like to simplify this output by removing duplications, moving > relevant information to the top. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HIVE-15616) Improve contents of qfile test output
[ https://issues.apache.org/jira/browse/HIVE-15616?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15922884#comment-15922884 ] Peter Vary commented on HIVE-15616: --- :) You were faster [~ashutoshc] :) > Improve contents of qfile test output > - > > Key: HIVE-15616 > URL: https://issues.apache.org/jira/browse/HIVE-15616 > Project: Hive > Issue Type: Improvement > Components: Tests >Affects Versions: 2.1.1 >Reporter: Barna Zsombor Klara >Assignee: Barna Zsombor Klara >Priority: Minor > Fix For: 2.2.0 > > Attachments: HIVE-15616.1.patch, HIVE-15616.2.patch, > HIVE-15616.3.patch, HIVE-15616.4.patch, HIVE-15616.patch > > > The current output of the failed qtests has a less than ideal signal to noise > ratio. > We have duplicated stack traces and messages between the error message/stack > trace/error out. > For diff errors the actual difference is missing from the error message and > can be found only in the standard out. > I would like to simplify this output by removing duplications, moving > relevant information to the top. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HIVE-16127) Separate database initialization from actual query run in TestBeeLineDriver
[ https://issues.apache.org/jira/browse/HIVE-16127?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15922896#comment-15922896 ] Peter Vary commented on HIVE-16127: --- Thanks [~kgyrtkirk] for the review and the commit! Eager to continue with the next ones :) > Separate database initialization from actual query run in TestBeeLineDriver > --- > > Key: HIVE-16127 > URL: https://issues.apache.org/jira/browse/HIVE-16127 > Project: Hive > Issue Type: Improvement > Components: Testing Infrastructure >Affects Versions: 2.2.0 >Reporter: Peter Vary >Assignee: Peter Vary >Priority: Minor > Fix For: 2.2.0 > > Attachments: HIVE-16127.01.patch, HIVE-16127.02.patch, > HIVE-16127.03.patch > > > Improve the TestBeeLineDriver, so when running multiple tests, then reuse the > default database for multiple runs. This helps to keep the runtimes in check. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Assigned] (HIVE-16127) Separate database initialization from actual query run in TestBeeLineDriver
[ https://issues.apache.org/jira/browse/HIVE-16127?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Peter Vary reassigned HIVE-16127: - > Separate database initialization from actual query run in TestBeeLineDriver > --- > > Key: HIVE-16127 > URL: https://issues.apache.org/jira/browse/HIVE-16127 > Project: Hive > Issue Type: Improvement > Components: Testing Infrastructure >Affects Versions: 2.2.0 >Reporter: Peter Vary >Assignee: Peter Vary >Priority: Minor > > Improve the TestBeeLineDriver, so when running multiple tests, then reuse the > default database for multiple runs. This helps to keep the runtimes in check. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HIVE-16116) Beeline throws NPE when beeline.hiveconfvariables={} in beeline.properties
[
https://issues.apache.org/jira/browse/HIVE-16116?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15897472#comment-15897472
]
Peter Vary commented on HIVE-16116:
---
Errors not related.
For the BeeLine one (TestBeeLineDriver.testCliDriver escape_comments) see
HIVE-16101.
Thanks for the patch [~rajesh.balamohan]!
+1 (non-binding)
> Beeline throws NPE when beeline.hiveconfvariables={} in beeline.properties
> --
>
> Key: HIVE-16116
> URL: https://issues.apache.org/jira/browse/HIVE-16116
> Project: Hive
> Issue Type: Bug
> Components: Beeline
>Reporter: Rajesh Balamohan
>Assignee: Rajesh Balamohan
>Priority: Minor
> Attachments: HIVE-16116.1.patch, HIVE-16116.2.patch
>
>
> Env: hive master
> Steps to reproduce:
> 1. clear previous beeline.properties (rm -rf ~/.beeline/beeline.properties)
> 2. Launch beeline, "!save" and exit. This would create new
> "~/.beeline/beeline.properties", which would have
> "beeline.hiveconfvariables={}"
> 3. Launch "beeline --hiveconf hive.tmp.dir=/tmp". This would throw NPE
> {noformat}
> Exception in thread "main" java.lang.NullPointerException
> at org.apache.hive.beeline.BeeLine.setHiveConfVar(BeeLine.java:885)
> at org.apache.hive.beeline.BeeLine.connectUsingArgs(BeeLine.java:832)
> at org.apache.hive.beeline.BeeLine.initArgs(BeeLine.java:775)
> at org.apache.hive.beeline.BeeLine.begin(BeeLine.java:1009)
> at
> org.apache.hive.beeline.BeeLine.mainWithInputRedirection(BeeLine.java:519)
> at org.apache.hive.beeline.BeeLine.main(BeeLine.java:501)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at org.apache.hadoop.util.RunJar.run(RunJar.java:233)
> at org.apache.hadoop.util.RunJar.main(RunJar.java:148)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HIVE-16089) "trustStorePassword" is logged as part of jdbc connection url
[
https://issues.apache.org/jira/browse/HIVE-16089?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15898027#comment-15898027
]
Peter Vary commented on HIVE-16089:
---
[~sfroehlich]: You could take a look at here:
https://archive.cloudera.com/cdh5/cdh/5/hive-1.1.0-cdh5.7.0.CHANGES.txt or
here:
https://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.3.4/bk_HDP_RelNotes/content/fixed_issues.html.
Pick your favorite :)
> "trustStorePassword" is logged as part of jdbc connection url
> -
>
> Key: HIVE-16089
> URL: https://issues.apache.org/jira/browse/HIVE-16089
> Project: Hive
> Issue Type: Bug
> Components: JDBC
>Affects Versions: 1.1.0
>Reporter: Sebastian Fröhlich
> Labels: security
>
> h5. General Story
> The use case is to connect via the Apache Hive JDBC driver to a Hive where
> SSL encryption is enabled.
> It was required to set the ssl-trust store password property
> {{trustStorePassword}} in the jdbc connection url.
> If the property is passed via "properties" parameter into
> {{Driver.connect(url, properties)}} this will not recognized.
> h5. Log message
> {code}
> 2017-03-03 09:57:58,385 [INFO] [InputInitializer {Map for sheets:[import]
> (fce7cd11-d489-4a13-a3a9-4c81d2907c87)} #0]
> |jdbc.Utils|: Resolved authority: :
> 2017-03-03 09:57:58,539 [INFO] [InputInitializer {Map for sheets:[import]
> (fce7cd11-d489-4a13-a3a9-4c81d2907c87)} #0] |jdbc.HiveConnection|: Will try
> to open client transport with JDBC Uri:
> jdbc:hive2://:/;ssl=true;sslTrustStore=/tmp/hs2keystore.jks;trustStorePassword=
> {code}
> E.g. produced by code {{org.apache.hive.jdbc.HiveConnection#openTransport()}}
> h5. Suggested Behavior
> The property {{trustStorePassword}} could be part of the "properties"
> parameter. This way the password is not part of the JDBC connection url.
> h5. Acceptance Criteria
> The ssl trust store password should not be logged as part of the JDBC
> connection string.
> Support the trust store password via the properties parameter within connect.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HIVE-16101) QTest failure BeeLine escape_comments after HIVE-16045
[
https://issues.apache.org/jira/browse/HIVE-16101?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15897992#comment-15897992
]
Peter Vary commented on HIVE-16101:
---
[~ngangam]: Yes, the first failures were because of {{No space left on
device}}. The second run was ok. [~kgyrtkirk] kindly reviewed the patch, so I
think it could be committed.
Thanks,
Peter
> QTest failure BeeLine escape_comments after HIVE-16045
> --
>
> Key: HIVE-16101
> URL: https://issues.apache.org/jira/browse/HIVE-16101
> Project: Hive
> Issue Type: Bug
> Components: Test
>Affects Versions: 2.2.0
>Reporter: Peter Vary
>Assignee: Peter Vary
>Priority: Minor
> Attachments: HIVE-16101.2.patch, HIVE-16101.patch
>
>
> HIVE-16045 committed immediately after HIVE-14459, and added two extra lines
> to the output which is written there with another thread. We should remove
> these lines before comparing the out file
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HIVE-16101) QTest failure BeeLine escape_comments after HIVE-16045
[ https://issues.apache.org/jira/browse/HIVE-16101?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15899189#comment-15899189 ] Peter Vary commented on HIVE-16101: --- Thanks [~kgyrtkirk] for the review and the commit! > QTest failure BeeLine escape_comments after HIVE-16045 > -- > > Key: HIVE-16101 > URL: https://issues.apache.org/jira/browse/HIVE-16101 > Project: Hive > Issue Type: Bug > Components: Test >Affects Versions: 2.2.0 >Reporter: Peter Vary >Assignee: Peter Vary >Priority: Minor > Fix For: 2.2.0 > > Attachments: HIVE-16101.2.patch, HIVE-16101.patch > > > HIVE-16045 committed immediately after HIVE-14459, and added two extra lines > to the output which is written there with another thread. We should remove > these lines before comparing the out file -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HIVE-16127) Separate database initialization from actual query run in TestBeeLineDriver
[ https://issues.apache.org/jira/browse/HIVE-16127?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Peter Vary updated HIVE-16127: -- Attachment: HIVE-16127.01.patch The first version. Open to suggestions, how to enhance. Please review. Thanks, Peter > Separate database initialization from actual query run in TestBeeLineDriver > --- > > Key: HIVE-16127 > URL: https://issues.apache.org/jira/browse/HIVE-16127 > Project: Hive > Issue Type: Improvement > Components: Testing Infrastructure >Affects Versions: 2.2.0 >Reporter: Peter Vary >Assignee: Peter Vary >Priority: Minor > Attachments: HIVE-16127.01.patch > > > Improve the TestBeeLineDriver, so when running multiple tests, then reuse the > default database for multiple runs. This helps to keep the runtimes in check. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HIVE-16127) Separate database initialization from actual query run in TestBeeLineDriver
[ https://issues.apache.org/jira/browse/HIVE-16127?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15899596#comment-15899596 ] Peter Vary commented on HIVE-16127: --- CC: [~kgyrtkirk], [~ngangam], [~spena], [~vihangk1], [~zsombor.klara] or anyone else, who is interested :) > Separate database initialization from actual query run in TestBeeLineDriver > --- > > Key: HIVE-16127 > URL: https://issues.apache.org/jira/browse/HIVE-16127 > Project: Hive > Issue Type: Improvement > Components: Testing Infrastructure >Affects Versions: 2.2.0 >Reporter: Peter Vary >Assignee: Peter Vary >Priority: Minor > Attachments: HIVE-16127.01.patch > > > Improve the TestBeeLineDriver, so when running multiple tests, then reuse the > default database for multiple runs. This helps to keep the runtimes in check. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HIVE-16057) SchemaTool ignores --passWord argument if hadoop.security.credential.provider.path is configured
[
https://issues.apache.org/jira/browse/HIVE-16057?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Peter Vary updated HIVE-16057:
--
Status: Patch Available (was: Reopened)
Forgot to press submit patch button.
Let's see what will happen
> SchemaTool ignores --passWord argument if
> hadoop.security.credential.provider.path is configured
>
>
> Key: HIVE-16057
> URL: https://issues.apache.org/jira/browse/HIVE-16057
> Project: Hive
> Issue Type: Bug
>Affects Versions: 2.2.0
>Reporter: Peter Vary
>Assignee: Peter Vary
> Fix For: 2.2.0
>
> Attachments: HIVE-16057.patch
>
>
> It the {{hadoop.security.credential.provider.path}} is defined in command
> line, but the correct {{HADOOP_CREDSTORE_PASSWORD}} is not provided the
> SchemaTool fails, even if the correct metastore password is provided with
> {{--passWord}}
> Could be reproduced if the hive-site.xml contains the following:
> {code}
>
> hadoop.security.credential.provider.path
>
> localjceks://file//Users/petervary/tmp/conf/creds.localjceks
>
> {code}
> {code}
> $ ../schemaTool --dbType=mysql --info --passWord=pwd
> Metastore connection URL:
> jdbc:mysql://localhost:3306/hive?useUnicode=true=UTF-8
> Metastore Connection Driver : com.mysql.jdbc.Driver
> Metastore connection User: hive
> org.apache.hadoop.hive.metastore.HiveMetaException: Failed to get schema
> version.
> *** schemaTool failed ***
> {code}
> The {{--passWord}} argument should override the errors from the credential
> provider
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HIVE-16057) SchemaTool ignores --passWord argument if hadoop.security.credential.provider.path is configured
[
https://issues.apache.org/jira/browse/HIVE-16057?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15896558#comment-15896558
]
Peter Vary commented on HIVE-16057:
---
The failures are not related.
> SchemaTool ignores --passWord argument if
> hadoop.security.credential.provider.path is configured
>
>
> Key: HIVE-16057
> URL: https://issues.apache.org/jira/browse/HIVE-16057
> Project: Hive
> Issue Type: Bug
>Affects Versions: 2.2.0
>Reporter: Peter Vary
>Assignee: Peter Vary
> Fix For: 2.2.0
>
> Attachments: HIVE-16057.02.patch, HIVE-16057.03.patch,
> HIVE-16057.patch
>
>
> It the {{hadoop.security.credential.provider.path}} is defined in command
> line, but the correct {{HADOOP_CREDSTORE_PASSWORD}} is not provided the
> SchemaTool fails, even if the correct metastore password is provided with
> {{--passWord}}
> Could be reproduced if the hive-site.xml contains the following:
> {code}
>
> hadoop.security.credential.provider.path
>
> localjceks://file//Users/petervary/tmp/conf/creds.localjceks
>
> {code}
> {code}
> $ ../schemaTool --dbType=mysql --info --passWord=pwd
> Metastore connection URL:
> jdbc:mysql://localhost:3306/hive?useUnicode=true=UTF-8
> Metastore Connection Driver : com.mysql.jdbc.Driver
> Metastore connection User: hive
> org.apache.hadoop.hive.metastore.HiveMetaException: Failed to get schema
> version.
> *** schemaTool failed ***
> {code}
> The {{--passWord}} argument should override the errors from the credential
> provider
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Updated] (HIVE-16057) SchemaTool ignores --passWord argument if hadoop.security.credential.provider.path is configured
[
https://issues.apache.org/jira/browse/HIVE-16057?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Peter Vary updated HIVE-16057:
--
Attachment: HIVE-16057.03.patch
Was not aware, that there are itests for SchemaTool too... Always learning :D
Modified the test to set username and password before running the schematool.
[~aihuaxu]: What do you think about this change? Is it ok, or using the
schematool programmatically is an intended use so the doInit method should try
to load the username/password too?
Thanks,
Peter
> SchemaTool ignores --passWord argument if
> hadoop.security.credential.provider.path is configured
>
>
> Key: HIVE-16057
> URL: https://issues.apache.org/jira/browse/HIVE-16057
> Project: Hive
> Issue Type: Bug
>Affects Versions: 2.2.0
>Reporter: Peter Vary
>Assignee: Peter Vary
> Fix For: 2.2.0
>
> Attachments: HIVE-16057.02.patch, HIVE-16057.03.patch,
> HIVE-16057.patch
>
>
> It the {{hadoop.security.credential.provider.path}} is defined in command
> line, but the correct {{HADOOP_CREDSTORE_PASSWORD}} is not provided the
> SchemaTool fails, even if the correct metastore password is provided with
> {{--passWord}}
> Could be reproduced if the hive-site.xml contains the following:
> {code}
>
> hadoop.security.credential.provider.path
>
> localjceks://file//Users/petervary/tmp/conf/creds.localjceks
>
> {code}
> {code}
> $ ../schemaTool --dbType=mysql --info --passWord=pwd
> Metastore connection URL:
> jdbc:mysql://localhost:3306/hive?useUnicode=true=UTF-8
> Metastore Connection Driver : com.mysql.jdbc.Driver
> Metastore connection User: hive
> org.apache.hadoop.hive.metastore.HiveMetaException: Failed to get schema
> version.
> *** schemaTool failed ***
> {code}
> The {{--passWord}} argument should override the errors from the credential
> provider
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Updated] (HIVE-16057) SchemaTool ignores --passWord argument if hadoop.security.credential.provider.path is configured
[
https://issues.apache.org/jira/browse/HIVE-16057?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Peter Vary updated HIVE-16057:
--
Attachment: HIVE-16057.02.patch
Build infrastructure failure:
{code}
+ curl -s -S --location --retry 3
https://issues.apache.org/jira/browse/HIVE-16057
curl: (7) Failed to connect to issues.apache.org port 443: No route to host
{code}
Let's see if it is solved, or not
> SchemaTool ignores --passWord argument if
> hadoop.security.credential.provider.path is configured
>
>
> Key: HIVE-16057
> URL: https://issues.apache.org/jira/browse/HIVE-16057
> Project: Hive
> Issue Type: Bug
>Affects Versions: 2.2.0
>Reporter: Peter Vary
>Assignee: Peter Vary
> Fix For: 2.2.0
>
> Attachments: HIVE-16057.02.patch, HIVE-16057.patch
>
>
> It the {{hadoop.security.credential.provider.path}} is defined in command
> line, but the correct {{HADOOP_CREDSTORE_PASSWORD}} is not provided the
> SchemaTool fails, even if the correct metastore password is provided with
> {{--passWord}}
> Could be reproduced if the hive-site.xml contains the following:
> {code}
>
> hadoop.security.credential.provider.path
>
> localjceks://file//Users/petervary/tmp/conf/creds.localjceks
>
> {code}
> {code}
> $ ../schemaTool --dbType=mysql --info --passWord=pwd
> Metastore connection URL:
> jdbc:mysql://localhost:3306/hive?useUnicode=true=UTF-8
> Metastore Connection Driver : com.mysql.jdbc.Driver
> Metastore connection User: hive
> org.apache.hadoop.hive.metastore.HiveMetaException: Failed to get schema
> version.
> *** schemaTool failed ***
> {code}
> The {{--passWord}} argument should override the errors from the credential
> provider
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HIVE-16101) QTest failure BeeLine escape_comments after HIVE-16045
[ https://issues.apache.org/jira/browse/HIVE-16101?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15896501#comment-15896501 ] Peter Vary commented on HIVE-16101: --- [~ngangam]: Could you please review. I do not want the BeeLine tests start as a flaky ones :) Thanks, Peter > QTest failure BeeLine escape_comments after HIVE-16045 > -- > > Key: HIVE-16101 > URL: https://issues.apache.org/jira/browse/HIVE-16101 > Project: Hive > Issue Type: Bug > Components: Test >Affects Versions: 2.2.0 >Reporter: Peter Vary >Assignee: Peter Vary >Priority: Minor > Attachments: HIVE-16101.patch > > > HIVE-16045 committed immediately after HIVE-14459, and added two extra lines > to the output which is written there with another thread. We should remove > these lines before comparing the out file -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HIVE-16116) Beeline throws NPE when beeline.hiveconfvariables={} in beeline.properties
[
https://issues.apache.org/jira/browse/HIVE-16116?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15896964#comment-15896964
]
Peter Vary commented on HIVE-16116:
---
Thanks for the patch [~rajesh.balamohan]!
Just a quick question:
- Have not tested yet, but is not {{setHiveVariables}} effected with this too?
Thanks,
Peter
> Beeline throws NPE when beeline.hiveconfvariables={} in beeline.properties
> --
>
> Key: HIVE-16116
> URL: https://issues.apache.org/jira/browse/HIVE-16116
> Project: Hive
> Issue Type: Bug
> Components: Beeline
>Reporter: Rajesh Balamohan
>Assignee: Rajesh Balamohan
>Priority: Minor
> Attachments: HIVE-16116.1.patch
>
>
> Env: hive master
> Steps to reproduce:
> 1. clear previous beeline.properties (rm -rf ~/.beeline/beeline.properties)
> 2. Launch beeline, "!save" and exit. This would create new
> "~/.beeline/beeline.properties", which would have
> "beeline.hiveconfvariables={}"
> 3. Launch "beeline --hiveconf hive.tmp.dir=/tmp". This would throw NPE
> {noformat}
> Exception in thread "main" java.lang.NullPointerException
> at org.apache.hive.beeline.BeeLine.setHiveConfVar(BeeLine.java:885)
> at org.apache.hive.beeline.BeeLine.connectUsingArgs(BeeLine.java:832)
> at org.apache.hive.beeline.BeeLine.initArgs(BeeLine.java:775)
> at org.apache.hive.beeline.BeeLine.begin(BeeLine.java:1009)
> at
> org.apache.hive.beeline.BeeLine.mainWithInputRedirection(BeeLine.java:519)
> at org.apache.hive.beeline.BeeLine.main(BeeLine.java:501)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at org.apache.hadoop.util.RunJar.run(RunJar.java:233)
> at org.apache.hadoop.util.RunJar.main(RunJar.java:148)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Updated] (HIVE-16101) QTest failure BeeLine escape_comments after HIVE-16045
[
https://issues.apache.org/jira/browse/HIVE-16101?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Peter Vary updated HIVE-16101:
--
Attachment: HIVE-16101.2.patch
Thanks for the review [~kgyrtkirk]!
Retriggering the precommit with the same patch.
About the {{QFileClient.java}}, I have plans, just want to clean up this stuff
first. Will CC you in the new jira. Thanks again! :)
> QTest failure BeeLine escape_comments after HIVE-16045
> --
>
> Key: HIVE-16101
> URL: https://issues.apache.org/jira/browse/HIVE-16101
> Project: Hive
> Issue Type: Bug
> Components: Test
>Affects Versions: 2.2.0
>Reporter: Peter Vary
>Assignee: Peter Vary
>Priority: Minor
> Attachments: HIVE-16101.2.patch, HIVE-16101.patch
>
>
> HIVE-16045 committed immediately after HIVE-14459, and added two extra lines
> to the output which is written there with another thread. We should remove
> these lines before comparing the out file
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HIVE-16101) QTest failure BeeLine escape_comments after HIVE-16045
[ https://issues.apache.org/jira/browse/HIVE-16101?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15897063#comment-15897063 ] Peter Vary commented on HIVE-16101: --- Failures are not related. > QTest failure BeeLine escape_comments after HIVE-16045 > -- > > Key: HIVE-16101 > URL: https://issues.apache.org/jira/browse/HIVE-16101 > Project: Hive > Issue Type: Bug > Components: Test >Affects Versions: 2.2.0 >Reporter: Peter Vary >Assignee: Peter Vary >Priority: Minor > Attachments: HIVE-16101.2.patch, HIVE-16101.patch > > > HIVE-16045 committed immediately after HIVE-14459, and added two extra lines > to the output which is written there with another thread. We should remove > these lines before comparing the out file -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HIVE-16101) QTest failure BeeLine escape_comments after HIVE-16045
[ https://issues.apache.org/jira/browse/HIVE-16101?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Peter Vary updated HIVE-16101: -- Attachment: HIVE-16101.patch Patch removes the log lines before diff > QTest failure BeeLine escape_comments after HIVE-16045 > -- > > Key: HIVE-16101 > URL: https://issues.apache.org/jira/browse/HIVE-16101 > Project: Hive > Issue Type: Bug > Components: Test >Affects Versions: 2.2.0 >Reporter: Peter Vary >Assignee: Peter Vary >Priority: Minor > Attachments: HIVE-16101.patch > > > HIVE-16045 committed immediately after HIVE-14459, and added two extra lines > to the output which is written there with another thread. We should remove > these lines before comparing the out file -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HIVE-16101) QTest failure BeeLine escape_comments after HIVE-16045
[ https://issues.apache.org/jira/browse/HIVE-16101?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Peter Vary updated HIVE-16101: -- Status: Patch Available (was: Open) > QTest failure BeeLine escape_comments after HIVE-16045 > -- > > Key: HIVE-16101 > URL: https://issues.apache.org/jira/browse/HIVE-16101 > Project: Hive > Issue Type: Bug > Components: Test >Affects Versions: 2.2.0 >Reporter: Peter Vary >Assignee: Peter Vary >Priority: Minor > Attachments: HIVE-16101.patch > > > HIVE-16045 committed immediately after HIVE-14459, and added two extra lines > to the output which is written there with another thread. We should remove > these lines before comparing the out file -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HIVE-16101) QTest failure BeeLine escape_comments after HIVE-16045
[ https://issues.apache.org/jira/browse/HIVE-16101?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15894095#comment-15894095 ] Peter Vary commented on HIVE-16101: --- [~kgyrtkirk]: Could you review please? Concurrent changes in HIVE-16045 caused the test fail. Thanks, Peter > QTest failure BeeLine escape_comments after HIVE-16045 > -- > > Key: HIVE-16101 > URL: https://issues.apache.org/jira/browse/HIVE-16101 > Project: Hive > Issue Type: Bug > Components: Test >Affects Versions: 2.2.0 >Reporter: Peter Vary >Assignee: Peter Vary >Priority: Minor > Attachments: HIVE-16101.patch > > > HIVE-16045 committed immediately after HIVE-14459, and added two extra lines > to the output which is written there with another thread. We should remove > these lines before comparing the out file -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Assigned] (HIVE-16101) QTest failure BeeLine escape_comments after HIVE-16045
[ https://issues.apache.org/jira/browse/HIVE-16101?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Peter Vary reassigned HIVE-16101: - > QTest failure BeeLine escape_comments after HIVE-16045 > -- > > Key: HIVE-16101 > URL: https://issues.apache.org/jira/browse/HIVE-16101 > Project: Hive > Issue Type: Bug > Components: Test >Affects Versions: 2.2.0 >Reporter: Peter Vary >Assignee: Peter Vary >Priority: Minor > > HIVE-16045 committed immediately after HIVE-14459, and added two extra lines > to the output which is written there with another thread. We should remove > these lines before comparing the out file -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HIVE-16345) BeeLineDriver should be able to run qtest files which are using default database tables
[ https://issues.apache.org/jira/browse/HIVE-16345?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Peter Vary updated HIVE-16345: -- Attachment: HIVE-16345.2.patch > BeeLineDriver should be able to run qtest files which are using default > database tables > --- > > Key: HIVE-16345 > URL: https://issues.apache.org/jira/browse/HIVE-16345 > Project: Hive > Issue Type: Improvement > Components: Testing Infrastructure >Reporter: Peter Vary >Assignee: Peter Vary > Attachments: HIVE-16345.2.patch, HIVE-16345.patch > > > It would be good to be able to run the default clientpositive tests. > Currently we can not do that, since we start with a specific database. We > should filter the query input and replace the table references -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HIVE-16345) BeeLineDriver should be able to run qtest files which are using default database tables
[
https://issues.apache.org/jira/browse/HIVE-16345?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15956807#comment-15956807
]
Peter Vary commented on HIVE-16345:
---
About the test results:
- HIVE-15776 - Flaky test: TestMiniLlapLocalCliDriver vector_if_expr
- org.apache.hive.jdbc.TestJdbcDriver2.testResultSetMetaData failing for a
while now
- TestTextFileHCatStorer errors are not reproducible locally
[~ychena], [~kgyrtkirk]: Could you please review?
[~aihuaxu], [~stakiar]: Your two below patches were reviewed, and tested
parallel, and combined they cause TestBeeLineDriver fails, which are handled
here:
- HIVE-16297: Improving hive logging configuration variables
- HIVE-16308: PreExecutePrinter and PostExecutePrinter should log to INFO level
instead of ERROR
[~sershe], [~vikram.dixit]: I have changed the IOContext key and you were the
ones who modified this part of the code at the last time. Could you please
review, at least this small change?
{code:title=Utilities.java}
public static void setMapRedWork(Configuration conf, MapredWork w, Path
hiveScratchDir) {
String useName = conf.get(INPUT_NAME);
if (useName == null) {
useName = "mapreduce:" + hiveScratchDir;
}
conf.set(INPUT_NAME, useName);
setMapWork(conf, w.getMapWork(), hiveScratchDir, true);
if (w.getReduceWork() != null) {
conf.set(INPUT_NAME, useName);
setReduceWork(conf, w.getReduceWork(), hiveScratchDir, true);
}
}
{code}
Thanks,
Peter
> BeeLineDriver should be able to run qtest files which are using default
> database tables
> ---
>
> Key: HIVE-16345
> URL: https://issues.apache.org/jira/browse/HIVE-16345
> Project: Hive
> Issue Type: Improvement
> Components: Testing Infrastructure
>Reporter: Peter Vary
>Assignee: Peter Vary
> Attachments: HIVE-16345.2.patch, HIVE-16345.patch
>
>
> It would be good to be able to run the default clientpositive tests.
> Currently we can not do that, since we start with a specific database. We
> should filter the query input and replace the table references
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Updated] (HIVE-16345) BeeLineDriver should be able to run qtest files which are using default database tables
[ https://issues.apache.org/jira/browse/HIVE-16345?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Peter Vary updated HIVE-16345: -- Attachment: HIVE-16345.3.patch Instead of the needsContinuation method refactor moved the getCommands to BeeLine, so we can reuse the whole file parsing algorithm in the tests. Modified the test classes to use the BeeLine.getCommands method Added a warning message, when use database command is used in the test script > BeeLineDriver should be able to run qtest files which are using default > database tables > --- > > Key: HIVE-16345 > URL: https://issues.apache.org/jira/browse/HIVE-16345 > Project: Hive > Issue Type: Improvement > Components: Testing Infrastructure >Reporter: Peter Vary >Assignee: Peter Vary > Attachments: HIVE-16345.2.patch, HIVE-16345.3.patch, HIVE-16345.patch > > > It would be good to be able to run the default clientpositive tests. > Currently we can not do that, since we start with a specific database. We > should filter the query input and replace the table references -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HIVE-16313) Flaky test: TestBeeLineDriver[drop_with_concurrency]
[
https://issues.apache.org/jira/browse/HIVE-16313?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15958346#comment-15958346
]
Peter Vary commented on HIVE-16313:
---
[~vihangk1], thanks for reporting!
Caused by these parallel changes:
- HIVE-16297: Improving hive logging configuration variables
- HIVE-16308: PreExecutePrinter and PostExecutePrinter should log to INFO level
instead of ERROR
Addressed by: HIVE-16345
> Flaky test: TestBeeLineDriver[drop_with_concurrency]
>
>
> Key: HIVE-16313
> URL: https://issues.apache.org/jira/browse/HIVE-16313
> Project: Hive
> Issue Type: Sub-task
> Components: Test
>Reporter: Peter Vary
>Assignee: Peter Vary
> Fix For: 3.0.0
>
> Attachments: HIVE-16313.patch
>
>
> TestBeeLineDriver test become flaky:
> {code}
> 28a29 > Acquired the compile lock
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HIVE-16334) Query lock contains the query string, which can cause OOM on ZooKeeper
[ https://issues.apache.org/jira/browse/HIVE-16334?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15960376#comment-15960376 ] Peter Vary commented on HIVE-16334: --- The failed tests are not related: - TestBeeLineDriver error will be solved with HIVE-16345 - HIVE-16405 - Flaky Test: org.apache.hive.jdbc.TestJdbcDriver2.testResultSetMetaData - HIVE-15165 - Flaky test: TestSparkNegativeCliDriver I think the patch is ready to commit. Thanks, Peter > Query lock contains the query string, which can cause OOM on ZooKeeper > -- > > Key: HIVE-16334 > URL: https://issues.apache.org/jira/browse/HIVE-16334 > Project: Hive > Issue Type: Improvement > Components: Locking >Reporter: Peter Vary >Assignee: Peter Vary > Attachments: HIVE-16334.2.patch, HIVE-16334.3.patch, > HIVE-16334.4.patch, HIVE-16334.patch > > > When there are big number of partitions in a query this will result in a huge > number of locks on ZooKeeper. Since the query object contains the whole query > string this might cause serious memory pressure on the ZooKeeper services. > It would be good to have the possibility to truncate the query strings that > are written into the locks -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HIVE-16345) BeeLineDriver should be able to run qtest files which are using default database tables
[ https://issues.apache.org/jira/browse/HIVE-16345?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15960385#comment-15960385 ] Peter Vary commented on HIVE-16345: --- Test failure not related: - HIVE-16387 - Fix failing test org.apache.hive.jdbc.TestJdbcDriver2.testResultSetMetaData Any more review comments? Thanks, Peter > BeeLineDriver should be able to run qtest files which are using default > database tables > --- > > Key: HIVE-16345 > URL: https://issues.apache.org/jira/browse/HIVE-16345 > Project: Hive > Issue Type: Improvement > Components: Testing Infrastructure >Reporter: Peter Vary >Assignee: Peter Vary > Attachments: HIVE-16345.2.patch, HIVE-16345.3.patch, HIVE-16345.patch > > > It would be good to be able to run the default clientpositive tests. > Currently we can not do that, since we start with a specific database. We > should filter the query input and replace the table references -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HIVE-16007) When the query does not complie the LogRunnable never stops
[
https://issues.apache.org/jira/browse/HIVE-16007?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Peter Vary updated HIVE-16007:
--
Attachment: HIVE-16007.6.patch
The BeeLine tests error were relevant. The last lines of the log was not
fetched. Removed the Exception instead.
> When the query does not complie the LogRunnable never stops
> ---
>
> Key: HIVE-16007
> URL: https://issues.apache.org/jira/browse/HIVE-16007
> Project: Hive
> Issue Type: Bug
> Components: Beeline
>Affects Versions: 2.2.0
>Reporter: Peter Vary
>Assignee: Peter Vary
> Attachments: HIVE-16007.02.patch, HIVE-16007.2.patch,
> HIVE-16007.3.patch, HIVE-16007.4.patch, HIVE-16007.5.patch,
> HIVE-16007.6.patch, HIVE-16007.patch
>
>
> When issuing a sql command which does not compile then the LogRunnable thread
> is never closed.
> The issue can be easily detected when running beeline with showWarnings=true.
> {code}
> $ ./beeline -u "jdbc:hive2://localhost:1 pvary pvary" --showWarnings=true
> [..]
> Connecting to jdbc:hive2://localhost:1
> Connected to: Apache Hive (version 2.2.0-SNAPSHOT)
> Driver: Hive JDBC (version 2.2.0-SNAPSHOT)
> Transaction isolation: TRANSACTION_REPEATABLE_READ
> Beeline version 2.2.0-SNAPSHOT by Apache Hive
> 0: jdbc:hive2://localhost:1> selekt;
> Warning: java.sql.SQLException: Method getQueryLog() failed. Because the
> stmtHandle in HiveStatement is null and the statement execution might fail.
> (state=,code=0)
> [..]
> Warning: java.sql.SQLException: Can't getQueryLog after statement has been
> closed (state=,code=0)
> [..]
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HIVE-16146) If possible find a better way to filter the TestBeeLineDriver output
[ https://issues.apache.org/jira/browse/HIVE-16146?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15960928#comment-15960928 ] Peter Vary commented on HIVE-16146: --- [~kgyrtkirk], [~ychena], [~aihuaxu], [~vihangk1] you can review if you have time. Thanks, Peter > If possible find a better way to filter the TestBeeLineDriver output > > > Key: HIVE-16146 > URL: https://issues.apache.org/jira/browse/HIVE-16146 > Project: Hive > Issue Type: Improvement > Components: Testing Infrastructure >Affects Versions: 2.2.0 >Reporter: Peter Vary >Assignee: Peter Vary > Attachments: HIVE-16146.02.patch, HIVE-16146.03.patch, > HIVE-16146.04.patch, HIVE-16146.05.patch, HIVE-16146.patch > > > Currently we apply a blacklist to filter the output of the BeeLine Qtest runs. > It might be a good idea to go thorough of the possibilities and find a better > way, if possible. > I think our main goal could be for the TestBeeLineDriver test output to match > the TestCliDriver output of the came query file. Or if it is not possible, > then at least a similar one > CC: [~vihangk1] -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HIVE-16146) If possible find a better way to filter the TestBeeLineDriver output
[ https://issues.apache.org/jira/browse/HIVE-16146?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15960923#comment-15960923 ] Peter Vary commented on HIVE-16146: --- The test failures are not related: - HIVE-16405 - Flaky Test: org.apache.hive.jdbc.TestJdbcDriver2.testResultSetMetaData - HIVE-15776 - Flaky test: TestMiniLlapLocalCliDriver vector_if_expr > If possible find a better way to filter the TestBeeLineDriver output > > > Key: HIVE-16146 > URL: https://issues.apache.org/jira/browse/HIVE-16146 > Project: Hive > Issue Type: Improvement > Components: Testing Infrastructure >Affects Versions: 2.2.0 >Reporter: Peter Vary >Assignee: Peter Vary > Attachments: HIVE-16146.02.patch, HIVE-16146.03.patch, > HIVE-16146.04.patch, HIVE-16146.05.patch, HIVE-16146.patch > > > Currently we apply a blacklist to filter the output of the BeeLine Qtest runs. > It might be a good idea to go thorough of the possibilities and find a better > way, if possible. > I think our main goal could be for the TestBeeLineDriver test output to match > the TestCliDriver output of the came query file. Or if it is not possible, > then at least a similar one > CC: [~vihangk1] -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HIVE-16334) Query lock contains the query string, which can cause OOM on ZooKeeper
[ https://issues.apache.org/jira/browse/HIVE-16334?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15960825#comment-15960825 ] Peter Vary commented on HIVE-16334: --- Thanks for the commit [~ctang.ma]! Updated the document, added the new property. > Query lock contains the query string, which can cause OOM on ZooKeeper > -- > > Key: HIVE-16334 > URL: https://issues.apache.org/jira/browse/HIVE-16334 > Project: Hive > Issue Type: Improvement > Components: Locking >Reporter: Peter Vary >Assignee: Peter Vary > Fix For: 3.0.0 > > Attachments: HIVE-16334.2.patch, HIVE-16334.3.patch, > HIVE-16334.4.patch, HIVE-16334.patch > > > When there are big number of partitions in a query this will result in a huge > number of locks on ZooKeeper. Since the query object contains the whole query > string this might cause serious memory pressure on the ZooKeeper services. > It would be good to have the possibility to truncate the query strings that > are written into the locks -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HIVE-16451) Race condition between HiveStatement.getQueryLog and HiveStatement.runAsyncOnServer
[
https://issues.apache.org/jira/browse/HIVE-16451?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15972328#comment-15972328
]
Peter Vary commented on HIVE-16451:
---
This 2 are known flaky:
- HIVE-15776 - Flaky test: TestMiniLlapLocalCliDriver vector_if_expr
- HIVE-16362- Flaky test:
TestMiniLlapLocalCliDriver.testCliDriver[vector_count_distinct]
Registered 2 new flaky:
- HIVE-16466 - Flaky test: TestJdbcDriver2.testSelectExecAsync2
- HIVE-16467 - Flaky test: TestCliDriver.testCliDriver[vector_order_null]
So I think the test failures are not related.
[~vgumashta], [~thejas]: Could you please review, since this patch affects
Ambari specific code as well in {{HiveStatement.executeAsync}}.
Thanks,
Peter
> Race condition between HiveStatement.getQueryLog and
> HiveStatement.runAsyncOnServer
> ---
>
> Key: HIVE-16451
> URL: https://issues.apache.org/jira/browse/HIVE-16451
> Project: Hive
> Issue Type: Bug
> Components: Beeline
>Affects Versions: 3.0.0
>Reporter: Peter Vary
>Assignee: Peter Vary
> Attachments: HIVE-16451.02.patch, HIVE-16451.03.patch,
> HIVE-16451.patch
>
>
> During the BeeLineDriver testing I have met the following race condition:
> - Run the query asynchronously through BeeLine
> - Querying the logs in the BeeLine
> In the following code:
> {code:title=HiveStatement.runAsyncOnServer}
> private void runAsyncOnServer(String sql) throws SQLException {
> checkConnection("execute");
> closeClientOperation();
> initFlags();
> [..]
> }
> {code}
> {code:title=HiveStatement.getQueryLog}
> public List getQueryLog(boolean incremental, int fetchSize)
> throws SQLException, ClosedOrCancelledStatementException {
> [..]
> try {
> if (stmtHandle != null) {
> [..]
> } else {
> if (isQueryClosed) {
> throw new ClosedOrCancelledStatementException("Method getQueryLog()
> failed. The " +
> "statement has been closed or cancelled.");
> } else {
> return logs;
> }
> }
> } catch (SQLException e) {
> [..]
> }
> [..]
> }
> {code}
> The runAsyncOnServer {{closeClientOperation}} sets {{isQueryClosed}} flag to
> true:
> {code:title=HiveStatement.closeClientOperation}
> void closeClientOperation() throws SQLException {
> [..]
> isQueryClosed = true;
> isExecuteStatementFailed = false;
> stmtHandle = null;
> }
> {code}
> The {{initFlags}} sets it to false:
> {code}
> private void initFlags() {
> isCancelled = false;
> isQueryClosed = false;
> isLogBeingGenerated = true;
> isExecuteStatementFailed = false;
> isOperationComplete = false;
> }
> {code}
> If the {{getQueryLog}} is called after the {{closeClientOperation}}, but
> before the {{initFlags}}, then we will have a following warning if verbose
> mode is set to true in BeeLine:
> {code}
> Warning: org.apache.hive.jdbc.ClosedOrCancelledStatementException: Method
> getQueryLog() failed. The statement has been closed or cancelled.
> (state=,code=0)
> {code}
> This caused this fail:
> https://builds.apache.org/job/PreCommit-HIVE-Build/4691/testReport/org.apache.hadoop.hive.cli/TestBeeLineDriver/testCliDriver_smb_mapjoin_11_/
> {code}
> Error Message
> Client result comparison failed with error code = 1 while executing
> fname=smb_mapjoin_11
> 16a17
> > Warning: org.apache.hive.jdbc.ClosedOrCancelledStatementException: Method
> > getQueryLog() failed. The statement has been closed or cancelled.
> > (state=,code=0)
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Updated] (HIVE-16449) BeeLineDriver should handle query result sorting
[ https://issues.apache.org/jira/browse/HIVE-16449?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Peter Vary updated HIVE-16449: -- Attachment: HIVE-16449.04.patch Rebased the patch, handled checkstyle and findbugs errors, and made it even more clear, that the converters are working only in test mode > BeeLineDriver should handle query result sorting > > > Key: HIVE-16449 > URL: https://issues.apache.org/jira/browse/HIVE-16449 > Project: Hive > Issue Type: Improvement > Components: Testing Infrastructure >Affects Versions: 3.0.0 >Reporter: Peter Vary >Assignee: Peter Vary > Attachments: HIVE-16449.02.patch, HIVE-16449.03.patch, > HIVE-16449.04.patch, HIVE-16449.patch > > > The CLI driver supports the following features: > -- SORT_QUERY_RESULTS > -- HASH_QUERY_RESULTS > -- SORT_AND_HASH_QUERY_RESULTS > BeeLineDriver should find a way to support these -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HIVE-16451) Race condition between HiveStatement.getQueryLog and HiveStatement.runAsyncOnServer
[
https://issues.apache.org/jira/browse/HIVE-16451?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Peter Vary updated HIVE-16451:
--
Attachment: HIVE-16451.02.patch
Addressed test failures
> Race condition between HiveStatement.getQueryLog and
> HiveStatement.runAsyncOnServer
> ---
>
> Key: HIVE-16451
> URL: https://issues.apache.org/jira/browse/HIVE-16451
> Project: Hive
> Issue Type: Bug
> Components: Beeline
>Affects Versions: 3.0.0
>Reporter: Peter Vary
>Assignee: Peter Vary
> Attachments: HIVE-16451.02.patch, HIVE-16451.patch
>
>
> During the BeeLineDriver testing I have met the following race condition:
> - Run the query asynchronously through BeeLine
> - Querying the logs in the BeeLine
> In the following code:
> {code:title=HiveStatement.runAsyncOnServer}
> private void runAsyncOnServer(String sql) throws SQLException {
> checkConnection("execute");
> closeClientOperation();
> initFlags();
> [..]
> }
> {code}
> {code:title=HiveStatement.getQueryLog}
> public List getQueryLog(boolean incremental, int fetchSize)
> throws SQLException, ClosedOrCancelledStatementException {
> [..]
> try {
> if (stmtHandle != null) {
> [..]
> } else {
> if (isQueryClosed) {
> throw new ClosedOrCancelledStatementException("Method getQueryLog()
> failed. The " +
> "statement has been closed or cancelled.");
> } else {
> return logs;
> }
> }
> } catch (SQLException e) {
> [..]
> }
> [..]
> }
> {code}
> The runAsyncOnServer {{closeClientOperation}} sets {{isQueryClosed}} flag to
> true:
> {code:title=HiveStatement.closeClientOperation}
> void closeClientOperation() throws SQLException {
> [..]
> isQueryClosed = true;
> isExecuteStatementFailed = false;
> stmtHandle = null;
> }
> {code}
> The {{initFlags}} sets it to false:
> {code}
> private void initFlags() {
> isCancelled = false;
> isQueryClosed = false;
> isLogBeingGenerated = true;
> isExecuteStatementFailed = false;
> isOperationComplete = false;
> }
> {code}
> If the {{getQueryLog}} is called after the {{closeClientOperation}}, but
> before the {{initFlags}}, then we will have a following warning if verbose
> mode is set to true in BeeLine:
> {code}
> Warning: org.apache.hive.jdbc.ClosedOrCancelledStatementException: Method
> getQueryLog() failed. The statement has been closed or cancelled.
> (state=,code=0)
> {code}
> This caused this fail:
> https://builds.apache.org/job/PreCommit-HIVE-Build/4691/testReport/org.apache.hadoop.hive.cli/TestBeeLineDriver/testCliDriver_smb_mapjoin_11_/
> {code}
> Error Message
> Client result comparison failed with error code = 1 while executing
> fname=smb_mapjoin_11
> 16a17
> > Warning: org.apache.hive.jdbc.ClosedOrCancelledStatementException: Method
> > getQueryLog() failed. The statement has been closed or cancelled.
> > (state=,code=0)
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HIVE-16467) Flaky test: TestCliDriver.testCliDriver[vector_order_null]
[
https://issues.apache.org/jira/browse/HIVE-16467?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15972263#comment-15972263
]
Peter Vary commented on HIVE-16467:
---
Well, since every query has an {{ORDER BY}} clause, it is not a good idea, to
add an extra sorting :)
The test file is added by HIVE-16448.
[~mmccline], [~gopalv]: Could you please check if it is enough to change every
{{ORDER BY a asc}} with {{ORDER BY a asc, b asc}}, and regenerate the golden
files?
Thanks,
Peter
> Flaky test: TestCliDriver.testCliDriver[vector_order_null]
> --
>
> Key: HIVE-16467
> URL: https://issues.apache.org/jira/browse/HIVE-16467
> Project: Hive
> Issue Type: Sub-task
>Reporter: Peter Vary
>
> Seems to me, that we have to add a {{-- SORT_QUERY_RESULTS}} to the query
> test.
> The error message is
> {code}
> Client Execution succeeded but contained differences (error code = 1) after
> executing vector_order_null.q
> 153a154
> > 2 B
> 156d156
> < 2 B
> 247a248
> > 2 B
> 250d250
> < 2 B
> 531a532
> > 2 B
> 534d534
> < 2 B
> 626a627
> > 2 B
> 629d629
> < 2 B
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HIVE-16467) Flaky test: TestCliDriver.testCliDriver[vector_order_null]
[
https://issues.apache.org/jira/browse/HIVE-16467?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15972450#comment-15972450
]
Peter Vary commented on HIVE-16467:
---
[~mmccline], the vector_order_null query file is added by HIVE-16448, and the
only problem with that the source data is:
{code}
insert into src_null values (1, 'A');
insert into src_null values (null, null);
insert into src_null values (3, null);
insert into src_null values (2, null);
insert into src_null values (2, 'A');
insert into src_null values (2, 'B');
{code}
And the results are only ordered by a column, and not by b, so as I see both
results are valid:
{code}
NULLNULL
1 A
2 B
2 A
2 NULL
3 NULL
{code}
or
{code}
NULLNULL
1 A
2 A
2 B
2 NULL
3 NULL
{code}
What do you think?
Thanks,
Peter
> Flaky test: TestCliDriver.testCliDriver[vector_order_null]
> --
>
> Key: HIVE-16467
> URL: https://issues.apache.org/jira/browse/HIVE-16467
> Project: Hive
> Issue Type: Sub-task
>Reporter: Peter Vary
>
> Seems to me, that we have to add a {{-- SORT_QUERY_RESULTS}} to the query
> test.
> The error message is
> {code}
> Client Execution succeeded but contained differences (error code = 1) after
> executing vector_order_null.q
> 153a154
> > 2 B
> 156d156
> < 2 B
> 247a248
> > 2 B
> 250d250
> < 2 B
> 531a532
> > 2 B
> 534d534
> < 2 B
> 626a627
> > 2 B
> 629d629
> < 2 B
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Assigned] (HIVE-16467) Flaky test: TestCliDriver.testCliDriver[vector_order_null]
[
https://issues.apache.org/jira/browse/HIVE-16467?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Peter Vary reassigned HIVE-16467:
-
Assignee: Peter Vary
> Flaky test: TestCliDriver.testCliDriver[vector_order_null]
> --
>
> Key: HIVE-16467
> URL: https://issues.apache.org/jira/browse/HIVE-16467
> Project: Hive
> Issue Type: Sub-task
>Reporter: Peter Vary
>Assignee: Peter Vary
>
> Seems to me, that we have to add a {{-- SORT_QUERY_RESULTS}} to the query
> test.
> The error message is
> {code}
> Client Execution succeeded but contained differences (error code = 1) after
> executing vector_order_null.q
> 153a154
> > 2 B
> 156d156
> < 2 B
> 247a248
> > 2 B
> 250d250
> < 2 B
> 531a532
> > 2 B
> 534d534
> < 2 B
> 626a627
> > 2 B
> 629d629
> < 2 B
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Updated] (HIVE-16467) Flaky test: TestCliDriver.testCliDriver[vector_order_null]
[
https://issues.apache.org/jira/browse/HIVE-16467?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Peter Vary updated HIVE-16467:
--
Status: Patch Available (was: Open)
> Flaky test: TestCliDriver.testCliDriver[vector_order_null]
> --
>
> Key: HIVE-16467
> URL: https://issues.apache.org/jira/browse/HIVE-16467
> Project: Hive
> Issue Type: Sub-task
>Reporter: Peter Vary
>Assignee: Peter Vary
> Attachments: HIVE-16467.patch
>
>
> Seems to me, that we have to add a {{-- SORT_QUERY_RESULTS}} to the query
> test.
> The error message is
> {code}
> Client Execution succeeded but contained differences (error code = 1) after
> executing vector_order_null.q
> 153a154
> > 2 B
> 156d156
> < 2 B
> 247a248
> > 2 B
> 250d250
> < 2 B
> 531a532
> > 2 B
> 534d534
> < 2 B
> 626a627
> > 2 B
> 629d629
> < 2 B
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Updated] (HIVE-16467) Flaky test: TestCliDriver.testCliDriver[vector_order_null]
[
https://issues.apache.org/jira/browse/HIVE-16467?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Peter Vary updated HIVE-16467:
--
Attachment: HIVE-16467.patch
It is easier to create the patch, than describe :)
> Flaky test: TestCliDriver.testCliDriver[vector_order_null]
> --
>
> Key: HIVE-16467
> URL: https://issues.apache.org/jira/browse/HIVE-16467
> Project: Hive
> Issue Type: Sub-task
>Reporter: Peter Vary
>Assignee: Peter Vary
> Attachments: HIVE-16467.patch
>
>
> Seems to me, that we have to add a {{-- SORT_QUERY_RESULTS}} to the query
> test.
> The error message is
> {code}
> Client Execution succeeded but contained differences (error code = 1) after
> executing vector_order_null.q
> 153a154
> > 2 B
> 156d156
> < 2 B
> 247a248
> > 2 B
> 250d250
> < 2 B
> 531a532
> > 2 B
> 534d534
> < 2 B
> 626a627
> > 2 B
> 629d629
> < 2 B
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HIVE-16467) Flaky test: TestCliDriver.testCliDriver[vector_order_null]
[
https://issues.apache.org/jira/browse/HIVE-16467?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15972534#comment-15972534
]
Peter Vary commented on HIVE-16467:
---
Hi [~mmccline],
Judging from the test name, the queries and the jira, I think the test aims to
test the order by clause. So if we add the SORT_QUERY_RESULTS then we defeat
the original intention of the test.
I was asking you, because the IDEA Annotate tells me, you were the one who
added this test with HIVE-16448 :)
Thanks,
Peter
> Flaky test: TestCliDriver.testCliDriver[vector_order_null]
> --
>
> Key: HIVE-16467
> URL: https://issues.apache.org/jira/browse/HIVE-16467
> Project: Hive
> Issue Type: Sub-task
>Reporter: Peter Vary
>Assignee: Peter Vary
> Attachments: HIVE-16467.patch
>
>
> Seems to me, that we have to add a {{-- SORT_QUERY_RESULTS}} to the query
> test.
> The error message is
> {code}
> Client Execution succeeded but contained differences (error code = 1) after
> executing vector_order_null.q
> 153a154
> > 2 B
> 156d156
> < 2 B
> 247a248
> > 2 B
> 250d250
> < 2 B
> 531a532
> > 2 B
> 534d534
> < 2 B
> 626a627
> > 2 B
> 629d629
> < 2 B
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HIVE-16357) Failed folder creation when creating a new table is reported incorrectly
[
https://issues.apache.org/jira/browse/HIVE-16357?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16108643#comment-16108643
]
Peter Vary commented on HIVE-16357:
---
Thanks for the patch [~zsombor.klara]!
I like this solution. +1 pending tests.
Peter
> Failed folder creation when creating a new table is reported incorrectly
>
>
> Key: HIVE-16357
> URL: https://issues.apache.org/jira/browse/HIVE-16357
> Project: Hive
> Issue Type: Bug
> Components: Metastore
>Affects Versions: 2.3.0, 3.0.0
>Reporter: Barna Zsombor Klara
>Assignee: Barna Zsombor Klara
>Priority: Minor
> Fix For: 3.0.0
>
> Attachments: HIVE-16357.01.patch, HIVE-16357.02.patch,
> HIVE-16357.03.patch, HIVE-16357.04.patch
>
>
> If the directory for a Hive table could not be created, them the HMS will
> throw a metaexception:
> {code}
> if (tblPath != null) {
> if (!wh.isDir(tblPath)) {
> if (!wh.mkdirs(tblPath, true)) {
> throw new MetaException(tblPath
> + " is not a directory or unable to create one");
> }
> madeDir = true;
> }
> }
> {code}
> However in the finally block we always try to call the
> DbNotificationListener, which in turn will also throw an exception because
> the directory is missing, overwriting the initial exception with a
> FileNotFoundException.
> Actual stacktrace seen by the caller:
> {code}
> 2017-04-03T05:58:00,128 ERROR [pool-7-thread-2] metastore.RetryingHMSHandler:
> MetaException(message:java.lang.RuntimeException:
> java.io.FileNotFoundException: File file:/.../0 does not exist)
> at
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.newMetaException(HiveMetaStore.java:6074)
> at
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.create_table_with_environment_context(HiveMetaStore.java:1496)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.apache.hadoop.hive.metastore.RetryingHMSHandler.invokeInternal(RetryingHMSHandler.java:148)
> at
> org.apache.hadoop.hive.metastore.RetryingHMSHandler.invoke(RetryingHMSHandler.java:107)
> at com.sun.proxy.$Proxy28.create_table_with_environment_context(Unknown
> Source)
> at
> org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$create_table_with_environment_context.getResult(ThriftHiveMetastore.java:11125)
> at
> org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$create_table_with_environment_context.getResult(ThriftHiveMetastore.java:11109)
> at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
> at
> org.apache.hadoop.hive.metastore.TUGIBasedProcessor$1.run(TUGIBasedProcessor.java:110)
> at
> org.apache.hadoop.hive.metastore.TUGIBasedProcessor$1.run(TUGIBasedProcessor.java:106)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)
> at
> org.apache.hadoop.hive.metastore.TUGIBasedProcessor.process(TUGIBasedProcessor.java:118)
> at
> org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:286)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: java.lang.RuntimeException: java.io.FileNotFoundException: File
> file:/.../0 does not exist
> at
> org.apache.hive.hcatalog.listener.DbNotificationListener$FileIterator.(DbNotificationListener.java:203)
> at
> org.apache.hive.hcatalog.listener.DbNotificationListener.onCreateTable(DbNotificationListener.java:137)
> at
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.create_table_core(HiveMetaStore.java:1463)
> at
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.create_table_with_environment_context(HiveMetaStore.java:1482)
> ... 20 more
> Caused by: java.io.FileNotFoundException: File file:/.../0 does not exist
> at
> org.apache.hadoop.fs.RawLocalFileSystem.listStatus(RawLocalFileSystem.java:429)
> at org.apache.hadoop.fs.FileSystem.listStatus(FileSystem.java:1515)
> at org.apache.hadoop.fs.FileSystem.listStatus(FileSystem.java:1555)
> at
>
[jira] [Assigned] (HIVE-17230) Timestamp format different in HiveCLI and Beeline
[
https://issues.apache.org/jira/browse/HIVE-17230?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Peter Vary reassigned HIVE-17230:
-
> Timestamp format different in HiveCLI and Beeline
> -
>
> Key: HIVE-17230
> URL: https://issues.apache.org/jira/browse/HIVE-17230
> Project: Hive
> Issue Type: Bug
> Components: Beeline, CLI
>Reporter: Peter Vary
>Assignee: Peter Vary
>
> The issue can be reproduced with the following commands:
> {code}
> create table timestamp_test(t timestamp);
> insert into table timestamp_test values('2000-01-01 01:00:00');
> select * from timestamp_test;
> {code}
> The timestamp is displayed without nanoseconds in HiveCLI:
> {code}
> 2000-01-01 01:00:00
> {code}
> When the exact same timestamp is displayed in BeeLine it displays:
> {code}
> 2000-01-01 01:00:00.0
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HIVE-17230) Timestamp format different in HiveCLI and Beeline
[
https://issues.apache.org/jira/browse/HIVE-17230?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16110874#comment-16110874
]
Peter Vary commented on HIVE-17230:
---
The root cause of the issue is, that HiveCLI writes out the value with
TimestampWritable.toString() method which specifically changes the output of
the timeStamp.toString() for an unknown reason since the "inception" of this
code :)
{code:title=TimestampWritable}
@Override
public String toString() {
if (timestampEmpty) {
populateTimestamp();
}
String timestampString = timestamp.toString();
if (timestampString.length() > 19) {
if (timestampString.length() == 21) {
if (timestampString.substring(19).compareTo(".0") == 0) {
return threadLocalDateFormat.get().format(timestamp);
}
}
return threadLocalDateFormat.get().format(timestamp) +
timestampString.substring(19);
}
return threadLocalDateFormat.get().format(timestamp);
}
{code}
The BeeLine receives a Timestamp object and calls it's toString without any
changes:
{code:title=org.apache.hive.beeline.Rows}
Row(int size, ResultSet rs) throws SQLException {
[..]
for (int i = 0; i < size; i++) {
if (numberFormat != null) {
Object o = rs.getObject(i + 1);
if (o == null) {
values[i] = null;
} else if (o instanceof Number) {
values[i] = numberFormat.format(o);
} else {
values[i] = o.toString(); <--- This row prints out the
Timestamp object
}
} else {
values[i] = rs.getString(i + 1);
}
values[i] = values[i] == null ? nullStr : values[i];
sizes[i] = values[i].length();
}
}
{code}
> Timestamp format different in HiveCLI and Beeline
> -
>
> Key: HIVE-17230
> URL: https://issues.apache.org/jira/browse/HIVE-17230
> Project: Hive
> Issue Type: Bug
> Components: Beeline, CLI
>Reporter: Peter Vary
>Assignee: Peter Vary
>
> The issue can be reproduced with the following commands:
> {code}
> create table timestamp_test(t timestamp);
> insert into table timestamp_test values('2000-01-01 01:00:00');
> select * from timestamp_test;
> {code}
> The timestamp is displayed without nanoseconds in HiveCLI:
> {code}
> 2000-01-01 01:00:00
> {code}
> When the exact same timestamp is displayed in BeeLine it displays:
> {code}
> 2000-01-01 01:00:00.0
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HIVE-17230) Timestamp format different in HiveCLI and Beeline
[
https://issues.apache.org/jira/browse/HIVE-17230?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16110888#comment-16110888
]
Peter Vary commented on HIVE-17230:
---
I have found this issue when I was trying to run several more tests with the
TestBeeLineDriver to see if there is any further differences between the output
of BeeLine, and HiveCLI.
[~vihangk1], [~aihuaxu]: What do you think we should do with this difference?
- Change the way BeeLine prints out timestamps?
- Change the way HiveCLI prints out timestamps?
- Document this as a known issue?
I personally do not understand/like the HiveCLI version where the Timestamp
nanoseconds are not displayed - in my view this column was specifically defined
as a Timestamp because we expect it to be an exact time, so I would go for
removing the formatting changes from the HiveCLI, but this is a backward
incompatible change, and I might not see every aspect of this change.
I would greatly value your input here [~aihuaxu], [~vihangk1].
Thanks,
Peter
> Timestamp format different in HiveCLI and Beeline
> -
>
> Key: HIVE-17230
> URL: https://issues.apache.org/jira/browse/HIVE-17230
> Project: Hive
> Issue Type: Bug
> Components: Beeline, CLI
>Reporter: Peter Vary
>Assignee: Peter Vary
>
> The issue can be reproduced with the following commands:
> {code}
> create table timestamp_test(t timestamp);
> insert into table timestamp_test values('2000-01-01 01:00:00');
> select * from timestamp_test;
> {code}
> The timestamp is displayed without nanoseconds in HiveCLI:
> {code}
> 2000-01-01 01:00:00
> {code}
> When the exact same timestamp is displayed in BeeLine it displays:
> {code}
> 2000-01-01 01:00:00.0
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HIVE-16294) Support snapshot for truncate table
[ https://issues.apache.org/jira/browse/HIVE-16294?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16112497#comment-16112497 ] Peter Vary commented on HIVE-16294: --- There is a non public API to find out, if a directory is snapshottable or not: - DFSClient.getSnapshottableDirListing(), or - DistributedFileSystem.getSnapshottableDirListing()) IMO, here is how we should handle this: - Create a Hadoop jira for requesting a public API. - Until the API is available it is enough to check if the fs is an instance of DistributedFileSystem, and the filename is ".snapshot" - Add a comment to remove this code, when the API is available. But I would like to know how the more experienced guys think about this. What do you think about this:[~aihuaxu], [~andrew.wang]? Thanks, Peter > Support snapshot for truncate table > --- > > Key: HIVE-16294 > URL: https://issues.apache.org/jira/browse/HIVE-16294 > Project: Hive > Issue Type: Sub-task > Components: Query Processor >Reporter: Vihang Karajgaonkar >Assignee: Barna Zsombor Klara > Attachments: HIVE-16294.01.patch, HIVE-16294.02.patch > > -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HIVE-17144) export of temporary tables not working and it seems to be using distcp rather than filesystem copy
[ https://issues.apache.org/jira/browse/HIVE-17144?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16112539#comment-16112539 ] Peter Vary commented on HIVE-17144: --- Thanks [~zsombor.klara] for localizing this issue. Committed the remaining files using the attached patch file. I hope there is no difference between the pull request and the patch file :) [~anishek]: Could you please file a jira and add the remaining apache headers to the new files? I was not sure if I am allowed to change the patch, so committed it without change to unblock others. > export of temporary tables not working and it seems to be using distcp rather > than filesystem copy > -- > > Key: HIVE-17144 > URL: https://issues.apache.org/jira/browse/HIVE-17144 > Project: Hive > Issue Type: Bug > Components: Hive, HiveServer2 >Affects Versions: 3.0.0 >Reporter: anishek >Assignee: anishek > Fix For: 3.0.0 > > Attachments: HIVE-17144.1.patch > > > create temporary table t1 (i int); > insert into t1 values (3); > export table t1 to 'hdfs://somelocation'; > above fails. additionally it should use filesystem copy and not distcp to do > the job. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HIVE-16845) INSERT OVERWRITE a table with dynamic partitions on S3 fails with NPE
[
https://issues.apache.org/jira/browse/HIVE-16845?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Peter Vary updated HIVE-16845:
--
Resolution: Fixed
Fix Version/s: 3.0.0
Status: Resolved (was: Patch Available)
Pushed to master.
Thanks [~kuczoram] for the patch!
> INSERT OVERWRITE a table with dynamic partitions on S3 fails with NPE
> -
>
> Key: HIVE-16845
> URL: https://issues.apache.org/jira/browse/HIVE-16845
> Project: Hive
> Issue Type: Bug
>Affects Versions: 2.1.1
>Reporter: Marta Kuczora
>Assignee: Marta Kuczora
> Fix For: 3.0.0
>
> Attachments: HIVE-16845.1.patch, HIVE-16845.2.patch,
> HIVE-16845.3.patch, HIVE-16845.4.patch
>
>
> *How to reproduce*
> - Create a partitioned table on S3:
> {noformat}
> CREATE EXTERNAL TABLE s3table(user_id string COMMENT '', event_name string
> COMMENT '') PARTITIONED BY (reported_date string, product_id int) LOCATION
> 's3a://';
> {noformat}
> - Create a temp table:
> {noformat}
> create table tmp_table (id string, name string, date string, pid int) row
> format delimited fields terminated by '\t' lines terminated by '\n' stored as
> textfile;
> {noformat}
> - Load the following rows to the tmp table:
> {noformat}
> u1value1 2017-04-10 1
> u2value2 2017-04-10 1
> u3value3 2017-04-10 10001
> {noformat}
> - Set the following parameters:
> -- hive.exec.dynamic.partition.mode=nonstrict
> -- mapreduce.input.fileinputformat.split.maxsize=10
> -- hive.blobstore.optimizations.enabled=true
> -- hive.blobstore.use.blobstore.as.scratchdir=false
> -- hive.merge.mapfiles=true
> - Insert the rows from the temp table into the s3 table:
> {noformat}
> INSERT OVERWRITE TABLE s3table
> PARTITION (reported_date, product_id)
> SELECT
> t.id as user_id,
> t.name as event_name,
> t.date as reported_date,
> t.pid as product_id
> FROM tmp_table t;
> {noformat}
> A NPE will occur with the following stacktrace:
> {noformat}
> 2017-05-08 21:32:50,607 ERROR
> org.apache.hive.service.cli.operation.Operation:
> [HiveServer2-Background-Pool: Thread-184028]: Error running hive query:
> org.apache.hive.service.cli.HiveSQLException: Error while processing
> statement: FAILED: Execution Error, return code -101 from
> org.apache.hadoop.hive.ql.exec.ConditionalTask. null
> at
> org.apache.hive.service.cli.operation.Operation.toSQLException(Operation.java:400)
> at
> org.apache.hive.service.cli.operation.SQLOperation.runQuery(SQLOperation.java:239)
> at
> org.apache.hive.service.cli.operation.SQLOperation.access$300(SQLOperation.java:88)
> at
> org.apache.hive.service.cli.operation.SQLOperation$3$1.run(SQLOperation.java:293)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:415)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1920)
> at
> org.apache.hive.service.cli.operation.SQLOperation$3.run(SQLOperation.java:306)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
> at java.util.concurrent.FutureTask.run(FutureTask.java:262)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: java.lang.NullPointerException
> at
> org.apache.hadoop.hive.ql.plan.ConditionalResolverMergeFiles.generateActualTasks(ConditionalResolverMergeFiles.java:290)
> at
> org.apache.hadoop.hive.ql.plan.ConditionalResolverMergeFiles.getTasks(ConditionalResolverMergeFiles.java:175)
> at
> org.apache.hadoop.hive.ql.exec.ConditionalTask.execute(ConditionalTask.java:81)
> at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:214)
> at
> org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:100)
> at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:1977)
> at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1690)
> at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1422)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1206)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1201)
> at
> org.apache.hive.service.cli.operation.SQLOperation.runQuery(SQLOperation.java:237)
> ... 11 more
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HIVE-16294) Support snapshot for truncate table
[ https://issues.apache.org/jira/browse/HIVE-16294?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16114161#comment-16114161 ] Peter Vary commented on HIVE-16294: --- +1 pending tests > Support snapshot for truncate table > --- > > Key: HIVE-16294 > URL: https://issues.apache.org/jira/browse/HIVE-16294 > Project: Hive > Issue Type: Sub-task > Components: Query Processor >Reporter: Vihang Karajgaonkar >Assignee: Barna Zsombor Klara > Attachments: HIVE-16294.01.patch, HIVE-16294.02.patch, > HIVE-16294.03.patch, HIVE-16294.04.patch > > -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HIVE-17311) Numeric overflow in the HiveConf
[
https://issues.apache.org/jira/browse/HIVE-17311?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16127405#comment-16127405
]
Peter Vary commented on HIVE-17311:
---
[~olegd]: Could you please update the description field, to match the patch :D
Thanks,
Peter
> Numeric overflow in the HiveConf
>
>
> Key: HIVE-17311
> URL: https://issues.apache.org/jira/browse/HIVE-17311
> Project: Hive
> Issue Type: Bug
>Reporter: Oleg Danilov
>Assignee: Oleg Danilov
>Priority: Minor
> Attachments: HIVE-17311.patch
>
>
> multiplierFor() method contains a typo, which causes wrong parsing of the
> rare suffixes ('tb' & 'pb').
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (HIVE-17311) Numeric overflow in the HiveConf
[
https://issues.apache.org/jira/browse/HIVE-17311?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Peter Vary updated HIVE-17311:
--
Resolution: Fixed
Fix Version/s: 3.0.0
Status: Resolved (was: Patch Available)
Pushed to master.
Thanks for the patch [~olegd]!
> Numeric overflow in the HiveConf
>
>
> Key: HIVE-17311
> URL: https://issues.apache.org/jira/browse/HIVE-17311
> Project: Hive
> Issue Type: Bug
>Reporter: Oleg Danilov
>Assignee: Oleg Danilov
>Priority: Minor
> Fix For: 3.0.0
>
> Attachments: HIVE-17311.patch
>
>
> multiplierFor() method contains a typo, which causes wrong parsing of the
> rare suffixes ('tb' & 'pb').
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Issue Comment Deleted] (HIVE-17311) Numeric overflow in the HiveConf
[
https://issues.apache.org/jira/browse/HIVE-17311?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Peter Vary updated HIVE-17311:
--
Comment: was deleted
(was: [~olegd]: Could you please update the description field, to match the
patch :D
Thanks,
Peter)
> Numeric overflow in the HiveConf
>
>
> Key: HIVE-17311
> URL: https://issues.apache.org/jira/browse/HIVE-17311
> Project: Hive
> Issue Type: Bug
>Reporter: Oleg Danilov
>Assignee: Oleg Danilov
>Priority: Minor
> Attachments: HIVE-17311.patch
>
>
> multiplierFor() method contains a typo, which causes wrong parsing of the
> rare suffixes ('tb' & 'pb').
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (HIVE-17305) New insert overwrite dynamic partitions qtest need to have the golden file regenerated
[ https://issues.apache.org/jira/browse/HIVE-17305?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Peter Vary updated HIVE-17305: -- Resolution: Fixed Fix Version/s: 3.0.0 Status: Resolved (was: Patch Available) Pushed to master. Thanks [~zsombor.klara] for the patch! > New insert overwrite dynamic partitions qtest need to have the golden file > regenerated > -- > > Key: HIVE-17305 > URL: https://issues.apache.org/jira/browse/HIVE-17305 > Project: Hive > Issue Type: Bug > Components: Tests >Affects Versions: 3.0.0 >Reporter: Barna Zsombor Klara >Assignee: Barna Zsombor Klara >Priority: Trivial > Fix For: 3.0.0 > > Attachments: HIVE-17305.01.patch > > -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HIVE-17322) Serialise BeeLine qtest execution to prevent flakyness
[ https://issues.apache.org/jira/browse/HIVE-17322?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16127401#comment-16127401 ] Peter Vary commented on HIVE-17322: --- +1 LGTM Could you please run the test 2-3 more times before commit to be sure this solves the problem? And after the commit we should create a follow-up jira to solve the root cause of this issue, or at least identify it. Thanks for the patch [~zsombor.klara]! Peter > Serialise BeeLine qtest execution to prevent flakyness > -- > > Key: HIVE-17322 > URL: https://issues.apache.org/jira/browse/HIVE-17322 > Project: Hive > Issue Type: Bug >Affects Versions: 3.0.0 >Reporter: Barna Zsombor Klara >Assignee: Barna Zsombor Klara >Priority: Minor > Attachments: HIVE-17322.01.patch, HIVE-17322.02.patch, > HIVE-17322.03.patch > > -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HIVE-17292) Change TestMiniSparkOnYarnCliDriver test configuration to use the configured cores
[
https://issues.apache.org/jira/browse/HIVE-17292?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Peter Vary updated HIVE-17292:
--
Attachment: HIVE-17292.5.patch
Well, this patch become huge :(
The actual code/configuration change is minimal:
- QTestUtil.java - to check for 4 cores before allowing to run a query
- SparkSessionImpl.java - to use the same method to calculating cores was with
spark.master="spark.\*"
- Hadoop23Shims.java - to change the scheduler allocation minimum, this way
allowing the MiniCluster to create 2 nodes
- The others are only q.out changes
-- Number of executors 2->4
-- Number of result files are higher because the executor number is higher
-- When there is no order by in the query the resulting lines are mixed in some
cases (union.q.out, union11.q.out, union14.q.out, union15.q.out, union7.q.out,
union_null.q.out) - We might have to apply {{-- SORT_QUERY_RESULTS}} if they
become flaky
-- The overall size of the result files become bigger (union_remove_10.q.out,
union_remove_13.q.out, union_remove_15.q.out, union_remove_16.q.out,
union_remove_7.q.out, union_remove_8.q.out, union_remove_9.q.out) - I think the
number of the files, and the overhead of the RCFileOutputFormat causes this
issue
- spark_dynamic_partition_pruning_mapjoin_only.q.out is changed - See:
HIVE-16948
What do you think about this change [~lirui]?
Shall we bite the bullet, and review/commit it - do we have a good way to
validate the changes?
Or shall we chicken out, and change the configuration back to use only 1
executor with 2 cores, and then only configuration change is needed?
> Change TestMiniSparkOnYarnCliDriver test configuration to use the configured
> cores
> --
>
> Key: HIVE-17292
> URL: https://issues.apache.org/jira/browse/HIVE-17292
> Project: Hive
> Issue Type: Sub-task
> Components: Spark, Test
>Affects Versions: 3.0.0
>Reporter: Peter Vary
>Assignee: Peter Vary
> Attachments: HIVE-17292.1.patch, HIVE-17292.2.patch,
> HIVE-17292.3.patch, HIVE-17292.5.patch
>
>
> Currently the {{hive-site.xml}} for the {{TestMiniSparkOnYarnCliDriver}} test
> defines 2 cores, and 2 executors, but only 1 is used, because the MiniCluster
> does not allows the creation of the 3rd container.
> The FairScheduler uses 1GB increments for memory, but the containers would
> like to use only 512MB. We should change the fairscheduler configuration to
> use only the requested 512MB
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (HIVE-17268) WebUI / QueryPlan: query plan is sometimes null when explain output conf is on
[ https://issues.apache.org/jira/browse/HIVE-17268?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Peter Vary updated HIVE-17268: -- Resolution: Fixed Fix Version/s: 3.0.0 Status: Resolved (was: Patch Available) Pushed to master. Thanks for your contribution [~klcopp]! > WebUI / QueryPlan: query plan is sometimes null when explain output conf is on > -- > > Key: HIVE-17268 > URL: https://issues.apache.org/jira/browse/HIVE-17268 > Project: Hive > Issue Type: Bug >Reporter: Karen Coppage >Assignee: Karen Coppage >Priority: Minor > Fix For: 3.0.0 > > Attachments: HIVE-17268.2.patch, HIVE-17268.3.patch, HIVE-17268.patch > > > The Hive WebUI's Query Plan tab displays "SET hive.log.explain.output TO true > TO VIEW PLAN" even when hive.log.explain.output is set to true, when the > query cannot be compiled, because the plan is null in this case. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HIVE-17292) Change TestMiniSparkOnYarnCliDriver test configuration to use the configured cores
[
https://issues.apache.org/jira/browse/HIVE-17292?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16128601#comment-16128601
]
Peter Vary commented on HIVE-17292:
---
[~lirui]: HIVE-17292.3.patch is the version where we change only the mini-yarn
test. It needs some polish but the relevant changes could be seen there. I
would be happy to provide that too if we decide so.
Thanks,
Peter
> Change TestMiniSparkOnYarnCliDriver test configuration to use the configured
> cores
> --
>
> Key: HIVE-17292
> URL: https://issues.apache.org/jira/browse/HIVE-17292
> Project: Hive
> Issue Type: Sub-task
> Components: Spark, Test
>Affects Versions: 3.0.0
>Reporter: Peter Vary
>Assignee: Peter Vary
> Attachments: HIVE-17292.1.patch, HIVE-17292.2.patch,
> HIVE-17292.3.patch, HIVE-17292.5.patch
>
>
> Currently the {{hive-site.xml}} for the {{TestMiniSparkOnYarnCliDriver}} test
> defines 2 cores, and 2 executors, but only 1 is used, because the MiniCluster
> does not allows the creation of the 3rd container.
> The FairScheduler uses 1GB increments for memory, but the containers would
> like to use only 512MB. We should change the fairscheduler configuration to
> use only the requested 512MB
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HIVE-17325) Clean up intermittently failing uni tests
[ https://issues.apache.org/jira/browse/HIVE-17325?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16128616#comment-16128616 ] Peter Vary commented on HIVE-17325: --- Thanks [~alangates] to bring this up! I really hate these continuously failing tests so I agree that we should solve them. I have information about some of them: - TestBeeLineDriver.testCliDriver.insert_overwrite_local_directory_1 - For the BeeLine tests [~zsombor.klara] has a patch which most probably solves the problem: HIVE-17322 - TestMiniSparkOnYarnCliDriver.testCliDriver.spark_dynamic_partition_pruning_mapjoin_only - I have a patch (HIVE-17292) where I will be update the golden files. It was missing from HIVE-16948. - TestMiniSparkOnYarnCliDriver.testCliDriver.spark_dynamic_partition_pruning - I have committed HIVE-17305 yesterday which should solve this issue I know about these, and not sure about how to handle them. Why we are not reverting the patch which caused the failures? - TestMiniSparkOnYarnCliDriver.testCliDriver.spark_vectorized_dynamic_partition_pruning - Most probably cause by HIVE-16273 - TestHCatClient - Caused by HIVE-16844 Thanks, Peter > Clean up intermittently failing uni tests > - > > Key: HIVE-17325 > URL: https://issues.apache.org/jira/browse/HIVE-17325 > Project: Hive > Issue Type: Test > Components: Tests >Reporter: Alan Gates >Assignee: Alan Gates > > We have a number of intermittently failing tests. I propose to disable these > so that we can get clean (or at least cleaner) CI runs. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
