[jira] [Updated] (HIVE-15859) HoS: Write RPC messages in event loop

2017-02-26 Thread Rui Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-15859?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rui Li updated HIVE-15859: -- Affects Version/s: (was: 2.2.0) 2.1.1 > HoS: Write RPC messages in event loop >

[jira] [Updated] (HIVE-15859) HoS: Write RPC messages in event loop

2017-02-26 Thread Rui Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-15859?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rui Li updated HIVE-15859: -- Summary: HoS: Write RPC messages in event loop (was: Hive client side shows Spark Driver disconnected while

[jira] [Updated] (HIVE-15859) Hive client side shows Spark Driver disconnected while Spark Driver side could not get RPC header

2017-02-26 Thread Rui Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-15859?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rui Li updated HIVE-15859: -- Attachment: HIVE-15859.3.patch Thanks [~KaiXu] for the clarifications. Update patch v3 to make sure we log the

[jira] [Commented] (HIVE-15882) HS2 generating high memory pressure with many partitions and concurrent queries

2017-02-23 Thread Rui Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-15882?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15882157#comment-15882157 ] Rui Li commented on HIVE-15882: --- The patch looks good to me overall. I left some minor comments on the RB.

[jira] [Commented] (HIVE-15882) HS2 generating high memory pressure with many partitions and concurrent queries

2017-02-23 Thread Rui Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-15882?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15881733#comment-15881733 ] Rui Li commented on HIVE-15882: --- Thanks [~mi...@cloudera.com] for the benchmark results. It looks promising.

[jira] [Commented] (HIVE-15859) Hive client side shows Spark Driver disconnected while Spark Driver side could not get RPC header

2017-02-23 Thread Rui Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-15859?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15880098#comment-15880098 ] Rui Li commented on HIVE-15859: --- I think I have managed to reproduce the issue by introducing some sleep

[jira] [Commented] (HIVE-15859) Hive client side shows Spark Driver disconnected while Spark Driver side could not get RPC header

2017-02-22 Thread Rui Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-15859?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15879995#comment-15879995 ] Rui Li commented on HIVE-15859: --- Hi [~xuefuz], netty's channel is thread safe. We can write to it

[jira] [Commented] (HIVE-15882) HS2 generating high memory pressure with many partitions and concurrent queries

2017-02-22 Thread Rui Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-15882?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15879917#comment-15879917 ] Rui Li commented on HIVE-15882: --- Hi [~mi...@cloudera.com], I guess one possible issue of String::intern is

[jira] [Commented] (HIVE-15859) Hive client side shows Spark Driver disconnected while Spark Driver side could not get RPC header

2017-02-22 Thread Rui Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-15859?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15878013#comment-15878013 ] Rui Li commented on HIVE-15859: --- Another way to fix (and also mentioned in the Livy PR) is to combine the

[jira] [Resolved] (HIVE-15946) Failing test : TestCliDriver cbo_rp_auto_join1

2017-02-21 Thread Rui Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-15946?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rui Li resolved HIVE-15946. --- Resolution: Resolved Fix Version/s: 2.2.0 Fixed in HIVE-15948 > Failing test : TestCliDriver

[jira] [Commented] (HIVE-15948) Failing test: TestCliDriver, TestSparkCliDriver join31

2017-02-20 Thread Rui Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-15948?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15875334#comment-15875334 ] Rui Li commented on HIVE-15948: --- [~stakiar] thanks for the update. +1 pending test > Failing test:

[jira] [Commented] (HIVE-15948) Failing test: TestCliDriver, TestSparkCliDriver join31

2017-02-20 Thread Rui Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-15948?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15875267#comment-15875267 ] Rui Li commented on HIVE-15948: --- Thanks for the patch [~stakiar]. I noticed there're some other places that

[jira] [Commented] (HIVE-15893) Followup on HIVE-15671

2017-02-20 Thread Rui Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-15893?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15874676#comment-15874676 ] Rui Li commented on HIVE-15893: --- Hi [~xuefuz], our RPC channel has a handler that monitors the channel

[jira] [Commented] (HIVE-15882) HS2 generating high memory pressure with many partitions and concurrent queries

2017-02-17 Thread Rui Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-15882?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15871561#comment-15871561 ] Rui Li commented on HIVE-15882: --- I also noticed lots of duplicated Strings in {{ListSinkOperator::res}} when

[jira] [Updated] (HIVE-15710) HS2 Stopped when running in background

2017-02-16 Thread Rui Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-15710?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rui Li updated HIVE-15710: -- Resolution: Fixed Fix Version/s: 2.2.0 Status: Resolved (was: Patch Available) Committed to

[jira] [Commented] (HIVE-15710) HS2 Stopped when running in background

2017-02-16 Thread Rui Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-15710?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15871059#comment-15871059 ] Rui Li commented on HIVE-15710: --- The age1 failure can't be reproduced locally. I'll commit shortly if no

[jira] [Commented] (HIVE-15912) Executor kill task and Failed to get spark memory/core info

2017-02-16 Thread Rui Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-15912?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15869505#comment-15869505 ] Rui Li commented on HIVE-15912: --- No I don't think the warning can be the cause. For yarn-client mode, you

[jira] [Commented] (HIVE-15926) Hive 2.1.1 is not supporting any version of Spark

2017-02-16 Thread Rui Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-15926?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15869485#comment-15869485 ] Rui Li commented on HIVE-15926: --- Hi [~hrishidypim], Hive-2.1.1 doesn't support Spark-2.0.0 or later.

[jira] [Updated] (HIVE-15710) HS2 Stopped when running in background

2017-02-15 Thread Rui Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-15710?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rui Li updated HIVE-15710: -- Attachment: HIVE-15710.3.patch Update to address comments. I think CLI also needs this logic. [~Ferd] let me

[jira] [Updated] (HIVE-15710) HS2 Stopped when running in background

2017-02-15 Thread Rui Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-15710?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rui Li updated HIVE-15710: -- Attachment: HIVE-15710.2.patch Update patch v2 based on Ferdinand's comments. [~Ferd] please let me know if it

[jira] [Commented] (HIVE-15710) HS2 Stopped when running in background

2017-02-15 Thread Rui Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-15710?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15867495#comment-15867495 ] Rui Li commented on HIVE-15710: --- Sorry I meant to start HS2, user can either call {{bin/hiveserver2}}, or

[jira] [Updated] (HIVE-15859) Hive client side shows Spark Driver disconnected while Spark Driver side could not get RPC header

2017-02-15 Thread Rui Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-15859?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rui Li updated HIVE-15859: -- Attachment: HIVE-15859.2.patch Fix test. > Hive client side shows Spark Driver disconnected while Spark Driver

[jira] [Commented] (HIVE-15710) HS2 Stopped when running in background

2017-02-14 Thread Rui Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-15710?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15867372#comment-15867372 ] Rui Li commented on HIVE-15710: --- Thanks [~Ferd] for your suggestions. I guess the redundancy is not a big

[jira] [Updated] (HIVE-15710) HS2 Stopped when running in background

2017-02-14 Thread Rui Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-15710?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rui Li updated HIVE-15710: -- Status: Patch Available (was: Open) > HS2 Stopped when running in background >

[jira] [Updated] (HIVE-15710) HS2 Stopped when running in background

2017-02-14 Thread Rui Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-15710?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rui Li updated HIVE-15710: -- Attachment: HIVE-15710.1.patch Move the check to bin/hive in patch v1. Also pinging [~Ferd] for reivew. > HS2

[jira] [Commented] (HIVE-15912) Executor kill task and Failed to get spark memory/core info

2017-02-14 Thread Rui Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-15912?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15867198#comment-15867198 ] Rui Li commented on HIVE-15912: --- The failure to get mem/core info is just a warning. So it's not related to

[jira] [Commented] (HIVE-15887) could not get APP ID and cause failed to connect to spark driver on yarn-client mode

2017-02-14 Thread Rui Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-15887?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15867150#comment-15867150 ] Rui Li commented on HIVE-15887: --- The symptom is quite similar to that of HIVE-12650. Is this a busy cluster?

[jira] [Commented] (HIVE-15710) HS2 Stopped when running in background

2017-02-14 Thread Rui Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-15710?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15865389#comment-15865389 ] Rui Li commented on HIVE-15710: --- Ping [~xuefuz], [~mohitsabharwal] again. > HS2 Stopped when running in

[jira] [Commented] (HIVE-15887) could not get APP ID and cause failed to connect to spark driver on yarn-client mode

2017-02-14 Thread Rui Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-15887?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15865384#comment-15865384 ] Rui Li commented on HIVE-15887: --- In the AM log, the "failure to connect to driver" happened around 5:05.

[jira] [Updated] (HIVE-15859) Hive client side shows Spark Driver disconnected while Spark Driver side could not get RPC header

2017-02-13 Thread Rui Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-15859?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rui Li updated HIVE-15859: -- Status: Patch Available (was: Open) > Hive client side shows Spark Driver disconnected while Spark Driver side

[jira] [Updated] (HIVE-15859) Hive client side shows Spark Driver disconnected while Spark Driver side could not get RPC header

2017-02-13 Thread Rui Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-15859?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rui Li updated HIVE-15859: -- Attachment: HIVE-15859.1.patch Patch v1 based on the Livy PR. [~KaiXu], could you test if the patch fixes your

[jira] [Assigned] (HIVE-15859) Hive client side shows Spark Driver disconnected while Spark Driver side could not get RPC header

2017-02-13 Thread Rui Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-15859?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rui Li reassigned HIVE-15859: - Assignee: Rui Li > Hive client side shows Spark Driver disconnected while Spark Driver side > could not

[jira] [Comment Edited] (HIVE-15859) Hive client side shows Spark Driver disconnected while Spark Driver side could not get RPC header

2017-02-13 Thread Rui Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-15859?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15865239#comment-15865239 ] Rui Li edited comment on HIVE-15859 at 2/14/17 7:11 AM: Hi [~vanzin], thanks for

[jira] [Commented] (HIVE-15859) Hive client side shows Spark Driver disconnected while Spark Driver side could not get RPC header

2017-02-13 Thread Rui Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-15859?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15865239#comment-15865239 ] Rui Li commented on HIVE-15859: --- Hi [~vanzin], thanks for providing the Livy PR. That's exactly the same

[jira] [Commented] (HIVE-15893) Followup on HIVE-15671

2017-02-13 Thread Rui Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-15893?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15864892#comment-15864892 ] Rui Li commented on HIVE-15893: --- My hunch is HIVE-15860 doesn't help here. It's only intended for RSC

[jira] [Commented] (HIVE-15887) could not get APP ID and cause failed to connect to spark driver on yarn-client mode

2017-02-13 Thread Rui Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-15887?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15863645#comment-15863645 ] Rui Li commented on HIVE-15887: --- Seems the application is ACCEPTED but didn't reach RUNNING. YARN log may be

[jira] [Commented] (HIVE-15859) Hive client side shows Spark Driver disconnected while Spark Driver side could not get RPC header

2017-02-13 Thread Rui Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-15859?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15863622#comment-15863622 ] Rui Li commented on HIVE-15859: --- I noticed writing to the channel is protected by channelLock in

[jira] [Updated] (HIVE-15859) Hive client side shows Spark Driver disconnected while Spark Driver side could not get RPC header

2017-02-13 Thread Rui Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-15859?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rui Li updated HIVE-15859: -- Description: Hive on Spark, failed with error: {noformat} 2017-02-08 09:50:59,331 Stage-2_0: 1039(+2)/1041

[jira] [Updated] (HIVE-15860) RemoteSparkJobMonitor may hang when RemoteDriver exits abnormally

2017-02-13 Thread Rui Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-15860?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rui Li updated HIVE-15860: -- Resolution: Fixed Fix Version/s: 2.2.0 Status: Resolved (was: Patch Available) Committed to

[jira] [Commented] (HIVE-15887) could not get APP ID and cause failed to connect to spark driver on yarn-client mode

2017-02-13 Thread Rui Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-15887?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15863160#comment-15863160 ] Rui Li commented on HIVE-15887: --- Hi [~KaiXu], the failure is just a warning and it shouldn't make the query

[jira] [Updated] (HIVE-15887) could not get APP ID and cause failed to connect to spark driver on yarn-client mode

2017-02-13 Thread Rui Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-15887?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rui Li updated HIVE-15887: -- Description: {noformat} 2017-02-13T03:10:01,639 INFO [stderr-redir-1] client.SparkClientImpl: 17/02/13

[jira] [Commented] (HIVE-15860) RemoteSparkJobMonitor may hang when RemoteDriver exits abnormally

2017-02-13 Thread Rui Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-15860?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15863228#comment-15863228 ] Rui Li commented on HIVE-15860: --- Latest failure not related. > RemoteSparkJobMonitor may hang when

[jira] [Updated] (HIVE-15860) RemoteSparkJobMonitor may hang when RemoteDriver exits abnormally

2017-02-12 Thread Rui Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-15860?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rui Li updated HIVE-15860: -- Attachment: HIVE-15860.2.patch > RemoteSparkJobMonitor may hang when RemoteDriver exits abnormally >

[jira] [Updated] (HIVE-15860) RemoteSparkJobMonitor may hang when RemoteDriver exits abnormally

2017-02-10 Thread Rui Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-15860?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rui Li updated HIVE-15860: -- Attachment: HIVE-15860.2.patch Patch v2 adds the check only when job is in STARTED state and we can't get the

[jira] [Commented] (HIVE-15860) RemoteSparkJobMonitor may hang when RemoteDriver exits abnormally

2017-02-10 Thread Rui Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-15860?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15861036#comment-15861036 ] Rui Li commented on HIVE-15860: --- A more specific way to fix it is just add the check when job has started

[jira] [Commented] (HIVE-15860) RemoteSparkJobMonitor may hang when RemoteDriver exits abnormally

2017-02-10 Thread Rui Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-15860?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15861020#comment-15861020 ] Rui Li commented on HIVE-15860: --- [~xuefuz] - yeah the monitor loops forever in that case. For the monitor,

[jira] [Updated] (HIVE-15860) RemoteSparkJobMonitor may hang when RemoteDriver exits abnormally

2017-02-10 Thread Rui Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-15860?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rui Li updated HIVE-15860: -- Status: Patch Available (was: Open) > RemoteSparkJobMonitor may hang when RemoteDriver exits abnormally >

[jira] [Updated] (HIVE-15860) RemoteSparkJobMonitor may hang when RemoteDriver exits abnormally

2017-02-10 Thread Rui Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-15860?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rui Li updated HIVE-15860: -- Attachment: HIVE-15860.1.patch Actually Rpc's listener can detect the remote end becomes inactive and mark

[jira] [Commented] (HIVE-15860) RemoteSparkJobMonitor may hang when RemoteDriver exits abnormally

2017-02-09 Thread Rui Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-15860?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15859410#comment-15859410 ] Rui Li commented on HIVE-15860: --- We cannot simply extend the monitor timeout to cover {{JobSubmitted}},

[jira] [Updated] (HIVE-15860) RemoteSparkJobMonitor may hang when RemoteDriver exits abnormally

2017-02-09 Thread Rui Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-15860?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rui Li updated HIVE-15860: -- Description: It happens when RemoteDriver crashes between {{JobStarted}} and {{JobSubmitted}}, e.g. killed by

[jira] [Assigned] (HIVE-15860) RemoteSparkJobMonitor may hang when RemoteDriver exists abnormally

2017-02-09 Thread Rui Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-15860?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rui Li reassigned HIVE-15860: - > RemoteSparkJobMonitor may hang when RemoteDriver exists abnormally >

[jira] [Updated] (HIVE-15860) RemoteSparkJobMonitor may hang when RemoteDriver exits abnormally

2017-02-09 Thread Rui Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-15860?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rui Li updated HIVE-15860: -- Summary: RemoteSparkJobMonitor may hang when RemoteDriver exits abnormally (was: RemoteSparkJobMonitor may

[jira] [Commented] (HIVE-15671) RPCServer.registerClient() erroneously uses server/client handshake timeout for connection timeout

2017-02-09 Thread Rui Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-15671?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15859267#comment-15859267 ] Rui Li commented on HIVE-15671: --- Hi [~xuefuz], can you get a thread dump of HS2/CLI when you hit the 2nd

[jira] [Commented] (HIVE-15671) RPCServer.registerClient() erroneously uses server/client handshake timeout for connection timeout

2017-01-23 Thread Rui Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-15671?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15834115#comment-15834115 ] Rui Li commented on HIVE-15671: --- Hi [~xuefuz], I tried your case but didn't reproduce your issue. Here's my

[jira] [Updated] (HIVE-15519) BitSet not computed properly for ColumnBuffer subset

2017-01-22 Thread Rui Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-15519?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rui Li updated HIVE-15519: -- Resolution: Fixed Fix Version/s: 2.2.0 1.3.0 Status: Resolved (was: Patch

[jira] [Commented] (HIVE-15519) BitSet not computed properly for ColumnBuffer subset

2017-01-22 Thread Rui Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-15519?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15833829#comment-15833829 ] Rui Li commented on HIVE-15519: --- Committed to master and branch-1. Thanks Thejas for the review! > BitSet

[jira] [Commented] (HIVE-15519) BitSet not computed properly for ColumnBuffer subset

2017-01-22 Thread Rui Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-15519?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15833819#comment-15833819 ] Rui Li commented on HIVE-15519: --- [~thejas], thanks for reviewing and the suggestion. I have locally run

[jira] [Commented] (HIVE-15671) RPCServer.registerClient() erroneously uses server/client handshake timeout for connection timeout

2017-01-20 Thread Rui Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-15671?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15831452#comment-15831452 ] Rui Li commented on HIVE-15671: --- Hi [~xuefuz], I tried your patch locally.

[jira] [Commented] (HIVE-15519) BitSet not computed properly for ColumnBuffer subset

2017-01-20 Thread Rui Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-15519?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15831399#comment-15831399 ] Rui Li commented on HIVE-15519: --- The last test is for v5 patch for branch-1. I tried some of the failures

[jira] [Commented] (HIVE-15519) BitSet not computed properly for ColumnBuffer subset

2017-01-18 Thread Rui Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-15519?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15829151#comment-15829151 ] Rui Li commented on HIVE-15519: --- Hi [~thejas], could you have a look at the latest patches? FYI v6 is for

[jira] [Commented] (HIVE-15580) Replace Spark's groupByKey operator with something with bounded memory

2017-01-18 Thread Rui Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-15580?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15828249#comment-15828249 ] Rui Li commented on HIVE-15580: --- Hmm Spark does make it clear groupByKey can't handle large keys:

[jira] [Commented] (HIVE-15580) Replace Spark's groupByKey operator with something with bounded memory

2017-01-18 Thread Rui Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-15580?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15828183#comment-15828183 ] Rui Li commented on HIVE-15580: --- [~xuefuz], thanks for your explanations. It makes sense. So in general, the

[jira] [Commented] (HIVE-15580) Replace Spark's groupByKey operator with something with bounded memory

2017-01-18 Thread Rui Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-15580?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15828143#comment-15828143 ] Rui Li commented on HIVE-15580: --- Hi [~xuefuz], I'd like to check my understanding too. Before the patch, we

[jira] [Commented] (HIVE-15519) BitSet not computed properly for ColumnBuffer subset

2017-01-18 Thread Rui Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-15519?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15828082#comment-15828082 ] Rui Li commented on HIVE-15519: --- The last test is for v6 patch. The patch is also in RB. Test failures with

[jira] [Updated] (HIVE-15519) BitSet not computed properly for ColumnBuffer subset

2017-01-18 Thread Rui Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-15519?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rui Li updated HIVE-15519: -- Attachment: (was: HIVE-15519.5-branch-1.patch) > BitSet not computed properly for ColumnBuffer subset >

[jira] [Updated] (HIVE-15519) BitSet not computed properly for ColumnBuffer subset

2017-01-18 Thread Rui Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-15519?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rui Li updated HIVE-15519: -- Attachment: HIVE-15519.5-branch-1.patch > BitSet not computed properly for ColumnBuffer subset >

[jira] [Updated] (HIVE-15519) BitSet not computed properly for ColumnBuffer subset

2017-01-17 Thread Rui Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-15519?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rui Li updated HIVE-15519: -- Attachment: HIVE-15519.6.patch > BitSet not computed properly for ColumnBuffer subset >

[jira] [Updated] (HIVE-15519) BitSet not computed properly for ColumnBuffer subset

2017-01-17 Thread Rui Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-15519?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rui Li updated HIVE-15519: -- Attachment: (was: HIVE-15519.6.patch) > BitSet not computed properly for ColumnBuffer subset >

[jira] [Updated] (HIVE-15519) BitSet not computed properly for ColumnBuffer subset

2017-01-15 Thread Rui Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-15519?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rui Li updated HIVE-15519: -- Attachment: (was: HIVE-15519.6.patch) > BitSet not computed properly for ColumnBuffer subset >

[jira] [Updated] (HIVE-15519) BitSet not computed properly for ColumnBuffer subset

2017-01-15 Thread Rui Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-15519?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rui Li updated HIVE-15519: -- Attachment: HIVE-15519.6.patch > BitSet not computed properly for ColumnBuffer subset >

[jira] [Updated] (HIVE-15519) BitSet not computed properly for ColumnBuffer subset

2017-01-13 Thread Rui Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-15519?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rui Li updated HIVE-15519: -- Attachment: (was: HIVE-15519.6.patch) > BitSet not computed properly for ColumnBuffer subset >

[jira] [Updated] (HIVE-15519) BitSet not computed properly for ColumnBuffer subset

2017-01-13 Thread Rui Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-15519?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rui Li updated HIVE-15519: -- Attachment: HIVE-15519.6.patch > BitSet not computed properly for ColumnBuffer subset >

[jira] [Updated] (HIVE-15519) BitSet not computed properly for ColumnBuffer subset

2017-01-12 Thread Rui Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-15519?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rui Li updated HIVE-15519: -- Attachment: HIVE-15519.6.patch > BitSet not computed properly for ColumnBuffer subset >

[jira] [Updated] (HIVE-15519) BitSet not computed properly for ColumnBuffer subset

2017-01-12 Thread Rui Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-15519?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rui Li updated HIVE-15519: -- Attachment: (was: HIVE-15519.4.patch) > BitSet not computed properly for ColumnBuffer subset >

[jira] [Updated] (HIVE-15519) BitSet not computed properly for ColumnBuffer subset

2017-01-12 Thread Rui Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-15519?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rui Li updated HIVE-15519: -- Attachment: HIVE-15519.4.patch Upload the patch for master to run test again. I'll look into the test failures

[jira] [Updated] (HIVE-15519) BitSet not computed properly for ColumnBuffer subset

2017-01-11 Thread Rui Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-15519?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rui Li updated HIVE-15519: -- Attachment: HIVE-15519.5-branch-1.patch Patch v5 is for branch-1. > BitSet not computed properly for

[jira] [Updated] (HIVE-15519) BitSet not computed properly for ColumnBuffer subset

2017-01-11 Thread Rui Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-15519?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rui Li updated HIVE-15519: -- Attachment: HIVE-15519.4.patch Patch v4 merges the new UT with the current test. > BitSet not computed

[jira] [Commented] (HIVE-15519) BitSet not computed properly for ColumnBuffer subset

2017-01-11 Thread Rui Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-15519?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15818525#comment-15818525 ] Rui Li commented on HIVE-15519: --- Just realised we already have a UT for ColumnBuffer named TestColumn. I'll

[jira] [Commented] (HIVE-15527) Memory usage is unbound in SortByShuffler for Spark

2017-01-11 Thread Rui Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-15527?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15818027#comment-15818027 ] Rui Li commented on HIVE-15527: --- Hi [~csun], it seems the current solution relies on how the iterators are

[jira] [Updated] (HIVE-15299) Yarn-cluster and yarn-client deprecated in Spark 2.0

2017-01-11 Thread Rui Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-15299?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rui Li updated HIVE-15299: -- Resolution: Fixed Fix Version/s: 2.2.0 Status: Resolved (was: Patch Available) Committed to

[jira] [Commented] (HIVE-15519) BitSet not computed properly for ColumnBuffer subset

2017-01-11 Thread Rui Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-15519?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15817679#comment-15817679 ] Rui Li commented on HIVE-15519: --- I tried the failures with age 1 and they can't be reproduced. > BitSet not

[jira] [Commented] (HIVE-15519) BitSet not computed properly for ColumnBuffer subset

2017-01-10 Thread Rui Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-15519?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15816955#comment-15816955 ] Rui Li commented on HIVE-15519: --- [~thejas], thanks for reviewing. I've updated patch to address your

[jira] [Updated] (HIVE-15519) BitSet not computed properly for ColumnBuffer subset

2017-01-10 Thread Rui Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-15519?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rui Li updated HIVE-15519: -- Attachment: HIVE-15519.3.patch > BitSet not computed properly for ColumnBuffer subset >

[jira] [Updated] (HIVE-15299) Yarn-cluster and yarn-client deprecated in Spark 2.0

2017-01-10 Thread Rui Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-15299?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rui Li updated HIVE-15299: -- Attachment: HIVE-15299.2.patch Fix tests. > Yarn-cluster and yarn-client deprecated in Spark 2.0 >

[jira] [Updated] (HIVE-15519) BitSet not computed properly for ColumnBuffer subset

2017-01-10 Thread Rui Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-15519?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rui Li updated HIVE-15519: -- Summary: BitSet not computed properly for ColumnBuffer subset (was: Hive Decimal Type column scale is

[jira] [Commented] (HIVE-15519) Hive Decimal Type column scale is returning as zero

2017-01-10 Thread Rui Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-15519?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15814317#comment-15814317 ] Rui Li commented on HIVE-15519: --- Ping [~thejas] again. [~xuefuz], could you also have a look? Thanks. >

[jira] [Updated] (HIVE-15299) Yarn-cluster and yarn-client deprecated in Spark 2.0

2017-01-09 Thread Rui Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-15299?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rui Li updated HIVE-15299: -- Attachment: HIVE-15299.1.patch > Yarn-cluster and yarn-client deprecated in Spark 2.0 >

[jira] [Updated] (HIVE-15299) Yarn-cluster and yarn-client deprecated in Spark 2.0

2017-01-09 Thread Rui Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-15299?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rui Li updated HIVE-15299: -- Status: Patch Available (was: Open) > Yarn-cluster and yarn-client deprecated in Spark 2.0 >

[jira] [Commented] (HIVE-15474) Extend limit propagation for chain of RS-GB-RS operators

2017-01-05 Thread Rui Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-15474?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15803369#comment-15803369 ] Rui Li commented on HIVE-15474: --- Thanks for the update [~jcamachorodriguez]. +1 > Extend limit propagation

[jira] [Commented] (HIVE-15474) Extend limit propagation for chain of RS-GB-RS operators

2017-01-05 Thread Rui Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-15474?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15800947#comment-15800947 ] Rui Li commented on HIVE-15474: --- I see. Thanks [~jcamachorodriguez]. Instead of having another

[jira] [Commented] (HIVE-15543) Don't try to get memory/cores to decide parallelism when Spark dynamic allocation is enabled

2017-01-04 Thread Rui Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-15543?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15800594#comment-15800594 ] Rui Li commented on HIVE-15543: --- +1 > Don't try to get memory/cores to decide parallelism when Spark

[jira] [Updated] (HIVE-15526) Some tests need SORT_QUERY_RESULTS

2017-01-03 Thread Rui Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-15526?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rui Li updated HIVE-15526: -- Resolution: Fixed Fix Version/s: 2.2.0 Status: Resolved (was: Patch Available) Committed to

[jira] [Commented] (HIVE-15474) Extend limit propagation for chain of RS-GB-RS operators

2017-01-03 Thread Rui Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-15474?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15797276#comment-15797276 ] Rui Li commented on HIVE-15474: --- Thanks [~jcamachorodriguez] for enabling the Spark test. It looks good to

[jira] [Commented] (HIVE-15526) Some tests need SORT_QUERY_RESULTS

2017-01-03 Thread Rui Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-15526?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15797092#comment-15797092 ] Rui Li commented on HIVE-15526: --- [~xuefuz], thanks for your comments. I chose order by for

[jira] [Commented] (HIVE-15526) Some tests need SORT_QUERY_RESULTS

2017-01-02 Thread Rui Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-15526?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15794403#comment-15794403 ] Rui Li commented on HIVE-15526: --- Failures not related. [~xuefuz], could you have a look? The change is

[jira] [Updated] (HIVE-15526) Some tests need SORT_QUERY_RESULTS

2017-01-02 Thread Rui Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-15526?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rui Li updated HIVE-15526: -- Description: {{temp_table_gb1.q}} and {{vector_between_in.q}} > Some tests need SORT_QUERY_RESULTS >

[jira] [Commented] (HIVE-15313) Add export spark.yarn.archive or spark.yarn.jars variable in Hive on Spark document

2017-01-02 Thread Rui Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-15313?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15794215#comment-15794215 ] Rui Li commented on HIVE-15313: --- [~Ferd], yeah I'm OK to update the wiki for performance first. I'll update

[jira] [Updated] (HIVE-15526) Some tests need SORT_QUERY_RESULTS

2017-01-02 Thread Rui Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-15526?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rui Li updated HIVE-15526: -- Status: Patch Available (was: Open) > Some tests need SORT_QUERY_RESULTS > --

[jira] [Updated] (HIVE-15526) Some tests need SORT_QUERY_RESULTS

2017-01-02 Thread Rui Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-15526?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rui Li updated HIVE-15526: -- Attachment: HIVE-15526.1.patch > Some tests need SORT_QUERY_RESULTS > -- > >

[jira] [Commented] (HIVE-15527) Memory usage is unbound in SortByShuffler for Spark

2017-01-02 Thread Rui Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-15527?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15793903#comment-15793903 ] Rui Li commented on HIVE-15527: --- Since the HiveKVResultCache here only stores values for the same key, I

<    3   4   5   6   7   8   9   10   11   12   >