[jira] [Commented] (HIVE-15526) Some tests need SORT_QUERY_RESULTS

2017-01-03 Thread Rui Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-15526?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15797092#comment-15797092 ] Rui Li commented on HIVE-15526: --- [~xuefuz], thanks for your comments. I chose order by for v

[jira] [Commented] (HIVE-15526) Some tests need SORT_QUERY_RESULTS

2017-01-02 Thread Rui Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-15526?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15794403#comment-15794403 ] Rui Li commented on HIVE-15526: --- Failures not related. [~xuefuz], could you have a look? The

[jira] [Updated] (HIVE-15526) Some tests need SORT_QUERY_RESULTS

2017-01-02 Thread Rui Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-15526?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rui Li updated HIVE-15526: -- Description: {{temp_table_gb1.q}} and {{vector_between_in.q}} > Some tests need SORT_QUERY_RESULTS > ---

[jira] [Commented] (HIVE-15313) Add export spark.yarn.archive or spark.yarn.jars variable in Hive on Spark document

2017-01-02 Thread Rui Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-15313?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15794215#comment-15794215 ] Rui Li commented on HIVE-15313: --- [~Ferd], yeah I'm OK to update the wiki for performance fir

[jira] [Updated] (HIVE-15526) Some tests need SORT_QUERY_RESULTS

2017-01-02 Thread Rui Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-15526?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rui Li updated HIVE-15526: -- Status: Patch Available (was: Open) > Some tests need SORT_QUERY_RESULTS > -- >

[jira] [Updated] (HIVE-15526) Some tests need SORT_QUERY_RESULTS

2017-01-02 Thread Rui Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-15526?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rui Li updated HIVE-15526: -- Attachment: HIVE-15526.1.patch > Some tests need SORT_QUERY_RESULTS > -- > >

[jira] [Commented] (HIVE-15527) Memory usage is unbound in SortByShuffler for Spark

2017-01-02 Thread Rui Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-15527?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15793903#comment-15793903 ] Rui Li commented on HIVE-15527: --- Since the HiveKVResultCache here only stores values for the

[jira] [Commented] (HIVE-15474) Extend limit propagation for chain of RS-GB-RS operators

2016-12-29 Thread Rui Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-15474?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15786788#comment-15786788 ] Rui Li commented on HIVE-15474: --- [~jcamachorodriguez], thanks very much for the detailed exp

[jira] [Commented] (HIVE-15519) Hive Decimal Type column scale is returning as zero

2016-12-29 Thread Rui Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-15519?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15786741#comment-15786741 ] Rui Li commented on HIVE-15519: --- The failures are not related because I can't reproduce them

[jira] [Commented] (HIVE-15519) Hive Decimal Type column scale is returning as zero

2016-12-28 Thread Rui Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-15519?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15784767#comment-15784767 ] Rui Li commented on HIVE-15519: --- Update patch v2 to add a UT. And also put it in an RB entry

[jira] [Updated] (HIVE-15519) Hive Decimal Type column scale is returning as zero

2016-12-28 Thread Rui Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-15519?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rui Li updated HIVE-15519: -- Attachment: HIVE-15519.2.patch > Hive Decimal Type column scale is returning as zero > -

[jira] [Commented] (HIVE-8373) OOM for a simple query with spark.master=local [Spark Branch]

2016-12-28 Thread Rui Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-8373?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15782500#comment-15782500 ] Rui Li commented on HIVE-8373: -- Thanks [~asears] for the explanations. [~kellyzly], yes please

[jira] [Updated] (HIVE-15519) Hive Decimal Type column scale is returning as zero

2016-12-28 Thread Rui Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-15519?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rui Li updated HIVE-15519: -- Status: Patch Available (was: Open) > Hive Decimal Type column scale is returning as zero > ---

[jira] [Updated] (HIVE-15519) Hive Decimal Type column scale is returning as zero

2016-12-28 Thread Rui Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-15519?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rui Li updated HIVE-15519: -- Attachment: HIVE-15519.1.patch > Hive Decimal Type column scale is returning as zero > -

[jira] [Commented] (HIVE-15519) Hive Decimal Type column scale is returning as zero

2016-12-28 Thread Rui Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-15519?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15782474#comment-15782474 ] Rui Li commented on HIVE-15519: --- Thanks [~bharatviswa] for raising the issue. I think we hav

[jira] [Assigned] (HIVE-15519) Hive Decimal Type column scale is returning as zero

2016-12-28 Thread Rui Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-15519?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rui Li reassigned HIVE-15519: - Assignee: Rui Li > Hive Decimal Type column scale is returning as zero >

[jira] [Updated] (HIVE-15519) Hive Decimal Type column scale is returning as zero

2016-12-28 Thread Rui Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-15519?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rui Li updated HIVE-15519: -- Description: Hive decimal type column precision is returning as zero, even though column has precision set. Exa

[jira] [Commented] (HIVE-8373) OOM for a simple query with spark.master=local [Spark Branch]

2016-12-22 Thread Rui Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-8373?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15771778#comment-15771778 ] Rui Li commented on HIVE-8373: -- Thanks [~asears] for the inputs. Just curious, in which case s

[jira] [Updated] (HIVE-15357) Fix and re-enable the spark-only tests

2016-12-22 Thread Rui Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-15357?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rui Li updated HIVE-15357: -- Resolution: Fixed Fix Version/s: 2.2.0 Status: Resolved (was: Patch Available) Committed to m

[jira] [Commented] (HIVE-8373) OOM for a simple query with spark.master=local [Spark Branch]

2016-12-21 Thread Rui Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-8373?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15769309#comment-15769309 ] Rui Li commented on HIVE-8373: -- Thanks [~kellyzly] for working on this. Have you verified the

[jira] [Commented] (HIVE-15474) Extend limit propagation for chain of RS-GB-RS operators

2016-12-21 Thread Rui Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-15474?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15766929#comment-15766929 ] Rui Li commented on HIVE-15474: --- Hi [~jcamachorodriguez], I think it's an interesting optimi

[jira] [Commented] (HIVE-15357) Fix and re-enable the spark-only tests

2016-12-20 Thread Rui Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-15357?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15765903#comment-15765903 ] Rui Li commented on HIVE-15357: --- Failures are not related. [~csun] could you take a look? Th

[jira] [Updated] (HIVE-15357) Fix and re-enable the spark-only tests

2016-12-20 Thread Rui Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-15357?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rui Li updated HIVE-15357: -- Status: Patch Available (was: Open) > Fix and re-enable the spark-only tests >

[jira] [Updated] (HIVE-15357) Fix and re-enable the spark-only tests

2016-12-20 Thread Rui Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-15357?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rui Li updated HIVE-15357: -- Attachment: HIVE-15357.1.patch All the update to golden files are in the query plan. No result changes. Some ch

[jira] [Commented] (HIVE-9153) Perf enhancement on CombineHiveInputFormat and HiveInputFormat

2016-12-19 Thread Rui Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-9153?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15763302#comment-15763302 ] Rui Li commented on HIVE-9153: -- I guess no configuration is suitable for all cases :) If I rem

[jira] [Updated] (HIVE-15428) HoS DPP doesn't remove cyclic dependency

2016-12-18 Thread Rui Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-15428?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rui Li updated HIVE-15428: -- Resolution: Fixed Fix Version/s: 2.2.0 Status: Resolved (was: Patch Available) Committed to m

[jira] [Commented] (HIVE-13278) Avoid FileNotFoundException when map/reduce.xml is not available

2016-12-18 Thread Rui Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-13278?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15760001#comment-15760001 ] Rui Li commented on HIVE-13278: --- Thanks [~csun] for the update. The latest patch looks good

[jira] [Updated] (HIVE-13293) Cache RDD to improve parallel order by performance for HoS

2016-12-16 Thread Rui Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-13293?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rui Li updated HIVE-13293: -- Summary: Cache RDD to improve parallel order by performance for HoS (was: Query occurs performance degradation

[jira] [Assigned] (HIVE-15272) "LEFT OUTER JOIN" Is not populating correct records with Hive On Spark

2016-12-15 Thread Rui Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-15272?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rui Li reassigned HIVE-15272: - Assignee: Rui Li > "LEFT OUTER JOIN" Is not populating correct records with Hive On Spark > -

[jira] [Commented] (HIVE-15272) "LEFT OUTER JOIN" Is not populating correct records with Hive On Spark

2016-12-15 Thread Rui Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-15272?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15753212#comment-15753212 ] Rui Li commented on HIVE-15272: --- OK I'll look into this. [~VPareek], I think the two tables

[jira] [Commented] (HIVE-15428) HoS DPP doesn't remove cyclic dependency

2016-12-15 Thread Rui Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-15428?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15753161#comment-15753161 ] Rui Li commented on HIVE-15428: --- Test failures are not related. > HoS DPP doesn't remove cy

[jira] [Comment Edited] (HIVE-13278) Avoid FileNotFoundException when map/reduce.xml is not available

2016-12-15 Thread Rui Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-13278?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15753143#comment-15753143 ] Rui Li edited comment on HIVE-13278 at 12/16/16 1:57 AM: - Hi [~csu

[jira] [Commented] (HIVE-13278) Avoid FileNotFoundException when map/reduce.xml is not available

2016-12-15 Thread Rui Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-13278?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15753143#comment-15753143 ] Rui Li commented on HIVE-13278: --- Hi [~csun], sorry maybe I was being misleading. What I have

[jira] [Updated] (HIVE-15428) HoS DPP doesn't remove cyclic dependency

2016-12-15 Thread Rui Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-15428?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rui Li updated HIVE-15428: -- Status: Patch Available (was: Open) > HoS DPP doesn't remove cyclic dependency > --

[jira] [Updated] (HIVE-15428) HoS DPP doesn't remove cyclic dependency

2016-12-15 Thread Rui Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-15428?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rui Li updated HIVE-15428: -- Attachment: HIVE-15428.1.patch Patch to add the detection. Basically it's just copied from Tez. And manually ve

[jira] [Commented] (HIVE-13278) Avoid FileNotFoundException when map/reduce.xml is not available

2016-12-15 Thread Rui Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-13278?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15750856#comment-15750856 ] Rui Li commented on HIVE-13278: --- Hi [~csun], I think Tez also calls setMapWork/setReduceWork

[jira] [Commented] (HIVE-15432) java.lang.ClassCastException is thrown when setting "hive.input.format" as "org.apache.hadoop.hive.ql.io.CombineHiveInputFormat" in hive on spark

2016-12-15 Thread Rui Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-15432?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15750782#comment-15750782 ] Rui Li commented on HIVE-15432: --- [~kellyzly], I think this is just a warning in the log and

[jira] [Commented] (HIVE-13278) Avoid FileNotFoundException when map/reduce.xml is not available

2016-12-14 Thread Rui Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-13278?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15750696#comment-15750696 ] Rui Li commented on HIVE-13278: --- Hi [~xuefuz], I just think it'll be even simpler to go the

[jira] [Commented] (HIVE-13278) Avoid FileNotFoundException when map/reduce.xml is not available

2016-12-14 Thread Rui Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-13278?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15750452#comment-15750452 ] Rui Li commented on HIVE-13278: --- Sorry about the delay. I have a concern about using flag: i

[jira] [Commented] (HIVE-15428) HoS DPP doesn't remove cyclic dependency

2016-12-14 Thread Rui Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-15428?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15748659#comment-15748659 ] Rui Li commented on HIVE-15428: --- The reason why we didn't see this before is related to how

[jira] [Commented] (HIVE-13278) Many redundant 'File not found' messages appeared in container log during query execution with Hive on Spark

2016-12-13 Thread Rui Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-13278?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15747436#comment-15747436 ] Rui Li commented on HIVE-13278: --- [~csun], I think the error is found in container's log. Spa

[jira] [Updated] (HIVE-15428) HoS DPP doesn't remove cyclic dependency

2016-12-13 Thread Rui Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-15428?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rui Li updated HIVE-15428: -- Description: More details in HIVE-15357 > HoS DPP doesn't remove cyclic dependency > ---

[jira] [Commented] (HIVE-15357) Fix and re-enable the spark-only tests

2016-12-13 Thread Rui Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-15357?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15747039#comment-15747039 ] Rui Li commented on HIVE-15357: --- Hi [~csun], I created HIVE-15428 for it. > Fix and re-enab

[jira] [Commented] (HIVE-13278) Many redundant 'File not found' messages appeared in container log during query execution with Hive on Spark

2016-12-13 Thread Rui Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-13278?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15747031#comment-15747031 ] Rui Li commented on HIVE-13278: --- [~csun], thanks for the patch. I think Xuefu's concerns are

[jira] [Commented] (HIVE-15357) Fix and re-enable the spark-only tests

2016-12-13 Thread Rui Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-15357?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15745110#comment-15745110 ] Rui Li commented on HIVE-15357: --- When I tried to re-enable these tests, I found this issue w

[jira] [Updated] (HIVE-15386) Expose Spark task counts and stage Ids information in SparkTask from SparkJobMonitor

2016-12-12 Thread Rui Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-15386?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rui Li updated HIVE-15386: -- Affects Version/s: (was: 2.2.0) 2.1.1 > Expose Spark task counts and stage Ids inform

[jira] [Updated] (HIVE-15386) Expose Spark task counts and stage Ids information in SparkTask from SparkJobMonitor

2016-12-12 Thread Rui Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-15386?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rui Li updated HIVE-15386: -- Resolution: Fixed Fix Version/s: 2.2.0 Status: Resolved (was: Patch Available) Committed to m

[jira] [Commented] (HIVE-13278) Many redundant 'File not found' messages appeared in container log during query execution with Hive on Spark

2016-12-12 Thread Rui Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-13278?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15744366#comment-15744366 ] Rui Li commented on HIVE-13278: --- Hi [~xuefuz], the conclusion is we somehow try to read redu

[jira] [Commented] (HIVE-15386) Expose Spark task counts and stage Ids information in SparkTask from SparkJobMonitor

2016-12-12 Thread Rui Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-15386?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15743862#comment-15743862 ] Rui Li commented on HIVE-15386: --- Thanks for the update [~zxu]. +1. [~xuefuz] do you have any

[jira] [Commented] (HIVE-15386) Expose Spark task counts and stage Ids information in SparkTask from SparkJobMonitor

2016-12-11 Thread Rui Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-15386?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15741232#comment-15741232 ] Rui Li commented on HIVE-15386: --- Thanks [~zxu] for the explanations. Then I think submitTime

[jira] [Comment Edited] (HIVE-15386) Expose Spark task counts and stage Ids information in SparkTask from SparkJobMonitor

2016-12-11 Thread Rui Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-15386?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15740845#comment-15740845 ] Rui Li edited comment on HIVE-15386 at 12/12/16 3:23 AM: - Hi [~zxu

[jira] [Commented] (HIVE-15386) Expose Spark task counts and stage Ids information in SparkTask from SparkJobMonitor

2016-12-11 Thread Rui Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-15386?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15740845#comment-15740845 ] Rui Li commented on HIVE-15386: --- Hi [~zxu], the v2 patch looks better. One question is what'

[jira] [Commented] (HIVE-9927) MR doesn't produce correct result for runtime_skewjoin_mapjoin_spark

2016-12-09 Thread Rui Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-9927?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15734982#comment-15734982 ] Rui Li commented on HIVE-9927: -- [~wenli], thanks for looking into this issue. Do you want to w

[jira] [Commented] (HIVE-15386) Expose Spark task counts and stage Ids information in SparkTask from SparkJobMonitor

2016-12-08 Thread Rui Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-15386?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15732341#comment-15732341 ] Rui Li commented on HIVE-15386: --- [~zxu], could you elaborate on how these task infos will be

[jira] [Updated] (HIVE-15357) Fix and re-enable the spark-only tests

2016-12-05 Thread Rui Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-15357?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rui Li updated HIVE-15357: -- Description: Defined by {{spark.only.query.files}}. > Fix and re-enable the spark-only tests > -

[jira] [Commented] (HIVE-15239) hive on spark combine equivalent work get wrong result because of TS operation compare

2016-12-05 Thread Rui Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-15239?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15722020#comment-15722020 ] Rui Li commented on HIVE-15239: --- Also filed HIVE-15357 for the tests. > hive on spark combi

[jira] [Updated] (HIVE-15239) hive on spark combine equivalent work get wrong result because of TS operation compare

2016-12-05 Thread Rui Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-15239?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rui Li updated HIVE-15239: -- Resolution: Fixed Fix Version/s: 2.2.0 Status: Resolved (was: Patch Available) Latest failure

[jira] [Updated] (HIVE-15239) hive on spark combine equivalent work get wrong result because of TS operation compare

2016-12-05 Thread Rui Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-15239?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rui Li updated HIVE-15239: -- Summary: hive on spark combine equivalent work get wrong result because of TS operation compare (was: hive on

[jira] [Updated] (HIVE-15239) hive on spark combine equivalentwork get wrong result because of tablescan operation compare

2016-12-04 Thread Rui Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-15239?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rui Li updated HIVE-15239: -- Attachment: HIVE-15239.4.patch > hive on spark combine equivalentwork get wrong result because of tablescan >

[jira] [Updated] (HIVE-15239) hive on spark combine equivalentwork get wrong result because of tablescan operation compare

2016-12-04 Thread Rui Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-15239?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rui Li updated HIVE-15239: -- Attachment: (was: HIVE-15239.4.patch) > hive on spark combine equivalentwork get wrong result because of ta

[jira] [Updated] (HIVE-15239) hive on spark combine equivalentwork get wrong result because of tablescan operation compare

2016-12-04 Thread Rui Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-15239?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rui Li updated HIVE-15239: -- Attachment: HIVE-15239.4.patch > hive on spark combine equivalentwork get wrong result because of tablescan >

[jira] [Updated] (HIVE-15239) hive on spark combine equivalentwork get wrong result because of tablescan operation compare

2016-12-04 Thread Rui Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-15239?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rui Li updated HIVE-15239: -- Attachment: (was: HIVE-15239.4.patch) > hive on spark combine equivalentwork get wrong result because of ta

[jira] [Updated] (HIVE-15239) hive on spark combine equivalentwork get wrong result because of tablescan operation compare

2016-12-01 Thread Rui Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-15239?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rui Li updated HIVE-15239: -- Attachment: HIVE-15239.4.patch Thanks [~xuefuz] for the review. Update patch v4 to address your comment. > hive

[jira] [Commented] (HIVE-15302) Relax the requirement that HoS needs Spark built w/o Hive

2016-12-01 Thread Rui Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-15302?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15713754#comment-15713754 ] Rui Li commented on HIVE-15302: --- [~kellyzly], if we use the config to make HoS run against S

[jira] [Updated] (HIVE-15239) hive on spark combine equivalentwork get wrong result because of tablescan operation compare

2016-12-01 Thread Rui Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-15239?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rui Li updated HIVE-15239: -- Attachment: (was: HIVE-15239.3.patch) > hive on spark combine equivalentwork get wrong result because of ta

[jira] [Updated] (HIVE-15239) hive on spark combine equivalentwork get wrong result because of tablescan operation compare

2016-12-01 Thread Rui Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-15239?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rui Li updated HIVE-15239: -- Attachment: HIVE-15239.3.patch Try again > hive on spark combine equivalentwork get wrong result because of ta

[jira] [Commented] (HIVE-15302) Relax the requirement that HoS needs Spark built w/o Hive

2016-12-01 Thread Rui Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-15302?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15711818#comment-15711818 ] Rui Li commented on HIVE-15302: --- We only care about yarn-client and yarn-cluster, because sp

[jira] [Commented] (HIVE-15302) Relax the requirement that HoS needs Spark built w/o Hive

2016-11-30 Thread Rui Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-15302?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15711207#comment-15711207 ] Rui Li commented on HIVE-15302: --- [~kellyzly], you're right about the ideas. But the needed s

[jira] [Updated] (HIVE-15239) hive on spark combine equivalentwork get wrong result because of tablescan operation compare

2016-11-30 Thread Rui Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-15239?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rui Li updated HIVE-15239: -- Attachment: HIVE-15239.3.patch [~xuefuz] I see your point. Update patch to address your comment. I also moved t

[jira] [Commented] (HIVE-15313) Add export spark.yarn.archive or spark.yarn.jars variable in Hive on Spark document

2016-11-30 Thread Rui Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-15313?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15710505#comment-15710505 ] Rui Li commented on HIVE-15313: --- Seems these two configs are useful in several ways :) I'm a

[jira] [Commented] (HIVE-15302) Relax the requirement that HoS needs Spark built w/o Hive

2016-11-30 Thread Rui Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-15302?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15710478#comment-15710478 ] Rui Li commented on HIVE-15302: --- We don't only depend on Spark jars, but also the scripts li

[jira] [Commented] (HIVE-15202) Concurrent compactions for the same partition may generate malformed folder structure

2016-11-30 Thread Rui Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-15202?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15710450#comment-15710450 ] Rui Li commented on HIVE-15202: --- I see. Thanks for the explanations Eugene :) > Concurrent

[jira] [Commented] (HIVE-15239) hive on spark combine equivalentwork get wrong result because of tablescan operation compare

2016-11-29 Thread Rui Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-15239?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15707619#comment-15707619 ] Rui Li commented on HIVE-15239: --- Pinging [~xuefuz] > hive on spark combine equivalentwork g

[jira] [Commented] (HIVE-15202) Concurrent compactions for the same partition may generate malformed folder structure

2016-11-29 Thread Rui Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-15202?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15707463#comment-15707463 ] Rui Li commented on HIVE-15202: --- Hi [~ekoifman], I have one question. Suppose we have a comp

[jira] [Commented] (HIVE-15302) Relax the requirement that HoS needs Spark built w/o Hive

2016-11-29 Thread Rui Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-15302?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15707443#comment-15707443 ] Rui Li commented on HIVE-15302: --- Thanks for your suggestions, Marcelo. I'll use spark.yarn.j

[jira] [Commented] (HIVE-15302) Relax the requirement that HoS needs Spark built w/o Hive

2016-11-29 Thread Rui Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-15302?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15707362#comment-15707362 ] Rui Li commented on HIVE-15302: --- To clarify, the method here only works for yarn-cluster mod

[jira] [Commented] (HIVE-15302) Relax the requirement that HoS needs Spark built w/o Hive

2016-11-29 Thread Rui Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-15302?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15707353#comment-15707353 ] Rui Li commented on HIVE-15302: --- Yeah my plan is to put the jars to HDFS. For example, if us

[jira] [Commented] (HIVE-15302) Relax the requirement that HoS needs Spark built w/o Hive

2016-11-29 Thread Rui Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-15302?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15707343#comment-15707343 ] Rui Li commented on HIVE-15302: --- Hi [~vanzin], the potential conflicts introduced by transit

[jira] [Commented] (HIVE-15302) Relax the requirement that HoS needs Spark built w/o Hive

2016-11-28 Thread Rui Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-15302?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15704529#comment-15704529 ] Rui Li commented on HIVE-15302: --- Basically HoS only needs the "core" functionalities of spar

[jira] [Commented] (HIVE-15302) Relax the requirement that HoS needs Spark built w/o Hive

2016-11-28 Thread Rui Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-15302?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15704508#comment-15704508 ] Rui Li commented on HIVE-15302: --- With Spark 2.0, we can use {{spark.yarn.archive}} or {{spar

[jira] [Commented] (HIVE-15239) hive on spark combine equivalentwork get wrong result because of tablescan operation compare

2016-11-27 Thread Rui Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-15239?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15700706#comment-15700706 ] Rui Li commented on HIVE-15239: --- Thanks [~csun] for the explanations. I think these tests ha

[jira] [Commented] (HIVE-13997) Insert overwrite directory doesn't overwrite existing files

2016-11-27 Thread Rui Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-13997?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15700678#comment-15700678 ] Rui Li commented on HIVE-13997: --- [~ajithshetty28], I guess that depends on whether they cher

[jira] [Updated] (HIVE-15168) Flaky test: TestSparkClient.testJobSubmission (still flaky)

2016-11-24 Thread Rui Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-15168?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rui Li updated HIVE-15168: -- Resolution: Fixed Status: Resolved (was: Patch Available) Committed to master. Thanks Barna for the pat

[jira] [Commented] (HIVE-15239) hive on spark combine equivalentwork get wrong result because of tablescan operation compare

2016-11-24 Thread Rui Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-15239?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15694624#comment-15694624 ] Rui Li commented on HIVE-15239: --- Latest failures are not related. And I guess {{spark.only.q

[jira] [Commented] (HIVE-14825) Figure out the minimum set of required jars for Hive on Spark after bumping up to Spark 2.0.0

2016-11-24 Thread Rui Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-14825?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15694590#comment-15694590 ] Rui Li commented on HIVE-14825: --- [~kellyzly] btw I think you're pinging the wrong Rui Li :)

[jira] [Commented] (HIVE-14825) Figure out the minimum set of required jars for Hive on Spark after bumping up to Spark 2.0.0

2016-11-24 Thread Rui Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-14825?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15694587#comment-15694587 ] Rui Li commented on HIVE-14825: --- I don't think so. Figuring out how spark prepares the class

[jira] [Commented] (HIVE-15168) Flaky test: TestSparkClient.testJobSubmission (still flaky)

2016-11-24 Thread Rui Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-15168?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15694580#comment-15694580 ] Rui Li commented on HIVE-15168: --- +1. The latest failures are not related > Flaky test: Test

[jira] [Updated] (HIVE-15239) hive on spark combine equivalentwork get wrong result because of tablescan operation compare

2016-11-24 Thread Rui Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-15239?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rui Li updated HIVE-15239: -- Attachment: HIVE-15239.2.patch Fix NPE and address Xuefu's comments. I also add the example in description as a

[jira] [Commented] (HIVE-15239) hive on spark combine equivalentwork get wrong result because of tablescan operation compare

2016-11-24 Thread Rui Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-15239?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15692721#comment-15692721 ] Rui Li commented on HIVE-15239: --- Thanks [~xuefuz] for the suggestions. 1. Not sure if I'm f

[jira] [Comment Edited] (HIVE-15239) hive on spark combine equivalentwork get wrong result because of tablescan operation compare

2016-11-24 Thread Rui Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-15239?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15690417#comment-15690417 ] Rui Li edited comment on HIVE-15239 at 11/24/16 8:43 AM: - Thanks [

[jira] [Updated] (HIVE-15237) Propagate Spark job failure to Hive

2016-11-24 Thread Rui Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-15237?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rui Li updated HIVE-15237: -- Resolution: Fixed Fix Version/s: 2.2.0 Status: Resolved (was: Patch Available) Committed to m

[jira] [Commented] (HIVE-15168) Flaky test: TestSparkClient.testJobSubmission (still flaky)

2016-11-23 Thread Rui Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-15168?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15691953#comment-15691953 ] Rui Li commented on HIVE-15168: --- [~zsombor.klara], would you mind update the patch to addres

[jira] [Updated] (HIVE-15239) hive on spark combine equivalentwork get wrong result because of tablescan operation compare

2016-11-23 Thread Rui Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-15239?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rui Li updated HIVE-15239: -- Status: Patch Available (was: Open) > hive on spark combine equivalentwork get wrong result because of tablesc

[jira] [Updated] (HIVE-15239) hive on spark combine equivalentwork get wrong result because of tablescan operation compare

2016-11-23 Thread Rui Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-15239?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rui Li updated HIVE-15239: -- Attachment: HIVE-15239.1.patch Thanks [~wenli] for reporting the issue. The problem is we can really tell whethe

[jira] [Commented] (HIVE-15168) Flaky test: TestSparkClient.testJobSubmission (still flaky)

2016-11-23 Thread Rui Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-15168?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15690100#comment-15690100 ] Rui Li commented on HIVE-15168: --- I'd prefer to remove the verification in the test - don't w

[jira] [Assigned] (HIVE-15239) hive on spark combine equivalentwork get wrong result because of tablescan operation compare

2016-11-22 Thread Rui Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-15239?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rui Li reassigned HIVE-15239: - Assignee: Rui Li > hive on spark combine equivalentwork get wrong result because of tablescan > operati

[jira] [Commented] (HIVE-15261) Exception in thread "main" java.lang.IllegalArgumentException: Unrecognized Hadoop major version number: 3.0.0-alpha1

2016-11-22 Thread Rui Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-15261?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15689012#comment-15689012 ] Rui Li commented on HIVE-15261: --- I think currently Hive doesn't run against Hadoop-3.0 > Ex

[jira] [Commented] (HIVE-15259) The deserialization time of HOS20 is longer than what in HOS16

2016-11-22 Thread Rui Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-15259?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15688996#comment-15688996 ] Rui Li commented on HIVE-15259: --- That's interesting. Yeah please go ahead and find out. Than

[jira] [Commented] (HIVE-15237) Propagate Spark job failure to Hive

2016-11-22 Thread Rui Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-15237?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15688985#comment-15688985 ] Rui Li commented on HIVE-15237: --- Thanks [~xuefuz]. The latest failures are not related. I'll

[jira] [Commented] (HIVE-15202) Concurrent compactions for the same partition may generate malformed folder structure

2016-11-22 Thread Rui Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-15202?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15688821#comment-15688821 ] Rui Li commented on HIVE-15202: --- Thanks for the explanations [~ekoifman]! Do you have any ot

<    4   5   6   7   8   9   10   11   12   13   >