[jira] [Commented] (HIVE-15474) Extend limit propagation for chain of RS-GB-RS operators

[ https://issues.apache.org/jira/browse/HIVE-15474?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15786788#comment-15786788 ] Rui Li commented on HIVE-15474: --- [~jcamachorodriguez], thanks very much for the detailed explanations :) For

[jira] [Commented] (HIVE-15519) Hive Decimal Type column scale is returning as zero

[ https://issues.apache.org/jira/browse/HIVE-15519?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15786741#comment-15786741 ] Rui Li commented on HIVE-15519: --- The failures are not related because I can't reproduce them locally.

[jira] [Commented] (HIVE-15519) Hive Decimal Type column scale is returning as zero

[ https://issues.apache.org/jira/browse/HIVE-15519?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15784767#comment-15784767 ] Rui Li commented on HIVE-15519: --- Update patch v2 to add a UT. And also put it in an RB entry. [~thejas], I

[jira] [Updated] (HIVE-15519) Hive Decimal Type column scale is returning as zero

[ https://issues.apache.org/jira/browse/HIVE-15519?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rui Li updated HIVE-15519: -- Attachment: HIVE-15519.2.patch > Hive Decimal Type column scale is returning as zero >

[jira] [Commented] (HIVE-8373) OOM for a simple query with spark.master=local [Spark Branch]

[ https://issues.apache.org/jira/browse/HIVE-8373?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15782500#comment-15782500 ] Rui Li commented on HIVE-8373: -- Thanks [~asears] for the explanations. [~kellyzly], yes please go ahead update

[jira] [Updated] (HIVE-15519) Hive Decimal Type column scale is returning as zero

[ https://issues.apache.org/jira/browse/HIVE-15519?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rui Li updated HIVE-15519: -- Status: Patch Available (was: Open) > Hive Decimal Type column scale is returning as zero >

[jira] [Updated] (HIVE-15519) Hive Decimal Type column scale is returning as zero

[ https://issues.apache.org/jira/browse/HIVE-15519?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rui Li updated HIVE-15519: -- Attachment: HIVE-15519.1.patch > Hive Decimal Type column scale is returning as zero >

[jira] [Commented] (HIVE-15519) Hive Decimal Type column scale is returning as zero

[ https://issues.apache.org/jira/browse/HIVE-15519?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15782474#comment-15782474 ] Rui Li commented on HIVE-15519: --- Thanks [~bharatviswa] for raising the issue. I think we have a bug in

[jira] [Assigned] (HIVE-15519) Hive Decimal Type column scale is returning as zero

[ https://issues.apache.org/jira/browse/HIVE-15519?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rui Li reassigned HIVE-15519: - Assignee: Rui Li > Hive Decimal Type column scale is returning as zero >

[jira] [Updated] (HIVE-15519) Hive Decimal Type column scale is returning as zero

[ https://issues.apache.org/jira/browse/HIVE-15519?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rui Li updated HIVE-15519: -- Description: Hive decimal type column precision is returning as zero, even though column has precision set.

[jira] [Commented] (HIVE-8373) OOM for a simple query with spark.master=local [Spark Branch]

[ https://issues.apache.org/jira/browse/HIVE-8373?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15771778#comment-15771778 ] Rui Li commented on HIVE-8373: -- Thanks [~asears] for the inputs. Just curious, in which case should

[jira] [Updated] (HIVE-15357) Fix and re-enable the spark-only tests

[ https://issues.apache.org/jira/browse/HIVE-15357?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rui Li updated HIVE-15357: -- Resolution: Fixed Fix Version/s: 2.2.0 Status: Resolved (was: Patch Available) Committed to

[jira] [Commented] (HIVE-8373) OOM for a simple query with spark.master=local [Spark Branch]

[ https://issues.apache.org/jira/browse/HIVE-8373?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15769309#comment-15769309 ] Rui Li commented on HIVE-8373: -- Thanks [~kellyzly] for working on this. Have you verified the OOM is related

[jira] [Commented] (HIVE-15474) Extend limit propagation for chain of RS-GB-RS operators

[ https://issues.apache.org/jira/browse/HIVE-15474?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15766929#comment-15766929 ] Rui Li commented on HIVE-15474: --- Hi [~jcamachorodriguez], I think it's an interesting optimization and

[jira] [Commented] (HIVE-15357) Fix and re-enable the spark-only tests

[ https://issues.apache.org/jira/browse/HIVE-15357?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15765903#comment-15765903 ] Rui Li commented on HIVE-15357: --- Failures are not related. [~csun] could you take a look? Thanks! > Fix and

[jira] [Updated] (HIVE-15357) Fix and re-enable the spark-only tests

[ https://issues.apache.org/jira/browse/HIVE-15357?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rui Li updated HIVE-15357: -- Status: Patch Available (was: Open) > Fix and re-enable the spark-only tests >

[jira] [Updated] (HIVE-15357) Fix and re-enable the spark-only tests

[ https://issues.apache.org/jira/browse/HIVE-15357?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rui Li updated HIVE-15357: -- Attachment: HIVE-15357.1.patch All the update to golden files are in the query plan. No result changes. Some

[jira] [Commented] (HIVE-9153) Perf enhancement on CombineHiveInputFormat and HiveInputFormat

[ https://issues.apache.org/jira/browse/HIVE-9153?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15763302#comment-15763302 ] Rui Li commented on HIVE-9153: -- I guess no configuration is suitable for all cases :) If I remember, smaller

[jira] [Updated] (HIVE-15428) HoS DPP doesn't remove cyclic dependency

[ https://issues.apache.org/jira/browse/HIVE-15428?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rui Li updated HIVE-15428: -- Resolution: Fixed Fix Version/s: 2.2.0 Status: Resolved (was: Patch Available) Committed to

[jira] [Commented] (HIVE-13278) Avoid FileNotFoundException when map/reduce.xml is not available

[ https://issues.apache.org/jira/browse/HIVE-13278?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15760001#comment-15760001 ] Rui Li commented on HIVE-13278: --- Thanks [~csun] for the update. The latest patch looks good to me. +1 I also

[jira] [Updated] (HIVE-13293) Cache RDD to improve parallel order by performance for HoS

[ https://issues.apache.org/jira/browse/HIVE-13293?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rui Li updated HIVE-13293: -- Summary: Cache RDD to improve parallel order by performance for HoS (was: Query occurs performance degradation

[jira] [Assigned] (HIVE-15272) "LEFT OUTER JOIN" Is not populating correct records with Hive On Spark

[ https://issues.apache.org/jira/browse/HIVE-15272?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rui Li reassigned HIVE-15272: - Assignee: Rui Li > "LEFT OUTER JOIN" Is not populating correct records with Hive On Spark >

[jira] [Commented] (HIVE-15272) "LEFT OUTER JOIN" Is not populating correct records with Hive On Spark

[ https://issues.apache.org/jira/browse/HIVE-15272?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15753212#comment-15753212 ] Rui Li commented on HIVE-15272: --- OK I'll look into this. [~VPareek], I think the two tables have same DDL

[jira] [Commented] (HIVE-15428) HoS DPP doesn't remove cyclic dependency

[ https://issues.apache.org/jira/browse/HIVE-15428?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15753161#comment-15753161 ] Rui Li commented on HIVE-15428: --- Test failures are not related. > HoS DPP doesn't remove cyclic dependency

[jira] [Comment Edited] (HIVE-13278) Avoid FileNotFoundException when map/reduce.xml is not available

[ https://issues.apache.org/jira/browse/HIVE-13278?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15753143#comment-15753143 ] Rui Li edited comment on HIVE-13278 at 12/16/16 1:57 AM: - Hi [~csun], sorry maybe

[jira] [Commented] (HIVE-13278) Avoid FileNotFoundException when map/reduce.xml is not available

[ https://issues.apache.org/jira/browse/HIVE-13278?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15753143#comment-15753143 ] Rui Li commented on HIVE-13278: --- Hi [~csun], sorry maybe I was being misleading. What I have in mind is

[jira] [Updated] (HIVE-15428) HoS DPP doesn't remove cyclic dependency

[ https://issues.apache.org/jira/browse/HIVE-15428?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rui Li updated HIVE-15428: -- Status: Patch Available (was: Open) > HoS DPP doesn't remove cyclic dependency >

[jira] [Updated] (HIVE-15428) HoS DPP doesn't remove cyclic dependency

[ https://issues.apache.org/jira/browse/HIVE-15428?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rui Li updated HIVE-15428: -- Attachment: HIVE-15428.1.patch Patch to add the detection. Basically it's just copied from Tez. And manually

[jira] [Commented] (HIVE-13278) Avoid FileNotFoundException when map/reduce.xml is not available

[ https://issues.apache.org/jira/browse/HIVE-13278?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15750856#comment-15750856 ] Rui Li commented on HIVE-13278: --- Hi [~csun], I think Tez also calls setMapWork/setReduceWork to associate

[jira] [Commented] (HIVE-15432) java.lang.ClassCastException is thrown when setting "hive.input.format" as "org.apache.hadoop.hive.ql.io.CombineHiveInputFormat" in hive on spark

[ https://issues.apache.org/jira/browse/HIVE-15432?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15750782#comment-15750782 ] Rui Li commented on HIVE-15432: --- [~kellyzly], I think this is just a warning in the log and doesn't cause

[jira] [Commented] (HIVE-13278) Avoid FileNotFoundException when map/reduce.xml is not available

[ https://issues.apache.org/jira/browse/HIVE-13278?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15750696#comment-15750696 ] Rui Li commented on HIVE-13278: --- Hi [~xuefuz], I just think it'll be even simpler to go the checking RS way

[jira] [Commented] (HIVE-13278) Avoid FileNotFoundException when map/reduce.xml is not available

[ https://issues.apache.org/jira/browse/HIVE-13278?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15750452#comment-15750452 ] Rui Li commented on HIVE-13278: --- Sorry about the delay. I have a concern about using flag: it seems

[jira] [Commented] (HIVE-15428) HoS DPP doesn't remove cyclic dependency

[ https://issues.apache.org/jira/browse/HIVE-15428?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15748659#comment-15748659 ] Rui Li commented on HIVE-15428: --- The reason why we didn't see this before is related to how we generate the

[jira] [Commented] (HIVE-13278) Many redundant 'File not found' messages appeared in container log during query execution with Hive on Spark

[ https://issues.apache.org/jira/browse/HIVE-13278?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15747436#comment-15747436 ] Rui Li commented on HIVE-13278: --- [~csun], I think the error is found in container's log. SparkPlanGenerator

[jira] [Updated] (HIVE-15428) HoS DPP doesn't remove cyclic dependency

[ https://issues.apache.org/jira/browse/HIVE-15428?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rui Li updated HIVE-15428: -- Description: More details in HIVE-15357 > HoS DPP doesn't remove cyclic dependency >

[jira] [Commented] (HIVE-15357) Fix and re-enable the spark-only tests

[ https://issues.apache.org/jira/browse/HIVE-15357?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15747039#comment-15747039 ] Rui Li commented on HIVE-15357: --- Hi [~csun], I created HIVE-15428 for it. > Fix and re-enable the

[jira] [Commented] (HIVE-13278) Many redundant 'File not found' messages appeared in container log during query execution with Hive on Spark

[ https://issues.apache.org/jira/browse/HIVE-13278?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15747031#comment-15747031 ] Rui Li commented on HIVE-13278: --- [~csun], thanks for the patch. I think Xuefu's concerns are valid. In

[jira] [Commented] (HIVE-15357) Fix and re-enable the spark-only tests

[ https://issues.apache.org/jira/browse/HIVE-15357?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15745110#comment-15745110 ] Rui Li commented on HIVE-15357: --- When I tried to re-enable these tests, I found this issue with DPP. Running

[jira] [Updated] (HIVE-15386) Expose Spark task counts and stage Ids information in SparkTask from SparkJobMonitor

[ https://issues.apache.org/jira/browse/HIVE-15386?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rui Li updated HIVE-15386: -- Affects Version/s: (was: 2.2.0) 2.1.1 > Expose Spark task counts and stage Ids

[jira] [Updated] (HIVE-15386) Expose Spark task counts and stage Ids information in SparkTask from SparkJobMonitor

[ https://issues.apache.org/jira/browse/HIVE-15386?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rui Li updated HIVE-15386: -- Resolution: Fixed Fix Version/s: 2.2.0 Status: Resolved (was: Patch Available) Committed to

[jira] [Commented] (HIVE-13278) Many redundant 'File not found' messages appeared in container log during query execution with Hive on Spark

[ https://issues.apache.org/jira/browse/HIVE-13278?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15744366#comment-15744366 ] Rui Li commented on HIVE-13278: --- Hi [~xuefuz], the conclusion is we somehow try to read reduce.xml for

[jira] [Commented] (HIVE-15386) Expose Spark task counts and stage Ids information in SparkTask from SparkJobMonitor

[ https://issues.apache.org/jira/browse/HIVE-15386?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15743862#comment-15743862 ] Rui Li commented on HIVE-15386: --- Thanks for the update [~zxu]. +1. [~xuefuz] do you have any further

[jira] [Commented] (HIVE-15386) Expose Spark task counts and stage Ids information in SparkTask from SparkJobMonitor

[ https://issues.apache.org/jira/browse/HIVE-15386?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15741232#comment-15741232 ] Rui Li commented on HIVE-15386: --- Thanks [~zxu] for the explanations. Then I think submitTime can also be

[jira] [Comment Edited] (HIVE-15386) Expose Spark task counts and stage Ids information in SparkTask from SparkJobMonitor

[ https://issues.apache.org/jira/browse/HIVE-15386?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15740845#comment-15740845 ] Rui Li edited comment on HIVE-15386 at 12/12/16 3:23 AM: - Hi [~zxu], the v1 patch

[jira] [Commented] (HIVE-15386) Expose Spark task counts and stage Ids information in SparkTask from SparkJobMonitor

[ https://issues.apache.org/jira/browse/HIVE-15386?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15740845#comment-15740845 ] Rui Li commented on HIVE-15386: --- Hi [~zxu], the v2 patch looks better. One question is what's the difference

[jira] [Commented] (HIVE-9927) MR doesn't produce correct result for runtime_skewjoin_mapjoin_spark

[ https://issues.apache.org/jira/browse/HIVE-9927?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15734982#comment-15734982 ] Rui Li commented on HIVE-9927: -- [~wenli], thanks for looking into this issue. Do you want to work on it? > MR

[jira] [Commented] (HIVE-15386) Expose Spark task counts and stage Ids information in SparkTask from SparkJobMonitor

[ https://issues.apache.org/jira/browse/HIVE-15386?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15732341#comment-15732341 ] Rui Li commented on HIVE-15386: --- [~zxu], could you elaborate on how these task infos will be used? I don't

[jira] [Updated] (HIVE-15357) Fix and re-enable the spark-only tests

[ https://issues.apache.org/jira/browse/HIVE-15357?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rui Li updated HIVE-15357: -- Description: Defined by {{spark.only.query.files}}. > Fix and re-enable the spark-only tests >

[jira] [Commented] (HIVE-15239) hive on spark combine equivalent work get wrong result because of TS operation compare

[ https://issues.apache.org/jira/browse/HIVE-15239?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15722020#comment-15722020 ] Rui Li commented on HIVE-15239: --- Also filed HIVE-15357 for the tests. > hive on spark combine equivalent

[jira] [Updated] (HIVE-15239) hive on spark combine equivalent work get wrong result because of TS operation compare

[ https://issues.apache.org/jira/browse/HIVE-15239?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rui Li updated HIVE-15239: -- Resolution: Fixed Fix Version/s: 2.2.0 Status: Resolved (was: Patch Available) Latest

[jira] [Updated] (HIVE-15239) hive on spark combine equivalent work get wrong result because of TS operation compare

[ https://issues.apache.org/jira/browse/HIVE-15239?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rui Li updated HIVE-15239: -- Summary: hive on spark combine equivalent work get wrong result because of TS operation compare (was: hive on

[jira] [Updated] (HIVE-15239) hive on spark combine equivalentwork get wrong result because of tablescan operation compare

[ https://issues.apache.org/jira/browse/HIVE-15239?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rui Li updated HIVE-15239: -- Attachment: HIVE-15239.4.patch > hive on spark combine equivalentwork get wrong result because of tablescan >

[jira] [Updated] (HIVE-15239) hive on spark combine equivalentwork get wrong result because of tablescan operation compare

[ https://issues.apache.org/jira/browse/HIVE-15239?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rui Li updated HIVE-15239: -- Attachment: (was: HIVE-15239.4.patch) > hive on spark combine equivalentwork get wrong result because of

[jira] [Updated] (HIVE-15239) hive on spark combine equivalentwork get wrong result because of tablescan operation compare

[ https://issues.apache.org/jira/browse/HIVE-15239?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rui Li updated HIVE-15239: -- Attachment: HIVE-15239.4.patch > hive on spark combine equivalentwork get wrong result because of tablescan >

[jira] [Updated] (HIVE-15239) hive on spark combine equivalentwork get wrong result because of tablescan operation compare

[ https://issues.apache.org/jira/browse/HIVE-15239?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rui Li updated HIVE-15239: -- Attachment: (was: HIVE-15239.4.patch) > hive on spark combine equivalentwork get wrong result because of

[jira] [Updated] (HIVE-15239) hive on spark combine equivalentwork get wrong result because of tablescan operation compare

[ https://issues.apache.org/jira/browse/HIVE-15239?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rui Li updated HIVE-15239: -- Attachment: HIVE-15239.4.patch Thanks [~xuefuz] for the review. Update patch v4 to address your comment. >

[jira] [Commented] (HIVE-15302) Relax the requirement that HoS needs Spark built w/o Hive

[ https://issues.apache.org/jira/browse/HIVE-15302?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15713754#comment-15713754 ] Rui Li commented on HIVE-15302: --- [~kellyzly], if we use the config to make HoS run against Spark built with

[jira] [Updated] (HIVE-15239) hive on spark combine equivalentwork get wrong result because of tablescan operation compare

[ https://issues.apache.org/jira/browse/HIVE-15239?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rui Li updated HIVE-15239: -- Attachment: (was: HIVE-15239.3.patch) > hive on spark combine equivalentwork get wrong result because of

[jira] [Updated] (HIVE-15239) hive on spark combine equivalentwork get wrong result because of tablescan operation compare

[ https://issues.apache.org/jira/browse/HIVE-15239?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rui Li updated HIVE-15239: -- Attachment: HIVE-15239.3.patch Try again > hive on spark combine equivalentwork get wrong result because of

[jira] [Commented] (HIVE-15302) Relax the requirement that HoS needs Spark built w/o Hive

[ https://issues.apache.org/jira/browse/HIVE-15302?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15711818#comment-15711818 ] Rui Li commented on HIVE-15302: --- We only care about yarn-client and yarn-cluster, because spark.yarn.archive

[jira] [Commented] (HIVE-15302) Relax the requirement that HoS needs Spark built w/o Hive

[ https://issues.apache.org/jira/browse/HIVE-15302?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15711207#comment-15711207 ] Rui Li commented on HIVE-15302: --- [~kellyzly], you're right about the ideas. But the needed spark jars may

[jira] [Updated] (HIVE-15239) hive on spark combine equivalentwork get wrong result because of tablescan operation compare

[ https://issues.apache.org/jira/browse/HIVE-15239?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rui Li updated HIVE-15239: -- Attachment: HIVE-15239.3.patch [~xuefuz] I see your point. Update patch to address your comment. I also moved

[jira] [Commented] (HIVE-15313) Add export spark.yarn.archive or spark.yarn.jars variable in Hive on Spark document

[ https://issues.apache.org/jira/browse/HIVE-15313?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15710505#comment-15710505 ] Rui Li commented on HIVE-15313: --- Seems these two configs are useful in several ways :) I'm also looking at

[jira] [Commented] (HIVE-15302) Relax the requirement that HoS needs Spark built w/o Hive

[ https://issues.apache.org/jira/browse/HIVE-15302?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15710478#comment-15710478 ] Rui Li commented on HIVE-15302: --- We don't only depend on Spark jars, but also the scripts like spark-submit.

[jira] [Commented] (HIVE-15202) Concurrent compactions for the same partition may generate malformed folder structure

[ https://issues.apache.org/jira/browse/HIVE-15202?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15710450#comment-15710450 ] Rui Li commented on HIVE-15202: --- I see. Thanks for the explanations Eugene :) > Concurrent compactions for

[jira] [Commented] (HIVE-15239) hive on spark combine equivalentwork get wrong result because of tablescan operation compare

[ https://issues.apache.org/jira/browse/HIVE-15239?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15707619#comment-15707619 ] Rui Li commented on HIVE-15239: --- Pinging [~xuefuz] > hive on spark combine equivalentwork get wrong result

[jira] [Commented] (HIVE-15202) Concurrent compactions for the same partition may generate malformed folder structure

[ https://issues.apache.org/jira/browse/HIVE-15202?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15707463#comment-15707463 ] Rui Li commented on HIVE-15202: --- Hi [~ekoifman], I have one question. Suppose we have a compaction in

[jira] [Commented] (HIVE-15302) Relax the requirement that HoS needs Spark built w/o Hive

[ https://issues.apache.org/jira/browse/HIVE-15302?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15707443#comment-15707443 ] Rui Li commented on HIVE-15302: --- Thanks for your suggestions, Marcelo. I'll use spark.yarn.jars instead.

[jira] [Commented] (HIVE-15302) Relax the requirement that HoS needs Spark built w/o Hive

[ https://issues.apache.org/jira/browse/HIVE-15302?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15707362#comment-15707362 ] Rui Li commented on HIVE-15302: --- To clarify, the method here only works for yarn-cluster mode. For

[jira] [Commented] (HIVE-15302) Relax the requirement that HoS needs Spark built w/o Hive

[ https://issues.apache.org/jira/browse/HIVE-15302?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15707353#comment-15707353 ] Rui Li commented on HIVE-15302: --- Yeah my plan is to put the jars to HDFS. For example, if user doesn't

[jira] [Commented] (HIVE-15302) Relax the requirement that HoS needs Spark built w/o Hive

[ https://issues.apache.org/jira/browse/HIVE-15302?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15707343#comment-15707343 ] Rui Li commented on HIVE-15302: --- Hi [~vanzin], the potential conflicts introduced by transitive dep have

[jira] [Commented] (HIVE-15302) Relax the requirement that HoS needs Spark built w/o Hive

[ https://issues.apache.org/jira/browse/HIVE-15302?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15704529#comment-15704529 ] Rui Li commented on HIVE-15302: --- Basically HoS only needs the "core" functionalities of spark, so I guess we

[jira] [Commented] (HIVE-15302) Relax the requirement that HoS needs Spark built w/o Hive

[ https://issues.apache.org/jira/browse/HIVE-15302?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15704508#comment-15704508 ] Rui Li commented on HIVE-15302: --- With Spark 2.0, we can use {{spark.yarn.archive}} or {{spark.yarn.jars}} to

[jira] [Commented] (HIVE-15239) hive on spark combine equivalentwork get wrong result because of tablescan operation compare

[ https://issues.apache.org/jira/browse/HIVE-15239?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15700706#comment-15700706 ] Rui Li commented on HIVE-15239: --- Thanks [~csun] for the explanations. I think these tests have been ignored

[jira] [Commented] (HIVE-13997) Insert overwrite directory doesn't overwrite existing files

[ https://issues.apache.org/jira/browse/HIVE-13997?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15700678#comment-15700678 ] Rui Li commented on HIVE-13997: --- [~ajithshetty28], I guess that depends on whether they cherry picked the

[jira] [Updated] (HIVE-15168) Flaky test: TestSparkClient.testJobSubmission (still flaky)

[ https://issues.apache.org/jira/browse/HIVE-15168?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rui Li updated HIVE-15168: -- Resolution: Fixed Status: Resolved (was: Patch Available) Committed to master. Thanks Barna for the

[jira] [Commented] (HIVE-15239) hive on spark combine equivalentwork get wrong result because of tablescan operation compare

[ https://issues.apache.org/jira/browse/HIVE-15239?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15694624#comment-15694624 ] Rui Li commented on HIVE-15239: --- Latest failures are not related. And I guess {{spark.only.query.files}} are

[jira] [Commented] (HIVE-14825) Figure out the minimum set of required jars for Hive on Spark after bumping up to Spark 2.0.0

[ https://issues.apache.org/jira/browse/HIVE-14825?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15694590#comment-15694590 ] Rui Li commented on HIVE-14825: --- [~kellyzly] btw I think you're pinging the wrong Rui Li :) > Figure out

[jira] [Commented] (HIVE-14825) Figure out the minimum set of required jars for Hive on Spark after bumping up to Spark 2.0.0

[ https://issues.apache.org/jira/browse/HIVE-14825?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15694587#comment-15694587 ] Rui Li commented on HIVE-14825: --- I don't think so. Figuring out how spark prepares the classpath for

[jira] [Commented] (HIVE-15168) Flaky test: TestSparkClient.testJobSubmission (still flaky)

[ https://issues.apache.org/jira/browse/HIVE-15168?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15694580#comment-15694580 ] Rui Li commented on HIVE-15168: --- +1. The latest failures are not related > Flaky test:

[jira] [Updated] (HIVE-15239) hive on spark combine equivalentwork get wrong result because of tablescan operation compare

[ https://issues.apache.org/jira/browse/HIVE-15239?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rui Li updated HIVE-15239: -- Attachment: HIVE-15239.2.patch Fix NPE and address Xuefu's comments. I also add the example in description as a

[jira] [Commented] (HIVE-15239) hive on spark combine equivalentwork get wrong result because of tablescan operation compare

[ https://issues.apache.org/jira/browse/HIVE-15239?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15692721#comment-15692721 ] Rui Li commented on HIVE-15239: --- Thanks [~xuefuz] for the suggestions. 1. Not sure if I'm following your

[jira] [Comment Edited] (HIVE-15239) hive on spark combine equivalentwork get wrong result because of tablescan operation compare

[ https://issues.apache.org/jira/browse/HIVE-15239?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15690417#comment-15690417 ] Rui Li edited comment on HIVE-15239 at 11/24/16 8:43 AM: - Thanks [~wenli] for

[jira] [Updated] (HIVE-15237) Propagate Spark job failure to Hive

[ https://issues.apache.org/jira/browse/HIVE-15237?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rui Li updated HIVE-15237: -- Resolution: Fixed Fix Version/s: 2.2.0 Status: Resolved (was: Patch Available) Committed to

[jira] [Commented] (HIVE-15168) Flaky test: TestSparkClient.testJobSubmission (still flaky)

[ https://issues.apache.org/jira/browse/HIVE-15168?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15691953#comment-15691953 ] Rui Li commented on HIVE-15168: --- [~zsombor.klara], would you mind update the patch to address Xuefu's

[jira] [Updated] (HIVE-15239) hive on spark combine equivalentwork get wrong result because of tablescan operation compare

[ https://issues.apache.org/jira/browse/HIVE-15239?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rui Li updated HIVE-15239: -- Status: Patch Available (was: Open) > hive on spark combine equivalentwork get wrong result because of

[jira] [Updated] (HIVE-15239) hive on spark combine equivalentwork get wrong result because of tablescan operation compare

[ https://issues.apache.org/jira/browse/HIVE-15239?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rui Li updated HIVE-15239: -- Attachment: HIVE-15239.1.patch Thanks [~wenli] for reporting the issue. The problem is we can really tell

[jira] [Commented] (HIVE-15168) Flaky test: TestSparkClient.testJobSubmission (still flaky)

[ https://issues.apache.org/jira/browse/HIVE-15168?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15690100#comment-15690100 ] Rui Li commented on HIVE-15168: --- I'd prefer to remove the verification in the test - don't want to add extra

[jira] [Assigned] (HIVE-15239) hive on spark combine equivalentwork get wrong result because of tablescan operation compare

[ https://issues.apache.org/jira/browse/HIVE-15239?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rui Li reassigned HIVE-15239: - Assignee: Rui Li > hive on spark combine equivalentwork get wrong result because of tablescan >

[jira] [Commented] (HIVE-15261) Exception in thread "main" java.lang.IllegalArgumentException: Unrecognized Hadoop major version number: 3.0.0-alpha1

[ https://issues.apache.org/jira/browse/HIVE-15261?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15689012#comment-15689012 ] Rui Li commented on HIVE-15261: --- I think currently Hive doesn't run against Hadoop-3.0 > Exception in

[jira] [Commented] (HIVE-15259) The deserialization time of HOS20 is longer than what in HOS16

[ https://issues.apache.org/jira/browse/HIVE-15259?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15688996#comment-15688996 ] Rui Li commented on HIVE-15259: --- That's interesting. Yeah please go ahead and find out. Thanks. > The

[jira] [Commented] (HIVE-15237) Propagate Spark job failure to Hive

[ https://issues.apache.org/jira/browse/HIVE-15237?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15688985#comment-15688985 ] Rui Li commented on HIVE-15237: --- Thanks [~xuefuz]. The latest failures are not related. I'll commit this if

[jira] [Commented] (HIVE-15202) Concurrent compactions for the same partition may generate malformed folder structure

[ https://issues.apache.org/jira/browse/HIVE-15202?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15688821#comment-15688821 ] Rui Li commented on HIVE-15202: --- Thanks for the explanations [~ekoifman]! Do you have any other solutions in

[jira] [Updated] (HIVE-15237) Propagate Spark job failure to Hive

[ https://issues.apache.org/jira/browse/HIVE-15237?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rui Li updated HIVE-15237: -- Attachment: HIVE-15237.2.patch Cannot reproduce the failures. Try again. > Propagate Spark job failure to Hive

[jira] [Commented] (HIVE-15168) Flaky test: TestSparkClient.testJobSubmission (still flaky)

[ https://issues.apache.org/jira/browse/HIVE-15168?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15688740#comment-15688740 ] Rui Li commented on HIVE-15168: --- [~zsombor.klara], thanks for the investigation. I also tried adding some

[jira] [Commented] (HIVE-14825) Figure out the minimum set of required jars for Hive on Spark after bumping up to Spark 2.0.0

[ https://issues.apache.org/jira/browse/HIVE-14825?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15688618#comment-15688618 ] Rui Li commented on HIVE-14825: --- Hi [~kellyzly], it's in the [Configuring

[jira] [Commented] (HIVE-15168) Flaky test: TestSparkClient.testJobSubmission (still flaky)

[ https://issues.apache.org/jira/browse/HIVE-15168?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15686633#comment-15686633 ] Rui Li commented on HIVE-15168: --- [~zsombor.klara], not sure if I correctly understand your explanation about

[jira] [Commented] (HIVE-15259) The deserialization time of HOS20 is longer than what in HOS16

[ https://issues.apache.org/jira/browse/HIVE-15259?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15686282#comment-15686282 ] Rui Li commented on HIVE-15259: --- With Spark 2.0, you don't have to copy all the jars to Hive lib. Please

[jira] [Updated] (HIVE-15237) Propagate Spark job failure to Hive

[ https://issues.apache.org/jira/browse/HIVE-15237?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rui Li updated HIVE-15237: -- Attachment: HIVE-15237.2.patch Thanks [~xuefuz] for the patch. I made some modifications based on your work.

[jira] [Updated] (HIVE-15237) Propagate Spark job failure to Hive

[ https://issues.apache.org/jira/browse/HIVE-15237?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rui Li updated HIVE-15237: -- Status: Patch Available (was: Open) > Propagate Spark job failure to Hive >

<    4   5   6   7   8   9   10   11   12   13   >