[jira] [Updated] (SPARK-44379) Broadcast Joins taking up too much memory

[ https://issues.apache.org/jira/browse/SPARK-44379?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shardul Mahadik updated SPARK-44379: Description: Context: After migrating to Spark 3 with AQE, we saw a significant increase

[jira] [Commented] (SPARK-44379) Broadcast Joins taking up too much memory

[ https://issues.apache.org/jira/browse/SPARK-44379?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17742126#comment-17742126 ] Shardul Mahadik commented on SPARK-44379: - cc: [~cloud_fan] [~joshrosen] [~mridul] Would be

[jira] [Updated] (SPARK-44379) Broadcast Joins taking up too much memory

[ https://issues.apache.org/jira/browse/SPARK-44379?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shardul Mahadik updated SPARK-44379: Description: Context: After migrating to Spark 3 with AQE, we saw a significant increase

[jira] [Updated] (SPARK-44379) Broadcast Joins taking up too much memory

[ https://issues.apache.org/jira/browse/SPARK-44379?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shardul Mahadik updated SPARK-44379: Attachment: screenshot-1.png > Broadcast Joins taking up too much memory >

[jira] [Updated] (SPARK-44379) Broadcast Joins taking up too much memory

[ https://issues.apache.org/jira/browse/SPARK-44379?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shardul Mahadik updated SPARK-44379: Attachment: screenshot-2.png > Broadcast Joins taking up too much memory >

[jira] [Created] (SPARK-44379) Broadcast Joins taking up too much memory

Shardul Mahadik created SPARK-44379: --- Summary: Broadcast Joins taking up too much memory Key: SPARK-44379 URL: https://issues.apache.org/jira/browse/SPARK-44379 Project: Spark Issue Type:

[jira] [Created] (SPARK-42290) Spark Driver hangs on OOM during Broadcast when AQE is enabled

Shardul Mahadik created SPARK-42290: --- Summary: Spark Driver hangs on OOM during Broadcast when AQE is enabled Key: SPARK-42290 URL: https://issues.apache.org/jira/browse/SPARK-42290 Project: Spark

[jira] [Commented] (SPARK-41557) Union of tables with and without metadata column fails when used in join

[ https://issues.apache.org/jira/browse/SPARK-41557?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17679004#comment-17679004 ] Shardul Mahadik commented on SPARK-41557: - Confirmed that the test work fine in master now.

[jira] [Resolved] (SPARK-41557) Union of tables with and without metadata column fails when used in join

[ https://issues.apache.org/jira/browse/SPARK-41557?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shardul Mahadik resolved SPARK-41557. - Resolution: Fixed > Union of tables with and without metadata column fails when used in

[jira] [Commented] (SPARK-41557) Union of tables with and without metadata column fails when used in join

[ https://issues.apache.org/jira/browse/SPARK-41557?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17648866#comment-17648866 ] Shardul Mahadik commented on SPARK-41557: - cc: [~Gengliang.Wang] [~cloud_fan] > Union of

[jira] [Updated] (SPARK-41557) Union of tables with and without metadata column fails when used in join

[ https://issues.apache.org/jira/browse/SPARK-41557?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shardul Mahadik updated SPARK-41557: Description: Here is a test case that can be added to {{MetadataColumnSuite}} to

[jira] [Updated] (SPARK-41557) Union of tables with and without metadata column fails when used in join

[ https://issues.apache.org/jira/browse/SPARK-41557?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shardul Mahadik updated SPARK-41557: Description: Here is a test case that can be added to {{MetadataColumnSuite}} to

[jira] [Created] (SPARK-41557) Union of tables with and without metadata column fails when used in join

Shardul Mahadik created SPARK-41557: --- Summary: Union of tables with and without metadata column fails when used in join Key: SPARK-41557 URL: https://issues.apache.org/jira/browse/SPARK-41557

[jira] [Updated] (SPARK-41162) Anti-join must not be pushed below aggregation with ambiguous predicates

[ https://issues.apache.org/jira/browse/SPARK-41162?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shardul Mahadik updated SPARK-41162: Labels: correctness (was: ) > Anti-join must not be pushed below aggregation with

[jira] [Comment Edited] (SPARK-41162) Anti-join must not be pushed below aggregation with ambiguous predicates

[ https://issues.apache.org/jira/browse/SPARK-41162?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17648751#comment-17648751 ] Shardul Mahadik edited comment on SPARK-41162 at 12/16/22 6:27 PM: ---

[jira] [Commented] (SPARK-41162) Anti-join must not be pushed below aggregation with ambiguous predicates

[ https://issues.apache.org/jira/browse/SPARK-41162?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17648751#comment-17648751 ] Shardul Mahadik commented on SPARK-41162: - [~cloud_fan] Can you help take a look at this? This

[jira] [Commented] (SPARK-40262) Expensive UDF evaluation pushed down past a join leads to performance issues

[ https://issues.apache.org/jira/browse/SPARK-40262?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17601021#comment-17601021 ] Shardul Mahadik commented on SPARK-40262: - [~cloud_fan] [~viirya] [~joshrosen] Gentle ping on

[jira] [Commented] (SPARK-40262) Expensive UDF evaluation pushed down past a join leads to performance issues

[ https://issues.apache.org/jira/browse/SPARK-40262?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17597436#comment-17597436 ] Shardul Mahadik commented on SPARK-40262: - cc: [~cloud_fan] [~xkrogen] [~mridulm80]  >

[jira] [Created] (SPARK-40262) Expensive UDF evaluation pushed down past a join leads to performance issues

Shardul Mahadik created SPARK-40262: --- Summary: Expensive UDF evaluation pushed down past a join leads to performance issues Key: SPARK-40262 URL: https://issues.apache.org/jira/browse/SPARK-40262

[jira] [Comment Edited] (SPARK-35253) Upgrade Janino from 3.0.16 to 3.1.4

[ https://issues.apache.org/jira/browse/SPARK-35253?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17524638#comment-17524638 ] Shardul Mahadik edited comment on SPARK-35253 at 4/20/22 12:31 AM: --- Hi

[jira] [Commented] (SPARK-35253) Upgrade Janino from 3.0.16 to 3.1.4

[ https://issues.apache.org/jira/browse/SPARK-35253?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17524638#comment-17524638 ] Shardul Mahadik commented on SPARK-35253: - Hi folks. This issue found in SPARK-35578 is now

[jira] [Commented] (SPARK-35253) Upgrade Janino from 3.0.16 to 3.1.4

[ https://issues.apache.org/jira/browse/SPARK-35253?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17513801#comment-17513801 ] Shardul Mahadik commented on SPARK-35253: - Hi folks, what is the path forward for this ticket?

[jira] [Created] (SPARK-38510) Failure fetching JSON representation of Spark plans with Hive UDFs

Shardul Mahadik created SPARK-38510: --- Summary: Failure fetching JSON representation of Spark plans with Hive UDFs Key: SPARK-38510 URL: https://issues.apache.org/jira/browse/SPARK-38510 Project:

[jira] [Commented] (SPARK-38030) Query with cast containing non-nullable columns fails with AQE on Spark 3.1.1

[ https://issues.apache.org/jira/browse/SPARK-38030?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17489016#comment-17489016 ] Shardul Mahadik commented on SPARK-38030: - During the PR reviews, we used a different approach

[jira] [Commented] (SPARK-38030) Query with cast containing non-nullable columns fails with AQE on Spark 3.1.1

[ https://issues.apache.org/jira/browse/SPARK-38030?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17482233#comment-17482233 ] Shardul Mahadik commented on SPARK-38030: - I plan to create a PR to change the canonicalization

[jira] [Created] (SPARK-38030) Query with cast containing non-nullable columns fails with AQE on Spark 3.1.1

Shardul Mahadik created SPARK-38030: --- Summary: Query with cast containing non-nullable columns fails with AQE on Spark 3.1.1 Key: SPARK-38030 URL: https://issues.apache.org/jira/browse/SPARK-38030

[jira] [Created] (SPARK-37822) SQL function `split` should return an array of non-nullable elements

Shardul Mahadik created SPARK-37822: --- Summary: SQL function `split` should return an array of non-nullable elements Key: SPARK-37822 URL: https://issues.apache.org/jira/browse/SPARK-37822 Project:

[jira] [Created] (SPARK-37602) Add config property to set default Spark listeners

Shardul Mahadik created SPARK-37602: --- Summary: Add config property to set default Spark listeners Key: SPARK-37602 URL: https://issues.apache.org/jira/browse/SPARK-37602 Project: Spark

[jira] [Created] (SPARK-37569) View Analysis incorrectly marks nested fields as nullable

Shardul Mahadik created SPARK-37569: --- Summary: View Analysis incorrectly marks nested fields as nullable Key: SPARK-37569 URL: https://issues.apache.org/jira/browse/SPARK-37569 Project: Spark

[jira] [Commented] (SPARK-36877) Calling ds.rdd with AQE enabled leads to jobs being run, eventually causing reruns

[ https://issues.apache.org/jira/browse/SPARK-36877?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17434515#comment-17434515 ] Shardul Mahadik commented on SPARK-36877: - Was able to get around this by re-using the RDD for

[jira] [Resolved] (SPARK-36877) Calling ds.rdd with AQE enabled leads to jobs being run, eventually causing reruns

[ https://issues.apache.org/jira/browse/SPARK-36877?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shardul Mahadik resolved SPARK-36877. - Resolution: Not A Problem > Calling ds.rdd with AQE enabled leads to jobs being run,

[jira] [Commented] (SPARK-36877) Calling ds.rdd with AQE enabled leads to jobs being run, eventually causing reruns

[ https://issues.apache.org/jira/browse/SPARK-36877?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17427825#comment-17427825 ] Shardul Mahadik commented on SPARK-36877: - {quote} Getting RDD means the physical plan is

[jira] [Commented] (SPARK-36905) Reading Hive view without explicit column names fails in Spark

[ https://issues.apache.org/jira/browse/SPARK-36905?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17422966#comment-17422966 ] Shardul Mahadik commented on SPARK-36905: - This cannot be reproduced with a view created from

[jira] [Comment Edited] (SPARK-36905) Reading Hive view without explicit column names fails in Spark

[ https://issues.apache.org/jira/browse/SPARK-36905?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17422958#comment-17422958 ] Shardul Mahadik edited comment on SPARK-36905 at 9/30/21, 6:21 PM: ---

[jira] [Commented] (SPARK-36905) Reading Hive view without explicit column names fails in Spark

[ https://issues.apache.org/jira/browse/SPARK-36905?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17422958#comment-17422958 ] Shardul Mahadik commented on SPARK-36905: - This worked fine prior to

[jira] [Updated] (SPARK-36905) Reading Hive view without explicit column names fails in Spark

[ https://issues.apache.org/jira/browse/SPARK-36905?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shardul Mahadik updated SPARK-36905: Description: Consider a Hive view in which some columns are not explicitly named

[jira] [Updated] (SPARK-36905) Reading Hive view without explicit column names fails in Spark

[ https://issues.apache.org/jira/browse/SPARK-36905?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shardul Mahadik updated SPARK-36905: Description: Consider a Hive view in which some columns are not explicitly named

[jira] [Created] (SPARK-36905) Reading Hive view without explicit column names fails in Spark

Shardul Mahadik created SPARK-36905: --- Summary: Reading Hive view without explicit column names fails in Spark Key: SPARK-36905 URL: https://issues.apache.org/jira/browse/SPARK-36905 Project: Spark

[jira] [Commented] (SPARK-35874) AQE Shuffle should wait for its subqueries to finish before materializing

[ https://issues.apache.org/jira/browse/SPARK-35874?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17421780#comment-17421780 ] Shardul Mahadik commented on SPARK-35874: - [~dongjoon] Should this be linked in SPARK-33828? >

[jira] [Updated] (SPARK-36877) Calling ds.rdd with AQE enabled leads to jobs being run, eventually causing reruns

[ https://issues.apache.org/jira/browse/SPARK-36877?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shardul Mahadik updated SPARK-36877: Summary: Calling ds.rdd with AQE enabled leads to jobs being run, eventually causing

[jira] [Commented] (SPARK-36877) Calling ds.rdd with AQE enabled leads to being jobs being run, eventually causing reruns

[ https://issues.apache.org/jira/browse/SPARK-36877?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17421510#comment-17421510 ] Shardul Mahadik commented on SPARK-36877: - cc: [~cloud_fan] [~mridulm80] > Calling ds.rdd with

[jira] [Created] (SPARK-36877) Calling ds.rdd with AQE enabled leads to being jobs being run, eventually causing reruns

Shardul Mahadik created SPARK-36877: --- Summary: Calling ds.rdd with AQE enabled leads to being jobs being run, eventually causing reruns Key: SPARK-36877 URL: https://issues.apache.org/jira/browse/SPARK-36877

[jira] [Updated] (SPARK-36877) Calling ds.rdd with AQE enabled leads to being jobs being run, eventually causing reruns

[ https://issues.apache.org/jira/browse/SPARK-36877?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shardul Mahadik updated SPARK-36877: Attachment: Screen Shot 2021-09-28 at 09.32.20.png > Calling ds.rdd with AQE enabled

[jira] [Commented] (SPARK-36673) Incorrect Unions of struct with mismatched field name case

[ https://issues.apache.org/jira/browse/SPARK-36673?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17410477#comment-17410477 ] Shardul Mahadik commented on SPARK-36673: - [~mgaido] [~cloud_fan] Since you guys were involved

[jira] [Updated] (SPARK-36673) Incorrect Unions of struct with mismatched field name case

[ https://issues.apache.org/jira/browse/SPARK-36673?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shardul Mahadik updated SPARK-36673: Description: If a nested field has different casing on two sides of the union, the

[jira] [Updated] (SPARK-36673) Incorrect Unions of struct with mismatched field name case

[ https://issues.apache.org/jira/browse/SPARK-36673?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shardul Mahadik updated SPARK-36673: Description: If a nested field has different casing on two sides of the union, the

[jira] [Created] (SPARK-36673) Incorrect Unions of struct with mismatched field name case

Shardul Mahadik created SPARK-36673: --- Summary: Incorrect Unions of struct with mismatched field name case Key: SPARK-36673 URL: https://issues.apache.org/jira/browse/SPARK-36673 Project: Spark

[jira] [Created] (SPARK-36215) Add logging for slow fetches to diagnose external shuffle service issues

Shardul Mahadik created SPARK-36215: --- Summary: Add logging for slow fetches to diagnose external shuffle service issues Key: SPARK-36215 URL: https://issues.apache.org/jira/browse/SPARK-36215

[jira] [Comment Edited] (SPARK-28266) data duplication when `path` serde property is present

[ https://issues.apache.org/jira/browse/SPARK-28266?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17380170#comment-17380170 ] Shardul Mahadik edited comment on SPARK-28266 at 7/13/21, 9:27 PM: --- I

[jira] [Comment Edited] (SPARK-28266) data duplication when `path` serde property is present

[ https://issues.apache.org/jira/browse/SPARK-28266?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17380170#comment-17380170 ] Shardul Mahadik edited comment on SPARK-28266 at 7/13/21, 9:27 PM: --- I

[jira] [Commented] (SPARK-28266) data duplication when `path` serde property is present

[ https://issues.apache.org/jira/browse/SPARK-28266?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17380170#comment-17380170 ] Shardul Mahadik commented on SPARK-28266: - I would like to propose another angle to look at the

[jira] [Created] (SPARK-35074) spark.jars.xxx configs should be moved to config/package.scala

Shardul Mahadik created SPARK-35074: --- Summary: spark.jars.xxx configs should be moved to config/package.scala Key: SPARK-35074 URL: https://issues.apache.org/jira/browse/SPARK-35074 Project: Spark

[jira] [Updated] (SPARK-35072) spark.jars.ivysettings should support local:// and hdfs:// schemes

[ https://issues.apache.org/jira/browse/SPARK-35072?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shardul Mahadik updated SPARK-35072: Description: During reviews of SPARK-34472, there was a desire to support local:// and

[jira] [Created] (SPARK-35073) SparkContext.addJar with an ivy path should not fail with a custom ivySettings file in non-YARN cluster modes

Shardul Mahadik created SPARK-35073: --- Summary: SparkContext.addJar with an ivy path should not fail with a custom ivySettings file in non-YARN cluster modes Key: SPARK-35073 URL:

[jira] [Created] (SPARK-35072) spark.jars.ivysettings should support local:// and hdfs:// schemes

Shardul Mahadik created SPARK-35072: --- Summary: spark.jars.ivysettings should support local:// and hdfs:// schemes Key: SPARK-35072 URL: https://issues.apache.org/jira/browse/SPARK-35072 Project:

[jira] [Created] (SPARK-34624) Filter non-jar dependencies from ivy/maven coordinates

Shardul Mahadik created SPARK-34624: --- Summary: Filter non-jar dependencies from ivy/maven coordinates Key: SPARK-34624 URL: https://issues.apache.org/jira/browse/SPARK-34624 Project: Spark

[jira] [Commented] (SPARK-34472) SparkContext.addJar with an ivy path fails in cluster mode with a custom ivySettings file

[ https://issues.apache.org/jira/browse/SPARK-34472?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17289258#comment-17289258 ] Shardul Mahadik commented on SPARK-34472: - [~xkrogen] raised a good point at

[jira] [Updated] (SPARK-34506) ADD JAR with ivy coordinates should be compatible with Hive behavior

[ https://issues.apache.org/jira/browse/SPARK-34506?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shardul Mahadik updated SPARK-34506: Description: SPARK-33084 added the ability to use ivy coordinates with

[jira] [Updated] (SPARK-34506) ADD JAR with ivy coordinates should be compatible with Hive transitive behavior

[ https://issues.apache.org/jira/browse/SPARK-34506?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shardul Mahadik updated SPARK-34506: Summary: ADD JAR with ivy coordinates should be compatible with Hive transitive behavior

[jira] [Updated] (SPARK-34506) ADD JAR with ivy coordinates should be compatible with Hive behavior

[ https://issues.apache.org/jira/browse/SPARK-34506?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shardul Mahadik updated SPARK-34506: Summary: ADD JAR with ivy coordinates should be compatible with Hive behavior (was: ADD

[jira] [Created] (SPARK-34506) ADD JAR with ivy coordinates should transitively fetch dependencies by default

Shardul Mahadik created SPARK-34506: --- Summary: ADD JAR with ivy coordinates should transitively fetch dependencies by default Key: SPARK-34506 URL: https://issues.apache.org/jira/browse/SPARK-34506

[jira] [Created] (SPARK-34477) Kryo NPEs when serializing Avro GenericData objects (except GenericRecord)

Shardul Mahadik created SPARK-34477: --- Summary: Kryo NPEs when serializing Avro GenericData objects (except GenericRecord) Key: SPARK-34477 URL: https://issues.apache.org/jira/browse/SPARK-34477

[jira] [Commented] (SPARK-34472) SparkContext.addJar with an ivy path fails in cluster mode with a custom ivySettings file

[ https://issues.apache.org/jira/browse/SPARK-34472?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17286977#comment-17286977 ] Shardul Mahadik commented on SPARK-34472: - I will be sending a PR for this soon. >

[jira] [Created] (SPARK-34472) SparkContext.addJar with an ivy path fails in cluster mode with a custom ivySettings file

Shardul Mahadik created SPARK-34472: --- Summary: SparkContext.addJar with an ivy path fails in cluster mode with a custom ivySettings file Key: SPARK-34472 URL: https://issues.apache.org/jira/browse/SPARK-34472