[jira] [Assigned] (SPARK-32946) Add withField function to SparkR

[ https://issues.apache.org/jira/browse/SPARK-32946?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-32946: Assignee: Apache Spark > Add withField function to SparkR >

[jira] [Assigned] (SPARK-32946) Add withField function to SparkR

[ https://issues.apache.org/jira/browse/SPARK-32946?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-32946: Assignee: (was: Apache Spark) > Add withField function to SparkR >

[jira] [Commented] (SPARK-32946) Add withField function to SparkR

[ https://issues.apache.org/jira/browse/SPARK-32946?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17198907#comment-17198907 ] Apache Spark commented on SPARK-32946: -- User 'zero323' has created a pull request for this issue:

[jira] [Created] (SPARK-32946) Add withField function to SparkR

Maciej Szymkiewicz created SPARK-32946: -- Summary: Add withField function to SparkR Key: SPARK-32946 URL: https://issues.apache.org/jira/browse/SPARK-32946 Project: Spark Issue Type:

[jira] [Commented] (SPARK-32799) Make unionByName optionally fill missing columns with nulls in SparkR

[ https://issues.apache.org/jira/browse/SPARK-32799?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17198889#comment-17198889 ] Apache Spark commented on SPARK-32799: -- User 'zero323' has created a pull request for this issue:

[jira] [Assigned] (SPARK-32799) Make unionByName optionally fill missing columns with nulls in SparkR

[ https://issues.apache.org/jira/browse/SPARK-32799?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-32799: Assignee: Apache Spark > Make unionByName optionally fill missing columns with nulls in

[jira] [Commented] (SPARK-32799) Make unionByName optionally fill missing columns with nulls in SparkR

[ https://issues.apache.org/jira/browse/SPARK-32799?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17198890#comment-17198890 ] Apache Spark commented on SPARK-32799: -- User 'zero323' has created a pull request for this issue:

[jira] [Assigned] (SPARK-32799) Make unionByName optionally fill missing columns with nulls in SparkR

[ https://issues.apache.org/jira/browse/SPARK-32799?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-32799: Assignee: (was: Apache Spark) > Make unionByName optionally fill missing columns

[jira] [Commented] (SPARK-32937) DecomissionSuite in k8s integration tests is failing.

[ https://issues.apache.org/jira/browse/SPARK-32937?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17198879#comment-17198879 ] Hyukjin Kwon commented on SPARK-32937: -- Sure, if there's a technical reason to revert, I believe

[jira] [Commented] (SPARK-32937) DecomissionSuite in k8s integration tests is failing.

[ https://issues.apache.org/jira/browse/SPARK-32937?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17198867#comment-17198867 ] Holden Karau commented on SPARK-32937: -- So there are additional proposed changes to this chunk of

[jira] [Commented] (SPARK-32937) DecomissionSuite in k8s integration tests is failing.

[ https://issues.apache.org/jira/browse/SPARK-32937?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17198866#comment-17198866 ] Hyukjin Kwon commented on SPARK-32937: -- This is a usual approach to make a followup if that can be

[jira] [Commented] (SPARK-32937) DecomissionSuite in k8s integration tests is failing.

[ https://issues.apache.org/jira/browse/SPARK-32937?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17198863#comment-17198863 ] Holden Karau commented on SPARK-32937: -- I want to be clear, I'm asking for a revert then fix. I do

[jira] [Commented] (SPARK-32937) DecomissionSuite in k8s integration tests is failing.

[ https://issues.apache.org/jira/browse/SPARK-32937?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17198862#comment-17198862 ] Hyukjin Kwon commented on SPARK-32937: -- [~Ngone51] can you take a look? If this isn't easy to fix,

[jira] [Created] (SPARK-32945) Avoid project combinatin when it will hurst performance

L. C. Hsieh created SPARK-32945: --- Summary: Avoid project combinatin when it will hurst performance Key: SPARK-32945 URL: https://issues.apache.org/jira/browse/SPARK-32945 Project: Spark Issue

[jira] [Commented] (SPARK-32937) DecomissionSuite in k8s integration tests is failing.

[ https://issues.apache.org/jira/browse/SPARK-32937?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17198860#comment-17198860 ] Holden Karau commented on SPARK-32937: -- It looks like someone changed the logging message again

[jira] [Commented] (SPARK-32912) Sparksql authentication

[ https://issues.apache.org/jira/browse/SPARK-32912?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17198857#comment-17198857 ] Hyukjin Kwon commented on SPARK-32912: -- Let's discuss this in the mailing list instead of filing an

[jira] [Resolved] (SPARK-32912) Sparksql authentication

[ https://issues.apache.org/jira/browse/SPARK-32912?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-32912. -- Resolution: Invalid > Sparksql authentication > --- > >

[jira] [Created] (SPARK-32944) Avoid push down Filter through Project when it will hurts performance

L. C. Hsieh created SPARK-32944: --- Summary: Avoid push down Filter through Project when it will hurts performance Key: SPARK-32944 URL: https://issues.apache.org/jira/browse/SPARK-32944 Project: Spark

[jira] [Comment Edited] (SPARK-32925) Support push-based shuffle in multiple deployment environments

[ https://issues.apache.org/jira/browse/SPARK-32925?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17198856#comment-17198856 ] Hyukjin Kwon edited comment on SPARK-32925 at 9/20/20, 1:53 AM: cc

[jira] [Comment Edited] (SPARK-32925) Support push-based shuffle in multiple deployment environments

[ https://issues.apache.org/jira/browse/SPARK-32925?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17198856#comment-17198856 ] Hyukjin Kwon edited comment on SPARK-32925 at 9/20/20, 1:53 AM: cc

[jira] [Commented] (SPARK-32925) Support push-based shuffle in multiple deployment environments

[ https://issues.apache.org/jira/browse/SPARK-32925?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17198856#comment-17198856 ] Hyukjin Kwon commented on SPARK-32925: -- cc [~mridul] FYI > Support push-based shuffle in multiple

[jira] [Commented] (SPARK-32937) DecomissionSuite in k8s integration tests is failing.

[ https://issues.apache.org/jira/browse/SPARK-32937?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17198855#comment-17198855 ] Hyukjin Kwon commented on SPARK-32937: -- cc [~holden]can you take a look please? > DecomissionSuite

[jira] [Created] (SPARK-32943) Add improvement to filter pushdown and project combination

L. C. Hsieh created SPARK-32943: --- Summary: Add improvement to filter pushdown and project combination Key: SPARK-32943 URL: https://issues.apache.org/jira/browse/SPARK-32943 Project: Spark

[jira] [Commented] (SPARK-32937) DecomissionSuite in k8s integration tests is failing.

[ https://issues.apache.org/jira/browse/SPARK-32937?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17198853#comment-17198853 ] Hyukjin Kwon commented on SPARK-32937: -- Oops, I left a comment in the wrong JIRA. I removed it back

[jira] [Resolved] (SPARK-32938) Spark can not cast long value from Kafka

[ https://issues.apache.org/jira/browse/SPARK-32938?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-32938. -- Resolution: Invalid It's just you're trying to cast binary to long type which isn't allowed.

[jira] [Issue Comment Deleted] (SPARK-32937) DecomissionSuite in k8s integration tests is failing.

[ https://issues.apache.org/jira/browse/SPARK-32937?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-32937: - Comment: was deleted (was: It's just you're trying to cast binary to long type which isn't

[jira] [Commented] (SPARK-32937) DecomissionSuite in k8s integration tests is failing.

[ https://issues.apache.org/jira/browse/SPARK-32937?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17198852#comment-17198852 ] Hyukjin Kwon commented on SPARK-32937: -- It's just you're trying to cast binary to long type which

[jira] [Assigned] (SPARK-32941) Optimize WithFields expression chain

[ https://issues.apache.org/jira/browse/SPARK-32941?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-32941: Assignee: Apache Spark (was: L. C. Hsieh) > Optimize WithFields expression chain >

[jira] [Updated] (SPARK-32938) Spark can not cast long value from Kafka

[ https://issues.apache.org/jira/browse/SPARK-32938?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-32938: - Priority: Major (was: Blocker) > Spark can not cast long value from Kafka >

[jira] [Commented] (SPARK-32941) Optimize WithFields expression chain

[ https://issues.apache.org/jira/browse/SPARK-32941?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17198851#comment-17198851 ] Apache Spark commented on SPARK-32941: -- User 'viirya' has created a pull request for this issue:

[jira] [Assigned] (SPARK-32941) Optimize WithFields expression chain

[ https://issues.apache.org/jira/browse/SPARK-32941?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-32941: Assignee: L. C. Hsieh (was: Apache Spark) > Optimize WithFields expression chain >

[jira] [Updated] (SPARK-32941) Optimize WithFields expression chain

[ https://issues.apache.org/jira/browse/SPARK-32941?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] L. C. Hsieh updated SPARK-32941: Parent: SPARK-32942 Issue Type: Sub-task (was: Improvement) > Optimize WithFields

[jira] [Created] (SPARK-32942) Optimize catalyst expression chain

L. C. Hsieh created SPARK-32942: --- Summary: Optimize catalyst expression chain Key: SPARK-32942 URL: https://issues.apache.org/jira/browse/SPARK-32942 Project: Spark Issue Type: Umbrella

[jira] [Created] (SPARK-32941) Optimize WithFields expression chain

L. C. Hsieh created SPARK-32941: --- Summary: Optimize WithFields expression chain Key: SPARK-32941 URL: https://issues.apache.org/jira/browse/SPARK-32941 Project: Spark Issue Type: Improvement

[jira] [Commented] (SPARK-32940) Collect, first and last should be deterministic aggregate functions

[ https://issues.apache.org/jira/browse/SPARK-32940?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17198820#comment-17198820 ] Apache Spark commented on SPARK-32940: -- User 'tanelk' has created a pull request for this issue:

[jira] [Assigned] (SPARK-32940) Collect, first and last should be deterministic aggregate functions

[ https://issues.apache.org/jira/browse/SPARK-32940?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-32940: Assignee: (was: Apache Spark) > Collect, first and last should be deterministic

[jira] [Assigned] (SPARK-32940) Collect, first and last should be deterministic aggregate functions

[ https://issues.apache.org/jira/browse/SPARK-32940?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-32940: Assignee: Apache Spark > Collect, first and last should be deterministic aggregate

[jira] [Created] (SPARK-32940) Collect, first and last should be deterministic aggregate functions

Tanel Kiis created SPARK-32940: -- Summary: Collect, first and last should be deterministic aggregate functions Key: SPARK-32940 URL: https://issues.apache.org/jira/browse/SPARK-32940 Project: Spark

[jira] [Comment Edited] (SPARK-16190) Worker registration failed: Duplicate worker ID

[ https://issues.apache.org/jira/browse/SPARK-16190?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17198736#comment-17198736 ] Valentina Glagoleva edited comment on SPARK-16190 at 9/19/20, 2:21 PM:

[jira] [Comment Edited] (SPARK-16190) Worker registration failed: Duplicate worker ID

[ https://issues.apache.org/jira/browse/SPARK-16190?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17198736#comment-17198736 ] Valentina Glagoleva edited comment on SPARK-16190 at 9/19/20, 2:21 PM:

[jira] [Comment Edited] (SPARK-16190) Worker registration failed: Duplicate worker ID

[ https://issues.apache.org/jira/browse/SPARK-16190?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17198736#comment-17198736 ] Valentina Glagoleva edited comment on SPARK-16190 at 9/19/20, 2:20 PM:

[jira] [Comment Edited] (SPARK-16190) Worker registration failed: Duplicate worker ID

[ https://issues.apache.org/jira/browse/SPARK-16190?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17198736#comment-17198736 ] Valentina Glagoleva edited comment on SPARK-16190 at 9/19/20, 2:18 PM:

[jira] [Assigned] (SPARK-16190) Worker registration failed: Duplicate worker ID

[ https://issues.apache.org/jira/browse/SPARK-16190?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-16190: Assignee: (was: Apache Spark) > Worker registration failed: Duplicate worker ID >

[jira] [Assigned] (SPARK-16190) Worker registration failed: Duplicate worker ID

[ https://issues.apache.org/jira/browse/SPARK-16190?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-16190: Assignee: Apache Spark > Worker registration failed: Duplicate worker ID >

[jira] [Commented] (SPARK-16190) Worker registration failed: Duplicate worker ID

[ https://issues.apache.org/jira/browse/SPARK-16190?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17198737#comment-17198737 ] Apache Spark commented on SPARK-16190: -- User 'valan4ik' has created a pull request for this issue:

[jira] [Comment Edited] (SPARK-16190) Worker registration failed: Duplicate worker ID

[ https://issues.apache.org/jira/browse/SPARK-16190?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17198736#comment-17198736 ] Valentina Glagoleva edited comment on SPARK-16190 at 9/19/20, 1:46 PM:

[jira] [Commented] (SPARK-16190) Worker registration failed: Duplicate worker ID

[ https://issues.apache.org/jira/browse/SPARK-16190?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17198736#comment-17198736 ] Valentina Glagoleva commented on SPARK-16190: - We faced similar issue and found possible

[jira] [Updated] (SPARK-30294) Read-only state store unnecessarily creates and deletes the temp file for delta file every batch

[ https://issues.apache.org/jira/browse/SPARK-30294?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jungtaek Lim updated SPARK-30294: - Affects Version/s: (was: 3.0.0) 3.1.0 > Read-only state store

[jira] [Updated] (SPARK-30294) Read-only state store unnecessarily creates and deletes the temp file for delta file every batch

[ https://issues.apache.org/jira/browse/SPARK-30294?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jungtaek Lim updated SPARK-30294: - Issue Type: Improvement (was: Bug) > Read-only state store unnecessarily creates and deletes

[jira] [Commented] (SPARK-30294) Read-only state store unnecessarily creates and deletes the temp file for delta file every batch

[ https://issues.apache.org/jira/browse/SPARK-30294?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17198682#comment-17198682 ] Jungtaek Lim commented on SPARK-30294: -- Agree. Let me update the type. > Read-only state store

[jira] [Commented] (SPARK-30602) SPIP: Support push-based shuffle to improve shuffle efficiency

[ https://issues.apache.org/jira/browse/SPARK-30602?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17198680#comment-17198680 ] Apache Spark commented on SPARK-30602: -- User 'otterc' has created a pull request for this issue:

[jira] [Commented] (SPARK-30602) SPIP: Support push-based shuffle to improve shuffle efficiency

[ https://issues.apache.org/jira/browse/SPARK-30602?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17198678#comment-17198678 ] Apache Spark commented on SPARK-30602: -- User 'otterc' has created a pull request for this issue:

[jira] [Assigned] (SPARK-30602) SPIP: Support push-based shuffle to improve shuffle efficiency

[ https://issues.apache.org/jira/browse/SPARK-30602?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-30602: Assignee: (was: Apache Spark) > SPIP: Support push-based shuffle to improve shuffle

[jira] [Assigned] (SPARK-30602) SPIP: Support push-based shuffle to improve shuffle efficiency

[ https://issues.apache.org/jira/browse/SPARK-30602?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-30602: Assignee: Apache Spark > SPIP: Support push-based shuffle to improve shuffle efficiency

[jira] [Commented] (SPARK-30602) SPIP: Support push-based shuffle to improve shuffle efficiency

[ https://issues.apache.org/jira/browse/SPARK-30602?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17198679#comment-17198679 ] Chandni Singh commented on SPARK-30602: --- The reference PR for the consolidated change: