[jira] [Resolved] (SPARK-38590) New SQL function: try_to_binary

2022-04-06 Thread Max Gekk (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38590?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Max Gekk resolved SPARK-38590. -- Fix Version/s: 3.3.0 Resolution: Fixed Issue resolved by pull request 35897

[jira] [Assigned] (SPARK-38800) Explicitly document the supported pandas version.

2022-04-06 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38800?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-38800: Assignee: (was: Apache Spark) > Explicitly document the supported pandas version. >

[jira] [Commented] (SPARK-38800) Explicitly document the supported pandas version.

2022-04-06 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38800?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17518566#comment-17518566 ] Apache Spark commented on SPARK-38800: -- User 'itholic' has created a pull request for this issue:

[jira] [Assigned] (SPARK-38800) Explicitly document the supported pandas version.

2022-04-06 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38800?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-38800: Assignee: Apache Spark > Explicitly document the supported pandas version. >

[jira] [Commented] (SPARK-38801) ISO-8859-1 encoding doesn't work for text format

2022-04-06 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38801?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17518565#comment-17518565 ] Hyukjin Kwon commented on SPARK-38801: -- Text file doesn't yet support encoding option. >

[jira] [Commented] (SPARK-38806) Unable to initialize the empty pyspark.pandas dataframe

2022-04-06 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38806?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17518564#comment-17518564 ] Hyukjin Kwon commented on SPARK-38806: -- Seems like this is an issue from your env itself instead of

[jira] [Commented] (SPARK-38810) Upgrade Python black in /dev/reformat-python

2022-04-06 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38810?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17518563#comment-17518563 ] Hyukjin Kwon commented on SPARK-38810: -- I faced the same issue. This can be fixed by reinstalling

[jira] [Assigned] (SPARK-38814) Migrate Junit4 to Junit5

2022-04-06 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38814?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-38814: Assignee: Apache Spark > Migrate Junit4 to Junit5 > > >

[jira] [Assigned] (SPARK-38814) Migrate Junit4 to Junit5

2022-04-06 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38814?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-38814: Assignee: (was: Apache Spark) > Migrate Junit4 to Junit5 >

[jira] [Updated] (SPARK-38800) Explicitly document the supported pandas version.

2022-04-06 Thread Haejoon Lee (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38800?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Haejoon Lee updated SPARK-38800: Component/s: Documentation > Explicitly document the supported pandas version. >

[jira] [Created] (SPARK-38814) Migrate Junit4 to Junit5

2022-04-06 Thread Yang Jie (Jira)
Yang Jie created SPARK-38814: Summary: Migrate Junit4 to Junit5 Key: SPARK-38814 URL: https://issues.apache.org/jira/browse/SPARK-38814 Project: Spark Issue Type: Improvement

[jira] [Assigned] (SPARK-38813) Remove TimestampNTZ type support in Spark 3.3

2022-04-06 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38813?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-38813: Assignee: Apache Spark (was: Gengliang Wang) > Remove TimestampNTZ type support in

[jira] [Commented] (SPARK-38813) Remove TimestampNTZ type support in Spark 3.3

2022-04-06 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38813?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17518552#comment-17518552 ] Apache Spark commented on SPARK-38813: -- User 'gengliangwang' has created a pull request for this

[jira] [Assigned] (SPARK-38813) Remove TimestampNTZ type support in Spark 3.3

2022-04-06 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38813?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-38813: Assignee: Gengliang Wang (was: Apache Spark) > Remove TimestampNTZ type support in

[jira] [Created] (SPARK-38813) Remove TimestampNTZ type support in Spark 3.2

2022-04-06 Thread Gengliang Wang (Jira)
Gengliang Wang created SPARK-38813: -- Summary: Remove TimestampNTZ type support in Spark 3.2 Key: SPARK-38813 URL: https://issues.apache.org/jira/browse/SPARK-38813 Project: Spark Issue

[jira] [Updated] (SPARK-38813) Remove TimestampNTZ type support in Spark 3.3

2022-04-06 Thread Gengliang Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38813?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gengliang Wang updated SPARK-38813: --- Summary: Remove TimestampNTZ type support in Spark 3.3 (was: Remove TimestampNTZ type

[jira] [Updated] (SPARK-38162) Optimize one row plan in normal and AQE Optimizer

2022-04-06 Thread XiDuo You (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38162?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] XiDuo You updated SPARK-38162: -- Parent: SPARK-37063 Issue Type: Sub-task (was: Improvement) > Optimize one row plan in

[jira] [Updated] (SPARK-38812) when i clean data ,I hope one rdd spill two rdd according clean data rule

2022-04-06 Thread gaokui (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38812?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] gaokui updated SPARK-38812: --- Description: when id do clean data,one rdd according one value(>or <) filter data, and then generate two

[jira] [Updated] (SPARK-38812) when i clean data ,I hope one rdd spill two rdd according clean data rule

2022-04-06 Thread gaokui (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38812?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] gaokui updated SPARK-38812: --- Summary: when i clean data ,I hope one rdd spill two rdd according clean data rule (was: rdd spil two rdd)

[jira] [Updated] (SPARK-38808) Windows, Spark 3.2.1: spark-shell command throwing this error: SparkContext: Error initializing SparkContext

2022-04-06 Thread Yuming Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38808?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuming Wang updated SPARK-38808: Target Version/s: (was: 3.2.1) > Windows, Spark 3.2.1: spark-shell command throwing this error:

[jira] [Updated] (SPARK-38812) rdd spil two rdd

2022-04-06 Thread gaokui (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38812?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] gaokui updated SPARK-38812: --- Description: when id do clean data, i hope rdd according one value(>or <) filter data, and then to

[jira] [Created] (SPARK-38812) rdd spilt，when id do clean data, i hope rdd according one value(>or <) filter data to two different set，one write error file， another is

2022-04-06 Thread gaokui (Jira)
gaokui created SPARK-38812: -- Summary: rdd spilt，when id do clean data, i hope rdd according one value(>or <) filter data to two different set，one write error file， another is Key: SPARK-38812 URL:

[jira] [Assigned] (SPARK-38772) Formatting the log plan in AdaptiveSparkPlanExec

2022-04-06 Thread Yuming Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38772?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuming Wang reassigned SPARK-38772: --- Assignee: Yuming Wang > Formatting the log plan in AdaptiveSparkPlanExec >

[jira] [Resolved] (SPARK-38772) Formatting the log plan in AdaptiveSparkPlanExec

2022-04-06 Thread Yuming Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38772?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuming Wang resolved SPARK-38772. - Fix Version/s: 3.4.0 Resolution: Fixed Issue resolved by pull request 36045

[jira] [Commented] (SPARK-38391) Datasource v2 supports partial topN push-down

2022-04-06 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38391?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17518527#comment-17518527 ] Apache Spark commented on SPARK-38391: -- User 'beliefer' has created a pull request for this issue:

[jira] [Commented] (SPARK-38768) If limit could pushed down and Data source only have one partition, DS V2 should not do limit again

2022-04-06 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38768?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17518526#comment-17518526 ] Apache Spark commented on SPARK-38768: -- User 'beliefer' has created a pull request for this issue:

[jira] [Commented] (SPARK-38391) Datasource v2 supports partial topN push-down

2022-04-06 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38391?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17518525#comment-17518525 ] Apache Spark commented on SPARK-38391: -- User 'beliefer' has created a pull request for this issue:

[jira] [Assigned] (SPARK-38811) Support ALTER TABLE ADD COLUMN commands with DEFAULT values

2022-04-06 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38811?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-38811: Assignee: Apache Spark > Support ALTER TABLE ADD COLUMN commands with DEFAULT values >

[jira] [Commented] (SPARK-38811) Support ALTER TABLE ADD COLUMN commands with DEFAULT values

2022-04-06 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38811?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17518520#comment-17518520 ] Apache Spark commented on SPARK-38811: -- User 'dtenedor' has created a pull request for this issue:

[jira] [Assigned] (SPARK-38811) Support ALTER TABLE ADD COLUMN commands with DEFAULT values

2022-04-06 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38811?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-38811: Assignee: (was: Apache Spark) > Support ALTER TABLE ADD COLUMN commands with DEFAULT

[jira] [Updated] (SPARK-38795) Support INSERT INTO user specified column lists with DEFAULT values

2022-04-06 Thread Daniel (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38795?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daniel updated SPARK-38795: --- Target Version/s: 3.4.0 > Support INSERT INTO user specified column lists with DEFAULT values >

[jira] [Updated] (SPARK-38811) Support ALTER TABLE ADD COLUMN commands with DEFAULT values

2022-04-06 Thread Daniel (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38811?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daniel updated SPARK-38811: --- Shepherd: Gengliang Wang Target Version/s: 3.4.0 > Support ALTER TABLE ADD COLUMN commands with

[jira] [Created] (SPARK-38811) Support ALTER TABLE ADD COLUMN commands with DEFAULT values

2022-04-06 Thread Daniel (Jira)
Daniel created SPARK-38811: -- Summary: Support ALTER TABLE ADD COLUMN commands with DEFAULT values Key: SPARK-38811 URL: https://issues.apache.org/jira/browse/SPARK-38811 Project: Spark Issue Type:

[jira] [Commented] (SPARK-38809) Implement option to skip null values in symmetric hash impl of stream-stream joins

2022-04-06 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38809?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17518483#comment-17518483 ] Apache Spark commented on SPARK-38809: -- User 'anishshri-db' has created a pull request for this

[jira] [Assigned] (SPARK-38809) Implement option to skip null values in symmetric hash impl of stream-stream joins

2022-04-06 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38809?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-38809: Assignee: (was: Apache Spark) > Implement option to skip null values in symmetric

[jira] [Commented] (SPARK-38809) Implement option to skip null values in symmetric hash impl of stream-stream joins

2022-04-06 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38809?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17518481#comment-17518481 ] Apache Spark commented on SPARK-38809: -- User 'anishshri-db' has created a pull request for this

[jira] [Assigned] (SPARK-38809) Implement option to skip null values in symmetric hash impl of stream-stream joins

2022-04-06 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38809?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-38809: Assignee: Apache Spark > Implement option to skip null values in symmetric hash impl of

[jira] [Comment Edited] (SPARK-38787) Possible correctness issue on stream-stream join when handling edge case

2022-04-06 Thread Jungtaek Lim (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38787?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17518471#comment-17518471 ] Jungtaek Lim edited comment on SPARK-38787 at 4/6/22 9:33 PM: -- (Somehow the

[jira] [Commented] (SPARK-38787) Possible correctness issue on stream-stream join when handling edge case

2022-04-06 Thread Jungtaek Lim (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38787?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17518471#comment-17518471 ] Jungtaek Lim commented on SPARK-38787: -- (Somehow the account is not recognized in the merge

[jira] [Assigned] (SPARK-38787) Possible correctness issue on stream-stream join when handling edge case

2022-04-06 Thread Jungtaek Lim (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38787?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jungtaek Lim reassigned SPARK-38787: Assignee: Anish Shrigondekar > Possible correctness issue on stream-stream join when

[jira] [Resolved] (SPARK-38787) Possible correctness issue on stream-stream join when handling edge case

2022-04-06 Thread Jungtaek Lim (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38787?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jungtaek Lim resolved SPARK-38787. -- Fix Version/s: 3.3.0 3.2.2 Resolution: Fixed Issue resolved by pull

[jira] [Created] (SPARK-38810) Upgrade Python black in /dev/reformat-python

2022-04-06 Thread Jira
Bjørn Jørgensen created SPARK-38810: --- Summary: Upgrade Python black in /dev/reformat-python Key: SPARK-38810 URL: https://issues.apache.org/jira/browse/SPARK-38810 Project: Spark Issue

[jira] [Commented] (SPARK-38792) Regression in time executor takes to do work sometime after v3.0.1 ?

2022-04-06 Thread Danny Guinther (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38792?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17518460#comment-17518460 ] Danny Guinther commented on SPARK-38792: I added another kind of dummy job that doesn't farm

[jira] [Commented] (SPARK-37660) Spark-3.2.0 Fetch Hbase Data not working

2022-04-06 Thread Thomas Prelle (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37660?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17518446#comment-17518446 ] Thomas Prelle commented on SPARK-37660: --- Hi [~Bhavya Raj Sharma], i found why on my case

[jira] [Commented] (SPARK-38809) Implement option to skip null values in symmetric hash impl of stream-stream joins

2022-04-06 Thread Anish Shrigondekar (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38809?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17518344#comment-17518344 ] Anish Shrigondekar commented on SPARK-38809: Working on this PR and will send the change out

[jira] [Comment Edited] (SPARK-38809) Implement option to skip null values in symmetric hash impl of stream-stream joins

2022-04-06 Thread Anish Shrigondekar (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38809?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17518344#comment-17518344 ] Anish Shrigondekar edited comment on SPARK-38809 at 4/6/22 6:38 PM:

[jira] [Created] (SPARK-38809) Implement option to skip null values in symmetric hash impl of stream-stream joins

2022-04-06 Thread Anish Shrigondekar (Jira)
Anish Shrigondekar created SPARK-38809: -- Summary: Implement option to skip null values in symmetric hash impl of stream-stream joins Key: SPARK-38809 URL: https://issues.apache.org/jira/browse/SPARK-38809

[jira] [Updated] (SPARK-38792) Regression in time executor takes to do work sometime after v3.0.1 ?

2022-04-06 Thread Danny Guinther (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38792?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Danny Guinther updated SPARK-38792: --- Summary: Regression in time executor takes to do work sometime after v3.0.1 ? (was:

[jira] [Updated] (SPARK-38792) Regression in time executor takes to do work since v3.0.1 ?

2022-04-06 Thread Danny Guinther (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38792?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Danny Guinther updated SPARK-38792: --- Attachment: dummy-job-query.png > Regression in time executor takes to do work since v3.0.1

[jira] [Commented] (SPARK-38792) Regression in time executor takes to do work since v3.0.1 ?

2022-04-06 Thread Danny Guinther (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38792?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17518323#comment-17518323 ] Danny Guinther commented on SPARK-38792: Things move through the Spark UI for my application too

[jira] [Updated] (SPARK-38792) Regression in time executor takes to do work since v3.0.1 ?

2022-04-06 Thread Danny Guinther (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38792?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Danny Guinther updated SPARK-38792: --- Attachment: dummy-job-job.jpg > Regression in time executor takes to do work since v3.0.1 ?

[jira] [Resolved] (SPARK-38762) Provide query context in Decimal overflow errors

2022-04-06 Thread Gengliang Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38762?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gengliang Wang resolved SPARK-38762. Fix Version/s: 3.3.0 Resolution: Fixed Issue resolved by pull request 36040

[jira] [Commented] (SPARK-38808) Windows, Spark 3.2.1: spark-shell command throwing this error: SparkContext: Error initializing SparkContext

2022-04-06 Thread Taras (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38808?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17518308#comment-17518308 ] Taras commented on SPARK-38808: --- Explanation of the bug and one workaround can be found below

[jira] [Commented] (SPARK-38807) Error when starting spark shell on Windows system

2022-04-06 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38807?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17518299#comment-17518299 ] Apache Spark commented on SPARK-38807: -- User '1104056452' has created a pull request for this

[jira] [Assigned] (SPARK-38807) Error when starting spark shell on Windows system

2022-04-06 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38807?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-38807: Assignee: Apache Spark > Error when starting spark shell on Windows system >

[jira] [Assigned] (SPARK-38807) Error when starting spark shell on Windows system

2022-04-06 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38807?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-38807: Assignee: (was: Apache Spark) > Error when starting spark shell on Windows system >

[jira] [Resolved] (SPARK-38790) Unexpected behaviour for "IN" operator in spark when null is involved in array

2022-04-06 Thread Vishnu K Suman (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38790?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vishnu K Suman resolved SPARK-38790. Resolution: Not A Problem > Unexpected behaviour for "IN" operator in spark when null is

[jira] [Commented] (SPARK-38790) Unexpected behaviour for "IN" operator in spark when null is involved in array

2022-04-06 Thread Vishnu K Suman (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38790?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17518274#comment-17518274 ] Vishnu K Suman commented on SPARK-38790: [~bersprockets] That was very helpful, thanks for the

[jira] [Updated] (SPARK-38792) Regression in time executor takes to do work since v3.0.1 ?

2022-04-06 Thread Danny Guinther (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38792?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Danny Guinther updated SPARK-38792: --- Attachment: min-time-way-up.jpg > Regression in time executor takes to do work since v3.0.1

[jira] [Updated] (SPARK-38792) Regression in time executor takes to do work since v3.0.1 ?

2022-04-06 Thread Danny Guinther (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38792?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Danny Guinther updated SPARK-38792: --- Attachment: (was: min-time-way-up.jpg) > Regression in time executor takes to do work

[jira] [Commented] (SPARK-38792) Regression in time executor takes to do work since v3.0.1 ?

2022-04-06 Thread Danny Guinther (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38792?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17518273#comment-17518273 ] Danny Guinther commented on SPARK-38792: To further try to understand what is going on here, I

[jira] [Updated] (SPARK-38792) Regression in time executor takes to do work since v3.0.1 ?

2022-04-06 Thread Danny Guinther (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38792?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Danny Guinther updated SPARK-38792: --- Attachment: min-time-way-up.jpg > Regression in time executor takes to do work since v3.0.1

[jira] [Resolved] (SPARK-38768) If limit could pushed down and Data source only have one partition, DS V2 should not do limit again

2022-04-06 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38768?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-38768. - Fix Version/s: 3.4.0 Resolution: Fixed Issue resolved by pull request 36043

[jira] [Created] (SPARK-38808) Windows, Spark 3.2.1: spark-shell command throwing this error: SparkContext: Error initializing SparkContext

2022-04-06 Thread Taras (Jira)
Taras created SPARK-38808: - Summary: Windows, Spark 3.2.1: spark-shell command throwing this error: SparkContext: Error initializing SparkContext Key: SPARK-38808 URL: https://issues.apache.org/jira/browse/SPARK-38808

[jira] [Created] (SPARK-38807) Error when starting spark shell on Windows system

2022-04-06 Thread Ming Li (Jira)
Ming Li created SPARK-38807: --- Summary: Error when starting spark shell on Windows system Key: SPARK-38807 URL: https://issues.apache.org/jira/browse/SPARK-38807 Project: Spark Issue Type: Bug

[jira] [Created] (SPARK-38806) Unable to initialize the empty pyspark.pandas dataframe

2022-04-06 Thread Prakhar Sandhu (Jira)
Prakhar Sandhu created SPARK-38806: -- Summary: Unable to initialize the empty pyspark.pandas dataframe Key: SPARK-38806 URL: https://issues.apache.org/jira/browse/SPARK-38806 Project: Spark

[jira] [Resolved] (SPARK-38799) Fix scala license declaration

2022-04-06 Thread Sean R. Owen (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38799?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean R. Owen resolved SPARK-38799. -- Fix Version/s: 3.4.0 Resolution: Fixed Issue resolved by pull request 36081

[jira] [Assigned] (SPARK-38799) Fix scala license declaration

2022-04-06 Thread Sean R. Owen (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38799?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean R. Owen reassigned SPARK-38799: Assignee: Kent Yao > Fix scala license declaration > - > >

[jira] [Updated] (SPARK-38799) Fix scala license declaration

2022-04-06 Thread Sean R. Owen (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38799?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean R. Owen updated SPARK-38799: - Priority: Minor (was: Major) > Fix scala license declaration > - >

[jira] [Commented] (SPARK-38805) Remove an expired indexFilePath from the ESS shuffleIndexCache or the PBS indexCache to save memory.

2022-04-06 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38805?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17518092#comment-17518092 ] Apache Spark commented on SPARK-38805: -- User 'weixiuli' has created a pull request for this issue:

[jira] [Assigned] (SPARK-38805) Remove an expired indexFilePath from the ESS shuffleIndexCache or the PBS indexCache to save memory.

2022-04-06 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38805?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-38805: Assignee: (was: Apache Spark) > Remove an expired indexFilePath from the ESS

[jira] [Commented] (SPARK-38805) Remove an expired indexFilePath from the ESS shuffleIndexCache or the PBS indexCache to save memory.

2022-04-06 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38805?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17518091#comment-17518091 ] Apache Spark commented on SPARK-38805: -- User 'weixiuli' has created a pull request for this issue:

[jira] [Assigned] (SPARK-38805) Remove an expired indexFilePath from the ESS shuffleIndexCache or the PBS indexCache to save memory.

2022-04-06 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38805?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-38805: Assignee: Apache Spark > Remove an expired indexFilePath from the ESS shuffleIndexCache

[jira] [Created] (SPARK-38805) Remove an expired indexFilePath from the ESS shuffleIndexCache or the PBS indexCache to save memory.

2022-04-06 Thread weixiuli (Jira)
weixiuli created SPARK-38805: Summary: Remove an expired indexFilePath from the ESS shuffleIndexCache or the PBS indexCache to save memory. Key: SPARK-38805 URL: https://issues.apache.org/jira/browse/SPARK-38805

[jira] [Comment Edited] (SPARK-38404) Spark does not find CTE inside nested CTE

2022-04-06 Thread Jarno Rajala (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38404?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17518011#comment-17518011 ] Jarno Rajala edited comment on SPARK-38404 at 4/6/22 10:13 AM: --- I've

[jira] [Commented] (SPARK-38404) Spark does not find CTE inside nested CTE

2022-04-06 Thread Jarno Rajala (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38404?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17518011#comment-17518011 ] Jarno Rajala commented on SPARK-38404: -- I've experienced the same bug in Databricks. I've also

[jira] [Commented] (SPARK-38802) Support spark.kubernetes.test.(driver|executor)RequestCores

2022-04-06 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38802?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17517970#comment-17517970 ] Apache Spark commented on SPARK-38802: -- User 'Yikun' has created a pull request for this issue:

[jira] [Assigned] (SPARK-38802) Support spark.kubernetes.test.(driver|executor)RequestCores

2022-04-06 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38802?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-38802: Assignee: (was: Apache Spark) > Support

[jira] [Assigned] (SPARK-38802) Support spark.kubernetes.test.(driver|executor)RequestCores

2022-04-06 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38802?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-38802: Assignee: Apache Spark > Support spark.kubernetes.test.(driver|executor)RequestCores >

[jira] [Assigned] (SPARK-38804) Add StreamingQueryManager.removeListener in PySpark

2022-04-06 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38804?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-38804: Assignee: Apache Spark > Add StreamingQueryManager.removeListener in PySpark >

[jira] [Commented] (SPARK-38804) Add StreamingQueryManager.removeListener in PySpark

2022-04-06 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38804?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17517936#comment-17517936 ] Apache Spark commented on SPARK-38804: -- User 'HyukjinKwon' has created a pull request for this

[jira] [Assigned] (SPARK-38804) Add StreamingQueryManager.removeListener in PySpark

2022-04-06 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38804?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-38804: Assignee: (was: Apache Spark) > Add StreamingQueryManager.removeListener in PySpark

[jira] [Assigned] (SPARK-38803) Add `minio` tag for K8s IT

2022-04-06 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38803?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-38803: Assignee: (was: Apache Spark) > Add `minio` tag for K8s IT >

[jira] [Commented] (SPARK-38803) Add `minio` tag for K8s IT

2022-04-06 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38803?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17517931#comment-17517931 ] Apache Spark commented on SPARK-38803: -- User 'Yikun' has created a pull request for this issue:

[jira] [Commented] (SPARK-38803) Add `minio` tag for K8s IT

2022-04-06 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38803?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17517930#comment-17517930 ] Apache Spark commented on SPARK-38803: -- User 'Yikun' has created a pull request for this issue:

[jira] [Assigned] (SPARK-38803) Add `minio` tag for K8s IT

2022-04-06 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38803?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-38803: Assignee: Apache Spark > Add `minio` tag for K8s IT > -- > >

[jira] [Created] (SPARK-38804) Add StreamingQueryManager.removeListener in PySpark

2022-04-06 Thread Hyukjin Kwon (Jira)
Hyukjin Kwon created SPARK-38804: Summary: Add StreamingQueryManager.removeListener in PySpark Key: SPARK-38804 URL: https://issues.apache.org/jira/browse/SPARK-38804 Project: Spark Issue

[jira] [Created] (SPARK-38803) Add `minio` tag for K8s IT

2022-04-06 Thread Yikun Jiang (Jira)
Yikun Jiang created SPARK-38803: --- Summary: Add `minio` tag for K8s IT Key: SPARK-38803 URL: https://issues.apache.org/jira/browse/SPARK-38803 Project: Spark Issue Type: Bug

[jira] [Created] (SPARK-38802) Support spark.kubernetes.test.(driver|executor)RequestCores

2022-04-06 Thread Yikun Jiang (Jira)
Yikun Jiang created SPARK-38802: --- Summary: Support spark.kubernetes.test.(driver|executor)RequestCores Key: SPARK-38802 URL: https://issues.apache.org/jira/browse/SPARK-38802 Project: Spark

[jira] [Updated] (SPARK-38801) ISO-8859-1 encoding doesn't work for text format

2022-04-06 Thread Himanshu Arora (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38801?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Himanshu Arora updated SPARK-38801: --- Description: When reading text files from spark which are not in UTF-8 charset it doesn't

[jira] [Updated] (SPARK-38801) ISO-8859-1 encoding doesn't work for text format

2022-04-06 Thread Himanshu Arora (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38801?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Himanshu Arora updated SPARK-38801: --- Environment: Databricks runtime 10.3 (spark 3.2.1, scala 2.12) (was: I tested this issue

[jira] [Updated] (SPARK-38801) ISO-8859-1 encoding doesn't work for text format

2022-04-06 Thread Himanshu Arora (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38801?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Himanshu Arora updated SPARK-38801: --- Attachment: Screenshot 2022-04-06 at 09.30.02.png > ISO-8859-1 encoding doesn't work for

[jira] [Updated] (SPARK-38801) ISO-8859-1 encoding doesn't work for text format

2022-04-06 Thread Himanshu Arora (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38801?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Himanshu Arora updated SPARK-38801: --- Attachment: Screenshot 2022-04-06 at 09.29.24.png > ISO-8859-1 encoding doesn't work for

[jira] [Created] (SPARK-38801) ISO-8859-1 encoding doesn't work for text format

2022-04-06 Thread Himanshu Arora (Jira)
Himanshu Arora created SPARK-38801: -- Summary: ISO-8859-1 encoding doesn't work for text format Key: SPARK-38801 URL: https://issues.apache.org/jira/browse/SPARK-38801 Project: Spark Issue