[jira] [Assigned] (SPARK-38076) Remove redundant null-check is covered by further condition

[ https://issues.apache.org/jira/browse/SPARK-38076?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun reassigned SPARK-38076: - Assignee: Yang Jie > Remove redundant null-check is covered by further condition >

[jira] [Resolved] (SPARK-38076) Remove redundant null-check is covered by further condition

[ https://issues.apache.org/jira/browse/SPARK-38076?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-38076. --- Fix Version/s: 3.3.0 Resolution: Fixed Issue resolved by pull request 35369

[jira] [Assigned] (SPARK-38080) Flaky test: StreamingQueryManagerSuite: 'awaitAnyTermination with timeout and resetTerminated'

[ https://issues.apache.org/jira/browse/SPARK-38080?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun reassigned SPARK-38080: - Assignee: Shixiong Zhu > Flaky test: StreamingQueryManagerSuite: 'awaitAnyTermination

[jira] [Resolved] (SPARK-38080) Flaky test: StreamingQueryManagerSuite: 'awaitAnyTermination with timeout and resetTerminated'

[ https://issues.apache.org/jira/browse/SPARK-38080?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-38080. --- Fix Version/s: 3.3.0 3.2.2 Resolution: Fixed Issue resolved by

[jira] [Resolved] (SPARK-37397) Inline type hints for python/pyspark/ml/base.py

[ https://issues.apache.org/jira/browse/SPARK-37397?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-37397. --- Fix Version/s: 3.3.0 Resolution: Fixed Issue resolved by pull request 35289

[jira] [Assigned] (SPARK-37397) Inline type hints for python/pyspark/ml/base.py

[ https://issues.apache.org/jira/browse/SPARK-37397?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun reassigned SPARK-37397: - Assignee: Maciej Szymkiewicz > Inline type hints for python/pyspark/ml/base.py >

[jira] [Updated] (SPARK-38013) AQE can change bhj to smj if no extra shuffle introduce

[ https://issues.apache.org/jira/browse/SPARK-38013?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-38013: -- Component/s: Tests > AQE can change bhj to smj if no extra shuffle introduce >

[jira] [Reopened] (SPARK-38013) AQE can change bhj to smj if no extra shuffle introduce

[ https://issues.apache.org/jira/browse/SPARK-38013?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun reopened SPARK-38013: --- > AQE can change bhj to smj if no extra shuffle introduce >

[jira] [Resolved] (SPARK-38013) AQE can change bhj to smj if no extra shuffle introduce

[ https://issues.apache.org/jira/browse/SPARK-38013?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-38013. --- Fix Version/s: 3.3.0 3.2.2 Resolution: Fixed This JIRA added a

[jira] [Assigned] (SPARK-38013) AQE can change bhj to smj if no extra shuffle introduce

[ https://issues.apache.org/jira/browse/SPARK-38013?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun reassigned SPARK-38013: - Assignee: XiDuo You > AQE can change bhj to smj if no extra shuffle introduce >

[jira] [Updated] (SPARK-38087) select doesnt validate if the column already exists

[ https://issues.apache.org/jira/browse/SPARK-38087?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Deepa Vasanthkumar updated SPARK-38087: --- Description:   Select doesnt validate whether the alias column is already present

[jira] [Updated] (SPARK-38087) select doesnt validate if the column already exists

[ https://issues.apache.org/jira/browse/SPARK-38087?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Deepa Vasanthkumar updated SPARK-38087: --- Attachment: select vs drop.png > select doesnt validate if the column already

[jira] [Updated] (SPARK-38087) select doesnt validate if the column already exists

[ https://issues.apache.org/jira/browse/SPARK-38087?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Deepa Vasanthkumar updated SPARK-38087: --- Description:   Select doesnt validate whether the alias column is already present

[jira] [Created] (SPARK-38087) select doesnt validate if the column already exists

Deepa Vasanthkumar created SPARK-38087: -- Summary: select doesnt validate if the column already exists Key: SPARK-38087 URL: https://issues.apache.org/jira/browse/SPARK-38087 Project: Spark

[jira] [Comment Edited] (SPARK-25789) Support for Dataset of Avro

[ https://issues.apache.org/jira/browse/SPARK-25789?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17485580#comment-17485580 ] Prashant Pandey edited comment on SPARK-25789 at 2/2/22, 5:51 AM: -- Is

[jira] [Comment Edited] (SPARK-25789) Support for Dataset of Avro

[ https://issues.apache.org/jira/browse/SPARK-25789?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17485580#comment-17485580 ] Prashant Pandey edited comment on SPARK-25789 at 2/2/22, 5:50 AM: -- Is

[jira] [Commented] (SPARK-25789) Support for Dataset of Avro

[ https://issues.apache.org/jira/browse/SPARK-25789?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17485580#comment-17485580 ] Prashant Pandey commented on SPARK-25789: - Is there any update on this ticket? We are facing

[jira] [Updated] (SPARK-38078) Aggregation with Watermark in AppendMode is holding data beyong water mark boundary.

[ https://issues.apache.org/jira/browse/SPARK-38078?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] krishna updated SPARK-38078: Description:  I am struggling with a unique issue. I am not sure if my understanding is wrong or this is

[jira] [Commented] (SPARK-38059) Incorrect query ordering with flatMap() and distinct()

[ https://issues.apache.org/jira/browse/SPARK-38059?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17485550#comment-17485550 ] Vu Tan commented on SPARK-38059: I ran your above  Java app and got the below result (on spark 3.2.0)

[jira] [Commented] (SPARK-38086) Make ArrowColumnVector Extendable

[ https://issues.apache.org/jira/browse/SPARK-38086?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17485530#comment-17485530 ] Kazuyuki Tanimura commented on SPARK-38086: --- I am working on this > Make ArrowColumnVector

[jira] [Created] (SPARK-38086) Make ArrowColumnVector Extendable

Kazuyuki Tanimura created SPARK-38086: - Summary: Make ArrowColumnVector Extendable Key: SPARK-38086 URL: https://issues.apache.org/jira/browse/SPARK-38086 Project: Spark Issue Type:

[jira] [Commented] (SPARK-35801) SPIP: Row-level operations in Data Source V2

[ https://issues.apache.org/jira/browse/SPARK-35801?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17485508#comment-17485508 ] L. C. Hsieh commented on SPARK-35801: - I think we can leave this open and put sub-tasks under this,

[jira] [Updated] (SPARK-38047) Add OUTLIER_NO_FALLBACK executor roll policy

[ https://issues.apache.org/jira/browse/SPARK-38047?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-38047: -- Summary: Add OUTLIER_NO_FALLBACK executor roll policy (was: Provide an option to only roll

[jira] [Created] (SPARK-38085) DataSource V2: Handle DELETE commands for group-based sources

Anton Okolnychyi created SPARK-38085: Summary: DataSource V2: Handle DELETE commands for group-based sources Key: SPARK-38085 URL: https://issues.apache.org/jira/browse/SPARK-38085 Project: Spark

[jira] [Updated] (SPARK-35801) SPIP: Row-level operations in Data Source V2

[ https://issues.apache.org/jira/browse/SPARK-35801?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anton Okolnychyi updated SPARK-35801: - Affects Version/s: 3.3.0 (was: 3.2.0) > SPIP: Row-level

[jira] [Commented] (SPARK-35801) SPIP: Row-level operations in Data Source V2

[ https://issues.apache.org/jira/browse/SPARK-35801?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17485502#comment-17485502 ] Anton Okolnychyi commented on SPARK-35801: -- [~viirya], shall we keep this one open until the

[jira] [Updated] (SPARK-35801) SPIP: Row-level operations in Data Source V2

[ https://issues.apache.org/jira/browse/SPARK-35801?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anton Okolnychyi updated SPARK-35801: - Labels: SPIP (was: ) > SPIP: Row-level operations in Data Source V2 >

[jira] [Resolved] (SPARK-38084) Support `SKIP_PYTHON` and `SKIP_R` in `run-tests.py`

[ https://issues.apache.org/jira/browse/SPARK-38084?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-38084. --- Fix Version/s: 3.3.0 3.2.2 Resolution: Fixed Issue resolved by

[jira] [Assigned] (SPARK-38084) Support `SKIP_PYTHON` and `SKIP_R` in `run-tests.py`

[ https://issues.apache.org/jira/browse/SPARK-38084?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun reassigned SPARK-38084: - Assignee: Dongjoon Hyun > Support `SKIP_PYTHON` and `SKIP_R` in `run-tests.py` >

[jira] [Updated] (SPARK-38075) Hive script transform with order by and limit will return fake rows

[ https://issues.apache.org/jira/browse/SPARK-38075?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Holden Karau updated SPARK-38075: - Fix Version/s: 3.1.3 (was: 3.1.4) > Hive script transform with order by

[jira] [Comment Edited] (SPARK-24156) Enable no-data micro batches for more eager streaming state clean up

[ https://issues.apache.org/jira/browse/SPARK-24156?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17483838#comment-17483838 ] krishna edited comment on SPARK-24156 at 2/1/22, 9:04 PM: -- Hi [~kcsrms] [~tdas]

[jira] [Updated] (SPARK-38075) Hive script transform with order by and limit will return fake rows

[ https://issues.apache.org/jira/browse/SPARK-38075?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Holden Karau updated SPARK-38075: - Fix Version/s: 3.1.4 (was: 3.1.3) Affects Version/s: 3.1.3 >

[jira] [Commented] (SPARK-38084) Support `SKIP_PYTHON` and `SKIP_R` in `run-tests.py`

[ https://issues.apache.org/jira/browse/SPARK-38084?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17485430#comment-17485430 ] Apache Spark commented on SPARK-38084: -- User 'dongjoon-hyun' has created a pull request for this

[jira] [Assigned] (SPARK-38084) Support `SKIP_PYTHON` and `SKIP_R` in `run-tests.py`

[ https://issues.apache.org/jira/browse/SPARK-38084?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-38084: Assignee: Apache Spark > Support `SKIP_PYTHON` and `SKIP_R` in `run-tests.py` >

[jira] [Assigned] (SPARK-38084) Support `SKIP_PYTHON` and `SKIP_R` in `run-tests.py`

[ https://issues.apache.org/jira/browse/SPARK-38084?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-38084: Assignee: (was: Apache Spark) > Support `SKIP_PYTHON` and `SKIP_R` in `run-tests.py`

[jira] [Created] (SPARK-38084) Support `SKIP_PYTHON` and `SKIP_R` in `run-tests.py`

Dongjoon Hyun created SPARK-38084: - Summary: Support `SKIP_PYTHON` and `SKIP_R` in `run-tests.py` Key: SPARK-38084 URL: https://issues.apache.org/jira/browse/SPARK-38084 Project: Spark Issue

[jira] [Resolved] (SPARK-38047) Provide an option to only roll executors if they are outliers

[ https://issues.apache.org/jira/browse/SPARK-38047?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-38047. --- Fix Version/s: 3.3.0 Resolution: Fixed Issue resolved by pull request 35373

[jira] [Commented] (SPARK-27826) saveAsTable() function case table have "HiveFileFormat" "ParquetFileFormat" format issue

[ https://issues.apache.org/jira/browse/SPARK-27826?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17485393#comment-17485393 ] Bandhu Gupta commented on SPARK-27826: -- Hi Fengtlyer , We are facing exactly the same issue which

[jira] [Updated] (SPARK-38083) set the amount of explained variance as parameter of pyspark.ml.feature.PCA

[ https://issues.apache.org/jira/browse/SPARK-38083?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nicola updated SPARK-38083: --- Summary: set the amount of explained variance as parameter of pyspark.ml.feature.PCA (was: set the amout

[jira] [Updated] (SPARK-38083) set the amout of explained variance as parameter of pyspark.ml.feature.PCA

[ https://issues.apache.org/jira/browse/SPARK-38083?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nicola updated SPARK-38083: --- Affects Version/s: 3.2.2 (was: 3.2.1) > set the amout of explained variance as

[jira] [Updated] (SPARK-38083) set the amout of explained variance as parameter of pyspark.ml.feature.PCA

[ https://issues.apache.org/jira/browse/SPARK-38083?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nicola updated SPARK-38083: --- Component/s: ML > set the amout of explained variance as parameter of pyspark.ml.feature.PCA >

[jira] [Created] (SPARK-38083) set the amout of explained variance as parameter of pyspark.ml.feature.PCA

Nicola created SPARK-38083: -- Summary: set the amout of explained variance as parameter of pyspark.ml.feature.PCA Key: SPARK-38083 URL: https://issues.apache.org/jira/browse/SPARK-38083 Project: Spark

[jira] [Resolved] (SPARK-38081) Support cloud-backend in K8s IT with SBT

[ https://issues.apache.org/jira/browse/SPARK-38081?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-38081. --- Fix Version/s: 3.3.0 Resolution: Fixed Issue resolved by pull request 35376

[jira] [Assigned] (SPARK-38081) Support cloud-backend in K8s IT with SBT

[ https://issues.apache.org/jira/browse/SPARK-38081?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun reassigned SPARK-38081: - Assignee: Dongjoon Hyun > Support cloud-backend in K8s IT with SBT >

[jira] [Updated] (SPARK-38082) Update minimum numpy version

[ https://issues.apache.org/jira/browse/SPARK-38082?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Maciej Szymkiewicz updated SPARK-38082: --- Description: Currently, we use set numpy version in {{extras_require}} to be

[jira] [Updated] (SPARK-38082) Update minimum numpy version

[ https://issues.apache.org/jira/browse/SPARK-38082?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Maciej Szymkiewicz updated SPARK-38082: --- Description: Currently, we use set numpy version in {{extras_require}} to be

[jira] [Updated] (SPARK-38082) Update minimum numpy version

[ https://issues.apache.org/jira/browse/SPARK-38082?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Maciej Szymkiewicz updated SPARK-38082: --- Description: Currently, we use set numpy version in {{extras_require}} to be

[jira] [Updated] (SPARK-38082) Update minimum numpy version

[ https://issues.apache.org/jira/browse/SPARK-38082?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Maciej Szymkiewicz updated SPARK-38082: --- Description: Currently, we use set numpy version in {{extras_require}} to be

[jira] [Commented] (SPARK-38082) Update minimum numpy version

[ https://issues.apache.org/jira/browse/SPARK-38082?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17485327#comment-17485327 ] Maciej Szymkiewicz commented on SPARK-38082: cc [~hyukjin.kwon] [~WeichenXu123] [~huaxingao]

[jira] [Updated] (SPARK-38082) Update minimum numpy version

[ https://issues.apache.org/jira/browse/SPARK-38082?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Maciej Szymkiewicz updated SPARK-38082: --- Description: Currently, we use set numpy version in {{extras_require}} to be

[jira] [Updated] (SPARK-38082) Update minimum numpy version

[ https://issues.apache.org/jira/browse/SPARK-38082?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Maciej Szymkiewicz updated SPARK-38082: --- Description: Currently, we use set numpy version in {{extras_require}} to be

[jira] [Created] (SPARK-38082) Update minimum numpy version

Maciej Szymkiewicz created SPARK-38082: -- Summary: Update minimum numpy version Key: SPARK-38082 URL: https://issues.apache.org/jira/browse/SPARK-38082 Project: Spark Issue Type:

[jira] [Commented] (SPARK-37417) Inline type hints for python/pyspark/ml/linalg/__init__.py

[ https://issues.apache.org/jira/browse/SPARK-37417?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17485313#comment-17485313 ] Apache Spark commented on SPARK-37417: -- User 'zero323' has created a pull request for this issue:

[jira] [Assigned] (SPARK-37417) Inline type hints for python/pyspark/ml/linalg/__init__.py

[ https://issues.apache.org/jira/browse/SPARK-37417?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-37417: Assignee: Apache Spark > Inline type hints for python/pyspark/ml/linalg/__init__.py >

[jira] [Assigned] (SPARK-37417) Inline type hints for python/pyspark/ml/linalg/__init__.py

[ https://issues.apache.org/jira/browse/SPARK-37417?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-37417: Assignee: (was: Apache Spark) > Inline type hints for

[jira] [Commented] (SPARK-37417) Inline type hints for python/pyspark/ml/linalg/__init__.py

[ https://issues.apache.org/jira/browse/SPARK-37417?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17485312#comment-17485312 ] Apache Spark commented on SPARK-37417: -- User 'zero323' has created a pull request for this issue:

[jira] [Commented] (SPARK-37980) Extend METADATA column to support row indices for file based data sources

[ https://issues.apache.org/jira/browse/SPARK-37980?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17485215#comment-17485215 ] Cheng Lian commented on SPARK-37980: [~prakharjain09], as you've mentioned, it's not super

[jira] [Assigned] (SPARK-38067) Inconsistent missing values handling in Pandas on Spark to_json

[ https://issues.apache.org/jira/browse/SPARK-38067?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Maciej Szymkiewicz reassigned SPARK-38067: -- Assignee: Bjørn Jørgensen > Inconsistent missing values handling in Pandas

[jira] [Resolved] (SPARK-38067) Inconsistent missing values handling in Pandas on Spark to_json

[ https://issues.apache.org/jira/browse/SPARK-38067?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Maciej Szymkiewicz resolved SPARK-38067. Fix Version/s: 3.3.0 Resolution: Fixed Issue resolved by pull request

[jira] [Updated] (SPARK-38067) Inconsistent missing values handling in Pandas on Spark to_json

[ https://issues.apache.org/jira/browse/SPARK-38067?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Maciej Szymkiewicz updated SPARK-38067: --- Description: If {{ps.DataFrame.to_json}} is called without {{path}} argument,

[jira] [Updated] (SPARK-38067) Inconsistent missing values handling in Pandas on Spark to_json

[ https://issues.apache.org/jira/browse/SPARK-38067?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Maciej Szymkiewicz updated SPARK-38067: --- Summary: Inconsistent missing values handling in Pandas on Spark to_json (was: