[jira] [Commented] (SPARK-31891) `ALTER TABLE multipartIdentifier RECOVER PARTITIONS` should drop partition if partition specific location is not exist any more

[ https://issues.apache.org/jira/browse/SPARK-31891?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17289745#comment-17289745 ] Apache Spark commented on SPARK-31891: -- User 'MaxGekk' has created a pull request f

[jira] [Commented] (SPARK-31891) `ALTER TABLE multipartIdentifier RECOVER PARTITIONS` should drop partition if partition specific location is not exist any more

[ https://issues.apache.org/jira/browse/SPARK-31891?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17289744#comment-17289744 ] Apache Spark commented on SPARK-31891: -- User 'MaxGekk' has created a pull request f

[jira] [Commented] (SPARK-33212) Upgrade to Hadoop 3.2.2 and move to shaded clients for Hadoop 3.x profile

[ https://issues.apache.org/jira/browse/SPARK-33212?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17289728#comment-17289728 ] Xiaochen Ouyang commented on SPARK-33212: - Thanks for your reply [~csun] ! Subm

[jira] [Updated] (SPARK-34515) Fix NPE if InSet contains null value during getPartitionsByFilter

[ https://issues.apache.org/jira/browse/SPARK-34515?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Takeshi Yamamuro updated SPARK-34515: - Affects Version/s: 3.1.2 > Fix NPE if InSet contains null value during getPartitionsByFi

[jira] [Resolved] (SPARK-34152) CreateViewStatement.child should be a real child

[ https://issues.apache.org/jira/browse/SPARK-34152?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-34152. - Fix Version/s: 3.2.0 Resolution: Fixed Issue resolved by pull request 31273 [https://gith

[jira] [Resolved] (SPARK-34414) OptimizeMetadataOnlyQuery should only apply for deterministic filters

[ https://issues.apache.org/jira/browse/SPARK-34414?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yesheng Ma resolved SPARK-34414. Resolution: Invalid > OptimizeMetadataOnlyQuery should only apply for deterministic filters >

[jira] [Comment Edited] (SPARK-34516) Spark 3.0.1 encounter parquet PageHerder IO issue

[ https://issues.apache.org/jira/browse/SPARK-34516?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17289680#comment-17289680 ] angerszhu edited comment on SPARK-34516 at 2/24/21, 6:34 AM: -

[jira] [Comment Edited] (SPARK-34516) Spark 3.0.1 encounter parquet PageHerder IO issue

[ https://issues.apache.org/jira/browse/SPARK-34516?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17289684#comment-17289684 ] angerszhu edited comment on SPARK-34516 at 2/24/21, 6:32 AM: -

[jira] [Commented] (SPARK-34516) Spark 3.0.1 encounter parquet PageHerder IO issue

[ https://issues.apache.org/jira/browse/SPARK-34516?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17289684#comment-17289684 ] angerszhu commented on SPARK-34516: --- [~maropu] Sort it out after desensitization and u

[jira] [Commented] (SPARK-34516) Spark 3.0.1 encounter parquet PageHerder IO issue

[ https://issues.apache.org/jira/browse/SPARK-34516?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17289680#comment-17289680 ] angerszhu commented on SPARK-34516: --- For this error , I found some related issue: [ht

[jira] [Commented] (SPARK-34516) Spark 3.0.1 encounter parquet PageHerder IO issue

[ https://issues.apache.org/jira/browse/SPARK-34516?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17289677#comment-17289677 ] Takeshi Yamamuro commented on SPARK-34516: -- What's a query to reproduce this is

[jira] [Created] (SPARK-34516) Spark 3.0.1 encounter parquet PageHerder IO issue

angerszhu created SPARK-34516: - Summary: Spark 3.0.1 encounter parquet PageHerder IO issue Key: SPARK-34516 URL: https://issues.apache.org/jira/browse/SPARK-34516 Project: Spark Issue Type: Impro

[jira] [Updated] (SPARK-34494) Move data source options from Python and Scala into a single page.

[ https://issues.apache.org/jira/browse/SPARK-34494?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-34494: - Affects Version/s: (was: 3.0.2) 3.2.0 > Move data source options from

[jira] [Updated] (SPARK-34493) Create "TEXT Files" page for Data Source documents.

[ https://issues.apache.org/jira/browse/SPARK-34493?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-34493: - Affects Version/s: (was: 3.0.2) 3.2.0 > Create "TEXT Files" page for

[jira] [Updated] (SPARK-34492) Create "CSV Files" page for Data Source documents.

[ https://issues.apache.org/jira/browse/SPARK-34492?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-34492: - Affects Version/s: (was: 3.0.2) 3.2.0 > Create "CSV Files" page for D

[jira] [Resolved] (SPARK-34246) New type coercion syntax rules in ANSI mode

[ https://issues.apache.org/jira/browse/SPARK-34246?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gengliang Wang resolved SPARK-34246. Fix Version/s: 3.2.0 Resolution: Fixed Issue resolved by pull request 31349 [https:

[jira] [Updated] (SPARK-34497) JDBC connection provider is not removing kerberos credentials from JVM security context

[ https://issues.apache.org/jira/browse/SPARK-34497?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Takeshi Yamamuro updated SPARK-34497: - Affects Version/s: 3.2.0 > JDBC connection provider is not removing kerberos credentials

[jira] [Updated] (SPARK-34497) JDBC connection provider is not removing kerberos credentials from JVM security context

[ https://issues.apache.org/jira/browse/SPARK-34497?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Takeshi Yamamuro updated SPARK-34497: - Affects Version/s: (was: 3.1.0) 3.1.2 > JDBC connection provi

[jira] [Commented] (SPARK-34497) JDBC connection provider is not removing kerberos credentials from JVM security context

[ https://issues.apache.org/jira/browse/SPARK-34497?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17289669#comment-17289669 ] Takeshi Yamamuro commented on SPARK-34497: -- Please fill the description. > JDB

[jira] [Commented] (SPARK-34515) Fix NPE if InSet contains null value during getPartitionsByFilter

[ https://issues.apache.org/jira/browse/SPARK-34515?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17289667#comment-17289667 ] Apache Spark commented on SPARK-34515: -- User 'ulysses-you' has created a pull reque

[jira] [Assigned] (SPARK-34515) Fix NPE if InSet contains null value during getPartitionsByFilter

[ https://issues.apache.org/jira/browse/SPARK-34515?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-34515: Assignee: (was: Apache Spark) > Fix NPE if InSet contains null value during getPartit

[jira] [Assigned] (SPARK-34515) Fix NPE if InSet contains null value during getPartitionsByFilter

[ https://issues.apache.org/jira/browse/SPARK-34515?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-34515: Assignee: Apache Spark > Fix NPE if InSet contains null value during getPartitionsByFilte

[jira] [Commented] (SPARK-34515) Fix NPE if InSet contains null value during getPartitionsByFilter

[ https://issues.apache.org/jira/browse/SPARK-34515?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17289665#comment-17289665 ] Apache Spark commented on SPARK-34515: -- User 'ulysses-you' has created a pull reque

[jira] [Resolved] (SPARK-34290) Support v2 TRUNCATE TABLE

[ https://issues.apache.org/jira/browse/SPARK-34290?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-34290. - Fix Version/s: 3.2.0 Resolution: Fixed Issue resolved by pull request 31605 [https://gith

[jira] [Assigned] (SPARK-34290) Support v2 TRUNCATE TABLE

[ https://issues.apache.org/jira/browse/SPARK-34290?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan reassigned SPARK-34290: --- Assignee: Maxim Gekk > Support v2 TRUNCATE TABLE > - > >

[jira] [Resolved] (SPARK-34245) Master may not remove the finished executor when Worker fails to send ExecutorStateChanged

[ https://issues.apache.org/jira/browse/SPARK-34245?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-34245. - Fix Version/s: 3.2.0 Resolution: Fixed Issue resolved by pull request 31348 [https://gith

[jira] [Assigned] (SPARK-34245) Master may not remove the finished executor when Worker fails to send ExecutorStateChanged

[ https://issues.apache.org/jira/browse/SPARK-34245?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan reassigned SPARK-34245: --- Assignee: wuyi > Master may not remove the finished executor when Worker fails to send > E

[jira] [Updated] (SPARK-34515) Fix NPE if InSet contains null value during getPartitionsByFilter

[ https://issues.apache.org/jira/browse/SPARK-34515?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ulysses you updated SPARK-34515: Description: Spark will convert InSet to `>= and <=` if it's values size over `spark.sql.hive.met

[jira] [Commented] (SPARK-33504) The application log in the Spark history server contains sensitive attributes such as password that should be redated instead of plain text

[ https://issues.apache.org/jira/browse/SPARK-33504?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17289653#comment-17289653 ] Apache Spark commented on SPARK-33504: -- User 'viirya' has created a pull request fo

[jira] [Commented] (SPARK-33504) The application log in the Spark history server contains sensitive attributes such as password that should be redated instead of plain text

[ https://issues.apache.org/jira/browse/SPARK-33504?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17289654#comment-17289654 ] Apache Spark commented on SPARK-33504: -- User 'viirya' has created a pull request fo

[jira] [Created] (SPARK-34515) Fix NPE if InSet contains null value during getPartitionsByFilter

ulysses you created SPARK-34515: --- Summary: Fix NPE if InSet contains null value during getPartitionsByFilter Key: SPARK-34515 URL: https://issues.apache.org/jira/browse/SPARK-34515 Project: Spark

[jira] [Commented] (SPARK-33212) Upgrade to Hadoop 3.2.2 and move to shaded clients for Hadoop 3.x profile

[ https://issues.apache.org/jira/browse/SPARK-33212?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17289652#comment-17289652 ] Chao Sun commented on SPARK-33212: -- Thanks for the details [~ouyangxc.zte]! {quote} Ge

[jira] [Commented] (SPARK-34488) Support task Metrics Distributions and executor Metrics Distributions in the REST API call for a specified stage

[ https://issues.apache.org/jira/browse/SPARK-34488?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17289646#comment-17289646 ] Ron Hu commented on SPARK-34488: It should be noted that this Jira  addresses query para

[jira] [Commented] (SPARK-34092) support filtering by task status in REST API call for a specific stage

[ https://issues.apache.org/jira/browse/SPARK-34092?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17289644#comment-17289644 ] Ron Hu commented on SPARK-34092: It should be noted that this Jira  addresses query para

[jira] [Commented] (SPARK-34507) Spark artefacts built against Scala 2.13 incorrectly depend on Scala 2.12

[ https://issues.apache.org/jira/browse/SPARK-34507?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17289638#comment-17289638 ] Yang Jie commented on SPARK-34507: -- It seems that the  [https://repository.apache.org/c

[jira] [Commented] (SPARK-34507) Spark artefacts built against Scala 2.13 incorrectly depend on Scala 2.12

[ https://issues.apache.org/jira/browse/SPARK-34507?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17289637#comment-17289637 ] Yang Jie commented on SPARK-34507: -- I think scala-2.13 profile should overrdie this pro

[jira] [Assigned] (SPARK-34514) Push down limit for LEFT SEMI and LEFT ANTI join

[ https://issues.apache.org/jira/browse/SPARK-34514?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-34514: Assignee: Apache Spark > Push down limit for LEFT SEMI and LEFT ANTI join > -

[jira] [Commented] (SPARK-34514) Push down limit for LEFT SEMI and LEFT ANTI join

[ https://issues.apache.org/jira/browse/SPARK-34514?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17289625#comment-17289625 ] Apache Spark commented on SPARK-34514: -- User 'c21' has created a pull request for t

[jira] [Assigned] (SPARK-34514) Push down limit for LEFT SEMI and LEFT ANTI join

[ https://issues.apache.org/jira/browse/SPARK-34514?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-34514: Assignee: Apache Spark > Push down limit for LEFT SEMI and LEFT ANTI join > -

[jira] [Assigned] (SPARK-34514) Push down limit for LEFT SEMI and LEFT ANTI join

[ https://issues.apache.org/jira/browse/SPARK-34514?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-34514: Assignee: (was: Apache Spark) > Push down limit for LEFT SEMI and LEFT ANTI join > --

[jira] [Comment Edited] (SPARK-33212) Upgrade to Hadoop 3.2.2 and move to shaded clients for Hadoop 3.x profile

[ https://issues.apache.org/jira/browse/SPARK-33212?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17289618#comment-17289618 ] Xiaochen Ouyang edited comment on SPARK-33212 at 2/24/21, 3:33 AM: ---

[jira] [Comment Edited] (SPARK-33212) Upgrade to Hadoop 3.2.2 and move to shaded clients for Hadoop 3.x profile

[ https://issues.apache.org/jira/browse/SPARK-33212?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17289618#comment-17289618 ] Xiaochen Ouyang edited comment on SPARK-33212 at 2/24/21, 3:29 AM: ---

[jira] [Updated] (SPARK-34514) Push down limit for LEFT SEMI and LEFT ANTI join

[ https://issues.apache.org/jira/browse/SPARK-34514?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Cheng Su updated SPARK-34514: - Description: I found out during code review of [https://github.com/apache/spark/pull/31567|https://gith

[jira] [Comment Edited] (SPARK-33212) Upgrade to Hadoop 3.2.2 and move to shaded clients for Hadoop 3.x profile

[ https://issues.apache.org/jira/browse/SPARK-33212?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17289618#comment-17289618 ] Xiaochen Ouyang edited comment on SPARK-33212 at 2/24/21, 3:28 AM: ---

[jira] [Comment Edited] (SPARK-33212) Upgrade to Hadoop 3.2.2 and move to shaded clients for Hadoop 3.x profile

[ https://issues.apache.org/jira/browse/SPARK-33212?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17289618#comment-17289618 ] Xiaochen Ouyang edited comment on SPARK-33212 at 2/24/21, 3:28 AM: ---

[jira] [Commented] (SPARK-33212) Upgrade to Hadoop 3.2.2 and move to shaded clients for Hadoop 3.x profile

[ https://issues.apache.org/jira/browse/SPARK-33212?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17289618#comment-17289618 ] Xiaochen Ouyang commented on SPARK-33212: - Hi [~csun], we submit a spark applica

[jira] [Created] (SPARK-34514) Push down limit for LEFT SEMI and LEFT ANTI join

Cheng Su created SPARK-34514: Summary: Push down limit for LEFT SEMI and LEFT ANTI join Key: SPARK-34514 URL: https://issues.apache.org/jira/browse/SPARK-34514 Project: Spark Issue Type: Improvem

[jira] [Assigned] (SPARK-32703) Enable dictionary filtering for Parquet vectorized reader

[ https://issues.apache.org/jira/browse/SPARK-32703?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-32703: Assignee: Apache Spark > Enable dictionary filtering for Parquet vectorized reader >

[jira] [Assigned] (SPARK-32703) Enable dictionary filtering for Parquet vectorized reader

[ https://issues.apache.org/jira/browse/SPARK-32703?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-32703: Assignee: (was: Apache Spark) > Enable dictionary filtering for Parquet vectorized re

[jira] [Updated] (SPARK-32703) Enable dictionary filtering for Parquet vectorized reader

[ https://issues.apache.org/jira/browse/SPARK-32703?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-32703: - Fix Version/s: (was: 3.2.0) > Enable dictionary filtering for Parquet vectorized reader > --

[jira] [Reopened] (SPARK-32703) Enable dictionary filtering for Parquet vectorized reader

[ https://issues.apache.org/jira/browse/SPARK-32703?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon reopened SPARK-32703: -- Assignee: (was: Chao Sun) Reverted at https://github.com/apache/spark/commit/80bad086c8

[jira] [Updated] (SPARK-34513) Kubernetes Spark Driver Pod Name Length Limitation

[ https://issues.apache.org/jira/browse/SPARK-34513?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] John updated SPARK-34513: - Description: Hi, We are using Spark in Airflow with the k8s-master. Airflow is attaching to our spark-driver p

[jira] [Created] (SPARK-34513) Kubernetes Spark Driver Pod Name Length Limitation

John created SPARK-34513: Summary: Kubernetes Spark Driver Pod Name Length Limitation Key: SPARK-34513 URL: https://issues.apache.org/jira/browse/SPARK-34513 Project: Spark Issue Type: Bug

[jira] [Commented] (SPARK-25390) Data source V2 API refactoring

[ https://issues.apache.org/jira/browse/SPARK-25390?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17289577#comment-17289577 ] Rafael commented on SPARK-25390: Sorry for late response, I was able to migrate my proj

[jira] [Commented] (SPARK-33044) Add a Jenkins build and test job for Scala 2.13

[ https://issues.apache.org/jira/browse/SPARK-33044?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17289542#comment-17289542 ] shane knapp commented on SPARK-33044: - hey! sorry, i've been pretty slammed these p

[jira] [Commented] (SPARK-33044) Add a Jenkins build and test job for Scala 2.13

[ https://issues.apache.org/jira/browse/SPARK-33044?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17289527#comment-17289527 ] shane knapp commented on SPARK-33044: - 1) logins are temporarily disabled due to new

[jira] [Resolved] (SPARK-26138) Pushdown limit through InnerLike when condition is empty

[ https://issues.apache.org/jira/browse/SPARK-26138?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuming Wang resolved SPARK-26138. - Fix Version/s: 3.2.0 Resolution: Fixed Issue resolved by pull request 31567 [https://gith

[jira] [Commented] (SPARK-32195) Standardize warning types and messages

[ https://issues.apache.org/jira/browse/SPARK-32195?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17289505#comment-17289505 ] RISHAV DUTTA commented on SPARK-32195: -- I am working on it. Can you takeup another

[jira] [Assigned] (SPARK-26138) Pushdown limit through InnerLike when condition is empty

[ https://issues.apache.org/jira/browse/SPARK-26138?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuming Wang reassigned SPARK-26138: --- Assignee: Yuming Wang > Pushdown limit through InnerLike when condition is empty >

[jira] [Commented] (SPARK-33044) Add a Jenkins build and test job for Scala 2.13

[ https://issues.apache.org/jira/browse/SPARK-33044?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17289470#comment-17289470 ] shane knapp commented on SPARK-33044: - welp, i manually created a test job and it fa

[jira] [Resolved] (SPARK-34503) Use zstd for spark.eventLog.compression.codec by default

[ https://issues.apache.org/jira/browse/SPARK-34503?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-34503. --- Fix Version/s: 3.2.0 Resolution: Fixed Issue resolved by pull request 31618 [https://

[jira] [Assigned] (SPARK-34503) Use zstd for spark.eventLog.compression.codec by default

[ https://issues.apache.org/jira/browse/SPARK-34503?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun reassigned SPARK-34503: - Assignee: Dongjoon Hyun > Use zstd for spark.eventLog.compression.codec by default > --

[jira] [Updated] (SPARK-34512) Disable validate default values when parsing Avro schemas

[ https://issues.apache.org/jira/browse/SPARK-34512?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuming Wang updated SPARK-34512: Description: This is a regression problem. How to reproduce this issue: {code:scala} // Add this

[jira] [Updated] (SPARK-34512) Disable validate default values when parsing Avro schemas

[ https://issues.apache.org/jira/browse/SPARK-34512?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuming Wang updated SPARK-34512: Description: How to reproduce this issue: {code:scala} // Add this test to HiveSerDeReadWriteSui

[jira] [Created] (SPARK-34512) Disable validate default values when parsing Avro schemas

Yuming Wang created SPARK-34512: --- Summary: Disable validate default values when parsing Avro schemas Key: SPARK-34512 URL: https://issues.apache.org/jira/browse/SPARK-34512 Project: Spark Issue

[jira] [Assigned] (SPARK-31891) `ALTER TABLE multipartIdentifier RECOVER PARTITIONS` should drop partition if partition specific location is not exist any more

[ https://issues.apache.org/jira/browse/SPARK-31891?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun reassigned SPARK-31891: - Assignee: Maxim Gekk > `ALTER TABLE multipartIdentifier RECOVER PARTITIONS` should drop

[jira] [Resolved] (SPARK-31891) `ALTER TABLE multipartIdentifier RECOVER PARTITIONS` should drop partition if partition specific location is not exist any more

[ https://issues.apache.org/jira/browse/SPARK-31891?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-31891. --- Fix Version/s: 3.2.0 Resolution: Fixed Issue resolved by pull request 31499 [https://

[jira] [Resolved] (SPARK-34508) skip HiveExternalCatalogVersionsSuite if network is down

[ https://issues.apache.org/jira/browse/SPARK-34508?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-34508. --- Fix Version/s: 3.2.0 3.1.1 Resolution: Fixed Issue resolved by pul

[jira] [Assigned] (SPARK-34508) skip HiveExternalCatalogVersionsSuite if network is down

[ https://issues.apache.org/jira/browse/SPARK-34508?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun reassigned SPARK-34508: - Assignee: Wenchen Fan > skip HiveExternalCatalogVersionsSuite if network is down >

[jira] [Commented] (SPARK-34511) Current Security vulnerabilities in spark libraries

[ https://issues.apache.org/jira/browse/SPARK-34511?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17289339#comment-17289339 ] Dongjoon Hyun commented on SPARK-34511: --- Hi, [~emac3060] . This is outdated alread

[jira] [Updated] (SPARK-34511) Current Security vulnerabilities in spark libraries

[ https://issues.apache.org/jira/browse/SPARK-34511?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-34511: -- Flags: (was: Patch) > Current Security vulnerabilities in spark libraries >

[jira] [Updated] (SPARK-34511) Current Security vulnerabilities in spark libraries

[ https://issues.apache.org/jira/browse/SPARK-34511?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-34511: -- Component/s: (was: Spark Core) Build > Current Security vulnerabilities i

[jira] [Updated] (SPARK-34510) .foreachPartition command hangs when ran inside Python package but works when ran from Python file outside the package on EMR

[ https://issues.apache.org/jira/browse/SPARK-34510?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuriy updated SPARK-34510: -- Description: I'm running on EMR Pyspark 3.0.0. with project structure below, process.py is what controls the

[jira] [Updated] (SPARK-34510) .foreachPartition command hangs when ran inside Python package but works when ran from Python file outside the package on EMR

[ https://issues.apache.org/jira/browse/SPARK-34510?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuriy updated SPARK-34510: -- Description: I'm running on EMR Pyspark 3.0.0. with a project structure below, process.py is what controls th

[jira] [Updated] (SPARK-34510) .foreachPartition command hangs when ran inside Python package but works when ran from Python file outside the package on EMR

[ https://issues.apache.org/jira/browse/SPARK-34510?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuriy updated SPARK-34510: -- Description: I'm running on EMR Pyspark 3.0.0. with a project structure below, process.py is what controls th

[jira] [Updated] (SPARK-34510) .foreachPartition command hangs when ran inside Python package but works when ran from Python file outside the package on EMR

[ https://issues.apache.org/jira/browse/SPARK-34510?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuriy updated SPARK-34510: -- Description: I'm running on EMR Pyspark 3.0.0. with a project structure below, process.py is what controls th

[jira] [Updated] (SPARK-34510) .foreachPartition command hangs when ran inside Python package but works when ran from Python file outside the package on EMR

[ https://issues.apache.org/jira/browse/SPARK-34510?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuriy updated SPARK-34510: -- Description: I'm running on EMR Pyspark 3.0.0. with a project structure below, process.py is what controls th

[jira] [Assigned] (SPARK-34502) Remove unused parameters in join methods

[ https://issues.apache.org/jira/browse/SPARK-34502?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] L. C. Hsieh reassigned SPARK-34502: --- Assignee: Huaxin Gao > Remove unused parameters in join methods > -

[jira] [Resolved] (SPARK-34502) Remove unused parameters in join methods

[ https://issues.apache.org/jira/browse/SPARK-34502?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] L. C. Hsieh resolved SPARK-34502. - Fix Version/s: 3.2.0 Resolution: Fixed Issue resolved by pull request 31617 [https://gith

[jira] [Assigned] (SPARK-34380) Support ifExists for ALTER TABLE ... UNSET TBLPROPERTIES

[ https://issues.apache.org/jira/browse/SPARK-34380?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-34380: Assignee: Terry Kim (was: Apache Spark) > Support ifExists for ALTER TABLE ... UNSET TBL

[jira] [Assigned] (SPARK-34380) Support ifExists for ALTER TABLE ... UNSET TBLPROPERTIES

[ https://issues.apache.org/jira/browse/SPARK-34380?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-34380: Assignee: Apache Spark (was: Terry Kim) > Support ifExists for ALTER TABLE ... UNSET TBL

[jira] [Reopened] (SPARK-34380) Support ifExists for ALTER TABLE ... UNSET TBLPROPERTIES

[ https://issues.apache.org/jira/browse/SPARK-34380?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan reopened SPARK-34380: - > Support ifExists for ALTER TABLE ... UNSET TBLPROPERTIES > ---

[jira] [Updated] (SPARK-34380) Support ifExists for ALTER TABLE ... UNSET TBLPROPERTIES

[ https://issues.apache.org/jira/browse/SPARK-34380?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan updated SPARK-34380: Fix Version/s: (was: 3.2.0) > Support ifExists for ALTER TABLE ... UNSET TBLPROPERTIES > -

[jira] [Commented] (SPARK-34472) SparkContext.addJar with an ivy path fails in cluster mode with a custom ivySettings file

[ https://issues.apache.org/jira/browse/SPARK-34472?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17289258#comment-17289258 ] Shardul Mahadik commented on SPARK-34472: - [~xkrogen] raised a good point at ht

[jira] [Created] (SPARK-34511) Current Security vulnerabilities in spark libraries

eoin created SPARK-34511: Summary: Current Security vulnerabilities in spark libraries Key: SPARK-34511 URL: https://issues.apache.org/jira/browse/SPARK-34511 Project: Spark Issue Type: Dependency up

[jira] [Updated] (SPARK-34510) .foreachPartition command hangs when ran inside Python package but works when ran from Python file outside the package on EMR

[ https://issues.apache.org/jira/browse/SPARK-34510?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuriy updated SPARK-34510: -- Summary: .foreachPartition command hangs when ran inside Python package but works when ran from Python file ou

[jira] [Updated] (SPARK-34510) .foreachPartition command hangs when ran inside Python package but works when ran from Python file outside the package

[ https://issues.apache.org/jira/browse/SPARK-34510?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuriy updated SPARK-34510: -- Attachment: Code.zip > .foreachPartition command hangs when ran inside Python package but works when > ran fr

[jira] [Commented] (SPARK-33212) Upgrade to Hadoop 3.2.2 and move to shaded clients for Hadoop 3.x profile

[ https://issues.apache.org/jira/browse/SPARK-33212?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17289200#comment-17289200 ] Chao Sun commented on SPARK-33212: -- Thanks for the report [~ouyangxc.zte]. Can you prov

[jira] [Created] (SPARK-34510) .foreachPartition command hangs when ran inside Python package but works when ran from Python file outside the package

Yuriy created SPARK-34510: - Summary: .foreachPartition command hangs when ran inside Python package but works when ran from Python file outside the package Key: SPARK-34510 URL: https://issues.apache.org/jira/browse/SPARK

[jira] [Commented] (SPARK-34509) Make dynamic allocation upscaling more progressive on K8S

[ https://issues.apache.org/jira/browse/SPARK-34509?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17289158#comment-17289158 ] Attila Zsolt Piros commented on SPARK-34509: I am working on this. > Make d

[jira] [Created] (SPARK-34509) Make dynamic allocation upscaling more progressive on K8S

Attila Zsolt Piros created SPARK-34509: -- Summary: Make dynamic allocation upscaling more progressive on K8S Key: SPARK-34509 URL: https://issues.apache.org/jira/browse/SPARK-34509 Project: Spark

[jira] [Commented] (SPARK-34508) skip HiveExternalCatalogVersionsSuite if network is down

[ https://issues.apache.org/jira/browse/SPARK-34508?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17289140#comment-17289140 ] Apache Spark commented on SPARK-34508: -- User 'cloud-fan' has created a pull request

[jira] [Assigned] (SPARK-34508) skip HiveExternalCatalogVersionsSuite if network is down

[ https://issues.apache.org/jira/browse/SPARK-34508?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-34508: Assignee: (was: Apache Spark) > skip HiveExternalCatalogVersionsSuite if network is d

[jira] [Assigned] (SPARK-34508) skip HiveExternalCatalogVersionsSuite if network is down

[ https://issues.apache.org/jira/browse/SPARK-34508?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-34508: Assignee: Apache Spark > skip HiveExternalCatalogVersionsSuite if network is down > -

[jira] [Created] (SPARK-34508) skip HiveExternalCatalogVersionsSuite if network is down

Wenchen Fan created SPARK-34508: --- Summary: skip HiveExternalCatalogVersionsSuite if network is down Key: SPARK-34508 URL: https://issues.apache.org/jira/browse/SPARK-34508 Project: Spark Issue

[jira] [Comment Edited] (SPARK-34448) Binary logistic regression incorrectly computes the intercept and coefficients when data is not centered

[ https://issues.apache.org/jira/browse/SPARK-34448?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17289113#comment-17289113 ] Yakov Kerzhner edited comment on SPARK-34448 at 2/23/21, 2:54 PM:

[jira] [Commented] (SPARK-34168) Support DPP in AQE When the join is Broadcast hash join before applying the AQE rules

[ https://issues.apache.org/jira/browse/SPARK-34168?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17289116#comment-17289116 ] Apache Spark commented on SPARK-34168: -- User 'cloud-fan' has created a pull request

[jira] [Commented] (SPARK-34448) Binary logistic regression incorrectly computes the intercept and coefficients when data is not centered

[ https://issues.apache.org/jira/browse/SPARK-34448?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17289113#comment-17289113 ] Yakov Kerzhner commented on SPARK-34448: As I said in the description, I do not

[jira] [Created] (SPARK-34507) Spark artefacts built against Scala 2.13 incorrectly depend on Scala 2.12

Guillaume Martres created SPARK-34507: - Summary: Spark artefacts built against Scala 2.13 incorrectly depend on Scala 2.12 Key: SPARK-34507 URL: https://issues.apache.org/jira/browse/SPARK-34507 P

[jira] [Commented] (SPARK-25075) Build and test Spark against Scala 2.13

[ https://issues.apache.org/jira/browse/SPARK-25075?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17289104#comment-17289104 ] Guillaume Martres commented on SPARK-25075: --- I've opened https://issues.apache

  1   2   >