[jira] [Updated] (SPARK-36701) Structured streaming maxOffsetsPerTrigger Invalidation

2021-09-08 Thread liujinhui (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36701?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] liujinhui updated SPARK-36701: -- Attachment: image-2021-09-09-13-57-21-175.png > Structured streaming maxOffsetsPerTrigger

[jira] [Updated] (SPARK-36701) Structured streaming maxOffsetsPerTrigger Invalidation

2021-09-08 Thread liujinhui (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36701?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] liujinhui updated SPARK-36701: -- Description: Why does maxOffsetsPerTrigger not work when consuming Kafka, the task fails, and the

[jira] [Updated] (SPARK-36701) Structured streaming maxOffsetsPerTrigger Invalidation

2021-09-08 Thread liujinhui (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36701?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] liujinhui updated SPARK-36701: -- Description: Why does maxOffsetsPerTrigger not work when consuming Kafka, the task fails, and the

[jira] [Updated] (SPARK-36701) Structured streaming maxOffsetsPerTrigger Invalidation

2021-09-08 Thread liujinhui (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36701?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] liujinhui updated SPARK-36701: -- Description: Why does maxOffsetsPerTrigger not work when consuming Kafka, the task fails, and the

[jira] [Created] (SPARK-36701) Structured streaming maxOffsetsPerTrigger Invalidation

2021-09-08 Thread liujinhui (Jira)
liujinhui created SPARK-36701: - Summary: Structured streaming maxOffsetsPerTrigger Invalidation Key: SPARK-36701 URL: https://issues.apache.org/jira/browse/SPARK-36701 Project: Spark Issue

[jira] [Commented] (SPARK-34479) Add zstandard codec to spark.sql.avro.compression.codec

2021-09-08 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34479?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17412337#comment-17412337 ] Apache Spark commented on SPARK-34479: -- User 'viirya' has created a pull request for this issue:

[jira] [Commented] (SPARK-35011) Avoid Block Manager registerations when StopExecutor msg is in-flight.

2021-09-08 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35011?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17412331#comment-17412331 ] Apache Spark commented on SPARK-35011: -- User 'Ngone51' has created a pull request for this issue:

[jira] [Commented] (SPARK-35011) Avoid Block Manager registerations when StopExecutor msg is in-flight.

2021-09-08 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35011?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17412330#comment-17412330 ] Apache Spark commented on SPARK-35011: -- User 'Ngone51' has created a pull request for this issue:

[jira] [Resolved] (SPARK-36674) Support ILIKE - case insensitive Like

2021-09-08 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36674?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-36674. - Fix Version/s: 3.3.0 Resolution: Fixed > Support ILIKE - case insensitive Like >

[jira] [Assigned] (SPARK-36686) Fix SimplifyConditionalsInPredicate to be null-safe

2021-09-08 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36686?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan reassigned SPARK-36686: --- Assignee: Andrew > Fix SimplifyConditionalsInPredicate to be null-safe >

[jira] [Resolved] (SPARK-36686) Fix SimplifyConditionalsInPredicate to be null-safe

2021-09-08 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36686?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-36686. - Fix Version/s: 3.2.0 Resolution: Fixed Issue resolved by pull request 33928

[jira] [Assigned] (SPARK-36699) Reuse compatible executors for stage-level scheduling

2021-09-08 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36699?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-36699: Assignee: (was: Apache Spark) > Reuse compatible executors for stage-level

[jira] [Assigned] (SPARK-36699) Reuse compatible executors for stage-level scheduling

2021-09-08 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36699?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-36699: Assignee: Apache Spark > Reuse compatible executors for stage-level scheduling >

[jira] [Commented] (SPARK-36699) Reuse compatible executors for stage-level scheduling

2021-09-08 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36699?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17412297#comment-17412297 ] Apache Spark commented on SPARK-36699: -- User 'xwu99' has created a pull request for this issue:

[jira] [Commented] (SPARK-36669) Fail to load Lz4 codec

2021-09-08 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36669?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17412295#comment-17412295 ] Apache Spark commented on SPARK-36669: -- User 'viirya' has created a pull request for this issue:

[jira] [Commented] (SPARK-36669) Fail to load Lz4 codec

2021-09-08 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36669?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17412294#comment-17412294 ] Apache Spark commented on SPARK-36669: -- User 'viirya' has created a pull request for this issue:

[jira] [Commented] (SPARK-36700) BlockManager re-registration is broken due to deferred removal of BlockManager

2021-09-08 Thread wuyi (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36700?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17412290#comment-17412290 ] wuyi commented on SPARK-36700: -- I'm working on the fix. > BlockManager re-registration is broken due to

[jira] [Updated] (SPARK-36699) Reuse compatible executors for stage-level scheduling

2021-09-08 Thread Xiaochang Wu (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36699?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiaochang Wu updated SPARK-36699: - Description: Current stage-level scheduling allocated separated set of executors for different

[jira] [Updated] (SPARK-36699) Reuse compatible executors for stage-level scheduling

2021-09-08 Thread Xiaochang Wu (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36699?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiaochang Wu updated SPARK-36699: - Description: Current stage-level scheduling allocated separated set of executors for different

[jira] [Commented] (SPARK-36700) BlockManager re-registration is broken due to deferred removal of BlockManager

2021-09-08 Thread wuyi (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36700?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17412285#comment-17412285 ] wuyi commented on SPARK-36700: -- cc [~mridulm80] [~sumeet.gajjar] [~attilapiros] cc [~gengliang] for the

[jira] [Created] (SPARK-36700) BlockManager re-registration is broken due to deferred removal of BlockManager

2021-09-08 Thread wuyi (Jira)
wuyi created SPARK-36700: Summary: BlockManager re-registration is broken due to deferred removal of BlockManager Key: SPARK-36700 URL: https://issues.apache.org/jira/browse/SPARK-36700 Project: Spark

[jira] [Commented] (SPARK-36670) Add end-to-end codec test cases for main datasources

2021-09-08 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36670?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17412280#comment-17412280 ] Apache Spark commented on SPARK-36670: -- User 'viirya' has created a pull request for this issue:

[jira] [Created] (SPARK-36699) Reuse compatible executors for stage-level scheduling

2021-09-08 Thread Xiaochang Wu (Jira)
Xiaochang Wu created SPARK-36699: Summary: Reuse compatible executors for stage-level scheduling Key: SPARK-36699 URL: https://issues.apache.org/jira/browse/SPARK-36699 Project: Spark Issue

[jira] [Commented] (SPARK-36681) Fail to load Snappy codec

2021-09-08 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36681?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17412274#comment-17412274 ] Hyukjin Kwon commented on SPARK-36681: -- (y) > Fail to load Snappy codec >

[jira] [Commented] (SPARK-36694) Unable to build Spark on Azure DevOps with ubuntu-latest

2021-09-08 Thread Kousuke Saruta (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36694?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17412272#comment-17412272 ] Kousuke Saruta commented on SPARK-36694: [~LvffY] Could you make sure that Java8 is used to

[jira] [Commented] (SPARK-36681) Fail to load Snappy codec

2021-09-08 Thread L. C. Hsieh (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36681?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17412273#comment-17412273 ] L. C. Hsieh commented on SPARK-36681: - After discussing with some PMC members, as this only affects

[jira] [Updated] (SPARK-36681) Fail to load Snappy codec

2021-09-08 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36681?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-36681: - Target Version/s: (was: 3.2.0) > Fail to load Snappy codec > - > >

[jira] [Updated] (SPARK-36681) Fail to load Snappy codec

2021-09-08 Thread L. C. Hsieh (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36681?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] L. C. Hsieh updated SPARK-36681: Priority: Major (was: Blocker) > Fail to load Snappy codec > - > >

[jira] [Resolved] (SPARK-36692) Improve Error statement when requesting thread dump while executor already stopped

2021-09-08 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36692?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-36692. -- Fix Version/s: 3.3.0 Resolution: Fixed Issue resolved by pull request 33935

[jira] [Assigned] (SPARK-36692) Improve Error statement when requesting thread dump while executor already stopped

2021-09-08 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36692?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon reassigned SPARK-36692: Assignee: angerszhu > Improve Error statement when requesting thread dump while executor

[jira] [Resolved] (SPARK-36697) Fix dropping all columns of a DataFrame

2021-09-08 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36697?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-36697. -- Fix Version/s: 3.2.0 Resolution: Fixed Issue resolved by pull request 33938

[jira] [Assigned] (SPARK-36697) Fix dropping all columns of a DataFrame

2021-09-08 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36697?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon reassigned SPARK-36697: Assignee: Xinrong Meng > Fix dropping all columns of a DataFrame >

[jira] [Updated] (SPARK-36698) Allow expand 'quotedRegexColumnNames' in all expressions when it’s expanded to one named expression

2021-09-08 Thread Pablo Langa Blanco (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36698?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Pablo Langa Blanco updated SPARK-36698: --- Description: With {code:java} spark.sql.parser.quotedRegexColumnNames=true{code}

[jira] [Resolved] (SPARK-36661) Support TimestampNTZ in Py4J

2021-09-08 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36661?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-36661. -- Fix Version/s: 3.3.0 Resolution: Fixed Issue resolved by pull request 33877

[jira] [Assigned] (SPARK-36661) Support TimestampNTZ in Py4J

2021-09-08 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36661?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon reassigned SPARK-36661: Assignee: Hyukjin Kwon > Support TimestampNTZ in Py4J > > >

[jira] [Resolved] (SPARK-36625) Support TimestampNTZ in pandas API on Spark

2021-09-08 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36625?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-36625. -- Fix Version/s: 3.3.0 Resolution: Fixed Issue resolved by pull request 33877

[jira] [Assigned] (SPARK-36625) Support TimestampNTZ in pandas API on Spark

2021-09-08 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36625?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon reassigned SPARK-36625: Assignee: Hyukjin Kwon > Support TimestampNTZ in pandas API on Spark >

[jira] [Commented] (SPARK-36698) Allow expand 'quotedRegexColumnNames' in all expressions when it’s expanded to one named expression

2021-09-08 Thread Pablo Langa Blanco (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36698?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17412220#comment-17412220 ] Pablo Langa Blanco commented on SPARK-36698: I'm working on it > Allow expand

[jira] [Created] (SPARK-36698) Allow expand 'quotedRegexColumnNames' in all expressions when it’s expanded to one named expression

2021-09-08 Thread Pablo Langa Blanco (Jira)
Pablo Langa Blanco created SPARK-36698: -- Summary: Allow expand 'quotedRegexColumnNames' in all expressions when it’s expanded to one named expression Key: SPARK-36698 URL:

[jira] [Assigned] (SPARK-36697) Fix dropping all columns of a DataFrame

2021-09-08 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36697?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-36697: Assignee: Apache Spark > Fix dropping all columns of a DataFrame >

[jira] [Commented] (SPARK-36697) Fix dropping all columns of a DataFrame

2021-09-08 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36697?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17412193#comment-17412193 ] Apache Spark commented on SPARK-36697: -- User 'xinrong-databricks' has created a pull request for

[jira] [Assigned] (SPARK-36697) Fix dropping all columns of a DataFrame

2021-09-08 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36697?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-36697: Assignee: (was: Apache Spark) > Fix dropping all columns of a DataFrame >

[jira] [Commented] (SPARK-34276) Check the unreleased/unresolved JIRAs/PRs of Parquet 1.11 and 1.12

2021-09-08 Thread Takuya Ueshin (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34276?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17412183#comment-17412183 ] Takuya Ueshin commented on SPARK-34276: --- SPARK-36696 seems to be the actual issue caused by 

[jira] [Commented] (SPARK-36697) Fix dropping all columns of a DataFrame

2021-09-08 Thread Xinrong Meng (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36697?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17412179#comment-17412179 ] Xinrong Meng commented on SPARK-36697: -- I am working on that now. > Fix dropping all columns of a

[jira] [Created] (SPARK-36697) Fix dropping all columns of a DataFrame

2021-09-08 Thread Xinrong Meng (Jira)
Xinrong Meng created SPARK-36697: Summary: Fix dropping all columns of a DataFrame Key: SPARK-36697 URL: https://issues.apache.org/jira/browse/SPARK-36697 Project: Spark Issue Type: Sub-task

[jira] [Commented] (SPARK-36696) spark.read.parquet loads empty dataset

2021-09-08 Thread Chao Sun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36696?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17412167#comment-17412167 ] Chao Sun commented on SPARK-36696: --

[jira] [Commented] (SPARK-36696) spark.read.parquet loads empty dataset

2021-09-08 Thread Chao Sun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36696?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17412164#comment-17412164 ] Chao Sun commented on SPARK-36696: -- This looks like the same issue as in PARQUET-2078. The file offset

[jira] [Updated] (SPARK-36696) spark.read.parquet loads empty dataset

2021-09-08 Thread Xiao Li (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36696?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Li updated SPARK-36696: Target Version/s: 3.2.0 > spark.read.parquet loads empty dataset > --

[jira] [Updated] (SPARK-36696) spark.read.parquet loads empty dataset

2021-09-08 Thread Xiao Li (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36696?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Li updated SPARK-36696: Priority: Blocker (was: Major) > spark.read.parquet loads empty dataset >

[jira] [Updated] (SPARK-36696) spark.read.parquet loads empty dataset

2021-09-08 Thread Takuya Ueshin (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36696?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Takuya Ueshin updated SPARK-36696: -- Attachment: example.parquet > spark.read.parquet loads empty dataset >

[jira] [Created] (SPARK-36696) spark.read.parquet loads empty dataset

2021-09-08 Thread Takuya Ueshin (Jira)
Takuya Ueshin created SPARK-36696: - Summary: spark.read.parquet loads empty dataset Key: SPARK-36696 URL: https://issues.apache.org/jira/browse/SPARK-36696 Project: Spark Issue Type: Bug

[jira] [Assigned] (SPARK-35822) Spark UI-Executor tab is empty in IE11

2021-09-08 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35822?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-35822: Assignee: Apache Spark > Spark UI-Executor tab is empty in IE11 >

[jira] [Assigned] (SPARK-35822) Spark UI-Executor tab is empty in IE11

2021-09-08 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35822?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-35822: Assignee: (was: Apache Spark) > Spark UI-Executor tab is empty in IE11 >

[jira] [Commented] (SPARK-35822) Spark UI-Executor tab is empty in IE11

2021-09-08 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35822?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17412089#comment-17412089 ] Apache Spark commented on SPARK-35822: -- User 'dgd-contributor' has created a pull request for this

[jira] [Commented] (SPARK-35823) Spark UI-Stages DAG visualization is empty in IE11

2021-09-08 Thread dgd_contributor (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35823?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17412080#comment-17412080 ] dgd_contributor commented on SPARK-35823: - I'm working on this. > Spark UI-Stages DAG

[jira] [Commented] (SPARK-35822) Spark UI-Executor tab is empty in IE11

2021-09-08 Thread dgd_contributor (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35822?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17412079#comment-17412079 ] dgd_contributor commented on SPARK-35822: - I'm working on this. > Spark UI-Executor tab is

[jira] [Created] (SPARK-36695) Allow passing V2 functions to data sources via V2 filters

2021-09-08 Thread Chao Sun (Jira)
Chao Sun created SPARK-36695: Summary: Allow passing V2 functions to data sources via V2 filters Key: SPARK-36695 URL: https://issues.apache.org/jira/browse/SPARK-36695 Project: Spark Issue

[jira] [Commented] (SPARK-33772) Build and Run Spark on Java 17

2021-09-08 Thread Lukas Rytz (Jira)
[ https://issues.apache.org/jira/browse/SPARK-33772?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17411984#comment-17411984 ] Lukas Rytz commented on SPARK-33772: > Scala 2.12 doesn't seem to fit well because there are some

[jira] [Commented] (SPARK-33772) Build and Run Spark on Java 17

2021-09-08 Thread Lukas Rytz (Jira)
[ https://issues.apache.org/jira/browse/SPARK-33772?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17411980#comment-17411980 ] Lukas Rytz commented on SPARK-33772: > JDK8 will fail at startup if you give `--add-opens`. The

[jira] [Updated] (SPARK-36671) Support Series.__and__ for Integral

2021-09-08 Thread dgd_contributor (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36671?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] dgd_contributor updated SPARK-36671: Summary: Support Series.__and__ for Integral (was: Support __and__ in num_ops.py) >

[jira] [Updated] (SPARK-36694) Unable to build Spark on Azure DevOps with ubuntu-latest

2021-09-08 Thread Armand BERGES (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36694?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Armand BERGES updated SPARK-36694: -- Description: Hello With my team we're currently triying to set up some test environment by

[jira] [Created] (SPARK-36694) Unable to build Spark on Azure DevOps with ubuntu-latest

2021-09-08 Thread Armand BERGES (Jira)
Armand BERGES created SPARK-36694: - Summary: Unable to build Spark on Azure DevOps with ubuntu-latest Key: SPARK-36694 URL: https://issues.apache.org/jira/browse/SPARK-36694 Project: Spark

[jira] [Commented] (SPARK-36622) spark.history.kerberos.principal doesn't take value _HOST

2021-09-08 Thread pralabhkumar (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36622?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17411859#comment-17411859 ] pralabhkumar commented on SPARK-36622: -- [~angerszhuuu] [~tgraves]   Please review the PR  >

[jira] [Updated] (SPARK-36575) Executor lost may cause spark stage to hang

2021-09-08 Thread hujiahua (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36575?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] hujiahua updated SPARK-36575: - Description: When a executor finished a task of some stage, the driver will receive a `StatusUpdate`

[jira] [Updated] (SPARK-36575) Executor lost may cause spark stage to hang

2021-09-08 Thread hujiahua (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36575?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] hujiahua updated SPARK-36575: - Description: When a executor finished a task of some stage, the driver will receive a `StatusUpdate`

[jira] [Updated] (SPARK-36575) Executor lost may cause spark stage to hang

2021-09-08 Thread hujiahua (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36575?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] hujiahua updated SPARK-36575: - Description: When a executor finished a task of some stage, the driver will receive a `StatusUpdate`

[jira] [Commented] (SPARK-36683) Support secent and cosecent

2021-09-08 Thread Kousuke Saruta (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36683?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17411808#comment-17411808 ] Kousuke Saruta commented on SPARK-36683: [~YActs] This issue is still open so let's not set Fix

[jira] [Updated] (SPARK-36683) Support secent and cosecent

2021-09-08 Thread Kousuke Saruta (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36683?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kousuke Saruta updated SPARK-36683: --- Fix Version/s: (was: 3.3.0) > Support secent and cosecent > ---

[jira] [Commented] (SPARK-36693) Implement spark-shell idle timeouts

2021-09-08 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36693?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17411789#comment-17411789 ] Apache Spark commented on SPARK-36693: -- User 'gyogal' has created a pull request for this issue:

[jira] [Assigned] (SPARK-36693) Implement spark-shell idle timeouts

2021-09-08 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36693?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-36693: Assignee: (was: Apache Spark) > Implement spark-shell idle timeouts >

[jira] [Assigned] (SPARK-36693) Implement spark-shell idle timeouts

2021-09-08 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36693?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-36693: Assignee: Apache Spark > Implement spark-shell idle timeouts >

[jira] [Commented] (SPARK-36693) Implement spark-shell idle timeouts

2021-09-08 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36693?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17411788#comment-17411788 ] Apache Spark commented on SPARK-36693: -- User 'gyogal' has created a pull request for this issue:

[jira] [Updated] (SPARK-36686) Fix SimplifyConditionalsInPredicate to be null-safe

2021-09-08 Thread Andrew (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36686?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew updated SPARK-36686: --- Description: SimplifyConditionalsInPredicate rule is not null-safe and leads to incorrect results (was:

[jira] [Updated] (SPARK-36686) Fix SimplifyConditionalsInPredicate to be null-safe

2021-09-08 Thread Andrew (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36686?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew updated SPARK-36686: --- Description: SimplifyConditionalsInPredicate rule is not null-safe and leads to incorrect results  

[jira] [Updated] (SPARK-36686) Fix SimplifyConditionalsInPredicate to be null-safe

2021-09-08 Thread Andrew (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36686?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew updated SPARK-36686: --- Description: SimplifyConditionalsInPredicate rule is not null-safe and leads to incorrect results  

[jira] [Created] (SPARK-36693) Implement spark-shell idle timeouts

2021-09-08 Thread Gyorgy Gal (Jira)
Gyorgy Gal created SPARK-36693: -- Summary: Implement spark-shell idle timeouts Key: SPARK-36693 URL: https://issues.apache.org/jira/browse/SPARK-36693 Project: Spark Issue Type: New Feature

[jira] [Assigned] (SPARK-36692) Improve Error statement when requesting thread dump while executor already stopped

2021-09-08 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36692?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-36692: Assignee: Apache Spark > Improve Error statement when requesting thread dump while

[jira] [Assigned] (SPARK-36692) Improve Error statement when requesting thread dump while executor already stopped

2021-09-08 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36692?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-36692: Assignee: (was: Apache Spark) > Improve Error statement when requesting thread dump

[jira] [Commented] (SPARK-36692) Improve Error statement when requesting thread dump while executor already stopped

2021-09-08 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36692?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17411771#comment-17411771 ] Apache Spark commented on SPARK-36692: -- User 'AngersZh' has created a pull request for this

[jira] [Updated] (SPARK-36686) Fix SimplifyConditionalsInPredicate to be null-safe

2021-09-08 Thread Andrew (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36686?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew updated SPARK-36686: --- Description: SimplifyConditionalsInPredicate rule is not null-safe and leads to incorrect results  

[jira] [Updated] (SPARK-36686) Fix SimplifyConditionalsInPredicate to be null-safe

2021-09-08 Thread Andrew (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36686?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew updated SPARK-36686: --- Description: SimplifyConditionalsInPredicate rule is not null-safe and leads to incorrect results  

[jira] [Created] (SPARK-36692) Improve Error statement when requesting thread dump while executor already stopped

2021-09-08 Thread angerszhu (Jira)
angerszhu created SPARK-36692: - Summary: Improve Error statement when requesting thread dump while executor already stopped Key: SPARK-36692 URL: https://issues.apache.org/jira/browse/SPARK-36692

[jira] [Updated] (SPARK-36686) Fix SimplifyConditionalsInPredicate to be null-safe

2021-09-08 Thread Andrew (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36686?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew updated SPARK-36686: --- Description: SimplifyConditionalsInPredicate rule is not null-safe and leads to incorrect results  

[jira] [Updated] (SPARK-36686) Fix SimplifyConditionalsInPredicate to be null-safe

2021-09-08 Thread Andrew (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36686?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew updated SPARK-36686: --- Summary: Fix SimplifyConditionalsInPredicate to be null-safe (was: fix SimplifyConditionalsInPredicate to

[jira] [Resolved] (SPARK-36690) Clean up deprecated api usage related to commons-pool2

2021-09-08 Thread Jungtaek Lim (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36690?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jungtaek Lim resolved SPARK-36690. -- Fix Version/s: 3.3.0 Resolution: Fixed Issue resolved by pull request 33933

[jira] [Assigned] (SPARK-36690) Clean up deprecated api usage related to commons-pool2

2021-09-08 Thread Jungtaek Lim (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36690?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jungtaek Lim reassigned SPARK-36690: Assignee: Yang Jie > Clean up deprecated api usage related to commons-pool2 >

[jira] [Updated] (SPARK-36681) Fail to load Snappy codec

2021-09-08 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36681?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-36681: - Target Version/s: 3.2.0 > Fail to load Snappy codec > - > >

[jira] [Commented] (SPARK-36681) Fail to load Snappy codec

2021-09-08 Thread L. C. Hsieh (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36681?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17411717#comment-17411717 ] L. C. Hsieh commented on SPARK-36681: - yea. > Fail to load Snappy codec > -

[jira] [Assigned] (SPARK-36691) PythonRunner failed should pass error message to ApplicationMaster too

2021-09-08 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36691?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-36691: Assignee: (was: Apache Spark) > PythonRunner failed should pass error message to

[jira] [Commented] (SPARK-36691) PythonRunner failed should pass error message to ApplicationMaster too

2021-09-08 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36691?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17411715#comment-17411715 ] Apache Spark commented on SPARK-36691: -- User 'AngersZh' has created a pull request for this

[jira] [Commented] (SPARK-36691) PythonRunner failed should pass error message to ApplicationMaster too

2021-09-08 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36691?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17411714#comment-17411714 ] Apache Spark commented on SPARK-36691: -- User 'AngersZh' has created a pull request for this

[jira] [Assigned] (SPARK-36691) PythonRunner failed should pass error message to ApplicationMaster too

2021-09-08 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36691?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-36691: Assignee: Apache Spark > PythonRunner failed should pass error message to

[jira] [Commented] (SPARK-36690) Clean up deprecated api usage related to commons-pool2

2021-09-08 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36690?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17411713#comment-17411713 ] Apache Spark commented on SPARK-36690: -- User 'LuciferYang' has created a pull request for this

[jira] [Assigned] (SPARK-36690) Clean up deprecated api usage related to commons-pool2

2021-09-08 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36690?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-36690: Assignee: Apache Spark > Clean up deprecated api usage related to commons-pool2 >

[jira] [Assigned] (SPARK-36690) Clean up deprecated api usage related to commons-pool2

2021-09-08 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36690?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-36690: Assignee: (was: Apache Spark) > Clean up deprecated api usage related to

[jira] [Updated] (SPARK-36691) PythonRunner failed should pass error message to ApplicationMaster too

2021-09-08 Thread angerszhu (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36691?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] angerszhu updated SPARK-36691: -- Description: In current pyspark, stderr and stdout are print together, if py-script failed, will

[jira] [Commented] (SPARK-36685) Fix wrong assert statement

2021-09-08 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36685?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17411706#comment-17411706 ] Hyukjin Kwon commented on SPARK-36685: -- Please go ahead for a PR. > Fix wrong assert statement >

[jira] [Commented] (SPARK-36681) Fail to load Snappy codec

2021-09-08 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36681?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17411707#comment-17411707 ] Hyukjin Kwon commented on SPARK-36681: -- Is it 3.2 blocker? > Fail to load Snappy codec >

[jira] [Created] (SPARK-36691) PythonRunner failed should pass error message to ApplicationMaster too

2021-09-08 Thread angerszhu (Jira)
angerszhu created SPARK-36691: - Summary: PythonRunner failed should pass error message to ApplicationMaster too Key: SPARK-36691 URL: https://issues.apache.org/jira/browse/SPARK-36691 Project: Spark