[jira] [Resolved] (SPARK-26808) Pruned schema should not change nullability

[ https://issues.apache.org/jira/browse/SPARK-26808?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Liang-Chi Hsieh resolved SPARK-26808. - Resolution: Won't Fix > Pruned schema should not change nullability >

[jira] [Commented] (SPARK-18161) Default PickleSerializer pickle protocol doesn't handle > 4GB objects

[ https://issues.apache.org/jira/browse/SPARK-18161?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16758906#comment-16758906 ] Boris Shminke commented on SPARK-18161: --- [~ssimmons] thanks for starting this work. 

[jira] [Assigned] (SPARK-26813) Consolidate java version across language compilers and build tools

[ https://issues.apache.org/jira/browse/SPARK-26813?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-26813: Assignee: Apache Spark > Consolidate java version across language compilers and build

[jira] [Assigned] (SPARK-26813) Consolidate java version across language compilers and build tools

[ https://issues.apache.org/jira/browse/SPARK-26813?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-26813: Assignee: (was: Apache Spark) > Consolidate java version across language compilers

[jira] [Updated] (SPARK-26813) Consolidate java version across language compilers and build tools

[ https://issues.apache.org/jira/browse/SPARK-26813?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chenxiao Mao updated SPARK-26813: - Description: The java version here means versions of javac source, javac target, scalac

[jira] [Updated] (SPARK-26813) Consolidate java version across language compilers and build tools

[ https://issues.apache.org/jira/browse/SPARK-26813?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chenxiao Mao updated SPARK-26813: - Description: The java version here means versions of javac source, javac target, scalac

[jira] [Updated] (SPARK-26813) Consolidate java version across language compilers and build tools

[ https://issues.apache.org/jira/browse/SPARK-26813?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chenxiao Mao updated SPARK-26813: - Description: The java version here means versions of javac source, javac target, scalac

[jira] [Created] (SPARK-26813) Consolidate java version across language compilers and build tools

Chenxiao Mao created SPARK-26813: Summary: Consolidate java version across language compilers and build tools Key: SPARK-26813 URL: https://issues.apache.org/jira/browse/SPARK-26813 Project: Spark

[jira] [Commented] (SPARK-26810) Fixing SPARK-25072 broke existing code and fails to show error message

[ https://issues.apache.org/jira/browse/SPARK-26810?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16758878#comment-16758878 ] Hyukjin Kwon commented on SPARK-26810: -- Also workaround is super super easy. Just put one {{*}}:

[jira] [Commented] (SPARK-26810) Fixing SPARK-25072 broke existing code and fails to show error message

[ https://issues.apache.org/jira/browse/SPARK-26810?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16758876#comment-16758876 ] Hyukjin Kwon commented on SPARK-26810: -- {code} Traceback (most recent call last): File "", line

[jira] [Commented] (SPARK-26809) insert overwrite directory + concat function => error

[ https://issues.apache.org/jira/browse/SPARK-26809?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16758873#comment-16758873 ] Hyukjin Kwon commented on SPARK-26809: -- Is it able to post a self-contained reproducer? It will

[jira] [Resolved] (SPARK-26796) Testcases failing with "org.apache.hadoop.fs.ChecksumException" error

[ https://issues.apache.org/jira/browse/SPARK-26796?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-26796. -- Resolution: Cannot Reproduce > Testcases failing with

[jira] [Commented] (SPARK-26804) Spark sql carries newline char from last csv column when imported

[ https://issues.apache.org/jira/browse/SPARK-26804?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16758872#comment-16758872 ] Hyukjin Kwon commented on SPARK-26804: -- Can you show your input file? It would be easier to verify

[jira] [Resolved] (SPARK-26804) Spark sql carries newline char from last csv column when imported

[ https://issues.apache.org/jira/browse/SPARK-26804?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-26804. -- Resolution: Incomplete > Spark sql carries newline char from last csv column when imported >

[jira] [Commented] (SPARK-26801) Spark unable to read valid avro types

[ https://issues.apache.org/jira/browse/SPARK-26801?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16758869#comment-16758869 ] Hyukjin Kwon commented on SPARK-26801: -- Thanks for reporting this. Would you be interested in

[jira] [Commented] (SPARK-26796) Testcases failing with "org.apache.hadoop.fs.ChecksumException" error

[ https://issues.apache.org/jira/browse/SPARK-26796?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16758868#comment-16758868 ] Hyukjin Kwon commented on SPARK-26796: -- I'm unable to reproduce this in my local, and the tests

[jira] [Commented] (SPARK-26791) Some scala codes doesn't show friendly and some description about foreachBatch is misleading

[ https://issues.apache.org/jira/browse/SPARK-26791?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16758867#comment-16758867 ] Hyukjin Kwon commented on SPARK-26791: -- Can you post a PR to improve the doc? > Some scala codes

[jira] [Commented] (SPARK-26807) Confusing documentation regarding installation from PyPi

[ https://issues.apache.org/jira/browse/SPARK-26807?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16758866#comment-16758866 ] Hyukjin Kwon commented on SPARK-26807: -- Can you post a PR? > Confusing documentation regarding

[jira] [Assigned] (SPARK-26651) Use Proleptic Gregorian calendar

[ https://issues.apache.org/jira/browse/SPARK-26651?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-26651: Assignee: Maxim Gekk (was: Apache Spark) > Use Proleptic Gregorian calendar >

[jira] [Assigned] (SPARK-26651) Use Proleptic Gregorian calendar

[ https://issues.apache.org/jira/browse/SPARK-26651?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-26651: Assignee: Apache Spark (was: Maxim Gekk) > Use Proleptic Gregorian calendar >

[jira] [Resolved] (SPARK-26651) Use Proleptic Gregorian calendar

[ https://issues.apache.org/jira/browse/SPARK-26651?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-26651. -- Resolution: Fixed Fix Version/s: 3.0.0 Issue resolved by pull request 23722

[jira] [Updated] (SPARK-26651) Use Proleptic Gregorian calendar

[ https://issues.apache.org/jira/browse/SPARK-26651?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-26651: - Fix Version/s: (was: 3.0.0) > Use Proleptic Gregorian calendar >

[jira] [Reopened] (SPARK-26651) Use Proleptic Gregorian calendar

[ https://issues.apache.org/jira/browse/SPARK-26651?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon reopened SPARK-26651: -- > Use Proleptic Gregorian calendar > > > Key:

[jira] [Resolved] (SPARK-18161) Default PickleSerializer pickle protocol doesn't handle > 4GB objects

[ https://issues.apache.org/jira/browse/SPARK-18161?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-18161. -- Resolution: Fixed Fix Version/s: 3.0.0 This is fixed by upgrading cloudpickle at

[jira] [Commented] (SPARK-21733) ERROR executor.CoarseGrainedExecutorBackend: RECEIVED SIGNAL TERM

[ https://issues.apache.org/jira/browse/SPARK-21733?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16758825#comment-16758825 ] Rajesh Chandramohan commented on SPARK-21733: - Its based on symptom from the actual issue. 

[jira] [Created] (SPARK-26812) PushProjectionThroughUnion nullability issue

Bogdan Raducanu created SPARK-26812: --- Summary: PushProjectionThroughUnion nullability issue Key: SPARK-26812 URL: https://issues.apache.org/jira/browse/SPARK-26812 Project: Spark Issue

[jira] [Resolved] (SPARK-26714) The job whose partiton num is zero not shown in WebUI

[ https://issues.apache.org/jira/browse/SPARK-26714?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen resolved SPARK-26714. --- Resolution: Fixed Fix Version/s: 3.0.0 Issue resolved by pull request 23637

[jira] [Assigned] (SPARK-26714) The job whose partiton num is zero not shown in WebUI

[ https://issues.apache.org/jira/browse/SPARK-26714?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen reassigned SPARK-26714: - Assignee: deshanxiao > The job whose partiton num is zero not shown in WebUI >

[jira] [Resolved] (SPARK-26771) Make .unpersist(), .destroy() consistently non-blocking by default

[ https://issues.apache.org/jira/browse/SPARK-26771?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen resolved SPARK-26771. --- Resolution: Fixed Fix Version/s: 3.0.0 Issue resolved by pull request 23685

[jira] [Assigned] (SPARK-26754) Add hasTrainingSummary to replace duplicate code in PySpark

[ https://issues.apache.org/jira/browse/SPARK-26754?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen reassigned SPARK-26754: - Assignee: Huaxin Gao > Add hasTrainingSummary to replace duplicate code in PySpark >

[jira] [Resolved] (SPARK-26754) Add hasTrainingSummary to replace duplicate code in PySpark

[ https://issues.apache.org/jira/browse/SPARK-26754?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen resolved SPARK-26754. --- Resolution: Fixed Fix Version/s: 3.0.0 Issue resolved by pull request 23676

[jira] [Commented] (SPARK-26786) Handle to treat escaped newline characters('\r','\n') in spark csv

[ https://issues.apache.org/jira/browse/SPARK-26786?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16758677#comment-16758677 ] vishnuram selvaraj commented on SPARK-26786: Thanks [~hyukjin.kwon]. I have raised a git

[jira] [Commented] (SPARK-24541) TCP based shuffle

[ https://issues.apache.org/jira/browse/SPARK-24541?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16758673#comment-16758673 ] Jungtaek Lim commented on SPARK-24541: -- Same understanding here: while I think there's pretty less

[jira] [Updated] (SPARK-26651) Use Proleptic Gregorian calendar

[ https://issues.apache.org/jira/browse/SPARK-26651?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Maxim Gekk updated SPARK-26651: --- Labels: ReleaseNote (was: ) > Use Proleptic Gregorian calendar >

[jira] [Updated] (SPARK-26651) Use Proleptic Gregorian calendar

[ https://issues.apache.org/jira/browse/SPARK-26651?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Maxim Gekk updated SPARK-26651: --- Description: Spark 2.4 and previous versions use a hybrid calendar - Julian + Gregorian in

[jira] [Updated] (SPARK-26651) Use Proleptic Gregorian calendar

[ https://issues.apache.org/jira/browse/SPARK-26651?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Maxim Gekk updated SPARK-26651: --- Description: Spark 2.4 and previous versions use a hybrid calendar - Julian + Gregorian in

[jira] [Created] (SPARK-26811) Add DataSourceV2 capabilities to check support for batch append, overwrite, truncate during analysis.

Ryan Blue created SPARK-26811: - Summary: Add DataSourceV2 capabilities to check support for batch append, overwrite, truncate during analysis. Key: SPARK-26811 URL: https://issues.apache.org/jira/browse/SPARK-26811

[jira] [Assigned] (SPARK-26651) Use Proleptic Gregorian calendar

[ https://issues.apache.org/jira/browse/SPARK-26651?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-26651: Assignee: Maxim Gekk (was: Apache Spark) > Use Proleptic Gregorian calendar >

[jira] [Assigned] (SPARK-26651) Use Proleptic Gregorian calendar

[ https://issues.apache.org/jira/browse/SPARK-26651?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-26651: Assignee: Apache Spark (was: Maxim Gekk) > Use Proleptic Gregorian calendar >

[jira] [Updated] (SPARK-26806) EventTimeStats.merge doesn't handle "zero.merge(zero)" correctly

[ https://issues.apache.org/jira/browse/SPARK-26806?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu updated SPARK-26806: - Affects Version/s: 2.3.3 > EventTimeStats.merge doesn't handle "zero.merge(zero)" correctly >

[jira] [Created] (SPARK-26810) Fixing SPARK-25072 broke existing code and fails to show error message

Arttu Voutilainen created SPARK-26810: - Summary: Fixing SPARK-25072 broke existing code and fails to show error message Key: SPARK-26810 URL: https://issues.apache.org/jira/browse/SPARK-26810

[jira] [Updated] (SPARK-26806) EventTimeStats.merge doesn't handle "zero.merge(zero)" correctly

[ https://issues.apache.org/jira/browse/SPARK-26806?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu updated SPARK-26806: - Fix Version/s: (was: 2.3.3) 2.3.4 > EventTimeStats.merge doesn't handle

[jira] [Updated] (SPARK-26806) EventTimeStats.merge doesn't handle "zero.merge(zero)" correctly

[ https://issues.apache.org/jira/browse/SPARK-26806?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu updated SPARK-26806: - Affects Version/s: 2.2.2 2.2.3 > EventTimeStats.merge doesn't handle

[jira] [Resolved] (SPARK-26806) EventTimeStats.merge doesn't handle "zero.merge(zero)" correctly

[ https://issues.apache.org/jira/browse/SPARK-26806?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu resolved SPARK-26806. -- Resolution: Fixed Fix Version/s: 3.0.0 2.4.1

[jira] [Commented] (SPARK-24961) sort operation causes out of memory

[ https://issues.apache.org/jira/browse/SPARK-24961?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16758557#comment-16758557 ] Mono Shiro commented on SPARK-24961: Spark Version 2.3.2.  I have a very similar issue when simply

[jira] [Commented] (SPARK-24541) TCP based shuffle

[ https://issues.apache.org/jira/browse/SPARK-24541?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16758488#comment-16758488 ] Jose Torres commented on SPARK-24541: - I'm not gonna lie, I didn't put a tremendous amount of

[jira] [Commented] (SPARK-24541) TCP based shuffle

[ https://issues.apache.org/jira/browse/SPARK-24541?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16758482#comment-16758482 ] Imran Rashid commented on SPARK-24541: -- well, rpc is over tcp, so I'm still not really sure what

[jira] [Commented] (SPARK-23685) Spark Structured Streaming Kafka 0.10 Consumer Can't Handle Non-consecutive Offsets (i.e. Log Compaction)

[ https://issues.apache.org/jira/browse/SPARK-23685?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16758351#comment-16758351 ] Gabor Somogyi commented on SPARK-23685: --- [~sindiri] We've tried to reproduce the issue without

[jira] [Commented] (SPARK-26783) Kafka parameter documentation doesn't match with the reality (upper/lowercase)

[ https://issues.apache.org/jira/browse/SPARK-26783?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16758348#comment-16758348 ] Gabor Somogyi commented on SPARK-26783: --- [~zsxwing] [~kabhwan] The more I'm playing with the

[jira] [Updated] (SPARK-26734) StackOverflowError on WAL serialization caused by large receivedBlockQueue

[ https://issues.apache.org/jira/browse/SPARK-26734?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gabor Somogyi updated SPARK-26734: -- Component/s: DStreams > StackOverflowError on WAL serialization caused by large

[jira] [Created] (SPARK-26809) insert overwrite directory + concat function => error

ant_nebula created SPARK-26809: -- Summary: insert overwrite directory + concat function => error Key: SPARK-26809 URL: https://issues.apache.org/jira/browse/SPARK-26809 Project: Spark Issue

[jira] [Assigned] (SPARK-26797) Start using the new logical types API of Parquet 1.11.0 instead of the deprecated one

[ https://issues.apache.org/jira/browse/SPARK-26797?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-26797: Assignee: (was: Apache Spark) > Start using the new logical types API of Parquet

[jira] [Assigned] (SPARK-26797) Start using the new logical types API of Parquet 1.11.0 instead of the deprecated one

[ https://issues.apache.org/jira/browse/SPARK-26797?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-26797: Assignee: Apache Spark > Start using the new logical types API of Parquet 1.11.0 instead

[jira] [Commented] (SPARK-23155) YARN-aggregated executor/driver logs appear unavailable when NM is down

[ https://issues.apache.org/jira/browse/SPARK-23155?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16758102#comment-16758102 ] Gera Shegalov commented on SPARK-23155: --- [~kabhwan], [~vanzin] I would still be interested to be

[jira] [Commented] (SPARK-26792) Apply custom log URL to Spark UI

[ https://issues.apache.org/jira/browse/SPARK-26792?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16758095#comment-16758095 ] Gera Shegalov commented on SPARK-26792: --- [~kabhwan] thanks for doing this work. I verified that I