[jira] [Commented] (SPARK-32965) pyspark reading csv files with utf_16le encoding

2020-09-22 Thread Takeshi Yamamuro (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32965?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17200578#comment-17200578 ] Takeshi Yamamuro commented on SPARK-32965: -- Is this issue almost the same with SPARK-32961? >

[jira] [Resolved] (SPARK-32959) Fix the "Relation: view text" test in DataSourceV2SQLSuite

2020-09-22 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32959?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-32959. - Fix Version/s: 3.1.0 Resolution: Fixed Issue resolved by pull request 29811

[jira] [Assigned] (SPARK-32959) Fix the "Relation: view text" test in DataSourceV2SQLSuite

2020-09-22 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32959?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan reassigned SPARK-32959: --- Assignee: Terry Kim > Fix the "Relation: view text" test in DataSourceV2SQLSuite >

[jira] [Updated] (SPARK-32966) Spark| PartitionBy is taking long time to process

2020-09-22 Thread Sujit Das (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32966?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sujit Das updated SPARK-32966: -- Environment: EMR - 5.30.0; Hadoop - 2.8.5; Spark - 2.4.5 (was: EMR - 5.30.0; Hadoop -2.8.5; Spark-

[jira] [Commented] (SPARK-32966) Spark| PartitionBy is taking long time to process

2020-09-22 Thread Takeshi Yamamuro (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32966?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17200571#comment-17200571 ] Takeshi Yamamuro commented on SPARK-32966: -- Is this a question? At least, I think you need to

[jira] [Resolved] (SPARK-32966) Spark| PartitionBy is taking long time to process

2020-09-22 Thread Takeshi Yamamuro (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32966?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Takeshi Yamamuro resolved SPARK-32966. -- Resolution: Invalid > Spark| PartitionBy is taking long time to process >

[jira] [Commented] (SPARK-32306) `approx_percentile` in Spark SQL gives incorrect results

2020-09-22 Thread Sean Malory (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32306?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17200564#comment-17200564 ] Sean Malory commented on SPARK-32306: - Thank you. > `approx_percentile` in Spark SQL gives

[jira] [Updated] (SPARK-32961) PySpark CSV read with UTF-16 encoding is not working correctly

2020-09-22 Thread Takeshi Yamamuro (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32961?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Takeshi Yamamuro updated SPARK-32961: - Component/s: (was: Spark Core) SQL > PySpark CSV read with UTF-16

[jira] [Commented] (SPARK-32961) PySpark CSV read with UTF-16 encoding is not working correctly

2020-09-22 Thread Takeshi Yamamuro (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32961?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17200551#comment-17200551 ] Takeshi Yamamuro commented on SPARK-32961: -- cc: [~yumwang] > PySpark CSV read with UTF-16

[jira] [Updated] (SPARK-32778) Accidental Data Deletion on calling saveAsTable

2020-09-22 Thread Takeshi Yamamuro (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32778?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Takeshi Yamamuro updated SPARK-32778: - Issue Type: Improvement (was: Bug) > Accidental Data Deletion on calling saveAsTable >

[jira] [Resolved] (SPARK-31618) Pushdown Distinct through Join in IntersectDistinct based on stats

2020-09-22 Thread Takeshi Yamamuro (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31618?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Takeshi Yamamuro resolved SPARK-31618. -- Resolution: Won't Fix I'll close this because the corresponding PR has been closed.

[jira] [Resolved] (SPARK-32870) Make sure that all expressions have their ExpressionDescription properly filled

2020-09-22 Thread Takeshi Yamamuro (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32870?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Takeshi Yamamuro resolved SPARK-32870. -- Fix Version/s: 3.1.0 Assignee: Tanel Kiis Resolution: Fixed Resolved

[jira] [Updated] (SPARK-32961) PySpark CSV read with UTF-16 encoding is not working correctly

2020-09-22 Thread Bui Bao Anh (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32961?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bui Bao Anh updated SPARK-32961: Attachment: sendo_sample.csv > PySpark CSV read with UTF-16 encoding is not working correctly >

[jira] [Comment Edited] (SPARK-27872) Driver and executors use a different service account breaking pull secrets

2020-09-22 Thread Neelesh Srinivas Salian (Jira)
[ https://issues.apache.org/jira/browse/SPARK-27872?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17194557#comment-17194557 ] Neelesh Srinivas Salian edited comment on SPARK-27872 at 9/23/20, 12:36 AM:

[jira] [Comment Edited] (SPARK-27872) Driver and executors use a different service account breaking pull secrets

2020-09-22 Thread Neelesh Srinivas Salian (Jira)
[ https://issues.apache.org/jira/browse/SPARK-27872?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17194557#comment-17194557 ] Neelesh Srinivas Salian edited comment on SPARK-27872 at 9/23/20, 12:36 AM:

[jira] [Issue Comment Deleted] (SPARK-27872) Driver and executors use a different service account breaking pull secrets

2020-09-22 Thread Neelesh Srinivas Salian (Jira)
[ https://issues.apache.org/jira/browse/SPARK-27872?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Neelesh Srinivas Salian updated SPARK-27872: Comment: was deleted (was: I have a patch to add this fix to the 2.4.x

[jira] [Comment Edited] (SPARK-27872) Driver and executors use a different service account breaking pull secrets

2020-09-22 Thread Neelesh Srinivas Salian (Jira)
[ https://issues.apache.org/jira/browse/SPARK-27872?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17194557#comment-17194557 ] Neelesh Srinivas Salian edited comment on SPARK-27872 at 9/23/20, 12:36 AM:

[jira] [Resolved] (SPARK-32017) Make Pyspark Hadoop 3.2+ Variant available in PyPI

2020-09-22 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32017?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-32017. -- Fix Version/s: 3.1.0 Resolution: Fixed Issue resolved by pull request 29703

[jira] [Assigned] (SPARK-32017) Make Pyspark Hadoop 3.2+ Variant available in PyPI

2020-09-22 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32017?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon reassigned SPARK-32017: Assignee: Hyukjin Kwon > Make Pyspark Hadoop 3.2+ Variant available in PyPI >

[jira] [Resolved] (SPARK-32933) Use keyword-only syntax for keyword_only methods

2020-09-22 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32933?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-32933. -- Fix Version/s: 3.1.0 Resolution: Fixed Issue resolved by pull request 29799

[jira] [Assigned] (SPARK-32933) Use keyword-only syntax for keyword_only methods

2020-09-22 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32933?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon reassigned SPARK-32933: Assignee: Maciej Szymkiewicz > Use keyword-only syntax for keyword_only methods >

[jira] [Commented] (SPARK-27872) Driver and executors use a different service account breaking pull secrets

2020-09-22 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-27872?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17200439#comment-17200439 ] Apache Spark commented on SPARK-27872: -- User 'nssalian' has created a pull request for this issue:

[jira] [Commented] (SPARK-27872) Driver and executors use a different service account breaking pull secrets

2020-09-22 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-27872?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17200438#comment-17200438 ] Apache Spark commented on SPARK-27872: -- User 'nssalian' has created a pull request for this issue:

[jira] [Assigned] (SPARK-17556) Executor side broadcast for broadcast joins

2020-09-22 Thread L. C. Hsieh (Jira)
[ https://issues.apache.org/jira/browse/SPARK-17556?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] L. C. Hsieh reassigned SPARK-17556: --- Assignee: (was: L. C. Hsieh) > Executor side broadcast for broadcast joins >

[jira] [Issue Comment Deleted] (SPARK-17556) Executor side broadcast for broadcast joins

2020-09-22 Thread L. C. Hsieh (Jira)
[ https://issues.apache.org/jira/browse/SPARK-17556?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] L. C. Hsieh updated SPARK-17556: Comment: was deleted (was: We will recently try to pick this up again.) > Executor side

[jira] [Updated] (SPARK-32932) AQE local shuffle reader breaks repartitioning for dynamic partition overwrite

2020-09-22 Thread Manu Zhang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32932?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Manu Zhang updated SPARK-32932: --- Description: With AQE, local shuffle reader breaks users' repartitioning for dynamic partition

[jira] [Commented] (SPARK-32956) Duplicate Columns in a csv file

2020-09-22 Thread Chen Zhang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32956?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17200382#comment-17200382 ] Chen Zhang commented on SPARK-32956: Okay, I will submit a PR later. > Duplicate Columns in a csv

[jira] [Commented] (SPARK-29250) Upgrade to Hadoop 3.2.1

2020-09-22 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29250?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17200366#comment-17200366 ] Apache Spark commented on SPARK-29250: -- User 'sunchao' has created a pull request for this issue:

[jira] [Assigned] (SPARK-29250) Upgrade to Hadoop 3.2.1

2020-09-22 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29250?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-29250: Assignee: Apache Spark > Upgrade to Hadoop 3.2.1 > --- > >

[jira] [Assigned] (SPARK-29250) Upgrade to Hadoop 3.2.1

2020-09-22 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29250?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-29250: Assignee: (was: Apache Spark) > Upgrade to Hadoop 3.2.1 > --- >

[jira] [Commented] (SPARK-29250) Upgrade to Hadoop 3.2.1

2020-09-22 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29250?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17200365#comment-17200365 ] Apache Spark commented on SPARK-29250: -- User 'sunchao' has created a pull request for this issue:

[jira] [Updated] (SPARK-32306) `approx_percentile` in Spark SQL gives incorrect results

2020-09-22 Thread L. C. Hsieh (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32306?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] L. C. Hsieh updated SPARK-32306: Issue Type: Documentation (was: Bug) > `approx_percentile` in Spark SQL gives incorrect results

[jira] [Commented] (SPARK-32306) `approx_percentile` in Spark SQL gives incorrect results

2020-09-22 Thread L. C. Hsieh (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32306?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17200337#comment-17200337 ] L. C. Hsieh commented on SPARK-32306: - Resolved by https://github.com/apache/spark/pull/29835. >

[jira] [Updated] (SPARK-32306) `approx_percentile` in Spark SQL gives incorrect results

2020-09-22 Thread L. C. Hsieh (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32306?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] L. C. Hsieh updated SPARK-32306: Affects Version/s: 3.1.0 3.0.0 > `approx_percentile` in Spark SQL gives

[jira] [Resolved] (SPARK-32306) `approx_percentile` in Spark SQL gives incorrect results

2020-09-22 Thread L. C. Hsieh (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32306?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] L. C. Hsieh resolved SPARK-32306. - Fix Version/s: 3.1.0 Assignee: Maxim Gekk Resolution: Fixed >

[jira] [Commented] (SPARK-32019) Add spark.sql.files.minPartitionNum config

2020-09-22 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32019?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17200327#comment-17200327 ] Apache Spark commented on SPARK-32019: -- User 'tanelk' has created a pull request for this issue:

[jira] [Commented] (SPARK-32019) Add spark.sql.files.minPartitionNum config

2020-09-22 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32019?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17200325#comment-17200325 ] Apache Spark commented on SPARK-32019: -- User 'tanelk' has created a pull request for this issue:

[jira] [Commented] (SPARK-32019) Add spark.sql.files.minPartitionNum config

2020-09-22 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32019?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17200326#comment-17200326 ] Apache Spark commented on SPARK-32019: -- User 'tanelk' has created a pull request for this issue:

[jira] [Assigned] (SPARK-32970) Reduce the runtime of unit test for SPARK-32019

2020-09-22 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32970?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-32970: Assignee: (was: Apache Spark) > Reduce the runtime of unit test for SPARK-32019 >

[jira] [Commented] (SPARK-32970) Reduce the runtime of unit test for SPARK-32019

2020-09-22 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32970?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17200323#comment-17200323 ] Apache Spark commented on SPARK-32970: -- User 'tanelk' has created a pull request for this issue:

[jira] [Assigned] (SPARK-32970) Reduce the runtime of unit test for SPARK-32019

2020-09-22 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32970?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-32970: Assignee: Apache Spark > Reduce the runtime of unit test for SPARK-32019 >

[jira] [Commented] (SPARK-27733) Upgrade to Avro 1.10.0

2020-09-22 Thread Xinli Shang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-27733?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17200307#comment-17200307 ] Xinli Shang commented on SPARK-27733: - We talked about the Parquet 1.11.0 adoption in Spark in

[jira] [Created] (SPARK-32970) Reduce the runtime of unit test for SPARK-32019

2020-09-22 Thread Tanel Kiis (Jira)
Tanel Kiis created SPARK-32970: -- Summary: Reduce the runtime of unit test for SPARK-32019 Key: SPARK-32970 URL: https://issues.apache.org/jira/browse/SPARK-32970 Project: Spark Issue Type:

[jira] [Updated] (SPARK-32969) Spark Submit process not exiting after session.stop()

2020-09-22 Thread El R (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32969?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] El R updated SPARK-32969: - Affects Version/s: (was: 3.0.1) > Spark Submit process not exiting after session.stop() >

[jira] [Created] (SPARK-32969) Spark Submit process not exiting after session.stop()

2020-09-22 Thread El R (Jira)
El R created SPARK-32969: Summary: Spark Submit process not exiting after session.stop() Key: SPARK-32969 URL: https://issues.apache.org/jira/browse/SPARK-32969 Project: Spark Issue Type: Bug

[jira] [Commented] (SPARK-20525) ClassCast exception when interpreting UDFs from a String in spark-shell

2020-09-22 Thread Igor Kamyshnikov (Jira)
[ https://issues.apache.org/jira/browse/SPARK-20525?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17200284#comment-17200284 ] Igor Kamyshnikov commented on SPARK-20525: -- I bet the issue is in JDK, but it could be solved

[jira] [Resolved] (SPARK-32964) Pass all `streaming` module UTs in Scala 2.13

2020-09-22 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32964?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-32964. --- Fix Version/s: 3.1.0 Resolution: Fixed Issue resolved by pull request 29836

[jira] [Assigned] (SPARK-32964) Pass all `streaming` module UTs in Scala 2.13

2020-09-22 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32964?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun reassigned SPARK-32964: - Assignee: Yang Jie > Pass all `streaming` module UTs in Scala 2.13 >

[jira] [Comment Edited] (SPARK-19938) java.lang.ClassCastException: cannot assign instance of scala.collection.immutable.List$SerializationProxy to field

2020-09-22 Thread Igor Kamyshnikov (Jira)
[ https://issues.apache.org/jira/browse/SPARK-19938?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17200272#comment-17200272 ] Igor Kamyshnikov edited comment on SPARK-19938 at 9/22/20, 5:55 PM:

[jira] [Commented] (SPARK-19938) java.lang.ClassCastException: cannot assign instance of scala.collection.immutable.List$SerializationProxy to field

2020-09-22 Thread Igor Kamyshnikov (Jira)
[ https://issues.apache.org/jira/browse/SPARK-19938?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17200272#comment-17200272 ] Igor Kamyshnikov commented on SPARK-19938: -- [~rdblue], my analysis shows the different root

[jira] [Updated] (SPARK-32968) Column pruning for CsvToStructs

2020-09-22 Thread L. C. Hsieh (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32968?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] L. C. Hsieh updated SPARK-32968: Description: We could do column pruning for CsvToStructs expression if we only require some

[jira] [Created] (SPARK-32968) Column pruning for CsvToStructs

2020-09-22 Thread L. C. Hsieh (Jira)
L. C. Hsieh created SPARK-32968: --- Summary: Column pruning for CsvToStructs Key: SPARK-32968 URL: https://issues.apache.org/jira/browse/SPARK-32968 Project: Spark Issue Type: Sub-task

[jira] [Created] (SPARK-32967) Optimize csv expression chain

2020-09-22 Thread L. C. Hsieh (Jira)
L. C. Hsieh created SPARK-32967: --- Summary: Optimize csv expression chain Key: SPARK-32967 URL: https://issues.apache.org/jira/browse/SPARK-32967 Project: Spark Issue Type: Sub-task

[jira] [Created] (SPARK-32966) Spark| PartitionBy is taking long time to process

2020-09-22 Thread Sujit Das (Jira)
Sujit Das created SPARK-32966: - Summary: Spark| PartitionBy is taking long time to process Key: SPARK-32966 URL: https://issues.apache.org/jira/browse/SPARK-32966 Project: Spark Issue Type:

[jira] [Commented] (SPARK-32659) Fix the data issue of inserted DPP on non-atomic type

2020-09-22 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32659?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17200176#comment-17200176 ] Apache Spark commented on SPARK-32659: -- User 'cloud-fan' has created a pull request for this issue:

[jira] [Created] (SPARK-32965) pyspark reading csv files with utf_16le encoding

2020-09-22 Thread Punit Shah (Jira)
Punit Shah created SPARK-32965: -- Summary: pyspark reading csv files with utf_16le encoding Key: SPARK-32965 URL: https://issues.apache.org/jira/browse/SPARK-32965 Project: Spark Issue Type: Bug

[jira] [Commented] (SPARK-32956) Duplicate Columns in a csv file

2020-09-22 Thread Punit Shah (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32956?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17200163#comment-17200163 ] Punit Shah commented on SPARK-32956: That may work > Duplicate Columns in a csv file >

[jira] [Comment Edited] (SPARK-32153) .m2 repository corruption happens

2020-09-22 Thread Kousuke Saruta (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32153?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17200112#comment-17200112 ] Kousuke Saruta edited comment on SPARK-32153 at 9/22/20, 2:31 PM: --

[jira] [Updated] (SPARK-32153) .m2 repository corruption happens

2020-09-22 Thread Kousuke Saruta (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32153?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kousuke Saruta updated SPARK-32153: --- Affects Version/s: 2.4.8 > .m2 repository corruption happens >

[jira] [Resolved] (SPARK-16190) Worker registration failed: Duplicate worker ID

2020-09-22 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-16190?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-16190. --- Fix Version/s: 3.0.0 Resolution: Duplicate This is fixed via SPARK-23191 . Please

[jira] [Reopened] (SPARK-32153) .m2 repository corruption happens

2020-09-22 Thread Kousuke Saruta (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32153?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kousuke Saruta reopened SPARK-32153: > .m2 repository corruption happens > - > >

[jira] [Commented] (SPARK-32153) .m2 repository corruption happens

2020-09-22 Thread Kousuke Saruta (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32153?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17200112#comment-17200112 ] Kousuke Saruta commented on SPARK-32153: [~shaneknapp]This issue seems to happen again

[jira] [Updated] (SPARK-32153) .m2 repository corruption happens

2020-09-22 Thread Kousuke Saruta (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32153?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kousuke Saruta updated SPARK-32153: --- Summary: .m2 repository corruption happens (was: .m2 repository corruption can happen on

[jira] [Updated] (SPARK-32956) Duplicate Columns in a csv file

2020-09-22 Thread Chen Zhang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32956?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chen Zhang updated SPARK-32956: --- Component/s: (was: Spark Core) SQL > Duplicate Columns in a csv file >

[jira] [Commented] (SPARK-32956) Duplicate Columns in a csv file

2020-09-22 Thread Chen Zhang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32956?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17200092#comment-17200092 ] Chen Zhang commented on SPARK-32956: In SPARK-16896, if the CSV data has duplicate column headers,

[jira] [Commented] (SPARK-32757) Physical InSubqueryExec should be consistent with logical InSubquery

2020-09-22 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32757?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17200079#comment-17200079 ] Apache Spark commented on SPARK-32757: -- User 'cloud-fan' has created a pull request for this issue:

[jira] [Commented] (SPARK-32659) Fix the data issue of inserted DPP on non-atomic type

2020-09-22 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32659?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17200063#comment-17200063 ] Apache Spark commented on SPARK-32659: -- User 'cloud-fan' has created a pull request for this issue:

[jira] [Commented] (SPARK-31882) DAG-viz is not rendered correctly with pagination.

2020-09-22 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31882?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17200035#comment-17200035 ] Apache Spark commented on SPARK-31882: -- User 'zhli1142015' has created a pull request for this

[jira] [Commented] (SPARK-31882) DAG-viz is not rendered correctly with pagination.

2020-09-22 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31882?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17200034#comment-17200034 ] Apache Spark commented on SPARK-31882: -- User 'zhli1142015' has created a pull request for this

[jira] [Commented] (SPARK-32938) Spark can not cast long value from Kafka

2020-09-22 Thread Vinod KC (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32938?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17200028#comment-17200028 ] Vinod KC commented on SPARK-32938: -- [~maseiler],  Can you please test with this example?  {code:java}

[jira] [Commented] (SPARK-32925) Support push-based shuffle in multiple deployment environments

2020-09-22 Thread qingwu.fu (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32925?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17200023#comment-17200023 ] qingwu.fu commented on SPARK-32925: --- Should send data to remote shuffle servioce bypass sort and spill

[jira] [Commented] (SPARK-32463) Document Data Type inference rule in SQL reference

2020-09-22 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32463?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=1721#comment-1721 ] Apache Spark commented on SPARK-32463: -- User 'planga82' has created a pull request for this issue:

[jira] [Assigned] (SPARK-32463) Document Data Type inference rule in SQL reference

2020-09-22 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32463?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-32463: Assignee: Apache Spark > Document Data Type inference rule in SQL reference >

[jira] [Assigned] (SPARK-32463) Document Data Type inference rule in SQL reference

2020-09-22 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32463?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-32463: Assignee: (was: Apache Spark) > Document Data Type inference rule in SQL reference >

[jira] [Commented] (SPARK-32964) Pass all `streaming` module UTs in Scala 2.13

2020-09-22 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32964?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=1715#comment-1715 ] Apache Spark commented on SPARK-32964: -- User 'LuciferYang' has created a pull request for this

[jira] [Commented] (SPARK-32964) Pass all `streaming` module UTs in Scala 2.13

2020-09-22 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32964?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=1717#comment-1717 ] Apache Spark commented on SPARK-32964: -- User 'LuciferYang' has created a pull request for this

[jira] [Assigned] (SPARK-32964) Pass all `streaming` module UTs in Scala 2.13

2020-09-22 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32964?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-32964: Assignee: (was: Apache Spark) > Pass all `streaming` module UTs in Scala 2.13 >

[jira] [Assigned] (SPARK-32964) Pass all `streaming` module UTs in Scala 2.13

2020-09-22 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32964?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-32964: Assignee: Apache Spark > Pass all `streaming` module UTs in Scala 2.13 >

[jira] [Updated] (SPARK-32964) Pass all `streaming` module UTs in Scala 2.13

2020-09-22 Thread Yang Jie (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32964?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yang Jie updated SPARK-32964: - Description: There is only one failed case of `streaming` module in Scala 2.13: * `start with

[jira] [Updated] (SPARK-32964) Pass all `streaming` module UTs in Scala 2.13

2020-09-22 Thread Yang Jie (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32964?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yang Jie updated SPARK-32964: - Description: There is only one failed case of `streaming` module in Scala 2.13: * `start with

[jira] [Created] (SPARK-32964) Pass all `streaming` module UTs in Scala 2.13

2020-09-22 Thread Yang Jie (Jira)
Yang Jie created SPARK-32964: Summary: Pass all `streaming` module UTs in Scala 2.13 Key: SPARK-32964 URL: https://issues.apache.org/jira/browse/SPARK-32964 Project: Spark Issue Type: Sub-task

[jira] [Commented] (SPARK-32306) `approx_percentile` in Spark SQL gives incorrect results

2020-09-22 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32306?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17199953#comment-17199953 ] Apache Spark commented on SPARK-32306: -- User 'MaxGekk' has created a pull request for this issue:

[jira] [Assigned] (SPARK-32306) `approx_percentile` in Spark SQL gives incorrect results

2020-09-22 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32306?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-32306: Assignee: (was: Apache Spark) > `approx_percentile` in Spark SQL gives incorrect

[jira] [Assigned] (SPARK-32306) `approx_percentile` in Spark SQL gives incorrect results

2020-09-22 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32306?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-32306: Assignee: Apache Spark > `approx_percentile` in Spark SQL gives incorrect results >

[jira] [Commented] (SPARK-32306) `approx_percentile` in Spark SQL gives incorrect results

2020-09-22 Thread Maxim Gekk (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32306?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17199952#comment-17199952 ] Maxim Gekk commented on SPARK-32306: I opened PR https://github.com/apache/spark/pull/29835 with

[jira] [Commented] (SPARK-32963) empty string should be consistent for schema name in SparkGetSchemasOperation

2020-09-22 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32963?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17199933#comment-17199933 ] Apache Spark commented on SPARK-32963: -- User 'yaooqinn' has created a pull request for this issue:

[jira] [Assigned] (SPARK-32963) empty string should be consistent for schema name in SparkGetSchemasOperation

2020-09-22 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32963?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-32963: Assignee: Apache Spark > empty string should be consistent for schema name in

[jira] [Assigned] (SPARK-32963) empty string should be consistent for schema name in SparkGetSchemasOperation

2020-09-22 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32963?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-32963: Assignee: (was: Apache Spark) > empty string should be consistent for schema name in

[jira] [Commented] (SPARK-32963) empty string should be consistent for schema name in SparkGetSchemasOperation

2020-09-22 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32963?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17199936#comment-17199936 ] Apache Spark commented on SPARK-32963: -- User 'yaooqinn' has created a pull request for this issue:

[jira] [Created] (SPARK-32963) empty string should be consistent for schema name in SparkGetSchemasOperation

2020-09-22 Thread Kent Yao (Jira)
Kent Yao created SPARK-32963: Summary: empty string should be consistent for schema name in SparkGetSchemasOperation Key: SPARK-32963 URL: https://issues.apache.org/jira/browse/SPARK-32963 Project: Spark

[jira] [Updated] (SPARK-32962) Spark Streaming

2020-09-22 Thread Amit Menashe (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32962?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Amit Menashe updated SPARK-32962: - Priority: Trivial (was: Major) > Spark Streaming > --- > > Key:

[jira] [Created] (SPARK-32962) Spark Streaming

2020-09-22 Thread Amit Menashe (Jira)
Amit Menashe created SPARK-32962: Summary: Spark Streaming Key: SPARK-32962 URL: https://issues.apache.org/jira/browse/SPARK-32962 Project: Spark Issue Type: Bug Components:

[jira] [Commented] (SPARK-32886) '.../jobs/undefined' link from "Event Timeline" in jobs page

2020-09-22 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32886?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17199898#comment-17199898 ] Apache Spark commented on SPARK-32886: -- User 'zhli1142015' has created a pull request for this

[jira] [Commented] (SPARK-32886) '.../jobs/undefined' link from "Event Timeline" in jobs page

2020-09-22 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32886?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17199900#comment-17199900 ] Apache Spark commented on SPARK-32886: -- User 'zhli1142015' has created a pull request for this

[jira] [Updated] (SPARK-32898) totalExecutorRunTimeMs is too big

2020-09-22 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32898?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-32898: -- Fix Version/s: 2.4.8 > totalExecutorRunTimeMs is too big > -

[jira] [Commented] (SPARK-32306) `approx_percentile` in Spark SQL gives incorrect results

2020-09-22 Thread Sean Malory (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32306?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17199846#comment-17199846 ] Sean Malory commented on SPARK-32306: - [~maxgekk]; thanks for the definition. Can we please update

[jira] [Commented] (SPARK-32306) `approx_percentile` in Spark SQL gives incorrect results

2020-09-22 Thread Sean Malory (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32306?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17199844#comment-17199844 ] Sean Malory commented on SPARK-32306: - Exactly; you should get the median, which is defined, almost

[jira] [Commented] (SPARK-32306) `approx_percentile` in Spark SQL gives incorrect results

2020-09-22 Thread Maxim Gekk (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32306?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17199839#comment-17199839 ] Maxim Gekk commented on SPARK-32306: The function returns an element of the input sequence, see

[jira] [Commented] (SPARK-32898) totalExecutorRunTimeMs is too big

2020-09-22 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32898?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17199840#comment-17199840 ] Apache Spark commented on SPARK-32898: -- User 'Ngone51' has created a pull request for this issue: