[jira] [Resolved] (SPARK-29923) Set `io.netty.tryReflectionSetAccessible` for Arrow on JDK9+

2019-11-15 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29923?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-29923. --- Fix Version/s: 3.0.0 Resolution: Fixed Issue resolved by pull request 26552

[jira] [Assigned] (SPARK-29923) Set `io.netty.tryReflectionSetAccessible` for Arrow on JDK9+

2019-11-15 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29923?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun reassigned SPARK-29923: - Assignee: Dongjoon Hyun > Set `io.netty.tryReflectionSetAccessible` for Arrow on JDK9+

[jira] [Updated] (SPARK-26362) Remove 'spark.driver.allowMultipleContexts' to disallow multiple Spark contexts

2019-11-15 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-26362?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-26362: -- Labels: release-notes (was: releasenotes) > Remove 'spark.driver.allowMultipleContexts' to

[jira] [Updated] (SPARK-26651) Use Proleptic Gregorian calendar

2019-11-15 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-26651?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-26651: -- Labels: release-notes (was: ReleaseNote) > Use Proleptic Gregorian calendar >

[jira] [Commented] (SPARK-29890) Unable to fill na with 0 with duplicate columns

2019-11-15 Thread Terry Kim (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29890?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16975609#comment-16975609 ] Terry Kim commented on SPARK-29890: --- Sure. I will take a look. > Unable to fill na with 0 with

[jira] [Commented] (SPARK-29890) Unable to fill na with 0 with duplicate columns

2019-11-15 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29890?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16975606#comment-16975606 ] Wenchen Fan commented on SPARK-29890: - seems like another self-join bug. [~imback82] can you take a

[jira] [Resolved] (SPARK-29867) add __repr__ in Python ML Models

2019-11-15 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29867?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-29867. --- Fix Version/s: 3.0.0 Resolution: Fixed Issue resolved by pull request 26489

[jira] [Assigned] (SPARK-29867) add __repr__ in Python ML Models

2019-11-15 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29867?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun reassigned SPARK-29867: - Assignee: Huaxin Gao > add __repr__ in Python ML Models >

[jira] [Updated] (SPARK-29890) Unable to fill na with 0 with duplicate columns

2019-11-15 Thread sandeshyapuram (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29890?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] sandeshyapuram updated SPARK-29890: --- Affects Version/s: 2.4.3 > Unable to fill na with 0 with duplicate columns >

[jira] [Assigned] (SPARK-29834) DESC DATABASE should look up catalog like v2 commands

2019-11-15 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29834?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun reassigned SPARK-29834: - Assignee: Hu Fuwang > DESC DATABASE should look up catalog like v2 commands >

[jira] [Resolved] (SPARK-29834) DESC DATABASE should look up catalog like v2 commands

2019-11-15 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29834?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-29834. --- Fix Version/s: 3.0.0 Resolution: Fixed Issue resolved by pull request 26513

[jira] [Resolved] (SPARK-29127) Add a Python, Pandas and PyArrow versions in clue at SQL query tests

2019-11-15 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29127?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-29127. --- Fix Version/s: 3.0.0 Assignee: Hyukjin Kwon Resolution: Fixed This is

[jira] [Updated] (SPARK-29127) Add a Python, Pandas and PyArrow versions in clue at SQL query tests

2019-11-15 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29127?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-29127: -- Reporter: Hyukjin Kwon (was: Burak Yavuz) > Add a Python, Pandas and PyArrow versions in

[jira] [Updated] (SPARK-29908) Add a Python, Pandas and PyArrow versions in clue at SQL query tests

2019-11-15 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29908?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-29908: -- Priority: Blocker (was: Major) > Add a Python, Pandas and PyArrow versions in clue at SQL

[jira] [Updated] (SPARK-29127) Add a Python, Pandas and PyArrow versions in clue at SQL query tests

2019-11-15 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29127?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-29127: -- Summary: Add a Python, Pandas and PyArrow versions in clue at SQL query tests (was: Support

[jira] [Updated] (SPARK-29127) Add a Python, Pandas and PyArrow versions in clue at SQL query tests

2019-11-15 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29127?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-29127: -- Description: Once Python test cases is failed in integrated UDF test cases, it's difficult

[jira] [Updated] (SPARK-29908) Support partitioning for DataSource V2 tables in DataFrameWriter.save

2019-11-15 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29908?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-29908: -- Summary: Support partitioning for DataSource V2 tables in DataFrameWriter.save (was: Add a

[jira] [Updated] (SPARK-29908) Add a Python, Pandas and PyArrow versions in clue at SQL query tests

2019-11-15 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29908?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-29908: -- Component/s: (was: PySpark) > Add a Python, Pandas and PyArrow versions in clue at SQL

[jira] [Updated] (SPARK-29127) Support partitioning for DataSource V2 tables in DataFrameWriter.save

2019-11-15 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29127?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-29127: -- Priority: Major (was: Blocker) > Support partitioning for DataSource V2 tables in

[jira] [Updated] (SPARK-29127) Support partitioning for DataSource V2 tables in DataFrameWriter.save

2019-11-15 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29127?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-29127: -- Component/s: PySpark > Support partitioning for DataSource V2 tables in DataFrameWriter.save

[jira] [Updated] (SPARK-29908) Support partitioning for DataSource V2 tables in DataFrameWriter.save

2019-11-15 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29908?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-29908: -- Description: Currently, any data source that that upgrades to DataSource V2 loses the

[jira] [Updated] (SPARK-29908) Support partitioning for DataSource V2 tables in DataFrameWriter.save

2019-11-15 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29908?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-29908: -- Reporter: Burak Yavuz (was: Hyukjin Kwon) > Support partitioning for DataSource V2 tables in

[jira] [Commented] (SPARK-29127) Support partitioning for DataSource V2 tables in DataFrameWriter.save

2019-11-15 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29127?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16975539#comment-16975539 ] Dongjoon Hyun commented on SPARK-29127: --- Hi, [~brkyvz] and [~hyukjin.kwon]. Sorry, but I'll

[jira] [Commented] (SPARK-29924) Document Arrow requirement in JDK9+

2019-11-15 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29924?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16975537#comment-16975537 ] Dongjoon Hyun commented on SPARK-29924: --- cc [~bryanc] > Document Arrow requirement in JDK9+ >

[jira] [Created] (SPARK-29924) Document Arrow requirement in JDK9+

2019-11-15 Thread Dongjoon Hyun (Jira)
Dongjoon Hyun created SPARK-29924: - Summary: Document Arrow requirement in JDK9+ Key: SPARK-29924 URL: https://issues.apache.org/jira/browse/SPARK-29924 Project: Spark Issue Type: Sub-task

[jira] [Updated] (SPARK-29923) Set `io.netty.tryReflectionSetAccessible` for Arrow on JDK9+

2019-11-15 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29923?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-29923: -- Parent: SPARK-29194 Issue Type: Sub-task (was: Improvement) > Set

[jira] [Created] (SPARK-29923) Set `io.netty.tryReflectionSetAccessible` for Arrow on JDK9+

2019-11-15 Thread Dongjoon Hyun (Jira)
Dongjoon Hyun created SPARK-29923: - Summary: Set `io.netty.tryReflectionSetAccessible` for Arrow on JDK9+ Key: SPARK-29923 URL: https://issues.apache.org/jira/browse/SPARK-29923 Project: Spark

[jira] [Resolved] (SPARK-29833) Add FileNotFoundException check for spark.yarn.jars

2019-11-15 Thread Marcelo Masiero Vanzin (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29833?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Masiero Vanzin resolved SPARK-29833. Fix Version/s: 3.0.0 Assignee: ulysses you Resolution:

[jira] [Resolved] (SPARK-29904) Parse timestamps in microsecond precision by JSON/CSV datasources

2019-11-15 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29904?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-29904. --- Fix Version/s: 2.4.5 Assignee: Maxim Gekk Resolution: Fixed This is

[jira] [Updated] (SPARK-29904) Parse timestamps in microsecond precision by JSON/CSV datasources

2019-11-15 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29904?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-29904: -- Issue Type: Bug (was: Improvement) > Parse timestamps in microsecond precision by JSON/CSV

[jira] [Updated] (SPARK-29829) SHOW TABLE EXTENDED should look up catalog/table like v2 commands

2019-11-15 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29829?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-29829: -- Fix Version/s: (was: 3.1.0) 3.0.0 > SHOW TABLE EXTENDED should look up

[jira] [Resolved] (SPARK-29829) SHOW TABLE EXTENDED should look up catalog/table like v2 commands

2019-11-15 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29829?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-29829. --- Fix Version/s: 3.1.0 Resolution: Fixed Issue resolved by pull request 26540

[jira] [Assigned] (SPARK-29829) SHOW TABLE EXTENDED should look up catalog/table like v2 commands

2019-11-15 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29829?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun reassigned SPARK-29829: - Assignee: Pablo Langa Blanco > SHOW TABLE EXTENDED should look up catalog/table like

[jira] [Updated] (SPARK-29794) Column level compression

2019-11-15 Thread Anirudh Vyas (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29794?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anirudh Vyas updated SPARK-29794: - Affects Version/s: 3.0.0 > Column level compression > > >

[jira] [Updated] (SPARK-29334) Supported vector operators in scala should have parity with pySpark

2019-11-15 Thread Sean R. Owen (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29334?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean R. Owen updated SPARK-29334: - Shepherd: (was: Sean R. Owen) > Supported vector operators in scala should have parity with

[jira] [Commented] (SPARK-29911) Cache table may memory leak when session closed

2019-11-15 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29911?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16975307#comment-16975307 ] Dongjoon Hyun commented on SPARK-29911: --- Hi, [~cltlfcjin]. Since is reported as a memory leakage

[jira] [Commented] (SPARK-29918) RecordBinaryComparator should check endianness when compared by long

2019-11-15 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29918?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16975305#comment-16975305 ] Dongjoon Hyun commented on SPARK-29918: --- Hi, [~EdisonWang]. What about the older Spark versions?

[jira] [Commented] (SPARK-29900) make relation lookup behavior consistent within Spark SQL

2019-11-15 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29900?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16975303#comment-16975303 ] Dongjoon Hyun commented on SPARK-29900: --- Thank you for pinging me, [~cloud_fan]. [~imback82].

[jira] [Updated] (SPARK-29906) Reading of csv file fails with adaptive execution turned on

2019-11-15 Thread koert kuipers (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29906?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] koert kuipers updated SPARK-29906: -- Labels: correctness (was: ) > Reading of csv file fails with adaptive execution turned on >

[jira] [Commented] (SPARK-29906) Reading of csv file fails with adaptive execution turned on

2019-11-15 Thread koert kuipers (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29906?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16975286#comment-16975286 ] koert kuipers commented on SPARK-29906: --- note that with the default option for csv being

[jira] [Created] (SPARK-29922) SHOW FUNCTIONS should look up catalog/table like v2 commands

2019-11-15 Thread Pablo Langa Blanco (Jira)
Pablo Langa Blanco created SPARK-29922: -- Summary: SHOW FUNCTIONS should look up catalog/table like v2 commands Key: SPARK-29922 URL: https://issues.apache.org/jira/browse/SPARK-29922 Project:

[jira] [Commented] (SPARK-29922) SHOW FUNCTIONS should look up catalog/table like v2 commands

2019-11-15 Thread Pablo Langa Blanco (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29922?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16975272#comment-16975272 ] Pablo Langa Blanco commented on SPARK-29922: I'm working on this > SHOW FUNCTIONS should

[jira] [Created] (SPARK-29921) SparkContext LiveListenerBus

2019-11-15 Thread Arun sethia (Jira)
Arun sethia created SPARK-29921: --- Summary: SparkContext LiveListenerBus Key: SPARK-29921 URL: https://issues.apache.org/jira/browse/SPARK-29921 Project: Spark Issue Type: Improvement

[jira] [Created] (SPARK-29920) Parsing failure on interval '20 15' day to hour

2019-11-15 Thread Maxim Gekk (Jira)
Maxim Gekk created SPARK-29920: -- Summary: Parsing failure on interval '20 15' day to hour Key: SPARK-29920 URL: https://issues.apache.org/jira/browse/SPARK-29920 Project: Spark Issue Type: Bug

[jira] [Created] (SPARK-29919) remove python2 test execution

2019-11-15 Thread Shane Knapp (Jira)
Shane Knapp created SPARK-29919: --- Summary: remove python2 test execution Key: SPARK-29919 URL: https://issues.apache.org/jira/browse/SPARK-29919 Project: Spark Issue Type: Sub-task

[jira] [Updated] (SPARK-29918) RecordBinaryComparator should check endianness when compared by long

2019-11-15 Thread EdisonWang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29918?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] EdisonWang updated SPARK-29918: --- Labels: correctness (was: ) > RecordBinaryComparator should check endianness when compared by long

[jira] [Created] (SPARK-29918) RecordBinaryComparator should check endianness when compared by long

2019-11-15 Thread EdisonWang (Jira)
EdisonWang created SPARK-29918: -- Summary: RecordBinaryComparator should check endianness when compared by long Key: SPARK-29918 URL: https://issues.apache.org/jira/browse/SPARK-29918 Project: Spark

[jira] [Created] (SPARK-29917) Provide functionality to rename Receivers on Spark Streaming Page

2019-11-15 Thread Jira
Burak KÖSE created SPARK-29917: -- Summary: Provide functionality to rename Receivers on Spark Streaming Page Key: SPARK-29917 URL: https://issues.apache.org/jira/browse/SPARK-29917 Project: Spark

[jira] [Resolved] (SPARK-29902) Add listener event queue capacity configuration to documentation

2019-11-15 Thread Sean R. Owen (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29902?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean R. Owen resolved SPARK-29902. -- Fix Version/s: 3.1.0 Resolution: Fixed Issue resolved by pull request 26529

[jira] [Assigned] (SPARK-29902) Add listener event queue capacity configuration to documentation

2019-11-15 Thread Sean R. Owen (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29902?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean R. Owen reassigned SPARK-29902: Assignee: shahid > Add listener event queue capacity configuration to documentation >

[jira] [Updated] (SPARK-29902) Add listener event queue capacity configuration to documentation

2019-11-15 Thread Sean R. Owen (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29902?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean R. Owen updated SPARK-29902: - Fix Version/s: (was: 3.1.0) 3.0.0 > Add listener event queue capacity

[jira] [Created] (SPARK-29916) spark on kubernetes fails with hadoop-3.2 due to the user not existing in executor pod

2019-11-15 Thread Jira
Michał Wesołowski created SPARK-29916: - Summary: spark on kubernetes fails with hadoop-3.2 due to the user not existing in executor pod Key: SPARK-29916 URL: https://issues.apache.org/jira/browse/SPARK-29916

[jira] [Commented] (SPARK-29748) Remove sorting of fields in PySpark SQL Row creation

2019-11-15 Thread Maciej Szymkiewicz (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29748?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16975054#comment-16975054 ] Maciej Szymkiewicz commented on SPARK-29748: [~jhereth] {quote}With simply removing sorting

[jira] [Created] (SPARK-29915) spark-py and spark-r images are not created with docker-image-tool.sh

2019-11-15 Thread Jira
Michał Wesołowski created SPARK-29915: - Summary: spark-py and spark-r images are not created with docker-image-tool.sh Key: SPARK-29915 URL: https://issues.apache.org/jira/browse/SPARK-29915

[jira] [Updated] (SPARK-29114) Dataset.coalesce(10) throw ChunkFetchFailureException when original Dataset partition size is big

2019-11-15 Thread ZhanxiongWang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29114?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ZhanxiongWang updated SPARK-29114: -- Description: Updated time:15/Nov/19 We saw this blog solved our confusion.

[jira] [Updated] (SPARK-29114) Dataset.coalesce(10) throw ChunkFetchFailureException when original Dataset partition size is big

2019-11-15 Thread ZhanxiongWang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29114?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ZhanxiongWang updated SPARK-29114: -- Description: Updated time:15/Nov/19 We saw this blog solved our confusion.

[jira] [Updated] (SPARK-29114) Dataset.coalesce(10) throw ChunkFetchFailureException when original Dataset partition size is big

2019-11-15 Thread ZhanxiongWang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29114?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ZhanxiongWang updated SPARK-29114: -- Description: Updated time:15/Nov/19 We saw this blog solved our confusion.

[jira] [Updated] (SPARK-29114) Dataset.coalesce(10) throw ChunkFetchFailureException when original Dataset partition size is big

2019-11-15 Thread ZhanxiongWang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29114?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ZhanxiongWang updated SPARK-29114: -- Description: Updated time:15/Nov/19 I discussed this issue with my colleagues today. We

[jira] [Updated] (SPARK-29114) Dataset.coalesce(10) throw ChunkFetchFailureException when original Dataset partition size is big

2019-11-15 Thread ZhanxiongWang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29114?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ZhanxiongWang updated SPARK-29114: -- Description: Updated time:15/Nov/19 I discussed this issue with my colleagues today. We think

[jira] [Created] (SPARK-29914) ML models append metadata in `transform`/`transformSchema`

2019-11-15 Thread zhengruifeng (Jira)
zhengruifeng created SPARK-29914: Summary: ML models append metadata in `transform`/`transformSchema` Key: SPARK-29914 URL: https://issues.apache.org/jira/browse/SPARK-29914 Project: Spark

[jira] [Created] (SPARK-29913) Improve Exception in postgreCastToBoolean

2019-11-15 Thread jobit mathew (Jira)
jobit mathew created SPARK-29913: Summary: Improve Exception in postgreCastToBoolean Key: SPARK-29913 URL: https://issues.apache.org/jira/browse/SPARK-29913 Project: Spark Issue Type:

[jira] [Resolved] (SPARK-29619) Add retry times when reading the daemon port.

2019-11-15 Thread Sean R. Owen (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29619?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean R. Owen resolved SPARK-29619. -- Resolution: Won't Fix > Add retry times when reading the daemon port. >

[jira] [Updated] (SPARK-29894) Add Codegen Stage Id to Spark plan graphs in Web UI SQL Tab

2019-11-15 Thread Luca Canali (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29894?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Luca Canali updated SPARK-29894: Attachment: Physical_plan_Annotated.png > Add Codegen Stage Id to Spark plan graphs in Web UI SQL

[jira] [Updated] (SPARK-29912) Pruning shuffle exchange and coalesce when input and output both are one partition

2019-11-15 Thread ulysses you (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29912?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ulysses you updated SPARK-29912: Description: It is meaningless to do `repartition(1)` or `coalesce(1)` when a child plan just

[jira] [Updated] (SPARK-29912) Pruning shuffle exchange and coalesce when input and output both are one partition

2019-11-15 Thread ulysses you (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29912?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ulysses you updated SPARK-29912: Description: It is meaningless to do `repartition(1)` or `coalesce(1)` when a plan child just

[jira] [Updated] (SPARK-29912) Pruning shuffle exchange and coalesce when input and output both are one partition

2019-11-15 Thread ulysses you (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29912?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ulysses you updated SPARK-29912: Description: It is meaningless to do `repartition(1)` or `coalesce(1)` when a plan child just

[jira] [Updated] (SPARK-29912) Pruning shuffle exchange and coalesce when input and output both are one partition

2019-11-15 Thread ulysses you (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29912?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ulysses you updated SPARK-29912: Summary: Pruning shuffle exchange and coalesce when input and output both are one partition

[jira] [Created] (SPARK-29912) Pruning shuffle exchange when input and output both are one partition

2019-11-15 Thread ulysses you (Jira)
ulysses you created SPARK-29912: --- Summary: Pruning shuffle exchange when input and output both are one partition Key: SPARK-29912 URL: https://issues.apache.org/jira/browse/SPARK-29912 Project: Spark