[jira] [Updated] (SPARK-19528) external shuffle service registration timeout is very short with heavy workloads when dynamic allocation is enabled

[ https://issues.apache.org/jira/browse/SPARK-19528?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] KaiXu updated SPARK-19528: -- Attachment: SPARK-19528.1.spark2.patch > external shuffle service registration timeout is very short with

[jira] [Updated] (SPARK-23775) Flaky test: DataFrameRangeSuite

[ https://issues.apache.org/jira/browse/SPARK-23775?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gabor Somogyi updated SPARK-23775: -- Attachment: filtered.log > Flaky test: DataFrameRangeSuite > --- >

[jira] [Commented] (SPARK-23772) Provide an option to ignore column of all null values or empty map/array during JSON schema inference

[ https://issues.apache.org/jira/browse/SPARK-23772?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16410791#comment-16410791 ] Takeshi Yamamuro commented on SPARK-23772: -- ya, I'll take this, thanks! > Provide an option to

[jira] [Commented] (SPARK-23772) Provide an option to ignore column of all null values or empty map/array during JSON schema inference

[ https://issues.apache.org/jira/browse/SPARK-23772?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16410790#comment-16410790 ] Xiao Li commented on SPARK-23772: - cc [~maropu] Are you interested in this? > Provide an option to

[jira] [Resolved] (SPARK-23614) Union produces incorrect results when caching is used

[ https://issues.apache.org/jira/browse/SPARK-23614?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-23614. - Resolution: Fixed Fix Version/s: 2.3.1 2.4.0 Issue resolved by pull

[jira] [Commented] (SPARK-19459) ORC tables cannot be read when they contain char/varchar columns

[ https://issues.apache.org/jira/browse/SPARK-19459?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16410781#comment-16410781 ] Apache Spark commented on SPARK-19459: -- User 'dongjoon-hyun' has created a pull request for this

[jira] [Commented] (SPARK-17570) Avoid Hash and Exchange in Sort Merge join if bucketing factor is multiple for tables

[ https://issues.apache.org/jira/browse/SPARK-17570?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16410752#comment-16410752 ] Xiaoju Wu commented on SPARK-17570: --- [~tejasp] When you join 3 tables with bucket number 4,8,12, if

[jira] [Created] (SPARK-23776) pyspark-sql tests should display build instructions when components are missing

Bruce Robbins created SPARK-23776: - Summary: pyspark-sql tests should display build instructions when components are missing Key: SPARK-23776 URL: https://issues.apache.org/jira/browse/SPARK-23776

[jira] [Updated] (SPARK-23775) Flaky test: DataFrameRangeSuite

[ https://issues.apache.org/jira/browse/SPARK-23775?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gabor Somogyi updated SPARK-23775: -- Description: DataFrameRangeSuite.test("Cancelling stage in a query with Range.") stays

[jira] [Commented] (SPARK-23775) Flaky test: DataFrameRangeSuite

[ https://issues.apache.org/jira/browse/SPARK-23775?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16410728#comment-16410728 ] Apache Spark commented on SPARK-23775: -- User 'gaborgsomogyi' has created a pull request for this

[jira] [Assigned] (SPARK-23775) Flaky test: DataFrameRangeSuite

[ https://issues.apache.org/jira/browse/SPARK-23775?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-23775: Assignee: Apache Spark > Flaky test: DataFrameRangeSuite >

[jira] [Assigned] (SPARK-23775) Flaky test: DataFrameRangeSuite

[ https://issues.apache.org/jira/browse/SPARK-23775?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-23775: Assignee: (was: Apache Spark) > Flaky test: DataFrameRangeSuite >

[jira] [Updated] (SPARK-23774) `Cast` to CHAR/VARCHAR should truncate the values

[ https://issues.apache.org/jira/browse/SPARK-23774?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-23774: -- Affects Version/s: 2.1.2 > `Cast` to CHAR/VARCHAR should truncate the values >

[jira] [Commented] (SPARK-23775) Flaky test: DataFrameRangeSuite

[ https://issues.apache.org/jira/browse/SPARK-23775?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16410715#comment-16410715 ] Gabor Somogyi commented on SPARK-23775: --- Working on it. > Flaky test: DataFrameRangeSuite >

[jira] [Updated] (SPARK-23775) Flaky test: DataFrameRangeSuite

[ https://issues.apache.org/jira/browse/SPARK-23775?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gabor Somogyi updated SPARK-23775: -- Description: DataFrameRangeSuite.test("Cancelling stage in a query with Range.") stays

[jira] [Created] (SPARK-23775) Flaky test: DataFrameRangeSuite

Gabor Somogyi created SPARK-23775: - Summary: Flaky test: DataFrameRangeSuite Key: SPARK-23775 URL: https://issues.apache.org/jira/browse/SPARK-23775 Project: Spark Issue Type: Bug

[jira] [Commented] (SPARK-19528) external shuffle service registration timeout is very short with heavy workloads when dynamic allocation is enabled

[ https://issues.apache.org/jira/browse/SPARK-19528?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16410655#comment-16410655 ] KaiXu commented on SPARK-19528: --- spark2.0.2 also found this issue recently! > external shuffle service

[jira] [Updated] (SPARK-19528) external shuffle service registration timeout is very short with heavy workloads when dynamic allocation is enabled

[ https://issues.apache.org/jira/browse/SPARK-19528?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] KaiXu updated SPARK-19528: -- Affects Version/s: 2.0.2 > external shuffle service registration timeout is very short with heavy > workloads

[jira] [Updated] (SPARK-23759) Unable to bind Spark UI to specific host name / IP

[ https://issues.apache.org/jira/browse/SPARK-23759?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Felix updated SPARK-23759: -- Description: Ideally, exporting SPARK_LOCAL_IP= in spark2 environment should allow Spark2 History server to

[jira] [Updated] (SPARK-23759) Unable to bind Spark UI to specific host name / IP

[ https://issues.apache.org/jira/browse/SPARK-23759?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Felix updated SPARK-23759: -- Summary: Unable to bind Spark UI to specific host name / IP (was: Unable to bind Spark2 history server to

[jira] [Updated] (SPARK-23774) `Cast` to CHAR/VARCHAR should truncate the values

[ https://issues.apache.org/jira/browse/SPARK-23774?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-23774: -- Description: This issue aims to fix the following `CAST` behavior on `CHAR/VARCHAR` types.

[jira] [Updated] (SPARK-23774) `Cast` to CHAR/VARCHAR should truncate the values

[ https://issues.apache.org/jira/browse/SPARK-23774?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-23774: -- Description: This issue aims to fix the following `CAST` behavior on `CHAR/VARCHAR` types.

[jira] [Assigned] (SPARK-23774) `Cast` to CHAR/VARCHAR should truncate the values

[ https://issues.apache.org/jira/browse/SPARK-23774?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-23774: Assignee: Apache Spark > `Cast` to CHAR/VARCHAR should truncate the values >

[jira] [Assigned] (SPARK-23774) `Cast` to CHAR/VARCHAR should truncate the values

[ https://issues.apache.org/jira/browse/SPARK-23774?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-23774: Assignee: (was: Apache Spark) > `Cast` to CHAR/VARCHAR should truncate the values >

[jira] [Commented] (SPARK-23774) `Cast` to CHAR/VARCHAR should truncate the values

[ https://issues.apache.org/jira/browse/SPARK-23774?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16410551#comment-16410551 ] Apache Spark commented on SPARK-23774: -- User 'dongjoon-hyun' has created a pull request for this

[jira] [Updated] (SPARK-23774) `Cast` to CHAR/VARCHAR should truncate the values

[ https://issues.apache.org/jira/browse/SPARK-23774?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-23774: -- Labels: Correctness (was: ) > `Cast` to CHAR/VARCHAR should truncate the values >

[jira] [Updated] (SPARK-23774) `Cast` to CHAR/VARCHAR should truncate the values

[ https://issues.apache.org/jira/browse/SPARK-23774?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-23774: -- Labels: (was: Correctness) > `Cast` to CHAR/VARCHAR should truncate the values >

[jira] [Created] (SPARK-23774) `Cast` to CHAR/VARCHAR should truncate the values

Dongjoon Hyun created SPARK-23774: - Summary: `Cast` to CHAR/VARCHAR should truncate the values Key: SPARK-23774 URL: https://issues.apache.org/jira/browse/SPARK-23774 Project: Spark Issue

[jira] [Commented] (SPARK-19724) create a managed table with an existed default location should throw an exception

[ https://issues.apache.org/jira/browse/SPARK-19724?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16410526#comment-16410526 ] Apache Spark commented on SPARK-19724: -- User 'gengliangwang' has created a pull request for this

[jira] [Commented] (SPARK-22239) User-defined window functions with pandas udf

[ https://issues.apache.org/jira/browse/SPARK-22239?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16410521#comment-16410521 ] Herman van Hovell commented on SPARK-22239: --- Unbouded windows should be relatively

[jira] [Assigned] (SPARK-23724) Custom record separator for jsons in charsets different from UTF-8

[ https://issues.apache.org/jira/browse/SPARK-23724?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-23724: Assignee: Apache Spark > Custom record separator for jsons in charsets different from

[jira] [Assigned] (SPARK-23724) Custom record separator for jsons in charsets different from UTF-8

[ https://issues.apache.org/jira/browse/SPARK-23724?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-23724: Assignee: (was: Apache Spark) > Custom record separator for jsons in charsets

[jira] [Commented] (SPARK-23765) Supports line separator for json datasource

[ https://issues.apache.org/jira/browse/SPARK-23765?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16410507#comment-16410507 ] Apache Spark commented on SPARK-23765: -- User 'MaxGekk' has created a pull request for this issue:

[jira] [Commented] (SPARK-23724) Custom record separator for jsons in charsets different from UTF-8

[ https://issues.apache.org/jira/browse/SPARK-23724?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16410506#comment-16410506 ] Apache Spark commented on SPARK-23724: -- User 'MaxGekk' has created a pull request for this issue:

[jira] [Commented] (SPARK-23773) JacksonGenerator does not include keys that have null value for StructTypes

[ https://issues.apache.org/jira/browse/SPARK-23773?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16410361#comment-16410361 ] Apache Spark commented on SPARK-23773: -- User 'makagonov' has created a pull request for this issue:

[jira] [Assigned] (SPARK-23773) JacksonGenerator does not include keys that have null value for StructTypes

[ https://issues.apache.org/jira/browse/SPARK-23773?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-23773: Assignee: Apache Spark > JacksonGenerator does not include keys that have null value for

[jira] [Assigned] (SPARK-23773) JacksonGenerator does not include keys that have null value for StructTypes

[ https://issues.apache.org/jira/browse/SPARK-23773?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-23773: Assignee: (was: Apache Spark) > JacksonGenerator does not include keys that have null

[jira] [Commented] (SPARK-19767) API Doc pages for Streaming with Kafka 0.10 not current

[ https://issues.apache.org/jira/browse/SPARK-19767?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16410334#comment-16410334 ] Nick Afshartous commented on SPARK-19767: - Running the full doc build using {{jekyll build}} I

[jira] [Commented] (SPARK-23759) Unable to bind Spark2 history server to specific host name / IP

[ https://issues.apache.org/jira/browse/SPARK-23759?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16410164#comment-16410164 ] Apache Spark commented on SPARK-23759: -- User 'felixalbani' has created a pull request for this

[jira] [Closed] (SPARK-23579) Provide context information in overview of Spark documentation

[ https://issues.apache.org/jira/browse/SPARK-23579?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Riaas Mokiem closed SPARK-23579. > Provide context information in overview of Spark documentation >

[jira] [Resolved] (SPARK-23579) Provide context information in overview of Spark documentation

[ https://issues.apache.org/jira/browse/SPARK-23579?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Riaas Mokiem resolved SPARK-23579. -- Resolution: Invalid The diagram was a bit too detailed to be included, and contained some

[jira] [Resolved] (SPARK-23599) The UUID() expression is too non-deterministic

[ https://issues.apache.org/jira/browse/SPARK-23599?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Herman van Hovell resolved SPARK-23599. --- Resolution: Fixed Fix Version/s: 2.4.0 > The UUID() expression is too

[jira] [Created] (SPARK-23773) JacksonGenerator does not include keys that have null value for StructTypes

Sergey created SPARK-23773: -- Summary: JacksonGenerator does not include keys that have null value for StructTypes Key: SPARK-23773 URL: https://issues.apache.org/jira/browse/SPARK-23773 Project: Spark

[jira] [Commented] (SPARK-13757) support quoted column names in schema string at types.py#_parse_datatype_string

[ https://issues.apache.org/jira/browse/SPARK-13757?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16409862#comment-16409862 ] Wenchen Fan commented on SPARK-13757: - thanks, closed. > support quoted column names in schema

[jira] [Resolved] (SPARK-13757) support quoted column names in schema string at types.py#_parse_datatype_string

[ https://issues.apache.org/jira/browse/SPARK-13757?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-13757. - Resolution: Duplicate Fix Version/s: 2.3.0 > support quoted column names in schema string

[jira] [Created] (SPARK-23772) Provide an option to ignore column of all null values or empty map/array during JSON schema inference

Xiangrui Meng created SPARK-23772: - Summary: Provide an option to ignore column of all null values or empty map/array during JSON schema inference Key: SPARK-23772 URL:

[jira] [Commented] (SPARK-23429) Add executor memory metrics to heartbeat and expose in executors REST API

[ https://issues.apache.org/jira/browse/SPARK-23429?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16409775#comment-16409775 ] Edwina Lu commented on SPARK-23429: --- [~assia6], it's being reviewed internally right now. I will post

[jira] [Commented] (SPARK-22865) Publish Official Apache Spark Docker images

[ https://issues.apache.org/jira/browse/SPARK-22865?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16409609#comment-16409609 ] Andrew Korzhuev commented on SPARK-22865: - Here is an example how this can be done with Travis CI

[jira] [Commented] (SPARK-23682) Memory issue with Spark structured streaming

[ https://issues.apache.org/jira/browse/SPARK-23682?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16409538#comment-16409538 ] Cristian Cifuentes commented on SPARK-23682: I have found the same behavior with this

[jira] [Updated] (SPARK-23682) Memory issue with Spark structured streaming

[ https://issues.apache.org/jira/browse/SPARK-23682?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Cristian Cifuentes updated SPARK-23682: --- Attachment: image-2018-03-22-14-46-31-960.png > Memory issue with Spark structured

[jira] [Created] (SPARK-23771) Uneven Rowgroup size after repartition

Johannes Mayer created SPARK-23771: -- Summary: Uneven Rowgroup size after repartition Key: SPARK-23771 URL: https://issues.apache.org/jira/browse/SPARK-23771 Project: Spark Issue Type: Bug

[jira] [Assigned] (SPARK-23769) Remove unnecessary scalastyle check disabling

[ https://issues.apache.org/jira/browse/SPARK-23769?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-23769: Assignee: Apache Spark > Remove unnecessary scalastyle check disabling >

[jira] [Commented] (SPARK-23769) Remove unnecessary scalastyle check disabling

[ https://issues.apache.org/jira/browse/SPARK-23769?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16409448#comment-16409448 ] Apache Spark commented on SPARK-23769: -- User 'arucard21' has created a pull request for this issue:

[jira] [Assigned] (SPARK-23769) Remove unnecessary scalastyle check disabling

[ https://issues.apache.org/jira/browse/SPARK-23769?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-23769: Assignee: (was: Apache Spark) > Remove unnecessary scalastyle check disabling >

[jira] [Commented] (SPARK-23769) Remove unnecessary scalastyle check disabling

[ https://issues.apache.org/jira/browse/SPARK-23769?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16409446#comment-16409446 ] Riaas Mokiem commented on SPARK-23769: -- A few of us actually went through all of the "scalastyle

[jira] [Commented] (SPARK-23769) Remove unnecessary scalastyle check disabling

[ https://issues.apache.org/jira/browse/SPARK-23769?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16409441#comment-16409441 ] Hyukjin Kwon commented on SPARK-23769: -- I think this needs no JIRA though. Mind checking similar

[jira] [Created] (SPARK-23770) Expose repartitionByRange in SparkR

Hyukjin Kwon created SPARK-23770: Summary: Expose repartitionByRange in SparkR Key: SPARK-23770 URL: https://issues.apache.org/jira/browse/SPARK-23770 Project: Spark Issue Type: Improvement

[jira] [Created] (SPARK-23769) Remove unnecessary scalastyle check disabling

Riaas Mokiem created SPARK-23769: Summary: Remove unnecessary scalastyle check disabling Key: SPARK-23769 URL: https://issues.apache.org/jira/browse/SPARK-23769 Project: Spark Issue Type:

[jira] [Created] (SPARK-23768) Proxy configuration for extraJavaOptions in defaults conf

Praveen created SPARK-23768: --- Summary: Proxy configuration for extraJavaOptions in defaults conf Key: SPARK-23768 URL: https://issues.apache.org/jira/browse/SPARK-23768 Project: Spark Issue Type:

[jira] [Commented] (SPARK-22865) Publish Official Apache Spark Docker images

[ https://issues.apache.org/jira/browse/SPARK-22865?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16409351#comment-16409351 ] Andrew Korzhuev commented on SPARK-22865: - What is the plan on resolving this issue? I see that

[jira] [Commented] (SPARK-13757) support quoted column names in schema string at types.py#_parse_datatype_string

[ https://issues.apache.org/jira/browse/SPARK-13757?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16409283#comment-16409283 ] Hyukjin Kwon commented on SPARK-13757: -- [~cloud_fan], seems it's fine to resolve this in favour of

[jira] [Created] (SPARK-23767) DirectStream is producing the incorrect type of message

Eduard Biceri Matei created SPARK-23767: --- Summary: DirectStream is producing the incorrect type of message Key: SPARK-23767 URL: https://issues.apache.org/jira/browse/SPARK-23767 Project: Spark

[jira] [Created] (SPARK-23766) Not able to execute multiple queries in spark structured streaming

Apeksha Agnihotri created SPARK-23766: - Summary: Not able to execute multiple queries in spark structured streaming Key: SPARK-23766 URL: https://issues.apache.org/jira/browse/SPARK-23766

[jira] [Assigned] (SPARK-23765) Supports line separator for json datasource

[ https://issues.apache.org/jira/browse/SPARK-23765?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-23765: Assignee: Apache Spark > Supports line separator for json datasource >

[jira] [Commented] (SPARK-23765) Supports line separator for json datasource

[ https://issues.apache.org/jira/browse/SPARK-23765?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16409150#comment-16409150 ] Apache Spark commented on SPARK-23765: -- User 'HyukjinKwon' has created a pull request for this

[jira] [Assigned] (SPARK-23765) Supports line separator for json datasource

[ https://issues.apache.org/jira/browse/SPARK-23765?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-23765: Assignee: (was: Apache Spark) > Supports line separator for json datasource >

[jira] [Commented] (SPARK-23653) Show sql statement in spark SQL UI

[ https://issues.apache.org/jira/browse/SPARK-23653?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16409143#comment-16409143 ] Apache Spark commented on SPARK-23653: -- User 'LantaoJin' has created a pull request for this issue:

[jira] [Created] (SPARK-23765) Supports line separator for json datasource

Hyukjin Kwon created SPARK-23765: Summary: Supports line separator for json datasource Key: SPARK-23765 URL: https://issues.apache.org/jira/browse/SPARK-23765 Project: Spark Issue Type: