[jira] [Commented] (SPARK-24705) Spark.sql.adaptive.enabled=true is enabled and self-join query

[ https://issues.apache.org/jira/browse/SPARK-24705?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16533212#comment-16533212 ] Takeshi Yamamuro commented on SPARK-24705: -- ok, will do. > Spark.sql.adaptive.enabled=true is

[jira] [Commented] (SPARK-24723) Discuss necessary info and access in barrier mode + YARN

[ https://issues.apache.org/jira/browse/SPARK-24723?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16533205#comment-16533205 ] Saisai Shao commented on SPARK-24723: - Hi [~mengxr], I would like to know the goal of this ticket?

[jira] [Commented] (SPARK-24705) Spark.sql.adaptive.enabled=true is enabled and self-join query

[ https://issues.apache.org/jira/browse/SPARK-24705?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16533195#comment-16533195 ] Xiao Li commented on SPARK-24705: - [~maropu]Thank you for taking this. Please try to fix it? >

[jira] [Commented] (SPARK-24739) PySpark does not work with Python 3.7.0

[ https://issues.apache.org/jira/browse/SPARK-24739?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16533191#comment-16533191 ] Hyukjin Kwon commented on SPARK-24739: -- For this fix particularly, the fix is small and safe. In

[jira] [Commented] (SPARK-24739) PySpark does not work with Python 3.7.0

[ https://issues.apache.org/jira/browse/SPARK-24739?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16533189#comment-16533189 ] Saisai Shao commented on SPARK-24739: - Do we have to fix it in 2.3.2? I don't think it is even

[jira] [Resolved] (SPARK-24698) In Pyspark's ML, an Identifiable's UID has 20 random characters rather than the 12 mentioned in the documentation.

[ https://issues.apache.org/jira/browse/SPARK-24698?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-24698. -- Resolution: Fixed Fix Version/s: 2.4.0 Issue resolved by pull request 21675

[jira] [Assigned] (SPARK-24698) In Pyspark's ML, an Identifiable's UID has 20 random characters rather than the 12 mentioned in the documentation.

[ https://issues.apache.org/jira/browse/SPARK-24698?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon reassigned SPARK-24698: Assignee: Thomas Dunne > In Pyspark's ML, an Identifiable's UID has 20 random characters

[jira] [Resolved] (SPARK-24549) Support DecimalType push down to the parquet data sources

[ https://issues.apache.org/jira/browse/SPARK-24549?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-24549. -- Resolution: Duplicate > Support DecimalType push down to the parquet data sources >

[jira] [Updated] (SPARK-24741) Have a built-in AVRO data source implementation

[ https://issues.apache.org/jira/browse/SPARK-24741?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Li updated SPARK-24741: Description: Apache Avro (https://avro.apache.org) is a data serialization format. It is widely used in

[jira] [Updated] (SPARK-24741) Have a built-in AVRO data source implementation

[ https://issues.apache.org/jira/browse/SPARK-24741?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Li updated SPARK-24741: Description: Apache Avro (https://avro.apache.org) is a data serialization format. It is widely used in

[jira] [Updated] (SPARK-24741) Have a built-in AVRO data source implementation

[ https://issues.apache.org/jira/browse/SPARK-24741?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Li updated SPARK-24741: Component/s: Structured Streaming > Have a built-in AVRO data source implementation >

[jira] [Created] (SPARK-24741) Have a built-in AVRO data source implementation

Xiao Li created SPARK-24741: --- Summary: Have a built-in AVRO data source implementation Key: SPARK-24741 URL: https://issues.apache.org/jira/browse/SPARK-24741 Project: Spark Issue Type: New

[jira] [Commented] (SPARK-24741) Have a built-in AVRO data source implementation

[ https://issues.apache.org/jira/browse/SPARK-24741?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16533137#comment-16533137 ] Xiao Li commented on SPARK-24741: - cc [~Gengliang.Wang] > Have a built-in AVRO data source

[jira] [Commented] (SPARK-17213) Parquet String Pushdown for Non-Eq Comparisons Broken

[ https://issues.apache.org/jira/browse/SPARK-17213?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16533083#comment-16533083 ] Apache Spark commented on SPARK-17213: -- User 'gatorsmile' has created a pull request for this

[jira] [Commented] (SPARK-23682) Memory issue with Spark structured streaming

[ https://issues.apache.org/jira/browse/SPARK-23682?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16533078#comment-16533078 ] Jungtaek Lim commented on SPARK-23682: -- [~bondyk] [~ccifuentes] [~akorzhuev] This may not due to

[jira] [Updated] (SPARK-24739) PySpark does not work with Python 3.7.0

[ https://issues.apache.org/jira/browse/SPARK-24739?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen updated SPARK-24739: -- Target Version/s: 2.3.2, 2.4.0 (was: 2.1.3, 2.2.2, 2.3.1, 2.4.0) > PySpark does not work with Python

[jira] [Updated] (SPARK-24672) No job is running but there are active tasks

[ https://issues.apache.org/jira/browse/SPARK-24672?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen updated SPARK-24672: -- Target Version/s: (was: 2.2.1) > No job is running but there are active tasks >

[jira] [Updated] (SPARK-24710) Information Gain Ratio for decision trees

[ https://issues.apache.org/jira/browse/SPARK-24710?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen updated SPARK-24710: -- Target Version/s: (was: 2.3.1) Fix Version/s: (was: 2.3.1) > Information Gain Ratio for

[jira] [Updated] (SPARK-24530) Sphinx doesn't render autodoc_docstring_signature correctly (with Python 2?) and pyspark.ml docs are broken

[ https://issues.apache.org/jira/browse/SPARK-24530?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen updated SPARK-24530: -- Target Version/s: 2.3.2, 2.4.0 (was: 2.1.3, 2.2.2, 2.3.2, 2.4.0) > Sphinx doesn't render

[jira] [Assigned] (SPARK-24740) PySpark tests do not pass with NumPy 0.14.x+

[ https://issues.apache.org/jira/browse/SPARK-24740?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-24740: Assignee: Hyukjin Kwon (was: Apache Spark) > PySpark tests do not pass with NumPy

[jira] [Commented] (SPARK-24740) PySpark tests do not pass with NumPy 0.14.x+

[ https://issues.apache.org/jira/browse/SPARK-24740?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16532928#comment-16532928 ] Apache Spark commented on SPARK-24740: -- User 'HyukjinKwon' has created a pull request for this

[jira] [Assigned] (SPARK-24740) PySpark tests do not pass with NumPy 0.14.x+

[ https://issues.apache.org/jira/browse/SPARK-24740?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-24740: Assignee: Apache Spark (was: Hyukjin Kwon) > PySpark tests do not pass with NumPy

[jira] [Comment Edited] (SPARK-24474) Cores are left idle when there are a lot of tasks to run

[ https://issues.apache.org/jira/browse/SPARK-24474?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16532910#comment-16532910 ] Al M edited comment on SPARK-24474 at 7/4/18 4:17 PM: -- My initial tests suggest

[jira] [Commented] (SPARK-24739) PySpark does not work with Python 3.7.0

[ https://issues.apache.org/jira/browse/SPARK-24739?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16532912#comment-16532912 ] Apache Spark commented on SPARK-24739: -- User 'HyukjinKwon' has created a pull request for this

[jira] [Assigned] (SPARK-24739) PySpark does not work with Python 3.7.0

[ https://issues.apache.org/jira/browse/SPARK-24739?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-24739: Assignee: Hyukjin Kwon (was: Apache Spark) > PySpark does not work with Python 3.7.0 >

[jira] [Assigned] (SPARK-24739) PySpark does not work with Python 3.7.0

[ https://issues.apache.org/jira/browse/SPARK-24739?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-24739: Assignee: Apache Spark (was: Hyukjin Kwon) > PySpark does not work with Python 3.7.0 >

[jira] [Commented] (SPARK-24474) Cores are left idle when there are a lot of tasks to run

[ https://issues.apache.org/jira/browse/SPARK-24474?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16532910#comment-16532910 ] Al M commented on SPARK-24474: -- My initial tests suggest that this stops the issue from happening.  Thanks! 

[jira] [Commented] (SPARK-24740) PySpark tests do not pass with NumPy 0.14.x+

[ https://issues.apache.org/jira/browse/SPARK-24740?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16532884#comment-16532884 ] Hyukjin Kwon commented on SPARK-24740: -- I am working on this > PySpark tests do not pass with

[jira] [Created] (SPARK-24740) PySpark tests do not pass with NumPy 0.14.x+

Hyukjin Kwon created SPARK-24740: Summary: PySpark tests do not pass with NumPy 0.14.x+ Key: SPARK-24740 URL: https://issues.apache.org/jira/browse/SPARK-24740 Project: Spark Issue Type:

[jira] [Updated] (SPARK-24739) PySpark does not work with Python 3.7.0

[ https://issues.apache.org/jira/browse/SPARK-24739?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-24739: - Target Version/s: 2.3.1, 2.2.2, 2.1.3, 2.4.0 (was: 2.1.3, 2.2.2, 2.3.1) > PySpark does not

[jira] [Commented] (SPARK-24438) Empty strings and null strings are written to the same partition

[ https://issues.apache.org/jira/browse/SPARK-24438?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16532775#comment-16532775 ] Wenchen Fan commented on SPARK-24438: - AFAIK this is the same behavior from Hive. null and empty

[jira] [Created] (SPARK-24739) PySpark does not work with Python 3.7.0

Hyukjin Kwon created SPARK-24739: Summary: PySpark does not work with Python 3.7.0 Key: SPARK-24739 URL: https://issues.apache.org/jira/browse/SPARK-24739 Project: Spark Issue Type: Bug

[jira] [Assigned] (SPARK-24739) PySpark does not work with Python 3.7.0

[ https://issues.apache.org/jira/browse/SPARK-24739?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon reassigned SPARK-24739: Assignee: Hyukjin Kwon > PySpark does not work with Python 3.7.0 >

[jira] [Commented] (SPARK-24739) PySpark does not work with Python 3.7.0

[ https://issues.apache.org/jira/browse/SPARK-24739?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16532771#comment-16532771 ] Hyukjin Kwon commented on SPARK-24739: -- I am working on this. > PySpark does not work with Python

[jira] [Resolved] (SPARK-24635) Remove Blocks class

[ https://issues.apache.org/jira/browse/SPARK-24635?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-24635. - Resolution: Fixed Fix Version/s: 2.4.0 Issue resolved by pull request 21619

[jira] [Assigned] (SPARK-24635) Remove Blocks class

[ https://issues.apache.org/jira/browse/SPARK-24635?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan reassigned SPARK-24635: --- Assignee: Liang-Chi Hsieh > Remove Blocks class > --- > >

[jira] [Created] (SPARK-24738) [HistoryServer] FsHistoryProvider clean outdated event logs at start

ShuMing Li created SPARK-24738: -- Summary: [HistoryServer] FsHistoryProvider clean outdated event logs at start Key: SPARK-24738 URL: https://issues.apache.org/jira/browse/SPARK-24738 Project:

[jira] [Resolved] (SPARK-24716) Refactor ParquetFilters

[ https://issues.apache.org/jira/browse/SPARK-24716?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-24716. - Resolution: Fixed Fix Version/s: 2.4.0 Issue resolved by pull request 21696

[jira] [Assigned] (SPARK-24716) Refactor ParquetFilters

[ https://issues.apache.org/jira/browse/SPARK-24716?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan reassigned SPARK-24716: --- Assignee: Yuming Wang > Refactor ParquetFilters > --- > >

[jira] [Assigned] (SPARK-24727) The cache 100 in CodeGenerator is too small for streaming

[ https://issues.apache.org/jira/browse/SPARK-24727?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan reassigned SPARK-24727: --- Assignee: Takeshi Yamamuro > The cache 100 in CodeGenerator is too small for streaming >

[jira] [Resolved] (SPARK-24727) The cache 100 in CodeGenerator is too small for streaming

[ https://issues.apache.org/jira/browse/SPARK-24727?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-24727. - Resolution: Fixed Fix Version/s: 2.4.0 Issue resolved by pull request 21705

[jira] [Comment Edited] (SPARK-19498) Discussion: Making MLlib APIs extensible for 3rd party libraries

[ https://issues.apache.org/jira/browse/SPARK-19498?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16532576#comment-16532576 ] Peter Knight edited comment on SPARK-19498 at 7/4/18 11:36 AM: --- Just

[jira] [Comment Edited] (SPARK-19498) Discussion: Making MLlib APIs extensible for 3rd party libraries

[ https://issues.apache.org/jira/browse/SPARK-19498?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16532576#comment-16532576 ] Peter Knight edited comment on SPARK-19498 at 7/4/18 11:27 AM: --- Just

[jira] [Commented] (SPARK-24474) Cores are left idle when there are a lot of tasks to run

[ https://issues.apache.org/jira/browse/SPARK-24474?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16532599#comment-16532599 ] Yuming Wang commented on SPARK-24474: - Can you try to set {{spark.locality.wait=0}}? > Cores are

[jira] [Updated] (SPARK-24731) java.io.IOException: s3n://bucketname: 400 : Bad Request

[ https://issues.apache.org/jira/browse/SPARK-24731?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] sivakphani updated SPARK-24731: --- Description: I wrote code for connecting aws s3 bucket for read json file through pyspark. when i

[jira] [Commented] (SPARK-19498) Discussion: Making MLlib APIs extensible for 3rd party libraries

[ https://issues.apache.org/jira/browse/SPARK-19498?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16532576#comment-16532576 ] Peter Knight commented on SPARK-19498: -- Just wanted to add to Lucas' post that I agree with them. I

[jira] [Commented] (SPARK-13127) Upgrade Parquet to 1.9 (Fixes parquet sorting)

[ https://issues.apache.org/jira/browse/SPARK-13127?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16532534#comment-16532534 ] Al M commented on SPARK-13127: -- Would be great to get this resolved in Spark 2.3.2.  Especially since

[jira] [Assigned] (SPARK-24737) Type coercion between StructTypes.

[ https://issues.apache.org/jira/browse/SPARK-24737?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-24737: Assignee: Apache Spark > Type coercion between StructTypes. >

[jira] [Assigned] (SPARK-24737) Type coercion between StructTypes.

[ https://issues.apache.org/jira/browse/SPARK-24737?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-24737: Assignee: (was: Apache Spark) > Type coercion between StructTypes. >

[jira] [Commented] (SPARK-24737) Type coercion between StructTypes.

[ https://issues.apache.org/jira/browse/SPARK-24737?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16532432#comment-16532432 ] Apache Spark commented on SPARK-24737: -- User 'ueshin' has created a pull request for this issue:

[jira] [Created] (SPARK-24737) Type coercion between StructTypes.

Takuya Ueshin created SPARK-24737: - Summary: Type coercion between StructTypes. Key: SPARK-24737 URL: https://issues.apache.org/jira/browse/SPARK-24737 Project: Spark Issue Type: Improvement

[jira] [Resolved] (SPARK-24704) The order of stages in the DAG graph is incorrect

[ https://issues.apache.org/jira/browse/SPARK-24704?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Herman van Hovell resolved SPARK-24704. --- Resolution: Fixed Assignee: StanZhai Fix Version/s: 2.4.0 > The

[jira] [Updated] (SPARK-24570) SparkSQL - show schemas/tables in dropdowns of SQL client tools (ie Squirrel SQL, DBVisualizer.etc)

[ https://issues.apache.org/jira/browse/SPARK-24570?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] t oo updated SPARK-24570: - Attachment: hive.png spark.png > SparkSQL - show schemas/tables in dropdowns of SQL client

[jira] [Commented] (SPARK-24570) SparkSQL - show schemas/tables in dropdowns of SQL client tools (ie Squirrel SQL, DBVisualizer.etc)

[ https://issues.apache.org/jira/browse/SPARK-24570?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16532403#comment-16532403 ] t oo commented on SPARK-24570: -- hive.png shows tables, spark.png does not. Using simba driver > SparkSQL -

[jira] [Commented] (SPARK-22384) Refine partition pruning when attribute is wrapped in Cast

[ https://issues.apache.org/jira/browse/SPARK-22384?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16532400#comment-16532400 ] Apache Spark commented on SPARK-22384: -- User 'cloud-fan' has created a pull request for this issue:

[jira] [Commented] (SPARK-24438) Empty strings and null strings are written to the same partition

[ https://issues.apache.org/jira/browse/SPARK-24438?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16532392#comment-16532392 ] Liang-Chi Hsieh commented on SPARK-24438: - >From the code, looks like we intentionally treat

[jira] [Commented] (SPARK-24467) VectorAssemblerEstimator

[ https://issues.apache.org/jira/browse/SPARK-24467?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16532324#comment-16532324 ] Liang-Chi Hsieh commented on SPARK-24467: - It sounds good to me for the approach similar to one