[jira] [Updated] (SPARK-24743) Update the DirectKafkaWordCount example to support the new API of Kafka

[ https://issues.apache.org/jira/browse/SPARK-24743?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] luochuan updated SPARK-24743: - Description: when I ran the example JavaDirectKafkaWordCount as follows: ./bin/spark-submit --class or

[jira] [Created] (SPARK-24743) Update the DirectKafkaWordCount example to support the new API of Kafka

luochuan created SPARK-24743: Summary: Update the DirectKafkaWordCount example to support the new API of Kafka Key: SPARK-24743 URL: https://issues.apache.org/jira/browse/SPARK-24743 Project: Spark

[jira] [Commented] (SPARK-24742) Field Metadata raises NullPointerException in hashCode method

[ https://issues.apache.org/jira/browse/SPARK-24742?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16533286#comment-16533286 ] Kaya Kupferschmidt commented on SPARK-24742: I will provide a patch later.

[jira] [Created] (SPARK-24742) Field Metadata raises NullPointerException in hashCode method

Kaya Kupferschmidt created SPARK-24742: -- Summary: Field Metadata raises NullPointerException in hashCode method Key: SPARK-24742 URL: https://issues.apache.org/jira/browse/SPARK-24742 Project: Sp

[jira] [Commented] (SPARK-24705) Spark.sql.adaptive.enabled=true is enabled and self-join query

[ https://issues.apache.org/jira/browse/SPARK-24705?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16533212#comment-16533212 ] Takeshi Yamamuro commented on SPARK-24705: -- ok, will do. > Spark.sql.adaptive.

[jira] [Commented] (SPARK-24723) Discuss necessary info and access in barrier mode + YARN

[ https://issues.apache.org/jira/browse/SPARK-24723?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16533205#comment-16533205 ] Saisai Shao commented on SPARK-24723: - Hi [~mengxr], I would like to know the goal o

[jira] [Commented] (SPARK-24705) Spark.sql.adaptive.enabled=true is enabled and self-join query

[ https://issues.apache.org/jira/browse/SPARK-24705?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16533195#comment-16533195 ] Xiao Li commented on SPARK-24705: - [~maropu]Thank you for taking this. Please try to fix

[jira] [Commented] (SPARK-24739) PySpark does not work with Python 3.7.0

[ https://issues.apache.org/jira/browse/SPARK-24739?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16533191#comment-16533191 ] Hyukjin Kwon commented on SPARK-24739: -- For this fix particularly, the fix is small

[jira] [Commented] (SPARK-24739) PySpark does not work with Python 3.7.0

[ https://issues.apache.org/jira/browse/SPARK-24739?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16533189#comment-16533189 ] Saisai Shao commented on SPARK-24739: - Do we have to fix it in 2.3.2? I don't think

[jira] [Resolved] (SPARK-24698) In Pyspark's ML, an Identifiable's UID has 20 random characters rather than the 12 mentioned in the documentation.

[ https://issues.apache.org/jira/browse/SPARK-24698?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-24698. -- Resolution: Fixed Fix Version/s: 2.4.0 Issue resolved by pull request 21675 [https://gi

[jira] [Assigned] (SPARK-24698) In Pyspark's ML, an Identifiable's UID has 20 random characters rather than the 12 mentioned in the documentation.

[ https://issues.apache.org/jira/browse/SPARK-24698?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon reassigned SPARK-24698: Assignee: Thomas Dunne > In Pyspark's ML, an Identifiable's UID has 20 random characters

[jira] [Resolved] (SPARK-24549) Support DecimalType push down to the parquet data sources

[ https://issues.apache.org/jira/browse/SPARK-24549?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-24549. -- Resolution: Duplicate > Support DecimalType push down to the parquet data sources > --

[jira] [Updated] (SPARK-24741) Have a built-in AVRO data source implementation

[ https://issues.apache.org/jira/browse/SPARK-24741?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Li updated SPARK-24741: Description: Apache Avro (https://avro.apache.org) is a data serialization format. It is widely used in t

[jira] [Updated] (SPARK-24741) Have a built-in AVRO data source implementation

[ https://issues.apache.org/jira/browse/SPARK-24741?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Li updated SPARK-24741: Description: Apache Avro (https://avro.apache.org) is a data serialization format. It is widely used in t

[jira] [Updated] (SPARK-24741) Have a built-in AVRO data source implementation

[ https://issues.apache.org/jira/browse/SPARK-24741?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Li updated SPARK-24741: Component/s: Structured Streaming > Have a built-in AVRO data source implementation >

[jira] [Created] (SPARK-24741) Have a built-in AVRO data source implementation

Xiao Li created SPARK-24741: --- Summary: Have a built-in AVRO data source implementation Key: SPARK-24741 URL: https://issues.apache.org/jira/browse/SPARK-24741 Project: Spark Issue Type: New Feature

[jira] [Commented] (SPARK-24741) Have a built-in AVRO data source implementation

[ https://issues.apache.org/jira/browse/SPARK-24741?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16533137#comment-16533137 ] Xiao Li commented on SPARK-24741: - cc [~Gengliang.Wang] > Have a built-in AVRO data sou

[jira] [Commented] (SPARK-17213) Parquet String Pushdown for Non-Eq Comparisons Broken

[ https://issues.apache.org/jira/browse/SPARK-17213?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16533083#comment-16533083 ] Apache Spark commented on SPARK-17213: -- User 'gatorsmile' has created a pull reques

[jira] [Commented] (SPARK-23682) Memory issue with Spark structured streaming

[ https://issues.apache.org/jira/browse/SPARK-23682?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16533078#comment-16533078 ] Jungtaek Lim commented on SPARK-23682: -- [~bondyk] [~ccifuentes] [~akorzhuev] This

[jira] [Updated] (SPARK-24739) PySpark does not work with Python 3.7.0

[ https://issues.apache.org/jira/browse/SPARK-24739?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen updated SPARK-24739: -- Target Version/s: 2.3.2, 2.4.0 (was: 2.1.3, 2.2.2, 2.3.1, 2.4.0) > PySpark does not work with Python

[jira] [Updated] (SPARK-24672) No job is running but there are active tasks

[ https://issues.apache.org/jira/browse/SPARK-24672?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen updated SPARK-24672: -- Target Version/s: (was: 2.2.1) > No job is running but there are active tasks >

[jira] [Updated] (SPARK-24710) Information Gain Ratio for decision trees

[ https://issues.apache.org/jira/browse/SPARK-24710?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen updated SPARK-24710: -- Target Version/s: (was: 2.3.1) Fix Version/s: (was: 2.3.1) > Information Gain Ratio for d

[jira] [Updated] (SPARK-24530) Sphinx doesn't render autodoc_docstring_signature correctly (with Python 2?) and pyspark.ml docs are broken

[ https://issues.apache.org/jira/browse/SPARK-24530?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen updated SPARK-24530: -- Target Version/s: 2.3.2, 2.4.0 (was: 2.1.3, 2.2.2, 2.3.2, 2.4.0) > Sphinx doesn't render autodoc_docs

[jira] [Assigned] (SPARK-24740) PySpark tests do not pass with NumPy 0.14.x+

[ https://issues.apache.org/jira/browse/SPARK-24740?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-24740: Assignee: Hyukjin Kwon (was: Apache Spark) > PySpark tests do not pass with NumPy 0.14.x

[jira] [Commented] (SPARK-24740) PySpark tests do not pass with NumPy 0.14.x+

[ https://issues.apache.org/jira/browse/SPARK-24740?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16532928#comment-16532928 ] Apache Spark commented on SPARK-24740: -- User 'HyukjinKwon' has created a pull reque

[jira] [Assigned] (SPARK-24740) PySpark tests do not pass with NumPy 0.14.x+

[ https://issues.apache.org/jira/browse/SPARK-24740?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-24740: Assignee: Apache Spark (was: Hyukjin Kwon) > PySpark tests do not pass with NumPy 0.14.x

[jira] [Comment Edited] (SPARK-24474) Cores are left idle when there are a lot of tasks to run

[ https://issues.apache.org/jira/browse/SPARK-24474?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16532910#comment-16532910 ] Al M edited comment on SPARK-24474 at 7/4/18 4:17 PM: -- My initial t

[jira] [Commented] (SPARK-24739) PySpark does not work with Python 3.7.0

[ https://issues.apache.org/jira/browse/SPARK-24739?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16532912#comment-16532912 ] Apache Spark commented on SPARK-24739: -- User 'HyukjinKwon' has created a pull reque

[jira] [Assigned] (SPARK-24739) PySpark does not work with Python 3.7.0

[ https://issues.apache.org/jira/browse/SPARK-24739?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-24739: Assignee: Hyukjin Kwon (was: Apache Spark) > PySpark does not work with Python 3.7.0 > -

[jira] [Assigned] (SPARK-24739) PySpark does not work with Python 3.7.0

[ https://issues.apache.org/jira/browse/SPARK-24739?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-24739: Assignee: Apache Spark (was: Hyukjin Kwon) > PySpark does not work with Python 3.7.0 > -

[jira] [Commented] (SPARK-24474) Cores are left idle when there are a lot of tasks to run

[ https://issues.apache.org/jira/browse/SPARK-24474?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16532910#comment-16532910 ] Al M commented on SPARK-24474: -- My initial tests suggest that this stops the issue from hap

[jira] [Commented] (SPARK-24740) PySpark tests do not pass with NumPy 0.14.x+

[ https://issues.apache.org/jira/browse/SPARK-24740?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16532884#comment-16532884 ] Hyukjin Kwon commented on SPARK-24740: -- I am working on this > PySpark tests do no

[jira] [Created] (SPARK-24740) PySpark tests do not pass with NumPy 0.14.x+

Hyukjin Kwon created SPARK-24740: Summary: PySpark tests do not pass with NumPy 0.14.x+ Key: SPARK-24740 URL: https://issues.apache.org/jira/browse/SPARK-24740 Project: Spark Issue Type: Test

[jira] [Updated] (SPARK-24739) PySpark does not work with Python 3.7.0

[ https://issues.apache.org/jira/browse/SPARK-24739?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-24739: - Target Version/s: 2.3.1, 2.2.2, 2.1.3, 2.4.0 (was: 2.1.3, 2.2.2, 2.3.1) > PySpark does not work

[jira] [Commented] (SPARK-24438) Empty strings and null strings are written to the same partition

[ https://issues.apache.org/jira/browse/SPARK-24438?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16532775#comment-16532775 ] Wenchen Fan commented on SPARK-24438: - AFAIK this is the same behavior from Hive. nu

[jira] [Assigned] (SPARK-24739) PySpark does not work with Python 3.7.0

[ https://issues.apache.org/jira/browse/SPARK-24739?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon reassigned SPARK-24739: Assignee: Hyukjin Kwon > PySpark does not work with Python 3.7.0 > --

[jira] [Created] (SPARK-24739) PySpark does not work with Python 3.7.0

Hyukjin Kwon created SPARK-24739: Summary: PySpark does not work with Python 3.7.0 Key: SPARK-24739 URL: https://issues.apache.org/jira/browse/SPARK-24739 Project: Spark Issue Type: Bug

[jira] [Commented] (SPARK-24739) PySpark does not work with Python 3.7.0

[ https://issues.apache.org/jira/browse/SPARK-24739?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16532771#comment-16532771 ] Hyukjin Kwon commented on SPARK-24739: -- I am working on this. > PySpark does not w

[jira] [Resolved] (SPARK-24635) Remove Blocks class

[ https://issues.apache.org/jira/browse/SPARK-24635?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-24635. - Resolution: Fixed Fix Version/s: 2.4.0 Issue resolved by pull request 21619 [https://gith

[jira] [Assigned] (SPARK-24635) Remove Blocks class

[ https://issues.apache.org/jira/browse/SPARK-24635?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan reassigned SPARK-24635: --- Assignee: Liang-Chi Hsieh > Remove Blocks class > --- > > K

[jira] [Created] (SPARK-24738) [HistoryServer] FsHistoryProvider clean outdated event logs at start

ShuMing Li created SPARK-24738: -- Summary: [HistoryServer] FsHistoryProvider clean outdated event logs at start Key: SPARK-24738 URL: https://issues.apache.org/jira/browse/SPARK-24738 Project: Spark

[jira] [Resolved] (SPARK-24716) Refactor ParquetFilters

[ https://issues.apache.org/jira/browse/SPARK-24716?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-24716. - Resolution: Fixed Fix Version/s: 2.4.0 Issue resolved by pull request 21696 [https://gith

[jira] [Assigned] (SPARK-24716) Refactor ParquetFilters

[ https://issues.apache.org/jira/browse/SPARK-24716?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan reassigned SPARK-24716: --- Assignee: Yuming Wang > Refactor ParquetFilters > --- > >

[jira] [Assigned] (SPARK-24727) The cache 100 in CodeGenerator is too small for streaming

[ https://issues.apache.org/jira/browse/SPARK-24727?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan reassigned SPARK-24727: --- Assignee: Takeshi Yamamuro > The cache 100 in CodeGenerator is too small for streaming > --

[jira] [Resolved] (SPARK-24727) The cache 100 in CodeGenerator is too small for streaming

[ https://issues.apache.org/jira/browse/SPARK-24727?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-24727. - Resolution: Fixed Fix Version/s: 2.4.0 Issue resolved by pull request 21705 [https://gith

[jira] [Comment Edited] (SPARK-19498) Discussion: Making MLlib APIs extensible for 3rd party libraries

[ https://issues.apache.org/jira/browse/SPARK-19498?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16532576#comment-16532576 ] Peter Knight edited comment on SPARK-19498 at 7/4/18 11:36 AM: ---

[jira] [Comment Edited] (SPARK-19498) Discussion: Making MLlib APIs extensible for 3rd party libraries

[ https://issues.apache.org/jira/browse/SPARK-19498?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16532576#comment-16532576 ] Peter Knight edited comment on SPARK-19498 at 7/4/18 11:27 AM: ---

[jira] [Commented] (SPARK-24474) Cores are left idle when there are a lot of tasks to run

[ https://issues.apache.org/jira/browse/SPARK-24474?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16532599#comment-16532599 ] Yuming Wang commented on SPARK-24474: - Can you try to set {{spark.locality.wait=0}}?

[jira] [Updated] (SPARK-24731) java.io.IOException: s3n://bucketname: 400 : Bad Request

[ https://issues.apache.org/jira/browse/SPARK-24731?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] sivakphani updated SPARK-24731: --- Description: I wrote code for connecting aws s3 bucket for read json file through pyspark. when i s

[jira] [Commented] (SPARK-19498) Discussion: Making MLlib APIs extensible for 3rd party libraries

[ https://issues.apache.org/jira/browse/SPARK-19498?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16532576#comment-16532576 ] Peter Knight commented on SPARK-19498: -- Just wanted to add to Lucas' post that I ag

[jira] [Commented] (SPARK-13127) Upgrade Parquet to 1.9 (Fixes parquet sorting)

[ https://issues.apache.org/jira/browse/SPARK-13127?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16532534#comment-16532534 ] Al M commented on SPARK-13127: -- Would be great to get this resolved in Spark 2.3.2.  Especi

[jira] [Assigned] (SPARK-24737) Type coercion between StructTypes.

[ https://issues.apache.org/jira/browse/SPARK-24737?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-24737: Assignee: Apache Spark > Type coercion between StructTypes. > ---

[jira] [Assigned] (SPARK-24737) Type coercion between StructTypes.

[ https://issues.apache.org/jira/browse/SPARK-24737?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-24737: Assignee: (was: Apache Spark) > Type coercion between StructTypes. >

[jira] [Commented] (SPARK-24737) Type coercion between StructTypes.

[ https://issues.apache.org/jira/browse/SPARK-24737?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16532432#comment-16532432 ] Apache Spark commented on SPARK-24737: -- User 'ueshin' has created a pull request fo

[jira] [Created] (SPARK-24737) Type coercion between StructTypes.

Takuya Ueshin created SPARK-24737: - Summary: Type coercion between StructTypes. Key: SPARK-24737 URL: https://issues.apache.org/jira/browse/SPARK-24737 Project: Spark Issue Type: Improvement

[jira] [Resolved] (SPARK-24704) The order of stages in the DAG graph is incorrect

[ https://issues.apache.org/jira/browse/SPARK-24704?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Herman van Hovell resolved SPARK-24704. --- Resolution: Fixed Assignee: StanZhai Fix Version/s: 2.4.0 > The orde

[jira] [Updated] (SPARK-24570) SparkSQL - show schemas/tables in dropdowns of SQL client tools (ie Squirrel SQL, DBVisualizer.etc)

[ https://issues.apache.org/jira/browse/SPARK-24570?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] t oo updated SPARK-24570: - Attachment: hive.png spark.png > SparkSQL - show schemas/tables in dropdowns of SQL client tools

[jira] [Commented] (SPARK-24570) SparkSQL - show schemas/tables in dropdowns of SQL client tools (ie Squirrel SQL, DBVisualizer.etc)

[ https://issues.apache.org/jira/browse/SPARK-24570?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16532403#comment-16532403 ] t oo commented on SPARK-24570: -- hive.png shows tables, spark.png does not. Using simba driv

[jira] [Commented] (SPARK-22384) Refine partition pruning when attribute is wrapped in Cast

[ https://issues.apache.org/jira/browse/SPARK-22384?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16532400#comment-16532400 ] Apache Spark commented on SPARK-22384: -- User 'cloud-fan' has created a pull request

[jira] [Commented] (SPARK-24438) Empty strings and null strings are written to the same partition

[ https://issues.apache.org/jira/browse/SPARK-24438?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16532392#comment-16532392 ] Liang-Chi Hsieh commented on SPARK-24438: - >From the code, looks like we intenti