[jira] [Commented] (SPARK-25940) Add the ability to tag datasets so that tags appear with RDDs in StageCompleted events when the corresponding sources are read/written

[ https://issues.apache.org/jira/browse/SPARK-25940?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16674705#comment-16674705 ] Aby Philip commented on SPARK-25940: Sorry. Removed. > Add the ability to tag datasets so that tags

[jira] [Updated] (SPARK-25940) Add the ability to tag datasets so that tags appear with RDDs in StageCompleted events when the corresponding sources are read/written

[ https://issues.apache.org/jira/browse/SPARK-25940?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aby Philip updated SPARK-25940: --- Target Version/s: (was: 2.3.0, 2.3.1) > Add the ability to tag datasets so that tags appear with

[jira] [Updated] (SPARK-25940) Add the ability to tag datasets so that tags appear with RDDs in StageCompleted events when the corresponding sources are read/written

[ https://issues.apache.org/jira/browse/SPARK-25940?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aby Philip updated SPARK-25940: --- Fix Version/s: (was: 2.3.1) (was: 2.3.0) > Add the ability to tag

[jira] [Commented] (SPARK-25940) Add the ability to tag datasets so that tags appear with RDDs in StageCompleted events when the corresponding sources are read/written

[ https://issues.apache.org/jira/browse/SPARK-25940?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16674704#comment-16674704 ] Yuming Wang commented on SPARK-25940: - [~aphilp] Please avoid set the {{Fix Version/s}} and {{Target

[jira] [Created] (SPARK-25940) Add the ability to tag datasets so that tags appear with RDDs in StageCompleted events when the corresponding sources are read/written

Aby Philip created SPARK-25940: -- Summary: Add the ability to tag datasets so that tags appear with RDDs in StageCompleted events when the corresponding sources are read/written Key: SPARK-25940 URL:

[jira] [Commented] (SPARK-25098) ‘Cast’ will return NULL when input string starts/ends with special character(s)

[ https://issues.apache.org/jira/browse/SPARK-25098?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16674690#comment-16674690 ] Apache Spark commented on SPARK-25098: -- User 'wangyum' has created a pull request for this issue:

[jira] [Commented] (SPARK-25098) ‘Cast’ will return NULL when input string starts/ends with special character(s)

[ https://issues.apache.org/jira/browse/SPARK-25098?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16674691#comment-16674691 ] Apache Spark commented on SPARK-25098: -- User 'wangyum' has created a pull request for this issue:

[jira] [Commented] (SPARK-25588) SchemaParseException: Can't redefine: list when reading from Parquet

[ https://issues.apache.org/jira/browse/SPARK-25588?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16674689#comment-16674689 ] Wenchen Fan commented on SPARK-25588: - This looks like a parquet bug when it interacts with avro. I

[jira] [Commented] (SPARK-25937) Support user-defined schema in Kafka Source & Sink

[ https://issues.apache.org/jira/browse/SPARK-25937?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16674665#comment-16674665 ] Jungtaek Lim commented on SPARK-25937: -- Another thought for my side: maybe we can classify various

[jira] [Commented] (SPARK-25884) Add TBLPROPERTIES and COMMENT, and use LOCATION when SHOW CREATE TABLE.

[ https://issues.apache.org/jira/browse/SPARK-25884?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16674661#comment-16674661 ] Apache Spark commented on SPARK-25884: -- User 'ueshin' has created a pull request for this issue:

[jira] [Commented] (SPARK-25932) why GraphX vertexId should be Long ?

[ https://issues.apache.org/jira/browse/SPARK-25932?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16674659#comment-16674659 ] Ali Zadedehbalaei commented on SPARK-25932: --- Thanks [~dongjoon] . > why GraphX vertexId

[jira] [Commented] (SPARK-25937) Support user-defined schema in Kafka Source & Sink

[ https://issues.apache.org/jira/browse/SPARK-25937?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16674630#comment-16674630 ] Yuanjian Li commented on SPARK-25937: - The problem described here need to be resolved, but

[jira] [Created] (SPARK-25939) Spark 'jshell' support on JDK 11

Hyukjin Kwon created SPARK-25939: Summary: Spark 'jshell' support on JDK 11 Key: SPARK-25939 URL: https://issues.apache.org/jira/browse/SPARK-25939 Project: Spark Issue Type: Sub-task

[jira] [Commented] (SPARK-24417) Build and Run Spark on JDK11

[ https://issues.apache.org/jira/browse/SPARK-24417?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16674601#comment-16674601 ] Hyukjin Kwon commented on SPARK-24417: -- Thanks, done. > Build and Run Spark on JDK11 >

[jira] [Commented] (SPARK-25928) NoSuchMethodError net.jpountz.lz4.LZ4BlockInputStream.(Ljava/io/InputStream;Z)V

[ https://issues.apache.org/jira/browse/SPARK-25928?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16674596#comment-16674596 ] Jerry Chabot commented on SPARK-25928: -- We got the job to succeed by changing to snappy

[jira] [Resolved] (SPARK-25913) Unary SparkPlan nodes should extend UnaryExecNode

[ https://issues.apache.org/jira/browse/SPARK-25913?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Li resolved SPARK-25913. - Resolution: Fixed Assignee: Maxim Gekk Fix Version/s: 3.0.0 > Unary SparkPlan nodes

[jira] [Resolved] (SPARK-23194) from_json in FAILFAST mode doesn't fail fast, instead it just returns nulls

[ https://issues.apache.org/jira/browse/SPARK-23194?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-23194. -- Resolution: Duplicate > from_json in FAILFAST mode doesn't fail fast, instead it just returns

[jira] [Updated] (SPARK-25778) WriteAheadLogBackedBlockRDD in YARN Cluster Mode Fails due lack of access to tmpDir from $PWD to HDFS

[ https://issues.apache.org/jira/browse/SPARK-25778?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Greg Senia updated SPARK-25778: --- Attachment: SPARK-25778.2.patch > WriteAheadLogBackedBlockRDD in YARN Cluster Mode Fails due lack

[jira] [Commented] (SPARK-25928) NoSuchMethodError net.jpountz.lz4.LZ4BlockInputStream.(Ljava/io/InputStream;Z)V

[ https://issues.apache.org/jira/browse/SPARK-25928?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16674559#comment-16674559 ] Jerry Chabot commented on SPARK-25928: -- Spark is failing because it can't find

[jira] [Resolved] (SPARK-25932) why GraphX vertexId should be Long ?

[ https://issues.apache.org/jira/browse/SPARK-25932?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-25932. --- Resolution: Invalid Sorry, [~alizadehei]. We don't use JIRA as a Q Please see `Have

[jira] [Closed] (SPARK-25932) why GraphX vertexId should be Long ?

[ https://issues.apache.org/jira/browse/SPARK-25932?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun closed SPARK-25932. - > why GraphX vertexId should be Long ? > > >

[jira] [Updated] (SPARK-25938) Action on cached dataset causes WARN java.lang.AssertionError in log4j logs.

[ https://issues.apache.org/jira/browse/SPARK-25938?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Benedikt Beckermann updated SPARK-25938: Description: Caching and counting any DataSet causes an AssertionError in the

[jira] [Updated] (SPARK-25938) Action on cached dataset causes WARN java.lang.AssertionError in log4j logs.

[ https://issues.apache.org/jira/browse/SPARK-25938?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Benedikt Beckermann updated SPARK-25938: Description: Caching and counting any DataSet causes an AssertionError in the

[jira] [Created] (SPARK-25938) Action on cached dataset causes WARN java.lang.AssertionError in log4j logs.

Benedikt Beckermann created SPARK-25938: --- Summary: Action on cached dataset causes WARN java.lang.AssertionError in log4j logs. Key: SPARK-25938 URL: https://issues.apache.org/jira/browse/SPARK-25938

[jira] [Commented] (SPARK-23194) from_json in FAILFAST mode doesn't fail fast, instead it just returns nulls

[ https://issues.apache.org/jira/browse/SPARK-23194?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16674468#comment-16674468 ] Maxim Gekk commented on SPARK-23194: This changes

[jira] [Commented] (SPARK-24417) Build and Run Spark on JDK11

[ https://issues.apache.org/jira/browse/SPARK-24417?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16674459#comment-16674459 ] Dongjoon Hyun commented on SPARK-24417: --- It sounds like a great idea, [~hyukjin.kwon]. Could you

[jira] [Commented] (SPARK-25776) The disk write buffer size must be greater than 12.

[ https://issues.apache.org/jira/browse/SPARK-25776?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16674442#comment-16674442 ] Kazuaki Ishizaki commented on SPARK-25776: -- Issue resolved by pull request 22754

[jira] [Resolved] (SPARK-25312) Add description for the conf spark.network.crypto.keyFactoryIterations

[ https://issues.apache.org/jira/browse/SPARK-25312?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen resolved SPARK-25312. --- Resolution: Duplicate > Add description for the conf spark.network.crypto.keyFactoryIterations >

[jira] [Updated] (SPARK-25776) The disk write buffer size must be greater than 12.

[ https://issues.apache.org/jira/browse/SPARK-25776?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kazuaki Ishizaki updated SPARK-25776: - Fix Version/s: 3.0.0 > The disk write buffer size must be greater than 12. >

[jira] [Resolved] (SPARK-25852) we should filter the workOffers with freeCores>=CPUS_PER_TASK at first for better performance

[ https://issues.apache.org/jira/browse/SPARK-25852?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen resolved SPARK-25852. --- Resolution: Not A Problem > we should filter the workOffers with freeCores>=CPUS_PER_TASK at first

[jira] [Resolved] (SPARK-25776) The disk write buffer size must be greater than 12.

[ https://issues.apache.org/jira/browse/SPARK-25776?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kazuaki Ishizaki resolved SPARK-25776. -- Resolution: Fixed Assignee: liuxian > The disk write buffer size must be

[jira] [Updated] (SPARK-25937) Support user-defined schema in Kafka Source & Sink

[ https://issues.apache.org/jira/browse/SPARK-25937?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jackey Lee updated SPARK-25937: --- Description:     Kafka Source & Sink is widely used in Spark and has the highest frequency in

[jira] [Created] (SPARK-25937) Support user-defined schema in Kafka Source & Sink

Jackey Lee created SPARK-25937: -- Summary: Support user-defined schema in Kafka Source & Sink Key: SPARK-25937 URL: https://issues.apache.org/jira/browse/SPARK-25937 Project: Spark Issue Type:

[jira] [Commented] (SPARK-25936) InsertIntoDataSourceCommand does not use Cached Data

[ https://issues.apache.org/jira/browse/SPARK-25936?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16674378#comment-16674378 ] Apache Spark commented on SPARK-25936: -- User 'wangyum' has created a pull request for this issue:

[jira] [Commented] (SPARK-25936) InsertIntoDataSourceCommand does not use Cached Data

[ https://issues.apache.org/jira/browse/SPARK-25936?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16674377#comment-16674377 ] Apache Spark commented on SPARK-25936: -- User 'wangyum' has created a pull request for this issue:

[jira] [Assigned] (SPARK-25936) InsertIntoDataSourceCommand does not use Cached Data

[ https://issues.apache.org/jira/browse/SPARK-25936?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-25936: Assignee: Apache Spark > InsertIntoDataSourceCommand does not use Cached Data >

[jira] [Assigned] (SPARK-25936) InsertIntoDataSourceCommand does not use Cached Data

[ https://issues.apache.org/jira/browse/SPARK-25936?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-25936: Assignee: (was: Apache Spark) > InsertIntoDataSourceCommand does not use Cached Data

[jira] [Commented] (SPARK-24417) Build and Run Spark on JDK11

[ https://issues.apache.org/jira/browse/SPARK-24417?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16674375#comment-16674375 ] Hyukjin Kwon commented on SPARK-24417: -- BTW, do we plan to add

[jira] [Assigned] (SPARK-25446) Add schema_of_json() and schema_of_csv() to R

[ https://issues.apache.org/jira/browse/SPARK-25446?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-25446: Assignee: Hyukjin Kwon (was: Apache Spark) > Add schema_of_json() and schema_of_csv()

[jira] [Assigned] (SPARK-25446) Add schema_of_json() and schema_of_csv() to R

[ https://issues.apache.org/jira/browse/SPARK-25446?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-25446: Assignee: Apache Spark (was: Hyukjin Kwon) > Add schema_of_json() and schema_of_csv()

[jira] [Commented] (SPARK-25446) Add schema_of_json() and schema_of_csv() to R

[ https://issues.apache.org/jira/browse/SPARK-25446?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16674371#comment-16674371 ] Apache Spark commented on SPARK-25446: -- User 'HyukjinKwon' has created a pull request for this

[jira] [Updated] (SPARK-25446) Add schema_of_json() and schema_of_csv to R

[ https://issues.apache.org/jira/browse/SPARK-25446?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-25446: - Priority: Major (was: Minor) > Add schema_of_json() and schema_of_csv to R >

[jira] [Updated] (SPARK-25446) Add schema_of_json() and schema_of_csv() to R

[ https://issues.apache.org/jira/browse/SPARK-25446?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-25446: - Summary: Add schema_of_json() and schema_of_csv() to R (was: Add schema_of_json() and

[jira] [Updated] (SPARK-25446) Add schema_of_json() and schema_of_csv to R

[ https://issues.apache.org/jira/browse/SPARK-25446?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-25446: - Summary: Add schema_of_json() and schema_of_csv to R (was: Add schema_of_json() to R) > Add

[jira] [Assigned] (SPARK-25446) Add schema_of_json() and schema_of_csv to R

[ https://issues.apache.org/jira/browse/SPARK-25446?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon reassigned SPARK-25446: Assignee: Hyukjin Kwon > Add schema_of_json() and schema_of_csv to R >

[jira] [Updated] (SPARK-25446) Add schema_of_json() and schema_of_csv to R

[ https://issues.apache.org/jira/browse/SPARK-25446?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-25446: - Description: The function schem_of_json() and schem_of_csv() are exposed in Scala/Java and

[jira] [Updated] (SPARK-25936) InsertIntoDataSourceCommand does not use Cached Data

[ https://issues.apache.org/jira/browse/SPARK-25936?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuming Wang updated SPARK-25936: Description: How to reproduce this issue: {code:scala} spark.sql(""" CREATE TABLE jdbcTable

[jira] [Created] (SPARK-25936) InsertIntoDataSourceCommand does not use Cached Data

Yuming Wang created SPARK-25936: --- Summary: InsertIntoDataSourceCommand does not use Cached Data Key: SPARK-25936 URL: https://issues.apache.org/jira/browse/SPARK-25936 Project: Spark Issue

[jira] [Commented] (SPARK-25935) Prevent null rows from JSON parser

[ https://issues.apache.org/jira/browse/SPARK-25935?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16674346#comment-16674346 ] Apache Spark commented on SPARK-25935: -- User 'MaxGekk' has created a pull request for this issue:

[jira] [Assigned] (SPARK-25935) Prevent null rows from JSON parser

[ https://issues.apache.org/jira/browse/SPARK-25935?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-25935: Assignee: Apache Spark > Prevent null rows from JSON parser >

[jira] [Commented] (SPARK-25935) Prevent null rows from JSON parser

[ https://issues.apache.org/jira/browse/SPARK-25935?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16674344#comment-16674344 ] Apache Spark commented on SPARK-25935: -- User 'MaxGekk' has created a pull request for this issue:

[jira] [Assigned] (SPARK-25935) Prevent null rows from JSON parser

[ https://issues.apache.org/jira/browse/SPARK-25935?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-25935: Assignee: (was: Apache Spark) > Prevent null rows from JSON parser >

[jira] [Updated] (SPARK-25935) Prevent null rows from JSON parser

[ https://issues.apache.org/jira/browse/SPARK-25935?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Maxim Gekk updated SPARK-25935: --- Summary: Prevent null rows from JSON parser (was: Prevent nulls from JSON parser) > Prevent null

[jira] [Created] (SPARK-25935) Prevent nulls from JSON parser

Maxim Gekk created SPARK-25935: -- Summary: Prevent nulls from JSON parser Key: SPARK-25935 URL: https://issues.apache.org/jira/browse/SPARK-25935 Project: Spark Issue Type: Improvement

[jira] [Updated] (SPARK-25729) It is better to replace `minPartitions` with `defaultParallelism` , when `minPartitions` is less than `defaultParallelism`

[ https://issues.apache.org/jira/browse/SPARK-25729?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] liuxian updated SPARK-25729: Description: For ‘WholeTextFileInputFormat’， when `minPartitions` is less than `defaultParallelism`， it

[jira] [Resolved] (SPARK-25491) pandas_udf(GROUPED_MAP) fails when using ArrayType(ArrayType(DoubleType()))

[ https://issues.apache.org/jira/browse/SPARK-25491?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-25491. -- Resolution: Won't Fix > pandas_udf(GROUPED_MAP) fails when using

[jira] [Commented] (SPARK-25491) pandas_udf(GROUPED_MAP) fails when using ArrayType(ArrayType(DoubleType()))

[ https://issues.apache.org/jira/browse/SPARK-25491?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16674330#comment-16674330 ] Hyukjin Kwon commented on SPARK-25491: -- Let me leave this resolved then. We will likely bump up the

[jira] [Commented] (SPARK-25491) pandas_udf(GROUPED_MAP) fails when using ArrayType(ArrayType(DoubleType()))

[ https://issues.apache.org/jira/browse/SPARK-25491?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16674328#comment-16674328 ] bnaya commented on SPARK-25491: --- it seems that this issue is solved in pyarrow 0.11.1 {code:java}  

[jira] [Commented] (SPARK-17967) Support for list or other types as an option for datasources

[ https://issues.apache.org/jira/browse/SPARK-17967?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16674327#comment-16674327 ] Hyukjin Kwon commented on SPARK-17967: -- That works but IMHO less pretty actually. We can add

[jira] [Assigned] (SPARK-25638) Convert structs to CSV strings

[ https://issues.apache.org/jira/browse/SPARK-25638?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon reassigned SPARK-25638: Assignee: Maxim Gekk > Convert structs to CSV strings > -- >

[jira] [Resolved] (SPARK-25638) Convert structs to CSV strings

[ https://issues.apache.org/jira/browse/SPARK-25638?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-25638. -- Resolution: Fixed Fix Version/s: 3.0.0 Issue resolved by pull request 22626