[jira] [Comment Edited] (SPARK-24130) Data Source V2: Join Push Down

[ https://issues.apache.org/jira/browse/SPARK-24130?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16473765#comment-16473765 ] Jia Li edited comment on SPARK-24130 at 5/14/18 4:48 AM: - Thank you for letting

[jira] [Commented] (SPARK-24130) Data Source V2: Join Push Down

[ https://issues.apache.org/jira/browse/SPARK-24130?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16473765#comment-16473765 ] Jia Li commented on SPARK-24130: Thank you for letting me know the permission issue, Ryan Blue. I have 

[jira] [Resolved] (SPARK-24228) Fix the lint error

[ https://issues.apache.org/jira/browse/SPARK-24228?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-24228. -- Resolution: Fixed Fix Version/s: 2.4.0 Fixed in

[jira] [Commented] (SPARK-24232) Allow referring to kubernetes secrets as env variable

[ https://issues.apache.org/jira/browse/SPARK-24232?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16473719#comment-16473719 ] Apache Spark commented on SPARK-24232: -- User 'skonto' has created a pull request for this issue:

[jira] [Assigned] (SPARK-24232) Allow referring to kubernetes secrets as env variable

[ https://issues.apache.org/jira/browse/SPARK-24232?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-24232: Assignee: (was: Apache Spark) > Allow referring to kubernetes secrets as env variable

[jira] [Assigned] (SPARK-24232) Allow referring to kubernetes secrets as env variable

[ https://issues.apache.org/jira/browse/SPARK-24232?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-24232: Assignee: Apache Spark > Allow referring to kubernetes secrets as env variable >

[jira] [Commented] (SPARK-24266) Spark client terminates while driver is still running

[ https://issues.apache.org/jira/browse/SPARK-24266?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16473717#comment-16473717 ] Chun Chen commented on SPARK-24266: --- cc [~anirudh] > Spark client terminates while driver is still

[jira] [Updated] (SPARK-24266) Spark client terminates while driver is still running

[ https://issues.apache.org/jira/browse/SPARK-24266?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chun Chen updated SPARK-24266: -- Description: {code} Warning: Ignoring non-spark config property: Default=system properties included

[jira] [Updated] (SPARK-24266) Spark client terminates while driver is still running

[ https://issues.apache.org/jira/browse/SPARK-24266?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chun Chen updated SPARK-24266: -- Description: {code} Warning: Ignoring non-spark config property: Default=system properties included

[jira] [Updated] (SPARK-24266) Spark client terminates while driver is still running

[ https://issues.apache.org/jira/browse/SPARK-24266?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chun Chen updated SPARK-24266: -- Description: {code} Warning: Ignoring non-spark config property: Default=system properties included

[jira] [Updated] (SPARK-24266) Spark client terminates while driver is still running

[ https://issues.apache.org/jira/browse/SPARK-24266?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chun Chen updated SPARK-24266: -- Description: {code} Warning: Ignoring non-spark config property: Default=system properties included

[jira] [Created] (SPARK-24266) Spark client terminates while driver is still running

Chun Chen created SPARK-24266: - Summary: Spark client terminates while driver is still running Key: SPARK-24266 URL: https://issues.apache.org/jira/browse/SPARK-24266 Project: Spark Issue Type:

[jira] [Commented] (SPARK-24262) Fix typo in UDF error message

[ https://issues.apache.org/jira/browse/SPARK-24262?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16473692#comment-16473692 ] Holden Karau commented on SPARK-24262: -- Just a JIRA issue kept me from closing, was going to try

[jira] [Assigned] (SPARK-17916) CSV data source treats empty string as null no matter what nullValue option is

[ https://issues.apache.org/jira/browse/SPARK-17916?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon reassigned SPARK-17916: Assignee: Maxim Gekk > CSV data source treats empty string as null no matter what

[jira] [Resolved] (SPARK-17916) CSV data source treats empty string as null no matter what nullValue option is

[ https://issues.apache.org/jira/browse/SPARK-17916?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-17916. -- Resolution: Fixed Fix Version/s: 2.4.0 Issue resolved by pull request 21273

[jira] [Updated] (SPARK-24260) Support for multi-statement SQL in SparkSession.sql API

[ https://issues.apache.org/jira/browse/SPARK-24260?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Takeshi Yamamuro updated SPARK-24260: - Component/s: (was: Spark Core) SQL > Support for multi-statement

[jira] [Commented] (SPARK-24262) Fix typo in UDF error message

[ https://issues.apache.org/jira/browse/SPARK-24262?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16473687#comment-16473687 ] Takeshi Yamamuro commented on SPARK-24262: -- cc: [~holdenk] Probably, you forgot to close this?

[jira] [Resolved] (SPARK-24186) add array reverse and concat

[ https://issues.apache.org/jira/browse/SPARK-24186?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-24186. -- Resolution: Fixed Fix Version/s: 2.4.0 Issue resolved by pull request 21307

[jira] [Assigned] (SPARK-24186) add array reverse and concat

[ https://issues.apache.org/jira/browse/SPARK-24186?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon reassigned SPARK-24186: Assignee: Huaxin Gao > add array reverse and concat > - > >

[jira] [Commented] (SPARK-24261) Spark cannot read renamed managed Hive table

[ https://issues.apache.org/jira/browse/SPARK-24261?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16473662#comment-16473662 ] Suraj Nayak commented on SPARK-24261: - Also, need some pointers here from committers so as was there

[jira] [Commented] (SPARK-24265) lintr checks not failing PR build

[ https://issues.apache.org/jira/browse/SPARK-24265?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16473651#comment-16473651 ] Felix Cheung commented on SPARK-24265: -- example:

[jira] [Assigned] (SPARK-20538) Dataset.reduce operator should use withNewExecutionId (as foreach or foreachPartition)

[ https://issues.apache.org/jira/browse/SPARK-20538?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-20538: Assignee: Apache Spark > Dataset.reduce operator should use withNewExecutionId (as

[jira] [Assigned] (SPARK-20538) Dataset.reduce operator should use withNewExecutionId (as foreach or foreachPartition)

[ https://issues.apache.org/jira/browse/SPARK-20538?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-20538: Assignee: (was: Apache Spark) > Dataset.reduce operator should use withNewExecutionId

[jira] [Commented] (SPARK-20538) Dataset.reduce operator should use withNewExecutionId (as foreach or foreachPartition)

[ https://issues.apache.org/jira/browse/SPARK-20538?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16473650#comment-16473650 ] Apache Spark commented on SPARK-20538: -- User 'sohama4' has created a pull request for this issue:

[jira] [Updated] (SPARK-23780) Failed to use googleVis library with new SparkR

[ https://issues.apache.org/jira/browse/SPARK-23780?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Felix Cheung updated SPARK-23780: - Affects Version/s: 2.3.0 > Failed to use googleVis library with new SparkR >

[jira] [Updated] (SPARK-23780) Failed to use googleVis library with new SparkR

[ https://issues.apache.org/jira/browse/SPARK-23780?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Felix Cheung updated SPARK-23780: - Labels: regression (was: regresion) > Failed to use googleVis library with new SparkR >

[jira] [Updated] (SPARK-23780) Failed to use googleVis library with new SparkR

[ https://issues.apache.org/jira/browse/SPARK-23780?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Felix Cheung updated SPARK-23780: - Labels: regresion (was: ) > Failed to use googleVis library with new SparkR >

[jira] [Assigned] (SPARK-23780) Failed to use googleVis library with new SparkR

[ https://issues.apache.org/jira/browse/SPARK-23780?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-23780: Assignee: Apache Spark > Failed to use googleVis library with new SparkR >

[jira] [Commented] (SPARK-24261) Spark cannot read renamed managed Hive table

[ https://issues.apache.org/jira/browse/SPARK-24261?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16473640#comment-16473640 ] Chitral Verma commented on SPARK-24261: --- I can see that in

[jira] [Assigned] (SPARK-23780) Failed to use googleVis library with new SparkR

[ https://issues.apache.org/jira/browse/SPARK-23780?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-23780: Assignee: (was: Apache Spark) > Failed to use googleVis library with new SparkR >

[jira] [Commented] (SPARK-23780) Failed to use googleVis library with new SparkR

[ https://issues.apache.org/jira/browse/SPARK-23780?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16473641#comment-16473641 ] Apache Spark commented on SPARK-23780: -- User 'felixcheung' has created a pull request for this

[jira] [Created] (SPARK-24265) lintr checks not failing PR build

Felix Cheung created SPARK-24265: Summary: lintr checks not failing PR build Key: SPARK-24265 URL: https://issues.apache.org/jira/browse/SPARK-24265 Project: Spark Issue Type: Bug

[jira] [Updated] (SPARK-24263) SparkR java check breaks on openjdk

[ https://issues.apache.org/jira/browse/SPARK-24263?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Felix Cheung updated SPARK-24263: - Description: testing with openjdk, noticed that it breaks because the version string is

[jira] [Updated] (SPARK-24263) SparkR java check breaks on openjdk

[ https://issues.apache.org/jira/browse/SPARK-24263?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Felix Cheung updated SPARK-24263: - Description: testing with openjdk, noticed that it breaks because the version string is

[jira] [Updated] (SPARK-24264) [Structured Streaming] Remove 'mergeSchema' option from Parquet source configuration

[ https://issues.apache.org/jira/browse/SPARK-24264?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gerard Maas updated SPARK-24264: Issue Type: Bug (was: Improvement) > [Structured Streaming] Remove 'mergeSchema' option from

[jira] [Commented] (SPARK-24263) SparkR java check breaks on openjdk

[ https://issues.apache.org/jira/browse/SPARK-24263?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16473632#comment-16473632 ] Apache Spark commented on SPARK-24263: -- User 'felixcheung' has created a pull request for this

[jira] [Assigned] (SPARK-24263) SparkR java check breaks on openjdk

[ https://issues.apache.org/jira/browse/SPARK-24263?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-24263: Assignee: (was: Apache Spark) > SparkR java check breaks on openjdk >

[jira] [Updated] (SPARK-24263) SparkR java check breaks on openjdk

[ https://issues.apache.org/jira/browse/SPARK-24263?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Felix Cheung updated SPARK-24263: - Priority: Blocker (was: Major) > SparkR java check breaks on openjdk >

[jira] [Updated] (SPARK-24263) SparkR java check breaks on openjdk

[ https://issues.apache.org/jira/browse/SPARK-24263?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Felix Cheung updated SPARK-24263: - Affects Version/s: (was: 2.3.0) 2.3.1 > SparkR java check breaks on

[jira] [Assigned] (SPARK-24263) SparkR java check breaks on openjdk

[ https://issues.apache.org/jira/browse/SPARK-24263?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-24263: Assignee: Apache Spark > SparkR java check breaks on openjdk >

[jira] [Created] (SPARK-24264) [Structured Streaming] Remove 'mergeSchema' option from Parquet source configuration

Gerard Maas created SPARK-24264: --- Summary: [Structured Streaming] Remove 'mergeSchema' option from Parquet source configuration Key: SPARK-24264 URL: https://issues.apache.org/jira/browse/SPARK-24264

[jira] [Created] (SPARK-24263) SparkR java check breaks on openjdk

Felix Cheung created SPARK-24263: Summary: SparkR java check breaks on openjdk Key: SPARK-24263 URL: https://issues.apache.org/jira/browse/SPARK-24263 Project: Spark Issue Type: Bug

[jira] [Assigned] (SPARK-24262) Fix typo in UDF error message

[ https://issues.apache.org/jira/browse/SPARK-24262?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] holdenk reassigned SPARK-24262: --- Assignee: Kelley Robinson > Fix typo in UDF error message > - > >

[jira] [Assigned] (SPARK-24262) Fix typo in UDF error message

[ https://issues.apache.org/jira/browse/SPARK-24262?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-24262: Assignee: Apache Spark > Fix typo in UDF error message > - >

[jira] [Assigned] (SPARK-24262) Fix typo in UDF error message

[ https://issues.apache.org/jira/browse/SPARK-24262?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-24262: Assignee: (was: Apache Spark) > Fix typo in UDF error message >

[jira] [Commented] (SPARK-24262) Fix typo in UDF error message

[ https://issues.apache.org/jira/browse/SPARK-24262?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16473623#comment-16473623 ] Apache Spark commented on SPARK-24262: -- User 'robinske' has created a pull request for this issue:

[jira] [Created] (SPARK-24262) Fix typo in UDF error message

holdenk created SPARK-24262: --- Summary: Fix typo in UDF error message Key: SPARK-24262 URL: https://issues.apache.org/jira/browse/SPARK-24262 Project: Spark Issue Type: Improvement

[jira] [Updated] (SPARK-20568) Delete files after processing in structured streaming

[ https://issues.apache.org/jira/browse/SPARK-20568?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-20568: -- Affects Version/s: 2.2.1 > Delete files after processing in structured streaming >

[jira] [Updated] (SPARK-24261) Spark cannot read renamed managed Hive table

[ https://issues.apache.org/jira/browse/SPARK-24261?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Suraj Nayak updated SPARK-24261: Description: When spark creates hive table using df.write.saveAsTable, it creates managed table

[jira] [Updated] (SPARK-24261) Spark cannot read renamed managed Hive table

[ https://issues.apache.org/jira/browse/SPARK-24261?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Suraj Nayak updated SPARK-24261: Attachment: some_db.some_new_table_buggy_path.ddl > Spark cannot read renamed managed Hive table >

[jira] [Updated] (SPARK-24261) Spark cannot read renamed managed Hive table

[ https://issues.apache.org/jira/browse/SPARK-24261?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Suraj Nayak updated SPARK-24261: Attachment: some_db.some_table.ddl some_db.some_new_table.ddl

[jira] [Updated] (SPARK-24261) Spark cannot read renamed managed Hive table

[ https://issues.apache.org/jira/browse/SPARK-24261?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Suraj Nayak updated SPARK-24261: Attachment: (was: some_db.some_new_table_buggy_path.ddl) > Spark cannot read renamed managed

[jira] [Updated] (SPARK-24261) Spark cannot read renamed managed Hive table

[ https://issues.apache.org/jira/browse/SPARK-24261?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Suraj Nayak updated SPARK-24261: Description: When spark creates hive table using df.write.saveAsTable, it creates managed table

[jira] [Updated] (SPARK-24261) Spark cannot read renamed managed Hive table

[ https://issues.apache.org/jira/browse/SPARK-24261?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Suraj Nayak updated SPARK-24261: Description: When spark creates hive table using df.write.saveAsTable, it creates managed table

[jira] [Created] (SPARK-24261) Spark cannot read renamed managed Hive table

Suraj Nayak created SPARK-24261: --- Summary: Spark cannot read renamed managed Hive table Key: SPARK-24261 URL: https://issues.apache.org/jira/browse/SPARK-24261 Project: Spark Issue Type: Bug

[jira] [Commented] (SPARK-24130) Data Source V2: Join Push Down

[ https://issues.apache.org/jira/browse/SPARK-24130?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16473594#comment-16473594 ] Ryan Blue commented on SPARK-24130: --- [~jliwork] could you please open up access to allow open

[jira] [Commented] (SPARK-24229) Upgrade to the latest Apache Thrift 0.10.0 release

[ https://issues.apache.org/jira/browse/SPARK-24229?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16473595#comment-16473595 ] Ray Donnelly commented on SPARK-24229: -- Ok I'll need to figure out how to test it then. Thanks for

[jira] [Commented] (SPARK-24229) Upgrade to the latest Apache Thrift 0.10.0 release

[ https://issues.apache.org/jira/browse/SPARK-24229?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16473590#comment-16473590 ] Marcelo Vanzin commented on SPARK-24229: The best approach if you really want this fixed is to 

[jira] [Updated] (SPARK-24260) Support for multi-statement SQL in SparkSession.sql API

[ https://issues.apache.org/jira/browse/SPARK-24260?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ravindra Nath Kakarla updated SPARK-24260: -- Description: sparkSession.sql API only supports a single SQL statement to be

[jira] [Created] (SPARK-24260) Support for multi-statement SQL in SparkSession.sql API

Ravindra Nath Kakarla created SPARK-24260: - Summary: Support for multi-statement SQL in SparkSession.sql API Key: SPARK-24260 URL: https://issues.apache.org/jira/browse/SPARK-24260 Project:

[jira] [Commented] (SPARK-19618) Inconsistency wrt max. buckets allowed from Dataframe API vs SQL

[ https://issues.apache.org/jira/browse/SPARK-19618?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16473584#comment-16473584 ] Fernando Pereira commented on SPARK-19618: -- [~cloud_fan] I have created the Jira and an

[jira] [Commented] (SPARK-18580) Use spark.streaming.backpressure.initialRate in DirectKafkaInputDStream

[ https://issues.apache.org/jira/browse/SPARK-18580?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16473557#comment-16473557 ] Farah Fertassi commented on SPARK-18580: Hello folks, is the fix version 2.4.0 ? Thank you :) >

[jira] [Commented] (SPARK-24187) add array join

[ https://issues.apache.org/jira/browse/SPARK-24187?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16473547#comment-16473547 ] Apache Spark commented on SPARK-24187: -- User 'huaxingao' has created a pull request for this issue:

[jira] [Assigned] (SPARK-24187) add array join

[ https://issues.apache.org/jira/browse/SPARK-24187?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-24187: Assignee: (was: Apache Spark) > add array join > -- > >

[jira] [Assigned] (SPARK-24187) add array join

[ https://issues.apache.org/jira/browse/SPARK-24187?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-24187: Assignee: Apache Spark > add array join > -- > > Key:

[jira] [Resolved] (SPARK-16658) Add EdgePartition.withVertexAttributes

[ https://issues.apache.org/jira/browse/SPARK-16658?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen resolved SPARK-16658. --- Resolution: Won't Fix > Add EdgePartition.withVertexAttributes >

[jira] [Resolved] (SPARK-23884) hasLaunchedTask should be true when launchedAnyTask be true

[ https://issues.apache.org/jira/browse/SPARK-23884?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen resolved SPARK-23884. --- Resolution: Not A Problem Fix Version/s: (was: 2.3.0) > hasLaunchedTask should be true

[jira] [Resolved] (SPARK-15739) Expose aggregateMessagesWithActiveSet to users.

[ https://issues.apache.org/jira/browse/SPARK-15739?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen resolved SPARK-15739. --- Resolution: Won't Fix > Expose aggregateMessagesWithActiveSet to users. >

[jira] [Resolved] (SPARK-23812) DFS should be removed from unsupportedHiveNativeCommands in SqlBase.g4

[ https://issues.apache.org/jira/browse/SPARK-23812?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen resolved SPARK-23812. --- Resolution: Not A Problem > DFS should be removed from unsupportedHiveNativeCommands in SqlBase.g4 >