[jira] [Updated] (SPARK-22646) Spark on Kubernetes - basic submission client

2017-11-29 Thread Anirudh Ramanathan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22646?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anirudh Ramanathan updated SPARK-22646: --- Description: The submission client is responsible for creating the Kubernetes pod

[jira] [Created] (SPARK-22648) Documentation for Kubernetes Scheduler Backend

2017-11-29 Thread Anirudh Ramanathan (JIRA)
Anirudh Ramanathan created SPARK-22648: -- Summary: Documentation for Kubernetes Scheduler Backend Key: SPARK-22648 URL: https://issues.apache.org/jira/browse/SPARK-22648 Project: Spark

[jira] [Created] (SPARK-22647) Docker files for image creation

2017-11-29 Thread Anirudh Ramanathan (JIRA)
Anirudh Ramanathan created SPARK-22647: -- Summary: Docker files for image creation Key: SPARK-22647 URL: https://issues.apache.org/jira/browse/SPARK-22647 Project: Spark Issue Type:

[jira] [Resolved] (SPARK-22645) Add Kubernetes Scheduler Backend

2017-11-29 Thread Anirudh Ramanathan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22645?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anirudh Ramanathan resolved SPARK-22645. Resolution: Fixed Fixed by https://github.com/apache/spark/pull/19468 > Add

[jira] [Created] (SPARK-22645) Add Kubernetes Scheduler Backend

2017-11-29 Thread Anirudh Ramanathan (JIRA)
Anirudh Ramanathan created SPARK-22645: -- Summary: Add Kubernetes Scheduler Backend Key: SPARK-22645 URL: https://issues.apache.org/jira/browse/SPARK-22645 Project: Spark Issue Type:

[jira] [Updated] (SPARK-22645) Add Scheduler Backend with static allocation of executors

2017-11-29 Thread Anirudh Ramanathan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22645?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anirudh Ramanathan updated SPARK-22645: --- Summary: Add Scheduler Backend with static allocation of executors (was: Add

[jira] [Resolved] (SPARK-22605) OutputMetrics empty for DataFrame writes

2017-11-29 Thread Wenchen Fan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22605?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-22605. - Resolution: Fixed Fix Version/s: 2.3.0 Issue resolved by pull request 19833

[jira] [Assigned] (SPARK-22649) localCheckpoint support in Dataset API

2017-11-29 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22649?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-22649: Assignee: Apache Spark > localCheckpoint support in Dataset API >

[jira] [Created] (SPARK-22650) spark2.2 on yarn streaming can't connect hbase

2017-11-29 Thread ZHOUBEIHUA (JIRA)
ZHOUBEIHUA created SPARK-22650: -- Summary: spark2.2 on yarn streaming can't connect hbase Key: SPARK-22650 URL: https://issues.apache.org/jira/browse/SPARK-22650 Project: Spark Issue Type:

[jira] [Commented] (SPARK-22635) FileNotFoundException again while reading ORC files containing special characters

2017-11-29 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22635?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16270505#comment-16270505 ] Apache Spark commented on SPARK-22635: -- User 'mgaido91' has created a pull request for this issue:

[jira] [Assigned] (SPARK-22635) FileNotFoundException again while reading ORC files containing special characters

2017-11-29 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22635?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-22635: Assignee: (was: Apache Spark) > FileNotFoundException again while reading ORC files

[jira] [Assigned] (SPARK-22635) FileNotFoundException again while reading ORC files containing special characters

2017-11-29 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22635?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-22635: Assignee: Apache Spark > FileNotFoundException again while reading ORC files containing

[jira] [Commented] (SPARK-22393) spark-shell can't find imported types in class constructors, extends clause

2017-11-29 Thread Mark Petruska (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22393?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16270605#comment-16270605 ] Mark Petruska commented on SPARK-22393: --- Based on the comments above and the comments in the Scala

[jira] [Assigned] (SPARK-22605) OutputMetrics empty for DataFrame writes

2017-11-29 Thread Wenchen Fan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22605?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan reassigned SPARK-22605: --- Assignee: Wenchen Fan > OutputMetrics empty for DataFrame writes >

[jira] [Created] (SPARK-22649) localCheckpoint support in Dataset API

2017-11-29 Thread Fernando Pereira (JIRA)
Fernando Pereira created SPARK-22649: Summary: localCheckpoint support in Dataset API Key: SPARK-22649 URL: https://issues.apache.org/jira/browse/SPARK-22649 Project: Spark Issue Type:

[jira] [Closed] (SPARK-22598) ExecutorAllocationManager does not requests new executors when executor fail and target has not change

2017-11-29 Thread Lijia Liu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22598?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lijia Liu closed SPARK-22598. - > ExecutorAllocationManager does not requests new executors when executor fail > and target has not change

[jira] [Resolved] (SPARK-22598) ExecutorAllocationManager does not requests new executors when executor fail and target has not change

2017-11-29 Thread Lijia Liu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22598?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lijia Liu resolved SPARK-22598. --- Resolution: Unresolved > ExecutorAllocationManager does not requests new executors when executor

[jira] [Commented] (SPARK-22649) localCheckpoint support in Dataset API

2017-11-29 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22649?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16270440#comment-16270440 ] Apache Spark commented on SPARK-22649: -- User 'ferdonline' has created a pull request for this issue:

[jira] [Assigned] (SPARK-22649) localCheckpoint support in Dataset API

2017-11-29 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22649?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-22649: Assignee: (was: Apache Spark) > localCheckpoint support in Dataset API >

[jira] [Created] (SPARK-22646) Spark on Kubernetes - basic submission client

2017-11-29 Thread Anirudh Ramanathan (JIRA)
Anirudh Ramanathan created SPARK-22646: -- Summary: Spark on Kubernetes - basic submission client Key: SPARK-22646 URL: https://issues.apache.org/jira/browse/SPARK-22646 Project: Spark

[jira] [Updated] (SPARK-22651) Calling ImageSchema.readImages initiate multiple Hive clients

2017-11-29 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22651?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-22651: - Component/s: ML > Calling ImageSchema.readImages initiate multiple Hive clients >

[jira] [Updated] (SPARK-22642) the createdTempDir will not be deleted if an exception occurs

2017-11-29 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22642?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen updated SPARK-22642: -- Priority: Minor (was: Critical) This is hardly critical > the createdTempDir will not be deleted if

[jira] [Assigned] (SPARK-22651) Calling ImageSchema.readImages initiate multiple Hive clients

2017-11-29 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22651?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-22651: Assignee: Apache Spark > Calling ImageSchema.readImages initiate multiple Hive clients >

[jira] [Resolved] (SPARK-22650) spark2.2 on yarn streaming can't connect hbase

2017-11-29 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22650?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen resolved SPARK-22650. --- Resolution: Invalid Questions to the mailing list please > spark2.2 on yarn streaming can't

[jira] [Assigned] (SPARK-22651) Calling ImageSchema.readImages initiate multiple Hive clients

2017-11-29 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22651?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-22651: Assignee: (was: Apache Spark) > Calling ImageSchema.readImages initiate multiple Hive

[jira] [Commented] (SPARK-22651) Calling ImageSchema.readImages initiate multiple Hive clients

2017-11-29 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22651?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16270721#comment-16270721 ] Apache Spark commented on SPARK-22651: -- User 'HyukjinKwon' has created a pull request for this

[jira] [Commented] (SPARK-22633) spark-submit.cmd cannot handle long arguments

2017-11-29 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22633?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16270753#comment-16270753 ] Hyukjin Kwon commented on SPARK-22633: -- {{spark-submit2.cmd}} is only there for the purpose of

[jira] [Created] (SPARK-22651) Calling ImageSchema.readImages initiate multiple Hive clients

2017-11-29 Thread Hyukjin Kwon (JIRA)
Hyukjin Kwon created SPARK-22651: Summary: Calling ImageSchema.readImages initiate multiple Hive clients Key: SPARK-22651 URL: https://issues.apache.org/jira/browse/SPARK-22651 Project: Spark

[jira] [Assigned] (SPARK-22393) spark-shell can't find imported types in class constructors, extends clause

2017-11-29 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22393?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-22393: Assignee: (was: Apache Spark) > spark-shell can't find imported types in class

[jira] [Assigned] (SPARK-22652) remove set methods in ColumnarRow

2017-11-29 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22652?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-22652: Assignee: Apache Spark (was: Wenchen Fan) > remove set methods in ColumnarRow >

[jira] [Commented] (SPARK-22636) row count not being set correctly (always 0) after Statement.executeUpdate()

2017-11-29 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22636?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16270956#comment-16270956 ] Sean Owen commented on SPARK-22636: --- Yes that's my understanding. We can wait a beat here to see if

[jira] [Assigned] (SPARK-22652) remove set methods in ColumnarRow

2017-11-29 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22652?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-22652: Assignee: Wenchen Fan (was: Apache Spark) > remove set methods in ColumnarRow >

[jira] [Commented] (SPARK-22652) remove set methods in ColumnarRow

2017-11-29 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22652?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16270953#comment-16270953 ] Apache Spark commented on SPARK-22652: -- User 'cloud-fan' has created a pull request for this issue:

[jira] [Commented] (SPARK-22393) spark-shell can't find imported types in class constructors, extends clause

2017-11-29 Thread Mark Petruska (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22393?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16270912#comment-16270912 ] Mark Petruska commented on SPARK-22393: --- [~srowen], [~rdub], I can now confirm that the original

[jira] [Updated] (SPARK-22641) Pyspark UDF relying on column added with withColumn after distinct

2017-11-29 Thread Andrew Duffy (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22641?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Duffy updated SPARK-22641: - Description: We seem to have found an issue with PySpark UDFs interacting with {{withColumn}}

[jira] [Commented] (SPARK-22393) spark-shell can't find imported types in class constructors, extends clause

2017-11-29 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22393?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16270897#comment-16270897 ] Sean Owen commented on SPARK-22393: --- OK, so this is basically "fixed for Scala 2.12 only"? >

[jira] [Commented] (SPARK-22636) row count not being set correctly (always 0) after Statement.executeUpdate()

2017-11-29 Thread Sergio Lob (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22636?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16270920#comment-16270920 ] Sergio Lob commented on SPARK-22636: Since we are using the Hive JDBC driver to access Spark, I guess

[jira] [Updated] (SPARK-22641) Pyspark UDF relying on column added with withColumn after distinct

2017-11-29 Thread Andrew Duffy (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22641?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Duffy updated SPARK-22641: - Description: We seem to have found an issue with PySpark UDFs interacting with {{withColumn}}

[jira] [Commented] (SPARK-22653) executorAddress registered in CoarseGrainedSchedulerBackend.executorDataMap is null

2017-11-29 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22653?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16270977#comment-16270977 ] Thomas Graves commented on SPARK-22653: --- will have patch up shortly > executorAddress registered

[jira] [Created] (SPARK-22653) executorAddress registered in CoarseGrainedSchedulerBackend.executorDataMap is null

2017-11-29 Thread Thomas Graves (JIRA)
Thomas Graves created SPARK-22653: - Summary: executorAddress registered in CoarseGrainedSchedulerBackend.executorDataMap is null Key: SPARK-22653 URL: https://issues.apache.org/jira/browse/SPARK-22653

[jira] [Assigned] (SPARK-22646) Spark on Kubernetes - basic submission client

2017-11-29 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22646?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-22646: Assignee: Apache Spark > Spark on Kubernetes - basic submission client >

[jira] [Assigned] (SPARK-22646) Spark on Kubernetes - basic submission client

2017-11-29 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22646?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-22646: Assignee: (was: Apache Spark) > Spark on Kubernetes - basic submission client >

[jira] [Commented] (SPARK-22646) Spark on Kubernetes - basic submission client

2017-11-29 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22646?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16271011#comment-16271011 ] Apache Spark commented on SPARK-22646: -- User 'liyinan926' has created a pull request for this issue:

[jira] [Commented] (SPARK-22634) Update Bouncy castle dependency

2017-11-29 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22634?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16271012#comment-16271012 ] Sean Owen commented on SPARK-22634: --- Right, I see core depends on jets3t, so it's not just provided by

[jira] [Commented] (SPARK-22636) row count not being set correctly (always 0) after Statement.executeUpdate()

2017-11-29 Thread Sergio Lob (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22636?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16270865#comment-16270865 ] Sergio Lob commented on SPARK-22636: Are you implying that its not "fixable" in neither Hive nor

[jira] [Created] (SPARK-22652) remove set methods in ColumnarRow

2017-11-29 Thread Wenchen Fan (JIRA)
Wenchen Fan created SPARK-22652: --- Summary: remove set methods in ColumnarRow Key: SPARK-22652 URL: https://issues.apache.org/jira/browse/SPARK-22652 Project: Spark Issue Type: Sub-task

[jira] [Commented] (SPARK-22636) row count not being set correctly (always 0) after Statement.executeUpdate()

2017-11-29 Thread Sergio Lob (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22636?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16271013#comment-16271013 ] Sergio Lob commented on SPARK-22636: OK > row count not being set correctly (always 0) after

[jira] [Commented] (SPARK-22636) row count not being set correctly (always 0) after Statement.executeUpdate()

2017-11-29 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22636?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16270895#comment-16270895 ] Sean Owen commented on SPARK-22636: --- No. I'm saying the ticket you linked to does not sound like a bug

[jira] [Assigned] (SPARK-22393) spark-shell can't find imported types in class constructors, extends clause

2017-11-29 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22393?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-22393: Assignee: Apache Spark > spark-shell can't find imported types in class constructors,

[jira] [Commented] (SPARK-22393) spark-shell can't find imported types in class constructors, extends clause

2017-11-29 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22393?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16270903#comment-16270903 ] Apache Spark commented on SPARK-22393: -- User 'mpetruska' has created a pull request for this issue:

[jira] [Updated] (SPARK-22599) Avoid extra reading for cached table

2017-11-29 Thread Nan Zhu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22599?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nan Zhu updated SPARK-22599: Description: In the current implementation of Spark, InMemoryTableExec read all data in a cached table,

[jira] [Commented] (SPARK-22625) Properly cleanup inheritable thread-locals

2017-11-29 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22625?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16271041#comment-16271041 ] Sean Owen commented on SPARK-22625: --- I don't have any additional info for you. You can propose a PR.

[jira] [Assigned] (SPARK-22641) Pyspark UDF relying on column added with withColumn after distinct

2017-11-29 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22641?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-22641: Assignee: (was: Apache Spark) > Pyspark UDF relying on column added with withColumn

[jira] [Assigned] (SPARK-22641) Pyspark UDF relying on column added with withColumn after distinct

2017-11-29 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22641?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-22641: Assignee: Apache Spark > Pyspark UDF relying on column added with withColumn after

[jira] [Commented] (SPARK-22641) Pyspark UDF relying on column added with withColumn after distinct

2017-11-29 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22641?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16271085#comment-16271085 ] Apache Spark commented on SPARK-22641: -- User 'sethah' has created a pull request for this issue:

[jira] [Commented] (SPARK-22653) executorAddress registered in CoarseGrainedSchedulerBackend.executorDataMap is null

2017-11-29 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22653?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16271254#comment-16271254 ] Apache Spark commented on SPARK-22653: -- User 'tgravescs' has created a pull request for this issue:

[jira] [Created] (SPARK-22654) Retry download of Spark from ASF mirror in HiveExternalCatalogVersionsSuite

2017-11-29 Thread Sean Owen (JIRA)
Sean Owen created SPARK-22654: - Summary: Retry download of Spark from ASF mirror in HiveExternalCatalogVersionsSuite Key: SPARK-22654 URL: https://issues.apache.org/jira/browse/SPARK-22654 Project: Spark

[jira] [Assigned] (SPARK-22654) Retry download of Spark from ASF mirror in HiveExternalCatalogVersionsSuite

2017-11-29 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22654?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-22654: Assignee: Sean Owen (was: Apache Spark) > Retry download of Spark from ASF mirror in

[jira] [Commented] (SPARK-22654) Retry download of Spark from ASF mirror in HiveExternalCatalogVersionsSuite

2017-11-29 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22654?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16271365#comment-16271365 ] Apache Spark commented on SPARK-22654: -- User 'srowen' has created a pull request for this issue:

[jira] [Assigned] (SPARK-22654) Retry download of Spark from ASF mirror in HiveExternalCatalogVersionsSuite

2017-11-29 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22654?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-22654: Assignee: Apache Spark (was: Sean Owen) > Retry download of Spark from ASF mirror in

[jira] [Resolved] (SPARK-22622) OutOfMemory thrown by Closure Serializer without proper failure propagation

2017-11-29 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22622?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen resolved SPARK-22622. --- Resolution: Duplicate It looks like, nevertheless, something huge is in your closure. It's probably

[jira] [Commented] (SPARK-22625) Properly cleanup inheritable thread-locals

2017-11-29 Thread Tolstopyatov Vsevolod (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22625?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16271038#comment-16271038 ] Tolstopyatov Vsevolod commented on SPARK-22625: --- Ping [~srowen] > Properly cleanup

[jira] [Resolved] (SPARK-22615) Handle more cases in PropagateEmptyRelation

2017-11-29 Thread Xiao Li (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22615?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Li resolved SPARK-22615. - Resolution: Fixed Assignee: Gengliang Wang Fix Version/s: 2.3.0 > Handle more cases in

[jira] [Commented] (SPARK-22162) Executors and the driver use inconsistent Job IDs during the new RDD commit protocol

2017-11-29 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22162?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16271154#comment-16271154 ] Apache Spark commented on SPARK-22162: -- User 'rezasafi' has created a pull request for this issue:

[jira] [Resolved] (SPARK-20650) Remove JobProgressListener (and other unneeded classes)

2017-11-29 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20650?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin resolved SPARK-20650. Resolution: Fixed Fix Version/s: 2.3.0 Issue resolved by pull request 19750

[jira] [Resolved] (SPARK-18935) Use Mesos "Dynamic Reservation" resource for Spark

2017-11-29 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18935?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin resolved SPARK-18935. Resolution: Fixed Assignee: Stavros Kontopoulos Fix Version/s: 2.3.0 > Use

[jira] [Created] (SPARK-22656) Upgrade Arrow to 0.8.0

2017-11-29 Thread Shixiong Zhu (JIRA)
Shixiong Zhu created SPARK-22656: Summary: Upgrade Arrow to 0.8.0 Key: SPARK-22656 URL: https://issues.apache.org/jira/browse/SPARK-22656 Project: Spark Issue Type: Bug Components:

[jira] [Created] (SPARK-22655) Fail task instead of complete task silently in PythonRunner during shutdown

2017-11-29 Thread Li Jin (JIRA)
Li Jin created SPARK-22655: -- Summary: Fail task instead of complete task silently in PythonRunner during shutdown Key: SPARK-22655 URL: https://issues.apache.org/jira/browse/SPARK-22655 Project: Spark

[jira] [Assigned] (SPARK-22655) Fail task instead of complete task silently in PythonRunner during shutdown

2017-11-29 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22655?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-22655: Assignee: (was: Apache Spark) > Fail task instead of complete task silently in

[jira] [Commented] (SPARK-22655) Fail task instead of complete task silently in PythonRunner during shutdown

2017-11-29 Thread Li Jin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22655?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16271620#comment-16271620 ] Li Jin commented on SPARK-22655: PR: https://github.com/apache/spark/pull/19852 > Fail task instead of

[jira] [Commented] (SPARK-22655) Fail task instead of complete task silently in PythonRunner during shutdown

2017-11-29 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22655?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16271626#comment-16271626 ] Apache Spark commented on SPARK-22655: -- User 'icexelloss' has created a pull request for this issue:

[jira] [Assigned] (SPARK-22655) Fail task instead of complete task silently in PythonRunner during shutdown

2017-11-29 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22655?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-22655: Assignee: Apache Spark > Fail task instead of complete task silently in PythonRunner

[jira] [Created] (SPARK-22659) Use {{scala.binary.version}} in the artifactId in the pom.xml of common/tags and streaming

2017-11-29 Thread liyunzhang (JIRA)
liyunzhang created SPARK-22659: -- Summary: Use {{scala.binary.version}} in the artifactId in the pom.xml of common/tags and streaming Key: SPARK-22659 URL: https://issues.apache.org/jira/browse/SPARK-22659

[jira] [Updated] (SPARK-22657) Hadoop fs implementation classes are not loaded if they are part of the app jar or other jar when --packages flag is used

2017-11-29 Thread Stavros Kontopoulos (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22657?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Stavros Kontopoulos updated SPARK-22657: Description: To reproduce this issue run: ./bin/spark-submit --master

[jira] [Updated] (SPARK-22657) Hadoop fs implementation classes are not loaded if they are part of the app jar or other jar when --packages flag is used

2017-11-29 Thread Stavros Kontopoulos (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22657?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Stavros Kontopoulos updated SPARK-22657: Description: To reproduce this issue run: ./bin/spark-submit --master

[jira] [Resolved] (SPARK-22585) Url encoding of jar path expected?

2017-11-29 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22585?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-22585. -- Resolution: Fixed Fix Version/s: 2.3.0 Issue resolved by pull request 19834

[jira] [Commented] (SPARK-22657) Hadoop fs implementation classes are not loaded if they are part of the app jar or other jar when --packages flag is used

2017-11-29 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22657?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16271944#comment-16271944 ] Sean Owen commented on SPARK-22657: --- I wouldn't have expected that to work. The user app classloader

[jira] [Commented] (SPARK-22659) Use {{scala.binary.version}} in the artifactId in the pom.xml of common/tags and streaming

2017-11-29 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22659?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16272045#comment-16272045 ] Sean Owen commented on SPARK-22659: --- You can't put vars in the artifact names in Maven. Believe me if

[jira] [Commented] (SPARK-22659) remove sun.misc.Cleaner references

2017-11-29 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22659?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16272117#comment-16272117 ] Sean Owen commented on SPARK-22659: --- This isn't related to Scala 2.12 now. JDK 9 isn't supported, and

[jira] [Updated] (SPARK-22657) Hadoop fs implementation classes are not loaded if they are part of the app jar or other jar when --packages flag is used

2017-11-29 Thread Stavros Kontopoulos (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22657?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Stavros Kontopoulos updated SPARK-22657: Description: To reproduce this issue run: ./bin/spark-submit --master

[jira] [Updated] (SPARK-22658) SPIP: TeansorFlowOnSpark as a Scalable Deep Learning Lib of Apache Spark

2017-11-29 Thread Andy Feng (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22658?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andy Feng updated SPARK-22658: -- Description: SPIP: TeansorFlowOnSpark as a Scalable Deep Learning Lib of Apache Spark Authors: Lee

[jira] [Commented] (SPARK-22658) SPIP: TeansorFlowOnSpark as a Scalable Deep Learning Lib of Apache Spark

2017-11-29 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22658?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16271930#comment-16271930 ] Sean Owen commented on SPARK-22658: --- I don't see a strong reason this needs to be part of Spark. It

[jira] [Resolved] (SPARK-22608) Avoid code duplication regarding CodeGeneration.splitExpressions()

2017-11-29 Thread Wenchen Fan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22608?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-22608. - Resolution: Fixed Fix Version/s: 2.3.0 Issue resolved by pull request 19821

[jira] [Assigned] (SPARK-22608) Avoid code duplication regarding CodeGeneration.splitExpressions()

2017-11-29 Thread Wenchen Fan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22608?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan reassigned SPARK-22608: --- Assignee: Kazuaki Ishizaki > Avoid code duplication regarding

[jira] [Updated] (SPARK-22659) remove sun.misc.Cleaner references

2017-11-29 Thread liyunzhang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22659?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] liyunzhang updated SPARK-22659: --- Summary: remove sun.misc.Cleaner references (was: Use {{scala.binary.version}} in the artifactId in

[jira] [Updated] (SPARK-22659) remove sun.misc.Cleaner references

2017-11-29 Thread liyunzhang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22659?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] liyunzhang updated SPARK-22659: --- Description: build with scala-2.12 with following steps 1. change the pom.xml with scala-2.12 {code}

[jira] [Commented] (SPARK-22659) remove sun.misc.Cleaner references

2017-11-29 Thread liyunzhang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22659?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16272123#comment-16272123 ] liyunzhang commented on SPARK-22659: So when you compile with -Pscala-2.12 and jdk8, there is no

[jira] [Commented] (SPARK-22647) Docker files for image creation

2017-11-29 Thread Yinan Li (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22647?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16271813#comment-16271813 ] Yinan Li commented on SPARK-22647: -- Note: Some reference Dockerfiles are included in

[jira] [Updated] (SPARK-22657) Hadoop fs implementation classes are not loaded if they are part of the app jar or other jar when --packages flag is used

2017-11-29 Thread Stavros Kontopoulos (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22657?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Stavros Kontopoulos updated SPARK-22657: Description: To reproduce this issue run: ./bin/spark-submit --master

[jira] [Updated] (SPARK-22657) Hadoop fs implementation classes are not loaded if they are part of the app jar or other jar when --packages flag is used

2017-11-29 Thread Stavros Kontopoulos (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22657?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Stavros Kontopoulos updated SPARK-22657: Description: To reproduce this issue run: ./bin/spark-submit --master

[jira] [Updated] (SPARK-22657) Hadoop fs implementation classes are not loaded if they are part of the app jar or other jar when --packages flag is used

2017-11-29 Thread Stavros Kontopoulos (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22657?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Stavros Kontopoulos updated SPARK-22657: Description: To reproduce this issue run: ./bin/spark-submit --master

[jira] [Updated] (SPARK-22657) Hadoop fs implementation classes are not loaded if they are part of the app jar or other jar when --packages flag is used

2017-11-29 Thread Stavros Kontopoulos (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22657?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Stavros Kontopoulos updated SPARK-22657: Description: To reproduce this issue run: ./bin/spark-submit --master

[jira] [Commented] (SPARK-22630) Consolidate all configuration properties into one page

2017-11-29 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22630?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16272033#comment-16272033 ] Hyukjin Kwon commented on SPARK-22630: -- +1 for ^. > Consolidate all configuration properties into

[jira] [Comment Edited] (SPARK-22659) Use {{scala.binary.version}} in the artifactId in the pom.xml of common/tags and streaming

2017-11-29 Thread liyunzhang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22659?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16272048#comment-16272048 ] liyunzhang edited comment on SPARK-22659 at 11/30/17 2:31 AM: -- I saw the

[jira] [Commented] (SPARK-22659) Use {{scala.binary.version}} in the artifactId in the pom.xml of common/tags and streaming

2017-11-29 Thread liyunzhang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22659?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16272048#comment-16272048 ] liyunzhang commented on SPARK-22659: where is the script? > Use {{scala.binary.version}} in the

[jira] [Updated] (SPARK-22657) Hadoop fs implementation classes are not loaded if they are part of the app jar or other jar when --packages flag is used

2017-11-29 Thread Stavros Kontopoulos (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22657?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Stavros Kontopoulos updated SPARK-22657: Description: To reproduce this issue run: ``` ./bin/spark-submit --master

[jira] [Created] (SPARK-22657) Hadoop fs implementation classes are not loaded if they are part of the app jar or other jar when --packages flag is used

2017-11-29 Thread Stavros Kontopoulos (JIRA)
Stavros Kontopoulos created SPARK-22657: --- Summary: Hadoop fs implementation classes are not loaded if they are part of the app jar or other jar when --packages flag is used Key: SPARK-22657 URL:

[jira] [Updated] (SPARK-22658) SPIP: TeansorFlowOnSpark as a Scalable Deep Learning Lib of Apache Spark

2017-11-29 Thread Andy Feng (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22658?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andy Feng updated SPARK-22658: -- Description: TensorFlowOnSpark (TFoS) was released at github for distributed TensorFlow training and

[jira] [Commented] (SPARK-22659) remove sun.misc.Cleaner references

2017-11-29 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22659?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16272187#comment-16272187 ] Apache Spark commented on SPARK-22659: -- User 'kellyzly' has created a pull request for this issue:

[jira] [Assigned] (SPARK-22659) remove sun.misc.Cleaner references

2017-11-29 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22659?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-22659: Assignee: Apache Spark > remove sun.misc.Cleaner references >

  1   2   >