[jira] [Commented] (SPARK-25328) Add a test case for having two columns as the grouping key in group aggregate pandas UDF

2018-09-03 Thread Xiao Li (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25328?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16602630#comment-16602630 ] Xiao Li commented on SPARK-25328: - cc [~icexelloss] [~bryanc] [~hyukjin.kwon] > Add a test case for

[jira] [Created] (SPARK-25328) Add a test case for having two columns as the grouping key in group aggregate pandas UDF

2018-09-03 Thread Xiao Li (JIRA)
Xiao Li created SPARK-25328: --- Summary: Add a test case for having two columns as the grouping key in group aggregate pandas UDF Key: SPARK-25328 URL: https://issues.apache.org/jira/browse/SPARK-25328

[jira] [Resolved] (SPARK-25310) ArraysOverlap may throw a CompileException

2018-09-03 Thread Takuya Ueshin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25310?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Takuya Ueshin resolved SPARK-25310. --- Resolution: Fixed Assignee: Kazuaki Ishizaki Fix Version/s: 2.4.0 Issue

[jira] [Resolved] (SPARK-25307) ArraySort function may return a error in the code generation phase.

2018-09-03 Thread Takuya Ueshin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25307?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Takuya Ueshin resolved SPARK-25307. --- Resolution: Fixed Assignee: Dilip Biswal Fix Version/s: 2.4.0 Issue

[jira] [Resolved] (SPARK-25308) ArrayContains function may return a error in the code generation phase.

2018-09-03 Thread Takuya Ueshin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25308?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Takuya Ueshin resolved SPARK-25308. --- Resolution: Fixed Assignee: Dilip Biswal Fix Version/s: 2.4.0 Issue

[jira] [Updated] (SPARK-25319) Spark MLlib, GraphX 2.4 QA umbrella

2018-09-03 Thread Weichen Xu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25319?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Weichen Xu updated SPARK-25319: --- Target Version/s: 2.4.0 (was: 2.3.0) > Spark MLlib, GraphX 2.4 QA umbrella >

[jira] [Updated] (SPARK-25319) Spark MLlib, GraphX 2.4 QA umbrella

2018-09-03 Thread Weichen Xu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25319?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Weichen Xu updated SPARK-25319: --- Fix Version/s: (was: 2.3.0) 2.4.0 > Spark MLlib, GraphX 2.4 QA umbrella >

[jira] [Updated] (SPARK-25325) ML, Graph 2.4 QA: Update user guide for new features & APIs

2018-09-03 Thread Weichen Xu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25325?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Weichen Xu updated SPARK-25325: --- Summary: ML, Graph 2.4 QA: Update user guide for new features & APIs (was: ML, Graph 2.3 QA:

[jira] [Updated] (SPARK-25327) Update MLlib, GraphX websites for 2.4

2018-09-03 Thread Weichen Xu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25327?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Weichen Xu updated SPARK-25327: --- Affects Version/s: 2.4.0 Target Version/s: 2.4.0 Summary: Update MLlib, GraphX

[jira] [Updated] (SPARK-25325) ML, Graph 2.3 QA: Update user guide for new features & APIs

2018-09-03 Thread Weichen Xu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25325?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Weichen Xu updated SPARK-25325: --- Affects Version/s: 2.4.0 Target Version/s: 2.4.0 Fix Version/s: (was: 2.3.0)

[jira] [Updated] (SPARK-25326) ML, Graph 2.4 QA: Programming guide update and migration guide

2018-09-03 Thread Weichen Xu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25326?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Weichen Xu updated SPARK-25326: --- Affects Version/s: (was: 2.3.0) 2.4.0 Target Version/s: 2.4.0 (was:

[jira] [Updated] (SPARK-25324) ML 2.4 QA: API: Java compatibility, docs

2018-09-03 Thread Weichen Xu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25324?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Weichen Xu updated SPARK-25324: --- Affects Version/s: 2.4.0 Target Version/s: 2.4.0 Fix Version/s: (was: 2.3.0)

[jira] [Updated] (SPARK-25321) ML, Graph 2.4 QA: API: New Scala APIs, docs

2018-09-03 Thread Weichen Xu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25321?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Weichen Xu updated SPARK-25321: --- Affects Version/s: (was: 2.3.0) 2.4.0 Target Version/s: 2.4.0 (was:

[jira] [Updated] (SPARK-25323) ML 2.4 QA: API: Python API coverage

2018-09-03 Thread Weichen Xu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25323?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Weichen Xu updated SPARK-25323: --- Target Version/s: 2.4.0 Summary: ML 2.4 QA: API: Python API coverage (was: CLONE - ML

[jira] [Updated] (SPARK-25323) CLONE - ML 2.3 QA: API: Python API coverage

2018-09-03 Thread Weichen Xu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25323?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Weichen Xu updated SPARK-25323: --- Affects Version/s: (was: 2.3.0) 2.4.0 Target Version/s: (was:

[jira] [Updated] (SPARK-25320) ML, Graph 2.4 QA: API: Binary incompatible changes

2018-09-03 Thread Weichen Xu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25320?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Weichen Xu updated SPARK-25320: --- Affects Version/s: (was: 2.3.0) 2.4.0 Target Version/s: 2.4.0 (was:

[jira] [Updated] (SPARK-25322) ML, Graph 2.4 QA: API: Experimental, DeveloperApi, final, sealed audit

2018-09-03 Thread Weichen Xu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25322?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Weichen Xu updated SPARK-25322: --- Affects Version/s: 2.4.0 Fix Version/s: (was: 2.3.0) Summary: ML, Graph

[jira] [Updated] (SPARK-25321) ML, Graph 2.4 QA: API: New Scala APIs, docs

2018-09-03 Thread Weichen Xu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25321?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Weichen Xu updated SPARK-25321: --- Description: Audit new public Scala APIs added to MLlib & GraphX. Take note of: * Protected/public

[jira] [Updated] (SPARK-25319) Spark MLlib, GraphX 2.4 QA umbrella

2018-09-03 Thread Weichen Xu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25319?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Weichen Xu updated SPARK-25319: --- Description: This JIRA lists tasks for the next Spark release's QA period for MLlib and GraphX.

[jira] [Created] (SPARK-25326) CLONE - ML, Graph 2.3 QA: Programming guide update and migration guide

2018-09-03 Thread Weichen Xu (JIRA)
Weichen Xu created SPARK-25326: -- Summary: CLONE - ML, Graph 2.3 QA: Programming guide update and migration guide Key: SPARK-25326 URL: https://issues.apache.org/jira/browse/SPARK-25326 Project: Spark

[jira] [Created] (SPARK-25324) CLONE - ML 2.3 QA: API: Java compatibility, docs

2018-09-03 Thread Weichen Xu (JIRA)
Weichen Xu created SPARK-25324: -- Summary: CLONE - ML 2.3 QA: API: Java compatibility, docs Key: SPARK-25324 URL: https://issues.apache.org/jira/browse/SPARK-25324 Project: Spark Issue Type:

[jira] [Created] (SPARK-25323) CLONE - ML 2.3 QA: API: Python API coverage

2018-09-03 Thread Weichen Xu (JIRA)
Weichen Xu created SPARK-25323: -- Summary: CLONE - ML 2.3 QA: API: Python API coverage Key: SPARK-25323 URL: https://issues.apache.org/jira/browse/SPARK-25323 Project: Spark Issue Type: Sub-task

[jira] [Updated] (SPARK-25320) ML, Graph 2.4 QA: API: Binary incompatible changes

2018-09-03 Thread Weichen Xu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25320?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Weichen Xu updated SPARK-25320: --- Summary: ML, Graph 2.4 QA: API: Binary incompatible changes (was: CLONE - ML, Graph 2.3 QA: API:

[jira] [Created] (SPARK-25320) CLONE - ML, Graph 2.3 QA: API: Binary incompatible changes

2018-09-03 Thread Weichen Xu (JIRA)
Weichen Xu created SPARK-25320: -- Summary: CLONE - ML, Graph 2.3 QA: API: Binary incompatible changes Key: SPARK-25320 URL: https://issues.apache.org/jira/browse/SPARK-25320 Project: Spark Issue

[jira] [Created] (SPARK-25322) CLONE - ML, Graph 2.3 QA: API: Experimental, DeveloperApi, final, sealed audit

2018-09-03 Thread Weichen Xu (JIRA)
Weichen Xu created SPARK-25322: -- Summary: CLONE - ML, Graph 2.3 QA: API: Experimental, DeveloperApi, final, sealed audit Key: SPARK-25322 URL: https://issues.apache.org/jira/browse/SPARK-25322 Project:

[jira] [Created] (SPARK-25327) CLONE - Update MLlib, GraphX websites for 2.3

2018-09-03 Thread Weichen Xu (JIRA)
Weichen Xu created SPARK-25327: -- Summary: CLONE - Update MLlib, GraphX websites for 2.3 Key: SPARK-25327 URL: https://issues.apache.org/jira/browse/SPARK-25327 Project: Spark Issue Type:

[jira] [Created] (SPARK-25321) CLONE - ML, Graph 2.3 QA: API: New Scala APIs, docs

2018-09-03 Thread Weichen Xu (JIRA)
Weichen Xu created SPARK-25321: -- Summary: CLONE - ML, Graph 2.3 QA: API: New Scala APIs, docs Key: SPARK-25321 URL: https://issues.apache.org/jira/browse/SPARK-25321 Project: Spark Issue Type:

[jira] [Created] (SPARK-25325) CLONE - ML, Graph 2.3 QA: Update user guide for new features & APIs

2018-09-03 Thread Weichen Xu (JIRA)
Weichen Xu created SPARK-25325: -- Summary: CLONE - ML, Graph 2.3 QA: Update user guide for new features & APIs Key: SPARK-25325 URL: https://issues.apache.org/jira/browse/SPARK-25325 Project: Spark

[jira] [Created] (SPARK-25319) Spark MLlib, GraphX 2.4 QA umbrella

2018-09-03 Thread Weichen Xu (JIRA)
Weichen Xu created SPARK-25319: -- Summary: Spark MLlib, GraphX 2.4 QA umbrella Key: SPARK-25319 URL: https://issues.apache.org/jira/browse/SPARK-25319 Project: Spark Issue Type: Umbrella

[jira] [Assigned] (SPARK-25318) Add exception handling when wrapping the input stream during the the fetch or stage retry in response to a corrupted block

2018-09-03 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25318?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-25318: Assignee: (was: Apache Spark) > Add exception handling when wrapping the input

[jira] [Commented] (SPARK-25318) Add exception handling when wrapping the input stream during the the fetch or stage retry in response to a corrupted block

2018-09-03 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25318?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16602571#comment-16602571 ] Apache Spark commented on SPARK-25318: -- User 'rezasafi' has created a pull request for this issue:

[jira] [Assigned] (SPARK-25318) Add exception handling when wrapping the input stream during the the fetch or stage retry in response to a corrupted block

2018-09-03 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25318?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-25318: Assignee: Apache Spark > Add exception handling when wrapping the input stream during

[jira] [Updated] (SPARK-25318) Add exception handling when wrapping the input stream during the the fetch or stage retry in response to a corrupted block

2018-09-03 Thread Reza Safi (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25318?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Reza Safi updated SPARK-25318: -- Description: SPARK-4105 provided a solution to block corruption issue by retrying the fetch or the

[jira] [Commented] (SPARK-25317) MemoryBlock performance regression

2018-09-03 Thread Kazuaki Ishizaki (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25317?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16602567#comment-16602567 ] Kazuaki Ishizaki commented on SPARK-25317: -- I confirmed this performance difference even after

[jira] [Commented] (SPARK-25318) Add exception handling when wrapping the input stream during the the fetch or stage retry in response to a corrupted block

2018-09-03 Thread Reza Safi (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25318?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16602565#comment-16602565 ] Reza Safi commented on SPARK-25318: --- I will send a pr for this shortly > Add exception handling when

[jira] [Created] (SPARK-25318) Add exception handling when wrapping the input stream during the the fetch or stage retry in response to a corrupted block

2018-09-03 Thread Reza Safi (JIRA)
Reza Safi created SPARK-25318: - Summary: Add exception handling when wrapping the input stream during the the fetch or stage retry in response to a corrupted block Key: SPARK-25318 URL:

[jira] [Commented] (SPARK-25237) FileScanRdd's inputMetrics is wrong when select the datasource table with limit

2018-09-03 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25237?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16602542#comment-16602542 ] Apache Spark commented on SPARK-25237: -- User 'maropu' has created a pull request for this issue:

[jira] [Commented] (SPARK-25237) FileScanRdd's inputMetrics is wrong when select the datasource table with limit

2018-09-03 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25237?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16602543#comment-16602543 ] Apache Spark commented on SPARK-25237: -- User 'maropu' has created a pull request for this issue:

[jira] [Commented] (SPARK-25293) Dataframe write to csv saves part files in outputDireotry/task-xx/part-xxx instead of directly saving in outputDir

2018-09-03 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25293?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16602528#comment-16602528 ] Hyukjin Kwon commented on SPARK-25293: -- [~omkar999], would you be able to test this and see if the

[jira] [Commented] (SPARK-25317) MemoryBlock performance regression

2018-09-03 Thread Jungtaek Lim (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25317?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16602512#comment-16602512 ] Jungtaek Lim commented on SPARK-25317: -- Why not running test with JMH, applying warmup and

[jira] [Assigned] (SPARK-20395) Update Scala to 2.11.11 and zinc to 0.3.15

2018-09-03 Thread Dongjoon Hyun (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20395?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun reassigned SPARK-20395: - Assignee: DB Tsai > Update Scala to 2.11.11 and zinc to 0.3.15 >

[jira] [Comment Edited] (SPARK-20395) Update Scala to 2.11.11 and zinc to 0.3.15

2018-09-03 Thread Dongjoon Hyun (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20395?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16602510#comment-16602510 ] Dongjoon Hyun edited comment on SPARK-20395 at 9/4/18 12:59 AM: In

[jira] [Resolved] (SPARK-20395) Update Scala to 2.11.11 and zinc to 0.3.15

2018-09-03 Thread Dongjoon Hyun (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20395?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-20395. --- Resolution: Fixed Fix Version/s: 2.4.0 In SPARK-24418, Scala version becomes 2.11.12

[jira] [Reopened] (SPARK-20395) Update Scala to 2.11.11 and zinc to 0.3.15

2018-09-03 Thread Dongjoon Hyun (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20395?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun reopened SPARK-20395: --- > Update Scala to 2.11.11 and zinc to 0.3.15 > -- > >

[jira] [Commented] (SPARK-25317) MemoryBlock performance regression

2018-09-03 Thread Kazuaki Ishizaki (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25317?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16602506#comment-16602506 ] Kazuaki Ishizaki commented on SPARK-25317: -- Let me run this on 2.3 and master. One question.

[jira] [Updated] (SPARK-25317) MemoryBlock performance regression

2018-09-03 Thread Kazuaki Ishizaki (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25317?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kazuaki Ishizaki updated SPARK-25317: - Description: eThere is a performance regression when calculating hash code for

[jira] [Comment Edited] (SPARK-25317) MemoryBlock performance regression

2018-09-03 Thread Wenchen Fan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25317?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16602503#comment-16602503 ] Wenchen Fan edited comment on SPARK-25317 at 9/4/18 12:24 AM: -- cc [~kiszk]

[jira] [Commented] (SPARK-25317) MemoryBlock performance regression

2018-09-03 Thread Wenchen Fan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25317?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16602503#comment-16602503 ] Wenchen Fan commented on SPARK-25317: - cc [~kiszk] > MemoryBlock performance regression >

[jira] [Created] (SPARK-25317) MemoryBlock performance regression

2018-09-03 Thread Wenchen Fan (JIRA)
Wenchen Fan created SPARK-25317: --- Summary: MemoryBlock performance regression Key: SPARK-25317 URL: https://issues.apache.org/jira/browse/SPARK-25317 Project: Spark Issue Type: Bug

[jira] [Updated] (SPARK-25317) MemoryBlock performance regression

2018-09-03 Thread Wenchen Fan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25317?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan updated SPARK-25317: Priority: Blocker (was: Major) > MemoryBlock performance regression >

[jira] [Created] (SPARK-25316) Spark error - ERROR ContextCleaner: Error cleaning broadcast 22, Exception thrown in awaitResult:

2018-09-03 Thread Vidya (JIRA)
Vidya created SPARK-25316: - Summary: Spark error - ERROR ContextCleaner: Error cleaning broadcast 22, Exception thrown in awaitResult: Key: SPARK-25316 URL: https://issues.apache.org/jira/browse/SPARK-25316

[jira] [Commented] (SPARK-25279) Throw exception: zzcclp java.io.NotSerializableException: org.apache.spark.sql.TypedColumn in Spark-shell when run example of doc

2018-09-03 Thread Dilip Biswal (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25279?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16602440#comment-16602440 ] Dilip Biswal commented on SPARK-25279: -- [~zzcclp] Hmmn.. i don't know whats happening in the :paste

[jira] [Resolved] (SPARK-25117) Add EXEPT ALL and INTERSECT ALL support in R.

2018-09-03 Thread Felix Cheung (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25117?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Felix Cheung resolved SPARK-25117. -- Resolution: Fixed Assignee: Dilip Biswal > Add EXEPT ALL and INTERSECT ALL support in

[jira] [Updated] (SPARK-25117) Add EXEPT ALL and INTERSECT ALL support in R.

2018-09-03 Thread Felix Cheung (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25117?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Felix Cheung updated SPARK-25117: - Fix Version/s: 2.4.0 > Add EXEPT ALL and INTERSECT ALL support in R. >

[jira] [Closed] (SPARK-22190) Add Spark executor task metrics to Dropwizard metrics

2018-09-03 Thread Luca Canali (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22190?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Luca Canali closed SPARK-22190. --- > Add Spark executor task metrics to Dropwizard metrics >

[jira] [Closed] (SPARK-21829) Enable config to permanently blacklist a list of nodes

2018-09-03 Thread Luca Canali (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21829?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Luca Canali closed SPARK-21829. --- > Enable config to permanently blacklist a list of nodes >

[jira] [Closed] (SPARK-21519) Add an option to the JDBC data source to initialize the environment of the remote database session

2018-09-03 Thread Luca Canali (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21519?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Luca Canali closed SPARK-21519. --- > Add an option to the JDBC data source to initialize the environment of the > remote database session

[jira] [Updated] (SPARK-25309) Sci-kit Learn like Auto Pipeline Parallelization in Spark

2018-09-03 Thread Ravi (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25309?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ravi updated SPARK-25309: - Description: SPARK-19357 and SPARK-21911 haveĀ helped parallelize Pipelines in Spark. However, instead of

[jira] [Comment Edited] (SPARK-25279) Throw exception: zzcclp java.io.NotSerializableException: org.apache.spark.sql.TypedColumn in Spark-shell when run example of doc

2018-09-03 Thread Zhichao Zhang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25279?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16602308#comment-16602308 ] Zhichao Zhang edited comment on SPARK-25279 at 9/3/18 4:12 PM:

[jira] [Commented] (SPARK-25279) Throw exception: zzcclp java.io.NotSerializableException: org.apache.spark.sql.TypedColumn in Spark-shell when run example of doc

2018-09-03 Thread Zhichao Zhang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25279?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16602308#comment-16602308 ] Zhichao Zhang commented on SPARK-25279: [~dkbiswal], I followed your steps to run code

[jira] [Created] (SPARK-25315) setting "auto.offset.reset" to "earliest" has no effect in Structured Streaming with Spark 2.3.1 and Kafka 1.0

2018-09-03 Thread Zhenhao Li (JIRA)
Zhenhao Li created SPARK-25315: -- Summary: setting "auto.offset.reset" to "earliest" has no effect in Structured Streaming with Spark 2.3.1 and Kafka 1.0 Key: SPARK-25315 URL:

[jira] [Resolved] (SPARK-24767) Propagate MDC to spark-submit thread in InProcessAppHandle

2018-09-03 Thread Yifei Huang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24767?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yifei Huang resolved SPARK-24767. - Resolution: Won't Fix > Propagate MDC to spark-submit thread in InProcessAppHandle >

[jira] [Assigned] (SPARK-25298) spark-tools build failure for Scala 2.12

2018-09-03 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25298?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen reassigned SPARK-25298: - Assignee: Darcy Shen > spark-tools build failure for Scala 2.12 >

[jira] [Resolved] (SPARK-25298) spark-tools build failure for Scala 2.12

2018-09-03 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25298?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen resolved SPARK-25298. --- Resolution: Fixed Fix Version/s: 2.4.0 Issue resolved by pull request 22310

[jira] [Updated] (SPARK-25298) spark-tools build failure for Scala 2.12

2018-09-03 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25298?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen updated SPARK-25298: -- Priority: Minor (was: Major) > spark-tools build failure for Scala 2.12 >

[jira] [Commented] (SPARK-19728) PythonUDF with multiple parents shouldn't be pushed down when used as a predicate

2018-09-03 Thread Sergey Bahchissaraitsev (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19728?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16602040#comment-16602040 ] Sergey Bahchissaraitsev commented on SPARK-19728: - This is still happening in 2.3.1 when

[jira] [Created] (SPARK-25314) Invalid PythonUDF - requires attributes from more than one child - in "on" join condition

2018-09-03 Thread Sergey Bahchissaraitsev (JIRA)
Sergey Bahchissaraitsev created SPARK-25314: --- Summary: Invalid PythonUDF - requires attributes from more than one child - in "on" join condition Key: SPARK-25314 URL:

[jira] [Commented] (SPARK-25262) Make Spark local dir volumes configurable with Spark on Kubernetes

2018-09-03 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25262?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16601921#comment-16601921 ] Apache Spark commented on SPARK-25262: -- User 'rvesse' has created a pull request for this issue:

[jira] [Commented] (SPARK-25262) Make Spark local dir volumes configurable with Spark on Kubernetes

2018-09-03 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25262?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16601919#comment-16601919 ] Apache Spark commented on SPARK-25262: -- User 'rvesse' has created a pull request for this issue:

[jira] [Commented] (SPARK-25312) Add description for the conf spark.network.crypto.keyFactoryIterations

2018-09-03 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25312?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16601857#comment-16601857 ] Apache Spark commented on SPARK-25312: -- User 'npoberezkin' has created a pull request for this

[jira] [Commented] (SPARK-25312) Add description for the conf spark.network.crypto.keyFactoryIterations

2018-09-03 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25312?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16601856#comment-16601856 ] Apache Spark commented on SPARK-25312: -- User 'npoberezkin' has created a pull request for this

[jira] [Assigned] (SPARK-25312) Add description for the conf spark.network.crypto.keyFactoryIterations

2018-09-03 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25312?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-25312: Assignee: (was: Apache Spark) > Add description for the conf

[jira] [Assigned] (SPARK-25312) Add description for the conf spark.network.crypto.keyFactoryIterations

2018-09-03 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25312?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-25312: Assignee: Apache Spark > Add description for the conf

[jira] [Assigned] (SPARK-25313) Fix regression in FileFormatWriter output schema

2018-09-03 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25313?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-25313: Assignee: (was: Apache Spark) > Fix regression in FileFormatWriter output schema >

[jira] [Assigned] (SPARK-25313) Fix regression in FileFormatWriter output schema

2018-09-03 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25313?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-25313: Assignee: Apache Spark > Fix regression in FileFormatWriter output schema >

[jira] [Assigned] (SPARK-25313) Fix regression in FileFormatWriter output schema

2018-09-03 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25313?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-25313: Assignee: (was: Apache Spark) > Fix regression in FileFormatWriter output schema >

[jira] [Assigned] (SPARK-25313) Fix regression in FileFormatWriter output schema

2018-09-03 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25313?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-25313: Assignee: Apache Spark > Fix regression in FileFormatWriter output schema >

[jira] [Assigned] (SPARK-25313) Fix regression in FileFormatWriter output schema

2018-09-03 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25313?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-25313: Assignee: (was: Apache Spark) > Fix regression in FileFormatWriter output schema >

[jira] [Assigned] (SPARK-25313) Fix regression in FileFormatWriter output schema

2018-09-03 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25313?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-25313: Assignee: Apache Spark > Fix regression in FileFormatWriter output schema >

[jira] [Assigned] (SPARK-25313) Fix regression in FileFormatWriter output schema

2018-09-03 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25313?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-25313: Assignee: Apache Spark > Fix regression in FileFormatWriter output schema >

[jira] [Assigned] (SPARK-25313) Fix regression in FileFormatWriter output schema

2018-09-03 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25313?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-25313: Assignee: (was: Apache Spark) > Fix regression in FileFormatWriter output schema >

[jira] [Assigned] (SPARK-25313) Fix regression in FileFormatWriter output schema

2018-09-03 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25313?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-25313: Assignee: Apache Spark > Fix regression in FileFormatWriter output schema >

[jira] [Assigned] (SPARK-25313) Fix regression in FileFormatWriter output schema

2018-09-03 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25313?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-25313: Assignee: (was: Apache Spark) > Fix regression in FileFormatWriter output schema >

[jira] [Assigned] (SPARK-25313) Fix regression in FileFormatWriter output schema

2018-09-03 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25313?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-25313: Assignee: (was: Apache Spark) > Fix regression in FileFormatWriter output schema >

[jira] [Assigned] (SPARK-25313) Fix regression in FileFormatWriter output schema

2018-09-03 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25313?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-25313: Assignee: Apache Spark > Fix regression in FileFormatWriter output schema >

[jira] [Assigned] (SPARK-25313) Fix regression in FileFormatWriter output schema

2018-09-03 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25313?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-25313: Assignee: (was: Apache Spark) > Fix regression in FileFormatWriter output schema >

[jira] [Commented] (SPARK-25312) Add description for the conf spark.network.crypto.keyFactoryIterations

2018-09-03 Thread Nikita Poberezkin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25312?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16601834#comment-16601834 ] Nikita Poberezkin commented on SPARK-25312: --- I will add description > Add description for the

[jira] [Commented] (SPARK-25313) Fix regression in FileFormatWriter output schema

2018-09-03 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25313?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16601833#comment-16601833 ] Apache Spark commented on SPARK-25313: -- User 'gengliangwang' has created a pull request for this

[jira] [Assigned] (SPARK-25313) Fix regression in FileFormatWriter output schema

2018-09-03 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25313?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-25313: Assignee: Apache Spark > Fix regression in FileFormatWriter output schema >

[jira] [Created] (SPARK-25313) Fix regression in FileFormatWriter output schema

2018-09-03 Thread Gengliang Wang (JIRA)
Gengliang Wang created SPARK-25313: -- Summary: Fix regression in FileFormatWriter output schema Key: SPARK-25313 URL: https://issues.apache.org/jira/browse/SPARK-25313 Project: Spark Issue

[jira] [Updated] (SPARK-25313) Fix regression in FileFormatWriter output schema

2018-09-03 Thread Gengliang Wang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25313?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gengliang Wang updated SPARK-25313: --- Description: In the follow example: val location = "/tmp/t" val df =

[jira] [Comment Edited] (SPARK-25303) A DStream that is checkpointed should allow its parent(s) to be removed and not persisted

2018-09-03 Thread Nikunj Bansal (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25303?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16599566#comment-16599566 ] Nikunj Bansal edited comment on SPARK-25303 at 9/3/18 6:41 AM: --- I have a

[jira] [Updated] (SPARK-25311) `SPARK_LOCAL_HOSTNAME` unsupport IPV6 when do host checking

2018-09-03 Thread Xiaochen Ouyang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25311?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiaochen Ouyang updated SPARK-25311: Description: IPV4 can pass the follwing check {code:java} def checkHost(host: String,