[jira] [Updated] (SPARK-25584) Document libsvm data source in doc site

2018-10-01 Thread Xiangrui Meng (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25584?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiangrui Meng updated SPARK-25584: -- Component/s: ML > Document libsvm data source in doc site >

[jira] [Updated] (SPARK-25347) Document image data source in doc site

2018-10-01 Thread Xiangrui Meng (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25347?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiangrui Meng updated SPARK-25347: -- Component/s: ML > Document image data source in doc site >

[jira] [Commented] (SPARK-25524) Spark datasource for image/libsvm user guide

2018-10-01 Thread Xiangrui Meng (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25524?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16634347#comment-16634347 ] Xiangrui Meng commented on SPARK-25524: --- Marked as duplicate and create SPARK-25584 for libsvm

[jira] [Updated] (SPARK-25584) Document libsvm data source in doc site

2018-10-01 Thread Xiangrui Meng (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25584?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiangrui Meng updated SPARK-25584: -- Description: Currently, we only have Scala/Java API docs for libsvm data source. It would be

[jira] [Created] (SPARK-25584) Document libsvm data source in doc site

2018-10-01 Thread Xiangrui Meng (JIRA)
Xiangrui Meng created SPARK-25584: - Summary: Document libsvm data source in doc site Key: SPARK-25584 URL: https://issues.apache.org/jira/browse/SPARK-25584 Project: Spark Issue Type: Story

[jira] [Resolved] (SPARK-25524) Spark datasource for image/libsvm user guide

2018-10-01 Thread Xiangrui Meng (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25524?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiangrui Meng resolved SPARK-25524. --- Resolution: Duplicate > Spark datasource for image/libsvm user guide >

[jira] [Commented] (SPARK-25378) ArrayData.toArray(StringType) assume UTF8String in 2.4

2018-10-01 Thread Xiangrui Meng (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25378?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16634335#comment-16634335 ] Xiangrui Meng commented on SPARK-25378: --- I don't think I'm the right person to decide here because

[jira] [Updated] (SPARK-25322) ML, Graph 2.4 QA: API: Experimental, DeveloperApi, final, sealed audit

2018-09-21 Thread Xiangrui Meng (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25322?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiangrui Meng updated SPARK-25322: -- Priority: Critical (was: Blocker) > ML, Graph 2.4 QA: API: Experimental, DeveloperApi,

[jira] [Commented] (SPARK-25319) Spark MLlib, GraphX 2.4 QA umbrella

2018-09-21 Thread Xiangrui Meng (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25319?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16624124#comment-16624124 ] Xiangrui Meng commented on SPARK-25319: --- [~WeichenXu123] Could you check the OPEN subtasks and see

[jira] [Assigned] (SPARK-25326) ML, Graph 2.4 QA: Programming guide update and migration guide

2018-09-21 Thread Xiangrui Meng (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25326?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiangrui Meng reassigned SPARK-25326: - Assignee: (was: Nick Pentreath) > ML, Graph 2.4 QA: Programming guide update and

[jira] [Assigned] (SPARK-25325) ML, Graph 2.4 QA: Update user guide for new features & APIs

2018-09-21 Thread Xiangrui Meng (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25325?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiangrui Meng reassigned SPARK-25325: - Assignee: (was: Nick Pentreath) > ML, Graph 2.4 QA: Update user guide for new

[jira] [Assigned] (SPARK-25327) Update MLlib, GraphX websites for 2.4

2018-09-21 Thread Xiangrui Meng (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25327?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiangrui Meng reassigned SPARK-25327: - Assignee: (was: Nick Pentreath) > Update MLlib, GraphX websites for 2.4 >

[jira] [Resolved] (SPARK-25327) Update MLlib, GraphX websites for 2.4

2018-09-21 Thread Xiangrui Meng (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25327?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiangrui Meng resolved SPARK-25327. --- Resolution: Won't Do > Update MLlib, GraphX websites for 2.4 >

[jira] [Assigned] (SPARK-25322) ML, Graph 2.4 QA: API: Experimental, DeveloperApi, final, sealed audit

2018-09-21 Thread Xiangrui Meng (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25322?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiangrui Meng reassigned SPARK-25322: - Assignee: (was: Nick Pentreath) > ML, Graph 2.4 QA: API: Experimental,

[jira] [Commented] (SPARK-25323) ML 2.4 QA: API: Python API coverage

2018-09-21 Thread Xiangrui Meng (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25323?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16624123#comment-16624123 ] Xiangrui Meng commented on SPARK-25323: --- [~WeichenXu123] anyone is working on this? I reduced the

[jira] [Updated] (SPARK-25323) ML 2.4 QA: API: Python API coverage

2018-09-21 Thread Xiangrui Meng (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25323?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiangrui Meng updated SPARK-25323: -- Priority: Critical (was: Blocker) > ML 2.4 QA: API: Python API coverage >

[jira] [Assigned] (SPARK-25323) ML 2.4 QA: API: Python API coverage

2018-09-21 Thread Xiangrui Meng (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25323?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiangrui Meng reassigned SPARK-25323: - Assignee: (was: Bryan Cutler) > ML 2.4 QA: API: Python API coverage >

[jira] [Resolved] (SPARK-25324) ML 2.4 QA: API: Java compatibility, docs

2018-09-21 Thread Xiangrui Meng (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25324?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiangrui Meng resolved SPARK-25324. --- Resolution: Done > ML 2.4 QA: API: Java compatibility, docs >

[jira] [Commented] (SPARK-25324) ML 2.4 QA: API: Java compatibility, docs

2018-09-21 Thread Xiangrui Meng (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25324?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16624122#comment-16624122 ] Xiangrui Meng commented on SPARK-25324: --- See discussion in

[jira] [Assigned] (SPARK-25321) ML, Graph 2.4 QA: API: New Scala APIs, docs

2018-09-21 Thread Xiangrui Meng (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25321?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiangrui Meng reassigned SPARK-25321: - Assignee: Weichen Xu (was: Yanbo Liang) > ML, Graph 2.4 QA: API: New Scala APIs, docs

[jira] [Assigned] (SPARK-25320) ML, Graph 2.4 QA: API: Binary incompatible changes

2018-09-21 Thread Xiangrui Meng (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25320?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiangrui Meng reassigned SPARK-25320: - Assignee: Weichen Xu (was: Bago Amirbekian) > ML, Graph 2.4 QA: API: Binary

[jira] [Commented] (SPARK-25320) ML, Graph 2.4 QA: API: Binary incompatible changes

2018-09-21 Thread Xiangrui Meng (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25320?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16624116#comment-16624116 ] Xiangrui Meng commented on SPARK-25320: --- We will keep "predict()" change because it only breaks

[jira] [Resolved] (SPARK-25320) ML, Graph 2.4 QA: API: Binary incompatible changes

2018-09-21 Thread Xiangrui Meng (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25320?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiangrui Meng resolved SPARK-25320. --- Resolution: Fixed > ML, Graph 2.4 QA: API: Binary incompatible changes >

[jira] [Commented] (SPARK-25320) ML, Graph 2.4 QA: API: Binary incompatible changes

2018-09-21 Thread Xiangrui Meng (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25320?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16624115#comment-16624115 ] Xiangrui Meng commented on SPARK-25320: --- We revert the tree Node incompatible change and one LDA

[jira] [Resolved] (SPARK-25321) ML, Graph 2.4 QA: API: New Scala APIs, docs

2018-09-21 Thread Xiangrui Meng (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25321?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiangrui Meng resolved SPARK-25321. --- Resolution: Fixed Issue resolved by pull request 22510

[jira] [Reopened] (SPARK-25321) ML, Graph 2.4 QA: API: New Scala APIs, docs

2018-09-21 Thread Xiangrui Meng (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25321?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiangrui Meng reopened SPARK-25321: --- > ML, Graph 2.4 QA: API: New Scala APIs, docs > --- > >

[jira] [Resolved] (SPARK-25321) ML, Graph 2.4 QA: API: New Scala APIs, docs

2018-09-21 Thread Xiangrui Meng (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25321?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiangrui Meng resolved SPARK-25321. --- Resolution: Fixed Fix Version/s: 2.4.0 Issue resolved by pull request 22492

[jira] [Comment Edited] (SPARK-25321) ML, Graph 2.4 QA: API: New Scala APIs, docs

2018-09-17 Thread Xiangrui Meng (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25321?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16618495#comment-16618495 ] Xiangrui Meng edited comment on SPARK-25321 at 9/18/18 5:21 AM:

[jira] [Commented] (SPARK-25321) ML, Graph 2.4 QA: API: New Scala APIs, docs

2018-09-17 Thread Xiangrui Meng (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25321?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16618495#comment-16618495 ] Xiangrui Meng commented on SPARK-25321: --- [~WeichenXu123] Could you check whether mleap is

[jira] [Commented] (SPARK-25378) ArrayData.toArray(StringType) assume UTF8String in 2.4

2018-09-12 Thread Xiangrui Meng (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25378?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16612399#comment-16612399 ] Xiangrui Meng commented on SPARK-25378: --- Comments from [~vomjom] at

[jira] [Updated] (SPARK-25378) ArrayData.toArray(StringType) assume UTF8String in 2.4

2018-09-10 Thread Xiangrui Meng (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25378?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiangrui Meng updated SPARK-25378: -- Summary: ArrayData.toArray(StringType) assume UTF8String in 2.4 (was: ArrayData.toArray

[jira] [Commented] (SPARK-25378) ArrayData.toArray assume UTF8String

2018-09-10 Thread Xiangrui Meng (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25378?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16609467#comment-16609467 ] Xiangrui Meng commented on SPARK-25378: --- I sent a PR to spark-tensorflow-connector at

[jira] [Commented] (SPARK-25378) ArrayData.toArray assume UTF8String

2018-09-08 Thread Xiangrui Meng (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25378?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16608171#comment-16608171 ] Xiangrui Meng commented on SPARK-25378: --- Btw, thanks for suggesting the right way to get strings

[jira] [Commented] (SPARK-25378) ArrayData.toArray assume UTF8String

2018-09-08 Thread Xiangrui Meng (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25378?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16608165#comment-16608165 ] Xiangrui Meng commented on SPARK-25378: --- This is a breaking change anyway. The error comes from

[jira] [Updated] (SPARK-25382) Remove ImageSchema.readImages in 3.0

2018-09-08 Thread Xiangrui Meng (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25382?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiangrui Meng updated SPARK-25382: -- Description: A follow-up task from SPARK-25345. We might need to support sampling

[jira] [Created] (SPARK-25383) Image data source supports sample pushdown

2018-09-08 Thread Xiangrui Meng (JIRA)
Xiangrui Meng created SPARK-25383: - Summary: Image data source supports sample pushdown Key: SPARK-25383 URL: https://issues.apache.org/jira/browse/SPARK-25383 Project: Spark Issue Type: New

[jira] [Updated] (SPARK-25345) Deprecate readImages APIs from ImageSchema

2018-09-08 Thread Xiangrui Meng (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25345?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiangrui Meng updated SPARK-25345: -- Description: After SPARK-22666, we can deprecate the public APIs in ImageSchema

[jira] [Created] (SPARK-25382) Remove ImageSchema.readImages in 3.0

2018-09-08 Thread Xiangrui Meng (JIRA)
Xiangrui Meng created SPARK-25382: - Summary: Remove ImageSchema.readImages in 3.0 Key: SPARK-25382 URL: https://issues.apache.org/jira/browse/SPARK-25382 Project: Spark Issue Type: Task

[jira] [Updated] (SPARK-25345) Deprecate readImages APIs from ImageSchema

2018-09-08 Thread Xiangrui Meng (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25345?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiangrui Meng updated SPARK-25345: -- Summary: Deprecate readImages APIs from ImageSchema (was: Deprecate public APIs from

[jira] [Assigned] (SPARK-25345) Deprecate public APIs from ImageSchema

2018-09-08 Thread Xiangrui Meng (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25345?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiangrui Meng reassigned SPARK-25345: - Assignee: Weichen Xu > Deprecate public APIs from ImageSchema >

[jira] [Resolved] (SPARK-25345) Deprecate public APIs from ImageSchema

2018-09-08 Thread Xiangrui Meng (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25345?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiangrui Meng resolved SPARK-25345. --- Resolution: Fixed Fix Version/s: 2.4.0 Issue resolved by pull request 22349

[jira] [Commented] (SPARK-25378) ArrayData.toArray assume UTF8String

2018-09-08 Thread Xiangrui Meng (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25378?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16608092#comment-16608092 ] Xiangrui Meng commented on SPARK-25378: --- Seems caused by SPARK-23875. cc: [~viirya] [~hvanhovell]

[jira] [Created] (SPARK-25378) ArrayData.toArray assume UTF8String

2018-09-08 Thread Xiangrui Meng (JIRA)
Xiangrui Meng created SPARK-25378: - Summary: ArrayData.toArray assume UTF8String Key: SPARK-25378 URL: https://issues.apache.org/jira/browse/SPARK-25378 Project: Spark Issue Type: Bug

[jira] [Updated] (SPARK-25376) Scenarios we should handle but missed in 2.4 for barrier execution mode

2018-09-08 Thread Xiangrui Meng (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25376?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiangrui Meng updated SPARK-25376: -- Summary: Scenarios we should handle but missed in 2.4 for barrier execution mode (was:

[jira] [Updated] (SPARK-25376) Scenarios we should handle but missed for barrier execution mode

2018-09-08 Thread Xiangrui Meng (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25376?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiangrui Meng updated SPARK-25376: -- Description: [~irashid] You mentioned that there are couple scenarios we should handle in

[jira] [Updated] (SPARK-25376) Scenarios we should handle but missed for barrier execution mode

2018-09-07 Thread Xiangrui Meng (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25376?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiangrui Meng updated SPARK-25376: -- Description: [~irashid] You mentioned that there are couple scenarios we should handle in

[jira] [Created] (SPARK-25376) Scenarios we should handle but missed for barrier execution mode

2018-09-07 Thread Xiangrui Meng (JIRA)
Xiangrui Meng created SPARK-25376: - Summary: Scenarios we should handle but missed for barrier execution mode Key: SPARK-25376 URL: https://issues.apache.org/jira/browse/SPARK-25376 Project: Spark

[jira] [Updated] (SPARK-25347) Document image data source in doc site

2018-09-05 Thread Xiangrui Meng (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25347?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiangrui Meng updated SPARK-25347: -- Summary: Document image data source in doc site (was: Document image data sources in doc

[jira] [Updated] (SPARK-25345) Deprecate public APIs from ImageSchema

2018-09-05 Thread Xiangrui Meng (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25345?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiangrui Meng updated SPARK-25345: -- Description: After SPARK-22328, we can deprecate the public APIs in ImageSchema

[jira] [Created] (SPARK-25349) Support sample pushdown in Data Source V2

2018-09-05 Thread Xiangrui Meng (JIRA)
Xiangrui Meng created SPARK-25349: - Summary: Support sample pushdown in Data Source V2 Key: SPARK-25349 URL: https://issues.apache.org/jira/browse/SPARK-25349 Project: Spark Issue Type:

[jira] [Created] (SPARK-25348) Data source for binary files

2018-09-05 Thread Xiangrui Meng (JIRA)
Xiangrui Meng created SPARK-25348: - Summary: Data source for binary files Key: SPARK-25348 URL: https://issues.apache.org/jira/browse/SPARK-25348 Project: Spark Issue Type: Story

[jira] [Created] (SPARK-25347) Document image data sources in doc site

2018-09-05 Thread Xiangrui Meng (JIRA)
Xiangrui Meng created SPARK-25347: - Summary: Document image data sources in doc site Key: SPARK-25347 URL: https://issues.apache.org/jira/browse/SPARK-25347 Project: Spark Issue Type: Story

[jira] [Updated] (SPARK-25346) Document Spark builtin data sources

2018-09-05 Thread Xiangrui Meng (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25346?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiangrui Meng updated SPARK-25346: -- Summary: Document Spark builtin data sources (was: Document Spark built-in data sources) >

[jira] [Updated] (SPARK-25346) Document Spark built-in data sources

2018-09-05 Thread Xiangrui Meng (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25346?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiangrui Meng updated SPARK-25346: -- Summary: Document Spark built-in data sources (was: Document Spark buit-in data sources) >

[jira] [Created] (SPARK-25346) Document Spark buit-in data sources

2018-09-05 Thread Xiangrui Meng (JIRA)
Xiangrui Meng created SPARK-25346: - Summary: Document Spark buit-in data sources Key: SPARK-25346 URL: https://issues.apache.org/jira/browse/SPARK-25346 Project: Spark Issue Type: Story

[jira] [Created] (SPARK-25345) Deprecate public APIs from ImageSchema

2018-09-05 Thread Xiangrui Meng (JIRA)
Xiangrui Meng created SPARK-25345: - Summary: Deprecate public APIs from ImageSchema Key: SPARK-25345 URL: https://issues.apache.org/jira/browse/SPARK-25345 Project: Spark Issue Type: Story

[jira] [Resolved] (SPARK-22666) Spark datasource for image format

2018-09-05 Thread Xiangrui Meng (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22666?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiangrui Meng resolved SPARK-22666. --- Resolution: Fixed Fix Version/s: 2.4.0 Issue resolved by pull request 22328

[jira] [Resolved] (SPARK-25248) Audit barrier APIs for Spark 2.4

2018-09-04 Thread Xiangrui Meng (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25248?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiangrui Meng resolved SPARK-25248. --- Resolution: Fixed Fix Version/s: 2.4.0 Issue resolved by pull request 22240

[jira] [Assigned] (SPARK-22666) Spark datasource for image format

2018-09-04 Thread Xiangrui Meng (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22666?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiangrui Meng reassigned SPARK-22666: - Assignee: Weichen Xu > Spark datasource for image format >

[jira] [Resolved] (SPARK-25266) Fix memory leak in Barrier Execution Mode

2018-08-29 Thread Xiangrui Meng (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25266?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiangrui Meng resolved SPARK-25266. --- Resolution: Fixed Fix Version/s: 2.4.0 Issue resolved by pull request 22258

[jira] [Created] (SPARK-25248) Audit barrier APIs for Spark 2.4

2018-08-26 Thread Xiangrui Meng (JIRA)
Xiangrui Meng created SPARK-25248: - Summary: Audit barrier APIs for Spark 2.4 Key: SPARK-25248 URL: https://issues.apache.org/jira/browse/SPARK-25248 Project: Spark Issue Type: Story

[jira] [Created] (SPARK-25247) Make RDDBarrier configurable

2018-08-26 Thread Xiangrui Meng (JIRA)
Xiangrui Meng created SPARK-25247: - Summary: Make RDDBarrier configurable Key: SPARK-25247 URL: https://issues.apache.org/jira/browse/SPARK-25247 Project: Spark Issue Type: Story

[jira] [Resolved] (SPARK-25234) SparkR:::parallelize doesn't handle integer overflow properly

2018-08-24 Thread Xiangrui Meng (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25234?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiangrui Meng resolved SPARK-25234. --- Resolution: Fixed Fix Version/s: 2.3.2 2.4.0 Issue resolved by

[jira] [Assigned] (SPARK-25234) SparkR:::parallelize doesn't handle integer overflow properly

2018-08-24 Thread Xiangrui Meng (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25234?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiangrui Meng reassigned SPARK-25234: - Assignee: Xiangrui Meng > SparkR:::parallelize doesn't handle integer overflow

[jira] [Updated] (SPARK-25234) SparkR:::parallelize doesn't handle integer overflow properly

2018-08-24 Thread Xiangrui Meng (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25234?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiangrui Meng updated SPARK-25234: -- Description: parallelize uses integer multiplication, which cannot handle size over ~47000.

[jira] [Created] (SPARK-25234) SparkR:::parallelize doesn't handle integer overflow properly

2018-08-24 Thread Xiangrui Meng (JIRA)
Xiangrui Meng created SPARK-25234: - Summary: SparkR:::parallelize doesn't handle integer overflow properly Key: SPARK-25234 URL: https://issues.apache.org/jira/browse/SPARK-25234 Project: Spark

[jira] [Resolved] (SPARK-25095) Python support for BarrierTaskContext

2018-08-21 Thread Xiangrui Meng (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25095?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiangrui Meng resolved SPARK-25095. --- Resolution: Fixed Fix Version/s: 2.4.0 Issue resolved by pull request 22085

[jira] [Assigned] (SPARK-25095) Python support for BarrierTaskContext

2018-08-21 Thread Xiangrui Meng (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25095?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiangrui Meng reassigned SPARK-25095: - Assignee: Jiang Xingbo > Python support for BarrierTaskContext >

[jira] [Assigned] (SPARK-25161) Fix several bugs in failure handling of barrier execution mode

2018-08-21 Thread Xiangrui Meng (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25161?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiangrui Meng reassigned SPARK-25161: - Assignee: Jiang Xingbo > Fix several bugs in failure handling of barrier execution

[jira] [Resolved] (SPARK-25161) Fix several bugs in failure handling of barrier execution mode

2018-08-21 Thread Xiangrui Meng (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25161?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiangrui Meng resolved SPARK-25161. --- Resolution: Fixed Fix Version/s: 2.4.0 Issue resolved by pull request 22158

[jira] [Resolved] (SPARK-24819) Fail fast when no enough slots to launch the barrier stage on job submitted

2018-08-15 Thread Xiangrui Meng (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24819?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiangrui Meng resolved SPARK-24819. --- Resolution: Fixed Fix Version/s: 2.4.0 Issue resolved by pull request 22001

[jira] [Assigned] (SPARK-24819) Fail fast when no enough slots to launch the barrier stage on job submitted

2018-08-15 Thread Xiangrui Meng (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24819?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiangrui Meng reassigned SPARK-24819: - Assignee: Jiang Xingbo > Fail fast when no enough slots to launch the barrier stage on

[jira] [Resolved] (SPARK-25045) Make `RDDBarrier.mapParititions` similar to `RDD.mapPartitions`

2018-08-07 Thread Xiangrui Meng (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25045?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiangrui Meng resolved SPARK-25045. --- Resolution: Fixed Fix Version/s: 2.4.0 Issue resolved by pull request 22026

[jira] [Assigned] (SPARK-25045) Make `RDDBarrier.mapParititions` similar to `RDD.mapPartitions`

2018-08-07 Thread Xiangrui Meng (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25045?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiangrui Meng reassigned SPARK-25045: - Assignee: Jiang Xingbo > Make `RDDBarrier.mapParititions` similar to

[jira] [Updated] (SPARK-25030) SparkSubmit.doSubmit will not return result if the mainClass submitted creates a Timer()

2018-08-06 Thread Xiangrui Meng (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25030?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiangrui Meng updated SPARK-25030: -- Summary: SparkSubmit.doSubmit will not return result if the mainClass submitted creates a

[jira] [Commented] (SPARK-25030) SparkSubmit will not return result if the mainClass submitted creates a Timer()

2018-08-06 Thread Xiangrui Meng (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25030?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16570951#comment-16570951 ] Xiangrui Meng commented on SPARK-25030: --- [~jiangxb1987] Could you create a PR to demonstrate the

[jira] [Assigned] (SPARK-24954) Fail fast on job submit if run a barrier stage with dynamic resource allocation enabled

2018-08-03 Thread Xiangrui Meng (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24954?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiangrui Meng reassigned SPARK-24954: - Assignee: Jiang Xingbo > Fail fast on job submit if run a barrier stage with dynamic

[jira] [Resolved] (SPARK-24954) Fail fast on job submit if run a barrier stage with dynamic resource allocation enabled

2018-08-03 Thread Xiangrui Meng (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24954?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiangrui Meng resolved SPARK-24954. --- Resolution: Fixed Fix Version/s: 2.4.0 Issue resolved by pull request 21915

[jira] [Resolved] (SPARK-24821) Fail fast when submitted job compute on a subset of all the partitions for a barrier stage

2018-08-02 Thread Xiangrui Meng (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24821?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiangrui Meng resolved SPARK-24821. --- Resolution: Fixed Fix Version/s: 2.4.0 Issue resolved by pull request 21927

[jira] [Assigned] (SPARK-24821) Fail fast when submitted job compute on a subset of all the partitions for a barrier stage

2018-08-02 Thread Xiangrui Meng (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24821?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiangrui Meng reassigned SPARK-24821: - Assignee: Jiang Xingbo > Fail fast when submitted job compute on a subset of all the

[jira] [Assigned] (SPARK-24820) Fail fast when submitted job contains PartitionPruningRDD in a barrier stage

2018-08-02 Thread Xiangrui Meng (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24820?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiangrui Meng reassigned SPARK-24820: - Assignee: Jiang Xingbo > Fail fast when submitted job contains PartitionPruningRDD in

[jira] [Resolved] (SPARK-24820) Fail fast when submitted job contains PartitionPruningRDD in a barrier stage

2018-08-02 Thread Xiangrui Meng (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24820?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiangrui Meng resolved SPARK-24820. --- Resolution: Fixed Fix Version/s: 2.4.0 Issue resolved by pull request 21927

[jira] [Commented] (SPARK-24719) ClusteringEvaluator supports integer type labels

2018-08-02 Thread Xiangrui Meng (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24719?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16566425#comment-16566425 ] Xiangrui Meng commented on SPARK-24719: --- [~mgaido] Sorry it was my bad! The error happened when a

[jira] [Resolved] (SPARK-24719) ClusteringEvaluator supports integer type labels

2018-08-02 Thread Xiangrui Meng (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24719?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiangrui Meng resolved SPARK-24719. --- Resolution: Not A Problem > ClusteringEvaluator supports integer type labels >

[jira] [Resolved] (SPARK-24557) ClusteringEvaluator support array input

2018-08-02 Thread Xiangrui Meng (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24557?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiangrui Meng resolved SPARK-24557. --- Resolution: Fixed Fix Version/s: 2.4.0 Issue resolved by pull request 21563

[jira] [Assigned] (SPARK-24557) ClusteringEvaluator support array input

2018-08-02 Thread Xiangrui Meng (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24557?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiangrui Meng reassigned SPARK-24557: - Assignee: zhengruifeng > ClusteringEvaluator support array input >

[jira] [Resolved] (SPARK-24726) Discuss necessary info and access in barrier mode + Standalone

2018-07-19 Thread Xiangrui Meng (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24726?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiangrui Meng resolved SPARK-24726. --- Resolution: Resolved Target Version/s: 2.4.0 (was: 3.0.0) > Discuss necessary

[jira] [Commented] (SPARK-24726) Discuss necessary info and access in barrier mode + Standalone

2018-07-19 Thread Xiangrui Meng (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24726?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16550050#comment-16550050 ] Xiangrui Meng commented on SPARK-24726: --- I'm closing this ticket as resolved since with

[jira] [Assigned] (SPARK-24726) Discuss necessary info and access in barrier mode + Standalone

2018-07-19 Thread Xiangrui Meng (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24726?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiangrui Meng reassigned SPARK-24726: - Assignee: Xiangrui Meng > Discuss necessary info and access in barrier mode +

[jira] [Updated] (SPARK-24723) Discuss necessary info and access in barrier mode + YARN

2018-07-19 Thread Xiangrui Meng (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24723?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiangrui Meng updated SPARK-24723: -- Description: In barrier mode, to run hybrid distributed DL training jobs, we need to provide

[jira] [Commented] (SPARK-24724) Discuss necessary info and access in barrier mode + Kubernetes

2018-07-19 Thread Xiangrui Meng (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24724?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16550049#comment-16550049 ] Xiangrui Meng commented on SPARK-24724: --- [~liyinan926] Any updates? > Discuss necessary info and

[jira] [Assigned] (SPARK-24723) Discuss necessary info and access in barrier mode + YARN

2018-07-19 Thread Xiangrui Meng (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24723?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiangrui Meng reassigned SPARK-24723: - Assignee: Saisai Shao > Discuss necessary info and access in barrier mode + YARN >

[jira] [Commented] (SPARK-24723) Discuss necessary info and access in barrier mode + YARN

2018-07-19 Thread Xiangrui Meng (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24723?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16550048#comment-16550048 ] Xiangrui Meng commented on SPARK-24723: --- [~jerryshao] Does YARN have the feature that will by

[jira] [Commented] (SPARK-24615) Accelerator-aware task scheduling for Spark

2018-07-19 Thread Xiangrui Meng (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24615?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16550023#comment-16550023 ] Xiangrui Meng commented on SPARK-24615: --- [~tgraves] Could you help link some past requests on

[jira] [Updated] (SPARK-24374) SPIP: Support Barrier Execution Mode in Apache Spark

2018-07-19 Thread Xiangrui Meng (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24374?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiangrui Meng updated SPARK-24374: -- Summary: SPIP: Support Barrier Execution Mode in Apache Spark (was: SPIP: Support Barrier

[jira] [Updated] (SPARK-24615) Accelerator-aware task scheduling for Spark

2018-07-16 Thread Xiangrui Meng (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24615?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiangrui Meng updated SPARK-24615: -- Summary: Accelerator-aware task scheduling for Spark (was: Accelerator aware task scheduling

[jira] [Assigned] (SPARK-24615) Accelerator aware task scheduling for Spark

2018-07-12 Thread Xiangrui Meng (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24615?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiangrui Meng reassigned SPARK-24615: - Assignee: Saisai Shao > Accelerator aware task scheduling for Spark >

[jira] [Updated] (SPARK-24747) Make spark.ml.util.Instrumentation class more flexible

2018-07-05 Thread Xiangrui Meng (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24747?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiangrui Meng updated SPARK-24747: -- Shepherd: Xiangrui Meng > Make spark.ml.util.Instrumentation class more flexible >

[jira] [Assigned] (SPARK-24747) Make spark.ml.util.Instrumentation class more flexible

2018-07-05 Thread Xiangrui Meng (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24747?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiangrui Meng reassigned SPARK-24747: - Assignee: Bago Amirbekian > Make spark.ml.util.Instrumentation class more flexible >

[jira] [Commented] (SPARK-24579) SPIP: Standardize Optimized Data Exchange between Spark and DL/AI frameworks

2018-07-03 Thread Xiangrui Meng (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24579?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16531737#comment-16531737 ] Xiangrui Meng commented on SPARK-24579: --- [~sethah] [~kiszk] [~rxin] Please request comment

<    1   2   3   4   5   6   7   8   9   10   >