[jira] [Commented] (SPARK-15694) Implement ScriptTransformation in sql/core

2016-07-15 Thread Reynold Xin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15694?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15380522#comment-15380522 ] Reynold Xin commented on SPARK-15694: - [~tejasp] any update on this? > Implement

[jira] [Commented] (SPARK-15799) Release SparkR on CRAN

2016-07-15 Thread Shivaram Venkataraman (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15799?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15380515#comment-15380515 ] Shivaram Venkataraman commented on SPARK-15799: --- I think the higher level packages I had in

[jira] [Commented] (SPARK-16581) Making JVM backend calling functions public

2016-07-15 Thread Shivaram Venkataraman (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16581?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15380513#comment-15380513 ] Shivaram Venkataraman commented on SPARK-16581: --- Thanks [~sunrui]. This is not targeted for

[jira] [Commented] (SPARK-16519) Handle SparkR RDD generics that create warnings in R CMD check

2016-07-15 Thread Shivaram Venkataraman (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16519?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15380512#comment-15380512 ] Shivaram Venkataraman commented on SPARK-16519: --- Thanks - the only things to keep in mind

[jira] [Commented] (SPARK-16508) Fix documentation warnings found by R CMD check

2016-07-15 Thread Shivaram Venkataraman (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16508?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15380509#comment-15380509 ] Shivaram Venkataraman commented on SPARK-16508: --- Sorry the JIRA isn't very clear. I divided

[jira] [Commented] (SPARK-16584) Move regexp unit tests to RegexpExpressionsSuite

2016-07-15 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16584?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15380505#comment-15380505 ] Apache Spark commented on SPARK-16584: -- User 'rxin' has created a pull request for this issue:

[jira] [Assigned] (SPARK-16584) Move regexp unit tests to RegexpExpressionsSuite

2016-07-15 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16584?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-16584: Assignee: Reynold Xin (was: Apache Spark) > Move regexp unit tests to

[jira] [Assigned] (SPARK-16584) Move regexp unit tests to RegexpExpressionsSuite

2016-07-15 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16584?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-16584: Assignee: Apache Spark (was: Reynold Xin) > Move regexp unit tests to

[jira] [Created] (SPARK-16584) Move regexp unit tests to RegexpExpressionsSuite

2016-07-15 Thread Reynold Xin (JIRA)
Reynold Xin created SPARK-16584: --- Summary: Move regexp unit tests to RegexpExpressionsSuite Key: SPARK-16584 URL: https://issues.apache.org/jira/browse/SPARK-16584 Project: Spark Issue Type:

[jira] [Commented] (SPARK-16519) Handle SparkR RDD generics that create warnings in R CMD check

2016-07-15 Thread Felix Cheung (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16519?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15380460#comment-15380460 ] Felix Cheung commented on SPARK-16519: -- I can take this > Handle SparkR RDD generics that

[jira] [Commented] (SPARK-16508) Fix documentation warnings found by R CMD check

2016-07-15 Thread Felix Cheung (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16508?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15380456#comment-15380456 ] Felix Cheung commented on SPARK-16508: -- Sure, from checking

[jira] [Issue Comment Deleted] (SPARK-16508) Fix documentation warnings found by R CMD check

2016-07-15 Thread Felix Cheung (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16508?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Felix Cheung updated SPARK-16508: - Comment: was deleted (was: Sure but I thought your PR is already handling most of this, and

[jira] [Commented] (SPARK-16579) Add a spark install function

2016-07-15 Thread Felix Cheung (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16579?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15380453#comment-15380453 ] Felix Cheung commented on SPARK-16579: -- we should download from an official apache release mirror.

[jira] [Comment Edited] (SPARK-16508) Fix documentation warnings found by R CMD check

2016-07-15 Thread Felix Cheung (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16508?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15380436#comment-15380436 ] Felix Cheung edited comment on SPARK-16508 at 7/16/16 3:05 AM: --- Sure but I

[jira] [Commented] (SPARK-16508) Fix documentation warnings found by R CMD check

2016-07-15 Thread Felix Cheung (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16508?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15380436#comment-15380436 ] Felix Cheung commented on SPARK-16508: -- Sure but I thought your PR is already handling most of this,

[jira] [Assigned] (SPARK-16447) LDA wrapper in SparkR

2016-07-15 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16447?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-16447: Assignee: Apache Spark (was: Xusen Yin) > LDA wrapper in SparkR > -

[jira] [Assigned] (SPARK-16447) LDA wrapper in SparkR

2016-07-15 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16447?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-16447: Assignee: Xusen Yin (was: Apache Spark) > LDA wrapper in SparkR > -

[jira] [Commented] (SPARK-16447) LDA wrapper in SparkR

2016-07-15 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16447?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15380420#comment-15380420 ] Apache Spark commented on SPARK-16447: -- User 'yinxusen' has created a pull request for this issue:

[jira] [Assigned] (SPARK-16583) Improve Partition Pruning in InMemoryTableScanExec

2016-07-15 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16583?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-16583: Assignee: Apache Spark > Improve Partition Pruning in InMemoryTableScanExec >

[jira] [Commented] (SPARK-16583) Improve Partition Pruning in InMemoryTableScanExec

2016-07-15 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16583?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15380405#comment-15380405 ] Apache Spark commented on SPARK-16583: -- User 'gatorsmile' has created a pull request for this issue:

[jira] [Assigned] (SPARK-16583) Improve Partition Pruning in InMemoryTableScanExec

2016-07-15 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16583?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-16583: Assignee: (was: Apache Spark) > Improve Partition Pruning in InMemoryTableScanExec >

[jira] [Created] (SPARK-16583) Improve Partition Pruning in InMemoryTableScanExec

2016-07-15 Thread Xiao Li (JIRA)
Xiao Li created SPARK-16583: --- Summary: Improve Partition Pruning in InMemoryTableScanExec Key: SPARK-16583 URL: https://issues.apache.org/jira/browse/SPARK-16583 Project: Spark Issue Type:

[jira] [Commented] (SPARK-16582) Explicitly define isNull = false for non-nullable expressions

2016-07-15 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16582?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15380349#comment-15380349 ] Apache Spark commented on SPARK-16582: -- User 'sameeragarwal' has created a pull request for this

[jira] [Assigned] (SPARK-16582) Explicitly define isNull = false for non-nullable expressions

2016-07-15 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16582?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-16582: Assignee: (was: Apache Spark) > Explicitly define isNull = false for non-nullable

[jira] [Assigned] (SPARK-16582) Explicitly define isNull = false for non-nullable expressions

2016-07-15 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16582?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-16582: Assignee: Apache Spark > Explicitly define isNull = false for non-nullable expressions >

[jira] [Created] (SPARK-16582) Explicitly define isNull = false for non-nullable expressions

2016-07-15 Thread Sameer Agarwal (JIRA)
Sameer Agarwal created SPARK-16582: -- Summary: Explicitly define isNull = false for non-nullable expressions Key: SPARK-16582 URL: https://issues.apache.org/jira/browse/SPARK-16582 Project: Spark

[jira] [Commented] (SPARK-16334) [SQL] SQL query on parquet table java.lang.ArrayIndexOutOfBoundsException

2016-07-15 Thread Sameer Agarwal (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16334?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15380330#comment-15380330 ] Sameer Agarwal commented on SPARK-16334: cc [~vivanov] too > [SQL] SQL query on parquet table

[jira] [Commented] (SPARK-16581) Making JVM backend calling functions public

2016-07-15 Thread Sun Rui (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16581?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15380307#comment-15380307 ] Sun Rui commented on SPARK-16581: - I can work on this if this is not so urgent:) > Making JVM backend

[jira] [Comment Edited] (SPARK-16574) Distribute computing to each node based on certain hints

2016-07-15 Thread Norman He (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16574?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15380046#comment-15380046 ] Norman He edited comment on SPARK-16574 at 7/15/16 11:30 PM: - worker rdd is

[jira] [Commented] (SPARK-15393) Writing empty Dataframes doesn't save any _metadata files

2016-07-15 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15393?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15380292#comment-15380292 ] Hyukjin Kwon commented on SPARK-15393: -- Yes please, could you please close this one? And actually,

[jira] [Commented] (SPARK-16574) Distribute computing to each node based on certain hints

2016-07-15 Thread Norman He (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16574?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15380279#comment-15380279 ] Norman He commented on SPARK-16574: --- exactly like you said, right now there is no guarentee this will

[jira] [Commented] (SPARK-16334) [SQL] SQL query on parquet table java.lang.ArrayIndexOutOfBoundsException

2016-07-15 Thread Egor Pahomov (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16334?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15380249#comment-15380249 ] Egor Pahomov commented on SPARK-16334: -- Sure, would test on Monday. > [SQL] SQL query on parquet

[jira] [Commented] (SPARK-15799) Release SparkR on CRAN

2016-07-15 Thread Felix Cheung (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15799?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15380246#comment-15380246 ] Felix Cheung commented on SPARK-15799: -- re: higher level R packages that depends on SparkR - I think

[jira] [Closed] (SPARK-14048) Aggregation operations on structs fail when the structs have fields with special characters

2016-07-15 Thread Reynold Xin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-14048?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Reynold Xin closed SPARK-14048. --- Resolution: Not A Problem Target Version/s: (was: 2.0.0) Closing this as it is a

[jira] [Resolved] (SPARK-13071) Coalescing HadoopRDD overwrites existing input metrics

2016-07-15 Thread Reynold Xin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-13071?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Reynold Xin resolved SPARK-13071. - Resolution: Fixed Fix Version/s: 2.0.0 > Coalescing HadoopRDD overwrites existing input

[jira] [Closed] (SPARK-15434) improve EmbedSerializerInFilter rule

2016-07-15 Thread Reynold Xin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15434?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Reynold Xin closed SPARK-15434. --- Resolution: Won't Fix > improve EmbedSerializerInFilter rule >

[jira] [Updated] (SPARK-15944) Make spark.ml package backward compatible with spark.mllib vectors

2016-07-15 Thread Reynold Xin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15944?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Reynold Xin updated SPARK-15944: Target Version/s: 2.1.0 (was: 2.0.0) > Make spark.ml package backward compatible with spark.mllib

[jira] [Updated] (SPARK-15703) Spark UI doesn't show all tasks as completed when it should

2016-07-15 Thread Reynold Xin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15703?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Reynold Xin updated SPARK-15703: Target Version/s: 2.0.1 (was: 2.0.0) > Spark UI doesn't show all tasks as completed when it

[jira] [Commented] (SPARK-15393) Writing empty Dataframes doesn't save any _metadata files

2016-07-15 Thread Reynold Xin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15393?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15380202#comment-15380202 ] Reynold Xin commented on SPARK-15393: - [~hyukjin.kwon] should we close this ticket and SPARK-10216

[jira] [Updated] (SPARK-16032) Audit semantics of various insertion operations related to partitioned tables

2016-07-15 Thread Reynold Xin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16032?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Reynold Xin updated SPARK-16032: Target Version/s: 2.1.0 (was: 2.0.0) > Audit semantics of various insertion operations related to

[jira] [Closed] (SPARK-15340) Limit the size of the map used to cache JobConfs to void OOM

2016-07-15 Thread Reynold Xin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15340?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Reynold Xin closed SPARK-15340. --- Resolution: Not A Problem [~DoingDone9] I'm gong to mark this as not a problem for now. Please

[jira] [Updated] (SPARK-16295) Extract SQL programming guide example snippets from source files instead of hard code them

2016-07-15 Thread Reynold Xin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16295?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Reynold Xin updated SPARK-16295: Target Version/s: 2.0.1 (was: 2.0.0) > Extract SQL programming guide example snippets from source

[jira] [Updated] (SPARK-16380) Update SQL examples and programming guide for Python language binding

2016-07-15 Thread Reynold Xin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16380?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Reynold Xin updated SPARK-16380: Target Version/s: 2.0.1 (was: 2.0.0) > Update SQL examples and programming guide for Python

[jira] [Closed] (SPARK-15252) add accumulator wrapper to have more control of it

2016-07-15 Thread Reynold Xin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15252?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Reynold Xin closed SPARK-15252. --- Resolution: Won't Fix > add accumulator wrapper to have more control of it >

[jira] [Resolved] (SPARK-12420) Have a built-in CSV data source implementation

2016-07-15 Thread Reynold Xin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12420?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Reynold Xin resolved SPARK-12420. - Resolution: Fixed Fix Version/s: 2.0.0 > Have a built-in CSV data source implementation >

[jira] [Updated] (SPARK-14385) Use FunctionIdentifier in FunctionRegistry/SessionCatalog

2016-07-15 Thread Reynold Xin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-14385?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Reynold Xin updated SPARK-14385: Target Version/s: (was: 2.0.0) > Use FunctionIdentifier in FunctionRegistry/SessionCatalog >

[jira] [Commented] (SPARK-16334) [SQL] SQL query on parquet table java.lang.ArrayIndexOutOfBoundsException

2016-07-15 Thread Reynold Xin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16334?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15380187#comment-15380187 ] Reynold Xin commented on SPARK-16334: - [~epakhomov] can you try the patch and see if it fixes your

[jira] [Assigned] (SPARK-16580) [WARN] class Accumulator in package spark is deprecated: use AccumulatorV2

2016-07-15 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16580?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-16580: Assignee: Apache Spark > [WARN] class Accumulator in package spark is deprecated: use

[jira] [Commented] (SPARK-16580) [WARN] class Accumulator in package spark is deprecated: use AccumulatorV2

2016-07-15 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16580?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15380185#comment-15380185 ] Apache Spark commented on SPARK-16580: -- User 'keypointt' has created a pull request for this issue:

[jira] [Assigned] (SPARK-16580) [WARN] class Accumulator in package spark is deprecated: use AccumulatorV2

2016-07-15 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16580?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-16580: Assignee: (was: Apache Spark) > [WARN] class Accumulator in package spark is

[jira] [Commented] (SPARK-16574) Distribute computing to each node based on certain hints

2016-07-15 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16574?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15380176#comment-15380176 ] Sean Owen commented on SPARK-16574: --- Sure, let's say you have 10 machines with 2 GPUs. Then you want to

[jira] [Commented] (SPARK-16581) Making JVM backend calling functions public

2016-07-15 Thread Shivaram Venkataraman (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16581?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15380162#comment-15380162 ] Shivaram Venkataraman commented on SPARK-16581: --- [~sunrui] Would you have time to work on

[jira] [Assigned] (SPARK-16334) [SQL] SQL query on parquet table java.lang.ArrayIndexOutOfBoundsException

2016-07-15 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16334?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-16334: Assignee: Apache Spark > [SQL] SQL query on parquet table

[jira] [Commented] (SPARK-16580) [WARN] class Accumulator in package spark is deprecated: use AccumulatorV2

2016-07-15 Thread Xin Ren (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16580?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15380169#comment-15380169 ] Xin Ren commented on SPARK-16580: - You are right this one is hard, it's kindof all over the place. I'd

[jira] [Commented] (SPARK-16334) [SQL] SQL query on parquet table java.lang.ArrayIndexOutOfBoundsException

2016-07-15 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16334?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15380170#comment-15380170 ] Apache Spark commented on SPARK-16334: -- User 'sameeragarwal' has created a pull request for this

[jira] [Assigned] (SPARK-16334) [SQL] SQL query on parquet table java.lang.ArrayIndexOutOfBoundsException

2016-07-15 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16334?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-16334: Assignee: (was: Apache Spark) > [SQL] SQL query on parquet table

[jira] [Commented] (SPARK-16581) Making JVM backend calling functions public

2016-07-15 Thread Shivaram Venkataraman (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16581?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15380164#comment-15380164 ] Shivaram Venkataraman commented on SPARK-16581: --- And we can probably use Apache Toree or

[jira] [Commented] (SPARK-16580) [WARN] class Accumulator in package spark is deprecated: use AccumulatorV2

2016-07-15 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16580?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15380155#comment-15380155 ] Sean Owen commented on SPARK-16580: --- Yes, there's much more than this. The ones that occur in tests are

[jira] [Created] (SPARK-16581) Making JVM backend calling functions public

2016-07-15 Thread Shivaram Venkataraman (JIRA)
Shivaram Venkataraman created SPARK-16581: - Summary: Making JVM backend calling functions public Key: SPARK-16581 URL: https://issues.apache.org/jira/browse/SPARK-16581 Project: Spark

[jira] [Created] (SPARK-16580) [WARN] class Accumulator in package spark is deprecated: use AccumulatorV2

2016-07-15 Thread Xin Ren (JIRA)
Xin Ren created SPARK-16580: --- Summary: [WARN] class Accumulator in package spark is deprecated: use AccumulatorV2 Key: SPARK-16580 URL: https://issues.apache.org/jira/browse/SPARK-16580 Project: Spark

[jira] [Reopened] (SPARK-16230) Executors self-killing after being assigned tasks while still in init

2016-07-15 Thread Shixiong Zhu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16230?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu reopened SPARK-16230: -- > Executors self-killing after being assigned tasks while still in init >

[jira] [Updated] (SPARK-16230) Executors self-killing after being assigned tasks while still in init

2016-07-15 Thread Shixiong Zhu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16230?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu updated SPARK-16230: - Assignee: Tejas Patil Fix Version/s: 2.1.0 2.0.1 > Executors

[jira] [Resolved] (SPARK-16230) Executors self-killing after being assigned tasks while still in init

2016-07-15 Thread Shixiong Zhu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16230?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu resolved SPARK-16230. -- Resolution: Fixed > Executors self-killing after being assigned tasks while still in init >

[jira] [Commented] (SPARK-16508) Fix documentation warnings found by R CMD check

2016-07-15 Thread Shivaram Venkataraman (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16508?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15380142#comment-15380142 ] Shivaram Venkataraman commented on SPARK-16508: --- [~dongjoon] [~felixcheung] Would one of

[jira] [Commented] (SPARK-16579) Add a spark install function

2016-07-15 Thread Shivaram Venkataraman (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16579?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15380139#comment-15380139 ] Shivaram Venkataraman commented on SPARK-16579: --- One thing to note is that I think the

[jira] [Commented] (SPARK-16334) [SQL] SQL query on parquet table java.lang.ArrayIndexOutOfBoundsException

2016-07-15 Thread Sameer Agarwal (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16334?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15380137#comment-15380137 ] Sameer Agarwal commented on SPARK-16334: While I've not been able to reproduce this bug, looking

[jira] [Created] (SPARK-16579) Add a spark install function

2016-07-15 Thread Shivaram Venkataraman (JIRA)
Shivaram Venkataraman created SPARK-16579: - Summary: Add a spark install function Key: SPARK-16579 URL: https://issues.apache.org/jira/browse/SPARK-16579 Project: Spark Issue Type:

[jira] [Commented] (SPARK-16579) Add a spark install function

2016-07-15 Thread Shivaram Venkataraman (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16579?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15380138#comment-15380138 ] Shivaram Venkataraman commented on SPARK-16579: --- cc [~junyangq] > Add a spark install

[jira] [Commented] (SPARK-16522) [MESOS] Spark application throws exception on exit

2016-07-15 Thread Michael Gummelt (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16522?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15380130#comment-15380130 ] Michael Gummelt commented on SPARK-16522: - This shouldn't affect functionality > [MESOS] Spark

[jira] [Created] (SPARK-16578) Configurable hostname for RBackend

2016-07-15 Thread Shivaram Venkataraman (JIRA)
Shivaram Venkataraman created SPARK-16578: - Summary: Configurable hostname for RBackend Key: SPARK-16578 URL: https://issues.apache.org/jira/browse/SPARK-16578 Project: Spark Issue

[jira] [Commented] (SPARK-16578) Configurable hostname for RBackend

2016-07-15 Thread Shivaram Venkataraman (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16578?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15380116#comment-15380116 ] Shivaram Venkataraman commented on SPARK-16578: --- cc [~junyangq] [~falaki] > Configurable

[jira] [Created] (SPARK-16577) Add check-cran script to Jenkins

2016-07-15 Thread Shivaram Venkataraman (JIRA)
Shivaram Venkataraman created SPARK-16577: - Summary: Add check-cran script to Jenkins Key: SPARK-16577 URL: https://issues.apache.org/jira/browse/SPARK-16577 Project: Spark Issue

[jira] [Commented] (SPARK-16576) Move plan SQL generation code from SQLBuilder into logical operators

2016-07-15 Thread Dongjoon Hyun (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16576?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15380098#comment-15380098 ] Dongjoon Hyun commented on SPARK-16576: --- Sure. I will add a broadcast hint in `[SPARK-16475][SQL]

[jira] [Commented] (SPARK-16576) Move plan SQL generation code from SQLBuilder into logical operators

2016-07-15 Thread Reynold Xin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16576?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15380094#comment-15380094 ] Reynold Xin commented on SPARK-16576: - cc [~lian cheng] - who wrote the original code for this. >

[jira] [Commented] (SPARK-16576) Move plan SQL generation code from SQLBuilder into logical operators

2016-07-15 Thread Reynold Xin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16576?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15380093#comment-15380093 ] Reynold Xin commented on SPARK-16576: - cc [~dongjoon] this is related to the broadcast issue. How

[jira] [Created] (SPARK-16576) Move plan SQL generation code from SQLBuilder into logical operators

2016-07-15 Thread Reynold Xin (JIRA)
Reynold Xin created SPARK-16576: --- Summary: Move plan SQL generation code from SQLBuilder into logical operators Key: SPARK-16576 URL: https://issues.apache.org/jira/browse/SPARK-16576 Project: Spark

[jira] [Updated] (SPARK-15232) Add subquery SQL building tests to LogicalPlanToSQLSuite

2016-07-15 Thread Reynold Xin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15232?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Reynold Xin updated SPARK-15232: Issue Type: Sub-task (was: Improvement) Parent: SPARK-16576 > Add subquery SQL building

[jira] [Updated] (SPARK-16575) partition calculation mismatch with sc.binaryFiles

2016-07-15 Thread Suhas (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16575?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Suhas updated SPARK-16575: -- Component/s: Spark Shell Java API Input/Output > partition calculation

[jira] [Commented] (SPARK-14817) ML, Graph, R 2.0 QA: Programming guide update and migration guide

2016-07-15 Thread Joseph K. Bradley (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-14817?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15380077#comment-15380077 ] Joseph K. Bradley commented on SPARK-14817: --- I just merged

[jira] [Updated] (SPARK-15747) Support SPARK_CONF_DIR/spark-defaults.d/*.conf drop-in style config files

2016-07-15 Thread Reynold Xin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15747?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Reynold Xin updated SPARK-15747: Target Version/s: (was: 2.0.0) > Support SPARK_CONF_DIR/spark-defaults.d/*.conf drop-in style

[jira] [Updated] (SPARK-14146) Imported implicits can't be found in Spark REPL in some cases

2016-07-15 Thread Reynold Xin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-14146?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Reynold Xin updated SPARK-14146: Target Version/s: (was: 2.0.0) > Imported implicits can't be found in Spark REPL in some cases >

[jira] [Closed] (SPARK-14823) Fix all references to HiveContext in comments and docs

2016-07-15 Thread Reynold Xin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-14823?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Reynold Xin closed SPARK-14823. --- Resolution: Fixed This was fixed when we updated the SQL programming guide. > Fix all references

[jira] [Updated] (SPARK-13393) Column mismatch issue in left_outer join using Spark DataFrame

2016-07-15 Thread Yin Huai (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-13393?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yin Huai updated SPARK-13393: - Target Version/s: (was: 2.0.0) > Column mismatch issue in left_outer join using Spark DataFrame >

[jira] [Updated] (SPARK-15232) Add subquery SQL building tests to LogicalPlanToSQLSuite

2016-07-15 Thread Reynold Xin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15232?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Reynold Xin updated SPARK-15232: Target Version/s: (was: 2.0.0) > Add subquery SQL building tests to LogicalPlanToSQLSuite >

[jira] [Resolved] (SPARK-13393) Column mismatch issue in left_outer join using Spark DataFrame

2016-07-15 Thread Yin Huai (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-13393?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yin Huai resolved SPARK-13393. -- Resolution: Duplicate The underlying issue shown by this jira is SPARK-13801. I am closing this one.

[jira] [Commented] (SPARK-16575) partition calculation mismatch with sc.binaryFiles

2016-07-15 Thread Suhas (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16575?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15380065#comment-15380065 ] Suhas commented on SPARK-16575: --- There is a workaround for this, by using sc.binaryFiles(path,

[jira] [Created] (SPARK-16575) partition calculation mismatch with sc.binaryFiles

2016-07-15 Thread Suhas (JIRA)
Suhas created SPARK-16575: - Summary: partition calculation mismatch with sc.binaryFiles Key: SPARK-16575 URL: https://issues.apache.org/jira/browse/SPARK-16575 Project: Spark Issue Type: Bug

[jira] [Commented] (SPARK-16522) [MESOS] Spark application throws exception on exit

2016-07-15 Thread Reynold Xin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16522?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15380058#comment-15380058 ] Reynold Xin commented on SPARK-16522: - Any update? > [MESOS] Spark application throws exception on

[jira] [Updated] (SPARK-13753) Column nullable is derived incorrectly

2016-07-15 Thread Reynold Xin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-13753?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Reynold Xin updated SPARK-13753: Target Version/s: (was: 2.0.0) > Column nullable is derived incorrectly >

[jira] [Resolved] (SPARK-13959) Audit MiMa excludes added in SPARK-13948 to make sure none are unintended incompatibilities

2016-07-15 Thread Reynold Xin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-13959?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Reynold Xin resolved SPARK-13959. - Resolution: Fixed Assignee: Reynold Xin Fix Version/s: 2.0.0 Target

[jira] [Commented] (SPARK-16574) Distribute computing to each node based on certain hints

2016-07-15 Thread Norman He (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16574?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15380046#comment-15380046 ] Norman He commented on SPARK-16574: --- worker rdd is 40 tuples. They are equivalent. no data locality

[jira] [Updated] (SPARK-13393) Column mismatch issue in left_outer join using Spark DataFrame

2016-07-15 Thread Yin Huai (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-13393?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yin Huai updated SPARK-13393: - Priority: Major (was: Critical) > Column mismatch issue in left_outer join using Spark DataFrame >

[jira] [Updated] (SPARK-13393) Column mismatch issue in left_outer join using Spark DataFrame

2016-07-15 Thread Yin Huai (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-13393?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yin Huai updated SPARK-13393: - Assignee: (was: Xiang Zhong) > Column mismatch issue in left_outer join using Spark DataFrame >

[jira] [Updated] (SPARK-16046) Add Spark SQL Dataset Tutorial

2016-07-15 Thread Reynold Xin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16046?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Reynold Xin updated SPARK-16046: Target Version/s: (was: 2.0.0) > Add Spark SQL Dataset Tutorial > --

[jira] [Commented] (SPARK-5569) Checkpoints cannot reference classes defined outside of Spark's assembly

2016-07-15 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-5569?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15380024#comment-15380024 ] Sean Owen commented on SPARK-5569: -- I'm not sure it was fully resolved; see the last 3-4 comments. It is

[jira] [Commented] (SPARK-5569) Checkpoints cannot reference classes defined outside of Spark's assembly

2016-07-15 Thread Michael Li (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-5569?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15379976#comment-15379976 ] Michael Li commented on SPARK-5569: --- It seems the PR (https://github.com/apache/spark/pull/8955) has

[jira] [Commented] (SPARK-16574) Distribute computing to each node based on certain hints

2016-07-15 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16574?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15379923#comment-15379923 ] Sean Owen commented on SPARK-16574: --- spark.yarn.executor.nodeLabelExpression is described at

[jira] [Commented] (SPARK-16574) Distribute computing to each node based on certain hints

2016-07-15 Thread Norman He (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16574?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15379921#comment-15379921 ] Norman He commented on SPARK-16574: --- do you have a link? Each task should only use one gpu. and The

[jira] [Commented] (SPARK-16573) executor stderr processing tools

2016-07-15 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16573?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15379892#comment-15379892 ] Sean Owen commented on SPARK-16573: --- I'm not sure it's mysterious how that would work, but, doesn't

[jira] [Commented] (SPARK-16574) Distribute computing to each node based on certain hints

2016-07-15 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16574?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15379890#comment-15379890 ] Sean Owen commented on SPARK-16574: --- You can target which machines to choose with something like YARN

  1   2   3   >