[jira] [Commented] (SPARK-3828) Spark returns inconsistent results when building with different Hadoop version

2014-10-06 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3828?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14161564#comment-14161564 ] Patrick Wendell commented on SPARK-3828: Yeah fair point - I think for now though

[jira] [Commented] (SPARK-3828) Spark returns inconsistent results when building with different Hadoop version

2014-10-06 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3828?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14161552#comment-14161552 ] Sean Owen commented on SPARK-3828: -- (Agree, although there's an interesting point in here

[jira] [Commented] (SPARK-3412) Add Missing Types for Row API

2014-10-06 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3412?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14161549#comment-14161549 ] Apache Spark commented on SPARK-3412: - User 'davies' has created a pull request for th

[jira] [Resolved] (SPARK-3828) Spark returns inconsistent results when building with different Hadoop version

2014-10-06 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3828?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell resolved SPARK-3828. Resolution: Not a Problem I believe this issue is simply due to different behavior in the Ha

[jira] [Commented] (SPARK-3828) Spark returns inconsistent results when building with different Hadoop version

2014-10-06 Thread Xiangrui Meng (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3828?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14161540#comment-14161540 ] Xiangrui Meng commented on SPARK-3828: -- `text8` doesn't contain any line feed charact

[jira] [Updated] (SPARK-3828) Spark returns inconsistent results when building with different Hadoop version

2014-10-06 Thread Liquan Pei (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3828?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Liquan Pei updated SPARK-3828: -- Summary: Spark returns inconsistent results when building with different Hadoop version (was: Spark re

[jira] [Updated] (SPARK-3828) Spark returns inconsistent results when building with different HADOOP version

2014-10-06 Thread Liquan Pei (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3828?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Liquan Pei updated SPARK-3828: -- Summary: Spark returns inconsistent results when building with different HADOOP version (was: Spark re

[jira] [Created] (SPARK-3828) Spark returns inconsistent result when compiling with different HADOOP version

2014-10-06 Thread Liquan Pei (JIRA)
Liquan Pei created SPARK-3828: - Summary: Spark returns inconsistent result when compiling with different HADOOP version Key: SPARK-3828 URL: https://issues.apache.org/jira/browse/SPARK-3828 Project: Spar

[jira] [Updated] (SPARK-3461) Support external groupByKey using repartitionAndSortWithinPartitions

2014-10-06 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3461?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell updated SPARK-3461: --- Priority: Critical (was: Major) > Support external groupByKey using repartitionAndSortWithinP

[jira] [Commented] (SPARK-3819) Jenkins should compile Spark against multiple versions of Hadoop

2014-10-06 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3819?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14161513#comment-14161513 ] Patrick Wendell commented on SPARK-3819: It's not feasible to run against multiple

[jira] [Updated] (SPARK-3778) newAPIHadoopRDD doesn't properly pass credentials for secure hdfs on yarn

2014-10-06 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3778?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell updated SPARK-3778: --- Target Version/s: 1.1.1, 1.2.0 (was: 1.1.1) > newAPIHadoopRDD doesn't properly pass credentia

[jira] [Commented] (SPARK-3814) Bitwise & does not work in Hive

2014-10-06 Thread Ravindra Pesala (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3814?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14161472#comment-14161472 ] Ravindra Pesala commented on SPARK-3814: Currently there is no support of Bitwise

[jira] [Commented] (SPARK-3825) Log more information when unrolling a block fails

2014-10-06 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3825?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14161461#comment-14161461 ] Apache Spark commented on SPARK-3825: - User 'andrewor14' has created a pull request fo

[jira] [Commented] (SPARK-3827) Very long RDD names are not rendered properly in web UI

2014-10-06 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3827?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14161433#comment-14161433 ] Apache Spark commented on SPARK-3827: - User 'falaki' has created a pull request for th

[jira] [Created] (SPARK-3827) Very long RDD names are not rendered properly in web UI

2014-10-06 Thread Hossein Falaki (JIRA)
Hossein Falaki created SPARK-3827: - Summary: Very long RDD names are not rendered properly in web UI Key: SPARK-3827 URL: https://issues.apache.org/jira/browse/SPARK-3827 Project: Spark Issue

[jira] [Commented] (SPARK-2308) Add KMeans MiniBatch clustering algorithm to MLlib

2014-10-06 Thread Derrick Burns (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-2308?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14161390#comment-14161390 ] Derrick Burns commented on SPARK-2308: -- I submitted PR #2634 to address the issues th

[jira] [Comment Edited] (SPARK-2585) Remove special handling of Hadoop JobConf

2014-10-06 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-2585?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14161382#comment-14161382 ] Patrick Wendell edited comment on SPARK-2585 at 10/7/14 2:33 AM: ---

[jira] [Commented] (SPARK-2585) Remove special handling of Hadoop JobConf

2014-10-06 Thread Josh Rosen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-2585?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14161383#comment-14161383 ] Josh Rosen commented on SPARK-2585: --- [~pwendell] I obtained these numbers by adding this

[jira] [Commented] (SPARK-2585) Remove special handling of Hadoop JobConf

2014-10-06 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-2585?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14161382#comment-14161382 ] Patrick Wendell commented on SPARK-2585: Hey [~joshrosen] what happens if you run

[jira] [Commented] (SPARK-3261) KMeans clusterer can return duplicate cluster centers

2014-10-06 Thread Derrick Burns (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3261?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14161380#comment-14161380 ] Derrick Burns commented on SPARK-3261: -- Another possible source of duplicate cluster

[jira] [Commented] (SPARK-3174) Provide elastic scaling within a Spark application

2014-10-06 Thread Andrew Or (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3174?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14161378#comment-14161378 ] Andrew Or commented on SPARK-3174: -- @[~tgraves] Replying inline: bq. Just to make sure,

[jira] [Comment Edited] (SPARK-3174) Provide elastic scaling within a Spark application

2014-10-06 Thread Andrew Or (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3174?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14161372#comment-14161372 ] Andrew Or edited comment on SPARK-3174 at 10/7/14 2:20 AM: --- @[~s

[jira] [Commented] (SPARK-3174) Provide elastic scaling within a Spark application

2014-10-06 Thread Andrew Or (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3174?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14161372#comment-14161372 ] Andrew Or commented on SPARK-3174: -- @[~sandyr] Replying inline: bq. I would expect prope

[jira] [Commented] (SPARK-3824) Spark SQL should cache in MEMORY_AND_DISK by default

2014-10-06 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3824?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14161366#comment-14161366 ] Apache Spark commented on SPARK-3824: - User 'liancheng' has created a pull request for

[jira] [Commented] (SPARK-3826) enable hive-thriftserver support hive-0.13.1

2014-10-06 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3826?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14161352#comment-14161352 ] Apache Spark commented on SPARK-3826: - User 'scwf' has created a pull request for this

[jira] [Closed] (SPARK-3793) use hiveconf when parse hive ql

2014-10-06 Thread wangfei (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3793?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] wangfei closed SPARK-3793. -- Resolution: Fixed should fix it in #2241 > use hiveconf when parse hive ql > --- >

[jira] [Updated] (SPARK-3261) KMeans clusterer can return duplicate cluster centers

2014-10-06 Thread Xiangrui Meng (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3261?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiangrui Meng updated SPARK-3261: - Assignee: Derrick Burns > KMeans clusterer can return duplicate cluster centers >

[jira] [Updated] (SPARK-3424) KMeans Plus Plus is too slow

2014-10-06 Thread Xiangrui Meng (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3424?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiangrui Meng updated SPARK-3424: - Assignee: Derrick Burns > KMeans Plus Plus is too slow > > >

[jira] [Created] (SPARK-3826) enable hive-thriftserver support hive-0.13.1

2014-10-06 Thread wangfei (JIRA)
wangfei created SPARK-3826: -- Summary: enable hive-thriftserver support hive-0.13.1 Key: SPARK-3826 URL: https://issues.apache.org/jira/browse/SPARK-3826 Project: Spark Issue Type: Improvement

[jira] [Closed] (SPARK-3370) The simple test error

2014-10-06 Thread Xiangrui Meng (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3370?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiangrui Meng closed SPARK-3370. Resolution: Duplicate This is a known issue. We can fix it by checkpointing intermediate RDDs. For

[jira] [Updated] (SPARK-1006) MLlib ALS gets stack overflow with too many iterations

2014-10-06 Thread Xiangrui Meng (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-1006?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiangrui Meng updated SPARK-1006: - Component/s: MLlib > MLlib ALS gets stack overflow with too many iterations >

[jira] [Commented] (SPARK-3803) ArrayIndexOutOfBoundsException found in executing computePrincipalComponents

2014-10-06 Thread Xiangrui Meng (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3803?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14161281#comment-14161281 ] Xiangrui Meng commented on SPARK-3803: -- In `computeCovariance`, we generate a warning

[jira] [Closed] (SPARK-3493) Unrolling behavior is too aggressive in dropping blocks

2014-10-06 Thread Andrew Or (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3493?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Or closed SPARK-3493. Resolution: Won't Fix False alarm. We only ever call `ensureFreeSpace` if we can't reserve more memory. [~t

[jira] [Created] (SPARK-3825) Log more information when unrolling a block fails

2014-10-06 Thread Andrew Or (JIRA)
Andrew Or created SPARK-3825: Summary: Log more information when unrolling a block fails Key: SPARK-3825 URL: https://issues.apache.org/jira/browse/SPARK-3825 Project: Spark Issue Type: Bug

[jira] [Commented] (SPARK-2546) Configuration object thread safety issue

2014-10-06 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-2546?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14161214#comment-14161214 ] Apache Spark commented on SPARK-2546: - User 'JoshRosen' has created a pull request for

[jira] [Created] (SPARK-3824) Spark SQL should cache in MEMORY_AND_DISK by default

2014-10-06 Thread Patrick Wendell (JIRA)
Patrick Wendell created SPARK-3824: -- Summary: Spark SQL should cache in MEMORY_AND_DISK by default Key: SPARK-3824 URL: https://issues.apache.org/jira/browse/SPARK-3824 Project: Spark Issue

[jira] [Commented] (SPARK-2585) Remove special handling of Hadoop JobConf

2014-10-06 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-2585?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14161166#comment-14161166 ] Apache Spark commented on SPARK-2585: - User 'JoshRosen' has created a pull request for

[jira] [Resolved] (SPARK-2530) Relax incorrect assumption of one ExternalAppendOnlyMap per thread

2014-10-06 Thread Matei Zaharia (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-2530?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matei Zaharia resolved SPARK-2530. -- Resolution: Fixed Fix Version/s: 1.1.0 This was fixed by SPARK-2711. > Relax incorrect a

[jira] [Commented] (SPARK-3633) Fetches failure observed after SPARK-2711

2014-10-06 Thread Matei Zaharia (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3633?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14161149#comment-14161149 ] Matei Zaharia commented on SPARK-3633: -- BTW one other possibility is that ExternalApp

[jira] [Commented] (SPARK-3633) Fetches failure observed after SPARK-2711

2014-10-06 Thread Matei Zaharia (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3633?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14161142#comment-14161142 ] Matei Zaharia commented on SPARK-3633: -- In that case though, the problem might be tha

[jira] [Commented] (SPARK-3633) Fetches failure observed after SPARK-2711

2014-10-06 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3633?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14161120#comment-14161120 ] Marcelo Vanzin commented on SPARK-3633: --- [~matei], as Nishkam mentions above, we rev

[jira] [Commented] (SPARK-3633) Fetches failure observed after SPARK-2711

2014-10-06 Thread Matei Zaharia (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3633?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14161117#comment-14161117 ] Matei Zaharia commented on SPARK-3633: -- I'm curious, why do you think this is caused

[jira] [Updated] (SPARK-3633) Fetches failure observed after SPARK-2711

2014-10-06 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3633?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell updated SPARK-3633: --- Target Version/s: 1.2.0 > Fetches failure observed after SPARK-2711 >

[jira] [Updated] (SPARK-3633) Fetches failure observed after SPARK-2711

2014-10-06 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3633?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell updated SPARK-3633: --- Priority: Critical (was: Major) > Fetches failure observed after SPARK-2711 > ---

[jira] [Updated] (SPARK-3797) Run the shuffle service inside the YARN NodeManager as an AuxiliaryService

2014-10-06 Thread Sandy Ryza (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3797?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sandy Ryza updated SPARK-3797: -- Summary: Run the shuffle service inside the YARN NodeManager as an AuxiliaryService (was: Enable runnin

[jira] [Updated] (SPARK-3797) Run the shuffle service inside the YARN NodeManager as an AuxiliaryService

2014-10-06 Thread Sandy Ryza (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3797?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sandy Ryza updated SPARK-3797: -- Description: It's also worth considering running the shuffle service in a YARN container beside the exec

[jira] [Commented] (SPARK-3174) Provide elastic scaling within a Spark application

2014-10-06 Thread Sandy Ryza (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3174?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14161048#comment-14161048 ] Sandy Ryza commented on SPARK-3174: --- Ah, misread. My opinion is that, for a first cut w

[jira] [Updated] (SPARK-3823) Spark Hive SQL readColumn is not reset each time for a new query

2014-10-06 Thread Alex Liu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3823?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alex Liu updated SPARK-3823: Description: After a few queries running in the same hiveContext, "hive.io.file.readcolumn.ids"" and "hive.

[jira] [Commented] (SPARK-2546) Configuration object thread safety issue

2014-10-06 Thread Andrew Ash (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-2546?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14161013#comment-14161013 ] Andrew Ash commented on SPARK-2546: --- Excellent research Josh! I agree that we should pa

[jira] [Updated] (SPARK-3479) Have Jenkins show which category of tests failed in his GitHub messages

2014-10-06 Thread Nicholas Chammas (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3479?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nicholas Chammas updated SPARK-3479: Summary: Have Jenkins show which category of tests failed in his GitHub messages (was: Have

[jira] [Commented] (SPARK-3174) Provide elastic scaling within a Spark application

2014-10-06 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3174?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14160988#comment-14160988 ] Thomas Graves commented on SPARK-3174: -- Perhaps I misread it then because the proposa

[jira] [Resolved] (SPARK-3479) Have Jenkins show which tests failed in his GitHub messages

2014-10-06 Thread Josh Rosen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3479?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Josh Rosen resolved SPARK-3479. --- Resolution: Fixed Fix Version/s: 1.2.0 Issue resolved by pull request 2606 [https://github.com/

[jira] [Commented] (SPARK-3710) YARN integration test is flaky

2014-10-06 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3710?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14160966#comment-14160966 ] Apache Spark commented on SPARK-3710: - User 'vanzin' has created a pull request for th

[jira] [Commented] (SPARK-3174) Provide elastic scaling within a Spark application

2014-10-06 Thread Sandy Ryza (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3174?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14160960#comment-14160960 ] Sandy Ryza commented on SPARK-3174: --- bq. for instance, lets say I do some ETL stuff wher

[jira] [Resolved] (SPARK-3773) Sphinx build warnings

2014-10-06 Thread Josh Rosen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3773?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Josh Rosen resolved SPARK-3773. --- Resolution: Fixed Fix Version/s: 1.2.0 Issue resolved by pull request 2653 [https://github.com/

[jira] [Updated] (SPARK-3797) Enable running shuffle service in separate process from executor

2014-10-06 Thread Sandy Ryza (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3797?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sandy Ryza updated SPARK-3797: -- Description: This could either mean * Running the shuffle service inside the YARN NodeManager as an Auxi

[jira] [Updated] (SPARK-3797) Enable running shuffle service in separate process from executor

2014-10-06 Thread Sandy Ryza (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3797?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sandy Ryza updated SPARK-3797: -- Description: This could either mean * Running the shuffle service inside the YARN NodeManager as an auxi

[jira] [Resolved] (SPARK-3786) Speedup tests of PySpark

2014-10-06 Thread Josh Rosen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3786?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Josh Rosen resolved SPARK-3786. --- Resolution: Fixed Fix Version/s: 1.2.0 Issue resolved by pull request 2646 [https://github.com/

[jira] [Updated] (SPARK-3797) Enable running shuffle service in separate process from executor

2014-10-06 Thread Sandy Ryza (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3797?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sandy Ryza updated SPARK-3797: -- Summary: Enable running shuffle service in separate process from executor (was: Integrate shuffle servi

[jira] [Resolved] (SPARK-2461) Add a toString method to GeneralizedLinearModel

2014-10-06 Thread Josh Rosen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-2461?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Josh Rosen resolved SPARK-2461. --- Resolution: Fixed Fix Version/s: 1.2.0 Issue resolved by pull request 2625 [https://github.com/

[jira] [Commented] (SPARK-3174) Provide elastic scaling within a Spark application

2014-10-06 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3174?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14160939#comment-14160939 ] Thomas Graves commented on SPARK-3174: -- Good write up Andrew. Just to make sure, the

[jira] [Commented] (SPARK-3174) Provide elastic scaling within a Spark application

2014-10-06 Thread Sandy Ryza (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3174?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14160937#comment-14160937 ] Sandy Ryza commented on SPARK-3174: --- Thanks for posting the detailed design, Andrew. A

[jira] [Commented] (SPARK-3133) Piggyback get location RPC call to fetch small blocks

2014-10-06 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3133?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14160923#comment-14160923 ] Apache Spark commented on SPARK-3133: - User 'davies' has created a pull request for th

[jira] [Commented] (SPARK-3174) Provide elastic scaling within a Spark application

2014-10-06 Thread Andrew Or (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3174?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14160914#comment-14160914 ] Andrew Or commented on SPARK-3174: -- Hi all, I have attached an updated design doc that de

[jira] [Created] (SPARK-3823) Spark Hive SQL readColumn is not reset each time for a new query

2014-10-06 Thread Alex Liu (JIRA)
Alex Liu created SPARK-3823: --- Summary: Spark Hive SQL readColumn is not reset each time for a new query Key: SPARK-3823 URL: https://issues.apache.org/jira/browse/SPARK-3823 Project: Spark Issue T

[jira] [Updated] (SPARK-3174) Provide elastic scaling within a Spark application

2014-10-06 Thread Andrew Or (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3174?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Or updated SPARK-3174: - Description: A common complaint with Spark in a multi-tenant environment is that applications have a fixe

[jira] [Updated] (SPARK-3174) Provide elastic scaling within a Spark application

2014-10-06 Thread Andrew Or (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3174?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Or updated SPARK-3174: - Description: A common complaint with Spark in a multi-tenant environment is that applications have a fixe

[jira] [Updated] (SPARK-3822) Expose a mechanism for SparkContext to ask for / remove Yarn containers

2014-10-06 Thread Andrew Or (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3822?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Or updated SPARK-3822: - Issue Type: Sub-task (was: Bug) Parent: SPARK-3174 > Expose a mechanism for SparkContext to ask f

[jira] [Created] (SPARK-3822) Expose a mechanism for SparkContext to ask for / remove Yarn containers

2014-10-06 Thread Andrew Or (JIRA)
Andrew Or created SPARK-3822: Summary: Expose a mechanism for SparkContext to ask for / remove Yarn containers Key: SPARK-3822 URL: https://issues.apache.org/jira/browse/SPARK-3822 Project: Spark

[jira] [Commented] (SPARK-3314) Script creation of AMIs

2014-10-06 Thread Nicholas Chammas (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3314?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14160901#comment-14160901 ] Nicholas Chammas commented on SPARK-3314: - Sounds good. I've created [SPARK-3821]

[jira] [Created] (SPARK-3821) Develop an automated way of creating Spark images (AMI, Docker, and others)

2014-10-06 Thread Nicholas Chammas (JIRA)
Nicholas Chammas created SPARK-3821: --- Summary: Develop an automated way of creating Spark images (AMI, Docker, and others) Key: SPARK-3821 URL: https://issues.apache.org/jira/browse/SPARK-3821 Proje

[jira] [Commented] (SPARK-2546) Configuration object thread safety issue

2014-10-06 Thread Josh Rosen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-2546?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14160892#comment-14160892 ] Josh Rosen commented on SPARK-2546: --- For now, let's ignore the design issue of whether t

[jira] [Updated] (SPARK-3174) Provide elastic scaling within a Spark application

2014-10-06 Thread Andrew Or (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3174?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Or updated SPARK-3174: - Attachment: dynamic-scaling-executors-10-6-14.pdf > Provide elastic scaling within a Spark application > -

[jira] [Created] (SPARK-3820) Specialize columnSimilarity() without any threshold

2014-10-06 Thread Xiangrui Meng (JIRA)
Xiangrui Meng created SPARK-3820: Summary: Specialize columnSimilarity() without any threshold Key: SPARK-3820 URL: https://issues.apache.org/jira/browse/SPARK-3820 Project: Spark Issue Type:

[jira] [Commented] (SPARK-2546) Configuration object thread safety issue

2014-10-06 Thread Josh Rosen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-2546?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14160842#comment-14160842 ] Josh Rosen commented on SPARK-2546: --- Here are a few "in the wild" examples of how {{sc.h

[jira] [Commented] (SPARK-2546) Configuration object thread safety issue

2014-10-06 Thread Josh Rosen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-2546?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14160790#comment-14160790 ] Josh Rosen commented on SPARK-2546: --- I've decided to go with the cloning approach, since

[jira] [Resolved] (SPARK-3718) FsHistoryProvider should consider spark.eventLog.dir not only spark.history.fs.logDirectory

2014-10-06 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3718?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin resolved SPARK-3718. --- Resolution: Not a Problem I think it was decided that this is not a feature we want (see discu

[jira] [Commented] (SPARK-2585) Remove special handling of Hadoop JobConf

2014-10-06 Thread Andrew Ash (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-2585?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14160642#comment-14160642 ] Andrew Ash commented on SPARK-2585: --- I also vote for "correct by default" and there are

[jira] [Commented] (SPARK-2585) Remove special handling of Hadoop JobConf

2014-10-06 Thread Josh Rosen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-2585?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14160589#comment-14160589 ] Josh Rosen commented on SPARK-2585: --- I tried benchmarking the time need to create a new

[jira] [Assigned] (SPARK-3797) Integrate shuffle service in YARN's pluggable shuffle

2014-10-06 Thread Andrew Or (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3797?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Or reassigned SPARK-3797: Assignee: Andrew Or > Integrate shuffle service in YARN's pluggable shuffle > -

[jira] [Commented] (SPARK-3339) Support for skipping json lines that fail to parse

2014-10-06 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3339?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14160585#comment-14160585 ] Apache Spark commented on SPARK-3339: - User 'yhuai' has created a pull request for thi

[jira] [Created] (SPARK-3819) Jenkins should compile Spark against multiple versions of Hadoop

2014-10-06 Thread Matt Cheah (JIRA)
Matt Cheah created SPARK-3819: - Summary: Jenkins should compile Spark against multiple versions of Hadoop Key: SPARK-3819 URL: https://issues.apache.org/jira/browse/SPARK-3819 Project: Spark Iss

[jira] [Commented] (SPARK-3818) Graph coarsening

2014-10-06 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3818?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14160474#comment-14160474 ] Apache Spark commented on SPARK-3818: - User 'ankurdave' has created a pull request for

[jira] [Created] (SPARK-3818) Graph coarsening

2014-10-06 Thread Ankur Dave (JIRA)
Ankur Dave created SPARK-3818: - Summary: Graph coarsening Key: SPARK-3818 URL: https://issues.apache.org/jira/browse/SPARK-3818 Project: Spark Issue Type: New Feature Components: GraphX

[jira] [Created] (SPARK-3817) BlockManagerMasterActor: Got two different block manager registrations with Mesos

2014-10-06 Thread Timothy Chen (JIRA)
Timothy Chen created SPARK-3817: --- Summary: BlockManagerMasterActor: Got two different block manager registrations with Mesos Key: SPARK-3817 URL: https://issues.apache.org/jira/browse/SPARK-3817 Project

[jira] [Commented] (SPARK-3813) Support "case when" conditional functions in Spark SQL

2014-10-06 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3813?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14160439#comment-14160439 ] Apache Spark commented on SPARK-3813: - User 'ravipesala' has created a pull request fo

[jira] [Commented] (SPARK-3619) Upgrade to Mesos 0.21 to work around MESOS-1688

2014-10-06 Thread Timothy Chen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3619?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14160437#comment-14160437 ] Timothy Chen commented on SPARK-3619: - [~matei] > Upgrade to Mesos 0.21 to work aroun

[jira] [Commented] (SPARK-3619) Upgrade to Mesos 0.21 to work around MESOS-1688

2014-10-06 Thread Timothy Chen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3619?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14160436#comment-14160436 ] Timothy Chen commented on SPARK-3619: - I can do this, please assign it to me > Upgrad

[jira] [Closed] (SPARK-2616) Update Mesos to 0.19.1

2014-10-06 Thread Timothy Chen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-2616?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Timothy Chen closed SPARK-2616. --- Resolution: Fixed SPARK-3619 is going to update to 0.21 > Update Mesos to 0.19.1 > --

[jira] [Commented] (SPARK-3816) Add configureOutputJobPropertiesForStorageHandler to JobConf in SparkHadoopWriter

2014-10-06 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3816?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14160429#comment-14160429 ] Apache Spark commented on SPARK-3816: - User 'alexliu68' has created a pull request for

[jira] [Updated] (SPARK-3816) Add configureOutputJobPropertiesForStorageHandler to JobConf in SparkHadoopWriter

2014-10-06 Thread Alex Liu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3816?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alex Liu updated SPARK-3816: Summary: Add configureOutputJobPropertiesForStorageHandler to JobConf in SparkHadoopWriter (was: Add config

[jira] [Updated] (SPARK-3816) Add configureInputJobPropertiesForStorageHandler to JobConf in SparkHadoopWriter

2014-10-06 Thread Alex Liu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3816?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alex Liu updated SPARK-3816: Description: It's similar to SPARK-2846. We should add PlanUtils.configureInputJobPropertiesForStorageHandle

[jira] [Created] (SPARK-3816) Add configureInputJobPropertiesForStorageHandler to JobConf in SparkHadoopWriter

2014-10-06 Thread Alex Liu (JIRA)
Alex Liu created SPARK-3816: --- Summary: Add configureInputJobPropertiesForStorageHandler to JobConf in SparkHadoopWriter Key: SPARK-3816 URL: https://issues.apache.org/jira/browse/SPARK-3816 Project: Spark

[jira] [Commented] (SPARK-3813) Support "case when" conditional functions in Spark SQL

2014-10-06 Thread Ravindra Pesala (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3813?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14160368#comment-14160368 ] Ravindra Pesala commented on SPARK-3813: The below code gives the exception. {code

[jira] [Commented] (SPARK-3778) newAPIHadoopRDD doesn't properly pass credentials for secure hdfs on yarn

2014-10-06 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3778?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14160367#comment-14160367 ] Apache Spark commented on SPARK-3778: - User 'tgravescs' has created a pull request for

[jira] [Comment Edited] (SPARK-1391) BlockManager cannot transfer blocks larger than 2G in size

2014-10-06 Thread Gilberto Tin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-1391?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14160356#comment-14160356 ] Gilberto Tin edited comment on SPARK-1391 at 10/6/14 2:55 PM: --

[jira] [Comment Edited] (SPARK-1391) BlockManager cannot transfer blocks larger than 2G in size

2014-10-06 Thread Gilberto Tin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-1391?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14160356#comment-14160356 ] Gilberto Tin edited comment on SPARK-1391 at 10/6/14 2:55 PM: --

[jira] [Commented] (SPARK-1391) BlockManager cannot transfer blocks larger than 2G in size

2014-10-06 Thread Gilberto Tin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-1391?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14160356#comment-14160356 ] Gilberto Tin commented on SPARK-1391: - I am having the same issue spark 1.1.0. 6 node

[jira] [Assigned] (SPARK-3778) newAPIHadoopRDD doesn't properly pass credentials for secure hdfs on yarn

2014-10-06 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3778?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves reassigned SPARK-3778: Assignee: Thomas Graves > newAPIHadoopRDD doesn't properly pass credentials for secure hdfs

[jira] [Created] (SPARK-3815) LPAD function does not work in where predicate

2014-10-06 Thread Yana Kadiyska (JIRA)
Yana Kadiyska created SPARK-3815: Summary: LPAD function does not work in where predicate Key: SPARK-3815 URL: https://issues.apache.org/jira/browse/SPARK-3815 Project: Spark Issue Type: Bug

  1   2   >