[jira] [Created] (SPARK-1405) parallel Latent Dirichlet Allocation (LDA) atop of spark in MLlib

Xusen Yin created SPARK-1405: Summary: parallel Latent Dirichlet Allocation (LDA) atop of spark in MLlib Key: SPARK-1405 URL: https://issues.apache.org/jira/browse/SPARK-1405 Project: Spark Issu

[jira] [Created] (SPARK-1404) Non-exported spark-env.sh variables are no longer present in spark-shell

Aaron Davidson created SPARK-1404: - Summary: Non-exported spark-env.sh variables are no longer present in spark-shell Key: SPARK-1404 URL: https://issues.apache.org/jira/browse/SPARK-1404 Project: Spa

[jira] [Created] (SPARK-1403) java.lang.ClassNotFoundException - spark on mesos

Bharath Bhushan created SPARK-1403: -- Summary: java.lang.ClassNotFoundException - spark on mesos Key: SPARK-1403 URL: https://issues.apache.org/jira/browse/SPARK-1403 Project: Spark Issue Ty

[jira] [Created] (SPARK-1402) 3 more compression algorithms for in-memory columnar storage

Cheng Lian created SPARK-1402: - Summary: 3 more compression algorithms for in-memory columnar storage Key: SPARK-1402 URL: https://issues.apache.org/jira/browse/SPARK-1402 Project: Spark Issue T

[jira] [Updated] (SPARK-1157) L-BFGS Optimizer

[ https://issues.apache.org/jira/browse/SPARK-1157?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] DB Tsai updated SPARK-1157: --- Description: L-BFGS (Limited-memory BFGS) is an optimization algorithm like BFGS which uses an approximation

[jira] [Closed] (SPARK-1401) Use mapParitions instead of map to avoid creating expensive object in GradientDescent optimizer

[ https://issues.apache.org/jira/browse/SPARK-1401?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] DB Tsai closed SPARK-1401. -- Resolution: Duplicate Fix Version/s: 0.9.1 > Use mapParitions instead of map to avoid creating expensive

[jira] [Commented] (SPARK-1401) Use mapParitions instead of map to avoid creating expensive object in GradientDescent optimizer

[ https://issues.apache.org/jira/browse/SPARK-1401?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13958438#comment-13958438 ] DB Tsai commented on SPARK-1401: In SPARK-1212, this issue is addressed by aggregate. Gonn

[jira] [Resolved] (SPARK-1364) DataTypes missing from ScalaReflection

[ https://issues.apache.org/jira/browse/SPARK-1364?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell resolved SPARK-1364. Resolution: Fixed > DataTypes missing from ScalaReflection > --

[jira] [Created] (SPARK-1401) Use mapParitions instead of map to avoid creating expensive object in GradientDescent optimizer

DB Tsai created SPARK-1401: -- Summary: Use mapParitions instead of map to avoid creating expensive object in GradientDescent optimizer Key: SPARK-1401 URL: https://issues.apache.org/jira/browse/SPARK-1401 Pro

[jira] [Created] (SPARK-1400) Spark Streaming's received data is not cleaned up from BlockManagers when not needed any more

Tathagata Das created SPARK-1400: Summary: Spark Streaming's received data is not cleaned up from BlockManagers when not needed any more Key: SPARK-1400 URL: https://issues.apache.org/jira/browse/SPARK-1400

[jira] [Updated] (SPARK-1118) Executor state shows as KILLED even the application is finished normally

[ https://issues.apache.org/jira/browse/SPARK-1118?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell updated SPARK-1118: --- Priority: Critical (was: Major) > Executor state shows as KILLED even the application is fin

[jira] [Updated] (SPARK-1118) Executor state shows as KILLED even the application is finished normally

[ https://issues.apache.org/jira/browse/SPARK-1118?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell updated SPARK-1118: --- Assignee: Kan Zhang > Executor state shows as KILLED even the application is finished normall

[jira] [Created] (SPARK-1399) Reason for Stage Failure should be shown in UI

Kay Ousterhout created SPARK-1399: - Summary: Reason for Stage Failure should be shown in UI Key: SPARK-1399 URL: https://issues.apache.org/jira/browse/SPARK-1399 Project: Spark Issue Type: Bu

[jira] [Commented] (SPARK-1118) Executor state shows as KILLED even the application is finished normally

[ https://issues.apache.org/jira/browse/SPARK-1118?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13958329#comment-13958329 ] Kan Zhang commented on SPARK-1118: -- I took a look running SparkPi on my single node clust

[jira] [Commented] (SPARK-1398) FindBugs 1 --> FindBugs 2

[ https://issues.apache.org/jira/browse/SPARK-1398?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13958262#comment-13958262 ] Sean Owen commented on SPARK-1398: -- +1 I would also volunteer to run it and propose fixes

[jira] [Created] (SPARK-1398) FindBugs 1 --> FindBugs 2

Mark Hamstra created SPARK-1398: --- Summary: FindBugs 1 --> FindBugs 2 Key: SPARK-1398 URL: https://issues.apache.org/jira/browse/SPARK-1398 Project: Spark Issue Type: Improvement Comp

[jira] [Commented] (SPARK-1097) ConcurrentModificationException

[ https://issues.apache.org/jira/browse/SPARK-1097?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13958248#comment-13958248 ] Nishkam Ravi commented on SPARK-1097: - Patch submitted against Hadoop-10456. > Concur

[jira] [Created] (SPARK-1397) SparkListeners should be notified when stages are cancelled

Kay Ousterhout created SPARK-1397: - Summary: SparkListeners should be notified when stages are cancelled Key: SPARK-1397 URL: https://issues.apache.org/jira/browse/SPARK-1397 Project: Spark

[jira] [Assigned] (SPARK-1396) DAGScheduler has a memory leak for cancelled jobs

[ https://issues.apache.org/jira/browse/SPARK-1396?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kay Ousterhout reassigned SPARK-1396: - Assignee: Kay Ousterhout > DAGScheduler has a memory leak for cancelled jobs > --

[jira] [Commented] (SPARK-1396) DAGScheduler has a memory leak for cancelled jobs

[ https://issues.apache.org/jira/browse/SPARK-1396?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13958187#comment-13958187 ] Kay Ousterhout commented on SPARK-1396: --- https://github.com/apache/spark/pull/305 >

[jira] [Created] (SPARK-1396) DAGScheduler has a memory leak for cancelled jobs

Kay Ousterhout created SPARK-1396: - Summary: DAGScheduler has a memory leak for cancelled jobs Key: SPARK-1396 URL: https://issues.apache.org/jira/browse/SPARK-1396 Project: Spark Issue Type:

[jira] [Commented] (SPARK-1395) Cannot launch jobs on Yarn cluster with "local:" scheme in SPARK_JAR

[ https://issues.apache.org/jira/browse/SPARK-1395?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13958165#comment-13958165 ] Marcelo Vanzin commented on SPARK-1395: --- https://github.com/apache/spark/pull/303 >

[jira] [Updated] (SPARK-1133) Add a new small files input for MLlib, which will return an RDD[(fileName, content)]

[ https://issues.apache.org/jira/browse/SPARK-1133?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matei Zaharia updated SPARK-1133: - Assignee: Xusen Yin > Add a new small files input for MLlib, which will return an RDD[(fileName,

[jira] [Updated] (SPARK-1395) Cannot launch jobs on Yarn cluster with "local:" scheme in SPARK_JAR

[ https://issues.apache.org/jira/browse/SPARK-1395?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin updated SPARK-1395: -- Description: If you define SPARK_JAR and friends to use "local:" URIs, you cannot submit a job

[jira] [Updated] (SPARK-1395) Cannot launch jobs on Yarn cluster with "local:" scheme in SPARK_JAR

[ https://issues.apache.org/jira/browse/SPARK-1395?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin updated SPARK-1395: -- Summary: Cannot launch jobs on Yarn cluster with "local:" scheme in SPARK_JAR (was: Cannoit la

[jira] [Created] (SPARK-1395) Cannoit launch jobs on Yarn cluster with "local:" scheme in SPARK_JAR

Marcelo Vanzin created SPARK-1395: - Summary: Cannoit launch jobs on Yarn cluster with "local:" scheme in SPARK_JAR Key: SPARK-1395 URL: https://issues.apache.org/jira/browse/SPARK-1395 Project: Spark

[jira] [Resolved] (SPARK-1373) Compression for In-Memory Columnar storage

[ https://issues.apache.org/jira/browse/SPARK-1373?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michael Armbrust resolved SPARK-1373. - Resolution: Fixed > Compression for In-Memory Columnar storage >

[jira] [Reopened] (SPARK-1371) HashAggregate should stream tuples and avoid doing an extra count

[ https://issues.apache.org/jira/browse/SPARK-1371?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michael Armbrust reopened SPARK-1371: - > HashAggregate should stream tuples and avoid doing an extra count > ---

[jira] [Resolved] (SPARK-1371) HashAggregate should stream tuples and avoid doing an extra count

[ https://issues.apache.org/jira/browse/SPARK-1371?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell resolved SPARK-1371. Resolution: Fixed > HashAggregate should stream tuples and avoid doing an extra count > ---

[jira] [Updated] (SPARK-1154) Spark fills up disk with app-* folders

[ https://issues.apache.org/jira/browse/SPARK-1154?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell updated SPARK-1154: --- Fix Version/s: 1.0.0 > Spark fills up disk with app-* folders > -

[jira] [Updated] (SPARK-1154) Spark fills up disk with app-* folders

[ https://issues.apache.org/jira/browse/SPARK-1154?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell updated SPARK-1154: --- Priority: Critical (was: Major) > Spark fills up disk with app-* folders > -

[jira] [Updated] (SPARK-1391) BlockManager cannot transfer blocks larger than 2G in size

[ https://issues.apache.org/jira/browse/SPARK-1391?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matei Zaharia updated SPARK-1391: - Assignee: Min Zhou > BlockManager cannot transfer blocks larger than 2G in size > ---

[jira] [Commented] (SPARK-1391) BlockManager cannot transfer blocks larger than 2G in size

[ https://issues.apache.org/jira/browse/SPARK-1391?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13958036#comment-13958036 ] Min Zhou commented on SPARK-1391: - [~shivaram] Yeah, I meant the fastutils involved by spa

[jira] [Commented] (SPARK-1391) BlockManager cannot transfer blocks larger than 2G in size

[ https://issues.apache.org/jira/browse/SPARK-1391?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13958014#comment-13958014 ] Shivaram Venkataraman commented on SPARK-1391: -- Oh and yes, I'd be happy to t

[jira] [Commented] (SPARK-1391) BlockManager cannot transfer blocks larger than 2G in size

[ https://issues.apache.org/jira/browse/SPARK-1391?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13958013#comment-13958013 ] Shivaram Venkataraman commented on SPARK-1391: -- I am not using any fastutil v

[jira] [Commented] (SPARK-1391) BlockManager cannot transfer blocks larger than 2G in size

[ https://issues.apache.org/jira/browse/SPARK-1391?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13958003#comment-13958003 ] Min Zhou commented on SPARK-1391: - Thank you guys. [~shivaram] Which fastutils version di

[jira] [Updated] (SPARK-1394) calling system.platform on worker raises IOError

[ https://issues.apache.org/jira/browse/SPARK-1394?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Idan Zalzberg updated SPARK-1394: - Summary: calling system.platform on worker raises IOError (was: calling system.platform on worke

[jira] [Created] (SPARK-1394) calling system.platform on worker raises exception

Idan Zalzberg created SPARK-1394: Summary: calling system.platform on worker raises exception Key: SPARK-1394 URL: https://issues.apache.org/jira/browse/SPARK-1394 Project: Spark Issue Type:

[jira] [Commented] (SPARK-1097) ConcurrentModificationException

[ https://issues.apache.org/jira/browse/SPARK-1097?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13957958#comment-13957958 ] Nishkam Ravi commented on SPARK-1097: - We can consider putting a workaround in Spark a

[jira] [Commented] (SPARK-1391) BlockManager cannot transfer blocks larger than 2G in size

[ https://issues.apache.org/jira/browse/SPARK-1391?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13957957#comment-13957957 ] Shivaram Venkataraman commented on SPARK-1391: -- Yeah I think the right soluti

[jira] [Commented] (SPARK-1391) BlockManager cannot transfer blocks larger than 2G in size

[ https://issues.apache.org/jira/browse/SPARK-1391?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13957953#comment-13957953 ] Shivaram Venkataraman commented on SPARK-1391: -- [~coderplay] That sounds grea

[jira] [Commented] (SPARK-1391) BlockManager cannot transfer blocks larger than 2G in size

[ https://issues.apache.org/jira/browse/SPARK-1391?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13957949#comment-13957949 ] Sean Owen commented on SPARK-1391: -- Of course! Hardly my issue. Well, you could try my pa

[jira] [Commented] (SPARK-1391) BlockManager cannot transfer blocks larger than 2G in size

[ https://issues.apache.org/jira/browse/SPARK-1391?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13957939#comment-13957939 ] Min Zhou commented on SPARK-1391: - [~shivaram] [~sowen], if you don't mind, I will take

[jira] [Commented] (SPARK-1391) BlockManager cannot transfer blocks larger than 2G in size

[ https://issues.apache.org/jira/browse/SPARK-1391?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13957889#comment-13957889 ] Sean Owen commented on SPARK-1391: -- No fastutil is nothing to do with it, if in fact the

[jira] [Commented] (SPARK-1391) BlockManager cannot transfer blocks larger than 2G in size

[ https://issues.apache.org/jira/browse/SPARK-1391?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13957844#comment-13957844 ] Shivaram Venkataraman commented on SPARK-1391: -- I just copied the stack trace

[jira] [Commented] (SPARK-1384) spark-shell on yarn on spark 0.9 branch doesn't always work with secure hdfs

[ https://issues.apache.org/jira/browse/SPARK-1384?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13957650#comment-13957650 ] Thomas Graves commented on SPARK-1384: -- https://github.com/apache/spark/pull/287 > s

[jira] [Assigned] (SPARK-1384) spark-shell on yarn on spark 0.9 branch doesn't always work with secure hdfs

[ https://issues.apache.org/jira/browse/SPARK-1384?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves reassigned SPARK-1384: Assignee: Thomas Graves > spark-shell on yarn on spark 0.9 branch doesn't always work with

[jira] [Updated] (SPARK-1393) fix computePreferredLocations signature to not depend on underlying implementation

[ https://issues.apache.org/jira/browse/SPARK-1393?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mridul Muralidharan updated SPARK-1393: --- Description: computePreferredLocations in core/src/main/scala/org/apache/spark/schedu

[jira] [Commented] (SPARK-1391) BlockManager cannot transfer blocks larger than 2G in size

[ https://issues.apache.org/jira/browse/SPARK-1391?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13957523#comment-13957523 ] Min Zhou commented on SPARK-1391: - The JIT intrinsic will replace the original JNI impleme

[jira] [Comment Edited] (SPARK-1391) BlockManager cannot transfer blocks larger than 2G in size

[ https://issues.apache.org/jira/browse/SPARK-1391?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13957522#comment-13957522 ] Min Zhou edited comment on SPARK-1391 at 4/2/14 10:29 AM: -- Finall

[jira] [Commented] (SPARK-1391) BlockManager cannot transfer blocks larger than 2G in size

[ https://issues.apache.org/jira/browse/SPARK-1391?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13957522#comment-13957522 ] Min Zhou commented on SPARK-1391: - Finally, I can explain the whole thing. Apologies to Sh

[jira] [Created] (SPARK-1393) fix computePreferredLocations signature to not depend on underlying implementation

Mridul Muralidharan created SPARK-1393: -- Summary: fix computePreferredLocations signature to not depend on underlying implementation Key: SPARK-1393 URL: https://issues.apache.org/jira/browse/SPARK-1393

[jira] [Commented] (SPARK-1391) BlockManager cannot transfer blocks larger than 2G in size

[ https://issues.apache.org/jira/browse/SPARK-1391?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13957492#comment-13957492 ] Min Zhou commented on SPARK-1391: - >From the line number, the fastutils version should be

[jira] [Commented] (SPARK-1367) NPE when joining Parquet Relations

[ https://issues.apache.org/jira/browse/SPARK-1367?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13957476#comment-13957476 ] Andre Schumacher commented on SPARK-1367: - That should be now fixed when https://g

[jira] [Commented] (SPARK-1391) BlockManager cannot transfer blocks larger than 2G in size

[ https://issues.apache.org/jira/browse/SPARK-1391?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13957468#comment-13957468 ] Min Zhou commented on SPARK-1391: - [~sowen] Yes, neither of the two simulation can exactly

[jira] [Commented] (SPARK-1391) BlockManager cannot transfer blocks larger than 2G in size

[ https://issues.apache.org/jira/browse/SPARK-1391?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13957450#comment-13957450 ] Sean Owen commented on SPARK-1391: -- Oops yes I mean offset of course. Good investigation

[jira] [Comment Edited] (SPARK-1391) BlockManager cannot transfer blocks larger than 2G in size

[ https://issues.apache.org/jira/browse/SPARK-1391?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13957446#comment-13957446 ] Min Zhou edited comment on SPARK-1391 at 4/2/14 8:17 AM: - [~sowen]

[jira] [Commented] (SPARK-1391) BlockManager cannot transfer blocks larger than 2G in size

[ https://issues.apache.org/jira/browse/SPARK-1391?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13957446#comment-13957446 ] Min Zhou commented on SPARK-1391: - [~sowen] Yes, there is possibility. It's not a line nu

[jira] [Commented] (SPARK-1391) BlockManager cannot transfer blocks larger than 2G in size

[ https://issues.apache.org/jira/browse/SPARK-1391?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13957437#comment-13957437 ] Sean Owen commented on SPARK-1391: -- It is possible that it constructs AIOOBE without the

[jira] [Commented] (SPARK-1391) BlockManager cannot transfer blocks larger than 2G in size

[ https://issues.apache.org/jira/browse/SPARK-1391?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13957432#comment-13957432 ] Min Zhou commented on SPARK-1391: - It's supposed to have some detailed message after "jav

[jira] [Assigned] (SPARK-1367) NPE when joining Parquet Relations

[ https://issues.apache.org/jira/browse/SPARK-1367?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andre Schumacher reassigned SPARK-1367: --- Assignee: Andre Schumacher (was: Michael Armbrust) > NPE when joining Parquet Relati

[jira] [Comment Edited] (SPARK-1391) BlockManager cannot transfer blocks larger than 2G in size

[ https://issues.apache.org/jira/browse/SPARK-1391?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13957417#comment-13957417 ] Min Zhou edited comment on SPARK-1391 at 4/2/14 7:16 AM: - Hi Shiva

[jira] [Commented] (SPARK-1391) BlockManager cannot transfer blocks larger than 2G in size

[ https://issues.apache.org/jira/browse/SPARK-1391?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13957417#comment-13957417 ] Min Zhou commented on SPARK-1391: - Hi Shivaram Is that the entire exception stack? > Blo