[jira] [Commented] (SPARK-3431) Parallelize execution of tests

2014-12-03 Thread Nicholas Chammas (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3431?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14233568#comment-14233568 ] Nicholas Chammas commented on SPARK-3431: - [~joshrosen] I tried [that patch you po

[jira] [Commented] (SPARK-3431) Parallelize execution of tests

2014-12-04 Thread Nicholas Chammas (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3431?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14234702#comment-14234702 ] Nicholas Chammas commented on SPARK-3431: - I think I'm on to something, but I need

[jira] [Commented] (SPARK-3431) Parallelize execution of tests

2014-12-04 Thread Nicholas Chammas (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3431?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14234783#comment-14234783 ] Nicholas Chammas commented on SPARK-3431: - As an aside, I expect there to be some

[jira] [Commented] (SPARK-3431) Parallelize execution of tests

2014-12-05 Thread Nicholas Chammas (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3431?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14236540#comment-14236540 ] Nicholas Chammas commented on SPARK-3431: - Here's an example failure I don't under

[jira] [Comment Edited] (SPARK-3431) Parallelize execution of tests

2014-12-05 Thread Nicholas Chammas (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3431?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14234702#comment-14234702 ] Nicholas Chammas edited comment on SPARK-3431 at 12/6/14 3:11 AM: --

[jira] [Commented] (SPARK-3431) Parallelize execution of tests

2014-12-08 Thread Nicholas Chammas (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3431?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14238330#comment-14238330 ] Nicholas Chammas commented on SPARK-3431: - I am currently (and have been) actively

[jira] [Commented] (SPARK-3431) Parallelize execution of tests

2014-12-08 Thread Nicholas Chammas (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3431?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14238596#comment-14238596 ] Nicholas Chammas commented on SPARK-3431: - Thanks for assigning the issue to me, J

[jira] [Commented] (SPARK-3431) Parallelize execution of tests

2014-12-08 Thread Nicholas Chammas (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3431?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14238666#comment-14238666 ] Nicholas Chammas commented on SPARK-3431: - OK, here's a patch for {{pom.xml}} that

[jira] [Comment Edited] (SPARK-3431) Parallelize execution of tests

2014-12-08 Thread Nicholas Chammas (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3431?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14238666#comment-14238666 ] Nicholas Chammas edited comment on SPARK-3431 at 12/8/14 11:36 PM: -

[jira] [Commented] (SPARK-3431) Parallelize execution of tests

2014-12-08 Thread Nicholas Chammas (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3431?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14238703#comment-14238703 ] Nicholas Chammas commented on SPARK-3431: - Here are some of the errors: {code} Ru

[jira] [Commented] (SPARK-3431) Parallelize execution of tests

2014-12-08 Thread Nicholas Chammas (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3431?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14238769#comment-14238769 ] Nicholas Chammas commented on SPARK-3431: - [~nkeywal] - I took a quick look at HBa

[jira] [Commented] (SPARK-3431) Parallelize execution of tests

2014-12-09 Thread Nicholas Chammas (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3431?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14240519#comment-14240519 ] Nicholas Chammas commented on SPARK-3431: - OK, thanks for the updates, Sean and Ni

[jira] [Comment Edited] (SPARK-3431) Parallelize execution of tests

2014-12-09 Thread Nicholas Chammas (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3431?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14240519#comment-14240519 ] Nicholas Chammas edited comment on SPARK-3431 at 12/10/14 2:01 AM: -

[jira] [Commented] (SPARK-3431) Parallelize execution of tests

2014-12-09 Thread Nicholas Chammas (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3431?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14240530#comment-14240530 ] Nicholas Chammas commented on SPARK-3431: - For the record, the suite that I'm runn

[jira] [Commented] (SPARK-4826) Possible flaky tests in WriteAheadLogBackedBlockRDDSuite: "java.lang.IllegalStateException: File exists and there is no append support!"

2014-12-14 Thread Nicholas Chammas (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-4826?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14246091#comment-14246091 ] Nicholas Chammas commented on SPARK-4826: - This raises an interesting test infrast

[jira] [Commented] (SPARK-4826) Possible flaky tests in WriteAheadLogBackedBlockRDDSuite: "java.lang.IllegalStateException: File exists and there is no append support!"

2014-12-14 Thread Nicholas Chammas (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-4826?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14246122#comment-14246122 ] Nicholas Chammas commented on SPARK-4826: - I just cooked up a quick way of invokin

[jira] [Created] (SPARK-4868) Twitter DStream.map() throws "Task not serializable"

2014-12-16 Thread Nicholas Chammas (JIRA)
Nicholas Chammas created SPARK-4868: --- Summary: Twitter DStream.map() throws "Task not serializable" Key: SPARK-4868 URL: https://issues.apache.org/jira/browse/SPARK-4868 Project: Spark Issu

[jira] [Commented] (SPARK-4868) Twitter DStream.map() throws "Task not serializable"

2014-12-16 Thread Nicholas Chammas (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-4868?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14249387#comment-14249387 ] Nicholas Chammas commented on SPARK-4868: - cc [~tdas], [~adav] > Twitter DStream.

[jira] [Commented] (SPARK-4868) Twitter DStream.map() throws "Task not serializable"

2014-12-16 Thread Nicholas Chammas (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-4868?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14249582#comment-14249582 ] Nicholas Chammas commented on SPARK-4868: - Replacing {{noop()}} with this {code}

[jira] [Updated] (SPARK-3432) Fix logging of unit test execution time

2014-12-17 Thread Nicholas Chammas (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3432?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nicholas Chammas updated SPARK-3432: Description: [Per Reynold|http://mail-archives.apache.org/mod_mbox/spark-dev/201408.mbox/%3

[jira] [Commented] (SPARK-3533) Add saveAsTextFileByKey() method to RDDs

2014-12-17 Thread Nicholas Chammas (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3533?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14250227#comment-14250227 ] Nicholas Chammas commented on SPARK-3533: - [~alilee] Do you mean [this answer|http

[jira] [Commented] (SPARK-3561) Allow for pluggable execution contexts in Spark

2014-12-20 Thread Nicholas Chammas (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3561?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14254605#comment-14254605 ] Nicholas Chammas commented on SPARK-3561: - {quote} One of the main reasons for thi

[jira] [Commented] (SPARK-2541) Standalone mode can't access secure HDFS anymore

2014-12-20 Thread Nicholas Chammas (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-2541?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14254625#comment-14254625 ] Nicholas Chammas commented on SPARK-2541: - Hey [~tgraves], is there someone we sho

[jira] [Commented] (SPARK-3432) Fix logging of unit test execution time

2014-12-21 Thread Nicholas Chammas (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3432?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14255106#comment-14255106 ] Nicholas Chammas commented on SPARK-3432: - [~rxin] Is this still an issue? As far

[jira] [Commented] (SPARK-2541) Standalone mode can't access secure HDFS anymore

2014-12-22 Thread Nicholas Chammas (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-2541?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14256357#comment-14256357 ] Nicholas Chammas commented on SPARK-2541: - I was just wondering if we needed to pi

[jira] [Commented] (SPARK-3821) Develop an automated way of creating Spark images (AMI, Docker, and others)

2014-12-22 Thread Nicholas Chammas (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3821?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14256683#comment-14256683 ] Nicholas Chammas commented on SPARK-3821: - Per the discussion earlier, I've [upda

[jira] [Reopened] (SPARK-4325) Improve spark-ec2 cluster launch times

2014-12-22 Thread Nicholas Chammas (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-4325?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nicholas Chammas reopened SPARK-4325: - Hey [~joshrosen], though [#3195|https://github.com/apache/spark/pull/3195] relates to this JI

[jira] [Commented] (SPARK-4241) spark_ec2.py support China AWS region: cn-north-1

2014-12-22 Thread Nicholas Chammas (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-4241?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14256694#comment-14256694 ] Nicholas Chammas commented on SPARK-4241: - [~joshrosen] - I noticed you linked thi

[jira] [Created] (SPARK-4947) Use EC2 status checks to know when to test SSH availability

2014-12-23 Thread Nicholas Chammas (JIRA)
Nicholas Chammas created SPARK-4947: --- Summary: Use EC2 status checks to know when to test SSH availability Key: SPARK-4947 URL: https://issues.apache.org/jira/browse/SPARK-4947 Project: Spark

[jira] [Resolved] (SPARK-4947) Use EC2 status checks to know when to test SSH availability

2014-12-23 Thread Nicholas Chammas (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-4947?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nicholas Chammas resolved SPARK-4947. - Resolution: Fixed Resolved in [#3195|https://github.com/apache/spark/pull/3195]. > Use EC

[jira] [Commented] (SPARK-4325) Improve spark-ec2 cluster launch times

2014-12-23 Thread Nicholas Chammas (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-4325?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14257939#comment-14257939 ] Nicholas Chammas commented on SPARK-4325: - OK, I created a [sub-task|https://issue

[jira] [Created] (SPARK-4948) Use pssh instead of bash-isms and remove unnecessary operations

2014-12-23 Thread Nicholas Chammas (JIRA)
Nicholas Chammas created SPARK-4948: --- Summary: Use pssh instead of bash-isms and remove unnecessary operations Key: SPARK-4948 URL: https://issues.apache.org/jira/browse/SPARK-4948 Project: Spark

[jira] [Created] (SPARK-4958) Bake common tools like ganglia into Spark AMI

2014-12-24 Thread Nicholas Chammas (JIRA)
Nicholas Chammas created SPARK-4958: --- Summary: Bake common tools like ganglia into Spark AMI Key: SPARK-4958 URL: https://issues.apache.org/jira/browse/SPARK-4958 Project: Spark Issue Type:

[jira] [Updated] (SPARK-2528) Default spark-ec2 security group permissions are too open

2014-12-25 Thread Nicholas Chammas (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-2528?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nicholas Chammas updated SPARK-2528: Description: {{spark-ec2}} by default configures EC2 security groups with ports [open to the

[jira] [Updated] (SPARK-2528) Default spark-ec2 security group permissions are too open

2014-12-25 Thread Nicholas Chammas (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-2528?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nicholas Chammas updated SPARK-2528: Summary: Default spark-ec2 security group permissions are too open (was: spark-ec2 security

[jira] [Updated] (SPARK-3522) Make spark-ec2 verbosity configurable

2014-12-25 Thread Nicholas Chammas (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3522?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nicholas Chammas updated SPARK-3522: Labels: starter (was: ) > Make spark-ec2 verbosity configurable > -

[jira] [Updated] (SPARK-4868) Twitter DStream.map() throws "Task not serializable"

2014-12-25 Thread Nicholas Chammas (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-4868?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nicholas Chammas updated SPARK-4868: Environment: * Spark 1.1.1 or 1.2.0 * EC2 cluster with 1 slave spun up using {{spark-e

[jira] [Commented] (SPARK-4977) spark-ec2 start resets all the spark/conf configurations

2014-12-27 Thread Nicholas Chammas (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-4977?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14259472#comment-14259472 ] Nicholas Chammas commented on SPARK-4977: - cc [~shivaram] > spark-ec2 start reset

[jira] [Created] (SPARK-4983) Tag EC2 instances in the same call that launches them

2014-12-28 Thread Nicholas Chammas (JIRA)
Nicholas Chammas created SPARK-4983: --- Summary: Tag EC2 instances in the same call that launches them Key: SPARK-4983 URL: https://issues.apache.org/jira/browse/SPARK-4983 Project: Spark Iss

[jira] [Commented] (SPARK-4983) Tag EC2 instances in the same call that launches them

2014-12-28 Thread Nicholas Chammas (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-4983?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14259816#comment-14259816 ] Nicholas Chammas commented on SPARK-4983: - FYI [~shivaram], [~joshrosen] > Tag EC

[jira] [Commented] (SPARK-4983) Tag EC2 instances in the same call that launches them

2014-12-29 Thread Nicholas Chammas (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-4983?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14260668#comment-14260668 ] Nicholas Chammas commented on SPARK-4983: - It's something we overlooked. I'm not s

[jira] [Created] (SPARK-4997) Check if Spark's conf needs to be put ahead of Hadoop's (for log4j purposes)

2014-12-29 Thread Nicholas Chammas (JIRA)
Nicholas Chammas created SPARK-4997: --- Summary: Check if Spark's conf needs to be put ahead of Hadoop's (for log4j purposes) Key: SPARK-4997 URL: https://issues.apache.org/jira/browse/SPARK-4997 Proj

[jira] [Commented] (SPARK-4997) Check if Spark's conf needs to be put ahead of Hadoop's (for log4j purposes)

2014-12-30 Thread Nicholas Chammas (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-4997?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14261234#comment-14261234 ] Nicholas Chammas commented on SPARK-4997: - Could be, though this issue is specific

[jira] [Commented] (SPARK-4997) Check if Spark's conf needs to be put ahead of Hadoop's (for log4j purposes)

2014-12-30 Thread Nicholas Chammas (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-4997?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14261308#comment-14261308 ] Nicholas Chammas commented on SPARK-4997: - OK, I'll close this issue as a duplicat

[jira] [Closed] (SPARK-4997) Check if Spark's conf needs to be put ahead of Hadoop's (for log4j purposes)

2014-12-30 Thread Nicholas Chammas (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-4997?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nicholas Chammas closed SPARK-4997. --- Resolution: Duplicate > Check if Spark's conf needs to be put ahead of Hadoop's (for log4j pur

[jira] [Commented] (SPARK-1010) Update all unit tests to use SparkConf instead of system properties

2014-12-30 Thread Nicholas Chammas (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-1010?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14261394#comment-14261394 ] Nicholas Chammas commented on SPARK-1010: - [~joshrosen] Haven't you been doing som

[jira] [Commented] (SPARK-2394) Make it easier to read LZO-compressed files from EC2 clusters

2014-12-30 Thread Nicholas Chammas (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-2394?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14261663#comment-14261663 ] Nicholas Chammas commented on SPARK-2394: - [~joshrosen] Put together a script as p

[jira] [Resolved] (SPARK-2394) Make it easier to read LZO-compressed files from EC2 clusters

2014-12-30 Thread Nicholas Chammas (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-2394?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nicholas Chammas resolved SPARK-2394. - Resolution: Won't Fix Resolving as "Won't Fix" since this seems like a relatively minor us

[jira] [Commented] (SPARK-3821) Develop an automated way of creating Spark images (AMI, Docker, and others)

2015-01-01 Thread Nicholas Chammas (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3821?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14262720#comment-14262720 ] Nicholas Chammas commented on SPARK-3821: - For lulz, I've benchmarked the start ti

[jira] [Commented] (SPARK-3821) Develop an automated way of creating Spark images (AMI, Docker, and others)

2015-01-02 Thread Nicholas Chammas (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3821?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14263187#comment-14263187 ] Nicholas Chammas commented on SPARK-3821: - I need to brush up on my statistics, bu

[jira] [Commented] (SPARK-3821) Develop an automated way of creating Spark images (AMI, Docker, and others)

2015-01-02 Thread Nicholas Chammas (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3821?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14263206#comment-14263206 ] Nicholas Chammas commented on SPARK-3821: - I have Packer configured to run {{creat

[jira] [Commented] (SPARK-4898) Replace cloudpickle with Dill

2015-01-05 Thread Nicholas Chammas (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-4898?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14264801#comment-14264801 ] Nicholas Chammas commented on SPARK-4898: - cc [~davies] > Replace cloudpickle wit

[jira] [Created] (SPARK-5122) Remove Shark from spark-ec2 modules

2015-01-06 Thread Nicholas Chammas (JIRA)
Nicholas Chammas created SPARK-5122: --- Summary: Remove Shark from spark-ec2 modules Key: SPARK-5122 URL: https://issues.apache.org/jira/browse/SPARK-5122 Project: Spark Issue Type: Improveme

[jira] [Updated] (SPARK-5122) Remove Shark from spark-ec2

2015-01-06 Thread Nicholas Chammas (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-5122?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nicholas Chammas updated SPARK-5122: Summary: Remove Shark from spark-ec2 (was: Remove Shark from spark-ec2 modules) > Remove S

[jira] [Commented] (SPARK-5122) Remove Shark from spark-ec2

2015-01-06 Thread Nicholas Chammas (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-5122?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14267281#comment-14267281 ] Nicholas Chammas commented on SPARK-5122: - cc [~shivaram] - Is it appropriate to j

[jira] [Updated] (SPARK-5122) Remove Shark from spark-ec2

2015-01-06 Thread Nicholas Chammas (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-5122?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nicholas Chammas updated SPARK-5122: Description: Since Shark has been replaced by Spark SQL, we don't need it in {{spark-ec2}} a

[jira] [Updated] (SPARK-5122) Remove Shark from spark-ec2

2015-01-06 Thread Nicholas Chammas (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-5122?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nicholas Chammas updated SPARK-5122: Description: Since Shark has been replaced by Spark SQL, we don't need it in {{spark-ec2}} a

[jira] [Commented] (SPARK-926) spark_ec2 script when ssh/scp-ing should pipe UserknowHostFile to /dev/null

2015-01-06 Thread Nicholas Chammas (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-926?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14267327#comment-14267327 ] Nicholas Chammas commented on SPARK-926: [~shay] / [~shayping] (pinging both names

[jira] [Resolved] (SPARK-4948) Use pssh instead of bash-isms and remove unnecessary operations

2015-01-06 Thread Nicholas Chammas (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-4948?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nicholas Chammas resolved SPARK-4948. - Resolution: Fixed Resolved by: https://github.com/mesos/spark-ec2/pull/86 > Use pssh inst

[jira] [Updated] (SPARK-4948) Use pssh instead of bash-isms and remove unnecessary operations

2015-01-06 Thread Nicholas Chammas (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-4948?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nicholas Chammas updated SPARK-4948: Target Version/s: 1.3.0 > Use pssh instead of bash-isms and remove unnecessary operations >

[jira] [Commented] (SPARK-4948) Use pssh instead of bash-isms and remove unnecessary operations

2015-01-06 Thread Nicholas Chammas (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-4948?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14267331#comment-14267331 ] Nicholas Chammas commented on SPARK-4948: - [~shivaram] Could you assign this issue

[jira] [Created] (SPARK-5125) Time major blocks of code in spark-ec2/setup.sh

2015-01-06 Thread Nicholas Chammas (JIRA)
Nicholas Chammas created SPARK-5125: --- Summary: Time major blocks of code in spark-ec2/setup.sh Key: SPARK-5125 URL: https://issues.apache.org/jira/browse/SPARK-5125 Project: Spark Issue Typ

[jira] [Commented] (SPARK-5125) Time major blocks of code in spark-ec2/setup.sh

2015-01-06 Thread Nicholas Chammas (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-5125?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14267333#comment-14267333 ] Nicholas Chammas commented on SPARK-5125: - [~shivaram] Please assign this issue to

[jira] [Resolved] (SPARK-5125) Time major blocks of code in spark-ec2/setup.sh

2015-01-07 Thread Nicholas Chammas (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-5125?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nicholas Chammas resolved SPARK-5125. - Resolution: Fixed Fix Version/s: 1.3.0 Target Version/s: 1.3.0 Resolved b

[jira] [Updated] (SPARK-4983) Tag EC2 instances in the same call that launches them

2015-01-07 Thread Nicholas Chammas (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-4983?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nicholas Chammas updated SPARK-4983: Labels: starter (was: ) > Tag EC2 instances in the same call that launches them > -

[jira] [Commented] (SPARK-2541) Standalone mode can't access secure HDFS anymore

2015-01-07 Thread Nicholas Chammas (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-2541?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14268475#comment-14268475 ] Nicholas Chammas commented on SPARK-2541: - By the way, should this issue be linked

[jira] [Commented] (SPARK-4983) Tag EC2 instances in the same call that launches them

2015-01-08 Thread Nicholas Chammas (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-4983?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14270147#comment-14270147 ] Nicholas Chammas commented on SPARK-4983: - Yeah, I took a quick look at the boto A

[jira] [Updated] (SPARK-3431) Parallelize Scala/Java test execution

2015-01-08 Thread Nicholas Chammas (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3431?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nicholas Chammas updated SPARK-3431: Summary: Parallelize Scala/Java test execution (was: Parallelize execution of tests) > Par

[jira] [Created] (SPARK-5161) Parallelize Python test execution

2015-01-08 Thread Nicholas Chammas (JIRA)
Nicholas Chammas created SPARK-5161: --- Summary: Parallelize Python test execution Key: SPARK-5161 URL: https://issues.apache.org/jira/browse/SPARK-5161 Project: Spark Issue Type: Improvement

[jira] [Commented] (SPARK-3431) Parallelize Scala/Java test execution

2015-01-08 Thread Nicholas Chammas (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3431?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14270324#comment-14270324 ] Nicholas Chammas commented on SPARK-3431: - Generic update: * For those not followi

[jira] [Created] (SPARK-5178) Integrate Python unit tests into Jenkins

2015-01-09 Thread Nicholas Chammas (JIRA)
Nicholas Chammas created SPARK-5178: --- Summary: Integrate Python unit tests into Jenkins Key: SPARK-5178 URL: https://issues.apache.org/jira/browse/SPARK-5178 Project: Spark Issue Type: Impr

[jira] [Commented] (SPARK-5178) Integrate Python unit tests into Jenkins

2015-01-09 Thread Nicholas Chammas (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-5178?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14271858#comment-14271858 ] Nicholas Chammas commented on SPARK-5178: - Relevant links: * http://stackoverflow.

[jira] [Commented] (SPARK-4924) Factor out code to launch Spark applications into a separate library

2015-01-09 Thread Nicholas Chammas (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-4924?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14272315#comment-14272315 ] Nicholas Chammas commented on SPARK-4924: - [~vanzin] - How does this proposal affe

[jira] [Commented] (SPARK-4924) Factor out code to launch Spark applications into a separate library

2015-01-09 Thread Nicholas Chammas (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-4924?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14272338#comment-14272338 ] Nicholas Chammas commented on SPARK-4924: - Okie doke. I ask because I've been wond

[jira] [Commented] (SPARK-2004) QA Automation

2015-01-09 Thread Nicholas Chammas (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-2004?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14272373#comment-14272373 ] Nicholas Chammas commented on SPARK-2004: - I recently had a [related discussion|h

[jira] [Commented] (SPARK-5008) Persistent HDFS does not recognize EBS Volumes

2015-01-10 Thread Nicholas Chammas (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-5008?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14272585#comment-14272585 ] Nicholas Chammas commented on SPARK-5008: - cc [~shivaram] [~brdwrd] - What was th

[jira] [Commented] (SPARK-4778) PySpark Json and groupByKey broken

2015-01-10 Thread Nicholas Chammas (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-4778?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14272587#comment-14272587 ] Nicholas Chammas commented on SPARK-4778: - Also, though the environment is on EC2,

[jira] [Updated] (SPARK-4778) PySpark Json and groupByKey broken

2015-01-10 Thread Nicholas Chammas (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-4778?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nicholas Chammas updated SPARK-4778: Component/s: (was: EC2) > PySpark Json and groupByKey broken > -

[jira] [Commented] (SPARK-4399) Support multiple cloud providers

2015-01-10 Thread Nicholas Chammas (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-4399?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14272589#comment-14272589 ] Nicholas Chammas commented on SPARK-4399: - Now that [Spark Packages|http://spark-p

[jira] [Commented] (SPARK-1422) Add scripts for launching Spark on Google Compute Engine

2015-01-10 Thread Nicholas Chammas (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-1422?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14272590#comment-14272590 ] Nicholas Chammas commented on SPARK-1422: - We have [a package for launching Spark

[jira] [Commented] (SPARK-2396) Spark EC2 scripts fail when trying to log in to EC2 instances

2015-01-10 Thread Nicholas Chammas (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-2396?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14272596#comment-14272596 ] Nicholas Chammas commented on SPARK-2396: - [~enraged_ginger] - Can you confirm tha

[jira] [Commented] (SPARK-1555) enable ec2/spark_ec2.py to stop/delete cluster non-interactively

2015-01-10 Thread Nicholas Chammas (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-1555?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14272597#comment-14272597 ] Nicholas Chammas commented on SPARK-1555: - [~joshrosen]'s workaround is precisely

[jira] [Resolved] (SPARK-5086) Specifying the master instance type

2015-01-10 Thread Nicholas Chammas (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-5086?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nicholas Chammas resolved SPARK-5086. - Resolution: Duplicate Thanks for reporting this. I believe this is a duplicate of [SPARK-1

[jira] [Commented] (SPARK-1204) EC2 scripts upload private key

2015-01-10 Thread Nicholas Chammas (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-1204?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14272600#comment-14272600 ] Nicholas Chammas commented on SPARK-1204: - cc [~shivaram] Do you know off the top

[jira] [Commented] (SPARK-1532) provide option for more restrictive firewall rule in ec2/spark_ec2.py

2015-01-10 Thread Nicholas Chammas (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-1532?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14272601#comment-14272601 ] Nicholas Chammas commented on SPARK-1532: - I believe this capability is now presen

[jira] [Resolved] (SPARK-2528) Default spark-ec2 security group permissions are too open

2015-01-10 Thread Nicholas Chammas (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-2528?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nicholas Chammas resolved SPARK-2528. - Resolution: Duplicate > Default spark-ec2 security group permissions are too open > --

[jira] [Resolved] (SPARK-2649) EC2: Ganglia-httpd broken on hvm based machines like r3.4xlarge

2015-01-10 Thread Nicholas Chammas (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-2649?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nicholas Chammas resolved SPARK-2649. - Resolution: Duplicate This issue appears to be a duplicate of [SPARK-1302]. Feel free to r

[jira] [Commented] (SPARK-1302) httpd doesn't start in spark-ec2 (cc2.8xlarge)

2015-01-10 Thread Nicholas Chammas (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-1302?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14272605#comment-14272605 ] Nicholas Chammas commented on SPARK-1302: - There is an open PR by [~fredcons] here

[jira] [Resolved] (SPARK-4778) PySpark Json and groupByKey broken

2015-01-10 Thread Nicholas Chammas (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-4778?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nicholas Chammas resolved SPARK-4778. - Resolution: Cannot Reproduce > PySpark Json and groupByKey broken > --

[jira] [Commented] (SPARK-1555) enable ec2/spark_ec2.py to stop/delete cluster non-interactively

2015-01-10 Thread Nicholas Chammas (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-1555?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14272615#comment-14272615 ] Nicholas Chammas commented on SPARK-1555: - The other thing to consider, if we supp

[jira] [Created] (SPARK-5189) Reorganize EC2 scripts so that nodes can be provisioned independent of Spark master

2015-01-10 Thread Nicholas Chammas (JIRA)
Nicholas Chammas created SPARK-5189: --- Summary: Reorganize EC2 scripts so that nodes can be provisioned independent of Spark master Key: SPARK-5189 URL: https://issues.apache.org/jira/browse/SPARK-5189

[jira] [Commented] (SPARK-5189) Reorganize EC2 scripts so that nodes can be provisioned independent of Spark master

2015-01-10 Thread Nicholas Chammas (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-5189?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14272660#comment-14272660 ] Nicholas Chammas commented on SPARK-5189: - cc [~joshrosen] and [~shivaram] - What

[jira] [Resolved] (SPARK-1204) EC2 scripts upload private key

2015-01-10 Thread Nicholas Chammas (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-1204?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nicholas Chammas resolved SPARK-1204. - Resolution: Fixed Fix Version/s: 0.9.0 Resolving this issue per Shivaram's comment.

[jira] [Commented] (SPARK-5008) Persistent HDFS does not recognize EBS Volumes

2015-01-10 Thread Nicholas Chammas (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-5008?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14272846#comment-14272846 ] Nicholas Chammas commented on SPARK-5008: - Though I'm not too familiar with this s

[jira] [Commented] (SPARK-5008) Persistent HDFS does not recognize EBS Volumes

2015-01-11 Thread Nicholas Chammas (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-5008?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14273007#comment-14273007 ] Nicholas Chammas commented on SPARK-5008: - Use [{{copy-dir}}|https://github.com/me

[jira] [Commented] (SPARK-3821) Develop an automated way of creating Spark images (AMI, Docker, and others)

2015-01-11 Thread Nicholas Chammas (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3821?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14273187#comment-14273187 ] Nicholas Chammas commented on SPARK-3821: - Updated launch stats: * Launching clust

[jira] [Commented] (SPARK-1422) Add scripts for launching Spark on Google Compute Engine

2015-01-11 Thread Nicholas Chammas (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-1422?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14273197#comment-14273197 ] Nicholas Chammas commented on SPARK-1422: - [~pwendell] - I would consider doing th

[jira] [Commented] (SPARK-3821) Develop an automated way of creating Spark images (AMI, Docker, and others)

2015-01-12 Thread Nicholas Chammas (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3821?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14274535#comment-14274535 ] Nicholas Chammas commented on SPARK-3821: - That's correct. All those paths are jus

[jira] [Commented] (SPARK-5008) Persistent HDFS does not recognize EBS Volumes

2015-01-13 Thread Nicholas Chammas (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-5008?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14275656#comment-14275656 ] Nicholas Chammas commented on SPARK-5008: - [~brdwrd] - Thank you for documenting t

[jira] [Updated] (SPARK-3185) SPARK launch on Hadoop 2 in EC2 throws Tachyon exception when Formatting JOURNAL_FOLDER

2015-01-13 Thread Nicholas Chammas (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3185?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nicholas Chammas updated SPARK-3185: Description: {code} org.apache.hadoop.ipc.RemoteException: Server IPC version 7 cannot commu

<    3   4   5   6   7   8   9   10   11   >