[jira] [Commented] (SPARK-5335) Destroying cluster in VPC with --delete-groups fails to remove security groups

2015-02-05 Thread Nicholas Chammas (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-5335?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14308133#comment-14308133 ] Nicholas Chammas commented on SPARK-5335: - For the record: [AWS says|https

PSA: Maven supports parallel builds

2015-02-05 Thread Nicholas Chammas
Y’all may already know this, but I haven’t seen it mentioned anywhere in our docs on here and it’s a pretty easy win. Maven supports parallel builds https://cwiki.apache.org/confluence/display/MAVEN/Parallel+builds+in+Maven+3 with the -T command line option. For example: ./build/mvn -T 1C

[jira] [Comment Edited] (SPARK-5335) Destroying cluster in VPC with --delete-groups fails to remove security groups

2015-02-05 Thread Nicholas Chammas (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-5335?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14308133#comment-14308133 ] Nicholas Chammas edited comment on SPARK-5335 at 2/6/15 1:15 AM

[jira] [Comment Edited] (SPARK-5637) Expose spark_ec2 as as StarCluster Plugin

2015-02-05 Thread Nicholas Chammas (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-5637?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14308394#comment-14308394 ] Nicholas Chammas edited comment on SPARK-5637 at 2/6/15 1:19 AM

[jira] [Updated] (SPARK-5637) Expose spark_ec2 as as StarCluster Plugin

2015-02-05 Thread Nicholas Chammas (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-5637?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nicholas Chammas updated SPARK-5637: Component/s: EC2 Expose spark_ec2 as as StarCluster Plugin

[jira] [Commented] (SPARK-5637) Expose spark_ec2 as as StarCluster Plugin

2015-02-05 Thread Nicholas Chammas (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-5637?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14308394#comment-14308394 ] Nicholas Chammas commented on SPARK-5637: - [~agrothberg] - Can you expand on how

[jira] [Commented] (SPARK-5403) Ignore UserKnownHostsFile in SSH calls

2015-02-05 Thread Nicholas Chammas (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-5403?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14307904#comment-14307904 ] Nicholas Chammas commented on SPARK-5403: - I believe this issue duplicates

[jira] [Created] (SPARK-5627) Enhance spark-ec2 for some programmatic use cases

2015-02-05 Thread Nicholas Chammas (JIRA)
Nicholas Chammas created SPARK-5627: --- Summary: Enhance spark-ec2 for some programmatic use cases Key: SPARK-5627 URL: https://issues.apache.org/jira/browse/SPARK-5627 Project: Spark Issue

[jira] [Created] (SPARK-5628) Add option to return spark-ec2 version

2015-02-05 Thread Nicholas Chammas (JIRA)
Nicholas Chammas created SPARK-5628: --- Summary: Add option to return spark-ec2 version Key: SPARK-5628 URL: https://issues.apache.org/jira/browse/SPARK-5628 Project: Spark Issue Type

[jira] [Created] (SPARK-5629) Add spark-ec2 option to return info about cluster for programmatic consumption

2015-02-05 Thread Nicholas Chammas (JIRA)
Nicholas Chammas created SPARK-5629: --- Summary: Add spark-ec2 option to return info about cluster for programmatic consumption Key: SPARK-5629 URL: https://issues.apache.org/jira/browse/SPARK-5629

[jira] [Updated] (SPARK-3185) SPARK launch on Hadoop 2 in EC2 throws Tachyon exception when Formatting JOURNAL_FOLDER

2015-02-04 Thread Nicholas Chammas (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3185?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nicholas Chammas updated SPARK-3185: Component/s: EC2 SPARK launch on Hadoop 2 in EC2 throws Tachyon exception when Formatting

Re: [VOTE] Release Apache Spark 1.2.1 (RC3)

2015-02-03 Thread Nicholas Chammas
I believe this was changed for 1.2.1. Here are the relevant JIRA issues https://issues.apache.org/jira/browse/SPARK-5289?jql=project%20%3D%20SPARK%20AND%20fixVersion%20%3D%201.2.1%20AND%20text%20~%20%22publish%22%20order%20by%20priority . On Tue Feb 03 2015 at 10:43:59 AM Dirceu Semighini Filho

Re: Welcoming three new committers

2015-02-03 Thread Nicholas Chammas
Congratulations guys! On Tue Feb 03 2015 at 2:36:12 PM Matei Zaharia matei.zaha...@gmail.com wrote: Hi all, The PMC recently voted to add three new committers: Cheng Lian, Joseph Bradley and Sean Owen. All three have been major contributors to Spark in the past year: Cheng on Spark SQL,

Re: Building Spark with Pants

2015-02-02 Thread Nicholas Chammas
for sbt and with a little bit of tweaking with maven as well. 2015-02-02 16:25 GMT-08:00 Nicholas Chammas nicholas.cham...@gmail.com: Does anyone here have experience with Pants http://pantsbuild.github.io/index.html or interest in trying to build Spark with it? Pants has an interesting story

[jira] [Commented] (SPARK-2005) Investigate linux container-based solution

2015-02-02 Thread Nicholas Chammas (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-2005?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14302554#comment-14302554 ] Nicholas Chammas commented on SPARK-2005: - [~mengxr] - Do you mind if I renamed

Re: Building Spark with Pants

2015-02-02 Thread Nicholas Chammas
2015 at 4:40:45 PM Nicholas Chammas nicholas.cham...@gmail.com wrote: I'm asking from an experimental standpoint; this is not happening anytime soon. Of course, if the experiment turns out very well, Pants would replace both sbt and Maven (like it has at Twitter, for example). Pants also works

[jira] [Commented] (SPARK-5541) Allow running Maven or SBT in run-tests

2015-02-02 Thread Nicholas Chammas (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-5541?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14302525#comment-14302525 ] Nicholas Chammas commented on SPARK-5541: - Dup of SPARK-3355? Allow running

[jira] [Commented] (SPARK-2004) QA Automation

2015-02-02 Thread Nicholas Chammas (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-2004?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14302549#comment-14302549 ] Nicholas Chammas commented on SPARK-2004: - [~mengxr] - Do you mind if I renamed

Building Spark with Pants

2015-02-02 Thread Nicholas Chammas
Does anyone here have experience with Pants http://pantsbuild.github.io/index.html or interest in trying to build Spark with it? Pants has an interesting story. It was born at Twitter to help them build their Scala, Java, and Python projects as several independent components in one monolithic

Re: [VOTE] Release Apache Spark 1.2.1 (RC2)

2015-01-31 Thread Nicholas Chammas
Do we have any open JIRA issues to add automated testing on Windows to Jenkins? I assume that's something we want to do. On Sat Jan 31 2015 at 10:37:42 PM Matei Zaharia matei.zaha...@gmail.com wrote: This looks like a pretty serious problem, thanks! Glad people are testing on Windows. Matei

Re: spark 1.2 ec2 launch script hang

2015-01-28 Thread Nicholas Chammas
clusters on EC2 since with no problems.) On Wed Jan 28 2015 at 12:05:43 PM Nicholas Chammas nicholas.cham...@gmail.com wrote: Ey-chih, That makes more sense. This is a known issue that will be fixed as part of SPARK-5242 https://issues.apache.org/jira/browse/SPARK-5242. Charles, Thanks

Re: spark 1.2 ec2 launch script hang

2015-01-28 Thread Nicholas Chammas
. But the second execution when things worked with an absolute path could have worked because the random hosts that came up on EC2 were never in my known_hosts. On Wed Jan 28 2015 at 3:45:36 PM Nicholas Chammas nicholas.cham...@gmail.com wrote: Hmm, I can’t see why using ~ would be problematic

Re: spark 1.2 ec2 launch script hang

2015-01-28 Thread Nicholas Chammas
If that was indeed the problem, I suggest updating your answer on SO http://stackoverflow.com/a/28005151/877069 to help others who may run into this same problem. ​ On Wed Jan 28 2015 at 9:40:39 PM Nicholas Chammas nicholas.cham...@gmail.com wrote: Thanks for sending this over, Peter. What

[jira] [Created] (SPARK-5473) Expose SSH failures after status checks pass

2015-01-28 Thread Nicholas Chammas (JIRA)
Nicholas Chammas created SPARK-5473: --- Summary: Expose SSH failures after status checks pass Key: SPARK-5473 URL: https://issues.apache.org/jira/browse/SPARK-5473 Project: Spark Issue Type

Re: Extending Scala style checks

2015-01-28 Thread Nicholas Chammas
...@databricks.com wrote: Thanks. I added one. On Wed, Oct 8, 2014 at 8:49 AM, Nicholas Chammas nicholas.cham...@gmail.com wrote: I've created SPARK-3849: Automate remaining Scala style rules https://issues.apache.org/jira/browse/SPARK-3849. Please create sub-tasks on this issue for rules

[jira] [Created] (SPARK-5434) Preserve spaces in path to spark-ec2

2015-01-27 Thread Nicholas Chammas (JIRA)
Nicholas Chammas created SPARK-5434: --- Summary: Preserve spaces in path to spark-ec2 Key: SPARK-5434 URL: https://issues.apache.org/jira/browse/SPARK-5434 Project: Spark Issue Type: Bug

[jira] [Commented] (SPARK-2008) Enhance spark-ec2 to be able to add and remove slaves to an existing cluster

2015-01-27 Thread Nicholas Chammas (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-2008?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14293805#comment-14293805 ] Nicholas Chammas commented on SPARK-2008: - This isn't implemented. {{--use

Re: saving rdd to multiple files named by the key

2015-01-27 Thread Nicholas Chammas
There is also SPARK-3533 https://issues.apache.org/jira/browse/SPARK-3533, which proposes to add a convenience method for this. ​ On Mon Jan 26 2015 at 10:38:56 PM Aniket Bhatnagar aniket.bhatna...@gmail.com wrote: This might be helpful:

Re: spark 1.2 ec2 launch script hang

2015-01-27 Thread Nicholas Chammas
For those who found that absolute vs. relative path for the pem file mattered, what OS and shell are you using? What version of Spark are you using? ~/ vs. absolute path shouldn’t matter. Your shell will expand the ~/ to the absolute path before sending it to spark-ec2. (i.e. tilde expansion.)

[jira] [Created] (SPARK-5398) Support the eu-central-1 region for spark-ec2

2015-01-24 Thread Nicholas Chammas (JIRA)
Nicholas Chammas created SPARK-5398: --- Summary: Support the eu-central-1 region for spark-ec2 Key: SPARK-5398 URL: https://issues.apache.org/jira/browse/SPARK-5398 Project: Spark Issue Type

Re: Analyzing data from non-standard data sources (e.g. AWS Redshift)

2015-01-24 Thread Nicholas Chammas
I believe databricks provides an rdd interface to redshift. Did you check spark-packages.org? On 2015년 1월 24일 (토) at 오전 6:45 Denis Mikhalkin deni...@yahoo.com.invalid wrote: Hello, we've got some analytics data in AWS Redshift. The data is being constantly updated. I'd like to be able to

Re: Discourse: A proposed alternative to the Spark User list

2015-01-23 Thread Nicholas Chammas
or communication fora - provided that they allow exporting the conversation if those sites were to change course. However, the state of the art stands as such. - Patrick On Wed, Jan 21, 2015 at 8:43 AM, Nicholas Chammas nicholas.cham...@gmail.com wrote: Josh / Patrick, What do y’all think

Re: Discourse: A proposed alternative to the Spark User list

2015-01-23 Thread Nicholas Chammas
https://issues.apache.org/jira/browse/SPARK-5390 On Fri Jan 23 2015 at 12:05:00 PM Gerard Maas gerard.m...@gmail.com wrote: +1 On Fri, Jan 23, 2015 at 5:58 PM, Nicholas Chammas nicholas.cham...@gmail.com wrote: That sounds good to me. Shall I open a JIRA / PR about updating the site

[jira] [Created] (SPARK-5390) Encourage users to post on Stack Overflow in Community Docs

2015-01-23 Thread Nicholas Chammas (JIRA)
Nicholas Chammas created SPARK-5390: --- Summary: Encourage users to post on Stack Overflow in Community Docs Key: SPARK-5390 URL: https://issues.apache.org/jira/browse/SPARK-5390 Project: Spark

[jira] [Updated] (SPARK-5390) Encourage users to post on Stack Overflow in Community Docs

2015-01-23 Thread Nicholas Chammas (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-5390?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nicholas Chammas updated SPARK-5390: Description: As [discussed extensively on the user list|http://apache-spark-user-list

[jira] [Updated] (SPARK-5390) Encourage users to post on Stack Overflow in Community Docs

2015-01-23 Thread Nicholas Chammas (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-5390?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nicholas Chammas updated SPARK-5390: Description: As [discussed extensively on the user list|http://apache-spark-user-list

[jira] [Updated] (SPARK-5390) Encourage users to post on Stack Overflow in Community Docs

2015-01-23 Thread Nicholas Chammas (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-5390?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nicholas Chammas updated SPARK-5390: Description: As [discussed extensively on the user list|http://apache-spark-user-list

[jira] [Commented] (SPARK-5390) Encourage users to post on Stack Overflow in Community Docs

2015-01-23 Thread Nicholas Chammas (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-5390?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14290185#comment-14290185 ] Nicholas Chammas commented on SPARK-5390: - Updated accordingly. Encourage users

[jira] [Comment Edited] (SPARK-5390) Encourage users to post on Stack Overflow in Community Docs

2015-01-23 Thread Nicholas Chammas (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-5390?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14290187#comment-14290187 ] Nicholas Chammas edited comment on SPARK-5390 at 1/23/15 11:08 PM

[jira] [Commented] (SPARK-5390) Encourage users to post on Stack Overflow in Community Docs

2015-01-23 Thread Nicholas Chammas (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-5390?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14290187#comment-14290187 ] Nicholas Chammas commented on SPARK-5390: - cc [~pwendell] Encourage users

Re: Discourse: A proposed alternative to the Spark User list

2015-01-22 Thread Nicholas Chammas
I agree with Sean that a Spark-specific Stack Exchange likely won't help and almost certainly won't make it out of Area 51. The idea certainly sounds nice from our perspective as Spark users, but it doesn't mesh with the structure of Stack Exchange or the criteria for creating new sites. On Thu

Re: Discourse: A proposed alternative to the Spark User list

2015-01-22 Thread Nicholas Chammas
, Nicholas Chammas wrote: I think a few things need to be laid out clearly: 1. This mailing list is the “official” user discussion platform. That is, it is sponsored and managed by the ASF. 2. Users are free to organize independent discussion platforms focusing on Spark

[jira] [Commented] (SPARK-5366) check for mode of private key file

2015-01-22 Thread Nicholas Chammas (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-5366?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14288480#comment-14288480 ] Nicholas Chammas commented on SPARK-5366: - If the problem is no warnings

Re: Discourse: A proposed alternative to the Spark User list

2015-01-21 Thread Nicholas Chammas
Josh / Patrick, What do y’all think of the idea of promoting Stack Overflow as a place to ask questions over this list, as long as the questions fit SO’s guidelines ( how-to-ask http://stackoverflow.com/help/how-to-ask, dont-ask http://stackoverflow.com/help/dont-ask)? The apache-spark

Re: Discourse: A proposed alternative to the Spark User list

2015-01-21 Thread Nicholas Chammas
I think a few things need to be laid out clearly: 1. This mailing list is the “official” user discussion platform. That is, it is sponsored and managed by the ASF. 2. Users are free to organize independent discussion platforms focusing on Spark, and there is already one such platform

Standardized Spark dev environment

2015-01-20 Thread Nicholas Chammas
What do y'all think of creating a standardized Spark development environment, perhaps encoded as a Vagrantfile, and publishing it under `dev/`? The goal would be to make it easier for new developers to get started with all the right configs and tools pre-installed. If we use something like

Re: Standardized Spark dev environment

2015-01-20 Thread Nicholas Chammas
Message - From: Nicholas Chammas nicholas.cham...@gmail.com To: Spark dev list dev@spark.apache.org Sent: Tuesday, January 20, 2015 6:13:31 PM Subject: Standardized Spark dev environment What do y'all think of creating a standardized Spark development environment, perhaps encoded

Re: pyspark sc.textFile uses only 4 out of 32 threads per node

2015-01-20 Thread Nicholas Chammas
anyone reproduce it with v1.1? On Wed, Dec 17, 2014 at 2:14 AM, Nicholas Chammas nicholas.cham...@gmail.com wrote: Rui is correct. Check how many partitions your RDD has after loading the gzipped files. e.g. rdd.getNumPartitions(). If that number is way less than the number of cores

[jira] [Commented] (SPARK-5246) spark/spark-ec2.py cannot start Spark master in VPC if local DNS name does not resolve

2015-01-18 Thread Nicholas Chammas (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-5246?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14282107#comment-14282107 ] Nicholas Chammas commented on SPARK-5246: - [~shivaram] - Should this issue

[jira] [Comment Edited] (SPARK-1532) provide option for more restrictive firewall rule in ec2/spark_ec2.py

2015-01-18 Thread Nicholas Chammas (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-1532?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14282103#comment-14282103 ] Nicholas Chammas edited comment on SPARK-1532 at 1/19/15 3:15 AM

[jira] [Updated] (SPARK-3244) Add fate sharing across related files in Jenkins

2015-01-18 Thread Nicholas Chammas (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3244?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nicholas Chammas updated SPARK-3244: Component/s: (was: Spark Core) Project Infra Add fate sharing across

[jira] [Commented] (SPARK-3244) Add fate sharing across related files in Jenkins

2015-01-18 Thread Nicholas Chammas (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3244?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14282105#comment-14282105 ] Nicholas Chammas commented on SPARK-3244: - [~andrewor14] - I updated

[jira] [Resolved] (SPARK-2396) Spark EC2 scripts fail when trying to log in to EC2 instances

2015-01-18 Thread Nicholas Chammas (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-2396?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nicholas Chammas resolved SPARK-2396. - Resolution: Cannot Reproduce Resolving this issue as Cannot Reproduce. Feel free

Re: Cluster hangs in 'ssh-ready' state using Spark 1.2 EC2 launch script

2015-01-18 Thread Nicholas Chammas
Nathan, I posted a bunch of questions for you as a comment on your question http://stackoverflow.com/q/28002443/877069 on Stack Overflow. If you answer them (don't forget to @ping me) I may be able to help you. Nick On Sat Jan 17 2015 at 3:49:54 PM gen tang gen.tan...@gmail.com wrote: Hi,

[jira] [Commented] (SPARK-5298) Spark not starting on EC2 using spark-ec2

2015-01-17 Thread Nicholas Chammas (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-5298?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14281540#comment-14281540 ] Nicholas Chammas commented on SPARK-5298: - Ah, I found the issue. You have

[jira] [Resolved] (SPARK-5298) Spark not starting on EC2 using spark-ec2

2015-01-17 Thread Nicholas Chammas (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-5298?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nicholas Chammas resolved SPARK-5298. - Resolution: Invalid I'm resolving this as invalid. If you believe this is incorrect

[jira] [Commented] (SPARK-5299) Is http://www.apache.org/dist/spark/KEYS out of date?

2015-01-17 Thread Nicholas Chammas (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-5299?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14281511#comment-14281511 ] Nicholas Chammas commented on SPARK-5299: - cc [~pwendell] Is http

[jira] [Commented] (SPARK-5298) Spark not starting on EC2 using spark-ec2

2015-01-17 Thread Nicholas Chammas (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-5298?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14281514#comment-14281514 ] Nicholas Chammas commented on SPARK-5298: - A few questions for you: 1. What

[jira] [Commented] (SPARK-5298) Spark not starting on EC2 using spark-ec2

2015-01-17 Thread Nicholas Chammas (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-5298?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14281575#comment-14281575 ] Nicholas Chammas commented on SPARK-5298: - Yes, {{mesos/spark-ec2

Re: Discourse: A proposed alternative to the Spark User list

2015-01-17 Thread Nicholas Chammas
The Stack Exchange community will not support creating a whole new site just for Spark (otherwise you’d see dedicated sites for much larger topics like “Python”). Their tagging system works well enough to separate questions about different topics, and the apache-spark

Re: dockerized spark executor on mesos?

2015-01-15 Thread Nicholas Chammas
The AMPLab maintains a bunch of Docker files for Spark here: https://github.com/amplab/docker-scripts Hasn't been updated since 1.0.0, but might be a good starting point. On Wed Jan 14 2015 at 12:14:13 PM Josh J joshjd...@gmail.com wrote: We have dockerized Spark Master and worker(s)

[jira] [Commented] (SPARK-3821) Develop an automated way of creating Spark images (AMI, Docker, and others)

2015-01-14 Thread Nicholas Chammas (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3821?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14277104#comment-14277104 ] Nicholas Chammas commented on SPARK-3821: - Hmm, I doubt that was intentional since

[jira] [Updated] (SPARK-1805) Error launching cluster when master and slaves machines are of different visualization types

2015-01-13 Thread Nicholas Chammas (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-1805?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nicholas Chammas updated SPARK-1805: Issue Type: Bug (was: Improvement) Error launching cluster when master and slaves

[jira] [Updated] (SPARK-1805) Error launching cluster when master and slave machines are of different virtualization types

2015-01-13 Thread Nicholas Chammas (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-1805?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nicholas Chammas updated SPARK-1805: Description: In the current EC2 script, the AMI image object is loaded only once

[jira] [Commented] (SPARK-3821) Develop an automated way of creating Spark images (AMI, Docker, and others)

2015-01-13 Thread Nicholas Chammas (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3821?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14276471#comment-14276471 ] Nicholas Chammas commented on SPARK-3821: - [~shivaram] Are we ready to open a PR

[jira] [Updated] (SPARK-3185) SPARK launch on Hadoop 2 in EC2 throws Tachyon exception when Formatting JOURNAL_FOLDER

2015-01-13 Thread Nicholas Chammas (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3185?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nicholas Chammas updated SPARK-3185: Description: {code} org.apache.hadoop.ipc.RemoteException: Server IPC version 7 cannot

[jira] [Commented] (SPARK-3821) Develop an automated way of creating Spark images (AMI, Docker, and others)

2015-01-13 Thread Nicholas Chammas (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3821?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14276411#comment-14276411 ] Nicholas Chammas commented on SPARK-3821: - Hi [~florianverhein] and thanks

[jira] [Commented] (SPARK-3821) Develop an automated way of creating Spark images (AMI, Docker, and others)

2015-01-12 Thread Nicholas Chammas (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3821?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14274535#comment-14274535 ] Nicholas Chammas commented on SPARK-3821: - That's correct. All those paths

[jira] [Commented] (SPARK-5008) Persistent HDFS does not recognize EBS Volumes

2015-01-11 Thread Nicholas Chammas (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-5008?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14273007#comment-14273007 ] Nicholas Chammas commented on SPARK-5008: - Use [{{copy-dir}}|https://github.com

[jira] [Commented] (SPARK-3821) Develop an automated way of creating Spark images (AMI, Docker, and others)

2015-01-11 Thread Nicholas Chammas (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3821?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14273187#comment-14273187 ] Nicholas Chammas commented on SPARK-3821: - Updated launch stats: * Launching

[jira] [Commented] (SPARK-1422) Add scripts for launching Spark on Google Compute Engine

2015-01-11 Thread Nicholas Chammas (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-1422?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14273197#comment-14273197 ] Nicholas Chammas commented on SPARK-1422: - [~pwendell] - I would consider doing

[jira] [Commented] (SPARK-1555) enable ec2/spark_ec2.py to stop/delete cluster non-interactively

2015-01-10 Thread Nicholas Chammas (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-1555?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14272597#comment-14272597 ] Nicholas Chammas commented on SPARK-1555: - [~joshrosen]'s workaround is precisely

[jira] [Resolved] (SPARK-5086) Specifying the master instance type

2015-01-10 Thread Nicholas Chammas (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-5086?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nicholas Chammas resolved SPARK-5086. - Resolution: Duplicate Thanks for reporting this. I believe this is a duplicate of [SPARK

[jira] [Commented] (SPARK-4399) Support multiple cloud providers

2015-01-10 Thread Nicholas Chammas (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-4399?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14272589#comment-14272589 ] Nicholas Chammas commented on SPARK-4399: - Now that [Spark Packages|http://spark

[jira] [Commented] (SPARK-1204) EC2 scripts upload private key

2015-01-10 Thread Nicholas Chammas (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-1204?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14272600#comment-14272600 ] Nicholas Chammas commented on SPARK-1204: - cc [~shivaram] Do you know off the top

[jira] [Commented] (SPARK-1302) httpd doesn't start in spark-ec2 (cc2.8xlarge)

2015-01-10 Thread Nicholas Chammas (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-1302?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14272605#comment-14272605 ] Nicholas Chammas commented on SPARK-1302: - There is an open PR by [~fredcons] here

[jira] [Commented] (SPARK-5008) Persistent HDFS does not recognize EBS Volumes

2015-01-10 Thread Nicholas Chammas (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-5008?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14272585#comment-14272585 ] Nicholas Chammas commented on SPARK-5008: - cc [~shivaram] [~brdwrd] - What

[jira] [Commented] (SPARK-1422) Add scripts for launching Spark on Google Compute Engine

2015-01-10 Thread Nicholas Chammas (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-1422?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14272590#comment-14272590 ] Nicholas Chammas commented on SPARK-1422: - We have [a package for launching Spark

[jira] [Commented] (SPARK-1532) provide option for more restrictive firewall rule in ec2/spark_ec2.py

2015-01-10 Thread Nicholas Chammas (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-1532?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14272601#comment-14272601 ] Nicholas Chammas commented on SPARK-1532: - I believe this capability is now

[jira] [Resolved] (SPARK-2528) Default spark-ec2 security group permissions are too open

2015-01-10 Thread Nicholas Chammas (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-2528?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nicholas Chammas resolved SPARK-2528. - Resolution: Duplicate Default spark-ec2 security group permissions are too open

[jira] [Commented] (SPARK-4778) PySpark Json and groupByKey broken

2015-01-10 Thread Nicholas Chammas (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-4778?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14272587#comment-14272587 ] Nicholas Chammas commented on SPARK-4778: - Also, though the environment is on EC2

[jira] [Updated] (SPARK-4778) PySpark Json and groupByKey broken

2015-01-10 Thread Nicholas Chammas (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-4778?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nicholas Chammas updated SPARK-4778: Component/s: (was: EC2) PySpark Json and groupByKey broken

[jira] [Resolved] (SPARK-2649) EC2: Ganglia-httpd broken on hvm based machines like r3.4xlarge

2015-01-10 Thread Nicholas Chammas (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-2649?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nicholas Chammas resolved SPARK-2649. - Resolution: Duplicate This issue appears to be a duplicate of [SPARK-1302]. Feel free

[jira] [Resolved] (SPARK-4778) PySpark Json and groupByKey broken

2015-01-10 Thread Nicholas Chammas (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-4778?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nicholas Chammas resolved SPARK-4778. - Resolution: Cannot Reproduce PySpark Json and groupByKey broken

[jira] [Commented] (SPARK-2396) Spark EC2 scripts fail when trying to log in to EC2 instances

2015-01-10 Thread Nicholas Chammas (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-2396?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14272596#comment-14272596 ] Nicholas Chammas commented on SPARK-2396: - [~enraged_ginger] - Can you confirm

[jira] [Commented] (SPARK-1555) enable ec2/spark_ec2.py to stop/delete cluster non-interactively

2015-01-10 Thread Nicholas Chammas (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-1555?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14272615#comment-14272615 ] Nicholas Chammas commented on SPARK-1555: - The other thing to consider, if we

[jira] [Created] (SPARK-5189) Reorganize EC2 scripts so that nodes can be provisioned independent of Spark master

2015-01-10 Thread Nicholas Chammas (JIRA)
Nicholas Chammas created SPARK-5189: --- Summary: Reorganize EC2 scripts so that nodes can be provisioned independent of Spark master Key: SPARK-5189 URL: https://issues.apache.org/jira/browse/SPARK-5189

[jira] [Commented] (SPARK-5189) Reorganize EC2 scripts so that nodes can be provisioned independent of Spark master

2015-01-10 Thread Nicholas Chammas (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-5189?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14272660#comment-14272660 ] Nicholas Chammas commented on SPARK-5189: - cc [~joshrosen] and [~shivaram] - What

[jira] [Resolved] (SPARK-1204) EC2 scripts upload private key

2015-01-10 Thread Nicholas Chammas (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-1204?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nicholas Chammas resolved SPARK-1204. - Resolution: Fixed Fix Version/s: 0.9.0 Resolving this issue per Shivaram's comment

[jira] [Commented] (SPARK-5008) Persistent HDFS does not recognize EBS Volumes

2015-01-10 Thread Nicholas Chammas (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-5008?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14272846#comment-14272846 ] Nicholas Chammas commented on SPARK-5008: - Though I'm not too familiar

[jira] [Commented] (SPARK-2004) QA Automation

2015-01-09 Thread Nicholas Chammas (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-2004?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14272373#comment-14272373 ] Nicholas Chammas commented on SPARK-2004: - I recently had a [related discussion

[jira] [Commented] (SPARK-4924) Factor out code to launch Spark applications into a separate library

2015-01-09 Thread Nicholas Chammas (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-4924?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14272338#comment-14272338 ] Nicholas Chammas commented on SPARK-4924: - Okie doke. I ask because I've been

[jira] [Commented] (SPARK-4924) Factor out code to launch Spark applications into a separate library

2015-01-09 Thread Nicholas Chammas (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-4924?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14272315#comment-14272315 ] Nicholas Chammas commented on SPARK-4924: - [~vanzin] - How does this proposal

Re: Accidental kill in UI

2015-01-09 Thread Nicholas Chammas
As Sean said, this definitely sounds like something worth a JIRA issue (and PR). On Fri Jan 09 2015 at 8:17:34 AM Sean Owen so...@cloudera.com wrote: (FWIW yes I think this should certainly be a POST. The link can become a miniature form to achieve this and then the endpoint just needs to

Re: Results of tests

2015-01-09 Thread Nicholas Chammas
Just created: Integrate Python unit tests into Jenkins https://issues.apache.org/jira/browse/SPARK-5178 Nick On Fri Jan 09 2015 at 2:48:48 PM Josh Rosen rosenvi...@gmail.com wrote: The Test Result pages for Jenkins builds shows some nice statistics for the test run, including individual

[jira] [Created] (SPARK-5178) Integrate Python unit tests into Jenkins

2015-01-09 Thread Nicholas Chammas (JIRA)
Nicholas Chammas created SPARK-5178: --- Summary: Integrate Python unit tests into Jenkins Key: SPARK-5178 URL: https://issues.apache.org/jira/browse/SPARK-5178 Project: Spark Issue Type

[jira] [Commented] (SPARK-5178) Integrate Python unit tests into Jenkins

2015-01-09 Thread Nicholas Chammas (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-5178?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14271858#comment-14271858 ] Nicholas Chammas commented on SPARK-5178: - Relevant links: * http

Re: Spark development with IntelliJ

2015-01-08 Thread Nicholas Chammas
Side question: Should this section https://cwiki.apache.org/confluence/display/SPARK/Contributing+to+Spark#ContributingtoSpark-IDESetup in the wiki link to Useful Developer Tools https://cwiki.apache.org/confluence/display/SPARK/Useful+Developer+Tools? On Thu Jan 08 2015 at 6:19:55 PM Sean Owen

[jira] [Commented] (SPARK-3431) Parallelize Scala/Java test execution

2015-01-08 Thread Nicholas Chammas (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3431?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14270324#comment-14270324 ] Nicholas Chammas commented on SPARK-3431: - Generic update: * For those

[jira] [Created] (SPARK-5161) Parallelize Python test execution

2015-01-08 Thread Nicholas Chammas (JIRA)
Nicholas Chammas created SPARK-5161: --- Summary: Parallelize Python test execution Key: SPARK-5161 URL: https://issues.apache.org/jira/browse/SPARK-5161 Project: Spark Issue Type

<    9   10   11   12   13   14   15   16   17   18   >