Re: Spark 1.3.1 / Hadoop 2.6 package has broken S3 access

epath=org/apache/hadoop/hadoop-aws/2.6.0/hadoop-aws-2.6.0.jar > > And add: > > export CLASSPATH=$CLASSPATH:hadoop-aws-2.6.0.jar > > And try to relaunch. > > Thanks, > Peter Rudenko > > > On 2015-05-07 19:30, Nicholas Chammas wrote: > > Hmm, I just tried chang

Re: Spark 1.3.1 / Hadoop 2.6 package has broken S3 access

sspath. Wich hadoop vendor (Hortonworks, Cloudera, MapR) do you use? > > Thanks, > Peter Rudenko > > On 2015-05-07 19:25, Nicholas Chammas wrote: > > Details are here: https://issues.apache.org/jira/browse/SPARK-7442 > > It looks like something specific to building against Hadoop 2.6? > > Nick > > > >

Spark 1.3.1 / Hadoop 2.6 package has broken S3 access

Details are here: https://issues.apache.org/jira/browse/SPARK-7442 It looks like something specific to building against Hadoop 2.6? Nick

Re: [discuss] ending support for Java 6?

:18 PM Nicholas Chammas wrote: > I understand the concern about cutting out users who still use Java 6, and > I don't have numbers about how many people are still using Java 6. > > But I want to say at a high level that I support deprecating older > versions of stuff to reduce

Re: [discuss] ending support for Java 6?

I understand the concern about cutting out users who still use Java 6, and I don't have numbers about how many people are still using Java 6. But I want to say at a high level that I support deprecating older versions of stuff to reduce our maintenance burden and let us use more modern patterns in

Re: Pandas' Shift in Dataframe

the data scientists I'm working with are using it, and it > would be therefore useful for me to translate Pandas code to Spark... > > Isn't the goal of Spark Dataframe to allow all the features of Pandas/R > Dataframe using Spark ? > > Regards, > > Olivier. > &g

Re: Pandas' Shift in Dataframe

You can check JIRA for any existing plans. If there isn't any, then feel free to create a JIRA and make the case there for why this would be a good feature to add. Nick On Wed, Apr 29, 2015 at 7:30 AM Olivier Girardot < o.girar...@lateral-thoughts.com> wrote: > Hi, > Is there any plan to add the

Re: github pull request builder FAIL, now WIN(-ish)

And unfortunately, many Jenkins executor slots are being taken by stale Spark PRs... On Mon, Apr 27, 2015 at 2:25 PM shane knapp wrote: > anyways, the build queue is SLAMMED... we're going to need at least a day > to catch up w/this. i'll be keeping an eye on system loads and whatnot all > day

Re: Design docs: consolidation and discoverability

, I like your idea of keeping it in a separate git repository. It > seems to combine the advantages of the present Google Docs approach with > the crisper history, discoverability, and text format simplicity of GitHub > wikis. > > Punya > On Mon, Apr 27, 2015 at 1:30 PM Nicholas

Re: Design docs: consolidation and discoverability

I like the idea of having design docs be kept up to date and tracked in git. If the Apache repo isn't a good fit, perhaps we can have a separate repo just for design docs? Maybe something like github.com/spark-docs/spark-docs/ ? If there's other stuff we want to track but haven't, perhaps we can

Re: Should we let everyone set Assignee?

To repeat what Patrick said (literally): If an issue is “assigned” to person X, but some other person Y submits a great patch for it, I think we have some obligation to Spark users and to the community to merge the better patch. So the idea of reserving the right to add a feature, it just seems ov

Re: Spark build time

I suggest searching the archives for this list as there were several previous discussions about this problem. JIRA also has several issues related to this. Some pointers: - SPARK-3431 : Parallelize Scala/Java test execution - http://a

Re: Is spark-ec2 for production use?

environments so > things can be validated post build/deploy and enhance CI process so if you > choose to deploy via bigtop in test/prod/etc you know things have gone > through a certain amount of rigor beforehand > > Nate > > -Original Message- > From: Patrick Wend

Is spark-ec2 for production use?

Is spark-ec2 intended for spinning up production Spark clusters? I think the answer is no. However, the docs for spark-ec2 very much leave that possibility open, and indeed I see many people asking questions or opening issues that stem from

Gitter chat room for Spark

Would we be interested in having a public chat room? Gitter offers them for free for open source projects. It's like web-based IRC. Check out the Docker room for example: https://gitter.im/docker/docker And if people prefer to use actual IRC, Gitter offers a bridge for that <

Re: wait time between start master and start slaves

utput /dev/null \ --write-out "%{http_code}" localhost:8080 )"done spark/sbin/start-slaves.sh Turns out that the master typically takes 3-4 seconds to come up. That’s 15 seconds saved. Hurray for yak shaving! Nick ​ On Sun, Apr 12, 2015 at 5:56 PM Nicholas Chammas wrote:

Re: Fwd: [jira] [Commented] (SPARK-6889) Streamline contribution process with update to Contribution wiki, JIRA rules

t; > planning to date and what can we learn? > > > > This overlaps with other discussion on SPARK-6889 but per Nicholas > > wanted to surface this > > > > -- Forwarded message -- > > From: Nicholas Chammas (JIRA) > > Date: Tue, Apr 14, 201

Re: Streamline contribution process with update to Contribution wiki, JIRA rules

Wow, I had an open email draft to whine (yet again) about our open PR count and provide some suggestions. Will redirect that to the JIRA Sean created. Sweet! Nick On Mon, Apr 13, 2015 at 7:05 PM Patrick Wendell wrote: > Would just like to encourage everyone who is active in day-to-day > develo

Re: wait time between start master and start slaves

, SparkUI.DEFAULT_PORT) > } > Better retrieve effective UI port before probing. > > Cheers > > On Sat, Apr 11, 2015 at 2:38 PM, Nicholas Chammas < > nicholas.cham...@gmail.com> wrote: > >> So basically, to tell if the master is ready to accept slaves, just poll >>

Re: wait time between start master and start slaves

f a good > way to check if the master is up though. I guess we could poll the Master > Web UI and see if we get a 200/ok response > > Shivaram > > On Fri, Apr 10, 2015 at 8:24 PM, Nicholas Chammas < > nicholas.cham...@gmail.com> wrote: > >> Check this o

wait time between start master and start slaves

Check this out (from spark-ec2): # Start Master$BIN_FOLDER/start-master.sh # Pause sleep 20 # Start Workers$BIN_FOLDER/start-slaves.sh I know this was probably done defensively, bu

Re: Guidance for becoming Spark contributor

Have you reviewed this guide? https://cwiki.apache.org/confluence/display/SPARK/Contributing+to+Spark Nick On Fri, Apr 10, 2015 at 7:29 PM Nitin Mathur wrote: > Hi Spark Dev Team, > > I want to start contributing to Spark Open source. This is the first time I > will be doing any open source co

Re: Contributor CLAs

You would have a very hard time arguing that you made a > contribution, didn't state anything about the license, but did not > intend somehow that the work could be licensed as the rest of the > project is. For reference Apache projects do not in general require a > CLA. > >

Contributor CLAs

I've seen many other OSS projects ask contributors to sign CLAs. I've never seen us do that. I assume it's not an issue, since people opening PRs generally understand what it means. But legally I'm sure there's some danger in taking an implied vs. explicit license to do something. So: Do we need

Windows CI infra

I've seen other projects use Appveyor for CI on Windows. Has anyone used them before? I've seen on more than one occasion something break on Windows without us knowing, so it might be worth looking into using something like this if it's relatively straightforward. Nick

Test all the things (Was: Unit test logs in Jenkins?)

t; > On Thu, Apr 2, 2015 at 8:54 AM, Nicholas Chammas < > nicholas.cham...@gmail.com> wrote: > >> This is secondary to Marcelo’s question, but I wanted to comment on this: >> >> Its main limitation is more cultural than technical: you need to get >> people >>

Re: Unit test logs in Jenkins?

This is secondary to Marcelo’s question, but I wanted to comment on this: Its main limitation is more cultural than technical: you need to get people to care about intermittent test runs, otherwise you can end up with failures that nobody keeps on top of This is a problem that plagues Spark as we

Re: May we merge into branch-1.3 at this point?

Looks like the release is out: http://spark.apache.org/releases/spark-release-1-3-0.html Though, interestingly, I think we are missing the appropriate v1.3.0 tag: https://github.com/apache/spark/releases Nick On Fri, Mar 13, 2015 at 6:07 AM Sean Owen wrote: > Is the release certain enough that

PSA: Link to files at fixed version

*TL;DR*: Hit y on any file page on GitHub to update the URL to a permanent link. Many of you probably already know this. Here’s a handy tip for the rest. So you’re on Github and you want to link to a file in an email, PR, or JIRA report. Or better yet, you want to link to some specific lines in a

Re: spark-ec2 default to Hadoop 2

e should shift towards >> 2.x at least as defaults. >> >> On Sun, Mar 1, 2015 at 10:59 PM, Nicholas Chammas >> wrote: >> > https://github.com/apache/spark/blob/fd8d283eeb98e310b1e85ef8c3a8af >> 9e547ab5e0/ec2/spark_ec2.py#L162-L164 >> > >> >

spark-ec2 default to Hadoop 2

https://github.com/apache/spark/blob/fd8d283eeb98e310b1e85ef8c3a8af9e547ab5e0/ec2/spark_ec2.py#L162-L164 Is there any reason we shouldn't update the default Hadoop major version in spark-ec2 to 2? Nick

Re: Some praise and comments on Spark

Thanks for sharing the feedback about what works well for you! It's nice to get that; as we all probably know, people generally reach out only when they have problems. On Wed, Feb 25, 2015 at 5:38 PM Reynold Xin wrote: > Thanks for the email and encouragement, Devl. Responses to the 3 requests:

Re: [jenkins infra -- pls read ] installing anaconda, moving default python from 2.6 -> 2.7

The first concern for Spark will probably be to ensure that we still build and test against Python 2.6, since that's the minimum version of Python we support. Otherwise this seems OK. We use numpy and other Python packages in PySpark, but I don't think we're pinned to any particular version of tho

Re: textFile() ordering and header rows

I guess on a technicality the docs just say "first item in this RDD", not "first line in the source text file". AFAIK there is no way apart from filtering to remove header lines . As long as first() always returns the same value for a given RDD, I think

Re: Improving metadata in Spark JIRA

sts to something that contains a few > actionable steps, instead of broad open-ended wishes > - Work on a fix. In an ideal world people are willing to work to resolve > JIRAs they open, and don't fire-and-forget > > > If everyone did this, not only would it advance the house-

Git Achievements

For fun: http://acha-acha.co/#/repo/https://github.com/apache/spark I just added Spark to this site. Some of these “achievements” are hilarious. Leo Tolstoy: More than 10 lines in a commit message Dangerous Game: Commit after 6PM friday Nick ​

Re: Improving metadata in Spark JIRA

others for the cleanup! Nick On Sat Feb 07 2015 at 8:29:42 PM Nicholas Chammas nicholas.cham...@gmail.com <http://mailto:nicholas.cham...@gmail.com> wrote: Oh derp, missed the YARN component. > > JIRA, does allow admins to make fields mandatory: > https://confluence.atlas

Re: Building Spark with Pants

FYI: Here is the matching discussion over on the Pants dev list. <https://groups.google.com/forum/#!topic/pants-devel/rTaU-iIOIFE> On Mon Feb 02 2015 at 4:50:33 PM Nicholas Chammas nicholas.cham...@gmail.com <http://mailto:nicholas.cham...@gmail.com> wrote: To reiterate, I'

Re: 1.2.1 start-all.sh broken?

following working > (against a directory with space in its name): > > #!/usr/bin/env bash > OLDIFS="$IFS" # save it > IFS="" # don't split on any white space > dir="$1/*" > for f in "$dir"; do > cat $f > done > IFS=$O

Re: 1.2.1 start-all.sh broken?

The tragic thing here is that I was asked to review the patch that introduced this <https://github.com/apache/spark/pull/3377#issuecomment-68077315>, and totally missed it... :( On Wed Feb 11 2015 at 2:46:35 PM Nicholas Chammas < nicholas.cham...@gmail.com> wrote: > lol yeah, I c

Re: 1.2.1 start-all.sh broken?

lol yeah, I changed the path for the email... turned out to be the issue itself. On Wed Feb 11 2015 at 2:43:09 PM Ted Yu wrote: > I see. > '/path/to/spark-1.2.1-bin-hadoop2.4' didn't contain space :-) > > On Wed, Feb 11, 2015 at 2:41 PM, Nicholas Chammas < >

Re: 1.2.1 start-all.sh broken?

Found it: https://github.com/apache/spark/compare/v1.2.0...v1.2.1#diff-73058f8e51951ec0b4cb3d48ade91a1fR73 GRRR BASH WORD SPLITTING My path has a space in it... Nick On Wed Feb 11 2015 at 2:37:39 PM Nicholas Chammas < nicholas.cham...@gmail.com> wrote: > This is what get: > >

Re: 1.2.1 start-all.sh broken?

amples-1.2.1-hadoop2.4.0.jar > > FYI > > On Wed, Feb 11, 2015 at 2:27 PM, Nicholas Chammas < > nicholas.cham...@gmail.com> wrote: > >> I just downloaded 1.2.1 pre-built for Hadoop 2.4+ and ran >> sbin/start-all.sh >> on my OS X. >> >> Failed t

1.2.1 start-all.sh broken?

I just downloaded 1.2.1 pre-built for Hadoop 2.4+ and ran sbin/start-all.sh on my OS X. Failed to find Spark assembly in /path/to/spark-1.2.1-bin-hadoop2.4/lib You need to build Spark before running this program. Did the same for 1.2.0 and it worked fine. Nick ​

numpy on PyPy - potential benefit to PySpark

Random question for the PySpark and Python experts/enthusiasts on here: How big of a deal would it be for PySpark and PySpark users if you could run numpy on PyPy? PySpark already supports running on PyPy , but libraries like MLlib that use numpy are not

Re: Keep or remove Debian packaging in Spark?

+1 to an "official" deprecation + redirecting users to some other project that will or already is taking this on. Nate? On Mon Feb 09 2015 at 10:08:27 AM Patrick Wendell wrote: > I have wondered whether we should sort of deprecated it more > officially, since otherwise I think people have the

Re: Using CUDA within Spark / boosting linear algebra

Lemme butt in randomly here and say there is an interesting discussion on this Spark PR about netlib-java, JBLAS, Breeze, and other things I know nothing of, that y'all may find interesting. Among the participants is the author of netlib-java. On Sun Feb

Re: Improving metadata in Spark JIRA

y merge them into Spark Core. > > On Fri, Feb 6, 2015 at 11:53 AM, Nicholas Chammas > wrote: > > Do we need some new components to be added to the JIRA project? > > > > Like: > > > >- > > > >scheduler > > - > >

Re: Improving metadata in Spark JIRA

we already have a YARN component. > > https://issues.apache.org/jira/issues/?jql=project%20% > 3D%20SPARK%20AND%20component%20%3D%20YARN > > I don't think JIRA allows it to be mandatory, but if it does, that > would be useful. > > On Sat, Feb 7, 2015 at 5:08 PM, Nicholas

Re: Improving metadata in Spark JIRA

Do we need some new components to be added to the JIRA project? Like: - scheduler - YARN - spark-submit - …? Nick ​ On Fri Feb 06 2015 at 10:50:41 AM Nicholas Chammas < nicholas.cham...@gmail.com> wrote: > +9000 on cleaning up JIRA. > > Thank you Sean for

Re: Improving metadata in Spark JIRA

+9000 on cleaning up JIRA. Thank you Sean for laying out some specific things to tackle. I will assist with this. Regarding email, I think Sandy is right. I only get JIRA email for issues I'm watching. Nick On Fri Feb 06 2015 at 9:52:58 AM Sandy Ryza wrote: > JIRA updates don't go to this lis

PSA: Maven supports parallel builds

Y’all may already know this, but I haven’t seen it mentioned anywhere in our docs on here and it’s a pretty easy win. Maven supports parallel builds with the -T command line option. For example: ./build/mvn -T 1C -Dha

Re: Welcoming three new committers

Congratulations guys! On Tue Feb 03 2015 at 2:36:12 PM Matei Zaharia wrote: > Hi all, > > The PMC recently voted to add three new committers: Cheng Lian, Joseph > Bradley and Sean Owen. All three have been major contributors to Spark in > the past year: Cheng on Spark SQL, Joseph on MLlib, and S

Re: [VOTE] Release Apache Spark 1.2.1 (RC3)

I believe this was changed for 1.2.1. Here are the relevant JIRA issues . On Tue Feb 03 2015 at 10:43:59 AM Dirceu Semighini Filho

Re: Building Spark with Pants

hare. On Mon Feb 02 2015 at 4:40:45 PM Nicholas Chammas < nicholas.cham...@gmail.com> wrote: > I'm asking from an experimental standpoint; this is not happening anytime > soon. > > Of course, if the experiment turns out very well, Pants would replace both > sbt and Mave

Re: Building Spark with Pants

sently > for sbt and with a little bit of tweaking with maven as well. > > 2015-02-02 16:25 GMT-08:00 Nicholas Chammas : > >> Does anyone here have experience with Pants >> > <http://pantsbuild.github.io/index.html> or interest in trying to build > > >> Sp

Building Spark with Pants

Does anyone here have experience with Pants or interest in trying to build Spark with it? Pants has an interesting story. It was born at Twitter to help them build their Scala, Java, and Python projects as several independent components in one monolithic re

Spark Master Maven with YARN build is broken

https://amplab.cs.berkeley.edu/jenkins/view/Spark/job/Spark-Master-Maven-with-YARN/HADOOP_PROFILE=hadoop-2.4,label=centos/ Is this is a known issue? It seems to have been broken since last night. Here’s a snippet from the build output of one of the builds

Re: [VOTE] Release Apache Spark 1.2.1 (RC2)

Do we have any open JIRA issues to add automated testing on Windows to Jenkins? I assume that's something we want to do. On Sat Jan 31 2015 at 10:37:42 PM Matei Zaharia wrote: > This looks like a pretty serious problem, thanks! Glad people are testing > on Windows. > > Matei > > > On Jan 31, 201

Re: Extending Scala style checks

Reynold Xin wrote: > Thanks. I added one. > > > On Wed, Oct 8, 2014 at 8:49 AM, Nicholas Chammas < > nicholas.cham...@gmail.com> wrote: > >> I've created SPARK-3849: Automate remaining Scala style rules >> > <https://issues.apache.org/jira/browse/SPARK-

Does spark-ec2 support Windows?

Is spark-ec2 supposed to run normally from Windows (e.g. to launch a cluster)? I ask because I don’t see mention of Windows anywhere in relation to spark-ec2, and there is an open PR that checks file permis

Re: Standardized Spark dev environment

t; > > best, > wb > > > - Original Message - > > From: "Nicholas Chammas" > > To: "Spark dev list" > > Sent: Tuesday, January 20, 2015 6:13:31 PM > > Subject: Standardized Spark dev environment > > > > What do y'all

Standardized Spark dev environment

What do y'all think of creating a standardized Spark development environment, perhaps encoded as a Vagrantfile, and publishing it under `dev/`? The goal would be to make it easier for new developers to get started with all the right configs and tools pre-installed. If we use something like Vagran

Re: Results of tests

Just created: "Integrate Python unit tests into Jenkins" https://issues.apache.org/jira/browse/SPARK-5178 Nick On Fri Jan 09 2015 at 2:48:48 PM Josh Rosen wrote: > The "Test Result" pages for Jenkins builds shows some nice statistics for > the test run, including individual test times: > > ht

Re: Spark development with IntelliJ

Side question: Should this section in the wiki link to Useful Developer Tools ? On Thu Jan 08 2015 at 6:19:55 PM Sean Owe

Re: Sample Spark Program Error

You sent this to the dev list. Please send it instead to the user list. We use the dev list to discuss development on Spark itself, new features, fixes to known bugs, and so forth. The user list is to discuss issues using Spark, which I believe is what you are looking for. Nick On Tue Dec 30 2

Re: ANNOUNCE: New build script ./build/mvn

Linkies for the curious: - SPARK-4501 : Create build/mvn to automatically download maven/zinc/scalac - https://github.com/apache/spark/pull/3707 - New build folder (mvn and sbt): https://github.com/apache/spark/tree/master/build Nic

SQL specification for reference during Spark SQL development

Do we have access to the SQL specification (say, SQL-92) for reference during Spark SQL development? I know it's not freely available on the web. Usually, you can only access drafts. I know that, generally, we look to other systems (especially Hive) when figuring out how something in Spark SQL sho

Re: Starting with Spark

t; https://github.com/apache/spark > > Search with "Build Spark with Maven" > > On Thu, Dec 25, 2014 at 1:49 AM, Nicholas Chammas < > nicholas.cham...@gmail.com> wrote: > >> The correct docs link is: >> https://spark.apache.org/docs/1.2.0/buildin

Re: Starting with Spark

The correct docs link is: https://spark.apache.org/docs/1.2.0/building-spark.html Where did you get that bad link from? Nick On Thu Dec 25 2014 at 12:00:53 AM Naveen Madhire wrote: > Hi All, > > I am starting to use Spark. I am having trouble getting the latest code > from git. > I am using I

Re: [ANNOUNCE] Requiring JIRA for inclusion in release credits

Does this include contributions made against the spark-ec2 repo? On Wed Dec 17 2014 at 12:29:19 AM Patrick Wendell wrote: > Hey All, > > Due to the very high volume of contributions, we're switching to an > automated process for generating release credits. Th

Re: Announcing Spark Packages

aken a close look > at this and I think we're in good shape her vis-a-vis this policy. > > - Patrick > > On Mon, Dec 22, 2014 at 5:29 PM, Nicholas Chammas > wrote: > > Hitesh, > > > > From your link: > > > > You may not use ASF trademarks such as &

Re: Announcing Spark Packages

Hitesh, >From your link : You may not use ASF trademarks such as “Apache” or “ApacheFoo” or “Foo” in your own domain names if that use would be likely to confuse a relevant consumer about the source of software or services provided through your web

Re: Handling stale PRs

:08 PM Nicholas Chammas < nicholas.cham...@gmail.com> wrote: > I recently came across this blog post, which reminded me of this thread. > > How to Discourage Open Source Contributions > <http://danluu.com/discourage-oss/> > > We are currently at 320+ open PRs, many of wh

Re: Scala's Jenkins setup looks neat

Shot down again. <https://issues.apache.org/jira/browse/INFRA-7367?focusedCommentId=14249382&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14249382> ​ On Tue Dec 16 2014 at 9:41:39 PM Nicholas Chammas < nicholas.cham...@gmail.com> wrote: > I see

Re: Scala's Jenkins setup looks neat

et: https://issues.apache.org/jira/browse/INFRA-7918 > > On Tue, Dec 16, 2014 at 6:23 PM, Nicholas Chammas < > nicholas.cham...@gmail.com> wrote: >> >> Actually, reading through the existing issue opened for this >> <https://issues.apache.org/jira/browse/INFRA-7367> back in F

Re: Scala's Jenkins setup looks neat

mission to code as well as statuses. > >> > >> As I understand it, ASF Infra has said no in the past to granting access > >> to statuses because it also granted push access. > >> > >> If so, this no longer appears to be the case. > >> > >&g

Re: Scala's Jenkins setup looks neat

with ASF Infra to give us OAuth keys with repo:status access? Nick On Sat Sep 06 2014 at 1:29:53 PM Nicholas Chammas < nicholas.cham...@gmail.com> wrote: Aww, that's a bummer... > > > On Sat, Sep 6, 2014 at 1:10 PM, Reynold Xin wrote: > >> that would require github

Re: Archiving XML test reports for analysis

e > XML for each build and just dump it there. > > what builds are we thinking about? spark pull request builder? what > others? > > On Mon, Dec 15, 2014 at 1:33 AM, Nicholas Chammas < > nicholas.cham...@gmail.com> wrote: >> >> Every time we run a test cycle

Re: Spark JIRA Report

> seem like a good way to tackle the wall of text effect that I'm worried > about. > > On Sun, Dec 14, 2014 at 12:28 PM, Nicholas Chammas < > nicholas.cham...@gmail.com> wrote: > >> Taking after Andrew’s suggestion, perhaps the report can just focus on >

Archiving XML test reports for analysis

Every time we run a test cycle on our Jenkins cluster, we generate hundreds of XML reports covering all the tests we have (e.g. `streaming/target/test-reports/org.apache.spark.streaming.util.WriteAheadLogSuite.xml`). These reports contain interesting information about whether tests succeeded or fa

Re: Spark JIRA Report

debug logging - [Dec 17, 2012] SPARK-636 <https://issues.apache.org/jira/browse/SPARK-636>: Add mechanism to run system management/configuration tasks on all workers Andrew, Does that seem more useful? Nick ​ On Sun Dec 14 2014 at 3:20:54 AM Nicholas Chammas < nicholas.cha

Re: Spark JIRA Report

rowse/SPARK-3561> Allow for > pluggable > execution contexts in Spark > > Andrew > > On Sat, Dec 13, 2014 at 11:31 PM, Nicholas Chammas < > nicholas.cham...@gmail.com> wrote: > >> What do y’all think of a report like this emailed out to the dev list on a >&g

Spark JIRA Report

What do y’all think of a report like this emailed out to the dev list on a monthly basis? The goal would be to increase visibility into our open issues and encourage developers to tend to our issue tracker more frequently. Nick There are 1,236 unresolved issues

Is Apache JIRA down?

For example: https://issues.apache.org/jira/browse/SPARK-3431 Where do we report/track issues with JIRA itself being down? Nick

Re: Is Apache JIRA down?

Nevermind, seems to be back up now. On Wed Dec 10 2014 at 7:46:30 PM Nicholas Chammas < nicholas.cham...@gmail.com> wrote: > For example: https://issues.apache.org/jira/browse/SPARK-3431 > > Where do we report/track issues with JIRA itself being down? > > Nick >

Re: Is this a little bug in BlockTransferMessage ?

think I have a solution for Maven on > SPARK-4159. > > On Tue, Dec 9, 2014 at 11:30 PM, Nicholas Chammas > wrote: > > So all this time the tests that Jenkins has been running via Jenkins and > SBT > > + ScalaTest... those haven't been running any of the Java unit tests

Re: Is this a little bug in BlockTransferMessage ?

So all this time the tests that Jenkins has been running via Jenkins and SBT + ScalaTest... those haven't been running any of the Java unit tests? SPARK-4159 only mentions Maven as a problem, but I'm wondering how these tests got through Jenkins O

Re: Handling stale PRs

ious things we (as contributors) could do to make the committers¹ > lives easier? Thanks! > > -Ilya > > On 12/8/14, 11:58 AM, "Nicholas Chammas" > wrote: > > >I recently came across this blog post, which reminded me of this thread. > > > >How to Di

Re: Handling stale PRs

s, would it help if an automated email went out to the dev list once a week that a) reported the number of stale PRs, and b) directly linked to the 5 least recently updated PRs? Nick On Sat Aug 30 2014 at 3:41:39 AM Nicholas Chammas < nicholas.cham...@gmail.com> wrote: > On Tue, Aug 26,

Re: Unit tests in < 5 minutes

Ted, I posted some updates on JIRA on my progress (or lack thereof) getting SBT to parallelize test suites properly. I'm currently stu

Re: Unit tests in < 5 minutes

I’d love to hear it (either on this thread or in the JIRA issue). Nick ​ On Sun Sep 07 2014 at 8:28:51 PM Nicholas Chammas < nicholas.cham...@gmail.com> wrote: > On Fri, Aug 8, 2014 at 1:12 PM, Reynold Xin wrote: > >> Nick, >> >> Would you like to file a tick

Re: zinc invocation examples

have to do > anything for each build. > > On Wed, Dec 3, 2014 at 3:44 PM, Nicholas Chammas > wrote: > > https://github.com/apache/spark/blob/master/docs/ > building-spark.md#speeding-up-compilation-with-zinc > > > > Could someone summarize how they invo

zinc invocation examples

https://github.com/apache/spark/blob/master/docs/building-spark.md#speeding-up-compilation-with-zinc Could someone summarize how they invoke zinc as part of a regular build-test-etc. cycle? I'll add it in to the aforelinked page if appropriate. Nick

Re: Spurious test failures, testing best practices

- currently the docs only contain information about building with maven, and even then don’t cover many important cases All other points aside, I just want to point out that the docs document both how to use Maven and SBT and clearly state

Re: Time taken to merge Spark PR's?

1.1.1 was just released, and 1.2 is close to a release. That, plus Thanksgiving in the US (where most Spark committers AFAIK are located), probably means a temporary lull in committer activity on non-critical items should be expected. On Mon Nov 24 2014 at 9:33:27 AM York, Brennon wrote: > All,

Re: Troubleshooting JVM OOM during Spark Unit Tests

gt; > On Fri, Nov 21, 2014 at 1:50 PM, Nicholas Chammas < > nicholas.cham...@gmail.com> wrote: > >> Howdy folks, >> >> I’m trying to understand why I’m getting “insufficient memory” errors when >> trying to run Spark Units tests within a CentOS Docke

Troubleshooting JVM OOM during Spark Unit Tests

Howdy folks, I’m trying to understand why I’m getting “insufficient memory” errors when trying to run Spark Units tests within a CentOS Docker container. I’m building Spark and running the tests as follows: # build sbt/sbt -Pyarn -Phadoop-2.3 -Dhadoop.version=2.3.0 -Pkinesis-asl -Phive -Phive-th

Re: mvn or sbt for studying and developing Spark?

The docs on using sbt are here: https://github.com/apache/spark/blob/master/docs/building-spark.md#building-with-sbt They'll be published with 1.2.0 presumably. On 2014년 11월 17일 (월) at 오후 2:49 Michael Armbrust wrote: > > > > * I moved from sbt to maven in June specifically due to Andrew Or's > >

Re: JIRA + PR backlog

gt; official docs or the wiki... > > On Tue, Nov 11, 2014 at 12:23 PM, Nicholas Chammas > wrote: > > Yeah, kudos to Josh for putting that together. > > > > On Tue, Nov 11, 2014 at 3:26 AM, Yu Ishikawa < > yuu.ishikawa+sp...@gmail.com> > > wrote: > > >

<    1   2   3   4   5   6   >