Re: Thoughts on release cadence?

I think we need to up date the FEATURE section on the version policy page to match.  It says feature releases are every 4 months. TomOn Monday, July 31, 2017, 2:23:10 PM CDT, Sean Owen wrote: Done at https://spark.apache.org/versioning-policy.html On Mon, Jul 31, 2017 at

Re: spark pypy support?

update that to 2.5+ since we aren't testing with 2.3 anymore? On Mon, Aug 14, 2017 at 3:09 PM, Tom Graves <tgraves...@yahoo.com.invalid> wrote: I tried 5.7 and 2.5.1 so its probably something in my setup.  I'll investigate that more, wanted to make sure it was still supported because

Re: spark pypy support?

yspark-sql', 'pyspark-streaming'] >> >> Starting test(python2.7): pyspark.mllib.tests >> >> Starting test(pypy): pyspark.sql.tests >> >> Starting test(pypy): pyspark.tests >> >> Starting test(pypy): pyspark.streaming.tests >> >> Finished test(pypy

spark pypy support?

Anyone know if pypy works with spark. Saw a jira that it was supported back in Spark 1.2 but getting an error when trying and not sure if its something with my pypy version of just something spark doesn't support. AttributeError: 'builtin-code' object has no attribute 'co_filename' Traceback

Re: [VOTE] [SPIP] SPARK-18085: Better History Server scalability

+1.  Tom On Monday, July 31, 2017, 12:28:02 PM CDT, Marcelo Vanzin wrote: Hey all, Following the SPIP process, I'm putting this SPIP up for a vote. It's been open for comments as an SPIP for about 3 weeks now, and had been open without the SPIP label for about 9 months

PR permission to kick Jenkins?

Does  anyone know how to configure Jenkins to allow committers to tell it to test prs?  I used to have this access but lately it is either not working or only intermittently working. The commands like "ok to test", "test this please", etc.. Thanks,Tom

Re: [VOTE] Apache Spark 2.1.1 (RC4)

+1 Tom Graves On Thursday, April 27, 2017 5:37 PM, vaquar khan <vaquar.k...@gmail.com> wrote: +1  Regards, Vaquar khan On Apr 27, 2017 4:11 PM, "Holden Karau" <hol...@pigscanfly.ca> wrote: +1 (non-binding) PySpark packaging issue from the earlier RC seems to ha

Re: planning & discussion for larger scheduler changes

filed [SPARK-20178] Improve Scheduler fetch failures - ASF JIRA | | | | || | | | | | [SPARK-20178] Improve Scheduler fetch failures - ASF JIRA | | | | Tom On Thursday, March 30, 2017 1:21 PM, Tom Graves <tgraves...@yahoo.com> wrote:

Re: planning & discussion for larger scheduler changes

imilar things that could be done in other parts of the scheduler. Tom's comments re: (2) are more about performance improvements rather than readability / testability / debuggability, but also seem important and it does seem useful to have a JIRA tracking those. -Kay On Mon, Mar 27, 2017 at 11:06 A

Re: planning & discussion for larger scheduler changes

1) I think this depends on individual case by case jira.  I haven't looked in detail at spark-14649 seems much larger although more the way I think we want to go. While SPARK-13669 seems less risky and easily configurable. 2) I don't know whether it needs an entire rewrite but I think there need

Re: Spark Improvement Proposals

Another thing I think you should send out is when exactly does this take affect.  Is it any major new feature without a pull request?   Is it anything major starting with the 2.3 release?   Tom On Monday, March 13, 2017 1:08 PM, Tom Graves <tgraves...@yahoo.com.INVALID> wrote:

Re: Spark Improvement Proposals

en around a long time with no further comment, and I called several times for more input. That's pretty strong lazy consensus of the form we use every day.  On Mon, Mar 13, 2017 at 5:30 PM Tom Graves <tgraves...@yahoo.com> wrote: It seems like if you are adding responsibilities you sho

Re: Spark Improvement Proposals

change, instead of precipitating a meta-vote.However, the text that's on the web site now can certainly be further amended if anyone wants to propose a change from here. On Mon, Mar 13, 2017 at 1:50 PM Tom Graves <tgraves...@yahoo.com> wrote: I think a vote here would be good.

Re: Spark Improvement Proposals

I think a vote here would be good. I think most of the discussion was done by 4 or 5 people and its a long thread.  If nothing else it summarizes everything and gets people attention to the change. Tom On Thursday, March 9, 2017 10:55 AM, Sean Owen wrote: I think a

Re: [VOTE] Release Apache Spark 2.0.1 (RC4)

+1 Tom On Wednesday, September 28, 2016 9:15 PM, Reynold Xin wrote: Please vote on releasing the following candidate as Apache Spark version 2.0.1. The vote is open until Sat, Oct 1, 2016 at 20:00 PDT and passes if a majority of at least 3+1 PMC votes are cast. [

Re: [discuss] Spark 2.x release cadence

+1 to 4 months. Tom On Tuesday, September 27, 2016 2:07 PM, Reynold Xin wrote: We are 2 months past releasing Spark 2.0.0, an important milestone for the project. Spark 2.0.0 deviated (took 6 month from the regular release cadence we had for the 1.x line, and we

Re: [discuss] separate API annotation into two components: InterfaceAudience & InterfaceStability

ping, did this discussion conclude or did we decide what we are doing? Tom On Friday, May 13, 2016 3:19 PM, Michael Armbrust wrote: +1 to the general structure of Reynold's proposal.  I've found what we do currently a little confusing.  In particular, it

Re: [DISCUSS] Minimize use of MINOR, BUILD, and HOTFIX w/ no JIRA

s a JIRA. Also: we have some hot-fixes here that aren't connected to JIRAs. Either they belong with an existing JIRA and aren't tagged correctly, or, again, are patching changes that weren't really trivial enough to skip a JIRA to begin with. On Thu, Jul 7, 2016 at 7:47 PM, Tom Graves <tgrave

Re: [DISCUSS] Minimize use of MINOR, BUILD, and HOTFIX w/ no JIRA

Popping this back up to the dev list again.  I see a bunch of checkins with minor or hotfix.   It seems to me we shouldn't be doing this, but I would like to hear thoughts from others.  I see no reason we can't have a jira for each of those issues, it only takes a few seconds to file one and it

Re: cutting 1.6.2 rc and 2.0.0 rc this week?

+1 Tom On Wednesday, June 15, 2016 2:01 PM, Reynold Xin wrote: It's been a while and we have accumulated quite a few bug fixes in branch-1.6. I'm thinking about cutting 1.6.2 rc this week. Any patches somebody want to get in last minute? On a related note, I'm

Re: Spark 2.0.0-preview artifacts still not available in Maven

On Tue, Jun 7, 2016 at 4:01 PM, Tom Graves <tgraves...@yahoo.com> wrote: > I just checked and I don't see the 2.0 preview release at all anymore on > .http://spark.apache.org/downloads.html, is it in transition?    The only > place I can see it is at > http://spark.apache.org/news/

Re: Spark 2.0.0-preview artifacts still not available in Maven

The documentation for the preview release also seem to be missing? Also what happens if we want to do a second preview release?  The naming doesn't seem to allow then unless we call it preview 2. Tom On Wednesday, June 1, 2016 6:27 PM, Sean Owen wrote: On Wed, Jun

Re: [VOTE] Removing module maintainer process

+1 (binding) Tom On Sunday, May 22, 2016 7:34 PM, Matei Zaharia wrote: It looks like the discussion thread on this has only had positive replies, so I'm going to call a VOTE. The proposal is to remove the maintainer process in

Re: [DISCUSS] Removing or changing maintainer process

+1 (binding) Tom On Thursday, May 19, 2016 10:35 AM, Matei Zaharia wrote: Hi folks, Around 1.5 years ago, Spark added a maintainer process for reviewing API and architectural changes

Re: [discuss] separate API annotation into two components: InterfaceAudience & InterfaceStability

So we definitely need to be careful here.  I know you didn't mention it but it mentioned by others so I would not recommend using LimitedPrivate.  I had started a discussion on Hadoop about some of this due to the way Spark needed to use some of the

Re: YARN Shuffle service and its compatibility

It would be nice if we could keep this compatible between 1.6 and 2.0 so I'm more for Option B at this point since the change made seems minor and we can change to have shuffle service do internally like Marcelo mention. Then lets try to keep compatible, but if there is a forcing function lets

Re: [discuss] ending support for Java 7 in Spark 2.0

Steve, those are good points, I had forgotten Hadoop had those issues.    We run with jdk 8, hadoop is built for jdk7 compatibility, we are running hadoop 2.7 on our clusters and by the time Spark 2.0 is out I would expected a mix of Hadoop 2.7 and 2.8.  We also don't use spnego. I didn't quite

Re: [discuss] ending support for Java 7 in Spark 2.0

+1. Tom On Tuesday, March 29, 2016 1:17 PM, Reynold Xin wrote: They work. On Tue, Mar 29, 2016 at 10:01 AM, Koert Kuipers wrote: if scala prior to sbt 2.10.4 didn't support java 8, does that mean that 3rd party scala libraries compiled with a

Re: A proposal for Spark 2.0

Do we have a summary of all the discussions and what is planned for 2.0 then?   Perhaps we should put on the wiki for reference. Tom On Tuesday, December 22, 2015 12:12 AM, Reynold Xin wrote: FYI I updated the master branch's Spark version to 2.0.0-SNAPSHOT.  On

Re: [VOTE] Release Apache Spark 1.6.0 (RC3)

+1.  Ran some regression tests on Spark on Yarn (hadoop 2.6 and 2.7). Tom On Wednesday, December 16, 2015 3:32 PM, Michael Armbrust wrote: Please vote on releasing the following candidate as Apache Spark version 1.6.0! The vote is open until Saturday, December

Re: [VOTE] Release Apache Spark 1.5.2 (RC2)

 While running our regression tests I found  https://issues.apache.org/jira/browse/SPARK-11555.  It is a break in backwards compatibility but its using the old spark-class and --num-workers interface which I hope no one is still using.   I'm a +0 as it doesn't seem super critical but I hate to

Re: [VOTE] Release Apache Spark 1.5.2 (RC2)

t work at > all on YARN unless dynamic allocation is on? the fix is easy, but > sounds like it could be a Blocker. > > On Fri, Nov 6, 2015 at 2:51 PM, Tom Graves <tgraves...@yahoo.com> wrote: >>  While running our regression tests I found >> https://issues.apach

Re: [ANNOUNCE] Announcing Spark 1.5.1

I know there are multiple things being talked about here, but  I agree with Patrick here, we vote on the source distribution - src tarball (and of course the tag should match).  Perhaps in principle we vote on all the other specific binary distributions since they are generated from source

Re: [VOTE] Release Apache Spark 1.5.1 (RC1)

+1. Tested Spark on Yarn on Hadoop 2.6 and 2.7. Tom On Thursday, September 24, 2015 2:34 AM, Reynold Xin wrote: Please vote on releasing the following candidate as Apache Spark version 1.5.1. The vote is open until Sun, Sep 27, 2015 at 10:00 UTC and passes if

Re: [VOTE] Release Apache Spark 1.5.0 (RC3)

+1. Tested on Yarn with Hadoop 2.6.  A few of the things tested: pyspark, hive integration, aux shuffle handler, history server, basic submit cli behavior, distributed cache behavior, cluster and client mode... Tom On Tuesday, September 1, 2015 3:42 PM, Reynold Xin

Re: [VOTE] Release Apache Spark 1.5.0 (RC1)

Is there a jira to update the sql hive docs?Spark SQL and DataFrames - Spark 1.5.0 Documentation |   | |   |   |   |   |   | | Spark SQL and DataFrames - Spark 1.5.0 DocumentationSpark SQL and DataFrame Guide Overview DataFrames Starting Point: SQLContext Creating DataFrames DataFrame

Re: [VOTE] Release Apache Spark 1.5.0 (RC1)

On Tuesday, August 25, 2015 1:56 PM, Tom Graves tgraves...@yahoo.com.INVALID wrote: Is there a jira to update the sql hive docs?Spark SQL and DataFrames - Spark 1.5.0 Documentation |   | |   |   |   |   |   | | Spark SQL and DataFrames - Spark 1.5.0 DocumentationSpark SQL and DataFrame

Re: [VOTE] Release Apache Spark 1.4.1 (RC4)

+1 Tom On Thursday, July 9, 2015 12:55 AM, Patrick Wendell pwend...@gmail.com wrote: Please vote on releasing the following candidate as Apache Spark version 1.4.1! This release fixes a handful of known issues in Spark 1.4.0, listed here: http://s.apache.org/spark-1.4.1 The tag to

Re: [VOTE] Release Apache Spark 1.4.1

+1. Tested on yarn on hadoop 2.6 cluster Tom On Monday, June 29, 2015 2:04 AM, Tathagata Das tathagata.das1...@gmail.com wrote: @Ted, could you elaborate more on what was the test command that you ran? What profiles, using SBT or Maven?  TD On Sun, Jun 28, 2015 at 12:21 PM, Patrick

Re: [VOTE] Release Apache Spark 1.4.1

So is this open for vote then or are we waiting on other things? Tom On Thursday, June 25, 2015 10:32 AM, Andrew Ash and...@andrewash.com wrote: I would guess that many tickets targeted at 1.4.1 were set that way during the tail end of the 1.4.0 voting process as people realized

kryo version?

Hey folks, I had a customer ask about updating the version of kryo to get fix:  https://github.com/EsotericSoftware/kryo/pull/164 which is in 2.23.Spark currently pull sin chill 0.5.0 which pulls in kryo 2.21.  I don't see a newer version of chill that has updated to kryo 2.23.   Anyone familiar

Re: [discuss] ending support for Java 6?

...@databricks.com wrote: OK I sent an email. On Tue, May 5, 2015 at 2:47 PM, shane knapp skn...@berkeley.edu wrote: +1 to an announce to user and dev.  java6 is so old and sad. On Tue, May 5, 2015 at 2:24 PM, Tom Graves tgraves...@yahoo.com wrote: +1. I haven't seen major objections here so

Re: [discuss] ending support for Java 6?

+1. I haven't seen major objections here so I would say send announcement and see if any users have objections Tom On Tuesday, May 5, 2015 5:09 AM, Patrick Wendell pwend...@gmail.com wrote: If there is broad consensus here to drop Java 1.6 in Spark 1.5, should we do an ANNOUNCE to

Spark Sql reading hive partitioned tables?

Hey, I was trying out spark sql using the HiveContext and doing a select on a partitioned table with lots of partitions (16,000+). It took over 6 minutes before it even started the job. It looks like it was querying the Hive metastore and got a good chunk of data back.  Which I'm guessing is

Re: [VOTE] Release Apache Spark 1.3.1 (RC2)

+1. Tested spark on yarn against hadoop 2.6. Tom On Wednesday, April 8, 2015 6:15 AM, Sean Owen so...@cloudera.com wrote: Still a +1 from me; same result (except that now of course the UISeleniumSuite test does not fail) On Wed, Apr 8, 2015 at 1:46 AM, Patrick Wendell

Re: [VOTE] Release Apache Spark 1.3.0 (RC1)

Trying to run pyspark on yarn in client mode with basic wordcount example I see the following error when doing the collect: Error from python worker:  /usr/bin/python: No module named sqlPYTHONPATH was: 

Re: [VOTE] Release Apache Spark 1.2.0 (RC2)

+1 built and tested on Yarn on Hadoop 2.x cluster. Tom On Saturday, December 13, 2014 12:48 AM, Denny Lee denny.g@gmail.com wrote: +1 Tested on OSX Tested Scala 2.10.3, SparkSQL with Hive 0.12 / Hadoop 2.5, Thrift Server, MLLib SVD On Fri Dec 12 2014 at 8:57:16 PM Mark Hamstra

Re: [VOTE] Release Apache Spark 1.2.0 (RC1)

+1 tested on yarn. Tom On Friday, November 28, 2014 11:18 PM, Patrick Wendell pwend...@gmail.com wrote: Please vote on releasing the following candidate as Apache Spark version 1.2.0! The tag to be voted on is v1.2.0-rc1 (commit 1056e9ec1):

Re: [VOTE] Designating maintainers for some Spark components

+1. Tom On Wednesday, November 5, 2014 9:21 PM, Matei Zaharia matei.zaha...@gmail.com wrote: BTW, my own vote is obviously +1 (binding). Matei On Nov 5, 2014, at 5:31 PM, Matei Zaharia matei.zaha...@gmail.com wrote: Hi all, I wanted to share a discussion we've been having on

Re: RFC: Deprecating YARN-alpha API's

Any other comments or objections on this? Thanks,Tom On Tuesday, September 9, 2014 4:39 PM, Chester Chen ches...@alpinenow.com wrote: We were using it until recently, we are talking to our customers and see if we can get off it. Chester Alpine Data Labs On Tue, Sep 9, 2014 at

Re: Spark authenticate enablement

Spark authentication does work in standalone mode (atleast it did, I haven't tested it in a while). The same shared secret has to be set on all the daemons (master and workers) and then also in the configs of any applications submitted. Since everyone shares the same secret its by no means

Re: [VOTE] Release Apache Spark 1.1.0 (RC4)

+1. Ran spark on yarn on hadoop 0.23 and 2.x. Tom On Wednesday, September 3, 2014 2:25 AM, Patrick Wendell pwend...@gmail.com wrote: Please vote on releasing the following candidate as Apache Spark version 1.1.0! The tag to be voted on is v1.1.0-rc4 (commit 2f9b2bd):

Re: [VOTE] Release Apache Spark 1.0.1 (RC2)

+1. Ran some Spark on yarn jobs on a hadoop 2.4 cluster with authentication on. Tom On Friday, July 4, 2014 2:39 PM, Patrick Wendell pwend...@gmail.com wrote: Please vote on releasing the following candidate as Apache Spark version 1.0.1! The tag to be voted on is v1.0.1-rc1 (commit

Re: [VOTE] Release Apache Spark 1.0.0 (RC11)

Testing... Resending as it appears my message didn't go through last week. Tom On Wednesday, May 28, 2014 4:12 PM, Tom Graves tgraves...@yahoo.com wrote: +1. Tested spark on yarn (cluster mode, client mode, pyspark, spark-shell) on hadoop 0.23 and 2.4.  Tom On Wednesday, May 28, 2014 3

Re: [VOTE] Release Apache Spark 1.0.0 (RC11)

+1. Tested spark on yarn (cluster mode, client mode, pyspark, spark-shell) on hadoop 0.23 and 2.4.  Tom On Wednesday, May 28, 2014 3:07 PM, Sean McNamara sean.mcnam...@webtrends.com wrote: Pulled down, compiled, and tested examples on OS X and ubuntu. Deployed app we are building on spark

Re: [VOTE] Release Apache Spark 1.0.0 (RC10)

Has anyone tried pyspark on yarn and got it to work?  I was having issues when I built spark on redhat but when I built on my mac it had worked,  but now when I build it on my mac it also doesn't work. Tom On Tuesday, May 20, 2014 3:14 PM, Tathagata Das tathagata.das1...@gmail.com wrote:

Re: [VOTE] Release Apache Spark 1.0.0 (RC10)

I don't think Kevin's issue would be with an api change in YarnClientImpl since in both cases he says he is using hadoop 2.3.0.  I'll take a look at his post in the user list. Tom On Wednesday, May 21, 2014 7:01 PM, Colin McCabe cmcc...@alumni.cmu.edu wrote: Hi Kevin, Can you try

Re: [VOTE] Release Apache Spark 1.0.0 (rc9)

I assume we will have an rc10 to fix the issues Matei found? Tom On Sunday, May 18, 2014 9:08 PM, Patrick Wendell pwend...@gmail.com wrote: Hey Matei - the issue you found is not related to security. This patch a few days ago broke builds for Hadoop 1 with YARN support enabled. The patch

Re: [VOTE] Release Apache Spark 1.0.0 (rc9)

no ideas off hand, I'll take a look tomorrow. Tom On Sunday, May 18, 2014 7:28 PM, Matei Zaharia matei.zaha...@gmail.com wrote: Alright, I’ve opened https://github.com/apache/spark/pull/819 with the Windows fixes. I also found one other likely bug,

Re: [VOTE] Release Apache Spark 0.9.1 (RC3)

I put up a pull request with documentation changes  https://github.com/apache/spark/pull/314   Tom On Wednesday, April 2, 2014 8:47 AM, Tom Graves tgraves...@yahoo.com wrote: Note I'm +1 with the doc changed to tell users to export SPARK_YARN_MODE=true before using spark-shell on yarn. I

Re: [VOTE] Release Apache Spark 0.9.1 (RC3)

Note I'm +1 with the doc changed to tell users to export SPARK_YARN_MODE=true before using spark-shell on yarn. I tested it on both hadoop 0.23 and 2.3 clusters using secure hdfs on linux. Tom On Tuesday, April 1, 2014 1:44 PM, Tom Graves tgraves...@yahoo.com wrote: No one else has reported

Re: [VOTE] Release Apache Spark 0.9.1 (RC3)

I should probably pull this off into another thread, but going forward can we try to not have the release votes end on a weekend? Since we only seem to give 3 days, it makes it really hard for anyone who is offline for the weekend to try it out.   Either that or extend the voting for more then

Re: Spark 0.9.1 release

Thanks for the heads up, saw that and will make sure that is resolved before pulling into 0.9.  Unless I'm missing something, they should just use sc.addJar to distributed the jar rather then relying on SPARK_YARN_APP_JAR. Tom On Thursday, March 20, 2014 3:31 PM, Patrick Wendell

cloudera repo down again - mqtt

It appears the cloudera repo for the mqtt stuff is down again.  Did someone  ping them the last time?   Can we pick this up from some other repo? [ERROR] Failed to execute goal org.apache.maven.plugins:maven-remote-resources-plugin:1.4:process (default) on project spark-examples_2.10: Error

Re: cloudera repo down again - mqtt

| London On Fri, Mar 14, 2014 at 7:37 AM, Tom Graves tgraves...@yahoo.com wrote: It appears the cloudera repo for the mqtt stuff is down again. Did someone  ping them the last time? Can we pick this up from some other repo? [ERROR] Failed to execute goal org.apache.maven.plugins:maven

Re: YARN Maven build questions

the default ones and at least one of the documented ones fail. Cheers, Lars On Fri, Feb 28, 2014 at 3:05 PM, Tom Graves tgraves...@yahoo.com wrote: what build command are you using?    What do you mean when you say YARN branch? The yarn builds have been working fine for me with maven

<    1   2