lots of test warning messages from UISeleniumSuite

2015-05-13 Thread Reynold Xin
Was looking at a PR test log just now. Can somebody take a look and remove the warnings (or just hide them)? 15/05/13 01:49:35 INFO UISeleniumSuite: Trying to start HiveThriftServer2: port=13125, mode=binary, attempt=0 15/05/13 01:50:28 INFO UISeleniumSuite: HiveThriftServer2 started

Re: [build system] brief downtime tomorrow morning (5-12-15, 7am PDT)

2015-05-13 Thread shane knapp
this is already done On Tue, May 12, 2015 at 1:14 PM, shane knapp skn...@berkeley.edu wrote: i will need to restart jenkins to finish a plugin install and resolve https://issues.apache.org/jira/browse/SPARK-7561 this will be very brief, and i'll retrigger any errant jobs i kill. please let

Re: How to link code pull request with JIRA ID?

2015-05-13 Thread Ted Yu
Subproject tag should follow SPARK JIRA number. e.g. [SPARK-5277][SQL] ... Cheers On Wed, May 13, 2015 at 11:50 AM, Stephen Boesch java...@gmail.com wrote: following up from Nicholas, it is [SPARK-12345] Your PR description where 12345 is the jira number. One thing I tend to forget is

Re: How to link code pull request with JIRA ID?

2015-05-13 Thread Stephen Boesch
following up from Nicholas, it is [SPARK-12345] Your PR description where 12345 is the jira number. One thing I tend to forget is when/where to include the subproject tag e.g. [MLLIB] 2015-05-13 11:11 GMT-07:00 Nicholas Chammas nicholas.cham...@gmail.com: That happens automatically when

Re: Task scheduling times

2015-05-13 Thread Reynold Xin
Maybe JIT? The 1st stage -- the scheduler code isn't JITed yet. On Wed, May 13, 2015 at 9:18 AM, Akshat Aranya aara...@gmail.com wrote: Hi, Any input on this? I'm willing to instrument further and experiment if there are any ideas. On Mon, May 4, 2015 at 11:27 AM, Akshat Aranya

[build system] scheduled datacenter downtime, sunday may 17th

2015-05-13 Thread shane knapp
our datacenter is rejiggering our network (read: fully re-engineering large portions from the ground up) and has downtime scheduled from 9am-3pm PDT, this sunday may17th. this means our jenkins instance will not be available to the outside world, and i will be putting jenkins in to quiet mode the

Re: How to link code pull request with JIRA ID?

2015-05-13 Thread Markus Weimer
Hi, how did you set this up? Over in the REEF incubation project, we painstakingly create the forwards- and backwards links despite having the IDs in the PR descriptions... Thanks! Markus On 2015-05-13 11:56, Ted Yu wrote: Subproject tag should follow SPARK JIRA number. e.g.

Re: [PySpark DataFrame] When a Row is not a Row

2015-05-13 Thread Nicholas Chammas
Is there some way around this? For example, can Row just be an implementation of namedtuple throughout? from collections import namedtuple class Row(namedtuple): ... From a user perspective, it’s confusing that there are 2 different implementations of the Row class with the same name. In my

Re: s3 vfs on Mesos Slaves

2015-05-13 Thread Akhil Das
Did you happened to have a look at this https://github.com/abashev/vfs-s3 Thanks Best Regards On Tue, May 12, 2015 at 11:33 PM, Stephen Carman scar...@coldlight.com wrote: We have a small mesos cluster and these slaves need to have a vfs setup on them so that the slaves can pull down the data

Re: s3 vfs on Mesos Slaves

2015-05-13 Thread jay vyas
Might I ask why vfs? I'm new to vfs and not sure wether or not it predates the hadoop file system interfaces (HCFS). After all spark natively supports any HCFS by leveraging the hadoop FileSystem api and class loaders and so on. So simply putting those resources on your classpath should be

Re: @since version tag for all dataframe/sql methods

2015-05-13 Thread Nicholas Chammas
Are we not doing the same thing for the Python API? On Wed, May 13, 2015 at 10:43 AM Olivier Girardot ssab...@gmail.com wrote: that's a great idea ! Le mer. 13 mai 2015 à 07:38, Reynold Xin r...@databricks.com a écrit : I added @since version tag for all public dataframe/sql

Re: Task scheduling times

2015-05-13 Thread Akshat Aranya
Hi, Any input on this? I'm willing to instrument further and experiment if there are any ideas. On Mon, May 4, 2015 at 11:27 AM, Akshat Aranya aara...@gmail.com wrote: Hi, I have been investigating scheduling delays in Spark and I found some unexplained anomalies. In my use case, I have two

Re: [IMPORTANT] Committers please update merge script

2015-05-13 Thread Patrick Wendell
Hi All - unfortunately the fix introduced another bug, which is that fixVersion was not updated properly. I've updated the script and had one other person test it. So committers please pull from master again thanks! - Patrick On Tue, May 12, 2015 at 6:25 PM, Patrick Wendell pwend...@gmail.com

Re: Change for submitting to yarn in 1.3.1

2015-05-13 Thread Patrick Wendell
Hey Chester, Thanks for sending this. It's very helpful to have this list. The reason we made the Client API private was that it was never intended to be used by third parties programmatically and we don't intend to support it in its current form as a stable API. We thought the fact that it was

Re: Change for submitting to yarn in 1.3.1

2015-05-13 Thread Chester @work
Patrick Thanks for responding. Yes. many of are features requests not private client related. These are the things I have been working with since last year. I have trying to push the PR for these changes. If the new Launcher lib is the way to go , we will try to work with new APIs.

How to link code pull request with JIRA ID?

2015-05-13 Thread Chandrashekhar Kotekar
Hi, I am new to open source contribution and trying to understand the process starting from pulling code to uploading patch. I have managed to pull code from GitHub. In JIRA I saw that each JIRA issue is connected with pull request. I would like to know how do people attach pull request details

Re: How to link code pull request with JIRA ID?

2015-05-13 Thread Nicholas Chammas
There's no magic to it. We're doing the same, except Josh automated it in the PR dashboard he created. https://spark-prs.appspot.com/ Nick On Wed, May 13, 2015 at 6:20 PM Markus Weimer mar...@weimo.de wrote: Hi, how did you set this up? Over in the REEF incubation project, we painstakingly