Re: Welcoming three new committers

2015-02-03 Thread Joseph Bradley
Thanks to everyone in the community for past collaborations, and I look forward to continuing in the future! Joseph On Tue, Feb 3, 2015 at 6:23 PM, Shixiong Zhu zsxw...@gmail.com wrote: Congrats guys! Best Regards, Shixiong Zhu 2015-02-04 6:34 GMT+08:00 Matei Zaharia

ASF Git / GitHub sync is down

2015-02-03 Thread Reynold Xin
Haven't sync-ed anything for the last 4 hours. Seems like this little piece of infrastructure always stops working around our own code freeze time ...

Re: Welcoming three new committers

2015-02-03 Thread Manish Amde
Congratulations Cheng, Joseph and Sean. On Tuesday, February 3, 2015, Zhan Zhang zzh...@hortonworks.com wrote: Congratulations! On Feb 3, 2015, at 2:34 PM, Matei Zaharia matei.zaha...@gmail.com javascript:; wrote: Hi all, The PMC recently voted to add three new committers: Cheng Lian,

Re: Welcoming three new committers

2015-02-03 Thread Ye Xianjin
Congratulations! -- Ye Xianjin Sent with Sparrow (http://www.sparrowmailapp.com/?sig) On Wednesday, February 4, 2015 at 6:34 AM, Matei Zaharia wrote: Hi all, The PMC recently voted to add three new committers: Cheng Lian, Joseph Bradley and Sean Owen. All three have been major

Re: Welcoming three new committers

2015-02-03 Thread Zhan Zhang
Congratulations! On Feb 3, 2015, at 2:34 PM, Matei Zaharia matei.zaha...@gmail.com wrote: Hi all, The PMC recently voted to add three new committers: Cheng Lian, Joseph Bradley and Sean Owen. All three have been major contributors to Spark in the past year: Cheng on Spark SQL, Joseph on

Re: ASF Git / GitHub sync is down

2015-02-03 Thread Reynold Xin
I filed an INFRA ticket: https://issues.apache.org/jira/browse/INFRA-9115 I wish ASF can reconsider requests like this in order to handle downtime gracefully https://issues.apache.org/jira/browse/INFRA-8738 On Tue, Feb 3, 2015 at 9:09 PM, Reynold Xin r...@databricks.com wrote: Haven't

Re: Welcoming three new committers

2015-02-03 Thread Shixiong Zhu
Congrats guys! Best Regards, Shixiong Zhu 2015-02-04 6:34 GMT+08:00 Matei Zaharia matei.zaha...@gmail.com: Hi all, The PMC recently voted to add three new committers: Cheng Lian, Joseph Bradley and Sean Owen. All three have been major contributors to Spark in the past year: Cheng on Spark

Re: Welcoming three new committers

2015-02-03 Thread prabeesh k
Congratulations! On 4 February 2015 at 02:34, Matei Zaharia matei.zaha...@gmail.com wrote: Hi all, The PMC recently voted to add three new committers: Cheng Lian, Joseph Bradley and Sean Owen. All three have been major contributors to Spark in the past year: Cheng on Spark SQL, Joseph on

Re: SparkSubmit.scala and stderr

2015-02-03 Thread Marcelo Vanzin
Hi Jay, On Tue, Feb 3, 2015 at 6:28 AM, jayhutfles jayhutf...@gmail.com wrote: // Exposed for testing private[spark] var printStream: PrintStream = System.err But as the comment states that it's for testing, maybe I'm misunderstanding its intent... The comment is there to tell

Re: SparkSubmit.scala and stderr

2015-02-03 Thread Reynold Xin
We can use ScalaTest's privateMethodTester also instead of exposing that. On Tue, Feb 3, 2015 at 2:18 PM, Marcelo Vanzin van...@cloudera.com wrote: Hi Jay, On Tue, Feb 3, 2015 at 6:28 AM, jayhutfles jayhutf...@gmail.com wrote: // Exposed for testing private[spark] var printStream:

Re: Jenkins install reference

2015-02-03 Thread shane knapp
here's the wiki describing the system setup: https://cwiki.apache.org/confluence/display/SPARK/Spark+QA+Infrastructure we have 1 master and 8 worker nodes, 12 executors per worker (we'd be better off w/more and smaller worker nodes however). you don't need to install sbt -- it's in the build/

Re: [spark-sql] JsonRDD

2015-02-03 Thread Daniil Osipov
Thanks Reynold, Case sensitivity issues are definitely orthogonal. I'll submit a bug or PR. Is there a way to rename the object to eliminate the confusion? Not sure how locked down the API is at this time, but it seems like a potential confusion point for developers. On Mon, Feb 2, 2015 at 4:30

Re: Accessing indices and values in SparseVector

2015-02-03 Thread Sean Owen
When you are describing an error, you should say what the error is. Here I'm pretty sure it says there is no such member of Vector, right? You explicitly made the type of sv2 Vector and not SparseVector, and the trait does not have any indices member. No it's not a problem, and I think the

Re: Accessing indices and values in SparseVector

2015-02-03 Thread Manoj Kumar
Alright, thanks for the quick clarification.

Accessing indices and values in SparseVector

2015-02-03 Thread Manoj Kumar
Hello, This is related to one of the issues that I'm working on. I am not sure if this is expected behavior or not. This works fine. val sv2 = new SparseVector(3, Array(0, 2), Array(1.1, 3.0)) sv2.indices But when I do this val sv2: Vector = Vectors.sparse(3, Array(0, 2), Array(1.1, 3.0))

Re: Can spark provide an option to start reduce stage early?

2015-02-03 Thread Kay Ousterhout
There's a JIRA tracking this here: https://issues.apache.org/jira/browse/SPARK-2387 On Mon, Feb 2, 2015 at 9:48 PM, Xuelin Cao xuelincao2...@gmail.com wrote: In hadoop MR, there is an option *mapred.reduce.slowstart.completed.maps* which can be used to start reducer stage when X% mappers are

Re: [VOTE] Release Apache Spark 1.2.1 (RC3)

2015-02-03 Thread Dirceu Semighini Filho
Hi Patrick, I work in an Startup and we want make one of our projects as open source. This project is based on Spark, and it will help users to instantiate spark clusters in a cloud environment. But for that project we need to use the repl, hive and thrift-server. Can the decision of not

Re: [VOTE] Release Apache Spark 1.2.1 (RC3)

2015-02-03 Thread Nicholas Chammas
I believe this was changed for 1.2.1. Here are the relevant JIRA issues https://issues.apache.org/jira/browse/SPARK-5289?jql=project%20%3D%20SPARK%20AND%20fixVersion%20%3D%201.2.1%20AND%20text%20~%20%22publish%22%20order%20by%20priority . On Tue Feb 03 2015 at 10:43:59 AM Dirceu Semighini Filho

SparkSubmit.scala and stderr

2015-02-03 Thread jayhutfles
Hi all, I just saw that the SparkSubmit.scala class has the following lines: object SparkSubmit { ... // Exposed for testing private[spark] var printStream: PrintStream = System.err ... } This causes all verbose logging messages elsewhere in SparkSubmit to go to stderr, not

Re: [VOTE] Release Apache Spark 1.2.1 (RC3)

2015-02-03 Thread Chip Senkbeil
+1 Tested the REPL release against the Spark Kernel project (compilation/testing/manual execution). Everything still checks out fine. Signed, Chip Senkbeil IBM Emerging Technologies Software Engineer On Tue Feb 03 2015 at 12:50:12 PM Nicholas Chammas nicholas.cham...@gmail.com wrote: I

[ANNOUNCE] branch-1.3 has been cut

2015-02-03 Thread Patrick Wendell
Hey All, Just wanted to announce that we've cut the 1.3 branch which will become the 1.3 release after community testing. There are still some features that will go in (in higher level libraries, and some stragglers in spark core), but overall this indicates the end of major feature development

Re: Welcoming three new committers

2015-02-03 Thread Ted Yu
Congratulations, Cheng, Joseph and Sean. On Tue, Feb 3, 2015 at 2:53 PM, Nicholas Chammas nicholas.cham...@gmail.com wrote: Congratulations guys! On Tue Feb 03 2015 at 2:36:12 PM Matei Zaharia matei.zaha...@gmail.com wrote: Hi all, The PMC recently voted to add three new committers:

Welcoming three new committers

2015-02-03 Thread Matei Zaharia
Hi all, The PMC recently voted to add three new committers: Cheng Lian, Joseph Bradley and Sean Owen. All three have been major contributors to Spark in the past year: Cheng on Spark SQL, Joseph on MLlib, and Sean on ML and many pieces throughout Spark Core. Join me in welcoming them as

Re: Welcoming three new committers

2015-02-03 Thread Nicholas Chammas
Congratulations guys! On Tue Feb 03 2015 at 2:36:12 PM Matei Zaharia matei.zaha...@gmail.com wrote: Hi all, The PMC recently voted to add three new committers: Cheng Lian, Joseph Bradley and Sean Owen. All three have been major contributors to Spark in the past year: Cheng on Spark SQL,

Re: Welcoming three new committers

2015-02-03 Thread Hari Shreedharan
Congrats Cheng, Joseph and Owen! Well done! Thanks, Hari On Tue, Feb 3, 2015 at 2:55 PM, Ted Yu yuzhih...@gmail.com wrote: Congratulations, Cheng, Joseph and Sean. On Tue, Feb 3, 2015 at 2:53 PM, Nicholas Chammas nicholas.cham...@gmail.com wrote: Congratulations guys! On Tue Feb 03

RE: Welcoming three new committers

2015-02-03 Thread Pritish Nawlakhe
Congrats and welcome back!! Thank you!! Regards Pritish Nirvana International Inc. Big Data, Hadoop, Oracle EBS and IT Solutions VA - SWaM, MD - MBE Certified Company prit...@nirvana-international.com http://www.nirvana-international.com Twitter: @nirvanainternat -Original

Re: Welcoming three new committers

2015-02-03 Thread Timothy Chen
Congrats all! Tim On Feb 4, 2015, at 7:10 AM, Pritish Nawlakhe prit...@nirvana-international.com wrote: Congrats and welcome back!! Thank you!! Regards Pritish Nirvana International Inc. Big Data, Hadoop, Oracle EBS and IT Solutions VA - SWaM, MD - MBE Certified Company

Re: Welcoming three new committers

2015-02-03 Thread Evan Chan
Congrats everyone!!! On Tue, Feb 3, 2015 at 3:17 PM, Timothy Chen tnac...@gmail.com wrote: Congrats all! Tim On Feb 4, 2015, at 7:10 AM, Pritish Nawlakhe prit...@nirvana-international.com wrote: Congrats and welcome back!! Thank you!! Regards Pritish Nirvana International Inc.

Re: SparkSubmit.scala and stderr

2015-02-03 Thread Evan Chan
Why not just use SLF4J? On Tue, Feb 3, 2015 at 2:22 PM, Reynold Xin r...@databricks.com wrote: We can use ScalaTest's privateMethodTester also instead of exposing that. On Tue, Feb 3, 2015 at 2:18 PM, Marcelo Vanzin van...@cloudera.com wrote: Hi Jay, On Tue, Feb 3, 2015 at 6:28 AM,

Re: IDF for ml pipeline

2015-02-03 Thread masaki rikitoku
Thank you for your reply. I will do it. — Mailbox から送信 On Tue, Feb 3, 2015 at 6:12 PM, Xiangrui Meng men...@gmail.com wrote: Yes, we need a wrapper under spark.ml. Feel free to create a JIRA for it. -Xiangrui On Mon, Feb 2, 2015 at 8:56 PM, masaki rikitoku rikima3...@gmail.com wrote: Hi

Re: Welcoming three new committers

2015-02-03 Thread Corey Nolet
Congrats guys! On Tue, Feb 3, 2015 at 7:01 PM, Evan Chan velvia.git...@gmail.com wrote: Congrats everyone!!! On Tue, Feb 3, 2015 at 3:17 PM, Timothy Chen tnac...@gmail.com wrote: Congrats all! Tim On Feb 4, 2015, at 7:10 AM, Pritish Nawlakhe prit...@nirvana-international.com

Re: Welcoming three new committers

2015-02-03 Thread Xuefeng Wu
Congratulations!well done. Yours, Xuefeng Wu 吴雪峰 敬上 On 2015年2月4日, at 上午6:34, Matei Zaharia matei.zaha...@gmail.com wrote: Hi all, The PMC recently voted to add three new committers: Cheng Lian, Joseph Bradley and Sean Owen. All three have been major contributors to Spark in the past

Re: IDF for ml pipeline

2015-02-03 Thread Xiangrui Meng
Yes, we need a wrapper under spark.ml. Feel free to create a JIRA for it. -Xiangrui On Mon, Feb 2, 2015 at 8:56 PM, masaki rikitoku rikima3...@gmail.com wrote: Hi all I am trying the ml pipeline for text classfication now. recently, i succeed to execute the pipeline processing in ml

Re: SparkSubmit.scala and stderr

2015-02-03 Thread Sean Owen
Despite its name, stderr is frequently used as the destination for anything that's not the output of the program, which includes log messages. That way, for example, you can redirect the output of such a program to capture its result without also capturing log or error messages, which will still

Re: Welcoming three new committers

2015-02-03 Thread Nan Zhu
Congratulations! -- Nan Zhu http://codingcat.me On Tuesday, February 3, 2015 at 8:08 PM, Xuefeng Wu wrote: Congratulations!well done. Yours, Xuefeng Wu 吴雪峰 敬上 On 2015年2月4日, at 上午6:34, Matei Zaharia matei.zaha...@gmail.com (mailto:matei.zaha...@gmail.com) wrote: Hi all,

Re: Welcoming three new committers

2015-02-03 Thread Mridul Muralidharan
Congratulations ! Keep up the good work :-) Regards Mridul On Tuesday, February 3, 2015, Matei Zaharia matei.zaha...@gmail.com wrote: Hi all, The PMC recently voted to add three new committers: Cheng Lian, Joseph Bradley and Sean Owen. All three have been major contributors to Spark in

Re: 2GB limit for partitions?

2015-02-03 Thread Mridul Muralidharan
That is fairly out of date (we used to run some of our jobs on it ... But that is forked off 1.1 actually). Regards Mridul On Tuesday, February 3, 2015, Imran Rashid iras...@cloudera.com wrote: Thanks for the explanations, makes sense. For the record looks like this was worked on a while

Re: Welcoming three new committers

2015-02-03 Thread Chao Chen
Congratulations guys, well done! 在 15-2-4 上午9:26, Nan Zhu 写道: Congratulations! -- Nan Zhu http://codingcat.me On Tuesday, February 3, 2015 at 8:08 PM, Xuefeng Wu wrote: Congratulations!well done. Yours, Xuefeng Wu 吴雪峰 敬上 On 2015年2月4日, at 上午6:34, Matei Zaharia matei.zaha...@gmail.com

Re: Welcoming three new committers

2015-02-03 Thread Denny Lee
Awesome stuff - congratulations! :) On Tue Feb 03 2015 at 5:34:06 PM Chao Chen crazy...@gmail.com wrote: Congratulations guys, well done! 在 15-2-4 上午9:26, Nan Zhu 写道: Congratulations! -- Nan Zhu http://codingcat.me On Tuesday, February 3, 2015 at 8:08 PM, Xuefeng Wu wrote:

Re: Welcoming three new committers

2015-02-03 Thread Debasish Das
Congratulations ! Keep helping the community :-) On Tue, Feb 3, 2015 at 5:34 PM, Denny Lee denny.g@gmail.com wrote: Awesome stuff - congratulations! :) On Tue Feb 03 2015 at 5:34:06 PM Chao Chen crazy...@gmail.com wrote: Congratulations guys, well done! 在 15-2-4 上午9:26, Nan Zhu