Re: Big-Endian (IBM Power7) Spark Serialization issue

2014-06-16 Thread Reynold Xin
Thanks for sending the update. Do you mind posting a link to the bug reported in the lzf project here as well? Cheers. On Sun, Jun 15, 2014 at 7:04 PM, gchen chenguanch...@gmail.com wrote: To anyone who is interested in this issue, the root cause if from a third party code

Re: Spark 1.1 Window and 1.0 Wrap-up

2014-06-16 Thread Bhaskar Dutta
On Tue, Jun 3, 2014 at 11:51 AM, Patrick Wendell pwend...@gmail.com wrote: Hey All, I wanted to announce the the Spark 1.1 release window: June 1 - Merge window opens July 25 - Cut-off for new pull requests August 1 - Merge window closes (code freeze), QA period starts August 15+ - RC's

Re: Big-Endian (IBM Power7) Spark Serialization issue

2014-06-16 Thread gchen
Hi Reynold, thanks for your interest on this issue. The work here is part of incorporating Spark into PowerLinux ecosystem. Here is the bug raised in ning by my colleague: https://github.com/ning/compress/issues/37 Would you mind to share whether some insights of Spark's support for Big Enidan

Re: Big-Endian (IBM Power7) Spark Serialization issue

2014-06-16 Thread Reynold Xin
I think you guys are / will be leading the effort on that :) On Mon, Jun 16, 2014 at 4:15 PM, gchen chenguanch...@gmail.com wrote: Hi Reynold, thanks for your interest on this issue. The work here is part of incorporating Spark into PowerLinux ecosystem. Here is the bug raised in ning by

Re: Big-Endian (IBM Power7) Spark Serialization issue

2014-06-16 Thread gchen
I didn't find ning's source code in Spark git repository (or maybe I missed it?), so next time when we meet bug caused by third party code, can we do something (to fix the bug) based on the Spark repository? -- View this message in context:

Re: Big-Endian (IBM Power7) Spark Serialization issue

2014-06-16 Thread Reynold Xin
It is here: https://github.com/apache/spark/blob/master/core/src/main/scala/org/apache/spark/io/CompressionCodec.scala On Mon, Jun 16, 2014 at 4:26 PM, gchen chenguanch...@gmail.com wrote: I didn't find ning's source code in Spark git repository (or maybe I missed it?), so next time when we

encounter jvm problem when integreation spark with mesos

2014-06-16 Thread qingyang li
hi, I encounter jvm problem when integreation spark with mesos, here is the log when i run spark-shell: -48ce131dc5af 14/06/17 12:24:55 INFO HttpServer: Starting HTTP Server 14/06/17 12:24:55 INFO SparkUI: Started Spark Web UI at http://bigdata001:4040 # # A fatal error has been detected by the

Compile failure with SBT on master

2014-06-16 Thread Andrew Ash
I can't run sbt/sbt gen-idea on a clean checkout of Spark master. I get resolution errors on junit#junit;4.10!junit.zip(source) As shown below: aash@aash-mbp /tmp/git/spark$ sbt/sbt gen-idea Using /Library/Java/JavaVirtualMachines/jdk1.7.0_45.jdk/Contents/Home as default JAVA_HOME. Note, this

Re: Compile failure with SBT on master

2014-06-16 Thread Ted Yu
I used the same command on Linux and it passed: Linux k.net 2.6.32-220.23.1.el6.YAHOO.20120713.x86_64 #1 SMP Fri Jul 13 11:40:51 CDT 2012 x86_64 x86_64 x86_64 GNU/Linux Cheers On Mon, Jun 16, 2014 at 9:29 PM, Andrew Ash and...@andrewash.com wrote: I can't run sbt/sbt gen-idea on a clean

Re: Compile failure with SBT on master

2014-06-16 Thread Andrew Ash
Maybe it's a Mac OS X thing? On Mon, Jun 16, 2014 at 9:57 PM, Ted Yu yuzhih...@gmail.com wrote: I used the same command on Linux and it passed: Linux k.net 2.6.32-220.23.1.el6.YAHOO.20120713.x86_64 #1 SMP Fri Jul 13 11:40:51 CDT 2012 x86_64 x86_64 x86_64 GNU/Linux Cheers On Mon, Jun