RE: Specifying Scala types when calling methods from SparkR

Hi Sun Rui, I’ve had some luck simply using “objectFile” when saving from SparkR directly. The problem is that if you do it that way, the model object will only work if you continue to use the current Spark Context, and I think model persistence should really enable you to use the model at a

Re: A bug in Spark standalone? Worker registration and deregistration

On Thu, Dec 10, 2015 at 8:10 PM, Shixiong Zhu wrote: > Jacek, could you create a JIRA for it? I just reproduced it. It's a bug in > how Master handles the Worker disconnection. Hi Shixiong, I'm saved. Kept thinking I'm lost in the sources and see ghosts :-)

Re: A bug in Spark standalone? Worker registration and deregistration

Jacek, could you create a JIRA for it? I just reproduced it. It's a bug in how Master handles the Worker disconnection. Best Regards, Shixiong Zhu 2015-12-10 2:45 GMT-08:00 Jacek Laskowski : > Hi, > > I'm on yesterday's master HEAD. > > Pozdrawiam, > Jacek > > -- > Jacek

Re: A bug in Spark standalone? Worker registration and deregistration

Hi Jacek, I also recently noticed those messages, and some others, and am wondering if there is an issue. I am also seeing the following when I have event logging enabled. The first application is submitted and executes fine, but all subsequent attempts produce an error log, but the master

A bug in Spark standalone? Worker registration and deregistration

Hi, While toying with Spark Standalone I've noticed the following messages in the logs of the master: INFO Master: Registering worker 192.168.1.6:59919 with 2 cores, 2.0 GB RAM INFO Master: localhost:59920 got disassociated, removing it. ... WARN Master: Removing

Re: [VOTE] Release Apache Spark 1.6.0 (RC1)

We are getting close to merging patches for SPARK-12155 and SPARK-12253 . I'll be cutting RC2 shortly after that. Michael On Tue, Dec 8, 2015 at 10:31 AM, Michael Armbrust

Re: A bug in Spark standalone? Worker registration and deregistration

Hi, I'm on yesterday's master HEAD. Pozdrawiam, Jacek -- Jacek Laskowski | https://medium.com/@jaceklaskowski/ | http://blog.jaceklaskowski.pl Mastering Spark https://jaceklaskowski.gitbooks.io/mastering-apache-spark/ Follow me at https://twitter.com/jaceklaskowski Upvote at

Re: [build system] jenkins downtime, thursday 12/10/15 7am PDT

Note also that ASF Jira goes down at 19:00 UTC on friday, it *should* be back up ... the @infrabot twitter feed will the best channel for updates -- Forwarded message -- From: Daniel Takamori Date: 8 December 2015 at 18:59 Subject: Jira reboot 11-12-2015 at

Re: releasing Spark 1.4.2

Hi, I can see that there is 1.4.2 rc release prepared already[1], but wasn't called the vote for some reason. Was there any particular reason behind for stop the voting process? Just wondering whether we can get 1.4.2 released? [1]

Re: [build system] jenkins downtime, thursday 12/10/15 7am PDT

Hi Shane, I found that Jenkins has been in the status of "Jenkins is going to shut down" for at least 4 hours (from ~23:30 Dec 9 to 3:45 Dec 10, PDT). Not sure whether this is part of the schedule or related? Cheng On Thu, Dec 10, 2015 at 3:56 AM, shane knapp wrote: >

RE: Specifying Scala types when calling methods from SparkR

Hi, Chris, I know your point: objectFile and saveAsObjectFile pair in SparkR can only be used in SparkR context, as the content of RDD is assumed to be serialized R objects. It’s fine to drop down to JVM level in the case the model is saved as objectFile in Scala, and load it in SparkR. But I

Re: [VOTE] Release Apache Spark 1.6.0 (RC1)

Cutting RC2 now. On Thu, Dec 10, 2015 at 12:59 PM, Michael Armbrust wrote: > We are getting close to merging patches for SPARK-12155 > and SPARK-12253 > . I'll be

Re: Does RDD[Type1, Iterable[Type2]] split into multiple partitions?

No, since the signature itself limits it. On Thu, Dec 10, 2015 at 9:19 PM, JaeSung Jun wrote: > Hi, > > I'm currently working on Iterable type of RDD, which is like : > > val keyValueIterableRDD[CaseClass1, Iterable[CaseClass2]] = buildRDD(...) > > If there is only one

Does RDD[Type1, Iterable[Type2]] split into multiple partitions?

Hi, I'm currently working on Iterable type of RDD, which is like : val keyValueIterableRDD[CaseClass1, Iterable[CaseClass2]] = buildRDD(...) If there is only one unique key and Iterable is big enough, would this Iterable be partitioned across all executors like followings ? (executor1) (xxx,

coalesce at DataFrame missing argument for shuffle.

Hi all, I accidentally met coalesce() function and found this taking arguments different for RDD and DataFrame. It looks shuffle option is missing for DataFrame. I understand repartition() exactly works as coalesce() with shuffling but it looks a bit weird that the same functions take different

JIRA: Wrong dates from imported JIRAs

Hi, I've been digging into JIRA a bit and found a couple of old issues (~250) and I just assume that they are all from the old JIRA. Here's one example: Old: New:

Re: [build system] jenkins downtime, thursday 12/10/15 7am PDT

this is happening now. On Wed, Dec 9, 2015 at 11:56 AM, shane knapp wrote: > here's the security advisory for the update: > https://wiki.jenkins-ci.org/display/SECURITY/Jenkins+Security+Advisory+2015-12-09 > > On Wed, Dec 9, 2015 at 9:55 AM, shane knapp

Re: [build system] jenkins downtime, thursday 12/10/15 7am PDT

jenkins is done, but we'll also be updating the firewall. this shouldn't take very long and i'll let everyone know when we're done. On Thu, Dec 10, 2015 at 6:35 AM, shane knapp wrote: > this is happening now. > > On Wed, Dec 9, 2015 at 11:56 AM, shane knapp

Re: [build system] jenkins downtime, thursday 12/10/15 7am PDT

and we're done! this was a quick one. :) On Thu, Dec 10, 2015 at 6:54 AM, shane knapp wrote: > jenkins is done, but we'll also be updating the firewall. this > shouldn't take very long and i'll let everyone know when we're done. > > On Thu, Dec 10, 2015 at 6:35 AM, shane

Re: [build system] jenkins downtime, thursday 12/10/15 7am PDT

that probably was me, sorry. i pulled up the rest api command on my phone before i fell asleep and must have accidentally put jenkins in to quiet mode. sorry about that! On Thu, Dec 10, 2015 at 3:49 AM, Cheng Lian wrote: > Hi Shane, > > I found that Jenkins has been in