RE: spark-shell fails to redefine values

Can you ask for eee inbetween each reassign? The memory address at the end 1ec5bf62 != 2c6beb3e or 66cb003 – so what’s going on there? From: Yang [mailto:tedd...@gmail.com] Sent: 21 December 2016 18:37 To: user Subject: spark-shell fails to redefine values summary:

BinaryClassificationMetrics - get raw tp/fp/tn/fn stats per threshold?

Hi, BinaryClassificationMetrics expose recall and precision byThreshold. Is there a way to true negatives / false negatives etc per threshold? I have weighted my genuines and would like the adjusted precision / FPR. (Unless there is an option that I've missed, although I have been over the

RE: Date / time stuff with spark.

Hi Andy, Sorry this is in Scala but you may be able to do something similar? I use Joda's DateTime class. I ran into a lot of difficulties with the serializer, but if you are an admin on the box you'll have less issues by adding in some Kryo serializers. import org.joda.time val

RE: Spark SQL . How to enlarge output rows ?

I forgot to add this is (I think) from 1.5.0. And yeah that looks like a Python – I’m not hot with Python but it may be capitalised as False or FALSE? From: Eli Super [mailto:eli.su...@gmail.com] Sent: 21 January 2016 14:48 To: Spencer, Alex (Santander) Cc: user@spark.apache.org Subject: Re

RE: NPE when using Joda DateTime

into DateTime locally as needed. Not great. Or if possible use Java 8, where Joda APIs are part of the JDK. Possibly it works then. On Fri, Jan 15, 2016 at 9:32 AM, Spencer, Alex (Santander) <alex.spen...@santander.co.uk> wrote: > Hi, > > > > I tried Zhu’s recommendation a

RE: jobs much slower in cluster mode vs local

That's not that much of a difference given the overhead of cluster management. I would have thought a job should take minutes before you'll see a performance improvement on using cluster mode? Kind Regards, Alex. From: saif.a.ell...@wellsfargo.com [mailto:saif.a.ell...@wellsfargo.com] Sent: 15

RE: NPE when using Joda DateTime

) then: tempDate.minusHours(1) works fine. Kind Regards, Alex. From: Romain Sagean [mailto:romain.sag...@hupi.fr] Sent: 15 January 2016 13:08 To: Sean Owen Cc: Spencer, Alex (Santander); Shixiong(Ryan) Zhu; user@spark.apache.org Subject: Re: NPE when using Joda DateTime Hi, I had a similar problem with Joda Time

RE: NPE when using Joda DateTime

...@databricks.com] Sent: 14 January 2016 21:57 To: Durgesh Verma Cc: Spencer, Alex (Santander); Todd Nist; Sean Owen; user@spark.apache.org Subject: Re: NPE when using Joda DateTime Could you try to use "Kryo.setDefaultSerializer" like this: class YourKryoRegistrator extends KryoRegistrator { ov

NPE when using Joda DateTime

Hello, I was wondering if somebody is able to help me get to the bottom of a null pointer exception I'm seeing in my code. I've managed to narrow down a problem in a larger class to my use of Joda's DateTime functions. I've successfully run my code in scala, but I've hit a few problems when

RE: NPE when using Joda DateTime

Sean Owen [mailto:so...@cloudera.com] Sent: 14 January 2016 14:07 To: Spencer, Alex (Santander) Cc: user@spark.apache.org Subject: Re: NPE when using Joda DateTime It does look somehow like the state of the DateTime object isn't being recreated properly on deserialization somehow, given where the NPE oc

RE: Spark SQL . How to enlarge output rows ?

Hi, Try …..show(false) public void show(int numRows, boolean truncate) Kind Regards, Alex. From: Eli Super [mailto:eli.su...@gmail.com] Sent: 14 January 2016 13:09 To: user@spark.apache.org Subject: Spark SQL . How to enlarge output rows ? Hi After executing sql

RE: NPE when using Joda DateTime

tried that yet. Kind Regards, Alex. From: Todd Nist [mailto:tsind...@gmail.com] Sent: 14 January 2016 16:28 To: Spencer, Alex (Santander) Cc: Sean Owen; user@spark.apache.org Subject: Re: NPE when using Joda DateTime I had a similar problem a while back and leveraged these Kryo serializers, https