Re: NullPointerException with joda time

2015-11-12 Thread Romain Sagean
I Still can't make the logger work inside a map function. I can use "logInfo("")" in the main but not in the function. Anyway I rewrite my program to use java.util.Date instead joda time and I don't have NPE anymore. I will stick with this solution for the moment even if I find java Date ugly.

Re: NullPointerException with joda time

2015-11-12 Thread Ted Yu
Even if log4j didn't work, you can still get some clue by wrapping the following call with try block: currentDate = currentDate.plusDays(1) catching NPE and rethrowing with an exception that shows the value of currentDate Cheers On Thu, Nov 12, 2015 at 1:56 AM, Romain Sagean

Re: NullPointerException with joda time

2015-11-12 Thread Koert Kuipers
i remember us having issues with joda classes not serializing property and coming out null "on the other side" in tasks On Thu, Nov 12, 2015 at 10:12 AM, Ted Yu wrote: > Even if log4j didn't work, you can still get some clue by wrapping the > following call with try block:

Re: NullPointerException with joda time

2015-11-11 Thread Ted Yu
In case you need to adjust log4j properties, see the following thread: http://search-hadoop.com/m/q3RTtJHkzb1t0J66=Re+Spark+Streaming+Log4j+Inside+Eclipse Cheers On Tue, Nov 10, 2015 at 1:28 PM, Ted Yu wrote: > I took a look at >

NullPointerException with joda time

2015-11-10 Thread romain sagean
Hi community, I try to apply the function below during a flatMapValues or a map but I get a nullPointerException with the plusDays(1). What did I miss ? def allDates(dateSeq: Seq[DateTime], dateEnd: DateTime): Seq[DateTime] = { if (dateSeq.last.isBefore(dateEnd)){ allDates(dateSeq:+

Re: NullPointerException with joda time

2015-11-10 Thread Ted Yu
Can you show the stack trace for the NPE ? Which release of Spark are you using ? Cheers On Tue, Nov 10, 2015 at 8:20 AM, romain sagean wrote: > Hi community, > I try to apply the function below during a flatMapValues or a map but I > get a nullPointerException with the

Re: NullPointerException with joda time

2015-11-10 Thread Romain Sagean
see below a more complete version of the code. the firstDate (previously minDate) should not be null, I even added an extra "filter( _._2 != null)" before the flatMap and the error is still there. What I don't understand is why I have the error on dateSeq.las.plusDays and not on

Re: NullPointerException with joda time

2015-11-10 Thread Ted Yu
I took a look at https://github.com/JodaOrg/joda-time/blob/master/src/main/java/org/joda/time/DateTime.java Looks like the NPE came from line below: long instant = getChronology().days().add(getMillis(), days); Maybe catch the NPE and print out the value of currentDate to see if there is more