Joda Time best practice?
Hi, I am having trouble with Joda Time in a Spark application and saw by now that I am not the only one (generally seems to have to do with serialization and internal caches of the Joda Time objects). Is there a known best practice to work around these issues? Jan - To unsubscribe, e-mail: user-unsubscr...@spark.apache.org For additional commands, e-mail: user-h...@spark.apache.org
Re: Joda Time best practice?
Hey Jan, Can you provide more details on the serialization and cache issues. If you are looking for datetime functionality with spark-sql please consider: https://github.com/SparklineData/spark-datetime It provides a simple way to combine joda datetime expressions with spark sql. regards, Harish. On Mon, Jul 20, 2015 at 7:37 AM, algermissen1971 algermissen1...@icloud.com wrote: Hi, I am having trouble with Joda Time in a Spark application and saw by now that I am not the only one (generally seems to have to do with serialization and internal caches of the Joda Time objects). Is there a known best practice to work around these issues? Jan - To unsubscribe, e-mail: user-unsubscr...@spark.apache.org For additional commands, e-mail: user-h...@spark.apache.org
Re: Joda Time best practice?
Can you post details on how to reproduce the NPE On Mon, Jul 20, 2015 at 1:19 PM, algermissen1971 algermissen1...@icloud.com wrote: Hi Harish, On 20 Jul 2015, at 20:37, Harish Butani rhbutani.sp...@gmail.com wrote: Hey Jan, Can you provide more details on the serialization and cache issues. My symptom is that I have a Joda DateTime on which I can call toString and getMillis without problems, but when I call getYear I get a NPE out of the internal AbstractDateTime. Totally strange but seems to align with issues others have. I am now changing the app to work with millis internally, as that seems to be a performance improvement regarding serialization anyhow. Thanks, Jan If you are looking for datetime functionality with spark-sql please consider: https://github.com/SparklineData/spark-datetime It provides a simple way to combine joda datetime expressions with spark sql. regards, Harish. On Mon, Jul 20, 2015 at 7:37 AM, algermissen1971 algermissen1...@icloud.com wrote: Hi, I am having trouble with Joda Time in a Spark application and saw by now that I am not the only one (generally seems to have to do with serialization and internal caches of the Joda Time objects). Is there a known best practice to work around these issues? Jan - To unsubscribe, e-mail: user-unsubscr...@spark.apache.org For additional commands, e-mail: user-h...@spark.apache.org
Re: Joda Time best practice?
On 20 Jul 2015, at 23:20, Harish Butani rhbutani.sp...@gmail.com wrote: Can you post details on how to reproduce the NPE Essentially it is like this: I have a scala case class that contains a Joda DateTime attribute and instances of this class are updated using updateStateByKey. When a certain condition is reached the instances are turned to Json (using spray.json) and are stored in ElasticSearch. just before creating the JSON I call getYear on the date attribute and that fails with NPE. When insert a getMillis or toString right before the getYear these work just fine. Jan On Mon, Jul 20, 2015 at 1:19 PM, algermissen1971 algermissen1...@icloud.com wrote: Hi Harish, On 20 Jul 2015, at 20:37, Harish Butani rhbutani.sp...@gmail.com wrote: Hey Jan, Can you provide more details on the serialization and cache issues. My symptom is that I have a Joda DateTime on which I can call toString and getMillis without problems, but when I call getYear I get a NPE out of the internal AbstractDateTime. Totally strange but seems to align with issues others have. I am now changing the app to work with millis internally, as that seems to be a performance improvement regarding serialization anyhow. Thanks, Jan If you are looking for datetime functionality with spark-sql please consider: https://github.com/SparklineData/spark-datetime It provides a simple way to combine joda datetime expressions with spark sql. regards, Harish. On Mon, Jul 20, 2015 at 7:37 AM, algermissen1971 algermissen1...@icloud.com wrote: Hi, I am having trouble with Joda Time in a Spark application and saw by now that I am not the only one (generally seems to have to do with serialization and internal caches of the Joda Time objects). Is there a known best practice to work around these issues? Jan - To unsubscribe, e-mail: user-unsubscr...@spark.apache.org For additional commands, e-mail: user-h...@spark.apache.org - To unsubscribe, e-mail: user-unsubscr...@spark.apache.org For additional commands, e-mail: user-h...@spark.apache.org
Re: Joda Time best practice?
Hi Harish, On 20 Jul 2015, at 20:37, Harish Butani rhbutani.sp...@gmail.com wrote: Hey Jan, Can you provide more details on the serialization and cache issues. My symptom is that I have a Joda DateTime on which I can call toString and getMillis without problems, but when I call getYear I get a NPE out of the internal AbstractDateTime. Totally strange but seems to align with issues others have. I am now changing the app to work with millis internally, as that seems to be a performance improvement regarding serialization anyhow. Thanks, Jan If you are looking for datetime functionality with spark-sql please consider: https://github.com/SparklineData/spark-datetime It provides a simple way to combine joda datetime expressions with spark sql. regards, Harish. On Mon, Jul 20, 2015 at 7:37 AM, algermissen1971 algermissen1...@icloud.com wrote: Hi, I am having trouble with Joda Time in a Spark application and saw by now that I am not the only one (generally seems to have to do with serialization and internal caches of the Joda Time objects). Is there a known best practice to work around these issues? Jan - To unsubscribe, e-mail: user-unsubscr...@spark.apache.org For additional commands, e-mail: user-h...@spark.apache.org - To unsubscribe, e-mail: user-unsubscr...@spark.apache.org For additional commands, e-mail: user-h...@spark.apache.org
