I'll try the hackier way for now - given the limitation of not being able to modify the environment we've been given. Thanks all for your help so far.
Kind Regards, Alex. -----Original Message----- From: Sean Owen [mailto:so...@cloudera.com] Sent: 15 January 2016 12:17 To: Spencer, Alex (Santander) Cc: Shixiong(Ryan) Zhu; user@spark.apache.org Subject: Re: NPE when using Joda DateTime I haven't dug into this, but I agree that something that is transient isn't meant to be restored by the default Java serialization mechanism. I'd expect the class handles restoring that value as needed or in a custom readObject method. And then I don't know how Kryo interacts with that. I don't think you need to install anything. You may end up writing your own serialization for Kryo. Try not using Kryo just to narrow it down? Hackier solution: send around long timestamps and then make them into DateTime locally as needed. Not great. Or if possible use Java 8, where Joda APIs are part of the JDK. Possibly it works then. On Fri, Jan 15, 2016 at 9:32 AM, Spencer, Alex (Santander) <alex.spen...@santander.co.uk> wrote: > Hi, > > > > I tried Zhu’s recommendation and sadly got the same error. (Again, > single map worked by the groupBy / flatMap generates this error). > > > > Does Kryo has a bug i.e. it’s not serialising all components needed, > or do I just need to get our IT team to install those magro > Serializers as suggested by Todd? If that variable is transient then > actually that means Kryo is working as it’s meant to? > > > > Am I at the point where I should pull apart the source code and build > my own DateTime class? I hate reinventing the wheel though. > > > > Thanks, > > Alex. > > Emails aren't always secure, and they may be intercepted or changed after they've been sent. Santander doesn't accept liability if this happens. If you think someone may have interfered with this email, please get in touch with the sender another way. This message doesn't create or change any contract. Santander doesn't accept responsibility for damage caused by any viruses contained in this email or its attachments. Emails may be monitored. If you've received this email by mistake, please let the sender know at once that it's gone to the wrong person and then destroy it without copying, using, or telling anyone about its contents. Santander UK plc Reg. No. 2294747 and Abbey National Treasury Services plc Reg. No. 2338548 Registered Offices: 2 Triton Square, Regent's Place, London NW1 3AN. Registered in England. www.santander.co.uk. Authorised by the Prudential Regulation Authority and regulated by the Financial Conduct Authority and the Prudential Regulation Authority. FCA Reg. No. 106054 and 146003 respectively. Santander Sharedealing is a trading name of Abbey Stockbrokers Limited Reg. No. 02666793. Registered Office: Kingfisher House, Radford Way, Billericay, Essex CM12 0GZ. Authorised and regulated by the Financial Conduct Authority. FCA Reg. No. 154210. You can check this on the Financial Services Register by visiting the FCA’s website www.fca.org.uk/register or by contacting the FCA on 0800 111 6768. Santander UK plc is also licensed by the Financial Supervision Commission of the Isle of Man for its branch in the Isle of Man. Deposits held with the Isle of Man branch are covered by the Isle of Man Depositors’ Compensation Scheme as set out in the Isle of Man Depositors’ Compensation Scheme Regulations 2010. In the Isle of Man, Santander UK plc’s principal place of business is at 19/21 Prospect Hill, Douglas, Isle of Man, IM1 1ET. Santander and the flame logo are registered trademarks. Santander Asset Finance plc. Reg. No. 1533123. Registered Office: 2 Triton Square, Regent’s Place, London NW1 3AN. Registered in England. Santander Corporate & Commercial is a brand name used by Santander UK plc, Abbey National Treasury Services plc and Santander Asset Finance plc. Ref:[PDB#1-4A]
