If the user classes are not Serializable, how does adding Serializable to the WindowedValue help? The user class, which is stored in a field in the WindowedValue will still be non-serializable, and thus cause problems.
On Thu, Aug 24, 2017 at 11:59 AM Kobi Salant <kobi.sal...@gmail.com> wrote: > right, if it is not kryo or java serializable then we should allow user to > set caching off > > 2017-08-24 21:55 GMT+03:00 Reuven Lax <re...@google.com.invalid>: > > > However user classes are not guaranteed to be java serializable either. > > > > On Thu, Aug 24, 2017 at 11:31 AM, Kobi Salant <kobi.sal...@gmail.com> > > wrote: > > > > > Thank you for your replies, > > > > > > Lukasz, The kryo does not fail on the WindowedValue but on the user > class > > > which in some cases is not kryo-serializable usually when there is no > > > default constructor. Switching to java serialization does fail on > > > WindowedValue > > > as it is not defined serializable > > > > > > Thomas, We thought about writing a custom serializer but it cannot work > > > without knowing the exact coders of each RDD, it is similar to the > > > discussion that was in the past is it mandatory for the user to set > > coders > > > for each PCollection or not. This suggestion is like asking Beam to > auto > > > detect coders for all Pcollections. > > > > > > Ben, Spark uses java serialization to send closures and it will not > work > > if > > > the cluster contains different JVMs that are not compatible. We agree > > with > > > you all that it will be more efficient to use coders for caching but > that > > > means > > > a major rewrite of the Spark runner and changing almost all > > > methods signatures and internal RDDs from a specific type to byte[]. > > > > > > As i said before this is fallback for users who have non serializable > > user > > > classes, even today if an RDD is cached with kryo it doesn't pass the > > coder > > > so we are not worsening the situation but enabling users to use the > Spark > > > runner. > > > > > > Thanks > > > Kobi > > > > > > > > > > > > בתאריך 24 באוג' 2017 20:37, "Thomas Weise" <t...@apache.org> כתב: > > > > > > > Would a custom Kryo serializer that uses the coders to perform > > > > serialization help? > > > > > > > > There are various ways Kryo let's you annotate such serializer > without > > > full > > > > surgery, including @Bind on the field level or at class level. > > > > > > > > Thanks, > > > > Thomas > > > > > > > > > > > > On Thu, Aug 24, 2017 at 9:57 AM, Lukasz Cwik > <lc...@google.com.invalid > > > > > > > wrote: > > > > > > > > > How does Kryo's FieldSerializer fail on WindowedValue/PaneInfo, it > > > seems > > > > > like those are pretty simple types. > > > > > > > > > > Also, I don't see why they can't be tagged with Serializable but I > > > think > > > > > the original reasoning was that coders should be used to guarantee > a > > > > stable > > > > > representation across JVM versions. > > > > > > > > > > On Thu, Aug 24, 2017 at 5:38 AM, Kobi Salant < > kobi.sal...@gmail.com> > > > > > wrote: > > > > > > > > > > > Hi All, > > > > > > > > > > > > I am working on Spark runner issue > > > > > > https://issues.apache.org/jira/browse/BEAM-2669 > > > > > > "Kryo serialization exception when DStreams containing > > > > > > non-Kryo-serializable data are cached" > > > > > > > > > > > > Currently, Spark runner enforce the kryo serializer and in > > shuffling > > > > > > scenarios we always uses coders to transfer bytes over the > network. > > > But > > > > > > Spark can also serialize data for caching/persisting and > currently > > it > > > > > uses > > > > > > kryo for this. > > > > > > > > > > > > Today, when the user uses a class which is not kryo serializable > > the > > > > > > caching fails and we thought to open the option to use java > > > > serialization > > > > > > as a fallback. > > > > > > > > > > > > Our RDDs/DStreams behind the PCollections are usually typed > > > > > > RDD/DStream<WindowedValue<InputT>> > > > > > > and when Spark tries to java serialize them for caching purposes > it > > > > fails > > > > > > on WindowedValue not being java serializable. > > > > > > > > > > > > Is there any objection to add Serializable implements to SDK > > classes > > > > like > > > > > > WindowedValue, PaneInfo and others? > > > > > > > > > > > > Again, i want to emphasise that coders are a big part of the > Spark > > > > runner > > > > > > code and we do use them whenever a shuffle is expected. Changing > > the > > > > > types > > > > > > of the RDDs/DStreams to byte[] will make the code pretty > > unreadable > > > > and > > > > > > will weaken type safety checks. > > > > > > > > > > > > Thanks > > > > > > Kobi > > > > > > > > > > > > > > > > > > > > >
