+1 On 26-Jan-2016 7:01 am, "Amol Kekre" <[email protected]> wrote:
> +1 for util function from me too. > > Thks > Amol > > On Mon, Jan 25, 2016 at 4:59 PM, Timothy Farkas <[email protected]> > wrote: > > > +1 for util function > > > > Thanks, > > Tim > > > > On Mon, Jan 25, 2016 at 4:52 PM, Siyuan Hua <[email protected]> > > wrote: > > > > > Sandesh and I found an issue in almost all the operators that use Kryo > to > > > clone operator itself in definePartition method. They all use default > > > Kryo instance without setting context classloader from the thread that > > > execute the code. It would fail in local mode if it tries to load class > > > that only in app package classpath. > > > > > > When I'm trying to fix it in all the places, I found repeated pattern > in > > > definePartitions. And it's very error-prone since no one actually did > it > > in > > > right way until we found the issue. I'm thinking to make a Util > functin > > to > > > create instance from operator itself and also hide the Kryo dependency > > from > > > the operator. Do you have any better suggestions to do this? > > > > > > Thanks! > > > > > > Siyuan > > > > > >
