What I seem to be don’t get is how my code ends up being on Worker node. My understanding was that jar file, which I use to start the job should automatically be copied into Worker nodes and added to classpath. It seems to be not the case. But if my jar is not copied into Worker nodes, then how my code is able to run on Workers? I know that Driver serializes my functions and send them to Workers, but most of my functions are using some other classes (from the jar). It seems that along with functions code, Driver has to be able to copy other classes, too. Is that so?
That actually might explain, why KryoRegistrator is not being found on Worker - there are no functions, which use it directly, so it never copied to Workers. Could you, please, explain of how code is end up on Worker or give me a hint where I can find it in the sources? On 08 Jul 2015, at 17:40, Eugene Morozov <fathers...@list.ru> wrote: > Hello. > > I have an issue with CustomKryoRegistrator, which causes ClassNotFound on > Worker. > The issue is resolved if call SparkConf.setJar with path to the same jar I > run. > > It is a workaround, but it requires to specify the same jar file twice. The > first time I use it to actually run the job, and second time in properties > file, which looks weird and unclear to as why I should do that. > > What is the reason for it? I thought the jar file has to be copied into all > Worker nodes (or else it’s not possible to run the job on Wokrers). Can > anyone shed some light on this? > > Thanks > -- > Eugene Morozov > fathers...@list.ru > > > > Eugene Morozov fathers...@list.ru
