Hey guys, So it looks like using Spark/Ignite on YARN together simply doesn't work.
Many of us have Hadoop appliances where we aren't allowed to install anything on the nodes. So the only option is YARN which barring a few bugs seems to work okay. But the IgniteContext within Spark doesn't allow you to read configuration files from YARN. So since you allow users to pass in an IgniteConfiguration we have tried to manually set configuration on the POJOs: https://github.com/apache/ignite/blob/master/modules/spark/src/main/scala/org/apache/ignite/spark/IgniteContext.scala But during any Spark distributed operation it will attempt to serialise this which is not possible since most of the classes contained within IgniteConfiguration e.g. TcpDiscoverySpi are not serializable. I am going to go through and see how many classes will need to be marked serializable (could be dozens) but a call will need to be made: 1. Mark everything within IgniteConfiguration as serializable. 2. Force ALL users of IgniteContext to either read config from HDFS or from a Local Filesystem. Both will go through Spring layer. What's the best way to get a decision on this ? Cheers, Naden
