Hi Naden,I am just wondering why you want to start ignite node in Yarn.You can stary the independent ignite node.What benefit you are getting by making part of yarn cluster.I am asking this because i too had the same queries but latet on i switched to starting independent ignite node on the same nodes where my node managers are running.I guess this too will give data locality etc.Also you can create RDD and save data directly to cache using RDD,you dont need to explicitly invoke ignite() or start().I can share the code if you want. Regards,Vij
Sent from Yahoo Mail on Android On Wed, 4 May, 2016 at 5:19 am, Franciscus, Naden<[email protected]> wrote: Hey Alexey, Sorry I meant HDFS :) It would be good for IgniteContext to have the option to read configuration from HDFS. Again many users of Hadoop aren't going to have the ability to modify data nodes to install Ignite. But I will raise that as a separate issue. The big issue is that there is a lot of code like this in documentation and elsewhere: ignite.cache("myCache").put(personId, person); Which we can't use in any RDD/DF map, filter etc because neither Ignite/IgniteCache are serializable. Cheers, Naden On 4/05/2016, 5:42 AM, "Alexey Goncharuk" <[email protected]> wrote: >Hi Naden, > > But the IgniteContext within Spark doesn't allow you to read >configuration >> files from YARN. >> > >I am a little bit confused. Ignite can be configured via basic Spring XML, >and you can definitely read those XML files off HDFS or any other source. >Is there any reason why XML does not work for you? > >Now it is impossible to make IgniteConfiguration serializable because it >contains such components as SPIs, which are non-serializable by it's >nature.
