I am new to Spark source code and looking to see if i can add push-down support of spark filters to the storage (in my case an object store). I am willing to consider how this can be generically done for any store that we might want to integrate with spark. I am looking to know the areas that I should look into to provide support for a new data store in this context. Following below are some of the questions I have to start with: 1. Do we need to create a new RDD class for the new store that we want to support? From Spark Context, we create an RDD and the operations on data including the filter are performed through the RDD methods. 2. When we specify the code for filter task in the RDD.filter() method, how does it get communicated to the Executor on the data node? Does the Executor need to compile this code on the fly and execute it? or how does it work? ( I have looked at the code for sometime, but not yet got to figuring this out, so i am looking for some pointers that can help me come a little up-to-speed in this part of the code) 3. How long the Executor holds the memory? and how does it decide when to release the memory/cache? Thank you in advance.
Regards, Rajendran. --------------------------------------------------------------------- To unsubscribe, e-mail: dev-unsubscr...@spark.apache.org For additional commands, e-mail: dev-h...@spark.apache.org