Re: how to send additional configuration to the RDD after it was lazily created

2015-09-21 Thread Romi Kuntsman
What new information do you know after creating the RDD, that you didn't know at the time of it's creation? I think the whole point is that RDD is immutable, you can't change it once it was created. Perhaps you need to refactor your logic to know the parameters earlier, or create a whole new RDD

how to send additional configuration to the RDD after it was lazily created

2015-09-17 Thread Gil Vernik
Hi, I have the following case, which i am not sure how to resolve. My code uses HadoopRDD and creates various RDDs on top of it (MapPartitionsRDD, and so on ) After all RDDs were lazily created, my code "knows" some new information and i want that "compute" method of the HadoopRDD will be