Hello All, I am in the middle of designing real time data enhancement services using spark streaming. As part of this, I have to look up some reference data while processing the incoming stream. I have below questions: 1) what is the maximum size of look up table / variable can be stored as Broadcast variable ()2) What is the impact of cluster performance, if I store a 10GB data in broadcast variable Any suggestions and thoughts are welcome. Thanks,Saravanan S.
- Maximum Size of Reference Look Up Table in Spark Saravanan Subramanian
- Re: Maximum Size of Reference Look Up Table in ... Jacek Laskowski
- Re: Maximum Size of Reference Look Up Table... Saravanan Subramanian
- Re: Maximum Size of Reference Look Up T... Jacek Laskowski