Thanks a LOT for your answer ! I've updated the diagram, at the same
address :
https://www.dropbox.com/s/q79taoce2ywdmf1/SparkStreaming.pdf?dl=0
I've addressed your more straightforward remarks directly in the diagram. A
couple questions:
- the location of instances (Executor, Master, Driver) is
Hey François,
Well, at a high-level here is what I thought about the diagram.
- ReceiverSupervisor handles only one Receiver.
- BlockGenerator is part of ReceiverSupervisor not ReceivedBlockHandler
- The blocks are inserted in BlockManager and if activated,
WriteAheadLogManager in parallel, not
I’ve been trying to produce an updated box diagram to refresh :
http://www.slideshare.net/spark-project/deep-divewithsparkstreaming-tathagatadassparkmeetup20130617/26
… after the SPARK-3129, and other switches (a surprising number of comments
still mention NetworkReceiver).
Here’s what I have
