Hi Flavio, let me try to clarify: The intention of this example is to demonstrate how different entities (drivers, passengers, etc') participates in a protocol (ride matching). For that we have the stateful functions application, and a standalone java application that just generates the events to trigger the simulation (i.e. a passenger requests a ride would be an event that the simulator emits). The visualization aspect of this example is not a part of what we are trying to demonstrate but rather, an out-of-band peek into the live system (websocket that duplicates the events directly to the UI) which we thought is a nice addition to the example that visualizes how cars are moving on the grid :-)
If your goal is to create a scalable visualization of car locations, then I guess your suggestion to break the grid into distinct geo-regions can work. For that you would need: 1. An additional egress backed by a Kafka topic. 2. Modify the driver function to send location changes to that egress, where you'd use the region id (geohash with a resolution to your liking) as key [1]. 3. At the consuming side subscribe to the partition that holds the region of interest. But, realistically speaking for this kind of a visualization you would be better off consuming from the locations update topic (from-driver topic) and populating an optimized geo-indexed store, and querying it directly. I hope this clarifies things, Igal [1] https://github.com/ververica/stateful-functions/blob/master/stateful-functions-examples/stateful-functions-ridesharing-example/stateful-functions-ridesharing-example-functions/src/main/java/com/ververica/statefun/examples/ridesharing/KafkaSpecs.java#L121 On Thu, Oct 31, 2019 at 6:21 PM Flavio Pompermaier <pomperma...@okkam.it> wrote: > Thanks Igal, this is more or less what I was expecting..this implies that > ALL events are received on the UI side. > I was concerned about the tradeoffs of this choice: when I zoom on the map > I could simply ignore messages outside the boundaries (but I still spend > many cpu resource in the reading of useless messages). > > In the case of a worldwide company (eg uber or simolar) it's probably better > to create a topic per geographical arean..but also in this case then the UI > should know when to attach or detach from queue topics when it reach a > section of the map served by different topics..in another case I could just > have too many events also in the single map section (let's think about some > big city with many user). > > Is there any talk at FF ir someone else that faced those issues too? > > Il Gio 31 Ott 2019, 17:44 Igal Shilman <i...@ververica.com> ha scritto: > >> For that particular example, the simulator [1] is responsible for >> simulating physical drivers and passengers that interact with their >> corresponding >> stateful functions [2]. >> The interaction between the simulator and the stateful functions is >> happening via four Kafka topics: >> * to-driver - messages that are sent from the physical drivers (the >> simulator [1]) to the stateful functions. The messages always carry a >> driver-id which acts as a routing key (I think that this is what you mean >> by correlation id) to a specific driver stateful function (FnDriver) >> * from-driver - messages that are sent from a stateful function with a >> specific driver id to the simulator >> * to-passenger - symmetric to to-driver >> * from-passenger - symmetric to from-driver. >> The ingress and egress definition are specified here [3], and you may >> want to checkout how to router is defined as well [4][5]. >> >> In addition the simulator is also feeding the UI directly by duplicating >> the messages to a web socket (see [6]) >> >> I hope this clarifies the examples. >> >> Igal. >> >> [1] >> https://github.com/ververica/stateful-functions/tree/master/stateful-functions-examples/stateful-functions-ridesharing-example/stateful-functions-ridesharing-example-simulator/src/main/java/com/ververica/statefun/examples/ridesharing/simulator >> [2] >> https://github.com/ververica/stateful-functions/tree/master/stateful-functions-examples/stateful-functions-ridesharing-example/stateful-functions-ridesharing-example-functions >> [3] >> https://github.com/ververica/stateful-functions/blob/master/stateful-functions-examples/stateful-functions-ridesharing-example/stateful-functions-ridesharing-example-functions/src/main/java/com/ververica/statefun/examples/ridesharing/KafkaSpecs.java#L43 >> [4] >> https://github.com/ververica/stateful-functions/blob/master/stateful-functions-examples/stateful-functions-ridesharing-example/stateful-functions-ridesharing-example-functions/src/main/java/com/ververica/statefun/examples/ridesharing/Module.java#L33 >> [5] >> https://github.com/ververica/stateful-functions/blob/master/stateful-functions-examples/stateful-functions-ridesharing-example/stateful-functions-ridesharing-example-functions/src/main/java/com/ververica/statefun/examples/ridesharing/InboundDriverRouter.java#L26 >> [6] >> https://github.com/ververica/stateful-functions/blob/master/stateful-functions-examples/stateful-functions-ridesharing-example/stateful-functions-ridesharing-example-simulator/src/main/java/com/ververica/statefun/examples/ridesharing/simulator/simulation/Driver.java#L70 >> >> On Thu, Oct 31, 2019 at 4:15 PM Flavio Pompermaier <pomperma...@okkam.it> >> wrote: >> >>> Yes, I'm interested in how to read data from a UI..which egress should I >>> use? If we use a kafka queue, how to filter data received in the topic? >>> Should I use a correlation id or use a new topic per user? >>> >>> Il Gio 31 Ott 2019, 16:08 Igal Shilman <i...@ververica.com> ha scritto: >>> >>>> Hi Flavio, >>>> >>>> We haven't included the UI source code just yet, we've only used it for >>>> demos and talks. >>>> >>>> The reason is that (1) we didn't put a lot of effort and time there (2) >>>> didn't check the time to go through the individual dependencies and >>>> licences. >>>> But we will add that very soon. >>>> >>>> Would having the UI code there would improve your understanding? or is >>>> there another reason? >>>> >>>> Thanks, >>>> Igal >>>> >>>> On Thu, Oct 31, 2019 at 5:44 AM Flavio Pompermaier < >>>> pomperma...@okkam.it> wrote: >>>> >>>>> Hi Vino, >>>>> I already checked that code but I can't find the UI part :( >>>>> >>>>> On Thu, Oct 31, 2019 at 12:32 PM vino yang <yanghua1...@gmail.com> >>>>> wrote: >>>>> >>>>>> Hi Flavio, >>>>>> >>>>>> Please see this link.[1] >>>>>> >>>>>> Best, >>>>>> Vino >>>>>> >>>>>> [1]: >>>>>> https://github.com/ververica/stateful-functions/tree/master/stateful-functions-examples/stateful-functions-ridesharing-example >>>>>> >>>>>> Flavio Pompermaier <pomperma...@okkam.it> 于2019年10月31日周四 下午4:53写道： >>>>>> >>>>>>> Hi to all, >>>>>>> yould it be possible to provide also the source code of the UI part >>>>>>> of the ride sharing example? It would be interesting to me how the UI is >>>>>>> reading the data from the Kafka egress. >>>>>>> >>>>>>> Best, >>>>>>> Flavio >>>>>>> >>>>>> >>>>>
