RE: Re: How best to push continuously updating ordered data on Apache ignite cache

[Alexandr Shapkin]

Hello!   You could take a look at putAll method that sends all updates to the server right away.   > *What is best way out of above to push continuously updating data like > market data?* For me it seems that the DataStreamer is more suitable for continuous data processing where you are not interested in getting acks from a server. Use StreamReceiver/StreamTransformer for additional data modification before the updating the values.   As Mikael suggested, you could implement some kind of an update counter. It feels that DateTime should work here.   Atomic sequences for the valid stock price could be helpful as well [1].   [1] - https://apacheignite.readme.io/docs/atomic-types   From: Mikael Sent: Friday, January 17, 2020 12:07 PM To: user@ignite.apache.org Subject: Re: How best to push continuously updating ordered data on Apache ignite cache   Hi!   Trusting that things come in the correct order could be tricky, the only way to do it I guess it to use put(), if you need fast writes you will have to use multiple threads and any ordering will be lost, can't you include some ordering data when you write them, a counter or whatever and use that to sort them in client 2 ? that would be uch safer.   You might need to include some kind of waiting process in case one entry has not arrived yet, but if you use a counter to write them you know that after 1 there will always ba a 2 and so on without any gaps so it should not be that messy.   Mikael   Den 2020-01-17 kl. 04:51, skrev trans: > *Usecase* >  > Here is the topology we are working on >  > *Server - 1* --> marketData cache, holds different share prices information >  > *Client - 1* --> Pushing data on the cache >  > *Client - 2* --> Continuous query, listening updates coming on marketData > cache per key basis >  > I want data to follow the order in which it was received and pushed on > queue. Reason is that client - 2 should not get old data. For example last > price data for an instrument moved from 100 to 101 then 102. Client - 2 > should get in the same order and to **not** get in order like 100 to 102 > then 101. > So for a key in my cache i want the messages to be pushed in order. >  > Ways of pushing data on cache: >  >   1. *put* seems to be safest way but looks like slow and full cache update > is happening and then thread moves to next statement. This might not be > suitable for pushing 20000 updates per second. >  >   2. *putAsync *seems to be good way, my understanding is that it uses > striped pool to push data on the cache. As the  striped pool > <https://apacheignite.readme.io/docs/thread-pools#section-striped-pool> > uses max(8, no of cores) so it should be faster. But as the multiple threads > are being processed in parallel so does it confirm the data ordering as it > was pushed in? >  >   3. *DataStreamer *seems to be best way as it also processes things in > parallel but again the problem is with the order of putting data into cache. > API documentation > <https://ignite.apache.org/releases/latest/dotnetdoc/api/Apache.Ignite.Core.Datastream.IDataStreamer-2.html> > also mention that ordering of data is not guaranteed. >  >  > Can someone please clarify above? I could not find document giving clear > deeper understanding for these ways. > *What is best way out of above to push continuously updating data like > market data?* >  >  >  > -- > Sent from: http://apache-ignite-users.70518.x6.nabble.com/