We have a requirement as follows: Large streams of Time-ordered data received form over a TCP source to be processed in a time-ordered (cannot do parallel processing of the records) manner. The raw data or processed data does not need to be retained/cached in the Ignite layer. The processed records need to be inserted into a DB.
The SLA is a) Lowest Latency b/w the stream ingestion till DB. b) Reliability : cannot afford losing any record/packet from the stream In view of the above SLAs, would appreciate any feedback/suggestions on the following : Q1) what would be a better choice for Stream Ingestion from TCP source : Ignite SocketStreamer OR Kafka Streamer ? Q2) since the records are time-ordered and need to be processed in the same manner, we need a FIFO Queue in which the Streamer can inject the raw records, which can be picked up by a ClusterSingletonService which can process the raw records in a time-ordered manner & then insert the processed record in a write-through Cache to DB. Pls note that to ensure Reliability , the take() of each record from the FIFO Queue by the ClusterSingletonService till it writes the processed record to the Cache --> needs to be completed within single User Transaction, such that in the event of any failure/node-crash/etc while processing the record, it must rollback the transaction and NOT remove the raw record from the FIFO Queue. I tried IgniteQueue, but it does not support User Transaction. What would be the best alternative for the FIFO Queue implementation ? Thanks =====-----=====-----===== Notice: The information contained in this e-mail message and/or attachments to it may contain confidential or privileged information. If you are not the intended recipient, any dissemination, use, review, distribution, printing or copying of the information contained in this e-mail message and/or attachments to it are strictly prohibited. If you have received this communication in error, please notify us by reply e-mail or telephone and immediately and permanently delete the message and any attachments. Thank you
