Hi, I am not sure I follow the relationship between per partition batching and client-side capabilities. Does this mean that the data streamer cannot do per partition batching on the server side, for example the service grid?
I understand that low intensity streaming defeats the purpose of having batching (whether client side or not), but my case is long lived high intensity data traffic. thanks. On Tue, Feb 4, 2020 at 4:14 AM Ilya Kasnacheev <ilya.kasnach...@gmail.com> wrote: > Hello! > > In case of long-lived, low-intensity streaming, Data Streamer will not be > able to utilize its client-side per-partition batching capabilities, > instead being just a wrapper over cache update operations, which are > available as part of Cache API. > > Regards, > -- > Ilya Kasnacheev > > > вт, 4 февр. 2020 г. в 03:41, Denis Magda <dma...@apache.org>: > >> Ilya, >> >> I don't quite understand why data streamer is not suitable as a >> long-running solution. Please don't mislead, otherwise, list out specific >> limitations. I don't see anything wrong by having an opened data >> streamer that transfer data to Ignite in real-time. >> >> Narges, if the streamer crashes then your service/app needs to resend >> those records that were not acknowledged. Probably, you might utilize Kafka >> Connect here that keeps track of committed/pending records. >> >> - >> Denis >> >> >> On Mon, Feb 3, 2020 at 6:13 AM Ilya Kasnacheev <ilya.kasnach...@gmail.com> >> wrote: >> >>> Hello! >>> >>> I think these benefits are imaginary. You will have to worry about >>> service more, rather about data streamer which may be recreated at any time. >>> >>> Regards, >>> -- >>> Ilya Kasnacheev >>> >>> >>> пн, 3 февр. 2020 г. в 16:58, narges saleh <snarges...@gmail.com>: >>> >>>> Thanks Ilya. >>>> I have to listen to these burst of data which arrive every few seconds >>>> meaning an almost constant bursts of data from different data sources. >>>> The main reason that the services grid is appealing to me is its >>>> resiliency; I don't have to worry about it. With the client side streamer, >>>> I will have to deploy it and keep it up running, and load/re balance it. >>>> >>>> On Mon, Feb 3, 2020 at 7:17 AM Ilya Kasnacheev < >>>> ilya.kasnach...@gmail.com> wrote: >>>> >>>>> Hello! >>>>> >>>>> I don't see why you would deploy it as a service, sounds like you will >>>>> have to send more data over network. If you have to pull batches in, then >>>>> service should work. I recommend re-acquiring data streamer for each >>>>> batch. >>>>> >>>>> Please note that Data Streamer is very scalable, so it is preferred to >>>>> tune it than trying to use more than one streamer. >>>>> >>>>> Regards, >>>>> -- >>>>> Ilya Kasnacheev >>>>> >>>>> >>>>> пн, 3 февр. 2020 г. в 16:11, narges saleh <snarges...@gmail.com>: >>>>> >>>>>> Hi Ilya >>>>>> The data comes in huge batches of records (each burst can be up to >>>>>> 50-100 MB, which I plan to spread across multiple streamers) so, the >>>>>> streamer seems to be the way to go. Also, I don't want to establish a >>>>>> JDBC >>>>>> connection each time. >>>>>> So, if the streamer is the way to go, is it feasible to deploy it as >>>>>> a service? >>>>>> thanks. >>>>>> >>>>>> On Mon, Feb 3, 2020 at 6:51 AM Ilya Kasnacheev < >>>>>> ilya.kasnach...@gmail.com> wrote: >>>>>> >>>>>>> Hello! >>>>>>> >>>>>>> Contrary to its name, data streamer is not actually suitable for >>>>>>> long-lived, low-intensity streaming. What it's good for is burst load of >>>>>>> large number of data in a short period of time. >>>>>>> >>>>>>> If your data arrives in large batches, you can use Data Streamer for >>>>>>> each batch. If not, better use Cache API. >>>>>>> >>>>>>> If you are worried that plain Cache API is slow, but also want >>>>>>> failure resilience, there's catch-22. The only way to make something >>>>>>> resilient is to put it into cache :) >>>>>>> >>>>>>> Regards, >>>>>>> -- >>>>>>> Ilya Kasnacheev >>>>>>> >>>>>>> >>>>>>> пн, 3 февр. 2020 г. в 14:34, narges saleh <snarges...@gmail.com>: >>>>>>> >>>>>>>> Hi, >>>>>>>> But services are by definition long lived, right? Here is my >>>>>>>> layout: The data is continuously generated and sent to the streamer >>>>>>>> services (via JDBC connection with set streaming on option), deployed, >>>>>>>> say, >>>>>>>> as node singleton (actually deployed also as microservices) to load the >>>>>>>> data into the caches. The streamers do flush data based on some timers. >>>>>>>> If the streamer crashes before the buffer is flushed, the client >>>>>>>> catches the exception and resends the batch. Any issue with this >>>>>>>> layout? >>>>>>>> >>>>>>>> thanks. >>>>>>>> >>>>>>>> On Mon, Feb 3, 2020 at 5:02 AM Ilya Kasnacheev < >>>>>>>> ilya.kasnach...@gmail.com> wrote: >>>>>>>> >>>>>>>>> Hello! >>>>>>>>> >>>>>>>>> It is not recommended to have long-lived data streamers, it's best >>>>>>>>> to acquire it when it is needed. >>>>>>>>> >>>>>>>>> If you have to keep data streamer around, don't forget to flush() >>>>>>>>> it. This way you don't have to worry about its queue. >>>>>>>>> >>>>>>>>> Regards, >>>>>>>>> -- >>>>>>>>> Ilya Kasnacheev >>>>>>>>> >>>>>>>>> >>>>>>>>> пн, 3 февр. 2020 г. в 13:24, narges saleh <snarges...@gmail.com>: >>>>>>>>> >>>>>>>>>> Hi, >>>>>>>>>> My specific question/concern is with regard to the state of the >>>>>>>>>> streamer when it run as a service, i.e. when it crashes and it gets >>>>>>>>>> redeployed. Specifically, what happens to the data? >>>>>>>>>> I have a similar question with regard to the state of a >>>>>>>>>> continuous query when it is deployed as a service, what happens to >>>>>>>>>> the data >>>>>>>>>> in the listener's queue? >>>>>>>>>> >>>>>>>>>> thanks. >>>>>>>>>> >>>>>>>>>> On Sun, Feb 2, 2020 at 4:18 PM Mikael <mikael-arons...@telia.com> >>>>>>>>>> wrote: >>>>>>>>>> >>>>>>>>>>> Hi! >>>>>>>>>>> >>>>>>>>>>> Not as far as I know, I have a number of services using >>>>>>>>>>> streamers >>>>>>>>>>> without any problems, do you have any specific problem with it ? >>>>>>>>>>> >>>>>>>>>>> Mikael >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> Den 2020-02-02 kl. 22:33, skrev narges saleh: >>>>>>>>>>> > Hi All, >>>>>>>>>>> > >>>>>>>>>>> > Is there a problem with running the datastreamer as a service, >>>>>>>>>>> being >>>>>>>>>>> > instantiated in init method? Or loading the data via JDBC >>>>>>>>>>> connection >>>>>>>>>>> > with streaming mode enabled? >>>>>>>>>>> > In either case, the deployment is affinity based. >>>>>>>>>>> > >>>>>>>>>>> > thanks. >>>>>>>>>>> >>>>>>>>>>
