Re: DataStreamer addData takes lot of time after 500 million writes
Hello!
This would happen if you have any non-inlined indexes. You need to load a
lot of pages for every write and you are running out of offheap.
4G is a tiny offheap.
Regards,
--
Ilya Kasnacheev
вс, 15 сент. 2019 г. в 15:49, KR Kumar :
> Hi all - Why data streamer take lot of time randomly after 500+ million
> writes, it frequently and very consistently takes lot of time to finish the
> writes and to the extent of 25 to 45 seconds for write. May be its flushing
> the data as i have flush frequency set but why not in the beginning and why
> only in the end. And also I see heap size going up after sometime and trend
> is consistently upwards.
>
> Here is the streamer configuration:
>
> dataStreamer.autoFlushFrequency(1);
> dataStreamer.perNodeBufferSize(32 * 1024);
> dataStreamer.perNodeParallelOperations(32);
>
> Not sure if this is of any use but here is the dataStorageConfiguration
>
>
> class="org.apache.ignite.configuration.DataStorageConfiguration">
> name="defaultDataRegionConfiguration">
> class="org.apache.ignite.configuration.DataRegionConfiguration">
> name="initialSize" value="#{512L * 1024 * 1024}">
> name="maxSize" value="#{4L * 1024 * 1024 * 1024}" />
> name="persistenceEnabled" value="true" />
> name="checkpointPageBufferSize" value="#{2L *1024 * 1024 * 1024}" />
>
>
> value="#{4 * 1024}" />
> name="storagePath" value="${grid.data}" />
> value="${grid.wal}" />
> name="walArchivePath" value="${grid.wal}/archive" />
> value="BACKGROUND" />
> name="walFlushFrequency" value="5000">
>
>
>
Re: DataStreamer addData takes lot of time after 500 million writes
Hi,
Is the problem specific to DataStreamer? Will the problem disappear if
insert the same data using IgniteCache.put (or putAll batches)?
вс, 15 сент. 2019 г. в 15:49, KR Kumar :
>
> Hi all - Why data streamer take lot of time randomly after 500+ million
> writes, it frequently and very consistently takes lot of time to finish the
> writes and to the extent of 25 to 45 seconds for write. May be its flushing
> the data as i have flush frequency set but why not in the beginning and why
> only in the end. And also I see heap size going up after sometime and trend
> is consistently upwards.
>
> Here is the streamer configuration:
>
> dataStreamer.autoFlushFrequency(1);
> dataStreamer.perNodeBufferSize(32 * 1024);
> dataStreamer.perNodeParallelOperations(32);
>
> Not sure if this is of any use but here is the dataStorageConfiguration
>
>
> class="org.apache.ignite.configuration.DataStorageConfiguration">
> name="defaultDataRegionConfiguration">
> class="org.apache.ignite.configuration.DataRegionConfiguration">
> name="initialSize" value="#{512L * 1024 * 1024}">
> name="maxSize" value="#{4L * 1024 * 1024 * 1024}" />
> name="persistenceEnabled" value="true" />
> name="checkpointPageBufferSize" value="#{2L *1024 * 1024 * 1024}" />
>
>
> value="#{4 * 1024}" />
> value="${grid.data}" />
> value="${grid.wal}" />
> name="walArchivePath" value="${grid.wal}/archive" />
> value="BACKGROUND" />
> name="walFlushFrequency" value="5000">
>
>
--
Best regards,
Ivan Pavlukhin
DataStreamer addData takes lot of time after 500 million writes
Hi all - Why data streamer take lot of time randomly after 500+ million writes, it frequently and very consistently takes lot of time to finish the writes and to the extent of 25 to 45 seconds for write. May be its flushing the data as i have flush frequency set but why not in the beginning and why only in the end. And also I see heap size going up after sometime and trend is consistently upwards. Here is the streamer configuration: dataStreamer.autoFlushFrequency(1); dataStreamer.perNodeBufferSize(32 * 1024); dataStreamer.perNodeParallelOperations(32); Not sure if this is of any use but here is the dataStorageConfiguration
