putAll() performance big difference, 2.7.5 vs 2.9.1

[Ping Hao]

Hi All,

Here it's a sample scala application, 

https://gist.github.com/PingHao/0369a6d14c1de7072de57041a3fec0f7

I had compiled it with both Ignite 2.7.5 and 2.9.1 libraries, and got result by 
run on a 3 server nodes + 1 client node cluster. all nodes have 56 cpu cores 
and 100G+ free memory. same nodes, same client code, same Java 11 env, but 
performance result show 2.9.1 is much slower.

Thread number set to 40, and batch size 2000.

Ignite 2.7.5: putAllEntries rate > 2M / sec , latency avg 13 ms, latency is 
measure time cost on putAll() call
Ignite 2.9.1: putAllEntries rate ~ 0.4M / sec, latency avg 120 ms

Note that cache key, value is  , all strings are size 10. 

I did ran yardstick test case, it didn't give so much difference. so here 
wander what's going on?

Cheers
Ping

Re: putAll() performance big difference, 2.7.5 vs 2.9.1

[jjimeno]

Hi,

Maybe it has to do with this:

Performance of Ignite as key-value datastore. C++ Thin Client

  



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: putAll() performance big difference, 2.7.5 vs 2.9.1

[Pavel Tupitsyn]

Hi,

1. Can you please attach the CacheConfiguration as well?
2. Have you tried DataStreamer
https://ignite.apache.org/docs/latest/data-streaming?

On Thu, Feb 4, 2021 at 12:27 PM jjimeno  wrote:

> Hi,
>
> Maybe it has to do with this:
>
> Performance of Ignite as key-value datastore. C++ Thin Client
> <
> http://apache-ignite-users.70518.x6.nabble.com/Performance-of-Ignite-as-key-value-datastore-C-Thin-Client-tt35274.html>
>
>
>
>
> --
> Sent from: http://apache-ignite-users.70518.x6.nabble.com/
>

Re: putAll() performance big difference, 2.7.5 vs 2.9.1

[haoping]

I don't know data-streaming, but will check it out later.
Here it is ignite config on one server node, same as other nodes.

http://www.springframework.org/schema/beans";
   xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance";
   xsi:schemaLocation="
   http://www.springframework.org/schema/beans
   http://www.springframework.org/schema/beans/spring-beans.xsd";>


































2s1b:37205
2s1c:37205
2s1d:37205
exp1:37205



















On Thu, Feb 4, 2021 at 7:27 AM Pavel Tupitsyn  wrote:

> Hi,
>
> 1. Can you please attach the CacheConfiguration as well?
> 2. Have you tried DataStreamer
> https://ignite.apache.org/docs/latest/data-streaming?
>
> On Thu, Feb 4, 2021 at 12:27 PM jjimeno  wrote:
>
>> Hi,
>>
>> Maybe it has to do with this:
>>
>> Performance of Ignite as key-value datastore. C++ Thin Client
>> <
>> http://apache-ignite-users.70518.x6.nabble.com/Performance-of-Ignite-as-key-value-datastore-C-Thin-Client-tt35274.html>
>>
>>
>>
>>
>> --
>> Sent from: http://apache-ignite-users.70518.x6.nabble.com/
>>
>

Re: putAll() performance big difference, 2.7.5 vs 2.9.1

[haoping]

Update with DataStreamer, Ignite 2.7.5 still much better than 2.9.1 on
throughput. However, my OP with putAll() in fact deliver highest throughput.

gist
https://gist.github.com/PingHao/f104c02e86bfbea9c0bd8d1abcfdb912

10 working threads, shared same DataStreamer instance, 40 thread pool size
of datastreamer in all ignite configuration. server nodes 4, driver nodes 1.
2.7.5: 1.8M/s
2.9.1: 0.46M/s




On Thu, Feb 4, 2021 at 8:43 AM haoping  wrote:

> I don't know data-streaming, but will check it out later.
> Here it is ignite config on one server node, same as other nodes.
>
> http://www.springframework.org/schema/beans";
>xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance";
>xsi:schemaLocation="
>http://www.springframework.org/schema/beans
>http://www.springframework.org/schema/beans/spring-beans.xsd";>
>
>  class="org.apache.ignite.configuration.IgniteConfiguration">
> 
> 
> 
> 
> 
>
> 
> 
> 
>
> 
>  class="org.apache.ignite.failure.StopNodeOrHaltFailureHandler">
> 
> 
>  name="ignoredFailureTypes">
> 
> 
>
> 
>  class="org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi">
> 
> 
> 
> 
>
> 
>  class="org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi">
> 
> 
>  class="org.apache.ignite.spi.discovery.tcp.ipfinder.vm.TcpDiscoveryVmIpFinder">
> 
> 
> 2s1b:37205
> 2s1c:37205
> 2s1d:37205
> exp1:37205
> 
> 
> 
> 
> 
> 
> 
>  class="org.apache.ignite.configuration.DataStorageConfiguration">
> 
>  class="org.apache.ignite.configuration.DataRegionConfiguration">
> 
> 
> 
> 
> 
> 
> 
> 
>
> On Thu, Feb 4, 2021 at 7:27 AM Pavel Tupitsyn 
> wrote:
>
>> Hi,
>>
>> 1. Can you please attach the CacheConfiguration as well?
>> 2. Have you tried DataStreamer
>> https://ignite.apache.org/docs/latest/data-streaming?
>>
>> On Thu, Feb 4, 2021 at 12:27 PM jjimeno  wrote:
>>
>>> Hi,
>>>
>>> Maybe it has to do with this:
>>>
>>> Performance of Ignite as key-value datastore. C++ Thin Client
>>> <
>>> http://apache-ignite-users.70518.x6.nabble.com/Performance-of-Ignite-as-key-value-datastore-C-Thin-Client-tt35274.html>
>>>
>>>
>>>
>>>
>>> --
>>> Sent from: http://apache-ignite-users.70518.x6.nabble.com/
>>>
>>

Re: putAll() performance big difference, 2.7.5 vs 2.9.1

[Ping Hao]

Thanks for heads up. Although for my case, the cache configuration is all by 
default, partition/atomic/zero backup. I guess some general logic was changed 
in 2.9.1 impact all type of atomicity modes.

On 2021/02/04 09:27:31, jjimeno  wrote: 
> Hi,
> 
> Maybe it has to do with this:
> 
> Performance of Ignite as key-value datastore. C++ Thin Client
> 
>   
> 
> 
> 
> --
> Sent from: http://apache-ignite-users.70518.x6.nabble.com/
> 

Re: putAll() performance big difference, 2.7.5 vs 2.9.1

[Pavel Tupitsyn]

There is a known performance drop of around 7% in 2.9 vs 2.8: [1]
However, it is nowhere near the difference you are getting.

Please make sure that no other apps affect the comparison.
Check our performance tuning documentation [2].

[1] https://issues.apache.org/jira/browse/IGNITE-13337
[2]
https://ignite.apache.org/docs/latest/perf-and-troubleshooting/general-perf-tips



On Thu, Feb 4, 2021 at 11:10 PM Ping Hao  wrote:

> Thanks for heads up. Although for my case, the cache configuration is all
> by default, partition/atomic/zero backup. I guess some general logic was
> changed in 2.9.1 impact all type of atomicity modes.
>
> On 2021/02/04 09:27:31, jjimeno  wrote:
> > Hi,
> >
> > Maybe it has to do with this:
> >
> > Performance of Ignite as key-value datastore. C++ Thin Client
> > <
> http://apache-ignite-users.70518.x6.nabble.com/Performance-of-Ignite-as-key-value-datastore-C-Thin-Client-tt35274.html>
>
> >
> >
> >
> > --
> > Sent from: http://apache-ignite-users.70518.x6.nabble.com/
> >
>