Re: Adding more ignite-clients slows down the first client

[Denis Magda]

Tobias,

I would also recommend to turn off Cassandra for now and confirm that the 
bottleneck is not in Ignite.

—
Denis

> On Nov 13, 2017, at 7:48 AM, Mikhail  wrote:
> 
> Hi Tobias,
> 
> if you need to do an initial loading of a big amount of data, please read
> the following section:
> https://apacheignite.readme.io/docs/streaming--cep
> 
> 
>> BUT when I add one more client to try to increased (scale horizontally) 
>> I clearly see that the INSERT speed decreases on the first client by a
> magnitude 
>> (dropping about 7,500 inserts/second from it’s 19,000 !!!)
> 
> you didn't scale it horizontally, you just added more load for one ignite
> server, if you need to scale it, you need
> to add more server nodes, you set backup==1, that means that you need to add
> at least 2 more nodes that see that it scales horizontally and linearly.
> 
>> Although the total speed increases to around ~25,000 INSERTs / second, this
> is still not good, and does not scale very well.
> 
> ignite scales pretty good, looks like *one* ignite server node reaches its
> limit, check cpu loading on the server node, I believe there would be 100%
> CPU utilization. So to scale it you need to add server nodes that will store
> data, because with you set up there's only 1 node that tries to handle
> traffic from 4 client nodes. 
> Also, you might be reach network IO limit, anyway in this case you also need
> to add more server nodes to split network a traffic from 4 clients to
> several server nodes.
> 
> 
> 
> 
> --
> Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: Adding more ignite-clients slows down the first client

[Mikhail]

Hi Tobias,

if you need to do an initial loading of a big amount of data, please read
the following section:
https://apacheignite.readme.io/docs/streaming--cep


>BUT when I add one more client to try to increased (scale horizontally) 
>I clearly see that the INSERT speed decreases on the first client by a
magnitude 
>(dropping about 7,500 inserts/second from it’s 19,000 !!!)

you didn't scale it horizontally, you just added more load for one ignite
server, if you need to scale it, you need
to add more server nodes, you set backup==1, that means that you need to add
at least 2 more nodes that see that it scales horizontally and linearly.

>Although the total speed increases to around ~25,000 INSERTs / second, this
is still not good, and does not scale very well.

ignite scales pretty good, looks like *one* ignite server node reaches its
limit, check cpu loading on the server node, I believe there would be 100%
CPU utilization. So to scale it you need to add server nodes that will store
data, because with you set up there's only 1 node that tries to handle
traffic from 4 client nodes. 
Also, you might be reach network IO limit, anyway in this case you also need
to add more server nodes to split network a traffic from 4 clients to
several server nodes.




--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Adding more ignite-clients slows down the first client

[Tobias Eriksson]

Hi
 I try to achieve very high inserts with Ignite-Cassandra
I have done some extensive testing, measuring all kinds of stuff
So long story short, here is my conclusion, I see the following when measuring 
INSERT rate per client
As you can see the first ignite-client has ~19,000 INSERTs / second
BUT when I add one more client to try to increased (scale horizontally) I 
clearly see that the INSERT speed decreases on the first client by a magnitude 
(dropping about 7,500 inserts/second from it’s 19,000 !!!)
Although the total speed increases to around ~25,000 INSERTs / second, this is 
still not good, and does not scale very well.
And as you can see adding a 3;rd and a 4;th client has the same affect.
Each client added steels overall performance from the other clients
I believe that Ignite is unable to handle this much inserts, can this be true…

Is this a known behavior ?
Is there a remedy ? (please say it is)

Details
Amazon C4.4XLarge   4 node cluster
Cassandra 3.3  4 Node cluster

Pojo Class with 50 attributes

Regards
Tobias


[cid:image001.png@01D35BF9.73D48C10]








java.lang.Long

com.qvantel.research.common.PojoProfile50























REPLICATION = {'class' : 'SimpleStrategy', 'replication_factor' :2}
AND DURABLE_WRITES = true


comment = 'Cache test'
AND read_repair_chance = 0.2






--
Tobias Eriksson
Chief Architect Research – CTO Office
Qvantel Sweden AB
Tel; +46 768 832453
e-mail; tobias.eriks...@qvantel.com