Dmitriy, Igniters,
I've got the first yardstick benchmarking results on Amazon EC2. Thanks
Nikolay for guidance and ready to use yardstick docker image.
Used configuration is the following - c4.xlarge, 5 server nodes, 1
backup, running put/get benchmark, manually stopping one instance during
the execution.
Time to warmup 60 seconds, execution time 150 seconds, 64 threads.
1) Failure detection timeout set to *300 ms.
*Unfortunately, a drop during a kill of one server nodes is significant.
Please see a resulting plot in ignite-results-failure-detection.zip.
Making the timeout lower doesn't improve the situation.
Right after that I've decided to run the same benchmark with failure
detection timeout ignored by setting several network related timeouts
explicitly (these timeouts were used before when we got insignificant drop).
TcpCommunicationSpi.setSocketWriteTimeout(200)
TcpDiscoverySpi.setAckTimeout(50)
TcpDiscoverySpi.setNetworkTimeout(5000)
TcpDiscoverySpi.setHeartbeatFrequency(100)
2) Explicitly set the timeouts above, run against the latest changes
including mine.
Here I saw pretty the same result - the drop is again signification.
Have a look at the plot in
ignite-results-no-failure-detection-explicit-timeouts.zip.
3) Well, the final sanity check was done over the latest release -
ignite-1.3.0-incubation that does NOT contain my changes. The timeouts
were the same as in 2).
Unfortunately, here I see the same drop as well. Look into
ignite-results-1.3.0.zip.
Seems that we got that drop even before my 'failure detection timeout'
changes were merged, if refer to 3). Will try to debug all this stuff
better tomorrow.
--
Denis
On 7/24/2015 7:15 PM, Dmitriy Setrakyan wrote:
Thanks Denis!
This feature significantly simplifies failure detection configuration in
Ignite - just one configuration flag now vs. don't even remember how many.
Have you run a yardstick test on Amazon EC2 with this new configuration
flag? If we kill a node in the middle, then drop should be insignificant.
Also, I want to note your excellent handling of Jira communication. The
ticket has been thoroughly updated every step of the way.
D.
On Fri, Jul 24, 2015 at 5:37 AM, Denis Magda <dma...@gridgain.com> wrote:
Igniters,
Have just back merged the changes into the main development branch. Thanks
Yakov and Dmitriy for spending your time on review!
From now it’s possible to detect failures at cluster nodes'
discovery/communication/network levels by altering a single parameter -
IgniteConfiguration.failureDetectionTimeout.
By setting the failure detection timeout for a server node it will be
possible to detect failed nodes in a cluster topology during the time equal
to timeout's value and switch to/keep working with only alive nodes.
By setting the timeout for a client node will let us to detect failures
between the client and its router node (a server node that is a part of a
topology).
In addition, bunch of other improvements and simplifications were done at
the level of TcpDiscoverySpi and TcpCommunicationSpi. Changes are
aggregated here:
https://issues.apache.org/jira/browse/IGNITE-752 <
https://issues.apache.org/jira/browse/IGNITE-752>
—
Denis
