Hello! Since Java is a garbage-collected language, you can expect that there will be some timeouts.
You should really be targeting 99th while having reasonably large timeout. Regards, -- Ilya Kasnacheev 2018-07-06 11:44 GMT+03:00 胡海麟 <h...@bulbit.jp>: > Hi, > > Thank you for your advice. > > We are checking if we can modify the connection part of the client. > > Meanwhile, we still have timeout issue on ignite. > Our write timeout setting is 20ms, but timeout happens several times 1 > minute. > The workload is like "Atomic Put, Throughput 100 - 150 Ops/Sec". > > https://www.gridgain.com/resources/benchmarks/gridgain-benchmarks-results > According the benchmark above, single digital ms per op is expected. > > Since we use golang redis client to communicate with ignite, does it > mean a great overhead? > Or any other client suggestion for golang? > > Thanks. > > On Fri, Jul 6, 2018 at 12:42 AM, Ilya Kasnacheev > <ilya.kasnach...@gmail.com> wrote: > > Hello! > > > > I have no idea why your node has lost zookeeper connection, but my bet > will > > be that you still failed to close some of connections, and they have > clogged > > each and every available socket on your machine (which are still only > 63k). > > > > Please make extra sure that when you open a socket, you close it > regardless > > of errors, etc. > > > > Regards, > > > > -- > > Ilya Kasnacheev > > > > 2018-07-05 13:33 GMT+03:00 胡海麟 <h...@bulbit.jp>: > >> > >> Hi, > >> > >> Challenged again and I'm back :) > >> > >> One of the node stopped for SEGMENTATION. > >> The Zookeeper is used by other services, too. It is supposed working > >> all the time. > >> > >> I'm not sure if raising session timeout of zookeeper work or not. Ask > >> for suggestions. > >> > >> Thanks. > >> > >> On Thu, Jul 5, 2018 at 6:24 PM, Ilya Kasnacheev > >> <ilya.kasnach...@gmail.com> wrote: > >> > Hello! > >> > > >> > The config looks correct. I'll check why ConnectorConfiguration > doesn't > >> > have > >> > proper string representation, hence: > >> > https://issues.apache.org/jira/browse/IGNITE-8935 > >> > > >> > Regards, > >> > > >> > -- > >> > Ilya Kasnacheev > >> > > >> > 2018-07-05 9:21 GMT+03:00 胡海麟 <h...@bulbit.jp>: > >> >> > >> >> Hi, > >> >> > >> >> I added the config above. But I don't know whether it is applied. > >> >> > >> >> > >> >> > >> >> connectorCfg=org.apache.ignite.configuration.ConnectorConfiguration@ > 7d8704ef > >> >> > >> >> There is such a piece of relevant info but no detail. Is there any > way > >> >> to confirm the details? > >> >> > >> >> Thanks. > >> >> > >> >> On Wed, Jul 4, 2018 at 7:25 PM, Ilya Kasnacheev > >> >> <ilya.kasnach...@gmail.com> wrote: > >> >> > Hello! > >> >> > > >> >> > You should add it to config/default-config.xml in the form of > >> >> > > >> >> > <property name="connectorConfiguration"> > >> >> > <bean > >> >> > class="org.apache.ignite.configuration.ConnectorConfiguration"> > >> >> > <property name="threadPoolSize" value="64"/> > >> >> > </bean> > >> >> > </property> > >> >> > > >> >> > inside IgniteConfiguration bean. > >> >> > > >> >> > Come to think of it, maybe your slowdown is caused not by thread > >> >> > starvation > >> >> > but by genuine locking due to repeated usage of the same keys. Are > >> >> > you > >> >> > sure > >> >> > that you are using a lot of different keys and not just a couple > >> >> > which > >> >> > you > >> >> > over-write over and over? > >> >> > > >> >> > Makes sense to try both. > >> >> > > >> >> > Regards, > >> >> > > >> >> > -- > >> >> > Ilya Kasnacheev > >> >> > > >> >> > 2018-07-04 13:19 GMT+03:00 胡海麟 <h...@bulbit.jp>: > >> >> >> > >> >> >> Hi, > >> >> >> > >> >> >> After googling for 1 hour to get nothing, I'm back for some basic > >> >> >> knowledges. > >> >> >> > >> >> >> igniteConfiguration.setConnectorConfiguration(new > >> >> >> ConnectorConfiguration().setThreadPoolSize(64)) > >> >> >> > >> >> >> I suppose this is a piece of java code but I don't know how and > >> >> >> where > >> >> >> to execute it. > >> >> >> I started the cluster with `bin/ignite.sh > config/default-config.xml` > >> >> >> and use golang written program as a client. > >> >> >> > >> >> >> Am I missing something? > >> >> >> > >> >> >> Thanks. > >> >> >> > >> >> >> On Wed, Jul 4, 2018 at 6:22 PM, Ilya Kasnacheev > >> >> >> <ilya.kasnach...@gmail.com> wrote: > >> >> >> > Hello! > >> >> >> > > >> >> >> > You can try increasing number of threads in REST thread pool by > >> >> >> > setting > >> >> >> > igniteConfiguration.setConnectorConfiguration(new > >> >> >> > ConnectorConfiguration().setThreadPoolSize(64)) > >> >> >> > Or the corresponding Spring XML. > >> >> >> > > >> >> >> > This is as per https://apacheignite.readme.io/docs/rest-api > >> >> >> > > >> >> >> > Regards, > >> >> >> > > >> >> >> > -- > >> >> >> > Ilya Kasnacheev > >> >> >> > > >> >> >> > 2018-07-04 12:04 GMT+03:00 胡海麟 <h...@bulbit.jp>: > >> >> >> >> > >> >> >> >> Hi, > >> >> >> >> > >> >> >> >> Here is the thread dump. > >> >> >> >> > >> >> >> >> Thank you. > >> >> >> >> > >> >> >> >> On Wed, Jul 4, 2018 at 5:52 PM, Ilya Kasnacheev > >> >> >> >> <ilya.kasnach...@gmail.com> wrote: > >> >> >> >> > Hello! > >> >> >> >> > > >> >> >> >> > Can you provide the thread dump collected when the system is > >> >> >> >> > under > >> >> >> >> > peak > >> >> >> >> > load? > >> >> >> >> > > >> >> >> >> > I think it's some other thread pool, such as client pool or > >> >> >> >> > management > >> >> >> >> > pool, > >> >> >> >> > but have to take a look at the thread dump to be sure. > >> >> >> >> > > >> >> >> >> > Regards, > >> >> >> >> > > >> >> >> >> > -- > >> >> >> >> > Ilya Kasnacheev > >> >> >> >> > > >> >> >> >> > 2018-07-04 11:33 GMT+03:00 胡海麟 <h...@bulbit.jp>: > >> >> >> >> >> > >> >> >> >> >> Hi, > >> >> >> >> >> > >> >> >> >> >> We use ignite as a redis server. > >> >> >> >> >> > >> >> >> >> >> The use case is > >> >> >> >> >> a. Write timeout is 15ms on the client side. > >> >> >> >> >> b. 2 server nodes. each is an EC2 r4.4xlarge instance. > >> >> >> >> >> c. Write req/s is about 120,000. In another word, 60,000 for > >> >> >> >> >> each > >> >> >> >> >> node. > >> >> >> >> >> > >> >> >> >> >> The problem is that timeout happens frequently, several ones > >> >> >> >> >> per > >> >> >> >> >> second. > >> >> >> >> >> A lower write req/s results less timeout. I guest we have > >> >> >> >> >> bottleneck > >> >> >> >> >> somewhere. > >> >> >> >> >> > >> >> >> >> >> ========== > >> >> >> >> >> $ tail -f > >> >> >> >> >> > >> >> >> >> >> > >> >> >> >> >> /opt/apache-ignite-fabric-2.5.0-bin/work/log/ignite- > ee4f25ed.0.log > >> >> >> >> >> | grep pool > >> >> >> >> >> ^-- Public thread pool [active=0, idle=0, qSize=0] > >> >> >> >> >> ^-- System thread pool [active=0, idle=16, qSize=0] > >> >> >> >> >> ========== > >> >> >> >> >> system thread pool seems not busy at all. > >> >> >> >> >> > >> >> >> >> >> ========== > >> >> >> >> >> $ tail -f > >> >> >> >> >> > >> >> >> >> >> > >> >> >> >> >> /opt/apache-ignite-fabric-2.5.0-bin/work/log/ignite- > ee4f25ed.0.log > >> >> >> >> >> | grep "CPU " > >> >> >> >> >> ^-- CPU [cur=14.77%, avg=6.21%, GC=0%] > >> >> >> >> >> ^-- CPU [cur=13.43%, avg=6.23%, GC=0%] > >> >> >> >> >> ========== > >> >> >> >> >> CPU is not busy, either. > >> >> >> >> >> > >> >> >> >> >> We expected milli second level performance and we have too > >> >> >> >> >> many > >> >> >> >> >> timeout > >> >> >> >> >> now. > >> >> >> >> >> Any idea for optimizing the performance? > >> >> >> >> >> > >> >> >> >> >> Thanks. > >> >> >> >> > > >> >> >> >> > > >> >> >> > > >> >> >> > > >> >> > > >> >> > > >> > > >> > > > > > >
