Ignite cache.getAll takes a long time

[Justin Ji]

Hi all

I have a cluster with 3 server nodes which have 4 CPU cores and 8Gb memory.
When applications call cache.getAll(set), it often takes a long time almost
500ms(about 0.6%), even the set size is less than 3. 

And I do not think it was caused by the network, because the network latency
between application and server is less than 2 milliseconds.

The CPU of Ignite servers were between 3%~5% and the QPS are about 1000 read
and  1000 write.

Here is my code segment:

Cache:
CacheConfiguration<String, BinaryObject> cacheCfg =
DefaultIgniteConfiguration.getCacheConfiguration(IgniteCacheKey.DATA_POINT.getCode());
        cacheCfg.setBackups(2);
        cacheCfg.setDataRegionName(Constants.THREE_GB_REGION);
       
cacheCfg.setCacheStoreFactory(FactoryBuilder.factoryOf(DataPointCacheStore.class));
        cacheCfg.setWriteThrough(true);
        cacheCfg.setWriteBehindEnabled(true);
        cacheCfg.setWriteBehindFlushThreadCount(2);
        //每15秒刷新一次
        cacheCfg.setWriteBehindFlushFrequency(15 * 1000);
        cacheCfg.setWriteBehindFlushSize(409600);
        cacheCfg.setWriteBehindBatchSize(1024);
        cacheCfg.setStoreKeepBinary(true);
        cacheCfg.setQueryParallelism(4);
        //2M
        cacheCfg.setRebalanceBatchSize(2 * 1024 * 1024);


getAll:

   public Map<String, BinaryObject> getAll(Set<String> keys) {
        long total = totalRead.incrementAndGet();

        long startTime = System.currentTimeMillis();
        Map<String, BinaryObject> all = igniteCache.getAll(keys);
        long currentTime = System.currentTimeMillis();

        long took = currentTime - startTime;
        if (took > 500) {
            readGreaterThan500.incrementAndGet();
            logger.warn("read complete, cost:{}ms", took);
        }
        if (total % 10000 == 0) {
            double percent = 1000.0 * readGreaterThan500.get() / total;
            logger.info("percent that more 500ms:{}‰",
readGreaterThan500.get(), total, format.format(percent));
        }
        return all;
    }

putAllAsync:

public IgniteFuture<Void> putAllAsync(Map<String, DpCache> map) {
        long total = totalWrite.incrementAndGet();

        long startTime = System.currentTimeMillis();

        Map<String, BinaryObject> objectMap = Maps.newHashMap();
        for (Map.Entry<String, DpCache> entry : map.entrySet()) {
            objectMap.put(entry.getKey(),
ignite.binary().toBinary(entry.getValue()));
        }
        IgniteFuture<Void> future = igniteCache.putAllAsync(objectMap);

        long currentTime = System.currentTimeMillis();
        long took = currentTime - startTime;
        if (took > 500) {
            writeGreaterThan500.incrementAndGet();
            logger.warn("write complete, cost:{}ms", took);
        }
        if (total % 10000 == 0) {
            double percent = 1000.0 * writeGreaterThan500.get() / total;
            logger.info("percent that more than 500ms :{}‰",
writeGreaterThan500.get(), total,
                    format.format(percent));
        }
        return future;
    }




--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/