Using OFF_HEAP_TIERED and Replicated Heap continously grows eventually heap crash

[styriver]

We have a cache configured as Replicated OFF_HEAP_TIERED and when we execute
our load test we watch Heap space grow in JConsole to the point where the
application becomes unresponsive when it reaches heap limit. We see no signs
in JConsole or Cache Metrics that off heap is growing?

Here is our test configuration. 

public class TestLoadIgniteCacheServiceImpl
{

   private static final String keystoreValue = "xxxxxxxxxxxxxxxxxxxx";
   private static final String seed = "xxxxxxxxxxxxxxxxxxxxxxx";
   private Ignite ignite;

   public TestLoadIgniteCacheServiceImpl (String foo)
   {
      CacheConfiguration cacheConfig = new CacheConfiguration();
      cacheConfig.setCacheMode(CacheMode.REPLICATED);
      cacheConfig.setMemoryMode(CacheMemoryMode.OFFHEAP_TIERED);
      cacheConfig.setSwapEnabled(false);
      cacheConfig.setIndexedTypes(java.util.UUID.class,
com.xxx.documentviewer.imaging.cache.beans.TiffPage.class);
      cacheConfig.setOffHeapMaxMemory(0);
      cacheConfig.setSqlOnheapRowCacheSize(100000);
      cacheConfig.setStatisticsEnabled(true);
      cacheConfig.setName("Foo");

      TcpCommunicationSpi tcpCommunicationSpi = new TcpCommunicationSpi();
      tcpCommunicationSpi.setSocketWriteTimeout(4000);

      TcpDiscoverySpi tcpDiscoverySpi = new TcpDiscoverySpi();
      TcpDiscoveryMulticastIpFinder tcpDiscoveryMulticastIpFinder = new
TcpDiscoveryMulticastIpFinder();
      tcpDiscoveryMulticastIpFinder.setMulticastGroup("000.00.00.1");
      tcpDiscoverySpi.setIpFinder(tcpDiscoveryMulticastIpFinder);

      IgniteConfiguration igniteConfig = new IgniteConfiguration();
      igniteConfig.setCommunicationSpi(tcpCommunicationSpi);
      igniteConfig.setDiscoverySpi(tcpDiscoverySpi);
      igniteConfig.setPeerClassLoadingEnabled(false);
      igniteConfig.setSslContextFactory(initializeSecurity());
      igniteConfig.setCacheConfiguration(cacheConfig);

      Ignition.setClientMode(true);
      ignite = Ignition.start(igniteConfig);

      ignite.getOrCreateCache(cacheConfig);

      String nodeId = ignite.cluster().localNode().id().toString();
      System.out.println("Cache Configuration loaded for " +
"ignite-config-" + Environment.RUN_MODE + ". Ignite Node started with Node
Id-" + nodeId);
   }


   private SslContextFactory initializeSecurity ()
   {
      SslContextFactory sslFactory = new SslContextFactory();
      EncryptionHelper helper = new EncryptionHelper(seed);
      char[] decrypted = helper.decrypt(keystoreValue).toCharArray();
     
sslFactory.setKeyStoreFilePath("/home2/cachemanager/xxx-ignite-client.jks");
      sslFactory.setKeyStorePassword(decrypted);
     
sslFactory.setTrustManagers(SslContextFactory.getDisabledTrustManager());
      sslFactory.setProtocol("TLSv1.2");

      return sslFactory;
   }

This is how we are running our load test.

    public String loadCache (int numberOfDocuments,
                            int numberOfpages,
                            TestLoadIgniteCacheServiceImpl
igniteServerCache)
   {
      StrBuilder builder = new StrBuilder();
      try {
         String imageFilePath = System.getProperty("imagePath",
"/home2/cachemanager/testImage.png");
         byte[] image = FileUtils.readFileToByteArray(new
File(imageFilePath));

         IgniteCache<UUID, ImagePage> imageCache =
igniteServerCache.getCache("Foo");
         IgniteDataStreamer<UUID, ImagePage> streamer =
igniteServerCache.getStreamer(IgniteCaches.VIEWER_IMAGE_CACHE_NAME.getName());
         int offHeapSize = imageCache.size(CachePeekMode.OFFHEAP);
         int onHeapSize = imageCache.size(CachePeekMode.ONHEAP);
         builder.appendln("Starting On Heap Size \"" + onHeapSize + "\"");
         builder.appendln("Starting Off Heap Size \"" + offHeapSize + "\"");

         // loop for documents
         for (int i = 1; i <= numberOfDocuments; i++) {
            int documentId = i * -1;
            // loop pages
            for (int p = 1; p <= numberOfpages; p++) {
               TiffPage tiffPage = new
TiffPage(Integer.toString(documentId), image, "png", p);
               //streamer.addData(tiffPage.getId(), tiffPage);
               imageCache.put(tiffPage.getId(), tiffPage);
            }
            builder.appendln("Added document \"" + documentId + "\"");
         }

         offHeapSize = imageCache.size(CachePeekMode.OFFHEAP);
         onHeapSize = imageCache.size(CachePeekMode.ONHEAP);
         builder.appendln("Finish  On Heap Size \"" + onHeapSize + "\"");
         builder.appendln("Finish Off Heap Size \"" + offHeapSize + "\"");

      } catch (Throwable throwable) {
         throwable.printStackTrace();
      }

      return builder.toString();
   }



--
View this message in context: 
http://apache-ignite-users.70518.x6.nabble.com/Using-OFF-HEAP-TIERED-and-Replicated-Heap-continously-grows-eventually-heap-crash-tp8604.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.