Re: recoveryBallotBoxes in MvccProcessorImpl memory leak?

[mvkarp]

Let me know if these help or if you need anything more specific.
recoveryBallotBoxes.zip

  


ilya.kasnacheev wrote
> Hello!
> 
> Can you please check whether there are any especially large objects inside
> recoveryBallotBoxes object graph? Sorting by retained heap may help in
> determining this. It would be nice to know what is the type histogram of
> what's inside recoveryBallotBoxes and where the bulk of heap usage
> resides.
> 
> Regards,
> -- 
> Ilya Kasnacheev
> 
> 
> чт, 7 нояб. 2019 г. в 06:23, mvkarp 

> liquid_ninja2k@

> :
> 
>> I've attached another set of screenshots, might be more clear.
>> heap.zip
>> http://apache-ignite-users.70518.x6.nabble.com/file/t2658/heap.zip;
>>
>>
>> mvkarp wrote
>> > I've attached some extra screenshots showing what is inside these
>> records
>> > and path to GC roots. heap.zip
>> > 
>> http://apache-ignite-users.70518.x6.nabble.com/file/t2658/heap.zip;
>> >
>> >
>> >
>> >
>> >
>> > --
>> > Sent from: http://apache-ignite-users.70518.x6.nabble.com/
>>
>>
>>
>>
>>
>> --
>> Sent from: http://apache-ignite-users.70518.x6.nabble.com/
>>





--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: Kubernetes- Failed to retrieve Ignite pods IP addresses.

[Gokulnath Chidambaram]

Thanks Andrei. the solution described in
https://apacheignite.readme.io/docs/rbac-authorization works with
minorchange in Role access. If we have restricted access, Role and
Rolebinding works.

Thanks again for your help,

-gokul

On Thu, Nov 7, 2019 at 5:09 PM Andrei Aleksandrov 
wrote:

> Hi,
>
> Using the search I found that current issue was resolved in next thread:
>
>
> http://apache-ignite-users.70518.x6.nabble.com/Ignite-on-RBAC-enabled-K8s-cluster-td22165.html
> I guess that you should also read next article:
>
> https://apacheignite.readme.io/docs/rbac-authorization
>
> BR,
> Andrei
>
> 11/7/2019 2:27 PM, Gokulnath Chidambaram пишет:
>
> Hello,
>
> I am trying to install apache ignite:2.7.6 in kubernetes cluster (deployed
> in aws).
> I created
>
> 1.service account
> 2. role access
> 3.rolebinding
> 4.deployment.
> 4. added namesapce (bean property)  in xml configuration file.
>
> I am getting the following error message.
>
> 11:18:39,724][INFO][main][PartitionsEvictManager] Evict partition
> permits=2
> [11:18:44,719][INFO][main][ClientListenerProcessor] Client connector
> processor has started on TCP port 10800
> [11:18:46,418][INFO][main][GridTcpRestProtocol] Command protocol
> successfully started [name=TCP binary, host=0.0.0.0/0.0.0.0, port=11211]
> [11:18:48,617][WARNING][jvm-pause-detector-worker][IgniteKernal] Possible
> too long JVM pause: 850 milliseconds.
> [11:18:52,420][INFO][main][GridJettyRestProtocol] Command protocol
> successfully started [name=Jetty REST, host=/0.0.0.0, port=8080]
> [11:18:53,721][INFO][main][IgniteKernal] Non-loopback local IPs:
> 10.42.5.132
> [11:18:53,721][INFO][main][IgniteKernal] Enabled local MACs: 363E412C208F
> [11:18:54,625][INFO][main][TcpDiscoverySpi] Connection check threshold is
> calculated: 1
> [11:18:55,024][INFO][main][TcpDiscoverySpi] Successfully bound to TCP port
> [port=47500, localHost=0.0.0.0/0.0.0.0,
> locNodeId=6bc33d41-6dba-422b-832d-ed8cae326a00]
> [11:19:03,423][SEVERE][main][TcpDiscoverySpi] Failed to get registered
> addresses from IP finder on start (retrying every 2000ms; change
> 'reconnectDelay' to configure the frequency of retries).
> class org.apache.ignite.spi.IgniteSpiException: Failed to retrieve Ignite
> pods IP addresses.
> at
> org.apache.ignite.spi.discovery.tcp.ipfinder.kubernetes.TcpDiscoveryKubernetesIpFinder.getRegisteredAddresses(TcpDiscoveryKubernetesIpFinder.java:172)
>
> at
> org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi.registeredAddresses(TcpDiscoverySpi.java:1900)
>
> at
> org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi.resolvedAddresses(TcpDiscoverySpi.java:1848)
>
> at
> org.apache.ignite.spi.discovery.tcp.ServerImpl.sendJoinRequestMessage(ServerImpl.java:1049)
>
> at
> org.apache.ignite.spi.discovery.tcp.ServerImpl.joinTopology(ServerImpl.java:910)
>
> at
> org.apache.ignite.spi.discovery.tcp.ServerImpl.spiStart(ServerImpl.java:391)
>
> at
> org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi.spiStart(TcpDiscoverySpi.java:2020)
>
> at
> org.apache.ignite.internal.managers.GridManagerAdapter.startSpi(GridManagerAdapter.java:297)
>
> at
> org.apache.ignite.internal.managers.discovery.GridDiscoveryManager.start(GridDiscoveryManager.java:939)
>
> at
> org.apache.ignite.internal.IgniteKernal.startManager(IgniteKernal.java:1682)
>
> at org.apache.ignite.internal.IgniteKernal.start(IgniteKernal.java:1066)
> at
> org.apache.ignite.internal.IgnitionEx$IgniteNamedInstance.start0(IgnitionEx.java:2038)
>
> at
> org.apache.ignite.internal.IgnitionEx$IgniteNamedInstance.start(IgnitionEx.java:1730)
>
> at org.apache.ignite.internal.IgnitionEx.start0(IgnitionEx.java:1158)
> at
> org.apache.ignite.internal.IgnitionEx.startConfigurations(IgnitionEx.java:1076)
>
> at org.apache.ignite.internal.IgnitionEx.start(IgnitionEx.java:962)
> at org.apache.ignite.internal.IgnitionEx.start(IgnitionEx.java:861)
> at org.apache.ignite.internal.IgnitionEx.start(IgnitionEx.java:731)
> at org.apache.ignite.internal.IgnitionEx.start(IgnitionEx.java:700)
> at org.apache.ignite.Ignition.start(Ignition.java:348)
> at
> org.apache.ignite.startup.cmdline.CommandLineStartup.main(CommandLineStartup.java:301)
>
> Caused by: java.io.IOException: Server returned HTTP response code: 403
> for URL:
> https://kubernetes.default.svc.cluster.local:443/api/v1/namespaces/dataobjns/endpoints/ignite
> at
> sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1894)
>
> at
> sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1492)
>
> at
> sun.net.www.protocol.https.HttpsURLConnectionImpl.getInputStream(HttpsURLConnectionImpl.java:263)
>
> at
> org.apache.ignite.spi.discovery.tcp.ipfinder.kubernetes.TcpDiscoveryKubernetesIpFinder.getRegisteredAddresses(TcpDiscoveryKubernetesIpFinder.java:153)
>
> ... 20 more
> [11:19:15,821][WARNING][jvm-pause-detector-worker][IgniteKernal] Possible
> too long JVM pause: 500 milliseconds.
>
> any help is appreciated.
>
>

Blocked Worker Thread

[Conrad Mukai (cmukai)]

We are running a cache in a 4 node cluster with atomicityMode set to ATOMIC and 
have persistence enabled. We repeatedly get a SYSTEM_WORKER_BLOCKED error on 
one node which is disabling the entire cluster. We were seeing a lot of sockets 
in TIME_WAIT state which was blocking clients from connecting so we did the 
following on all the nodes:

# ignore TIME_WAIT state on sockets
echo "1" > /proc/sys/net/ipv4/tcp_tw_reuse
echo "1" > /proc/sys/net/ipv4/tcp_tw_recycle

This made that issue go away, but may play a part in this new issue. First 
question is what is the root cause of the error? The second question is why 
does this bring down the entire cluster?

Here is the error message:

[2019-11-07 16:36:22,037][ERROR][tcp-disco-msg-worker-#2][root] Critical system 
error detected. Will be handled accordingly to configured handler 
[hnd=StopNodeOrHaltFailureHandler [tryStop=false, timeout=0, 
super=AbstractFailureHandler [ignoredFailureTypes=[SYSTEM_WORKER_BLOCKED, 
SYSTEM_CRITICAL_OPERATION_TIMEOUT]]], failureCtx=FailureContext 
[type=SYSTEM_WORKER_BLOCKED, err=class o.a.i.IgniteException: GridWorker 
[name=partition-exchanger, igniteInstanceName=null, finished=false, 
heartbeatTs=1573090509428]]]
class org.apache.ignite.IgniteException: GridWorker [name=partition-exchanger, 
igniteInstanceName=null, finished=false, heartbeatTs=1573090509428]
at 
org.apache.ignite.internal.IgnitionEx$IgniteNamedInstance$2.apply(IgnitionEx.java:1831)
at 
org.apache.ignite.internal.IgnitionEx$IgniteNamedInstance$2.apply(IgnitionEx.java:1826)
at 
org.apache.ignite.internal.worker.WorkersRegistry.onIdle(WorkersRegistry.java:233)
at 
org.apache.ignite.internal.util.worker.GridWorker.onIdle(GridWorker.java:297)
at 
org.apache.ignite.spi.discovery.tcp.ServerImpl$RingMessageWorker.lambda$new$0(ServerImpl.java:2663)
at 
org.apache.ignite.spi.discovery.tcp.ServerImpl$MessageWorker.body(ServerImpl.java:7181)
at 
org.apache.ignite.spi.discovery.tcp.ServerImpl$RingMessageWorker.body(ServerImpl.java:2700)
at 
org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:120)
at 
org.apache.ignite.spi.discovery.tcp.ServerImpl$MessageWorkerThread.body(ServerImpl.java:7119)
at org.apache.ignite.spi.IgniteSpiThread.run(IgniteSpiThread.java:62)

This is followed by a warning and a thread dump:

[2019-11-07 16:36:22,038][WARN ][tcp-disco-msg-worker-#2][FailureProcessor] No 
deadlocked threads detected.
[2019-11-07 16:36:22,328][WARN ][tcp-disco-msg-worker-#2][FailureProcessor] 
Thread dump at 2019/11/07 16:36:22 GMT

For the particular thread in the error and warning messages here is the thread 
dump:

Thread [name="tcp-disco-msg-worker-#2", id=113, state=RUNNABLE, blockCnt=211, 
waitCnt=4745368]
at sun.management.ThreadImpl.dumpThreads0(Native Method)
at sun.management.ThreadImpl.dumpAllThreads(ThreadImpl.java:454)
at o.a.i.i.util.IgniteUtils.dumpThreads(IgniteUtils.java:1368)
at 
o.a.i.i.processors.failure.FailureProcessor.process(FailureProcessor.java:128)
- locked o.a.i.i.processors.failure.FailureProcessor@7e65ceba
at 
o.a.i.i.processors.failure.FailureProcessor.process(FailureProcessor.java:104)
at o.a.i.i.IgnitionEx$IgniteNamedInstance$2.apply(IgnitionEx.java:1829)
at o.a.i.i.IgnitionEx$IgniteNamedInstance$2.apply(IgnitionEx.java:1826)
at o.a.i.i.worker.WorkersRegistry.onIdle(WorkersRegistry.java:233)
at o.a.i.i.util.worker.GridWorker.onIdle(GridWorker.java:297)
at 
o.a.i.spi.discovery.tcp.ServerImpl$RingMessageWorker.lambda$new$0(ServerImpl.java:2663)
at 
o.a.i.spi.discovery.tcp.ServerImpl$RingMessageWorker$$Lambda$47/1047515321.run(Unknown
 Source)
at 
o.a.i.spi.discovery.tcp.ServerImpl$MessageWorker.body(ServerImpl.java:7181)
at 
o.a.i.spi.discovery.tcp.ServerImpl$RingMessageWorker.body(ServerImpl.java:2700)
at o.a.i.i.util.worker.GridWorker.run(GridWorker.java:120)
at 
o.a.i.spi.discovery.tcp.ServerImpl$MessageWorkerThread.body(ServerImpl.java:7119)
at o.a.i.spi.IgniteSpiThread.run(IgniteSpiThread.java:62)

In addition all the system threads are in TIMED_WAITING state:

Thread [name="sys-#7099", id=9252, state=TIMED_WAITING, blockCnt=0, waitCnt=1]
Lock 
[object=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@677ec573,
 ownerName=null, ownerId=-1]
at sun.misc.Unsafe.park(Native Method)
at 
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
at 
java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467)
at 
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073)
at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
at 

Re: resetLostPartitions is blocked inside event listener

[Ilya Kasnacheev]

Hello!

Event listener is invoked synchronously from internal threads. If partition
reset has to happen from the same thread, then obviously there will be a
deadlock.

Cache listeners have same property, i.e., you should avoid doing cache
operations from them.

This is tradeoff between performance and usability which was resolved in
favor of former.

Regards,
-- 
Ilya Kasnacheev


чт, 7 нояб. 2019 г. в 20:30, Prasad Bhalerao :

> Do you mean to say, spawn a different thread from event listener and reset
> the lost partition in that thread?
>
> I tried this and it works.
>
> But wanted to understand the reason, why this call get blocked in event
> listener?
>
> Thanks,
> Prasad
>
> On Thu 7 Nov, 2019, 9:28 PM Ilya Kasnacheev  wrote:
>
>> Hello!
>>
>> It is not advisable to call any blocking methods from event listeners.
>> Just fire resetLostPartitions from another thread.
>>
>> Regards,
>> --
>> Ilya Kasnacheev
>>
>>
>> чт, 7 нояб. 2019 г. в 15:17, Akash Shinde :
>>
>>> Hi,
>>> I am trying to handle lost partition scenario.
>>> I have written event listener listening  to
>>> EventType.EVT_CACHE_REBALANCE_PART_DATA_LOST event.
>>> I want to reset lost partition state of cache after cache loading  is
>>> done.
>>> *Issue:* ignite.resetLostPartitions(caheName) is getting blocked and
>>> not completing.
>>>
>>> Please find the code for Event Listener. Someone can help on this. *Why
>>> this resetLostPartitions getting blocked.*
>>>
>>> public class IgniteEventListner implements 
>>> IgnitePredicate {
>>>private static final Logger LOGGER = 
>>> LoggerFactory.getLogger(IgniteEventListner.class);
>>>
>>>   private final Ignite ignite;
>>>
>>>   public IgniteEventListner(Ignite ignite) {
>>> this.ignite = ignite;
>>>   }
>>>
>>>   @Override
>>>   public boolean apply(CacheRebalancingEvent evt) {
>>>
>>> IgniteCache cache = 
>>> ignite.getOrCreateCache(CacheName.ASSET_GROUP_CACHE.name());
>>> Collection lostPartitions = cache.lostPartitions();
>>> reloadCache(lostPartitions); //perform partition based cache loading
>>>
>>>* 
>>> ignite.resetLostPartitions(Arrays.asList(CacheName.ASSET_GROUP_CACHE.name()));
>>>   //Reset partitions*
>>>
>>> System.out.println("Check-1, Partition lost event processed");
>>>
>>> return true;
>>>   }
>>> }
>>>
>>> *Cache Configuration*
>>>
>>> private CacheConfiguration assetGroupCacheCfg() {
>>> CacheConfiguration assetGroupCacheCfg = new 
>>> CacheConfiguration<>(CacheName.ASSET_GROUP_CACHE.name());
>>> assetGroupCacheCfg.setAtomicityMode(CacheAtomicityMode.TRANSACTIONAL);
>>> assetGroupCacheCfg.setWriteThrough(false);
>>> assetGroupCacheCfg.setReadThrough(false);
>>> assetGroupCacheCfg.setRebalanceMode(CacheRebalanceMode.ASYNC);
>>> assetGroupCacheCfg.setBackups(0);
>>> assetGroupCacheCfg.setCacheMode(CacheMode.PARTITIONED);
>>> assetGroupCacheCfg.setIndexedTypes(DefaultDataAffinityKey.class, 
>>> AssetGroupData.class);
>>> assetGroupCacheCfg.setSqlIndexMaxInlineSize(100);
>>> RendezvousAffinityFunction affinityFunction = new 
>>> RendezvousAffinityFunction();
>>> assetGroupCacheCfg.setAffinity(affinityFunction);
>>> assetGroupCacheCfg.setStatisticsEnabled(true);
>>>
>>> assetGroupCacheCfg.setPartitionLossPolicy(PartitionLossPolicy.READ_WRITE_SAFE);
>>> return assetGroupCacheCfg;
>>>   }
>>>
>>> *Ignite Configuration*
>>>
>>> private IgniteConfiguration getIgniteConfiguration() {
>>>
>>>   TcpDiscoveryVmIpFinder ipFinder = new TcpDiscoveryVmIpFinder();
>>>   String[] hosts = {"127.0.0.1:47500..47509"};
>>>   ipFinder.setAddresses(Arrays.asList(hosts));
>>>
>>>   TcpDiscoverySpi discoSpi = new TcpDiscoverySpi();
>>>   discoSpi.setIpFinder(ipFinder);
>>>
>>>   IgniteConfiguration cfg = new IgniteConfiguration();
>>>   cfg.setDiscoverySpi(discoSpi);
>>>   cfg.setIgniteInstanceName("springDataNode");
>>>   cfg.setPeerClassLoadingEnabled(false);
>>>   cfg.setRebalanceThreadPoolSize(4);
>>>   DataStorageConfiguration storageCfg = new DataStorageConfiguration();
>>>   DataRegionConfiguration regionConfiguration = new 
>>> DataRegionConfiguration();
>>>   regionConfiguration.setInitialSize(3L * 1024 * 1024 * 1024);
>>>   regionConfiguration.setMaxSize(3L * 1024 * 1024 * 1024);
>>>   regionConfiguration.setMetricsEnabled(true);
>>>
>>>   storageCfg.setDefaultDataRegionConfiguration(regionConfiguration);
>>>   storageCfg.setStoragePath("c:/ignite-storage/storage");
>>>   storageCfg.setWalPath("c:/ignite-storage/storage/wal");
>>>   storageCfg.setWalArchivePath("c:/ignite-storage/storage/wal-archive");
>>>   storageCfg.setMetricsEnabled(true);
>>>   cfg.setDataStorageConfiguration(storageCfg);
>>>   
>>> cfg.setIncludeEventTypes(EventType.EVT_CACHE_REBALANCE_PART_DATA_LOST,EventType.EVT_NODE_FAILED);
>>>   cfg.setCacheConfiguration(getCacheConfigurations());
>>>   return cfg;
>>> }
>>>
>>>
>>> Thanks,
>>>
>>> Akash
>>>
>>>

Re: Service grid webinar

[Prasad Bhalerao]

Can you please share the recording of webinar  in case  if you have it?

On Wed 6 Nov, 2019, 3:30 AM Denis Mekhanikov  Hi Igniters!
>
> I’ve been working on the Service Grid functionality in Apache Ignite for a
> while, and at some point I've decided to make a webinar with a high-level
> overview of this part of the project.
>
> If you want to learn more about services, look at some use-cases or just
> ask a few questions somebody who takes part in the development, please feel
> free to join the presentation on November 6th, at 10 AM PST.
>
> You can sign up by the following link:
> https://www.gridgain.com/resources/webinars/best-practices-microservices-architecture-apache-ignite
>
> Denis
>

Re: resetLostPartitions is blocked inside event listener

[Prasad Bhalerao]

Do you mean to say, spawn a different thread from event listener and reset
the lost partition in that thread?

I tried this and it works.

But wanted to understand the reason, why this call get blocked in event
listener?

Thanks,
Prasad

On Thu 7 Nov, 2019, 9:28 PM Ilya Kasnacheev  Hello!
>
> It is not advisable to call any blocking methods from event listeners.
> Just fire resetLostPartitions from another thread.
>
> Regards,
> --
> Ilya Kasnacheev
>
>
> чт, 7 нояб. 2019 г. в 15:17, Akash Shinde :
>
>> Hi,
>> I am trying to handle lost partition scenario.
>> I have written event listener listening  to
>> EventType.EVT_CACHE_REBALANCE_PART_DATA_LOST event.
>> I want to reset lost partition state of cache after cache loading  is
>> done.
>> *Issue:* ignite.resetLostPartitions(caheName) is getting blocked and not
>> completing.
>>
>> Please find the code for Event Listener. Someone can help on this. *Why
>> this resetLostPartitions getting blocked.*
>>
>> public class IgniteEventListner implements 
>> IgnitePredicate {
>>private static final Logger LOGGER = 
>> LoggerFactory.getLogger(IgniteEventListner.class);
>>
>>   private final Ignite ignite;
>>
>>   public IgniteEventListner(Ignite ignite) {
>> this.ignite = ignite;
>>   }
>>
>>   @Override
>>   public boolean apply(CacheRebalancingEvent evt) {
>>
>> IgniteCache cache = 
>> ignite.getOrCreateCache(CacheName.ASSET_GROUP_CACHE.name());
>> Collection lostPartitions = cache.lostPartitions();
>> reloadCache(lostPartitions); //perform partition based cache loading
>>
>>* 
>> ignite.resetLostPartitions(Arrays.asList(CacheName.ASSET_GROUP_CACHE.name()));
>>   //Reset partitions*
>>
>> System.out.println("Check-1, Partition lost event processed");
>>
>> return true;
>>   }
>> }
>>
>> *Cache Configuration*
>>
>> private CacheConfiguration assetGroupCacheCfg() {
>> CacheConfiguration assetGroupCacheCfg = new 
>> CacheConfiguration<>(CacheName.ASSET_GROUP_CACHE.name());
>> assetGroupCacheCfg.setAtomicityMode(CacheAtomicityMode.TRANSACTIONAL);
>> assetGroupCacheCfg.setWriteThrough(false);
>> assetGroupCacheCfg.setReadThrough(false);
>> assetGroupCacheCfg.setRebalanceMode(CacheRebalanceMode.ASYNC);
>> assetGroupCacheCfg.setBackups(0);
>> assetGroupCacheCfg.setCacheMode(CacheMode.PARTITIONED);
>> assetGroupCacheCfg.setIndexedTypes(DefaultDataAffinityKey.class, 
>> AssetGroupData.class);
>> assetGroupCacheCfg.setSqlIndexMaxInlineSize(100);
>> RendezvousAffinityFunction affinityFunction = new 
>> RendezvousAffinityFunction();
>> assetGroupCacheCfg.setAffinity(affinityFunction);
>> assetGroupCacheCfg.setStatisticsEnabled(true);
>>
>> assetGroupCacheCfg.setPartitionLossPolicy(PartitionLossPolicy.READ_WRITE_SAFE);
>> return assetGroupCacheCfg;
>>   }
>>
>> *Ignite Configuration*
>>
>> private IgniteConfiguration getIgniteConfiguration() {
>>
>>   TcpDiscoveryVmIpFinder ipFinder = new TcpDiscoveryVmIpFinder();
>>   String[] hosts = {"127.0.0.1:47500..47509"};
>>   ipFinder.setAddresses(Arrays.asList(hosts));
>>
>>   TcpDiscoverySpi discoSpi = new TcpDiscoverySpi();
>>   discoSpi.setIpFinder(ipFinder);
>>
>>   IgniteConfiguration cfg = new IgniteConfiguration();
>>   cfg.setDiscoverySpi(discoSpi);
>>   cfg.setIgniteInstanceName("springDataNode");
>>   cfg.setPeerClassLoadingEnabled(false);
>>   cfg.setRebalanceThreadPoolSize(4);
>>   DataStorageConfiguration storageCfg = new DataStorageConfiguration();
>>   DataRegionConfiguration regionConfiguration = new 
>> DataRegionConfiguration();
>>   regionConfiguration.setInitialSize(3L * 1024 * 1024 * 1024);
>>   regionConfiguration.setMaxSize(3L * 1024 * 1024 * 1024);
>>   regionConfiguration.setMetricsEnabled(true);
>>
>>   storageCfg.setDefaultDataRegionConfiguration(regionConfiguration);
>>   storageCfg.setStoragePath("c:/ignite-storage/storage");
>>   storageCfg.setWalPath("c:/ignite-storage/storage/wal");
>>   storageCfg.setWalArchivePath("c:/ignite-storage/storage/wal-archive");
>>   storageCfg.setMetricsEnabled(true);
>>   cfg.setDataStorageConfiguration(storageCfg);
>>   
>> cfg.setIncludeEventTypes(EventType.EVT_CACHE_REBALANCE_PART_DATA_LOST,EventType.EVT_NODE_FAILED);
>>   cfg.setCacheConfiguration(getCacheConfigurations());
>>   return cfg;
>> }
>>
>>
>> Thanks,
>>
>> Akash
>>
>>

Re: resetLostPartitions is blocked inside event listener

[Ilya Kasnacheev]

Hello!

It is not advisable to call any blocking methods from event listeners. Just
fire resetLostPartitions from another thread.

Regards,
-- 
Ilya Kasnacheev


чт, 7 нояб. 2019 г. в 15:17, Akash Shinde :

> Hi,
> I am trying to handle lost partition scenario.
> I have written event listener listening  to
> EventType.EVT_CACHE_REBALANCE_PART_DATA_LOST event.
> I want to reset lost partition state of cache after cache loading  is done.
> *Issue:* ignite.resetLostPartitions(caheName) is getting blocked and not
> completing.
>
> Please find the code for Event Listener. Someone can help on this. *Why
> this resetLostPartitions getting blocked.*
>
> public class IgniteEventListner implements 
> IgnitePredicate {
>private static final Logger LOGGER = 
> LoggerFactory.getLogger(IgniteEventListner.class);
>
>   private final Ignite ignite;
>
>   public IgniteEventListner(Ignite ignite) {
> this.ignite = ignite;
>   }
>
>   @Override
>   public boolean apply(CacheRebalancingEvent evt) {
>
> IgniteCache cache = 
> ignite.getOrCreateCache(CacheName.ASSET_GROUP_CACHE.name());
> Collection lostPartitions = cache.lostPartitions();
> reloadCache(lostPartitions); //perform partition based cache loading
>
>* 
> ignite.resetLostPartitions(Arrays.asList(CacheName.ASSET_GROUP_CACHE.name()));
>   //Reset partitions*
>
> System.out.println("Check-1, Partition lost event processed");
>
> return true;
>   }
> }
>
> *Cache Configuration*
>
> private CacheConfiguration assetGroupCacheCfg() {
> CacheConfiguration assetGroupCacheCfg = new 
> CacheConfiguration<>(CacheName.ASSET_GROUP_CACHE.name());
> assetGroupCacheCfg.setAtomicityMode(CacheAtomicityMode.TRANSACTIONAL);
> assetGroupCacheCfg.setWriteThrough(false);
> assetGroupCacheCfg.setReadThrough(false);
> assetGroupCacheCfg.setRebalanceMode(CacheRebalanceMode.ASYNC);
> assetGroupCacheCfg.setBackups(0);
> assetGroupCacheCfg.setCacheMode(CacheMode.PARTITIONED);
> assetGroupCacheCfg.setIndexedTypes(DefaultDataAffinityKey.class, 
> AssetGroupData.class);
> assetGroupCacheCfg.setSqlIndexMaxInlineSize(100);
> RendezvousAffinityFunction affinityFunction = new 
> RendezvousAffinityFunction();
> assetGroupCacheCfg.setAffinity(affinityFunction);
> assetGroupCacheCfg.setStatisticsEnabled(true);
>
> assetGroupCacheCfg.setPartitionLossPolicy(PartitionLossPolicy.READ_WRITE_SAFE);
> return assetGroupCacheCfg;
>   }
>
> *Ignite Configuration*
>
> private IgniteConfiguration getIgniteConfiguration() {
>
>   TcpDiscoveryVmIpFinder ipFinder = new TcpDiscoveryVmIpFinder();
>   String[] hosts = {"127.0.0.1:47500..47509"};
>   ipFinder.setAddresses(Arrays.asList(hosts));
>
>   TcpDiscoverySpi discoSpi = new TcpDiscoverySpi();
>   discoSpi.setIpFinder(ipFinder);
>
>   IgniteConfiguration cfg = new IgniteConfiguration();
>   cfg.setDiscoverySpi(discoSpi);
>   cfg.setIgniteInstanceName("springDataNode");
>   cfg.setPeerClassLoadingEnabled(false);
>   cfg.setRebalanceThreadPoolSize(4);
>   DataStorageConfiguration storageCfg = new DataStorageConfiguration();
>   DataRegionConfiguration regionConfiguration = new DataRegionConfiguration();
>   regionConfiguration.setInitialSize(3L * 1024 * 1024 * 1024);
>   regionConfiguration.setMaxSize(3L * 1024 * 1024 * 1024);
>   regionConfiguration.setMetricsEnabled(true);
>
>   storageCfg.setDefaultDataRegionConfiguration(regionConfiguration);
>   storageCfg.setStoragePath("c:/ignite-storage/storage");
>   storageCfg.setWalPath("c:/ignite-storage/storage/wal");
>   storageCfg.setWalArchivePath("c:/ignite-storage/storage/wal-archive");
>   storageCfg.setMetricsEnabled(true);
>   cfg.setDataStorageConfiguration(storageCfg);
>   
> cfg.setIncludeEventTypes(EventType.EVT_CACHE_REBALANCE_PART_DATA_LOST,EventType.EVT_NODE_FAILED);
>   cfg.setCacheConfiguration(getCacheConfigurations());
>   return cfg;
> }
>
>
> Thanks,
>
> Akash
>
>

Re: Persistence store

[slut_hunter]

The problem was that instead StatefulSet I used Deployment 

Thanks for the help



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: Ignite singel instance

[Ilya Kasnacheev]

Hello!

Have you tried collecting thread dumps from node under load, to identify
any bottlenecks?

What is the CPU usage by Ignite?

Regards,
-- 
Ilya Kasnacheev


ср, 6 нояб. 2019 г. в 09:08, Siew Wai Yow :

> Hi Stan
>
> The result will be the same without transaction, even with atomic mode. So
> wondering if there is other reason capped the performance. Attach is our
> POC source code which give around 1700 TPS, with or without transaction,
> may be you can advise what's wrong with it.
>
> -DIGNITE_MAX_INDEX_PAYLOAD_SIZE=64 is using already.
>
> We choose ODBC because it is more compatible to existing product's API,
> which will need to create table/cache on the fly without pre-define
> C-object. Ignite C++ need to have the cache C-object ready before run.
>
> Also in Ignite cluster, how can we connect to cluster, just connect to one
> of the host? And when host crash we need to change manually connect to
> another node? Seems ODBC connection string not allow to input multiple host
> address.
>
> Thanks in advance.
>
> Regards,
> Yow
>
>
> --
> *From:* Stanislav Lukyanov 
> *Sent:* Wednesday, November 6, 2019 12:33 AM
> *To:* user@ignite.apache.org 
> *Subject:* Re: Ignite singel instance
>
> First, 1700 TPS given your transaction structure is 17 simple
> operations per second, which is quite substantial - especially if you're
> doing that from a single thread / single ODBC client.
>
> Second, note that TRANSACTIONAL_SNAPSHOT is in beta and is not ready for
> production use. There are no claims about performance or stability of that
> cache mode.
>
> Third, you don't need the index on key01 - it is created automatically
> because it is a primary key. But you do need to set INLINE SIZE of the
> default index. Run your Ignite server with system property or environment
> varialy IGNITE_MAX_INDEX_PAYLOAD_SIZE=64.
>
> Finally, the operations you're doing don't look like something to be done
> with SQL. Consider using key-value API in Ignite C++ instead -
> https://apacheignite-cpp.readme.io/docs.
>
> Stan
>
> On Sat, Nov 2, 2019 at 8:15 PM Evgenii Zhuravlev 
> wrote:
>
> Hi,
>
> Do you use only one ODBC client? Can you start one more ODBC client and
> check the performance?
>
> Thanks,
> Evgenii
>
> сб, 2 нояб. 2019 г. в 16:47, Siew Wai Yow :
>
> Hi,
>
> We are doing POC on ignite performance using ODBC driver but the
> performance is always capped at around 1700 TPS which is too slow. It is
> local ignite service. All tuning tips from Ignite page has been applied, no
> bottleneck from CPU and Memory. At the moment not turn on persistence yet,
> it will be worse if turn on. This POC is very crucial to our product
> roadmap.
>
> Any tips? Thank you.
>
> Test case,
> 1 x insert --> 49 x select -->49 x update --> 1 x delete
> repeat for 5 times.
>
> *const char* insert_sql = "insert into CDRTEST
> values(?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?)";*
> *const char* update_sql = "update CDRTEST set
> value01=?,value02=?,value03=?,value04=?,value05=?,value21=?,value22=?,value23=?,value24=?,value25=?
> where key01=?";*
> *const char* delete_sql = "delete from CDRTEST where key01=?";*
> *const char* select_sql = "select value01 from CDRTEST where key01=?";*
>
> *retcode = SQLExecDirect(hstmt,
> reinterpret_cast(const_cast("CREATE TABLE IF NOT EXISTS
> CDRTEST ( "*
> *  "key01 VARCHAR PRIMARY KEY, "*
> *  "value01 LONG, "*
> *  "value02 LONG, "*
> *  "value03 LONG, "*
> *  "value04 LONG, "*
> *  "value05 LONG, "*
> *  "value06 LONG, "*
> *  "value07 LONG, "*
> *  "value08 LONG, "*
> *  "value09 LONG, "*
> *  "value10 LONG, "*
> *  "value11 LONG, "*
> *  "value12 LONG, "*
> *  "value13 LONG, "*
> *  "value14 LONG, "*
> *  "value15 LONG, "*
> *  "value16 LONG, "*
> *  "value17 LONG, "*
> *  "value18 LONG, "*
> *  "value19 LONG, "*
> *  "value20 LONG, "   *
> *  "value21 VARCHAR, "*
> *  "value22 VARCHAR, "*
> *  "value23 VARCHAR, "*
> *  "value24 VARCHAR, "*
> *  "value25 VARCHAR, "*
> *  "value26 VARCHAR, "*
> *  "value27 VARCHAR, "*
> *  "value28 VARCHAR, "*
> *  "value29 VARCHAR, "*
> *  "value30 VARCHAR, "*
> *  "value31 VARCHAR, "*
> *  "value32 VARCHAR, "*
> *  "value33 VARCHAR, "*
> *  "value34 VARCHAR, "*
> *  "value35 VARCHAR, "*
> *  "value36 VARCHAR, "*
> *  "value37 VARCHAR, "*
> *  "value38 VARCHAR, "*
> *  "value39 VARCHAR) "*
> *  "WITH
> \"template=partitioned,atomicity=TRANSACTIONAL_SNAPSHOT,WRITE_SYNCHRONIZATION_MODE=FULL_ASYNC\"")),
> SQL_NTS);*
> *CHECK_ERROR(retcode, "Fail to create table", hstmt,
> SQL_HANDLE_STMT);*
>
> *retcode = SQLExecDirect(hstmt,
> reinterpret_cast(const_cast("CREATE INDEX key01t_idx ON
> CDRTEST(key01) INLINE_SIZE 64")), SQL_NTS);*
>
> *CHECK_ERROR(retcode, "Fail to create index", hstmt,
> SQL_HANDLE_STMT); *
>
>
> Below are configuration we used,
>
> **
>
> *http://www.springframework.org/schema/beans
> 

Re: Cluster in AWS can not have more than 100 nodes?

[Ilya Kasnacheev]

Hello!

List of hosts present in IP finder has very little bearing on list of nodes
in cluster.

Server node's host do not have to be present in IP finder in order for this
node to function. Cluster will be operational until there exists at least
one node which is a) alive, b) reachable, and c) listed in IP finder. In
your case there are 100 of such nodes.

Please share the latest "Topology snapshot" messages from your nodes.

Regards,
-- 
Ilya Kasnacheev


вт, 5 нояб. 2019 г. в 18:51, Stanislav Lukyanov :

> Each node is supposed to add its own IP and port to the S3 bucket when it
> starts. That said, I wouldn't check the cluster state based on the contents
> of the bucket alone.
> Check your logs for errors. Try using some tools (e.g. check out Web
> Console - either the one in Ignite
> https://apacheignite-tools.readme.io/docs/getting-started or from
> GridGain
> https://www.gridgain.com/products/software/enterprise-edition/management-tool)
> to see whether all nodes are there.
>
> If you don't see any clues, please share your logs from any of the running
> servers. Ideally, a log from one of the 100 servers which joined the
> cluster, and a log from one of the 28 that didn't.
>
> Stan
>
> On Fri, Nov 1, 2019 at 2:41 AM codeboyyong  wrote:
>
>> No . The thing is we started ignite in embedded mode as web app, we can
>> see
>> there are 128 instances.
>> But when we check he S3 bucket , it always only have 100 its, which means
>> only 100 nodes joined the ignite cluster.  Not sure if there are another
>> way
>> to get the cluster size.
>> Since we are running as web app, we actually dont have all the access of
>> command line tools.
>>
>> Thank You
>>
>>
>>
>> --
>> Sent from: http://apache-ignite-users.70518.x6.nabble.com/
>>
>

resetLostPartitions is blocked inside event listener

[Akash Shinde]

Hi,
I am trying to handle lost partition scenario.
I have written event listener listening  to
EventType.EVT_CACHE_REBALANCE_PART_DATA_LOST event.
I want to reset lost partition state of cache after cache loading  is done.
*Issue:* ignite.resetLostPartitions(caheName) is getting blocked and not
completing.

Please find the code for Event Listener. Someone can help on this. *Why
this resetLostPartitions getting blocked.*

public class IgniteEventListner implements
IgnitePredicate {
   private static final Logger LOGGER =
LoggerFactory.getLogger(IgniteEventListner.class);

  private final Ignite ignite;

  public IgniteEventListner(Ignite ignite) {
this.ignite = ignite;
  }

  @Override
  public boolean apply(CacheRebalancingEvent evt) {

IgniteCache cache =
ignite.getOrCreateCache(CacheName.ASSET_GROUP_CACHE.name());
Collection lostPartitions = cache.lostPartitions();
reloadCache(lostPartitions); //perform partition based cache loading

   * 
ignite.resetLostPartitions(Arrays.asList(CacheName.ASSET_GROUP_CACHE.name()));
 //Reset partitions*

System.out.println("Check-1, Partition lost event processed");

return true;
  }
}

*Cache Configuration*

private CacheConfiguration assetGroupCacheCfg() {
CacheConfiguration assetGroupCacheCfg = new
CacheConfiguration<>(CacheName.ASSET_GROUP_CACHE.name());
assetGroupCacheCfg.setAtomicityMode(CacheAtomicityMode.TRANSACTIONAL);
assetGroupCacheCfg.setWriteThrough(false);
assetGroupCacheCfg.setReadThrough(false);
assetGroupCacheCfg.setRebalanceMode(CacheRebalanceMode.ASYNC);
assetGroupCacheCfg.setBackups(0);
assetGroupCacheCfg.setCacheMode(CacheMode.PARTITIONED);
assetGroupCacheCfg.setIndexedTypes(DefaultDataAffinityKey.class,
AssetGroupData.class);
assetGroupCacheCfg.setSqlIndexMaxInlineSize(100);
RendezvousAffinityFunction affinityFunction = new
RendezvousAffinityFunction();
assetGroupCacheCfg.setAffinity(affinityFunction);
assetGroupCacheCfg.setStatisticsEnabled(true);
   
assetGroupCacheCfg.setPartitionLossPolicy(PartitionLossPolicy.READ_WRITE_SAFE);
return assetGroupCacheCfg;
  }

*Ignite Configuration*

private IgniteConfiguration getIgniteConfiguration() {

  TcpDiscoveryVmIpFinder ipFinder = new TcpDiscoveryVmIpFinder();
  String[] hosts = {"127.0.0.1:47500..47509"};
  ipFinder.setAddresses(Arrays.asList(hosts));

  TcpDiscoverySpi discoSpi = new TcpDiscoverySpi();
  discoSpi.setIpFinder(ipFinder);

  IgniteConfiguration cfg = new IgniteConfiguration();
  cfg.setDiscoverySpi(discoSpi);
  cfg.setIgniteInstanceName("springDataNode");
  cfg.setPeerClassLoadingEnabled(false);
  cfg.setRebalanceThreadPoolSize(4);
  DataStorageConfiguration storageCfg = new DataStorageConfiguration();
  DataRegionConfiguration regionConfiguration = new DataRegionConfiguration();
  regionConfiguration.setInitialSize(3L * 1024 * 1024 * 1024);
  regionConfiguration.setMaxSize(3L * 1024 * 1024 * 1024);
  regionConfiguration.setMetricsEnabled(true);

  storageCfg.setDefaultDataRegionConfiguration(regionConfiguration);
  storageCfg.setStoragePath("c:/ignite-storage/storage");
  storageCfg.setWalPath("c:/ignite-storage/storage/wal");
  storageCfg.setWalArchivePath("c:/ignite-storage/storage/wal-archive");
  storageCfg.setMetricsEnabled(true);
  cfg.setDataStorageConfiguration(storageCfg);
  
cfg.setIncludeEventTypes(EventType.EVT_CACHE_REBALANCE_PART_DATA_LOST,EventType.EVT_NODE_FAILED);
  cfg.setCacheConfiguration(getCacheConfigurations());
  return cfg;
}


Thanks,

Akash

Re: Kubernetes- Failed to retrieve Ignite pods IP addresses.

[Andrei Aleksandrov]

Hi,

Using the search I found that current issue was resolved in next thread:

http://apache-ignite-users.70518.x6.nabble.com/Ignite-on-RBAC-enabled-K8s-cluster-td22165.html

I guess that you should also read next article:

https://apacheignite.readme.io/docs/rbac-authorization

BR,
Andrei

11/7/2019 2:27 PM, Gokulnath Chidambaram пишет:

Hello,

I am trying to install apache ignite:2.7.6 in kubernetes cluster 
(deployed in aws).

I created

1.service account
2. role access
3.rolebinding
4.deployment.
4. added namesapce (bean property)  in xml configuration file.

I am getting the following error message.

11:18:39,724][INFO][main][PartitionsEvictManager] Evict partition 
permits=2
[11:18:44,719][INFO][main][ClientListenerProcessor] Client connector 
processor has started on TCP port 10800
[11:18:46,418][INFO][main][GridTcpRestProtocol] Command protocol 
successfully started [name=TCP binary, host=0.0.0.0/0.0.0.0 
, port=11211]
[11:18:48,617][WARNING][jvm-pause-detector-worker][IgniteKernal] 
Possible too long JVM pause: 850 milliseconds.
[11:18:52,420][INFO][main][GridJettyRestProtocol] Command protocol 
successfully started [name=Jetty REST, host=/0.0.0.0 , 
port=8080]
[11:18:53,721][INFO][main][IgniteKernal] Non-loopback local IPs: 
10.42.5.132

[11:18:53,721][INFO][main][IgniteKernal] Enabled local MACs: 363E412C208F
[11:18:54,625][INFO][main][TcpDiscoverySpi] Connection check threshold 
is calculated: 1
[11:18:55,024][INFO][main][TcpDiscoverySpi] Successfully bound to TCP 
port [port=47500, localHost=0.0.0.0/0.0.0.0 , 
locNodeId=6bc33d41-6dba-422b-832d-ed8cae326a00]
[11:19:03,423][SEVERE][main][TcpDiscoverySpi] Failed to get registered 
addresses from IP finder on start (retrying every 2000ms; change 
'reconnectDelay' to configure the frequency of retries).
class org.apache.ignite.spi.IgniteSpiException: Failed to retrieve 
Ignite pods IP addresses.
at 
org.apache.ignite.spi.discovery.tcp.ipfinder.kubernetes.TcpDiscoveryKubernetesIpFinder.getRegisteredAddresses(TcpDiscoveryKubernetesIpFinder.java:172) 

at 
org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi.registeredAddresses(TcpDiscoverySpi.java:1900) 

at 
org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi.resolvedAddresses(TcpDiscoverySpi.java:1848) 

at 
org.apache.ignite.spi.discovery.tcp.ServerImpl.sendJoinRequestMessage(ServerImpl.java:1049) 

at 
org.apache.ignite.spi.discovery.tcp.ServerImpl.joinTopology(ServerImpl.java:910) 

at 
org.apache.ignite.spi.discovery.tcp.ServerImpl.spiStart(ServerImpl.java:391) 

at 
org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi.spiStart(TcpDiscoverySpi.java:2020) 

at 
org.apache.ignite.internal.managers.GridManagerAdapter.startSpi(GridManagerAdapter.java:297) 

at 
org.apache.ignite.internal.managers.discovery.GridDiscoveryManager.start(GridDiscoveryManager.java:939) 

at 
org.apache.ignite.internal.IgniteKernal.startManager(IgniteKernal.java:1682) 


at org.apache.ignite.internal.IgniteKernal.start(IgniteKernal.java:1066)
at 
org.apache.ignite.internal.IgnitionEx$IgniteNamedInstance.start0(IgnitionEx.java:2038) 

at 
org.apache.ignite.internal.IgnitionEx$IgniteNamedInstance.start(IgnitionEx.java:1730) 


at org.apache.ignite.internal.IgnitionEx.start0(IgnitionEx.java:1158)
at 
org.apache.ignite.internal.IgnitionEx.startConfigurations(IgnitionEx.java:1076) 


at org.apache.ignite.internal.IgnitionEx.start(IgnitionEx.java:962)
at org.apache.ignite.internal.IgnitionEx.start(IgnitionEx.java:861)
at org.apache.ignite.internal.IgnitionEx.start(IgnitionEx.java:731)
at org.apache.ignite.internal.IgnitionEx.start(IgnitionEx.java:700)
at org.apache.ignite.Ignition.start(Ignition.java:348)
at 
org.apache.ignite.startup.cmdline.CommandLineStartup.main(CommandLineStartup.java:301) 

Caused by: java.io.IOException: Server returned HTTP response code: 
403 for URL: 
https://kubernetes.default.svc.cluster.local:443/api/v1/namespaces/dataobjns/endpoints/ignite 

at 
sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1894) 

at 
sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1492) 

at 
sun.net.www.protocol.https.HttpsURLConnectionImpl.getInputStream(HttpsURLConnectionImpl.java:263) 

at 
org.apache.ignite.spi.discovery.tcp.ipfinder.kubernetes.TcpDiscoveryKubernetesIpFinder.getRegisteredAddresses(TcpDiscoveryKubernetesIpFinder.java:153) 


... 20 more
[11:19:15,821][WARNING][jvm-pause-detector-worker][IgniteKernal] 
Possible too long JVM pause: 500 milliseconds.


any help is appreciated.

Re: Spring Data native Query support

[Ilya Kasnacheev]

Hello!

Judging from stack trace, you ended up with Hibernate repository and not
with Ignite repository.

Regards,
-- 
Ilya Kasnacheev


ср, 30 окт. 2019 г. в 23:25, niamin :

> Does Ignite provide native Query support using Spring Data? I am not able
> to
> bootstrap my application when I configure my Repository to include a method
> that implements a native query as below:
>
> @Repository
> @RepositoryConfig(cacheName = "FOO")
> public interface ARInvoiceRepository
> extends IgniteRepository {
> @Query(value = "SELECT * FROM AR_INVOICE i where i.customerId =
> :customerId " +
> "and i.installmentFl = 'N' and i.excludeFromArFl = 'N' and
> ((i.installmentNo > 0 " +
> "and i.installmentStartDt <= :toDate) or i.tranDt <= :toDate)
> and i.status != 'H' ",
> nativeQuery = true)
> List getInvoicesForOIStatement(@Param("customerId") String
> customerId, @Param("toDate") Date statementDate);
> }
>
> When I start my application I get an error as below:
>
> Caused by: java.lang.IllegalArgumentException: Not a managed type: class
> com.gpc.rpm.bo.ARInvoice
> at
>
> org.hibernate.metamodel.internal.MetamodelImpl.managedType(MetamodelImpl.java:473)
> ~[hibernate-core-5.2.17.Final.jar:5.2.17.Final]
> at
>
> org.springframework.data.jpa.repository.support.JpaMetamodelEntityInformation.(JpaMetamodelEntityInformation.java:73)
> ~[spring-data-jpa-2.0.9.RELEASE.jar:2.0.9.RELEASE]
> at
>
> org.springframework.data.jpa.repository.support.JpaEntityInformationSupport.getEntityInformation(JpaEntityInformationSupport.java:66)
> ~[spring-data-jpa-2.0.9.RELEASE.jar:2.0.9.RELEASE]
> at
>
> org.springframework.data.jpa.repository.support.JpaRepositoryFactory.getEntityInformation(JpaRepositoryFactory.java:180)
> ~[spring-data-jpa-2.0.9.RELEASE.jar:2.0.9.RELEASE]
> at
>
> org.springframework.data.jpa.repository.support.JpaRepositoryFactory.getTargetRepository(JpaRepositoryFactory.java:118)
> ~[spring-data-jpa-2.0.9.RELEASE.jar:2.0.9.RELEASE]
> at
>
> org.springframework.data.jpa.repository.support.JpaRepositoryFactory.getTargetRepository(JpaRepositoryFactory.java:101)
> ~[spring-data-jpa-2.0.9.RELEASE.jar:2.0.9.RELEASE]
> at
>
> org.springframework.data.repository.core.support.RepositoryFactorySupport.getRepository(RepositoryFactorySupport.java:304)
> ~[spring-data-commons-2.0.14.RELEASE.jar:2.0.14.RELEASE]
> at
>
> org.springframework.data.repository.core.support.RepositoryFactoryBeanSupport.lambda$afterPropertiesSet$4(RepositoryFactoryBeanSupport.java:290)
> ~[spring-data-commons-2.0.14.RELEASE.jar:2.0.14.RELEASE]
> at org.springframework.data.util.Lazy.getNullable(Lazy.java:141)
> ~[spring-data-commons-2.0.14.RELEASE.jar:2.0.14.RELEASE]
> at org.springframework.data.util.Lazy.get(Lazy.java:63)
> ~[spring-data-commons-2.0.14.RELEASE.jar:2.0.14.RELEASE]
>
>
> I've added @Entity annotation to ARInvoice class but that didn't change the
> error.
>
>
>
> --
> Sent from: http://apache-ignite-users.70518.x6.nabble.com/
>

Kubernetes- Failed to retrieve Ignite pods IP addresses.

[Gokulnath Chidambaram]

Hello,

I am trying to install apache ignite:2.7.6 in kubernetes cluster (deployed
in aws).
I created

1.service account
2. role access
3.rolebinding
4.deployment.
4. added namesapce (bean property)  in xml configuration file.

I am getting the following error message.

11:18:39,724][INFO][main][PartitionsEvictManager] Evict partition permits=2
[11:18:44,719][INFO][main][ClientListenerProcessor] Client connector
processor has started on TCP port 10800
[11:18:46,418][INFO][main][GridTcpRestProtocol] Command protocol
successfully started [name=TCP binary, host=0.0.0.0/0.0.0.0, port=11211]
[11:18:48,617][WARNING][jvm-pause-detector-worker][IgniteKernal] Possible
too long JVM pause: 850 milliseconds.
[11:18:52,420][INFO][main][GridJettyRestProtocol] Command protocol
successfully started [name=Jetty REST, host=/0.0.0.0, port=8080]
[11:18:53,721][INFO][main][IgniteKernal] Non-loopback local IPs:
10.42.5.132
[11:18:53,721][INFO][main][IgniteKernal] Enabled local MACs: 363E412C208F
[11:18:54,625][INFO][main][TcpDiscoverySpi] Connection check threshold is
calculated: 1
[11:18:55,024][INFO][main][TcpDiscoverySpi] Successfully bound to TCP port
[port=47500, localHost=0.0.0.0/0.0.0.0,
locNodeId=6bc33d41-6dba-422b-832d-ed8cae326a00]
[11:19:03,423][SEVERE][main][TcpDiscoverySpi] Failed to get registered
addresses from IP finder on start (retrying every 2000ms; change
'reconnectDelay' to configure the frequency of retries).
class org.apache.ignite.spi.IgniteSpiException: Failed to retrieve Ignite
pods IP addresses.
at
org.apache.ignite.spi.discovery.tcp.ipfinder.kubernetes.TcpDiscoveryKubernetesIpFinder.getRegisteredAddresses(TcpDiscoveryKubernetesIpFinder.java:172)

at
org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi.registeredAddresses(TcpDiscoverySpi.java:1900)

at
org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi.resolvedAddresses(TcpDiscoverySpi.java:1848)

at
org.apache.ignite.spi.discovery.tcp.ServerImpl.sendJoinRequestMessage(ServerImpl.java:1049)

at
org.apache.ignite.spi.discovery.tcp.ServerImpl.joinTopology(ServerImpl.java:910)

at
org.apache.ignite.spi.discovery.tcp.ServerImpl.spiStart(ServerImpl.java:391)

at
org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi.spiStart(TcpDiscoverySpi.java:2020)

at
org.apache.ignite.internal.managers.GridManagerAdapter.startSpi(GridManagerAdapter.java:297)

at
org.apache.ignite.internal.managers.discovery.GridDiscoveryManager.start(GridDiscoveryManager.java:939)

at
org.apache.ignite.internal.IgniteKernal.startManager(IgniteKernal.java:1682)

at org.apache.ignite.internal.IgniteKernal.start(IgniteKernal.java:1066)
at
org.apache.ignite.internal.IgnitionEx$IgniteNamedInstance.start0(IgnitionEx.java:2038)

at
org.apache.ignite.internal.IgnitionEx$IgniteNamedInstance.start(IgnitionEx.java:1730)

at org.apache.ignite.internal.IgnitionEx.start0(IgnitionEx.java:1158)
at
org.apache.ignite.internal.IgnitionEx.startConfigurations(IgnitionEx.java:1076)

at org.apache.ignite.internal.IgnitionEx.start(IgnitionEx.java:962)
at org.apache.ignite.internal.IgnitionEx.start(IgnitionEx.java:861)
at org.apache.ignite.internal.IgnitionEx.start(IgnitionEx.java:731)
at org.apache.ignite.internal.IgnitionEx.start(IgnitionEx.java:700)
at org.apache.ignite.Ignition.start(Ignition.java:348)
at
org.apache.ignite.startup.cmdline.CommandLineStartup.main(CommandLineStartup.java:301)

Caused by: java.io.IOException: Server returned HTTP response code: 403 for
URL:
https://kubernetes.default.svc.cluster.local:443/api/v1/namespaces/dataobjns/endpoints/ignite
at
sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1894)

at
sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1492)

at
sun.net.www.protocol.https.HttpsURLConnectionImpl.getInputStream(HttpsURLConnectionImpl.java:263)

at
org.apache.ignite.spi.discovery.tcp.ipfinder.kubernetes.TcpDiscoveryKubernetesIpFinder.getRegisteredAddresses(TcpDiscoveryKubernetesIpFinder.java:153)

... 20 more
[11:19:15,821][WARNING][jvm-pause-detector-worker][IgniteKernal] Possible
too long JVM pause: 500 milliseconds.

any help is appreciated.

Re: apache ignite installation fails in kubernetes

[Gokulnath Chidambaram]

thanks Andre. this works.

unfortunately, i am getting to other famous issue which is  Failed to
retrieve Ignite pods IP addresses.
will post as new issue.

On Wed, Nov 6, 2019 at 7:06 PM Andrei Aleksandrov 
wrote:

> Hi,
>
> As I know it should be fixed in Apache Ignite 2.7.6. Also as an option
> you can try to modify your current Docker file as next (master version):
>
>
> https://github.com/apache/ignite/blob/master/docker/apache-ignite/Dockerfile
>
>
> BR,
> Andrei
>
> 11/6/2019 10:33 AM, Gokulnath Chidambaram пишет:
> > Hello,
> >
> > I am trying to run Apache ignite : 2.5.0 inside the kubernetes
> > cluster. My organization security policy doesn't allow to run as
> > 'root' inside any container. I tried to add security context
> > (runAsNonRoot) in kubernetes yaml file. I am always getting the
> > following error.
> >
> > cp: can't create '/opt/ignite/apache-ignite-fabric/libs/README.txt':
> > File exists
> > cp: can't create
> > '/opt/ignite/apache-ignite-fabric/libs/ignite-kubernetes-2.5.0.jar':
> > Permission denied
> > cp: can't create
> > '/opt/ignite/apache-ignite-fabric/libs/jackson-core-asl-1.9.13.jar':
> > Permission denied
> > cp: can't create
> > '/opt/ignite/apache-ignite-fabric/libs/jackson-mapper-asl-1.9.13.jar':
> > Permission denied
> > cp: can't create
> > '/opt/ignite/apache-ignite-fabric/libs/licenses/apache-2.0.txt': File
> > exists
> > cp: can't create
> >
> '/opt/ignite/apache-ignite-fabric/libs/licenses/ignite-kubernetes-licenses.txt':
>
> > Permission denied
> >
> > Any insights is appreciated.
> >
> > Thanks
>

Re: recoveryBallotBoxes in MvccProcessorImpl memory leak?

[Ilya Kasnacheev]

Hello!

Can you please check whether there are any especially large objects inside
recoveryBallotBoxes object graph? Sorting by retained heap may help in
determining this. It would be nice to know what is the type histogram of
what's inside recoveryBallotBoxes and where the bulk of heap usage resides.

Regards,
-- 
Ilya Kasnacheev


чт, 7 нояб. 2019 г. в 06:23, mvkarp :

> I've attached another set of screenshots, might be more clear.
> heap.zip
> 
>
>
> mvkarp wrote
> > I've attached some extra screenshots showing what is inside these records
> > and path to GC roots. heap.zip
> > 
> http://apache-ignite-users.70518.x6.nabble.com/file/t2658/heap.zip;
> >
> >
> >
> >
> >
> > --
> > Sent from: http://apache-ignite-users.70518.x6.nabble.com/
>
>
>
>
>
> --
> Sent from: http://apache-ignite-users.70518.x6.nabble.com/
>

Re: Persistence store

[Ilya Kasnacheev]

Hello!

[2019-11-06 13:18:39,466][INFO ][main][GridCacheDatabaseSharedManager]
Checking memory state [lastValidPos=FileWALPointer [idx=0, fileOff=0,
len=0], lastMarked=FileWALPointer [idx=0, fileOff=0, len=0],
lastCheckpointId=----]
[2019-11-06 13:18:39,483][INFO ][main][GridCacheDatabaseSharedManager]
Applying lost cache updates since last checkpoint record
[lastMarked=FileWALPointer [idx=0, fileOff=0, len=0],
lastCheckpointId=----]
[2019-11-06 13:18:39,484][INFO ][main][GridCacheDatabaseSharedManager]
Finished applying WAL changes [updatesApplied=0, time=0ms]

I think this means you did not have any pre-existing persistence data on
that node after restart. Moreover, it seems that page store directory is
different. Are you sure that you did not clear
/opt/ignite/apache-ignite/work between restarts? What does it contain
currently?

Regards,
-- 
Ilya Kasnacheev


ср, 6 нояб. 2019 г. в 16:28, slut_hunter :

> In this screenshot, we see 3 nodes with a full cache.
> 
>
>
> In this screenshot, we see 3 nodes after restarting one.
> 
>
>
>
> Node log before restart
> befor_stop.log
> 
>
>
>
> Node log after restart
> after_start.log
> 
>
>
>
>
> --
> Sent from: http://apache-ignite-users.70518.x6.nabble.com/
>

Re: Data is lost during rebalance

[novacean.alex]

Upon further research it appears that disabling page eviction for the default
data region is not something i should do. 
According to Apache Ignite Documentation if Ignite Native Persistence is off
and page eviction is disabled for the default data region, OutOfMemory
errors will be thrown.
The 'persistenceEnabled' property is set to false in my config, as i am not
able to use it, so i need the page eviction enabled in order to avoid
OutOfMemory errors.



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: Data is lost during rebalance

[Вячеслав Коптилин]

Hi Alex,

I will take a look at log files.

Thanks,
S.

ср, 6 нояб. 2019 г. в 17:51, novacean.alex :

> This is the test i am performing:
>
> 1. I have an Ignite Cluster of 3 Server Nodes running in Kubernetes. (the
> cluster is created using a StatefulSet)
> 2. Once the cluster is up i use a Deployment to run 5 pods each with a
> Client Node that performs put operations on my cache.
> 3. Once i see in the logs "Page-based evictions started. Consider
> increasing
> 'maxSize' on Data Region configuration: default" on all 3 Server Nodes i
> delete the deployment, thus stopping the put opperations.
> 4. Once the deployment is deleted i restart node-2 from the cluster and
> once
> it is up again i wait for it to finish rebalancing.
> 5. The result of the test so far is that entries are missing from node-2
> after the rebalance is complete.
>
> Log Files before the restart:
> node-0.log
> 
> node-1.log
> 
> node-2.log
> 
>
> Log Files after the restart:
>
> node-0.log
> 
> node-1.log
> 
> node-2.log
> 
>
> I want to use the cluster to store web-session and i have ~400 request per
> second of cahe read and put operation.
> Would disabling the page eviction for the default data region end up and
> getting OOM Exceptions?
> If no, i will try and disable it as you suggested and perform more tests.
>
> Thanks,
> Alex.
>
>
>
> --
> Sent from: http://apache-ignite-users.70518.x6.nabble.com/
>