Re: Near cache configuration for partitioned cache

[Evgenii Zhuravlev]

Hi,

Near Cache configuration in xml creates near caches for all nodes,
including server nodes. As far as I understand, you want to have them on
client side only, right? If so, I'd recommend to create them dynamically:
https://www.gridgain.com/docs/latest/developers-guide/near-cache#creating-near-cache-dynamically-on-client-nodes

What kind of operations are you running? Are you trying to access data on
server from another server node? In any case, so many entries in Heap on
server nodes looks strange.

Evgenii

пн, 23 мар. 2020 г. в 07:08, Dominik Przybysz :

> Hi,
> I am using Ignite 2.7.6 and I have 2 server nodes with one partitioned
> cache and configuration:
>
> 
> http://www.springframework.org/schema/beans;
>xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance;
>xsi:schemaLocation="
>http://www.springframework.org/schema/beans
>http://www.springframework.org/schema/beans/spring-beans.xsd;>
>
>  class="org.apache.ignite.configuration.IgniteConfiguration">
> 
>  class="org.apache.ignite.configuration.CacheConfiguration">
> 
> 
> 
> 
> 
> 
>
> 
>  class="org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi">
> 
> 
> 
>
> 
>  class="org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi">
> 
> 
> 
>  class="org.apache.ignite.spi.discovery.tcp.ipfinder.vm.TcpDiscoveryVmIpFinder">
> 
> 
> ignite1:47100..47200
> ignite2:47100..47200
> 
> 
> 
> 
> 
> 
>
> 
>  class="org.apache.ignite.configuration.ClientConnectorConfiguration">
> 
> 
> 
>
> 
>  class="org.apache.ignite.configuration.DataStorageConfiguration">
> 
>  class="org.apache.ignite.configuration.DataRegionConfiguration">
> 
> 
> 
> 
> 
> 
> 
>
> 
>
>  value="{{ignite_system_thread_pool_size}}"/>
>  value="{{ignite_cluster_data_streamer_thread_pool_size}}"/>
> 
> 
>
> I loaded 1,5mln entries into cluster via data streamer.
> I tested this topology without near cache and everything was fine, but
> when I tried to add near cache to my client nodes then server nodes started
> to keep data on heap and reads rps dramatically fell down (150k rps to 10k
> rps).
>
> My clients' configuration:
>
> 
> http://www.springframework.org/schema/beans;
>xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance;
>xsi:schemaLocation="
>http://www.springframework.org/schema/beans
>http://www.springframework.org/schema/beans/spring-beans.xsd;>
>  class="org.apache.ignite.configuration.IgniteConfiguration">
> 
> 
>  class="org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi">
> 
>  class="org.apache.ignite.spi.discovery.tcp.ipfinder.vm.TcpDiscoveryVmIpFinder">
> 
> 
> ignite1:47100..47200
> ignite2:47100..47200
> 
> 
> 
> 
> 
> 
> 
> 
>
> 
>  class="org.apache.ignite.configuration.CacheConfiguration">
> 
> 
> 
> 
> 
>
> 
>  class="org.apache.ignite.configuration.NearCacheConfiguration">
> 
>  class="org.apache.ignite.cache.eviction.lru.LruEvictionPolicyFactory">
> 
> 
> 
> 
> 
> 
> 
> 
> 
>
> On visor i see:
>
> Nodes for: cache1(@c0)
>
> +=+
> |   Node ID8(@), IP   | CPUs | Heap Used | CPU Load |   Up Time
>  |Size (Primary / Backup)| Hi/Mi/Rd/Wr  |
>
> +=+
> | BCA8F378(@n2), 10.100.0.239 | 4| 32.32 %   | 2.17 %   | 00:38:33.071
> | Total: 55204 (55204 / 0)  | Hi: 1671212  |
> | |  |   |  |
>  |   Heap: 55204 (55204 / ) 

Re: Ignite python module pygridgain - Connected to

[Evgenii Zhuravlev]

Hi,

In pyignite, Client extends Connection class, which has __repr__
implementation, while in pygridgain Client doesn't extends Connection
class. In both cases Client doesn't have it's own representation.

Evgenii

сб, 21 мар. 2020 г. в 08:47, Dren Butković :

> Hi,
> I tested two Python Thin client modules and they do not give the same
> connection info.
>
> Python code and exaples:
>
> from pyignite import Client
> nodes = [('192.168.50.106', 10800)]
> ignite_client = Client(username='ignite', password='ignite', use_ssl=False)
> ignite_client.connect(nodes)
> print('Connected to {}'.format(ignite_client))
> ignite_client.close()
>
> Connected to 192.168.50.106:10800
>
> #
>
> from pygridgain import Client
> nodes = [('192.168.50.106', 10800)]
> ignite_client = Client(username='ignite', password='ignite', use_ssl=False)
> ignite_client.connect(nodes)
> print('Connected to {}'.format(ignite_client))
> ignite_client.close()
>
> Connected to 
>
>
>

Re: Python - Ignite 2.8.0 - java.lang.NullPointerException

[Evgenii Zhuravlev]

Hi,

I'm running this reproducer and I don't see any issues. Do you use pyignite
from the same version? Can you please share the full log file, maybe we're
missing something?

Thanks,
Evgenii

пт, 20 мар. 2020 г. в 08:11, dbutkovic :

> I have one small error in py code, please remove last "," from UPDATE
> statement in test_insert_update.py
>
> SQL_UPDATE =''' UPDATE TEST SET
> COL1 = ?,
> COL2 = ?,
> COL3 = ?,
> COL4 = ?,
> COL5 = ?,
> COL6 = ?,
> COL7 = ?,
> COL8 = ?,
> COL9 = ?,
> COL10= ?,
> COL11= ?,
> COL12= ?
> WHERE ID = ?
> '''
>
> regardless of this sintax error, insert on Ignite 2.8.0 is not ok.
>
>
>
>
> --
> Sent from: http://apache-ignite-users.70518.x6.nabble.com/
>

RE: RE: Re: Ignite 2.8.0. Heap mem issue

[Andrey Davydov]

It seems detached connection NEVER become attached to thread other it was born. Because borrow method always return object related to caller thread. I.e. all detached connection borned in joined thread are not collectable forewer. So possible reproduce scenario: start separate thread. Run in this thread some logic that creates detached connection, finish and join thread. Remove link to thread. Repeat. Andrey. От: Andrey DavydovОтправлено: 23 марта 2020 г. в 21:37Кому: user@ignite.apache.orgТема: RE: Re: Ignite 2.8.0. Heap mem issue As I see in heapDump: There are 29000+ connections stored here: this - value: org.apache.ignite.internal.processors.query.h2.H2ConnectionWrapper #1<- key - class: java.util.concurrent.ConcurrentHashMap$Node, value: org.apache.ignite.internal.processors.query.h2.H2ConnectionWrapper #1  <- [41919] - class: java.util.concurrent.ConcurrentHashMap$Node[], value: java.util.concurrent.ConcurrentHashMap$Node #32575   <- table - class: java.util.concurrent.ConcurrentHashMap, value: java.util.concurrent.ConcurrentHashMap$Node[] #11010    <- detachedConns - class: org.apache.ignite.internal.processors.query.h2.ConnectionManager, value: java.util.concurrent.ConcurrentHashMap #2109 But there are only 28 TCP connections in system.  I have already report some way for memory leaks in ConnectionManager in other mail thread, this is one more way. It seem that more problems is in ConnectionManager class.  Andrey. От: Evgenii ZhuravlevОтправлено: 23 марта 2020 г. в 20:37Кому: userТема: Re: Ignite 2.8.0. Heap mem issue Hi, I tried to reproduce the same problem using the provided information, but without success. In the log I see that there are operations running on the cluster that were not described in pseudo code, for example, SQL queries withh count. Are you running something else? Can you provide full reproducer or try to run all operations by one to find the operation, which can cause this behaviour? Evgenii ср, 18 мар. 2020 г. в 07:07, dbutkovic :Hi Alex, i did another test and collected all the logs, GC logs, Heap mem dump, fiewscreenshots.All files are in zip file. File is to big for upload, please download fromJumbo mail link.https://jumboiskon.tportal.hr/download/eeab9848-2494-4ab7-a2cb-88766db0fafaThanks, Dren--Sent from: http://apache-ignite-users.70518.x6.nabble.com/  

RE: Re: Ignite memory leaks in 2.8.0

[Andrey Davydov]

Sorry, It was to http://apache-ignite-users.70518.x6.nabble.com/Ignite-2-8-0-Heap-mem-issue-td31755.html thread Andrey. От: Andrey DavydovОтправлено: 23 марта 2020 г. в 23:00Кому: user@ignite.apache.orgТема: Re: Ignite memory leaks in 2.8.0 It seems detached connection NEVER become attached to thread other it was born. Because borrow method always return object related to caller thread. I.e. all detached connection borned in joined thread are not collectable forewer. So possible reproduce scenario: start separate thread. Run in this thread some logic that creates detached connection, finish and join thread. Remove link to thread. Repeat. пн, 23 мар. 2020 г., 15:49 Taras Ledkov :Hi,Thanks for your investigation.Root cause is clear. What use-case is causing the leak?I've created the issue to remove mess ThreadLocal logic from ConnectionManager. [1]We 've done it in GG Community Edition and it works OK.[1]. https://issues.apache.org/jira/browse/IGNITE-12804On 21.03.2020 22:50, Andrey Davydov wrote:A simple diagnostic utility I use to detect these problems:  import java.lang.ref.WeakReference;import java.util.ArrayList;import java.util.LinkedList;import java.util.List;import org.apache.ignite.Ignite;import org.apache.ignite.internal.GridComponent;import org.apache.ignite.internal.IgniteKernal;import org.apache.logging.log4j.LogManager;import org.apache.logging.log4j.Logger;public class IgniteWeakRefTracker {    private static final Logger LOGGER = LogManager.getLogger(IgniteWeakRefTracker.class);    private final String clazz;    private final String testName;    private final String name;    private final WeakReference innerRef;    private final List> componentRefs = new ArrayList<>(128);    private static final LinkedList refs = new LinkedList<>();    private IgniteWeakRefTracker(String testName, Ignite ignite) {        this.clazz = ignite.getClass().getCanonicalName();        this.innerRef = new WeakReference<>(ignite);        this.name = ignite.name();        this.testName = testName;        if (ignite instanceof IgniteKernal) {            IgniteKernal ik = (IgniteKernal) ignite;            List components = ik.context().components();            for (GridComponent c : components) {                componentRefs.add(new WeakReference<>(c));            }        }    }    public static void register(String testName, Ignite ignite) {        refs.add(new IgniteWeakRefTracker(testName, ignite));    }    public static void trimCollectedRefs() {        List toRemove = new ArrayList<>();        for (IgniteWeakRefTracker ref : refs) {            if (ref.isIgniteCollected()) {                LOGGER.info("Collected ignite: ignite {} from test {}", ref.getIgniteName(), ref.getTestName());                toRemove.add(ref);                if (ref.igniteComponentsNonCollectedCount() != 0) {                    throw new IllegalStateException("Non collected components for collected ignite.");                }            } else {                LOGGER.warn("Leaked ignite: ignite {} from test {}", ref.getIgniteName(), ref.getTestName());            }        }        refs.removeAll(toRemove);        LOGGER.info("Leaked ignites count:  {}", refs.size());    }    public static int getLeakedSize() {        return refs.size();    }    public boolean isIgniteCollected() {        return innerRef.get() == null;    }    public int igniteComponentsNonCollectedCount() {        int res = 0;        for (WeakReference cr : componentRefs) {            GridComponent gridComponent = cr.get();            if (gridComponent != null) {                LOGGER.warn("Uncollected component: {}", gridComponent.getClass().getSimpleName());                res++;            }        }        return res;    }    public String getClazz() {        return clazz;    }    public String getTestName() {        return testName;    }    public String getIgniteName() {        return name;    }}  On Fri, Mar 20, 2020 at 11:51 PM Andrey Davydov  wrote:I found one more way for leak and understand reason: this     - value: org.apache.ignite.internal.IgniteKernal #1 <- grid     - class: org.apache.ignite.internal.GridKernalContextImpl, value: org.apache.ignite.internal.IgniteKernal #1  <- ctx     - class: org.apache.ignite.internal.processors.timeout.GridTimeoutProcessor, value: org.apache.ignite.internal.GridKernalContextImpl #3   <- this$0     - class: org.apache.ignite.internal.processors.timeout.GridTimeoutProcessor$CancelableTask, value: org.apache.ignite.internal.processors.timeout.GridTimeoutProcessor #1    <- stmtCleanupTask     - class: org.apache.ignite.internal.processors.query.h2.ConnectionManager, value: org.apache.ignite.internal.processors.timeout.GridTimeoutProcessor$CancelableTask #11     <- arg$1     - class: org.apache.ignite.internal.processors.query.h2.ConnectionManager$$Lambda$174, value: org.apache.ignite.internal.processors.query.h2.ConnectionManager #1      <- recycler     - class: 

Re: Ignite memory leaks in 2.8.0

[Andrey Davydov]

It seems detached connection NEVER become attached to thread other it was
born. Because borrow method always return object related to caller thread.
I.e. all detached connection borned in joined thread are not collectable
forewer.

So possible reproduce scenario: start separate thread. Run in this thread
some logic that creates detached connection, finish and join thread. Remove
link to thread. Repeat.

пн, 23 мар. 2020 г., 15:49 Taras Ledkov :

> Hi,
>
> Thanks for your investigation.
> Root cause is clear. What use-case is causing the leak?
>
> I've created the issue to remove mess ThreadLocal logic from
> ConnectionManager. [1]
> We 've done it in GG Community Edition and it works OK.
>
> [1]. https://issues.apache.org/jira/browse/IGNITE-12804
> On 21.03.2020 22:50, Andrey Davydov wrote:
>
> A simple diagnostic utility I use to detect these problems:
>
> import java.lang.ref.WeakReference;
> import java.util.ArrayList;
> import java.util.LinkedList;
> import java.util.List;
> import org.apache.ignite.Ignite;
> import org.apache.ignite.internal.GridComponent;
> import org.apache.ignite.internal.IgniteKernal;
> import org.apache.logging.log4j.LogManager;
> import org.apache.logging.log4j.Logger;
>
> public class IgniteWeakRefTracker {
>
> private static final Logger LOGGER =
> LogManager.getLogger(IgniteWeakRefTracker.class);
>
> private final String clazz;
> private final String testName;
> private final String name;
> private final WeakReference innerRef;
> private final List> componentRefs = new
> ArrayList<>(128);
>
> private static final LinkedList refs = new
> LinkedList<>();
>
> private IgniteWeakRefTracker(String testName, Ignite ignite) {
> this.clazz = ignite.getClass().getCanonicalName();
> this.innerRef = new WeakReference<>(ignite);
> this.name = ignite.name();
> this.testName = testName;
>
> if (ignite instanceof IgniteKernal) {
> IgniteKernal ik = (IgniteKernal) ignite;
> List components = ik.context().components();
> for (GridComponent c : components) {
> componentRefs.add(new WeakReference<>(c));
> }
> }
> }
>
> public static void register(String testName, Ignite ignite) {
> refs.add(new IgniteWeakRefTracker(testName, ignite));
> }
>
> public static void trimCollectedRefs() {
>
> List toRemove = new ArrayList<>();
>
> for (IgniteWeakRefTracker ref : refs) {
> if (ref.isIgniteCollected()) {
> LOGGER.info("Collected ignite: ignite {} from test {}",
> ref.getIgniteName(), ref.getTestName());
> toRemove.add(ref);
> if (ref.igniteComponentsNonCollectedCount() != 0) {
> throw new IllegalStateException("Non collected
> components for collected ignite.");
> }
> } else {
> LOGGER.warn("Leaked ignite: ignite {} from test {}",
> ref.getIgniteName(), ref.getTestName());
> }
> }
>
> refs.removeAll(toRemove);
>
> LOGGER.info("Leaked ignites count:  {}", refs.size());
>
> }
>
> public static int getLeakedSize() {
> return refs.size();
> }
>
> public boolean isIgniteCollected() {
> return innerRef.get() == null;
> }
>
> public int igniteComponentsNonCollectedCount() {
> int res = 0;
>
> for (WeakReference cr : componentRefs) {
> GridComponent gridComponent = cr.get();
> if (gridComponent != null) {
> LOGGER.warn("Uncollected component: {}",
> gridComponent.getClass().getSimpleName());
> res++;
> }
> }
>
> return res;
> }
>
> public String getClazz() {
> return clazz;
> }
>
> public String getTestName() {
> return testName;
> }
>
> public String getIgniteName() {
> return name;
> }
>
> }
>
>
> On Fri, Mar 20, 2020 at 11:51 PM Andrey Davydov 
> wrote:
>
>> I found one more way for leak and understand reason:
>>
>>
>> this - value: org.apache.ignite.internal.IgniteKernal #1
>>  <- grid - class: org.apache.ignite.internal.GridKernalContextImpl,
>> value: org.apache.ignite.internal.IgniteKernal #1
>>   <- ctx - class:
>> org.apache.ignite.internal.processors.timeout.GridTimeoutProcessor, value:
>> org.apache.ignite.internal.GridKernalContextImpl #3
>><- this$0 - class:
>> org.apache.ignite.internal.processors.timeout.GridTimeoutProcessor$CancelableTask,
>> value: org.apache.ignite.internal.processors.timeout.GridTimeoutProcessor #1
>> <- stmtCleanupTask - class:
>> org.apache.ignite.internal.processors.query.h2.ConnectionManager, value:
>> org.apache.ignite.internal.processors.timeout.GridTimeoutProcessor$CancelableTask
>> #11
>>  <- arg$1 - class:
>> org.apache.ignite.internal.processors.query.h2.ConnectionManager$$Lambda$174,
>> value: 

RE: Re: Ignite 2.8.0. Heap mem issue

[dbutkovic]

Hi,

in attachment is code for heap mem issue simulation. Ignite_2.zip
  

# create cache TEST and insert random data into cache TEST
test_insert_in_cache.py

# read data from cache TEST and inset/update data in table TEST_TABLE via
SQL
read_cache_insert_update_table.py


Best regads

Dren




--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: No ignitevisorcmd.sh in Ignite 2.8

[joaogoncalves]

Yes, im running ignitevisor on the same machine.

I've been testing, trying to start ignite with example config files, and I'm
not able to connect to ignite with any of them. It starts up correctly, but
I can't connect. 

I get no errors, except on one of them:

Caused by: class org.apache.ignite.IgniteCheckedException: Failed to find
configuration in: file:/opt/ignite/examples/config/example-default.xml

This is the content of that file:







http://www.springframework.org/schema/beans;
   xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance;
   xmlns:util="http://www.springframework.org/schema/util;
   xsi:schemaLocation="
http://www.springframework.org/schema/beans
http://www.springframework.org/schema/beans/spring-beans.xsd
http://www.springframework.org/schema/util
http://www.springframework.org/schema/util/spring-util.xsd;>

































127.0.0.1:47500..47509










I assume this is some configuration error within the file itself.

I've given full read/write permissions to the whole IGNITE_HOME and still
nothing.

Are there any other requisites to using ignitevisor that aren't specified in
the docs? 
Is there any other way to inspect existing cache configurations?



 



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Loading Data from RDMS to ignite using Data Streamer

[nithin91]

Hi ,

I am new to Ignite. i am able to load data from oracle into ignite using
Cache Pojo Store method ignite.cache.("cachename").loadcache(null) . But in
the documentation it was mentioned that Data Steamers will provide better 
performance when loading data into ignite cache.

*Documentation Reference:*

https://apacheignite.readme.io/docs/data-streamers

Following is the example provided for Data Streamers in the documentation

// Get the data streamer reference and stream data.
try (IgniteDataStreamer stmr =
ignite.dataStreamer("myStreamCache")) {
// Stream entries.
for (int i = 0; i < 10; i++)
stmr.addData(i, Integer.toString(i));
}

I have following questions by taking  this example as reference to stream
data from oracle table to ignite 
cache.

1. Is an external streamer needed to stream data from oracle to ignite to
use data streaming technique
as mentioned in above example.

2. If an external streamer is not required, then do we need to loop through
the jdbc result set and add data to the cache.Please find below for the
pseudo code implementation of the same.

try (IgniteDataStreamer stmr = ignite.dataStreamer("CacheName")) {
// Stream entries.


while(rs.hasNext()){-->rs is JDBC result set

Key keyobj=new Key(rs.getString(1));
Value valueobj=new
Value(rs.getString(2),rs.getString(3),rs.getString(4));
stmr.addData(keyobj , valueobj);
}
}

When i tried to load data into ignite cache from oracle using this  method
it is taking lot of time which is as expected as we are looping through each
and every record.Is this right method of implementing Data Streamers to load
data from Oracle to Ignite.If this is not right way,can anyone share sample
code to stream data from any RDBMS table to Ignite cache.





--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: Data Isolation during Ignite Cache load

[Evgenii Zhuravlev]

Hi,

> ignite should display the data prior to data load instead of displaying
the data that isloaded till that point
*What do you mean? How it will help with a problem that you describing
after that? *
> he might get an empty response in case if the requested data is notyet
loaded when the cache load process is running

There is no data in Ignite, that's why it will not return any value. I
don't really understand, why in this case UI will allow user to query entry
which was not loaded yet.

Evgenii

пн, 23 мар. 2020 г. в 08:45, nithin91 <
nithinbharadwaj.govindar...@franklintempleton.com>:

> Hi ,
>
> But it is not behaving that way. I have tested it  using the following
> scenario.
>
> Scenario:
>
>
> Initially loading 1000 rows from oracle table with the following filter
> where rownum<=1000 into ignite cache.
> During the cache load process, i checked the count using sql fields query i
> got the response as number of records that are loaded into the cache till
> that point i.e  100,200 and finally 1000.
>
> Again started the load into ignite cache  from oracle table with the same
> filter where rownum<=1000.
> Now during the cache load time, till 1000 rows are loaded it is showing the
> count as 1000 but once the count crosses 1000 it is showing the count as
> 1001,1002 and finally 2000.
>
> Can you please help me fix this issue (i.e While loading Ignite Cache using
> JDBC POJO Ignite.cache.loadcache method,  ignite should display the data
> prior to the start of data load instead of displaying the data that is
> loaded till that point when a query is submitted during the load process.
> As
> we are using the ignite cache for front-end websites,there might be
> scenarios where the  user queries the cache against a specific value for
> which  he might get an empty response in case if the requested data is not
> yet loaded when the cache load process is running. Instead the problem will
> be resolved if ignite always shows the data prior to start of data load if
> the cache is queried during data load process).
>
>
>
>
>
>
>
>
>
> --
> Sent from: http://apache-ignite-users.70518.x6.nabble.com/
>

RE: Re: Ignite 2.8.0. Heap mem issue

[Andrey Davydov]

As I see in heapDump: There are 29000+ connections stored here: this - value: org.apache.ignite.internal.processors.query.h2.H2ConnectionWrapper #1 <- key - class: java.util.concurrent.ConcurrentHashMap$Node, value: org.apache.ignite.internal.processors.query.h2.H2ConnectionWrapper #1  <- [41919] - class: java.util.concurrent.ConcurrentHashMap$Node[], value: java.util.concurrent.ConcurrentHashMap$Node #32575   <- table - class: java.util.concurrent.ConcurrentHashMap, value: java.util.concurrent.ConcurrentHashMap$Node[] #11010    <- detachedConns - class: org.apache.ignite.internal.processors.query.h2.ConnectionManager, value: java.util.concurrent.ConcurrentHashMap #2109 But there are only 28 TCP connections in system.  I have already report some way for memory leaks in ConnectionManager in other mail thread, this is one more way. It seem that more problems is in ConnectionManager class.  Andrey. От: Evgenii ZhuravlevОтправлено: 23 марта 2020 г. в 20:37Кому: userТема: Re: Ignite 2.8.0. Heap mem issue Hi, I tried to reproduce the same problem using the provided information, but without success. In the log I see that there are operations running on the cluster that were not described in pseudo code, for example, SQL queries withh count. Are you running something else? Can you provide full reproducer or try to run all operations by one to find the operation, which can cause this behaviour? Evgenii ср, 18 мар. 2020 г. в 07:07, dbutkovic :Hi Alex, i did another test and collected all the logs, GC logs, Heap mem dump, fiewscreenshots.All files are in zip file. File is to big for upload, please download fromJumbo mail link.https://jumboiskon.tportal.hr/download/eeab9848-2494-4ab7-a2cb-88766db0fafaThanks, Dren--Sent from: http://apache-ignite-users.70518.x6.nabble.com/ 

Ignite.cache.loadcache.Does this method do Increamental Load?

[nithin91]

Hi 

I am trying to load the data into ignite cache using JDBC Pojo Store method
ignite.cache("cacheName").loadCache(null).I have used this method and got
the following results for the following scenarios.

*Scenario-1:Trying to load the same key which is available in cache*

In this case, the value part corresponding to the key is not updated in the
cache based on the latest available record.

*Scenario -2:When loading a key which is not present in cache.*

in this case, it is appending the new key and value pair to the cache and
preserving the old data.


But my doubt is why in scenario-1, it is not  updating the value
corresponding to the when i am  trying to load the same key.

Does this method do incremental load?.
Is this the expected behavior or do i need to set any additional property in
the bean file.Attaching you the bean configuration of the cache.

cache.xml
  









--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: Ignite 2.8.0. Heap mem issue

[Evgenii Zhuravlev]

Hi,

I tried to reproduce the same problem using the provided information, but
without success. In the log I see that there are operations running on the
cluster that were not described in pseudo code, for example, SQL queries
withh count. Are you running something else? Can you provide full
reproducer or try to run all operations by one to find the operation, which
can cause this behaviour?

Evgenii

ср, 18 мар. 2020 г. в 07:07, dbutkovic :

> Hi Alex,
>
> i did another test and collected all the logs, GC logs, Heap mem dump, fiew
> screenshots.
>
> All files are in zip file. File is to big for upload, please download from
> Jumbo mail link.
>
> https://jumboiskon.tportal.hr/download/eeab9848-2494-4ab7-a2cb-88766db0fafa
>
> Thanks, Dren
>
>
>
> --
> Sent from: http://apache-ignite-users.70518.x6.nabble.com/
>

Re: No ignitevisorcmd.sh in Ignite 2.8

[Stephen Darlington]

Are you starting ignitevisor on the same machine as your server? There’s no 
problem with your discovery configuration. I can use your configuration to 
start a server node and Visor on my laptop.

> On 23 Mar 2020, at 15:38, joaogoncalves  wrote:
> 
> I am starting ignite with my custom config file like:
> 
> ./bin/ignite.sh config.xml
> 
> Below is my config file:
> 
> http://www.springframework.org/schema/beans;
>  xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance;
>  xsi:schemaLocation="
>   http://www.springframework.org/schema/beans
>   http://www.springframework.org/schema/beans/spring-beans.xsd;>
> 
>   class="org.apache.ignite.configuration.IgniteConfiguration">
> 
>
>  
> 
>
>  
>  
>  
> 
> 
>
>  
>
> 
> 
>
>   class="org.apache.ignite.configuration.DataStorageConfiguration">
>
>  
> 
> class="org.apache.ignite.configuration.DataRegionConfiguration">
> 
>  
> 
> 
>  
> 
> 
>  
> 
> 
> 
>
>  
>
>  
>
> 
>
>  
>
>   class="org.apache.ignite.spi.discovery.tcp.ipfinder.vm.TcpDiscoveryVmIpFinder">
>
>  
> 
> 
> 
> 
>127.0.0.1:47500..47509
>  
>
>  
>
>  
>
> 
>  
> 
> 
> 
> I then run ignitevisorcmd.sh, execute the 'open' command and select the same
> config.xml.
> The status remains the same:
> 
> +-+
> | Status   | Disconnected |
> | Ignite instance name | |
> | Config path  | |
> | Uptime   | |
> +-+
> 
> I want to use ignitevisor so i can inspect a cache configuration that i
> suspect is not being loaded. If you noticed in my config, i have defined a
> cache template with the following setting:
> 
> 
> 
> This was supposed to make ignite case sensitive when creating tables, so
> that i get my table names and field names in lowercase. 
> 
> This is not directly related to ignitevisor and maybe i should move it to
> another post?
> 
> Thanks for your help so far.
> 
> 
> 
> 
> 
> --
> Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: Data Isolation during Ignite Cache load

[nithin91]

Hi ,

But it is not behaving that way. I have tested it  using the following
scenario.

Scenario:


Initially loading 1000 rows from oracle table with the following filter
where rownum<=1000 into ignite cache.
During the cache load process, i checked the count using sql fields query i
got the response as number of records that are loaded into the cache till
that point i.e  100,200 and finally 1000.

Again started the load into ignite cache  from oracle table with the same
filter where rownum<=1000.
Now during the cache load time, till 1000 rows are loaded it is showing the
count as 1000 but once the count crosses 1000 it is showing the count as
1001,1002 and finally 2000.

Can you please help me fix this issue (i.e While loading Ignite Cache using
JDBC POJO Ignite.cache.loadcache method,  ignite should display the data
prior to the start of data load instead of displaying the data that is
loaded till that point when a query is submitted during the load process. As
we are using the ignite cache for front-end websites,there might be
scenarios where the  user queries the cache against a specific value for
which  he might get an empty response in case if the requested data is not
yet loaded when the cache load process is running. Instead the problem will
be resolved if ignite always shows the data prior to start of data load if
the cache is queried during data load process).









--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: No ignitevisorcmd.sh in Ignite 2.8

[joaogoncalves]

I am starting ignite with my custom config file like:

./bin/ignite.sh config.xml

Below is my config file:

http://www.springframework.org/schema/beans;
  xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance;
  xsi:schemaLocation="
   http://www.springframework.org/schema/beans
   http://www.springframework.org/schema/beans/spring-beans.xsd;>
  
  


  


  
  
  
  
  

  




  

  


  
  

  
  

  
  

  
  

  

  



  

  

  




127.0.0.1:47500..47509
  

  

  


  



I then run ignitevisorcmd.sh, execute the 'open' command and select the same
config.xml.
The status remains the same:

+-+
| Status   | Disconnected |
| Ignite instance name | |
| Config path  | |
| Uptime   | |
+-+

I want to use ignitevisor so i can inspect a cache configuration that i
suspect is not being loaded. If you noticed in my config, i have defined a
cache template with the following setting:



This was supposed to make ignite case sensitive when creating tables, so
that i get my table names and field names in lowercase. 

This is not directly related to ignitevisor and maybe i should move it to
another post?

Thanks for your help so far.





--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: No ignitevisorcmd.sh in Ignite 2.8

[Stephen Darlington]

Are you getting any error messages? Are you giving it the same configuration 
file as you give the server nodes? It works when I try it with the default 
options.

> On 23 Mar 2020, at 12:03, joaogoncalves  wrote:
> 
> Hi again
> 
> I now downloaded the binary and started ignite in a screen.
> 
> I started ignitevisorcmd.sh but can't connect to my running node with the
> 'open' command.
> 
> I can connect to ignite with sqlline and inspect tables.
> 
> Any clues?
> 
> 
> 
> --
> Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Near cache configuration for partitioned cache

[Dominik Przybysz]

Hi,
I am using Ignite 2.7.6 and I have 2 server nodes with one partitioned
cache and configuration:


http://www.springframework.org/schema/beans;
   xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance;
   xsi:schemaLocation="
   http://www.springframework.org/schema/beans
   http://www.springframework.org/schema/beans/spring-beans.xsd;>

























ignite1:47100..47200
ignite2:47100..47200
































I loaded 1,5mln entries into cluster via data streamer.
I tested this topology without near cache and everything was fine, but when
I tried to add near cache to my client nodes then server nodes started to
keep data on heap and reads rps dramatically fell down (150k rps to 10k
rps).

My clients' configuration:


http://www.springframework.org/schema/beans;
   xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance;
   xsi:schemaLocation="
   http://www.springframework.org/schema/beans
   http://www.springframework.org/schema/beans/spring-beans.xsd;>








ignite1:47100..47200
ignite2:47100..47200































On visor i see:

Nodes for: cache1(@c0)
+=+
|   Node ID8(@), IP   | CPUs | Heap Used | CPU Load |   Up Time
 |Size (Primary / Backup)| Hi/Mi/Rd/Wr  |
+=+
| BCA8F378(@n2), 10.100.0.239 | 4| 32.32 %   | 2.17 %   | 00:38:33.071
| Total: 55204 (55204 / 0)  | Hi: 1671212  |
| |  |   |  |
 |   Heap: 55204 (55204 / ) | Mi: 35034768 |
| |  |   |  |
 |   Off-Heap: 0 (0 / 0) | Rd: 36705980 |
| |  |   |  |
 |   Off-Heap Memory: 0  | Wr: 0|
+-+--+---+--+--+---+--+
| 905F83EE(@n3), 10.100.0.230 | 4| 52.56 %   | 6.67 %   | 00:38:33.401
| Total: 54051 (54051 / 0)  | Hi: 1766495  |
| |  |   |  |
 |   Heap: 54051 (54051 / ) | Mi: 34283753 |
| |  |   |  |
 |   Off-Heap: 0 (0 / 0) | Rd: 36050248 |
| |  |   |  |
 |   Off-Heap Memory: 0  | Wr: 0|
+-+--+---+--+--+---+--+
| 793E1BC9(@n1), 10.100.0.206 | 4| 99.33 %   | 38.43 %  | 00:51:11.877
| Total: 2999836 (2230060 / 769776) | Hi: 17323596 |
| |  |   |  |
 |   Heap: 1499836 (1499836 / ) | Mi: 0|
| |  |   |  |
 |   Off-Heap: 150 (730224 / 769776) | Rd: 17323596 |
| |  |   |  |
 |   Off-Heap Memory:   | Wr: 0|
+-+--+---+--+--+---+--+
| 0147FB02(@n0), 10.100.0.205 | 4| 96.48 %   | 40.33 %  | 00:51:11.820
| Total: 2999814 (2269590 / 730224) | Hi: 17335702 |
| |  |   |  |
 |   Heap: 1499814 (1499814 / ) | Mi: 0  

Re: HOW TO CHANGE IGNITE JAVA THIN CLIENT PASSWORD

[Ilya Kasnacheev]

Hello!

I have just rechecked this case:
1) start persistent node.
2) try changing password - failure (cluster not active)
3) activate cluster (control.sh --activate, ignite/ignite)
4) try changing password - success.
5) check with sqlline - new password (test) accepted.
6) try connecting with thin client and ignite/ignite again - failure.

Are you sure you're not losing your cluster together with all data between
runs?

igniteClient.query(new SqlFieldsQuery("ALTER USER \"ignite\" WITH
PASSWORD 'test'")).getAll();


Regards,
-- 
Ilya Kasnacheev


ср, 18 мар. 2020 г. в 11:47, DS :

>
>
> igniteClient.query(new SqlFieldsQuery(" ALTER USER 'ignite' WITH PASSWORD
> 'password' "));
>
> igniteClient.query(new SqlFieldsQuery(" ALTER USER "ignite'' WITH PASSWORD
> 'password' "));
>
> 1) Both give the same result. i.e query runs without throwing any
> error/exception.
>
> 2) Unable to connect with the new password.
>
> 3) Again tried with 'ignite' as password, it connects back.
>
>
>
>
> --
> Sent from: http://apache-ignite-users.70518.x6.nabble.com/
>

Re: Ignite memory leaks in 2.8.0

[Taras Ledkov]

Hi,

Thanks for your investigation.
Root cause is clear. What use-case is causing the leak?

I've created the issue to remove mess ThreadLocal logic from 
ConnectionManager. [1]

We 've done it in GG Community Edition and it works OK.

[1]. https://issues.apache.org/jira/browse/IGNITE-12804

On 21.03.2020 22:50, Andrey Davydov wrote:

A simple diagnostic utility I use to detect these problems:

import java.lang.ref.WeakReference;
import java.util.ArrayList;
import java.util.LinkedList;
import java.util.List;
import org.apache.ignite.Ignite;
import org.apache.ignite.internal.GridComponent;
import org.apache.ignite.internal.IgniteKernal;
import org.apache.logging.log4j.LogManager;
import org.apache.logging.log4j.Logger;

public class IgniteWeakRefTracker {

    private static final Logger LOGGER = 
LogManager.getLogger(IgniteWeakRefTracker.class);


    private final String clazz;
    private final String testName;
    private final String name;
    private final WeakReference innerRef;
    private final List> componentRefs = 
new ArrayList<>(128);


    private static final LinkedList refs = new 
LinkedList<>();


    private IgniteWeakRefTracker(String testName, Ignite ignite) {
        this.clazz = ignite.getClass().getCanonicalName();
        this.innerRef = new WeakReference<>(ignite);
this.name  = ignite.name ();
        this.testName = testName;

        if (ignite instanceof IgniteKernal) {
            IgniteKernal ik = (IgniteKernal) ignite;
            List components = ik.context().components();
            for (GridComponent c : components) {
                componentRefs.add(new WeakReference<>(c));
            }
        }
    }

    public static void register(String testName, Ignite ignite) {
        refs.add(new IgniteWeakRefTracker(testName, ignite));
    }

    public static void trimCollectedRefs() {

        List toRemove = new ArrayList<>();

        for (IgniteWeakRefTracker ref : refs) {
            if (ref.isIgniteCollected()) {
                LOGGER.info("Collected ignite: ignite {} from test 
{}", ref.getIgniteName(), ref.getTestName());

                toRemove.add(ref);
                if (ref.igniteComponentsNonCollectedCount() != 0) {
                    throw new IllegalStateException("Non collected 
components for collected ignite.");

                }
            } else {
                LOGGER.warn("Leaked ignite: ignite {} from test {}", 
ref.getIgniteName(), ref.getTestName());

            }
        }

        refs.removeAll(toRemove);

        LOGGER.info("Leaked ignites count:  {}", refs.size());

    }

    public static int getLeakedSize() {
        return refs.size();
    }

    public boolean isIgniteCollected() {
        return innerRef.get() == null;
    }

    public int igniteComponentsNonCollectedCount() {
        int res = 0;

        for (WeakReference cr : componentRefs) {
            GridComponent gridComponent = cr.get();
            if (gridComponent != null) {
                LOGGER.warn("Uncollected component: {}", 
gridComponent.getClass().getSimpleName());

                res++;
            }
        }

        return res;
    }

    public String getClazz() {
        return clazz;
    }

    public String getTestName() {
        return testName;
    }

    public String getIgniteName() {
        return name;
    }

}


On Fri, Mar 20, 2020 at 11:51 PM Andrey Davydov 
mailto:andrey.davy...@gmail.com>> wrote:


I found one more way for leak and understand reason:


this     - value: org.apache.ignite.internal.IgniteKernal #1
 <- grid     - class:
org.apache.ignite.internal.GridKernalContextImpl, value:
org.apache.ignite.internal.IgniteKernal #1
  <- ctx     - class:
org.apache.ignite.internal.processors.timeout.GridTimeoutProcessor,
value: org.apache.ignite.internal.GridKernalContextImpl #3
   <- this$0     - class:

org.apache.ignite.internal.processors.timeout.GridTimeoutProcessor$CancelableTask,
value:
org.apache.ignite.internal.processors.timeout.GridTimeoutProcessor #1
    <- stmtCleanupTask     - class:
org.apache.ignite.internal.processors.query.h2.ConnectionManager,
value:

org.apache.ignite.internal.processors.timeout.GridTimeoutProcessor$CancelableTask
#11
     <- arg$1     - class:

org.apache.ignite.internal.processors.query.h2.ConnectionManager$$Lambda$174,
value:
org.apache.ignite.internal.processors.query.h2.ConnectionManager #1
      <- recycler     - class:
org.apache.ignite.internal.processors.query.h2.ThreadLocalObjectPool,
value:
org.apache.ignite.internal.processors.query.h2.ConnectionManager$$Lambda$174
#1
       <- this$0     - class:

org.apache.ignite.internal.processors.query.h2.ThreadLocalObjectPool$Reusable,
value:
org.apache.ignite.internal.processors.query.h2.ThreadLocalObjectPool
#1
        <- value     - class:

Re: Query timeout

[Taras Ledkov]

Hi,

Hm.. It works for me.
Looks at the Ignite test: 
JdbcThinStatementTimeoutSelfTest#testQueryTimeoutRetrival [1].


Please share your simple reproducer or test.

[1]. 
https://github.com/apache/ignite/blob/9f19e0160b1ca43a27908849ad46c65ebd8689f1/modules/clients/src/test/java/org/apache/ignite/jdbc/thin/JdbcThinStatementTimeoutSelfTest.java#L212


On 23.03.2020 14:36, breathem wrote:

I try jdbc:ignite:thin://192.168.1.138:10800;queryTimeout=1 and
jdbc:ignite:thin://192.168.1.138:10800?queryTimeout=1 both.
Query is still executed more then 1 sec without exception.
Is this parameter supported in 2.8.0?



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/


--
Taras Ledkov
Mail-To: tled...@gridgain.com

Re: Query timeout

[breathem]

I try jdbc:ignite:thin://192.168.1.138:10800;queryTimeout=1 and
jdbc:ignite:thin://192.168.1.138:10800?queryTimeout=1 both.
Query is still executed more then 1 sec without exception. 
Is this parameter supported in 2.8.0?



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Exporter usage of Ignite 2.8.0

[Kamlesh Joshi]

Hi Team,

Can you help how to use inbuilt exporter for Ignite 2.8.0. I read it in release 
notes that 'Added monitoring API - an exporter of Ignite metrics to external 
recipients'.

Do we have to manually expose these exporters on some port ? Or are they 
exposed on any default port ? So that these exporters then can directly be 
consumed by Prometheus.

Thanks and Regards,
Kamlesh Joshi

"Confidentiality Warning: This message and any attachments are intended only 
for the use of the intended recipient(s). 
are confidential and may be privileged. If you are not the intended recipient. 
you are hereby notified that any 
review. re-transmission. conversion to hard copy. copying. circulation or other 
use of this message and any attachments is 
strictly prohibited. If you are not the intended recipient. please notify the 
sender immediately by return email. 
and delete this message and any attachments from your system.

Virus Warning: Although the company has taken reasonable precautions to ensure 
no viruses are present in this email. 
The company cannot accept responsibility for any loss or damage arising from 
the use of this email or attachment."

Re: Query timeout

[Taras Ledkov]

Hi,

queryTimeout=N

in seconds according with JDBC specification: 
java.sql.Statement#setQueryTimeout


On 23.03.2020 12:47, breathem wrote:

Hi all,
Is there any way to set query timeout parameter in JDBC url?



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/


--
Taras Ledkov
Mail-To: tled...@gridgain.com

Query timeout

[breathem]

Hi all,
Is there any way to set query timeout parameter in JDBC url?



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: No ignitevisorcmd.sh in Ignite 2.8

[Petr Ivanov]

ignitevisorcmd was moved to docs by design, due to lack of time to configure 
package to install it correctly into system as ignite.sh is being installed 
(PATH, alternatives, etc.)


> On 20 Mar 2020, at 23:55, Stephen Darlington 
>  wrote:
> 
> Ah, apparently it doesn’t work when installed from a package. It’s in the doc 
> directory somewhere. See here for a discussion from when 2.7.5 was current: 
> http://apache-ignite-users.70518.x6.nabble.com/2-7-5-IGNITE-HOME-bin-ignitevisorcmd-sh-is-absent-td28629.html
>  
> 
> 
> Regards,
> Stephen
> 
>> On 20 Mar 2020, at 20:46, João Gonçalves > > wrote:
>> 
>> Thanks for the quick reply
>> 
>> I downloaded the rpm package 
>> https://ignite.apache.org/download.cgi#rpm-package 
>> 
>> 
>> Which one should I use?
>> 
>> 
>> Stephen Darlington > > escreveu no dia sexta, 20/03/2020 
>> à(s) 20:30:
>> Where did you download it from? Did you download the binary or source 
>> version? It’s there in my download:
>> 
>> ~ stephen.darlington$ cd $IGNITE_HOME
>> -ignite-2.8.0-bin stephen.darlington$ ls bin/*sh
>> bin/control.sh   bin/igniterouter.sh bin/sqlline.sh
>> bin/ignite.shbin/ignitevisorcmd.sh
>> apache-ignite-2.8.0-bin stephen.darlington$ 
>> 
>> Regards,
>> Stephen
>> 
>>> On 20 Mar 2020, at 19:19, joaogoncalves >> > wrote:
>>> 
>>> Hi
>>> Recently started working with Ignite on version 2.8.0
>>> 
>>> Can't find ignitevisorcmd.sh, was it removed? According to 
>>> https://apacheignite-tools.readme.io/docs/command-line-interface 
>>> 
>>> >> >   it
>>> should be under IGNITE_HOME/bin
>>> 
>>> I found IGNITE_HOME/bin/include/visorcmd/ignite-visor-console-2.8.0.jar and
>>> tried to run it but got: 
>>> "no main manifest attribute, in ignite-visor-console-2.8.0.jar"
>>> 
>>> Clearly it must run somehow but I can't find any documentation on how to do
>>> it.
>>> 
>>> 
>>> 
>>> --
>>> Sent from: http://apache-ignite-users.70518.x6.nabble.com/ 
>>> 
>> 
>> 
> 
>