waiting for partition map exchange questions
hi All, I just encountered a situation in my k8s cluster where I'm running a 3 node ignite setup, with 2 client nodes. The server nodes have 8GB of off-heap per node, 8GB JVM (with g1gc) and 4GB of OS memory without persistence. I'm using Ignite 2.7. One of the ignite nodes got killed due to some issue in the cluster. I believe this was the sequence of events: -> Data Eviction spikes on two nodes in the cluster (NODE A & B), then 15 mins later.. -> NODE C goes down -> NODE D comes up (to replace node C) --> NODE D attempts a PME --> NODE B log = "Local node has detected failed nodes and started cluster-wide procedure" --> During PME the Ignite JVM on NODE D is restarted since it was taking too long and was killed by a k8s liveness probe. --> NODE D comes back up and attempts another PME ---> Note: i see these messages from all the nodes "First 10 pending exchange futures [total=2]" The total keeps ascending. The highest number I see is total=14. ---> NODE D log = "Failed to wait for initial partition map exchange. Possible reasons are:..." ---> NODE B log = "Possible starvation in striped pool. queue=[], dealock = false, Completed: 991189487 ..." ---> NODE A log = "Client node considered as unreachable and will be dropped from cluster, because no metrics update messages received in interval: TcpDiscoverySpi.clientFailureDetectionTimeout() ms. It may be caused by network problems or long GC pause on client node, try to increase this parameter. [nodeId=c5a92006-c29a-4a37-b149-7ec7855dc401, clientFailureDetectionTimeout=3]" NOTE that NODE D kept restarting due to a k8s liveness probe. I think I'm going to remove the probe or make it much more relaxed. During this time the ignite cluster is completely frozen. Restarting NODE D and replacing it with NODE E did not solve the issue. The only way I could solve the problem is to restart NODE B. Any idea why this could have occurred or what I can do to prevent it in the future? I do see this from the failureHandler: "FailureContext [type=CRITICAL_ERROR, err=class org.apache.ignite.IgniteException: Failed to create string representation of binary object.]" but not sure if this is something that would have caused the cluster to seize up. Overall nodes go down in this environment and come back all the time without issues. But I've seen problem occur twice in the last few months. I have logs & thread dumps for all the nodes in the system so if you want me to check anything in particular let me know. thanks, Scott -- Sent from: http://apache-ignite-users.70518.x6.nabble.com/
Re: Running control.sh in docker container when JXM is started
Hello! I suggest filing a ticket against Apache Ignite JIRA about that problem. Regards, -- Ilya Kasnacheev вт, 9 апр. 2019 г. в 18:18, kresimir.horvat : > Hi, > > solution you have suggested does work. So setting JMX_MON= before calling > control.sh will run script without exception. > Starting docker with NOJMX=1 and NOJMXI=1 doesn't. I found that > /docker/apache-ignite/run.sh script will start ignite.sh without -nojmx all > the time. If ignite.sh is started without -nojmx script parseargs.sh will > override NOJMX=1 value I have passed as env variable. > I had to change run.sh and commit new docker container. > > Is it possible to add support for -nojmx flag when running docker > container? > > Regards, > Kresimir > > > > -- > Sent from: http://apache-ignite-users.70518.x6.nabble.com/ >
Re: Running control.sh in docker container when JXM is started
Hi, solution you have suggested does work. So setting JMX_MON= before calling control.sh will run script without exception. Starting docker with NOJMX=1 and NOJMXI=1 doesn't. I found that /docker/apache-ignite/run.sh script will start ignite.sh without -nojmx all the time. If ignite.sh is started without -nojmx script parseargs.sh will override NOJMX=1 value I have passed as env variable. I had to change run.sh and commit new docker container. Is it possible to add support for -nojmx flag when running docker container? Regards, Kresimir -- Sent from: http://apache-ignite-users.70518.x6.nabble.com/
Cache#replace(K,V,V) implementation
1) How does Ignite implement this Cache#boolean replace(K key, V oldValue, V
newValue) function?JSR107 tells that it's equivalent to: if
(cache.containsKey(key) && *equals*(cache.get(key), oldValue)) {
cache.put(key, newValue); return true; } else { return false; }but I assume
this is just logical equivalence.How Ignite really perform
"equals(cache.get(key), oldValue)"? Or it uses some more intelligent way,
like comparing some hash codes or other kind of metadata. Performing
"equals" on the values could be very expensive operation.My question is very
practical. I'm implementing a Web-service that's backed by Ignite, which has
to support "Last-Modified" or "ETag" HTTP headers. These headers would make
sure the data has not been sent over the wire unnecessarily if nothing has
changed, and the data has not been concurrently updated since the last time
it was requested. These headers allow client or proxy to cache the data,
especially if the data is of a substantial size. I'm struggling to find the
most optimal way to map it to my cache operations. Therefore, here is my
second question.2) How would you recommend implementing "Last-Modified" or
"ETag" semantics based on Ignite? (more specifically in the context of
replacing/updating the values)
--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
Data count decreases when node down
I'm new with Ignite and I'm trying to test data quality and availability of Ignite cluster. I use the below xml configuration for setting cluster, x.x.x.1:47500..47509 x.x.x.2:47500..47509 Also and jthe CacheConfiguration is, My scenarios are, Loaded the 5 million data when all the 3 nodes Bring one node down The count shows 3.75 million. (Data loss) Bringing the node up counts 5 million again. I tried backup 1,2,3 all resulted in the same data loss. As per Ignite documents, appears the data loss should not happen. If this fixed, I can try adding data when the node is down and check how it behaves. Any suggestions, please? -- Sent from: http://apache-ignite-users.70518.x6.nabble.com/
Re: sql exception while batch inserting records
Hello! I thin you should not do conn.commit(); Instead you should do SET STREAMING OFF; COMMIT; is a statement that's not INSERT, which is probably causing the error. Regards, -- Ilya Kasnacheev вт, 9 апр. 2019 г. в 15:23, shivakumar : > HI Ilya Kasnacheev, > > The cluster is deployed with 2.7.0 version of ignite. > if i use 2.6.0 version of jar then i'am not getting any exception but if i > use 2.7.0 version of jar then i'am getting this exception. > > regards, > shiva > > > > > -- > Sent from: http://apache-ignite-users.70518.x6.nabble.com/ >
Re: sql exception while batch inserting records
HI Ilya Kasnacheev, The cluster is deployed with 2.7.0 version of ignite. if i use 2.6.0 version of jar then i'am not getting any exception but if i use 2.7.0 version of jar then i'am getting this exception. regards, shiva -- Sent from: http://apache-ignite-users.70518.x6.nabble.com/
Re: Running control.sh in docker container when JXM is started
Hello! Have you tried it? Does it work? Regards, -- Ilya Kasnacheev пн, 8 апр. 2019 г. в 14:31, kresimir.horvat : > Hi, > > thank you for your replay. I will try solution you suggested. > > If I want to avoid modifications and commits of changes in docker container > I will have to do the following: > 1. start container with JMX_MON specified so ignite will start with open > JMX > ports > 2. pass NOJMX = 1 (ignite.sh) and NOJMXI = 1 (control.sh) so my JMX_MON > won't be overridden by ignite.sh or control.sh > 3. when starting control.sh manually pass empty JMX_MON and NOJMXI = 1 > > Please correct me if I am wrong. > > Regards, > Kresimir > > > > -- > Sent from: http://apache-ignite-users.70518.x6.nabble.com/ >
Re: DataStreamer: Failed to process selector key
Hello! I think they keep data on heap in hopes of further use which will not happen. Please make sure you close them explicitly. Regards, -- Ilya Kasnacheev пн, 8 апр. 2019 г. в 20:49, kellan : > But are they actually affecting anything? Like are my DataStreamers > dropping > data, or is it being retried? > > > > -- > Sent from: http://apache-ignite-users.70518.x6.nabble.com/ >
Re: sql exception while batch inserting records
Hello!
What is the version of Ignite on your cluster? Please make sure that driver
version exactly matches cluster nodes' version.
Regards,
--
Ilya Kasnacheev
пн, 8 апр. 2019 г. в 16:17, shivakumar :
> Hi all,
> I'am getting this below exception while doing batch insert.
>
> *WARNING: Exception during batch send on streamed connection close
> java.sql.BatchUpdateException: Streaming mode supports only INSERT commands
> without subqueries.
> at
>
> org.apache.ignite.internal.jdbc.thin.JdbcThinConnection$StreamState.readResponses(JdbcThinConnection.java:1016)
> at java.lang.Thread.run(Thread.java:748)*
>
> Scenario:
> insert sql statement:
> InsertSql = "INSERT INTO PUBLIC.MY_TABLE (Id,
> time_stamp,second_time,last_time,period,name,
>
> iid,ptype,pversion) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?);"
>
> conn = DriverManager.getConnection(url, IGNITE_USER, IGNITE_PASS);
> conn.setAutoCommit(false);
> stmt = conn.createStatement();
> stmt.execute("SET STREAMING ON;");
> stmt.close();
>
> pStmt = conn.prepareStatement(table.getInsertSql());
> while (n < generate_rows) {
> table.createBatch(pStmt);
> n++;
> }
> count = pStmt.executeBatch();
> conn.commit();
> pStmt.close();
> igniteConn.close();
>
>
> when last igniteConn.close(); got called it's throwing above sql exception
> when i use* 2.7.0 ignite-core jar * and if i use *2.6.0 ignite-core jar*
> i'am not getting this exception.
>
> thanks with regards,
> shiva
>
>
>
>
>
>
>
> --
> Sent from: http://apache-ignite-users.70518.x6.nabble.com/
>
