Re: Failed to get page IO instance (page content is corrupted) after onenode failed when trying to reboot.
Here's my analysis
Looks like I encountered this bug
https://issues.apache.org/jira/browse/IGNITE-8659
Because in log file ignite-9d66b750-first-restart-node1.log, I see
[2018-11-29T03:01:39,135][INFO
][exchange-worker-#162][GridCachePartitionExchangeManager] Rebalancing started
[top=AffinityTopologyVersion [topVer=11834, minorTopVer=0], evt=NODE_JOINED,
node=6018393e-a88c-40f5-8d77-d136d5226741]
[2018-11-29T03:01:39,136][INFO
][exchange-worker-#162][GridDhtPartitionDemander] Starting rebalancing
[grp=SQL_PUBLIC_WBXSITEACCOUNT, mode=ASYNC,
fromNode=6018393e-a88c-40f5-8d77-d136d5226741, partitionsCount=345,
topology=AffinityTopologyVersion [topVer=11834, minorTopVer=0], rebalanceId=47]
But why did rebalance started after two hours after the node started?
Is it because PME got stuck for two hours?
Also it looks like the PME got stuck again when rebalance started (This is when
I restarted node2 and node3).
Because in the same log file, I see
[2018-11-29T03:01:59,443][WARN
][exchange-worker-#162][GridDhtPartitionsExchangeFuture] Unable to await
partitions release latch within timeout: ServerLatch [permits=2,
pendingAcks=[6018393e-a88c-40f5-8d77-d136d5226741,
75a180ea-78de-4d63-8bd5-291557bd58f4], super=CompletableLatch [id=exchange,
topVer=AffinityTopologyVersion [topVer=11835, minorTopVer=0]]]
Based on this document
https://apacheignite.readme.io/docs/rebalancing#section-rebalance-modes, the
rebalance is async by default.
So what is block PME this time?
So basically I have three questions.
1. Why node1 can't join cluster("Still waiting for initial partition map
exchange" for two hours) when restarted?
Is it because node2 and node3 have some newly ingested data when node1 is down?
2. Why is node3 blocked by " Unable to await partitions release latch within
timeout " when restarted?
3. Is https://issues.apache.org/jira/browse/IGNITE-8659 the solution?
Andrew, can you take a look please?
I think it's a critical problem because the only way to get node1 working is to
delete data and wal folder.
No need to say, it will cause data loss.
Thanks
Ray wrote:
This issue happened again.
Here's the summary.
I'm running a three nodes of Ignite 2.6 cluster with these config
http://www.springframework.org/schema/beans";
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance";
xsi:schemaLocation="
http://www.springframework.org/schema/beans
http://www.springframework.org/schema/beans/spring-beans.xsd";>
node1:49500
node2:49500
node3:49500
I have a few caches setup with TTL with enabled persistence.
Why I'm mentioning this because I check this thread
http://apache-ignite-users.70518.x6.nabble.com/And-again-Failed-to-get-page-IO-instance-page-content-is-corrupted-td20095.html#a22037
and a few tickets mentioned in this ticket.
https://issues.apache.org/jira/browse/IGNITE-8659
https://issues.apache.org/jira/browse/IGNITE-5874
Other issues is ignored because they're already fixed in 2.6
Node1 goes down because of a long GC pause.
When I try to restart Ignite service on Node1, I got "Still waiting for
initial partition map exchange" warning log going on for more than 2 hours.
[WARN ][main][GridCachePartitionExchangeManager] Still waiting for initial
partition map exchange [fut=GridDhtPartitionsExchangeFuture
[firstDiscoEvt=DiscoveryEvent [evtNode=TcpDiscoveryNode
[id=9d66b750-09a3-4f0e-afa9-7cf24847ee6a, addrs=[10.252.4.60, 127.0.0.1],
sockAddrs=[rpsj1ign001.webex.com/10.252.4.60:49500, /127.0.0.1:49500],
discPort=49500, order=11813, intOrder=5909, lastExchangeTime=1543451981558,
loc=true, ver=2.6.0#20180709-sha1:5faffcee, isClient=false], topVer=11813,
nodeId8=9d66b750, msg=null, type=NODE_JOINED, tstamp=1543451943071],
crd=TcpDiscoveryNode [id=f14c8e36-9a20-4668-b52e-0de64c743700,
addrs=[10.252.10.20, 127.0.0.1],
sockAddrs=[rpsj1ign003.webex.com/10.252.10.20:49500, /127.0.0.1:4950
Ignite SQL function questions
I've been reading Ignite official document and going through Ignite source code for a few while now. From the source code, I see that Ignite can be a key/value store and the data is stored in a ConcurrentHashMap for every partition. So, this is one copy of data in memory. For Ignite SQL function, it seems like Ignite creates a H2 table in memory based on the fields user want to be query with SQL. So, does it mean for every key with @QuerySqlField annotation, there will be two copies of data in memory (one in ConcurrentHashMap and one in in-memory H2 table)?
Ignite cluster with persistent store enabled did not load from wal after restarting.
Hi All,
I’m using Ignite version 2.1.
I opened persistent sotre by adding these configuration in the xml file
according to https://apacheignite.readme.io/docs/distributed-persistent-store
Here’s the code I used to ingest data to ignite.
public class IgniteTest {
public static void main(String[] args) throws Exception {
try (Ignite ignite = Ignition.start("example-ignite.xml")) {
CacheConfiguration personCacheCfg = new
CacheConfiguration<>("person");
personCacheCfg.setIndexedTypes(Integer.class, Person.class);
IgniteCache personCache =
ignite.getOrCreateCache(personCacheCfg);
Person p1 = new Person(1, "John", "Doe", 2000);
Person p2 = new Person(1, "Jane", "Doe", 1000);
Person p3 = new Person(2, "John", "Smith", 1000);
Person p4 = new Person(2, "Jane", "Smith", 2000);
personCache.put(p1.getId(), p1);
personCache.put(p2.getId(), p2);
personCache.put(p3.getId(), p3);
personCache.put(p4.getId(), p4);
}
}
public static class Person implements Serializable {
private static int PERSON_ID = 0;
@QuerySqlField(index = true)
private Integer id;
@QuerySqlField(index = true)
private Integer orgId;
@QuerySqlField
private String firstName;
@QuerySqlField
private String lastName;
@QuerySqlField(index = true)
private double salary;
Person(Integer orgId, String firstName, String lastName, double salary)
{
id = PERSON_ID++;
this.orgId = orgId;
this.firstName = firstName;
this.lastName = lastName;
this.salary = salary;
}
public Integer getOrganizationId() {
return orgId;
}
public String getFirstName() {
return firstName;
}
public String getLastName() {
return lastName;
}
public double getSalary() {
return salary;
}
public Integer getId() {
return id;
}
}
}
And I can see the wal log files generated under /$IgniteHome/work/db folder.
After restarting the cluster, the “person” cache should be re-created
automatically, right?
But it didn’t.
Here’s some debug log may help locate the problem
[2017-09-15T16:34:13,090][INFO ][main][CacheObjectBinaryProcessorImpl] Resolved
directory for serialized binary metadata:
/apache-ignite-fabric-2.1.0-bin/work/binary_meta/0_0_0_0_0_0_0_1_10_140_48_140_127_0_0_1_2001_420_589a_1250_516a_c5d4_fe7d_863c_47500
[2017-09-15T16:34:13,225][DEBUG][main][GridCacheMvccManager] Cache manager
started.
[2017-09-15T16:34:13,225][DEBUG][main][GridCacheVersionManager] Cache manager
started.
[2017-09-15T16:34:13,243][DEBUG][main][IgniteTxManager] Cache manager started.
[2017-09-15T16:34:13,243][INFO ][main][FilePageStoreManager] Resolved page
store work directory:
//apache-ignite-fabric-2.1.0-bin/work/db/0_0_0_0_0_0_0_1_10_140_48_140_127_0_0_1_2001_420_589a_1250_516a_c5d4_fe7d_863c_47500
[2017-09-15T16:34:13,243][DEBUG][main][FilePageStoreManager] Cache manager
started.
[2017-09-15T16:34:13,243][INFO ][main][FileWriteAheadLogManager] Resolved write
ahead log work directory:
/apache-ignite-fabric-2.1.0-bin/work/db/wal/0_0_0_0_0_0_0_1_10_140_48_140_127_0_0_1_2001_420_589a_1250_516a_c5d4_fe7d_863c_47500
[2017-09-15T16:34:13,244][INFO ][main][FileWriteAheadLogManager] Resolved write
ahead log archive directory:
//apache-ignite-fabric-2.1.0-bin/work/db/wal/archive/0_0_0_0_0_0_0_1_10_140_48_140_127_0_0_1_2001_420_589a_1250_516a_c5d4_fe7d_863c_47500
[2017-09-15T16:34:13,253][DEBUG][main][FileWriteAheadLogManager] Cache manager
started.
Re: Issue when executing sql query using REST API
Hi Alexey,
Thanks for the information, the SqlFieldsQuery works fine for me.
But I still want to try out the sql query.
After uploading the jar file in the $IgniteHome/lib folder, a new exception
shows up.
The rest response says
No serializer found for class Ignite$Test and no properties discovered to
create BeanSerializer
Any ideas?
From: Alexey Kukushkin
Reply-To: "user@ignite.apache.org"
Date: Wednesday, 13 September 2017 at 17:46
To: "user@ignite.apache.org"
Subject: Re: Issue when executing sql query using REST API
Hi Ray,
You can avoid deploying classes to the server nodes by using Ignite API that
does not require building POJOs (IgniteTest$Person in your case).
* For SQL API use
SqlFieldsQuery<https://apacheignite.readme.io/docs/sql-queries#section-sqlfieldsqueries>
that, unlike SqlQuery, does not build a full object
* For REST API use command "qryfldexe"
<https://apacheignite.readme.io/docs/rest-api#sql-fields-query-execute> (that
will result in SqlFieldsQuery)
* For Java API use IgniteBinary
API<https://apacheignite.readme.io/docs/binary-marshaller#binaryobject-cache-api>.
Also, Ignite has Zero deployment
feature<https://apacheignite.readme.io/docs/zero-deployment> applicable only to
the computational code (compute closures and tasks) that enables automated
deployment. But as I said it is not applicable to the metadata (cache) classes.
On Wed, Sep 13, 2017 at 12:23 PM, Ray Liu (rayliu)
mailto:ray...@cisco.com>> wrote:
Do I have to deploy my classes on all of the server nodes manually?
Every time I added some new classes, I have to do this again?
From: Alexey Kukushkin
mailto:kukushkinale...@gmail.com>>
Reply-To: "user@ignite.apache.org<mailto:user@ignite.apache.org>"
mailto:user@ignite.apache.org>>
Date: Wednesday, 13 September 2017 at 17:20
To: "user@ignite.apache.org<mailto:user@ignite.apache.org>"
mailto:user@ignite.apache.org>>
Subject: Re: Issue when executing sql query using REST API
Ray, I am sorry - just read your second email and I realised I originally
misunderstood your question. You guessed right, your classes need to be
deployed on the server nodes and on the server nodes you need to set the
USER_LIBS environment variable like:
export USER_LIBS=/.jar
On Wed, Sep 13, 2017 at 12:15 PM, Ray Liu (rayliu)
mailto:ray...@cisco.com>> wrote:
Can you share your configuration xml?
I think it’s a binary masharller configuration issue.
From: Alexey Kukushkin
mailto:kukushkinale...@gmail.com>>
Reply-To: "user@ignite.apache.org<mailto:user@ignite.apache.org>"
mailto:user@ignite.apache.org>>
Date: Wednesday, 13 September 2017 at 17:12
To: "user@ignite.apache.org<mailto:user@ignite.apache.org>"
mailto:user@ignite.apache.org>>
Subject: Re: Issue when executing sql query using REST API
Hi Ray,
Just copied your code into a simple single file sample project, ran it and your
query worked fine for me:
1. Request URL:
http://127.0.0.1:8080/ignite?cacheName=person&pageSize=1&qry=limit+2&cmd=qryexe&type=Person&;
2. Request Method:
GET
3. Status Code:
200 OK
4. -
5.
{"successStatus":0,"error":null,"response":{"items":[{"key":0,"value":{"id":0,"firstName":"John","lastName":"Doe","salary":2000.0,"organizationId":1}}],"last":false,"fieldsMetadata":[],"queryId":2},"sessionToken":null}
--
Best regards,
Alexey
--
Best regards,
Alexey
Re: Issue when executing sql query using REST API
Do I have to deploy my classes on all of the server nodes manually?
Every time I added some new classes, I have to do this again?
From: Alexey Kukushkin
Reply-To: "user@ignite.apache.org"
Date: Wednesday, 13 September 2017 at 17:20
To: "user@ignite.apache.org"
Subject: Re: Issue when executing sql query using REST API
Ray, I am sorry - just read your second email and I realised I originally
misunderstood your question. You guessed right, your classes need to be
deployed on the server nodes and on the server nodes you need to set the
USER_LIBS environment variable like:
export USER_LIBS=/.jar
On Wed, Sep 13, 2017 at 12:15 PM, Ray Liu (rayliu)
mailto:ray...@cisco.com>> wrote:
Can you share your configuration xml?
I think it’s a binary masharller configuration issue.
From: Alexey Kukushkin
mailto:kukushkinale...@gmail.com>>
Reply-To: "user@ignite.apache.org<mailto:user@ignite.apache.org>"
mailto:user@ignite.apache.org>>
Date: Wednesday, 13 September 2017 at 17:12
To: "user@ignite.apache.org<mailto:user@ignite.apache.org>"
mailto:user@ignite.apache.org>>
Subject: Re: Issue when executing sql query using REST API
Hi Ray,
Just copied your code into a simple single file sample project, ran it and your
query worked fine for me:
1. Request URL:
http://127.0.0.1:8080/ignite?cacheName=person&pageSize=1&qry=limit+2&cmd=qryexe&type=Person&;
2. Request Method:
GET
3. Status Code:
200 OK
4. -
5.
{"successStatus":0,"error":null,"response":{"items":[{"key":0,"value":{"id":0,"firstName":"John","lastName":"Doe","salary":2000.0,"organizationId":1}}],"last":false,"fieldsMetadata":[],"queryId":2},"sessionToken":null}
--
Best regards,
Alexey
Re: Issue when executing sql query using REST API
Can you share your configuration xml?
I think it’s a binary masharller configuration issue.
From: Alexey Kukushkin
Reply-To: "user@ignite.apache.org"
Date: Wednesday, 13 September 2017 at 17:12
To: "user@ignite.apache.org"
Subject: Re: Issue when executing sql query using REST API
Hi Ray,
Just copied your code into a simple single file sample project, ran it and your
query worked fine for me:
1. Request URL:
http://127.0.0.1:8080/ignite?cacheName=person&pageSize=1&qry=limit+2&cmd=qryexe&type=Person&;
2. Request Method:
GET
3. Status Code:
200 OK
4. -
5.
{"successStatus":0,"error":null,"response":{"items":[{"key":0,"value":{"id":0,"firstName":"John","lastName":"Doe","salary":2000.0,"organizationId":1}}],"last":false,"fieldsMetadata":[],"queryId":2},"sessionToken":null}
Re: Issue when executing sql query using REST API
Also sql scan query is not working with the following rest call.
http://127.0.0.1:8080/ignite?cmd=qryscanexe&pageSize=10&cacheName=person&className=IgniteTest$Person
And the error response is
{
"successStatus": 1,
"error": "Failed to find target class: IgniteTest$Person",
"response": null,
"sessionToken": null
}
But sql fields query is working with the following rest call.
http://127.0.0.1:8080/ignite?cmd=qryfldexe&pageSize=10&cacheName=person&qry=select+firstName%2C+lastName+from+Person
So I’m guessing the Person class should be somehow uploaded to the lib folder
under Ignitehome folder?
From: "Ray Liu (rayliu)"
Reply-To: "user@ignite.apache.org"
Date: Wednesday, 13 September 2017 at 14:56
To: "user@ignite.apache.org"
Subject: Issue when executing sql query using REST API
Hi all,
I’m trying to execute a sql query using REST API with the following url.
http://127.0.0.1:8080/ignite?cacheName=person&pageSize=1&qry=limit+2&cmd=qryexe&type=Person&;
But I got the following error response
{
"successStatus": 1,
"error": "IgniteTest$Person",
"response": null,
"sessionToken": null
}
Here’s the code I used to ingest data to ignite.
public class IgniteTest {
public static void main(String[] args) throws Exception {
try (Ignite ignite = Ignition.start("example-ignite.xml")) {
CacheConfiguration personCacheCfg = new
CacheConfiguration<>("person");
personCacheCfg.setIndexedTypes(Integer.class, Person.class);
IgniteCache personCache =
ignite.getOrCreateCache(personCacheCfg);
Person p1 = new Person(1, "John", "Doe", 2000);
Person p2 = new Person(1, "Jane", "Doe", 1000);
Person p3 = new Person(2, "John", "Smith", 1000);
Person p4 = new Person(2, "Jane", "Smith", 2000);
personCache.put(p1.getId(), p1);
personCache.put(p2.getId(), p2);
personCache.put(p3.getId(), p3);
personCache.put(p4.getId(), p4);
}
}
/**
* Person class.
*/
public static class Person implements Serializable {
/** Person id. */
private static int PERSON_ID = 0;
/** Person ID (indexed). */
@QuerySqlField(index = true)
private Integer id;
/** Organization id. */
@QuerySqlField(index = true)
private Integer orgId;
/** First name (not-indexed). */
@QuerySqlField
private String firstName;
/** Last name (not indexed). */
@QuerySqlField
private String lastName;
/** Salary (indexed). */
@QuerySqlField(index = true)
private double salary;
/**
* @param firstName First name.
* @param lastName Last name.
* @param salary Salary.
*/
Person(Integer orgId, String firstName, String lastName, double salary)
{
id = PERSON_ID++;
this.orgId = orgId;
this.firstName = firstName;
this.lastName = lastName;
this.salary = salary;
}
/**
* @return Organization ID.
*/
public Integer getOrganizationId() {
return orgId;
}
/**
* @return First name.
*/
public String getFirstName() {
return firstName;
}
/**
* @return Last name.
*/
public String getLastName() {
return lastName;
}
/**
* @return Salary.
*/
public double getSalary() {
return salary;
}
/**
* @return Id.
*/
public Integer getId() {
return id;
}
}
}
And debug log generated by ignite
[2017-09-13T14:42:33,059][DEBUG][qtp379430898-73][GridJettyRestProtocol]
Handling request [target=/ignite, req=(GET
/ignite?cacheName=person&pageSize=1&qry=limit+2&cmd=qryexe&type=Person)@1162648735
org.eclipse.jetty.server.Request@454c9c9f, srvReq=(GET
/ignite?cacheName=person&pageSize=1&qry=limit+2&cmd=qryexe&type=Person)@1162648735
org.eclipse.jetty.server.Request@454c9c9f]
[2017-09-13T14:42:33,059][DEBUG][qtp379430898-73][GridJettyRestProtocol]
Initialized command request: GridRestRequest [destId=null, clientId=null,
addr=/127.0.0.1:50853, cmd=EXECUTE_SQL_QUERY]
[2017-09-13T14:42:33,060][DEBUG][rest-#42%null%][GridRestProcessor] Grid
runnable started: rest-proc-worker
[2017-09-13T14:42:33,060][DEBUG][rest-#42%null%][GridRestProcessor] Received
request from client: GridRestRequest [destId=null, clientId=null,
addr=/127.0.0.1:50853, cmd=EXECUTE_SQL_QUERY]
[2017-09-1
Issue when executing sql query using REST API
Hi all,
I’m trying to execute a sql query using REST API with the following url.
http://127.0.0.1:8080/ignite?cacheName=person&pageSize=1&qry=limit+2&cmd=qryexe&type=Person&;
But I got the following error response
{
"successStatus": 1,
"error": "IgniteTest$Person",
"response": null,
"sessionToken": null
}
Here’s the code I used to ingest data to ignite.
public class IgniteTest {
public static void main(String[] args) throws Exception {
try (Ignite ignite = Ignition.start("example-ignite.xml")) {
CacheConfiguration personCacheCfg = new
CacheConfiguration<>("person");
personCacheCfg.setIndexedTypes(Integer.class, Person.class);
IgniteCache personCache =
ignite.getOrCreateCache(personCacheCfg);
Person p1 = new Person(1, "John", "Doe", 2000);
Person p2 = new Person(1, "Jane", "Doe", 1000);
Person p3 = new Person(2, "John", "Smith", 1000);
Person p4 = new Person(2, "Jane", "Smith", 2000);
personCache.put(p1.getId(), p1);
personCache.put(p2.getId(), p2);
personCache.put(p3.getId(), p3);
personCache.put(p4.getId(), p4);
}
}
/**
* Person class.
*/
public static class Person implements Serializable {
/** Person id. */
private static int PERSON_ID = 0;
/** Person ID (indexed). */
@QuerySqlField(index = true)
private Integer id;
/** Organization id. */
@QuerySqlField(index = true)
private Integer orgId;
/** First name (not-indexed). */
@QuerySqlField
private String firstName;
/** Last name (not indexed). */
@QuerySqlField
private String lastName;
/** Salary (indexed). */
@QuerySqlField(index = true)
private double salary;
/**
* @param firstName First name.
* @param lastName Last name.
* @param salary Salary.
*/
Person(Integer orgId, String firstName, String lastName, double salary)
{
id = PERSON_ID++;
this.orgId = orgId;
this.firstName = firstName;
this.lastName = lastName;
this.salary = salary;
}
/**
* @return Organization ID.
*/
public Integer getOrganizationId() {
return orgId;
}
/**
* @return First name.
*/
public String getFirstName() {
return firstName;
}
/**
* @return Last name.
*/
public String getLastName() {
return lastName;
}
/**
* @return Salary.
*/
public double getSalary() {
return salary;
}
/**
* @return Id.
*/
public Integer getId() {
return id;
}
}
}
And debug log generated by ignite
[2017-09-13T14:42:33,059][DEBUG][qtp379430898-73][GridJettyRestProtocol]
Handling request [target=/ignite, req=(GET
/ignite?cacheName=person&pageSize=1&qry=limit+2&cmd=qryexe&type=Person)@1162648735
org.eclipse.jetty.server.Request@454c9c9f, srvReq=(GET
/ignite?cacheName=person&pageSize=1&qry=limit+2&cmd=qryexe&type=Person)@1162648735
org.eclipse.jetty.server.Request@454c9c9f]
[2017-09-13T14:42:33,059][DEBUG][qtp379430898-73][GridJettyRestProtocol]
Initialized command request: GridRestRequest [destId=null, clientId=null,
addr=/127.0.0.1:50853, cmd=EXECUTE_SQL_QUERY]
[2017-09-13T14:42:33,060][DEBUG][rest-#42%null%][GridRestProcessor] Grid
runnable started: rest-proc-worker
[2017-09-13T14:42:33,060][DEBUG][rest-#42%null%][GridRestProcessor] Received
request from client: GridRestRequest [destId=null, clientId=null,
addr=/127.0.0.1:50853, cmd=EXECUTE_SQL_QUERY]
[2017-09-13T14:42:33,060][DEBUG][rest-#42%null%][GridResourceProcessor]
Injecting resources
[target=org.apache.ignite.internal.processors.rest.handlers.query.QueryCommandHandler$ExecuteQueryCallable@12f55fbb]
[2017-09-13T14:42:33,060][DEBUG][rest-#42%null%][GridRestProcessor] Grid
runnable finished normally: rest-proc-worker
[2017-09-13T14:42:33,060][DEBUG][pub-#310%null%][GridClosureProcessor] Grid
runnable started: closure-proc-worker
[2017-09-13T14:42:33,061][DEBUG][pub-#310%null%][GridCacheProcessor] Getting
public cache for name: person
[2017-09-13T14:42:33,063][DEBUG][pub-#310%null%][IgniteH2Indexing] Set schema:
person
[2017-09-13T14:42:33,064][DEBUG][grid-timeout-worker-#15%null%][GridTimeoutProcessor]
Timeout has occurred: CancelableTask
[id=a5584d97e51-9027a323-d2b1-49c5-962d-e0dd53b5eba0, endTime=1505284953057,
period=3000, cancel=false,
task=org.apache.ignite.internal.processors.query.GridQueryProcessor$2@df38437]
[2017-09-13T14:42:33,070][DEBUG][pub-#310%null%][IgniteH2Indexing] Parsed
query: `SELECT "person"."PERSON"._KEY, "person"."PERSON"._VAL FROM
"person"."PERSON" limit 2` into two step que
