Re: Creating index failed
Hi Mario! I agree with you that the user wanted to index all the data in the region when using a partitioned region. But when the command is not successful, the cluster config is not updated. After the server restart, it will not have indexes as it is not stored in the cluster configuration. Interesting! If I understand you correctly, the initial request to each server succeeds, but later ones will fail because the index is already there. However, the first, successful request should also have updated the cluster config, right?. Am I misunderstanding something? From: Mario Kevo Sent: Tuesday, December 7, 2021 06:36 To: dev@geode.apache.org Subject: Odg: Creating index failed Hi Jason, I agree with you that the user wanted to index all the data in the region when using a partitioned region. But when the command is not successful, the cluster config is not updated. After the server restart, it will not have indexes as it is not stored in the cluster configuration. So there should be some changes, as the index is created on all members but the command is not successful. I'm working on a fix. As soon as possible I will create PR on the already mentioned ticket. BR, Mario Šalje: Jason Huynh Poslano: 6. prosinca 2021. 18:45 Prima: dev@geode.apache.org Predmet: Re: Creating index failed Hi Mario, A lot of the indexing code pre-dates GFSH. The behavior you are seeing is when an index is created on a partition region. When creating an index on a partition region, the idea is that the user wanted to index all the data in the region. So the server will let all other servers know to create an index on the partition region. This is slightly different for an index on a replicated region. That is when the index can be created on a per member basis, which is what I think the -member flag is for. GFSH however defaults to sending the create index message to all members for any index type from what I remember and from what is being described. That is why you’ll see the race condition with indexes created on partitioned regions but the end result being that the index that someone wanted to create is either created or already there. -Jason On 12/6/21, 6:37 AM, "Mario Kevo" wrote: Hi devs, While doing some testing, I found the issue which is already reported there. https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissues.apache.org%2Fjira%2Fbrowse%2FGEODE-7875&data=04%7C01%7Camurmann%40vmware.com%7C48b13b5a3485492868dd08d9b98efd67%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C1%7C637744846071934875%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=82bEcKRqw8yIP4MCx7hsVKgNprgdWu9Vh%2FNatImH2Vo%3D&reserved=0 If we run the create index command it will create an index locally and send a request to create an index on other members of that region. The problem happened if the remote request comes before the request from the locator, in that case, a request from the locator failed with the following message: Index "index1" already exists. Create failed due to duplicate name. This can be reproduced by running 6 servers with DEBUG log level(due to this system will be slower), creating a partitioned region, and then creating an index. Why does the server send remote requests to other members as they will get a request from the locator to create an index? Also when running the gfsh command to create an index on one member, it will send create index requests to all other members. In that case, what is the purpose of this --member flag? BR, Mario
Re: Creating index failed
The IndexTask is working as expected...To handle multiple request creating the same index. It looks like, when there is multiple index creation request; It is handled properly (using the Future task) What is the case when it throws Index already exists? The other option is if the index request matches (index type and expression) we should just ignore the exception; and consider index is successfully created. -Anil. On 12/7/21, 8:49 AM, "Mario Kevo" wrote: Are you thinking about not sending it to the remote nodes or not sending requests from locator to the each node? Also, there is one map where indexTask is stored, and there is putIfAbsent method which seems is not working properly. // This will return either the Index FutureTask or Index itself, based // on whether the index creation is in process or completed. Object ind = this.indexes.putIfAbsent(indexTask, indexFutureTask); In case we change it to something like: Object ind = null; if(!this.indexes.containsKey(indexTask)) { ind = this.indexes.put(indexTask, indexFutureTask); } If it already has that indexTask it will not go again to run creating it, whether or not that index is created by remote request or locally. And in that case, the command will be successful and the cluster config is updated. BR, Mario Šalje: Anilkumar Gingade Poslano: 7. prosinca 2021. 16:41 Prima: dev@geode.apache.org Predmet: Re: Creating index failed In case if you are planning to fix; the probable fix is not to send gfsh create command to all the nodes when its partitioned region.. On 12/7/21, 6:37 AM, "Mario Kevo" wrote: Hi Jason, I agree with you that the user wanted to index all the data in the region when using a partitioned region. But when the command is not successful, the cluster config is not updated. After the server restart, it will not have indexes as it is not stored in the cluster configuration. So there should be some changes, as the index is created on all members but the command is not successful. I'm working on a fix. As soon as possible I will create PR on the already mentioned ticket. BR, Mario Šalje: Jason Huynh Poslano: 6. prosinca 2021. 18:45 Prima: dev@geode.apache.org Predmet: Re: Creating index failed Hi Mario, A lot of the indexing code pre-dates GFSH. The behavior you are seeing is when an index is created on a partition region. When creating an index on a partition region, the idea is that the user wanted to index all the data in the region. So the server will let all other servers know to create an index on the partition region. This is slightly different for an index on a replicated region. That is when the index can be created on a per member basis, which is what I think the -member flag is for. GFSH however defaults to sending the create index message to all members for any index type from what I remember and from what is being described. That is why you’ll see the race condition with indexes created on partitioned regions but the end result being that the index that someone wanted to create is either created or already there. -Jason On 12/6/21, 6:37 AM, "Mario Kevo" wrote: Hi devs, While doing some testing, I found the issue which is already reported there. https://issues.apache.org/jira/browse/GEODE-7875 If we run the create index command it will create an index locally and send a request to create an index on other members of that region. The problem happened if the remote request comes before the request from the locator, in that case, a request from the locator failed with the following message: Index "index1" already exists. Create failed due to duplicate name. This can be reproduced by running 6 servers with DEBUG log level(due to this system will be slower), creating a partitioned region, and then creating an index. Why does the server send remote requests to other members as they will get a request from the locator to create an index? Also when running the gfsh command to create an index on one member, it will send create index requests to all other members. In that case, what is the purpose of this --member flag? BR, Mario
Odg: Creating index failed
Are you thinking about not sending it to the remote nodes or not sending requests from locator to the each node? Also, there is one map where indexTask is stored, and there is putIfAbsent method which seems is not working properly. // This will return either the Index FutureTask or Index itself, based // on whether the index creation is in process or completed. Object ind = this.indexes.putIfAbsent(indexTask, indexFutureTask); In case we change it to something like: Object ind = null; if(!this.indexes.containsKey(indexTask)) { ind = this.indexes.put(indexTask, indexFutureTask); } If it already has that indexTask it will not go again to run creating it, whether or not that index is created by remote request or locally. And in that case, the command will be successful and the cluster config is updated. BR, Mario Šalje: Anilkumar Gingade Poslano: 7. prosinca 2021. 16:41 Prima: dev@geode.apache.org Predmet: Re: Creating index failed In case if you are planning to fix; the probable fix is not to send gfsh create command to all the nodes when its partitioned region.. On 12/7/21, 6:37 AM, "Mario Kevo" wrote: Hi Jason, I agree with you that the user wanted to index all the data in the region when using a partitioned region. But when the command is not successful, the cluster config is not updated. After the server restart, it will not have indexes as it is not stored in the cluster configuration. So there should be some changes, as the index is created on all members but the command is not successful. I'm working on a fix. As soon as possible I will create PR on the already mentioned ticket. BR, Mario Šalje: Jason Huynh Poslano: 6. prosinca 2021. 18:45 Prima: dev@geode.apache.org Predmet: Re: Creating index failed Hi Mario, A lot of the indexing code pre-dates GFSH. The behavior you are seeing is when an index is created on a partition region. When creating an index on a partition region, the idea is that the user wanted to index all the data in the region. So the server will let all other servers know to create an index on the partition region. This is slightly different for an index on a replicated region. That is when the index can be created on a per member basis, which is what I think the -member flag is for. GFSH however defaults to sending the create index message to all members for any index type from what I remember and from what is being described. That is why you’ll see the race condition with indexes created on partitioned regions but the end result being that the index that someone wanted to create is either created or already there. -Jason On 12/6/21, 6:37 AM, "Mario Kevo" wrote: Hi devs, While doing some testing, I found the issue which is already reported there. https://issues.apache.org/jira/browse/GEODE-7875 If we run the create index command it will create an index locally and send a request to create an index on other members of that region. The problem happened if the remote request comes before the request from the locator, in that case, a request from the locator failed with the following message: Index "index1" already exists. Create failed due to duplicate name. This can be reproduced by running 6 servers with DEBUG log level(due to this system will be slower), creating a partitioned region, and then creating an index. Why does the server send remote requests to other members as they will get a request from the locator to create an index? Also when running the gfsh command to create an index on one member, it will send create index requests to all other members. In that case, what is the purpose of this --member flag? BR, Mario
Re: Creating index failed
In case if you are planning to fix; the probable fix is not to send gfsh create command to all the nodes when its partitioned region.. On 12/7/21, 6:37 AM, "Mario Kevo" wrote: Hi Jason, I agree with you that the user wanted to index all the data in the region when using a partitioned region. But when the command is not successful, the cluster config is not updated. After the server restart, it will not have indexes as it is not stored in the cluster configuration. So there should be some changes, as the index is created on all members but the command is not successful. I'm working on a fix. As soon as possible I will create PR on the already mentioned ticket. BR, Mario Šalje: Jason Huynh Poslano: 6. prosinca 2021. 18:45 Prima: dev@geode.apache.org Predmet: Re: Creating index failed Hi Mario, A lot of the indexing code pre-dates GFSH. The behavior you are seeing is when an index is created on a partition region. When creating an index on a partition region, the idea is that the user wanted to index all the data in the region. So the server will let all other servers know to create an index on the partition region. This is slightly different for an index on a replicated region. That is when the index can be created on a per member basis, which is what I think the -member flag is for. GFSH however defaults to sending the create index message to all members for any index type from what I remember and from what is being described. That is why you’ll see the race condition with indexes created on partitioned regions but the end result being that the index that someone wanted to create is either created or already there. -Jason On 12/6/21, 6:37 AM, "Mario Kevo" wrote: Hi devs, While doing some testing, I found the issue which is already reported there. https://issues.apache.org/jira/browse/GEODE-7875 If we run the create index command it will create an index locally and send a request to create an index on other members of that region. The problem happened if the remote request comes before the request from the locator, in that case, a request from the locator failed with the following message: Index "index1" already exists. Create failed due to duplicate name. This can be reproduced by running 6 servers with DEBUG log level(due to this system will be slower), creating a partitioned region, and then creating an index. Why does the server send remote requests to other members as they will get a request from the locator to create an index? Also when running the gfsh command to create an index on one member, it will send create index requests to all other members. In that case, what is the purpose of this --member flag? BR, Mario
Odg: Creating index failed
Hi Jason, I agree with you that the user wanted to index all the data in the region when using a partitioned region. But when the command is not successful, the cluster config is not updated. After the server restart, it will not have indexes as it is not stored in the cluster configuration. So there should be some changes, as the index is created on all members but the command is not successful. I'm working on a fix. As soon as possible I will create PR on the already mentioned ticket. BR, Mario Šalje: Jason Huynh Poslano: 6. prosinca 2021. 18:45 Prima: dev@geode.apache.org Predmet: Re: Creating index failed Hi Mario, A lot of the indexing code pre-dates GFSH. The behavior you are seeing is when an index is created on a partition region. When creating an index on a partition region, the idea is that the user wanted to index all the data in the region. So the server will let all other servers know to create an index on the partition region. This is slightly different for an index on a replicated region. That is when the index can be created on a per member basis, which is what I think the -member flag is for. GFSH however defaults to sending the create index message to all members for any index type from what I remember and from what is being described. That is why you’ll see the race condition with indexes created on partitioned regions but the end result being that the index that someone wanted to create is either created or already there. -Jason On 12/6/21, 6:37 AM, "Mario Kevo" wrote: Hi devs, While doing some testing, I found the issue which is already reported there. https://issues.apache.org/jira/browse/GEODE-7875 If we run the create index command it will create an index locally and send a request to create an index on other members of that region. The problem happened if the remote request comes before the request from the locator, in that case, a request from the locator failed with the following message: Index "index1" already exists. Create failed due to duplicate name. This can be reproduced by running 6 servers with DEBUG log level(due to this system will be slower), creating a partitioned region, and then creating an index. Why does the server send remote requests to other members as they will get a request from the locator to create an index? Also when running the gfsh command to create an index on one member, it will send create index requests to all other members. In that case, what is the purpose of this --member flag? BR, Mario