Re: Question related to gateway-receivers connection load balancing
Jakov,
I'm not sure about the coordinator / non-coordinator behavior you're
describing, but I see the other behavior. It doesn't seem quite right.
Here is a detailed explanation of what I see.
LocatorLoadSnapshot.getServerForConnection increments the LoadHolder's
connections which updates its load. In the receiver's case,
getServerForConnection is called by the remote sender to get a server to
connect to (as you said using a ClientConnectionRequest).
Normally, the LoadHolder's load is updated when load is received in
LocatorLoadSnapshot.updateMap (also as you said using a
CacheServerLoadMessage). This doesn't happen in the receiver's case.
Locator behavior:
When the receiver connects, it sends its profile to the locator which causes
its LoadHolder to be added to the __recv__group map. It is not added to the
null group map. Thats the map for normal servers that have no group. Based on
current implementation, it can't be added to this map or it would be used for
normal local (to the receiver) client connections.
LocatorLoadSnapshot.addServer location=192.168.1.5:5409; groups=[__recv__group]
LocatorLoadSnapshot.addGroups group=__recv__group; location=192.168.1.5:5409
LocatorLoadSnapshot.addGroups not adding to the null map group=__recv__group;
location=192.168.1.5:5409
When the load is received for the receiver, it is ignored. updateMap gets the
LoadHolder from the null group map. Since that map was not updated for the
receiver, there is no entry for it (holder=null below).
LocatorLoadSnapshot.updateLoad about to update connectionLoadMap
location=192.168.1.5:5409
LocatorLoadSnapshot.updateMap location=192.168.1.5:5409; load=0.0;
loadPerConnection=0.00125
LocatorLoadSnapshot.updateMap location=192.168.1.5:5409; load=0.0;
loadPerConnection=0.00125; holder=null
LocatorLoadSnapshot.updateLoad ignoring load location=192.168.1.5:5409
LocatorLoadSnapshot.updateLoad done update connectionLoadMap
location=192.168.1.5:5409
So, load is not updated in this way for a receiver.
When a request for a remote receiver is received, it uses the __recv__group
load to provide that server. It also increments the load to that server (in
LoadHolder.incConnections). This is how load is updated for a receiver.
LocatorLoadSnapshot.getServerForConnection group=__recv__group
LocatorLoadSnapshot.getServerForConnection group=__recv__group;
potentialServers={192.168.1.5:5409@192.168.1.5(ln-1:81083):41002=LoadHolder[0.0,
192.168.1.5:5409, loadPollInterval=5000, 0.00125]}
LoadHolder.incConnections location=192.168.1.5:5409; load=0.00125
LocatorLoadSnapshot.getServerForConnection group=__recv__group;
usingServer=192.168.1.5:5409
Receiver server behavior:
When a receiver gets a new connection, the ServerConnection.processHandShake
updates the LoadMonitor. The LoadMonitor explicitly does not update the
connection count because isClientOperations=false, so the load is never changed
on the server. This is interesting behavior. I'm not sure if it wasn't updated
because it was unnecessary given the behavior of the locator above.
ServerConnection.processHandShake about to update the LoadMonitor
communicationMode=gateway
LoadMonitor.connectionOpened isClientOperations=false;
type=GatewayReceiverStatistics
LoadMonitor.connectionOpened did not increment connectionCount=0;
type=GatewayReceiverStatistics
ServerConnection.processHandShake done update the LoadMonitor
The LoadMonitor on the server does send the load periodically to the locator
but only because skippedLoadUpdates>forceUpdateFrequency (which is 10 times
through the polling loop by default):
PollingThread.run got load type=GatewayReceiverStatistics; load=Load(0.0,
0.00125, 0.0, 1.0)
PollingThread.run forceUpdateFrequency=true
PollingThread.run about to send CacheServerLoadMessage
type=GatewayReceiverStatistics; load=Load(0.0, 0.00125, 0.0, 1.0)
So, the load sent by the server is not accurate and its ignored in the locator.
I haven't had a chance to look at your PR yet.
Barry
From: Jakov Varenina
Sent: Thursday, March 10, 2022 1:21 AM
To: dev@geode.apache.org
Subject: Question related to gateway-receivers connection load balancing
Hi devs,
We have observed some weird behavior related to load balancing of
gateway-receivers connections in the geode cluster. Currently,
gateway-receiver connection load is only updated on coordinator locator
when it provides server location to remote gateway-sender in
ClientConnectionRequest{group=__recv_group...}/ClientConnectionResponse
messages exchange. Other locators never update gateway-receiver
connection load, since they are not handling these messages.
Additionally, locators (including the coordinator) ignore
CacheServerLoadMessage messages that are carrying the receiver's
connection load. This means that locators will not adjust the load when
the connection on some receiver is shut down.
Is this expected behavior or this is a bug?
You can find more information in
Re: Question about INDEX_THRESHOLD_SIZE
As an fyi, in the past we disabled applying limits at the index level for range
indexes.
I’m surprised in this case that we would add all the entries to the
intermediate results instead of applying the filter first and checking the
condition before adding to the intermediate results..
I would have thought it would have to apply the condition as seen in
CompactRangeIndex.addToResultsFromEntries :
if (ok && runtimeItr != null && iterOps != null) {
ok = QueryUtils.applyCondition(iterOps, context);
}
I haven’t walked this query through the code, perhaps it’s hitting a different
index type (I’d think a map index but that probably is backed by
CompactRangeIndexes for each key…)
From: Jason Huynh
Date: Friday, March 11, 2022 at 12:47 PM
To: dev@geode.apache.org
Subject: Re: Question about INDEX_THRESHOLD_SIZE
I think https://github.com/apache/geode/pull/7010 may have changed what that
property represented. I believe it was some arbitrary threshold to abort using
index look ups (if the intermediate results were small, it was deemed faster to
just iterate through and not do a lookup – at least from my interpretation of
the code)
It looks like with the change, it now munges it with limit.. so now limit is
applied to that value.. gfsh happens to always pass in a limit too, so there is
possibly additional confusion
From the diff there is also one spot where a limit != -1 had not been added..
In CompactRangeIndex line 489:
if (limit < indexThresholdSize) {
limit = indexThresholdSize;
}
This might be affecting the usage of limit at the index level?
From: Anilkumar Gingade
Date: Friday, March 11, 2022 at 12:11 PM
To: dev@geode.apache.org
Subject: Re: Question about INDEX_THRESHOLD_SIZE
Mario,
There is similar test/example added by you in QueryWithRangeIndexDUnitTest.
testQueryWithWildcardAndIndexOnAttributeFromHashMap()
When I run that test (on develop); I see the results as expected:
*
Command result for :
Result : true
Limit : 100
Rows: 1
Query Trace : Query Executed in 85.1964 ms; indexesUsed(1):IdIndex(Results:
1)
Are you running your test with any additional change like as you are saying :
>> I was working on allowing INDEX_THRESHOLD_SIZE System property to override
>> CompiledValue.RESULT_LIMIT.
If so , you need to look at the change and see why its impacting that way.
If not, please let me know what change can be made in that test to reproduce
the issue you are seeing; that will help to debug/analyze the issue.
-Anil.
On 3/11/22, 12:18 AM, "Mario Kevo" wrote:
Hi,
It works without an index but it doesn't work with an index.
When I revert changes, it takes INDEX_THRESHOLD_SIZE default value(100).
And if the entry that matches the condition is not in that resultset it will
not be printed.
Without index:
gfsh>query --query="SELECT e.key, e.value from
/example-region.entrySet e where e.value.positions['SUN'] like 'someth%'"
Result : true
Limit : 100
Rows: 1
Query Trace : Query Executed in 11.502283 ms; indexesUsed(0)
key | value
--- |
300 |
{"ID":300,"indexKey":0,"pkid":"300","shortID":null,"position1":{"mktValue":1945.0,"secId":"ORCL","secIdIndexed":"ORCL","secType":null,"sharesOutstanding":1944000.0,"underlyer":null,"pid":1944,"portfolioId":300,..
With index:
gfsh>query --query="SELECT e.key, e.value from
/example-region.entrySet e where e.value.positions['SUN'] like 'someth%'"
Result : true
Limit : 100
Rows: 0
Query Trace : Query Executed in 8.784831 ms; indexesUsed(1):index1(Results:
100)
BR,
Mario
Šalje: Anilkumar Gingade
Poslano: 10. ožujka 2022. 23:16
Prima: dev@geode.apache.org
Predmet: Re: Question about INDEX_THRESHOLD_SIZE
Mario,
There are few changes happened around this area as part of GEODE-9632 fix;
can you please revert that change and see if the query works both with and
without index.
Looking at the code; it seems to restrict the number index look up that
needs to be performed; certain latency/throughput sensitive queries that or not
expecting exact result may use this (my guess) but by default it should not be
resulting in unexpected results.
-Anil.
On 3/10/22, 6:50 AM, "Mario Kevo" wrote:
Hi geode-dev,
Some time ago I was working on allowing INDEX_THRESHOLD_SIZE System
property to override CompiledValue.RESULT_LIMIT.
After this change, adding this attribute will take into a count if you
set it.
But I need some clarification of this INDEX_THRESHOLD_SIZE attribute.
Why is this set by default to
Re: Question about INDEX_THRESHOLD_SIZE
I think https://github.com/apache/geode/pull/7010 may have changed what that
property represented. I believe it was some arbitrary threshold to abort using
index look ups (if the intermediate results were small, it was deemed faster to
just iterate through and not do a lookup – at least from my interpretation of
the code)
It looks like with the change, it now munges it with limit.. so now limit is
applied to that value.. gfsh happens to always pass in a limit too, so there is
possibly additional confusion
From the diff there is also one spot where a limit != -1 had not been added..
In CompactRangeIndex line 489:
if (limit < indexThresholdSize) {
limit = indexThresholdSize;
}
This might be affecting the usage of limit at the index level?
From: Anilkumar Gingade
Date: Friday, March 11, 2022 at 12:11 PM
To: dev@geode.apache.org
Subject: Re: Question about INDEX_THRESHOLD_SIZE
Mario,
There is similar test/example added by you in QueryWithRangeIndexDUnitTest.
testQueryWithWildcardAndIndexOnAttributeFromHashMap()
When I run that test (on develop); I see the results as expected:
*
Command result for :
Result : true
Limit : 100
Rows: 1
Query Trace : Query Executed in 85.1964 ms; indexesUsed(1):IdIndex(Results:
1)
Are you running your test with any additional change like as you are saying :
>> I was working on allowing INDEX_THRESHOLD_SIZE System property to override
>> CompiledValue.RESULT_LIMIT.
If so , you need to look at the change and see why its impacting that way.
If not, please let me know what change can be made in that test to reproduce
the issue you are seeing; that will help to debug/analyze the issue.
-Anil.
On 3/11/22, 12:18 AM, "Mario Kevo" wrote:
Hi,
It works without an index but it doesn't work with an index.
When I revert changes, it takes INDEX_THRESHOLD_SIZE default value(100).
And if the entry that matches the condition is not in that resultset it will
not be printed.
Without index:
gfsh>query --query="SELECT e.key, e.value from
/example-region.entrySet e where e.value.positions['SUN'] like 'someth%'"
Result : true
Limit : 100
Rows: 1
Query Trace : Query Executed in 11.502283 ms; indexesUsed(0)
key | value
--- |
300 |
{"ID":300,"indexKey":0,"pkid":"300","shortID":null,"position1":{"mktValue":1945.0,"secId":"ORCL","secIdIndexed":"ORCL","secType":null,"sharesOutstanding":1944000.0,"underlyer":null,"pid":1944,"portfolioId":300,..
With index:
gfsh>query --query="SELECT e.key, e.value from
/example-region.entrySet e where e.value.positions['SUN'] like 'someth%'"
Result : true
Limit : 100
Rows: 0
Query Trace : Query Executed in 8.784831 ms; indexesUsed(1):index1(Results:
100)
BR,
Mario
Šalje: Anilkumar Gingade
Poslano: 10. ožujka 2022. 23:16
Prima: dev@geode.apache.org
Predmet: Re: Question about INDEX_THRESHOLD_SIZE
Mario,
There are few changes happened around this area as part of GEODE-9632 fix;
can you please revert that change and see if the query works both with and
without index.
Looking at the code; it seems to restrict the number index look up that
needs to be performed; certain latency/throughput sensitive queries that or not
expecting exact result may use this (my guess) but by default it should not be
resulting in unexpected results.
-Anil.
On 3/10/22, 6:50 AM, "Mario Kevo" wrote:
Hi geode-dev,
Some time ago I was working on allowing INDEX_THRESHOLD_SIZE System
property to override CompiledValue.RESULT_LIMIT.
After this change, adding this attribute will take into a count if you
set it.
But I need some clarification of this INDEX_THRESHOLD_SIZE attribute.
Why is this set by default to 100?
The main problem with this attribute is that if you want to get the
correct result, you need to know how many entries will be in the region while
starting servers and set it on that value or higher. Sometimes it is too hard
to know how many entries will be in the region, so maybe better will be to set
it by default to some higher number, something like Integer.MAX_VALUE.
Where this attribute is used?
It is used to get index results while doing queries.
What is the problem?
If we have INDEX_THRESHOLD_SIZE set to 500, and we have 1k entries it
can happen that while doing a query it will get only 500 entries and where
clause cannot be fulfilled and we got no results.
Let's see it by an example!
We have only one entry that matches the condition from the query,
INDEX_THRESHOLD_SIZE set to
Re: Question about INDEX_THRESHOLD_SIZE
Mario,
There is similar test/example added by you in QueryWithRangeIndexDUnitTest.
testQueryWithWildcardAndIndexOnAttributeFromHashMap()
When I run that test (on develop); I see the results as expected:
*
Command result for :
Result : true
Limit : 100
Rows: 1
Query Trace : Query Executed in 85.1964 ms; indexesUsed(1):IdIndex(Results:
1)
Are you running your test with any additional change like as you are saying :
>> I was working on allowing INDEX_THRESHOLD_SIZE System property to override
>> CompiledValue.RESULT_LIMIT.
If so , you need to look at the change and see why its impacting that way.
If not, please let me know what change can be made in that test to reproduce
the issue you are seeing; that will help to debug/analyze the issue.
-Anil.
On 3/11/22, 12:18 AM, "Mario Kevo" wrote:
Hi,
It works without an index but it doesn't work with an index.
When I revert changes, it takes INDEX_THRESHOLD_SIZE default value(100).
And if the entry that matches the condition is not in that resultset it will
not be printed.
Without index:
gfsh>query --query="SELECT e.key, e.value from
/example-region.entrySet e where e.value.positions['SUN'] like 'someth%'"
Result : true
Limit : 100
Rows: 1
Query Trace : Query Executed in 11.502283 ms; indexesUsed(0)
key | value
--- |
300 |
{"ID":300,"indexKey":0,"pkid":"300","shortID":null,"position1":{"mktValue":1945.0,"secId":"ORCL","secIdIndexed":"ORCL","secType":null,"sharesOutstanding":1944000.0,"underlyer":null,"pid":1944,"portfolioId":300,..
With index:
gfsh>query --query="SELECT e.key, e.value from
/example-region.entrySet e where e.value.positions['SUN'] like 'someth%'"
Result : true
Limit : 100
Rows: 0
Query Trace : Query Executed in 8.784831 ms; indexesUsed(1):index1(Results:
100)
BR,
Mario
Šalje: Anilkumar Gingade
Poslano: 10. ožujka 2022. 23:16
Prima: dev@geode.apache.org
Predmet: Re: Question about INDEX_THRESHOLD_SIZE
Mario,
There are few changes happened around this area as part of GEODE-9632 fix;
can you please revert that change and see if the query works both with and
without index.
Looking at the code; it seems to restrict the number index look up that
needs to be performed; certain latency/throughput sensitive queries that or not
expecting exact result may use this (my guess) but by default it should not be
resulting in unexpected results.
-Anil.
On 3/10/22, 6:50 AM, "Mario Kevo" wrote:
Hi geode-dev,
Some time ago I was working on allowing INDEX_THRESHOLD_SIZE System
property to override CompiledValue.RESULT_LIMIT.
After this change, adding this attribute will take into a count if you
set it.
But I need some clarification of this INDEX_THRESHOLD_SIZE attribute.
Why is this set by default to 100?
The main problem with this attribute is that if you want to get the
correct result, you need to know how many entries will be in the region while
starting servers and set it on that value or higher. Sometimes it is too hard
to know how many entries will be in the region, so maybe better will be to set
it by default to some higher number, something like Integer.MAX_VALUE.
Where this attribute is used?
It is used to get index results while doing queries.
What is the problem?
If we have INDEX_THRESHOLD_SIZE set to 500, and we have 1k entries it
can happen that while doing a query it will get only 500 entries and where
clause cannot be fulfilled and we got no results.
Let's see it by an example!
We have only one entry that matches the condition from the query,
INDEX_THRESHOLD_SIZE set to 500, and 1k entries in the region.
If we run the query without an index we got the result.
gfsh>query --query="SELECT e.key, e.value from
/example-region.entrySet e where e.value.positions['SUN'] like 'someth%'"
Result : true
Limit : 100
Rows: 1
Query Trace : Query Executed in 10.750238 ms; indexesUsed(0)
key | value
--- |
700 |
{"ID":700,"indexKey":0,"pkid":"700","shortID":null,"position1":{"mktValue":1945.0,"secId":"ORCL","secIdIndexed":"ORCL","secType":null,"sharesOutstanding":1944000.0,"underlyer":null,"pid":1944,"portfolioId":700,..
If we create an index and then run again this query there is
Odg: Question about INDEX_THRESHOLD_SIZE
Hi,
It works without an index but it doesn't work with an index.
When I revert changes, it takes INDEX_THRESHOLD_SIZE default value(100). And if
the entry that matches the condition is not in that resultset it will not be
printed.
Without index:
gfsh>query --query="SELECT e.key, e.value from /example-region.entrySet
e where e.value.positions['SUN'] like 'someth%'"
Result : true
Limit : 100
Rows: 1
Query Trace : Query Executed in 11.502283 ms; indexesUsed(0)
key | value
--- |
300 |
{"ID":300,"indexKey":0,"pkid":"300","shortID":null,"position1":{"mktValue":1945.0,"secId":"ORCL","secIdIndexed":"ORCL","secType":null,"sharesOutstanding":1944000.0,"underlyer":null,"pid":1944,"portfolioId":300,..
With index:
gfsh>query --query="SELECT e.key, e.value from /example-region.entrySet
e where e.value.positions['SUN'] like 'someth%'"
Result : true
Limit : 100
Rows: 0
Query Trace : Query Executed in 8.784831 ms; indexesUsed(1):index1(Results: 100)
BR,
Mario
Šalje: Anilkumar Gingade
Poslano: 10. ožujka 2022. 23:16
Prima: dev@geode.apache.org
Predmet: Re: Question about INDEX_THRESHOLD_SIZE
Mario,
There are few changes happened around this area as part of GEODE-9632 fix; can
you please revert that change and see if the query works both with and without
index.
Looking at the code; it seems to restrict the number index look up that needs
to be performed; certain latency/throughput sensitive queries that or not
expecting exact result may use this (my guess) but by default it should not be
resulting in unexpected results.
-Anil.
On 3/10/22, 6:50 AM, "Mario Kevo" wrote:
Hi geode-dev,
Some time ago I was working on allowing INDEX_THRESHOLD_SIZE System
property to override CompiledValue.RESULT_LIMIT.
After this change, adding this attribute will take into a count if you set
it.
But I need some clarification of this INDEX_THRESHOLD_SIZE attribute. Why
is this set by default to 100?
The main problem with this attribute is that if you want to get the correct
result, you need to know how many entries will be in the region while starting
servers and set it on that value or higher. Sometimes it is too hard to know
how many entries will be in the region, so maybe better will be to set it by
default to some higher number, something like Integer.MAX_VALUE.
Where this attribute is used?
It is used to get index results while doing queries.
What is the problem?
If we have INDEX_THRESHOLD_SIZE set to 500, and we have 1k entries it can
happen that while doing a query it will get only 500 entries and where clause
cannot be fulfilled and we got no results.
Let's see it by an example!
We have only one entry that matches the condition from the query,
INDEX_THRESHOLD_SIZE set to 500, and 1k entries in the region.
If we run the query without an index we got the result.
gfsh>query --query="SELECT e.key, e.value from
/example-region.entrySet e where e.value.positions['SUN'] like 'someth%'"
Result : true
Limit : 100
Rows: 1
Query Trace : Query Executed in 10.750238 ms; indexesUsed(0)
key | value
--- |
700 |
{"ID":700,"indexKey":0,"pkid":"700","shortID":null,"position1":{"mktValue":1945.0,"secId":"ORCL","secIdIndexed":"ORCL","secType":null,"sharesOutstanding":1944000.0,"underlyer":null,"pid":1944,"portfolioId":700,..
If we create an index and then run again this query there is no result.
gfsh>query --query="SELECT e.key, e.value from
/example-region.entrySet e where e.value.positions['SUN'] like 'someth%'"
Result : true
Limit : 100
Rows: 0
Query Trace : Query Executed in 22.079016 ms;
indexesUsed(1):index1(Results: 500)
This happened because we have no luck getting that entry that matches the
condition in the intermediate results for the index.
So the questions are:
What if more entries enter the region that will make the index return more
entries than this threshold sets? Then we're again in jeopardy that the query
condition will not match.
Why is this attribute set by default to 100?
Can we change it to the Integer.MAX_VALUE by default to be sure that we
have the correct result? What are the consequences?
BR,
Mario
