anuragaw opened a new pull request #3457: Fix bug in counting items for search query URL: https://github.com/apache/cloudstack/pull/3457 The implementation of GenericBaseDao#searchAndCount() can result in incorrect count. This happens because the implementation of the GenericBaseDao#getCount method ingores the groupBy components of the search queries completely. This means the count returned will always be larger than the actual count when not considering pagination. The change to ignore the group by components was brought in b0ce8fd which also fails to explain the rationale. This was probably because the groupby would be executed after select count(*) has finished. Further investigation of the getCount usage reveal they are always accompanied by search queries which include groupBy components via GenericBaseDao#searchIncludingRemoved method. Current code diff between search and count methods is as follows - ```diff --git a/framework/db/src/main/java/com/cloud/uddtils/db/GenericDaoBase.java b/framework/db/src/main/java/com/cloud/utils/db/GenericDaoBase.java @@ -371,7 +371,7 @@ public abstract class GenericDaoBase<T, ID extends Serializable> extends Compone clause = null; } - final StringBuilder str = createPartialSelectSql(sc, clause != null, enableQueryCache); + final StringBuilder str = createCountSelect(sc, clause != null); @@ -384,19 +384,12 @@ public abstract class GenericDaoBase<T, ID extends Serializable> extends Compone } } - List<Object> groupByValues = addGroupBy(str, sc); - addFilter(str, filter); - + // we have to disable group by in getting count, since count for groupBy clause will be different. + //List<Object> groupByValues = addGroupBy(str, sc); final TransactionLegacy txn = TransactionLegacy.currentTxn(); - if (lock != null) { - assert (txn.dbTxnStarted() == true) : "As nice as I can here now....how do you lock when there's no DB transaction? Review your db 101 course from college."; - str.append(lock ? FOR_UPDATE_CLAUSE : SHARE_MODE_CLAUSE); - } - @@ -410,20 +403,19 @@ public abstract class GenericDaoBase<T, ID extends Serializable> extends Compone + /* if (groupByValues != null) { for (Object value : groupByValues) { pstmt.setObject(i++, value); } } + */ - if (s_logger.isDebugEnabled() && lock != null) { - txn.registerLock(pstmt.toString()); - } final ResultSet rs = pstmt.executeQuery(); while (rs.next()) { - result.add(toEntityBean(rs, cache)); + return rs.getInt(1); } - return result; + return 0; -- 2.17.1``` The fix is to update the way we setup query for counting with group by params. We need to do the below query to get accurate count - `SELECT COUNT(DISTINCT GROUP_ROW1...) FROM.......` ## Description <!--- Describe your changes in detail --> <!-- For new features, provide link to FS, dev ML discussion etc. --> <!-- In case of bug fix, the expected and actual behaviours, steps to reproduce. --> <!-- When "Fixes: #<id>" is specified, the issue/PR will automatically be closed when this PR gets merged --> <!-- For addressing multiple issues/PRs, use multiple "Fixes: #<id>" --> <!-- Fixes: # --> ## Types of changes <!--- What types of changes does your code introduce? Put an `x` in all the boxes that apply: --> - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] New feature (non-breaking change which adds functionality) - [x] Bug fix (non-breaking change which fixes an issue) - [ ] Enhancement (improves an existing feature and functionality) - [ ] Cleanup (Code refactoring and cleanup, that may add test cases) ## Screenshots (if appropriate): ## How Has This Been Tested? <!-- Please describe in detail how you tested your changes. --> <!-- Include details of your testing environment, and the tests you ran to --> <!-- see how your change affects other areas of the code, etc. --> Example API call - `http://localhost:8080/client/api?command=listPublicIpAddresses&response=json&associatednetworkid=72954e98-7fa5-41a0-85f1-97b719f8ff0a&forloadbalancing=true&listall=true&_=1561816856393` Here is it's response - ``` { "listpublicipaddressesresponse": { "count": 2, "publicipaddress": [ { "id": "b65b9d02-9135-4f3a-89c7-c6d1948ad3ce", "ipaddress": "172.20.20.107", "allocated": "2019-06-29T19:21:01+0530", "zoneid": "c3476f05-69ea-4b4a-a38c-c7436188a727", "zonename": "KVM-advzone1", "issourcenat": false, "account": "admin", "domainid": "96fb4917-97e5-11e9-8e69-34e12d5f623e", "domain": "ROOT", "forvirtualnetwork": true, "vlanid": "8f7413bc-3d85-423e-b685-60910b91a819", "vlanname": "vlan://untagged", "isstaticnat": false, "issystem": false, "associatednetworkid": "72954e98-7fa5-41a0-85f1-97b719f8ff0a", "associatednetworkname": "tier1", "networkid": "5a0679dc-1f52-4e40-a7aa-1253848e4886", "state": "Allocated", "physicalnetworkid": "ac5912b6-6461-4d4a-b137-f730f29ef241", "vpcid": "89195bd2-28b1-4afa-b67f-6c8b468ac3f6", "tags": [ ], "isportable": false, "fordisplay": true } ] } } ``` After fix the correct count is returned. <!-- Please read the [CONTRIBUTING](https://github.com/apache/cloudstack/blob/master/CONTRIBUTING.md) document -->
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
