----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/73292/ -----------------------------------------------------------
(Updated May 10, 2021, 4:56 p.m.) Review request for atlas, Ashutosh Mestry, Jayendra Parab, Madhan Neethiraj, and Sarath Subramanian. Bugs: ATLAS-4254 https://issues.apache.org/jira/browse/ATLAS-4254 Repository: atlas Description (updated) ------- **Current Working of Pagination** - When user specifies page information i.e offset and limit in the request, except some condition, offset is reset to 0 - Query is then executed from 0 offset and limit - Filter out the results obtained from query - Generate result set by skipping the results from 0 to the given offset **New Working Structure** - New Parameter 'marker' (String) has been introduced in SearchParameters which will have encrypted string as value. - The marker should be set as '*' for the start of the page - The response of above request (AtlasSearchResult) will have 'nextMarker' field which will have encrypted String which is to be passed as value of 'marker' in the subsequent request - if marker is set then offset should not be greater than 0 **Perfomance details** Took usecase of ranger tagsync: - On searching tagged entities starting with offset: 0 and limit: 1000 returned 10000 entities in ~ 21s - On searching tagged entities starting with marker: * and limit: 1000 returned 10000 entities in ~ 10s (both the number are taken on the fresh start of the server, so as to avoid caching affecting the numbers) - Removed 'atlas.graph.index.search.max-result-set-size=150' property from atlas-application.properties - Around 51% improvement - Improvement is seen as the page size increases. Diffs ----- intg/src/main/java/org/apache/atlas/model/discovery/AtlasSearchResult.java e1c550ef7 intg/src/main/java/org/apache/atlas/model/discovery/SearchParameters.java 9d2cd4ffc repository/src/main/java/org/apache/atlas/discovery/ClassificationSearchProcessor.java 647ff9ca0 repository/src/main/java/org/apache/atlas/discovery/EntityDiscoveryService.java a3ab6e38c repository/src/main/java/org/apache/atlas/discovery/EntitySearchProcessor.java 5dcff3b16 repository/src/main/java/org/apache/atlas/discovery/FreeTextSearchProcessor.java 92152fff5 repository/src/main/java/org/apache/atlas/discovery/FullTextSearchProcessor.java b37d93a83 repository/src/main/java/org/apache/atlas/discovery/SearchContext.java aa49121f6 repository/src/main/java/org/apache/atlas/discovery/SearchProcessor.java f9832c3b2 repository/src/main/java/org/apache/atlas/discovery/TermSearchProcessor.java 45a815886 repository/src/test/java/org/apache/atlas/discovery/AtlasDiscoveryServiceTest.java 027827af2 repository/src/test/java/org/apache/atlas/discovery/ClassificationSearchProcessorTest.java e1ebbfc70 webapp/src/main/java/org/apache/atlas/web/rest/DiscoveryREST.java 4c7b6228d Diff: https://reviews.apache.org/r/73292/diff/4/ Testing ------- MAnual testing unit testcases Thanks, Pinal Shah