Franklinsam Paul created ATLAS-4919:
---------------------------------------
Summary: Need option to read records without traversing from
beginning
Key: ATLAS-4919
URL: https://issues.apache.org/jira/browse/ATLAS-4919
Project: Atlas
Issue Type: Improvement
Reporter: Franklinsam Paul
When we read records , with higher offset the more time it takes, it shows
that the search traverse from beginning for each api, check the example below.
{code:java}
Example:
1.
https://ATLAS_SERVER_FQDN:31443/api/atlas/v2/search/basic?excludeDeletedEntities=true&limit=1000&offset=0&typeName=hdfs_path";
- takes just 6 seconds
2.
https://ATLAS_SERVER_FQDN:31443/api/atlas/v2/search/basic?excludeDeletedEntities=true&limit=1000&offset=1000000&typeName=hdfs_path";
- takes about 40 second
3.
https://ATLAS_SERVER_FQDN:31443/api/atlas/v2/search/basic?excludeDeletedEntities=true&limit=1000&offset=4000000&typeName=hdfs_path";
- taken about 2 minute
4.
https://ATLAS_SERVER_FQDN:31443/api/atlas/v2/search/basic?excludeDeletedEntities=true&limit=1000&offset=4000000&typeName=hdfs_path";
- taken about 3 minutes
5.
https://ATLAS_SERVER_FQDN:31443/api/atlas/v2/search/basic?excludeDeletedEntities=true&limit=1000&offset=10000000&typeName=hdfs_path";
0 taken more time and failed with socket timeout ref:
https://issues.apache.org/jira/browse/ATLAS-4918
{code}
- Solr has "cursorMark" that will print the VALUE as NextCursorMark, which can
be used in the consecutive queries to directly read from the marker. REF:
[https://solr.apache.org/guide/6_6/pagination-of-results.html#using-cursors]
- I also see (“marker” : “...“,) option in atlas here
[https://atlas.apache.org/api/v2/json_SearchParameters.html] but it's unclear
how/whether this can be used in api.
ASK:
if Marker above is already there, provide it's usage in api, if not need a
similar implementation to avoid traversing through records from beginning to
avoid timeout and slowness
[~dishatalreja] [~jayendrap] [~amekala2514]
{code:java}
{code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
