[ https://issues.apache.org/jira/browse/HADOOP-14154?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16578007#comment-16578007 ]
Gabor Bota edited comment on HADOOP-14154 at 8/13/18 9:28 AM: -------------------------------------------------------------- I've uploaded the following: * perf-eval-v2.diff: contains an additional change on top of v1 that will cause an update on the metadata dir listing. This will not happen in perf-eval-v1.diff so the measurements will not show the decreased time for directory listing queries. * run-dir-perf-itest-v2.sh: corrected the v1 script, so we run a test without authoritative directory listings in LocalMetadataStore * all-logs-v2.txt: measurement results with the v2 diff and itest script. In the results you can see a decreased time in directory listings when {{Using metadata store DynamoDBMetadataStore\{region=us-west-2, tableName=gaborbota-oregon\}, authoritative=true}} is used. The first query takes longer, but then the cached entries can be reached from dynamo. was (Author: gabor.bota): I've uploaded the following: * perf-eval-v2.diff: contains an additional change on top of v1 that will cause an update on the metadata dir listing. This will not happen in perf-eval-v1.diff so the measurements will not show the decreased time for directory listing queries. * run-dir-perf-itest-v2.sh: corrected the v1 script, so we run a test without authoritative directory listings in LocalMetadataStore * all-logs-v2.txt: measurement results with the v2 diff and itest script. In the results you can see a decreased time in directory listings when {{Using metadata store DynamoDBMetadataStore{region=us-west-2, tableName=gaborbota-oregon}, authoritative=true}} is used. The first query takes longer, but then the cached entries can be reached from dynamo. > Persist isAuthoritative bit in DynamoDBMetaStore (authoritative mode support) > ----------------------------------------------------------------------------- > > Key: HADOOP-14154 > URL: https://issues.apache.org/jira/browse/HADOOP-14154 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 > Affects Versions: 3.0.0-beta1 > Reporter: Rajesh Balamohan > Assignee: Gabor Bota > Priority: Minor > Attachments: HADOOP-14154-HADOOP-13345.001.patch, > HADOOP-14154-HADOOP-13345.002.patch, HADOOP-14154-spec-001.pdf, > HADOOP-14154-spec-002.pdf, HADOOP-14154.001.patch, HADOOP-14154.002.patch, > HADOOP-14154.003.patch, HADOOP-14154.004.patch, all-logs-v2.txt, > perf-eval-v1.diff, perf-eval-v2.diff, run-dir-perf-itest-v2.sh, > run-dir-perf-itest.sh > > > Add support for "authoritative mode" for DynamoDBMetadataStore. > The missing feature is to persist the bit set in > {{DirListingMetadata.isAuthoritative}}. > This topic has been super confusing for folks so I will also file a > documentation Jira to explain the design better. > We may want to also rename the DirListingMetadata.isAuthoritative field to > .isFullListing to eliminate the multiple uses and meanings of the word > "authoritative". > -- This message was sent by Atlassian JIRA (v7.6.3#76005) --------------------------------------------------------------------- To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org