[ https://issues.apache.org/jira/browse/HDFS-6782?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Yi Liu updated HDFS-6782: ------------------------- Status: Patch Available (was: Open) > Improve FS editlog logSync > -------------------------- > > Key: HDFS-6782 > URL: https://issues.apache.org/jira/browse/HDFS-6782 > Project: Hadoop HDFS > Issue Type: Improvement > Components: namenode > Affects Versions: 3.0.0 > Reporter: Yi Liu > Assignee: Yi Liu > Attachments: HDFS-6782.001.patch > > > In NN, it uses a double buffer (bufCurrent, bufReady) for log sync, > bufCurrent it to buffer new coming edit ops and bufReady is for flushing. > This's efficient. When flush is ongoing, and bufCurrent is full, NN goes to > force log sync, and all new Ops are blocked (since force log sync is > protected by FSNameSystem write lock). After the flush finished, the new Ops > are still blocked, but actually at this time, bufCurrent is free and Ops can > go ahead and write to the buffer. The following diagram shows the detail. > This JIRA is for this improvement. Thanks [~umamaheswararao] for confirming > this issue. > {code} > edit1(txid1) ------ write to bufCurrent -------- logSync --------- (swap > buffer)flushing ------- > edit2(txid2) ------ write to bufCurrent -------- logSync --------- waiting > ------- > edit3(txid3) ------ write to bufCurrent -------- logSync --------- waiting > ------- > edit4(txid4) ------ write to bufCurrent -------- logSync --------- waiting > ------- > edit5(txid5) ------ write to bufCurrent --full-- force sync --------- waiting > ------- > edit6(txid6) ------ blocked > ... > editn(txidn) ------ blocked > {code} > After the flush, it becomes > {code} > edit1(txid1) ------ write to bufCurrent -------- logSync --------- finished > -------- > edit2(txid2) ------ write to bufCurrent -------- logSync --------- flushing > ------- > edit3(txid3) ------ write to bufCurrent -------- logSync --------- waiting > ------- > edit4(txid4) ------ write to bufCurrent -------- logSync --------- waiting > ------- > edit5(txid5) ------ write to bufCurrent --full-- force sync --------- waiting > ------- > edit6(txid6) ------ blocked > ... > editn(txidn) ------ blocked > {code} > After edit1 finished, bufCurrent is free, and the thread which flushes txid2 > will also flushes txid3-5, so we should return from the force sync of edit5 > and FSNamesystem write lock will be freed (Don't worry that edit5 Op will > return, since there will be a normal logSync after the force logSync and > there will wait for sync finished). This is the idea of this JIRA. -- This message was sent by Atlassian JIRA (v6.2#6252)