On Wed, Dec 4, 2013 at 3:56 PM, Stack <st...@duboce.net> wrote: > On Thu, Dec 5, 2013 at 7:46 AM, Enis Söztutar <enis....@gmail.com> wrote: > > > I did not know that we were reopening the log file for tailing. From what > > Nicolas talks about in > > https://issues.apache.org/jira/browse/HDFS-3219 it seems that the > visible > > length is not > > updated for the open stream which is a shame. However in the append > design, > > the primary can > > send the committed length ( minimum of RAs) to replicas, so that replicas > > can make that data visible > > to the client. > > > > > Nit: We could but then we'd have two 'tailing' workarounds in hbase; the > file reopen when we run of the end used by replication and then the > suggestion made here. > > > I've written a tailing experiment and I can say with the newish and poorly named FS.isFileClosed() call, we can get correctness. If and when better tailing facilities are added hdfs we can take advanage of those apis instead of the non-optimal version of today.
We'd have one version of this if we proceed with the work. > It would be good if we can implement this in hdfs. > > > > > Yes. This would be better all around. > > > > > About the minimum work agreed that we should not merge this in unless > there > > are real benefits > > demonstrated. That is why we proposed to do the work for phase 1 in a > > branch, and at the end of > > that, we are hoping we can have something useful and working (but without > > wal tailing and async > > wal replication), and we will have a more detailed plan for the remaining > > steps. We would love to > > hear more feedback of how to test / stabilize the feature at the merge > > discussions. > > > > Sounds good E. > > St.Ack > -- // Jonathan Hsieh (shay) // Software Engineer, Cloudera // j...@cloudera.com
