You are right, sorry I hadn't checked before stating that. The Jira at https://issues.apache.org/jira/browse/HDFS-4213 has some history around this.
Looking over your question again, you may not really require the update length flag (a forced NameNode RPC) at every writer and also the hsync altogether if all you want is a polling client to read the latest changes. What you need is a periodic or continual hflush from the writer, and a reinitiated client (new FileStatus query every time the client initiates, along with prior offset information) such as is done in the FsShell's Tail program example: https://github.com/apache/hadoop/blob/release-2.7.0/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/shell/Tail.java#L74-L80 Here's a demo: http://imgur.com/a/L90po and the code snippet used behind the writer: https://gist.github.com/QwertyManiac/76743cf6df52607a8441049928e046c6 On Thu, Aug 10, 2017, 1:15 AM David Robison <david.robi...@psgglobal.net> wrote: > I don’t see where I can pass the UPDATE_LENGTH flag to hflush. Should it > be there? David > > > > Best Regards, > > > > *David R Robison* > > *Senior Systems Engineer* > > [image: cid:image004.png@01D19182.F24CA3E0] > > > > *From:* Harsh J [mailto:ha...@cloudera.com] > *Sent:* Wednesday, August 9, 2017 3:01 PM > *To:* David Robison <david.robi...@psgglobal.net>; user@hadoop.apache.org > *Subject:* Re: Forcing a file to update its length > > > > I don't think it'd be safe for a reader to force an update of length at > the replica locations directly. Only the writer would be perfectly aware of > the DNs in use for the replicas and their states, and the precise count of > bytes entirely flushed out of the local buffer. Thereby only the writer is > in a confident position to ask the NameNode to update the visible length > during a hflush/hsync. > > > > Perhaps your app can perform the update length at the more "occassional" > lengths of time (or other such stimuli), instead of at every write? > > > > P.s. Unless you need strict persistence to disk, you may also choose to > use hflush with the UPDATE_LENGTH flag. > > On Wed, 9 Aug 2017 at 18:07 David Robison <david.robi...@psgglobal.net> > wrote: > > I understand that, when writing to a file, I can force it to update its > length on the namenode by using the following command: > > > > ((DFSOutputStream) > imageWriter.getWrappedStream()).hsync(EnumSet.of(SyncFlag.UPDATE_LENGTH)); > > > > Is there a way to force the update without having to open a > DFSOutputStream? Can I do this from the FileSystem class or some other Java > class? The reason for this is that I am mostly writing to HDFS and only > occasionally reading. However, when I go to read, I am most often reading > the most recent data written (reading the end of the file not the > beginning). If I could force the length update at the time of reading that > would save time by not having to make sure I update the length every time I > write to the file (which is about once per second). > > > > Thanks, David > > > > *David R Robison* > > *Senior Systems Engineer* > > O. +1 512 247 3700 <0512%20247%203700> > > M. +1 757 286 0022 <+1%20757-286-0022> > > david.robi...@psgglobal.net > > *www.psgglobal.net <http://www.psgglobal.net/>* > > [image: image001.png] > > *Prometheus Security Group Global, Inc.* > > 3019 Alvin Devane Boulevard > > Building 4, Suite 450 > > Austin, TX 78741 > > [image: image001.png] > > > >
--------------------------------------------------------------------- To unsubscribe, e-mail: user-unsubscr...@hadoop.apache.org For additional commands, e-mail: user-h...@hadoop.apache.org
