I don’t see where I can pass the UPDATE_LENGTH flag to hflush. Should it be there? David
Best Regards, David R Robison Senior Systems Engineer [cid:image004.png@01D19182.F24CA3E0] From: Harsh J [mailto:ha...@cloudera.com] Sent: Wednesday, August 9, 2017 3:01 PM To: David Robison <david.robi...@psgglobal.net>; user@hadoop.apache.org Subject: Re: Forcing a file to update its length I don't think it'd be safe for a reader to force an update of length at the replica locations directly. Only the writer would be perfectly aware of the DNs in use for the replicas and their states, and the precise count of bytes entirely flushed out of the local buffer. Thereby only the writer is in a confident position to ask the NameNode to update the visible length during a hflush/hsync. Perhaps your app can perform the update length at the more "occassional" lengths of time (or other such stimuli), instead of at every write? P.s. Unless you need strict persistence to disk, you may also choose to use hflush with the UPDATE_LENGTH flag. On Wed, 9 Aug 2017 at 18:07 David Robison <david.robi...@psgglobal.net<mailto:david.robi...@psgglobal.net>> wrote: I understand that, when writing to a file, I can force it to update its length on the namenode by using the following command: ((DFSOutputStream) imageWriter.getWrappedStream()).hsync(EnumSet.of(SyncFlag.UPDATE_LENGTH)); Is there a way to force the update without having to open a DFSOutputStream? Can I do this from the FileSystem class or some other Java class? The reason for this is that I am mostly writing to HDFS and only occasionally reading. However, when I go to read, I am most often reading the most recent data written (reading the end of the file not the beginning). If I could force the length update at the time of reading that would save time by not having to make sure I update the length every time I write to the file (which is about once per second). Thanks, David David R Robison Senior Systems Engineer O. +1 512 247 3700<tel:0512%20247%203700> M. +1 757 286 0022<tel:+1%20757-286-0022> david.robi...@psgglobal.net<mailto:david.robi...@psgglobal.net> www.psgglobal.net<http://www.psgglobal.net/> Prometheus Security Group Global, Inc. 3019 Alvin Devane Boulevard Building 4, Suite 450 Austin, TX 78741
