Thanks Colin for the detailed explanation. On Fri, Sep 19, 2014 at 9:38 PM, Colin McCabe <cmcc...@alumni.cmu.edu> wrote: > > On Thu, Sep 18, 2014 at 11:06 AM, Vinayakumar B <vinayakum...@apache.org> wrote: > > bq. I don't know about the merits of this, but I do know that native > > filesystems > > implement this by not raising the EOF exception on the seek() but only on > > the read ... some of the non-HDFS filesystems Hadoop support work this way. > > Pretty much all of them should. POSIX specifies that seeking past the > end of a file is not an error. Reading past the end of the file gives > an EOF, but the seek always succeeds. > > It would be nice if HDFS had this behavior as well. It seems like > this would have to be a 3.0 thing, since it's a potential > incompatibility. > > > I agree with you steve. read only will throw EOF. But when we know that > > file is being written and it can have fresh data, then polling can be done > > by calling available(). later we can continue read or call seek. >
Yes, I too agree that, if we are changing seek() behaviour, then definitely that is a 3.0 thing. > InputStream#available() has a really specific function in Java: > telling you approximately how much data is currently buffered by the > stream. > > As a side note, InputStream#available seems to be one of the most > misunderstood APIs in Java. It's pretty common for people to assume > that it means "how much data is left in the stream" or something like > that. I think I made that mistake at least once when getting started > with Java. I guess the JavaDoc is kind of vague-- it specifies that > available returns "an estimate of the number of bytes that can be read > (or skipped over) from this input stream without blocking." But in > practice, that means how much is buffered (for a file-backed stream, > to pull more bytes from the OS would require a syscall, which is > "blocking." Similarly for network-backed streams.) Yes, InputStream#available() javadoc says its the data which can be read non-blocking, It also says, impls can chose to return total number of bytes available in the stream, which is done in DFSInputStream > In any case, we certainly could create a new API to refresh > inputstream data. I guess the idea would be to check if the last > block we knew about had reached full length-- if so, we would ask the > NameNode for any new block locations. So it would be a DN operation > in most cases, but sometimes a NN operation. Correct, we can anyway have new API to refresh. But if clients uses just InputStream interface, then IMO its better to do this in available() itself. This will be inline with native FileInputStream. If the file is closed, then we can chose to return -1, else if no new data available then can return 0 as its doing now. As you mentioned, refresh can be done only from DNs, and if the block is full, then refresh from NN again. But also needs to think how we can handle this, if the proposed "variable length blocks" comes to HDFS. > Have you looked at https://issues.apache.org/jira/browse/HDFS-6633: > Support reading new data in a being written file until the file is > closed? That patch seems to take the approach of turning reading past > the end of the file into an operation that blocks until there is new > data. (when dfs.client.read.tail-follow is set.) I think I prefer > the idea of a new refresh API, just because it puts more control in > the hands of the user. Just now saw the Jira. Intention of the Jira is same as this discussion. Before seeing this mail, I had raised https://issues.apache.org/jira/browse/HDFS-7099, anyway we can duplicate this. But patch proposed in HDFS-6633, is a blocking call on read(), which may not be feasible for the clients. I feel polling for new data, instead of blocking read call(). Anyway this can be discussed more on the jira itself. > Another thing to consider is how this all interacts with the proposed > HDFS truncate operation (see HDFS-3107). I haven't seen this in detail. I will check it soon. > best, > Colin > > > > > > One simple example use case is tailing a file. > > > > Regards, > > Vinay > > > > On Thu, Sep 18, 2014 at 3:35 PM, Steve Loughran <ste...@hortonworks.com> > > wrote: > > > >> I don't know about the merits of this, but I do know that native > >> filesystems implement this by not raising the EOF exception on the seek() > >> but only on the read ... some of the non-HDFS filesystems Hadoop support > >> work this way. > >> > >> -I haven't ever looked to see what code assumes that it is the seek that > >> fails, not the read. > >> -PositionedReadable had better handle this too, even if it isn't done via a > >> seek()-read()-seek() sequence > >> > >> > >> On 18 September 2014 08:48, Vinayakumar B <vinayakum...@apache.org> wrote: > >> > >> > Hi all, > >> > > >> > Currently *DFSInputStream *doen't allow reading a write-inprogress file, > >> > once all written bytes, by the time of opening an input stream, are read. > >> > > >> > To read further update on the same file, needs to be read by opening > >> > another stream to the same file again. > >> > > >> > Instead how about refreshing length of such open files if the current > >> > position is at earlier EOF. > >> > > >> > May be this could be done in *available() *method, So that clients who > >> > knows that original writer will not close then read can continuously poll > >> > for new data using the same stream? > >> > > >> > PS: This is possible in local disk read using FileInputStream > >> > > >> > Regards, > >> > Vinay > >> > > >> > >> -- > >> CONFIDENTIALITY NOTICE > >> NOTICE: This message is intended for the use of the individual or entity to > >> which it is addressed and may contain information that is confidential, > >> privileged and exempt from disclosure under applicable law. If the reader > >> of this message is not the intended recipient, you are hereby notified that > >> any printing, copying, dissemination, distribution, disclosure or > >> forwarding of this communication is strictly prohibited. If you have > >> received this communication in error, please contact the sender immediately > >> and delete it from your system. Thank You. > >> Regards, Vinay
