Re: Appending and seeking files while writing
Hi. Should be out soon - Tom White is working hard on the release. Note that the > first release, 0.21.0, will be somewhat of a "development quality" release > not recommended for production use. Of course, the way it will become > production-worthy is by less risk-averse people trying it and finding the > bugs :) > > > > Will it work both with Fuse and HDFS API? > > > > I don't know that the Fuse code has been updated to call append. My guess > is that a small patch would be required. > > > > > > > Also, append does *not* add random write. It simply adds the ability to > > > re-open a file and add more data to the end. > > > > > > > > Just to clarify, even with append it won't be possible to: > > 1) Pause writing of new file, skip to any position, and update the data. > > 2) Open existing file, skip to any position and update the data. > > > > Correct, neither of those are allowed. > > Thanks for clarification.
Re: Appending and seeking files while writing
On Mon, Jun 14, 2010 at 4:28 AM, Stas Oskin wrote: > By the way, what about an ability for node to read file which is being > written by another node? > This is allowed, though there are some remaining bugs to be ironed out here. See https://issues.apache.org/jira/browse/HDFS-1057 for example. > Or the file must be written and closed completely, before it becomes > available for other nodes? > > (AFAIK in 0.18.3 the file appeared as 0 size until it was closed). > > Regards. > -- Todd Lipcon Software Engineer, Cloudera
Re: Appending and seeking files while writing
On Mon, Jun 14, 2010 at 4:00 AM, Stas Oskin wrote: > Hi. > > Thanks for clarification. > > Append will be supported fully in 0.21. > > > > > Any ETA for this version? > Should be out soon - Tom White is working hard on the release. Note that the first release, 0.21.0, will be somewhat of a "development quality" release not recommended for production use. Of course, the way it will become production-worthy is by less risk-averse people trying it and finding the bugs :) > Will it work both with Fuse and HDFS API? > > I don't know that the Fuse code has been updated to call append. My guess is that a small patch would be required. > > > Also, append does *not* add random write. It simply adds the ability to > > re-open a file and add more data to the end. > > > > > Just to clarify, even with append it won't be possible to: > 1) Pause writing of new file, skip to any position, and update the data. > 2) Open existing file, skip to any position and update the data. > > Correct, neither of those are allowed. > This will be even with FUSE. > > Is this correct? > > Regards. > -- Todd Lipcon Software Engineer, Cloudera
Re: Appending and seeking files while writing
By the way, what about an ability for node to read file which is being written by another node? Or the file must be written and closed completely, before it becomes available for other nodes? (AFAIK in 0.18.3 the file appeared as 0 size until it was closed). Regards.
Re: Appending and seeking files while writing
Hi. Thanks for clarification. Append will be supported fully in 0.21. > > Any ETA for this version? Will it work both with Fuse and HDFS API? > Also, append does *not* add random write. It simply adds the ability to > re-open a file and add more data to the end. > > Just to clarify, even with append it won't be possible to: 1) Pause writing of new file, skip to any position, and update the data. 2) Open existing file, skip to any position and update the data. This will be even with FUSE. Is this correct? Regards.
Re: Appending and seeking files while writing
On Sun, Jun 13, 2010 at 12:46 AM, Vidur Goyal wrote: > Append is supported in hadoop 0.20 . > > Append will be supported in the 0.20-append branch, which is still in progress. It is NOT supported in vanilla 0.20. You can turn on the config option but it is dangerous and highly discouraged for real use. Append will be supported fully in 0.21. Also, append does *not* add random write. It simply adds the ability to re-open a file and add more data to the end. -Todd > > > Hi. > > > > I think this really depends on the append functionality, any idea whether > > it > > supports such behaviour now? > > > > Regards. > > > > On Fri, Jun 11, 2010 at 10:41 AM, hadooprcoks > > wrote: > > > >> Stas, > >> > >> I also believe that there should be a seek interface on the write path > >> so > >> that the FS API is complete. The FsDataInputStream already support > >> seek() - > >> so should FsDataOutputStream. For File systems, that do not support the > >> seek > >> on the write path, the seek can be a no operation. > >> > >> Could you open a JIRA to track this. I am willing to provide the patch > >> if > >> you do not have the time to do so. > >> > >> thanks > >> hadooprocks > >> > >> > >> On Thu, Jun 10, 2010 at 5:05 AM, Stas Oskin > >> wrote: > >> > >> > Hi. > >> > > >> > Was the append functionality finally added to 0.20.1 version? > >> > > >> > Also, is the ability to seek file being written and write data in > >> other > >> > place also supported? > >> > > >> > Thanks in advance! > >> > > >> > > > > -- > > This message has been scanned for viruses and > > dangerous content by MailScanner, and is > > believed to be clean. > > > > > > > -- > This message has been scanned for viruses and > dangerous content by MailScanner, and is > believed to be clean. > > -- Todd Lipcon Software Engineer, Cloudera
Re: Appending and seeking files while writing
Append is supported in hadoop 0.20 . > Hi. > > I think this really depends on the append functionality, any idea whether > it > supports such behaviour now? > > Regards. > > On Fri, Jun 11, 2010 at 10:41 AM, hadooprcoks > wrote: > >> Stas, >> >> I also believe that there should be a seek interface on the write path >> so >> that the FS API is complete. The FsDataInputStream already support >> seek() - >> so should FsDataOutputStream. For File systems, that do not support the >> seek >> on the write path, the seek can be a no operation. >> >> Could you open a JIRA to track this. I am willing to provide the patch >> if >> you do not have the time to do so. >> >> thanks >> hadooprocks >> >> >> On Thu, Jun 10, 2010 at 5:05 AM, Stas Oskin >> wrote: >> >> > Hi. >> > >> > Was the append functionality finally added to 0.20.1 version? >> > >> > Also, is the ability to seek file being written and write data in >> other >> > place also supported? >> > >> > Thanks in advance! >> > >> > > -- > This message has been scanned for viruses and > dangerous content by MailScanner, and is > believed to be clean. > > -- This message has been scanned for viruses and dangerous content by MailScanner, and is believed to be clean.
Re: Appending and seeking files while writing
Hi. I think this really depends on the append functionality, any idea whether it supports such behaviour now? Regards. On Fri, Jun 11, 2010 at 10:41 AM, hadooprcoks wrote: > Stas, > > I also believe that there should be a seek interface on the write path so > that the FS API is complete. The FsDataInputStream already support seek() - > so should FsDataOutputStream. For File systems, that do not support the > seek > on the write path, the seek can be a no operation. > > Could you open a JIRA to track this. I am willing to provide the patch if > you do not have the time to do so. > > thanks > hadooprocks > > > On Thu, Jun 10, 2010 at 5:05 AM, Stas Oskin wrote: > > > Hi. > > > > Was the append functionality finally added to 0.20.1 version? > > > > Also, is the ability to seek file being written and write data in other > > place also supported? > > > > Thanks in advance! > > >
Re: Appending and seeking files while writing
hadooprocks, As a project requirement I have to do the same and write a seek() operation for FSDataoutputStream. It will be very kind of you to give me some insight on this. I have explored the web about recompiling Hadoop once I change the contents , Can you give me some documents that help me do that too. Thanks, Vidur > Stas, > > I also believe that there should be a seek interface on the write path so > that the FS API is complete. The FsDataInputStream already support seek() > - > so should FsDataOutputStream. For File systems, that do not support the > seek > on the write path, the seek can be a no operation. > > Could you open a JIRA to track this. I am willing to provide the patch if > you do not have the time to do so. > > thanks > hadooprocks > > > On Thu, Jun 10, 2010 at 5:05 AM, Stas Oskin wrote: > >> Hi. >> >> Was the append functionality finally added to 0.20.1 version? >> >> Also, is the ability to seek file being written and write data in other >> place also supported? >> >> Thanks in advance! >> > > -- > This message has been scanned for viruses and > dangerous content by MailScanner, and is > believed to be clean. > > -- This message has been scanned for viruses and dangerous content by MailScanner, and is believed to be clean.
Re: Appending and seeking files while writing
Stas, I also believe that there should be a seek interface on the write path so that the FS API is complete. The FsDataInputStream already support seek() - so should FsDataOutputStream. For File systems, that do not support the seek on the write path, the seek can be a no operation. Could you open a JIRA to track this. I am willing to provide the patch if you do not have the time to do so. thanks hadooprocks On Thu, Jun 10, 2010 at 5:05 AM, Stas Oskin wrote: > Hi. > > Was the append functionality finally added to 0.20.1 version? > > Also, is the ability to seek file being written and write data in other > place also supported? > > Thanks in advance! >
Re: Appending and seeking files while writing
Actually, I noticed FUSE finally being supported in 0.20.1, any idea if this supports RDWR mode for both writing and reading file at same time? Regards. On Thu, Jun 10, 2010 at 3:05 PM, Stas Oskin wrote: > Hi. > > Was the append functionality finally added to 0.20.1 version? > > Also, is the ability to seek file being written and write data in other > place also supported? > > Thanks in advance! >
