Re: Hadoop and Ceph client/mds view of modification time

2012-11-27 Thread Sage Weil
On Tue, 27 Nov 2012, Sam Lang wrote: Hi Noah, I was able to reproduce your issue with a similar test using the fuse client and the clock_offset option for the mds. This is what I see happening: clientA's clock is a few seconds behind the mds clock clientA creates the file - the

Re: Hadoop and Ceph client/mds view of modification time

2012-11-27 Thread Gregory Farnum
On Tuesday, November 27, 2012 at 8:45 AM, Sam Lang wrote: Hi Noah, I was able to reproduce your issue with a similar test using the fuse client and the clock_offset option for the mds. This is what I see happening: clientA's clock is a few seconds behind the mds clock clientA

Re: Hadoop and Ceph client/mds view of modification time

2012-11-27 Thread David Zafman
On Nov 27, 2012, at 9:03 AM, Sage Weil s...@inktank.com wrote: On Tue, 27 Nov 2012, Sam Lang wrote: 3. When a client acquires the cap for a file, have the mds provide its current time as well. As the client updates the mtime, it uses the timestamp provided by the mds and the time

Re: Hadoop and Ceph client/mds view of modification time

2012-11-27 Thread Sam Lang
On 11/27/2012 11:07 AM, Gregory Farnum wrote: On Tuesday, November 27, 2012 at 8:45 AM, Sam Lang wrote: Hi Noah, I was able to reproduce your issue with a similar test using the fuse client and the clock_offset option for the mds. This is what I see happening: clientA's clock is a few

Re: Hadoop and Ceph client/mds view of modification time

2012-11-27 Thread Sam Lang
On 11/27/2012 11:03 AM, Sage Weil wrote: On Tue, 27 Nov 2012, Sam Lang wrote: Hi Noah, I was able to reproduce your issue with a similar test using the fuse client and the clock_offset option for the mds. This is what I see happening: clientA's clock is a few seconds behind the mds clock

Re: Hadoop and Ceph client/mds view of modification time

2012-11-27 Thread Sage Weil
On Tue, 27 Nov 2012, David Zafman wrote: On Nov 27, 2012, at 9:03 AM, Sage Weil s...@inktank.com wrote: On Tue, 27 Nov 2012, Sam Lang wrote: 3. When a client acquires the cap for a file, have the mds provide its current time as well. As the client updates the mtime, it uses the

Re: Hadoop and Ceph client/mds view of modification time

2012-11-27 Thread Sam Lang
On 11/27/2012 12:01 PM, Sage Weil wrote: On Tue, 27 Nov 2012, David Zafman wrote: On Nov 27, 2012, at 9:03 AM, Sage Weil s...@inktank.com wrote: On Tue, 27 Nov 2012, Sam Lang wrote: 3. When a client acquires the cap for a file, have the mds provide its current time as well. As the client

Re: Hadoop and Ceph client/mds view of modification time

2012-11-27 Thread David Zafman
On Nov 27, 2012, at 11:05 AM, Sam Lang sam.l...@inktank.com wrote: On 11/27/2012 12:01 PM, Sage Weil wrote: On Tue, 27 Nov 2012, David Zafman wrote: On Nov 27, 2012, at 9:03 AM, Sage Weil s...@inktank.com wrote: On Tue, 27 Nov 2012, Sam Lang wrote: 3. When a client acquires the cap

Re: Hadoop and Ceph client/mds view of modification time

2012-11-27 Thread Sage Weil
On 11/27/2012 12:01 PM, Sage Weil wrote: On Tue, 27 Nov 2012, David Zafman wrote: On Nov 27, 2012, at 9:03 AM, Sage Weil s...@inktank.com wrote: On Tue, 27 Nov 2012, Sam Lang wrote: 3. When a client acquires the cap for a file, have the mds provide its current

Re: Hadoop and Ceph client/mds view of modification time

2012-11-27 Thread Sam Lang
On 11/27/2012 01:38 PM, David Zafman wrote: On Nov 27, 2012, at 11:05 AM, Sam Lang sam.l...@inktank.com wrote: On 11/27/2012 12:01 PM, Sage Weil wrote: On Tue, 27 Nov 2012, David Zafman wrote: On Nov 27, 2012, at 9:03 AM, Sage Weil s...@inktank.com wrote: On Tue, 27 Nov 2012, Sam Lang

Re: Hadoop and Ceph client/mds view of modification time

2012-11-27 Thread David Zafman
On Nov 27, 2012, at 1:14 PM, Sam Lang sam.l...@inktank.com wrote: On 11/27/2012 01:38 PM, David Zafman wrote: On Nov 27, 2012, at 11:05 AM, Sam Lang sam.l...@inktank.com wrote: On 11/27/2012 12:01 PM, Sage Weil wrote: On Tue, 27 Nov 2012, David Zafman wrote: On Nov 27, 2012, at 9:03

Re: Hadoop and Ceph client/mds view of modification time

2012-11-21 Thread Noah Watkins
(Sorry for the dupe message. vger rejected due to HTML). Thanks, I'll try this patch this morning. Client B should perform a single stat after a notification from Client A. But, won't Sage's patch still be required, since Client A needs the MDS time to pass to Client B? On Tue, Nov 20, 2012 at

Hadoop and Ceph client/mds view of modification time

2012-11-20 Thread Noah Watkins
This is a description of the clock synchronization issue we are facing in Hadoop: Components of Hadoop use mtime as a versioning mechanism. Here is an example where Client B tests the expected 'version' of a file created by Client A: Client A: create file, write data into file. Client A:

Re: Hadoop and Ceph client/mds view of modification time

2012-11-20 Thread Sam Lang
On 11/20/2012 01:44 PM, Noah Watkins wrote: This is a description of the clock synchronization issue we are facing in Hadoop: Components of Hadoop use mtime as a versioning mechanism. Here is an example where Client B tests the expected 'version' of a file created by Client A: Client A: