[jira] [Updated] (HDFS-744) Support hsync in HDFS

2012-06-04 Thread Tsz Wo (Nicholas), SZE (JIRA)

 [ 
https://issues.apache.org/jira/browse/HDFS-744?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Tsz Wo (Nicholas), SZE updated HDFS-744:


Fix Version/s: (was: 3.0.0)
   2.0.1-alpha

 Support hsync in HDFS
 -

 Key: HDFS-744
 URL: https://issues.apache.org/jira/browse/HDFS-744
 Project: Hadoop HDFS
  Issue Type: New Feature
  Components: data-node, hdfs client
Reporter: Hairong Kuang
Assignee: Lars Hofhansl
 Fix For: 2.0.1-alpha

 Attachments: HDFS-744-2.0-v1.patch, HDFS-744-2.0-v2.patch, 
 HDFS-744-trunk-v2.patch, HDFS-744-trunk-v3.patch, HDFS-744-trunk-v4.patch, 
 HDFS-744-trunk-v5.patch, HDFS-744-trunk-v6.patch, HDFS-744-trunk-v7.patch, 
 HDFS-744-trunk-v8.patch, HDFS-744-trunk.patch, hdfs-744-v2.txt, 
 hdfs-744-v3.txt, hdfs-744.txt


 HDFS-731 implements hsync by default as hflush. As descriibed in HADOOP-6313, 
 the real expected semantics should be flushes out to all replicas and all 
 replicas have done posix fsync equivalent - ie the OS has flushed it to the 
 disk device (but the disk may have it in its cache). This jira aims to 
 implement the expected behaviour.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira




[jira] [Updated] (HDFS-744) Support hsync in HDFS

2012-05-31 Thread Lars Hofhansl (JIRA)

 [ 
https://issues.apache.org/jira/browse/HDFS-744?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Lars Hofhansl updated HDFS-744:
---

Attachment: HDFS-744-2.0-v1.patch

This patch mostly applied to the 2.0 branch (with some offsets, no fuzz, and 
one hunk failed, which was easily fixable).

I understand if there's hesitation to pull this into 2.0 or even 1.0.

 Support hsync in HDFS
 -

 Key: HDFS-744
 URL: https://issues.apache.org/jira/browse/HDFS-744
 Project: Hadoop HDFS
  Issue Type: New Feature
  Components: data-node, hdfs client
Reporter: Hairong Kuang
Assignee: Lars Hofhansl
 Fix For: 3.0.0

 Attachments: HDFS-744-2.0-v1.patch, HDFS-744-trunk-v2.patch, 
 HDFS-744-trunk-v3.patch, HDFS-744-trunk-v4.patch, HDFS-744-trunk-v5.patch, 
 HDFS-744-trunk-v6.patch, HDFS-744-trunk-v7.patch, HDFS-744-trunk-v8.patch, 
 HDFS-744-trunk.patch, hdfs-744-v2.txt, hdfs-744-v3.txt, hdfs-744.txt


 HDFS-731 implements hsync by default as hflush. As descriibed in HADOOP-6313, 
 the real expected semantics should be flushes out to all replicas and all 
 replicas have done posix fsync equivalent - ie the OS has flushed it to the 
 disk device (but the disk may have it in its cache). This jira aims to 
 implement the expected behaviour.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira




[jira] [Updated] (HDFS-744) Support hsync in HDFS

2012-05-31 Thread Lars Hofhansl (JIRA)

 [ 
https://issues.apache.org/jira/browse/HDFS-744?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Lars Hofhansl updated HDFS-744:
---

Attachment: HDFS-744-2.0-v2.patch

Meh, forgot to add TestHSync to svn.

 Support hsync in HDFS
 -

 Key: HDFS-744
 URL: https://issues.apache.org/jira/browse/HDFS-744
 Project: Hadoop HDFS
  Issue Type: New Feature
  Components: data-node, hdfs client
Reporter: Hairong Kuang
Assignee: Lars Hofhansl
 Fix For: 3.0.0

 Attachments: HDFS-744-2.0-v1.patch, HDFS-744-2.0-v2.patch, 
 HDFS-744-trunk-v2.patch, HDFS-744-trunk-v3.patch, HDFS-744-trunk-v4.patch, 
 HDFS-744-trunk-v5.patch, HDFS-744-trunk-v6.patch, HDFS-744-trunk-v7.patch, 
 HDFS-744-trunk-v8.patch, HDFS-744-trunk.patch, hdfs-744-v2.txt, 
 hdfs-744-v3.txt, hdfs-744.txt


 HDFS-731 implements hsync by default as hflush. As descriibed in HADOOP-6313, 
 the real expected semantics should be flushes out to all replicas and all 
 replicas have done posix fsync equivalent - ie the OS has flushed it to the 
 disk device (but the disk may have it in its cache). This jira aims to 
 implement the expected behaviour.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira




[jira] [Updated] (HDFS-744) Support hsync in HDFS

2012-05-23 Thread Lars Hofhansl (JIRA)

 [ 
https://issues.apache.org/jira/browse/HDFS-744?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Lars Hofhansl updated HDFS-744:
---

Attachment: HDFS-744-trunk-v8.patch

* The synchronized was left over from the former hsync stuff (the one that just 
called hflush). I agree it should not be needed.
* Removed the tabs
* Added a better comment about multiple hsyncs in the test code
* Removed some unused variables from the test code

 Support hsync in HDFS
 -

 Key: HDFS-744
 URL: https://issues.apache.org/jira/browse/HDFS-744
 Project: Hadoop HDFS
  Issue Type: New Feature
  Components: data-node, hdfs client
Reporter: Hairong Kuang
Assignee: Lars Hofhansl
 Attachments: HDFS-744-trunk-v2.patch, HDFS-744-trunk-v3.patch, 
 HDFS-744-trunk-v4.patch, HDFS-744-trunk-v5.patch, HDFS-744-trunk-v6.patch, 
 HDFS-744-trunk-v7.patch, HDFS-744-trunk-v8.patch, HDFS-744-trunk.patch, 
 hdfs-744-v2.txt, hdfs-744-v3.txt, hdfs-744.txt


 HDFS-731 implements hsync by default as hflush. As descriibed in HADOOP-6313, 
 the real expected semantics should be flushes out to all replicas and all 
 replicas have done posix fsync equivalent - ie the OS has flushed it to the 
 disk device (but the disk may have it in its cache). This jira aims to 
 implement the expected behaviour.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira




[jira] [Updated] (HDFS-744) Support hsync in HDFS

2012-05-19 Thread Lars Hofhansl (JIRA)

 [ 
https://issues.apache.org/jira/browse/HDFS-744?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Lars Hofhansl updated HDFS-744:
---

Attachment: HDFS-744-trunk-v6.patch

New patch.

* only uses one flag
* SequenceFile.Writer implement Syncable.
* syncFS is deprecated.
* added metrics
* added test using metrics to verify behavior
* added replication test, to make sure the hsync is happening correctly on 
replicas.
* found a bug, where sometimes a sync is missed, happens when the currentPacket 
in hsync is null. Fixed now.

Currently I count syncs of out and checksumOut (in BlockReceiver) as two 
sync events. Would be hard to do otherwise, as both can be null resp, in which 
case no sync is happening.

In rare cases I found that the metrics on the DNs are not updated fast enough, 
and a test fails. Looks like there are existing tests that also use metrics 
that have the same problem. Not sure what to do about that.


 Support hsync in HDFS
 -

 Key: HDFS-744
 URL: https://issues.apache.org/jira/browse/HDFS-744
 Project: Hadoop HDFS
  Issue Type: New Feature
  Components: data-node, hdfs client
Reporter: Hairong Kuang
Assignee: Lars Hofhansl
 Attachments: HDFS-744-trunk-v2.patch, HDFS-744-trunk-v3.patch, 
 HDFS-744-trunk-v4.patch, HDFS-744-trunk-v5.patch, HDFS-744-trunk-v6.patch, 
 HDFS-744-trunk.patch, hdfs-744-v2.txt, hdfs-744-v3.txt, hdfs-744.txt


 HDFS-731 implements hsync by default as hflush. As descriibed in HADOOP-6313, 
 the real expected semantics should be flushes out to all replicas and all 
 replicas have done posix fsync equivalent - ie the OS has flushed it to the 
 disk device (but the disk may have it in its cache). This jira aims to 
 implement the expected behaviour.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira




[jira] [Updated] (HDFS-744) Support hsync in HDFS

2012-05-19 Thread Lars Hofhansl (JIRA)

 [ 
https://issues.apache.org/jira/browse/HDFS-744?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Lars Hofhansl updated HDFS-744:
---

Attachment: HDFS-744-trunk-v7.patch

v7 is almost identical to v6.
* Count a sync only once, but still measure combined time for syncing out and 
checksumOut


 Support hsync in HDFS
 -

 Key: HDFS-744
 URL: https://issues.apache.org/jira/browse/HDFS-744
 Project: Hadoop HDFS
  Issue Type: New Feature
  Components: data-node, hdfs client
Reporter: Hairong Kuang
Assignee: Lars Hofhansl
 Attachments: HDFS-744-trunk-v2.patch, HDFS-744-trunk-v3.patch, 
 HDFS-744-trunk-v4.patch, HDFS-744-trunk-v5.patch, HDFS-744-trunk-v6.patch, 
 HDFS-744-trunk-v7.patch, HDFS-744-trunk.patch, hdfs-744-v2.txt, 
 hdfs-744-v3.txt, hdfs-744.txt


 HDFS-731 implements hsync by default as hflush. As descriibed in HADOOP-6313, 
 the real expected semantics should be flushes out to all replicas and all 
 replicas have done posix fsync equivalent - ie the OS has flushed it to the 
 disk device (but the disk may have it in its cache). This jira aims to 
 implement the expected behaviour.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira




[jira] [Updated] (HDFS-744) Support hsync in HDFS

2012-05-17 Thread Lars Hofhansl (JIRA)

 [ 
https://issues.apache.org/jira/browse/HDFS-744?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Lars Hofhansl updated HDFS-744:
---

Attachment: HDFS-744-trunk-v5.patch

New version of the patch.

* Implemented all of Nicholas suggestions.
* Added some simple tests.
* Added a flushFS() method to SequeceFile.Writer.

I would still prefer to implement to hsync() as flushOrSync(syncBlock) rather 
than flushOrSync(true), but this works too.

 Support hsync in HDFS
 -

 Key: HDFS-744
 URL: https://issues.apache.org/jira/browse/HDFS-744
 Project: Hadoop HDFS
  Issue Type: New Feature
  Components: data-node, hdfs client
Reporter: Hairong Kuang
Assignee: Lars Hofhansl
 Attachments: HDFS-744-trunk-v2.patch, HDFS-744-trunk-v3.patch, 
 HDFS-744-trunk-v4.patch, HDFS-744-trunk-v5.patch, HDFS-744-trunk.patch, 
 hdfs-744-v2.txt, hdfs-744-v3.txt, hdfs-744.txt


 HDFS-731 implements hsync by default as hflush. As descriibed in HADOOP-6313, 
 the real expected semantics should be flushes out to all replicas and all 
 replicas have done posix fsync equivalent - ie the OS has flushed it to the 
 disk device (but the disk may have it in its cache). This jira aims to 
 implement the expected behaviour.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira




[jira] [Updated] (HDFS-744) Support hsync in HDFS

2012-05-12 Thread Lars Hofhansl (JIRA)

 [ 
https://issues.apache.org/jira/browse/HDFS-744?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Lars Hofhansl updated HDFS-744:
---

Attachment: HDFS-744-trunk-v4.patch

Really fix TestFilterFileSystem this time.
The TestTrash failure seems unrelated to my change.

 Support hsync in HDFS
 -

 Key: HDFS-744
 URL: https://issues.apache.org/jira/browse/HDFS-744
 Project: Hadoop HDFS
  Issue Type: New Feature
  Components: data-node, hdfs client
Reporter: Hairong Kuang
Assignee: Lars Hofhansl
 Attachments: HDFS-744-trunk-v2.patch, HDFS-744-trunk-v3.patch, 
 HDFS-744-trunk-v4.patch, HDFS-744-trunk.patch, hdfs-744-v2.txt, 
 hdfs-744-v3.txt, hdfs-744.txt


 HDFS-731 implements hsync by default as hflush. As descriibed in HADOOP-6313, 
 the real expected semantics should be flushes out to all replicas and all 
 replicas have done posix fsync equivalent - ie the OS has flushed it to the 
 disk device (but the disk may have it in its cache). This jira aims to 
 implement the expected behaviour.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira




[jira] [Updated] (HDFS-744) Support hsync in HDFS

2012-05-11 Thread Tsz Wo (Nicholas), SZE (JIRA)

 [ 
https://issues.apache.org/jira/browse/HDFS-744?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Tsz Wo (Nicholas), SZE updated HDFS-744:


Component/s: hdfs client
 data-node
   Assignee: Lars Hofhansl

 Support hsync in HDFS
 -

 Key: HDFS-744
 URL: https://issues.apache.org/jira/browse/HDFS-744
 Project: Hadoop HDFS
  Issue Type: New Feature
  Components: data-node, hdfs client
Reporter: Hairong Kuang
Assignee: Lars Hofhansl
 Attachments: HDFS-744-trunk-v2.patch, HDFS-744-trunk.patch, 
 hdfs-744-v2.txt, hdfs-744-v3.txt, hdfs-744.txt


 HDFS-731 implements hsync by default as hflush. As descriibed in HADOOP-6313, 
 the real expected semantics should be flushes out to all replicas and all 
 replicas have done posix fsync equivalent - ie the OS has flushed it to the 
 disk device (but the disk may have it in its cache). This jira aims to 
 implement the expected behaviour.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira




[jira] [Updated] (HDFS-744) Support hsync in HDFS

2012-05-11 Thread Tsz Wo (Nicholas), SZE (JIRA)

 [ 
https://issues.apache.org/jira/browse/HDFS-744?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Tsz Wo (Nicholas), SZE updated HDFS-744:


Status: Patch Available  (was: Open)

Let's try submitting it.  Will review the patch next week.

 Support hsync in HDFS
 -

 Key: HDFS-744
 URL: https://issues.apache.org/jira/browse/HDFS-744
 Project: Hadoop HDFS
  Issue Type: New Feature
  Components: data-node, hdfs client
Reporter: Hairong Kuang
Assignee: Lars Hofhansl
 Attachments: HDFS-744-trunk-v2.patch, HDFS-744-trunk.patch, 
 hdfs-744-v2.txt, hdfs-744-v3.txt, hdfs-744.txt


 HDFS-731 implements hsync by default as hflush. As descriibed in HADOOP-6313, 
 the real expected semantics should be flushes out to all replicas and all 
 replicas have done posix fsync equivalent - ie the OS has flushed it to the 
 disk device (but the disk may have it in its cache). This jira aims to 
 implement the expected behaviour.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira




[jira] [Updated] (HDFS-744) Support hsync in HDFS

2012-05-11 Thread Lars Hofhansl (JIRA)

 [ 
https://issues.apache.org/jira/browse/HDFS-744?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Lars Hofhansl updated HDFS-744:
---

Attachment: HDFS-744-trunk-v3.patch

Most tests fail due to an NPE in flush.
currentPacket can be null in DFSOutputStream.{hflush|hsync} even when 
bytesCurBlock  lastFlushOffset, which I found a bit surprising.
I think in that case it is OK not to send a sync packet (if FORCE is enabled, 
because there is nothing outstanding to sync).

Also fixed TestFilterFileSystem.
Not sure what's wrong with testTrash.

Also I could use guidance for:
* what types of tests should I add
* If I wanted to add a feature where optionally only the first DN in the 
replication chain does the fsync, how should a client convey that to the 
FS.create method (it does not make sense that add another CreateFlag).


 Support hsync in HDFS
 -

 Key: HDFS-744
 URL: https://issues.apache.org/jira/browse/HDFS-744
 Project: Hadoop HDFS
  Issue Type: New Feature
  Components: data-node, hdfs client
Reporter: Hairong Kuang
Assignee: Lars Hofhansl
 Attachments: HDFS-744-trunk-v2.patch, HDFS-744-trunk-v3.patch, 
 HDFS-744-trunk.patch, hdfs-744-v2.txt, hdfs-744-v3.txt, hdfs-744.txt


 HDFS-731 implements hsync by default as hflush. As descriibed in HADOOP-6313, 
 the real expected semantics should be flushes out to all replicas and all 
 replicas have done posix fsync equivalent - ie the OS has flushed it to the 
 disk device (but the disk may have it in its cache). This jira aims to 
 implement the expected behaviour.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira




[jira] [Updated] (HDFS-744) Support hsync in HDFS

2012-05-10 Thread Lars Hofhansl (JIRA)

 [ 
https://issues.apache.org/jira/browse/HDFS-744?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Lars Hofhansl updated HDFS-744:
---

Attachment: HDFS-744-trunk-v2.patch

Patch that I tested against HBase.
(I post the required HBase changes on the linked jira.)

HBase starts up, I can flush, and compact tables.
I verified via debugger that the sync path is correctly triggered.

*Please* have a look. For users like us (Salesforce.com) this is an important 
data safety feature.


 Support hsync in HDFS
 -

 Key: HDFS-744
 URL: https://issues.apache.org/jira/browse/HDFS-744
 Project: Hadoop HDFS
  Issue Type: New Feature
Reporter: Hairong Kuang
 Attachments: HDFS-744-trunk-v2.patch, HDFS-744-trunk.patch, 
 hdfs-744-v2.txt, hdfs-744-v3.txt, hdfs-744.txt


 HDFS-731 implements hsync by default as hflush. As descriibed in HADOOP-6313, 
 the real expected semantics should be flushes out to all replicas and all 
 replicas have done posix fsync equivalent - ie the OS has flushed it to the 
 disk device (but the disk may have it in its cache). This jira aims to 
 implement the expected behaviour.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira




[jira] [Updated] (HDFS-744) Support hsync in HDFS

2012-05-08 Thread Lars Hofhansl (JIRA)

 [ 
https://issues.apache.org/jira/browse/HDFS-744?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Lars Hofhansl updated HDFS-744:
---

Attachment: HDFS-744-trunk.patch

Here's a patch against trunk.
Principle is the same, CreateFlag now also has a FORCE flag.

It compiles and I did test with a simple client.

I have not managed to get HBase running with Hadoop trunk, yet.


 Support hsync in HDFS
 -

 Key: HDFS-744
 URL: https://issues.apache.org/jira/browse/HDFS-744
 Project: Hadoop HDFS
  Issue Type: New Feature
Reporter: Hairong Kuang
 Attachments: HDFS-744-trunk.patch, hdfs-744-v2.txt, hdfs-744-v3.txt, 
 hdfs-744.txt


 HDFS-731 implements hsync by default as hflush. As descriibed in HADOOP-6313, 
 the real expected semantics should be flushes out to all replicas and all 
 replicas have done posix fsync equivalent - ie the OS has flushed it to the 
 disk device (but the disk may have it in its cache). This jira aims to 
 implement the expected behaviour.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira




[jira] [Updated] (HDFS-744) Support hsync in HDFS

2012-05-07 Thread Lars Hofhansl (JIRA)

 [ 
https://issues.apache.org/jira/browse/HDFS-744?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Lars Hofhansl updated HDFS-744:
---

Attachment: hdfs-744.txt

Here's a sketch of a patch against 1.0.x.
I did only light testing on this.

It uses the packet's lastPacketInBlock byte to send flags to the Datanode. (An 
old client will work against a new data node, a new client should work against 
an old data node as long the sync feature is not used, but it would not fail 
gracefully if it did.)

The client can send two flags with a packet:
# sync block
# sync packet

The idea is that #1 would be sent with at least one packet per block in order 
to sync the block upon close.
#2 is sent by the client if a sync should be forced immediately on a partial 
block. If the client has outstanding data to send anyway the flag is pickbagged 
on the packet for that data, otherwise an empty sync packet is sent if needed.

Together they allow a client to guarantee that all bytes up to a certain point 
are guaranteed on disk.

Please have a look and let me know whether I'm off track with this.

If not, I'll clean it up, add some tests, create a trunk patch (which I imagine 
would look a bit differently), and maybe add a only-the-last-replica-syncs 
option.

Thanks.

 Support hsync in HDFS
 -

 Key: HDFS-744
 URL: https://issues.apache.org/jira/browse/HDFS-744
 Project: Hadoop HDFS
  Issue Type: New Feature
Reporter: Hairong Kuang
 Attachments: hdfs-744.txt


 HDFS-731 implements hsync by default as hflush. As descriibed in HADOOP-6313, 
 the real expected semantics should be flushes out to all replicas and all 
 replicas have done posix fsync equivalent - ie the OS has flushed it to the 
 disk device (but the disk may have it in its cache). This jira aims to 
 implement the expected behaviour.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira




[jira] [Updated] (HDFS-744) Support hsync in HDFS

2012-05-07 Thread Lars Hofhansl (JIRA)

 [ 
https://issues.apache.org/jira/browse/HDFS-744?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Lars Hofhansl updated HDFS-744:
---

Attachment: hdfs-744-v2.txt

Renamed sync to force added a few more comments. Fixed 
ChecksumFileSystem.create signature issue. Fixed endless loop when 
FileSystem.create with force is called on FS other than DFS.

 Support hsync in HDFS
 -

 Key: HDFS-744
 URL: https://issues.apache.org/jira/browse/HDFS-744
 Project: Hadoop HDFS
  Issue Type: New Feature
Reporter: Hairong Kuang
 Attachments: hdfs-744-v2.txt, hdfs-744.txt


 HDFS-731 implements hsync by default as hflush. As descriibed in HADOOP-6313, 
 the real expected semantics should be flushes out to all replicas and all 
 replicas have done posix fsync equivalent - ie the OS has flushed it to the 
 disk device (but the disk may have it in its cache). This jira aims to 
 implement the expected behaviour.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira




[jira] [Updated] (HDFS-744) Support hsync in HDFS

2012-05-07 Thread Lars Hofhansl (JIRA)

 [ 
https://issues.apache.org/jira/browse/HDFS-744?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Lars Hofhansl updated HDFS-744:
---

Attachment: hdfs-744-v3.txt

More changes needed for HBase (SequenceFiles need to support the force flag).

I made matching changes in HBase and will test this in a real cluster.

The patch has become big. There is no feedback, yet, from any HDFS folks.

Without a design approval more work might go to waste.


 Support hsync in HDFS
 -

 Key: HDFS-744
 URL: https://issues.apache.org/jira/browse/HDFS-744
 Project: Hadoop HDFS
  Issue Type: New Feature
Reporter: Hairong Kuang
 Attachments: hdfs-744-v2.txt, hdfs-744-v3.txt, hdfs-744.txt


 HDFS-731 implements hsync by default as hflush. As descriibed in HADOOP-6313, 
 the real expected semantics should be flushes out to all replicas and all 
 replicas have done posix fsync equivalent - ie the OS has flushed it to the 
 disk device (but the disk may have it in its cache). This jira aims to 
 implement the expected behaviour.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira