[jira] [Updated] (HDFS-2003) Separate FSEditLog reading logic from editLog memory state building logic
[ https://issues.apache.org/jira/browse/HDFS-2003?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ivan Kelly updated HDFS-2003: - Status: Patch Available (was: Open) > Separate FSEditLog reading logic from editLog memory state building logic > - > > Key: HDFS-2003 > URL: https://issues.apache.org/jira/browse/HDFS-2003 > Project: Hadoop HDFS > Issue Type: Improvement >Affects Versions: Edit log branch (HDFS-1073) >Reporter: Ivan Kelly >Assignee: Ivan Kelly > Fix For: Edit log branch (HDFS-1073) > > Attachments: HDFS-2003.diff > > > Currently FSEditLogLoader has code for reading from an InputStream > interleaved with code which updates the FSNameSystem and FSDirectory. This > makes it difficult to read an edit log without having a whole load of other > object initialised, which is problematic if you want to do things like count > how many transactions are in a file etc. > This patch separates the reading of the stream and the building of the memory > state. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-2003) Separate FSEditLog reading logic from editLog memory state building logic
[ https://issues.apache.org/jira/browse/HDFS-2003?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ivan Kelly updated HDFS-2003: - Attachment: HDFS-2003.diff Patch applies against 1073 branch. > Separate FSEditLog reading logic from editLog memory state building logic > - > > Key: HDFS-2003 > URL: https://issues.apache.org/jira/browse/HDFS-2003 > Project: Hadoop HDFS > Issue Type: Improvement >Affects Versions: Edit log branch (HDFS-1073) >Reporter: Ivan Kelly >Assignee: Ivan Kelly > Fix For: Edit log branch (HDFS-1073) > > Attachments: HDFS-2003.diff > > > Currently FSEditLogLoader has code for reading from an InputStream > interleaved with code which updates the FSNameSystem and FSDirectory. This > makes it difficult to read an edit log without having a whole load of other > object initialised, which is problematic if you want to do things like count > how many transactions are in a file etc. > This patch separates the reading of the stream and the building of the memory > state. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-2003) Separate FSEditLog reading logic from editLog memory state building logic
[ https://issues.apache.org/jira/browse/HDFS-2003?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ivan Kelly updated HDFS-2003: - Attachment: HDFS-2003.diff Cleaned up imports > Separate FSEditLog reading logic from editLog memory state building logic > - > > Key: HDFS-2003 > URL: https://issues.apache.org/jira/browse/HDFS-2003 > Project: Hadoop HDFS > Issue Type: Improvement >Affects Versions: Edit log branch (HDFS-1073) >Reporter: Ivan Kelly >Assignee: Ivan Kelly > Fix For: Edit log branch (HDFS-1073) > > Attachments: HDFS-2003.diff, HDFS-2003.diff > > > Currently FSEditLogLoader has code for reading from an InputStream > interleaved with code which updates the FSNameSystem and FSDirectory. This > makes it difficult to read an edit log without having a whole load of other > object initialised, which is problematic if you want to do things like count > how many transactions are in a file etc. > This patch separates the reading of the stream and the building of the memory > state. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-2003) Separate FSEditLog reading logic from editLog memory state building logic
[ https://issues.apache.org/jira/browse/HDFS-2003?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ivan Kelly updated HDFS-2003: - Attachment: HDFS-2003.diff Uploaded patch addressing Todd's comments > Separate FSEditLog reading logic from editLog memory state building logic > - > > Key: HDFS-2003 > URL: https://issues.apache.org/jira/browse/HDFS-2003 > Project: Hadoop HDFS > Issue Type: Improvement >Affects Versions: Edit log branch (HDFS-1073) >Reporter: Ivan Kelly >Assignee: Ivan Kelly > Fix For: Edit log branch (HDFS-1073) > > Attachments: HDFS-2003.diff, HDFS-2003.diff, HDFS-2003.diff > > > Currently FSEditLogLoader has code for reading from an InputStream > interleaved with code which updates the FSNameSystem and FSDirectory. This > makes it difficult to read an edit log without having a whole load of other > object initialised, which is problematic if you want to do things like count > how many transactions are in a file etc. > This patch separates the reading of the stream and the building of the memory > state. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-2003) Separate FSEditLog reading logic from editLog memory state building logic
[ https://issues.apache.org/jira/browse/HDFS-2003?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ivan Kelly updated HDFS-2003: - Attachment: HDFS-2003.diff > Separate FSEditLog reading logic from editLog memory state building logic > - > > Key: HDFS-2003 > URL: https://issues.apache.org/jira/browse/HDFS-2003 > Project: Hadoop HDFS > Issue Type: Improvement >Affects Versions: Edit log branch (HDFS-1073) >Reporter: Ivan Kelly >Assignee: Ivan Kelly > Fix For: Edit log branch (HDFS-1073) > > Attachments: HDFS-2003.diff, HDFS-2003.diff, HDFS-2003.diff, > HDFS-2003.diff > > > Currently FSEditLogLoader has code for reading from an InputStream > interleaved with code which updates the FSNameSystem and FSDirectory. This > makes it difficult to read an edit log without having a whole load of other > object initialised, which is problematic if you want to do things like count > how many transactions are in a file etc. > This patch separates the reading of the stream and the building of the memory > state. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-2003) Separate FSEditLog reading logic from editLog memory state building logic
[ https://issues.apache.org/jira/browse/HDFS-2003?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Todd Lipcon updated HDFS-2003: -- Attachment: 2003-delta.txt Here is a patch on top of your latest (https://issues.apache.org/jira/secure/attachment/12481572/HDFS-2003.diff) which makes the following changes: - made some methods static which didn't access any member vars - readOp() now returns null for an expected end-of-file -- this is more like how most iterator interfaces work, I think a bit clearer than always having to catch EOF - fixed a couple of warnings (unused var, unnecessary suppresswarnings) I had one thought as I was going through this: can this actually go into trunk and then be merged into the 1073 branch? There isn't anything here that doesn't make sense in trunk, and since it's a pretty clean refactor I don't see why we need to do it on branch. > Separate FSEditLog reading logic from editLog memory state building logic > - > > Key: HDFS-2003 > URL: https://issues.apache.org/jira/browse/HDFS-2003 > Project: Hadoop HDFS > Issue Type: Improvement >Affects Versions: Edit log branch (HDFS-1073) >Reporter: Ivan Kelly >Assignee: Ivan Kelly > Fix For: Edit log branch (HDFS-1073) > > Attachments: 2003-delta.txt, HDFS-2003.diff, HDFS-2003.diff, > HDFS-2003.diff, HDFS-2003.diff > > > Currently FSEditLogLoader has code for reading from an InputStream > interleaved with code which updates the FSNameSystem and FSDirectory. This > makes it difficult to read an edit log without having a whole load of other > object initialised, which is problematic if you want to do things like count > how many transactions are in a file etc. > This patch separates the reading of the stream and the building of the memory > state. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-2003) Separate FSEditLog reading logic from editLog memory state building logic
[ https://issues.apache.org/jira/browse/HDFS-2003?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Todd Lipcon updated HDFS-2003: -- Attachment: hdfs-2003.txt Rebased patch against trunk, using the new LayoutVersion support, etc. > Separate FSEditLog reading logic from editLog memory state building logic > - > > Key: HDFS-2003 > URL: https://issues.apache.org/jira/browse/HDFS-2003 > Project: Hadoop HDFS > Issue Type: Improvement >Affects Versions: Edit log branch (HDFS-1073) >Reporter: Ivan Kelly >Assignee: Ivan Kelly > Fix For: Edit log branch (HDFS-1073) > > Attachments: 2003-delta.txt, HDFS-2003.diff, HDFS-2003.diff, > HDFS-2003.diff, HDFS-2003.diff, hdfs-2003.txt > > > Currently FSEditLogLoader has code for reading from an InputStream > interleaved with code which updates the FSNameSystem and FSDirectory. This > makes it difficult to read an edit log without having a whole load of other > object initialised, which is problematic if you want to do things like count > how many transactions are in a file etc. > This patch separates the reading of the stream and the building of the memory > state. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-2003) Separate FSEditLog reading logic from editLog memory state building logic
[ https://issues.apache.org/jira/browse/HDFS-2003?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Todd Lipcon updated HDFS-2003: -- Attachment: hdfs-2003.txt - Updated patch against trunk to include the new ReassignLeaseOp. - I filed HDFS-2041 for the findbugs warnings - turns out they are legit, but I want to keep this as a refactor and commit the bug fix right afterwards. For me, I was getting an error that the "unchecked" suppression wasn't necessary - where is there an unchecked cast warning? > Separate FSEditLog reading logic from editLog memory state building logic > - > > Key: HDFS-2003 > URL: https://issues.apache.org/jira/browse/HDFS-2003 > Project: Hadoop HDFS > Issue Type: Improvement >Affects Versions: Edit log branch (HDFS-1073) >Reporter: Ivan Kelly >Assignee: Ivan Kelly > Fix For: Edit log branch (HDFS-1073) > > Attachments: 2003-delta.txt, HDFS-2003.diff, HDFS-2003.diff, > HDFS-2003.diff, HDFS-2003.diff, hdfs-2003.txt, hdfs-2003.txt > > > Currently FSEditLogLoader has code for reading from an InputStream > interleaved with code which updates the FSNameSystem and FSDirectory. This > makes it difficult to read an edit log without having a whole load of other > object initialised, which is problematic if you want to do things like count > how many transactions are in a file etc. > This patch separates the reading of the stream and the building of the memory > state. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-2003) Separate FSEditLog reading logic from editLog memory state building logic
[ https://issues.apache.org/jira/browse/HDFS-2003?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ivan Kelly updated HDFS-2003: - Attachment: HDFS-2003.diff Fixed the findbugs. I don't think its worth doing it in another JIRA as the changes for it are tiny and don't do very much. I commented the code out instead of removing so that it's clear what was meant to be read. The reason this didn't hit before was that timestamp and atime were being reused in FSEditLogLoader#loadEditRecords(). > Separate FSEditLog reading logic from editLog memory state building logic > - > > Key: HDFS-2003 > URL: https://issues.apache.org/jira/browse/HDFS-2003 > Project: Hadoop HDFS > Issue Type: Improvement >Affects Versions: Edit log branch (HDFS-1073) >Reporter: Ivan Kelly >Assignee: Ivan Kelly > Fix For: Edit log branch (HDFS-1073) > > Attachments: 2003-delta.txt, HDFS-2003.diff, HDFS-2003.diff, > HDFS-2003.diff, HDFS-2003.diff, HDFS-2003.diff, hdfs-2003.txt, hdfs-2003.txt > > > Currently FSEditLogLoader has code for reading from an InputStream > interleaved with code which updates the FSNameSystem and FSDirectory. This > makes it difficult to read an edit log without having a whole load of other > object initialised, which is problematic if you want to do things like count > how many transactions are in a file etc. > This patch separates the reading of the stream and the building of the memory > state. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-2003) Separate FSEditLog reading logic from editLog memory state building logic
[
https://issues.apache.org/jira/browse/HDFS-2003?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Todd Lipcon updated HDFS-2003:
--
Attachment: hdfs-2003.txt
It turns out that the bug mentioned above does not actually cause a problem. I
added a unit test here to be sure of it -- the BlockInfo objects do get created
with a replication count that's too low, but before the {{triplets}} array is
used, {{ensureCapacity}} will resize it appropriately.
I think this patch is ready for commit if someone can please review.
> Separate FSEditLog reading logic from editLog memory state building logic
> -
>
> Key: HDFS-2003
> URL: https://issues.apache.org/jira/browse/HDFS-2003
> Project: Hadoop HDFS
> Issue Type: Improvement
>Affects Versions: Edit log branch (HDFS-1073)
>Reporter: Ivan Kelly
>Assignee: Ivan Kelly
> Fix For: Edit log branch (HDFS-1073)
>
> Attachments: 2003-delta.txt, HDFS-2003.diff, HDFS-2003.diff,
> HDFS-2003.diff, HDFS-2003.diff, HDFS-2003.diff, hdfs-2003.txt, hdfs-2003.txt,
> hdfs-2003.txt
>
>
> Currently FSEditLogLoader has code for reading from an InputStream
> interleaved with code which updates the FSNameSystem and FSDirectory. This
> makes it difficult to read an edit log without having a whole load of other
> object initialised, which is problematic if you want to do things like count
> how many transactions are in a file etc.
> This patch separates the reading of the stream and the building of the memory
> state.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-2003) Separate FSEditLog reading logic from editLog memory state building logic
[ https://issues.apache.org/jira/browse/HDFS-2003?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ivan Kelly updated HDFS-2003: - Attachment: HDFS-2003-replicationfix-delta.diff > Separate FSEditLog reading logic from editLog memory state building logic > - > > Key: HDFS-2003 > URL: https://issues.apache.org/jira/browse/HDFS-2003 > Project: Hadoop HDFS > Issue Type: Improvement >Affects Versions: Edit log branch (HDFS-1073) >Reporter: Ivan Kelly >Assignee: Ivan Kelly > Fix For: Edit log branch (HDFS-1073) > > Attachments: 2003-delta.txt, HDFS-2003-replicationfix-delta.diff, > HDFS-2003.diff, HDFS-2003.diff, HDFS-2003.diff, HDFS-2003.diff, > HDFS-2003.diff, hdfs-2003.txt, hdfs-2003.txt, hdfs-2003.txt > > > Currently FSEditLogLoader has code for reading from an InputStream > interleaved with code which updates the FSNameSystem and FSDirectory. This > makes it difficult to read an edit log without having a whole load of other > object initialised, which is problematic if you want to do things like count > how many transactions are in a file etc. > This patch separates the reading of the stream and the building of the memory > state. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-2003) Separate FSEditLog reading logic from editLog memory state building logic
[ https://issues.apache.org/jira/browse/HDFS-2003?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ivan Kelly updated HDFS-2003: - Attachment: HDFS-2003.diff > Separate FSEditLog reading logic from editLog memory state building logic > - > > Key: HDFS-2003 > URL: https://issues.apache.org/jira/browse/HDFS-2003 > Project: Hadoop HDFS > Issue Type: Improvement >Affects Versions: Edit log branch (HDFS-1073) >Reporter: Ivan Kelly >Assignee: Ivan Kelly > Fix For: Edit log branch (HDFS-1073) > > Attachments: 2003-delta.txt, HDFS-2003-replicationfix-delta.diff, > HDFS-2003.diff, HDFS-2003.diff, HDFS-2003.diff, HDFS-2003.diff, > HDFS-2003.diff, HDFS-2003.diff, hdfs-2003.txt, hdfs-2003.txt, hdfs-2003.txt > > > Currently FSEditLogLoader has code for reading from an InputStream > interleaved with code which updates the FSNameSystem and FSDirectory. This > makes it difficult to read an edit log without having a whole load of other > object initialised, which is problematic if you want to do things like count > how many transactions are in a file etc. > This patch separates the reading of the stream and the building of the memory > state. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-2003) Separate FSEditLog reading logic from editLog memory state building logic
[ https://issues.apache.org/jira/browse/HDFS-2003?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ivan Kelly updated HDFS-2003: - Attachment: HDFS-2003.diff Resubmitting patch fixing error which caused hudson failure. As reading in the Block and constructing the BlockInfo have been separated, Block objects need to be persisted. i.e. The same block object cant be reused for all the reads. @Todd could you give this a quick test with your benchmark to make sure it doesn't hit performance. > Separate FSEditLog reading logic from editLog memory state building logic > - > > Key: HDFS-2003 > URL: https://issues.apache.org/jira/browse/HDFS-2003 > Project: Hadoop HDFS > Issue Type: Improvement >Affects Versions: Edit log branch (HDFS-1073) >Reporter: Ivan Kelly >Assignee: Ivan Kelly > Fix For: Edit log branch (HDFS-1073) > > Attachments: 2003-delta.txt, HDFS-2003-replicationfix-delta.diff, > HDFS-2003.diff, HDFS-2003.diff, HDFS-2003.diff, HDFS-2003.diff, > HDFS-2003.diff, HDFS-2003.diff, HDFS-2003.diff, hdfs-2003.txt, hdfs-2003.txt, > hdfs-2003.txt > > > Currently FSEditLogLoader has code for reading from an InputStream > interleaved with code which updates the FSNameSystem and FSDirectory. This > makes it difficult to read an edit log without having a whole load of other > object initialised, which is problematic if you want to do things like count > how many transactions are in a file etc. > This patch separates the reading of the stream and the building of the memory > state. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-2003) Separate FSEditLog reading logic from editLog memory state building logic
[ https://issues.apache.org/jira/browse/HDFS-2003?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ivan Kelly updated HDFS-2003: - Attachment: HDFS-2003.diff Addressed the two things from Todd's previous comment. > Separate FSEditLog reading logic from editLog memory state building logic > - > > Key: HDFS-2003 > URL: https://issues.apache.org/jira/browse/HDFS-2003 > Project: Hadoop HDFS > Issue Type: Improvement >Affects Versions: Edit log branch (HDFS-1073) >Reporter: Ivan Kelly >Assignee: Ivan Kelly > Fix For: Edit log branch (HDFS-1073) > > Attachments: 2003-delta.txt, HDFS-2003-replicationfix-delta.diff, > HDFS-2003.diff, HDFS-2003.diff, HDFS-2003.diff, HDFS-2003.diff, > HDFS-2003.diff, HDFS-2003.diff, HDFS-2003.diff, HDFS-2003.diff, > hdfs-2003.txt, hdfs-2003.txt, hdfs-2003.txt > > > Currently FSEditLogLoader has code for reading from an InputStream > interleaved with code which updates the FSNameSystem and FSDirectory. This > makes it difficult to read an edit log without having a whole load of other > object initialised, which is problematic if you want to do things like count > how many transactions are in a file etc. > This patch separates the reading of the stream and the building of the memory > state. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-2003) Separate FSEditLog reading logic from editLog memory state building logic
[ https://issues.apache.org/jira/browse/HDFS-2003?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Todd Lipcon updated HDFS-2003: -- Resolution: Fixed Hadoop Flags: [Reviewed] Status: Resolved (was: Patch Available) Committed to trunk, thanks Ivan! I will try to merge this into the 1073 branch this afternoon or evening. > Separate FSEditLog reading logic from editLog memory state building logic > - > > Key: HDFS-2003 > URL: https://issues.apache.org/jira/browse/HDFS-2003 > Project: Hadoop HDFS > Issue Type: Improvement >Affects Versions: Edit log branch (HDFS-1073) >Reporter: Ivan Kelly >Assignee: Ivan Kelly > Fix For: Edit log branch (HDFS-1073) > > Attachments: 2003-delta.txt, HDFS-2003-replicationfix-delta.diff, > HDFS-2003.diff, HDFS-2003.diff, HDFS-2003.diff, HDFS-2003.diff, > HDFS-2003.diff, HDFS-2003.diff, HDFS-2003.diff, HDFS-2003.diff, > hdfs-2003.txt, hdfs-2003.txt, hdfs-2003.txt > > > Currently FSEditLogLoader has code for reading from an InputStream > interleaved with code which updates the FSNameSystem and FSDirectory. This > makes it difficult to read an edit log without having a whole load of other > object initialised, which is problematic if you want to do things like count > how many transactions are in a file etc. > This patch separates the reading of the stream and the building of the memory > state. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-2003) Separate FSEditLog reading logic from editLog memory state building logic
[ https://issues.apache.org/jira/browse/HDFS-2003?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Todd Lipcon updated HDFS-2003: -- Affects Version/s: 0.23.0 Fix Version/s: 0.23.0 Updating fix versions, this was done in trunk as well. > Separate FSEditLog reading logic from editLog memory state building logic > - > > Key: HDFS-2003 > URL: https://issues.apache.org/jira/browse/HDFS-2003 > Project: Hadoop HDFS > Issue Type: Improvement >Affects Versions: Edit log branch (HDFS-1073), 0.23.0 >Reporter: Ivan Kelly >Assignee: Ivan Kelly > Fix For: Edit log branch (HDFS-1073), 0.23.0 > > Attachments: 2003-delta.txt, HDFS-2003-replicationfix-delta.diff, > HDFS-2003.diff, HDFS-2003.diff, HDFS-2003.diff, HDFS-2003.diff, > HDFS-2003.diff, HDFS-2003.diff, HDFS-2003.diff, HDFS-2003.diff, > hdfs-2003.txt, hdfs-2003.txt, hdfs-2003.txt > > > Currently FSEditLogLoader has code for reading from an InputStream > interleaved with code which updates the FSNameSystem and FSDirectory. This > makes it difficult to read an edit log without having a whole load of other > object initialised, which is problematic if you want to do things like count > how many transactions are in a file etc. > This patch separates the reading of the stream and the building of the memory > state. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
