Store data and checksums together in block file
---
Key: HDFS-2699
URL: https://issues.apache.org/jira/browse/HDFS-2699
Project: Hadoop HDFS
Issue Type: Improvement
Reporter: dhruba
[
https://issues.apache.org/jira/browse/HDFS-2699?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13171493#comment-13171493
]
dhruba borthakur commented on HDFS-2699:
The number of random reads issued by HBase
[
https://issues.apache.org/jira/browse/HDFS-2699?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13171531#comment-13171531
]
Luke Lu commented on HDFS-2699:
---
bq. The number of random reads issued by HBase is almost
[
https://issues.apache.org/jira/browse/HDFS-2699?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13171539#comment-13171539
]
Luke Lu commented on HDFS-2699:
---
bq. You'd need HFile v3 for this
Or come up new compression
[
https://issues.apache.org/jira/browse/HDFS-2687?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13171542#comment-13171542
]
Hudson commented on HDFS-2687:
--
Integrated in Hadoop-Hdfs-trunk #897 (See
[
https://issues.apache.org/jira/browse/HDFS-2694?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13171541#comment-13171541
]
Hudson commented on HDFS-2694:
--
Integrated in Hadoop-Hdfs-trunk #897 (See
[
https://issues.apache.org/jira/browse/HDFS-2640?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13171545#comment-13171545
]
Hudson commented on HDFS-2640:
--
Integrated in Hadoop-Hdfs-trunk #897 (See
[
https://issues.apache.org/jira/browse/HDFS-2684?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13171547#comment-13171547
]
Hudson commented on HDFS-2684:
--
Integrated in Hadoop-Hdfs-HAbranch-build #19 (See
[
https://issues.apache.org/jira/browse/HDFS-2640?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13171550#comment-13171550
]
Hudson commented on HDFS-2640:
--
Integrated in Hadoop-Hdfs-0.23-Build #110 (See
[
https://issues.apache.org/jira/browse/HDFS-2640?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13171557#comment-13171557
]
Hudson commented on HDFS-2640:
--
Integrated in Hadoop-Mapreduce-0.23-Build #130 (See
[
https://issues.apache.org/jira/browse/HDFS-2694?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13171565#comment-13171565
]
Hudson commented on HDFS-2694:
--
Integrated in Hadoop-Mapreduce-trunk #930 (See
[
https://issues.apache.org/jira/browse/HDFS-2687?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13171566#comment-13171566
]
Hudson commented on HDFS-2687:
--
Integrated in Hadoop-Mapreduce-trunk #930 (See
[
https://issues.apache.org/jira/browse/HDFS-2640?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13171569#comment-13171569
]
Hudson commented on HDFS-2640:
--
Integrated in Hadoop-Mapreduce-trunk #930 (See
[
https://issues.apache.org/jira/browse/HDFS-2486?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Uma Maheswara Rao G updated HDFS-2486:
--
Attachment: HDFS-2486.patch
Review issues with UnderReplicatedBlocks
[
https://issues.apache.org/jira/browse/HDFS-2486?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Uma Maheswara Rao G updated HDFS-2486:
--
Assignee: Uma Maheswara Rao G
Target Version/s: 0.24.0
Status:
[
https://issues.apache.org/jira/browse/HDFS-2486?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13171577#comment-13171577
]
Uma Maheswara Rao G commented on HDFS-2486:
---
This patch addresses the second
[
https://issues.apache.org/jira/browse/HDFS-2362?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13171582#comment-13171582
]
Uma Maheswara Rao G commented on HDFS-2362:
---
Can some please merge this
[
https://issues.apache.org/jira/browse/HDFS-2486?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13171586#comment-13171586
]
Hadoop QA commented on HDFS-2486:
-
-1 overall. Here are the results of testing the latest
[
https://issues.apache.org/jira/browse/HDFS-2553?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Uma Maheswara Rao G updated HDFS-2553:
--
Attachment: (was: HDFS-2553.patch)
BlockPoolSliceScanner spinning in loop
[
https://issues.apache.org/jira/browse/HDFS-2553?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Uma Maheswara Rao G updated HDFS-2553:
--
Attachment: HDFS-2553.patch
Attached the same patch again to trigger the Jenkins
TestDataNodeMultipleRegistrations is failing in trunk
-
Key: HDFS-2700
URL: https://issues.apache.org/jira/browse/HDFS-2700
Project: Hadoop HDFS
Issue Type: Test
Reporter: Uma
[
https://issues.apache.org/jira/browse/HDFS-2700?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13171596#comment-13171596
]
Uma Maheswara Rao G commented on HDFS-2700:
---
more info:
java.io.IOException:
[
https://issues.apache.org/jira/browse/HDFS-2553?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13171598#comment-13171598
]
Hadoop QA commented on HDFS-2553:
-
-1 overall. Here are the results of testing the latest
[
https://issues.apache.org/jira/browse/HDFS-2553?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13171599#comment-13171599
]
Uma Maheswara Rao G commented on HDFS-2553:
---
{quote}
-1 core tests. The patch
[
https://issues.apache.org/jira/browse/HDFS-2699?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13171601#comment-13171601
]
dhruba borthakur commented on HDFS-2699:
There are various alternatie like you
[
https://issues.apache.org/jira/browse/HDFS-2699?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13171604#comment-13171604
]
dhruba borthakur commented on HDFS-2699:
Another option that I am going to try is
[
https://issues.apache.org/jira/browse/HDFS-2700?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13171607#comment-13171607
]
Uma Maheswara Rao G commented on HDFS-2700:
---
Some more information:
Looks nor
[
https://issues.apache.org/jira/browse/HDFS-554?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Harsh J updated HDFS-554:
-
Attachment: HDFS-554.txt
Thanks for that catch Todd, you're right :)
BlockInfo.ensureCapacity
[
https://issues.apache.org/jira/browse/HDFS-2700?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13171616#comment-13171616
]
Uma Maheswara Rao G commented on HDFS-2700:
---
Here the problem is
[
https://issues.apache.org/jira/browse/HDFS-2700?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Uma Maheswara Rao G updated HDFS-2700:
--
Attachment: HDFS-2700.patch
TestDataNodeMultipleRegistrations is failing in trunk
[
https://issues.apache.org/jira/browse/HDFS-2700?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Uma Maheswara Rao G updated HDFS-2700:
--
Assignee: Uma Maheswara Rao G
Status: Patch Available (was: Open)
[
https://issues.apache.org/jira/browse/HDFS-2700?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Uma Maheswara Rao G updated HDFS-2700:
--
Issue Type: Bug (was: Test)
TestDataNodeMultipleRegistrations is failing in trunk
[
https://issues.apache.org/jira/browse/HDFS-554?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13171622#comment-13171622
]
Aaron T. Myers commented on HDFS-554:
-
The patch looks good to me. +1 pending clean
[
https://issues.apache.org/jira/browse/HDFS-2700?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13171635#comment-13171635
]
Hadoop QA commented on HDFS-2700:
-
-1 overall. Here are the results of testing the latest
[
https://issues.apache.org/jira/browse/HDFS-2700?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13171638#comment-13171638
]
Uma Maheswara Rao G commented on HDFS-2700:
---
Test failures are unrelated to this
[
https://issues.apache.org/jira/browse/HDFS-2699?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13171643#comment-13171643
]
Andrew Purtell commented on HDFS-2699:
--
IMHO, this is a design evolution question for
Cleanup FS* processIOError methods
--
Key: HDFS-2701
URL: https://issues.apache.org/jira/browse/HDFS-2701
Project: Hadoop HDFS
Issue Type: Improvement
Components: name-node
Affects Versions:
[
https://issues.apache.org/jira/browse/HDFS-2699?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13171653#comment-13171653
]
dhruba borthakur commented on HDFS-2699:
Hi andrew, all of the points you mentioned
[
https://issues.apache.org/jira/browse/HDFS-2701?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Eli Collins updated HDFS-2701:
--
Attachment: hdfs-2701.txt
Patch attached. Running test-patch now.
Cleanup FS*
[
https://issues.apache.org/jira/browse/HDFS-2699?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13171657#comment-13171657
]
Andrew Purtell commented on HDFS-2699:
--
@Dhruba, yes I agree with you fully. From the
A single failed name dir can cause the NN to exit
--
Key: HDFS-2702
URL: https://issues.apache.org/jira/browse/HDFS-2702
Project: Hadoop HDFS
Issue Type: Bug
Affects Versions: 0.20.205.0
[
https://issues.apache.org/jira/browse/HDFS-2702?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Eli Collins updated HDFS-2702:
--
Affects Version/s: (was: 1.1.0)
1.0.0
A single failed name dir can
removedStorageDirs is not updated everywhere we remove a storage dir
Key: HDFS-2703
URL: https://issues.apache.org/jira/browse/HDFS-2703
Project: Hadoop HDFS
Issue Type:
[
https://issues.apache.org/jira/browse/HDFS-2702?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Eli Collins updated HDFS-2702:
--
Target Version/s: 1.1.0
Affects Version/s: (was: 0.20.205.0)
1.1.0
[
https://issues.apache.org/jira/browse/HDFS-2701?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Eli Collins updated HDFS-2701:
--
Target Version/s: 1.1.0
Affects Version/s: (was: 0.20.205.0)
1.0.0
[
https://issues.apache.org/jira/browse/HDFS-2699?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13171660#comment-13171660
]
Allen Wittenauer commented on HDFS-2699:
Would it make sense to make the equivalent
NameNodeResouceChecker#checkAvailableResources should check for inodes
--
Key: HDFS-2704
URL: https://issues.apache.org/jira/browse/HDFS-2704
Project: Hadoop HDFS
Issue
[
https://issues.apache.org/jira/browse/HDFS-2701?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Eli Collins updated HDFS-2701:
--
Attachment: hdfs-2701.txt
test-patch results. Just cleanup so existing tests suffice. Will do a full
[
https://issues.apache.org/jira/browse/HDFS-2703?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Eli Collins updated HDFS-2703:
--
Attachment: hdfs-2703.txt
Patch attached. Verified that when the 2NN triggers a roll and a storage
[
https://issues.apache.org/jira/browse/HDFS-2702?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13171678#comment-13171678
]
Eli Collins commented on HDFS-2702:
---
This bug exists in FSEditLog#close as well.
[
https://issues.apache.org/jira/browse/HDFS-2703?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13171679#comment-13171679
]
Eli Collins commented on HDFS-2703:
---
This applies atop HDFS-2701.
[
https://issues.apache.org/jira/browse/HDFS-2702?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Eli Collins updated HDFS-2702:
--
Attachment: hdfs-2702.txt
Patch attached. Depends on HDFS-2701 and HDFS-2703. I verified the NN no
[
https://issues.apache.org/jira/browse/HDFS-2702?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13171685#comment-13171685
]
Eli Collins commented on HDFS-2702:
---
And verified the NN still exits after all storage
[
https://issues.apache.org/jira/browse/HDFS-2701?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Eli Collins updated HDFS-2701:
--
Attachment: hdfs-2701.txt
Minor update.
Cleanup FS* processIOError methods
[
https://issues.apache.org/jira/browse/HDFS-2677?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Eli Collins updated HDFS-2677:
--
Attachment: hdfs-2677.txt
Updated patch to reflect HADOOP-7925.
HA: Web UI should
[
https://issues.apache.org/jira/browse/HDFS-2679?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Eli Collins updated HDFS-2679:
--
Attachment: hdfs-2679.txt
Updated patch to reflect HADOOP-7925.
Add interface to
[
https://issues.apache.org/jira/browse/HDFS-2654?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Eli Collins updated HDFS-2654:
--
Attachment: hdfs-2654-b1-4-fix.patch
In hdfs-2654-b1-4.patch I replaced reader.close with
[
https://issues.apache.org/jira/browse/HDFS-2699?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13171698#comment-13171698
]
Todd Lipcon commented on HDFS-2699:
---
+1 on considering putting them in the same file.
[
https://issues.apache.org/jira/browse/HDFS-2335?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Uma Maheswara Rao G updated HDFS-2335:
--
Affects Version/s: 1.0.0
0.24.0
0.23.0
[
https://issues.apache.org/jira/browse/HDFS-2654?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13171704#comment-13171704
]
Todd Lipcon commented on HDFS-2654:
---
woops, sorry to have missed that.
[
https://issues.apache.org/jira/browse/HDFS-2699?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13171712#comment-13171712
]
Scott Carey commented on HDFS-2699:
---
That brings up a related question: Why a 4 byte crc
[
https://issues.apache.org/jira/browse/HDFS-2700?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13171732#comment-13171732
]
Todd Lipcon commented on HDFS-2700:
---
+1, thanks for taking care of this Uma. Will commit
[
https://issues.apache.org/jira/browse/HDFS-2700?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Todd Lipcon updated HDFS-2700:
--
Resolution: Fixed
Fix Version/s: 0.24.0
Hadoop Flags: Reviewed
Status: Resolved
[
https://issues.apache.org/jira/browse/HDFS-2553?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Uma Maheswara Rao G updated HDFS-2553:
--
Target Version/s: 0.24.0, 0.23.1 (was: 0.23.1, 0.24.0)
Status: Open
[
https://issues.apache.org/jira/browse/HDFS-2553?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Todd Lipcon resolved HDFS-2553.
---
Resolution: Fixed
Fix Version/s: 0.23.1
0.24.0
Target
[
https://issues.apache.org/jira/browse/HDFS-2553?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13171738#comment-13171738
]
Hudson commented on HDFS-2553:
--
Integrated in Hadoop-Hdfs-0.23-Commit #293 (See
[
https://issues.apache.org/jira/browse/HDFS-2553?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13171743#comment-13171743
]
Hudson commented on HDFS-2553:
--
Integrated in Hadoop-Common-0.23-Commit #304 (See
[
https://issues.apache.org/jira/browse/HDFS-2700?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13171740#comment-13171740
]
Hudson commented on HDFS-2700:
--
Integrated in Hadoop-Hdfs-trunk-Commit #1523 (See
[
https://issues.apache.org/jira/browse/HDFS-2553?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13171741#comment-13171741
]
Hudson commented on HDFS-2553:
--
Integrated in Hadoop-Hdfs-trunk-Commit #1523 (See
[
https://issues.apache.org/jira/browse/HDFS-2700?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13171744#comment-13171744
]
Hudson commented on HDFS-2700:
--
Integrated in Hadoop-Common-trunk-Commit #1450 (See
[
https://issues.apache.org/jira/browse/HDFS-2553?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13171745#comment-13171745
]
Hudson commented on HDFS-2553:
--
Integrated in Hadoop-Common-trunk-Commit #1450 (See
[
https://issues.apache.org/jira/browse/HDFS-2701?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13171747#comment-13171747
]
Todd Lipcon commented on HDFS-2701:
---
- The behavior of exitIfInvalidStreams is extremely
[
https://issues.apache.org/jira/browse/HDFS-2703?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13171748#comment-13171748
]
Todd Lipcon commented on HDFS-2703:
---
Looks good, but see the comment about a test plan in
[
https://issues.apache.org/jira/browse/HDFS-2553?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13171753#comment-13171753
]
Hudson commented on HDFS-2553:
--
Integrated in Hadoop-Mapreduce-trunk-Commit #1473 (See
[
https://issues.apache.org/jira/browse/HDFS-2700?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13171752#comment-13171752
]
Hudson commented on HDFS-2700:
--
Integrated in Hadoop-Mapreduce-trunk-Commit #1473 (See
[
https://issues.apache.org/jira/browse/HDFS-2553?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13171758#comment-13171758
]
Hudson commented on HDFS-2553:
--
Integrated in Hadoop-Mapreduce-0.23-Commit #315 (See
[
https://issues.apache.org/jira/browse/HDFS-554?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Harsh J updated HDFS-554:
-
Status: Patch Available (was: Open)
BlockInfo.ensureCapacity may get a speedup from System.arraycopy()
[
https://issues.apache.org/jira/browse/HDFS-554?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Harsh J updated HDFS-554:
-
Status: Open (was: Patch Available)
BlockInfo.ensureCapacity may get a speedup from System.arraycopy()
78 matches
Mail list logo