[jira] Updated: (HDFS-516) Low Latency distributed reads
[ https://issues.apache.org/jira/browse/HDFS-516?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jay Booth updated HDFS-516: --- Status: Patch Available (was: Open) Here's the initial patch. I have some pretty decent byte-consistency and integration tests wrapping it but no actual applications built using it. I started on a SequenceFileSearcher that implemented binary search but dealing with the sync points was sticky so it's not working yet (and commented out, with associated failing test). I'm hoping to get that working in the next couple weeks when I have some time as an example along with doing a performance comparison between nutch FsDirectory using DistributedFileSystem and RadFileSystem to see if this shows the gains that I'm thinking it will. My only change to core HDFS was to add getFile() to FSDatasetInterface so that the RadNode (plugged into datanode via ServicePlugin) can open FileChannels. > Low Latency distributed reads > - > > Key: HDFS-516 > URL: https://issues.apache.org/jira/browse/HDFS-516 > Project: Hadoop HDFS > Issue Type: New Feature >Reporter: Jay Booth >Priority: Minor > Attachments: radfs.patch > > Original Estimate: 168h > Remaining Estimate: 168h > > I created a method for low latency random reads using NIO on the server side > and simulated OS paging with LRU caching and lookahead on the client side. > Some applications could include lucene searching (term->doc and doc->offset > mappings are likely to be in local cache, thus much faster than nutch's > current FsDirectory impl and binary search through record files (bytes at > 1/2, 1/4, 1/8 marks are likely to be cached) -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (HDFS-516) Low Latency distributed reads
[ https://issues.apache.org/jira/browse/HDFS-516?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jay Booth updated HDFS-516: --- Attachment: radfs.patch > Low Latency distributed reads > - > > Key: HDFS-516 > URL: https://issues.apache.org/jira/browse/HDFS-516 > Project: Hadoop HDFS > Issue Type: New Feature >Reporter: Jay Booth >Priority: Minor > Attachments: radfs.patch > > Original Estimate: 168h > Remaining Estimate: 168h > > I created a method for low latency random reads using NIO on the server side > and simulated OS paging with LRU caching and lookahead on the client side. > Some applications could include lucene searching (term->doc and doc->offset > mappings are likely to be in local cache, thus much faster than nutch's > current FsDirectory impl and binary search through record files (bytes at > 1/2, 1/4, 1/8 marks are likely to be cached) -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (HDFS-516) Low Latency distributed reads
[ https://issues.apache.org/jira/browse/HDFS-516?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jay Booth updated HDFS-516: --- Attachment: radfs.tgz Oops. Shows what I get for posting at the end of the day, svn diff misses new files, duh. Here's a tarball of src/contrib/radfs. build.xml should play well with the rest of hadoop > Low Latency distributed reads > - > > Key: HDFS-516 > URL: https://issues.apache.org/jira/browse/HDFS-516 > Project: Hadoop HDFS > Issue Type: New Feature >Reporter: Jay Booth >Priority: Minor > Attachments: radfs.patch, radfs.tgz > > Original Estimate: 168h > Remaining Estimate: 168h > > I created a method for low latency random reads using NIO on the server side > and simulated OS paging with LRU caching and lookahead on the client side. > Some applications could include lucene searching (term->doc and doc->offset > mappings are likely to be in local cache, thus much faster than nutch's > current FsDirectory impl and binary search through record files (bytes at > 1/2, 1/4, 1/8 marks are likely to be cached) -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (HDFS-516) Low Latency distributed reads
[ https://issues.apache.org/jira/browse/HDFS-516?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jay Booth updated HDFS-516: --- Attachment: radfs.patch Full patch file from hdfs-trunk/. > Low Latency distributed reads > - > > Key: HDFS-516 > URL: https://issues.apache.org/jira/browse/HDFS-516 > Project: Hadoop HDFS > Issue Type: New Feature >Reporter: Jay Booth >Priority: Minor > Attachments: radfs.patch, radfs.patch, radfs.tgz > > Original Estimate: 168h > Remaining Estimate: 168h > > I created a method for low latency random reads using NIO on the server side > and simulated OS paging with LRU caching and lookahead on the client side. > Some applications could include lucene searching (term->doc and doc->offset > mappings are likely to be in local cache, thus much faster than nutch's > current FsDirectory impl and binary search through record files (bytes at > 1/2, 1/4, 1/8 marks are likely to be cached) -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (HDFS-516) Low Latency distributed reads
[ https://issues.apache.org/jira/browse/HDFS-516?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jay Booth updated HDFS-516: --- Attachment: (was: radfs.patch) > Low Latency distributed reads > - > > Key: HDFS-516 > URL: https://issues.apache.org/jira/browse/HDFS-516 > Project: Hadoop HDFS > Issue Type: New Feature >Reporter: Jay Booth >Priority: Minor > Attachments: radfs.patch > > Original Estimate: 168h > Remaining Estimate: 168h > > I created a method for low latency random reads using NIO on the server side > and simulated OS paging with LRU caching and lookahead on the client side. > Some applications could include lucene searching (term->doc and doc->offset > mappings are likely to be in local cache, thus much faster than nutch's > current FsDirectory impl and binary search through record files (bytes at > 1/2, 1/4, 1/8 marks are likely to be cached) -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (HDFS-516) Low Latency distributed reads
[ https://issues.apache.org/jira/browse/HDFS-516?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jay Booth updated HDFS-516: --- Attachment: (was: radfs.tgz) > Low Latency distributed reads > - > > Key: HDFS-516 > URL: https://issues.apache.org/jira/browse/HDFS-516 > Project: Hadoop HDFS > Issue Type: New Feature >Reporter: Jay Booth >Priority: Minor > Attachments: radfs.patch > > Original Estimate: 168h > Remaining Estimate: 168h > > I created a method for low latency random reads using NIO on the server side > and simulated OS paging with LRU caching and lookahead on the client side. > Some applications could include lucene searching (term->doc and doc->offset > mappings are likely to be in local cache, thus much faster than nutch's > current FsDirectory impl and binary search through record files (bytes at > 1/2, 1/4, 1/8 marks are likely to be cached) -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (HDFS-516) Low Latency distributed reads
[ https://issues.apache.org/jira/browse/HDFS-516?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jay Booth updated HDFS-516: --- Attachment: radfs.patch Attached wrong patch before, fixed > Low Latency distributed reads > - > > Key: HDFS-516 > URL: https://issues.apache.org/jira/browse/HDFS-516 > Project: Hadoop HDFS > Issue Type: New Feature >Reporter: Jay Booth >Priority: Minor > Original Estimate: 168h > Remaining Estimate: 168h > > I created a method for low latency random reads using NIO on the server side > and simulated OS paging with LRU caching and lookahead on the client side. > Some applications could include lucene searching (term->doc and doc->offset > mappings are likely to be in local cache, thus much faster than nutch's > current FsDirectory impl and binary search through record files (bytes at > 1/2, 1/4, 1/8 marks are likely to be cached) -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (HDFS-516) Low Latency distributed reads
[ https://issues.apache.org/jira/browse/HDFS-516?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jay Booth updated HDFS-516: --- Attachment: (was: radfs.patch) > Low Latency distributed reads > - > > Key: HDFS-516 > URL: https://issues.apache.org/jira/browse/HDFS-516 > Project: Hadoop HDFS > Issue Type: New Feature >Reporter: Jay Booth >Priority: Minor > Original Estimate: 168h > Remaining Estimate: 168h > > I created a method for low latency random reads using NIO on the server side > and simulated OS paging with LRU caching and lookahead on the client side. > Some applications could include lucene searching (term->doc and doc->offset > mappings are likely to be in local cache, thus much faster than nutch's > current FsDirectory impl and binary search through record files (bytes at > 1/2, 1/4, 1/8 marks are likely to be cached) -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (HDFS-516) Low Latency distributed reads
[ https://issues.apache.org/jira/browse/HDFS-516?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jay Booth updated HDFS-516: --- Attachment: (was: radfs.patch) > Low Latency distributed reads > - > > Key: HDFS-516 > URL: https://issues.apache.org/jira/browse/HDFS-516 > Project: Hadoop HDFS > Issue Type: New Feature >Reporter: Jay Booth >Priority: Minor > Original Estimate: 168h > Remaining Estimate: 168h > > I created a method for low latency random reads using NIO on the server side > and simulated OS paging with LRU caching and lookahead on the client side. > Some applications could include lucene searching (term->doc and doc->offset > mappings are likely to be in local cache, thus much faster than nutch's > current FsDirectory impl and binary search through record files (bytes at > 1/2, 1/4, 1/8 marks are likely to be cached) -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (HDFS-516) Low Latency distributed reads
[ https://issues.apache.org/jira/browse/HDFS-516?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jay Booth updated HDFS-516: --- Attachment: radfs.patch > Low Latency distributed reads > - > > Key: HDFS-516 > URL: https://issues.apache.org/jira/browse/HDFS-516 > Project: Hadoop HDFS > Issue Type: New Feature >Reporter: Jay Booth >Priority: Minor > Attachments: radfs.patch > > Original Estimate: 168h > Remaining Estimate: 168h > > I created a method for low latency random reads using NIO on the server side > and simulated OS paging with LRU caching and lookahead on the client side. > Some applications could include lucene searching (term->doc and doc->offset > mappings are likely to be in local cache, thus much faster than nutch's > current FsDirectory impl and binary search through record files (bytes at > 1/2, 1/4, 1/8 marks are likely to be cached) -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (HDFS-516) Low Latency distributed reads
[ https://issues.apache.org/jira/browse/HDFS-516?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jay Booth updated HDFS-516: --- Attachment: hdfs-516-20090824.patch Changed caching system to ehcache, which gives us a configurable api with better caching options. Fixed all known correctness bugs and added lots of test coverage. Got SequenceFileSearcher working, but it depends on HADOOP-6196 -- I could fold the searcher code and searcher test into HADOOP-6196 if people are interested in the functionality. Added benchmarking utils, .sh scripts intended to be run from HADOOP_HOME/bin. Note, if anyone installs, run ant bin-package from project root before trying to run radfs unit tests. TODO: benchmark javadoc, pretty up codebase, change unit tests to JUnit 4 style, add filesystem contract unit test I'll get a fork going on github soon if anyone's interested in trying it out. > Low Latency distributed reads > - > > Key: HDFS-516 > URL: https://issues.apache.org/jira/browse/HDFS-516 > Project: Hadoop HDFS > Issue Type: New Feature >Reporter: Jay Booth >Priority: Minor > Attachments: hdfs-516-20090824.patch, radfs.patch > > Original Estimate: 168h > Remaining Estimate: 168h > > I created a method for low latency random reads using NIO on the server side > and simulated OS paging with LRU caching and lookahead on the client side. > Some applications could include lucene searching (term->doc and doc->offset > mappings are likely to be in local cache, thus much faster than nutch's > current FsDirectory impl and binary search through record files (bytes at > 1/2, 1/4, 1/8 marks are likely to be cached) -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (HDFS-516) Low Latency distributed reads
[ https://issues.apache.org/jira/browse/HDFS-516?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jay Booth updated HDFS-516: --- Attachment: hdfs-516-20090831.patch New patch, IPC server was too slow for IO operations (like 40 times slower than DFS without caching) so I wrote a custom ByteServer that's streamlined to avoid object creation or byte copying whenever possible and defaults to tcp nodelay. Client connections pool using commons-pool. Uses static methods in hdfs.rad.ByteServiceProtocol for all serialization, faster than reflection. On the laptop in pseudodistributed, I'm seeing 5X faster than DFS for random searches. Refactored a bunch on the client side, eliminated a few redundant classes, still need to make lookahead happen via a separate thread in caching byteservice and tweak a couple things in ByteServer for performance, then this thing will be pretty fast. I'm gonna run some numbers on EC2 tonight/tomorrow and see what I come up with. Also, cleaned up unit tests to JUnit 4 and added some javadoc, probably missed a bunch of places and could certainly expand on all of it. Haven't added license to the header of every file yet, license explicitly granted here, will get to that for next patch. > Low Latency distributed reads > - > > Key: HDFS-516 > URL: https://issues.apache.org/jira/browse/HDFS-516 > Project: Hadoop HDFS > Issue Type: New Feature >Reporter: Jay Booth >Priority: Minor > Attachments: hdfs-516-20090824.patch, hdfs-516-20090831.patch, > radfs.patch > > Original Estimate: 168h > Remaining Estimate: 168h > > I created a method for low latency random reads using NIO on the server side > and simulated OS paging with LRU caching and lookahead on the client side. > Some applications could include lucene searching (term->doc and doc->offset > mappings are likely to be in local cache, thus much faster than nutch's > current FsDirectory impl and binary search through record files (bytes at > 1/2, 1/4, 1/8 marks are likely to be cached) -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (HDFS-516) Low Latency distributed reads
[ https://issues.apache.org/jira/browse/HDFS-516?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jay Booth updated HDFS-516: --- Attachment: (was: hdfs-516-20090831.patch) > Low Latency distributed reads > - > > Key: HDFS-516 > URL: https://issues.apache.org/jira/browse/HDFS-516 > Project: Hadoop HDFS > Issue Type: New Feature >Reporter: Jay Booth >Priority: Minor > Attachments: hdfs-516-20090912.patch > > Original Estimate: 168h > Remaining Estimate: 168h > > I created a method for low latency random reads using NIO on the server side > and simulated OS paging with LRU caching and lookahead on the client side. > Some applications could include lucene searching (term->doc and doc->offset > mappings are likely to be in local cache, thus much faster than nutch's > current FsDirectory impl and binary search through record files (bytes at > 1/2, 1/4, 1/8 marks are likely to be cached) -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (HDFS-516) Low Latency distributed reads
[ https://issues.apache.org/jira/browse/HDFS-516?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jay Booth updated HDFS-516: --- Attachment: hdfs-516-20090912.patch New patch: * README file with instructions for eclipse and running with hadoop * Javadoc and JUnit 4-style test cases, some new test cases * Benchmarks for random reads, binary search, and streaming * Illustrated 100% performance increase in streaming case, somehow, from 213 seconds to 112 seconds to stream 1GB from a remote HDFS file. Reproduced a couple times, using 16MB of cache with the lookahead mechanism. I suspect it uses a lot more CPU than conventional streaming, but still, that's a lot faster. * No longer requires any change to HDFS code, module is now entirely in contrib * Cleans up file handles better * Handles remote disconnect better from the client side What are people's thoughts on getting this into 0.21? It shows a lot of promise as far as performance but hasn't been tested on larger clusters, I'd be confident up to 200 nodes or so and then I'd start getting nervous. Given that it lives entirely in contrib and needs to be actively configured to turn it on, could we include this for 0.21? Anyone want to try running the benchmarks? I'll run the benchmarks one last time tomorrow to sanity check the latest patch (changed a couple things since the last time I ran in a cluster), then maybe we could consider committing? > Low Latency distributed reads > - > > Key: HDFS-516 > URL: https://issues.apache.org/jira/browse/HDFS-516 > Project: Hadoop HDFS > Issue Type: New Feature >Reporter: Jay Booth >Priority: Minor > Attachments: hdfs-516-20090912.patch > > Original Estimate: 168h > Remaining Estimate: 168h > > I created a method for low latency random reads using NIO on the server side > and simulated OS paging with LRU caching and lookahead on the client side. > Some applications could include lucene searching (term->doc and doc->offset > mappings are likely to be in local cache, thus much faster than nutch's > current FsDirectory impl and binary search through record files (bytes at > 1/2, 1/4, 1/8 marks are likely to be cached) -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (HDFS-516) Low Latency distributed reads
[ https://issues.apache.org/jira/browse/HDFS-516?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jay Booth updated HDFS-516: --- Attachment: (was: radfs.patch) > Low Latency distributed reads > - > > Key: HDFS-516 > URL: https://issues.apache.org/jira/browse/HDFS-516 > Project: Hadoop HDFS > Issue Type: New Feature >Reporter: Jay Booth >Priority: Minor > Attachments: hdfs-516-20090912.patch > > Original Estimate: 168h > Remaining Estimate: 168h > > I created a method for low latency random reads using NIO on the server side > and simulated OS paging with LRU caching and lookahead on the client side. > Some applications could include lucene searching (term->doc and doc->offset > mappings are likely to be in local cache, thus much faster than nutch's > current FsDirectory impl and binary search through record files (bytes at > 1/2, 1/4, 1/8 marks are likely to be cached) -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (HDFS-516) Low Latency distributed reads
[ https://issues.apache.org/jira/browse/HDFS-516?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jay Booth updated HDFS-516: --- Attachment: (was: hdfs-516-20090824.patch) > Low Latency distributed reads > - > > Key: HDFS-516 > URL: https://issues.apache.org/jira/browse/HDFS-516 > Project: Hadoop HDFS > Issue Type: New Feature >Reporter: Jay Booth >Priority: Minor > Attachments: hdfs-516-20090912.patch > > Original Estimate: 168h > Remaining Estimate: 168h > > I created a method for low latency random reads using NIO on the server side > and simulated OS paging with LRU caching and lookahead on the client side. > Some applications could include lucene searching (term->doc and doc->offset > mappings are likely to be in local cache, thus much faster than nutch's > current FsDirectory impl and binary search through record files (bytes at > 1/2, 1/4, 1/8 marks are likely to be cached) -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (HDFS-516) Low Latency distributed reads
[ https://issues.apache.org/jira/browse/HDFS-516?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jay Booth updated HDFS-516: --- Resolution: Won't Fix Status: Resolved (was: Patch Available) This work didn't really go anywhere, I tried multithreading on the client side to make a request-per-packet approach viable, but it wound up being even slower than singlethreaded request-per-packet. If there's interest in the client-side byte cache, I could create another issue with just that improvement. > Low Latency distributed reads > - > > Key: HDFS-516 > URL: https://issues.apache.org/jira/browse/HDFS-516 > Project: Hadoop HDFS > Issue Type: New Feature >Reporter: Jay Booth >Priority: Minor > Attachments: hdfs-516-20090912.patch, radfs.odp > > Original Estimate: 168h > Remaining Estimate: 168h > > I created a method for low latency random reads using NIO on the server side > and simulated OS paging with LRU caching and lookahead on the client side. > Some applications could include lucene searching (term->doc and doc->offset > mappings are likely to be in local cache, thus much faster than nutch's > current FsDirectory impl and binary search through record files (bytes at > 1/2, 1/4, 1/8 marks are likely to be cached) -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
