That is awesome! Let know your results when you get a chance. Aaron
On Mon, Aug 10, 2015 at 9:21 AM, Ravikumar Govindarajan < [email protected]> wrote: > Hadoop 2.7.1 is out and now handles mixed storage… A single > data-node/shard-server can run HDDs & SSDs together… > > More about this here… > > > http://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-hdfs/ArchivalStorage.html > > The policy I looked for was "SSD_One". The first-copy of index-data placed > on local-machine will be stored in SSD. The second & third-copies stored on > other machines will be in HDDs… > > This eliminates need for mixed setup using RACK1 & RACK2 I previously > thought of. Hadoop 2.7.1 helps me to achieve this in a single cluster of > machines running data-nodes + shard-servers > > Every machine stores primary copy in SSDs. Writes, Searches, Merges all > take advantage of it, while replication can be relegated to slower but > bigger capacity HDDs. These HDDs also serve as an online backup of less > fault-tolerant SSDs > > We have ported our in-house blur extension to hadoop-2.7.1. Will update on > test results shortly > > -- > Ravi > > On Mon, Jun 22, 2015 at 6:18 PM, Aaron McCurry <[email protected]> wrote: > > > On Thu, Jun 18, 2015 at 8:55 AM, Ravikumar Govindarajan < > > [email protected]> wrote: > > > > > Apologize for resurrecting this thread… > > > > > > One problem of lucene is OS buffer-cache pollution during segment > merges, > > > as documented here > > > > > > http://blog.mikemccandless.com/2010/06/lucene-and-fadvisemadvise.html > > > > > > This problem could occur in Blur, when short-circuit reads are > enabled... > > > > > > > True but Blur deals with this issue by not allowing (by default) the > merges > > to effect the Block Cache. > > > > > > > > > > My take on this… > > > > > > It may be possible to overcome the problem by simply re-directing > > > merge-read requests to a node other than local-node instead of fancy > > stuff > > > like O_DIRECT, FADVISE etc... > > > > > > > I have always thought of having merge occur in a Mapreduce (or Yarn) job > > instead of locally. > > > > > > > > > > In a mixed setup, this means merge requests need to be diverted to > > low-end > > > Rack2 machines {running only data-nodes} while short-circuit read > > requests > > > will continue to be served from high-end Rack1 machines {running both > > > shard-server and data-nodes} > > > > > > Hadoop 2.x provides a cool read-API "seekToNewSource" > > > API documentation says "Seek to given position on a node other than the > > > current node" > > > > > > > From blur code, it's just enough if we open a new FSDataInputStream for > > > merge-reads and issue seekToNewSource call. Once merges are done, it > can > > > closed & discarded… > > > > > > Please let know your view-points on this… > > > > > > > We could do this, but I find that reading the TIM file types over the > wire > > during a merge causes a HUGE slow down in merge performance. The fastest > > way to merge is to copy the TIM files involved in the merge locally to > run > > the merge and then delete them after the fact. > > > > Aaron > > > > > > > > > > -- > > > Ravi > > > > > > On Mon, Mar 9, 2015 at 5:45 PM, Ravikumar Govindarajan < > > > [email protected]> wrote: > > > > > > > > > > > On Sat, Mar 7, 2015 at 11:00 AM, Aaron McCurry <[email protected]> > > > wrote: > > > > > > > >> > > > >> I thought the normal hdfs replica rules were once local. One remote > > rack > > > >> once same rack. > > > >> > > > > > > > > Yes. One copy is local & other two copies on the same remote rack. > > > > > > > > How did > > > >> land on your current configuration ? > > > > > > > > > > > > When I was evaluating disk-budget, we were looking at 6 expensive > > drives > > > > per machine. It lead me to think what those 6 drives would do & how > we > > > can > > > > reduce the cost. Then stumbled on this two-rack setup and now we need > > > only > > > > 2 such drives... > > > > > > > > Apart from reduced disk-budget & write-overhead on cluster, it also > > helps > > > > in greater availability as rack-failure would be recoverable... > > > > > > > > -- > > > > Ravi > > > > > > > > > > > > > >
