Re: A question about RPC

2011-10-11 Thread Todd Lipcon
On Tue, Oct 11, 2011 at 12:44 PM, Stephan Gammeter wrote: > TL;DR > .proto + protocol buffer plugins for generating rpc clients and servers is > really handy. If writables or protobufs are faster needs to be benchmarked, > but probably both serialize faster than one can write. Looking at perform

Re: A question about RPC

2011-10-11 Thread Stephan Gammeter
When i started writing i was not aware of zmq yet, so the connection layer is just written using boost, but it would be quite simple to replace that with zmq, just have not gotten around to it yet. > i really don't see the need for people reinventing the low level rcp stuff over and over again.

Re: blocks with a huge size?

2011-10-11 Thread Will Maier
Hi Vincent- On Tue, Oct 11, 2011 at 07:12:55PM +0200, Vincent Boucher wrote: > Indeed, it's a similar setup, at a smaller scale: Tier2 for CMS at UCLouvain, > Belgium, with data&sim from the whole collaboration; stageout files from the > Grid for our local users ... It's always nice to see other

Re: blocks with a huge size?

2011-10-11 Thread Vincent Boucher
Hi Matt- Le Tuesday 11 October 2011, GOEKE, MATTHEW (AG/1000) a écrit : > Vincent, > > Just to clarify, are you using hdfs/Hadoop primarily as a storage layer > with little to no MR? I know that is the case for Will's team and I was > kind of assuming that as another tier2 you would probably be do

Re: blocks with a huge size?

2011-10-11 Thread Vincent Boucher
Hi Will- Le Tuesday 11 October 2011, Will Maier a écrit : > As I described earlier, our ~1PB cluster is very similar: some large > servers, some small ones. We store data recorded at and simulated for the > CMS detector at the LHC, as well as the products of our users' physics > analysis. Like you

Re: blocks with a huge size?

2011-10-11 Thread Todd Lipcon
Hi Vincent, Maybe you'd be better off using something like MogileFS for this application? -Todd On Tue, Oct 11, 2011 at 1:39 AM, Vincent Boucher wrote: > Hello again, > > * Our case: > > Most of the files we are dealing with are 10GB wide. Our hdfs configuration > would be the following: data i

Re: FileSystem API for 0.19

2011-10-11 Thread Arko Provo Mukherjee
Hi, Yeah, I fixed the problem, I compiled against the incorrect jars. The reason I use 0.19 is that the cloud I have access to is still being maintained at 0.19 :( Thanks much! Warm regards Arko On Oct 10, 2011, at 9:34 PM, Harsh J wrote: > Arko, > > It helps if we know what "doesn't work"

Re: Is there any Namenode block scheduler which considers the block load

2011-10-11 Thread Harsh J
Anil, Moving this to hdfs-dev@. The transfer thread load is definitely considered when building the DN replication pipeline. See the method ReplicationTargetChooser#isGoodTarget(…), which is called during each type of choice (local node, local rack, remote rack or random). One part of its analysi

RE: blocks with a huge size?

2011-10-11 Thread GOEKE, MATTHEW (AG/1000)
Vincent, Just to clarify, are you using hdfs/Hadoop primarily as a storage layer with little to no MR? I know that is the case for Will's team and I was kind of assuming that as another tier2 you would probably be doing the same. The reason why I ask is the design decisions you have made so far

Is there any Namenode block scheduler which considers the block load

2011-10-11 Thread AnilKumar B
Scenario: If I run huge number of jobs(all these jobs will use the same resources(input files)) on mini cluster(say 10-15 nodes), then every time namenode returning the first block of nearest data node. So in this case all the clients are trying to do read/write operations on same block. So is the

Re: blocks with a huge size?

2011-10-11 Thread Will Maier
Hi Vincent- On Tue, Oct 11, 2011 at 10:39:46AM +0200, Vincent Boucher wrote: > Most of the files we are dealing with are 10GB wide. Our hdfs configuration > would be the following: data is stored on mass storage servers (10x50TB) each > with RAID6; no replica for data. > > With a 64MB hdfs block

Re: blocks with a huge size?

2011-10-11 Thread Vincent Boucher
Le Tuesday 11 October 2011, Sudharsan Sampath a écrit : > Hello, > > I donot understand "having one server down will corrupt the full > filesystem". HDFS allows for high availability based on the replication > factor that you set. > > Do you have a constraint on setting the replication factor more

Re: blocks with a huge size?

2011-10-11 Thread Sudharsan Sampath
Hello, I donot understand "having one server down will corrupt the full filesystem". HDFS allows for high availability based on the replication factor that you set. Do you have a constraint on setting the replication factor more than 1 ? Thanks Sudharsan S On Tue, Oct 11, 2011 at 2:09 PM, Vince

blocks with a huge size?

2011-10-11 Thread Vincent Boucher
Hello again, * Our case: Most of the files we are dealing with are 10GB wide. Our hdfs configuration would be the following: data is stored on mass storage servers (10x50TB) each with RAID6; no replica for data. With a 64MB hdfs block size, it is extremely likely that all of our 10GB files wi

Re: hdfs file - datanode association

2011-10-11 Thread Vincent Boucher
It seems there's such a solution in 0.21, let's see how it integrates with 0.20.203 https://issues.apache.org/jira/browse/HDFS-385 http://hadoopblog.blogspot.com/2009/09/hdfs-block-replica-placement-in-your.html Do you have any hint about the release date of the stable 0.21th version? Vincent