Hi, you can limit the heap-size for libhdfs .. export LIBHDFS_OPTS="-Xmx128m"
best, Alex On Wed, Nov 30, 2011 at 3:25 PM, Zhanwei Wang <had...@wangzw.org> wrote: > Hi everyone**** > > ** ** > > I have a problem when I want to enable our distributed system to access > hdfs.**** > > ** ** > > The background:**** > > In our system, we have 4 ~ 6 segment instance on one physical node, and > each segment forks a new process to deal with a new session. So if a client > connect to our system, we will have 4~6 processes working on this session > in one physical node. We need to handle 250 sessions at the same time, that > means 1000~1500 processes on one physical node. That is ok for our system. > **** > > ** ** > > Now we want to enable our system to access hdfs. We use libhdfs to do > that. It works ok if we only deal with a few concurrent session, but if too > many client connect to the system, the system cannot access hdfs anymore.* > *** > > The reason is libhdfs create one JVM for each process, and our machine > cannot afford 1000~1500 JVMs. Libhdfs report the error that it cannot > create JVM due to memory limitation. **** > > ** ** > > We try to walk around with this problem using a few processes as proxies > to access hdfs and exchange data with other processes, but proxy process > could become the performance bottleneck.**** > > ** ** > > My questions:**** > > **1) **Can different processes share JVM if we use libhdfs to access > hdfs?**** > > **2) **Is there any other good solution? Maybe I am stupid to find > it out. **** > > **3) **Is there any ongoing project to implement a C/C++ hdfs > client, without JVM? Is it a good idea to create such a project? If it is, > I will be very glad to contribute my time.**** > > ** ** > > ** ** > > Thanks**** > > ** ** > > ** ** > > ** ** > > Best Regards**** > > ** ** > > ------------------------------**** > > ** ** > > Zhanwei Wang**** > > ** ** > -- Alexander Lorenz http://mapredit.blogspot.com *P **Think of the environment: please don't print this email unless you really need to.*
