Re: Monitoring hadoop?
You might also want to check out Chukwa, which is a Hadoop subproject aiming to do scalable monitoring and analysis of large clusters, particularly Hadoop clusters. It's still in development, so you'd be able to have an influence on the design and priorities that come out. --Ari On Fri, Jun 5, 2009 at 6:55 AM, Anthony McCulleyamccul...@gmail.com wrote: Hey all, I'm currently tasked to come up with a web/flex-based visualization/monitoring system for a cloud system using hadoop as part of a university research project. I was wondering if I could elicit some feedback from all of you with regards to: - If you were an engineer of a cloud system running hadoop, what information would you be interested in capturing, viewing, monitoring, etc? - Is there any sort of real-time stats or monitoring currently available for hadoop? if so, is in a web-friendly format? Thanks in advance, - Anthony -- Ari Rabkin asrab...@gmail.com UC Berkeley Computer Science Department
Re: HDFS as a logfile ??
Everything gets dumped into the same files. We don't assume anything at all about the format of the input data; it gets dumped into Hadoop sequence files, tagged with some metadata to say what machine and app it came from, and where it was in the original stream. There is a slight penalty from the log-to-local disk. In practice, you often want a local copy anyway, both for redundancy and because it's much more convenient for human inspection. Having a separate collector process is indeed inelegant. However, HDFS copes badly with many small files, so that pushes you to merge entries across either hosts or time partitions. And since HDFS doesn't have a flush(), having one log per source means that log entries don't become visible quickly enough. Hence, collectors. It isn't gorgeous, but it seems to work fine in practice. On Mon, Apr 13, 2009 at 8:01 AM, Ricky Ho r...@adobe.com wrote: Ari, thanks for your note. Like to understand more how Chukwa group log entries ... If I have appA running in machine X, Y and appB running in machine Y, Z. Each of them calling the Chukwa log API. Do I have all entries going in the same HDFS file ? or 4 separated HDFS files based on the App/Machine combination ? If the answer of first Q is yes, then what if appA and appB has different format of log entries ? If the answer of second Q is yes, then are all these HDFS files cut at the same time boundary ? Looks like in Chukwa, application first log to a daemon, which buffer-write the log entries into a local file. And there is a separate process to ship these data to a remote collector daemon which issue the actual HDFS write. I observe the following overhead ... 1) The overhead of extra write to local disk and ship the data over to the collector. If HDFS supports append, the application can then go directly to the HDFS 2) The centralized collector establish a bottleneck to the otherwise perfectly parallel HDFS architecture. Am I missing something here ? -- Ari Rabkin asrab...@gmail.com UC Berkeley Computer Science Department
Re: HDFS as a logfile ??
Chukwa is a Hadoop subproject aiming to do something similar, though particularly for the case of Hadoop logs. You may find it useful. Hadoop unfortunately does not support concurrent appends. As a result, the Chukwa project found itself creating a whole new demon, the chukwa collector, precisely to merge the event streams and write it out, just once. We're set to do a release within the next week or two, but in the meantime you can check it out from SVN at https://svn.apache.org/repos/asf/hadoop/chukwa/trunk --Ari On Fri, Apr 10, 2009 at 12:06 AM, Ricky Ho r...@adobe.com wrote: I want to analyze the traffic pattern and statistics of a distributed application. I am thinking of having the application write the events as log entries into HDFS and then later I can use a Map/Reduce task to do the analysis in parallel. Is this a good approach ? In this case, does HDFS support concurrent write (append) to a file ? Another question is whether the write API thread-safe ? Rgds, Ricky -- Ari Rabkin asrab...@gmail.com UC Berkeley Computer Science Department
Re: Chukwa documentation
Howdy. You do not need torque. It's not even helpful, as far as I know. You don't need a database, but if you don't have one, you'd probably need to do a bit more work to analyze the collected data in HDFS. If you were going to be using MapReduce for analysis anyway, that's probably a non-issue for you. We're working on documentation, but it's sort of chasing a moving target, since the Chukwa codebase and configuration interfaces are still in flux. --Ari On Tue, Feb 3, 2009 at 5:00 PM, xavier.quint...@orange-ftgroup.com wrote: Hi Everybody, I don't know if there is a mail list for Chukwa so I apologies in advance if this is not the right place to ask my questions. I have the following questions and comments: It was simple the configuration of the collector and the agent. However, there is other features that are not documented it all like: - torque (Do I have to install torque before? Yes? No? and Why?), - database, (Do I have to have a DB?) -what is queueinfo.properties, which kind of information provides me? -and there is more stuff that I need to dig in the code to understand. Could somebody update the documentation from Chukwa?. -- Ari Rabkin asrab...@gmail.com UC Berkeley Computer Science Department
Re: Append to Files..
File append is a major change, not a small bugfix. Probably, you need to bite the bullet and upgrade to a newer JDK. :( On Mon, Dec 1, 2008 at 4:29 AM, Sandeep Dhawan, Noida [EMAIL PROTECTED] wrote: Hello, I am currently using hadoop-0.18.0. I am not able to append files in DFS. I came across a fix which was done on version 0.19.0 (http://issues.apache.org/jira/browse/HADOOP-1700). But I cannot migrate to 0.19.0 version because it runs on JDK 1.6 and I have to stick to JDK 1.5 Therefore, I would like to know, if there is patch available for this bug for 0.18.0. Any assistance in this matter will be greatly appreciated. Eagerly waiting for your response. Thanks, Sandeep -- Ari Rabkin [EMAIL PROTECTED] UC Berkeley Computer Science Department
Re: Chukwa Support
Hey, glad to see that Chukwa is getting some attention and interest. An adaptor is a Java class that implements org.apache.hadoop.chukwa.datacollection.adaptor.Adaptor. The Adaptor javadoc should tell you what the methods need to do. You start an adaptor by sending a command of the form add [classname] [parameters] 0 to the Chukwa agent over TCP. By default, Chukwa listens on port 9093. I don't believe HICC has been publicly released yet, due to annoying GPL/Apache license incompatibilities. On Wed, Oct 15, 2008 at 3:27 PM, Alex Loddengaard [EMAIL PROTECTED] wrote: I'm trying to play with Chukwa, but I'm struggling to get anything going. I've been operating off of the wiki entry ( http://wiki.apache.org/hadoop/Chukwa_Quick_Start), making revisions as I go along. It's unclear to me how to 1) create an adapter and 2) start HICC (see the wiki for more information). I've gone through the wiki and created 'Document TODO:' items for each issue that I've run in to. Could someone familiar with Chukwa either comment on this issues on the mailing list or update the wiki? Chukwa seems like a great tool, but it's unclear exactly how to get it up and running. -- Ari Rabkin [EMAIL PROTECTED] UC Berkeley Computer Science Department
Re: Hadoop Profiling!
That code is in, unfortunately it doesn't quite solve the problem; you'd need to do some more work. You'd have to write subclasses that spit out the statistics you want. Then set the appropriate options in hadoop-site, so that those classes get loaded. On Wed, Oct 8, 2008 at 12:30 PM, George Porter [EMAIL PROTECTED] wrote: Hi Ashish, I believe that Ari committed two instrumentation classes, TaskTrackerInstrumentation and JobTrackerInstrumentation, (both in src/mapred/org/apache/hadoop/mapred) that can give you information on when components of your M/R jobs start and stop. I'm in the process of writing some additional instrumentation APIs that collect timing information about the RPC and HDFS layers, and will hopefully be able to submit a patch in a few weeks. Thanks, George Ashish Venugopal wrote: Are you interested in simply profiling your own code (in which case you can clearly use what ever java profiler you want), or your construction of the MapReduce job, ie how much time is being spent in the Map vs the sort vs the shuffle vs the Reduce. I am not aware of a good solution to the second problem, can anyone comment? Ashish On Wed, Oct 8, 2008 at 12:06 PM, Stefan Groschupf [EMAIL PROTECTED] wrote: Just run your map reduce job local and connect your profiler. I use yourkit. Works great! You can profile your map reduce job running the job in local mode as ant other java app as well. However we also profiled in a grid. You just need to install the yourkit agent into the jvm of the node you want to profile and than you connect to the node when the job runs. However you need to time things well, since the task jvm is shutdown as soon your job is done. Stefan ~~~ 101tec Inc., Menlo Park, California web: http://www.101tec.com blog: http://www.find23.net On Oct 8, 2008, at 11:27 AM, Gerardo Velez wrote: Hi! I've developed a Map/Reduce algorithm to analyze some logs from web application. So basically, we are ready to start QA test phase, so now, I would like to now how efficient is my application from performance point of view. So is there any procedure I could use to do some profiling? Basically I need basi data, like time excecution or code bottlenecks. Thanks in advance. -- Gerardo Velez -- George Porter, Sun Labs/CTO Sun Microsystems - San Diego, Calif. [EMAIL PROTECTED] 1.858.526.9328 -- Ari Rabkin [EMAIL PROTECTED] UC Berkeley Computer Science Department
