Google "Hadoop WholeFileInputFormat" or search it in book "Hadoop: The Definitive Guide" Yong
Date: Tue, 31 Dec 2013 09:39:58 +0800 Subject: Re: any suggestions on IIS log storage and analysis? From: raofeng...@gmail.com To: user@hadoop.apache.org Thanks, Yong! The dependence never cross files, but since HDFS splits files into blocks, it may cross blocks, which makes it difficult to write MR job. I don't quite understand what you mean by "WholeFileInputFormat ". Actually, I have no idea how to deal with dependence across blocks. 2013/12/31 java8964 <java8...@hotmail.com> I don't know any example of IIS log files. But from what you described, it looks like analyzing one line of log data depends on some previous lines data. You should be more clear about what is this dependence and what you are trying to do. Just based on your questions, you still have different options, which one is better depends on your requirements and data. 1) You know the existing default TextInputFormat not suitable for your case, you just need to find alternatives, or write your own. 2) If the dependences never cross the files, just cross lines, you can use WholeFileInputFormat (No such class coming from Hadoop itself, but very easy implemented by yourself)3) If the dependences cross the files, then you maybe have to enforce your business logics in reducer side, instead of mapper side. Without knowing your detail requirements of this dependence, it is hard to give you more detail, but you need to find out what are good KEY candidates for your dependence logic, send the data based on that to the reducers, and enforce your logic on the reducer sides. If one MR job is NOT enough to solve your dependence, you may need chain several MR jobs together. Yong Date: Mon, 30 Dec 2013 15:58:57 +0800 Subject: any suggestions on IIS log storage and analysis? From: raofeng...@gmail.com To: user@hadoop.apache.org Hi, HDFS splits files into blocks, and mapreduce runs a map task for each block. However, Fields could be changed in IIS log files, which means fields in one block may depend on another, and thus make it not suitable for mapreduce job. It seems there should be some preprocess before storing and analyzing the IIS log files. We plan to parse each line to the same fields and store in Avro files with compression. Any other alternatives? Hbase? or any suggestions on analyzing IIS log files? thanks!