Hi guys,
I wrote a patch to use MultiFIleInputFOrmat in hadoop 0.19 where
CombineInputFormat is not available.
We needed this for a system that had too many small files, which causes too
many mappers/reducers,
which causes overhead...
---------- Forwarded message ----------
From: Roberto Congiu <[email protected]>
Date: Mon, Feb 1, 2010 at 11:02 AM
Subject: Re: HIVE-74 and CombineFileInputFormat on pre-0.20 hadoop
To: Namit Jain <[email protected]>
Cc: "[email protected]" <[email protected]>
Reviving this old thread...just found the time to work on this...
I have a patch for using MultiFIleInputFormat in hadoop 0.19 as
CombineHiveInputFormat - setting
set hive.input.format=org.apache.hadoop.hive.ql.io.CombineHiveInputFormat;
(or the equivalent setting on hive-site.xml) will have hive use
MultiFIleInputFormat, packing many small files in
mapred.multifileinputformat.splits splits (if set), or guessing the size by
dividing the total input size by the DFS block size.
Patch attached...I checked that it passes all unit tests according to
http://wiki.apache.org/hadoop/Hive/HowToContribute#Setting_up_Eclipse_Development_Environment_.28Optional.29
On Wed, Sep 30, 2009 at 4:34 AM, Namit Jain <[email protected]> wrote:
> That’s right
>
>
>
>
> On 9/30/09 12:07 AM, "Roberto Congiu" <[email protected]> wrote:
>
> Hi Namit,
> that's what I thought. Right now unfortunately we can't migrate to 0.20.
> I realize we lose data locality but as you said, it would still be
> considerably better than now.
>
> I had a look at the shim code, shouldn't be difficult since it would
> be basically mimicking CombineFileInputFormat.
>
> Once I add the appropriate logic to the shim, I have to set
> hive.input.format to
> org.apache.hadoop.hive.ql.io.CombineHiveInputFormat to have hive
> actually use it, right ?
>
> Roberto
>
> 2009/9/29 Namit Jain <[email protected]>:
> > Hi Roberto,
> >
> > Talked with Raghu and Dhruba – it is possible to do so using
> > MutliFileInputFormat,
> > But the performance will not be very good since MutliFileInputFormat does
> > not
> > provide any locality. However, it will still be much better than the
> problem
> > you are
> > running into right now.
> >
> > Can you move to hadoop-0.20 ? That might be simpler.
> >
> > If not, you can definitely implement the shim using MultiFileInputFormat
> for
> > 0.19
> > (which should work even with 0.17). Do you need some help in
> understanding
> > the
> > current shim code ?
> >
> > Thanks,
> > -namit
> >
> >
> >
> >
> >
> > On 9/29/09 10:53 AM, "Namit Jain" <[email protected]> wrote:
> >
> > Just checked – CombineFileInputFormat and a lot of other related stuff
> went
> > to hadoop 0.20
> > So, it would be very difficult to add this for 0.19
> >
> >
> >
> > From: Namit Jain [mailto:[email protected]] <[email protected]]>
> > Sent: Monday, September 28, 2009 10:30 PM
> > To: [email protected]; [email protected]
> > Subject: Re: HIVE-74 and CombineFileInputFormat on pre-0.20 hadoop
> >
> > I am not sure whether CombineFileInputFormat (in hadoop) is available in
> > 0.19 -
> > If it is, we can add it, otherwise it will be very difficult.
> >
> >
> >
> > On 9/28/09 7:06 PM, "Raghu Murthy" <[email protected]> wrote:
> > Can we add MultiFileInputFormat as the CombineFileInputFormatShim for
> > hadoop-0.19?
> >
> > On 9/28/09 6:57 PM, "Roberto Congiu" <[email protected]> wrote:
> >
> >> Hi guys,
> >> I've been working on integrating hive with a legacy file format we use
> >> here. I wrote the appropriate InputFormat and SerDe and everything
> >> works, but it's painfully slow.
> >> The reason is that the files I am reading are many and hive uses one
> >> mapper for every file.
> >> I saw the HIVE-74 patches but those use CombineFileInputFormat which
> >> is available on hadoop 0.20...but we use 0.19. Is there any reason the
> >> same goal could not be achieved using the deprecated (but present <
> >> 0.20) MultiFileInputFormat ?
> >>
> >> Thanks,
> >> Roberto
> >
> >
> >
>
>
Index: shims/src/0.19/java/org/apache/hadoop/hive/shims/Hadoop19Shims.java
===================================================================
--- shims/src/0.19/java/org/apache/hadoop/hive/shims/Hadoop19Shims.java (revision 903519)
+++ shims/src/0.19/java/org/apache/hadoop/hive/shims/Hadoop19Shims.java (working copy)
@@ -17,20 +17,29 @@
*/
package org.apache.hadoop.hive.shims;
+import java.io.DataInput;
+import java.io.DataOutput;
import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.fs.FileStatus;
import org.apache.hadoop.fs.FileSystem;
import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.fs.PathFilter;
+import org.apache.hadoop.fs.FileUtil;
import org.apache.hadoop.io.Text;
import org.apache.hadoop.hdfs.MiniDFSCluster;
import org.apache.hadoop.mapred.InputFormat;
import org.apache.hadoop.mapred.JobClient;
import org.apache.hadoop.mapred.JobConf;
import org.apache.hadoop.mapred.TaskCompletionEvent;
-import org.apache.hadoop.mapred.TaskAttemptID;
import org.apache.hadoop.mapred.TaskID;
import java.io.IOException;
+import java.lang.reflect.Constructor;
+import org.apache.hadoop.mapred.InputSplit;
+import org.apache.hadoop.mapred.MultiFileInputFormat;
+import org.apache.hadoop.mapred.MultiFileSplit;
+import org.apache.hadoop.mapred.Reporter;
+import org.apache.hadoop.mapred.RecordReader;
/**
* Implemention of shims against Hadoop 0.19.0
@@ -97,20 +106,363 @@
}
public HadoopShims.CombineFileInputFormatShim getCombineFileInputFormat() {
- return null;
+ return new CombineFileInputFormatShim() {
+
+ @Override
+ public RecordReader getRecordReader(InputSplit split, JobConf arg1,
+ Reporter arg2) throws IOException {
+ throw new IOException(
+ "CombineFileInputFormat.getRecordReader not needed.");
+ }
+ };
}
public String getInputFormatClassName() {
return "org.apache.hadoop.hive.ql.io.HiveInputFormat";
}
+ /**
+ * MultiFileShim code here
+ *
+ *
+ */
+ public abstract static class CombineFileInputFormatShim<K, V> extends
+ MultiFileInputFormat<K, V> implements
+ HadoopShims.CombineFileInputFormatShim<K, V> {
+
+ /**
+ * gets the input paths from static method in parent class. Same code in the
+ * hadoop20shim, adapted for @link{MultiFileInputFormat}
+ *
+ * @param conf
+ * @return Path[] of all files to be processed.
+ */
+ public Path[] getInputPathsShim(JobConf conf) {
+ Path[] paths;
+ try {
+ paths = MultiFileInputFormat.getInputPaths(conf);
+ LOG.debug("MultiFileInputFormat returned " + paths.length + " paths.");
+ } catch (Exception e) {
+ throw new RuntimeException(e);
+ }
+ Path[] newPaths = new Path[paths.length];
+ // remove file:
+ for (int pos = 0; pos < paths.length; pos++) {
+ newPaths[pos] = new Path(paths[pos].toString().substring(5));
+ }
+ return newPaths;
+ }
+
+ /**
+ * Not supported by MultiFileInputFormat so it doesn't do anything
+ *
+ * @param conf
+ * @param filters
+ */
+ public void createPool(JobConf conf, PathFilter... filters) {
+ LOG.debug("createPool called.");
+ }
+
+ @Override
+ public InputSplitShim[] getSplits(JobConf job, int numSplits)
+ throws IOException {
+ long minSize = job.getLong("mapred.min.split.size", 0);
+
+ // For backward compatibility, let the above parameter be used
+ if (job.getLong("mapred.min.split.size.per.node", 0) == 0) {
+ super.setMinSplitSize(minSize);
+ }
+
+ if (job.getLong("mapred.min.split.size.per.rack", 0) == 0) {
+ super.setMinSplitSize(minSize);
+ }
+
+ numSplits = (int) job.getLong("mapred.multifileinputformat.splits",
+ estimateNumSplits(job, numSplits));
+
+ LOG.debug("Estimated splits (numSplits) = " + numSplits);
+ MultiFileSplit[] splits = (MultiFileSplit[]) super.getSplits(job,
+ numSplits);
+ LOG.debug("MultiFileSplit returned " + splits.length + " splits.");
+
+ InputSplitShim[] isplits = new InputSplitShim[splits.length];
+ for (int pos = 0; pos < splits.length; pos++) {
+ isplits[pos] = new InputSplitShim(job, splits[pos]);
+ }
+
+ return isplits;
+ }
+
+ public InputSplitShim getInputSplitShim() throws IOException {
+ return new InputSplitShim();
+ }
+
+ public RecordReader<K, V> getRecordReader(JobConf job,
+ HadoopShims.InputSplitShim split, Reporter reporter,
+ Class<RecordReader<K, V>> rrClass) throws IOException {
+ return new MultiFileRecordReader<K, V>(job, (InputSplitShim) split,
+ reporter, rrClass);
+ }
+
+ /**
+ * tries to guesstimate the optimal number of splits. We just calculate the
+ * total size of the job and divide it by the block size.
+ *
+ * @param job
+ * @param numSplits
+ * @return
+ */
+ private long estimateNumSplits(JobConf job, int numSplits)
+ throws IOException {
+ Path[] paths = FileUtil.stat2Paths(listStatus(job));
+
+ long blockSize = job.getLong("dfs.block.size", 67108864);
+
+ if (paths.length != 0) {
+ // HADOOP-1818: Manage splits only if there are paths
+ long[] lengths = new long[paths.length];
+ long totLength = 0;
+ for (int i = 0; i < paths.length; i++) {
+ FileSystem fs = paths[i].getFileSystem(job);
+ lengths[i] = fs.getContentSummary(paths[i]).getLength();
+ totLength += lengths[i];
+ }
+
+ numSplits = (int) Math.max(numSplits, (int) totLength / blockSize);
+ LOG.debug("Total Length " + totLength + ", block size " + blockSize
+ + ", number of splits: " + numSplits);
+
+ }
+ if (numSplits < 1)
+ numSplits = 1;
+
+ return numSplits;
+ }
+ }
+
+ /**
+ * We need to supply a RecordReader.
+ */
+ public static class MultiFileRecordReader<K, V> implements RecordReader<K, V> {
+
+ protected InputSplitShim split;
+ protected JobConf jc;
+ protected Reporter reporter;
+ protected Class<RecordReader<K, V>> rrClass;
+ protected Constructor<RecordReader<K, V>> rrConstructor;
+ protected FileSystem fs;
+ protected int idx;
+ protected long progress;
+ protected RecordReader<K, V> curReader;
+
+ public boolean next(K key, V value) throws IOException {
+
+ while ((curReader == null) || !curReader.next(key, value)) {
+ if (!initNextRecordReader()) {
+ return false;
+ }
+ }
+ return true;
+ }
+
+ public K createKey() {
+ return curReader.createKey();
+ }
+
+ public V createValue() {
+ return curReader.createValue();
+ }
+
+ /**
+ * return the amount of data processed
+ */
+ public long getPos() throws IOException {
+ return progress;
+ }
+
+ public void close() throws IOException {
+ if (curReader != null) {
+ curReader.close();
+ curReader = null;
+ }
+ }
+
+ /**
+ * return progress based on the amount of data processed so far.
+ */
+ public float getProgress() throws IOException {
+ return Math.min(1.0f, progress / (float) (split.getLength()));
+ }
+
+ static final Class[] constructorSignature = new Class[] { InputSplit.class,
+ Configuration.class, Reporter.class, Integer.class };
+
+ /**
+ * A generic RecordReader that can hand out different recordReaders for each
+ * chunk in the CombineFileSplit.
+ */
+ public MultiFileRecordReader(JobConf job, InputSplitShim split,
+ Reporter reporter, Class<RecordReader<K, V>> rrClass)
+ throws IOException {
+ this.split = split;
+ this.jc = job;
+ this.rrClass = rrClass;
+ this.reporter = reporter;
+ this.idx = 0;
+ this.curReader = null;
+ this.progress = 0;
+
+ try {
+ rrConstructor = rrClass.getDeclaredConstructor(constructorSignature);
+ rrConstructor.setAccessible(true);
+ } catch (Exception e) {
+ throw new RuntimeException(rrClass.getName()
+ + " does not have valid constructor", e);
+ }
+ initNextRecordReader();
+ }
+
+ /**
+ * Get the record reader for the next chunk in this CombineFileSplit.
+ */
+ protected boolean initNextRecordReader() throws IOException {
+ if (curReader != null) {
+ curReader.close();
+ curReader = null;
+ if (idx > 0) {
+ progress += split.getLength(idx - 1); // done processing so far
+ }
+ }
+
+ // if all chunks have been processed, nothing more to do.
+ if (idx == split.getNumPaths()) {
+ return false;
+ }
+
+ // get a record reader for the idx-th chunk
+ try {
+ /*
+ * in mapper task, the split is constructed with the default
+ * constructor, so it won't have the jc, resulting in
+ * NullPointerException
+ */
+ InputSplitShim newSplit = new InputSplitShim(jc, split.delegate);
+
+ curReader = rrConstructor.newInstance(new Object[] {
+ (InputSplit) newSplit, jc, reporter, Integer.valueOf(idx) });
+
+ // setup some helper config variables.
+ jc.set("map.input.file", split.getPath(idx).toString());
+ jc.setLong("map.input.start", 0); // split.getOffset(idx) instead of 0
+ // in CombineFileSplit
+ jc.setLong("map.input.length", split.getLength(idx));
+ } catch (Exception e) {
+ throw new RuntimeException(e);
+ }
+ idx++;
+ return true;
+ }
+ }
+
+ /**
+ * InputSplitShim
+ *
+ */
+ public static class InputSplitShim // extends MultiFileSplit
+ implements HadoopShims.InputSplitShim {
+
+ MultiFileSplit delegate;
+ long[] offsets;
+ JobConf job;
+ String constructor = "none";
+
+ public InputSplitShim() {
+ delegate = new MultiFileSplit(null, new Path[0], new long[0]);
+ onDelegateChange(delegate);
+ constructor = "default";
+ }
+
+ /**
+ * It encapsulate a set of files
+ *
+ * @param job
+ * @param old
+ * @throws IOException
+ */
+ public InputSplitShim(JobConf job, MultiFileSplit old) throws IOException {
+ delegate = new MultiFileSplit(job, old.getPaths(), old.getLengths());
+ offsets = new long[old.getPaths().length];
+ this.job = job;
+ constructor = "copy";
+ }
+
+ public long[] getStartOffsets() {
+ return offsets;
+ }
+
+ public long getOffset(int i) {
+ return 0;
+ }
+
+ public JobConf getJob() {
+ return job;
+ }
+
+ public long getLength() {
+ return delegate.getLength();
+ }
+
+ public long[] getLengths() {
+ return delegate.getLengths();
+ }
+
+ public long getLength(int i) {
+ return delegate.getLength(i);
+ }
+
+ public int getNumPaths() {
+ return delegate.getNumPaths();
+ }
+
+ public Path getPath(int i) {
+ return delegate.getPath(i);
+ }
+
+ public Path[] getPaths() {
+ return delegate.getPaths();
+ }
+
+ public String[] getLocations() throws IOException {
+ return delegate.getLocations();
+ }
+
+ public void readFields(DataInput in) throws IOException {
+ delegate.readFields(in);
+ onDelegateChange(delegate);
+ }
+
+ public void write(DataOutput out) throws IOException {
+ delegate.write(out);
+ }
+
+ private void onDelegateChange(MultiFileSplit delegate) {
+ if (delegate != null) {
+ offsets = new long[delegate.getNumPaths()];
+ }
+ }
+
+ private void setJob(JobConf jc) {
+ job = jc;
+ }
+ }
+
@Override
public long getAccessTime(FileStatus file) {
return file.getAccessTime();
}
+ String [] ret = new String[2];
- String [] ret = new String[2];
@Override
public String [] getTaskJobIDs(TaskCompletionEvent t) {
TaskID tid = t.getTaskAttemptId().getTaskID();