Hi
Can anyone share his experience or solution for the following problem?
I'm having to deal with a lot of different file formats, most of them csv.
Each of them shares similar semantics, ie. fields in file A exists in
file B as well.
What I'm not sure of is the exact index of the field in the csv file.
Fields in file A may also have different names for the same thing as in file B.
Simplified example:
affiliateA.csv:
Date; Clicks; Views; Orders
2009-03-10; 10; 20; 4
affiliateB.csv
Date; Orders; Impressions; Clicks
13/03/09; 40; 2000; 1000
Possible mapping file:
<field-mapping>
<field id="date" type="java.util.Date"/>
<field id="clicks" type="java.lang.Integer"/>
<field id="views" type="java.lang.Integer"/>
<file>
<path pattern="/affiliateA/*.csv">
<format type="csv">
<seperator>\t</seperator>
<quotes>"</quotes>
</format>
<columns>
<column name="Date" alias="date">
<format>YYYY-MM-DD</format>
</column>
<column name="Clicks" alias="clicks"/>
</columns>
</file>
<file>
<path pattern="/affiliateB/*.csv">
<format type="csv">
<seperator>;</seperator>
</format>
<columns>
<column index="1" alias="date">
<format>dd/MM/yyy</format>
</column>
<column index="2" alias="clicks"/>
</columns>
</file>
</field-mapping>
What I'd like to be able is to use this external descriptor for each
file with a custom hadoop InputFormat.
Instead of a line of text, my MR values would be a Map containing the
parsed values mapped to the field IDs.
map(key, fields) {
Date date = fields.get('date');
Integer clicks = fields.get('clicks');
}
This would allow me to uncouple my MR job from the actual file format
and also moves all csv handling code out of my mappers.
Does anyone know if such a solution already exists for hadoop? Any thoughts?
Stefan