Hi

Can anyone share his experience or solution for the following problem?
I'm having to deal with a lot of different file formats, most of them csv.
Each of them shares similar semantics, ie. fields in file A exists in
file B as well.
What I'm not sure of is the exact index of the field in the csv file.
Fields in file A may also have different names for the same thing as in file B.

Simplified example:

affiliateA.csv:
Date; Clicks; Views; Orders
2009-03-10; 10; 20; 4

affiliateB.csv
Date; Orders; Impressions; Clicks
13/03/09; 40; 2000; 1000


Possible mapping file:

<field-mapping>
  <field id="date" type="java.util.Date"/>
  <field id="clicks" type="java.lang.Integer"/>
  <field id="views" type="java.lang.Integer"/>

  <file>
     <path pattern="/affiliateA/*.csv">
     <format type="csv">
        <seperator>\t</seperator>
        <quotes>&quot;</quotes>
     </format>
     <columns>
        <column name="Date" alias="date">
             <format>YYYY-MM-DD</format>
        </column>
        <column name="Clicks" alias="clicks"/>
     </columns>
  </file>

  <file>
     <path pattern="/affiliateB/*.csv">
     <format type="csv">
        <seperator>;</seperator>
     </format>
     <columns>
        <column index="1" alias="date">
             <format>dd/MM/yyy</format>
        </column>
        <column index="2" alias="clicks"/>
     </columns>
  </file>
</field-mapping>


What I'd like to be able is to use this external descriptor for each
file with a custom hadoop InputFormat.
Instead of a line of text, my MR values would be a Map containing the
parsed values mapped to the field IDs.

map(key, fields) {
  Date date = fields.get('date');
  Integer clicks = fields.get('clicks');
}

This would allow me to uncouple my MR job from the actual file format
and also moves all csv handling code out of my mappers.

Does anyone know if such a solution already exists for hadoop? Any thoughts?

Stefan

Reply via email to