Hi Stuart,
Thanks for the link
Relating to the hadoop code I posted above, I got it to compile by putting
the code under a namespace and changing
the class names appropriately , but still I have no clue as to why the code
didn't work without the namespace.
I'll defenitely check the link though, thks.
regards,
rdsr
Here's the code with the modification
(ns com.rdsr.maxt
(:gen-class)
(:import [org.apache.hadoop.io IntWritable Text]
[org.apache.hadoop.fs Path]
[org.apache.hadoop.mapred
JobConf JobClient
FileInputFormat FileOutputFormat
Mapper Reducer MapReduceBase])
(:use [clojure.contrib.str-utils :only (re-split)]))
(gen-class
:name com.rdsr.mapper
:extends org.apache.hadoop.mapred.MapReduceBase
:implements [org.apache.hadoop.mapred.Mapper]
:prefix "m-")
(defn m-map [this key value output-collector reporter]
(let [line (str value)
year (subs line 15 19)
air-temperature (if (= (nth line 87) \+)
(Integer/parseInt (subs line 88 92))
(Integer/parseInt (subs line 87 92)))
quality (subs line 92 93)]
(if (and (not (= air-temperature 9999))
(re-matches #"[01459]" quality))
(.collect output-collector (Text. year) (IntWritable.
air-temperature)))))
(gen-class
:name com.rdsr.reducer
:extends org.apache.hadoop.mapred.MapReduceBase
:implements [org.apache.hadoop.mapred.Reducer]
:prefix "r-")
(defn r-reduce [this key values output-collector reporter]
(let [values-seq (iterator-seq values)]
(.collect output-collector key
(apply max (map (fn [n] (.get n)) values-seq)))))
(defn -main [& args]
(let [conf (JobConf. com.rdsr.maxt)]
(doto conf
(.setJobName "Max temperature")
(.setMapperClass com.rdsr.mapper)
(.setReducerClass com.rdsr.reducer)
(.setOutputKeyClass Text)
(.setOutputValueClass IntWritable))
(FileInputFormat/addInputPath conf (Path. (first args)))
(FileOutputFormat/setOutputPath conf (Path. (second args)))
(JobClient/runJob conf)))
(compile 'com.rdsr.maxt)
On Mon, Jul 27, 2009 at 9:46 PM, Stuart Sierra
<[email protected]>wrote:
>
> Hi rdsr,
>
> The problem is that you're trying to use the "maxtemperature" as a
> class name before it's finished compiling. To work around that, use
> (Class/forName "maxtemperature") instead.
>
> You can also see my (very task-specific) Hadoop/Clojure integration at
> <http://tinyurl.com/mqv2os>
> and <http://tinyurl.com/lznlg2>.
>
> -SS
>
>
> On Jul 27, 4:55 am, RD <[email protected]> wrote:
> > Hi all,
> > I'm having trouble converting the basic hadoop example to clojure.
> > Here's my code.
> >
> > (ns maxtemperature
> > (:gen-class)
> > (:import [org.apache.hadoop.io IntWritable Text]
> > [org.apache.hadoop.mapred
> > JobConf JobClient
> > FileInputFormat FileOutputFormat
> > Mapper Reducer MapReduceBase])
> > (:use [clojure.contrib.str-utils :only (re-split)]))
> >
> > (gen-class
> > :name mapper
> > :extends org.apache.hadoop.mapred.MapReduceBase
> > :implements [org.apache.hadoop.mapred.Mapper]
> > :prefix "m-")
> >
> > (defn m-map [this key value output-collector reporter]
> > (let [line (str value)
> > year (subs line 15 19)
> > air-temperature (if (= (nth line 87) \+)
> > (Integer/parseInt (subs line 88 92))
> > (Integer/parseInt (subs line 87 92)))
> > quality (subs line 92 93)]
> > (if (and (not (= air-temperature 9999))
> > (re-matches quality "[01459]"))
> > (.collect output-collector (Text. year) (IntWritable.
> > air-temperature)))))
> >
> > (gen-class
> > :name reducer
> > :extends org.apache.hadoop.mapred.MapReduceBase
> > :implements [org.apache.hadoop.mapred.Reducer]
> > :prefix "r-")
> >
> > (defn r-reduce [this key values output-collector]
> > (.collect output-collector key (apply max values)))
> >
> > (defn -main [s]
> > (let [conf (JobConf. maxtemperature)
> > args (re-split " +" s 2)]
> > (doto conf
> > (.setJobName "Max temperature")
> > (.setMapperClass mapper)
> > (.setReducerClass reducer)
> > (.setOutputKeyClass Text)
> > (.setOutputValueClass IntWritable))
> > (FileInputFormat/addInputPath conf (first args))
> > (FileOutputFormat/setOutputPath conf (second args))
> > (.runJob JobClient conf)))
> >
> > (compile 'maxtemperature)
> >
> > The problem is that, the compilation step fails when it encounters the
> > maxtemperature in the main function.
> > Without the main function the code works fine and even the compilation
> > succeeds. I can't figure out what's wrong.
> > Greatly appreciate help on this.
> >
> > thanks,
> > rdsr.
> >
>
--~--~---------~--~----~------------~-------~--~----~
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to [email protected]
Note that posts from new members are moderated - please be patient with your
first post.
To unsubscribe from this group, send email to
[email protected]
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
-~----------~----~----~----~------~----~------~--~---
