Hi Andreas, Here is what I did:
bin/hadoop jar build/hadoop-0.18.0-dev-examples.jar randomtextwriter -Dtest.randomtextwrite.min_words_key=40 -Dtest.randomtextwrite.max_words_key=50 -Dtest.randomtextwrite.maps_per_host=1 textinput (this would generate 1GB of text data with pretty long sentences. Refer src/examples/org/apache/hadoop/examples/RandomTextWriter.java) bin/hadoop jar build/hadoop-0.18.0-dev-examples.jar sort -Dmapred.min.split.size=536870912 -Dio.sort.mb=256 -inFormat org.apache.hadoop.mapred.KeyValueTextInputFormat -outFormat org.apache.hadoop.mapred.lib.NullOutputFormat -outKey org.apache.hadoop.io.Text -outValue org.apache.hadoop.io.Text textinput textout (This is similar to what you run. Notice that I have a pretty high value of the mapred.min.split.size and io.sort.mb to ensure that each invocation of qsort processes good amount of data) This ran perfectly well. I even tried reducing the length of the sentences by specifying 1 for all the four - min_words_key/value, max_words_key/value during data creation. That seemed to work fine too. So could you pls do this: 1) Generate data using RandomTextWriter having similar characteristics as your input data set where qsort fails. 2) Try to reproduce the issue (you may have to do a couple of runs of (1)). Let us know the configuration of RandomTextWriter with which you see StackOverflow errors in qsort. I hope I am not asking for too much.. Pls let us know if you need any help in this regard... Thanks a lot! Devaraj > -----Original Message----- > From: Andreas Kostyrka [mailto:[EMAIL PROTECTED] > Sent: Wednesday, June 04, 2008 4:56 AM > To: [email protected] > Subject: Re: Stackoverflow > > Ok, I've tried it out, the example sort bombs exactly like > streaming => > > http://heaven.kostyrka.org/test.log > > Any recommendations? > > Thanks, > > Andreas >
