Hi,
If you intend to use those counters in further functions then I think Hadoop
will take care this by itself, you can explore combiner for the same. In
Sequence file mapfile and index files has its own functionality and as per my
understanding index file expect offset to move over map file and logically you
can't use it for counter.
Sent via Rocket from my HTC
----- Reply message -----
From: "Elazar Leibovich" <elaz...@gmail.com>
To: <user@hadoop.apache.org>
Subject: Saving counters in Mapfile
Date: Wed, Jul 24, 2013 3:27 AM
Hi,
A common use case one want an ordered structure for, is for saving counters.
Naturally, I wanted to save the counters in a Mapfile:
for (long ix = 0; ix < MAXVALUE; ix++) {
mapfile.append(new Text("counter key of val " + ix), new
LongWritable(ix));
}
This however looks a bit inefficient. We'll store two files, and an index
file. The index file will contain an offset (long) to the sequence file,
which would contain a single long.
I'd rather have only the index file, that would store the counter value
instead of offsets.
Is there a way to do that with Mapfile? Perhaps there's a better way to
save searchable counters in HDFS file?
Thanks,
