Mappers don't just write data, they also sort it. Different parameters like the size of the sort buffer can have an impact on these kinds of metrics.
On Friday, February 4, 2011, Keith Wiley <kwi...@keithwiley.com> wrote: > I noticed that it takes much longer to write 55 MBs to a streaming output > than it takes to write 12 MBs (much more than 4-5X longer), so I broke the > output up, writing 1 MB at a time and discovered a perfectly linear slowdown. > Bottom line, the more data I have already written to stdout from a streaming > task, the longer it takes to write the next block of data. I have no idea if > this is intrinsic to producing stdout from any Unix process (I've never heard > of such a thing) or if this is a Hadoop issue. Does anyone have any idea > what's going on here? > > From pos 0, wrote 1048576 bytes. Next pos will be 1048576. > diffTimeWriteOneBlock: 0.31s > From pos 1048576, wrote 1048576 bytes. Next pos will be 2097152. > diffTimeWriteOneBlock: 0.9s > From pos 2097152, wrote 1048576 bytes. Next pos will be 3145728. > diffTimeWriteOneBlock: 1.46s > From pos 3145728, wrote 1048576 bytes. Next pos will be 4194304. > diffTimeWriteOneBlock: 1.98s > From pos 4194304, wrote 1048576 bytes. Next pos will be 5242880. > diffTimeWriteOneBlock: 2.47s > From pos 5242880, wrote 1048576 bytes. Next pos will be 6291456. > diffTimeWriteOneBlock: 3.06s > From pos 6291456, wrote 1048576 bytes. Next pos will be 7340032. > diffTimeWriteOneBlock: 3.53s > From pos 7340032, wrote 1048576 bytes. Next pos will be 8388608. > diffTimeWriteOneBlock: 3.96s > From pos 8388608, wrote 1048576 bytes. Next pos will be 9437184. > diffTimeWriteOneBlock: 4.24s > From pos 9437184, wrote 1048576 bytes. Next pos will be 10485760. > diffTimeWriteOneBlock: 4.74s > From pos 10485760, wrote 1048576 bytes. Next pos will be 11534336. > diffTimeWriteOneBlock: 5.24s > From pos 11534336, wrote 1048576 bytes. Next pos will be 12582912. > diffTimeWriteOneBlock: 5.72s > From pos 12582912, wrote 1048576 bytes. Next pos will be 13631488. > diffTimeWriteOneBlock: 6.25s > From pos 13631488, wrote 1048576 bytes. Next pos will be 14680064. > diffTimeWriteOneBlock: 6.77s > From pos 14680064, wrote 1048576 bytes. Next pos will be 15728640. > diffTimeWriteOneBlock: 7.37s > From pos 15728640, wrote 1048576 bytes. Next pos will be 16777216. > diffTimeWriteOneBlock: 7.76s > From pos 16777216, wrote 1048576 bytes. Next pos will be 17825792. > diffTimeWriteOneBlock: 8.74s > From pos 17825792, wrote 1048576 bytes. Next pos will be 18874368. > diffTimeWriteOneBlock: 8.99s > From pos 18874368, wrote 1048576 bytes. Next pos will be 19922944. > diffTimeWriteOneBlock: 9.35s > From pos 19922944, wrote 1048576 bytes. Next pos will be 20971520. > diffTimeWriteOneBlock: 9.85s > From pos 20971520, wrote 1048576 bytes. Next pos will be 22020096. > diffTimeWriteOneBlock: 10.43s > From pos 22020096, wrote 1048576 bytes. Next pos will be 23068672. > diffTimeWriteOneBlock: 11.05s > From pos 23068672, wrote 1048576 bytes. Next pos will be 24117248. > diffTimeWriteOneBlock: 11.52s > From pos 24117248, wrote 1048576 bytes. Next pos will be 25165824. > diffTimeWriteOneBlock: 12.23s > From pos 25165824, wrote 1048576 bytes. Next pos will be 26214400. > diffTimeWriteOneBlock: 12.49s > From pos 26214400, wrote 1048576 bytes. Next pos will be 27262976. > diffTimeWriteOneBlock: 13.1s > From pos 27262976, wrote 1048576 bytes. Next pos will be 28311552. > diffTimeWriteOneBlock: 13.83s > From pos 28311552, wrote 1048576 bytes. Next pos will be 29360128. > diffTimeWriteOneBlock: 14.31s > From pos 29360128, wrote 1048576 bytes. Next pos will be 30408704. > diffTimeWriteOneBlock: 14.65s > From pos 30408704, wrote 1048576 bytes. Next pos will be 31457280. > diffTimeWriteOneBlock: 15.32s > From pos 31457280, wrote 1048576 bytes. Next pos will be 32505856. > diffTimeWriteOneBlock: 15.88s > From pos 32505856, wrote 1048576 bytes. Next pos will be 33554432. > diffTimeWriteOneBlock: 16.77s > From pos 33554432, wrote 1048576 bytes. Next pos will be 34603008. > diffTimeWriteOneBlock: 16.9s > From pos 34603008, wrote 1048576 bytes. Next pos will be 35651584. > diffTimeWriteOneBlock: 17.39s > From pos 35651584, wrote 1048576 bytes. Next pos will be 36700160. > diffTimeWriteOneBlock: 18.12s > From pos 36700160, wrote 1048576 bytes. Next pos will be 37748736. > diffTimeWriteOneBlock: 18.69s > From pos 37748736, wrote 1048576 bytes. Next pos will be 38797312. > diffTimeWriteOneBlock: 19.09s > From pos 38797312, wrote 1048576 bytes. Next pos will be 39845888. > diffTimeWriteOneBlock: 19.7s > From pos 39845888, wrote 1048576 bytes. Next pos will be 40894464. > diffTimeWriteOneBlock: 20.65s > From pos 40894464, wrote 1048576 bytes. Next pos will be 41943040. > diffTimeWriteOneBlock: 20.71s > From pos 41943040, wrote 1048576 bytes. Next pos will be 42991616. > diffTimeWriteOneBlock: 21.49s > From pos 42991616, wrote 1048576 bytes. Next pos will be 44040192. > diffTimeWriteOneBlock: 22.1s > From pos 44040192, wrote 1048576 bytes. Next pos will be 45088768. > diffTimeWriteOneBlock: 22.48s > From pos 45088768, wrote 1048576 bytes. Next pos will be 46137344. > diffTimeWriteOneBlock: 23.36s > From pos 46137344, wrote 1048576 bytes. Next pos will be 47185920. > diffTimeWriteOneBlock: 23.84s > From pos 47185920, wrote 1048576 bytes. Next pos will be 48234496. > diffTimeWriteOneBlock: 24.23s > From pos 48234496, wrote 1048576 bytes. Next pos will be 49283072. > diffTimeWriteOneBlock: 25.07s > From pos 49283072, wrote 1048576 bytes. Next pos will be 50331648. > diffTimeWriteOneBlock: 25.56s > From pos 50331648, wrote 1048576 bytes. Next pos will be 51380224. > diffTimeWriteOneBlock: 26.27s > From pos 51380224, wrote 1048576 bytes. Next pos will be 52428800. > diffTimeWriteOneBlock: 26.79s > From pos 52428800, wrote 1048576 bytes. Next pos will be 53477376. > diffTimeWriteOneBlock: 27.47s > From pos 53477376, wrote 1048576 bytes. Next pos will be 54525952. > diffTimeWriteOneBlock: 27.98s > From pos 54525952, wrote 1048576 bytes. Next pos will be 55574528. > diffTimeWriteOneBlock: 28.53s > From pos 55574528, wrote 205312 bytes. Next pos will be 55779840. > diffTimeWriteOneBlock: 7.22s > > > ________________________________________________________________________________ > Keith Wiley kwi...@keithwiley.com keithwiley.com > music.keithwiley.com > > "I do not feel obliged to believe that the same God who has endowed us with > sense, reason, and intellect has intended us to forgo their use." > -- Galileo Galilei > ________________________________________________________________________________ > >
