Stack, I did some tests with different flushing parameters. I've touched following params: hbase.hregion.memstore.block.multiplier hbase.hregion.memstore.flush.size
When I've increased flushsize to 256Mb (64Mb by default) time of 25Kb test grew. When I've changed block.multiplier to 12 (was 10) I won about 40 seconds in 25Kb test (was 150-160 secs, became ~110 secs for one running instance that inserts 100K records). All tests I did was with WAL on. 2010/1/5 Dmitriy Lyfar <[email protected]> > Hello Stack, > > >> > And throughput without WAL is about 50 Mb/sec and about 15 Mb/sec with >> WAL >> > on. When I run clients in serial order (i.e. at the moment there is only >> > one >> > working script) time almost stable and not grows. >> > >> > >> > > See what the >> > > numbers are like uploading into a table that is pre-split? >> > >> > >> > Sorry, what you mean pre-split? You mean splitting regions before >> running >> > script? >> > >> > >> I was thinking you were uploading into a new table and that the region >> splits were happening inline with your upload. I was asking what the >> performance was like if the table had already had all its regions pre-made >> wondering if it ran faster but sounds like your table is already >> pre-split. >> >> So where are we at now? You tried running multiple separate upload >> processes and it still runs too slow? >> > > Yes, still too slow, especially with WAL on. Btw, I see the greater row > size, the greater impact has WAL. I'm not an expert in hbase internals, but > I begin think that the reason of throughput fall in case of 25Kb size > connected with flushing. I mean looks like we begin flush too often and it > impacts on throughput. > Also as I see from architecture description there are could be several > reasons, like rolling hlog too often and long compaction period. Would you > advice which log messages in region/master logs should warn me that > something going wrong? > > > -- > Regards, Lyfar Dmitriy > > -- Regards, Lyfar Dmitriy mailto: [email protected] jabber: [email protected]
