Hello St.Ack,
Thank you for your answers.
> 100K records
>
> >> We have about 6M records to insert at once, client creates thread per
> each
> >> So, loading script is a single-process of 60 threads?
>
Yes, I'm running client from namenode machine.
> > and
> > then wait until all threads will be finished. Each row is about 25Kb
> size.
> > Each thread creates its own HTable and HBaseConfiguration.
> >
>
> >> Reuse the configuration across threads otherwise it looks to zookeeper
> as
> >> though each thread is a new connection (long story -- I can tell you
> more if
> >> you are interested but basically if a new HBaseConfiguration, then we
> set up
> >> new connections else we reuse). Zookeeper has a basic protection
> against
> >> DOS attack not letting > 30 clients (by default) from the same host
> connect.
> >> This makes for interesting issues.
>
> > Something going wrong, because sometimes I get exception:
> >
> > Exception in thread "Thread-9" java.util.ConcurrentModificationException
> >
> >
> >> Can we see full stack trace to see which Map is throwing the CME?
>
Exception in thread "Thread-9" java.util.ConcurrentModificationException
at
java.util.AbstractList$Itr.checkForComodification(AbstractList.java:372)
at java.util.AbstractList$Itr.next(AbstractList.java:343)
at
org.apache.hadoop.conf.Configuration.loadResources(Configuration.java:1028)
at org.apache.hadoop.conf.Configuration.getProps(Configuration.java:979)
at
org.apache.hadoop.conf.Configuration.iterator(Configuration.java:1015)
at
org.apache.hadoop.hbase.HBaseConfiguration.hashCode(HBaseConfiguration.java:63)
at java.util.WeakHashMap.get(WeakHashMap.java:348)
at
org.apache.hadoop.hbase.client.HConnectionManager.getConnection(HConnectionManager.java:97)
at org.apache.hadoop.hbase.client.HTable.<init>(HTable.java:123)
at org.apache.hadoop.hbase.client.HTable.<init>(HTable.java:105)
at InserterThread.run(hbase_client.java:53)
at java.lang.Thread.run(Thread.java:619)
> > As for timings:
> > For 5Kb rows we have about 35-40K records per second.
> > For 25Kb rows -- about 1-2K records per second.
> >
> > So I have different throughput on different row size, looks illogical.
> >
> >
> Is it? Is same amount of data being carried?
>
> Or it could be that while the 25k is being sent, all other access to a
> particular node is blocked (Thats how hadoop RPC works -- one connection
> per
> process per server with request/response exclusive on the channel). Thread
> dump a few times or add some logging to see if you can figure if this is
> the
> case.
>
I mean that if I have 80-100 mb/sec throughput for 5Kb rows it should stay
the same for 25Kb rows. Of course I will insert less rows per second in
case of 25Kb, but throughput should stay the same. Now I'm trying to run
several instances of client each of them inserts 100K records (each record
is 25Kb). Time of execution grows for each client.
>
> In general, our client ain't to good at multiplexing because of such as the
> above noted limitation (our client does not yet do nio). If you want to
> test cluster performance, run multiple concurrent clients each to its own
> process. MapReduce is good for doing this. See the PerformanceEvaluation
> code for a sample MR job that floats many clients doing different loading
> types.
>
MapReduce is good idea, but actually we don't have data which is located in
hadoop, we processes data in realtime and insert it into hbase. So I think
it will be inefficient to write our data in hadoop and then run MapReduce
work which will insert that data into the tables.
>
> > Also I see that nodes load is almost idle. Hbase jvm heap size is 5Gb on
> > each node and only 300-500Mb is used during test.
> > I've used all performance tuning advices, like autoflush off, write
> buffer
> > =
> > 12 MB, WAL is off.
> >
>
> My guess is that you are not putting up sufficent loading. What if you
> start up ten other client processes? Do they all go at the same rate?
> Does
> the cluster then start to break a sweat?
>
>
Time with several clients is growing. For example when I'm running four
processes, each of them have one inserter thread I got following results:
1) Thread-1 have finished its work in 189 sec
2) Thread-1 have finished its work in 198 sec
3) Thread-1 have finished its work in 206 sec
4) Thread-1 have finished its work in 208 sec
I.e. each next process works longer than previous. It was timings for test
where each process inserts 100K 25Kb rows with WAL on. Btw WAL have great
impact on performance when I increase size of row. I have about 80 sec for
this test with WAL off. Also when running several clients nodes seems still
almost idle.
--
Regards, Lyfar Dmitriy
