Sorry, occasionally sent. Now exception says about *sendMultiAction*, before that is wan't multi.
Feb 09, 2016 8:45:17 PM *org.apache.hadoop.hbase.client.AsyncProcess sendMultiAction* WARNING: #127, the task was rejected by the pool. This is unexpected. Server is node02.admanager.pro,60020,1447338599407 java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@32c81537 rejected from java.util.concurrent.ThreadPoolExecutor@45b63ba6[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 1] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2047) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:823) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1369) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.sendMultiAction(AsyncProcess.java:956) at org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.access$000(AsyncProcess.java:574) at org.apache.hadoop.hbase.client.AsyncProcess.submitMultiActions(AsyncProcess.java:423) at org.apache.hadoop.hbase.client.AsyncProcess.submit(AsyncProcess.java:403) at org.apache.hadoop.hbase.client.AsyncProcess.submit(AsyncProcess.java:320) at org.apache.hadoop.hbase.client.BufferedMutatorImpl.backgroundFlushCommits(BufferedMutatorImpl.java:206) at org.apache.hadoop.hbase.client.BufferedMutatorImpl.close(BufferedMutatorImpl.java:158) at org.apache.hadoop.hbase.client.BufferedMutator$close$0.call(Unknown Source) 2016-02-09 18:46 GMT+01:00 Serega Sheypak <serega.shey...@gmail.com>: > I've modified my code: > > void saveUsers(Collection<User> users){ > if(users && !users.isEmpty()){ > > // Get Connection instance, instance created once. , BM is new > for each request. > def mutator = > getConnection().getBufferedMutator(getBufferedMutatorParams()) > > List<Put> putList = users.collect{toPut(it)} > mutator.mutate(putList) > *mutator.close() // exception here* > } > } > > Exception is still thrown > > 2016-02-09 15:43 GMT+01:00 Stack <st...@duboce.net>: > >> On Tue, Feb 9, 2016 at 6:37 AM, Serega Sheypak <serega.shey...@gmail.com> >> wrote: >> >> > Hi, thanks for reply! >> > >> > >> > > What should we add here to make the doc more clear on BufferedMutator? >> > > >> > >> > >> http://hbase.apache.org/apidocs/org/apache/hadoop/hbase/client/BufferedMutator.html >> > It's pretty clear. >> > >> > > And in the example? >> > Example shows that both connection and buffered mutator are thread safe >> and >> > they are closed at once. >> > - Should keep single instance of connection and bufferedmutator per >> thread? >> > >> >> No. Connection says you should generally share the Connection instance. >> Ditto for BufferedMutator. It is backed by a buffer so you are batching >> your writes when you use it. If you share the BM, your batching will more >> 'regular'. >> >> >> > - Should I keep connection per thread and instantiate mutator for each >> > request? >> > >> >> See above. >> >> >> > - what happens if autoflush enabled? >> > >> >> When the backing buffer is full, it gets flushed to the cluster. >> >> >> > - is it possible to turn on sync mode for saving data? >> > >> >> Yes. Don't use a BufferedMutator. Your throughput will go down as you do >> an >> RPC per write. >> >> >> > - Connection could get into failed/invalid state for some reason (RS >> down, >> > RS up, some networking partitioned happend). Is it possible? >> >> >> >> It could happen but I think by now we've seen nearly all of the ways in >> which a Connection can fail and internally, it compensates. >> >> >> >> > If it's >> > possible, then what is the right way to handle it: "close" failed >> > connection and ask for new one? >> > >> > >> Good question. Connection internally will retry and 'ride over' near all >> cluster issues but the catastrophic. >> >> St.Ack >> >> >> >> > > I assume users is collection of User's. >> > > Have you tried obtaining / closing mutator for each User instead of >> > sharing >> > > the mutator ? >> > >> > > If another flush, say because there were lots of puts, then when close >> > > comes in, we are out of threads and you'd get below. >> > Looks like it's the root cause, let me try! >> > >> > Thank you for detailed explanation! >> > >> > >> > >> > 2016-02-09 7:10 GMT+01:00 Stack <st...@duboce.net>: >> > >> > > On Mon, Feb 8, 2016 at 3:01 PM, Serega Sheypak < >> serega.shey...@gmail.com >> > > >> > > wrote: >> > > >> > > > Hi, I'm confused with new HBase 1.0 API. API says that application >> > should >> > > > manage connections (Previously HConnections) on their own. Nothing >> is >> > > > managed itnernally now. >> > > > >> > > > >> > > That is right. >> > > >> > > >> > > >> > > >> > > > Here is an example: >> > > > >> > > > >> > > >> > >> https://hbase.apache.org/xref/org/apache/hadoop/hbase/client/example/BufferedMutatorExample.html >> > > > >> > > > It gives no clue about lifecycle :( >> > > > >> > > >> > > >> > > Connection is fairly explicit: >> > > >> > > >> > >> http://hbase.apache.org/apidocs/org/apache/hadoop/hbase/client/Connection.html >> > > >> > > What should we add here to make the doc more clear on BufferedMutator? >> > > >> > > >> > >> http://hbase.apache.org/apidocs/org/apache/hadoop/hbase/client/BufferedMutator.html >> > > >> > > And in the example? >> > > >> > > >> > > >> > > > Right now I create single connection instance for servlet and >> > > > BufferedMutator per request. >> > > > >> > > > >> > > > >> > > The recommendation is a singled BufferedMutator shared across >> requests. >> > > >> > > >> > > >> > > >> > > > //getConnection returns single instance, it doesn't return new >> > connection >> > > > each time >> > > > def mutator = >> > > > getConnection().getBufferedMutator(getBufferedMutatorParams()) >> > > > >> > > >> > > >> > > getConnection is your method? >> > > >> > > >> > > getBufferedMutator creates a new one? >> > > >> > > >> > > >> > > > users.each{ mutator.mutate(toPut(it))} >> > > > mutator.close() //exception is thrown here >> > > > >> > > > >> > > The close is flushing out all the writes. >> > > >> > > If a BufferedMutator per servlet instance, there are probably many >> when >> > > many requests coming in. >> > > >> > > See what happens when you create one: >> > > >> > > >> > > >> > >> http://hbase.apache.org/xref/org/apache/hadoop/hbase/client/ConnectionImplementation.html#313 >> > > >> > > which calls through to here.... >> > > >> > > >> > >> http://hbase.apache.org/xref/org/apache/hadoop/hbase/client/HTable.html#126 >> > > >> > > ... which creates an executor of max 1 task only. >> > > >> > > If another flush, say because there were lots of puts, then when close >> > > comes in, we are out of threads and you'd get below. >> > > >> > > St.Ack >> > > >> > > >> > > >> > > >> > > > >> > > > And I get tons of exceptions thrown on "mutator.close()", what do I >> do >> > > > wrong? >> > > > >> > > > WARNING: #905, the task was rejected by the pool. This is >> unexpected. >> > > > Server is node04.server.com, 60020,1447338864601 >> > > > java.util.concurrent.RejectedExecutionException: Task >> > > > java.util.concurrent.FutureTask@5cff3b40 rejected from >> > > > java.util.concurrent.ThreadPoolExecutor@686c2853[Terminated, pool >> > size = >> > > > 0, >> > > > active threads = 0, queued tasks = 0, completed tasks = 1] >> > > > at >> > > > >> > > > >> > > >> > >> java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2047) >> > > > at >> > > > >> > > >> > >> java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:823) >> > > > at >> > > > >> > > > >> > > >> > >> java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1369) >> > > > at >> > > > >> > > > >> > > >> > >> java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) >> > > > at >> > > > >> > > > >> > > >> > >> org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.sendMultiAction(AsyncProcess.java:956) >> > > > at >> > > > >> > > > >> > > >> > >> org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.access$000(AsyncProcess.java:574) >> > > > at >> > > > >> > > > >> > > >> > >> org.apache.hadoop.hbase.client.AsyncProcess.submitMultiActions(AsyncProcess.java:423) >> > > > at >> > > > >> > >> org.apache.hadoop.hbase.client.AsyncProcess.submit(AsyncProcess.java:403) >> > > > at >> > > > >> > >> org.apache.hadoop.hbase.client.AsyncProcess.submit(AsyncProcess.java:320) >> > > > at >> > > > >> > > > >> > > >> > >> org.apache.hadoop.hbase.client.BufferedMutatorImpl.backgroundFlushCommits(BufferedMutatorImpl.java:206) >> > > > at >> > > > >> > > > >> > > >> > >> org.apache.hadoop.hbase.client.BufferedMutatorImpl.close(BufferedMutatorImpl.java:158) >> > > > at >> org.apache.hadoop.hbase.client.BufferedMutator$close$0.call(Unknown >> > > > Source) >> > > > >> > > >> > >> > >