My phoenix upsert batch size is 50. You mean to say that 50 is also a lot? However, AsyncProcess is complaining about 2000 actions.
I tried with upsert batch size of 5 also. But it didnt help. On Sun, Feb 14, 2016 at 7:37 PM, anil gupta <anilgupt...@gmail.com> wrote: > My phoenix upsert batch size is 50. You mean to say that 50 is also a lot? > > However, AsyncProcess is complaining about 2000 actions. > > I tried with upsert batch size of 5 also. But it didnt help. > > > On Sun, Feb 14, 2016 at 6:43 PM, Heng Chen <heng.chen.1...@gmail.com> > wrote: > >> 2016-02-14 12:34:23,593 INFO [main] >> org.apache.hadoop.hbase.client.AsyncProcess: #1, waiting for 2000 >> actions to finish >> >> It means your writes are too many, please decrease the batch size of your >> puts, and balance your requests on each RS. >> >> 2016-02-15 4:53 GMT+08:00 anil gupta <anilgupt...@gmail.com>: >> >> > After a while we also get this error: >> > 2016-02-14 12:45:10,515 WARN [main] >> > org.apache.phoenix.execute.MutationState: Swallowing exception and >> > retrying after clearing meta cache on connection. >> > java.sql.SQLException: ERROR 2008 (INT10): Unable to find cached index >> > metadata. ERROR 2008 (INT10): ERROR 2008 (INT10): Unable to find >> > cached index metadata. key=-594230549321118802 >> > region=BI.SALES,,1455470578449.44e39179789041b5a8c03316730260e7. Index >> > update failed >> > >> > We have already set: >> > >> > >> <name>phoenix.coprocessor.maxServerCacheTimeToLiveMs</name><value>180000</value> >> > >> > Upset batch size is 50. Write are quite frequent so the cache would >> > not timeout in 180000ms >> > >> > >> > On Sun, Feb 14, 2016 at 12:44 PM, anil gupta <anilgupt...@gmail.com> >> > wrote: >> > >> > > Hi, >> > > >> > > We are using phoenix4.4, hbase 1.1(hdp2.3.4). >> > > I have a MR job that is using PhoenixOutputFormat. My job keeps on >> > failing >> > > due to following error: >> > > >> > > 2016-02-14 12:29:43,182 INFO [main] >> > org.apache.hadoop.hbase.client.AsyncProcess: #1, waiting for 2000 >> actions >> > to finish >> > > 2016-02-14 12:29:53,197 INFO [main] >> > org.apache.hadoop.hbase.client.AsyncProcess: #1, waiting for 2000 >> actions >> > to finish >> > > 2016-02-14 12:30:03,212 INFO [main] >> > org.apache.hadoop.hbase.client.AsyncProcess: #1, waiting for 2000 >> actions >> > to finish >> > > 2016-02-14 12:30:13,225 INFO [main] >> > org.apache.hadoop.hbase.client.AsyncProcess: #1, waiting for 2000 >> actions >> > to finish >> > > 2016-02-14 12:30:23,239 INFO [main] >> > org.apache.hadoop.hbase.client.AsyncProcess: #1, waiting for 2000 >> actions >> > to finish >> > > 2016-02-14 12:30:33,253 INFO [main] >> > org.apache.hadoop.hbase.client.AsyncProcess: #1, waiting for 2000 >> actions >> > to finish >> > > 2016-02-14 12:30:43,266 INFO [main] >> > org.apache.hadoop.hbase.client.AsyncProcess: #1, waiting for 2000 >> actions >> > to finish >> > > 2016-02-14 12:30:53,279 INFO [main] >> > org.apache.hadoop.hbase.client.AsyncProcess: #1, waiting for 2000 >> actions >> > to finish >> > > 2016-02-14 12:31:03,293 INFO [main] >> > org.apache.hadoop.hbase.client.AsyncProcess: #1, waiting for 2000 >> actions >> > to finish >> > > 2016-02-14 12:31:13,305 INFO [main] >> > org.apache.hadoop.hbase.client.AsyncProcess: #1, waiting for 2000 >> actions >> > to finish >> > > 2016-02-14 12:31:23,318 INFO [main] >> > org.apache.hadoop.hbase.client.AsyncProcess: #1, waiting for 2000 >> actions >> > to finish >> > > 2016-02-14 12:31:33,331 INFO [main] >> > org.apache.hadoop.hbase.client.AsyncProcess: #1, waiting for 2000 >> actions >> > to finish >> > > 2016-02-14 12:31:43,345 INFO [main] >> > org.apache.hadoop.hbase.client.AsyncProcess: #1, waiting for 2000 >> actions >> > to finish >> > > 2016-02-14 12:31:53,358 INFO [main] >> > org.apache.hadoop.hbase.client.AsyncProcess: #1, waiting for 2000 >> actions >> > to finish >> > > 2016-02-14 12:32:03,371 INFO [main] >> > org.apache.hadoop.hbase.client.AsyncProcess: #1, waiting for 2000 >> actions >> > to finish >> > > 2016-02-14 12:32:13,385 INFO [main] >> > org.apache.hadoop.hbase.client.AsyncProcess: #1, waiting for 2000 >> actions >> > to finish >> > > 2016-02-14 12:32:23,399 INFO [main] >> > org.apache.hadoop.hbase.client.AsyncProcess: #1, waiting for 2000 >> actions >> > to finish >> > > 2016-02-14 12:32:33,412 INFO [main] >> > org.apache.hadoop.hbase.client.AsyncProcess: #1, waiting for 2000 >> actions >> > to finish >> > > 2016-02-14 12:32:43,428 INFO [main] >> > org.apache.hadoop.hbase.client.AsyncProcess: #1, waiting for 2000 >> actions >> > to finish >> > > 2016-02-14 12:32:53,443 INFO [main] >> > org.apache.hadoop.hbase.client.AsyncProcess: #1, waiting for 2000 >> actions >> > to finish >> > > 2016-02-14 12:33:03,457 INFO [main] >> > org.apache.hadoop.hbase.client.AsyncProcess: #1, waiting for 2000 >> actions >> > to finish >> > > 2016-02-14 12:33:13,472 INFO [main] >> > org.apache.hadoop.hbase.client.AsyncProcess: #1, waiting for 2000 >> actions >> > to finish >> > > 2016-02-14 12:33:23,486 INFO [main] >> > org.apache.hadoop.hbase.client.AsyncProcess: #1, waiting for 2000 >> actions >> > to finish >> > > 2016-02-14 12:33:33,524 INFO [main] >> > org.apache.hadoop.hbase.client.AsyncProcess: #1, waiting for 2000 >> actions >> > to finish >> > > 2016-02-14 12:33:43,538 INFO [main] >> > org.apache.hadoop.hbase.client.AsyncProcess: #1, waiting for 2000 >> actions >> > to finish >> > > 2016-02-14 12:33:53,551 INFO [main] >> > org.apache.hadoop.hbase.client.AsyncProcess: #1, waiting for 2000 >> actions >> > to finish >> > > 2016-02-14 12:34:03,565 INFO [main] >> > org.apache.hadoop.hbase.client.AsyncProcess: #1, waiting for 2000 >> actions >> > to finish >> > > 2016-02-14 12:34:03,953 INFO [hconnection-0xe82ca6e-shared--pool2-t16] >> > org.apache.hadoop.hbase.client.AsyncProcess: #1, table=BI.SALES, >> > attempt=10/35 failed=2000ops, last exception: null on hdp3.truecar.com >> ,16020,1455326291512, >> > tracking started null, retrying after=10086ms, replay=2000ops >> > > 2016-02-14 12:34:13,578 INFO [main] >> > org.apache.hadoop.hbase.client.AsyncProcess: #1, waiting for 2000 >> actions >> > to finish >> > > 2016-02-14 12:34:23,593 INFO [main] >> > org.apache.hadoop.hbase.client.AsyncProcess: #1, waiting for 2000 >> actions >> > to finish >> > > >> > > I have never seen anything like this. Can anyone give me pointers >> about >> > > this problem? >> > > >> > > -- >> > > Thanks & Regards, >> > > Anil Gupta >> > > >> > >> > >> > >> > -- >> > Thanks & Regards, >> > Anil Gupta >> > >> > > > > -- > Thanks & Regards, > Anil Gupta > -- Thanks & Regards, Anil Gupta