bq. at some point we changed it to 30. Do you remember when the change was made ? After the change, was there server restart ?
You showed part of log for 35188 but the RetriesExhaustedException was for 35118. Was this due to redaction or were you checking a different server ? The RetriesExhaustedException happend at Jul 18 13:49:59 PDT 2017 which didn't match the server log snippet (2017-07-14 22:25:58,692) The coloring didn't go thru. For SocketTimeoutException, please see hbase-client/src/main/java/org/apache/hadoop/hbase/client/RpcRetryingCaller.java around line 169 (in branch-1.3). For the remaining exception message, see getExceptionMessageAdditionalDetail() in hbase-client/src/main/java/org/apache/hadoop/hbase/client/RegionServerCallable.java On Tue, Jul 18, 2017 at 10:48 PM, S L <[email protected]> wrote: > Just wondering how changing TTL on a table affects minor and major > compaction when it's trying to combine/delete things. > > There is a rowkey that shows up when my program does a scan. According to > the TTL, it shouldn't be there but it is. It gives me > RetriesExhaustedException. Looking at the table description, the TTL is 30 > days. We initially had it at 14 days, then at some point we changed it to > 30. > > Table description > describe 'dbi_based_data' > Table dbi_based_data is ENABLED > > > dbi_based_data > > COLUMN FAMILIES DESCRIPTION > > > {NAME => 'raw_data', DATA_BLOCK_ENCODING => 'NONE', BLOOMFILTER => 'ROW', > REPLICATION_SCOPE => '0', VERSIONS => '1', COMPRESSION => 'SNAPPY', > MIN_VERSIONS => '0', T > > TL => '2592000 SECONDS (30 DAYS)', KEEP_DELETED_CELLS => 'FALSE', BLOCKSIZE > => '65536', IN_MEMORY => 'false', BLOCKCACHE => 'true'} > > 1 row(s) in 0.5310 seconds > > The rowkey exception shows up as follows in the hadoop output: > > Error: org.apache.hadoop.hbase.client.RetriesExhaustedException: Failed > after attempts=36, exceptions: Tue Jul 18 13:49:59 PDT 2017, null, > java.net.SocketTimeoutException: callTimeout=75000, callDuration=75301: > row > '170_p3431.db160044.hkg1.dbi_1476454680' on table 'dbi_based_data' at > region=dbi_based_data,170_p3431.db160044.hkg1.dbi_ > 1476454680,1490311699762.e00f8737bfc5010b6bd9be11520ca1d3., > hostname=hslave35118.ams9.mydomain.com,60020,1483570489305, > seqNum=16311452 > > I went to the region server above to look at the logs > (/var/log/hbase/hbase*.out). I didn't find anything with that rowkey and I > can see it is performing major_compaction on other elements. > > hbase-cmf-hbase-REGIONSERVER-hslave35188.ams9.mydomain.com. > log.out.7:2017-07-14 > 22:25:58,692 INFO org.apache.hadoop.hbase.regionserver.HStore: Completed > major compaction of 4 (all) file(s) in raw_data of > dbi_based_data,109_p3400.db160093.sjc5.dbi_1493088600,1495713795634. > 358bee5b1db95d10e7533c7fe0365c0e. > into 51a8c31e32b8445eb781bdc580c2d5d1(size=4.6 G), total size for store is > 4.6 G. This selection was in queue for 6mins, 21sec, and took 2mins, 21sec > to execute. > > Two questions: > 1) Can this change in TTL make some keys not get deleted due to timing or > whatever other reason? If so, how do I delete these undeleteable keys. > > 2) What is the name of the java file that outputs the exception message > above? What do the different fields from the output mean? > Tue Jul 18 13:49:59 PDT 2017, null, java.net.SocketTimeoutException: > callTimeout=75000, callDuration=75301: row > '170_p3431.db160044.hkg1.dbi_1476454680' on table 'dbi_based_data' at > region=dbi_based_data,170_p3431.db160044.hkg1.dbi_ > 1476454680,1490311699762. > e00f8737bfc5010b6bd9be11520ca1d3., hostname=hslave35118.ams9.mydomain.com > ,60020,1483570489305, seqNum=16311452 > > > I've color coded the fields that I don't know what they mean. Hopefully > you can see the color. > a) What does "null" mean (light blue)? > b) The yellow is the rowkey but the red is some timestamp that I don't > know. The red is from March 2017 so it's way past the 30 day TTL. > c) Don't know what the purple alphanumeric string is but it might be the > region name? > d) The orange timestamp again is past the 30day TTL and from Jan 2017 but > again, I don't know what it is from/for. > > Thanks in advance for your help. >
