Just wondering how changing TTL on a table affects minor and major
compaction when it's trying to combine/delete things.
There is a rowkey that shows up when my program does a scan. According to
the TTL, it shouldn't be there but it is. It gives me
RetriesExhaustedException. Looking at the table description, the TTL is 30
days. We initially had it at 14 days, then at some point we changed it to
30.
Table description
describe 'dbi_based_data'
Table dbi_based_data is ENABLED
dbi_based_data
COLUMN FAMILIES DESCRIPTION
{NAME => 'raw_data', DATA_BLOCK_ENCODING => 'NONE', BLOOMFILTER => 'ROW',
REPLICATION_SCOPE => '0', VERSIONS => '1', COMPRESSION => 'SNAPPY',
MIN_VERSIONS => '0', T
TL => '2592000 SECONDS (30 DAYS)', KEEP_DELETED_CELLS => 'FALSE', BLOCKSIZE
=> '65536', IN_MEMORY => 'false', BLOCKCACHE => 'true'}
1 row(s) in 0.5310 seconds
The rowkey exception shows up as follows in the hadoop output:
Error: org.apache.hadoop.hbase.client.RetriesExhaustedException: Failed
after attempts=36, exceptions: Tue Jul 18 13:49:59 PDT 2017, null,
java.net.SocketTimeoutException: callTimeout=75000, callDuration=75301: row
'170_p3431.db160044.hkg1.dbi_1476454680' on table 'dbi_based_data' at
region=dbi_based_data,170_p3431.db160044.hkg1.dbi_1476454680,1490311699762.e00f8737bfc5010b6bd9be11520ca1d3.,
hostname=hslave35118.ams9.mydomain.com,60020,1483570489305, seqNum=16311452
I went to the region server above to look at the logs
(/var/log/hbase/hbase*.out). I didn't find anything with that rowkey and I
can see it is performing major_compaction on other elements.
hbase-cmf-hbase-REGIONSERVER-hslave35188.ams9.mydomain.com.log.out.7:2017-07-14
22:25:58,692 INFO org.apache.hadoop.hbase.regionserver.HStore: Completed
major compaction of 4 (all) file(s) in raw_data of
dbi_based_data,109_p3400.db160093.sjc5.dbi_1493088600,1495713795634.358bee5b1db95d10e7533c7fe0365c0e.
into 51a8c31e32b8445eb781bdc580c2d5d1(size=4.6 G), total size for store is
4.6 G. This selection was in queue for 6mins, 21sec, and took 2mins, 21sec
to execute.
Two questions:
1) Can this change in TTL make some keys not get deleted due to timing or
whatever other reason? If so, how do I delete these undeleteable keys.
2) What is the name of the java file that outputs the exception message
above? What do the different fields from the output mean?
Tue Jul 18 13:49:59 PDT 2017, null, java.net.SocketTimeoutException:
callTimeout=75000, callDuration=75301: row
'170_p3431.db160044.hkg1.dbi_1476454680' on table 'dbi_based_data' at
region=dbi_based_data,170_p3431.db160044.hkg1.dbi_1476454680,1490311699762.
e00f8737bfc5010b6bd9be11520ca1d3., hostname=hslave35118.ams9.mydomain.com
,60020,1483570489305, seqNum=16311452
I've color coded the fields that I don't know what they mean. Hopefully
you can see the color.
a) What does "null" mean (light blue)?
b) The yellow is the rowkey but the red is some timestamp that I don't
know. The red is from March 2017 so it's way past the 30 day TTL.
c) Don't know what the purple alphanumeric string is but it might be the
region name?
d) The orange timestamp again is past the 30day TTL and from Jan 2017 but
again, I don't know what it is from/for.
Thanks in advance for your help.
