Interesting behavior, I just tried it out on my local setup (master/HEAD) out of curiosity to check if we can trick HBase into deleting this bad row and the following worked for me. I don't know how you ended up with that row though (bad bulk load? just guessing).
To have a table with the Long.MAX timestamp, I commented out some pieces of HBase code so that it doesn't override the timestamp with the current millis on the region server (otherwise, I just see the expected behavior of current ms). *Step1: Create a table and generate the problematic row* hbase(main):002:0> create 't1', 'f' Created table t1 -- patch hbase to accept Long.MAX_VALUE ts --- hbase(main):005:0> put 't1', 'row1', 'f:a', 'val', 9223372036854775807 Took 0.0054 seconds -- make sure the put with the ts is present -- hbase(main):006:0> scan 't1' ROW COLUMN+CELL row1 column=f:a, timestamp= *9223372036854775807*, value=val 1 row(s) Took 0.0226 seconds *Step 2: Hand craft an HFile with the delete marker* ...with this row/col/max ts [Let me know if you want the code, I can put it somewhere. I just used the StoreFileWriter utility ] -- dump the contents of hfile using the utility --- $ bin/hbase hfile -f file:///tmp/hfiles/f/bf84f424544f4675880494e09b750ce8 -p ...... Scanned kv count -> 1 K: row1/f:a/LATEST_TIMESTAMP/Delete/vlen=0/seqid=0 V: <==== Delete marker *Step 3: Bulk load this HFile with the delete marker * bin/hbase completebulkload file:///tmp/hfiles t1 *Step 4: Make sure the delete marker is inserted correctly.* hbase(main):001:0> scan 't1' ...... 0 row(s) Took 0.1387 seconds -- Raw scan to make sure the delete marker is inserted and nothing funky is happening --- hbase(main):003:0> scan 't1', {RAW=>true} ROW COLUMN+CELL row1 column=f:a, timestamp=9223372036854775807, type=Delete row1 column=f:a, timestamp=9223372036854775807, value=val 1 row(s) Took 0.0044 seconds Thoughts? On Tue, May 12, 2020 at 2:00 PM Alexander Batyrshin <0x62...@gmail.com> wrote: > Table is ~ 10TB SNAPPY data. I don’t have such a big time window on > production for re-inserting all data. > > I don’t know how we got those cells. I can only assume that this is > phoenix and/or replaying from WAL after region server crash. > > > On 12 May 2020, at 18:25, Wellington Chevreuil < > wellington.chevre...@gmail.com> wrote: > > > > How large is this table? Can you afford re-insert all current data on a > > new, temp table? If so, you could write a mapreduce job that scans this > > table and rewrite all its cells to this new, temp table. I had verified > > that 1.4.10 does have the timestamp replacing logic here: > > > https://github.com/apache/hbase/blob/branch-1.4/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java#L3395 > < > https://github.com/apache/hbase/blob/branch-1.4/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java#L3395 > > > > > > So if you re-insert all this table cells into a new one, the timestamps > > would be inserted correctly and you would then be able to delete those. > > Now, how those cells managed to get inserted with max timestamp? Was this > > cluster running on an old version that then got upgraded to 1.4.10? > > > > > > Em ter., 12 de mai. de 2020 às 13:49, Alexander Batyrshin < > 0x62...@gmail.com <mailto:0x62...@gmail.com>> > > escreveu: > > > >> Any ideas how to delete these rows? > >> > >> I see only this way: > >> - backup data from region that contains “damaged” rows > >> - close region > >> - remove region files from HDFS > >> - assign region > >> - copy needed rows from backup to recreated region > >> > >>> On 30 Apr 2020, at 21:00, Alexander Batyrshin <0x62...@gmail.com> > wrote: > >>> > >>> The same effect for CF: > >>> > >>> d = > >> > org.apache.hadoop.hbase.client.Delete.new("\x0439d58wj434dd".to_s.to_java_bytes) > >>> d.deleteFamily("d".to_s.to_java_bytes, > >> 9223372036854775807.to_java(Java::long)) > >>> table.delete(d) > >>> > >>> ROW > COLUMN+CELL > >>> \x0439d58wj434dd column=d:, > >> timestamp=1588269277879, type=DeleteFamily > >>> > >>> > >>>> On 29 Apr 2020, at 18:30, Wellington Chevreuil < > >> wellington.chevre...@gmail.com <mailto:wellington.chevre...@gmail.com> > <mailto:wellington.chevre...@gmail.com <mailto: > wellington.chevre...@gmail.com>>> > >> wrote: > >>>> > >>>> Well, it's weird that puts with such TS values were allowed, according > >> to > >>>> current code state. Can you afford delete the whole CF for those rows? > >>>> > >>>> Em qua., 29 de abr. de 2020 às 14:41, junhyeok park < > >> runnerren...@gmail.com <mailto:runnerren...@gmail.com> <mailto: > runnerren...@gmail.com <mailto:runnerren...@gmail.com>>> > >>>> escreveu: > >>>> > >>>>> I've been through the same thing. I use 2.2.0 > >>>>> > >>>>> 2020년 4월 29일 (수) 오후 10:32, Alexander Batyrshin <0x62...@gmail.com > <mailto:0x62...@gmail.com> > >> <mailto:0x62...@gmail.com <mailto:0x62...@gmail.com>>>님이 작성: > >>>>> > >>>>>> As you can see in example I already tried DELETE operation with > >> timestamp > >>>>>> = Long.MAX_VALUE without any success. > >>>>>> > >>>>>>> On 29 Apr 2020, at 12:41, Wellington Chevreuil < > >>>>>> wellington.chevre...@gmail.com <mailto: > wellington.chevre...@gmail.com> <mailto:wellington.chevre...@gmail.com > <mailto:wellington.chevre...@gmail.com>>> > >> wrote: > >>>>>>> > >>>>>>> That's expected behaviour [1]. If you are "travelling to the > future", > >>>>> you > >>>>>>> need to do a delete specifying Long.MAX_VALUE timestamp as the > >>>>> timestamp > >>>>>>> optional parameter in the delete operation [2], if you don't > specify > >>>>>>> timestamp on the delete, it will assume current time for the delete > >>>>>> marker, > >>>>>>> which will be smaller than the Long.MAX_VALUE set to your cells, so > >>>>> scans > >>>>>>> wouldn't filter it. > >>>>>>> > >>>>>>> [1] https://hbase.apache.org/book.html#version.delete < > https://hbase.apache.org/book.html#version.delete> < > >> https://hbase.apache.org/book.html#version.delete < > https://hbase.apache.org/book.html#version.delete>> > >>>>>>> [2] > >>>>>>> > >>>>>> > >>>>> > >> > https://github.com/apache/hbase/blob/branch-1.4/hbase-client/src/main/java/org/apache/hadoop/hbase/client/Delete.java#L98 > < > https://github.com/apache/hbase/blob/branch-1.4/hbase-client/src/main/java/org/apache/hadoop/hbase/client/Delete.java#L98 > > > >> < > >> > https://github.com/apache/hbase/blob/branch-1.4/hbase-client/src/main/java/org/apache/hadoop/hbase/client/Delete.java#L98 > >>> > >>>>>>> > >>>>>>> Em qua., 29 de abr. de 2020 às 08:57, Alexander Batyrshin < > >>>>>> 0x62...@gmail.com> > >>>>>>> escreveu: > >>>>>>> > >>>>>>>> Hello all, > >>>>>>>> We had faced with strange situation: table has rows with > >>>>> Long.MAX_VALUE > >>>>>>>> timestamp. > >>>>>>>> These rows impossible to delete, because DELETE mutation uses > >>>>>>>> System.currentTimeMillis() timestamp. > >>>>>>>> Is there any way to delete these rows? > >>>>>>>> We use HBase-1.4.10 > >>>>>>>> > >>>>>>>> Example: > >>>>>>>> > >>>>>>>> hbase(main):037:0> scan 'TRACET', { ROWPREFIXFILTER => > >>>>>> "\x0439d58wj434dd", > >>>>>>>> RAW=>true, VERSIONS=>10} > >>>>>>>> ROW > >>>>> COLUMN+CELL > >>>>>>>> \x0439d58wj434dd column=d:_0, > >>>>>>>> timestamp=9223372036854775807, value=x > >>>>>>>> > >>>>>>>> > >>>>>>>> hbase(main):045:0* delete 'TRACET', "\x0439d58wj434dd", "d:_0" > >>>>>>>> 0 row(s) in 0.0120 seconds > >>>>>>>> > >>>>>>>> hbase(main):046:0> scan 'TRACET', { ROWPREFIXFILTER => > >>>>>> "\x0439d58wj434dd", > >>>>>>>> RAW=>true, VERSIONS=>10} > >>>>>>>> ROW > >>>>> COLUMN+CELL > >>>>>>>> \x0439d58wj434dd column=d:_0, > >>>>>>>> timestamp=9223372036854775807, value=x > >>>>>>>> \x0439d58wj434dd column=d:_0, > >>>>>>>> timestamp=1588146570005, type=Delete > >>>>>>>> > >>>>>>>> > >>>>>>>> hbase(main):047:0> delete 'TRACET', "\x0439d58wj434dd", "d:_0", > >>>>>>>> 9223372036854775807 > >>>>>>>> 0 row(s) in 0.0110 seconds > >>>>>>>> > >>>>>>>> hbase(main):048:0> scan 'TRACET', { ROWPREFIXFILTER => > >>>>>> "\x0439d58wj434dd", > >>>>>>>> RAW=>true, VERSIONS=>10} > >>>>>>>> ROW > >>>>> COLUMN+CELL > >>>>>>>> \x0439d58wj434dd column=d:_0, > >>>>>>>> timestamp=9223372036854775807, value=x > >>>>>>>> \x0439d58wj434dd column=d:_0, > >>>>>>>> timestamp=1588146678086, type=Delete > >>>>>>>> \x0439d58wj434dd column=d:_0, > >>>>>>>> timestamp=1588146570005, type=Delete > >