> Can you share code that generate HFiles with delete markers? Here <https://gist.github.com/bharathv/1bc499e717366ea71940ab6df6a98d33> you go. You might want to use table.getDescriptor() and build the column families from that descriptor. I just hardcoded everything for my simple table. You might have to change a few things depending on your table properties. Also, the code works on the master branch, certain APIs may be different from older branches. I was actually thinking about a little more elegant way of doing this, so I was mucking with the importtsv tool to generate deletes rather than puts from an input tsv file. I gave up after sometime, but that is also worth trying I guess, or may be others know a better way of doing this.
> But what would happen after major-compaction? Its like a regular major compaction. In this case, it cleans up all the history because both edits undo each other. hbase(main):006:0> scan 't1', {RAW=>true} ROW COLUMN+CELL row1 column=f:a, timestamp=9223372036854775807, type=Delete row1 column=f:a, timestamp=9223372036854775807, value=val 1 row(s) Took 0.0193 seconds hbase(main):007:0> major_compact 't1' Took 0.0479 seconds hbase(main):008:0> scan 't1', {RAW=>true} ROW COLUMN+CELL 0 row(s) Took 0.0026 seconds > Maybe TS corruption issue some how linked with another issue that we got - https://issues.apache.org/jira/browse/HBASE-22862 We are running into this too. Our current theory is that it is caused by Phoenix indexes and an upgrade may fix it but I don't know if/how these two issues are linked. On Thu, May 14, 2020 at 7:22 AM Alexander Batyrshin <0x62...@gmail.com> wrote: > Thank you for this idea. Its looks promising. > > Can you share code that generate HFiles with delete markers? > > As I see delete markers was inserted correctly. > But what would happen after major-compaction? > > > On 13 May 2020, at 08:32, Bharath Vissapragada <bhara...@apache.org> > wrote: > > > > Interesting behavior, I just tried it out on my local setup (master/HEAD) > > out of curiosity to check if we can trick HBase into deleting this bad > row > > and the following worked for me. I don't know how you ended up with that > > row though (bad bulk load? just guessing). > > > > To have a table with the Long.MAX timestamp, I commented out some pieces > of > > HBase code so that it doesn't override the timestamp with the current > > millis on the region server (otherwise, I just see the expected behavior > of > > current ms). > > > > *Step1: Create a table and generate the problematic row* > > > > hbase(main):002:0> create 't1', 'f' > > Created table t1 > > > > -- patch hbase to accept Long.MAX_VALUE ts --- > > > > hbase(main):005:0> put 't1', 'row1', 'f:a', 'val', 9223372036854775807 > > Took 0.0054 seconds > > > > -- make sure the put with the ts is present -- > > hbase(main):006:0> scan 't1' > > ROW COLUMN+CELL > > > > row1 column=f:a, timestamp= > > *9223372036854775807*, value=val > > > > 1 row(s) > > Took 0.0226 seconds > > > > *Step 2: Hand craft an HFile with the delete marker* > > > > ...with this row/col/max ts [Let me know if you want the code, I can put > > it somewhere. I just used the StoreFileWriter utility ] > > > > -- dump the contents of hfile using the utility --- > > > > $ bin/hbase hfile -f > file:///tmp/hfiles/f/bf84f424544f4675880494e09b750ce8 > <file:///tmp/hfiles/f/bf84f424544f4675880494e09b750ce8> > > -p > > ...... > > Scanned kv count -> 1 > > K: row1/f:a/LATEST_TIMESTAMP/Delete/vlen=0/seqid=0 V: <==== Delete > marker > > > > *Step 3: Bulk load this HFile with the delete marker * > > > > bin/hbase completebulkload file:///tmp/hfiles <file:///tmp/hfiles> t1 > > > > *Step 4: Make sure the delete marker is inserted correctly.* > > > > hbase(main):001:0> scan 't1' > > ...... > > > > 0 row(s) > > Took 0.1387 seconds > > > > -- Raw scan to make sure the delete marker is inserted and nothing funky > is > > happening --- > > > > hbase(main):003:0> scan 't1', {RAW=>true} > > ROW COLUMN+CELL > > > > > > row1 column=f:a, > > timestamp=9223372036854775807, type=Delete > > > > row1 column=f:a, > > timestamp=9223372036854775807, value=val > > > > 1 row(s) > > Took 0.0044 seconds > > > > Thoughts? > > > > On Tue, May 12, 2020 at 2:00 PM Alexander Batyrshin <0x62...@gmail.com > <mailto:0x62...@gmail.com>> > > wrote: > > > >> Table is ~ 10TB SNAPPY data. I don’t have such a big time window on > >> production for re-inserting all data. > >> > >> I don’t know how we got those cells. I can only assume that this is > >> phoenix and/or replaying from WAL after region server crash. > >> > >>> On 12 May 2020, at 18:25, Wellington Chevreuil < > >> wellington.chevre...@gmail.com> wrote: > >>> > >>> How large is this table? Can you afford re-insert all current data on a > >>> new, temp table? If so, you could write a mapreduce job that scans this > >>> table and rewrite all its cells to this new, temp table. I had verified > >>> that 1.4.10 does have the timestamp replacing logic here: > >>> > >> > https://github.com/apache/hbase/blob/branch-1.4/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java#L3395 > >> < > >> > https://github.com/apache/hbase/blob/branch-1.4/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java#L3395 > < > https://github.com/apache/hbase/blob/branch-1.4/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java#L3395 > > > >>> > >>> > >>> So if you re-insert all this table cells into a new one, the timestamps > >>> would be inserted correctly and you would then be able to delete those. > >>> Now, how those cells managed to get inserted with max timestamp? Was > this > >>> cluster running on an old version that then got upgraded to 1.4.10? > >>> > >>> > >>> Em ter., 12 de mai. de 2020 às 13:49, Alexander Batyrshin < > >> 0x62...@gmail.com <mailto:0x62...@gmail.com> <mailto:0x62...@gmail.com > <mailto:0x62...@gmail.com>>> > >>> escreveu: > >>> > >>>> Any ideas how to delete these rows? > >>>> > >>>> I see only this way: > >>>> - backup data from region that contains “damaged” rows > >>>> - close region > >>>> - remove region files from HDFS > >>>> - assign region > >>>> - copy needed rows from backup to recreated region > >>>> > >>>>> On 30 Apr 2020, at 21:00, Alexander Batyrshin <0x62...@gmail.com > <mailto:0x62...@gmail.com>> > >> wrote: > >>>>> > >>>>> The same effect for CF: > >>>>> > >>>>> d = > >>>> > >> > org.apache.hadoop.hbase.client.Delete.new("\x0439d58wj434dd".to_s.to_java_bytes) > >>>>> d.deleteFamily("d".to_s.to_java_bytes, > >>>> 9223372036854775807.to_java(Java::long)) > >>>>> table.delete(d) > >>>>> > >>>>> ROW > >> COLUMN+CELL > >>>>> \x0439d58wj434dd > column=d:, > >>>> timestamp=1588269277879, type=DeleteFamily > >>>>> > >>>>> > >>>>>> On 29 Apr 2020, at 18:30, Wellington Chevreuil < > >>>> wellington.chevre...@gmail.com <mailto:wellington.chevre...@gmail.com> > <mailto:wellington.chevre...@gmail.com <mailto: > wellington.chevre...@gmail.com>> > >> <mailto:wellington.chevre...@gmail.com <mailto: > wellington.chevre...@gmail.com> <mailto: > >> wellington.chevre...@gmail.com <mailto:wellington.chevre...@gmail.com > >>>> > >>>> wrote: > >>>>>> > >>>>>> Well, it's weird that puts with such TS values were allowed, > according > >>>> to > >>>>>> current code state. Can you afford delete the whole CF for those > rows? > >>>>>> > >>>>>> Em qua., 29 de abr. de 2020 às 14:41, junhyeok park < > >>>> runnerren...@gmail.com <mailto:runnerren...@gmail.com> <mailto: > runnerren...@gmail.com <mailto:runnerren...@gmail.com>> <mailto: > >> runnerren...@gmail.com <mailto:runnerren...@gmail.com> <mailto: > runnerren...@gmail.com <mailto:runnerren...@gmail.com>>>> > >>>>>> escreveu: > >>>>>> > >>>>>>> I've been through the same thing. I use 2.2.0 > >>>>>>> > >>>>>>> 2020년 4월 29일 (수) 오후 10:32, Alexander Batyrshin <0x62...@gmail.com > <mailto:0x62...@gmail.com> > >> <mailto:0x62...@gmail.com <mailto:0x62...@gmail.com>> > >>>> <mailto:0x62...@gmail.com <mailto:0x62...@gmail.com> <mailto: > 0x62...@gmail.com <mailto:0x62...@gmail.com>>>>님이 작성: > >>>>>>> > >>>>>>>> As you can see in example I already tried DELETE operation with > >>>> timestamp > >>>>>>>> = Long.MAX_VALUE without any success. > >>>>>>>> > >>>>>>>>> On 29 Apr 2020, at 12:41, Wellington Chevreuil < > >>>>>>>> wellington.chevre...@gmail.com <mailto: > wellington.chevre...@gmail.com> <mailto: > >> wellington.chevre...@gmail.com <mailto:wellington.chevre...@gmail.com>> > <mailto:wellington.chevre...@gmail.com <mailto: > wellington.chevre...@gmail.com> > >> <mailto:wellington.chevre...@gmail.com <mailto: > wellington.chevre...@gmail.com>>>> > >>>> wrote: > >>>>>>>>> > >>>>>>>>> That's expected behaviour [1]. If you are "travelling to the > >> future", > >>>>>>> you > >>>>>>>>> need to do a delete specifying Long.MAX_VALUE timestamp as the > >>>>>>> timestamp > >>>>>>>>> optional parameter in the delete operation [2], if you don't > >> specify > >>>>>>>>> timestamp on the delete, it will assume current time for the > delete > >>>>>>>> marker, > >>>>>>>>> which will be smaller than the Long.MAX_VALUE set to your cells, > so > >>>>>>> scans > >>>>>>>>> wouldn't filter it. > >>>>>>>>> > >>>>>>>>> [1] https://hbase.apache.org/book.html#version.delete < > https://hbase.apache.org/book.html#version.delete> < > >> https://hbase.apache.org/book.html#version.delete < > https://hbase.apache.org/book.html#version.delete>> < > >>>> https://hbase.apache.org/book.html#version.delete < > https://hbase.apache.org/book.html#version.delete> < > >> https://hbase.apache.org/book.html#version.delete < > https://hbase.apache.org/book.html#version.delete>>> > >>>>>>>>> [2] > >>>>>>>>> > >>>>>>>> > >>>>>>> > >>>> > >> > https://github.com/apache/hbase/blob/branch-1.4/hbase-client/src/main/java/org/apache/hadoop/hbase/client/Delete.java#L98 > < > https://github.com/apache/hbase/blob/branch-1.4/hbase-client/src/main/java/org/apache/hadoop/hbase/client/Delete.java#L98 > > > >> < > >> > https://github.com/apache/hbase/blob/branch-1.4/hbase-client/src/main/java/org/apache/hadoop/hbase/client/Delete.java#L98 > < > https://github.com/apache/hbase/blob/branch-1.4/hbase-client/src/main/java/org/apache/hadoop/hbase/client/Delete.java#L98 > > > >>> > >>>> < > >>>> > >> > https://github.com/apache/hbase/blob/branch-1.4/hbase-client/src/main/java/org/apache/hadoop/hbase/client/Delete.java#L98 > < > https://github.com/apache/hbase/blob/branch-1.4/hbase-client/src/main/java/org/apache/hadoop/hbase/client/Delete.java#L98 > > > >>>>> > >>>>>>>>> > >>>>>>>>> Em qua., 29 de abr. de 2020 às 08:57, Alexander Batyrshin < > >>>>>>>> 0x62...@gmail.com> > >>>>>>>>> escreveu: > >>>>>>>>> > >>>>>>>>>> Hello all, > >>>>>>>>>> We had faced with strange situation: table has rows with > >>>>>>> Long.MAX_VALUE > >>>>>>>>>> timestamp. > >>>>>>>>>> These rows impossible to delete, because DELETE mutation uses > >>>>>>>>>> System.currentTimeMillis() timestamp. > >>>>>>>>>> Is there any way to delete these rows? > >>>>>>>>>> We use HBase-1.4.10 > >>>>>>>>>> > >>>>>>>>>> Example: > >>>>>>>>>> > >>>>>>>>>> hbase(main):037:0> scan 'TRACET', { ROWPREFIXFILTER => > >>>>>>>> "\x0439d58wj434dd", > >>>>>>>>>> RAW=>true, VERSIONS=>10} > >>>>>>>>>> ROW > >>>>>>> COLUMN+CELL > >>>>>>>>>> \x0439d58wj434dd column=d:_0, > >>>>>>>>>> timestamp=9223372036854775807, value=x > >>>>>>>>>> > >>>>>>>>>> > >>>>>>>>>> hbase(main):045:0* delete 'TRACET', "\x0439d58wj434dd", "d:_0" > >>>>>>>>>> 0 row(s) in 0.0120 seconds > >>>>>>>>>> > >>>>>>>>>> hbase(main):046:0> scan 'TRACET', { ROWPREFIXFILTER => > >>>>>>>> "\x0439d58wj434dd", > >>>>>>>>>> RAW=>true, VERSIONS=>10} > >>>>>>>>>> ROW > >>>>>>> COLUMN+CELL > >>>>>>>>>> \x0439d58wj434dd column=d:_0, > >>>>>>>>>> timestamp=9223372036854775807, value=x > >>>>>>>>>> \x0439d58wj434dd column=d:_0, > >>>>>>>>>> timestamp=1588146570005, type=Delete > >>>>>>>>>> > >>>>>>>>>> > >>>>>>>>>> hbase(main):047:0> delete 'TRACET', "\x0439d58wj434dd", "d:_0", > >>>>>>>>>> 9223372036854775807 > >>>>>>>>>> 0 row(s) in 0.0110 seconds > >>>>>>>>>> > >>>>>>>>>> hbase(main):048:0> scan 'TRACET', { ROWPREFIXFILTER => > >>>>>>>> "\x0439d58wj434dd", > >>>>>>>>>> RAW=>true, VERSIONS=>10} > >>>>>>>>>> ROW > >>>>>>> COLUMN+CELL > >>>>>>>>>> \x0439d58wj434dd column=d:_0, > >>>>>>>>>> timestamp=9223372036854775807, value=x > >>>>>>>>>> \x0439d58wj434dd column=d:_0, > >>>>>>>>>> timestamp=1588146678086, type=Delete > >>>>>>>>>> \x0439d58wj434dd column=d:_0, > >>>>>>>>>> timestamp=1588146570005, type=Delete > >