keith-turner commented on PR #4333:
URL: https://github.com/apache/accumulo/pull/4333#issuecomment-1974074548
I have been looking at logs at lot and running test while making this
change. While doing this realized that it would be nice to link fate id and
external compaction ids in a single line. This change does that linking.
Below is some output from running
`CompactionIT.testMultiStepCompactionThatDeletesAll()`. This test initiates a
table compaction on a tablet that has more files than a single compaction will
process.
```
$ grep 4a66e792f3b4760a Manager_1145795413.out
2024-03-01T17:40:01,940 82 [fate.Fate] INFO : Seeding
FATE:USER:4a66e792f3b4760a TABLE_COMPACT Compact table (1) with config
[iterators=[name:GrepIterator, priority:100,
class:org.apache.accumulo.core.iterators.user.GrepIterator,
properties:{term=keep}]]
2024-03-01T17:40:01,966 113 [tableOps.Utils] INFO : namespace +default
FATE:USER:4a66e792f3b4760a locked for READ operation: COMPACT
2024-03-01T17:40:01,971 113 [tableOps.Utils] INFO : table 1
FATE:USER:4a66e792f3b4760a locked for READ operation: COMPACT
2024-03-01T17:40:02,009 111 [tablet.files] TRACE: Selected files 1<<
FATE:USER:4a66e792f3b4760a [F0000004.rf, F0000005.rf, F0000006.rf, F0000007.rf,
F0000008.rf, F0000009.rf, F000000a.rf, F000000b.rf, F000000c.rf, F000000d.rf,
F000000e.rf, F000000f.rf, F000000g.rf, F000000h.rf, F000000i.rf, F000000j.rf,
F000000k.rf, F000000l.rf, F000000m.rf, F000000n.rf, F000000o.rf, F000000p.rf,
F000000q.rf, F000000r.rf, F000000s.rf, F000000t.rf, F000000u.rf, F000000v.rf,
F000000w.rf, F000000x.rf, F000000y.rf, F000000z.rf, F0000010.rf, F0000011.rf,
F0000012.rf, F0000013.rf, F0000014.rf, F0000015.rf, F0000016.rf, F0000017.rf,
F0000018.rf, F0000019.rf, F000001a.rf, F000001b.rf, F000001c.rf, F000001d.rf,
F000001e.rf, F000001f.rf, F000001g.rf, F000001h.rf, F000001i.rf, F000001j.rf,
F000001k.rf, F000001l.rf, F000001m.rf, F000001n.rf, F000001o.rf, F000001p.rf,
F000001q.rf, F000001r.rf]
2024-03-01T17:40:02,009 111 [compact.CompactionDriver] DEBUG:
FATE:USER:4a66e792f3b4760a tablet stats, total:1 complete:0 selected_now:1
selected_prev:0 selected_by_other:0 no_files:0 none_selected:0
user_compaction_requested:0 user_compaction_waiting:0 opids:0
scan_update_time:17ms
2024-03-01T17:40:02,009 111 [manager.EventCoordinator] DEBUG:
FATE:USER:4a66e792f3b4760a selected files for compaction for 1 tablets
2024-03-01T17:40:02,067 83 [tablet.files] DEBUG: Compacting 1<<
driver:FATE:USER:4a66e792f3b4760a id:ECID-bfff40f4-2dbd-43c0-a5a0-b3ce2b667618
group:default compactor:localhost:9134 priority:-2933 size:2 KB kind:USER
files:[F000000c.rf, F000000b.rf, F0000008.rf, F0000007.rf, F000000a.rf,
F0000006.rf, F0000005.rf, F000000d.rf, F0000004.rf, F0000009.rf]
2024-03-01T17:40:02,248 78 [tablet.files] DEBUG: Compacting 1<<
driver:FATE:USER:4a66e792f3b4760a id:ECID-c9c9be4c-464f-45ce-b85d-f1af0a5bcc03
group:default compactor:localhost:9134 priority:-2943 size:2 KB kind:USER
files:[F000000i.rf, F000000h.rf, F000000n.rf, F000000g.rf, F000000f.rf,
F000000m.rf, F000000l.rf, F000000e.rf, F000000k.rf, F000000j.rf]
2024-03-01T17:40:02,368 58 [tablet.files] DEBUG: Compacting 1<<
driver:FATE:USER:4a66e792f3b4760a id:ECID-5a8cf71d-9ce1-462a-893e-9f65519b3293
group:default compactor:localhost:9134 priority:-2953 size:2 KB kind:USER
files:[F000000x.rf, F000000w.rf, F000000p.rf, F000000o.rf, F000000v.rf,
F000000u.rf, F000000t.rf, F000000s.rf, F000000r.rf, F000000q.rf]
2024-03-01T17:40:02,471 74 [tablet.files] DEBUG: Compacting 1<<
driver:FATE:USER:4a66e792f3b4760a id:ECID-0e31175d-eedf-4c38-8b06-4f7f9cf5ee46
group:default compactor:localhost:9134 priority:-2963 size:2 KB kind:USER
files:[F0000015.rf, F0000014.rf, F0000013.rf, F0000012.rf, F000000z.rf,
F000000y.rf, F0000011.rf, F0000010.rf, F0000017.rf, F0000016.rf]
2024-03-01T17:40:02,525 111 [compact.CompactionDriver] DEBUG:
FATE:USER:4a66e792f3b4760a tablet stats, total:1 complete:0 selected_now:0
selected_prev:1 selected_by_other:0 no_files:0 none_selected:0
user_compaction_requested:0 user_compaction_waiting:0 opids:0
scan_update_time:5ms
2024-03-01T17:40:02,576 73 [tablet.files] DEBUG: Compacting 1<<
driver:FATE:USER:4a66e792f3b4760a id:ECID-ba089e00-d6d3-47ee-ab77-dcaace7dfd21
group:default compactor:localhost:9134 priority:-2973 size:2 KB kind:USER
files:[F000001f.rf, F000001e.rf, F000001d.rf, F000001c.rf, F0000019.rf,
F0000018.rf, F000001b.rf, F000001a.rf, F000001h.rf, F000001g.rf]
2024-03-01T17:40:02,758 64 [tablet.files] DEBUG: Compacting 1<<
driver:FATE:USER:4a66e792f3b4760a id:ECID-57ca0286-9fbf-4e88-9f98-c261754e7e79
group:default compactor:localhost:9134 priority:-2983 size:2 KB kind:USER
files:[F000001m.rf, F000001l.rf, F000001k.rf, F000001r.rf, F000001j.rf,
F000001q.rf, F000001p.rf, F000001i.rf, F000001o.rf, F000001n.rf]
2024-03-01T17:40:02,816 112 [commit.CommitCompaction] DEBUG: All selected
files compacted for 1<< setting compacted for FATE:USER:4a66e792f3b4760a
2024-03-01T17:40:03,043 114 [compact.CompactionDriver] DEBUG:
FATE:USER:4a66e792f3b4760a tablet stats, total:1 complete:1 selected_now:0
selected_prev:0 selected_by_other:0 no_files:0 none_selected:0
user_compaction_requested:0 user_compaction_waiting:0 opids:0
scan_update_time:3ms
2024-03-01T17:40:03,083 111 [compact.CleanUp] DEBUG:
FATE:USER:4a66e792f3b4760a removed 1 of 1 compacted markers for 1 tablets in 1ms
```
* The fate op `FATE:USER:4a66e792f3b4760a` drives the table compaction
* The fate operation selects a bunch of files for the only tablet in the
table. Can see that in the new log message ` [tablet.files] TRACE: Selected
files 1<< FATE:USER:4a66e792f3b4760a [F0000004.rf, `
* The selected files on the tablet cause compactions to start happening.
Can see these starting with messages like `Compacting 1<<
driver:FATE:USER:4a66e792f3b4760a id:ECID-ba089e00-d6d3-47ee-ab77-dcaace7dfd21
` This log message links the fate id driving the table compaction to
individual compaction ids.
Can take one of the compaction ids linked from the fate operation and grep
for it to get more information.
```
$ grep ECID-0e31175d-eedf-4c38-8b06-4f7f9cf5ee46 Manager_1145795413.out
2024-03-01T17:40:02,471 74 [tablet.files] DEBUG: Compacting 1<<
driver:FATE:USER:4a66e792f3b4760a id:ECID-0e31175d-eedf-4c38-8b06-4f7f9cf5ee46
group:default compactor:localhost:9134 priority:-2963 size:2 KB kind:USER
files:[F0000015.rf, F0000014.rf, F0000013.rf, F0000012.rf, F000000z.rf,
F000000y.rf, F0000011.rf, F0000010.rf, F0000017.rf, F0000016.rf]
2024-03-01T17:40:02,473 62 [coordinator.CompactionCoordinator] DEBUG:
Compaction status update, id: ECID-0e31175d-eedf-4c38-8b06-4f7f9cf5ee46,
timestamp: 1709332802472, update: TCompactionStatusUpdate(state:STARTED,
message:Compaction started, entriesToBeCompacted:-1, entriesRead:-1,
entriesWritten:-1)
2024-03-01T17:40:02,490 65 [coordinator.CompactionCoordinator] DEBUG:
Compaction status update, id: ECID-0e31175d-eedf-4c38-8b06-4f7f9cf5ee46,
timestamp: 1709332802490, update: TCompactionStatusUpdate(state:SUCCEEDED,
message:Compaction completed successfully, entriesToBeCompacted:-1,
entriesRead:-1, entriesWritten:-1)
2024-03-01T17:40:02,491 78 [coordinator.CompactionCoordinator] INFO :
Compaction completed, id: ECID-0e31175d-eedf-4c38-8b06-4f7f9cf5ee46, stats:
TCompactionStats(entriesRead:10, entriesWritten:0, fileSize:145), extent: 1<<
2024-03-01T17:40:02,500 78 [fate.Fate] INFO : Seeding
FATE:USER:6efa9c0980b8b4b2 Commit compaction
ECID-0e31175d-eedf-4c38-8b06-4f7f9cf5ee46
2024-03-01T17:40:02,543 114 [tablet.files] DEBUG:
ECID-0e31175d-eedf-4c38-8b06-4f7f9cf5ee46 compacted 1<< for USER created no
output from [F000000z.rf, F0000014.rf, F000000y.rf, F0000015.rf, F0000016.rf,
F0000017.rf, F0000011.rf, F0000013.rf, F0000010.rf, F0000012.rf]
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
