keith-turner commented on PR #287: URL: https://github.com/apache/accumulo-testing/pull/287#issuecomment-2481629433
The following is example output that shows running these changes where it continually bulk imports into 3 random tablets on a table with 20 tablets. Around every 6s it bulk imports 3 files to 3 tablets w/ one million total key values. ``` $ ./bin/cingest ingest -o test.ci.ingest.bulk.workdir=hdfs://localhost:8020/ci_bulk -o test.ci.ingest.max.tablets=3 2024-11-17T22:11:52,292 [testing.continuous.ContinuousIngest] INFO : Ingest instance ID: 655f7838-dde9-4c28-8c6f-a05211d5c4e1 current time: 1731881512290ms 2024-11-17T22:11:52,293 [testing.continuous.ContinuousIngest] INFO : A flush will occur after every 1000000 entries written 2024-11-17T22:11:52,305 [testing.continuous.ContinuousIngest] INFO : Total entries to be written: 9223372036854775807 2024-11-17T22:11:52,307 [testing.continuous.ContinuousIngest] INFO : DELETES will occur with a probability of 0.10 2024-11-17T22:11:54,385 [testing.continuous.BulkBatchWriter] DEBUG: Created new file bbw-00000.rf for range 0cccccccccccccce 1333333333333335 2024-11-17T22:11:55,480 [testing.continuous.BulkBatchWriter] DEBUG: Created new file bbw-00001.rf for range 6cccccccccccccd7 733333333333333e 2024-11-17T22:11:56,367 [testing.continuous.BulkBatchWriter] DEBUG: Created new file bbw-00002.rf for range 733333333333333e 79999999999999a5 2024-11-17T22:11:58,049 [testing.continuous.BulkBatchWriter] DEBUG: Bulk imported dir hdfs://localhost:8020/ci_bulk/5455e07d-ee97-4fcd-b526-65d75a31697c destinations:3 mutations:1000000 memUsed:331000000 time:791ms 2024-11-17T22:11:58,054 [testing.continuous.ContinuousIngest] INFO : FLUSH - duration: 4410ms, since last flush: 5749ms, total written: 1000000, total deleted: 0 2024-11-17T22:11:59,862 [testing.continuous.BulkBatchWriter] DEBUG: Created new file bbw-00000.rf for range 399999999999999f 4000000000000006 2024-11-17T22:12:00,930 [testing.continuous.BulkBatchWriter] DEBUG: Created new file bbw-00001.rf for range 466666666666666d 4cccccccccccccd4 2024-11-17T22:12:01,976 [testing.continuous.BulkBatchWriter] DEBUG: Created new file bbw-00002.rf for range 6000000000000009 666666666666667 2024-11-17T22:12:03,742 [testing.continuous.BulkBatchWriter] DEBUG: Bulk imported dir hdfs://localhost:8020/ci_bulk/05d3d626-265e-4b01-a1bd-6053d108cca6 destinations:3 mutations:1000000 memUsed:347000000 time:734ms 2024-11-17T22:12:03,744 [testing.continuous.ContinuousIngest] INFO : FLUSH - duration: 4515ms, since last flush: 5690ms, total written: 2000000, total deleted: 0 2024-11-17T22:12:05,524 [testing.continuous.BulkBatchWriter] DEBUG: Created new file bbw-00000.rf for range 199999999999999c 2000000000000003 2024-11-17T22:12:06,573 [testing.continuous.BulkBatchWriter] DEBUG: Created new file bbw-00001.rf for range 4000000000000006 466666666666666d 2024-11-17T22:12:07,602 [testing.continuous.BulkBatchWriter] DEBUG: Created new file bbw-00002.rf for range 59999999999999a2 6000000000000009 2024-11-17T22:12:09,342 [testing.continuous.BulkBatchWriter] DEBUG: Bulk imported dir hdfs://localhost:8020/ci_bulk/eee6174e-f539-4a81-a28f-84a0dd8d9932 destinations:3 mutations:1000000 memUsed:347000000 time:719ms 2024-11-17T22:12:09,344 [testing.continuous.ContinuousIngest] INFO : FLUSH - duration: 4427ms, since last flush: 5600ms, total written: 3000000, total deleted: 0 2024-11-17T22:12:11,094 [testing.continuous.BulkBatchWriter] DEBUG: Created new file bbw-00000.rf for range 2000000000000003 266666666666666a 2024-11-17T22:12:12,168 [testing.continuous.BulkBatchWriter] DEBUG: Created new file bbw-00001.rf for range 4000000000000006 466666666666666d 2024-11-17T22:12:13,186 [testing.continuous.BulkBatchWriter] DEBUG: Created new file bbw-00002.rf for range 59999999999999a2 6000000000000009 2024-11-17T22:12:14,924 [testing.continuous.BulkBatchWriter] DEBUG: Bulk imported dir hdfs://localhost:8020/ci_bulk/4cf6970c-fed7-4f51-bb1c-03274ae05c48 destinations:3 mutations:1000000 memUsed:347000000 time:715ms 2024-11-17T22:12:14,926 [testing.continuous.ContinuousIngest] INFO : FLUSH - duration: 4405ms, since last flush: 5582ms, total written: 4000000, total deleted: 0accu ``` I have run the bulk changes and the live ingest into the same table and then successfully ran the verify map reduce job. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
