Hi Dears,
I'm using ActiveMQ 5.15.9 and after about one month our customers started
to complain about the Kahadb folder increase until reaching its limit and
crash the system. Then they delete the folder to recover the system, but we
are losing data in this process.
We activate the kahadb log and seems that some old messages are getting
stuck for some reason and do not allow the ActiveMQ to delete the db-x.log
files:
*2022-09-29 17:39:51,555 | TRACE | ackMessageFileMap:* {244=[244, 243],
245=[123], 246=[244, 246], 247=[246, 247], 248=[123], 249=[249, 247],
250=[163, 211, 249, 250, 234, 237], 251=[250, 243, 251, 247], 252=[140,
108, 97], 253=[249, 253, 251], 254=[253, 254], 255=[140, 141], 256=[256,
250, 251, 253, 254], 257=[256, 257], 258=[141], 259=[257, 259], 260=[259,
260, 254], 261=[140], 262=[260, 262], 263=[262, 263], 264=[264, 262, 254,
263], 265=[264, 265, 253], 266=[265, 249, 266, 250, 253], 267=[159],
268=[266, 268, 253, 79], 269=[259, 268, 269, 262, 254, 263], 270=[159],
271=[269, 271], 272=[272, 265, 266, 271], 273=[163], 274=[272, 274, 271],
275=[274, 275], 276=[163], 277=[265, 274, 266, 275, 277], 278=[277, 278],
279=[163], 280=[280, 278, 266], 281=[280, 281], 282=[281, 282], 283=[159],
284=[284, 282], 285=[284, 285], 286=[174], 287=[285, 287], 288=[288, 287],
289=[174], 290=[288, 290], 291=[290, 291], 292=[163], 293=[293, 291],
294=[293, 294], 295=[294, 295], 296=[296, 295], 297=[296, 297], 298=[174,
159], 299=[297, 299], 300=[300, 299], 301=[180], 302=[300, 302], 303=[302,
303], 304=[180], 305=[305, 303], 306=[305, 306], 307=[182], 308=[308, 306,
287], 309=[308, 309], 310=[182], 311=[309, 311], 312=[312, 311], 313=[312,
313], 314=[313, 314], 315=[284, 314, 315], 316=[174], 317=[317, 315],
318=[317, 318], 319=[198], 320=[320, 318], 321=[320, 321], 322=[199],
323=[321, 323], 324=[324, 323], 325=[199], 326=[272, 274, 324, 326, 278,
294, 296], 327=[326, 327], 328=[328, 327], 329=[328, 329], 330=[329, 330],
331=[208], 332=[332, 330], 333=[332, 333], 334=[208], 335=[250, 251, 333,
253, 335], 336=[336, 335], 337=[336, 337, 211], 338=[20], 339=[337, 339],
342=[337, 342, 339], 343=[342, 343], 344=[20, 21], 345=[345, 343],
346=[345, 346], 347=[337, 345, 346, 347], 348=[348, 345, 347], 349=[348,
337, 349], 350=[349, 350], 351=[272, 337, 274, 294, 296, 350, 302, 351,
271], 352=[352, 337, 351], 353=[352, 353, 337, 285], 354=[21], 355=[353,
355], 356=[337, 274, 355, 356, 294, 299, 302], 357=[24], 358=[356, 358],
359=[358, 359], 360=[256, 272, 337, 274, 294, 359, 263, 360, 296, 265,
266], 361=[337, 274, 263, 359, 360, 361, 265, 253, 302], 362=[361, 362],
363=[360, 362, 363], 364=[364, 361, 363], 365=[24], 366=[364, 366],
367=[337, 345, 346, 366, 367, 351], 368=[21], 369=[369, 367], 370=[369,
370], 371=[28], 372=[337, 369, 370, 372, 345, 346, 351], 373=[372, 373],
374=[373, 374], 375=[374, 375], 376=[376, 345, 375, 351], 377=[376, 377],
378=[369, 374, 376, 377, 345, 378, 351], 379=[28], 380=[380, 378],
381=[380, 381], 382=[28], 383=[381, 383], 384=[384, 381, 383], 385=[384,
385, 366], 386=[352, 384, 385, 386, 350, 351, 367, 383], 387=[337, 385,
386, 387, 345, 378, 348, 366, 351], 388=[388, 387], 389=[352, 384, 388,
389, 351, 383], 390=[376, 369, 380, 389, 390, 374], 391=[390, 391],
392=[34], 393=[393, 387, 389, 390, 391], 394=[393, 394], 395=[34],
396=[385, 259, 387, 390, 296, 360, 265, 394, 266, 363, 396, 366, 337, 274,
243, 277, 278, 250, 284, 253], 397=[396, 397, 394], 398=[28, 34], 399=[380,
397, 374, 399], 400=[400, 384, 390, 362, 380, 396, 350, 399, 351],
401=[400, 401], 402=[37], 403=[401, 403], 404=[404, 403], 405=[37, 38],
406=[404, 406], 407=[406, 407], 408=[34, 38, 39], 409=[409, 407], 410=[409,
410], 411=[409, 410, 411], 412=[40, 39], 413=[413, 411], 414=[401, 409,
389, 413, 414], 415=[40], 416=[416, 414], 417=[416, 417], 418=[38],
419=[417, 419], 420=[420, 414, 419], 421=[400, 403, 420, 421], 422=[385,
419, 420, 421, 422], 423=[387, 419, 404, 420, 421, 422, 423, 394, 414],
424=[424, 419, 420, 423], 425=[424, 425], 426=[419, 404, 420, 422, 425,
426, 394, 414], 427=[426, 419, 427, 420], 428=[428, 427], 429=[400, 424,
403, 428, 429], 430=[48], 431=[419, 403, 404, 420, 424, 429, 414, 431],
432=[432, 431], 433=[48, 49], 434=[432, 434], 435=[424, 434, 435], 436=[49,
51], 437=[437, 435], 438=[428, 437, 438, 431], 439=[48, 49, 51, 52, 40],
440=[438, 440], 441=[440, 441], 442=[52, 53, 21], 443=[438, 441, 443],
444=[438, 440, 443, 444], 445=[438, 444, 445], 446=[445, 446], 447=[438,
446, 447], 448=[448, 438, 440, 445, 446, 447], 449=[448, 449], 450=[53,
54], 451=[449, 451], 452=[337, 451, 419, 452, 404, 420, 438, 394, 414],
453=[49, 54, 55, 40], 454=[452, 454], 455=[454, 455], 456=[55, 57],
457=[449, 438, 455, 457, 441], 458=[457, 458], 459=[458, 459], 460=[49, 53,
57, 58], 461=[437, 438, 459, 461], 462=[438, 458, 459, 461, 462], 463=[58,
60], 464=[464, 462], 465=[464, 465, 459], 466=[465, 466], 467=[466, 467],
468=[466, 467, 468], 469=[468, 469, 459], 470=[469, 470], 471=[470, 471],
472=[471, 472], 473=[472, 473], 474=[471, 472, 473, 474], 475=[474, 475],
476=[475, 476], 477=[476, 477], 478=[477, 478], 479=[34, 40, 60, 61],
480=[480, 478], 481=[480, 481], 482=[61], 483=[481, 483], 484=[483, 484],
485=[64, 40], 486=[484, 486], 487=[486, 487], 488=[487, 488], 489=[64, 66],
490=[488, 490], 491=[490, 491], 492=[66, 67], 493=[491, 493], 494=[493,
494], 495=[64, 67, 68], 496=[496, 464, 465, 494], 497=[496, 497, 438],
498=[497, 498], 499=[68, 70], 500=[465, 497, 498, 500, 438], 501=[500,
501], 502=[70, 71], 503=[420, 484, 501, 437, 438, 486, 503, 457, 490, 459,
446], 504=[503, 504], 505=[71, 73], 506=[504, 506], 507=[506, 507],
508=[507, 508], 509=[508, 509], 510=[509, 510], 511=[510, 511], 512=[512,
438, 511], 513=[73], 514=[512, 514], 515=[514, 515], 516=[76], 517=[515,
517], 518=[386, 387, 452, 517, 421, 518, 424, 459, 397, 429, 465, 500, 438,
503, 506], 519=[352, 337, 437, 518, 438, 519, 345, 381, 511], 520=[519,
520], 521=[519, 520, 521], 522=[521, 522], 523=[515, 521, 522, 523],
524=[76, 77], 525=[515, 521, 522, 523, 525], 526=[514, 515, 521, 525, 526],
527=[77, 79], 528=[528, 526], 529=[528, 529], 530=[79], 531=[529, 531],
532=[531, 532, 520], 533=[532, 533], 534=[533, 534], 535=[512, 517, 534,
535], 536=[81], 537=[535, 537], 538=[537, 538], 539=[81, 82, 71], 540=[538,
540], 541=[540, 541], 542=[82, 83], 543=[535, 541, 543], 544=[544, 543],
545=[544, 545], 546=[83], 547=[545, 547], 548=[544, 545, 547, 531, 548,
535, 521, 538, 543], 549=[79], 550=[548, 550], 551=[548, 550, 551],
552=[97], 553=[551, 553], 554=[337, 531, 438, 535, 553, 554], 555=[97],
556=[548, 554, 538, 556], 557=[545, 556, 557], 558=[557, 558], 559=[64, 68,
70], 560=[560, 544, 554, 558], 561=[560, 561], 562=[64, 82, 68, 70, 55,
105, 57], 563=[561, 563], 564=[563, 564, 537, 558], 565=[564, 565],
566=[565, 566, 556, 558], 567=[566, 567], 568=[567, 568], 569=[568, 569],
570=[566, 569, 570, 558], 571=[570, 571], 572=[211], 573=[571, 573],
574=[573, 574], 575=[211, 214], 576=[576, 574], 577=[576, 544, 577, 561,
554, 557], 578=[97, 105], 579=[577, 579, 565, 566], 580=[579, 580],
581=[214], 582=[580, 582], 583=[582, 583], 584=[566, 583, 584, 558],
585=[108], 586=[566, 584, 586], 587=[108], 588=[583, 586, 588], 589=[211],
590=[586, 588, 590], 591=[590, 591], 592=[211], 593=[593, 586, 591],
594=[593, 594, 586], 595=[594, 595], 596=[595, 596, 586], 597=[596, 597],
598=[111], 599=[596, 597, 599, 586], 600=[597, 599, 600], 601=[597, 599,
600, 601], 602=[601, 602, 586, 588], 603=[211], 604=[583, 599, 602, 586,
604], 605=[604, 605], 606=[586, 605, 606], 607=[113, 97, 108, 111],
608=[608, 596, 599, 586, 606], 609=[608, 609], 610=[113], 611=[609, 611,
586, 588], 612=[611, 612, 583, 599, 586], 613=[608, 612, 613, 599, 600],
614=[609, 596, 613, 614, 599, 586], 615=[614, 615], 616=[608, 615, 599,
616, 600, 586], 617=[615, 616, 617], 618=[611, 613, 617, 618], 619=[618,
619], 620=[234], 621=[611, 613, 619, 621], 622=[608, 609, 611, 621, 622],
623=[234], 624=[624, 608, 612, 613, 614, 615, 622], 625=[624, 625, 616,
621], 626=[234, 237], 627=[625, 627], 628=[624, 625, 627, 628, 615, 619,
622], 629=[237], 630=[628, 630], 631=[630, 631], 632=[97, 108, 79],
633=[624, 625, 630, 631, 633, 622], 634=[633, 634], 635=[599, 631, 634,
635, 621], 636=[97], 637=[635, 637], 638=[619, 637, 638, 622], 639=[243]} |
org.apache.activemq.store.kahadb.MessageDatabase | ActiveMQ Journal
Checkpoint Worker
*2022-09-29 17:39:51,557 | TRACE |* not removing data file: 244 as
contained ack(s) refer to referenced file: [244, 243] |
org.apache.activemq.store.kahadb.MessageDatabase | ActiveMQ Journal
Checkpoint Worker
*2022-09-29 17:39:51,559 | TRACE |* not removing data file: 245 as
contained ack(s) refer to referenced file: [123] |
org.apache.activemq.store.kahadb.MessageDatabase | ActiveMQ Journal
Checkpoint Worker
*2022-09-29 17:39:51,561 | TRACE |* not removing data file: 246 as
contained ack(s) refer to referenced file: [244, 246] |
org.apache.activemq.store.kahadb.MessageDatabase | ActiveMQ Journal
Checkpoint Worker
*2022-09-29 17:39:51,563 | TRACE |* not removing data file: 247 as
contained ack(s) refer to referenced file: [246, 247] |
org.apache.activemq.store.kahadb.MessageDatabase | ActiveMQ Journal
Checkpoint Worker
*2022-09-29 17:39:51,564 | WARN | *Stealing link for clientId
TR_VP-018:30246 From Connection Transport Connection to: tcp://
172.24.96.125:42412 | org.apache.activemq.broker.region.RegionBroker |
ActiveMQ Transport: tcp:///172.24.96.125:42414@1883
*2022-09-29 17:39:51,565 | TRACE |* not removing data file: 248 as
contained ack(s) refer to referenced file: [123] |
org.apache.activemq.store.kahadb.MessageDatabase | ActiveMQ Journal
Checkpoint Worker
*2022-09-29 17:39:51,565 | TRACE |* not removing data file: 249 as
contained ack(s) refer to referenced file: [249, 247] |
org.apache.activemq.store.kahadb.MessageDatabase | ActiveMQ Journal
Checkpoint Worker
*2022-09-29 17:39:51,565 | TRACE |* not removing data file: 250 as
contained ack(s) refer to referenced file: [163, 211, 249, 250, 234, 237] |
org.apache.activemq.store.kahadb.MessageDatabase | ActiveMQ Journal
Checkpoint Worker
*2022-09-29 17:39:51,565 | TRACE |* not removing data file: 251 as
contained ack(s) refer to referenced file: [250, 243, 251, 247] |
org.apache.activemq.store.kahadb.MessageDatabase | ActiveMQ Journal
Checkpoint Worker
*2022-09-29 17:39:51,565 | TRACE |* not removing data file: 248 as
contained ack(s) refer to referenced file: [123] |
org.apache.activemq.store.kahadb.MessageDatabase | ActiveMQ Journal
Checkpoint Worker
*2022-09-29 17:39:51,565 | TRACE |* not removing data file: 249 as
contained ack(s) refer to referenced file: [249, 247] |
org.apache.activemq.store.kahadb.MessageDatabase | ActiveMQ Journal
Checkpoint Worker
*2022-09-29 17:39:51,565 | TRACE |* not removing data file: 250 as
contained ack(s) refer to referenced file: [163, 211, 249, 250, 234, 237] |
org.apache.activemq.store.kahadb.MessageDatabase | ActiveMQ Journal
Checkpoint Worker
*2022-09-29 17:39:51,565 | TRACE |* not removing data file: 251 as
contained ack(s) refer to referenced file: [250, 243, 251, 247] |
org.apache.activemq.store.kahadb.MessageDatabase | ActiveMQ Journal
Checkpoint Worker
*2022-09-29 17:39:51,565 | TRACE |* not removing data file: 252 as
contained ack(s) refer to referenced file: [140, 108, 97] |
org.apache.activemq.store.kahadb.MessageDatabase | ActiveMQ Journal
Checkpoint Worker
*2022-09-29 17:39:51,565 | TRACE | *not removing data file: 253 as
contained ack(s) refer to referenced file: [249, 253, 251] |
org.apache.activemq.store.kahadb.MessageDatabase | ActiveMQ Journal
Checkpoint Worker
*2022-09-29 17:39:51,565 | TRACE |* not removing data file: 254 as
contained ack(s) refer to referenced file: [253, 254] |
org.apache.activemq.store.kahadb.MessageDatabase | ActiveMQ Journal
Checkpoint Worker
*2022-09-29 17:39:51,565 | TRACE |* not removing data file: 255 as
contained ack(s) refer to referenced file: [140, 141] |
org.apache.activemq.store.kahadb.MessageDatabase | ActiveMQ Journal
Checkpoint Worker not removing data file: 252 as contained ack(s) refer to
referenced file: [140, 108, 97] |
org.apache.activemq.store.kahadb.MessageDatabase | ActiveMQ Journal
Checkpoint Worker
*2022-09-29 17:39:51,565 | TRACE |* not removing data file: 253 as
contained ack(s) refer to referenced file: [249, 253, 251] |
org.apache.activemq.store.kahadb.MessageDatabase | ActiveMQ Journal
Checkpoint Worker
*2022-09-29 17:39:51,565 | TRACE | *not removing data file: 254 as
contained ack(s) refer to referenced file: [253, 254] |
org.apache.activemq.store.kahadb.MessageDatabase | ActiveMQ Journal
Checkpoint Worker
*2022-09-29 17:39:51,565 | TRACE |* not removing data file: 255 as
contained ack(s) refer to referenced file: [140, 141] |
org.apache.activemq.store.kahadb.MessageDatabase | ActiveMQ Journal
Checkpoint Worker
We also use durable topics as the devices that subscribe to these topics
and send MQTT messages to others work in a very bad network environment,
this is the way we found to avoid data loss while offline.
I think the problem is related to durable topics with offline
subscriptions, but how can I confirm this assumption, eg which topics? What
signatures?
I much appreciate any thoughts!
Thanks in advance
