Hello Marco,
First of All, Thanks for your reply and detailed explanation.
To avoid increasing the KahaDB with DLQ queues, we disable the DLQ, so
expired messages will be discarded automatically:
<deadLetterStrategy>
<sharedDeadLetterStrategy processExpired="false" />
</deadLetterStrategy>
Because of that configuration, I assume that the problems fall on durable
topics.
As we know the number of durable topics must be created every time our
system start, I'll study virtual topics and check if it is suitable for our
purposes. I'll let you ASAP.
Thanks!
*Rafael Pimenta*
Em dom., 2 de out. de 2022 às 08:08, Marco Bettiol <jamiro...@gmail.com>
escreveu:
> Hi Rafael,
> I’m just an ActiveMQ classic user.
>
> I will report the behavior of ActiveMQ I experienced.
>
> As far as I know the ActiveMQ classic journal is composed of append only
> files.
> Each file should be by default 32MB in size (this can be changed, I think
> is controlled by journalLogFileSize)
> A journal file gets released only when ALL the messages within are
> consumed.
> Furthermore, messages form different queues are appended to and mixed
> within the same file until the maximum size gets reached.
>
> As consequence if you have unprocessed messages (DLQ or offline consumers)
> the used storage will increase, usually in the same order of magnitude as
> the total traffic served by the broker: a small percentage of unconsumed
> messages will prevent journal file cleanup.
>
> For example an unprocessed 1KByte message will retain 32MB on disk.
>
> As far as I know there is no building functionality to perform journal
> compaction.
>
> In our case the developed internally a small application to offload
> unprocessed messages to an external database/ requeue DLQ messages to the
> same DLQ in order to free storage.
>
> In my experience persistent Topics are quite hard to monitor, if the list
> of subscribers is not dynamic (known when application starts) you could
> think to use VirtualTopics so that you will get back to a normal queue uses
> case for each consumer.
>
> Regards,
> Marc
>
> Il giorno gio 29 set 2022 alle ore 23:06 Rafael Pimenta <
> rafa.spime...@gmail.com> ha scritto:
>
> > Hi Dears,
> > I'm using ActiveMQ 5.15.9 and after about one month our customers started
> > to complain about the Kahadb folder increase until reaching its limit and
> > crash the system. Then they delete the folder to recover the system, but
> we
> > are losing data in this process.
> > We activate the kahadb log and seems that some old messages are getting
> > stuck for some reason and do not allow the ActiveMQ to delete the
> db-x.log
> > files:
> >
> > *2022-09-29 17:39:51,555 | TRACE | ackMessageFileMap:* {244=[244, 243],
> > 245=[123], 246=[244, 246], 247=[246, 247], 248=[123], 249=[249, 247],
> > 250=[163, 211, 249, 250, 234, 237], 251=[250, 243, 251, 247], 252=[140,
> > 108, 97], 253=[249, 253, 251], 254=[253, 254], 255=[140, 141], 256=[256,
> > 250, 251, 253, 254], 257=[256, 257], 258=[141], 259=[257, 259], 260=[259,
> > 260, 254], 261=[140], 262=[260, 262], 263=[262, 263], 264=[264, 262, 254,
> > 263], 265=[264, 265, 253], 266=[265, 249, 266, 250, 253], 267=[159],
> > 268=[266, 268, 253, 79], 269=[259, 268, 269, 262, 254, 263], 270=[159],
> > 271=[269, 271], 272=[272, 265, 266, 271], 273=[163], 274=[272, 274, 271],
> > 275=[274, 275], 276=[163], 277=[265, 274, 266, 275, 277], 278=[277, 278],
> > 279=[163], 280=[280, 278, 266], 281=[280, 281], 282=[281, 282],
> 283=[159],
> > 284=[284, 282], 285=[284, 285], 286=[174], 287=[285, 287], 288=[288,
> 287],
> > 289=[174], 290=[288, 290], 291=[290, 291], 292=[163], 293=[293, 291],
> > 294=[293, 294], 295=[294, 295], 296=[296, 295], 297=[296, 297], 298=[174,
> > 159], 299=[297, 299], 300=[300, 299], 301=[180], 302=[300, 302],
> 303=[302,
> > 303], 304=[180], 305=[305, 303], 306=[305, 306], 307=[182], 308=[308,
> 306,
> > 287], 309=[308, 309], 310=[182], 311=[309, 311], 312=[312, 311],
> 313=[312,
> > 313], 314=[313, 314], 315=[284, 314, 315], 316=[174], 317=[317, 315],
> > 318=[317, 318], 319=[198], 320=[320, 318], 321=[320, 321], 322=[199],
> > 323=[321, 323], 324=[324, 323], 325=[199], 326=[272, 274, 324, 326, 278,
> > 294, 296], 327=[326, 327], 328=[328, 327], 329=[328, 329], 330=[329,
> 330],
> > 331=[208], 332=[332, 330], 333=[332, 333], 334=[208], 335=[250, 251, 333,
> > 253, 335], 336=[336, 335], 337=[336, 337, 211], 338=[20], 339=[337, 339],
> > 342=[337, 342, 339], 343=[342, 343], 344=[20, 21], 345=[345, 343],
> > 346=[345, 346], 347=[337, 345, 346, 347], 348=[348, 345, 347], 349=[348,
> > 337, 349], 350=[349, 350], 351=[272, 337, 274, 294, 296, 350, 302, 351,
> > 271], 352=[352, 337, 351], 353=[352, 353, 337, 285], 354=[21], 355=[353,
> > 355], 356=[337, 274, 355, 356, 294, 299, 302], 357=[24], 358=[356, 358],
> > 359=[358, 359], 360=[256, 272, 337, 274, 294, 359, 263, 360, 296, 265,
> > 266], 361=[337, 274, 263, 359, 360, 361, 265, 253, 302], 362=[361, 362],
> > 363=[360, 362, 363], 364=[364, 361, 363], 365=[24], 366=[364, 366],
> > 367=[337, 345, 346, 366, 367, 351], 368=[21], 369=[369, 367], 370=[369,
> > 370], 371=[28], 372=[337, 369, 370, 372, 345, 346, 351], 373=[372, 373],
> > 374=[373, 374], 375=[374, 375], 376=[376, 345, 375, 351], 377=[376, 377],
> > 378=[369, 374, 376, 377, 345, 378, 351], 379=[28], 380=[380, 378],
> > 381=[380, 381], 382=[28], 383=[381, 383], 384=[384, 381, 383], 385=[384,
> > 385, 366], 386=[352, 384, 385, 386, 350, 351, 367, 383], 387=[337, 385,
> > 386, 387, 345, 378, 348, 366, 351], 388=[388, 387], 389=[352, 384, 388,
> > 389, 351, 383], 390=[376, 369, 380, 389, 390, 374], 391=[390, 391],
> > 392=[34], 393=[393, 387, 389, 390, 391], 394=[393, 394], 395=[34],
> > 396=[385, 259, 387, 390, 296, 360, 265, 394, 266, 363, 396, 366, 337,
> 274,
> > 243, 277, 278, 250, 284, 253], 397=[396, 397, 394], 398=[28, 34],
> 399=[380,
> > 397, 374, 399], 400=[400, 384, 390, 362, 380, 396, 350, 399, 351],
> > 401=[400, 401], 402=[37], 403=[401, 403], 404=[404, 403], 405=[37, 38],
> > 406=[404, 406], 407=[406, 407], 408=[34, 38, 39], 409=[409, 407],
> 410=[409,
> > 410], 411=[409, 410, 411], 412=[40, 39], 413=[413, 411], 414=[401, 409,
> > 389, 413, 414], 415=[40], 416=[416, 414], 417=[416, 417], 418=[38],
> > 419=[417, 419], 420=[420, 414, 419], 421=[400, 403, 420, 421], 422=[385,
> > 419, 420, 421, 422], 423=[387, 419, 404, 420, 421, 422, 423, 394, 414],
> > 424=[424, 419, 420, 423], 425=[424, 425], 426=[419, 404, 420, 422, 425,
> > 426, 394, 414], 427=[426, 419, 427, 420], 428=[428, 427], 429=[400, 424,
> > 403, 428, 429], 430=[48], 431=[419, 403, 404, 420, 424, 429, 414, 431],
> > 432=[432, 431], 433=[48, 49], 434=[432, 434], 435=[424, 434, 435],
> 436=[49,
> > 51], 437=[437, 435], 438=[428, 437, 438, 431], 439=[48, 49, 51, 52, 40],
> > 440=[438, 440], 441=[440, 441], 442=[52, 53, 21], 443=[438, 441, 443],
> > 444=[438, 440, 443, 444], 445=[438, 444, 445], 446=[445, 446], 447=[438,
> > 446, 447], 448=[448, 438, 440, 445, 446, 447], 449=[448, 449], 450=[53,
> > 54], 451=[449, 451], 452=[337, 451, 419, 452, 404, 420, 438, 394, 414],
> > 453=[49, 54, 55, 40], 454=[452, 454], 455=[454, 455], 456=[55, 57],
> > 457=[449, 438, 455, 457, 441], 458=[457, 458], 459=[458, 459], 460=[49,
> 53,
> > 57, 58], 461=[437, 438, 459, 461], 462=[438, 458, 459, 461, 462],
> 463=[58,
> > 60], 464=[464, 462], 465=[464, 465, 459], 466=[465, 466], 467=[466, 467],
> > 468=[466, 467, 468], 469=[468, 469, 459], 470=[469, 470], 471=[470, 471],
> > 472=[471, 472], 473=[472, 473], 474=[471, 472, 473, 474], 475=[474, 475],
> > 476=[475, 476], 477=[476, 477], 478=[477, 478], 479=[34, 40, 60, 61],
> > 480=[480, 478], 481=[480, 481], 482=[61], 483=[481, 483], 484=[483, 484],
> > 485=[64, 40], 486=[484, 486], 487=[486, 487], 488=[487, 488], 489=[64,
> 66],
> > 490=[488, 490], 491=[490, 491], 492=[66, 67], 493=[491, 493], 494=[493,
> > 494], 495=[64, 67, 68], 496=[496, 464, 465, 494], 497=[496, 497, 438],
> > 498=[497, 498], 499=[68, 70], 500=[465, 497, 498, 500, 438], 501=[500,
> > 501], 502=[70, 71], 503=[420, 484, 501, 437, 438, 486, 503, 457, 490,
> 459,
> > 446], 504=[503, 504], 505=[71, 73], 506=[504, 506], 507=[506, 507],
> > 508=[507, 508], 509=[508, 509], 510=[509, 510], 511=[510, 511], 512=[512,
> > 438, 511], 513=[73], 514=[512, 514], 515=[514, 515], 516=[76], 517=[515,
> > 517], 518=[386, 387, 452, 517, 421, 518, 424, 459, 397, 429, 465, 500,
> 438,
> > 503, 506], 519=[352, 337, 437, 518, 438, 519, 345, 381, 511], 520=[519,
> > 520], 521=[519, 520, 521], 522=[521, 522], 523=[515, 521, 522, 523],
> > 524=[76, 77], 525=[515, 521, 522, 523, 525], 526=[514, 515, 521, 525,
> 526],
> > 527=[77, 79], 528=[528, 526], 529=[528, 529], 530=[79], 531=[529, 531],
> > 532=[531, 532, 520], 533=[532, 533], 534=[533, 534], 535=[512, 517, 534,
> > 535], 536=[81], 537=[535, 537], 538=[537, 538], 539=[81, 82, 71],
> 540=[538,
> > 540], 541=[540, 541], 542=[82, 83], 543=[535, 541, 543], 544=[544, 543],
> > 545=[544, 545], 546=[83], 547=[545, 547], 548=[544, 545, 547, 531, 548,
> > 535, 521, 538, 543], 549=[79], 550=[548, 550], 551=[548, 550, 551],
> > 552=[97], 553=[551, 553], 554=[337, 531, 438, 535, 553, 554], 555=[97],
> > 556=[548, 554, 538, 556], 557=[545, 556, 557], 558=[557, 558], 559=[64,
> 68,
> > 70], 560=[560, 544, 554, 558], 561=[560, 561], 562=[64, 82, 68, 70, 55,
> > 105, 57], 563=[561, 563], 564=[563, 564, 537, 558], 565=[564, 565],
> > 566=[565, 566, 556, 558], 567=[566, 567], 568=[567, 568], 569=[568, 569],
> > 570=[566, 569, 570, 558], 571=[570, 571], 572=[211], 573=[571, 573],
> > 574=[573, 574], 575=[211, 214], 576=[576, 574], 577=[576, 544, 577, 561,
> > 554, 557], 578=[97, 105], 579=[577, 579, 565, 566], 580=[579, 580],
> > 581=[214], 582=[580, 582], 583=[582, 583], 584=[566, 583, 584, 558],
> > 585=[108], 586=[566, 584, 586], 587=[108], 588=[583, 586, 588],
> 589=[211],
> > 590=[586, 588, 590], 591=[590, 591], 592=[211], 593=[593, 586, 591],
> > 594=[593, 594, 586], 595=[594, 595], 596=[595, 596, 586], 597=[596, 597],
> > 598=[111], 599=[596, 597, 599, 586], 600=[597, 599, 600], 601=[597, 599,
> > 600, 601], 602=[601, 602, 586, 588], 603=[211], 604=[583, 599, 602, 586,
> > 604], 605=[604, 605], 606=[586, 605, 606], 607=[113, 97, 108, 111],
> > 608=[608, 596, 599, 586, 606], 609=[608, 609], 610=[113], 611=[609, 611,
> > 586, 588], 612=[611, 612, 583, 599, 586], 613=[608, 612, 613, 599, 600],
> > 614=[609, 596, 613, 614, 599, 586], 615=[614, 615], 616=[608, 615, 599,
> > 616, 600, 586], 617=[615, 616, 617], 618=[611, 613, 617, 618], 619=[618,
> > 619], 620=[234], 621=[611, 613, 619, 621], 622=[608, 609, 611, 621, 622],
> > 623=[234], 624=[624, 608, 612, 613, 614, 615, 622], 625=[624, 625, 616,
> > 621], 626=[234, 237], 627=[625, 627], 628=[624, 625, 627, 628, 615, 619,
> > 622], 629=[237], 630=[628, 630], 631=[630, 631], 632=[97, 108, 79],
> > 633=[624, 625, 630, 631, 633, 622], 634=[633, 634], 635=[599, 631, 634,
> > 635, 621], 636=[97], 637=[635, 637], 638=[619, 637, 638, 622],
> 639=[243]} |
> > org.apache.activemq.store.kahadb.MessageDatabase | ActiveMQ Journal
> > Checkpoint Worker
> > *2022-09-29 17:39:51,557 | TRACE |* not removing data file: 244 as
> > contained ack(s) refer to referenced file: [244, 243] |
> > org.apache.activemq.store.kahadb.MessageDatabase | ActiveMQ Journal
> > Checkpoint Worker
> > *2022-09-29 17:39:51,559 | TRACE |* not removing data file: 245 as
> > contained ack(s) refer to referenced file: [123] |
> > org.apache.activemq.store.kahadb.MessageDatabase | ActiveMQ Journal
> > Checkpoint Worker
> > *2022-09-29 17:39:51,561 | TRACE |* not removing data file: 246 as
> > contained ack(s) refer to referenced file: [244, 246] |
> > org.apache.activemq.store.kahadb.MessageDatabase | ActiveMQ Journal
> > Checkpoint Worker
> > *2022-09-29 17:39:51,563 | TRACE |* not removing data file: 247 as
> > contained ack(s) refer to referenced file: [246, 247] |
> > org.apache.activemq.store.kahadb.MessageDatabase | ActiveMQ Journal
> > Checkpoint Worker
> > *2022-09-29 17:39:51,564 | WARN | *Stealing link for clientId
> > TR_VP-018:30246 From Connection Transport Connection to: tcp://
> > 172.24.96.125:42412 | org.apache.activemq.broker.region.RegionBroker |
> > ActiveMQ Transport: tcp:///172.24.96.125:42414@1883
> > *2022-09-29 17:39:51,565 | TRACE |* not removing data file: 248 as
> > contained ack(s) refer to referenced file: [123] |
> > org.apache.activemq.store.kahadb.MessageDatabase | ActiveMQ Journal
> > Checkpoint Worker
> > *2022-09-29 17:39:51,565 | TRACE |* not removing data file: 249 as
> > contained ack(s) refer to referenced file: [249, 247] |
> > org.apache.activemq.store.kahadb.MessageDatabase | ActiveMQ Journal
> > Checkpoint Worker
> > *2022-09-29 17:39:51,565 | TRACE |* not removing data file: 250 as
> > contained ack(s) refer to referenced file: [163, 211, 249, 250, 234,
> 237] |
> > org.apache.activemq.store.kahadb.MessageDatabase | ActiveMQ Journal
> > Checkpoint Worker
> > *2022-09-29 17:39:51,565 | TRACE |* not removing data file: 251 as
> > contained ack(s) refer to referenced file: [250, 243, 251, 247] |
> > org.apache.activemq.store.kahadb.MessageDatabase | ActiveMQ Journal
> > Checkpoint Worker
> > *2022-09-29 17:39:51,565 | TRACE |* not removing data file: 248 as
> > contained ack(s) refer to referenced file: [123] |
> > org.apache.activemq.store.kahadb.MessageDatabase | ActiveMQ Journal
> > Checkpoint Worker
> > *2022-09-29 17:39:51,565 | TRACE |* not removing data file: 249 as
> > contained ack(s) refer to referenced file: [249, 247] |
> > org.apache.activemq.store.kahadb.MessageDatabase | ActiveMQ Journal
> > Checkpoint Worker
> > *2022-09-29 17:39:51,565 | TRACE |* not removing data file: 250 as
> > contained ack(s) refer to referenced file: [163, 211, 249, 250, 234,
> 237] |
> > org.apache.activemq.store.kahadb.MessageDatabase | ActiveMQ Journal
> > Checkpoint Worker
> > *2022-09-29 17:39:51,565 | TRACE |* not removing data file: 251 as
> > contained ack(s) refer to referenced file: [250, 243, 251, 247] |
> > org.apache.activemq.store.kahadb.MessageDatabase | ActiveMQ Journal
> > Checkpoint Worker
> > *2022-09-29 17:39:51,565 | TRACE |* not removing data file: 252 as
> > contained ack(s) refer to referenced file: [140, 108, 97] |
> > org.apache.activemq.store.kahadb.MessageDatabase | ActiveMQ Journal
> > Checkpoint Worker
> > *2022-09-29 17:39:51,565 | TRACE | *not removing data file: 253 as
> > contained ack(s) refer to referenced file: [249, 253, 251] |
> > org.apache.activemq.store.kahadb.MessageDatabase | ActiveMQ Journal
> > Checkpoint Worker
> > *2022-09-29 17:39:51,565 | TRACE |* not removing data file: 254 as
> > contained ack(s) refer to referenced file: [253, 254] |
> > org.apache.activemq.store.kahadb.MessageDatabase | ActiveMQ Journal
> > Checkpoint Worker
> > *2022-09-29 17:39:51,565 | TRACE |* not removing data file: 255 as
> > contained ack(s) refer to referenced file: [140, 141] |
> > org.apache.activemq.store.kahadb.MessageDatabase | ActiveMQ Journal
> > Checkpoint Worker not removing data file: 252 as contained ack(s) refer
> to
> > referenced file: [140, 108, 97] |
> > org.apache.activemq.store.kahadb.MessageDatabase | ActiveMQ Journal
> > Checkpoint Worker
> > *2022-09-29 17:39:51,565 | TRACE |* not removing data file: 253 as
> > contained ack(s) refer to referenced file: [249, 253, 251] |
> > org.apache.activemq.store.kahadb.MessageDatabase | ActiveMQ Journal
> > Checkpoint Worker
> > *2022-09-29 17:39:51,565 | TRACE | *not removing data file: 254 as
> > contained ack(s) refer to referenced file: [253, 254] |
> > org.apache.activemq.store.kahadb.MessageDatabase | ActiveMQ Journal
> > Checkpoint Worker
> > *2022-09-29 17:39:51,565 | TRACE |* not removing data file: 255 as
> > contained ack(s) refer to referenced file: [140, 141] |
> > org.apache.activemq.store.kahadb.MessageDatabase | ActiveMQ Journal
> > Checkpoint Worker
> >
> > We also use durable topics as the devices that subscribe to these topics
> > and send MQTT messages to others work in a very bad network environment,
> > this is the way we found to avoid data loss while offline.
> > I think the problem is related to durable topics with offline
> > subscriptions, but how can I confirm this assumption, eg which topics?
> What
> > signatures?
> >
> > I much appreciate any thoughts!
> > Thanks in advance
> >
>
