[jira] [Work logged] (AMQ-9494) Upgrade to maven-source-plugin 3.3.1
[ https://issues.apache.org/jira/browse/AMQ-9494?focusedWorklogId=920864=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-920864 ] ASF GitHub Bot logged work on AMQ-9494: --- Author: ASF GitHub Bot Created on: 25/May/24 05:45 Start Date: 25/May/24 05:45 Worklog Time Spent: 10m Work Description: jbonofre merged PR #1224: URL: https://github.com/apache/activemq/pull/1224 Issue Time Tracking --- Worklog Id: (was: 920864) Time Spent: 20m (was: 10m) > Upgrade to maven-source-plugin 3.3.1 > > > Key: AMQ-9494 > URL: https://issues.apache.org/jira/browse/AMQ-9494 > Project: ActiveMQ Classic > Issue Type: Dependency upgrade >Reporter: Jean-Baptiste Onofré >Assignee: Jean-Baptiste Onofré >Priority: Major > Fix For: 6.2.0 > > Time Spent: 20m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (AMQ-9494) Upgrade to maven-source-plugin 3.3.1
[ https://issues.apache.org/jira/browse/AMQ-9494?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17849428#comment-17849428 ] ASF subversion and git services commented on AMQ-9494: -- Commit d1ab8ef93d8807faf4fb5c9d28fb0a5a2c5b015d in activemq's branch refs/heads/main from JB Onofré [ https://gitbox.apache.org/repos/asf?p=activemq.git;h=d1ab8ef93 ] Merge pull request #1224 from jbonofre/AMQ-9494 AMQ-9494: Upgrade to maven-source-plugin 3.3.1 > Upgrade to maven-source-plugin 3.3.1 > > > Key: AMQ-9494 > URL: https://issues.apache.org/jira/browse/AMQ-9494 > Project: ActiveMQ Classic > Issue Type: Dependency upgrade >Reporter: Jean-Baptiste Onofré >Assignee: Jean-Baptiste Onofré >Priority: Major > Fix For: 6.2.0 > > Time Spent: 20m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (AMQ-9494) Upgrade to maven-source-plugin 3.3.1
[ https://issues.apache.org/jira/browse/AMQ-9494?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17849427#comment-17849427 ] ASF subversion and git services commented on AMQ-9494: -- Commit 7bc8b67fe03a4695c6dd4ed0e333c418b2c99a74 in activemq's branch refs/heads/main from JB Onofré [ https://gitbox.apache.org/repos/asf?p=activemq.git;h=7bc8b67fe ] AMQ-9494: Upgrade to maven-source-plugin 3.3.1 > Upgrade to maven-source-plugin 3.3.1 > > > Key: AMQ-9494 > URL: https://issues.apache.org/jira/browse/AMQ-9494 > Project: ActiveMQ Classic > Issue Type: Dependency upgrade >Reporter: Jean-Baptiste Onofré >Assignee: Jean-Baptiste Onofré >Priority: Major > Fix For: 6.2.0 > > Time Spent: 20m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (AMQ-9494) Upgrade to maven-source-plugin 3.3.1
[ https://issues.apache.org/jira/browse/AMQ-9494?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jean-Baptiste Onofré resolved AMQ-9494. --- Resolution: Fixed > Upgrade to maven-source-plugin 3.3.1 > > > Key: AMQ-9494 > URL: https://issues.apache.org/jira/browse/AMQ-9494 > Project: ActiveMQ Classic > Issue Type: Dependency upgrade >Reporter: Jean-Baptiste Onofré >Assignee: Jean-Baptiste Onofré >Priority: Major > Fix For: 6.2.0 > > Time Spent: 20m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Assigned] (AMQ-9505) ActiveMQ classic with postgres data source gives errors on start
[ https://issues.apache.org/jira/browse/AMQ-9505?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jean-Baptiste Onofré reassigned AMQ-9505: - Assignee: Jean-Baptiste Onofré > ActiveMQ classic with postgres data source gives errors on start > > > Key: AMQ-9505 > URL: https://issues.apache.org/jira/browse/AMQ-9505 > Project: ActiveMQ Classic > Issue Type: Bug >Affects Versions: 6.1.1 > Environment: Fresh 6.1.1 installation on Ubuntu + Java 17 >Reporter: Susinda Perera >Assignee: Jean-Baptiste Onofré >Priority: Major > > I get this error when I start ActiveMQ > {noformat} > 2024-05-21 23:59:32,959 | INFO | Using Persistence Adapter: > JDBCPersistenceAdapter(HikariDataSource (null)) | > org.apache.activemq.broker.BrokerService | main > 2024-05-21 23:59:32,968 | INFO | Starting Persistence Adapter: > JDBCPersistenceAdapter(HikariDataSource (null)) | > org.apache.activemq.broker.BrokerService | main > 2024-05-21 23:59:32,969 | INFO | HikariPool-1 - Starting... | > com.zaxxer.hikari.HikariDataSource | main > 2024-05-21 23:59:35,211 | INFO | HikariPool-1 - Added connection > org.postgresql.jdbc.PgConnection@70101687 | com.zaxxer.hikari.pool.HikariPool > | main > 2024-05-21 23:59:35,214 | INFO | HikariPool-1 - Start completed. | > com.zaxxer.hikari.HikariDataSource | main > 2024-05-21 23:59:35,433 | INFO | Database adapter driver override recognized > for : [postgresql_jdbc_driver] - adapter: class > org.apache.activemq.store.jdbc.adapter.PostgresqlJDBCAdapter | > org.apache.activemq.store.jdbc.JDBCPersistenceAdapter | main > 2024-05-21 23:59:38,423 | WARN | Could not create JDBC tables; they could > already exist. Failure was: ALTER TABLE ACTIVEMQ_ACKS DROP CONSTRAINT > "activemq_acks_pkey" Message: ERROR: constraint "activemq_acks_pkey" of > relation "activemq_acks" does not exist SQLState: 42704 Vendor code: 0 | > org.apache.activemq.store.jdbc.adapter.DefaultJDBCAdapter | main > 2024-05-21 23:59:38,423 | WARN | Failure details: ERROR: constraint > "activemq_acks_pkey" of relation "activemq_acks" does not exist | > org.apache.activemq.store.jdbc.JDBCPersistenceAdapter | main > org.postgresql.util.PSQLException: ERROR: constraint "activemq_acks_pkey" of > relation "activemq_acks" does not exist >at > org.postgresql.core.v3.QueryExecutorImpl.receiveErrorResponse(QueryExecutorImpl.java:2725) >at > org.postgresql.core.v3.QueryExecutorImpl.processResults(QueryExecutorImpl.java:2412) >at > org.postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:371) >at org.postgresql.jdbc.PgStatement.executeInternal(PgStatement.java:502) >at org.postgresql.jdbc.PgStatement.execute(PgStatement.java:419) >at org.postgresql.jdbc.PgStatement.executeWithFlags(PgStatement.java:341) >at org.postgresql.jdbc.PgStatement.executeCachedSql(PgStatement.java:326) >at org.postgresql.jdbc.PgStatement.executeWithFlags(PgStatement.java:302) >at org.postgresql.jdbc.PgStatement.execute(PgStatement.java:297) >at com.zaxxer.hikari.pool.ProxyStatement.execute(ProxyStatement.java:94) >at > com.zaxxer.hikari.pool.HikariProxyStatement.execute(HikariProxyStatement.java) >at > org.apache.activemq.store.jdbc.adapter.DefaultJDBCAdapter.executeStatement(DefaultJDBCAdapter.java:112) >at > org.apache.activemq.store.jdbc.adapter.DefaultJDBCAdapter.doCreateTables(DefaultJDBCAdapter.java:90) >at > org.apache.activemq.store.jdbc.JDBCPersistenceAdapter.init(JDBCPersistenceAdapter.java:318) >at > org.apache.activemq.broker.LockableServiceSupport.preStart(LockableServiceSupport.java:99) >at org.apache.activemq.util.ServiceSupport.start(ServiceSupport.java:54) >at > org.apache.activemq.broker.BrokerService.doStartPersistenceAdapter(BrokerService.java:681) >at > org.apache.activemq.broker.BrokerService.startPersistenceAdapter(BrokerService.java:663) >at org.apache.activemq.broker.BrokerService.start(BrokerService.java:627) >at > org.apache.activemq.xbean.XBeanBrokerService.afterPropertiesSet(XBeanBrokerService.java:73) >at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native > Method) >at > java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77) >at > java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) >at java.base/java.lang.reflect.Method.invoke(Method.java:568) >at > org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.invokeCustomInitMethod(AbstractAutowireCapableBeanFactory.java:1890) >at > org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.invokeInitMethods(AbstractAutowireCapableBeanFactory.java:1843) >at >
[jira] [Created] (ARTEMIS-4780) Artemis web console does not ever set 'artemisJmxDomain' in localStorage
Josh Byster created ARTEMIS-4780: Summary: Artemis web console does not ever set 'artemisJmxDomain' in localStorage Key: ARTEMIS-4780 URL: https://issues.apache.org/jira/browse/ARTEMIS-4780 Project: ActiveMQ Artemis Issue Type: Bug Affects Versions: 2.32.0 Reporter: Josh Byster We relocate our artemis packages as part of our shading process. I did set up: {code:java} aActiveMQServer.getConfiguration().setJMXDomain("com.abc.shaded.org.apache.activemq.artemis"); {code} However the web console does not show up. I noticed in the code it's because we try to read in localStorage['artemisJmxDomain'] but never actually set it as far as I can tell. Thus we always default to org.apache.activemq.artemis. Everything works properly when setting localStorage['artemisJmxDomain'] = "com.abc.shaded.org.apache.activemq.artemis" -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (ARTEMIS-4306) Add authn/z metrics
[ https://issues.apache.org/jira/browse/ARTEMIS-4306?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17849328#comment-17849328 ] Justin Bertram commented on ARTEMIS-4306: - I think support for {{CaffeineStatsCounter}} would be pretty straight-forward to implement. I've got some ideas. I'll send a PR soon. > Add authn/z metrics > --- > > Key: ARTEMIS-4306 > URL: https://issues.apache.org/jira/browse/ARTEMIS-4306 > Project: ActiveMQ Artemis > Issue Type: Improvement >Reporter: Justin Bertram >Priority: Major > > It would be useful to have metrics for authn/z successes and failures as well > as for metrics related to the corresponding caches. > See this discussion on the users mailing list for more details: > https://lists.apache.org/thread/g6ygyo4kb3xhygq8hpw7vsl3l2g5qt92 -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Comment Edited] (ARTEMIS-4306) Add authn/z metrics
[ https://issues.apache.org/jira/browse/ARTEMIS-4306?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17849099#comment-17849099 ] Mike Artz edited comment on ARTEMIS-4306 at 5/24/24 2:45 PM: - [~jbertram] - I made this [PR for micrometer to allow prefixing in the CacheMeterBinder|https://github.com/micrometer-metrics/micrometer/pull/4048], so that we could for example add the *"artemis.authentication."* prefix but the PR _kind of_ got stuck, and the PR got closed. Coming back to this now, _[if we use the CacheMeterBinder|https://github.com/micrometer-metrics/micrometer/blob/37883fa6fb4a6d3f83d01f6b53101cc9f52b3f78/micrometer-core/src/main/java/io/micrometer/core/instrument/binder/cache/CacheMeterBinder.java],_ would it be ok to not have *"artemis.authentication."* prefixes, and adding *authentication* and *authorization* as Tags? i.e. {{{}Tag("cacheName", "authentication"){}}}and {{{}Tag("{}}}{{{}cacheName{}}}{{{}", "authorization"){}}} It seems this might be more of a standard too for micrometer users possibly. However, now that we are using the [CaffeineCache|https://github.com/ben-manes/caffeine/blob/b4cedbc411130b8e78c51effdd527756bdf1ff55/caffeine/src/main/java/com/github/benmanes/caffeine/cache/Cache.java], I see there are two concrete classes - [CaffeineCacheMetrics|https://github.com/micrometer-metrics/micrometer/blob/37883fa6fb4a6d3f83d01f6b53101cc9f52b3f78/micrometer-core/src/main/java/io/micrometer/core/instrument/binder/cache/CaffeineCacheMetrics.java] and [CaffeineStatsCounter|https://github.com/micrometer-metrics/micrometer/blob/37883fa6fb4a6d3f83d01f6b53101cc9f52b3f78/micrometer-core/src/main/java/io/micrometer/core/instrument/binder/cache/CaffeineStatsCounter.java]. Both of these look ok at first (more or less), but I am stuck at this decision. [CaffeineStatsCounter|https://github.com/micrometer-metrics/micrometer/blob/37883fa6fb4a6d3f83d01f6b53101cc9f52b3f78/micrometer-core/src/main/java/io/micrometer/core/instrument/binder/cache/CaffeineStatsCounter.java] offers more detailed statistics than [CaffeineCacheMetrics|https://github.com/micrometer-metrics/micrometer/blob/37883fa6fb4a6d3f83d01f6b53101cc9f52b3f78/micrometer-core/src/main/java/io/micrometer/core/instrument/binder/cache/CaffeineCacheMetrics.java] and it allows name prefixes. However, [CaffeineStatsCounter|https://github.com/micrometer-metrics/micrometer/blob/37883fa6fb4a6d3f83d01f6b53101cc9f52b3f78/micrometer-core/src/main/java/io/micrometer/core/instrument/binder/cache/CaffeineStatsCounter.java] needs to pass the meterRegistry to the builder Caffeine object {_}before the cache is built with the build() call{_}. There is no direct support to add a [CaffeineStatsCounter|https://github.com/micrometer-metrics/micrometer/blob/37883fa6fb4a6d3f83d01f6b53101cc9f52b3f78/micrometer-core/src/main/java/io/micrometer/core/instrument/binder/cache/CaffeineStatsCounter.java] to a Caffeine Cache after the cache is built. So if you use the [CaffeineStatsCounter|https://github.com/micrometer-metrics/micrometer/blob/37883fa6fb4a6d3f83d01f6b53101cc9f52b3f78/micrometer-core/src/main/java/io/micrometer/core/instrument/binder/cache/CaffeineStatsCounter.java] you might have to do something like call the [MetricsConfiguration|https://github.com/apache/activemq-artemis/blob/c47713454caeece82df29a0a7fd4a2a39000576b/artemis-server/src/main/java/org/apache/activemq/artemis/core/config/MetricsConfiguration.java] from the *[SecurityStoreImpl|https://github.com/apache/activemq-artemis/blob/main/artemis-server/src/main/java/org/apache/activemq/artemis/core/security/impl/SecurityStoreImpl.java#L105-L108]* like {code:java} MeterRegistry registry = metricsConfiguration.getPlugin().getRegistry(); authenticationCache = Caffeine.newBuilder() .maximumSize(authenticationCacheSize) .expireAfterWrite(invalidationInterval, TimeUnit.MILLISECONDS) .recordStats(() -> new CaffeineStatsCounter(registry, "authentication")) .build();{code} And that doesnt make any sense because *MetricsConfiguration* happens after the *SecurityStoreImpl.* So this doesnt seem like the best option. *Other Options* * Use [CaffeineStatsCounter|https://github.com/micrometer-metrics/micrometer/blob/37883fa6fb4a6d3f83d01f6b53101cc9f52b3f78/micrometer-core/src/main/java/io/micrometer/core/instrument/binder/cache/CaffeineStatsCounter.java] could initialize the authn cache (and authz cache) from the MetricsManager class _(Couples the SecurityStoreImpl to the MetricsManager but maybe 2nd best option)_ * Use [CaffeineCacheMetrics|https://github.com/micrometer-metrics/micrometer/blob/37883fa6fb4a6d3f83d01f6b53101cc9f52b3f78/micrometer-core/src/main/java/io/micrometer/core/instrument/binder/cache/CaffeineCacheMetrics.java] _(Seems like best option)_ * Create another concrete class (or maybe even extend the concrete class
[jira] [Commented] (AMQ-9482) Broker crashes after runaway threads spawn
[ https://issues.apache.org/jira/browse/AMQ-9482?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17849318#comment-17849318 ] Matt Pavlovich commented on AMQ-9482: - There are a ton of environment and use case specific details to get a tuning exercise corrected-- very difficult to resolve over a JIRA. > Broker crashes after runaway threads spawn > -- > > Key: AMQ-9482 > URL: https://issues.apache.org/jira/browse/AMQ-9482 > Project: ActiveMQ Classic > Issue Type: Bug > Components: Broker >Affects Versions: 5.17.6, 6.0.1 > Environment: Bitnami created AMI in AWS >Reporter: Tom Tichy >Priority: Major > Attachments: activemq.tdump, brokerInfo-after-crash-redacted.json, > thread_dump.txt > > > Running on Bitnami created AMI in AWS. The broker has about 7000 devices > connected via MQTT. Each devices has its own topic name. > Broker stays up for about 4-5 days before being hobbled and unable to create > any new tasks/accept any new connections. > (There is identical setup for staging environment with about 100 devices > connected. It runs without any issues.) > I have troubleshot the cause to be the systemd task limit. The current > `TasksMax` is 18100. When running normally, the number of tasks is about 300. > Then (every 4-5 days) there is a quick spike to the max 18100 tasks and it > stays there never coming back down. The result is that the broker just sits > there, does nothing useful and keeps logging the following message > > {code:java} > [659914.788s][warning][os,thread] Failed to start thread "Unknown thread" - > pthread_create failed (EAGAIN) for attributes: stacksize: 1024k, g > uardsize: 0k, detached. > [659914.788s][warning][os,thread] Failed to start the native thread for > java.lang.Thread "ActiveMQ BrokerService[localhost] Task-281805" > ERROR | Scheduled task error > java.lang.OutOfMemoryError: unable to create native thread: possibly out of > memory or process/resource limits reached > at java.lang.Thread.start0(Native Method) ~[?:?] > at java.lang.Thread.start(Thread.java:809) ~[?:?] > at > java.util.concurrent.ThreadPoolExecutor.addWorker(ThreadPoolExecutor.java:945) > ~[?:?] > at > java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1364) > ~[?:?] > at > org.apache.activemq.thread.TaskRunnerFactory.execute(TaskRunnerFactory.java:173) > ~[activemq-client-6.0.1.jar:6.0.1] > at > org.apache.activemq.thread.TaskRunnerFactory.execute(TaskRunnerFactory.java:165) > ~[activemq-client-6.0.1.jar:6.0.1] > at org.apache.activemq.broker.region.Topic$7.run(Topic.java:820) > ~[activemq-broker-6.0.1.jar:6.0.1] > at > org.apache.activemq.thread.SchedulerTimerTask.run(SchedulerTimerTask.java:39) > ~[activemq-client-6.0.1.jar:6.0.1] > at java.util.TimerThread.mainLoop(Timer.java:566) ~[?:?] > at java.util.TimerThread.run(Timer.java:516) ~[?:?] > Exception in thread "ActiveMQ Broker[localhost] Scheduler" > java.lang.OutOfMemoryError: unable to create native thread: possibly out of > memory or process/resource limits reached > at java.base/java.lang.Thread.start0(Native Method) > at java.base/java.lang.Thread.start(Thread.java:809) > at > java.base/java.util.concurrent.ThreadPoolExecutor.addWorker(ThreadPoolExecutor.java:945) > at > java.base/java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1364) > at > org.apache.activemq.thread.TaskRunnerFactory.execute(TaskRunnerFactory.java:173) > at > org.apache.activemq.thread.TaskRunnerFactory.execute(TaskRunnerFactory.java:165) > at org.apache.activemq.broker.region.Topic$7.run(Topic.java:820) > at > org.apache.activemq.thread.SchedulerTimerTask.run(SchedulerTimerTask.java:39) > at java.base/java.util.TimerThread.mainLoop(Timer.java:566) > at java.base/java.util.TimerThread.run(Timer.java:516) > {code} > > The start command is > {code:java} > /opt/bitnami/java/bin/java -Xms2G -Xmx4G > -Djava.util.logging.config.file=logging.properties > -Djava.security.auth.login.config=/opt/bitnami/activemq/conf/login.config > -Dorg.apache.activemq.UseDedicatedTaskRunner=false > -Dcom.sun.management.jmxremote -Djava.awt.headless=true > -Djava.io.tmpdir=/opt/bitnami/activemq/tmp --add-reads=java.xml=java.logging > --add-opens java.base/java.security=ALL-UNNAMED --add-opens > java.base/java.net=ALL-UNNAMED --add-opens java.base/java.lang=ALL-UNNAMED > --add-opens java.base/java.util=ALL-UNNAMED --add-opens > java.naming/javax.naming.spi=ALL-UNNAMED --add-opens > java.rmi/sun.rmi.transport.tcp=ALL-UNNAMED --add-opens > java.base/java.util.concurrent=ALL-UNNAMED --add-opens > java.base/java.util.concurrent.atomic=ALL-UNNAMED >
[jira] [Commented] (AMQ-9482) Broker crashes after runaway threads spawn
[ https://issues.apache.org/jira/browse/AMQ-9482?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17849248#comment-17849248 ] Tom Tichy commented on AMQ-9482: Hi, I did add the `soTimeout` and `soWriteTimeout`, but alas no joy. I have a better thread dump with about 11.5k identical threads that are in the WAITING state Fastthread.io analysis says {quote} h1. 11440 threads with same stack trace !https://fastthread.io/assets/globally-shared/images/icon-error.svg! 11440 threads are WAITING on *_park()_* method in *_jdk.internal.misc.Unsafe_* file and they all have same stack trace. If multiple threads exhibit same stack trace, you might want to examine their stack trace. (Note: If your application is unresponsive or poorly responding, it might be caused because these threads). {panel} h2. {color:#cc3300}ActiveMQ BrokerService[localhost] Task-68826{color} PRIORITY : 5 THREAD ID : 0X7F1E3010FB90 NATIVE ID : 0XE3ABE NATIVE ID (DECIMAL) : 932542 STATE : TIMED_WAITING stackTrace: java.lang.Thread.State: TIMED_WAITING (parking) at jdk.internal.misc.Unsafe.park({color:#80}java.base@{*}17.0.9/Native Method{*}{color}{*}){*} - parking to wait for *<0x000700232d10>* (a java.util.concurrent.SynchronousQueue$TransferStack) at java.util.concurrent.locks.LockSupport.parkNanos({color:#80}java.base@{*}17.0.9/LockSupport.java:252{*}{color}{*}){*} at java.util.concurrent.SynchronousQueue$TransferStack.transfer({color:#80}java.base@{*}17.0.9/SynchronousQueue.java:401{*}{color}{*}){*} at java.util.concurrent.SynchronousQueue.poll({color:#80}java.base@{*}17.0.9/SynchronousQueue.java:903{*}{color}{*}){*} at java.util.concurrent.ThreadPoolExecutor.getTask({color:#80}java.base@{*}17.0.9/ThreadPoolExecutor.java:1061{*}{color}{*}){*} at java.util.concurrent.ThreadPoolExecutor.runWorker({color:#80}java.base@{*}17.0.9/ThreadPoolExecutor.java:1122{*}{color}{*}){*} at java.util.concurrent.ThreadPoolExecutor$Worker.run({color:#80}java.base@{*}17.0.9/ThreadPoolExecutor.java:635{*}{color}{*}){*} at java.lang.Thread.run({color:#80}java.base@{*}17.0.9/Thread.java:840{*}{color}{*}){*} Locked ownable synchronizers: - None{panel} {quote} Here is another piece of information that may be relevant. The devices that connect to our brokers do something silly and open a new MQTT connection everytime they want to send something. The MQTT spec allows this and ActiveMQ dutifully logs {code:java} WARN | Stealing link for clientId {code} [^thread_dump.txt] > Broker crashes after runaway threads spawn > -- > > Key: AMQ-9482 > URL: https://issues.apache.org/jira/browse/AMQ-9482 > Project: ActiveMQ Classic > Issue Type: Bug > Components: Broker >Affects Versions: 5.17.6, 6.0.1 > Environment: Bitnami created AMI in AWS >Reporter: Tom Tichy >Priority: Major > Attachments: activemq.tdump, brokerInfo-after-crash-redacted.json, > thread_dump.txt > > > Running on Bitnami created AMI in AWS. The broker has about 7000 devices > connected via MQTT. Each devices has its own topic name. > Broker stays up for about 4-5 days before being hobbled and unable to create > any new tasks/accept any new connections. > (There is identical setup for staging environment with about 100 devices > connected. It runs without any issues.) > I have troubleshot the cause to be the systemd task limit. The current > `TasksMax` is 18100. When running normally, the number of tasks is about 300. > Then (every 4-5 days) there is a quick spike to the max 18100 tasks and it > stays there never coming back down. The result is that the broker just sits > there, does nothing useful and keeps logging the following message > > {code:java} > [659914.788s][warning][os,thread] Failed to start thread "Unknown thread" - > pthread_create failed (EAGAIN) for attributes: stacksize: 1024k, g > uardsize: 0k, detached. > [659914.788s][warning][os,thread] Failed to start the native thread for > java.lang.Thread "ActiveMQ BrokerService[localhost] Task-281805" > ERROR | Scheduled task error > java.lang.OutOfMemoryError: unable to create native thread: possibly out of > memory or process/resource limits reached > at java.lang.Thread.start0(Native Method) ~[?:?] > at java.lang.Thread.start(Thread.java:809) ~[?:?] > at > java.util.concurrent.ThreadPoolExecutor.addWorker(ThreadPoolExecutor.java:945) > ~[?:?] > at > java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1364) > ~[?:?] > at > org.apache.activemq.thread.TaskRunnerFactory.execute(TaskRunnerFactory.java:173) > ~[activemq-client-6.0.1.jar:6.0.1] > at > org.apache.activemq.thread.TaskRunnerFactory.execute(TaskRunnerFactory.java:165) > ~[activemq-client-6.0.1.jar:6.0.1] > at
[jira] [Updated] (AMQ-9482) Broker crashes after runaway threads spawn
[ https://issues.apache.org/jira/browse/AMQ-9482?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tom Tichy updated AMQ-9482: --- Attachment: thread_dump.txt > Broker crashes after runaway threads spawn > -- > > Key: AMQ-9482 > URL: https://issues.apache.org/jira/browse/AMQ-9482 > Project: ActiveMQ Classic > Issue Type: Bug > Components: Broker >Affects Versions: 5.17.6, 6.0.1 > Environment: Bitnami created AMI in AWS >Reporter: Tom Tichy >Priority: Major > Attachments: activemq.tdump, brokerInfo-after-crash-redacted.json, > thread_dump.txt > > > Running on Bitnami created AMI in AWS. The broker has about 7000 devices > connected via MQTT. Each devices has its own topic name. > Broker stays up for about 4-5 days before being hobbled and unable to create > any new tasks/accept any new connections. > (There is identical setup for staging environment with about 100 devices > connected. It runs without any issues.) > I have troubleshot the cause to be the systemd task limit. The current > `TasksMax` is 18100. When running normally, the number of tasks is about 300. > Then (every 4-5 days) there is a quick spike to the max 18100 tasks and it > stays there never coming back down. The result is that the broker just sits > there, does nothing useful and keeps logging the following message > > {code:java} > [659914.788s][warning][os,thread] Failed to start thread "Unknown thread" - > pthread_create failed (EAGAIN) for attributes: stacksize: 1024k, g > uardsize: 0k, detached. > [659914.788s][warning][os,thread] Failed to start the native thread for > java.lang.Thread "ActiveMQ BrokerService[localhost] Task-281805" > ERROR | Scheduled task error > java.lang.OutOfMemoryError: unable to create native thread: possibly out of > memory or process/resource limits reached > at java.lang.Thread.start0(Native Method) ~[?:?] > at java.lang.Thread.start(Thread.java:809) ~[?:?] > at > java.util.concurrent.ThreadPoolExecutor.addWorker(ThreadPoolExecutor.java:945) > ~[?:?] > at > java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1364) > ~[?:?] > at > org.apache.activemq.thread.TaskRunnerFactory.execute(TaskRunnerFactory.java:173) > ~[activemq-client-6.0.1.jar:6.0.1] > at > org.apache.activemq.thread.TaskRunnerFactory.execute(TaskRunnerFactory.java:165) > ~[activemq-client-6.0.1.jar:6.0.1] > at org.apache.activemq.broker.region.Topic$7.run(Topic.java:820) > ~[activemq-broker-6.0.1.jar:6.0.1] > at > org.apache.activemq.thread.SchedulerTimerTask.run(SchedulerTimerTask.java:39) > ~[activemq-client-6.0.1.jar:6.0.1] > at java.util.TimerThread.mainLoop(Timer.java:566) ~[?:?] > at java.util.TimerThread.run(Timer.java:516) ~[?:?] > Exception in thread "ActiveMQ Broker[localhost] Scheduler" > java.lang.OutOfMemoryError: unable to create native thread: possibly out of > memory or process/resource limits reached > at java.base/java.lang.Thread.start0(Native Method) > at java.base/java.lang.Thread.start(Thread.java:809) > at > java.base/java.util.concurrent.ThreadPoolExecutor.addWorker(ThreadPoolExecutor.java:945) > at > java.base/java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1364) > at > org.apache.activemq.thread.TaskRunnerFactory.execute(TaskRunnerFactory.java:173) > at > org.apache.activemq.thread.TaskRunnerFactory.execute(TaskRunnerFactory.java:165) > at org.apache.activemq.broker.region.Topic$7.run(Topic.java:820) > at > org.apache.activemq.thread.SchedulerTimerTask.run(SchedulerTimerTask.java:39) > at java.base/java.util.TimerThread.mainLoop(Timer.java:566) > at java.base/java.util.TimerThread.run(Timer.java:516) > {code} > > The start command is > {code:java} > /opt/bitnami/java/bin/java -Xms2G -Xmx4G > -Djava.util.logging.config.file=logging.properties > -Djava.security.auth.login.config=/opt/bitnami/activemq/conf/login.config > -Dorg.apache.activemq.UseDedicatedTaskRunner=false > -Dcom.sun.management.jmxremote -Djava.awt.headless=true > -Djava.io.tmpdir=/opt/bitnami/activemq/tmp --add-reads=java.xml=java.logging > --add-opens java.base/java.security=ALL-UNNAMED --add-opens > java.base/java.net=ALL-UNNAMED --add-opens java.base/java.lang=ALL-UNNAMED > --add-opens java.base/java.util=ALL-UNNAMED --add-opens > java.naming/javax.naming.spi=ALL-UNNAMED --add-opens > java.rmi/sun.rmi.transport.tcp=ALL-UNNAMED --add-opens > java.base/java.util.concurrent=ALL-UNNAMED --add-opens > java.base/java.util.concurrent.atomic=ALL-UNNAMED > --add-exports=java.base/sun.net.www.protocol.http=ALL-UNNAMED > --add-exports=java.base/sun.net.www.protocol.https=ALL-UNNAMED >