Hi All, I am able to fix this issue. Posting my solution here, just in case anyone else faces a similar issue. I converted deployment to a statefulset (to get predictable names). Then used headless services for each pod. This enabled both master and slave pods to lookup each other just by pod name. In case anyone is looking for more details, feel free to connect.
Regards, Prateek Jain -------------------------------------------------------------- EXPECTATION : Causes all troubles...... -------------------------------------------------------------- On Wed, Mar 15, 2023 at 3:54 PM prateekjai...@gmail.com < prateekjai...@gmail.com> wrote: > > Hi Justin, > > Thanks for replying, I think artemis is failing to lookup POD by its name > because we cannot just get POD IP just by its name. Given below is the > error which is getting repeated in logs: > > 2023-03-15 10:04:06,611 INFO [org.apache.activemq.artemis] AMQ241002: > Artemis Jolokia REST API available at > http://jms-01-deployment-6d7dcb8ccd-j7pxl:8161/console/jolokia > 2023-03-15 10:04:06,611 INFO [org.apache.activemq.artemis] AMQ241004: > Artemis Console available at > http://jms-01-deployment-6d7dcb8ccd-j7pxl:8161/console > 2023-03-15 10:04:10,868 ERROR [org.apache.activemq.artemis.core.client] > AMQ214016: Failed to create netty connection > java.net.UnknownHostException: jms-01-deployment-6d7dcb8ccd-2t9g2 > at java.net.InetAddress$CachedAddresses.get(InetAddress.java:797) > ~[?:?] > at java.net.InetAddress.getAllByName0(InetAddress.java:1524) ~[?:?] > at java.net.InetAddress.getAllByName(InetAddress.java:1382) ~[?:?] > at java.net.InetAddress.getAllByName(InetAddress.java:1306) ~[?:?] > at java.net.InetAddress.getByName(InetAddress.java:1256) ~[?:?] > at io.netty.util.internal.SocketUtils$8.run(SocketUtils.java:156) > ~[netty-common-4.1.86.Final.jar:4.1.86.Final] > at io.netty.util.internal.SocketUtils$8.run(SocketUtils.java:153) > ~[netty-common-4.1.86.Final.jar:4.1.86.Final] > at java.security.AccessController.doPrivileged(Native Method) > ~[?:?] > at > io.netty.util.internal.SocketUtils.addressByName(SocketUtils.java:153) > ~[netty-common-4.1.86.Final.jar:4.1.86.Final] > at > io.netty.resolver.DefaultNameResolver.doResolve(DefaultNameResolver.java:41) > ~[netty-resolver-4.1.86.Final.jar:4.1.86.Final] > at > io.netty.resolver.SimpleNameResolver.resolve(SimpleNameResolver.java:61) > ~[netty-resolver-4.1.86.Final.jar:4.1.86.Final] > at > io.netty.resolver.SimpleNameResolver.resolve(SimpleNameResolver.java:53) > ~[netty-resolver-4.1.86.Final.jar:4.1.86.Final] > at > io.netty.resolver.InetSocketAddressResolver.doResolve(InetSocketAddressResolver.java:55) > ~[netty-resolver-4.1.86.Final.jar:4.1.86.Final] > at > io.netty.resolver.InetSocketAddressResolver.doResolve(InetSocketAddressResolver.java:31) > ~[netty-resolver-4.1.86.Final.jar:4.1.86.Final] > at > io.netty.resolver.AbstractAddressResolver.resolve(AbstractAddressResolver.java:106) > ~[netty-resolver-4.1.86.Final.jar:4.1.86.Final] > at > io.netty.bootstrap.Bootstrap.doResolveAndConnect0(Bootstrap.java:206) > ~[netty-transport-4.1.86.Final.jar:4.1.86.Final] > at io.netty.bootstrap.Bootstrap.access$000(Bootstrap.java:46) > ~[netty-transport-4.1.86.Final.jar:4.1.86.Final] > at > io.netty.bootstrap.Bootstrap$1.operationComplete(Bootstrap.java:180) > ~[netty-transport-4.1.86.Final.jar:4.1.86.Final] > at > io.netty.bootstrap.Bootstrap$1.operationComplete(Bootstrap.java:166) > ~[netty-transport-4.1.86.Final.jar:4.1.86.Final] > at > io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:590) > ~[netty-common-4.1.86.Final.jar:4.1.86.Final] > at > io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:557) > ~[netty-common-4.1.86.Final.jar:4.1.86.Final] > at > io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:492) > ~[netty-common-4.1.86.Final.jar:4.1.86.Final] > at > io.netty.util.concurrent.DefaultPromise.setValue0(DefaultPromise.java:636) > ~[netty-common-4.1.86.Final.jar:4.1.86.Final] > at > io.netty.util.concurrent.DefaultPromise.setSuccess0(DefaultPromise.java:625) > ~[netty-common-4.1.86.Final.jar:4.1.86.Final] > at > io.netty.util.concurrent.DefaultPromise.trySuccess(DefaultPromise.java:105) > ~[netty-common-4.1.86.Final.jar:4.1.86.Final] > at > io.netty.channel.DefaultChannelPromise.trySuccess(DefaultChannelPromise.java:84) > ~[netty-transport-4.1.86.Final.jar:4.1.86.Final] > at > io.netty.channel.AbstractChannel$AbstractUnsafe.safeSetSuccess(AbstractChannel.java:990) > ~[netty-transport-4.1.86.Final.jar:4.1.86.Final] > at > io.netty.channel.AbstractChannel$AbstractUnsafe.register0(AbstractChannel.java:516) > ~[netty-transport-4.1.86.Final.jar:4.1.86.Final] > at > io.netty.channel.AbstractChannel$AbstractUnsafe.access$200(AbstractChannel.java:429) > ~[netty-transport-4.1.86.Final.jar:4.1.86.Final] > at > io.netty.channel.AbstractChannel$AbstractUnsafe$1.run(AbstractChannel.java:486) > ~[netty-transport-4.1.86.Final.jar:4.1.86.Final] > at > io.netty.util.concurrent.AbstractEventExecutor.runTask(AbstractEventExecutor.java:174) > ~[netty-common-4.1.86.Final.jar:4.1.86.Final] > at > io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:167) > ~[netty-common-4.1.86.Final.jar:4.1.86.Final] > at > io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:470) > ~[netty-common-4.1.86.Final.jar:4.1.86.Final] > at > io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:406) > ~[netty-transport-classes-epoll-4.1.86.Final.jar:4.1.86.Final] > at > io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) > ~[netty-common-4.1.86.Final.jar:4.1.86.Final] > at > io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) > ~[netty-common-4.1.86.Final.jar:4.1.86.Final] > at > org.apache.activemq.artemis.utils.ActiveMQThreadFactory$1.run(ActiveMQThreadFactory.java:118) > ~[artemis-commons-2.28.0.jar:?] > 2023-03-15 10:04:10,868 ERROR [org.apache.activemq.artemis.core.client] > AMQ214016: Failed to create netty connection > java.net.UnknownHostException: jms-01-deployment-6d7dcb8ccd-2t9g2 > at java.net.InetAddress$CachedAddresses.get(InetAddress.java:797) > ~[?:?] > at java.net.InetAddress.getAllByName0(InetAddress.java:1524) ~[?:?] > at java.net.InetAddress.getAllByName(InetAddress.java:1382) ~[?:?] > at java.net.InetAddress.getAllByName(InetAddress.java:1306) ~[?:?] > at java.net.InetAddress.getByName(InetAddress.java:1256) ~[?:?] > > IMO, I need to find some way in which the pod name resolves to pod ip. If > you have any recommendations, let me know. > > Regards, > Prateek Jain > > > -------------------------------------------------------------- > EXPECTATION : Causes all troubles...... > -------------------------------------------------------------- > > > On Wed, Mar 15, 2023 at 2:23 PM Justin Bertram <jbert...@apache.org> > wrote: > >> ActiveMQ Artemis offers JGroups integration for cloud use-cases such as >> yours, but, generally speaking, we don't really provide support for >> JGroups >> in the ActiveMQ community since we don't maintain the JGroups code-base >> and >> don't know its inner workings. >> >> That said, I would suggest that the JGroups configuration using FILE_PING >> shipped in the examples may not be the best for your use-case. We chose >> the >> FILE_PING configuration because it was the simplest for the use-cases in >> the examples (i.e. where all the nodes are running on the same machine on >> different ports with access to the same disk). However, something like >> KUBE_PING [1] would probably be a better fit for your use-case since you >> are, in fact, using Kubernetes. >> >> In any event, it's hard to comment on what may be happening without more >> information. You've only provided a snippet of the stack-trace for the >> UnknownHostException so it's not clear what class is actually attempting >> that operation. >> >> >> Justin >> >> [1] https://github.com/jgroups-extras/jgroups-kubernetes >> >> On Wed, Mar 15, 2023 at 7:27 AM prateekjai...@gmail.com < >> prateekjai...@gmail.com> wrote: >> >> > Hi All, >> > >> > How are you? I am a bit confused about the errors reported by artemis >> slave >> > instance (running in clustered mode using jgroups). I might be missing >> > something here so please bear with me. >> > >> > Deployment details: I am running 2 pods of artemis in master/slave >> > configuration in k8s. It supports NFS v4. I am using jgroups >> configuration >> > (same file which is supplied in examples of artemis) and I can see that >> > correct pod names and IPs are updated in share directory >> > >> > -------- output from jgroups file ---------------- >> > root@jms-01-deployment-6d7dcb8ccd-2t9g2:/opt# cat >> > >> > >> /usr/share/artemis-jgroups/active_broadcast_channel/ab62408d-6a7a-4894-9923-2649d8bf3b28.jms-01-deployment-6d7dcb8ccd-2t9g2-29095.list >> > >> > jms-01-deployment-6d7dcb8ccd-2t9g2-29095 >> > ab62408d-6a7a-4894-9923-2649d8bf3b28 192.168.48.154:7800 T >> > jms-01-deployment-6d7dcb8ccd-j7pxl-34353 >> > 5b3d099b-8e5f-4dd0-ab43-604ad1ae87c3 192.168.199.2:7800 F >> > -------------------------------------------------------- >> > >> > POD "jms-01-deployment-6d7dcb8ccd-2t9g2" is the master and another is >> > slave. Correct POD ips are listed there. But in log/artemis.log file of >> > slave pod, I see these logs >> > >> > ------------------------------- >> > 2023-03-15 11:29:51,519 ERROR [org.apache.activemq.artemis.core.client] >> > AMQ214016: Failed to create netty connection >> > java.net.UnknownHostException: jms-01-deployment-6d7dcb8ccd-2t9g2 >> > at java.net.InetAddress$CachedAddresses.get(InetAddress.java:797) ~[?:?] >> > at java.net.InetAddress.getAllByName0(InetAddress.java:1524) ~[?:?] >> > at java.net.InetAddress.getAllByName(InetAddress.java:1382) ~[?:?] >> > ----------------------------------- >> > >> > jgroups is correctly creating and updating files. IPs of master/slave >> pods >> > are visible from each other. I have tested them using command >> > >> > "./artemis check node". >> > Right now I have created deployment objects. Is it necessary to create >> > stateful sets? >> > >> > Am I missing something here? Let me know if you need more information on >> > this. >> > >> > >> > Regards, >> > Prateek Jain >> > >> > >> > -------------------------------------------------------------- >> > EXPECTATION : Causes all troubles...... >> > -------------------------------------------------------------- >> > >> >
