Hello, I am trying to deploy an ignite based application on an Azure K8S using helm charts.
I test my application locally on a docker-desktop based K8S with obviously only one node (my laptop) That works quite well and when I increase the replicas, the new pods join the cluster and it works. The ignite service lists all the pods without any issues. When I deploy on Azure though not all of the pods join the cluster but some do and I fail to understand why. I started with only one node. The ignite service does list all the pods. I have no error message in none of the console. Any idea on how to investigate that? Some log of the different pods: >>> +----------------------------------------------------------------------+ >>> Ignite ver. 2.7.5#20190603-sha1:be4f2a158bcf79a52ab4f372a6576f20c4f86954 >>> +----------------------------------------------------------------------+ >>> OS name: Linux 4.15.0-1061-azure amd64 >>> CPU(s): 1 >>> Heap: 0.5GB >>> VM name: 1@v1-processing-grid.master-545c9d7594-7pcxz >>> Ignite instance name: com.wolterskluwer.processing >>> Local node [ID=AA51A811-7844-4E1D-839F-59832EE6FB2A, order=2, >>> clientMode=false] >>> Local node addresses: >>> [v1-processing-grid.master-545c9d7594-7pcxz/10.244.0.17, /127.0.0.1] >>> Local ports: TCP:10500 TCP:10800 TCP:11211 TCP:11400 TCP:47100 TCP:47500 INFO [org.apache.ignite.internal.managers.discovery.GridDiscoveryManager] (ServerService Thread Pool -- 6) Topology snapshot [ver=2, locNode=aa51a811, servers=2, clients=0, state=ACTIVE, CPUs=2, offheap=0.95GB, heap=1.0GB] >>> +----------------------------------------------------------------------+ >>> Ignite ver. 2.7.5#20190603-sha1:be4f2a158bcf79a52ab4f372a6576f20c4f86954 >>> +----------------------------------------------------------------------+ >>> OS name: Linux 4.15.0-1061-azure amd64 >>> CPU(s): 1 >>> Heap: 0.5GB >>> VM name: 1@v1-processing-grid.worker-866b666878-2g4j8 >>> Ignite instance name: com.wolterskluwer.processing >>> Local node [ID=0CD902B3-271E-43FD-9F83-9E76570743C2, order=1, >>> clientMode=false] >>> Local node addresses: >>> [v1-processing-grid.worker-866b666878-2g4j8/10.244.0.13, /127.0.0.1] >>> Local ports: TCP:10500 TCP:10800 TCP:11211 TCP:11400 TCP:47100 UDP:47400 >>> TCP:47500 INFO [org.apache.ignite.internal.managers.discovery.GridDiscoveryManager] (ServerService Thread Pool -- 15) Topology snapshot [ver=1, locNode=0cd902b3, servers=1, clients=0, state=ACTIVE, CPUs=1, offheap=0.48GB, heap=0.5GB] >>> +----------------------------------------------------------------------+ >>> Ignite ver. 2.7.5#20190603-sha1:be4f2a158bcf79a52ab4f372a6576f20c4f86954 >>> +----------------------------------------------------------------------+ >>> OS name: Linux 4.15.0-1061-azure amd64 >>> CPU(s): 1 >>> Heap: 0.5GB >>> VM name: 1@v1-processing-grid.worker-866b666878-4kwrq >>> Ignite instance name: com.wolterskluwer.processing >>> Local node [ID=6CA4D1A2-BDCD-4F12-B185-A89CA57B4430, order=1, >>> clientMode=false] >>> Local node addresses: >>> [v1-processing-grid.worker-866b666878-4kwrq/10.244.0.11, /127.0.0.1] >>> Local ports: TCP:10500 TCP:10800 TCP:11211 TCP:11400 TCP:47100 UDP:47400 >>> TCP:47500 INFO [org.apache.ignite.internal.managers.discovery.GridDiscoveryManager] (ServerService Thread Pool -- 6) Topology snapshot [ver=2, locNode=aa51a811, servers=2, clients=0, state=ACTIVE, CPUs=2, offheap=0.95GB, heap=1.0GB]-12-14 17:34:46,125 INFO [org.apache.ignite.internal.managers.discovery.GridDiscoveryManager] (ServerService Thread Pool -- 6) Topology snapshot [ver=1, locNode=6ca4d1a2, servers=1, clients=0, state=ACTIVE, CPUs=1, offheap=0.48GB, heap=0.5GB] >>> +----------------------------------------------------------------------+ >>> Ignite ver. 2.7.5#20190603-sha1:be4f2a158bcf79a52ab4f372a6576f20c4f86954 >>> +----------------------------------------------------------------------+ >>> OS name: Linux 4.15.0-1061-azure amd64 >>> CPU(s): 1 >>> Heap: 0.5GB >>> VM name: 1@v1-processing-grid.worker-866b666878-hdqdf >>> Ignite instance name: com.wolterskluwer.processing >>> Local node [ID=B9C318E9-42E2-43F4-BD58-8FC8C42C68B7, order=1, >>> clientMode=false] >>> Local node addresses: >>> [v1-processing-grid.worker-866b666878-hdqdf/10.244.0.15, /127.0.0.1] >>> Local ports: TCP:10500 TCP:10800 TCP:11211 TCP:11400 TCP:47100 UDP:47400 >>> TCP:47500 INFO [org.apache.ignite.internal.managers.discovery.GridDiscoveryManager] (ServerService Thread Pool -- 15) Topology snapshot [ver=1, locNode=b9c318e9, servers=1, clients=0, state=ACTIVE, CPUs=1, offheap=0.48GB, heap=0.5GB] >>> +----------------------------------------------------------------------+ >>> Ignite ver. 2.7.5#20190603-sha1:be4f2a158bcf79a52ab4f372a6576f20c4f86954 >>> +----------------------------------------------------------------------+ >>> OS name: Linux 4.15.0-1061-azure amd64 >>> CPU(s): 1 >>> Heap: 0.5GB >>> VM name: 1@v1-processing-grid.worker-866b666878-m7d8s >>> Ignite instance name: com.wolterskluwer.processing >>> Local node [ID=6698A357-FCF6-4BEC-85D7-D22CF1754A87, order=1, >>> clientMode=false] >>> Local node addresses: >>> [v1-processing-grid.worker-866b666878-m7d8s/10.244.0.12, /127.0.0.1] >>> Local ports: TCP:10500 TCP:10800 TCP:11211 TCP:11400 TCP:47100 UDP:47400 >>> TCP:47500 INFO [org.apache.ignite.internal.managers.discovery.GridDiscoveryManager] (ServerService Thread Pool -- 15) Topology snapshot [ver=1, locNode=6698a357, servers=1, clients=0, state=ACTIVE, CPUs=1, offheap=0.48GB, heap=0.5GB][0m[0m2019-12-14 17:34:45,820 INFO [org.apache.ignite.internal.exchange.time] (exchange-worker-#39%com.wolterskluwer.processing%) Started exchange init [topVer=AffinityTopologyVersion [topVer=1, minorTopVer=5], mvccCrd=MvccCoordinator [nodeId=6698a357-fcf6-4bec-85d7-d22cf1754a87, crdVer=1576344882399, topVer=AffinityTopologyVersion [topVer=1, minorTopVer=0]], mvccCrdChange=false, crd=true, evt=DISCOVERY_CUSTOM_EVT, evtNode=6698a357-fcf6-4bec-85d7-d22cf1754a87, customEvt=DynamicCacheChangeBatch [id=380f9750f61-8b71e6cb-20d5-44ec-a1e1-bbd9590cd29f, reqs=[DynamicCacheChangeRequest [cacheName=$$PART_CACHE$$, hasCfg=true, nodeId=6698a357-fcf6-4bec-85d7-d22cf1754a87, clientStartOnly=false, stop=false, destroy=false, disabledAfterStartfalse]], exchangeActions=Ex INFO [org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi] (tcp-disco-sock-reader-#7%com.wolterskluwer.processing%) Started serving remote node connection [rmtAddr=/10.244.0.17:57771, rmtPort=57771] INFO [org.apache.ignite.internal.managers.discovery.GridDiscoveryManager] (disco-event-worker-#38%com.wolterskluwer.processing%) Added new node to topology: TcpDiscoveryNode [id=aa51a811-7844-4e1d-839f-59832ee6fb2a, addrs=[10.244.0.17, 127.0.0.1], sockAddrs=[/10.244.0.17:47500, /127.0.0.1:47500], discPort=47500, order=2, intOrder=2, lastExchangeTime=1576344939198, loc=false, ver=2.7.5#20190603-sha1:be4f2a15, isClient=false] INFO [org.apache.ignite.internal.managers.discovery.GridDiscoveryManager] (disco-event-worker-#38%com.wolterskluwer.processing%) Topology snapshot [ver=2, locNode=6698a357, servers=2, clients=0, state=ACTIVE, CPUs=2, offheap=0.95GB, heap=1.0GB] >>> +----------------------------------------------------------------------+ >>> Ignite ver. 2.7.5#20190603-sha1:be4f2a158bcf79a52ab4f372a6576f20c4f86954 >>> +----------------------------------------------------------------------+ >>> OS name: Linux 4.15.0-1061-azure amd64 >>> CPU(s): 1 >>> Heap: 0.5GB >>> VM name: 1@v1-processing-grid.worker-866b666878-q2kln >>> Ignite instance name: com.wolterskluwer.processing >>> Local node [ID=FBFBA982-3470-49B3-A2E9-E6F426EE8034, order=1, >>> clientMode=false] >>> Local node addresses: >>> [v1-processing-grid.worker-866b666878-q2kln/10.244.0.16, /127.0.0.1] >>> Local ports: TCP:10500 TCP:10800 TCP:11211 TCP:11400 TCP:47100 UDP:47400 >>> TCP:47500 INFO [org.apache.ignite.internal.managers.discovery.GridDiscoveryManager] (ServerService Thread Pool -- 15) Topology snapshot [ver=1, locNode=fbfba982, servers=1, clients=0, state=ACTIVE, CPUs=1, offheap=0.48GB, heap=0.5GB] -- Sent from: http://apache-ignite-users.70518.x6.nabble.com/
