Re: Cluster in AWS can not have more than 100 nodes?
Hello! Please check logs from all nodes to determine if there are any nodes which did not join, and what is their reasoning for this. Regards, -- Ilya Kasnacheev сб, 9 нояб. 2019 г. в 02:57, codeboyyong : > Hi Ilya > yes from the message I can see there are more than 100 server nodes : > > Topology snapshot [ver=2104, locNode=63120c8d, servers=117, clients=129, > state=ACTIVE, CPUs=9240, offheap=2300.0GB, heap=7300.0GB] > > We are running a spark job keep writing to the database every minute. > At least from the long I can see there are 117 server nodes, comparing with > the 100 ip in S3 bucket, it is better but still no sure why it is not 128 > nodes we started . > > Thank You > Yong > > > > -- > Sent from: http://apache-ignite-users.70518.x6.nabble.com/ >
Re: Cluster in AWS can not have more than 100 nodes?
Hi Ilya yes from the message I can see there are more than 100 server nodes : Topology snapshot [ver=2104, locNode=63120c8d, servers=117, clients=129, state=ACTIVE, CPUs=9240, offheap=2300.0GB, heap=7300.0GB] We are running a spark job keep writing to the database every minute. At least from the long I can see there are 117 server nodes, comparing with the 100 ip in S3 bucket, it is better but still no sure why it is not 128 nodes we started . Thank You Yong -- Sent from: http://apache-ignite-users.70518.x6.nabble.com/
Re: Cluster in AWS can not have more than 100 nodes?
Hello! List of hosts present in IP finder has very little bearing on list of nodes in cluster. Server node's host do not have to be present in IP finder in order for this node to function. Cluster will be operational until there exists at least one node which is a) alive, b) reachable, and c) listed in IP finder. In your case there are 100 of such nodes. Please share the latest "Topology snapshot" messages from your nodes. Regards, -- Ilya Kasnacheev вт, 5 нояб. 2019 г. в 18:51, Stanislav Lukyanov : > Each node is supposed to add its own IP and port to the S3 bucket when it > starts. That said, I wouldn't check the cluster state based on the contents > of the bucket alone. > Check your logs for errors. Try using some tools (e.g. check out Web > Console - either the one in Ignite > https://apacheignite-tools.readme.io/docs/getting-started or from > GridGain > https://www.gridgain.com/products/software/enterprise-edition/management-tool) > to see whether all nodes are there. > > If you don't see any clues, please share your logs from any of the running > servers. Ideally, a log from one of the 100 servers which joined the > cluster, and a log from one of the 28 that didn't. > > Stan > > On Fri, Nov 1, 2019 at 2:41 AM codeboyyong wrote: > >> No . The thing is we started ignite in embedded mode as web app, we can >> see >> there are 128 instances. >> But when we check he S3 bucket , it always only have 100 its, which means >> only 100 nodes joined the ignite cluster. Not sure if there are another >> way >> to get the cluster size. >> Since we are running as web app, we actually dont have all the access of >> command line tools. >> >> Thank You >> >> >> >> -- >> Sent from: http://apache-ignite-users.70518.x6.nabble.com/ >> >
Re: Cluster in AWS can not have more than 100 nodes?
Each node is supposed to add its own IP and port to the S3 bucket when it starts. That said, I wouldn't check the cluster state based on the contents of the bucket alone. Check your logs for errors. Try using some tools (e.g. check out Web Console - either the one in Ignite https://apacheignite-tools.readme.io/docs/getting-started or from GridGain https://www.gridgain.com/products/software/enterprise-edition/management-tool) to see whether all nodes are there. If you don't see any clues, please share your logs from any of the running servers. Ideally, a log from one of the 100 servers which joined the cluster, and a log from one of the 28 that didn't. Stan On Fri, Nov 1, 2019 at 2:41 AM codeboyyong wrote: > No . The thing is we started ignite in embedded mode as web app, we can see > there are 128 instances. > But when we check he S3 bucket , it always only have 100 its, which means > only 100 nodes joined the ignite cluster. Not sure if there are another > way > to get the cluster size. > Since we are running as web app, we actually dont have all the access of > command line tools. > > Thank You > > > > -- > Sent from: http://apache-ignite-users.70518.x6.nabble.com/ >
Re: Cluster in AWS can not have more than 100 nodes?
There is no hard limit of this kind, here is the source code of the IP finder: https://github.com/apache/ignite/blob/master/modules/aws/src/main/java/org/apache/ignite/spi/discovery/tcp/ipfinder/s3/TcpDiscoveryS3IpFinder.java You can enable DEBUG level for Discovery and the IP Finder to collect more details. - Denis On Thu, Oct 31, 2019 at 4:41 PM codeboyyong wrote: > No . The thing is we started ignite in embedded mode as web app, we can see > there are 128 instances. > But when we check he S3 bucket , it always only have 100 its, which means > only 100 nodes joined the ignite cluster. Not sure if there are another > way > to get the cluster size. > Since we are running as web app, we actually dont have all the access of > command line tools. > > Thank You > > > > -- > Sent from: http://apache-ignite-users.70518.x6.nabble.com/ >
Re: Cluster in AWS can not have more than 100 nodes?
No . The thing is we started ignite in embedded mode as web app, we can see there are 128 instances. But when we check he S3 bucket , it always only have 100 its, which means only 100 nodes joined the ignite cluster. Not sure if there are another way to get the cluster size. Since we are running as web app, we actually dont have all the access of command line tools. Thank You -- Sent from: http://apache-ignite-users.70518.x6.nabble.com/
Re: Cluster in AWS can not have more than 100 nodes?
Maybe you have an instance limit of 100 on aws side ? > Am 31.10.2019 um 19:09 schrieb codeboyyong : > > Hi Friends, > I have a big ignite cluster running in private was like cloud, I use > TcpDiscoverySpi with TcpDiscoveryS3IpFinder and run with 96 nodes, it works > fine. > Today I redeploy it with 128 nodes and found in the S3 buckets, there are > always show 100 nodes. Is there any config or hard limit node numbers on > TcpDiscoverySpi ? > Can someone pls let us know how to expand more than 100 nodes ? > > > Thank You > Very much! > > > > -- > Sent from: http://apache-ignite-users.70518.x6.nabble.com/
Cluster in AWS can not have more than 100 nodes?
Hi Friends, I have a big ignite cluster running in private was like cloud, I use TcpDiscoverySpi with TcpDiscoveryS3IpFinder and run with 96 nodes, it works fine. Today I redeploy it with 128 nodes and found in the S3 buckets, there are always show 100 nodes. Is there any config or hard limit node numbers on TcpDiscoverySpi ? Can someone pls let us know how to expand more than 100 nodes ? Thank You Very much! -- Sent from: http://apache-ignite-users.70518.x6.nabble.com/
