Re: Failed to terminate member Error appearing repeatedly

2014-10-28 Thread Rajkumar Rajaratnam
Hmm yeah we are creating a new member, add it to the topology and sending InstanceSpawnedEvent. My concern was whether it is expensive to update the topology again and again. But then realized that we are doing it every time when CC receives each instance status event :) So yeah, no problem. We c

Re: Failed to terminate member Error appearing repeatedly

2014-10-28 Thread Nirmal Fernando
Each event meant to carry a new set of information. On Tue, Oct 28, 2014 at 11:07 AM, Rajkumar Rajaratnam wrote: > Hmm yeah we are creating a new member, add it to the topology and > sending InstanceSpawnedEvent. > > My concern was whether it is expensive to update the topology again and > again

Re: Failed to terminate member Error appearing repeatedly

2014-10-28 Thread Nirmal Fernando
MemberSpawned event will update it, isn't it? On Tue, Oct 28, 2014 at 10:53 AM, Rajkumar Rajaratnam wrote: > Seems we are having a problem here. If we are adding to the topology as > soon as CC gets a instance start up call, at that point we do not have > information such as member ip, allocated

Re: Failed to terminate member Error appearing repeatedly

2014-10-28 Thread Rajkumar Rajaratnam
Seems we are having a problem here. If we are adding to the topology as soon as CC gets a instance start up call, at that point we do not have information such as member ip, allocated port and so on. If we add it to the topology at this point, then we might need to update topology again once we get

Re: Failed to terminate member Error appearing repeatedly

2014-10-28 Thread Rajkumar Rajaratnam
On Tue, Oct 28, 2014 at 3:00 PM, Nirmal Fernando wrote: > I think the correct approach would be to introduce a new topology event to > fill the blind area between real instance creation via CC and member > spawned event. So, as soon as CC receives a instance start up call, CC > would add the memb

Re: Failed to terminate member Error appearing repeatedly

2014-10-28 Thread Nirmal Fernando
I think the correct approach would be to introduce a new topology event to fill the blind area between real instance creation via CC and member spawned event. So, as soon as CC receives a instance start up call, CC would add the member to topology and will send a new event with a new state possibly

Re: Failed to terminate member Error appearing repeatedly

2014-10-28 Thread Chamila De Alwis
Hi Raj, On Tue, Oct 28, 2014 at 2:23 PM, Rajkumar Rajaratnam wrote: > CC will add it to the topology, send member created event and send member > terminated soon after that. Now AS will get this event and remove the > member from its lists. We can't send member terminated events for members > w

Re: Failed to terminate member Error appearing repeatedly

2014-10-28 Thread Rajkumar Rajaratnam
Hi, I found the cause for this. Let me summarize. - AS asks CC to create a container - CC schedule a new task and returning member contexts to AS - AS adding the members to pending list - CC is also adding member contexts to its data holder - But CC will not add it to the topology

Re: Failed to terminate member Error appearing repeatedly

2014-10-27 Thread Rajkumar Rajaratnam
Hi Chamila, This happens when the container is actually killed by CC, but CC didn't send the member terminated event. Hence AS didn't remove the member from obsolete list. In next cycle it is asking CC again to terminated the instance. Since CC terminated it already, it will report this error. I g

Failed to terminate member Error appearing repeatedly

2014-10-27 Thread Chamila De Alwis
Hi, I'm doing a test run with the Python cartridge agent in Kubernetes. I have three pods running, with the two machine setup (discovery + master/minion). Stratos is from master, built about 8 hours ago. After a while the following error started appearing. [2014-10-28 01:03:07,947] INFO {org.a