Great! I think I found the problem with K8s services not getting cleaned up. I see a NPE at removeKubernetesServices(KubernetesIaas.java:1025) when undeploying an application.
[2015-09-21 00:31:16,750] ERROR {org.apache.stratos.messaging.message.receiver.cluster.status.ClusterStatusEventMessageDelegator} - Failed to retrieve cluster status event message java.lang.NullPointerException at org.apache.stratos.cloud.controller.iaases.kubernetes.KubernetesIaas.removeKubernetesServices(KubernetesIaas.java:1025) at org.apache.stratos.cloud.controller.messaging.topology.TopologyBuilder.handleClusterTerminatingEvent(TopologyBuilder.java:1068) at org.apache.stratos.cloud.controller.messaging.receiver.cluster.status.ClusterStatusTopicReceiver$5.onEvent(ClusterStatusTopicReceiver.java:86) at org.apache.stratos.messaging.listener.EventListener.update(EventListener.java:42) at java.util.Observable.notifyObservers(Observable.java:159) at org.apache.stratos.messaging.event.EventObservable.notifyEventListeners(EventObservable.java:51) at org.apache.stratos.messaging.message.processor.cluster.status.ClusterStatusClusterTerminatingMessageProcessor.process(ClusterStatusClusterTerminatingMessageProcessor.java:48) at org.apache.stratos.messaging.message.processor.cluster.status.ClusterStatusClusterTerminatedMessageProcessor.process(ClusterStatusClusterTerminatedMessageProcessor.java:55) at org.apache.stratos.messaging.message.processor.cluster.status.ClusterStatusClusterInactivateMessageProcessor.process(ClusterStatusClusterInactivateMessageProcessor.java:52) at org.apache.stratos.messaging.message.processor.cluster.status.ClusterStatusClusterActivatedMessageProcessor.process(ClusterStatusClusterActivatedMessageProcessor.java:52) at org.apache.stratos.messaging.message.processor.cluster.status.ClusterStatusClusterResetMessageProcessor.process(ClusterStatusClusterResetMessageProcessor.java:52) at org.apache.stratos.messaging.message.processor.MessageProcessorChain.process(MessageProcessorChain.java:61) at org.apache.stratos.messaging.message.receiver.cluster.status.ClusterStatusEventMessageDelegator.run(ClusterStatusEventMessageDelegator.java:71) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745) On Mon, Sep 21, 2015 at 12:30 AM, Imesh Gunaratne <im...@apache.org> wrote: > I have now refined this logic to trim all labels to 62 characters and > added following as labels: > > Pod: Application-ID, Cluster-Instance-ID, Member-ID > Service: Application-ID, Cluster-ID, Port-Name > > Now I'm seeing a different problem. When an application is un-deployed, > services do not get deleted. We need to fix this. > > Thanks > > On Mon, Sep 21, 2015 at 12:12 AM, Gayan Gunarathne <gay...@wso2.com> > wrote: > >> Hi Akila, >> >> On Sun, Sep 20, 2015 at 11:30 PM, Akila Ravihansa Perera < >> raviha...@wso2.com> wrote: >> >>> Hi Gayan, >>> >>> +1 for properly identifying attributes to be used for labels and >>> annotations. As I mentioned earlier I tried to use them as labels but ran >>> into length limitation issues. >>> >>> memberId and clusterId cannot be used directly since they are generated >>> as follows; >>> >>> memberId = clusterId + UUID >>> clusterId = applicationId + "." + alias + "." + cartridgeType + ".domain" >>> >>> As you can see those can go be beyond 63 characters since we don't have >>> a limitation for applicationId length. I'm -1 on imposing a character >>> limitation to applicationId only to get away with Kubernetes restriction. >>> What if some other provider comes out with a different limitation? It >>> should be restricted but only for good reasons concerning to Stratos. >>> >> >> Right. Let's say we are not going to restrict the application id length. >> So in that case we won't be able to use the application id as a pod label. >> IMO we used application id to identify the application. As it is a >> identifier I think it is OK to have a restriction.I think using a >> description for the application id is not practical.Also in most of the use >> cases identifiers comes up with some length restrictions.If we don't have a >> restriction on application id , users can enter description sort of a >> application id. >> >> >>> >>> I had an offline chat with Imesh and we were discussing about >>> implementing a sequence number based approach for generating clusterId and >>> memberId without using UUIDs. Any thoughts? >>> >> >> I am also +1 for this thinking. So how we are going to generate the >> cluster id? I think we need to generate the meaningful ids. >> >> >>> >>> Thanks. >>> >>> >>> >>> Thanks, >> Gayan >> >>> >>> >>> On Sun, Sep 20, 2015 at 10:55 PM, Gayan Gunarathne <gay...@wso2.com> >>> wrote: >>> >>>> As per Kubernetes definition I think we are using labels and annotation >>>> in a wrong way.As annotations are arbitrary non-identifying metadata , they >>>> might not shown them in their UI/CLI >>>> >>>> I think first we need to recognize what are the identifiers for the >>>> pod.In think we may need to use member id , application id >>>> >>>> Also what is the ration behind of using the md5Hex of the member id as >>>> a identifier. IMO we can directly use the member id. As member id is UUID >>>> it won't be more than 63 characters. >>>> >>>> Thanks, >>>> Gayan >>>> >>>> On Sun, Sep 20, 2015 at 10:13 PM, Imesh Gunaratne <im...@apache.org> >>>> wrote: >>>> >>>>> Hi Akila, >>>>> >>>>> Right, I get your point on the length limitation, but this is how >>>>> Kubernetes has described Labels and Annotations: >>>>> >>>>> *We'll eventually index and reverse-index labels for efficient queries >>>>> and watches, use them to sort and group in UIs and CLIs, etc. We don't >>>>> want >>>>> to pollute labels with non-identifying, especially large and/or >>>>> structured, >>>>> data. Non-identifying information should be recorded using annotations.* >>>>> >>>>> The idea is to use labels for all identifiers. >>>>> >>>>> http://kubernetes.io/v1.0/docs/user-guide/labels.html >>>>> >>>>> On Sun, Sep 20, 2015 at 10:05 PM, Akila Ravihansa Perera < >>>>> raviha...@wso2.com> wrote: >>>>> >>>>>> Hi Imesh, >>>>>> >>>>>> Yes, I tried adding them as labels but ran into a problem since label >>>>>> length cannot be greater than 63 characters. I'm not sure why we don't >>>>>> see >>>>>> annotations in the UI for pods created. But I can see the annotations >>>>>> added >>>>>> for services. Might have to raise this in K8s mailing list. >>>>>> >>>>>> Thanks. >>>>>> >>>>>> On Sun, Sep 20, 2015 at 10:00 PM, Imesh Gunaratne <im...@apache.org> >>>>>> wrote: >>>>>> >>>>>>> Hi Akila, >>>>>>> >>>>>>> We do not query pod labels, its handled by the Kubernetes service >>>>>>> selector, at the moment this is called "name". I have now fixed this >>>>>>> issue >>>>>>> locally, will push in few minutes. >>>>>>> >>>>>>> In addition I think it would be better to use labels instead of >>>>>>> annotations because they are not visible in the UI: >>>>>>> >>>>>>> >>>>>>> >>>>>>> Thanks >>>>>>> >>>>>>> On Sun, Sep 20, 2015 at 9:52 PM, Akila Ravihansa Perera < >>>>>>> raviha...@wso2.com> wrote: >>>>>>> >>>>>>>> Hi Imesh, >>>>>>>> >>>>>>>> Where do we query those pod labels? >>>>>>>> >>>>>>>> I'm in the process of moving label names to constants. I can apply >>>>>>>> the same in all the places. I think better to change that label to >>>>>>>> md5Hex-memberId to avoid any confusion when troubleshooting issues. One >>>>>>>> might mistakenly take that as the actual memberId when it is not the >>>>>>>> case. >>>>>>>> >>>>>>>> Thanks. >>>>>>>> >>>>>>>> On Sun, Sep 20, 2015 at 9:40 PM, Imesh Gunaratne <im...@apache.org> >>>>>>>> wrote: >>>>>>>> >>>>>>>>> I'm seeing a problem, we have removed the "name" label in the pod >>>>>>>>> and added a new label called "md5Hex-memberId". Due to this load >>>>>>>>> balancing >>>>>>>>> would not work: >>>>>>>>> >>>>>>>>> Map<String, String> podLabels = new HashMap<>(); >>>>>>>>> podLabels.put("md5Hex-memberId", >>>>>>>>> DigestUtils.md5Hex(memberContext.getMemberId())); >>>>>>>>> >>>>>>>>> Map<String, String> podAnnotations = new HashMap<>(); >>>>>>>>> podAnnotations.put("memberId", memberContext.getMemberId()); >>>>>>>>> podAnnotations.put("cartridgeType", memberContext.getCartridgeType()); >>>>>>>>> podAnnotations.put("applicationId", memberContext.getApplicationId()); >>>>>>>>> podAnnotations.put("clusterId", memberContext.getClusterId()); >>>>>>>>> podAnnotations.put("clusterInstanceId", >>>>>>>>> memberContext.getClusterInstanceId()); >>>>>>>>> >>>>>>>>> kubernetesApi.createPod(podId, podName, podLabels, podAnnotations, >>>>>>>>> dockerImage, cpu, memory, ports, >>>>>>>>> environmentVariables); >>>>>>>>> >>>>>>>>> >>>>>>>>> On Sun, Sep 20, 2015 at 2:51 PM, Akila Ravihansa Perera < >>>>>>>>> raviha...@wso2.com> wrote: >>>>>>>>> >>>>>>>>>> Hi all, >>>>>>>>>> >>>>>>>>>> I've pushed release artifacts to staging repo at [1]. Please take >>>>>>>>>> few mins to verify the release. Also I've pushed cartridge-base, php >>>>>>>>>> and >>>>>>>>>> tomcat Docker images with 4.1.3 artifacts to DockerHub. >>>>>>>>>> >>>>>>>>>> [1] >>>>>>>>>> https://repository.apache.org/content/repositories/orgapachestratos-1057/ >>>>>>>>>> >>>>>>>>>> Thanks. >>>>>>>>>> >>>>>>>>>> -- >>>>>>>>>> Akila Ravihansa Perera >>>>>>>>>> WSO2 Inc.; http://wso2.com/ >>>>>>>>>> >>>>>>>>>> Blog: http://ravihansa3000.blogspot.com >>>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> -- >>>>>>>>> Imesh Gunaratne >>>>>>>>> >>>>>>>>> Senior Technical Lead, WSO2 >>>>>>>>> Committer & PMC Member, Apache Stratos >>>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> -- >>>>>>>> Akila Ravihansa Perera >>>>>>>> WSO2 Inc.; http://wso2.com/ >>>>>>>> >>>>>>>> Blog: http://ravihansa3000.blogspot.com >>>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> -- >>>>>>> Imesh Gunaratne >>>>>>> >>>>>>> Senior Technical Lead, WSO2 >>>>>>> Committer & PMC Member, Apache Stratos >>>>>>> >>>>>> >>>>>> >>>>>> >>>>>> -- >>>>>> Akila Ravihansa Perera >>>>>> WSO2 Inc.; http://wso2.com/ >>>>>> >>>>>> Blog: http://ravihansa3000.blogspot.com >>>>>> >>>>> >>>>> >>>>> >>>>> -- >>>>> Imesh Gunaratne >>>>> >>>>> Senior Technical Lead, WSO2 >>>>> Committer & PMC Member, Apache Stratos >>>>> >>>> >>>> >>>> >>>> -- >>>> >>>> Gayan Gunarathne >>>> Technical Lead, WSO2 Inc. (http://wso2.com) >>>> Committer & PMC Member, Apache Stratos >>>> email : gay...@wso2.com | mobile : +94 775030545 <%2B94%20766819985> >>>> >>>> >>>> >>> >>> >>> >>> -- >>> Akila Ravihansa Perera >>> WSO2 Inc.; http://wso2.com/ >>> >>> Blog: http://ravihansa3000.blogspot.com >>> >> >> >> >> -- >> >> Gayan Gunarathne >> Technical Lead, WSO2 Inc. (http://wso2.com) >> Committer & PMC Member, Apache Stratos >> email : gay...@wso2.com | mobile : +94 775030545 <%2B94%20766819985> >> >> >> > > > > -- > Imesh Gunaratne > > Senior Technical Lead, WSO2 > Committer & PMC Member, Apache Stratos > -- Akila Ravihansa Perera WSO2 Inc.; http://wso2.com/ Blog: http://ravihansa3000.blogspot.com