Great!

I think I found the problem with K8s services not getting cleaned up. I see
a NPE at removeKubernetesServices(KubernetesIaas.java:1025) when
undeploying an application.

[2015-09-21 00:31:16,750] ERROR
{org.apache.stratos.messaging.message.receiver.cluster.status.ClusterStatusEventMessageDelegator}
-  Failed to retrieve cluster status event message
java.lang.NullPointerException
at
org.apache.stratos.cloud.controller.iaases.kubernetes.KubernetesIaas.removeKubernetesServices(KubernetesIaas.java:1025)
at
org.apache.stratos.cloud.controller.messaging.topology.TopologyBuilder.handleClusterTerminatingEvent(TopologyBuilder.java:1068)
at
org.apache.stratos.cloud.controller.messaging.receiver.cluster.status.ClusterStatusTopicReceiver$5.onEvent(ClusterStatusTopicReceiver.java:86)
at
org.apache.stratos.messaging.listener.EventListener.update(EventListener.java:42)
at java.util.Observable.notifyObservers(Observable.java:159)
at
org.apache.stratos.messaging.event.EventObservable.notifyEventListeners(EventObservable.java:51)
at
org.apache.stratos.messaging.message.processor.cluster.status.ClusterStatusClusterTerminatingMessageProcessor.process(ClusterStatusClusterTerminatingMessageProcessor.java:48)
at
org.apache.stratos.messaging.message.processor.cluster.status.ClusterStatusClusterTerminatedMessageProcessor.process(ClusterStatusClusterTerminatedMessageProcessor.java:55)
at
org.apache.stratos.messaging.message.processor.cluster.status.ClusterStatusClusterInactivateMessageProcessor.process(ClusterStatusClusterInactivateMessageProcessor.java:52)
at
org.apache.stratos.messaging.message.processor.cluster.status.ClusterStatusClusterActivatedMessageProcessor.process(ClusterStatusClusterActivatedMessageProcessor.java:52)
at
org.apache.stratos.messaging.message.processor.cluster.status.ClusterStatusClusterResetMessageProcessor.process(ClusterStatusClusterResetMessageProcessor.java:52)
at
org.apache.stratos.messaging.message.processor.MessageProcessorChain.process(MessageProcessorChain.java:61)
at
org.apache.stratos.messaging.message.receiver.cluster.status.ClusterStatusEventMessageDelegator.run(ClusterStatusEventMessageDelegator.java:71)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)




On Mon, Sep 21, 2015 at 12:30 AM, Imesh Gunaratne <im...@apache.org> wrote:

> I have now refined this logic to trim all labels to 62 characters and
> added following as labels:
>
> Pod: Application-ID, Cluster-Instance-ID, Member-ID
> Service: Application-ID, Cluster-ID, Port-Name
>
> Now I'm seeing a different problem. When an application is un-deployed,
> services do not get deleted. We need to fix this.
>
> Thanks
>
> On Mon, Sep 21, 2015 at 12:12 AM, Gayan Gunarathne <gay...@wso2.com>
> wrote:
>
>> Hi Akila,
>>
>> On Sun, Sep 20, 2015 at 11:30 PM, Akila Ravihansa Perera <
>> raviha...@wso2.com> wrote:
>>
>>> Hi Gayan,
>>>
>>> +1 for properly identifying attributes to be used for labels and
>>> annotations. As I mentioned earlier I tried to use them as labels but ran
>>> into length limitation issues.
>>>
>>> memberId and clusterId cannot be used directly since they are generated
>>> as follows;
>>>
>>> memberId = clusterId + UUID
>>> clusterId = applicationId + "." + alias + "." + cartridgeType + ".domain"
>>>
>>> As you can see those can go be beyond 63 characters since we don't have
>>> a limitation for applicationId length. I'm -1 on imposing a character
>>> limitation to applicationId only to get away with Kubernetes restriction.
>>> What if some other provider comes out with a different limitation? It
>>> should be restricted but only for good reasons concerning to Stratos.
>>>
>>
>> Right. Let's say we are not going to restrict the application id length.
>> So in that case we won't be able to use the application id as a pod label.
>> IMO we used application id to identify the application. As it is a
>> identifier I think it is OK to have a restriction.I think using a
>> description for the application id is not practical.Also in most of the use
>> cases identifiers comes up with some length restrictions.If we don't have a
>> restriction on application id , users can enter description sort of a
>> application id.
>>
>>
>>>
>>> I had an offline chat with Imesh and we were discussing about
>>> implementing a sequence number based approach for generating clusterId and
>>> memberId without using UUIDs. Any thoughts?
>>>
>>
>>  I am also +1 for this thinking. So how we are going to generate the
>> cluster id? I think we need to generate the meaningful ids.
>>
>>
>>>
>>> Thanks.
>>>
>>>
>>>
>>> Thanks,
>> Gayan
>>
>>>
>>>
>>> On Sun, Sep 20, 2015 at 10:55 PM, Gayan Gunarathne <gay...@wso2.com>
>>> wrote:
>>>
>>>> As per Kubernetes definition I think we are using labels and annotation
>>>> in a wrong way.As annotations are arbitrary non-identifying metadata , they
>>>> might not shown them in their UI/CLI
>>>>
>>>> I think first we need to recognize what are the identifiers for the
>>>> pod.In think we may need to use member id , application id
>>>>
>>>> Also what is the ration behind of using the md5Hex of the member id as
>>>> a identifier. IMO we can directly use the member id. As member id is UUID
>>>> it won't be more than 63 characters.
>>>>
>>>> Thanks,
>>>> Gayan
>>>>
>>>> On Sun, Sep 20, 2015 at 10:13 PM, Imesh Gunaratne <im...@apache.org>
>>>> wrote:
>>>>
>>>>> Hi Akila,
>>>>>
>>>>> Right, I get your point on the length limitation, but this is how
>>>>> Kubernetes has described Labels and Annotations:
>>>>>
>>>>> *We'll eventually index and reverse-index labels for efficient queries
>>>>> and watches, use them to sort and group in UIs and CLIs, etc. We don't 
>>>>> want
>>>>> to pollute labels with non-identifying, especially large and/or 
>>>>> structured,
>>>>> data. Non-identifying information should be recorded using annotations.*
>>>>>
>>>>> The idea is to use labels for all identifiers.
>>>>>
>>>>> http://kubernetes.io/v1.0/docs/user-guide/labels.html
>>>>>
>>>>> On Sun, Sep 20, 2015 at 10:05 PM, Akila Ravihansa Perera <
>>>>> raviha...@wso2.com> wrote:
>>>>>
>>>>>> Hi Imesh,
>>>>>>
>>>>>> Yes, I tried adding them as labels but ran into a problem since label
>>>>>> length cannot be greater than 63 characters. I'm not sure why we don't 
>>>>>> see
>>>>>> annotations in the UI for pods created. But I can see the annotations 
>>>>>> added
>>>>>> for services. Might have to raise this in K8s mailing list.
>>>>>>
>>>>>> Thanks.
>>>>>>
>>>>>> On Sun, Sep 20, 2015 at 10:00 PM, Imesh Gunaratne <im...@apache.org>
>>>>>> wrote:
>>>>>>
>>>>>>> Hi Akila,
>>>>>>>
>>>>>>> We do not query pod labels, its handled by the Kubernetes service
>>>>>>> selector, at the moment this is called "name". I have now fixed this 
>>>>>>> issue
>>>>>>> locally, will push in few minutes.
>>>>>>>
>>>>>>> In addition I think it would be better to use labels instead of
>>>>>>> annotations because they are not visible in the UI:
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> Thanks
>>>>>>>
>>>>>>> On Sun, Sep 20, 2015 at 9:52 PM, Akila Ravihansa Perera <
>>>>>>> raviha...@wso2.com> wrote:
>>>>>>>
>>>>>>>> Hi Imesh,
>>>>>>>>
>>>>>>>> Where do we query those pod labels?
>>>>>>>>
>>>>>>>> I'm in the process of moving label names to constants. I can apply
>>>>>>>> the same in all the places. I think better to change that label to
>>>>>>>> md5Hex-memberId to avoid any confusion when troubleshooting issues. One
>>>>>>>> might mistakenly take that as the actual memberId when it is not the 
>>>>>>>> case.
>>>>>>>>
>>>>>>>> Thanks.
>>>>>>>>
>>>>>>>> On Sun, Sep 20, 2015 at 9:40 PM, Imesh Gunaratne <im...@apache.org>
>>>>>>>> wrote:
>>>>>>>>
>>>>>>>>> I'm seeing a problem, we have removed the "name" label in the pod
>>>>>>>>> and added a new label called "md5Hex-memberId". Due to this load 
>>>>>>>>> balancing
>>>>>>>>> would not work:
>>>>>>>>>
>>>>>>>>> Map<String, String> podLabels = new HashMap<>();
>>>>>>>>> podLabels.put("md5Hex-memberId", 
>>>>>>>>> DigestUtils.md5Hex(memberContext.getMemberId()));
>>>>>>>>>
>>>>>>>>> Map<String, String> podAnnotations = new HashMap<>();
>>>>>>>>> podAnnotations.put("memberId", memberContext.getMemberId());
>>>>>>>>> podAnnotations.put("cartridgeType", memberContext.getCartridgeType());
>>>>>>>>> podAnnotations.put("applicationId", memberContext.getApplicationId());
>>>>>>>>> podAnnotations.put("clusterId", memberContext.getClusterId());
>>>>>>>>> podAnnotations.put("clusterInstanceId", 
>>>>>>>>> memberContext.getClusterInstanceId());
>>>>>>>>>
>>>>>>>>> kubernetesApi.createPod(podId, podName, podLabels, podAnnotations, 
>>>>>>>>> dockerImage, cpu, memory, ports,
>>>>>>>>>         environmentVariables);
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On Sun, Sep 20, 2015 at 2:51 PM, Akila Ravihansa Perera <
>>>>>>>>> raviha...@wso2.com> wrote:
>>>>>>>>>
>>>>>>>>>> Hi all,
>>>>>>>>>>
>>>>>>>>>> I've pushed release artifacts to staging repo at [1]. Please take
>>>>>>>>>> few mins to verify the release. Also I've pushed cartridge-base, php 
>>>>>>>>>> and
>>>>>>>>>> tomcat Docker images with 4.1.3 artifacts to DockerHub.
>>>>>>>>>>
>>>>>>>>>> [1]
>>>>>>>>>> https://repository.apache.org/content/repositories/orgapachestratos-1057/
>>>>>>>>>>
>>>>>>>>>> Thanks.
>>>>>>>>>>
>>>>>>>>>> --
>>>>>>>>>> Akila Ravihansa Perera
>>>>>>>>>> WSO2 Inc.;  http://wso2.com/
>>>>>>>>>>
>>>>>>>>>> Blog: http://ravihansa3000.blogspot.com
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> --
>>>>>>>>> Imesh Gunaratne
>>>>>>>>>
>>>>>>>>> Senior Technical Lead, WSO2
>>>>>>>>> Committer & PMC Member, Apache Stratos
>>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> --
>>>>>>>> Akila Ravihansa Perera
>>>>>>>> WSO2 Inc.;  http://wso2.com/
>>>>>>>>
>>>>>>>> Blog: http://ravihansa3000.blogspot.com
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>> Imesh Gunaratne
>>>>>>>
>>>>>>> Senior Technical Lead, WSO2
>>>>>>> Committer & PMC Member, Apache Stratos
>>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>> Akila Ravihansa Perera
>>>>>> WSO2 Inc.;  http://wso2.com/
>>>>>>
>>>>>> Blog: http://ravihansa3000.blogspot.com
>>>>>>
>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> Imesh Gunaratne
>>>>>
>>>>> Senior Technical Lead, WSO2
>>>>> Committer & PMC Member, Apache Stratos
>>>>>
>>>>
>>>>
>>>>
>>>> --
>>>>
>>>> Gayan Gunarathne
>>>> Technical Lead, WSO2 Inc. (http://wso2.com)
>>>> Committer & PMC Member, Apache Stratos
>>>> email : gay...@wso2.com  | mobile : +94 775030545 <%2B94%20766819985>
>>>>
>>>>
>>>>
>>>
>>>
>>>
>>> --
>>> Akila Ravihansa Perera
>>> WSO2 Inc.;  http://wso2.com/
>>>
>>> Blog: http://ravihansa3000.blogspot.com
>>>
>>
>>
>>
>> --
>>
>> Gayan Gunarathne
>> Technical Lead, WSO2 Inc. (http://wso2.com)
>> Committer & PMC Member, Apache Stratos
>> email : gay...@wso2.com  | mobile : +94 775030545 <%2B94%20766819985>
>>
>>
>>
>
>
>
> --
> Imesh Gunaratne
>
> Senior Technical Lead, WSO2
> Committer & PMC Member, Apache Stratos
>



-- 
Akila Ravihansa Perera
WSO2 Inc.;  http://wso2.com/

Blog: http://ravihansa3000.blogspot.com

Reply via email to