Hi Mentors, I will try using Puppet. It wasn't a problem with log path/pattern, in fact I used the same log path that I used last time. I believe it was a bug from Azure side, please check the attached screenshots. Shall we please have the meeting on Friday?
thank you, On Wed, Jul 20, 2016 at 6:50 PM, Isuru Haththotuwa <isu...@wso2.com> wrote: > Hi Osura, > > Shall we have a hangout on Thursday/ Friday to discuss and demonstrate the > current progress of the project? Please let us know your preference. > > On Wed, Jul 20, 2016 at 3:35 PM, Imesh Gunaratne <im...@wso2.com> wrote: > >> Hi Osura, >> >> On Mon, Jul 18, 2016 at 4:20 PM, Osura Rathnayake <osura...@gmail.com> >> wrote: >> >>> Hi Mentors, >>> >>> This is the progress so far. >>> >>> *Auto scaling* >>> >>> >>> Azure supports two paradigms of auto scaling, vertical and horizontal. >>> Vertical scaling, also known as scale up and scale down, means increasing >>> or decreasing virtual machine (VM) sizes in response to a workload. As I >>> explained in one of my previous emails, vertical auto scaling is achieved >>> through monitoring rules. We can set it to trigger an alarm when certain >>> conditions are met and also as the action we can set up a runbook to scale >>> up or down. I could successfully get VMs to scale up and down using this >>> feature. >>> >> >> We would use horizontal scaling in most scenarios. >> >> >>> >>> Horizontal scaling, also referred to as scale out and scale in, where >>> the number of VMs is altered depending on the workload. Horizontal scaling >>> is achieved through virtual machine scale sets (VMSS). One important thing >>> about VMSS is that the VMs included should be of the same size and have the >>> same OS image. All the VMs in the scale set are added to the load balancer, >>> as a backend pool. Backend pool is a pool of VMs which share the traffic >>> coming via the load balancer. We can add auto scale rules, as to when >>> additional VMs should be added and removed, based on the conditions. I >>> could test some auto scale rules and observed VMs getting added to the >>> backend pool. But one problem is that when these VMs getting added, it’s a >>> whole new VM. I’m trying to add custom made VMs which has a wso2 product >>> installed and configured. >>> >> >> Yes we would need a VM image with required WSO2 product and >> pre-requisites to test this. At WSO2 we use Puppet >> >> for automating the installation process [1]. With Puppet we can use one >> base VM image and start a VM for any WSO2 product. Puppet does the WSO2 >> product installation at the VM startup. >> >>> >>> Note: only a limited horizontal scaling features are supported in the >>> azure portal yet so I’m using REST API to create and manage auto scaling >>> features. >>> >>> >>> *Centralized logging* >>> >>> >>> I was able to get log to Log Analytics using custom logs. I have >>> collected logs generated from 2 wso2 application servers. You only have to >>> add respective VMs to the Log analytics and set the path where logs are >>> located. This feature was under public preview, which might have been one >>> reason why it didn’t work last time when I tried. >>> >> >> Was the issue with the log file path/pattern? Did it work once you >> remove that? >> >> [1] >> https://github.com/wso2/puppet-modules/wiki/Use-WSO2-Puppet-Modules-in-puppet-master-agent-Environment >> >> Thanks >> >>> >>> Thanks, >>> >>> On Mon, Jul 18, 2016 at 8:21 AM, Osura Rathnayake <osura...@gmail.com> >>> wrote: >>> >>>> Hi Isuru, >>>> >>>> Please accept my apologies I have messed up names in my last email. I'm >>>> not going to be available today due to an unavoidable circumstance so can >>>> we please have the meeting on Wednesday? Extremely sorry if it made any >>>> inconvenience. I will update you with a detailed email today for sure. >>>> >>>> Thanks in advance, >>>> >>>> On Fri, Jul 15, 2016 at 4:04 PM, Isuru Haththotuwa <isu...@wso2.com> >>>> wrote: >>>> >>>>> Hi Osura, >>>>> >>>>> Can you send a detailed updated on the current status? Shall we have a >>>>> meeting on Monday to review the progress. >>>>> >>>>> On Tue, Jul 12, 2016 at 2:03 PM, Osura Rathnayake <osura...@gmail.com> >>>>> wrote: >>>>> >>>>>> Hi Imesh, >>>>>> >>>>>> About dynamically adding members to the load balancer, I will cross >>>>>> check it with auto-scaling. I couldn't look into that from auto-scaling >>>>>> end >>>>>> since I couldn't test it yet. >>>>>> >>>>>> about monitoring, yes we can do a POC on that. >>>>>> >>>>>> thanks, >>>>>> >>>>>> On Tue, Jul 12, 2016 at 12:53 PM, Imesh Gunaratne <im...@wso2.com> >>>>>> wrote: >>>>>> >>>>>>> On Tue, Jul 12, 2016 at 12:09 PM, Osura Rathnayake < >>>>>>> osura...@gmail.com> wrote: >>>>>>> >>>>>>>> Hi Imesh, >>>>>>>> >>>>>>>> About centralized logging, I'm trying to get logs to the Log >>>>>>>> Analytics using few methods supported in azure. We can either parse >>>>>>>> logs in >>>>>>>> to syslogs and send to the Log Analytics or create custom logs >>>>>>>> specifying >>>>>>>> the logs location. As you said, logs shouldn't be in .txt extension, I >>>>>>>> got >>>>>>>> it clarified from a azure blog. Will update you soon after I could >>>>>>>> resolve >>>>>>>> it. >>>>>>>> >>>>>>>> Right, thanks for the update! >>>>>>> >>>>>>> >>>>>>>> No you can't dynamically add VMs to the load balancer. Backend >>>>>>>> pool, where all the VMs reside, should be predefined. >>>>>>>> >>>>>>> >>>>>>> Technically that capability should be there. Otherwise we would not >>>>>>> be able to autoscale a server cluster dynamically. >>>>>>> >>>>>>> >>>>>>> >>>>>>>> you can auto-scale using scale sets(I'm still researching about >>>>>>>> it), that's the equivalent of AWS auto scaling group . also you can >>>>>>>> scale >>>>>>>> up or down a VM if it exceeds a certain parameter like CPU usage, using >>>>>>>> monitoring rules. >>>>>>>> >>>>>>>> *Monitoring * >>>>>>>> >>>>>>>> >>>>>>>> Azure has a native monitoring tool which involves collecting and >>>>>>>> tracking metrics, analyzing log files, defining custom metrics and >>>>>>>> logging >>>>>>>> generated by specific applications or workloads running in Virtual >>>>>>>> Machines. Azure represents monitored data in a graphical manner using >>>>>>>> charts. Monitoring also facilitates triggering alarms when certain >>>>>>>> conditions are met and also it can be configured to take actions on >>>>>>>> the met >>>>>>>> conditions. Monitoring is done by the Diagnostic Extension and it has >>>>>>>> following capabilities. >>>>>>>> >>>>>>>> · Collects and uploads the system performance information >>>>>>>> from the Linux VM to the user's storage table, including diagnostic and >>>>>>>> syslog information. >>>>>>>> >>>>>>>> · Enables users to customize the data metrics that will be >>>>>>>> collected and uploaded. >>>>>>>> >>>>>>>> · Enables users to upload specified log files to a >>>>>>>> designated storage table. >>>>>>>> >>>>>>>> Note: Azure storage tables are a non-relational, key-value-pair, >>>>>>>> storage system suitable for storing massive amounts of unstructured >>>>>>>> data. >>>>>>>> >>>>>>>> >>>>>>>> We can add monitor rules so that when an alert triggers it notifies >>>>>>>> the admins via email. Furthermore we can set to take automated actions. >>>>>>>> Azure automate actions by running runbooks. A runbook is a set of tasks >>>>>>>> that perform some automated process in Azure Automation. We can create >>>>>>>> our >>>>>>>> own runbooks as well. Available runbooks include, >>>>>>>> >>>>>>>> · Restart VM >>>>>>>> >>>>>>>> · Stop VM >>>>>>>> >>>>>>>> · Remove VM >>>>>>>> >>>>>>>> · Scale up VM >>>>>>>> >>>>>>>> · Scale down VM >>>>>>>> >>>>>>>> When scaling up it sets the virtual machine to the next larger size >>>>>>>> within the size group and when scaling down it sets the virtual >>>>>>>> machine to >>>>>>>> the next smaller size within the size group. >>>>>>>> >>>>>>>> More about runbooks and automation [1] >>>>>>>> >>>>>>> >>>>>>> Sounds good, will us be able to do a POC on this? >>>>>>> >>>>>>> >>>>>>>> >>>>>>>> *Auto scaling * >>>>>>>> >>>>>>>> >>>>>>>> Auto-scaling is the process of dynamically allocating the resources >>>>>>>> required by an application to match performance requirements. Virtual >>>>>>>> machine scale sets are an Azure Compute resource you can use to deploy >>>>>>>> and >>>>>>>> manage a set of identical VMs. With all VMs configured the same, VM >>>>>>>> scale >>>>>>>> sets are designed to support true auto-scale no pre-provisioning of >>>>>>>> VMs is >>>>>>>> required – and as such makes it easier to build large-scale services >>>>>>>> targeting big compute, big data, and containerized workloads [2]. >>>>>>>> >>>>>>>> >>>>>>>> Note: I couldn’t practically do this as my azure free account lets >>>>>>>> me have only 4 cores and I have used all of them on my current >>>>>>>> deployment. >>>>>>>> I’m getting a new azure account from one of my friends in a day so >>>>>>>> hopefully I will do this on it and update you. >>>>>>>> >>>>>>> >>>>>>> Great! Thanks! >>>>>>> >>>>>>> >>>>>>>> >>>>>>>> [1] >>>>>>>> https://azure.microsoft.com/en-us/documentation/articles/automation-intro/ >>>>>>>> >>>>>>>> [2] >>>>>>>> https://azure.microsoft.com/en-us/documentation/articles/virtual-machine-scale-sets-overview/ >>>>>>>> >>>>>>>> >>>>>>>> thanks, >>>>>>>> >>>>>>>> >>>>>>>> On Mon, Jul 11, 2016 at 9:54 AM, Imesh Gunaratne <im...@wso2.com> >>>>>>>> wrote: >>>>>>>> >>>>>>>>> On Thu, Jul 7, 2016 at 7:11 PM, Osura Rathnayake < >>>>>>>>> osura...@gmail.com> wrote: >>>>>>>>> >>>>>>>>>> Hi Mentors, >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> In addition to refining the membership scheme code, I looked into >>>>>>>>>> following features of Azure. >>>>>>>>>> >>>>>>>>>> >>>>>>>>> Good findings Osura, please find few questions inline: >>>>>>>>> >>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> *Azure dynamic load balancing* >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> Azure has a native load balancer which is very easy to configure. >>>>>>>>>> It’s a layer 4 (TCP, UDP) load balancer which helps to spread >>>>>>>>>> traffic among >>>>>>>>>> healthy virtual machines. Following are some key terms you need to >>>>>>>>>> know. >>>>>>>>>> >>>>>>>>>> >>>>>>>>> Can members be dynamically added and removed to/from a load >>>>>>>>> balancer? To check this we may need to explore how autoscaling works. >>>>>>>>> On >>>>>>>>> AWS this is handled with autoscaling groups [3] >>>>>>>>> >>>>>>>>>> >>>>>>>>>> *Capturing Virtual Machine Images as templates* >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> Azure provides the feature of generalizing and capturing virtual >>>>>>>>>> machines so that they can be used as templates. This is very useful >>>>>>>>>> and >>>>>>>>>> time saving when the production environment has many instances of >>>>>>>>>> the same >>>>>>>>>> kind of virtual machine. When the virtual machine is being >>>>>>>>>> generalized all >>>>>>>>>> the data in user directories are erased so better to have wso2 >>>>>>>>>> product >>>>>>>>>> directory not in "/home/*". More about this can be found here [2]. >>>>>>>>>> >>>>>>>>>> Once the virtual machine is captured, it is stored in the storage >>>>>>>>>> account that is associated with the virtual machine. You can either >>>>>>>>>> download this or use directly by referring to the URI when you want >>>>>>>>>> to make >>>>>>>>>> other virtual machines with this template. What would be awesome is >>>>>>>>>> if we >>>>>>>>>> can fully configure the virtual machine with a given product and >>>>>>>>>> make it >>>>>>>>>> available to users. >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>> Yes, this is mandatory. Otherwise we would not be able to >>>>>>>>> autoscale a server cluster. >>>>>>>>> >>>>>>>>> I'm sorry I may have missed, how did it go with centralized >>>>>>>>> logging? >>>>>>>>> >>>>>>>>> [3] >>>>>>>>> http://docs.aws.amazon.com/autoscaling/latest/userguide/AutoScalingGroup.html >>>>>>>>> >>>>>>>>> Thanks >>>>>>>>> >>>>>>>>> >>>>>>>>>> [1] >>>>>>>>>> https://azure.microsoft.com/en-us/documentation/articles/load-balancer-overview/ >>>>>>>>>> >>>>>>>>>> [2] >>>>>>>>>> https://azure.microsoft.com/en-us/documentation/articles/virtual-machines-linux-capture-image/ >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> Thanks, >>>>>>>>>> >>>>>>>>>> On Wed, Jul 6, 2016 at 12:18 PM, Osura Rathnayake < >>>>>>>>>> osura...@gmail.com> wrote: >>>>>>>>>> >>>>>>>>>>> Hi Akila, >>>>>>>>>>> >>>>>>>>>>> Please refer to the screenshots that I have attached. When I >>>>>>>>>>> updated localMemberPort to 4200, I can see it being reflected in >>>>>>>>>>> logs when >>>>>>>>>>> members are joining. So should I still make modifications in the >>>>>>>>>>> code? >>>>>>>>>>> >>>>>>>>>>> .gitignore was added. >>>>>>>>>>> >>>>>>>>>>> okay I will write test cases in testNG and update >>>>>>>>>>> >>>>>>>>>>> thanks, >>>>>>>>>>> >>>>>>>>>>> On Wed, Jul 6, 2016 at 9:06 AM, Akila Ravihansa Perera < >>>>>>>>>>> raviha...@wso2.com> wrote: >>>>>>>>>>> >>>>>>>>>>>> Hi, >>>>>>>>>>>> >>>>>>>>>>>> On Tue, Jul 5, 2016 at 11:38 AM, Osura Rathnayake < >>>>>>>>>>>> osura...@gmail.com> wrote: >>>>>>>>>>>> >>>>>>>>>>>>> Hi Akila, >>>>>>>>>>>>> >>>>>>>>>>>>> Please check the modified code. It now takes the value which >>>>>>>>>>>>> is specified as localMemberPort in axis2.xml. >>>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> I still don't see any change to the logic of how member address >>>>>>>>>>>> is calculated. Can you double check? >>>>>>>>>>>> >>>>>>>>>>>> Please make sure "target/**" directories are ignored from >>>>>>>>>>>> .gitignore. These shouldn't be in the repo [1]. You might also >>>>>>>>>>>> need to >>>>>>>>>>>> ignore any IDE specific files. Have a look at .gitignore in >>>>>>>>>>>> Kubernetes >>>>>>>>>>>> artifacts [2]. >>>>>>>>>>>> >>>>>>>>>>>> I see that you have committed some test cases based on JUnit. >>>>>>>>>>>> Please note that as a platform we are moving to testng framework >>>>>>>>>>>> so better >>>>>>>>>>>> to use that. >>>>>>>>>>>> @Imesh, Isuru: Please correct me if I'm wrong. >>>>>>>>>>>> >>>>>>>>>>>> Shall we get a repo created under wso2-incubator for this? >>>>>>>>>>>> >>>>>>>>>>>> [1] >>>>>>>>>>>> https://github.com/osuran/azure-membership-scheme/tree/master/target >>>>>>>>>>>> [2] >>>>>>>>>>>> https://github.com/wso2/kubernetes-artifacts/blob/master/.gitignore >>>>>>>>>>>> >>>>>>>>>>>> Thanks. >>>>>>>>>>>> -- >>>>>>>>>>>> Akila Ravihansa Perera >>>>>>>>>>>> WSO2 Inc.; http://wso2.com/ >>>>>>>>>>>> >>>>>>>>>>>> Blog: http://ravihansa3000.blogspot.com >>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> -- >>>>>>>>>>> Regards, >>>>>>>>>>> Osura Rathnayake >>>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> -- >>>>>>>>>> Regards, >>>>>>>>>> Osura Rathnayake >>>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> -- >>>>>>>>> *Imesh Gunaratne* >>>>>>>>> Software Architect >>>>>>>>> WSO2 Inc: http://wso2.com >>>>>>>>> T: +94 11 214 5345 M: +94 77 374 2057 >>>>>>>>> W: https://medium.com/@imesh TW: @imesh >>>>>>>>> >>>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> -- >>>>>>>> Regards, >>>>>>>> Osura Rathnayake >>>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> -- >>>>>>> *Imesh Gunaratne* >>>>>>> Software Architect >>>>>>> WSO2 Inc: http://wso2.com >>>>>>> T: +94 11 214 5345 M: +94 77 374 2057 >>>>>>> W: https://medium.com/@imesh TW: @imesh >>>>>>> >>>>>>> >>>>>> >>>>>> >>>>>> -- >>>>>> Regards, >>>>>> Osura Rathnayake >>>>>> >>>>> >>>>> >>>>> >>>>> -- >>>>> Thanks and Regards, >>>>> >>>>> Isuru H. >>>>> +94 716 358 048* <http://wso2.com/>* >>>>> >>>>> >>>>> >>>> >>>> >>>> -- >>>> Regards, >>>> Osura Rathnayake >>>> >>> >>> >>> >>> -- >>> Regards, >>> Osura Rathnayake >>> >> >> >> >> -- >> *Imesh Gunaratne* >> Software Architect >> WSO2 Inc: http://wso2.com >> T: +94 11 214 5345 M: +94 77 374 2057 >> W: https://medium.com/@imesh TW: @imesh >> >> > > > -- > Thanks and Regards, > > Isuru H. > +94 716 358 048* <http://wso2.com/>* > > > -- Regards, Osura Rathnayake
_______________________________________________ Dev mailing list Dev@wso2.org http://wso2.org/cgi-bin/mailman/listinfo/dev
