Hi Osura, Shall we have a hangout on Thursday/ Friday to discuss and demonstrate the current progress of the project? Please let us know your preference.
On Wed, Jul 20, 2016 at 3:35 PM, Imesh Gunaratne <im...@wso2.com> wrote: > Hi Osura, > > On Mon, Jul 18, 2016 at 4:20 PM, Osura Rathnayake <osura...@gmail.com> > wrote: > >> Hi Mentors, >> >> This is the progress so far. >> >> *Auto scaling* >> >> >> Azure supports two paradigms of auto scaling, vertical and horizontal. >> Vertical scaling, also known as scale up and scale down, means increasing >> or decreasing virtual machine (VM) sizes in response to a workload. As I >> explained in one of my previous emails, vertical auto scaling is achieved >> through monitoring rules. We can set it to trigger an alarm when certain >> conditions are met and also as the action we can set up a runbook to scale >> up or down. I could successfully get VMs to scale up and down using this >> feature. >> > > We would use horizontal scaling in most scenarios. > > >> >> Horizontal scaling, also referred to as scale out and scale in, where the >> number of VMs is altered depending on the workload. Horizontal scaling is >> achieved through virtual machine scale sets (VMSS). One important thing >> about VMSS is that the VMs included should be of the same size and have the >> same OS image. All the VMs in the scale set are added to the load balancer, >> as a backend pool. Backend pool is a pool of VMs which share the traffic >> coming via the load balancer. We can add auto scale rules, as to when >> additional VMs should be added and removed, based on the conditions. I >> could test some auto scale rules and observed VMs getting added to the >> backend pool. But one problem is that when these VMs getting added, it’s a >> whole new VM. I’m trying to add custom made VMs which has a wso2 product >> installed and configured. >> > > Yes we would need a VM image with required WSO2 product and > pre-requisites to test this. At WSO2 we use Puppet > > for automating the installation process [1]. With Puppet we can use one > base VM image and start a VM for any WSO2 product. Puppet does the WSO2 > product installation at the VM startup. > >> >> Note: only a limited horizontal scaling features are supported in the >> azure portal yet so I’m using REST API to create and manage auto scaling >> features. >> >> >> *Centralized logging* >> >> >> I was able to get log to Log Analytics using custom logs. I have >> collected logs generated from 2 wso2 application servers. You only have to >> add respective VMs to the Log analytics and set the path where logs are >> located. This feature was under public preview, which might have been one >> reason why it didn’t work last time when I tried. >> > > Was the issue with the log file path/pattern? Did it work once you > remove that? > > [1] > https://github.com/wso2/puppet-modules/wiki/Use-WSO2-Puppet-Modules-in-puppet-master-agent-Environment > > Thanks > >> >> Thanks, >> >> On Mon, Jul 18, 2016 at 8:21 AM, Osura Rathnayake <osura...@gmail.com> >> wrote: >> >>> Hi Isuru, >>> >>> Please accept my apologies I have messed up names in my last email. I'm >>> not going to be available today due to an unavoidable circumstance so can >>> we please have the meeting on Wednesday? Extremely sorry if it made any >>> inconvenience. I will update you with a detailed email today for sure. >>> >>> Thanks in advance, >>> >>> On Fri, Jul 15, 2016 at 4:04 PM, Isuru Haththotuwa <isu...@wso2.com> >>> wrote: >>> >>>> Hi Osura, >>>> >>>> Can you send a detailed updated on the current status? Shall we have a >>>> meeting on Monday to review the progress. >>>> >>>> On Tue, Jul 12, 2016 at 2:03 PM, Osura Rathnayake <osura...@gmail.com> >>>> wrote: >>>> >>>>> Hi Imesh, >>>>> >>>>> About dynamically adding members to the load balancer, I will cross >>>>> check it with auto-scaling. I couldn't look into that from auto-scaling >>>>> end >>>>> since I couldn't test it yet. >>>>> >>>>> about monitoring, yes we can do a POC on that. >>>>> >>>>> thanks, >>>>> >>>>> On Tue, Jul 12, 2016 at 12:53 PM, Imesh Gunaratne <im...@wso2.com> >>>>> wrote: >>>>> >>>>>> On Tue, Jul 12, 2016 at 12:09 PM, Osura Rathnayake < >>>>>> osura...@gmail.com> wrote: >>>>>> >>>>>>> Hi Imesh, >>>>>>> >>>>>>> About centralized logging, I'm trying to get logs to the Log >>>>>>> Analytics using few methods supported in azure. We can either parse >>>>>>> logs in >>>>>>> to syslogs and send to the Log Analytics or create custom logs >>>>>>> specifying >>>>>>> the logs location. As you said, logs shouldn't be in .txt extension, I >>>>>>> got >>>>>>> it clarified from a azure blog. Will update you soon after I could >>>>>>> resolve >>>>>>> it. >>>>>>> >>>>>>> Right, thanks for the update! >>>>>> >>>>>> >>>>>>> No you can't dynamically add VMs to the load balancer. Backend pool, >>>>>>> where all the VMs reside, should be predefined. >>>>>>> >>>>>> >>>>>> Technically that capability should be there. Otherwise we would not >>>>>> be able to autoscale a server cluster dynamically. >>>>>> >>>>>> >>>>>> >>>>>>> you can auto-scale using scale sets(I'm still researching about it), >>>>>>> that's the equivalent of AWS auto scaling group . also you can scale up >>>>>>> or >>>>>>> down a VM if it exceeds a certain parameter like CPU usage, using >>>>>>> monitoring rules. >>>>>>> >>>>>>> *Monitoring * >>>>>>> >>>>>>> >>>>>>> Azure has a native monitoring tool which involves collecting and >>>>>>> tracking metrics, analyzing log files, defining custom metrics and >>>>>>> logging >>>>>>> generated by specific applications or workloads running in Virtual >>>>>>> Machines. Azure represents monitored data in a graphical manner using >>>>>>> charts. Monitoring also facilitates triggering alarms when certain >>>>>>> conditions are met and also it can be configured to take actions on the >>>>>>> met >>>>>>> conditions. Monitoring is done by the Diagnostic Extension and it has >>>>>>> following capabilities. >>>>>>> >>>>>>> · Collects and uploads the system performance information >>>>>>> from the Linux VM to the user's storage table, including diagnostic and >>>>>>> syslog information. >>>>>>> >>>>>>> · Enables users to customize the data metrics that will be >>>>>>> collected and uploaded. >>>>>>> >>>>>>> · Enables users to upload specified log files to a >>>>>>> designated storage table. >>>>>>> >>>>>>> Note: Azure storage tables are a non-relational, key-value-pair, >>>>>>> storage system suitable for storing massive amounts of unstructured >>>>>>> data. >>>>>>> >>>>>>> >>>>>>> We can add monitor rules so that when an alert triggers it notifies >>>>>>> the admins via email. Furthermore we can set to take automated actions. >>>>>>> Azure automate actions by running runbooks. A runbook is a set of tasks >>>>>>> that perform some automated process in Azure Automation. We can create >>>>>>> our >>>>>>> own runbooks as well. Available runbooks include, >>>>>>> >>>>>>> · Restart VM >>>>>>> >>>>>>> · Stop VM >>>>>>> >>>>>>> · Remove VM >>>>>>> >>>>>>> · Scale up VM >>>>>>> >>>>>>> · Scale down VM >>>>>>> >>>>>>> When scaling up it sets the virtual machine to the next larger size >>>>>>> within the size group and when scaling down it sets the virtual machine >>>>>>> to >>>>>>> the next smaller size within the size group. >>>>>>> >>>>>>> More about runbooks and automation [1] >>>>>>> >>>>>> >>>>>> Sounds good, will us be able to do a POC on this? >>>>>> >>>>>> >>>>>>> >>>>>>> *Auto scaling * >>>>>>> >>>>>>> >>>>>>> Auto-scaling is the process of dynamically allocating the resources >>>>>>> required by an application to match performance requirements. Virtual >>>>>>> machine scale sets are an Azure Compute resource you can use to deploy >>>>>>> and >>>>>>> manage a set of identical VMs. With all VMs configured the same, VM >>>>>>> scale >>>>>>> sets are designed to support true auto-scale no pre-provisioning of >>>>>>> VMs is >>>>>>> required – and as such makes it easier to build large-scale services >>>>>>> targeting big compute, big data, and containerized workloads [2]. >>>>>>> >>>>>>> >>>>>>> Note: I couldn’t practically do this as my azure free account lets >>>>>>> me have only 4 cores and I have used all of them on my current >>>>>>> deployment. >>>>>>> I’m getting a new azure account from one of my friends in a day so >>>>>>> hopefully I will do this on it and update you. >>>>>>> >>>>>> >>>>>> Great! Thanks! >>>>>> >>>>>> >>>>>>> >>>>>>> [1] >>>>>>> https://azure.microsoft.com/en-us/documentation/articles/automation-intro/ >>>>>>> >>>>>>> [2] >>>>>>> https://azure.microsoft.com/en-us/documentation/articles/virtual-machine-scale-sets-overview/ >>>>>>> >>>>>>> >>>>>>> thanks, >>>>>>> >>>>>>> >>>>>>> On Mon, Jul 11, 2016 at 9:54 AM, Imesh Gunaratne <im...@wso2.com> >>>>>>> wrote: >>>>>>> >>>>>>>> On Thu, Jul 7, 2016 at 7:11 PM, Osura Rathnayake < >>>>>>>> osura...@gmail.com> wrote: >>>>>>>> >>>>>>>>> Hi Mentors, >>>>>>>>> >>>>>>>>> >>>>>>>>> In addition to refining the membership scheme code, I looked into >>>>>>>>> following features of Azure. >>>>>>>>> >>>>>>>>> >>>>>>>> Good findings Osura, please find few questions inline: >>>>>>>> >>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> *Azure dynamic load balancing* >>>>>>>>> >>>>>>>>> >>>>>>>>> Azure has a native load balancer which is very easy to configure. >>>>>>>>> It’s a layer 4 (TCP, UDP) load balancer which helps to spread traffic >>>>>>>>> among >>>>>>>>> healthy virtual machines. Following are some key terms you need to >>>>>>>>> know. >>>>>>>>> >>>>>>>>> >>>>>>>> Can members be dynamically added and removed to/from a load >>>>>>>> balancer? To check this we may need to explore how autoscaling works. >>>>>>>> On >>>>>>>> AWS this is handled with autoscaling groups [3] >>>>>>>> >>>>>>>>> >>>>>>>>> *Capturing Virtual Machine Images as templates* >>>>>>>>> >>>>>>>>> >>>>>>>>> Azure provides the feature of generalizing and capturing virtual >>>>>>>>> machines so that they can be used as templates. This is very useful >>>>>>>>> and >>>>>>>>> time saving when the production environment has many instances of the >>>>>>>>> same >>>>>>>>> kind of virtual machine. When the virtual machine is being >>>>>>>>> generalized all >>>>>>>>> the data in user directories are erased so better to have wso2 product >>>>>>>>> directory not in "/home/*". More about this can be found here [2]. >>>>>>>>> >>>>>>>>> Once the virtual machine is captured, it is stored in the storage >>>>>>>>> account that is associated with the virtual machine. You can either >>>>>>>>> download this or use directly by referring to the URI when you want >>>>>>>>> to make >>>>>>>>> other virtual machines with this template. What would be awesome is >>>>>>>>> if we >>>>>>>>> can fully configure the virtual machine with a given product and make >>>>>>>>> it >>>>>>>>> available to users. >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>> Yes, this is mandatory. Otherwise we would not be able to >>>>>>>> autoscale a server cluster. >>>>>>>> >>>>>>>> I'm sorry I may have missed, how did it go with centralized >>>>>>>> logging? >>>>>>>> >>>>>>>> [3] >>>>>>>> http://docs.aws.amazon.com/autoscaling/latest/userguide/AutoScalingGroup.html >>>>>>>> >>>>>>>> Thanks >>>>>>>> >>>>>>>> >>>>>>>>> [1] >>>>>>>>> https://azure.microsoft.com/en-us/documentation/articles/load-balancer-overview/ >>>>>>>>> >>>>>>>>> [2] >>>>>>>>> https://azure.microsoft.com/en-us/documentation/articles/virtual-machines-linux-capture-image/ >>>>>>>>> >>>>>>>>> >>>>>>>>> Thanks, >>>>>>>>> >>>>>>>>> On Wed, Jul 6, 2016 at 12:18 PM, Osura Rathnayake < >>>>>>>>> osura...@gmail.com> wrote: >>>>>>>>> >>>>>>>>>> Hi Akila, >>>>>>>>>> >>>>>>>>>> Please refer to the screenshots that I have attached. When I >>>>>>>>>> updated localMemberPort to 4200, I can see it being reflected in >>>>>>>>>> logs when >>>>>>>>>> members are joining. So should I still make modifications in the >>>>>>>>>> code? >>>>>>>>>> >>>>>>>>>> .gitignore was added. >>>>>>>>>> >>>>>>>>>> okay I will write test cases in testNG and update >>>>>>>>>> >>>>>>>>>> thanks, >>>>>>>>>> >>>>>>>>>> On Wed, Jul 6, 2016 at 9:06 AM, Akila Ravihansa Perera < >>>>>>>>>> raviha...@wso2.com> wrote: >>>>>>>>>> >>>>>>>>>>> Hi, >>>>>>>>>>> >>>>>>>>>>> On Tue, Jul 5, 2016 at 11:38 AM, Osura Rathnayake < >>>>>>>>>>> osura...@gmail.com> wrote: >>>>>>>>>>> >>>>>>>>>>>> Hi Akila, >>>>>>>>>>>> >>>>>>>>>>>> Please check the modified code. It now takes the value which is >>>>>>>>>>>> specified as localMemberPort in axis2.xml. >>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> I still don't see any change to the logic of how member address >>>>>>>>>>> is calculated. Can you double check? >>>>>>>>>>> >>>>>>>>>>> Please make sure "target/**" directories are ignored from >>>>>>>>>>> .gitignore. These shouldn't be in the repo [1]. You might also need >>>>>>>>>>> to >>>>>>>>>>> ignore any IDE specific files. Have a look at .gitignore in >>>>>>>>>>> Kubernetes >>>>>>>>>>> artifacts [2]. >>>>>>>>>>> >>>>>>>>>>> I see that you have committed some test cases based on JUnit. >>>>>>>>>>> Please note that as a platform we are moving to testng framework so >>>>>>>>>>> better >>>>>>>>>>> to use that. >>>>>>>>>>> @Imesh, Isuru: Please correct me if I'm wrong. >>>>>>>>>>> >>>>>>>>>>> Shall we get a repo created under wso2-incubator for this? >>>>>>>>>>> >>>>>>>>>>> [1] >>>>>>>>>>> https://github.com/osuran/azure-membership-scheme/tree/master/target >>>>>>>>>>> [2] >>>>>>>>>>> https://github.com/wso2/kubernetes-artifacts/blob/master/.gitignore >>>>>>>>>>> >>>>>>>>>>> Thanks. >>>>>>>>>>> -- >>>>>>>>>>> Akila Ravihansa Perera >>>>>>>>>>> WSO2 Inc.; http://wso2.com/ >>>>>>>>>>> >>>>>>>>>>> Blog: http://ravihansa3000.blogspot.com >>>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> -- >>>>>>>>>> Regards, >>>>>>>>>> Osura Rathnayake >>>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> -- >>>>>>>>> Regards, >>>>>>>>> Osura Rathnayake >>>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> -- >>>>>>>> *Imesh Gunaratne* >>>>>>>> Software Architect >>>>>>>> WSO2 Inc: http://wso2.com >>>>>>>> T: +94 11 214 5345 M: +94 77 374 2057 >>>>>>>> W: https://medium.com/@imesh TW: @imesh >>>>>>>> >>>>>>>> >>>>>>> >>>>>>> >>>>>>> -- >>>>>>> Regards, >>>>>>> Osura Rathnayake >>>>>>> >>>>>> >>>>>> >>>>>> >>>>>> -- >>>>>> *Imesh Gunaratne* >>>>>> Software Architect >>>>>> WSO2 Inc: http://wso2.com >>>>>> T: +94 11 214 5345 M: +94 77 374 2057 >>>>>> W: https://medium.com/@imesh TW: @imesh >>>>>> >>>>>> >>>>> >>>>> >>>>> -- >>>>> Regards, >>>>> Osura Rathnayake >>>>> >>>> >>>> >>>> >>>> -- >>>> Thanks and Regards, >>>> >>>> Isuru H. >>>> +94 716 358 048* <http://wso2.com/>* >>>> >>>> >>>> >>> >>> >>> -- >>> Regards, >>> Osura Rathnayake >>> >> >> >> >> -- >> Regards, >> Osura Rathnayake >> > > > > -- > *Imesh Gunaratne* > Software Architect > WSO2 Inc: http://wso2.com > T: +94 11 214 5345 M: +94 77 374 2057 > W: https://medium.com/@imesh TW: @imesh > > -- Thanks and Regards, Isuru H. +94 716 358 048* <http://wso2.com/>*
_______________________________________________ Dev mailing list Dev@wso2.org http://wso2.org/cgi-bin/mailman/listinfo/dev
