Did you check on network QoS on VR as well if there is any .!
Vivek Kumar Sr. Manager - Cloud & DevOps TechOps | Indiqus Technologies vivek.ku...@indiqus.com <mailto:vivek.ku...@indiqus.com> www.indiqus.com <https://www.indiqus.com/> > On 10-Jul-2023, at 9:34 PM, Granwille Strauss <granwi...@namhost.com> wrote: > > Hi Levin > > I skipped all the VM testing and tested iperf straight from the KVM host to > determine a base line to the remote USA VM machine: > > - KVM to Remote USA VM: 113 Mbits/sec > - USA VM to KVM Host: 35.9 Mbits/sec > > I then ran the same test again, this this time the remote host was in a DC > that's close to our DC in the same country that we use: > > - KVM to remote host: 409 Mbits/sec > - Remote host to KVM: 477 Mbits/sec > > So do you think its safe to conclude that somewhere traffic from the USA VM > to the local KVM host gets throttled? Based on the results above, the > throttling doesn't seems to be in from ISPs inside our country. > > So yeah, somewhere some ISP is throttling during the USA routes. > > On 7/10/23 15:56, Levin Ng wrote: >> Hi Groete, >> >> I’m not sure what is your network setting in ACS, but test between two >> public IP with ~500Mbps sound like u are saturated by in/out bound traffics >> in the single network path, can you do a test from outside ACS to your VM >> using an same public network segment IP, it will avoid network routing and >> confusion., what is your ACS network driver using? If vxlan, better check >> with network switch multicast performance. >> >> The remote performances clearly shown the ISP put some limit on the line, >> you have to check with them. Unless your line is end-to-end, Metro-Ethernet >> etc… otherwise it is not always have guarantee throughput. >> >> >> On the disk performance, should you share your fio test command and result >> beforehand. I’m assuming you are doing something like >> >> fio -filename=./testfile.bin -direct=1 -iodepth 8 -thread -rw=randrw >> -rwmixread=50 -ioengine=psync -bs=4k -size=1000M -numjobs=30 -runtime=600 >> -group_reporting -name=mytest >> >> >> Regards, >> Levin >> On 10 Jul 2023 at 11:33 +0100, Granwille Strauss <granwi...@namhost.com> >> <mailto:granwi...@namhost.com>, wrote: >>> Hi Guys >>> Thank you, I have been running more tests now with the feedback you guys >>> gave. Firstly, I want to break this up into two sections: >>> 1. Network: >>> - So I have been running iperf tests between my VMs on their public >>> network, and my iperf tests gives me speeds of ~500 Mbps, keep in mind this >>> in between two local VMs on the same KVM but on public network. >>> - I then run iperf tests in and out from my local VMs to remote servers, >>> this is where it does funny things. From the remote VM in USA, I run an >>> iperf test to my local VM, the speeds show ~50 Mbps. And if I run a test >>> from my local VM to a remote USA VM the same ~50 Mbps speeds are >>> accomplished. I ran my iperf tests with 1 GB and 2GB flags and the results >>> remain constant >>> - During all these test I kept an eye on my VR resources, which use default >>> service offerings, it never spiked or reach thresholds. >>> Is it safe to assume that because of the MASSIVE distance between the >>> remote VM and my local VMs, the speed dropping to ~50 Mbps is normal? Keep >>> in mind the remote VM has 1 Gbps line too and this VM is managed by a big >>> ISP provider in the USA. To me its quite a massive drop from 1000 Mbps to >>> 50 Mbps, this kinda does not make sense to me. I would understand at least >>> 150 Mbps. >>> 2. Disk Write Speed: >>> - It seems the only changes that can be made is to implement disk cache >>> options. And so far I see write-back seems to be common practise for most >>> cloud providers, given that they have the necessary power redundancy and VM >>> backup images in place. >>> >>> But for now, other than write cache types are there anything else that can >>> be done to improve disk writing speeds. I checked RedHat guides on >>> optimising VMs and I seem to have most in place, but write speeds remain at >>> ~50 Mbps. >>> On 7/10/23 06:25, Jithin Raju wrote: >>>> Hi Groete, >>>> >>>> The VM virtual NIC network throttling is picked up from its compute >>>> offering. You may need to create a new compute offering and change the >>>> VM’s compute offering. If it is not specified in the compute offering, >>>> means it is taking the values from the global settings: >>>> vm.network.throttling.rate . >>>> >>>> >>>> >>>> -Jithin >>>> >>>> From: Levin Ng <levindec...@gmail.com> <mailto:levindec...@gmail.com> >>>> Date: Sunday, 9 July 2023 at 5:00 AM >>>> To: users@cloudstack.apache.org <mailto:users@cloudstack.apache.org> >>>> <users@cloudstack.apache.org> <mailto:users@cloudstack.apache.org>, >>>> Granwille Strauss <granwi...@namhost.com> <mailto:granwi...@namhost.com> >>>> Cc: vivek.ku...@indiqus.com <mailto:vivek.ku...@indiqus.com> >>>> <vivek.ku...@indiqus.com> <mailto:vivek.ku...@indiqus.com>, Nux >>>> <n...@li.nux.ro> <mailto:n...@li.nux.ro> >>>> Subject: Re: Write Speeds >>>> Dear Groete, >>>> >>>> https://github.com/shapeblue/cloudstack/blob/965856057d5147f12b86abe5c9c205cdc5e44615/plugins/hypervisors/kvm/src/main/java/com/cloud/hypervisor/kvm/resource/DirectVifDriver.java >>>> >>>> https://libvirt.org/formatnetwork.html#quality-of-service >>>> >>>> This is in kilobytes/second, u have to divide by 8 >>>> >>>> 1Gbps / 8bit = 128MB = 128000KBps >>>> >>>> You can verify by iperf test, and yes, u need to ensure both VR and VM >>>> match bandwidth settings to get a consistent result, something u also need >>>> to pay attention on VR resource, default system router resource offering >>>> is quite limited, the network speed may throttled if VR running are out of >>>> CPU resource. >>>> >>>> Regards, >>>> Levin >>>> >>>> >>>> >>>> On 8 Jul 2023 at 09:02 +0100, Granwille Strauss <granwi...@namhost.com> >>>> <mailto:granwi...@namhost.com>, wrote: >>>>> Hi Levin >>>>> Thank you very much, I do appreciate your feedback and time replying back >>>>> to me. I believe I have picked up on something. My VMs XML dump ALL show >>>>> the following: >>>>>> <interface type='bridge'> <mac address='xxxxxxx'/> <source >>>>>> bridge='cloudbr0'/> <bandwidth> <inbound average='128000' >>>>>> peak='128000'/> <outbound average='128000' peak='128000'/> </bandwidth> >>>>>> <target dev='vnet219'/> <model type='virtio'/> <link state='up'/> <alias >>>>>> name='net0'/> <rom bar='off' file=''/> <address type='pci' >>>>>> domain='0x0000' bus='0x00' slot='0x03' function='0x0'/> </interface> >>>>> Specifically: >>>>> <bandwidth> <inbound average='128000' peak='128000'/> <outbound >>>>> average='128000' peak='128000'/> </bandwidth> >>>>> All my VM has this in place, a 128 Mbps limit even the VR, which makes no >>>>> sense. I found this thread posted 10 years ago and Nux says that this >>>>> value is affected by the service offerings: >>>>> https://users.cloudstack.apache.narkive.com/T6Gx7BoV/cloudstack-network-limitation >>>>> But all my service offerings are set to 1000 Mbps. See attached >>>>> screenshots. The 4.18 documentation also confirms if the values are null, >>>>> which most default service offerings are, it takes the values set in >>>>> global settings network.throttling.rate and vm.network.throttling.rate, >>>>> which I also have set as 1000 as you can see in the screenshots. >>>>> >>>>> I then found this: >>>>> https://cwiki.apache.org/confluence/display/CLOUDSTACK/Network+throttling+in+CloudStack >>>>> with no KVM details, seems this part is missing to tell me how KVM >>>>> throttling is applied. So as DuJun said 10 years ago, I feel confused >>>>> about how Cloudstack limit the network rate for guest. And yes, I have >>>>> stopped my VMs and rebooted MANY times it doesn't update the XML at all. >>>>> Please also take into account documentation states that in shared >>>>> networking there's supposed to be no limits on incoming traffic >>>>> (ingress), as far as I understand it. >>>>> On 7/7/23 23:35, Levin Ng wrote: >>>>>> Hi Groete, >>>>>> >>>>>> >>>>>> IMO, You should bypass any ACS provisioning to troubleshoot the >>>>>> performance case first, which allow you get more idea on the hardware + >>>>>> kvm performance with minimal influent, then you can compare the libvirt >>>>>> xml different between plain KVM and ACS. That help you sort out the >>>>>> different where it come from, you will see QoS bandwidth setting in the >>>>>> VM xml if you do. >>>>>> >>>>>> We are trying to tell you, when you diagnose the throughput problem, you >>>>>> should first identify the bottleneck where it come first. Iperf is a >>>>>> tools that you can test the line speed end to end into your VM, if the >>>>>> result in 1Gbps network are near 800+ Mbps, you can focus on the VM >>>>>> performance or the copy protocol you are using, try different protocol, >>>>>> ssh/rsync/ftp/nfs, see any different. >>>>>> >>>>>> You are already test the write-back caching which will improve disk I/O >>>>>> performance, it is another story you need to deep dive the pro and cons >>>>>> on the write cache, there are risk to corrupt the VM filesystem in some >>>>>> case, this is what u need to learn about each cache mode. >>>>>> >>>>>> VM Guest performance are involved by many factor, you cannot expect VM >>>>>> perform nearly the bare metal does. There are long journey to do such >>>>>> optimization, take time and improve it gradually. There are lot of kvm >>>>>> tuning guide you can reference and prove it on your hardware. Read >>>>>> thoughtfully on each tuning that may bring improvement and also >>>>>> introduce risk factor. >>>>>> >>>>>> >>>>>> Regards, >>>>>> Levin >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> On 7 Jul 2023 at 21:24 +0100, Granwille Strauss <granwi...@namhost.com> >>>>>> <mailto:granwi...@namhost.com>, wrote: >>>>>>> Sorry that I have to ask, can you perhaps be a bit more specific, >>>>>>> please. The only QOS settings I see in service offering are "None", >>>>>>> "Hypervisor" and "Storage", which doesn't really seem network related. >>>>>>> Or am I missing the point? Take note that I use the default offerings >>>>>>> for the VR and VMs but with slight tweaks such as setting local storage >>>>>>> etc and only increased the Network rate from 200 Mbps to 1000 Mbps. >>>>>>> >>>>>>> So can you kindly explain by what QOS settings you guys are referring >>>>>>> to, please? >>>>>>> PS, the Write-back disk caching seems to give the VM a slight increase, >>>>>>> I now see writes at 190 Mbps from ~70 Mbps. >>>>>>> On 7/7/23 21:11, Vivek Kumar wrote: >>>>>>>> Hello, >>>>>>>> >>>>>>>> IPerf will simply tell you the bandwidth and the open pipe between 2 >>>>>>>> VMs, so I don’t think that it’s depends on disk performance, it’s >>>>>>>> better to check the network QoS at every layer, VR and VM. >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> Vivek Kumar >>>>>>>> Sr. Manager - Cloud & DevOps >>>>>>>> TechOps | Indiqus Technologies >>>>>>>> >>>>>>>> vivek.ku...@indiqus.com <mailto:vivek.ku...@indiqus.com> >>>>>>>> <mailto:vivek.ku...@indiqus.com> <mailto:vivek.ku...@indiqus.com> >>>>>>>> www.indiqus.com <http://www.indiqus.com/><http://www.indiqus.com> >>>>>>>> <http://www.indiqus.com/> <https://www.indiqus.com/> >>>>>>>> <https://www.indiqus.com/> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>> >>>>> On 07-Jul-2023, at 9:44 PM, Granwille Strauss >>>>> <granwi...@namhost.com.INVALID> <mailto:granwi...@namhost.com.INVALID> >>>>> wrote: >>>>>>>>> Hi Levin >>>>>>>>> >>>>>>>>> Thank you, I am aware of network offering, the first thing I did was >>>>>>>>> make sure it was set to accommodate the KVM's entire 1 Gbps uplink. >>>>>>>>> But now that I think if it iperf test prevousily were always stuck on >>>>>>>>> 50 Mbps, but this is because of the write speeds on the disk at least >>>>>>>>> that's what I believe causes the network bottle neck. I will >>>>>>>>> double-check this again. >>>>>>>>> >>>>>>>>> But there is some sort of limit on the VM disk in place. FIO tests >>>>>>>>> show that write speeds are in the range of 50 - 90 MB/s on the VM, >>>>>>>>> while fio test confirms on the KVM its over 400 MB/s. >>>>>>>>> >>>>>>>>> On 7/7/23 18:08, Levin Ng wrote: >>>>>>>>>> Hi Groete, >>>>>>>>>> >>>>>>>>>> Forgot to mention, when you are talking about file copies between >>>>>>>>>> remote server, you need to aware there are network QoS option in the >>>>>>>>>> offering, make sure the limits correctness. Do iperf test prove that >>>>>>>>>> too, test between server and via virtual router. Hope you can >>>>>>>>>> narrow down the problem soon. >>>>>>>>>> >>>>>>>>>> Regards, >>>>>>>>>> Levin >>>>>>>>>> >>>>>>>>>> On 7 Jul 2023 at 16:40 +0100, Granwille Strauss >>>>>>>>>> <granwi...@namhost.com> <mailto:granwi...@namhost.com> >>>>>>>>>> <mailto:granwi...@namhost.com> <mailto:granwi...@namhost.com>, wrote: >>>>>>>>>>> Hi Levin >>>>>>>>>>> Thank you, yes I leave IOPs empty. And the KVM host has SSDs in a >>>>>>>>>>> hardware RAID 5 configuration, of which I am using local storage >>>>>>>>>>> pool, yes. I will run fio test and also playing around with the >>>>>>>>>>> controller cache settings to see what happens and provide feedback >>>>>>>>>>> on this soon. >>>>>>>>>>> On 7/7/23 17:23, Levin Ng wrote: >>>>>>>>>>>> HI Groete, >>>>>>>>>>>> >>>>>>>>>>>> Should you run a fio test on the VM and the KVM host to get a >>>>>>>>>>>> baseline first. SSD are tricky device, when it fill up the cache >>>>>>>>>>>> or nearly full, the performance will drop significantly, >>>>>>>>>>>> especially consumer grade SSD. There are option to limit IOPs in >>>>>>>>>>>> ACS offering setting, I believe you leave it empty, so it is no >>>>>>>>>>>> limit. When you talking about KVM uses SSDs, I think you are using >>>>>>>>>>>> Local Disk Pool right? If you have RAID controller underlying, try >>>>>>>>>>>> toggle the controller cache, SSD may perform vary on different >>>>>>>>>>>> disk controller cache setting. >>>>>>>>>>>> >>>>>>>>>>>> Controller type scsi, or virtio performance are similar, no need >>>>>>>>>>>> to worry about it. Of coz, in general, using RAW format and thick >>>>>>>>>>>> provisioning could get a best io performance result, but consume >>>>>>>>>>>> space and lack of snapshot capabliblity , so most the time it is >>>>>>>>>>>> not prefer go this path. >>>>>>>>>>>> >>>>>>>>>>>> Please gather more information first >>>>>>>>>>>> >>>>>>>>>>>> Regards, >>>>>>>>>>>> Levin >>>>>>>>>>>> On 7 Jul 2023 at 15:30 +0100, Granwille Strauss >>>>>>>>>>>> <granwi...@namhost.com.invalid> >>>>>>>>>>>> <mailto:granwi...@namhost.com.invalid> >>>>>>>>>>>> <mailto:granwi...@namhost.com.invalid> >>>>>>>>>>>> <mailto:granwi...@namhost.com.invalid>, wrote: >>>>>>>>>>>>> Hi Guys >>>>>>>>>>>>> Does Cloudstack have a disk write speed limit somewhere in its >>>>>>>>>>>>> setting? We have been transferring many files from remote servers >>>>>>>>>>>>> to VM machines on our Cloudstack instance and we recently noticed >>>>>>>>>>>>> that the VM write speeds are all limited to about 5-8 MB/s. But >>>>>>>>>>>>> the underlying hardware of the KVM uses SSDs capable of write >>>>>>>>>>>>> speeds of 300 - 600 MB/s. My disk offering on my current vms are >>>>>>>>>>>>> set to "No Disk Cache" with thin provisioning, could this be the >>>>>>>>>>>>> reason? I understand that "Write Back Disk Cach" has better write >>>>>>>>>>>>> speeds. Also I have VMs set as virtio for its disk controller. >>>>>>>>>>>>> What could I be missing in this case? >>>>>>>>>>>>> -- >>>>>>>>>>>>> Regards / Groete >>>>>>>>>>>>> >>>>>>>>>>>>> Granwille Strauss // Senior Systems Admin >>>>>>>>>>>>> >>>>>>>>>>>>> e: granwi...@namhost.com <mailto:granwi...@namhost.com> >>>>>>>>>>>>> <mailto:granwi...@namhost.com> <mailto:granwi...@namhost.com> >>>>>>>>>>>>> m: +264 81 323 1260 >>>>>>>>>>>>> w: www.namhost.com >>>>>>>>>>>>> <http://www.namhost.com/><http://www.namhost.com> >>>>>>>>>>>>> <http://www.namhost.com/> <http://www.namhost.com/> >>>>>>>>>>>>> <http://www.namhost.com/> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> Namhost Internet Services (Pty) Ltd, >>>>>>>>>>>>> 24 Black Eagle Rd, Hermanus, 7210, RSA >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> The content of this message is confidential. If you have received >>>>>>>>>>>>> it by mistake, please inform us by email reply and then delete >>>>>>>>>>>>> the message. It is forbidden to copy, forward, or in any way >>>>>>>>>>>>> reveal the contents of this message to anyone without our >>>>>>>>>>>>> explicit consent. The integrity and security of this email cannot >>>>>>>>>>>>> be guaranteed over the Internet. Therefore, the sender will not >>>>>>>>>>>>> be held liable for any damage caused by the message. For our full >>>>>>>>>>>>> privacy policy and disclaimers, please go to >>>>>>>>>>>>> https://www.namhost.com/privacy-policy >>>>>>>>>>>>> >>>>>>>>>>> -- >>>>>>>>>>> Regards / Groete >>>>>>>>>>> >>>>>>>>>>> Granwille Strauss // Senior Systems Admin >>>>>>>>>>> >>>>>>>>>>> e: granwi...@namhost.com <mailto:granwi...@namhost.com> >>>>>>>>>>> <mailto:granwi...@namhost.com> <mailto:granwi...@namhost.com> >>>>>>>>>>> m: +264 81 323 1260 >>>>>>>>>>> w: www.namhost.com >>>>>>>>>>> <http://www.namhost.com/><http://www.namhost.com> >>>>>>>>>>> <http://www.namhost.com/> <http://www.namhost.com/> >>>>>>>>>>> <http://www.namhost.com/> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> Namhost Internet Services (Pty) Ltd, >>>>>>>>>>> 24 Black Eagle Rd, Hermanus, 7210, RSA >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> The content of this message is confidential. If you have received >>>>>>>>>>> it by mistake, please inform us by email reply and then delete the >>>>>>>>>>> message. It is forbidden to copy, forward, or in any way reveal the >>>>>>>>>>> contents of this message to anyone without our explicit consent. >>>>>>>>>>> The integrity and security of this email cannot be guaranteed over >>>>>>>>>>> the Internet. Therefore, the sender will not be held liable for any >>>>>>>>>>> damage caused by the message. For our full privacy policy and >>>>>>>>>>> disclaimers, please go to https://www.namhost.com/privacy-policy >>>>>>>>>>> >>>>>>>>> -- >>>>>>>>> Regards / Groete >>>>>>>>> >>>>>>>>> <https://www.namhost.com/> <https://www.namhost.com/> Granwille >>>>>>>>> Strauss // Senior Systems Admin >>>>>>>>> >>>>>>>>> e: granwi...@namhost.com <mailto:granwi...@namhost.com> >>>>>>>>> <mailto:granwi...@namhost.com> <mailto:granwi...@namhost.com> >>>>>>>>> m: +264 81 323 1260 <tel:+264813231260> <tel:+264813231260> >>>>>>>>> w: www.namhost.com <http://www.namhost.com/><http://www.namhost.com> >>>>>>>>> <http://www.namhost.com/> <https://www.namhost.com/> >>>>>>>>> <https://www.namhost.com/> >>>>>>>>> >>>>>>>>> <https://www.facebook.com/namhost> <https://www.facebook.com/namhost> >>>>>>>>> <https://twitter.com/namhost> <https://twitter.com/namhost> >>>>>>>>> <https://www.instagram.com/namhostinternetservices/> >>>>>>>>> <https://www.instagram.com/namhostinternetservices/> >>>>>>>>> <https://www.linkedin.com/company/namhos> >>>>>>>>> <https://www.linkedin.com/company/namhos> >>>>>>>>> <https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA> >>>>>>>>> <https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA> >>>>>>>>> >>>>>>>>> <https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner> >>>>>>>>> >>>>>>>>> <https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner> >>>>>>>>> >>>>>>>>> Namhost Internet Services (Pty) Ltd, 24 Black Eagle Rd, Hermanus, >>>>>>>>> 7210, RSA >>>>>>>>> >>>>>>>>> The content of this message is confidential. If you have received it >>>>>>>>> by mistake, please inform us by email reply and then delete the >>>>>>>>> message. It is forbidden to copy, forward, or in any way reveal the >>>>>>>>> contents of this message to anyone without our explicit consent. The >>>>>>>>> integrity and security of this email cannot be guaranteed over the >>>>>>>>> Internet. Therefore, the sender will not be held liable for any >>>>>>>>> damage caused by the message. For our full privacy policy and >>>>>>>>> disclaimers, please go to https://www.namhost.com/privacy-policy >>>>>>>>> <https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818> >>>>>>>>> >>>>>>>>> <https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818> >>>>>>> -- >>>>>>> Regards / Groete >>>>>>> >>>>>>> Granwille Strauss // Senior Systems Admin >>>>>>> >>>>>>> e: granwi...@namhost.com <mailto:granwi...@namhost.com> >>>>>>> m: +264 81 323 1260 >>>>>>> w: www.namhost.com <http://www.namhost.com/><http://www.namhost.com> >>>>>>> <http://www.namhost.com/> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> Namhost Internet Services (Pty) Ltd, >>>>>>> 24 Black Eagle Rd, Hermanus, 7210, RSA >>>>>>> >>>>>>> >>>>>>> The content of this message is confidential. If you have received it by >>>>>>> mistake, please inform us by email reply and then delete the message. >>>>>>> It is forbidden to copy, forward, or in any way reveal the contents of >>>>>>> this message to anyone without our explicit consent. The integrity and >>>>>>> security of this email cannot be guaranteed over the Internet. >>>>>>> Therefore, the sender will not be held liable for any damage caused by >>>>>>> the message. For our full privacy policy and disclaimers, please go to >>>>>>> https://www.namhost.com/privacy-policy >>>>>>> >>>>> -- >>>>> Regards / Groete >>>>> >>>>> Granwille Strauss // Senior Systems Admin >>>>> >>>>> e: granwi...@namhost.com <mailto:granwi...@namhost.com> >>>>> m: +264 81 323 1260 >>>>> w: www.namhost.com <http://www.namhost.com/><http://www.namhost.com> >>>>> <http://www.namhost.com/> >>>>> >>>>> >>>>> >>>>> >>>>> Namhost Internet Services (Pty) Ltd, >>>>> 24 Black Eagle Rd, Hermanus, 7210, RSA >>>>> >>>>> >>>>> The content of this message is confidential. If you have received it by >>>>> mistake, please inform us by email reply and then delete the message. It >>>>> is forbidden to copy, forward, or in any way reveal the contents of this >>>>> message to anyone without our explicit consent. The integrity and >>>>> security of this email cannot be guaranteed over the Internet. Therefore, >>>>> the sender will not be held liable for any damage caused by the message. >>>>> For our full privacy policy and disclaimers, please go to >>>>> https://www.namhost.com/privacy-policy >>>>> >>> -- >>> Regards / Groete >>> >>> Granwille Strauss // Senior Systems Admin >>> >>> e: granwi...@namhost.com <mailto:granwi...@namhost.com> >>> m: +264 81 323 1260 >>> w: www.namhost.com <http://www.namhost.com/> >>> >>> >>> >>> >>> Namhost Internet Services (Pty) Ltd, >>> 24 Black Eagle Rd, Hermanus, 7210, RSA >>> >>> >>> The content of this message is confidential. If you have received it by >>> mistake, please inform us by email reply and then delete the message. It is >>> forbidden to copy, forward, or in any way reveal the contents of this >>> message to anyone without our explicit consent. The integrity and security >>> of this email cannot be guaranteed over the Internet. Therefore, the sender >>> will not be held liable for any damage caused by the message. For our full >>> privacy policy and disclaimers, please go to >>> https://www.namhost.com/privacy-policy >>> > -- > Regards / Groete > > <https://www.namhost.com/> Granwille Strauss // Senior Systems Admin > > e: granwi...@namhost.com <mailto:granwi...@namhost.com> > m: +264 81 323 1260 <tel:+264813231260> > w: www.namhost.com <https://www.namhost.com/> > > <https://www.facebook.com/namhost> <https://twitter.com/namhost> > <https://www.instagram.com/namhostinternetservices/> > <https://www.linkedin.com/company/namhos> > <https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA> > > > <https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner> > > Namhost Internet Services (Pty) Ltd, 24 Black Eagle Rd, Hermanus, 7210, RSA > > The content of this message is confidential. If you have received it by > mistake, please inform us by email reply and then delete the message. It is > forbidden to copy, forward, or in any way reveal the contents of this message > to anyone without our explicit consent. The integrity and security of this > email cannot be guaranteed over the Internet. Therefore, the sender will not > be held liable for any damage caused by the message. For our full privacy > policy and disclaimers, please go to https://www.namhost.com/privacy-policy > > <https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818> -- This message is intended only for the use of the individual or entity to which it is addressed and may contain confidential and/or privileged information. If you are not the intended recipient, please delete the original message and any copy of it from your computer system. You are hereby notified that any dissemination, distribution or copying of this communication is strictly prohibited unless proper authorization has been obtained for such action. If you have received this communication in error, please notify the sender immediately. Although IndiQus attempts to sweep e-mail and attachments for viruses, it does not guarantee that both are virus-free and accepts no liability for any damage sustained as a result of viruses.
