Re: about realtime system
Paul Robert Marino,
Your comments are not constructive and make you seem very ignorant, pop culture politics polarized, [tool - of - those - in - power]
...
Your comment is just useless!
Regards,
Bill
=Vote ,and keep track of what they do.
Sent:Monday, August 25, 2014 at 3:05 AM
From:Paul Robert Marino prmari...@gmail.com
To:jdow j...@earthlink.net, scientific-linux-users@fnal.gov
Subject:Re: about realtime system
Jdow
Your comments are not constructive and make you seem very ignorant, pop culture politics polarized, and unaware of the actual conversations involved in this thread. For the sake of your own professional integrity, and our own happiness please stay out of this.
I am happy informing the VMware fan and I hope he will learn something and if he comes up with a good point I will be happy to learn something from him.
Your comment is just useless!
-- Sent from my HP Pre3
On Aug 24, 2014 7:50 PM, jdow j...@earthlink.net wrote:
The stock exchange could remove most of the problem, meaning high
frequency trades, by placing a purely random 0 to 1 second latency
on all incoming data and all outgoing data. The high frequency trading
reads to me as just another means of skimming now that theyre not
allowed to round down fractional pennies and pocket the change. Its
time to give mere mortals some practical access to the exchanges. And
this interest in microsecond clocks would simply vanish from the
exchanges.
On a different point, the word I can find is that the free version of
VMWare does not support this high latency sensitivity setting.
{o.o} Joanne, Just sayin
On 2014-08-24 13:42, Paul Robert Marino wrote:
John
reread the first and third paragraph of my previous email.
Trading firms care about low latency but never cared about the
accuracy millisecond of the clocks. Sock exchanges on the other hand
want predictable latency not necessarily low latency but absolutely
require millisecond and if possible microsecond accurate clocks.
The reason for this is trading funds are worried about getting quotes,
bids, executions, etc. to the exchanges gateways as fast as possible;
however the exchange has to be able to prove to both the member firms
and the regulators that everyone is treated fairly once they put an
order into the gateway.
While yes VMware says under very specific configurations with 3/4ths
of their features disabled and special network cards which offload
their virtual switchs work VMware can handle low latency, but they
still can not handle clocks accurate to the millisecond and certain
cant handle it to the microsecond.
Furthermore I find this article highly suspect because its talking
about reducing the latency overhead in their visualization stack to
the point where it becomes less noticeable not necessarily true low
latency. This makes it acceptable for small hedge funds which have
staff and equipment budget constraints, but not really good enough for
the big boys if they are smart. I would advise you to be careful with
VMwares technical marketing docs and blogs in this area because the
sale people will tell you anything to get you to buy it, their high
level engineers will actually tell you the truth if they know what you
are using it for. In true real time and high precision situations
their senior engineers will tell the sales department to wave you off
of using their product if your employer is a big enough name for for
the real senior engineers (not sales engineers) to look at your
design prior to sale.
If you dive deep into that article it says you need
1) very specific hardware support specifically network cards
2) you need to turn off vmotion and all of the other fault tolerance features
3) you need to have very specific features turned on
4) It makes strongly implied suggestion but doesnt state flat out
that for best performance you need to align the number of cores you
assign to the layout of the cache in your CPU so you dont get
multiple VMs sharing CPU cache even if that means assigning more
VCPUs than you need.
5) a separate physical network card for each VM
6) disable memory over committing (Same as KVM)
7) disable CPU over committing (Same as KVM)
Even with that all of that you still do not have a 10 microsecond
latency jitter in the network stack, and the accuracy of the clocks
are still not guaranteed to the millisecond. In no place in that
article or the blog is clock accuracy mentioned at all.
All they are talking about is better response time latency not real precision.
On Sun, Aug 24, 2014 at 3:46 PM, John Lauro john.la...@covenanteyes.com wrote:
The recommendation changed with 5.5.
http://blogs.vmware.com/performance/2013/09/deploying-extremely-latency-sensitive-applications-in-vmware-vsphere-5-5.html
... However, performance demands of latency-sensitive applications with very low latency requirements such as distributed in-memory data management, stock trading, and high-performance computing
Re: about realtime system
On 27/08/14 19:21, Ken Teh wrote: When I first worked with it, it ran the 1.3 kernel and it was really fast. A 6µsec latency. It got progressively worse with 2x kernels. But still much better than the 150µsec quoted earler. Just to clarify. 150µsec max latency is the certification criteria from Red Hat. Many hardware vendors does it far better. Some require more tweaking than others. But once you've done that on appropriate hardware, you can often reach down to 10-25µs max latency, with a std.dev around 1-2µs. But it is highly hardware and configuration sensitive, and the lower latency you want the harder it gets. In addition, software tuning is also a crucial point. It's needed to ensure that the latency sensitive software runs with realtime scheduling and its priority isn't too low, as well as the kernel threads it depends on (like networking tx/rx threads) - But also not too high priorities, otherwise it may block important kernel threads doing their job. But doing this correctly, you can get really good results. -- kind regards, David Sommerseth
Re: about realtime system
On 24/08/14 18:57, John Lauro wrote: Why spread FUD about Vmware. Anyways, to hear what they say on the subject: http://www.vmware.com/files/pdf/techpaper/latency-sensitive-perf-vsphere55.pdf Anyways, KVM will not handle latency any better than Vmware. You can currently not achieve true realtime characteristics when adding virtualization, no matter the technology. The reason is that realtime tasks must be able to preempt running tasks to be able to keep its deadlines. Consider running a virtualized realtime kernel running a realtime task, on a host VM with a realtime kernel. When the task gets CPU time, it preempts all other running tasks on the provided CPU core. But if the VM host is not aware of this happening, it may just as well not give enough runtime in the right time-window to the realtime guest OS. Thus increasing the latency quite noticeably. So for this to work, the guest OS kernel must be able to communicate to the host OS kernel that it has a task which needs attention right now. And AFAIK, this mechanism is not implemented anywhere. I know there has been done some research on this topic some years ago, and an interesting paper on it. But I don't know if this has come any further. http://lwn.net/images/conf/rtlws11/papers/proc/p18.pdf -- kind regards, David Sommerseth
Re: about realtime system
Hello, While I am thoroughly interested in SL topics, I rarely comment on threads. Today is an exception. I work for a real-time linux vendor. I concur with David Somerseth's summation. Real-time cannot be achieved under virtualization. Regards, -- Michael Duvall Systems Analyst, Real-Time michael.duv...@ccur.com (954) 973-5395 direct (954) 531-4538 mobile CONCURRENT | 2881 Gateway Drive | Pompano Beach, FL 33069 | www.real-time.ccur.com -Original Message- From: David Sommerseth sl+us...@lists.topphemmelig.net Reply-to: scientific-linux-us...@listserv.fnal.gov scientific-linux-us...@listserv.fnal.gov To: John Lauro john.la...@covenanteyes.com, Paul Robert Marino prmari...@gmail.com Cc: Brandon Vincent brandon.vinc...@asu.edu, llwa...@gmail.com llwa...@gmail.com, SCIENTIFIC-LINUX-USERS@FNAL.GOV scientific-linux-users@fnal.gov, Nico Kadel-Garcia nka...@gmail.com Subject: Re: about realtime system Date: Wed, 27 Aug 2014 12:27:50 -0400 On 24/08/14 18:57, John Lauro wrote: Why spread FUD about Vmware. Anyways, to hear what they say on the subject: http://www.vmware.com/files/pdf/techpaper/latency-sensitive-perf-vsphere55.pdf Anyways, KVM will not handle latency any better than Vmware. You can currently not achieve true realtime characteristics when adding virtualization, no matter the technology. The reason is that realtime tasks must be able to preempt running tasks to be able to keep its deadlines. Consider running a virtualized realtime kernel running a realtime task, on a host VM with a realtime kernel. When the task gets CPU time, it preempts all other running tasks on the provided CPU core. But if the VM host is not aware of this happening, it may just as well not give enough runtime in the right time-window to the realtime guest OS. Thus increasing the latency quite noticeably. So for this to work, the guest OS kernel must be able to communicate to the host OS kernel that it has a task which needs attention right now. And AFAIK, this mechanism is not implemented anywhere. I know there has been done some research on this topic some years ago, and an interesting paper on it. But I don't know if this has come any further. http://lwn.net/images/conf/rtlws11/papers/proc/p18.pdf
Re: about realtime system
I used to play with a realtime Linux system back in the 90s that had a sort of virtualization architecture. It had a realtime executive that could run realtime tasks. One of its tasks was the Linux kernel itself so this way the tasks could talk with linux processes and make use of linux capabilities it lacked. When I first worked with it, it ran the 1.3 kernel and it was really fast. A 6µsec latency. It got progressively worse with 2x kernels. But still much better than the 150µsec quoted earler. I can't remember what it was called. Somebody from New Mexico developed it. There was also an offshoot development by a group in Italy. On 08/27/2014 12:07 PM, Michael Duvall wrote: Hello, While I am thoroughly interested in SL topics, I rarely comment on threads. Today is an exception. I work for a real-time linux vendor. I concur with David Somerseth's summation. Real-time cannot be achieved under virtualization. Regards, -- *Michael Duvall* Systems Analyst, Real-Time michael.duv...@ccur.com mailto:michael.duv...@ccur.com (954) 973-5395 direct (954) 531-4538 mobile CONCURRENT | 2881 Gateway Drive | Pompano Beach, FL 33069 | www.real-time.ccur.com http://www.real-time.ccur.com/ -Original Message- *From*: David Sommerseth sl+us...@lists.topphemmelig.net mailto:david%20sommerseth%20%3csl+us...@lists.topphemmelig.net%3e *Reply-to*: scientific-linux-us...@listserv.fnal.gov scientific-linux-us...@listserv.fnal.gov *To*: John Lauro john.la...@covenanteyes.com mailto:john%20lauro%20%3cjohn.la...@covenanteyes.com%3e, Paul Robert Marino prmari...@gmail.com mailto:paul%20robert%20marino%20%3cprmari...@gmail.com%3e *Cc*: Brandon Vincent brandon.vinc...@asu.edu mailto:brandon%20vincent%20%3cbrandon.vinc...@asu.edu%3e, llwa...@gmail.com llwa...@gmail.com mailto:%22llwa...@gmail.com%22%20%3cllwa...@gmail.com%3e, SCIENTIFIC-LINUX-USERS@FNAL.GOV scientific-linux-users@fnal.gov mailto:%22scientific-linux-us...@fnal.gov%22%20%3cscientific-linux-us...@fnal.gov%3e, Nico Kadel-Garcia nka...@gmail.com mailto:nico%20kadel-garcia%20%3cnka...@gmail.com%3e *Subject*: Re: about realtime system *Date*: Wed, 27 Aug 2014 12:27:50 -0400 On 24/08/14 18:57, John Lauro wrote: Why spread FUD about Vmware. Anyways, to hear what they say on the subject: http://www.vmware.com/files/pdf/techpaper/latency-sensitive-perf-vsphere55.pdf Anyways, KVM will not handle latency any better than Vmware. You can currently not achieve true realtime characteristics when adding virtualization, no matter the technology. The reason is that realtime tasks must be able to preempt running tasks to be able to keep its deadlines. Consider running a virtualized realtime kernel running a realtime task, on a host VM with a realtime kernel. When the task gets CPU time, it preempts all other running tasks on the provided CPU core. But if the VM host is not aware of this happening, it may just as well not give enough runtime in the right time-window to the realtime guest OS. Thus increasing the latency quite noticeably. So for this to work, the guest OS kernel must be able to communicate to the host OS kernel that it has a task which needs attention right now. And AFAIK, this mechanism is not implemented anywhere. I know there has been done some research on this topic some years ago, and an interesting paper on it. But I don't know if this has come any further. http://lwn.net/images/conf/rtlws11/papers/proc/p18.pdf
Re: about realtime system
On Sat, Aug 23, 2014 at 11:20 PM, Brandon Vincent brandon.vinc...@asu.edu wrote: On Fri, Aug 22, 2014 at 9:41 AM, llwa...@gmail.com wrote: Does it really make difference in timing control comparing to non-realtime kernel? Thanks. Whether or not you need a RTOS depends on your specific needs. Since you're working with LabVIEW, I would check out their white paper on the subject. http://www.ni.com/white-paper/14238/en/ I assume you're running real hardware, instead of virtual systems? I've seen people try to run real-time systems in virtuaalized, CentOS based VMware systems, and the results for extremely low latency operations were not encouraging. I also spent a lot of time with that with CentOS 5 making sure the hosted partitions were 4096 byte block aligned, which made a *big* difference with NetApp backend disk image storage. I don't have NI's realtime system but I am looking for a free linux OS which supports realtime kernel. So I wonder does anyone have experience in this on scientific linux? Does it support realtime kernel? NI Linux Real-Time is based off of the PREEMPT_RT patch set which you can find on the kernel.org Wiki. https://rt.wiki.kernel.org/index.php/CONFIG_PREEMPT_RT_Patch I've never compiled an EL kernel with the patch, but there is no reason why it should not be possible. Brandon Vincent If you go this route, do look into creating kernel RPM's with the mock toolkit and with patches applied as part of the SRPM. It's extra work to start with, but lets you manage the kernels and report dependencies more effectively.
Re: about realtime system
Why spread FUD about Vmware. Anyways, to hear what they say on the subject: http://www.vmware.com/files/pdf/techpaper/latency-sensitive-perf-vsphere55.pdf Anyways, KVM will not handle latency any better than Vmware. - Original Message - From: Paul Robert Marino prmari...@gmail.com To: Nico Kadel-Garcia nka...@gmail.com Cc: Brandon Vincent brandon.vinc...@asu.edu, llwa...@gmail.com, SCIENTIFIC-LINUX-USERS@FNAL.GOV scientific-linux-users@fnal.gov Sent: Sunday, August 24, 2014 12:26:17 PM Subject: Re: about realtime system Wow I don't know how VMware got mentioned in this string but VMware is not capable of real time operation and if you ask the senior engineers at VMware they will tell you they don't want you even trying it on their product because they know it wont work. The reason is VMware plays games with the clock on the VM so the clocks can never be 100% accurate. It should be possible to do real time in KVM assuming you don't overbook your CPU Cores or RAM. Apparently Red Hat has been doing VM's with microsecond accurate clocks with PTP running on the visualization
Re: about realtime system
On Sun, Aug 24, 2014 at 12:57 PM, John Lauro john.la...@covenanteyes.com wrote: Why spread FUD about Vmware. Anyways, to hear what they say on the subject: http://www.vmware.com/files/pdf/techpaper/latency-sensitive-perf-vsphere55.pdf Anyways, KVM will not handle latency any better than Vmware. - Original Message - From: Paul Robert Marino prmari...@gmail.com To: Nico Kadel-Garcia nka...@gmail.com Cc: Brandon Vincent brandon.vinc...@asu.edu, llwa...@gmail.com, SCIENTIFIC-LINUX-USERS@FNAL.GOV scientific-linux-users@fnal.gov Sent: Sunday, August 24, 2014 12:26:17 PM Subject: Re: about realtime system Wow I don't know how VMware got mentioned in this string but VMware is not capable of real time operation and if you ask the senior engineers at VMware they will tell you they don't want you even trying it on their product because they know it wont work. The reason is VMware plays games with the clock on the VM so the clocks can never be 100% accurate. It should be possible to do real time in KVM assuming you don't overbook your CPU Cores or RAM. Apparently Red Hat has been doing VM's with microsecond accurate clocks with PTP running on the visualization I mentioned that I hope they were using real servers, not VM's. I'd had people try to run real-time systems in virtualization, specifically with VMware, and it wasn't workable for their needs. Also, high precision is not the same as low latency, although both are often grouped together for real-time operations. I'm curious if Paul needs both.
Re: about realtime system
On Sun, Aug 24, 2014 at 9:26 AM, Paul Robert Marino prmari...@gmail.com wrote: That said I've run hundreds of mission critical systems many accurate to within 5 milliseconds on the standard kernel with proper tuning and a stripped down OS install for nearly a decade and Ive never had an issue. The standard Linux kernel shipped by most distributions is actually quite effective in applications that require millisecond level scheduling precision. Paul's experience confirms this. If the need for smaller interrupt and scheduling latency arises, the stock kernel source allows you to generate a soft real-time kernel by choosing the Preemptible Kernel (PREEMPTDESKTOP) during kernel configuration and compilation. What the standard kernel source doesn't provide is reliable and consistent sub-millisecond scheduling. Applying the PREEMPT_RT patch generates what can for the most part be considered a hard RTOS kernel. For the most part in 99% of applications the standard kernel will suffice. Without knowing what Lee's application is, it is impossible to determine if he really needs a soft or hard real-time kernel. Brandon Vincent
Re: about realtime system
Nico Depending on the role of the particular system and or which company I was working for at the time I've need one the other or both. In my current role in the broadcast industry precision with predictable latency is more important for most of my systems. That said when I worked in the financial industry it changed based on what part of the industy I was working for. The stock exchanges I've worked for cared about precision because it was more important to them to make sure every one had the same latency and our logging was accurate for audit purposes. When I used to work for a managed systems vender for hedge funds they were all about low latency because the faster they got data in and out of the exchanges often determined if they had an edge over their competitors or not. quite literally an extra millisecond could cost them millions of dollars a second due to the nature of high frequency trading. Generally I don't think of real time kernels when I am thinking about low latency because oftent they increase the latency when dealing with multiple operations. however the reverse can true if you only have a box doing one specific task only but that rarely is the case. By the way one of those stock exchanges is where the VMware engineers told us never to use their product in production. In fact we had huge problems with VMware in our development environments because some of our applications would actually detect the clock instability in the VMware clocks and would shut themselves down rather than have inaccurate audit logs. as a result we found we had trouble even using it in our development environments. By the way Red hat only told me recently about guaranteeing the microsecond precision of the clocks in KVM on RHEV and said they have been doing it in financial for over a year. there are conditions though such as you need to turn off support for overbooking the CPU cores. last I checked VMware still says do not use their product anywhere where you need millisecond accurate clocks. Further more I dont know about that statement Anyways, KVM will not handle latency any better than Vmware. the article you pointed out talks about VCPU's going in and out of halted states, which is normal and completely expected in VMware because they always assume you are going to overbook your CPU cores. there is a slight difference when you talk about KVM in paravirtualized mode with overbooking disabled it directly maps the CPU cores the the VM so as long as you don't have power management enabled the CPU's are always operating at full speed further more you can directly map PCIe bus address to the VM (essentially assigning a card on your bus directly to the VM to be completely managed by its kernel) to reduce latency in other ways to hardware if you need too. On Sun, Aug 24, 2014 at 2:02 PM, Nico Kadel-Garcia nka...@gmail.com wrote: On Sun, Aug 24, 2014 at 12:57 PM, John Lauro john.la...@covenanteyes.com wrote: Why spread FUD about Vmware. Anyways, to hear what they say on the subject: http://www.vmware.com/files/pdf/techpaper/latency-sensitive-perf-vsphere55.pdf Anyways, KVM will not handle latency any better than Vmware. - Original Message - From: Paul Robert Marino prmari...@gmail.com To: Nico Kadel-Garcia nka...@gmail.com Cc: Brandon Vincent brandon.vinc...@asu.edu, llwa...@gmail.com, SCIENTIFIC-LINUX-USERS@FNAL.GOV scientific-linux-users@fnal.gov Sent: Sunday, August 24, 2014 12:26:17 PM Subject: Re: about realtime system Wow I don't know how VMware got mentioned in this string but VMware is not capable of real time operation and if you ask the senior engineers at VMware they will tell you they don't want you even trying it on their product because they know it wont work. The reason is VMware plays games with the clock on the VM so the clocks can never be 100% accurate. It should be possible to do real time in KVM assuming you don't overbook your CPU Cores or RAM. Apparently Red Hat has been doing VM's with microsecond accurate clocks with PTP running on the visualization I mentioned that I hope they were using real servers, not VM's. I'd had people try to run real-time systems in virtualization, specifically with VMware, and it wasn't workable for their needs. Also, high precision is not the same as low latency, although both are often grouped together for real-time operations. I'm curious if Paul needs both.
Re: about realtime system
PS. That last paragraph was intended to respond to John not Nico. On Sun, Aug 24, 2014 at 3:27 PM, Paul Robert Marino prmari...@gmail.com wrote: Nico Depending on the role of the particular system and or which company I was working for at the time I've need one the other or both. In my current role in the broadcast industry precision with predictable latency is more important for most of my systems. That said when I worked in the financial industry it changed based on what part of the industy I was working for. The stock exchanges I've worked for cared about precision because it was more important to them to make sure every one had the same latency and our logging was accurate for audit purposes. When I used to work for a managed systems vender for hedge funds they were all about low latency because the faster they got data in and out of the exchanges often determined if they had an edge over their competitors or not. quite literally an extra millisecond could cost them millions of dollars a second due to the nature of high frequency trading. Generally I don't think of real time kernels when I am thinking about low latency because oftent they increase the latency when dealing with multiple operations. however the reverse can true if you only have a box doing one specific task only but that rarely is the case. By the way one of those stock exchanges is where the VMware engineers told us never to use their product in production. In fact we had huge problems with VMware in our development environments because some of our applications would actually detect the clock instability in the VMware clocks and would shut themselves down rather than have inaccurate audit logs. as a result we found we had trouble even using it in our development environments. By the way Red hat only told me recently about guaranteeing the microsecond precision of the clocks in KVM on RHEV and said they have been doing it in financial for over a year. there are conditions though such as you need to turn off support for overbooking the CPU cores. last I checked VMware still says do not use their product anywhere where you need millisecond accurate clocks. Further more I dont know about that statement Anyways, KVM will not handle latency any better than Vmware. the article you pointed out talks about VCPU's going in and out of halted states, which is normal and completely expected in VMware because they always assume you are going to overbook your CPU cores. there is a slight difference when you talk about KVM in paravirtualized mode with overbooking disabled it directly maps the CPU cores the the VM so as long as you don't have power management enabled the CPU's are always operating at full speed further more you can directly map PCIe bus address to the VM (essentially assigning a card on your bus directly to the VM to be completely managed by its kernel) to reduce latency in other ways to hardware if you need too. On Sun, Aug 24, 2014 at 2:02 PM, Nico Kadel-Garcia nka...@gmail.com wrote: On Sun, Aug 24, 2014 at 12:57 PM, John Lauro john.la...@covenanteyes.com wrote: Why spread FUD about Vmware. Anyways, to hear what they say on the subject: http://www.vmware.com/files/pdf/techpaper/latency-sensitive-perf-vsphere55.pdf Anyways, KVM will not handle latency any better than Vmware. - Original Message - From: Paul Robert Marino prmari...@gmail.com To: Nico Kadel-Garcia nka...@gmail.com Cc: Brandon Vincent brandon.vinc...@asu.edu, llwa...@gmail.com, SCIENTIFIC-LINUX-USERS@FNAL.GOV scientific-linux-users@fnal.gov Sent: Sunday, August 24, 2014 12:26:17 PM Subject: Re: about realtime system Wow I don't know how VMware got mentioned in this string but VMware is not capable of real time operation and if you ask the senior engineers at VMware they will tell you they don't want you even trying it on their product because they know it wont work. The reason is VMware plays games with the clock on the VM so the clocks can never be 100% accurate. It should be possible to do real time in KVM assuming you don't overbook your CPU Cores or RAM. Apparently Red Hat has been doing VM's with microsecond accurate clocks with PTP running on the visualization I mentioned that I hope they were using real servers, not VM's. I'd had people try to run real-time systems in virtualization, specifically with VMware, and it wasn't workable for their needs. Also, high precision is not the same as low latency, although both are often grouped together for real-time operations. I'm curious if Paul needs both.
Re: about realtime system
The recommendation changed with 5.5. http://blogs.vmware.com/performance/2013/09/deploying-extremely-latency-sensitive-applications-in-vmware-vsphere-5-5.html ... However, performance demands of latency-sensitive applications with very low latency requirements such as distributed in-memory data management, stock trading, and high-performance computing have long been thought to be incompatible with virtualization. vSphere 5.5 includes a new feature for setting latency sensitivity in order to support virtual machines with strict latency requirements. - Original Message - From: Paul Robert Marino prmari...@gmail.com To: Nico Kadel-Garcia nka...@gmail.com Cc: John Lauro john.la...@covenanteyes.com, Brandon Vincent brandon.vinc...@asu.edu, Lee Kin llwa...@gmail.com, SCIENTIFIC-LINUX-USERS@FNAL.GOV scientific-linux-users@fnal.gov Sent: Sunday, August 24, 2014 3:27:39 PM Subject: Re: about realtime system ... By the way one of those stock exchanges is where the VMware engineers told us never to use their product in production. In fact we had huge problems with VMware in our development environments because some of our applications would actually detect the clock instability in the VMware clocks and would shut themselves down rather than have inaccurate audit logs. as a result we found we had trouble even using it in our development environments.
Re: about realtime system
John reread the first and third paragraph of my previous email. Trading firms care about low latency but never cared about the accuracy millisecond of the clocks. Sock exchanges on the other hand want predictable latency not necessarily low latency but absolutely require millisecond and if possible microsecond accurate clocks. The reason for this is trading funds are worried about getting quotes, bids, executions, etc. to the exchanges gateways as fast as possible; however the exchange has to be able to prove to both the member firms and the regulators that everyone is treated fairly once they put an order into the gateway. While yes VMware says under very specific configurations with 3/4ths of their features disabled and special network cards which offload their virtual switch's work VMware can handle low latency, but they still can not handle clocks accurate to the millisecond and certain cant handle it to the microsecond. Furthermore I find this article highly suspect because its talking about reducing the latency overhead in their visualization stack to the point where it becomes less noticeable not necessarily true low latency. This makes it acceptable for small hedge funds which have staff and equipment budget constraints, but not really good enough for the big boys if they are smart. I would advise you to be careful with VMwares technical marketing docs and blogs in this area because the sale people will tell you anything to get you to buy it, their high level engineers will actually tell you the truth if they know what you are using it for. In true real time and high precision situations their senior engineers will tell the sales department to wave you off of using their product if your employer is a big enough name for for the real senior engineers (not sales engineers) to look at your design prior to sale. If you dive deep into that article it say's you need 1) very specific hardware support specifically network cards 2) you need to turn off vmotion and all of the other fault tolerance features 3) you need to have very specific features turned on 4) It makes strongly implied suggestion but doesn't state flat out that for best performance you need to align the number of cores you assign to the layout of the cache in your CPU so you don't get multiple VM's sharing CPU cache even if that means assigning more VCPUs than you need. 5) a separate physical network card for each VM 6) disable memory over committing (Same as KVM) 7) disable CPU over committing (Same as KVM) Even with that all of that you still do not have a 10 microsecond latency jitter in the network stack, and the accuracy of the clocks are still not guaranteed to the millisecond. In no place in that article or the blog is clock accuracy mentioned at all. All they are talking about is better response time latency not real precision. On Sun, Aug 24, 2014 at 3:46 PM, John Lauro john.la...@covenanteyes.com wrote: The recommendation changed with 5.5. http://blogs.vmware.com/performance/2013/09/deploying-extremely-latency-sensitive-applications-in-vmware-vsphere-5-5.html ... However, performance demands of latency-sensitive applications with very low latency requirements such as distributed in-memory data management, stock trading, and high-performance computing have long been thought to be incompatible with virtualization. vSphere 5.5 includes a new feature for setting latency sensitivity in order to support virtual machines with strict latency requirements. - Original Message - From: Paul Robert Marino prmari...@gmail.com To: Nico Kadel-Garcia nka...@gmail.com Cc: John Lauro john.la...@covenanteyes.com, Brandon Vincent brandon.vinc...@asu.edu, Lee Kin llwa...@gmail.com, SCIENTIFIC-LINUX-USERS@FNAL.GOV scientific-linux-users@fnal.gov Sent: Sunday, August 24, 2014 3:27:39 PM Subject: Re: about realtime system ... By the way one of those stock exchanges is where the VMware engineers told us never to use their product in production. In fact we had huge problems with VMware in our development environments because some of our applications would actually detect the clock instability in the VMware clocks and would shut themselves down rather than have inaccurate audit logs. as a result we found we had trouble even using it in our development environments.
Re: about realtime system
The stock exchange could remove most of the problem, meaning high
frequency trades, by placing a purely random 0 to 1 second latency
on all incoming data and all outgoing data. The high frequency trading
reads to me as just another means of skimming now that they're not
allowed to round down fractional pennies and pocket the change. It's
time to give mere mortals some practical access to the exchanges. And
this interest in microsecond clocks would simply vanish from the
exchanges.
On a different point, the word I can find is that the free version of
VMWare does not support this high latency sensitivity setting.
{o.o} Joanne, Just sayin'
On 2014-08-24 13:42, Paul Robert Marino wrote:
John
reread the first and third paragraph of my previous email.
Trading firms care about low latency but never cared about the
accuracy millisecond of the clocks. Sock exchanges on the other hand
want predictable latency not necessarily low latency but absolutely
require millisecond and if possible microsecond accurate clocks.
The reason for this is trading funds are worried about getting quotes,
bids, executions, etc. to the exchanges gateways as fast as possible;
however the exchange has to be able to prove to both the member firms
and the regulators that everyone is treated fairly once they put an
order into the gateway.
While yes VMware says under very specific configurations with 3/4ths
of their features disabled and special network cards which offload
their virtual switch's work VMware can handle low latency, but they
still can not handle clocks accurate to the millisecond and certain
cant handle it to the microsecond.
Furthermore I find this article highly suspect because its talking
about reducing the latency overhead in their visualization stack to
the point where it becomes less noticeable not necessarily true low
latency. This makes it acceptable for small hedge funds which have
staff and equipment budget constraints, but not really good enough for
the big boys if they are smart. I would advise you to be careful with
VMwares technical marketing docs and blogs in this area because the
sale people will tell you anything to get you to buy it, their high
level engineers will actually tell you the truth if they know what you
are using it for. In true real time and high precision situations
their senior engineers will tell the sales department to wave you off
of using their product if your employer is a big enough name for for
the real senior engineers (not sales engineers) to look at your
design prior to sale.
If you dive deep into that article it say's you need
1) very specific hardware support specifically network cards
2) you need to turn off vmotion and all of the other fault tolerance
features
3) you need to have very specific features turned on
4) It makes strongly implied suggestion but doesn't state flat out
that for best performance you need to align the number of cores you
assign to the layout of the cache in your CPU so you don't get
multiple VM's sharing CPU cache even if that means assigning more
VCPUs than you need.
5) a separate physical network card for each VM
6) disable memory over committing (Same as KVM)
7) disable CPU over committing (Same as KVM)
Even with that all of that you still do not have a 10 microsecond
latency jitter in the network stack, and the accuracy of the clocks
are still not guaranteed to the millisecond. In no place in that
article or the blog is clock accuracy mentioned at all.
All they are talking about is better response time latency not real precision.
On Sun, Aug 24, 2014 at 3:46 PM, John Lauro john.la...@covenanteyes.com wrote:
The recommendation changed with 5.5.
http://blogs.vmware.com/performance/2013/09/deploying-extremely-latency-sensitive-applications-in-vmware-vsphere-5-5.html
... However, performance demands of latency-sensitive applications with very
low latency requirements such as distributed in-memory data management, stock
trading, and high-performance computing have long been thought to be incompatible
with virtualization.
vSphere 5.5 includes a new feature for setting latency sensitivity in order to
support virtual machines with strict latency requirements.
- Original Message -
From: Paul Robert Marino prmari...@gmail.com
To: Nico Kadel-Garcia nka...@gmail.com
Cc: John Lauro john.la...@covenanteyes.com, Brandon Vincent
brandon.vinc...@asu.edu, Lee Kin
llwa...@gmail.com, SCIENTIFIC-LINUX-USERS@FNAL.GOV
scientific-linux-users@fnal.gov
Sent: Sunday, August 24, 2014 3:27:39 PM
Subject: Re: about realtime system
...
By the way one of those stock exchanges is where the VMware engineers
told us never to use their product in production. In fact we had huge
problems with VMware in our development environments because some of
our applications would actually detect the clock instability in the
VMware clocks and would shut themselves down rather than have
inaccurate audit logs. as a result we found we had
Re: about realtime system
On Sun, Aug 24, 2014 at 3:27 PM, Paul Robert Marino prmari...@gmail.com wrote: Nico Depending on the role of the particular system and or which company I was working for at the time I've need one the other or both. In my current role in the broadcast industry precision with predictable latency is more important for most of my systems. That said when I worked in the financial industry it changed based on what part of the industy I was working for. I... need to step out of some parts of this disussion, due to NDA's. Further more I dont know about that statement Anyways, KVM will not handle latency any better than Vmware. the article you pointed out talks about VCPU's going in and out of halted states, which is normal I'm afraid that wasn't me. I just expressed my hope that the original poster was using real hardware, not virtualization, due to my poor experience and the architectural constraints of virtualization. and completely expected in VMware because they always assume you are going to overbook your CPU cores. there is a slight difference when you talk about KVM in paravirtualized mode with overbooking disabled it directly maps the CPU cores the the VM so as long as you don't have power management enabled the CPU's are always operating at full speed further more you can directly map PCIe bus address to the VM (essentially assigning a card on your bus directly to the VM to be completely managed by its kernel) to reduce latency in other ways to hardware if you need too. I'm not disagreeing. It's extra work, it's fragile in configuration, and that sort of fine tuning is very, very easy to get wrong.
Re: about realtime system
On Sun, Aug 24, 2014 at 7:50 PM, jdow j...@earthlink.net wrote:
The stock exchange could remove most of the problem, meaning high
frequency trades, by placing a purely random 0 to 1 second latency
on all incoming data and all outgoing data. The high frequency trading
reads to me as just another means of skimming now that they're not
allowed to round down fractional pennies and pocket the change. It's
time to give mere mortals some practical access to the exchanges. And
this interest in microsecond clocks would simply vanish from the
exchanges.
The whole high frequency trading, low latency mess is due to leave the
Linux world, at least for the hosts directly receiving the data, due
to the availability of FPGA's that can live on fiber optic
connections, physically adjacent to the stock market. I can say that
based on interviews I did some years back, without a signed NDA for
the interviews, and based on press articles on the technology.
Generating the rules for the FPGA's, now, *that* is an interesting
potential Linux market, and I can point people to job ads for
precisely this. Tying it back to Scientific Linux, I'd *love* to see
them using Scientific Linux for this due to the support available from
the Scientific Linux world for oddball scientific computing
requirements. And the Scientific Linux built-in integration for 3rd
party repositories like EPEL and ATrpms is invaluable.
On a different point, the word I can find is that the free version of
VMWare does not support this high latency sensitivity setting.
{o.o} Joanne, Just sayin'
Cool thanks for looking into it!
Re: about realtime system
Seriously lets take the high frequency trading thing off this list and any one else who wants to talk to me as well a out it, I'm perfectly happy to explain it.Further more I'm willing to explain the real problems with the world financial system but not on this list and certainly not on this thread-- Sent from my HP Pre3On Aug 24, 2014 8:55 PM, Nico Kadel-Garcia nka...@gmail.com wrote: On Sun, Aug 24, 2014 at 7:50 PM, jdow j...@earthlink.net wrote:
The stock exchange could remove most of the problem, meaning high
frequency trades, by placing a purely random 0 to 1 second latency
on all incoming data and all outgoing data. The high frequency trading
reads to me as just another means of skimming now that they're not
allowed to round down fractional pennies and pocket the change. It's
time to give mere mortals some practical access to the exchanges. And
this interest in microsecond clocks would simply vanish from the
exchanges.
The whole high frequency trading, low latency mess is due to leave the
Linux world, at least for the hosts directly receiving the data, due
to the availability of FPGA's that can live on fiber optic
connections, physically adjacent to the stock market. I can say that
based on interviews I did some years back, without a signed NDA for
the interviews, and based on press articles on the technology.
Generating the rules for the FPGA's, now, *that* is an interesting
potential Linux market, and I can point people to job ads for
precisely this. Tying it back to Scientific Linux, I'd *love* to see
them using Scientific Linux for this due to the support available from
the Scientific Linux world for oddball scientific computing
requirements. And the Scientific Linux built-in integration for 3rd
party repositories like EPEL and ATrpms is invaluable.
On a different point, the word I can find is that the free version of
VMWare does not support this "high latency sensitivity" setting.
{o.o} Joanne, Just sayin'
Cool thanks for looking into it!
about realtime system
Hi all, I find scientific linux in NI (Labview)'s page. I find that labview support linux systems including scientific linux from ver 2011. I am going to build a system with labview installed on this OS. The reason that I don't use MS windows is the time slice control is pretty bad and most time I need to control the timing in an acceptable precision. I don't have NI's realtime system but I am looking for a free linux OS which supports realtime kernel. So I wonder does anyone have experience in this on scientific linux? Does it support realtime kernel? Does it really make difference in timing control comparing to non-realtime kernel? Thanks.
