Hi all,
Here are two Python scripts which I have used to collect performance
benchmarks for this series. I am putting them here in case they are useful.
Ben
On 11/09/15 17:08, Ben Catterall wrote:
Hi all,
I have now finished my internship at Citrix and am posting this final version of
my RFC series. I would like to express my thanks to all of those who have taken
the time to review, comment and discuss this series, as well as to my colleagues
who have provided excellent guidance and help. I have learned a great deal and
have greatly enjoyed working with all of you. Thank you.
Hopefully the series will be beneficial. I believe that it has shown that a
deprivileged mode in Xen is a possible and viable option, as long as performance
impact vs security is carefully considered on a case-by-case basis. The end of
this series contains an example of moving some of the vpic into deprivileged
mode which has allowed me to test and verify that the feature works. There are
enhancements and some clean up which is needed but, after that, the feature
could be deployed to HVM devices currently found in Xen such as the VPIC.
Patches one to four are (hopefully) now fairly stable. Patch 5 is the new
system call and deprivileged dispatch mode which is new to this series. Patch 6
is also new and is a demonstration of using this for the vpic and hass mainly
been used to test and exercise this feature.
As this patch series is in RFC, there are some debug printks which should be
removed when/if it leaves RFC but, they are useful in fixing the known issue so
I have left them in until that can be resolved.
There are some efficiency savings that can be made and an instance of a general
issue (detailed later) which will need to be addressed.
Many thanks once again,
Ben
TODOs
-----
There is a set of TODOs in this patch series, some issues in the later patches
which need addressing and some other considerations which I've summarised here.
Patch 1:
- Consider hvm_deprivileged_map_* and an efficiency saving by mapping in
larger
pages. See the TODO at the top of the L4 version of this method.
Patch 2:
- We have a much more heavyweight version of the deprivileged mode context
switch after testing for AMD SVM found that this was necessary. However,
the FPU is currently also saved and this may not be necessary. Consideration
is needed to work out if we can cut this down even more.
Patch 4:
- The watchdog timer is hooked currently to kill deprivileged mode operations
that run for too long and is hardcoded to be at least one watchdog tick and
at most two. This may want to be refined.
Patch 5:
- Alias data for deprivileged mode. There is a large comment at the top of
deprivileged_syscall.c which outlines considerations.
- Check if we need to map_domain_page the pages when we do the copy in
hvm_deprivileged_copy_data{to/from}
- Check for unsigned integer wrapping on addition in
hvm_deprivileged_copy_data_{to/from}
- Move hvm_deprivileged_syscall into the syscall macro. It's a stub and
unless extra code is needed there it can be folded into the macro.
- Check maintainers' thoughts on the deprivileged mode function checks in
hvm_deprivileged_user_mode. See the TODO comment.
Patches 5 & 6:
- Fix/work around the GCC switch statement issue.
KNOWN ISSUES
------------
- Page fault for vpic_ioport_write due to GCC switch statements placing the
jump table in .rodata which is in the privileged mode area.
This has been traced to the first of the switch statements in the function.
Though other switches in that function may also be affected.
Compiled using GCC 4.9.2-10.
You can get the offset into this function by doing:
(RIP - (depriv_vpic_ioport_write - __hvm_deprivileged_text_start))
It appears to be a built-in default of GCC to put switch jump tables in
.rodata or .text and there does not appear to be a way to change this
(except to patch the compiler, though hopefully there _is_ another
option I just haven't been able to find...). Note that GCC will not
necessarily allocate jump tables for each switch statment, it appears to
depends on a number of factors such as the optimiser, the number of cases,
the type of the case, compiler version etc.
Thus, when we relocate a deprivileged method containing code using a switch
statement which GCC has created a jump table for, this leads to a page
fault. This is because we have not mapped in the rodata section
as we should not (depriv should not have access to it).
A workaround would be to patch the generated assembly so that this table is
moved into hvm_deprivileged.rodata. This can be done by adding,
.section .hvm_deprivileged.rodata, around the generated table. We can then
relocate this.
Note that GCC is using RIP-relative addressing for this, so the offset
of depriv .rodata to the depriv .text segment will need to be the same
when it is mapped in.
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel
#!/usr/bin/python3
import numpy as np
import matplotlib.pyplot as plt
#Use for running rsync
import subprocess
#Use for command line args
import sys, argparse
import array
import pickle
from matplotlib.pyplot import savefig
'''
Plot the data and output to a file
'''
def plot(xList,filename):
i = 0
colours = ["red", "blue"]
labels = ["Enabled", "Disabled"]
bins = array.array('f')
#Get max and min
max = np.amax(xList[0])
if np.amax(xList[1]) > max:
max = np.amax(xList[1])
min = np.amin(xList[0])
if np.amin(xList[1]) < min:
min = np.amin(xList[1])
intervalBins = np.linspace(min, max, num=100)
for data in xList:
n, bins, patches = plt.hist(data, intervalBins, facecolor=colours[i],
alpha=0.75, label=labels[i]);
print(str(n) + " " + str(bins) + " " + str(patches) + "\n")
i = i + 1
print("Mean: " + str(np.mean(data)) + " seconds\n")
print("Std: " + str(np.std(data,dtype=np.float64)) + " seconds\n")
print("Max: " + str(np.amax(data)) + " Min: " + str(np.amin(data)) + "\n")
plt.legend(loc='upper right')
plt.ylabel("Frequency")
plt.xlabel("Time/s")
plt.title("Graph of Xen deprivileged user performance")
plt.ticklabel_format(axis='x', style='sci', scilimits=(-2,2))
fg = plt.gcf()
fg.savefig(filename)
return
def main():
#Setup parser
parser = argparse.ArgumentParser(description="Test HVM depriv performance.")
parser.add_argument("-p", "--plot", nargs=2, metavar=("[DATA FILE NAME]", "[GRAPH FILE NAME]"),
help="Plot the data and output a graph at file name")
args = parser.parse_args()
#Process args
if args.plot:
print("Plotting...\n")
#Use Pickle to load in the array
timeData = pickle.load(open(args.plot[0], "rb"))
#Parse it and plot
plot(timeData, args.plot[1])
print("DONE")
if __name__ == '__main__':
main()
#!/usr/bin/python3
'''
To make use of this script, you will need to run it as root and have installed
the Python portio package.
You need to have patched the portio operation in vmx.c and svm.c which can be
found at xen/arch/x86/hvm/{vmx,svm}/{vmx.c,svm.c} as shown below so that it
will run the deprivileged mode when we perform a particular port operation.
My series has the basic hook in there for patch 2 but this is commented
out/removed in later patches as there was a functional change (you can
no longer just enter the mode, you need to supply an operation to jump to so
this will need to be updated to work with that).
This is in vmx_vmexit_handler and svm_vmexit_handler
Here is the svm part of this copied here for completeness:
uint16_t port = (vmcb->exitinfo1 >> 16) & 0xFFFF;
int bytes = ((vmcb->exitinfo1 >> 4) & 0x07);
int dir = (vmcb->exitinfo1 & 1) ? IOREQ_READ : IOREQ_WRITE;
/* DEBUG: Run only for a specific port */
if(port == 0x1000)
{
if( guest_cpu_user_regs()->eax == 0x1)
{
hvm_deprivileged_user_mode();
}
__update_guest_eip(regs, vmcb->exitinfo2 - vmcb->rip);
break;
}
if ( handle_pio(port, bytes, dir) )
__update_guest_eip(regs, vmcb->exitinfo2 - vmcb->rip);
}
'''
import array
#Use for command line args
import sys
import pickle
import time
from portio import ioperm, iopl, outb
def prepare_portio(port):
#Get access to all ports (needed on Linux pre 2.6.? kernel)
if( iopl(3) != 0 ):
print("ERROR: Elevating access to ports " + str(port) + "\n")
return False
#get access to port, only one port, and enable it
if( ioperm(port, 1, 1) != 0 ):
print("ERROR: Preparing port " + str(port) + "\n")
return False
return True
"""
Clean up after portio operations
"""
def release_portio(port):
if( ioperm(port, 1, 0) != 0 ):
print("ERROR: Releasing port " + str(port) + " \n")
return False
if( iopl(0) != 0 ):
print("ERROR: Clearing access to ports " + str(port) + "\n")
return False
return True
'''
Needs root to run
'''
def send_data(port, data):
outb(data, port)
def profile(outputTimes, numIterations, port, data):
i = 0
while (i < numIterations):
before = time.time()
send_data(port, data)
after = time.time()
timeDelta = after - before
outputTimes.append( timeDelta)
i = i + 1
def main():
outputTimesWith = array.array("f")
outputTimesWithout = array.array("f")
portWith = 0x1000
portWithout = 0x1000
dataWith = 0x1
dataWithout =0x2
numIterations = 100000
prepare_portio(portWith)
#Profile the system
#With the deprivileged op
profile(outputTimesWith, numIterations, portWith, dataWith);
release_portio(portWith)
#Without the deprivileged op
prepare_portio(portWithout)
profile(outputTimesWithout, numIterations, portWithout, dataWithout);
release_portio(portWithout)
data =[outputTimesWith, outputTimesWithout]
outputFile = open("data", "wb")
pickle.dump(data, outputFile)
outputFile.close()
return
if __name__ == '__main__':
main()
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel
