Hello Martin,
I don't have experience with running containers and VGL, but I have set up
headless servers with GPU cards and remote users doing server-side
rendering. I can confirm that CUDA programs will work in this scenario. The
users connect with FastX, but Turbo VNC would also work. We accomplish this
using a normal Xorg process (not xvfb) running on DISPLAY=:1 with the
proprietary nvidia driver configured as normal except for setting the
DISPLAY and the PCI_ID. We set the PCI ID due to using multiple GPUs in the
same box.
We also run the above config in KVM VMs with one GPU per VM and use
PCI-passthrough to pass the GPUs from the host to the guest. I don't know
how to have a single Xorg talk to multiple GPUs.
I've attached a file with commands that might help to set up an Nvidia card
on RHEL7
If you want to set up multiple Xorg processes, then just use the next
DISPLAY number and increment the numbers for the headlessx@Z service to
increment Z
** NOTE1: change the +s and +f options to -s and -f on the
vglserver_config command to restrict access. Allowing all users is useful
for troubleshooting/exploration.
** NOTE2: The headlessx@1 service will take over the console and make the
display blank while it's active. I haven't found a way to fix this.
I hope this helps.
Sincerely,
Jason
---------------------------------------------------------------------------
Jason Edgecombe | Linux Administrator
UNC Charlotte | Office of OneIT
9201 University City Blvd. | Charlotte, NC 28223-0001
Phone: 704-687-1943
[email protected] | http://engr.uncc.edu | Facebook
---------------------------------------------------------------------------
If you are not the intended recipient of this transmission or a person
responsible for delivering it to the intended recipient, any disclosure,
copying, distribution, or other use of any of the information in this
transmission is strictly prohibited. If you have received this transmission
in error, please notify me immediately by reply e-mail or by telephone at
704-687-1943. Thank you.
On Fri, Aug 21, 2020 at 8:44 AM Martin Pecka <[email protected]> wrote:
> [*Caution*: Email from External Sender. Do not click or open links or
> attachments unless you know this sender.]
>
> Hi, we're thinking about getting GLX support on our HPC cluster which
> (currently) is completely headless. The idea is that users should be able
> to run virtual containers which would be given access to HW rendering with
> OpenGL. EGL would be better, but we're stuck with OGRE rendering engine
> which doesn't have proper support for the nvidia EGL.
>
> Could you comment on my idea? Is it a supported scenario?
>
> The multi-GPU server would run a single "3D X server", probably Xorg. It
> would also run the virtualgl client. Containers that want to do some OpenGL
> stuff would call a combination of xvfb and vglrun. I.e. the whole setup
> only works with a single machine, not a pair connected via ssh -X.
>
> Is that possible? Is there a tutorial for this kind of setup?
>
> Thanks for help
>
> --
> You received this message because you are subscribed to the Google Groups
> "VirtualGL User Discussion/Support" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to [email protected].
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/virtualgl-users/d4256049-e3e7-40ab-97b8-634fc69a38e4n%40googlegroups.com
> <https://groups.google.com/d/msgid/virtualgl-users/d4256049-e3e7-40ab-97b8-634fc69a38e4n%40googlegroups.com?utm_medium=email&utm_source=footer>
> .
>
--
You received this message because you are subscribed to the Google Groups
"VirtualGL User Discussion/Support" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
To view this discussion on the web visit
https://groups.google.com/d/msgid/virtualgl-users/CAAR6MGBs_R9R4qtm1Y%3D4c9BkdR4dv9UigPs0YrMqBk3U%3Df_P6w%40mail.gmail.com.
# lightdm is probably not needed. The server can run the multi-user target
without a GUI.
systemctl set-default multi-user.target
# We need both 32bit and 64bit virtualGL if the application is 32bit.
rpm -e VirtualGL --allmatches
yum -y install VirtualGL.x86_64 VirtualGL.i386
##
# install the nvidia driver from elrepo
# Remove ocl-id because it conflicts with the nvidia driver because of OpenCL
yum -y remove ocl-icd.x86_64
yum -y install yum-plugin-fastestmirror nvidia-detect
nvidia-detect | grep -qs kmod && yum -y install $(nvidia-detect)
# reboot for the nvidia drives to detect everything.
reboot
# Grab the bus ID for the nvidia card. this should similar to "PCI:1:0:0".
lspci can provide the same info, but the format is slighly different.
nvidia-xconfig shuold provide a format that is compatible with "Xorg
-isolateDevice"
bus=$(nvidia-xconfig --query-gpu-info|grep PCI|awk '{print $4}')
systemctl isolate multi-user.target
# Remove any previous config file runs
rm -f /root/xorg.conf.new
# move any primar DISPLAY=":0 config out of the way
mv /etc/X11/xorg.conf /etc/X11/xorg.conf.bak
Xorg :1 -configure -isolateDevice $bus
# xorg written as /root/xorg.conf.new
# this seems to fail sometimes. Just use nvidia-xconfig and move
/etc/X/xorg.conf to /etc/X/xorg.1.conf
# Run "Xorg :1 -config /root/xorg.conf.new" and check for errors. Ctrl-C to
cancel
mv -f /root/xorg.conf.new /etc/X11/xorg.1.conf
# Add the magic nvidia bits to xorg.conf
nvidia-xconfig -a --allow-empty-initial-configuration --busid=$bus
--use-display-device=none --virtual=1920x1200 -c /etc/X11/xorg.1.conf
#mv /etc/X11/xorg.conf /etc/X11/xorg.1.conf
### Set up a systemd service to manage Xorg Display :1
### begin headlessx.service ##### Just copy and paste into bash
cat > /etc/systemd/system/[email protected] <<EOF
[Unit]
Description=VirtualGL Headless X server on display %I
After=multi-user.target
[Service]
Type=simple
ExecStart=/bin/bash -c "Xorg :%i -listen tcp -config /etc/X11/xorg.%i.conf"
Restart=on-failure
[Install]
WantedBy=multi-user.target
EOF
### end headlessx.service #####
systemctl daemon-reload
systemctl enable headlessx@1
systemctl restart headlessx@1
# configure virtualGL to restrict users to all all users (DANGER). disable XTEST
/opt/VirtualGL/bin/vglserver_config -config -s +f -t
# refresh X display permissions as changed by vglserver_config
systemctl isolate multi-user.target
systemctl stop headlessx@1
rmmod nvidia_drm nvidia_modeset nvidia
systemctl start headlessx@1
cat > /etc/profile.d/virtualgl.sh <<EOF
# Configure the default DISPLAY for VirtualGL vglrun
export VGL_DISPLAY=':1'
EOF
reboot
# check xorg log for errors:
# Ignore error (EE) Failed to load module "glxservernvidia" (module does not
exist, 0)
grep EE /var/log/Xorg.1.log
######################### end instructions ##############################
##### additional troubleshooting steps/notes #####
######## example commands below for running VirtualGL as a normal user. This is
intended for use with StarNet FastX.
VGL_DISPLAY=:1 vglrun -c proxy +v /opt/VirtualGL/bin/glxspheres64
