[Bug 1791220] Re: increased crash rate since 10.3 upgrade is available
** Changed in: open-vm-tools Status: Unknown => Fix Released -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1791220 Title: increased crash rate since 10.3 upgrade is available To manage notifications about this bug go to: https://bugs.launchpad.net/open-vm-tools/+bug/1791220/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1791220] Re: increased crash rate since 10.3 upgrade is available
This bug was fixed in the package open-vm-tools - 2:10.3.0-0ubuntu1~18.04.2 --- open-vm-tools (2:10.3.0-0ubuntu1~18.04.2) bionic; urgency=medium * d/p/ubuntu/lp-1791220-Disable-hgfsServer-not-VMware.patch: avoid crashing with segfaults when force starting the service in non VMWare environments. (LP: #1791220) * d/p/debian/scsi-udev-rule: fix applying of the scsi timeout (LP: #1790145) - revert "update to match changes in 10.3 (DEVTYPE checkadded upstream)" - refresh patch context to match upstream version 10.3 -- Christian Ehrhardt Mon, 10 Sep 2018 09:33:13 +0200 ** Changed in: open-vm-tools (Ubuntu Bionic) Status: Fix Committed => Fix Released -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1791220 Title: increased crash rate since 10.3 upgrade is available To manage notifications about this bug go to: https://bugs.launchpad.net/open-vm-tools/+bug/1791220/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1791220] Re: increased crash rate since 10.3 upgrade is available
Before Fix root@b-vmtools:~# vmtoolsd Segmentation fault (core dumped) Upgrade package (Reading database ... 28495 files and directories currently installed.) Preparing to unpack open-vm-tools_10.3.0-0ubuntu1~18.04.2_amd64.deb ... Unpacking open-vm-tools (2:10.3.0-0ubuntu1~18.04.2) over (2:10.3.0-0ubuntu1~18.04.1) ... Setting up open-vm-tools (2:10.3.0-0ubuntu1~18.04.2) ... Processing triggers for libc-bin (2.27-3ubuntu1) ... Processing triggers for ureadahead (0.100.0-20) ... Processing triggers for systemd (237-3ubuntu10.3) ... Processing triggers for man-db (2.8.3-2) ... After Fix root@b-vmtools:~# vmtoolsd root@b-vmtools:~# dpkg -l open-vm-tools | tee Working as planned on non-vmware platforms now ** Tags removed: verification-needed verification-needed-bionic ** Tags added: verification-done verification-done-bionic -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1791220 Title: increased crash rate since 10.3 upgrade is available To manage notifications about this bug go to: https://bugs.launchpad.net/open-vm-tools/+bug/1791220/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1791220] Re: increased crash rate since 10.3 upgrade is available
Hello , or anyone else affected, Accepted open-vm-tools into bionic-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/open-vm- tools/2:10.3.0-0ubuntu1~18.04.2 in a few hours, and then in the -proposed repository. Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users. If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested and change the tag from verification-needed-bionic to verification-done-bionic. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-bionic. In either case, without details of your testing we will not be able to proceed. Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping! N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days. ** Changed in: open-vm-tools (Ubuntu Bionic) Status: Triaged => Fix Committed ** Tags added: verification-needed verification-needed-bionic -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1791220 Title: increased crash rate since 10.3 upgrade is available To manage notifications about this bug go to: https://bugs.launchpad.net/open-vm-tools/+bug/1791220/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1791220] Re: increased crash rate since 10.3 upgrade is available
- SRU Template complete - Cosmic fixed and confirmed (I retested on vSphere) - Since other than the version in changelog latest -dev and LTS are kept in sync the MP we had for cosmic essentially applies here as well. Pushing the same fix to Bionic, now available in -unapproved for the SRU Teams consideration. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1791220 Title: increased crash rate since 10.3 upgrade is available To manage notifications about this bug go to: https://bugs.launchpad.net/open-vm-tools/+bug/1791220/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1791220] Re: increased crash rate since 10.3 upgrade is available
** Also affects: open-vm-tools (Ubuntu Bionic) Importance: Undecided Status: New ** Changed in: open-vm-tools (Ubuntu Bionic) Status: New => Triaged -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1791220 Title: increased crash rate since 10.3 upgrade is available To manage notifications about this bug go to: https://bugs.launchpad.net/open-vm-tools/+bug/1791220/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1791220] Re: increased crash rate since 10.3 upgrade is available
This bug was fixed in the package open-vm-tools - 2:10.3.0-0ubuntu2 --- open-vm-tools (2:10.3.0-0ubuntu2) cosmic; urgency=medium * d/p/ubuntu/lp-1791220-Disable-hgfsServer-not-VMware.patch: avoid crashing with segfaults when force starting the service in non VMWare environments. (LP: #1791220) * d/p/debian/scsi-udev-rule: fix applying of the scsi timeout (LP: #1790145) - revert "update to match changes in 10.3 (DEVTYPE checkadded upstream)" - refresh patch context to match upstream version 10.3 -- Christian Ehrhardt Mon, 10 Sep 2018 09:33:13 +0200 ** Changed in: open-vm-tools (Ubuntu) Status: In Progress => Fix Released -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1791220 Title: increased crash rate since 10.3 upgrade is available To manage notifications about this bug go to: https://bugs.launchpad.net/open-vm-tools/+bug/1791220/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1791220] Re: increased crash rate since 10.3 upgrade is available
With the insight so far we can remove the systemd task again. ** No longer affects: systemd (Ubuntu) -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1791220 Title: increased crash rate since 10.3 upgrade is available To manage notifications about this bug go to: https://bugs.launchpad.net/open-vm-tools/+bug/1791220/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1791220] Re: increased crash rate since 10.3 upgrade is available
** Changed in: open-vm-tools (Ubuntu) Status: New => In Progress ** Changed in: open-vm-tools (Ubuntu) Assignee: (unassigned) => Christian Ehrhardt (paelzer) -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1791220 Title: increased crash rate since 10.3 upgrade is available To manage notifications about this bug go to: https://bugs.launchpad.net/open-vm-tools/+bug/1791220/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1791220] Re: increased crash rate since 10.3 upgrade is available
** Description changed: + [Impact] + + * The new version of open-vm-tools we recently SRUed had an issue +that if started in non-VMWare environments (protected by systemd +condition-virtualization, but people fake VMware envs) would segfault. + + * It is fine to "not work there" but segfaulting is too hard + + * Upstream realized the issue, this is backporting the fix + + [Test Case] + + * Use a non-VMWare guest (e.g. a test system lxd container) and run + $ vmtoolsd + + * This will segfault, with the fix it will have a clean exit. + + [Regression Potential] + + * All of this only affects non-VMWare environments in a tool only needed +in those environments - so the risk should be low. It is closing a +corner case of faked environments to keep bug reports and crashes down. +I can think of a regression if the detection of the VMContext would be +wrong and therefore the service not properly initializing anymore in +systems it should do so - so far such an issue was not seen, btu that +would be the most likely potential regression. + + [Other Info] + + * n/a + + + --- + Hi, Crash reports like: vmtoolsd (11) /usr/lib/libvmtools.so.0.0.0+2839c → [stack]+1ecfc → /usr/lib/libvmtools.so.0.0.0+4eb05 → /usr/lib/open-vm-tools/plugins/common/libhgfsServer.so+ccf → /usr/bin/vmtoolsd+5e98 → /usr/bin/vmtoolsd+4ce3 → /usr/bin/vmtoolsd+3ea8 The cmdline of the failing process had: /usr/bin/vmtoolsd -n vmusr --blockFd 3 I have a disassembly thou which looks like: => 0x7fedb35b739c :in (%dx),%eax 0x7fedb35b739d : xchg %rax,(%rsp) 0x7fedb35b73a1 : mov %rdi,0x28(%rax) 0x7fedb35b73a5 : mov %rsi,0x20(%rax) 0x7fedb35b73a9 : mov %rdx,0x18(%rax) 0x7fedb35b73ad : mov %rcx,0x10(%rax) 0x7fedb35b73b1 : mov %rbx,0x8(%rax) 0x7fedb35b73b5 : popq (%rax) 0x7fedb35b73b7 : pop %rbx - 0x7fedb35b73b8 : retq + 0x7fedb35b73b8 : retq 0x7fedb35b73b9: nopl 0x0(%rax) 0x7fedb35b73c0 :push %rbx 0x7fedb35b73c1 : mov %rdi,%rax 0x7fedb35b73c4 : push %rbp 0x7fedb35b73c5 : push %rax 0x7fedb35b73c6 : mov 0x30(%rax),%rbp We don't have much more yet. I filed an issue upstream [2] last week in case this showed up somewhere else as well, but haven't seen any response yet. Crash reports overview (might need ubuntu signin) are at [3] and one example is [4] [1]: https://bugs.launchpad.net/ubuntu/+source/open-vm-tools/+bug/1784638 [2]: https://github.com/vmware/open-vm-tools/issues/284 [3]: https://errors.ubuntu.com/?release=Ubuntu%2018.04=open-vm-tools=week=2%3A10.3.0-0ubuntu1~18.04.1 [4]: https://errors.ubuntu.com/problem/bff98ff07d389a1a92b6cd538a823c1e2ce1a193 -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1791220 Title: increased crash rate since 10.3 upgrade is available To manage notifications about this bug go to: https://bugs.launchpad.net/open-vm-tools/+bug/1791220/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1791220] Re: increased crash rate since 10.3 upgrade is available
** Merge proposal linked: https://code.launchpad.net/~paelzer/ubuntu/+source/open-vm-tools/+git/open-vm-tools/+merge/354555 -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1791220 Title: increased crash rate since 10.3 upgrade is available To manage notifications about this bug go to: https://bugs.launchpad.net/open-vm-tools/+bug/1791220/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1791220] Re: increased crash rate since 10.3 upgrade is available
I tried backporting the upstream fix that came up in our discussion. It fixes the crash back to how it was in Xenial (no segfault). There still won't be much reason to run in non-VMWare envoronments, but this isn't what this was about. It is working just fine. Preparing a MP to review and a cosmic SRU template for Bionic later on ... -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1791220 Title: increased crash rate since 10.3 upgrade is available To manage notifications about this bug go to: https://bugs.launchpad.net/open-vm-tools/+bug/1791220/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1791220] Re: increased crash rate since 10.3 upgrade is available
I have an idea actually, these are setups faking the VMWare ID. Systemd checks dmidecode values which could be faked in other hipervisors. There the tool would start and then trigger the crash. IMHO - the solution is make the tool exit, but not crash -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1791220 Title: increased crash rate since 10.3 upgrade is available To manage notifications about this bug go to: https://bugs.launchpad.net/open-vm-tools/+bug/1791220/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1791220] Re: increased crash rate since 10.3 upgrade is available
We want to fix it in open-vm-tools but I also added a sytemd bug task to get more opinions. @systemd As ConditionVirtualization=vmware should have prevented the issue in the first place, is there any rumor or known issues of that not being reliable? ** Also affects: systemd (Ubuntu) Importance: Undecided Status: New -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1791220 Title: increased crash rate since 10.3 upgrade is available To manage notifications about this bug go to: https://bugs.launchpad.net/open-vm-tools/+bug/1791220/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1791220] Re: increased crash rate since 10.3 upgrade is available
Oh well we can. Debugging from what I learned off the ASM I found the following TL;DR that I put on the upstream bug: TL;DR: - the bug is trying to run vmtoolsd on non vmware systems - it is stupid and ConditionVirtualization=vmware should prevent that issue from existing - Old versions (10.2) of vmtoolsd did NOT segfault on this - Could we make this a non segfault again? Steps to reproduce: - take a non VMware guest - run /usr/bin/vmtoolsd on open-vm-tools 10.3 Waiting for upstreams response to that ... -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1791220 Title: increased crash rate since 10.3 upgrade is available To manage notifications about this bug go to: https://bugs.launchpad.net/open-vm-tools/+bug/1791220/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1791220] Re: increased crash rate since 10.3 upgrade is available
Code isn't generated:
open-vm-tools/lib/backdoor/backdoorGcc64.c
Backdoor_InOut(Backdoor_proto *myBp) // IN/OUT
{
uint64 dummy;
__asm__ __volatile__(
#ifdef __APPLE__
/*
* Save %rbx on the stack because the Mac OS GCC doesn't want us to
* clobber it - it erroneously thinks %rbx is the PIC register.
* (Radar bug 7304232)
*/
"pushq %%rbx" "\n\t"
#endif
"pushq %%rax" "\n\t"
"movq 40(%%rax), %%rdi" "\n\t"
"movq 32(%%rax), %%rsi" "\n\t"
"movq 24(%%rax), %%rdx" "\n\t"
"movq 16(%%rax), %%rcx" "\n\t"
"movq 8(%%rax), %%rbx" "\n\t"
"movq (%%rax), %%rax" "\n\t"
"inl %%dx, %%eax" "\n\t" /* NB: There is no inq instruction */
"xchgq %%rax, (%%rsp)" "\n\t"
"movq %%rdi, 40(%%rax)" "\n\t"
"movq %%rsi, 32(%%rax)" "\n\t"
"movq %%rdx, 24(%%rax)" "\n\t"
"movq %%rcx, 16(%%rax)" "\n\t"
"movq %%rbx, 8(%%rax)" "\n\t"
"popq (%%rax)" "\n\t"
#ifdef __APPLE__
"popq %%rbx""\n\t"
#endif
: "=a" (dummy)
: "0" (myBp)
/*
* vmware can modify the whole VM state without the compiler knowing
* it. So far it does not modify EFLAGS. --hpreg
*/
:
#ifndef __APPLE__
/* %rbx is unchanged at the end of the function on Mac OS. */
"rbx",
#endif
"rcx", "rdx", "rsi", "rdi", "memory"
);
}
Last functional change in 2011, so it isn't a change to the crashing
code itself.
The failing instruction is the port-io hypercall
See: http://www.felixcloutier.com/x86/IN.html
It does a 32bit read to EAX from source port defined in DX.
DX is lowest 16 bit of RDX
We have RDX from the crash: rdx 0x775658511101130328
It is that
The error being "dx" suggests that the source port would be wrong?
That would be 0x5658 or in decimal 22104
I don't know the VMware Hipercall interface - is that source port
reasonable?
That is not getting us anywhere without help I assume :-/
I'll post this reading of the call to the upstream bug.
The question is what happens on this hypercall and/or the call to it that
triggers the bug?
I'm afraid we have to wait for a reproducer due to upstream work on it
or one of the affected chiming in here.
--
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/1791220
Title:
increased crash rate since 10.3 upgrade is available
To manage notifications about this bug go to:
https://bugs.launchpad.net/open-vm-tools/+bug/1791220/+subscriptions
--
ubuntu-bugs mailing list
ubuntu-bugs@lists.ubuntu.com
https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1791220] Re: increased crash rate since 10.3 upgrade is available
Not bad: https://www.thezdi.com/blog/2017/8/1/pythonizing-the-vmware-backdoor https://sites.google.com/site/chitchatvmback/backdoor I think VMWare can reconstruct the actual call, lets add them the full regs. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1791220 Title: increased crash rate since 10.3 upgrade is available To manage notifications about this bug go to: https://bugs.launchpad.net/open-vm-tools/+bug/1791220/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1791220] Re: increased crash rate since 10.3 upgrade is available
>From the upstream bug:
Can someone who has seen this and make this happen obtain tools logs with the
HGFS server logging enabled?
How to set up the logging is as follows:
Create the following file:
/etc/vmware-tools/tools.conf
Edit it with the following details (modified to your file location of choice):
[logging]
log = true
vmtoolsd.level = debug
vmtoolsd.handler = file
vmtoolsd.data = c:/tmp/vmtoolsd.${USER}.log
vmsvc.level = debug
vmsvc.handler = file
vmsvc.data = c:/tmp/vmsvc.log
dndcp.level = debug
dndcp.handler = file
dndcp.data = c:/tmp/dndcp.log
vmusr.level = debug
vmusr.handler = file
vmusr.data = c:/tmp/vmusr.${USER}.log
hgfsServer.level = debug
hgfsd.level = debug
No need to restart any of the tools services, just check that the files start
to appear.
Do whatever steps are required to enable the crash to occur.
Update the tools log files here.
** Bug watch added: github.com/vmware/open-vm-tools/issues #284
https://github.com/vmware/open-vm-tools/issues/284
** Also affects: open-vm-tools via
https://github.com/vmware/open-vm-tools/issues/284
Importance: Unknown
Status: Unknown
--
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/1791220
Title:
increased crash rate since 10.3 upgrade is available
To manage notifications about this bug go to:
https://bugs.launchpad.net/open-vm-tools/+bug/1791220/+subscriptions
--
ubuntu-bugs mailing list
ubuntu-bugs@lists.ubuntu.com
https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1791220] Re: increased crash rate since 10.3 upgrade is available
Tagged regression update as that is what it seems. I still lack a reproducer of any kind, so if one affected by this bug finds this bug while searching for it, please help to fix this by sharing what exactly was done before hitting this issue. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1791220 Title: increased crash rate since 10.3 upgrade is available To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/open-vm-tools/+bug/1791220/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
