[Kernel-packages] [Bug 1470250] Re: [Hyper-V] Ubuntu 14.04.2 LTS Generation 2 SCSI Errors on VSS Based Backups

** Tags added: cscc -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1470250 Title: [Hyper-V] Ubuntu 14.04.2 LTS Generation 2 SCSI Errors on VSS Based Backups Status in linux package

[Kernel-packages] [Bug 1470250] Re: [Hyper-V] Ubuntu 14.04.2 LTS Generation 2 SCSI Errors on VSS Based Backups

** CVE added: https://cve.mitre.org/cgi-bin/cvename.cgi?name=2019-3462 -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1470250 Title: [Hyper-V] Ubuntu 14.04.2 LTS Generation 2 SCSI

[Kernel-packages] [Bug 1470250] Re: [Hyper-V] Ubuntu 14.04.2 LTS Generation 2 SCSI Errors on VSS Based Backups

** Changed in: linux (Ubuntu Trusty) Status: In Progress => Won't Fix -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1470250 Title: [Hyper-V] Ubuntu 14.04.2 LTS Generation 2 SCSI

[Kernel-packages] [Bug 1470250] Re: [Hyper-V] Ubuntu 14.04.2 LTS Generation 2 SCSI Errors on VSS Based Backups

This bug was fixed in the package linux - 4.4.0-75.96 --- linux (4.4.0-75.96) xenial; urgency=low * linux: 4.4.0-75.96 -proposed tracker (LP: #1684441) * [Hyper-V] hv: util: move waiting for release to hv_utils_transport itself (LP: #1682561) - Drivers: hv: util: move

[Kernel-packages] [Bug 1470250] Re: [Hyper-V] Ubuntu 14.04.2 LTS Generation 2 SCSI Errors on VSS Based Backups

This bug was fixed in the package linux - 4.8.0-49.52 --- linux (4.8.0-49.52) yakkety; urgency=low * linux: 4.8.0-49.52 -proposed tracker (LP: #1684427) * [Hyper-V] hv: util: move waiting for release to hv_utils_transport itself (LP: #1682561) - Drivers: hv: util: move

[Kernel-packages] [Bug 1470250] Re: [Hyper-V] Ubuntu 14.04.2 LTS Generation 2 SCSI Errors on VSS Based Backups

This bug is awaiting verification that the kernel in -proposed solves the problem. Please test the kernel and update this bug with the results. If the problem is solved, change the tag 'verification-needed- yakkety' to 'verification-done-yakkety'. If the problem still exists, change the tag

[Kernel-packages] [Bug 1470250] Re: [Hyper-V] Ubuntu 14.04.2 LTS Generation 2 SCSI Errors on VSS Based Backups

This bug is awaiting verification that the kernel in -proposed solves the problem. Please test the kernel and update this bug with the results. If the problem is solved, change the tag 'verification-needed- xenial' to 'verification-done-xenial'. If the problem still exists, change the tag

[Kernel-packages] [Bug 1470250] Re: [Hyper-V] Ubuntu 14.04.2 LTS Generation 2 SCSI Errors on VSS Based Backups

The fixes for Xenial, Yakkety and Zesty are committed or released. However, I built one more Trusty test kernel with all of the prereq commits and the patches. Would it be possible to have this kernel tested one more time before it is SRU'd? It would be good to confirm no regressions are

[Kernel-packages] [Bug 1470250] Re: [Hyper-V] Ubuntu 14.04.2 LTS Generation 2 SCSI Errors on VSS Based Backups

Trusty requires quite a few prereq commits(See comment #353), so it's SRU will be a little behind. However, it will be SRU'd shortly as well. -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu.

[Kernel-packages] [Bug 1470250] Re: [Hyper-V] Ubuntu 14.04.2 LTS Generation 2 SCSI Errors on VSS Based Backups

How about rusty? -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1470250 Title: [Hyper-V] Ubuntu 14.04.2 LTS Generation 2 SCSI Errors on VSS Based Backups Status in linux package in

[Kernel-packages] [Bug 1470250] Re: [Hyper-V] Ubuntu 14.04.2 LTS Generation 2 SCSI Errors on VSS Based Backups

** Changed in: linux (Ubuntu Zesty) Status: In Progress => Fix Released -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1470250 Title: [Hyper-V] Ubuntu 14.04.2 LTS Generation 2

[Kernel-packages] [Bug 1470250] Re: [Hyper-V] Ubuntu 14.04.2 LTS Generation 2 SCSI Errors on VSS Based Backups

** Changed in: linux (Ubuntu Xenial) Status: In Progress => Fix Committed ** Changed in: linux (Ubuntu Yakkety) Status: In Progress => Fix Committed -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu.

[Kernel-packages] [Bug 1470250] Re: [Hyper-V] Ubuntu 14.04.2 LTS Generation 2 SCSI Errors on VSS Based Backups

Sorry guys I had been swamped with other stuff and then simply forgot to test @alexng's patch I ran it overnight on my original test setup and it also worked for me. -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu.

[Kernel-packages] [Bug 1470250] Re: [Hyper-V] Ubuntu 14.04.2 LTS Generation 2 SCSI Errors on VSS Based Backups

The patch from @faulpeltz hasn't been mainlined because of some feedback that it shouldn't have to close and reopen the /dev/vmbus/hv_vss device after failure. I addressed this comment this in a modified version of @faulpeltz's patch (see comment #345). I haven't heard from @faulpeltz whether

[Kernel-packages] [Bug 1470250] Re: [Hyper-V] Ubuntu 14.04.2 LTS Generation 2 SCSI Errors on VSS Based Backups

By the way, the "operation times should match host expectation" patch in #343 (and listed in #347) is upstream: commit b357fd3908c1191f2f56e38aa77f2aecdae18bc8 Author: Alex Ng Date: Sun Nov 6 13:14:11 2016 -0800 Drivers: hv: vss: Operation timeouts should

[Kernel-packages] [Bug 1470250] Re: [Hyper-V] Ubuntu 14.04.2 LTS Generation 2 SCSI Errors on VSS Based Backups

That patch was submitted upstream under the title "[PATCH] Tools: hv: recover after hv_vss_daemon freeze times out" but I don't see that it was committed. I'll poke around. -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu.

[Kernel-packages] [Bug 1470250] Re: [Hyper-V] Ubuntu 14.04.2 LTS Generation 2 SCSI Errors on VSS Based Backups

The patch from @faulpeltz posted in comment #345 has never landed in mainline. Is there plans for this patch to be sent upstream, or do we just want to include it as a SAUCE patch? -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in

[Kernel-packages] [Bug 1470250] Re: [Hyper-V] Ubuntu 14.04.2 LTS Generation 2 SCSI Errors on VSS Based Backups

I can confirm that the trusty patches are preventing the filesystem failure. -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1470250 Title: [Hyper-V] Ubuntu 14.04.2 LTS Generation 2 SCSI

[Kernel-packages] [Bug 1470250] Re: [Hyper-V] Ubuntu 14.04.2 LTS Generation 2 SCSI Errors on VSS Based Backups

Just waiting on test results for Trusty per comments #352 and #353. It sounds like the test kernels resolved the issue? If that is the case, I'll submit the SRU request. -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu.

[Kernel-packages] [Bug 1470250] Re: [Hyper-V] Ubuntu 14.04.2 LTS Generation 2 SCSI Errors on VSS Based Backups

Hi @jsalisbury, Any status update on the patches for this issue? It appears the test kernels have resolved the issue. Let us know if you need additional testing or have questions about the patches. Thanks, Alex -- You received this bug notification because you are a member of Kernel

[Kernel-packages] [Bug 1470250] Re: [Hyper-V] Ubuntu 14.04.2 LTS Generation 2 SCSI Errors on VSS Based Backups

So I'm the OP on this bug report from what seems like ages ago... While a lot of hard work as been done by some of the members posting here to try and replicate it, it's been very difficult to track the specific event that has been causing this.While we "fixed" our problem a long time ago by

[Kernel-packages] [Bug 1470250] Re: [Hyper-V] Ubuntu 14.04.2 LTS Generation 2 SCSI Errors on VSS Based Backups

Yeah an official fix would be nice. I am tired of getting up early to see if I have to reboot any vm's :) -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1470250 Title: [Hyper-V] Ubuntu

[Kernel-packages] [Bug 1470250] Re: [Hyper-V] Ubuntu 14.04.2 LTS Generation 2 SCSI Errors on VSS Based Backups

It's been more than 1.5 years! When can we expect this fix to be released? $ uname -a Linux Server123 3.13.0-107-generic #154-Ubuntu SMP Tue Dec 20 09:57:27 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux $ lsb_release -a No LSB modules are available. Distributor ID: Ubuntu Description:Ubuntu

[Kernel-packages] [Bug 1470250] Re: [Hyper-V] Ubuntu 14.04.2 LTS Generation 2 SCSI Errors on VSS Based Backups

3.13.0-103-generic #150~lp1470250 SMP looks promising. Several days without crash. I encountered some IO errors but the filesystem remains rw. [Wed Jan 11 13:47:35 2017] hv_storvsc vmbus_0_1: cmd 0x35 scsi status 0x2 srb status 0x82 [Wed Jan 11 13:47:35 2017] hv_storvsc vmbus_0_1: stor pkt

[Kernel-packages] [Bug 1470250] Re: [Hyper-V] Ubuntu 14.04.2 LTS Generation 2 SCSI Errors on VSS Based Backups

Microsoft will test the trusty kernel in the coming days. The Xenial kernel may be getting all the patches from the 4.9 rebase that's currently in progress. -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu.

[Kernel-packages] [Bug 1470250] Re: [Hyper-V] Ubuntu 14.04.2 LTS Generation 2 SCSI Errors on VSS Based Backups

@Emsi and whoever can test Trusty. There is a test kernel available at: http://kernel.ubuntu.com/~jsalisbury/lp1470250/trusty This test kernel has the updated 3 patches. It also required 19 prerequisite patches. -- You received this bug notification because you are a member of Kernel

[Kernel-packages] [Bug 1470250] Re: [Hyper-V] Ubuntu 14.04.2 LTS Generation 2 SCSI Errors on VSS Based Backups

For those interested, the Trusty test kernel contains the following commits: e81d871 UBUNTU: SAUCE: (no-up) Tools: hv: vss: Thaw the filesystem and continue after freeze fails 01032c1 UBUNTU: SAUCE: (no-up) Drivers: hv: vss: Operation timeouts should match host expectation 4a6f680 Drivers: hv:

[Kernel-packages] [Bug 1470250] Re: [Hyper-V] Ubuntu 14.04.2 LTS Generation 2 SCSI Errors on VSS Based Backups

@Emsi, right now there are only Xenial and Yakkety test kernels with all the updated patches. I will have a Trusty test kernel available shortly. -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu.

[Kernel-packages] [Bug 1470250] Re: [Hyper-V] Ubuntu 14.04.2 LTS Generation 2 SCSI Errors on VSS Based Backups

The link to trusty kernels is gone :( http://kernel.ubuntu.com/~jsalisbury/lp1470250/AlexFaulpeltzPatch/trusty/ -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1470250 Title: [Hyper-V]

[Kernel-packages] [Bug 1470250] Re: [Hyper-V] Ubuntu 14.04.2 LTS Generation 2 SCSI Errors on VSS Based Backups

I built Yakkety and Xenial test kernels with the three patches mentioned in comment #347. It would be great if these test kernels can be tested. If they resolve the bug, I'll submit and SRU request to have them included in Yakkety and Xenial. The test kernels can be downloaded from: Xenial:

[Kernel-packages] [Bug 1470250] Re: [Hyper-V] Ubuntu 14.04.2 LTS Generation 2 SCSI Errors on VSS Based Backups

Thanks Joseph for compiling the list. The patches you've outlined should be sufficient. -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1470250 Title: [Hyper-V] Ubuntu 14.04.2 LTS

[Kernel-packages] [Bug 1470250] Re: [Hyper-V] Ubuntu 14.04.2 LTS Generation 2 SCSI Errors on VSS Based Backups

@faulpeltz, Thanks for the update. I'll build a test kernel with all the latest patches and backport them to prior releases. Just to confirm, so I build the test kernels with everything needed, we need the following: New patch from comment #343: Drivers: hv: vss: Operation timeouts should match

[Kernel-packages] [Bug 1470250] Re: [Hyper-V] Ubuntu 14.04.2 LTS Generation 2 SCSI Errors on VSS Based Backups

** Also affects: linux (Ubuntu Zesty) Importance: Critical Assignee: Joseph Salisbury (jsalisbury) Status: In Progress -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1470250

[Kernel-packages] [Bug 1470250] Re: [Hyper-V] Ubuntu 14.04.2 LTS Generation 2 SCSI Errors on VSS Based Backups

@Alex I will try as soon as I have some spare time @jsalisbury Unfortunately I didnt have time to test the kernel from #341. Including the #343 upstream commit should take care of our issue, and using the patch from #345 (replacing my initial patch) should prevent any hv_vss_daemon crashes in

[Kernel-packages] [Bug 1470250] Re: [Hyper-V] Ubuntu 14.04.2 LTS Generation 2 SCSI Errors on VSS Based Backups

@faulpeltz I agree. Your patch should also be included in case a FREEZE operation does exceed the increased timeout. I've attached a modified version of your patch that addresses some concerns we had offline. Could you give it a try? ** Patch added:

[Kernel-packages] [Bug 1470250] Re: [Hyper-V] Ubuntu 14.04.2 LTS Generation 2 SCSI Errors on VSS Based Backups

@Alex The modified timeout should take care of the issue, but I think its a good idea for the VSS daemon to issue a THAW before either exiting or trying to recover -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu.

[Kernel-packages] [Bug 1470250] Re: [Hyper-V] Ubuntu 14.04.2 LTS Generation 2 SCSI Errors on VSS Based Backups

I'll let @jsalisbury comment on the status of his backported patches. @jsalisbury, I'd also take this recently submitted patch from upstream kernel as well. It ensures that the VSS driver doesn't timeout any long running FREEZE operations too early. This should preclude the need for faulpeltz's

[Kernel-packages] [Bug 1470250] Re: [Hyper-V] Ubuntu 14.04.2 LTS Generation 2 SCSI Errors on VSS Based Backups

Any news on this? Recently it seemed like a fix was found, but no update for about a month now. Thanks guys! -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1470250 Title: [Hyper-V]

[Kernel-packages] [Bug 1470250] Re: [Hyper-V] Ubuntu 14.04.2 LTS Generation 2 SCSI Errors on VSS Based Backups

I was able to backport the two patches to Trusty. It required quite a few prerequisite commits. The following commits were backported to a Trusty test kernel: f9cc88d lp1470250: patch from faulpeltz 6f3e8a2 Drivers: hv: utils: Continue to poll VSS channel after handling requests. 725a85d

[Kernel-packages] [Bug 1470250] Re: [Hyper-V] Ubuntu 14.04.2 LTS Generation 2 SCSI Errors on VSS Based Backups

Thanks @faulpeltz for the info. I sent you a private message with the list of maintainers to send the patch (trying to avoid pasting it here in case spam bots can go through this archive). -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to

[Kernel-packages] [Bug 1470250] Re: [Hyper-V] Ubuntu 14.04.2 LTS Generation 2 SCSI Errors on VSS Based Backups

@Alex 1) Yes, but I might need some help with that. Which list/maintainer should I submit it to? 2) On our test machine, with both the hyper-v host as well as the guest under heavy i/o load, it was a few hundred ms but with high variance and spiking (quite often) in to the 2-4s range, with

[Kernel-packages] [Bug 1470250] Re: [Hyper-V] Ubuntu 14.04.2 LTS Generation 2 SCSI Errors on VSS Based Backups

Hi @faulpeltz, A few questions/comments about your patch: 1) Can you submit your patch to the upstream kernel? 2) Under load, were you able to measure how long the FIFREEZE operation took before it succeeded? I'm trying to see if we can increase the timeout of the kernel driver before it hits

[Kernel-packages] [Bug 1470250] Re: [Hyper-V] Ubuntu 14.04.2 LTS Generation 2 SCSI Errors on VSS Based Backups

There should be a Trusty test kernel soon. I've had to identify 13 prereq commits to get the two patches to apply and build properly. For Xenial and newer, there has been positive test results from Emsi and faulpeltz with Alexs' patch and the one from faulpeltz. Are we comfortable with

[Kernel-packages] [Bug 1470250] Re: [Hyper-V] Ubuntu 14.04.2 LTS Generation 2 SCSI Errors on VSS Based Backups

** No longer affects: linux (Ubuntu Wily) ** No longer affects: linux (Ubuntu Vivid) -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1470250 Title: [Hyper-V] Ubuntu 14.04.2 LTS

[Kernel-packages] [Bug 1470250] Re: [Hyper-V] Ubuntu 14.04.2 LTS Generation 2 SCSI Errors on VSS Based Backups

Thank you for the update. I'm staying tuned :) -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1470250 Title: [Hyper-V] Ubuntu 14.04.2 LTS Generation 2 SCSI Errors on VSS Based Backups

[Kernel-packages] [Bug 1470250] Re: [Hyper-V] Ubuntu 14.04.2 LTS Generation 2 SCSI Errors on VSS Based Backups

I'm still working on getting a Trusty test kernel build with both patches. Alexs' patch is requiring some prereq commits to work with Trusty. I should have a test kernel ready shortly. -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to

[Kernel-packages] [Bug 1470250] Re: [Hyper-V] Ubuntu 14.04.2 LTS Generation 2 SCSI Errors on VSS Based Backups

Thanks for the update. I'll rebuild the 14.04 test kernel, but this time with both patches. That is what the Xenial test kernel has. -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu.

[Kernel-packages] [Bug 1470250] Re: [Hyper-V] Ubuntu 14.04.2 LTS Generation 2 SCSI Errors on VSS Based Backups

No luck with 14.04. After couple of days two test machines restarted unexpectedly at the very same moment. I sustepct kernel crash but the virtual console is gone and nothing was logged. On the other hand the xenial machine is working for weeks now without any issues. -- You received this bug

[Kernel-packages] [Bug 1470250] Re: [Hyper-V] Ubuntu 14.04.2 LTS Generation 2 SCSI Errors on VSS Based Backups

Over 24h of trusty tests under heavy load and so good so far. IMHO the kernel patch should suffice. I don't see "trying to recover VSS connection" messages neither on 16.04 kernel nor 14.04. -- You received this bug notification because you are a member of Kernel Packages, which is subscribed

[Kernel-packages] [Bug 1470250] Re: [Hyper-V] Ubuntu 14.04.2 LTS Generation 2 SCSI Errors on VSS Based Backups

@faulpeltz, have had a chance to test the Trusty kernel, posted in comment #323? That kernel only has your patch and not Alex's. It would be good to know if both are needed. -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in

[Kernel-packages] [Bug 1470250] Re: [Hyper-V] Ubuntu 14.04.2 LTS Generation 2 SCSI Errors on VSS Based Backups

We moved some machines back to their regular backup schedule with the new kernel, no problems so far -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1470250 Title: [Hyper-V] Ubuntu

[Kernel-packages] [Bug 1470250] Re: [Hyper-V] Ubuntu 14.04.2 LTS Generation 2 SCSI Errors on VSS Based Backups

Er, not "filesystem system" but "filesystem consistent". -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1470250 Title: [Hyper-V] Ubuntu 14.04.2 LTS Generation 2 SCSI Errors on VSS Based

[Kernel-packages] [Bug 1470250] Re: [Hyper-V] Ubuntu 14.04.2 LTS Generation 2 SCSI Errors on VSS Based Backups

@trash1-z, Application-consistent backups will fall back to filesystem system backups as before as there is no API for informing Linux applications to complete operations for a backup. VSS has used filesystem freeze operations for years to do backups on Linux. Hyper-V integration services is

[Kernel-packages] [Bug 1470250] Re: [Hyper-V] Ubuntu 14.04.2 LTS Generation 2 SCSI Errors on VSS Based Backups

The xenial kernel works like a charm since the patch. Great work! :) I'm using application consistent backup (the Hyper-V signals snapshot to the guest OS) rather than crash consistent (snapshoting without informing the guest). The Hyper-h integration services in the guest are required for that

[Kernel-packages] [Bug 1470250] Re: [Hyper-V] Ubuntu 14.04.2 LTS Generation 2 SCSI Errors on VSS Based Backups

I was able to patch the kernel, however the fault still exists. I'm uncertain if issue is related to VSS, but in ballpark (see screenshots). Ubuntu 16.04.1 on same server smooth as silk. ** Attachment added: "Ubuntu Error 4c.png"

[Kernel-packages] [Bug 1470250] Re: [Hyper-V] Ubuntu 14.04.2 LTS Generation 2 SCSI Errors on VSS Based Backups

You can install the .deb packages by using the dpkg command from a terminal. For example: sudo dpkg -i linux- image-3.13.0-95-generic_3.13.0-95.142~lp1470250faulpeltzPatchBAckported_amd64.deb -- You received this bug notification because you are a member of Kernel Packages, which is subscribed

[Kernel-packages] [Bug 1470250] Re: [Hyper-V] Ubuntu 14.04.2 LTS Generation 2 SCSI Errors on VSS Based Backups

I'd love to. Can you advise how I do that? I'm a good typist, but cut n paste is so much better! -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1470250 Title: [Hyper-V] Ubuntu 14.04.2

[Kernel-packages] [Bug 1470250] Re: [Hyper-V] Ubuntu 14.04.2 LTS Generation 2 SCSI Errors on VSS Based Backups

I backported the patch from faulpeltz to trusty and built a test kernel. The test kernel can be downloaded from: http://kernel.ubuntu.com/~jsalisbury/lp1470250/trusty/ Can you give this kernel a test? -- You received this bug notification because you are a member of Kernel Packages, which is

[Kernel-packages] [Bug 1470250] Re: [Hyper-V] Ubuntu 14.04.2 LTS Generation 2 SCSI Errors on VSS Based Backups

I'm really looking forward to test the path on trusty. -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1470250 Title: [Hyper-V] Ubuntu 14.04.2 LTS Generation 2 SCSI Errors on VSS Based

[Kernel-packages] [Bug 1470250] Re: [Hyper-V] Ubuntu 14.04.2 LTS Generation 2 SCSI Errors on VSS Based Backups

Yes. Linux backup-01 4.4.0-36-generic #55~lp1470250PatchedAlexFaulpeltz SMP Wed Aug 31 16:14:16 UTC 20 x86_64 x86_64 x86_64 GNU/Linux Another 24h and no crash. -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu.

[Kernel-packages] [Bug 1470250] Re: [Hyper-V] Ubuntu 14.04.2 LTS Generation 2 SCSI Errors on VSS Based Backups

@Emsi, that is great news. Is this with the kernel that includes both the patch from Alex and the one from faulpeltz, which was posted in comment #317? -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu.

[Kernel-packages] [Bug 1470250] Re: [Hyper-V] Ubuntu 14.04.2 LTS Generation 2 SCSI Errors on VSS Based Backups

It looks like it does the trick! Already over 90h of testing with tiobench and it works like a charm! Usually it took less than 4h (several at best) to crash. -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu.

[Kernel-packages] [Bug 1470250] Re: [Hyper-V] Ubuntu 14.04.2 LTS Generation 2 SCSI Errors on VSS Based Backups

Alex's patch is in commit http://git.kernel.org/cgit/linux/kernel/git/next/linux- next.git/commit/?id=497af84b81b98b27e9ba7aebb8a373412e328497 -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu.

[Kernel-packages] [Bug 1470250] Re: [Hyper-V] Ubuntu 14.04.2 LTS Generation 2 SCSI Errors on VSS Based Backups

I built a Xenial test kernel with the first patch from Alex and the patch from faulpeltz. The test kernel can be downloaded from: http://kernel.ubuntu.com/~jsalisbury/lp1470250/AlexFaulpeltzPatch/xenial/ A little backporting is needed to get the patches to apply to Trusty, but I'll also post

[Kernel-packages] [Bug 1470250] Re: [Hyper-V] Ubuntu 14.04.2 LTS Generation 2 SCSI Errors on VSS Based Backups

@faulpeltz, that is great work creating the new patch. I'll build test kernels for all releases with the first patch from Alex and your patch posted in comment #313. -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu.

[Kernel-packages] [Bug 1470250] Re: [Hyper-V] Ubuntu 14.04.2 LTS Generation 2 SCSI Errors on VSS Based Backups

** Tags added: patch -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1470250 Title: [Hyper-V] Ubuntu 14.04.2 LTS Generation 2 SCSI Errors on VSS Based Backups Status in linux package

[Kernel-packages] [Bug 1470250] Re: [Hyper-V] Ubuntu 14.04.2 LTS Generation 2 SCSI Errors on VSS Based Backups

@Frederik: yes, as far as I have seen. The file systems are still frozen between FREEZE and THAW, which in the case of timeouts, is >10s, I have seen about 30s in some of our error cases. But they do recover. I only tried the patched 4.4.0-34 version for now, though Some testing would be

[Kernel-packages] [Bug 1470250] Re: [Hyper-V] Ubuntu 14.04.2 LTS Generation 2 SCSI Errors on VSS Based Backups

@faulpletz Do you mean with "and the guest systems continue to work normally" that the guest does not have any read-only problems any more with your patch? -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu.

[Kernel-packages] [Bug 1470250] Re: [Hyper-V] Ubuntu 14.04.2 LTS Generation 2 SCSI Errors on VSS Based Backups

I spent some time investigating our issue further. As far as I can tell, the main issue is that ioctl(FIFREEZE) can take a long time when running VSS backups, and the default timeout is 10s. This is very noticeable under load, with rare peaks of >5s seen, so 10s seem plausible If the timeout

[Kernel-packages] [Bug 1470250] Re: [Hyper-V] Ubuntu 14.04.2 LTS Generation 2 SCSI Errors on VSS Based Backups

One reason you may see the timeout messages is due to having a mismatch between the user-space hv_vss_daemon version and the kernel version. Can you rebuild the user-space hv_vss_daemon under the source tree's tools/hv directory and replace the one provided that's provided in Ubuntu by default?

[Kernel-packages] [Bug 1470250] Re: [Hyper-V] Ubuntu 14.04.2 LTS Generation 2 SCSI Errors on VSS Based Backups

And in response to Joseph's comment #309, the second patch shouldn't be required as it's related to a feature introduced in Windows Server 2016 (I'm assuming you folks are testing in Windows Server 2012 R2). -- You received this bug notification because you are a member of Kernel Packages, which

[Kernel-packages] [Bug 1470250] Re: [Hyper-V] Ubuntu 14.04.2 LTS Generation 2 SCSI Errors on VSS Based Backups

v4.8-rc2 (with patch 1/2 from #308) failed after 18h :( With the first patch applied, the VSS daemon decides to quit, but a THAW is missing after the FREEZE there are the usual syscall timeouts afterwards kernel: sd 2:0:0:0: [storvsc] Add. Sense: Changed operating definition kernel: sd 2:0:0:0:

[Kernel-packages] [Bug 1470250] Re: [Hyper-V] Ubuntu 14.04.2 LTS Generation 2 SCSI Errors on VSS Based Backups

I built a v4.8-rc2 test kernel with the first patch Alex mentions in comment #308. Can this kernel be tested to see if it still exhibits the bug? The test kernel can be downloaded from: http://kernel.ubuntu.com/~jsalisbury/lp1470250/081816-Patch/ Alex, do you think we need the second patch for

[Kernel-packages] [Bug 1470250] Re: [Hyper-V] Ubuntu 14.04.2 LTS Generation 2 SCSI Errors on VSS Based Backups

Might be worth trying this patchset: https://lkml.org/lkml/2016/8/18/859 The first patch in the set addresses some issues with VSS that would cause it to take a long time to initiate backup (and even may timeout). The second patch is not necessary (but will need VSS daemon to be replaced if you

[Kernel-packages] [Bug 1470250] Re: [Hyper-V] Ubuntu 14.04.2 LTS Generation 2 SCSI Errors on VSS Based Backups

unfortunately still the same problem using the v4.8 tools (on 4.4 and up it doesnt remount the filesystem read-only, it just hangs on any write operation) dmesg output: [30626.788513] hv_utils: VSS: timeout waiting for daemon to reply [30627.100164] hv_utils: VSS: Transaction not active

[Kernel-packages] [Bug 1470250] Re: [Hyper-V] Ubuntu 14.04.2 LTS Generation 2 SCSI Errors on VSS Based Backups

@faulpeltz, I created athe tools and uploaded them to: http://kernel.ubuntu.com/~jsalisbury/lp1470250/v4.8-rc2/ -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1470250 Title: [Hyper-V]

[Kernel-packages] [Bug 1470250] Re: [Hyper-V] Ubuntu 14.04.2 LTS Generation 2 SCSI Errors on VSS Based Backups

488347f seems stable after 25h 4.8-rc1 crashed/hung after ~7h, but I didnt have the 4.8 cloud tools (using the 4.4 ones), @jsalisbury: maybe you could build those? -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu.

[Kernel-packages] [Bug 1470250] Re: [Hyper-V] Ubuntu 14.04.2 LTS Generation 2 SCSI Errors on VSS Based Backups

@jsalisbury: Yes, as far as I can trust my results Currently running 488347f (which is reasonably close to 71425a9) If the current one doesnt fail I would test d215f91 next, then 71425a9 and the 4.8rc1 mainline -- You received this bug notification because you are a member of Kernel Packages,

[Kernel-packages] [Bug 1470250] Re: [Hyper-V] Ubuntu 14.04.2 LTS Generation 2 SCSI Errors on VSS Based Backups

@faulpeltz, so it sounds like we can confirm the latest commit to pass in 3.13.0-34 is 95d1181? If that is the case, it might be worthwhile for me to build a test kernel with d215f91 as tip of the tree and a second kernel with 71425a9 as the tip. Just let me know if you could test those two

[Kernel-packages] [Bug 1470250] Re: [Hyper-V] Ubuntu 14.04.2 LTS Generation 2 SCSI Errors on VSS Based Backups

Currently re-running a few test kernels. Current results: 3.13.0-35-generic #62-Commitd215f91Reverted: BAD 3.13.0-34-generic #61 @4c48c359b: GOOD 3.13.0-34-generic #61 @95d1181: GOOD -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux

[Kernel-packages] [Bug 1470250] Re: [Hyper-V] Ubuntu 14.04.2 LTS Generation 2 SCSI Errors on VSS Based Backups

@Tommy Eriksen, I believe the current status is that we have no idea. Every time when I think we are getting close on the cause of the problem, there comes proof that it is not the cause of the problems. For us it already critical for months. We are in the middle of moving our systems. I know

[Kernel-packages] [Bug 1470250] Re: [Hyper-V] Ubuntu 14.04.2 LTS Generation 2 SCSI Errors on VSS Based Backups

Hi Guys, Any news on the testing? We have an increasingly large number of servers failing after their backup run; it is getting a bit critical, unfortunately. Thanks a lot, Tommy -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in

[Kernel-packages] [Bug 1470250] Re: [Hyper-V] Ubuntu 14.04.2 LTS Generation 2 SCSI Errors on VSS Based Backups

Update (no real results yet) I tried to improve on the test cycle by stopping the hyper-v backup immediately after it has begun, then waiting until the delta disks have been merged back (rinse and repeat) It took some time to get stable but it seems to have a 6-7x speedup compared to the

[Kernel-packages] [Bug 1470250] Re: [Hyper-V] Ubuntu 14.04.2 LTS Generation 2 SCSI Errors on VSS Based Backups

I built a 3.13.0-35.62 kernel with d215f91 reverted. It is available here: http://kernel.ubuntu.com/~jsalisbury/lp1470250/d215f91-reverted/3.13.0-35/ Can you test this kernel when you have a chance? If it runs past the 80 hour mark, I'd say let it run for a week if you can. -- You received

[Kernel-packages] [Bug 1470250] Re: [Hyper-V] Ubuntu 14.04.2 LTS Generation 2 SCSI Errors on VSS Based Backups

The test kernel I built an post in comment #295 was Ubuntu-3.13.0-86.131 with d215f91 reverted. However, d215f91 was committed in Ubuntu-3.13.0-35.61. I wonder if it could be that the second bad commit is actually between 3.13.0-35.61 and 3.13.0-86.131. Can you continue to run 4c48c35 from the

[Kernel-packages] [Bug 1470250] Re: [Hyper-V] Ubuntu 14.04.2 LTS Generation 2 SCSI Errors on VSS Based Backups

Unfortunately, yes. I did not find any evidence of other Hyper-V/Host related problems. I will try to repro the crash at least once to be sure. -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu.

[Kernel-packages] [Bug 1470250] Re: [Hyper-V] Ubuntu 14.04.2 LTS Generation 2 SCSI Errors on VSS Based Backups

The Trusty test kernel with d215f91 reverted failed after 80 hours? That is the kernel that can be downloaded from here: http://kernel.ubuntu.com/~jsalisbury/lp1470250/d215f91-reverted/trusty/ -- You received this bug notification because you are a member of Kernel Packages, which is subscribed

[Kernel-packages] [Bug 1470250] Re: [Hyper-V] Ubuntu 14.04.2 LTS Generation 2 SCSI Errors on VSS Based Backups

Unfortunately it crashed after about 80 hours. I am currently running 4c48c35 from the original bisect (95h+) But things seem increasingly random at this point. I tweaked the IO load on the host and guest machine and it seems the crashes are now reproducible a bit faster, but I think we might

[Kernel-packages] [Bug 1470250] Re: [Hyper-V] Ubuntu 14.04.2 LTS Generation 2 SCSI Errors on VSS Based Backups

@faulpeltz, were you able to run the test kernel with d215f91 reverted for a long duration, like a week? -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1470250 Title: [Hyper-V] Ubuntu

[Kernel-packages] [Bug 1470250] Re: [Hyper-V] Ubuntu 14.04.2 LTS Generation 2 SCSI Errors on VSS Based Backups

i meant crashed after 12 hours *g* -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1470250 Title: [Hyper-V] Ubuntu 14.04.2 LTS Generation 2 SCSI Errors on VSS Based Backups Status in

[Kernel-packages] [Bug 1470250] Re: [Hyper-V] Ubuntu 14.04.2 LTS Generation 2 SCSI Errors on VSS Based Backups

3.14rc crashed after 3.14-rc1 However isn't the crash to be expected in all the mainline kernels >=3.14 since d215f91 has not been reverted in them? -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu.

[Kernel-packages] [Bug 1470250] Re: [Hyper-V] Ubuntu 14.04.2 LTS Generation 2 SCSI Errors on VSS Based Backups

Started run on 3.14-rc1 -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1470250 Title: [Hyper-V] Ubuntu 14.04.2 LTS Generation 2 SCSI Errors on VSS Based Backups Status in linux

[Kernel-packages] [Bug 1470250] Re: [Hyper-V] Ubuntu 14.04.2 LTS Generation 2 SCSI Errors on VSS Based Backups

That would be a great test. I will also be out next week, so it would be good timing for a week long run. The reverted test kernel can be downloaded from: http://kernel.ubuntu.com/~jsalisbury/lp1470250/d215f91-reverted/trusty/ Before starting that test, can you test the upstream v3.14-rc1

[Kernel-packages] [Bug 1470250] Re: [Hyper-V] Ubuntu 14.04.2 LTS Generation 2 SCSI Errors on VSS Based Backups

Both crashed, 3.15rc1 after about 20h, 3.14 only after 66h I am starting to wonder if it might be a good idea to run the good(?) 3.13.0-86(+revert) kernel for a week or two to make sure its actually good. -- You received this bug notification because you are a member of Kernel Packages, which

[Kernel-packages] [Bug 1470250] Re: [Hyper-V] Ubuntu 14.04.2 LTS Generation 2 SCSI Errors on VSS Based Backups

It's looking like the second bad commit came in the 3.14 or 3.15 kernel. Can you test the following two mainline kernels: 3.15-rc1: http://kernel.ubuntu.com/~kernel- ppa/mainline/v3.15-rc1-trusty/ 3.14 final: http://kernel.ubuntu.com/~kernel-ppa/mainline/v3.14-trusty/ These won't have hte cloud

[Kernel-packages] [Bug 1470250] Re: [Hyper-V] Ubuntu 14.04.2 LTS Generation 2 SCSI Errors on VSS Based Backups

Crashed after 30 and 12 hours. -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1470250 Title: [Hyper-V] Ubuntu 14.04.2 LTS Generation 2 SCSI Errors on VSS Based Backups Status in

[Kernel-packages] [Bug 1470250] Re: [Hyper-V] Ubuntu 14.04.2 LTS Generation 2 SCSI Errors on VSS Based Backups

Started run on 3.15, but there were no cloud tools in your build, so I used the linux-cloud-tools from 3.16.0-76 (jus copied the hv_* daemons over) -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu.

[Kernel-packages] [Bug 1470250] Re: [Hyper-V] Ubuntu 14.04.2 LTS Generation 2 SCSI Errors on VSS Based Backups

I built a 3.15 based kernel, which can be downloaded from: http://kernel.ubuntu.com/~jsalisbury/lp1470250/3.15/ Can you give this kernel a test? It never had commit 89fb4cd applied, so it did not need to be reverted. -- You received this bug notification because you are a member of Kernel

[Kernel-packages] [Bug 1470250] Re: [Hyper-V] Ubuntu 14.04.2 LTS Generation 2 SCSI Errors on VSS Based Backups

Thanks again for testing! It seems like we now need to perform another round of bisecting between 3.13 and 3.16 since reverting all the commits mentioned in comment #279 did not resolve the bug. So what we need to do is revert 89fb4cd before each step on the bisect. I'll get started on building

[Kernel-packages] [Bug 1470250] Re: [Hyper-V] Ubuntu 14.04.2 LTS Generation 2 SCSI Errors on VSS Based Backups

@faulpeltz I confirm the timing. I was never as specific as you are now. I already mentioned earlier that it always happens towards the end of the backup. I would not be surprised if the crash happens during the merge. -- You received this bug notification because you are a member of Kernel

  1   2   3   4   >