[Kernel-packages] [Bug 1470250] Re: [Hyper-V] Ubuntu 14.04.2 LTS Generation 2 SCSI Errors on VSS Based Backups

2016-06-29 Thread faulpeltz
And it crashed again after 10 hours.. Another thing: I looked at the timing of the (most recent) crashes a bit and it seems like they always happen after the backup has completed, when its merging back the backup checkpoint disk (which can be quite large under heavy I/O) -- You received this

[Kernel-packages] [Bug 1470250] Re: [Hyper-V] Ubuntu 14.04.2 LTS Generation 2 SCSI Errors on VSS Based Backups

2016-06-29 Thread faulpeltz
Unfortunately, it failed after a few hours. Trying to repro the crash a second time. -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1470250 Title: [Hyper-V] Ubuntu 14.04.2 LTS

[Kernel-packages] [Bug 1470250] Re: [Hyper-V] Ubuntu 14.04.2 LTS Generation 2 SCSI Errors on VSS Based Backups

2016-06-28 Thread Joseph Salisbury
Ok, I built one more test kernel with all of the commits to scsi_lib.c between v3.13 and 3.16 reverted. The list of commits reverted are: 39460b Revert "scsi: handle command allocation failure in scsi_reset_provider" 9ab4c8d Revert "fix regression in SCSI_IOCTL_SEND_COMMAND" 53bad98 Revert

[Kernel-packages] [Bug 1470250] Re: [Hyper-V] Ubuntu 14.04.2 LTS Generation 2 SCSI Errors on VSS Based Backups

2016-06-28 Thread Joseph Salisbury
I'm working on building a test kernel with all of the commits to scsi_lib.c between v3.13 and 3.16 reverted. Some of the commits need to be backported, but I should have a test kernel shortly. -- You received this bug notification because you are a member of Kernel Packages, which is subscribed

[Kernel-packages] [Bug 1470250] Re: [Hyper-V] Ubuntu 14.04.2 LTS Generation 2 SCSI Errors on VSS Based Backups

2016-06-24 Thread Emsi
http://orig00.deviantart.net/d4e7/f/2012/247/6/b/and_it_s_gone_by_celeith-d5div3y.jpg (sry, couldnt' resist it :) -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1470250 Title: [Hyper-V]

[Kernel-packages] [Bug 1470250] Re: [Hyper-V] Ubuntu 14.04.2 LTS Generation 2 SCSI Errors on VSS Based Backups

2016-06-24 Thread faulpeltz
.. and it failed :( -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1470250 Title: [Hyper-V] Ubuntu 14.04.2 LTS Generation 2 SCSI Errors on VSS Based Backups Status in linux package

[Kernel-packages] [Bug 1470250] Re: [Hyper-V] Ubuntu 14.04.2 LTS Generation 2 SCSI Errors on VSS Based Backups

2016-06-24 Thread faulpeltz
@jsalisbury: Your newest build is now at ~35hours without any issues; will keep it running over the weekend Could you maybe post the commits you reverted? -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu.

[Kernel-packages] [Bug 1470250] Re: [Hyper-V] Ubuntu 14.04.2 LTS Generation 2 SCSI Errors on VSS Based Backups

2016-06-22 Thread Joseph Salisbury
I built one more test kernel with 11 of the commits between 3.13 and 3.16 reverted. Can you test this kernel to see if one of this commits is the second issue? It can be downloaded from: http://kernel.ubuntu.com/~jsalisbury/lp1470250/revert-and-bisect/utopic/ -- You received this bug

[Kernel-packages] [Bug 1470250] Re: [Hyper-V] Ubuntu 14.04.2 LTS Generation 2 SCSI Errors on VSS Based Backups

2016-06-21 Thread faulpeltz
Unfortunately, it failed after 21 hours BTW we are at 240TB written on our test server :P -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1470250 Title: [Hyper-V] Ubuntu 14.04.2 LTS

[Kernel-packages] [Bug 1470250] Re: [Hyper-V] Ubuntu 14.04.2 LTS Generation 2 SCSI Errors on VSS Based Backups

2016-06-20 Thread faulpeltz
@jsalisbury: started testing 3.16 "double revert" -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1470250 Title: [Hyper-V] Ubuntu 14.04.2 LTS Generation 2 SCSI Errors on VSS Based

[Kernel-packages] [Bug 1470250] Re: [Hyper-V] Ubuntu 14.04.2 LTS Generation 2 SCSI Errors on VSS Based Backups

2016-06-20 Thread Joseph Salisbury
@faulpeltz, thanks again for testing. We're getting closer to narrowing down the second commit that needs to be reverted. I took a look of commits between 3.13 and 3.16. One sticks out as a possibility: bc85dc5. I built a Utopic test kernel with commits 89fb4cd and bc85dc5 reverted. The test

[Kernel-packages] [Bug 1470250] Re: [Hyper-V] Ubuntu 14.04.2 LTS Generation 2 SCSI Errors on VSS Based Backups

2016-06-20 Thread Tommy Eriksen
Hi, Do we have any kind of pattern on this yet? I have (so far) 4 VMs exhibiting this behaviour constantly - but out of maybe 200 Ubuntu VMs on Hyper-V. Problem is it's a production environment and I can't easily test different versions (as it requires proposing changes for each machine I want

[Kernel-packages] [Bug 1470250] Re: [Hyper-V] Ubuntu 14.04.2 LTS Generation 2 SCSI Errors on VSS Based Backups

2016-06-15 Thread faulpeltz
The second run of the 3.19 vivid kernel crashed after 16 hours The Utopic kernel (3.16) crashed after 23 hours (first run), restarting -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu.

[Kernel-packages] [Bug 1470250] Re: [Hyper-V] Ubuntu 14.04.2 LTS Generation 2 SCSI Errors on VSS Based Backups

2016-06-14 Thread Joseph Salisbury
@faulpeltz, yes sorry that test kernel was for vivid. Thanks for testing an narrowing it down further. I did build a Utopic 3.16 based kernel now. It has commit 89fb4cd1f7 reverted. It can be downloaded from: http://kernel.ubuntu.com/~jsalisbury/lp1470250/revert-and-bisect/utopic/ Can you

[Kernel-packages] [Bug 1470250] Re: [Hyper-V] Ubuntu 14.04.2 LTS Generation 2 SCSI Errors on VSS Based Backups

2016-06-13 Thread faulpeltz
The 3.19.0-61-generic you posted failed after 2 hours (currently re-running) Also, isn't this 3.19 a vivid kernel? sd 2:0:0:0: Device offlined - not ready after error recovery sd 2:0:0:0: [sda] FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_OK sd 2:0:0:0: [sda] CDB: Write(10): 2a 00 00 a1 cc 00

[Kernel-packages] [Bug 1470250] Re: [Hyper-V] Ubuntu 14.04.2 LTS Generation 2 SCSI Errors on VSS Based Backups

2016-06-13 Thread Joseph Salisbury
There are quite a few commits that may also need to be reverted from 3.13 to the 4.2 kernel. There is nothing evident that commit 89fb4cd1f does that can be easily identified the the Wily and newer code. It may be best to narrow down the specific kernel version that contains the second offending

[Kernel-packages] [Bug 1470250] Re: [Hyper-V] Ubuntu 14.04.2 LTS Generation 2 SCSI Errors on VSS Based Backups

2016-06-10 Thread Joseph Salisbury
Thanks for the update, @Emsi. I'll be traveling for the next 24 hours or so. That will give me some time to focus on the diffs between 3.13 and 4.2 and newer with that commit reverted. I'll post my findings here. -- You received this bug notification because you are a member of Kernel

[Kernel-packages] [Bug 1470250] Re: [Hyper-V] Ubuntu 14.04.2 LTS Generation 2 SCSI Errors on VSS Based Backups

2016-06-10 Thread Emsi
Keep in mind tough that I suggested that it might be some kind of memory corruption bug and the filesystem corruption might be just manifestation. I experiences different problems on 3.13.0-32-generic (random crashes and so on) thet were related to snapshoting. I guess the vmm integration is made

[Kernel-packages] [Bug 1470250] Re: [Hyper-V] Ubuntu 14.04.2 LTS Generation 2 SCSI Errors on VSS Based Backups

2016-06-10 Thread Emsi
Good news. # uname -a Linux backup-02 3.13.0-86-generic #131~lp1470250Commitd215f91Reverted SMP Wed May 25 20:47:20 UTC x86_64 x86_64 x86_64 GNU/Linux It is running since Jun 7 16:30:10 and still no crash. The only issue is that integration with VMM went awry and the hostname is reported

[Kernel-packages] [Bug 1470250] Re: [Hyper-V] Ubuntu 14.04.2 LTS Generation 2 SCSI Errors on VSS Based Backups

2016-06-09 Thread faulpeltz
@jsalisbury Yes I can confirm that. Both 4.X kernels were run twice to make sure the crash is reproducible, and the 3.13 which seems stable ran for a long time. -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu.

[Kernel-packages] [Bug 1470250] Re: [Hyper-V] Ubuntu 14.04.2 LTS Generation 2 SCSI Errors on VSS Based Backups

2016-06-08 Thread Joseph Salisbury
@faulpeltz, so per you testing it looks like: 3.13 based kernel with dfbdac2e(Same commit as 89fb4cd, but using the Trusty SHA1) reverted fixes the bug. 4.2 based kernel with 89fb4cd reverted still exhibits the bug. 4.4 based kernel with 89fb4cd reverted still exhibits the bug. This does seem

[Kernel-packages] [Bug 1470250] Re: [Hyper-V] Ubuntu 14.04.2 LTS Generation 2 SCSI Errors on VSS Based Backups

2016-06-08 Thread faulpeltz
4.2.0-36-89fb4cdReverted crashed after 10 hours, same errors -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1470250 Title: [Hyper-V] Ubuntu 14.04.2 LTS Generation 2 SCSI Errors on VSS

[Kernel-packages] [Bug 1470250] Re: [Hyper-V] Ubuntu 14.04.2 LTS Generation 2 SCSI Errors on VSS Based Backups

2016-06-07 Thread faulpeltz
No luck with the Xenial kernel (4.4.0-22), I could repro the crash 2 times (after a couple of hours each). Testing the Wily kernel next. Here is the relevant part of the logs (both crashes produced near identical logs): Jun 07 20:19:44 muchcrash02 kernel: sd 2:0:0:0: Device offlined - not ready

[Kernel-packages] [Bug 1470250] Re: [Hyper-V] Ubuntu 14.04.2 LTS Generation 2 SCSI Errors on VSS Based Backups

2016-06-07 Thread Emsi
Testing... -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1470250 Title: [Hyper-V] Ubuntu 14.04.2 LTS Generation 2 SCSI Errors on VSS Based Backups Status in linux package in Ubuntu:

[Kernel-packages] [Bug 1470250] Re: [Hyper-V] Ubuntu 14.04.2 LTS Generation 2 SCSI Errors on VSS Based Backups

2016-06-07 Thread Frederik Bosch
@emsi I had that problem too, you should ignore the binutils upgrade. I had a look at the CHANGELOG of binutils and there is no significant change. So the installation should continue although there is a dependency mismatch. I cannot recall exactly what I fix the mismatch. -- You received this

[Kernel-packages] [Bug 1470250] Re: [Hyper-V] Ubuntu 14.04.2 LTS Generation 2 SCSI Errors on VSS Based Backups

2016-06-07 Thread Emsi
I'm having difficulties installing those packages on trusty. I run into: linux-tools-3.13.0-86 : Depends: binutils (>= 2.24) but it is not going to be installed. Depends: binutils (< 2.25) but it is not going to be installed. Depends: libdw1 (>=

[Kernel-packages] [Bug 1470250] Re: [Hyper-V] Ubuntu 14.04.2 LTS Generation 2 SCSI Errors on VSS Based Backups

2016-06-06 Thread Joseph Salisbury
I built Wily and Xenial test kernels with commit 89fb4cd reverted(Same as commit d215f91 in Trusty). The test kernels can be downloaded from: Wily: http://kernel.ubuntu.com/~jsalisbury/lp1470250/d215f91-reverted/wily/ Xenial:

[Kernel-packages] [Bug 1470250] Re: [Hyper-V] Ubuntu 14.04.2 LTS Generation 2 SCSI Errors on VSS Based Backups

2016-06-06 Thread Joseph Salisbury
However, what is worrying is commit d215f91 was introduced in Ubuntu-3.13.0-35.61. @Emsi reports he hit the bug in 3.13.0-32. @Emsi, would it be possible for you to test the kernel posted in comment #248? -- You received this bug notification because you are a member of Kernel Packages, which

[Kernel-packages] [Bug 1470250] Re: [Hyper-V] Ubuntu 14.04.2 LTS Generation 2 SCSI Errors on VSS Based Backups

2016-06-06 Thread Joseph Salisbury
@faulpeltz, that is a positive sign. I'll build test kernels with this commit reverted in all the other kernels as well for testing. I'll post links shortly. -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu.

[Kernel-packages] [Bug 1470250] Re: [Hyper-V] Ubuntu 14.04.2 LTS Generation 2 SCSI Errors on VSS Based Backups

2016-06-03 Thread faulpeltz
d215f91-reverted stable for 90 hours -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1470250 Title: [Hyper-V] Ubuntu 14.04.2 LTS Generation 2 SCSI Errors on VSS Based Backups Status

[Kernel-packages] [Bug 1470250] Re: [Hyper-V] Ubuntu 14.04.2 LTS Generation 2 SCSI Errors on VSS Based Backups

2016-05-31 Thread Emsi
Are you aware of the fact that I have reported errors with 3.13.0-32-generic? -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1470250 Title: [Hyper-V] Ubuntu 14.04.2 LTS Generation 2

[Kernel-packages] [Bug 1470250] Re: [Hyper-V] Ubuntu 14.04.2 LTS Generation 2 SCSI Errors on VSS Based Backups

2016-05-31 Thread Dino
Sorry for the late reply @jrp: I haven't tested with backports kernel and I can't test it on the production machines. We also have an old VM with Ubuntu 12.04.5 LTS which works fine and without any issues. Maybe you guys can check the differences between this version and today's version. --

[Kernel-packages] [Bug 1470250] Re: [Hyper-V] Ubuntu 14.04.2 LTS Generation 2 SCSI Errors on VSS Based Backups

2016-05-30 Thread faulpeltz
a1dd8c87 failed after 70(!) hours (VSS backups started to fail after 17 hours or so, but the file system was remounted read-only after 70 hours total) Another observation, something which h I also noticed in previous "bad" runs: Also almost instantly after starting backups, there were I/O errors

[Kernel-packages] [Bug 1470250] Re: [Hyper-V] Ubuntu 14.04.2 LTS Generation 2 SCSI Errors on VSS Based Backups

2016-05-25 Thread Joseph Salisbury
I built a Trusty test kernel with commit d215f91 reverted. The test kernel can be downloaded from: http://kernel.ubuntu.com/~jsalisbury/lp1470250/d215f91-reverted/trusty I don't recall if I reverted that commit in a Trusty test kernel or in Xenial, so you may be correct. A merge after 3.13 may

[Kernel-packages] [Bug 1470250] Re: [Hyper-V] Ubuntu 14.04.2 LTS Generation 2 SCSI Errors on VSS Based Backups

2016-05-25 Thread Joakim Plate
I can see a possible issue with that commit. For an invalid sense value, sense_deferred with remain 0. and thus trigger the special logic. >From the rest of the code sense_deferred seem to only be valid in the case when sense_valid, but that is not checked. Given the large amount of invalid

[Kernel-packages] [Bug 1470250] Re: [Hyper-V] Ubuntu 14.04.2 LTS Generation 2 SCSI Errors on VSS Based Backups

2016-05-25 Thread Joseph Salisbury
I built the next test kernel up to commit: a1dd8c87 The test kernel can be downloaded from: http://kernel.ubuntu.com/~jsalisbury/lp1470250/third-bisect/a1dd8c87 I agree, the only commit in the current range that is suspect is: d215f91 I may have bad testing with that commit reverted, so I'll

[Kernel-packages] [Bug 1470250] Re: [Hyper-V] Ubuntu 14.04.2 LTS Generation 2 SCSI Errors on VSS Based Backups

2016-05-25 Thread faulpeltz
@Dino Yes we could also reproduce the issue on Jessie (3.16) and I've also seen it in testing/unstable, too @jsalisbury Assuming the bisect is correct this time, d215f91 seems the only likely suspect Which kernel version did you test with this commit reverted? Maybe some of the later merges

[Kernel-packages] [Bug 1470250] Re: [Hyper-V] Ubuntu 14.04.2 LTS Generation 2 SCSI Errors on VSS Based Backups

2016-05-25 Thread Joshua R. Poulson
@dino.m Indeed, since the 3.16 kernel in Jessie shares a common heritage with the 3.16 Ubuntu Utopic kernel it may indeed have the same problem. However, until we find root cause it is difficult to carry the search elsewhere. Since we're talking Jessie, are you having similar difficulties with the

[Kernel-packages] [Bug 1470250] Re: [Hyper-V] Ubuntu 14.04.2 LTS Generation 2 SCSI Errors on VSS Based Backups

2016-05-25 Thread Dino
Hi everybody. I came here through the link in the Microsoft Technet Forum. We also have these problems under Hyper-V on a Windows Server 2012R2, but with a Debian Jessie 3.16.0-4. We also use Altaro Hyper-V Backup, and once or twice a week after the backup we have to repair the file system. An

[Kernel-packages] [Bug 1470250] Re: [Hyper-V] Ubuntu 14.04.2 LTS Generation 2 SCSI Errors on VSS Based Backups

2016-05-25 Thread faulpeltz
488347f3f is good after 48 hours -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1470250 Title: [Hyper-V] Ubuntu 14.04.2 LTS Generation 2 SCSI Errors on VSS Based Backups Status in

[Kernel-packages] [Bug 1470250] Re: [Hyper-V] Ubuntu 14.04.2 LTS Generation 2 SCSI Errors on VSS Based Backups

2016-05-23 Thread Joseph Salisbury
I built the next test kernel up to commit: 488347f3f The test kernel can be downloaded from: http://kernel.ubuntu.com/~jsalisbury/lp1470250/third-bisect/488347f3f -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu.

[Kernel-packages] [Bug 1470250] Re: [Hyper-V] Ubuntu 14.04.2 LTS Generation 2 SCSI Errors on VSS Based Backups

2016-05-23 Thread faulpeltz
bb3becb good after 86 hours I stopped the test run for now -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1470250 Title: [Hyper-V] Ubuntu 14.04.2 LTS Generation 2 SCSI Errors on VSS

[Kernel-packages] [Bug 1470250] Re: [Hyper-V] Ubuntu 14.04.2 LTS Generation 2 SCSI Errors on VSS Based Backups

2016-05-21 Thread faulpeltz
bb3becb good after 38 hours -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1470250 Title: [Hyper-V] Ubuntu 14.04.2 LTS Generation 2 SCSI Errors on VSS Based Backups Status in linux

[Kernel-packages] [Bug 1470250] Re: [Hyper-V] Ubuntu 14.04.2 LTS Generation 2 SCSI Errors on VSS Based Backups

2016-05-19 Thread ubuntu
FYI, When I did the analysis of this last year I found a couple of additional key data points that may help with reproducing the error. I reported them at the time but they're well buried by now. 1) Timing of the bug appears to be load related. I found that generating the error could take

[Kernel-packages] [Bug 1470250] Re: [Hyper-V] Ubuntu 14.04.2 LTS Generation 2 SCSI Errors on VSS Based Backups

2016-05-19 Thread Joseph Salisbury
I built the next test kernel up to commit: bb3becb The test kernel can be downloaded from: http://kernel.ubuntu.com/~jsalisbury/lp1470250/third-bisect/bb3becbf1 For some reason, I'm now unable to reproduce the bug, even with old kernels that failed before. I'm going to investigate why that

[Kernel-packages] [Bug 1470250] Re: [Hyper-V] Ubuntu 14.04.2 LTS Generation 2 SCSI Errors on VSS Based Backups

2016-05-19 Thread faulpeltz
95d1181 good after 38 hours -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1470250 Title: [Hyper-V] Ubuntu 14.04.2 LTS Generation 2 SCSI Errors on VSS Based Backups Status in linux

[Kernel-packages] [Bug 1470250] Re: [Hyper-V] Ubuntu 14.04.2 LTS Generation 2 SCSI Errors on VSS Based Backups

2016-05-18 Thread faulpeltz
95d1181 is still good after 25 hours, will keep it running for another 10 or so -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1470250 Title: [Hyper-V] Ubuntu 14.04.2 LTS Generation 2

[Kernel-packages] [Bug 1470250] Re: [Hyper-V] Ubuntu 14.04.2 LTS Generation 2 SCSI Errors on VSS Based Backups

2016-05-17 Thread faulpeltz
4c48c359b is still good after 27 hours, starting on 95d1181 If I hit the error I will re-run it immediately to make sure its bad -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1470250

[Kernel-packages] [Bug 1470250] Re: [Hyper-V] Ubuntu 14.04.2 LTS Generation 2 SCSI Errors on VSS Based Backups

2016-05-17 Thread Joseph Salisbury
I started the next bisect between 4c48c359b and da1674843. The first test kernel is built up to the commit: 95d118176516b3aa16249b9bbdf579a67878d3c3 Can you test this kernel? It can be downloaded from: http://kernel.ubuntu.com/~jsalisbury/lp1470250/third-bisect/95d1181/ -- You received this

[Kernel-packages] [Bug 1470250] Re: [Hyper-V] Ubuntu 14.04.2 LTS Generation 2 SCSI Errors on VSS Based Backups

2016-05-17 Thread Joseph Salisbury
I tested a kernel with commit d215f91 reverted, and it unfortunately hit the bug. At least the range of commits is getting smaller. I'll restart the bisect between 4c48c359b..da1674843 and build the next test kernel. I'll start testing it and post it shortly. -- You received this bug

[Kernel-packages] [Bug 1470250] Re: [Hyper-V] Ubuntu 14.04.2 LTS Generation 2 SCSI Errors on VSS Based Backups

2016-05-17 Thread faulpeltz
@jsalisbury: As an update to my latest post: I re-ran a few even older builds on the weekend: adbb4e646 - good after 40hours (bad before, might have been another victim of the disk filling up, I think I might have screwed up this run too) da1674843 - bad after 8 hours (consistent with previous

[Kernel-packages] [Bug 1470250] Re: [Hyper-V] Ubuntu 14.04.2 LTS Generation 2 SCSI Errors on VSS Based Backups

2016-05-14 Thread faulpeltz
586fbce still running after 26 hours, I will keep it running over the weekend As far as is can tell, the previous 586fbce run might have been affected by the same issue as the other two versions (5e6cf71/8321521). I do have more confidence in the newer runs, but it would be good if multiple

[Kernel-packages] [Bug 1470250] Re: [Hyper-V] Ubuntu 14.04.2 LTS Generation 2 SCSI Errors on VSS Based Backups

2016-05-13 Thread faulpeltz
8321521 is good after 40 hours, moving to 586fbce -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1470250 Title: [Hyper-V] Ubuntu 14.04.2 LTS Generation 2 SCSI Errors on VSS Based

[Kernel-packages] [Bug 1470250] Re: [Hyper-V] Ubuntu 14.04.2 LTS Generation 2 SCSI Errors on VSS Based Backups

2016-05-11 Thread Joseph Salisbury
I'm still testing 1c8349a17 now. I increased the number of I/O threads to increase the change of triggering the bug. It's been running for almost 24 hours now. I'll let it run for up to 36 hours with this I/O load. I agree that if 8321521 does turn out good, we would want to test 586fbce again

[Kernel-packages] [Bug 1470250] Re: [Hyper-V] Ubuntu 14.04.2 LTS Generation 2 SCSI Errors on VSS Based Backups

2016-05-11 Thread faulpeltz
5e6cf71 still good after 26 hours, switching to 8321521 next, if that turns out good it might be good idea to re-test 586fbce as well or we can continue the bisect @jsalisbury were you able to reproduce the crash on the kernels with 1c8349a17 reverted? @benjamin-ihrig on one of our production

[Kernel-packages] [Bug 1470250] Re: [Hyper-V] Ubuntu 14.04.2 LTS Generation 2 SCSI Errors on VSS Based Backups

2016-05-10 Thread Benjamin Ihrig
First of all thanks for your effort guys (sorry, I cannot really help you out, I am a Java Developer and not into Ubuntu Kernel)! I do experience these problems with a Hyper-V environment I operate for a volunteer fire departement. I am not sure about the following fact, but testing around made

[Kernel-packages] [Bug 1470250] Re: [Hyper-V] Ubuntu 14.04.2 LTS Generation 2 SCSI Errors on VSS Based Backups

2016-05-10 Thread tcmbackwards
Let me clarify, I am using hyper-v and I got the same SCSI errors, not iSCSI. The SCSI errors were the same kernel errors about changing operating parameters. -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu.

[Kernel-packages] [Bug 1470250] Re: [Hyper-V] Ubuntu 14.04.2 LTS Generation 2 SCSI Errors on VSS Based Backups

2016-05-10 Thread tcmbackwards
I had corruption with btrfs months ago with the same iSCSI errors in hyperv. At the time I thought it was btrfs instability with docker overlays but changing to ext4 didn't help. I get the same errors everyone else in this thread has reported with ext4. -- You received this bug notification

[Kernel-packages] [Bug 1470250] Re: [Hyper-V] Ubuntu 14.04.2 LTS Generation 2 SCSI Errors on VSS Based Backups

2016-05-10 Thread Joseph Salisbury
That is a good idea about testing a file system other than ext4. There are two other ext4 commits in the bisect range we are testing. It would require 8321521 to be Good for it to be either of these: 265dabe ext4: fix wrong assert in ext4_mb_normalize_request() d4d2e7e ext4: fix zeroing of page

[Kernel-packages] [Bug 1470250] Re: [Hyper-V] Ubuntu 14.04.2 LTS Generation 2 SCSI Errors on VSS Based Backups

2016-05-10 Thread faulpeltz
@jsalisbury, I moved back to testing only a single machine at a given time, currently 5e6cf71 is running for ~6 hours, 83215219 is up next we had 5e6cf71 and 83215219 running at the same time without any issues for 24 hours *but* the problem seems to be easier to reproduce with only one machine

[Kernel-packages] [Bug 1470250] Re: [Hyper-V] Ubuntu 14.04.2 LTS Generation 2 SCSI Errors on VSS Based Backups

2016-05-10 Thread Joseph Salisbury
Hmm, I still have not hit the bug with the test kernel that has 1c8349a17 reverted. It's a good thing we have multiple testers, thanks so much, faulpeltz. For this bug, it's probably best to just finish up with the bisect to identify the exact commit without any guessing. @faulpeltz, were you

[Kernel-packages] [Bug 1470250] Re: [Hyper-V] Ubuntu 14.04.2 LTS Generation 2 SCSI Errors on VSS Based Backups

2016-05-10 Thread faulpeltz
>From 4.2.0-35-generic (lp1470250Commit1c8349a17Reverted), crashed after less >than 2 hours: [ 7016.076017] sd 2:0:0:0: [storvsc] Sense Key : Unit Attention [current] [ 7016.076062] sd 2:0:0:0: [storvsc] Add. Sense: Changed operating definition [ 7016.076262] sd 2:0:0:0: Warning! Received an

[Kernel-packages] [Bug 1470250] Re: [Hyper-V] Ubuntu 14.04.2 LTS Generation 2 SCSI Errors on VSS Based Backups

2016-05-09 Thread Joseph Salisbury
There is now a 4.2 based Wily test kernel available here: http://kernel.ubuntu.com/~jsalisbury/lp1470250/reverts/wily/ -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1470250 Title:

[Kernel-packages] [Bug 1470250] Re: [Hyper-V] Ubuntu 14.04.2 LTS Generation 2 SCSI Errors on VSS Based Backups

2016-05-09 Thread Joseph Salisbury
That is a good idea, Frederik. I'll build test kernel for all the other releases and post a link to them shortly. -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1470250 Title:

[Kernel-packages] [Bug 1470250] Re: [Hyper-V] Ubuntu 14.04.2 LTS Generation 2 SCSI Errors on VSS Based Backups

2016-05-09 Thread Frederik Bosch
@jsalisbury Maybe you can also create a 4.2.0 kernel with cd4842f4 reverted. Then I can test one. -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1470250 Title: [Hyper-V] Ubuntu 14.04.2

[Kernel-packages] [Bug 1470250] Re: [Hyper-V] Ubuntu 14.04.2 LTS Generation 2 SCSI Errors on VSS Based Backups

2016-05-08 Thread Joseph Salisbury
I have been testing the kernel with commit cd4842f4(1c8349a17 in mainline) reverted for three days now without hitting the bug. It's looking very promising that commit cd4842f4 is what introduced this bug. I'll continue to test this kernel for a few more days to ensure it's stable. It would be

[Kernel-packages] [Bug 1470250] Re: [Hyper-V] Ubuntu 14.04.2 LTS Generation 2 SCSI Errors on VSS Based Backups

2016-05-07 Thread faulpeltz
I stopped running 3.13.0-34.60 and dfbdac2e after nearly 90 hours and 500 backups with no issues Started re-running 5e6cf71 and 83215219 If 83215219 is bad, I will run the kernel with cd4842f4 reverted -- You received this bug notification because you are a member of Kernel Packages, which is

[Kernel-packages] [Bug 1470250] Re: [Hyper-V] Ubuntu 14.04.2 LTS Generation 2 SCSI Errors on VSS Based Backups

2016-05-05 Thread Joseph Salisbury
I have a test kernel with commit cd4842f4(1c8349a17 in mainline) reverted. It can be downloaded from: http://kernel.ubuntu.com/~jsalisbury/lp1470250/reverts/commit- cd4842f4-reverted/ Like I mentioned in the last comment, there is a later xfs commit which depended on commit 1c8349a17 for

[Kernel-packages] [Bug 1470250] Re: [Hyper-V] Ubuntu 14.04.2 LTS Generation 2 SCSI Errors on VSS Based Backups

2016-05-05 Thread Joseph Salisbury
I also plan on building a test kernel with just commit cd4842f4 reverted. It's not straight forward because a later xfs commit started using a function added by this commit(set_page_writeback_keepwrite()). I think the easiest thing for testing and confirming this is the bad commit is to just

Re: [Kernel-packages] [Bug 1470250] Re: [Hyper-V] Ubuntu 14.04.2 LTS Generation 2 SCSI Errors on VSS Based Backups

2016-05-05 Thread Frederik Bosch
@jsalisbury I would not be surprised by that. It would explain the pain in finding the cause of this bug. Its outside of any of our scopes. Moreover, when you read the commit message (https://github.com/torvalds/linux/commit/cd4842f4), it seems that there is quite some controversial, probably

[Kernel-packages] [Bug 1470250] Re: [Hyper-V] Ubuntu 14.04.2 LTS Generation 2 SCSI Errors on VSS Based Backups

2016-05-05 Thread Joseph Salisbury
If 5e6cf71 ends up being good, that put's one of the ext4 commits in the suspect range: cd4842f4 ext4: fix data integrity sync in ordered mode -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu.

[Kernel-packages] [Bug 1470250] Re: [Hyper-V] Ubuntu 14.04.2 LTS Generation 2 SCSI Errors on VSS Based Backups

2016-05-05 Thread Joseph Salisbury
I'm actually going to build a test kernel with those three ext4 commits reverted and test that. That will allow us to test in parallel and ensure if 5e6cf71 and 83215219 were in fact bad. -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to

[Kernel-packages] [Bug 1470250] Re: [Hyper-V] Ubuntu 14.04.2 LTS Generation 2 SCSI Errors on VSS Based Backups

2016-05-05 Thread Joseph Salisbury
I've also been running dfbdac2e for 48 hours without hitting the bug. I don't think bad commit is in the hv_storvsc driver or any HV specific code. I'm not sure how to explain that yet. I tested a kernel with all the HV related commits(c8c38b3, 7af024a and 6ad4874) reverted and the bug still

[Kernel-packages] [Bug 1470250] Re: [Hyper-V] Ubuntu 14.04.2 LTS Generation 2 SCSI Errors on VSS Based Backups

2016-05-05 Thread faulpeltz
both 3.13.0-34.60 and dfbdac2e running for 48 hours, no issues, both now have gone through 260 backup cycles i will keep them running for now -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu.

[Kernel-packages] [Bug 1470250] Re: [Hyper-V] Ubuntu 14.04.2 LTS Generation 2 SCSI Errors on VSS Based Backups

2016-05-05 Thread Alex Ng
Also, to add one other thing. There were a bunch of commits made upstream to the storvsc driver in the last few months. Can we try them out to see if they have any impact on this issue? In particular: 1) 81988a0e6b031bc80da15257201810ddcf989e64 - storvsc: get rid of bounce buffer 2)

[Kernel-packages] [Bug 1470250] Re: [Hyper-V] Ubuntu 14.04.2 LTS Generation 2 SCSI Errors on VSS Based Backups

2016-05-05 Thread Alex Ng
To help see if this is an issue in the hv_storvsc driver; I took the storvsc driver code from 3.13.0-34.60 (presumably a good build) and applied it to the 4.2.0-27 kernel (presumably a bad build). Ran tiobench with backups and was able to repro after about 48 hours. This implies that either: 1)

[Kernel-packages] [Bug 1470250] Re: [Hyper-V] Ubuntu 14.04.2 LTS Generation 2 SCSI Errors on VSS Based Backups

2016-05-04 Thread Joseph Salisbury
@faulpeltz, Yes, I did test a kernel with commit 6ad4874. I tried this before starting the bisect since it looked suspect. I'm still testing dfbdac2e as well and will let it run for another 24 hours. -- You received this bug notification because you are a member of Kernel Packages, which is

[Kernel-packages] [Bug 1470250] Re: [Hyper-V] Ubuntu 14.04.2 LTS Generation 2 SCSI Errors on VSS Based Backups

2016-05-04 Thread faulpeltz
@jsalisbury We have 3.13.0-34.60 already running for about 22hours straight, no problems yet, as well as dfbdac2e, which also runs fine for now. I'll just keep it running for a few days Also, unfortunately, our result for 5e6cf71 might be invalid because the test machine ran out of disk space on

[Kernel-packages] [Bug 1470250] Re: [Hyper-V] Ubuntu 14.04.2 LTS Generation 2 SCSI Errors on VSS Based Backups

2016-05-04 Thread Joseph Salisbury
@faulpeltz, I am continuing to test the next test kernel up to commit: dfbdac2e Are you testing 3.13.0-34.60 or would you be wiling to do that? Just to confirm it does not contain the bug and the bisect was started at the correct versions. -- You received this bug notification because you are

[Kernel-packages] [Bug 1470250] Re: [Hyper-V] Ubuntu 14.04.2 LTS Generation 2 SCSI Errors on VSS Based Backups

2016-05-04 Thread Emsi
I experienced it during migration and other means involving snapshots. -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1470250 Title: [Hyper-V] Ubuntu 14.04.2 LTS Generation 2 SCSI

[Kernel-packages] [Bug 1470250] Re: [Hyper-V] Ubuntu 14.04.2 LTS Generation 2 SCSI Errors on VSS Based Backups

2016-05-03 Thread faulpeltz
I did not explicitly test for that, but on our production server the issue went away completely But we can definitely try that, 48 hours of I/O torture should rule out any non-VSS related issues -- You received this bug notification because you are a member of Kernel Packages, which is

[Kernel-packages] [Bug 1470250] Re: [Hyper-V] Ubuntu 14.04.2 LTS Generation 2 SCSI Errors on VSS Based Backups

2016-05-03 Thread Frederik Bosch
@alexng-v I only had this while doing backups. Also, in the initial post on MS Technet refers to it as a backup problem: https://social.technet.microsoft.com/Forums/windowsserver/en-US /8807f61c-565e-45bc-abc4-af09abf59de2/ubuntu-14042-lts-generation-2 -scsi-errors-on-vss-based-backups. We are now

[Kernel-packages] [Bug 1470250] Re: [Hyper-V] Ubuntu 14.04.2 LTS Generation 2 SCSI Errors on VSS Based Backups

2016-05-03 Thread Alex Ng
Don't think this has been asked before, but has anyone had a repro when backups were turned off? Or does this only happen when backups are enabled? I'm verifying this on my own as well, but if this happens regardless of whether backup is enabled/disabled; then it'll help us narrow down the cause

[Kernel-packages] [Bug 1470250] Re: [Hyper-V] Ubuntu 14.04.2 LTS Generation 2 SCSI Errors on VSS Based Backups

2016-05-02 Thread Joseph Salisbury
I built the next test kernel up to commit: dfbdac2e The test kernel can be downloaded from: http://kernel.ubuntu.com/~jsalisbury/lp1470250/second-bisect/dfbdac2e It's probably best to ensure you don't hit the bug with 3.13.0-34.60 first, then test this next kernel. -- You received this bug

[Kernel-packages] [Bug 1470250] Re: [Hyper-V] Ubuntu 14.04.2 LTS Generation 2 SCSI Errors on VSS Based Backups

2016-05-02 Thread Joseph Salisbury
The next kernel is building. It would be good for you to confirm 3.13.0-34.60 is actually good, so we know the bisect is going in the right direction. -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu.

[Kernel-packages] [Bug 1470250] Re: [Hyper-V] Ubuntu 14.04.2 LTS Generation 2 SCSI Errors on VSS Based Backups

2016-05-02 Thread faulpeltz
5e6cf71 crashed after 12 hours until the next build is available I will let it churn on 3.13.0-34.60 just to make sure its stable there are only 26 commits in 3402ec8..5e6cf71 -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in

[Kernel-packages] [Bug 1470250] Re: [Hyper-V] Ubuntu 14.04.2 LTS Generation 2 SCSI Errors on VSS Based Backups

2016-05-01 Thread Joseph Salisbury
I built the next test kernel up to commit: 5e6cf71 I'll test this kernel next. It can also be downloaded from here if others want to test: http://kernel.ubuntu.com/~jsalisbury/lp1470250/second-bisect/5e6cf71 -- You received this bug notification because you are a member of Kernel Packages,

[Kernel-packages] [Bug 1470250] Re: [Hyper-V] Ubuntu 14.04.2 LTS Generation 2 SCSI Errors on VSS Based Backups

2016-04-29 Thread faulpeltz
8321521963a dead after 19 hours -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1470250 Title: [Hyper-V] Ubuntu 14.04.2 LTS Generation 2 SCSI Errors on VSS Based Backups Status in

[Kernel-packages] [Bug 1470250] Re: [Hyper-V] Ubuntu 14.04.2 LTS Generation 2 SCSI Errors on VSS Based Backups

2016-04-28 Thread faulpeltz
We observed that as well. The issue can occur just by creating a volume shadow copy of the volume the Hyper-V disk is stored on (with the Hyper-V VSS writer) Started running build 83215219 in the meantime. I also thought about experimenting with creating shadow copies (volatile, with writers)

[Kernel-packages] [Bug 1470250] Re: [Hyper-V] Ubuntu 14.04.2 LTS Generation 2 SCSI Errors on VSS Based Backups

2016-04-28 Thread Frederik Bosch
@elupus This is in line with our observations. Our guess is that the change to a read-only state occurs during/just befor/just after a FREEZE and/or THAW. -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu.

[Kernel-packages] [Bug 1470250] Re: [Hyper-V] Ubuntu 14.04.2 LTS Generation 2 SCSI Errors on VSS Based Backups

2016-04-27 Thread Joakim Plate
Some interesting additional observations. I'm running kernel 4.2.0-35 on 15.10 ubuntu. Last night it hung with readonly system ala- this report. What was interesting was that the machine that hung was NOT included in the backups run on the host system. The host hypervisor does run backups on other

[Kernel-packages] [Bug 1470250] Re: [Hyper-V] Ubuntu 14.04.2 LTS Generation 2 SCSI Errors on VSS Based Backups

2016-04-27 Thread Joseph Salisbury
Thanks for the update. The test of the kernel on my system is still running, so it's great you could reproduce the bug faster. I built the next kernel up to commit: 83215219 It can be downloaded from: http://kernel.ubuntu.com/~jsalisbury/lp1470250/second-bisect/83215219/ I'll see if there is a

[Kernel-packages] [Bug 1470250] Re: [Hyper-V] Ubuntu 14.04.2 LTS Generation 2 SCSI Errors on VSS Based Backups

2016-04-27 Thread faulpeltz
586fbce failed for me after 28 hours It would be nice if we could have packages for maybe 2 further versions in the bisect (the current one + good/fail one), so we can run new builds back to back. -- You received this bug notification because you are a member of Kernel Packages, which is

[Kernel-packages] [Bug 1470250] Re: [Hyper-V] Ubuntu 14.04.2 LTS Generation 2 SCSI Errors on VSS Based Backups

2016-04-25 Thread Joseph Salisbury
I wasn't able to reproduce the bug using the test kernel with commit da1674843 as the tip and Powershell as the backup client(Using the scripts posted in #182). I am able to reproduce the bug with that kernel using the Windows Server Backup GUI, it just took 35 hours. I'm going to keep using

[Kernel-packages] [Bug 1470250] Re: [Hyper-V] Ubuntu 14.04.2 LTS Generation 2 SCSI Errors on VSS Based Backups

2016-04-25 Thread Joseph Salisbury
I built the next test kernel up to commit: 586fbce I'll test this kernel next. It can also be downloaded from here if others want to test: http://kernel.ubuntu.com/~jsalisbury/lp1470250/second-bisect/586fbce -- You received this bug notification because you are a member of Kernel Packages,

[Kernel-packages] [Bug 1470250] Re: [Hyper-V] Ubuntu 14.04.2 LTS Generation 2 SCSI Errors on VSS Based Backups

2016-04-25 Thread faulpeltz
Oops I actually meant adbb4e646 .. The one provided in post #181 as a download -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1470250 Title: [Hyper-V] Ubuntu 14.04.2 LTS Generation 2

[Kernel-packages] [Bug 1470250] Re: [Hyper-V] Ubuntu 14.04.2 LTS Generation 2 SCSI Errors on VSS Based Backups

2016-04-25 Thread Alex Ng
Any updates on whether adbb4e646 test kernel is able to repro this issue? In any case, if we can get a next test kernel to try, I can help try repro on it as well. -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu.

[Kernel-packages] [Bug 1470250] Re: [Hyper-V] Ubuntu 14.04.2 LTS Generation 2 SCSI Errors on VSS Based Backups

2016-04-25 Thread faulpeltz
We have been running the da1674843 test kernel on another hyper-v server (an older test machine), as well as 4.4.0_21 for comparison; the test kernel failed after 14h the 4.4.0_21 after 9h. This takes much longer than on the original server (which was a 2-CPU 20-Core 256GB RAM machine), but we

[Kernel-packages] [Bug 1470250] Re: [Hyper-V] Ubuntu 14.04.2 LTS Generation 2 SCSI Errors on VSS Based Backups

2016-04-19 Thread Joseph Salisbury
Thanks for the suggestions @faulpeltz and @Emsi. I was using the Windows Server Backup GUI, which allowed me a maximum of 30 minutes backup intervals. I now have a Powershell script to kick of one backup after another: $i=1 while($true) { "###" "Starting backup $i"

[Kernel-packages] [Bug 1470250] Re: [Hyper-V] Ubuntu 14.04.2 LTS Generation 2 SCSI Errors on VSS Based Backups

2016-04-19 Thread faulpeltz
For the backup stress test I really just used: :start wbadmin start backup -quiet -backupTarget:\\myserver\dummyshare -hyperv:"MYTESTVM" goto start In our case the VM server did not run anything else and the Ubuntu guest was a minmal install, so the loop took only a couple of minutes,

<    1   2   3   4   >