[Bug 1676635] Re: [Hyper-V] Implement Hyper-V PTP Source
Hi Marcelo, I kicked the tires on 4.4.0-85-generic kernel from xenial-proposed. The fixes look good. I see the PTP device and TimeSync is not causing "Time has been changed" messages in systemd. I also see that apt-daily timer is no longer being randomly delayed due to the clock changes. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1676635 Title: [Hyper-V] Implement Hyper-V PTP Source To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1676635/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1676635] Re: [Hyper-V] Implement Hyper-V PTP Source
Please ignore my last comment #13, I intended it for another bug. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1676635 Title: [Hyper-V] Implement Hyper-V PTP Source To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1676635/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1676635] Re: [Hyper-V] Implement Hyper-V PTP Source
Here are a few upstream commits that should also be included. Drivers: hv: util: don't forget to init host_ts.lock https://git.kernel.org/pub/scm/linux/kernel/git/next/linux-next.git/commit/drivers/hv/hv_util.c?id=5a16dfc855127906fcd2935fb039bc8989313915 hv_utils: drop .getcrosststamp() support from PTP driver https://git.kernel.org/pub/scm/linux/kernel/git/next/linux-next.git/commit/drivers/hv/hv_util.c?id=4f9bac039a64f6306b613a0d90e6b7e75d7ab0c4 hv_utils: fix TimeSync work on pre-TimeSync-v4 hosts https://git.kernel.org/pub/scm/linux/kernel/git/next/linux-next.git/commit/drivers/hv/hv_util.c?id=1d10602d306cb7f70545b5e1166efc9409e7d384 -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1676635 Title: [Hyper-V] Implement Hyper-V PTP Source To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1676635/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1676635] Re: [Hyper-V] Implement Hyper-V PTP Source
@Alberto Ornaghi. Are you running in Azure or in your own Hyper-V host? If you have access to the Hyper-V host, you can disable the TimeSync integration service from the VM settings. Otherwise, you can create a script that runs "echo 2dd1ce17-079e- 403c-b352-a1921ee207ee > /sys/bus/vmbus/drivers/hv_util/unbind" upon startup. This disables TimeSync from within the guest VM. Hope this helps, Alex -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1676635 Title: [Hyper-V] Implement Hyper-V PTP Source To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1676635/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1470250] Re: [Hyper-V] Ubuntu 14.04.2 LTS Generation 2 SCSI Errors on VSS Based Backups
The patch from @faulpeltz hasn't been mainlined because of some feedback that it shouldn't have to close and reopen the /dev/vmbus/hv_vss device after failure. I addressed this comment this in a modified version of @faulpeltz's patch (see comment #345). I haven't heard from @faulpeltz whether he's tested it. From my testing, it seems fine and I expect to resubmit it upstream. If we can make it a SAUCE patch, that would be nice. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1470250 Title: [Hyper-V] Ubuntu 14.04.2 LTS Generation 2 SCSI Errors on VSS Based Backups To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1470250/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1470250] Re: [Hyper-V] Ubuntu 14.04.2 LTS Generation 2 SCSI Errors on VSS Based Backups
Hi @jsalisbury, Any status update on the patches for this issue? It appears the test kernels have resolved the issue. Let us know if you need additional testing or have questions about the patches. Thanks, Alex -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1470250 Title: [Hyper-V] Ubuntu 14.04.2 LTS Generation 2 SCSI Errors on VSS Based Backups To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1470250/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1470250] Re: [Hyper-V] Ubuntu 14.04.2 LTS Generation 2 SCSI Errors on VSS Based Backups
Thanks Joseph for compiling the list. The patches you've outlined should be sufficient. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1470250 Title: [Hyper-V] Ubuntu 14.04.2 LTS Generation 2 SCSI Errors on VSS Based Backups To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1470250/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1470250] Re: [Hyper-V] Ubuntu 14.04.2 LTS Generation 2 SCSI Errors on VSS Based Backups
@faulpeltz I agree. Your patch should also be included in case a FREEZE operation does exceed the increased timeout. I've attached a modified version of your patch that addresses some concerns we had offline. Could you give it a try? ** Patch added: "0004-Tools-hv-vss-Thaw-the-filesystem-and-continue-after-.patch" https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1470250/+attachment/4778824/+files/0004-Tools-hv-vss-Thaw-the-filesystem-and-continue-after-.patch -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1470250 Title: [Hyper-V] Ubuntu 14.04.2 LTS Generation 2 SCSI Errors on VSS Based Backups To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1470250/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1470250] Re: [Hyper-V] Ubuntu 14.04.2 LTS Generation 2 SCSI Errors on VSS Based Backups
I'll let @jsalisbury comment on the status of his backported patches. @jsalisbury, I'd also take this recently submitted patch from upstream kernel as well. It ensures that the VSS driver doesn't timeout any long running FREEZE operations too early. This should preclude the need for faulpeltz's patch in most cases, unless the FREEZE operation takes extraordinarily long (> 15 mins). https://git.kernel.org/cgit/linux/kernel/git/next/linux- next.git/commit/drivers/hv?id=b357fd3908c1191f2f56e38aa77f2aecdae18bc8 -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1470250 Title: [Hyper-V] Ubuntu 14.04.2 LTS Generation 2 SCSI Errors on VSS Based Backups To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1470250/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1470250] Re: [Hyper-V] Ubuntu 14.04.2 LTS Generation 2 SCSI Errors on VSS Based Backups
Thanks @faulpeltz for the info. I sent you a private message with the list of maintainers to send the patch (trying to avoid pasting it here in case spam bots can go through this archive). -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1470250 Title: [Hyper-V] Ubuntu 14.04.2 LTS Generation 2 SCSI Errors on VSS Based Backups To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1470250/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1470250] Re: [Hyper-V] Ubuntu 14.04.2 LTS Generation 2 SCSI Errors on VSS Based Backups
Hi @faulpeltz, A few questions/comments about your patch: 1) Can you submit your patch to the upstream kernel? 2) Under load, were you able to measure how long the FIFREEZE operation took before it succeeded? I'm trying to see if we can increase the timeout of the kernel driver before it hits the error condition that you encountered. Thanks, Alex -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1470250 Title: [Hyper-V] Ubuntu 14.04.2 LTS Generation 2 SCSI Errors on VSS Based Backups To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1470250/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1584597] Re: Hyper-V Memory Ballooning re-broken in 16.04
Evan, from your screenshots, I can see that the memory demand values and the /proc/meminfo values do match closely to each other. On the 16.04.01 VM, /proc/meminfo shows Committed_AS value is ~1800MB. The balloon driver then adds a buffer of ~300MB (this buffer is calculated according to how much total memory is assigned to the VM). So ~1800MB + ~300MB = 2100MB matches closely with the 2088 MB memory demand that you're seeing from the host. A similar calculation on the 15.10 VM also shows that the meminfo values match closely with the reported memory demand. (~2900MB committed + ~500MB buffer = 3400MB compared to 3600MB) The main culprit in both cases appears to be the large amount of committed memory. Would be worth investigating what the cause of this memory usage is. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1584597 Title: Hyper-V Memory Ballooning re-broken in 16.04 To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1584597/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1584597] Re: Hyper-V Memory Ballooning re-broken in 16.04
Bernhard(galmok), the memory corruption you're seeing is likely a known issue that happens when Linux guests are running on Windows Server 2012. This was fixed recently in the upstream Linux kernels, so hopefully a future update will have these patches. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1584597 Title: Hyper-V Memory Ballooning re-broken in 16.04 To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1584597/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1470250] Re: [Hyper-V] Ubuntu 14.04.2 LTS Generation 2 SCSI Errors on VSS Based Backups
And in response to Joseph's comment #309, the second patch shouldn't be required as it's related to a feature introduced in Windows Server 2016 (I'm assuming you folks are testing in Windows Server 2012 R2). -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1470250 Title: [Hyper-V] Ubuntu 14.04.2 LTS Generation 2 SCSI Errors on VSS Based Backups To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1470250/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1470250] Re: [Hyper-V] Ubuntu 14.04.2 LTS Generation 2 SCSI Errors on VSS Based Backups
One reason you may see the timeout messages is due to having a mismatch between the user-space hv_vss_daemon version and the kernel version. Can you rebuild the user-space hv_vss_daemon under the source tree's tools/hv directory and replace the one provided that's provided in Ubuntu by default? -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1470250 Title: [Hyper-V] Ubuntu 14.04.2 LTS Generation 2 SCSI Errors on VSS Based Backups To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1470250/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1470250] Re: [Hyper-V] Ubuntu 14.04.2 LTS Generation 2 SCSI Errors on VSS Based Backups
Might be worth trying this patchset: https://lkml.org/lkml/2016/8/18/859 The first patch in the set addresses some issues with VSS that would cause it to take a long time to initiate backup (and even may timeout). The second patch is not necessary (but will need VSS daemon to be replaced if you choose to apply it). -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1470250 Title: [Hyper-V] Ubuntu 14.04.2 LTS Generation 2 SCSI Errors on VSS Based Backups To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1470250/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1584597] Re: Hyper-V Memory Ballooning re-broken in 16.04
> Committed_AS: 7291440 kB The amount of committed memory reported by meminfo is about 7GB. So it shouldn't be surprising to see that the demand as seen by Hyper-V is also high. In general, the Hyper-V demand is calculated as Committed_AS value plus some buffer(roughly 700MB if you have 8GB of total ram). So for the above meminfo report, I would expect Hyper-V to show that about ~7-8GB is in use. Can you show /proc/meminfo when things have calmed down (as well as a screenshot of what Hyper-V is showing at the same time)? -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1584597 Title: Hyper-V Memory Ballooning re-broken in 16.04 To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1584597/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1584597] Re: Hyper-V Memory Ballooning re-broken in 16.04
The hv_balloon driver hasn't changed between 15.10 and 16.04, so there shouldn't be any difference in the way the driver reports demand to Hyper-V. To provide a further breakdown of the memory usage, can you show the output of "cat /proc/meminfo"? Might help to compare this info between 16.04 and 15.10. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1584597 Title: Hyper-V Memory Ballooning re-broken in 16.04 To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1584597/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1584597] Re: Hyper-V Memory Ballooning re-broken in 16.04
One other caveat I should mention: Even if the VM has reduced its memory consumption, Hyper-V does not necessarily reclaim the unused memory from that VM. Generally, Hyper-V reclaims unused memory from a VM if it's seeing memory pressure from other VMs or if the Hyper-V host itself is seeing memory pressure. The behavior you are seeing might be expected. Indeed, if your physical host has a lot of physical memory, Hyper-V may decide that enough memory is available to satisfy all VMs and it doesn't need to reclaim. If say a bunch of other VMs come online that also need a lot of memory, perhaps we would see Hyper-V start to reclaim the unused memory from the existing VMs. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1584597 Title: Hyper-V Memory Ballooning re-broken in 16.04 To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1584597/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1584597] Re: Hyper-V Memory Ballooning re-broken in 16.04
Thanks. In both VMs, it looks like the buff/cached memory dropped by 2GB. After some time, did the Hyper-V host eventually reclaim this memory? Can you check if the Hyper-V host's assigned memory also drops after some time? Also, can you tell us if you tried this on an older build where it doesn't repro? -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1584597 Title: Hyper-V Memory Ballooning re-broken in 16.04 To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1584597/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1584597] Re: Hyper-V Memory Ballooning re-broken in 16.04
In both screenshots, it appears that the "buff/cache" value is very high. Perhaps the balloon driver is reporting the "buff/cache" as in use to the host. Can you try running following commands to flush write buffers and free page cache? 1) sync 2) echo 3 > /proc/sys/vm/drop_caches After running the above, do you see that the "buff/cache" value decreases and does this result in the memory being reclaimed by Hyper-V? -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1584597 Title: Hyper-V Memory Ballooning re-broken in 16.04 To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1584597/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1584597] Re: Hyper-V Memory Ballooning re-broken in 16.04
Hi Jason, As Josh mentioned, the screenshot shows that demand as seen by Hyper-V is still quite high compared to what is displayed inside the guest. If this number doesn't settle, then Hyper-V would have no reason to reclaim memory. Does the Hyper-V memory demand number ever settle after some time? With the "free" command, you should be seeing the following if memory is being returned to the host successfully: - Because Linux guests return memory via ballooning instead of hot- unplug; you'll never see the "total" memory decrease (i.e. it'll always reflect the spike) - Instead, you should see that "used" memory increases and "free" memory decreases. This indicates the kernel allocated some memory so no one can use it, and that Hyper-V has reclaimed this memory. (i.e. the used memory should increase by roughly the amount that the host has reclaimed) -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1584597 Title: Hyper-V Memory Ballooning re-broken in 16.04 To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1584597/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1470250] Re: [Hyper-V] Ubuntu 14.04.2 LTS Generation 2 SCSI Errors on VSS Based Backups
Also, to add one other thing. There were a bunch of commits made upstream to the storvsc driver in the last few months. Can we try them out to see if they have any impact on this issue? In particular: 1) 81988a0e6b031bc80da15257201810ddcf989e64 - storvsc: get rid of bounce buffer 2) 3209f9d780d137cdb54c85e0a776cb19e723a170 - scsi: storvsc: Fix a bug in the handling of SRB status flags 3) 03996f2064a5c5b7c1bd942794d622179acf2d61 - storvsc: Tighten up the interrupt path -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1470250 Title: [Hyper-V] Ubuntu 14.04.2 LTS Generation 2 SCSI Errors on VSS Based Backups To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1470250/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1470250] Re: [Hyper-V] Ubuntu 14.04.2 LTS Generation 2 SCSI Errors on VSS Based Backups
To help see if this is an issue in the hv_storvsc driver; I took the storvsc driver code from 3.13.0-34.60 (presumably a good build) and applied it to the 4.2.0-27 kernel (presumably a bad build). Ran tiobench with backups and was able to repro after about 48 hours. This implies that either: 1) The bad commit is not in the hv_storvsc driver. 2) 3.13.0-34 is not a good build as we previously thought. I'll wait to hear if other people can repro this issue on 3.13.0-34. The storvsc driver code I took for this snapshot => http://kernel.ubuntu.com/git/ubuntu/ubuntu- trusty.git/tree/drivers/scsi/storvsc_drv.c?id=3402ec8cbd984e8e2a1f99b51ec6a15a23328223 -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1470250 Title: [Hyper-V] Ubuntu 14.04.2 LTS Generation 2 SCSI Errors on VSS Based Backups To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1470250/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1470250] Re: [Hyper-V] Ubuntu 14.04.2 LTS Generation 2 SCSI Errors on VSS Based Backups
Don't think this has been asked before, but has anyone had a repro when backups were turned off? Or does this only happen when backups are enabled? I'm verifying this on my own as well, but if this happens regardless of whether backup is enabled/disabled; then it'll help us narrow down the cause further to either a storage issue (more likely) or a VSS backup issue. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1470250 Title: [Hyper-V] Ubuntu 14.04.2 LTS Generation 2 SCSI Errors on VSS Based Backups To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1470250/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1470250] Re: [Hyper-V] Ubuntu 14.04.2 LTS Generation 2 SCSI Errors on VSS Based Backups
Any updates on whether adbb4e646 test kernel is able to repro this issue? In any case, if we can get a next test kernel to try, I can help try repro on it as well. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1470250 Title: [Hyper-V] Ubuntu 14.04.2 LTS Generation 2 SCSI Errors on VSS Based Backups To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1470250/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1470250] Re: [Hyper-V] Ubuntu 14.04.2 LTS Generation 2 SCSI Errors on VSS Based Backups
I'm also going to try reverting this commit on a 4.2.0-27 kernel that I was able to see this issue on; and see if I can repro it there as well. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1470250 Title: [Hyper-V] Ubuntu 14.04.2 LTS Generation 2 SCSI Errors on VSS Based Backups To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1470250/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1470250] Re: [Hyper-V] Ubuntu 14.04.2 LTS Generation 2 SCSI Errors on VSS Based Backups
Hi Joseph, Any updates on the bisect? -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1470250 Title: [Hyper-V] Ubuntu 14.04.2 LTS Generation 2 SCSI Errors on VSS Based Backups To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1470250/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1470250] Re: [Hyper-V] Ubuntu 14.04.2 LTS Generation 2 SCSI Errors on VSS Based Backups
Thanks for the update @jsalisbury. 7af024a isn't likely to be the cause, as that commit only changes behavior during VMBus shutdown (i.e. when we cleanup VMBus during guest shutdown). This leads me to think that commit c8c38b3 is more likely to be a factor. Nonetheless, I'll withhold further comment until we see results from testing commit 7af024a. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1470250 Title: [Hyper-V] Ubuntu 14.04.2 LTS Generation 2 SCSI Errors on VSS Based Backups To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1470250/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1470250] Re: [Hyper-V] Ubuntu 14.04.2 LTS Generation 2 SCSI Errors on VSS Based Backups
Thanks for going through this testing. Based on your results, I looked at the Hyper-V related commits in http://kernel.ubuntu.com/git/ubuntu/ubuntu-trusty.git between 3.13.0-26.48 (GOOD) and 3.13.0-35.62 (BAD). >From reviewing these commits, there doesn't seem to be an obvious culprit. But it's possible that anyone of the following could affect storage driver behavior. Looking forward to seeing what you uncover in your bisect. - a61e9104ea3b183a3b3cfdaa43145e1cd26b21ff Drivers: hv: vmbus: Negotiate version 3.0 when running on ws2012r2 hosts - c8c38b34f5790366d98306c1fc418ddff1b0357b Drivers: hv: vmbus: Fix a bug in the channel callback dispatch code -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1470250 Title: [Hyper-V] Ubuntu 14.04.2 LTS Generation 2 SCSI Errors on VSS Based Backups To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1470250/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1470250] Re: [Hyper-V] Ubuntu 14.04.2 LTS Generation 2 SCSI Errors on VSS Based Backups
Thanks for testing this. We will continue looking at this internally. It's possible that the issue's with the storage drivers. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1470250 Title: [Hyper-V] Ubuntu 14.04.2 LTS Generation 2 SCSI Errors on VSS Based Backups To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1470250/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1470250] Re: [Hyper-V] Ubuntu 14.04.2 LTS Generation 2 SCSI Errors on VSS Based Backups
While we're not completely certain this is the smoking gun, I've observed that in the logs being posted here that there are no freeze/thaw operations taking place when the issue occurs. The commits that @jrp identified, fix messaging between the VSS utility driver and the host; and will hopefully ensure expected freeze/thawing of the filesystem during backup. In our internal testing with the latest Linux-next kernel (which contains these commits) we were unable to hit this issue using scripts from @jsalisbury. We look forward to hearing back from the testing. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1470250 Title: [Hyper-V] Ubuntu 14.04.2 LTS Generation 2 SCSI Errors on VSS Based Backups To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1470250/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1470250] Re: [Hyper-V] Ubuntu 14.04.2 LTS Generation 2 SCSI Errors on VSS Based Backups
In addition to what @jrp mentioned, I would also add: b9830d120cbe155863399f25eaef6aa8353e767f "Drivers: hv: util: Pass the channel information during the init call" -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1470250 Title: [Hyper-V] Ubuntu 14.04.2 LTS Generation 2 SCSI Errors on VSS Based Backups To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1470250/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1508706] Re: Networking hangs on azure using hv_netvsc; bisected
Hi Jay, Does this issue occur even when VXLAN is not configured between the two instances? -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1508706 Title: Networking hangs on azure using hv_netvsc; bisected To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1508706/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1439780] Re: [Hyper-V] Fiber Channel critical target error
This test kernel fixes the issue. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1439780 Title: [Hyper-V] Fiber Channel critical target error To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1439780/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1439780] Re: [Hyper-V] Fiber Channel critical target error
Hi, I'm finished with the bisect and the following commit will fix this issue: dc45708ca9988656d706940df5fd102672c5de92 [storvsc: Set the SRB flags correctly when no data transfer is needed] Looks like we'll want to include this patch. Here's the rest of my bisect log, in case you're interested: # bad: [ba155e2d21f6bf05de86a78dbe5bfd8757604a65] Linux 4.1-rc5 # good: [e26081808edadfd257c6c9d81014e3b25e9a6118] Linux 4.1-rc4 git bisect start 'v4.1-rc5' 'v4.1-rc4' # good: [1d82b0baf9abd59ae6f53f3102f4e442043763a4] Merge branch 'for-linus' of git://git.kernel.org/pub/scm/linux/kernel/git/dtor/input git bisect good 1d82b0baf9abd59ae6f53f3102f4e442043763a4 # good: [0b6280c62026168f79ff4dd1437df131bdfd24f2] Merge git://git.kernel.org/pub/scm/linux/kernel/git/davem/net git bisect good 0b6280c62026168f79ff4dd1437df131bdfd24f2 # good: [b0f155ada4c819f06aa32b4c906e7e76350c7ec1] drm/exynos: dp: Lower level of EDID read success message git bisect good b0f155ada4c819f06aa32b4c906e7e76350c7ec1 # good: [7ce14f6ff26460819345fe8495cf2dd6538b7cdc] Merge branch 'for-linus-4.1' of git://git.kernel.org/pub/scm/linux/kernel/git/mason/linux-btrfs git bisect good 7ce14f6ff26460819345fe8495cf2dd6538b7cdc # good: [c5db6a3bdeb72f4238e1faefa4ce4eab7a64baea] Merge branch 'timers-urgent-for-linus' of git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip git bisect good c5db6a3bdeb72f4238e1faefa4ce4eab7a64baea # bad: [74856fbf441929918c49ff262ace9835048e4e6a] sd: Disable support for 256 byte/sector disks git bisect bad 74856fbf441929918c49ff262ace9835048e4e6a # bad: [8b2564ec7410928639db5c09a34d7d8330f1d759] lpfc: Fix breakage on big endian kernels git bisect bad 8b2564ec7410928639db5c09a34d7d8330f1d759 # bad: [dc45708ca9988656d706940df5fd102672c5de92] storvsc: Set the SRB flags correctly when no data transfer is needed git bisect bad dc45708ca9988656d706940df5fd102672c5de92 # first bad commit: [dc45708ca9988656d706940df5fd102672c5de92] storvsc: Set the SRB flags correctly when no data transfer is needed -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1439780 Title: [Hyper-V] Fiber Channel critical target error To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1439780/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1439780] Re: [Hyper-V] Fiber Channel critical target error
Hi Joseph, Kernel with commits up to 7ce14f6ff26460819345fe8495cf2dd6538b7cdc are also seeing this issue. I'm can run my own bisect as well so we can save some time going back and forth. Are you basing your bisect off of Linux-stable repository? Thanks, Alex -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1439780 Title: [Hyper-V] Fiber Channel critical target error To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1439780/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1439780] Re: [Hyper-V] Fiber Channel critical target error
Hi, The issue also shows up in this kernel. I can take over the bisect. My current bisect log is the same as yours. I'm currently building the next test kernel up to following commit: 74856fbf441929918c49ff262ace9835048e4e6a Will let you know when I narrow it down further. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1439780 Title: [Hyper-V] Fiber Channel critical target error To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1439780/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1439780] Re: [Hyper-V] Fiber Channel critical target error
The issue also shows in this kernel. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1439780 Title: [Hyper-V] Fiber Channel critical target error To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1439780/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1439780] Re: [Hyper-V] Fiber Channel critical target error
Still not working with the kernel up to this commit: 0b6280c62026168f79ff4dd1437df131bdfd24f2 -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1439780 Title: [Hyper-V] Fiber Channel critical target error To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1439780/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1439780] Re: [Hyper-V] Fiber Channel critical target error
The test kernel still hits this issue. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1439780 Title: [Hyper-V] Fiber Channel critical target error To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1439780/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1439780] Re: [Hyper-V] Fiber Channel critical target error
So I tried to repro this on the latest upstream v4.1-fc5 kernel from http://kernel.ubuntu.com/~kernel-ppa/mainline/v4.1-rc5-unstable/. It looks like the issue's been fixed there. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1439780 Title: [Hyper-V] Fiber Channel critical target error To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1439780/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1439780] Re: [Hyper-V] Fiber Channel critical target error
To answer your question, it looks like the issue can be reproduced as recently as v4.1-rc4. So the last version of the kernel that we see this bug is: v4.1-rc4 And the first version of the kernel we don't see this bug is: v4.1-rc5 -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1439780 Title: [Hyper-V] Fiber Channel critical target error To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1439780/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 805613] Re: mutt treats email address as file name when using attachments
The man page for mutt has for the -a option: -a file [...] Attach a file to your message using MIME. When attaching single or multiple files, separating filenames and recipient addresses with -- is mandatory, e.g. mutt -a image.jpg -- addr1 or mutt -a img.jpg *.png -- addr1 addr2. The -a option must be placed at the end of command line options. so it would appear the double dash '--' is mandatory, and is working as designed. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/805613 Title: mutt treats email address as file name when using attachments To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/mutt/+bug/805613/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs