Re: cam(4) timeouts in bhyve/kyua runs up on Jenkins?
Hi Neel, On Tue, May 26, 2015 at 9:06 AM, Neel Natu wrote: > Hi NGie, ... > Can you try to reproduce with a virtio-blk instead of ahci-hd? > > It will help narrow down the problem because both ahci-hd and > virtio-blk emulations share a common backend to read/write to the > disk. You'll have to ask Craig. I don't run BHyve at all, and I don't think that running BHyve over VMware Fusion would be the best "proof of concept" for determining whether or not there's an issue here as VMware Fusion suffers timeouts on occasion with mpt(4) under high load.. Thanks! ___ freebsd-current@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"
Re: cam(4) timeouts in bhyve/kyua runs up on Jenkins?
On 26.05.2015 09:06, Neel Natu wrote: > On Mon, May 25, 2015 at 1:40 PM, Garrett Cooper wrote: >> On Apr 28, 2015, at 0:54, Alexander Motin wrote: >>> On 27.04.2015 21:17, Garrett Cooper wrote: On Apr 27, 2015, at 11:16, Garrett Cooper wrote: > I was looking at the console log for the latest kyua run and I’ve > noticed that it’s timing out a bit more [1] than it was > previously [2]. I’ve seen some of your commits recently to cam(4) > dealing with bhyve — has there been a performance regression > there? Thanks! -NGie > > 1. > https://jenkins.freebsd.org/job/FreeBSD_HEAD-tests2/940/console > 2. > https://jenkins.freebsd.org/job/FreeBSD_HEAD-tests2/983/console (Sorry for not being more explicit for the archives) These are the timeouts I’m referring to: ahcich0: is cs ss 1f00 rs 1f00 tfd 50 serr cmd 1000dc17 (ada0:ahcich0:0:0:0): WRITE_FPDMA_QUEUED. ACB: 61 08 a8 54 1e 40 00 00 00 00 00 00 (ada0:ahcich0:0:0:0): CAM status: Command timeout (ada0:ahcich0:0:0:0): Retrying command > > Can you try to reproduce with a virtio-blk instead of ahci-hd? > > It will help narrow down the problem because both ahci-hd and > virtio-blk emulations share a common backend to read/write to the > disk. virtio-blk has no command timeouts, so it may be problematic to notice delayed commands there, unless they stuck forever. This won't be fair comparison. Though if not use TRIM, virtio-blk should be more efficient now, as it is supposed to. >>> Last time I was more working on bhyve host disk emulation, rather then >>> on cam(4) running on guest. Considering that, what guest and what host >>> versions are you running? Is there any other load on host except this VM >>> that could cause I/O delays high enough to trigger timeouts? What are >>> you using to back the virtual disk (file, zvol, ...)? >> >> I have no idea what the Jenkins slaves are running in terms of >> configuration/version/etc. You’ll have to ask jenkins-admin@… >> Thanks! >> -NGie -- Alexander Motin ___ freebsd-current@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"
Re: cam(4) timeouts in bhyve/kyua runs up on Jenkins?
Hi NGie, On Mon, May 25, 2015 at 1:40 PM, Garrett Cooper wrote: > On Apr 28, 2015, at 0:54, Alexander Motin wrote: > >> Hi. >> >> On 27.04.2015 21:17, Garrett Cooper wrote: >>> On Apr 27, 2015, at 11:16, Garrett Cooper >>> wrote: >>> I was looking at the console log for the latest kyua run and I’ve noticed that it’s timing out a bit more [1] than it was previously [2]. I’ve seen some of your commits recently to cam(4) dealing with bhyve — has there been a performance regression there? Thanks! -NGie 1. https://jenkins.freebsd.org/job/FreeBSD_HEAD-tests2/940/console 2. https://jenkins.freebsd.org/job/FreeBSD_HEAD-tests2/983/console >>> >>> (Sorry for not being more explicit for the archives) These are the >>> timeouts I’m referring to: >>> >>> ahcich0: is cs ss 1f00 rs 1f00 tfd 50 >>> serr cmd 1000dc17 (ada0:ahcich0:0:0:0): >>> WRITE_FPDMA_QUEUED. ACB: 61 08 a8 54 1e 40 00 00 00 00 00 00 >>> (ada0:ahcich0:0:0:0): CAM status: Command timeout >>> (ada0:ahcich0:0:0:0): Retrying command Can you try to reproduce with a virtio-blk instead of ahci-hd? It will help narrow down the problem because both ahci-hd and virtio-blk emulations share a common backend to read/write to the disk. best Neel >> >> Last time I was more working on bhyve host disk emulation, rather then >> on cam(4) running on guest. Considering that, what guest and what host >> versions are you running? Is there any other load on host except this VM >> that could cause I/O delays high enough to trigger timeouts? What are >> you using to back the virtual disk (file, zvol, ...)? > > I have no idea what the Jenkins slaves are running in terms of > configuration/version/etc. You’ll have to ask jenkins-admin@… > Thanks! > -NGie ___ freebsd-current@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"
Re: cam(4) timeouts in bhyve/kyua runs up on Jenkins?
On Apr 28, 2015, at 0:54, Alexander Motin wrote: > Hi. > > On 27.04.2015 21:17, Garrett Cooper wrote: >> On Apr 27, 2015, at 11:16, Garrett Cooper >> wrote: >> >>> I was looking at the console log for the latest kyua run and I’ve >>> noticed that it’s timing out a bit more [1] than it was >>> previously [2]. I’ve seen some of your commits recently to cam(4) >>> dealing with bhyve — has there been a performance regression >>> there? Thanks! -NGie >>> >>> 1. >>> https://jenkins.freebsd.org/job/FreeBSD_HEAD-tests2/940/console >>> 2. >>> https://jenkins.freebsd.org/job/FreeBSD_HEAD-tests2/983/console >> >> (Sorry for not being more explicit for the archives) These are the >> timeouts I’m referring to: >> >> ahcich0: is cs ss 1f00 rs 1f00 tfd 50 >> serr cmd 1000dc17 (ada0:ahcich0:0:0:0): >> WRITE_FPDMA_QUEUED. ACB: 61 08 a8 54 1e 40 00 00 00 00 00 00 >> (ada0:ahcich0:0:0:0): CAM status: Command timeout >> (ada0:ahcich0:0:0:0): Retrying command > > Last time I was more working on bhyve host disk emulation, rather then > on cam(4) running on guest. Considering that, what guest and what host > versions are you running? Is there any other load on host except this VM > that could cause I/O delays high enough to trigger timeouts? What are > you using to back the virtual disk (file, zvol, ...)? I have no idea what the Jenkins slaves are running in terms of configuration/version/etc. You’ll have to ask jenkins-admin@… Thanks! -NGie signature.asc Description: Message signed with OpenPGP using GPGMail
Re: cam(4) timeouts in bhyve/kyua runs up on Jenkins?
Hi. On 27.04.2015 21:17, Garrett Cooper wrote: > On Apr 27, 2015, at 11:16, Garrett Cooper > wrote: > >> I was looking at the console log for the latest kyua run and I’ve >> noticed that it’s timing out a bit more [1] than it was >> previously [2]. I’ve seen some of your commits recently to cam(4) >> dealing with bhyve — has there been a performance regression >> there? Thanks! -NGie >> >> 1. >> https://jenkins.freebsd.org/job/FreeBSD_HEAD-tests2/940/console >> 2. >> https://jenkins.freebsd.org/job/FreeBSD_HEAD-tests2/983/console > > (Sorry for not being more explicit for the archives) These are the > timeouts I’m referring to: > > ahcich0: is cs ss 1f00 rs 1f00 tfd 50 > serr cmd 1000dc17 (ada0:ahcich0:0:0:0): > WRITE_FPDMA_QUEUED. ACB: 61 08 a8 54 1e 40 00 00 00 00 00 00 > (ada0:ahcich0:0:0:0): CAM status: Command timeout > (ada0:ahcich0:0:0:0): Retrying command Last time I was more working on bhyve host disk emulation, rather then on cam(4) running on guest. Considering that, what guest and what host versions are you running? Is there any other load on host except this VM that could cause I/O delays high enough to trigger timeouts? What are you using to back the virtual disk (file, zvol, ...)? -- Alexander Motin ___ freebsd-current@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"
Re: cam(4) timeouts in bhyve/kyua runs up on Jenkins?
On Apr 27, 2015, at 11:16, Garrett Cooper wrote: > Hi mav! > I was looking at the console log for the latest kyua run and I’ve > noticed that it’s timing out a bit more [1] than it was previously [2]. I’ve > seen some of your commits recently to cam(4) dealing with bhyve — has there > been a performance regression there? > Thanks! > -NGie > > 1. https://jenkins.freebsd.org/job/FreeBSD_HEAD-tests2/940/console > 2. https://jenkins.freebsd.org/job/FreeBSD_HEAD-tests2/983/console (Sorry for not being more explicit for the archives) These are the timeouts I’m referring to: ahcich0: is cs ss 1f00 rs 1f00 tfd 50 serr cmd 1000dc17 (ada0:ahcich0:0:0:0): WRITE_FPDMA_QUEUED. ACB: 61 08 a8 54 1e 40 00 00 00 00 00 00 (ada0:ahcich0:0:0:0): CAM status: Command timeout (ada0:ahcich0:0:0:0): Retrying command signature.asc Description: Message signed with OpenPGP using GPGMail
cam(4) timeouts in bhyve/kyua runs up on Jenkins?
Hi mav! I was looking at the console log for the latest kyua run and I’ve noticed that it’s timing out a bit more [1] than it was previously [2]. I’ve seen some of your commits recently to cam(4) dealing with bhyve — has there been a performance regression there? Thanks! -NGie 1. https://jenkins.freebsd.org/job/FreeBSD_HEAD-tests2/940/console 2. https://jenkins.freebsd.org/job/FreeBSD_HEAD-tests2/983/console signature.asc Description: Message signed with OpenPGP using GPGMail
