Re: USB console support "was: NetBSD-7.0 boots OK and NetBSD-8.0 hangs/crashes during boot on a MacBook7,1)
At Thu, 09 Jul 2020 18:16:26 -0700, "Greg A. Woods" wrote: Subject: USB console support "was: NetBSD-7.0 boots OK and NetBSD-8.0 hangs/crashes during boot on a MacBook7,1) > > Oh, and I wanted to mention something else that I'd forgotten about but > stumbled across again the other day while debugging servers: > > Xen supports writing console messages to a special kind of USB port: > > "console=dbgp" indicates that Xen should use a USB debug port. > > http://xenbits.xenproject.org/docs/4.11-testing/misc/xen-command-line.html > > There's more about it in this thread: > > https://lists.xenproject.org/archives/html/xen-devel/2009-03/msg00436.html > https://lists.xenproject.org/archives/html/xen-devel/2009-03/msg00458.html For what it's worth my Dell servers and my MacBook Pro have such USB debug ports. The MacBook Pro even has two of them, and I'm pretty sure one of them is connected to the external ports. On the Dell though this seems to be the port that connects to the DRAC. From "lspci -vvv": 00:1d.7 USB controller: Intel Corporation 631xESB/632xESB/3100 Chipset EHCI USB2 Controller (rev 09) (prog-if 20 [EHCI]) Subsystem: Dell Device 01b2 Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR+ FastB2B- DisINTx- Status: Cap+ 66MHz- UDF- FastB2B+ ParErr- DEVSEL=medium >TAbort- SERR- Kelowna, BC +1 250 762-7675 RoboHack Planix, Inc. Avoncote Farms pgpwb_S2z2w5O.pgp Description: OpenPGP Digital Signature
USB console support "was: NetBSD-7.0 boots OK and NetBSD-8.0 hangs/crashes during boot on a MacBook7,1)
At Mon, 06 Jul 2020 13:13:03 -0700, "Greg A. Woods" wrote: Subject: Re: NetBSD-7.0 boots OK and NetBSD-8.0 hangs/crashes during boot on a MacBook7,1 > > Or indeed any device with any kind of USB port, e.g. a laptop. Oh, and I wanted to mention something else that I'd forgotten about but stumbled across again the other day while debugging servers: Xen supports writing console messages to a special kind of USB port: "console=dbgp" indicates that Xen should use a USB debug port. http://xenbits.xenproject.org/docs/4.11-testing/misc/xen-command-line.html There's more about it in this thread: https://lists.xenproject.org/archives/html/xen-devel/2009-03/msg00436.html https://lists.xenproject.org/archives/html/xen-devel/2009-03/msg00458.html Further of interest is that Xen also supports writing to both a COM port and the "vga" console simultaneously. Indeed it may support writing to "dbgp" at the same time as well. This is something I looked into for NetBSD/i386 some time ago, but never got it fully working. To me I think it would be super incredibly valuable for the boot code to be able to talk to both a serial port and the "pc" console simultaneously. It is less important for the kernel to do so, but it still would be nice. -- Greg A. Woods Kelowna, BC +1 250 762-7675 RoboHack Planix, Inc. Avoncote Farms pgpWgRz8XQc4Z.pgp Description: OpenPGP Digital Signature
USB keyboard and ddb [was Re: NetBSD-7.0 boots OK and NetBSD-8.0 hangs/crashes during boot on a MacBook7,1]
On 06/07/2020 10:34, Andreas Gustafsson wrote: > Martin Husemann wrote: >> USB keyboards as console in ddb worked fine last I tested. > > That has not been my experience. For example, > >PR 52569 Entering ddb using USB keyboard panics with "locking against myself" I just committed something that will help >PR 54599 Can't enter ddb using USB keyboard because console PR 53599 Can't enter ddb using USB keyboard because console I don't know about this one, sorry. Nick
Re: NetBSD-7.0 boots OK and NetBSD-8.0 hangs/crashes during boot on a MacBook7,1
At Mon, 6 Jul 2020 23:53:02 +0900, Rin Okuyama wrote: Subject: Re: NetBSD-7.0 boots OK and NetBSD-8.0 hangs/crashes during boot on a MacBook7,1 > > It seems that stride of framebuffer is not correctly set. > > Your laptop has an NVIDIA GPU, doesn't it? If so, nouveaufb(4) is used > instead of genfb(4), which is normally used when booted from UEFI. It > should be worth trying Yes, indeed, it has an NVIDIA GeForce 320M. > userconf disable nouveau* > > for UEFI bootloader. Oh, that sounded so very promising! However unfortunately it made not one bit of difference. Thank you for the idea though, and also thank you for pointing out the alternate framebuffer driver that might also be worth looking into. -- Greg A. Woods Kelowna, BC +1 250 762-7675 RoboHack Planix, Inc. Avoncote Farms pgpDKqSxprCns.pgp Description: OpenPGP Digital Signature
Re: NetBSD-7.0 boots OK and NetBSD-8.0 hangs/crashes during boot on a MacBook7,1
At Sun, 5 Jul 2020 21:09:27 -0700, Brian Buhrow wrote: Subject: Re: NetBSD-7.0 boots OK and NetBSD-8.0 hangs/crashes during boot on a MacBook7,1 > > Hello. I agree with Mouse, except that I also think it would be very > helpful and useful to have a serial console on USB only devices. Or indeed any device with any kind of USB port, e.g. a laptop. However what would be most generally useful, as opposed to ideal, would be for just the console output to appear on the first found USB serial adapter. So if the kernel can get far enough to probe a USB serial port, then it should dump the message buffer, and continue to copy everything added to the message buffer, to that USB serial device. That's the first and most important step. Make it simple, easy, and obvious how to capture all kernel messages on a modern machine without having to get all the way to the point where one can run "dmesg". Further allowing that port to be attached as the console would be "nice but not quite as necessary". Now ideally the kernel should make the best attempt to identify the first possible USB serial port as early as possible, and attach it as console, so that nothing can be missed, and so that any other bugs in device probing, etc., etc., etc., would not prevent use of DDB on this USB serial console. Even better would be to find out if the platform firmware can do some or all of this, and then to use that code both for the boot loader and the kernel console. E.g. on an EFI system, perhaps through a custom EFI driver? And for uBoot systems too? -- Greg A. Woods Kelowna, BC +1 250 762-7675 RoboHack Planix, Inc. Avoncote Farms pgpFUqYZ0ZSmB.pgp Description: OpenPGP Digital Signature
Re: NetBSD-7.0 boots OK and NetBSD-8.0 hangs/crashes during boot on a MacBook7,1
On Mon, Jul 06, 2020 at 05:07:00PM +0200, Martin Husemann wrote: > On Mon, Jul 06, 2020 at 07:55:29AM -0700, Brian Buhrow wrote: > > hello. In my case, there are times when I want a serial console, for > > set up or troubleshooting, but cannot use the built-in display for various > > reasons. So, I think it would be useful in more situations than might > > first appear. Yes, it wouldn't give you DDB on that console, but for > > environments where the kernel loads and runs, it would give you access to > > everything else over a serial port. > > Stupid question: are there now actually x86 boards that do *not* have a real > serial on-board? I have not seen any so far (none of the new ones come with > an external connector of course, but they can be added easily unless it is > a notebook). Not a stupid question at all. Short answer is "yes", unfortunately. I don't buy brand-new boards very frequently, so it hasn't completely impacted me yet, but dropping the connector already seems common, and last time I shopped ITX for a small home server I ended up eliminating a lot of candidates (maybe over 50%) because they didn't even have the COM header on the board so I could add the connector myself. I'd always hoped that, if anything, the "legacy" db9 connector might over time start turning into rj45 serial connectors with telco pinouts ala Cisco and other devices. But it doesn't seem a likely trend for PC systems at this point. Even some of the recent name-brand "enterprise" rackmount systems at $work didn't come with a serial console port out the back -- it was an add-on option at most. The vendors presumably want to push their GUI BMC paradigm, with the graphical server management goop. Some of it is admittedly nice functionality, but a lot of it is simply overkill when a good ole serial port would be fine. I think it's already been mentioned downthread, but other systems in this category are smaller devices like NUC and similar. The low power (noise, heat etc.) profile makes those attractive, but without a serial console they're not as nice. Best I've been able to do with a couple of those is a USB-serial terminal (!=console) which is better than nothing, e.g. to fix a broken network config, but doesn't help when you need to interact with boot stuff. Cheers, sr.
Re: NetBSD-7.0 boots OK and NetBSD-8.0 hangs/crashes during boot on a MacBook7,1
> On Jul 6, 2020, at 10:07 AM, Mike Pumford wrote: > > On 06/07/2020 16:07, Martin Husemann wrote: >> Stupid question: are there now actually x86 boards that do *not* have a real >> serial on-board? I have not seen any so far (none of the new ones come with >> an external connector of course, but they can be added easily unless it is >> a notebook). > A quick look around suggests that some of the very high end gaming ones > don't. Also assuming users will actually be able to find a cable to actually > hook up the motherboard COM port is optimistic. You would probably have to > get one second hand these days and if I remember correctly there are 2 > incompatible pinouts for the 10 pin header. :( Just in case anyone cares (I was looking to order one recently), there are definitely two pinouts. One is the so-called AT/Everex pinout, the other is the so-called DTK/Intel pinout. They are described here among other places: https://pinoutguide.com/Motherboard/rs232_header_pinout.shtml Cheers, Brook
Re: NetBSD-7.0 boots OK and NetBSD-8.0 hangs/crashes during boot on a MacBook7,1
On Mon, 6 Jul 2020, Martin Husemann wrote: On Mon, Jul 06, 2020 at 05:07:51PM +0100, Mike Pumford wrote: A quick look around suggests that some of the very high end gaming ones don't. Also assuming users will actually be able to find a cable to actually hook up the motherboard COM port is optimistic. You would probably have to get one second hand these days and if I remember correctly there are 2 incompatible pinouts for the 10 pin header. :( I had no trouble ordering new ones late last year. If you wanted a branded one (!), there's a Lenovo server option with SKU 7Z17A02577 (I don't know about the pinout off-hand, but I have one here I could buzz through if anyone really cared). -- Stephen
Re: NetBSD-7.0 boots OK and NetBSD-8.0 hangs/crashes during boot on a MacBook7,1
On Mon, Jul 06, 2020 at 05:07:51PM +0100, Mike Pumford wrote: > A quick look around suggests that some of the very high end gaming ones > don't. Also assuming users will actually be able to find a cable to actually > hook up the motherboard COM port is optimistic. You would probably have to > get one second hand these days and if I remember correctly there are 2 > incompatible pinouts for the 10 pin header. :( I had no trouble ordering new ones late last year. Martin
Re: NetBSD-7.0 boots OK and NetBSD-8.0 hangs/crashes during boot on a MacBook7,1
On 06/07/2020 16:07, Martin Husemann wrote: Stupid question: are there now actually x86 boards that do *not* have a real serial on-board? I have not seen any so far (none of the new ones come with an external connector of course, but they can be added easily unless it is a notebook). A quick look around suggests that some of the very high end gaming ones don't. Also assuming users will actually be able to find a cable to actually hook up the motherboard COM port is optimistic. You would probably have to get one second hand these days and if I remember correctly there are 2 incompatible pinouts for the 10 pin header. :( Mike
Re: NetBSD-7.0 boots OK and NetBSD-8.0 hangs/crashes during boot on a MacBook7,1
Brian Buhrow wrote: > Hello. I'm thinking of notebooks. Yes, they have screens and > keyboards, but those are not always usable and, having a serial console > over USB could let someone install to a notebook remotely. > Also, I've encountered some Intel based appliance boards that don't have > easily > used serial ports on them. When they're installed in cramped wiring > closets, it's much easier to get a USB serial port on them than it is to > get a screen and keyboard. It's not just laptops and appliance boards - even ATX sized PC motherboards have been made with no com ports for a long time, for example the Intel DH67CL from 2011. The specifications at https://ark.intel.com/content/www/us/en/ark/products/50101/intel-desktop-board-dh67cl.html say # of Serial Ports: 0 Serial Port via Internal Header: No and when booting NetBSD on one, the dmesg output contains no "com" entry. -- Andreas Gustafsson, g...@gson.org
Re: NetBSD-7.0 boots OK and NetBSD-8.0 hangs/crashes during boot on a MacBook7,1
Hello. I'm thinking of notebooks. Yes, they have screens and keyboards, but those are not always usable and, having a serial console over USB could let someone install to a notebook remotely. Also, I've encountered some Intel based appliance boards that don't have easily used serial ports on them. When they're installed in cramped wiring closets, it's much easier to get a USB serial port on them than it is to get a screen and keyboard. -Brian On Jul 6, 5:07pm, Martin Husemann wrote: } Subject: Re: NetBSD-7.0 boots OK and NetBSD-8.0 hangs/crashes during boot } On Mon, Jul 06, 2020 at 07:55:29AM -0700, Brian Buhrow wrote: } > hello. In my case, there are times when I want a serial console, for } > set up or troubleshooting, but cannot use the built-in display for various } > reasons. So, I think it would be useful in more situations than might } > first appear. Yes, it wouldn't give you DDB on that console, but for } > environments where the kernel loads and runs, it would give you access to } > everything else over a serial port. } } Stupid question: are there now actually x86 boards that do *not* have a real } serial on-board? I have not seen any so far (none of the new ones come with } an external connector of course, but they can be added easily unless it is } a notebook). } } Martin >-- End of excerpt from Martin Husemann
Re: NetBSD-7.0 boots OK and NetBSD-8.0 hangs/crashes during boot on a MacBook7,1
On Mon, Jul 06, 2020 at 07:55:29AM -0700, Brian Buhrow wrote: > hello. In my case, there are times when I want a serial console, for > set up or troubleshooting, but cannot use the built-in display for various > reasons. So, I think it would be useful in more situations than might > first appear. Yes, it wouldn't give you DDB on that console, but for > environments where the kernel loads and runs, it would give you access to > everything else over a serial port. Stupid question: are there now actually x86 boards that do *not* have a real serial on-board? I have not seen any so far (none of the new ones come with an external connector of course, but they can be added easily unless it is a notebook). Martin
Re: NetBSD-7.0 boots OK and NetBSD-8.0 hangs/crashes during boot on a MacBook7,1
hello. In my case, there are times when I want a serial console, for set up or troubleshooting, but cannot use the built-in display for various reasons. So, I think it would be useful in more situations than might first appear. Yes, it wouldn't give you DDB on that console, but for environments where the kernel loads and runs, it would give you access to everything else over a serial port. -Brian On Jul 6, 6:05am, Mouse wrote: } Subject: Re: NetBSD-7.0 boots OK and NetBSD-8.0 hangs/crashes during boot } > I agree with Mouse, except that I also think it would be very helpful } > and useful to have a serial console on USB only devices. } } Oh, sure, it'd be helpful/useful. Lots of difficult things would be. } } > I wonder if we could make the console a virtual device which is } > attached dynamically to a USB serial port if and when available. } } I have no doubt that could be done. } } I'm not sure how useful it would be. It seems to me that the set of } times when it's most important to have a serial console overlaps } heavily with the set of times when the USB stack is least likely to } work. } } So I guess my reaction is "probably better than nothing - but not by } all that much". } } Reworking the USB stack so serial ports can run polled would probably } help, but my own experience echos Mike Pumford's. I've even had times } when I've had boot troubles and a *PS/2* keyboard didn't work - and at } least half the times when I've used a USB keyboard as console keyboard } (which != a USB serial port as serial console) it's worked because the } BIOS and hardware have collaborated to give the illusion of a PS/2 } keyboard. } } /~\ The ASCII Mouse } \ / Ribbon Campaign } X Against HTML mo...@rodents-montreal.org } / \ Email! 7D C8 61 52 5D E7 2D 39 4E F1 31 3E E8 B3 27 4B >-- End of excerpt from Mouse
Re: NetBSD-7.0 boots OK and NetBSD-8.0 hangs/crashes during boot on a MacBook7,1
On 2020/07/06 10:05, Greg A. Woods wrote: NetBSD-7.2 boots fine from USB on the MacBook Pro (MacBook7,1) (with the help of rEFIT on a second USB stick). NetBSD-8.2 and newer, including the most recent -current, hangs during boot and the kernel messages appear to have torn video: http://www.planix.ca/~woods/macbookpro-netbsd-boot-fail.jpg ... In any case, what might have been changed after 8.0 that broke the video output? Where do I look? Is amd64 video now the genfb(4) device code? Or is it still vga(4)? If it's genfb(4), then I do see commits about doing anti-aliasing, and maybe the video junk I see could possibly be explained by such a thing. It seems that stride of framebuffer is not correctly set. Your laptop has an NVIDIA GPU, doesn't it? If so, nouveaufb(4) is used instead of genfb(4), which is normally used when booted from UEFI. It should be worth trying userconf disable nouveau* for UEFI bootloader. Thanks, rin
Re: NetBSD-7.0 boots OK and NetBSD-8.0 hangs/crashes during boot on a MacBook7,1
> I agree with Mouse, except that I also think it would be very helpful > and useful to have a serial console on USB only devices. Oh, sure, it'd be helpful/useful. Lots of difficult things would be. > I wonder if we could make the console a virtual device which is > attached dynamically to a USB serial port if and when available. I have no doubt that could be done. I'm not sure how useful it would be. It seems to me that the set of times when it's most important to have a serial console overlaps heavily with the set of times when the USB stack is least likely to work. So I guess my reaction is "probably better than nothing - but not by all that much". Reworking the USB stack so serial ports can run polled would probably help, but my own experience echos Mike Pumford's. I've even had times when I've had boot troubles and a *PS/2* keyboard didn't work - and at least half the times when I've used a USB keyboard as console keyboard (which != a USB serial port as serial console) it's worked because the BIOS and hardware have collaborated to give the illusion of a PS/2 keyboard. /~\ The ASCII Mouse \ / Ribbon Campaign X Against HTMLmo...@rodents-montreal.org / \ Email! 7D C8 61 52 5D E7 2D 39 4E F1 31 3E E8 B3 27 4B
Re: NetBSD-7.0 boots OK and NetBSD-8.0 hangs/crashes during boot on a MacBook7,1
On Mon, Jul 06, 2020 at 12:34:32PM +0300, Andreas Gustafsson wrote: > Why would acting as a device be needed? To emulate a serial console on an USB OTG port. Martin
Re: NetBSD-7.0 boots OK and NetBSD-8.0 hangs/crashes during boot on a MacBook7,1
Martin Husemann wrote: > USB keyboards as console in ddb worked fine last I tested. That has not been my experience. For example, PR 52569 Entering ddb using USB keyboard panics with "locking against myself" PR 54599 Can't enter ddb using USB keyboard because console > Running the usb host in polled mode however is quite a bit simpler than > doing the device part (where you have to obey timings from the host). Why would acting as a device be needed? -- Andreas Gustafsson, g...@gson.org
Re: NetBSD-7.0 boots OK and NetBSD-8.0 hangs/crashes during boot on a MacBook7,1
On 06/07/2020 10:01, Martin Husemann wrote: On Mon, Jul 06, 2020 at 09:52:31AM +0100, Mike Pumford wrote: Like it or not its probably going to be a long term requirement to have console USB support. If not for the serial output its becoming more and more necessary just to get a console keyboard for DDB. USB keyboards as console in ddb worked fine last I tested. Running the usb host in polled mode however is quite a bit simpler than doing the device part (where you have to obey timings from the host). That hasn't been my experience. Until I turned off ddb I had several situations when one of my systems crashed during boot and was stuck at a DDB prompt that needed a PS2 keyboard for interaction. Mike
Re: NetBSD-7.0 boots OK and NetBSD-8.0 hangs/crashes during boot on a MacBook7,1
On Mon, Jul 06, 2020 at 09:52:31AM +0100, Mike Pumford wrote: > Like it or not its probably going to be a long term requirement to have > console USB support. If not for the serial output its becoming more and more > necessary just to get a console keyboard for DDB. USB keyboards as console in ddb worked fine last I tested. Running the usb host in polled mode however is quite a bit simpler than doing the device part (where you have to obey timings from the host). Martin
Re: NetBSD-7.0 boots OK and NetBSD-8.0 hangs/crashes during boot on a MacBook7,1
On 06/07/2020 05:09, Brian Buhrow wrote: Hello. I agree with Mouse, except that I also think it would be very helpful and useful to have a serial console on USB only devices. I wonder if we could make the console a virtual device which is attached dynamically to a USB serial port if and when available. that would let the system think it has a console, but one would only see it when the kernel and the USB subsystem are up. Yes, I get this would make watching things boot challenging, but by the time you get to single user mode, the kernel is fully up and running and USB is or should be available by then. Like it or not its probably going to be a long term requirement to have console USB support. If not for the serial output its becoming more and more necessary just to get a console keyboard for DDB. I don't think its possible to by an amd64 motherboard (at least in the consumer space) that actually has a PS2 port for a keyboard any more. Mike
Re: NetBSD-7.0 boots OK and NetBSD-8.0 hangs/crashes during boot on a MacBook7,1
Hello. I agree with Mouse, except that I also think it would be very helpful and useful to have a serial console on USB only devices. I wonder if we could make the console a virtual device which is attached dynamically to a USB serial port if and when available. that would let the system think it has a console, but one would only see it when the kernel and the USB subsystem are up. Yes, I get this would make watching things boot challenging, but by the time you get to single user mode, the kernel is fully up and running and USB is or should be available by then. thoughts?
Re: NetBSD-7.0 boots OK and NetBSD-8.0 hangs/crashes during boot on a MacBook7,1
> That said, what would it take to wire the NetBSD console to a USB > serial adapter? "Too much". USB is horrible from this persective; you need quite a lot of infrastructure to do, well, pretty much anything over USB. Take a look at the code sometime. In the particular case of the console, you'd also have to figure out what to do with the console if the hardware goes away. It's one of the reasons I dislike USB, and closely related to most of the others. /~\ The ASCII Mouse \ / Ribbon Campaign X Against HTMLmo...@rodents-montreal.org / \ Email! 7D C8 61 52 5D E7 2D 39 4E F1 31 3E E8 B3 27 4B
NetBSD-7.0 boots OK and NetBSD-8.0 hangs/crashes during boot on a MacBook7,1
So, in my ongoing NetBSD on a MacBook saga NetBSD-7.2 boots fine from USB on the MacBook Pro (MacBook7,1) (with the help of rEFIT on a second USB stick). NetBSD-8.2 and newer, including the most recent -current, hangs during boot and the kernel messages appear to have torn video: http://www.planix.ca/~woods/macbookpro-netbsd-boot-fail.jpg However today I discovered that NetBSD-8.0 will often boot with the kernel messages properly visible in nice green on black in a full 52(?)-line display, but it hangs or crashes. (It is not reliable at booting though -- sometimes the boot loader just hangs without printing anything.) If the boot loader does work though, and if I boot "normally" it just hangs, with the last message being: pci0 at mainbus0 bus0: configuration mode 1 The caps-lock button is dead so I think the machine is well and truly frozen in a CPU loop (the CPU is hot, the fan runs fast). I'm guessing NetBSD-8.2 and everything more recent is also hanging at this same spot, but with the busted video mode it's hard to tell for sure. If I boot 8.0 with ACPI turned off (boot option #2 or from the boot prompt "boot -2"), it crashes into ddb after getting a bit further, but there are many errors about not being ablt to map PCI interrupts. If I boot 8.0 with "-vx", there are quite a number of "invalid config space" messages after the pci0 attachment: pci0 at mainbus0 bus0: configuration mode 1 acpi0: MCFG: 000:00:0: invalid config space (cfg[0x100]=0x, alias=false) The second and third numbers change in each following message, and in two of those messages the cfg[0x100] number is 0x. So it looks like ACPI is necessary, but support for using it in this MacBook7,1 is broken somehow. I can post a full-res photo of the screen in one or more or all of these states it someone wants to see it. In any case, what might have been changed after 8.0 that broke the video output? Where do I look? Is amd64 video now the genfb(4) device code? Or is it still vga(4)? If it's genfb(4), then I do see commits about doing anti-aliasing, and maybe the video junk I see could possibly be explained by such a thing. If I can get 7.2 installed (likely), so that I need only drop a kernel in place instead of building the whole installimage and writing the damn slow USB stick with a whole install image every time, then maybe I'll be able to try bisecting changes to get the video working right again. I really wish modern PC vendors were not still so bloody stupid with their firmware as to make it impossible to talk to them via a serial port of some kind (e.g. a USB serial adapter as console would be awesome!). That said, what would it take to wire the NetBSD console to a USB serial adapter? In lieu of that it would be nice if hitting ^S on the keyboard would at least pause the kernel messages from scrolling by during boot, but I get that such a thing might be a bit hard to arrange for in NetBSD. -- Greg A. Woods Kelowna, BC +1 250 762-7675 RoboHack Planix, Inc. Avoncote Farms pgpvllmWWoiDK.pgp Description: OpenPGP Digital Signature
