Re: Boot hang on Xen after r318347/(310418)

[Adam McDougall]

On 05/25/2017 09:28, Adam McDougall wrote:
> On Thu, May 25, 2017 at 10:41:03AM +0100, Roger Pau Monné wrote:
> 
>> On Wed, May 24, 2017 at 06:33:07PM -0400, Adam McDougall wrote:
>>> Hello,
>>>
>>> Recently I made a new build of 11-STABLE but encountered a boot hang
>>> at this state:
>>> http://www.egr.msu.edu/~mcdouga9/pics/r318347-smp-hang.png
>>>
>>> It is easy to reproduce, I can just boot from any 11 or 12 ISO that 
>>> contains the commit.
>>
>> I have just tested latest HEAD (r318861) and stable/11 (r318854) and
>> they both work fine on my environment (a VM with 4 vCPUs and 2GB of
>> RAM on OSS Xen 4.9). I'm also adding Colin in case he has some input,
>> he has been doing some tests on HEAD and AFAIK he hasn't seen any
>> issues.
>>
>>> I compiled various svn revisions to confirm that r318347 caused the 
>>> issue and r318346 is fine. With r318347 or later including the latest 
>>> 11-STABLE, the system will only boot with one virtual CPU in XenServer. 
>>> Any more cpus and it hangs. I also tried a 12 kernel from head this 
>>> afternoon and I have the same hang. I had this issue on XenServer 7 
>>> (Xen 4.7) and XenServer 6.5 (Xen 4.4). I did most of my testing on 7. I 
>>> also did much of my testing with a GENERIC kernel to try to rule out 
>>> kernel configuration mistakes. When it hangs, the performance 
>>> monitoring in Xen tells me at least one CPU is pegged. r318674 boots 
>>> fine on physical hardware without Xen involved.
>>>
>>> Looking at r318347 which mentions EARLY_AP_STARTUP and later seeing 
>>> r318763 which enables EARLY_AP_STARTUP in GENERIC, I tried adding it to 
>>> my kernel but it turned the hang into a panic but with any number of 
>>> CPUs: 
>>> http://www.egr.msu.edu/~mcdouga9/pics/r318347-early-ap-startup-panic.png
>>
>> I guess this is on stable/11 right? The panic looks easier to debug
>> that the hang, so let's start by this one. Can you enable the serial
>> console and kernel debug options in order to get a trace? With just
>> this it's almost impossible to know what went wrong.
> 
> Yes this was on stable/11 amd64.
> 
>>
>> Roger.

I worked on this today and the short version is recent kernels no longer
hang or panic with EARLY_AP_STARTUP which includes the 20170602 iso
images of 11 and 12. Adding EARLY_AP_STARTUP to my kernel config appears
to prevent the hang and something between r318855 (May 24) and r319554
(today, June 3) prevents the panic. I'm tempted to figure out which
commit but I already spent hours bisecting and building today, so since
this seems to be a forward working solution, I'm content. Thanks.
___
freebsd-stable@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"

Re: Errors with ports on 9.3..

[Mark Saad]

> On Jun 3, 2017, at 3:10 PM, Mark Saad  wrote:
> 
> 
> 
>> On Jun 3, 2017, at 2:24 PM, Adam Vande More  wrote:
>> 
>>> On Sat, Jun 3, 2017 at 12:19 PM, Howard Leadmon  wrote:
>>>Thanks for the update, I had the feeling the issue was from it being to 
>>> old.   I have a question, not sure if you know, but I will toss it out.   
>>> As I mentioned I update using svn for both src and ports, and I am curious 
>>> to know if I can actually bring my src tree up to the most current 10.x 
>>> stable, recompile, and install and have it all run?
> 
> So here is my take on the update . For starters if you are using a stock 
> binary 9.3-RELEASE you can use freebsd-update to go 9.3 -> 10.1 , 10.1 -> 
> 10.3  , 10.3 -> 11.0 . In theory freebsd-update should allow for 9.x -> 10.x 
> but there was some breakage in the 10's FreeBSD updates that prevented this .
> 
> If you decided to do a source build you can go 9.x to 10.3 w/o much. 

I meant to say trouble but I accidentally hit send . 


Another option is to download the binary sets from ftp.freebsd.org 

ftp://ftp.freebsd.org/pub/FreeBSD/releases/amd64/11.0-RELEASE/

Now with some care you can backup your 9 kernel to kernel.old and untar the 
kernel tar to /boot . Then reboot in single user . Backup your etc and extract 
the base tar and then using etcupdate for fixing etc or manually fix it with 
your backup . Then reboot and fix the ports using pkg -f install pkg && pkg 
upgrade .  However if you haven't done this before it can be error prone if . 

Also you can look into boot environments for zfs but if memory serves me right 
it's not fully baked into 9.x and it may not work right . 

> 
>>> 
>>>  In the past with much older versions, I know file system changes and such 
>>> make it pretty hard to jump major revisions,
>>> so have a little bit of fear about jumping from 9.x to 10.x, and possibly 
>>> even to 11.x if that is now stable.   I am using ZFS, so I guess that would 
>>> be one thing that is outside the norm, but should be part of the base 
>>> kernels now anyway.
>>> 
>>>  Any input on upgrading would be most appreciated...

Honestly 11 has been very stable . There are issues but nothing that has wanted 
me to roll back to 10 . 

I am using 10.3-STABLE from about a year ago for my routers and 11.0-STABLE 
from April for general use and it's been good and crash free. Knock on wood . 

One thing to remember is upgrading the zpool and zfs version/ feature flags . 
Al la zfs upgrade pool0/foo . This is a one time job ; with no way to go back . 
So save this for last after your box has settled down and you are comfortable. 

>> 
>> I don't know what you know I guess, but it should work following these 
>> instructions:
>> 
>> https://www.freebsd.org/doc/en_US.ISO8859-1/books/handbook/makeworld.html
>> 
>> or these:
>> 
>> https://www.freebsd.org/doc/handbook/updating-upgrading.html
>> 
>> or these:
>> 
>> https://www.freebsd.org/releases/11.0R/installation.html
>> 
>> It is probably wise to make a backup and do a test first.
>> 
>> -- 


>> Adam


---
Mark Saad | nones...@longcount.org


___
freebsd-stable@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"

Re: Errors with ports on 9.3..

[Mark Saad]

> On Jun 3, 2017, at 2:24 PM, Adam Vande More  wrote:
> 
>> On Sat, Jun 3, 2017 at 12:19 PM, Howard Leadmon  wrote:
>>Thanks for the update, I had the feeling the issue was from it being to 
>> old.   I have a question, not sure if you know, but I will toss it out.   As 
>> I mentioned I update using svn for both src and ports, and I am curious to 
>> know if I can actually bring my src tree up to the most current 10.x stable, 
>> recompile, and install and have it all run?

So here is my take on the update . For starters if you are using a stock binary 
9.3-RELEASE you can use freebsd-update to go 9.3 -> 10.1 , 10.1 -> 10.3  , 10.3 
-> 11.0 . In theory freebsd-update should allow for 9.x -> 10.x but there was 
some breakage in the 10's FreeBSD updates that prevented this .

If you decided to do a source build you can go 9.x to 10.3 w/o much.


>> 
>>  In the past with much older versions, I know file system changes and such 
>> make it pretty hard to jump major revisions,
>> so have a little bit of fear about jumping from 9.x to 10.x, and possibly 
>> even to 11.x if that is now stable.   I am using ZFS, so I guess that would 
>> be one thing that is outside the norm, but should be part of the base 
>> kernels now anyway.
>> 
>>  Any input on upgrading would be most appreciated...
> 
> I don't know what you know I guess, but it should work following these 
> instructions:
> 
> https://www.freebsd.org/doc/en_US.ISO8859-1/books/handbook/makeworld.html
> 
> or these:
> 
> https://www.freebsd.org/doc/handbook/updating-upgrading.html
> 
> or these:
> 
> https://www.freebsd.org/releases/11.0R/installation.html
> 
> It is probably wise to make a backup and do a test first.
> 
> -- 
> Adam
___
freebsd-stable@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"

Re: Errors with ports on 9.3..

[Adam Vande More]

On Sat, Jun 3, 2017 at 12:19 PM, Howard Leadmon  wrote:

>Thanks for the update, I had the feeling the issue was from it being to
> old.   I have a question, not sure if you know, but I will toss it out.
>  As I mentioned I update using svn for both src and ports, and I am curious
> to know if I can actually bring my src tree up to the most current 10.x
> stable, recompile, and install and have it all run?
>
>  In the past with much older versions, I know file system changes and such
> make it pretty hard to jump major revisions,

so have a little bit of fear about jumping from 9.x to 10.x, and possibly
> even to 11.x if that is now stable.   I am using ZFS, so I guess that would
> be one thing that is outside the norm, but should be part of the base
> kernels now anyway.
>
>  Any input on upgrading would be most appreciated...


I don't know what you know I guess, but it should work following these
instructions:

https://www.freebsd.org/doc/en_US.ISO8859-1/books/handbook/makeworld.html

or these:

https://www.freebsd.org/doc/handbook/updating-upgrading.html

or these:

https://www.freebsd.org/releases/11.0R/installation.html

It is probably wise to make a backup and do a test first.

-- 
Adam
___
freebsd-stable@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"

Re: Errors with ports on 9.3..

[Howard Leadmon]

   Thanks for the update, I had the feeling the issue was from it being 
to old.   I have a question, not sure if you know, but I will toss it 
out.   As I mentioned I update using svn for both src and ports, and I 
am curious to know if I can actually bring my src tree up to the most 
current 10.x stable, recompile, and install and have it all run?


 In the past with much older versions, I know file system changes and 
such make it pretty hard to jump major revisions, so have a little bit 
of fear about jumping from 9.x to 10.x, and possibly even to 11.x if 
that is now stable.   I am using ZFS, so I guess that would be one thing 
that is outside the norm, but should be part of the base kernels now anyway.


 Any input on upgrading would be most appreciated...

---
Howard Leadmon
PBW Communications, LLC
http://www.pbwcomm.com

On 6/3/2017 1:08 PM, Mark Saad wrote:

Howard
  There are 3 options . One upgrade to 10.x or 11.x and ports/pkg will work . 
Support for the 9 series and 10.1 and 10.2 was removed from ports a few months 
ago . Two check out ports via svn and use the tag RELEASE_9_EOL and use that as 
your starting point . This is ports before the support for 9 was dropped . It's 
not up to date but it will work .Lastly you could use pkgsrc from netbsd not 
Joynet , and do a full rebuild of what you need . Pkgsrc also has a pkgsrc to 
ports conversion tool if you want the subsequent pkgsrc packages registered in 
a FreeBSD standard pkg dir . This would work if you haven't converted 9 to use 
pkgng .

https://svnweb.freebsd.org/ports/tags/RELEASE_9_EOL/

https://www.pkgsrc.org/



---
Mark Saad | nones...@longcount.org


On Jun 3, 2017, at 11:39 AM, Howard Leadmon  wrote:

I know 9.3 is kinda dated, but it had been running fine so I left the servers 
alone as they were in a remote location many hours away.

That said, if I try and do anything with ports (I keep everything updated using 
SVN), I get the following error:

"/usr/ports/Mk/bsd.port.mk", line 1038: Unknown directive

The line is:   .export LANG LC_ALL


I even tried removing the ports tree, and bringing it all back down, but same 
error.   Maybe my googlefu is failing me, but I couldn't find this exact error 
listed anyplace.   I have a couple FBSD 9.3 machines still in existence, and 
they all do this with anything in ports now.

Any ideas or suggestions?  Would just taking SVN to the current 10.x fix this?

---
Howard Leadmon
PBW Communications, LLC
http://www.pbwcomm.com

___
freebsd-stable@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"

___
freebsd-stable@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"


___
freebsd-stable@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"

Re: Errors with ports on 9.3..

[Mark Saad]

Howard 
 There are 3 options . One upgrade to 10.x or 11.x and ports/pkg will work . 
Support for the 9 series and 10.1 and 10.2 was removed from ports a few months 
ago . Two check out ports via svn and use the tag RELEASE_9_EOL and use that as 
your starting point . This is ports before the support for 9 was dropped . It's 
not up to date but it will work .Lastly you could use pkgsrc from netbsd not 
Joynet , and do a full rebuild of what you need . Pkgsrc also has a pkgsrc to 
ports conversion tool if you want the subsequent pkgsrc packages registered in 
a FreeBSD standard pkg dir . This would work if you haven't converted 9 to use 
pkgng . 

https://svnweb.freebsd.org/ports/tags/RELEASE_9_EOL/

https://www.pkgsrc.org/



---
Mark Saad | nones...@longcount.org

> On Jun 3, 2017, at 11:39 AM, Howard Leadmon  wrote:
> 
> I know 9.3 is kinda dated, but it had been running fine so I left the servers 
> alone as they were in a remote location many hours away.
> 
> That said, if I try and do anything with ports (I keep everything updated 
> using SVN), I get the following error:
> 
> "/usr/ports/Mk/bsd.port.mk", line 1038: Unknown directive
> 
> The line is:   .export LANG LC_ALL
> 
> 
> I even tried removing the ports tree, and bringing it all back down, but same 
> error.   Maybe my googlefu is failing me, but I couldn't find this exact 
> error listed anyplace.   I have a couple FBSD 9.3 machines still in 
> existence, and they all do this with anything in ports now.
> 
> Any ideas or suggestions?  Would just taking SVN to the current 10.x fix this?
> 
> ---
> Howard Leadmon
> PBW Communications, LLC
> http://www.pbwcomm.com
> 
> ___
> freebsd-stable@freebsd.org mailing list
> https://lists.freebsd.org/mailman/listinfo/freebsd-stable
> To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"
___
freebsd-stable@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"

Errors with ports on 9.3..

[Howard Leadmon]

 I know 9.3 is kinda dated, but it had been running fine so I left the 
servers alone as they were in a remote location many hours away.


 That said, if I try and do anything with ports (I keep everything 
updated using SVN), I get the following error:


"/usr/ports/Mk/bsd.port.mk", line 1038: Unknown directive

The line is:   .export LANG LC_ALL


I even tried removing the ports tree, and bringing it all back down, but 
same error.   Maybe my googlefu is failing me, but I couldn't find this 
exact error listed anyplace.   I have a couple FBSD 9.3 machines still 
in existence, and they all do this with anything in ports now.


Any ideas or suggestions?  Would just taking SVN to the current 10.x fix 
this?


---
Howard Leadmon
PBW Communications, LLC
http://www.pbwcomm.com

___
freebsd-stable@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"