Re: [gentoo-user] What is the best way forward? - Update 2 - SUCCESS! - CURRENT!!!
On 3/8/21 5:59 PM, antlists wrote: As I remember, you always had to use eselect to switch versions ... and witness all the chaos with python at the moment ... I don't know. If you leave things "at the default", doesn't that screw you over when python/kernel/gcc etc upgrade and a depclean deletes your original default version? Or is that now fixed so you can't mess things up that way? I've not changed the kernel yet. I didn't knowingly have any problems with Python changes /related/ /to/ gcc. My Python problem was that my make.profile was pointing to the old portage directory that was still back at last March while emerge was using a newer incrementally updating version of portage. -- I consider this to be my fault. -- Grant. . . . unix || die
Re: [gentoo-user] What is the best way forward? - Update 2 - SUCCESS! - CURRENT!!!
On Tue, 9 Mar 2021 17:26:41 +0100, smurfd wrote: > Massive congrats :) > Great when things work out, after doing a ton of work to get there. As the great Douglas Adams once said "I am rarely happier than when spending an entire day programming my computer to perform automatically a task that would otherwise take me a good ten seconds to do by hand.”" > br smurfd > > On 2021-03-08 23:06, Grant Taylor wrote: > > On 2/25/21 5:31 PM, Grant Taylor wrote: > >> 10 have git switch to the next day > >> 20 emerge -aDUN @world > >> 30 assess / deal with masked packages > >> 40 goto 10 > >> > >> It /looks/ like things are working. > > > > *TL;DR* > > > > DenverCoder9: DEAR PEOPLE FROM THE FUTURE ... > > > > This method /does/ work. I have successfully brought the problem > > system from ~1 year old to ~current (Gentoo portage repo < 24 hours > > old). > > > > *Speed Bumps* > > > > These were the four things that caused the biggest slow down in this > > process. > > > > 1) Source packages / ebuilds no longer available. > > - I found and downloaded files to DISTDIR. > > - I copied some ebuilds from older versions of portage to my > > local repo. > > 2) make.profile not using PORTDIR definition in make.conf. > > - I ran into this while working on October ~ November '20 > > updates. 3) PYTHON_TARGETS & PYTHON_SINGLE_TARGET > > - I ran into this after fixing #2. > > - I had to add the following to pull Python 3.6 back in so that > > things would work to add Python 3.7, before allowing the system to > > remove Python 3.6 (again). > > PYTHON_TARGETS="python2_7 python3_6 python3_7" > > PYTHON_SINGLE_TARGET="python3_7" > > 4) Firefox & Thunderbird 68 disliking rust ≈ 1.48. > > - I had to give up on retaining version 68 of Firefox and > > Thunderbird. > > - The loss of some important extensions still really hurts. > > > > *How* > > > > The high level process that I used is a very close superset of what I > > hypothesized. > > > > 10 have git switch to the next day > > 20 emerge -DUN @world > > 21 emerge --depclean --verbose n > > 22 emerge @preserved-rebuild > > 23 revdep-rebuild > > 30 assess / deal with output from steps 20-23 > > 40 goto 10 > > > > Steps 21-23 added mid-stream to make comparison to previous message > > simpler. > > > > All of these steps were in a function, `e3` (see attached file), > > which relied on one variable, `d`, the count of how many days to go > > backwards and set the date (`D`) to that everything should act on. > > > > Aside: The next version of e3 would probably store `d` in a file and > > subsequently re-load it from said file on each invocation. Thus > > eliminating the reliance on the environment variable. I would > > probably store this file in /var/tmp as /tmp and /dev/shm are cleared > > on boot. > > > > After gaining enough trust in the overall process, I ended up running > > the following while loop: > > > > while e3; true; done > > > > This allowed the system to stay busy emerging things up to the point > > that something failed and needed attention. > > > > *Setup* > > > > I did a `git clone` of the Gentoo portage repo. Currently ~6 GB. > > > > I then created the branches in the git repo with the following > > command (from inside of the git repo directory): > > > > for ((age=1; age<1024; age++)); do eval $(printf 'git log > > --pretty=format:"%%H %%cd" --date=format:%%Y-%%m-%%d\ %%H:%%M:%%S > > --after=%s --before=%s | fgrep -m1 %s' $(date +%Y-%m-%d -d "$(($age + > > 1)) days ago") $(date +%Y-%m-%d -d "$(($age - 1)) days ago") $(date > > +%Y-%m-%d -d "$age days ago")) | read hash date time; time git > > checkout -b $date $hash; done > > > > Basically, this command starts at current; `stable`, and finds the > > first (most recent) commit for a given date and creates a branch, and > > works backwards for however many days configured; 1024 in the example. > > > > *Miscellany* > > > > I did `emerge -e @world` 3~5 times throughout the process just to > > make sure that everything was consistent. I will do this once more > > tomorrow after a full backup runs tonight. > > > > I did end up removing a small list of packages that were blocking > > emerge in one way or another. -- I decided that removing them to > > allow emerge to complete on it's own accord was more expedient than > > fighting them at the time. I will re-add them as necessary. > > > > - net-firewall/nftables > > - net-fs/ncpfs > > - media-gfx/gimp > > - dev-python/pycairo > > - dev-python/fido2 > > - net-analyzer/scapy > > - app-crypt/yubikey-manager > > > > Some of the packages were subsequently pulled back in. > > > > I did run into a bug with app-misc/pax-utils where I needed to add > > "-seccomp" for the package to be able to move forward. > > > > I also did the /usr/portage to /var/db/repos/gentoo et al. migration. > > > > "repo" can be ambiguous when there talking about both "Gentoo portage > > repo"
Re: [gentoo-user] What is the best way forward? - Update 2 - SUCCESS! - CURRENT!!!
Massive congrats :) Great when things work out, after doing a ton of work to get there. br smurfd On 2021-03-08 23:06, Grant Taylor wrote: On 2/25/21 5:31 PM, Grant Taylor wrote: 10 have git switch to the next day 20 emerge -aDUN @world 30 assess / deal with masked packages 40 goto 10 It /looks/ like things are working. *TL;DR* DenverCoder9: DEAR PEOPLE FROM THE FUTURE ... This method /does/ work. I have successfully brought the problem system from ~1 year old to ~current (Gentoo portage repo < 24 hours old). *Speed Bumps* These were the four things that caused the biggest slow down in this process. 1) Source packages / ebuilds no longer available. - I found and downloaded files to DISTDIR. - I copied some ebuilds from older versions of portage to my local repo. 2) make.profile not using PORTDIR definition in make.conf. - I ran into this while working on October ~ November '20 updates. 3) PYTHON_TARGETS & PYTHON_SINGLE_TARGET - I ran into this after fixing #2. - I had to add the following to pull Python 3.6 back in so that things would work to add Python 3.7, before allowing the system to remove Python 3.6 (again). PYTHON_TARGETS="python2_7 python3_6 python3_7" PYTHON_SINGLE_TARGET="python3_7" 4) Firefox & Thunderbird 68 disliking rust ≈ 1.48. - I had to give up on retaining version 68 of Firefox and Thunderbird. - The loss of some important extensions still really hurts. *How* The high level process that I used is a very close superset of what I hypothesized. 10 have git switch to the next day 20 emerge -DUN @world 21 emerge --depclean --verbose n 22 emerge @preserved-rebuild 23 revdep-rebuild 30 assess / deal with output from steps 20-23 40 goto 10 Steps 21-23 added mid-stream to make comparison to previous message simpler. All of these steps were in a function, `e3` (see attached file), which relied on one variable, `d`, the count of how many days to go backwards and set the date (`D`) to that everything should act on. Aside: The next version of e3 would probably store `d` in a file and subsequently re-load it from said file on each invocation. Thus eliminating the reliance on the environment variable. I would probably store this file in /var/tmp as /tmp and /dev/shm are cleared on boot. After gaining enough trust in the overall process, I ended up running the following while loop: while e3; true; done This allowed the system to stay busy emerging things up to the point that something failed and needed attention. *Setup* I did a `git clone` of the Gentoo portage repo. Currently ~6 GB. I then created the branches in the git repo with the following command (from inside of the git repo directory): for ((age=1; age<1024; age++)); do eval $(printf 'git log --pretty=format:"%%H %%cd" --date=format:%%Y-%%m-%%d\ %%H:%%M:%%S --after=%s --before=%s | fgrep -m1 %s' $(date +%Y-%m-%d -d "$(($age + 1)) days ago") $(date +%Y-%m-%d -d "$(($age - 1)) days ago") $(date +%Y-%m-%d -d "$age days ago")) | read hash date time; time git checkout -b $date $hash; done Basically, this command starts at current; `stable`, and finds the first (most recent) commit for a given date and creates a branch, and works backwards for however many days configured; 1024 in the example. *Miscellany* I did `emerge -e @world` 3~5 times throughout the process just to make sure that everything was consistent. I will do this once more tomorrow after a full backup runs tonight. I did end up removing a small list of packages that were blocking emerge in one way or another. -- I decided that removing them to allow emerge to complete on it's own accord was more expedient than fighting them at the time. I will re-add them as necessary. - net-firewall/nftables - net-fs/ncpfs - media-gfx/gimp - dev-python/pycairo - dev-python/fido2 - net-analyzer/scapy - app-crypt/yubikey-manager Some of the packages were subsequently pulled back in. I did run into a bug with app-misc/pax-utils where I needed to add "-seccomp" for the package to be able to move forward. I also did the /usr/portage to /var/db/repos/gentoo et al. migration. "repo" can be ambiguous when there talking about both "Gentoo portage repo" and "git repo". Especially when the latter is managing the former. The following packages take what seems like F O R E V E R to emerge: - gcc - rust - Firefox - Thunderbird Link - xkcd - Wisdom of the Ancients (a.k.a. DenverCoder9) - https://xkcd.com/979/ *Summary* Yes, there are probably faster and / or more efficient processes to get a Gentoo system that's ~1 year behind caught up to current. But I did learn some things along the way. -- I tried to outline the toe stubbers so others can avoid them. Ultimately, I believe I have done in the last 11 days what would have been done over the course of the last ~year. Even 11 days is longer than necessary as I started with the
Re: [gentoo-user] What is the best way forward? - Update 2 - SUCCESS! - CURRENT!!!
On Tue, 9 Mar 2021 00:59:29 +, antlists wrote: > > It means you probably spent a lot of time compile gcc versions only to > > carry on using the old version, but as you said, this wasn't about > > efficiency. You were going to emerge -e @world at the end anyway, > > which would get everything built with the latest toolchain. > > > As I remember, you always had to use eselect to switch versions ... and > witness all the chaos with python at the moment ... > > If you leave things "at the default", doesn't that screw you over when > python/kernel/gcc etc upgrade and a depclean deletes your original > default version? Or is that now fixed so you can't mess things up that > way? You can't because eselect always shows a fallback option, which will be used should you unmerge the selected version. -- Neil Bothwick IMPORTANT: The entire physical universe, including this message, may one day collapse back into an infinitesimally small space. Should another universe subsequently re-emerge, the existence of this message in that universe cannot be guaranteed. pgpRljpBtfCwz.pgp Description: OpenPGP digital signature
Re: [gentoo-user] What is the best way forward? - Update 2 - SUCCESS! - CURRENT!!!
On 08/03/2021 23:16, Neil Bothwick wrote: I don't remember what it was at the start, probably 8. or 9.. I did see 9.3 somewhere along the way. gcc -v says that 10.2.0 is currently installed. It means you probably spent a lot of time compile gcc versions only to carry on using the old version, but as you said, this wasn't about efficiency. You were going to emerge -e @world at the end anyway, which would get everything built with the latest toolchain. As I remember, you always had to use eselect to switch versions ... and witness all the chaos with python at the moment ... If you leave things "at the default", doesn't that screw you over when python/kernel/gcc etc upgrade and a depclean deletes your original default version? Or is that now fixed so you can't mess things up that way? Cheers, Wol
Re: [gentoo-user] What is the best way forward? - Update 2 - SUCCESS! - CURRENT!!!
On 3/8/21 5:35 PM, Neil Bothwick wrote: Not if you went up a slot, then the old version would still continue to be used until you ran gcc-config. However, if you were depcleaning at each step, that would remove the previous slot and you would stay current. So my overall method, which included depclean, did allow subsequent runs to use the new updated GCC. Thank you for clarifying. I tend to keep old copies of gcc around until I'm sure things play nicely with the new version: % gcc-config -l [1] x86_64-pc-linux-gnu-9.3.0 [2] x86_64-pc-linux-gnu-10.2.0 * [1] aarch64-unknown-linux-gnu-9.3.0 * (cyan *) [2] x86_64-pc-linux-gnu-10.2.0 * (green *) The aarch64* came in as part of @openwrt-prerequisites. I should probably remove that as I no longer need it. Thank you for your input Neil. -- Grant. . . . unix || die
Re: [gentoo-user] What is the best way forward? - Update 2 - SUCCESS! - CURRENT!!!
On Mon, 8 Mar 2021 16:44:35 -0700, Grant Taylor wrote: > > It means you probably spent a lot of time compile gcc versions only > > to carry on using the old version, but as you said, this wasn't about > > efficiency. > > Wouldn't the next execution of gcc, post Emerge & Installation use the > newly emerged binary? > > If not next package in a given emerge run didn't use the new gcc, I > would fully expect that subsequent emerges would use the new gcc. Not if you went up a slot, then the old version would still continue to be used until you ran gcc-config. However, if you were depcleaning at each step, that would remove the previous slot and you would stay current. I tend to keep old copies of gcc around until I'm sure things play nicely with the new version: % gcc-config -l [1] x86_64-pc-linux-gnu-9.3.0 [2] x86_64-pc-linux-gnu-10.2.0 * -- Neil Bothwick Psychiatrists say that 1 of 4 people are mentally ill. Check three friends. If they're OK, you're it. pgp1liLQnwr6t.pgp Description: OpenPGP digital signature
Re: [gentoo-user] What is the best way forward? - Update 2 - SUCCESS! - CURRENT!!!
On 3/8/21 4:16 PM, Neil Bothwick wrote: It would have to be done before the first update, when the repo was set to a date just after the last update. Yes and no. It really could have been done at any point along the way. Also, with the git version of the portage repo, I could switch back to the branch from any time I wanted to. You can rephrase that as "I left it at the default", which is an acceptable answer :) *nod* It means you probably spent a lot of time compile gcc versions only to carry on using the old version, but as you said, this wasn't about efficiency. Wouldn't the next execution of gcc, post Emerge & Installation use the newly emerged binary? If not next package in a given emerge run didn't use the new gcc, I would fully expect that subsequent emerges would use the new gcc. You were going to emerge -e @world at the end anyway, which would get everything built with the latest toolchain. Yes. I have initiated a full system backup. I'll start an `emerge -e @world` after that finishes. I'll actually do the full suite: 1) emerge -e @world 2) emerge --depclean --verbose n 3) emerge @preserved-rebuild 4) revdep-rebuild I expect that #3 should be a NoOp and just burn CPU cycles. I don't know anything else that can be done to make a Gentoo box happier (from a software standpoint). Most of the effort for you was developing the procedure. All the real effort was left to the computer. Exactly! Well, developing the method /and/ establishing trust therein. I was thinking of a week max. I suspect that would be quite safe. -- Grant. . . . unix || die
Re: [gentoo-user] What is the best way forward? - Update 2 - SUCCESS! - CURRENT!!!
On Mon, 8 Mar 2021 15:44:38 -0700, Grant Taylor wrote: > On 3/8/21 3:29 PM, Neil Bothwick wrote: > > With hindsight, removing firefox, thunderbird and libreoffice and > > replacing them with their -bin counterparts at the start of the > > process would have saved much time. You could switch back to the > > source options once the system is up to date. > > You're probably correct. > > However I don't think I would do that even if I had known / thought > about doing so. Partially because changing things was questionable at > / near the start and partially because this was about possibility, not > efficiency.It would ve to be done It would have to be done before the first update, when the repo was set to a date just after the last update. > > How did you manage gcc upgrades, did you run gcc-config manually > > whenever gcc was updated? > > Is "I ignored them and let emerge deal with it" count? I did see gcc > upgrades along the way. You can rephrase that as "I left it at the default", which is an acceptable answer :) > I don't remember what it was at the start, probably 8. or > 9.. I did see 9.3 somewhere along the way. gcc -v says > that 10.2.0 is currently installed. It means you probably spent a lot of time compile gcc versions only to carry on using the old version, but as you said, this wasn't about efficiency. You were going to emerge -e @world at the end anyway, which would get everything built with the latest toolchain. > > Do you feel it was worth the effort of updating for every day of the > > git history? > > I don't know if it was worth the effort or not. I initially did one > day at a time while testing the waters and going from theory to some > practical experience of the method. > > Very quickly I used a different version of e (1 or 2) that took the > date as a parameter. My command line was calling e with the date > derived from the d variable and then decrementing the d variable after > the e function finished. I.e. > > e $(date +%Y-%m-%d -d "$d days ago"); $((d=$d-1)) > > I would let that run, deal with any results, then hit the up arrow and > enter. > > I just let that process continue for a while. Then at some point I > optimized it into e3 and ran that for a while. Then I optimized that > into the while e3; do true; done loop. Most of the effort for you was developing the procedure. All the real effort was left to the computer. > But I stuck with single day steps mostly from inertia. It was working. > So ... stick with it. > > > Would a larger increment have saved time, or did you think minimising > > the number of issues to deal with after each emerge was more > > important? > > Maybe. If anything, it would have saved the time for emerge to process > all of it's meta data. Much like an initial emerge vs an emerge > --resume. But again, this was about the viability of the process, not > the efficiency thereof. > > I probably could have gone with a week at a time. I don't know if that > would have helped or not. I don't think I would go with more than a > week with a largely automated process. I was thinking of a week max. -- Neil Bothwick Distrust any enterprise that requires new clothes. - Henry David Thoreau (1817-1862) pgp7BOgh2uzc5.pgp Description: OpenPGP digital signature
Re: [gentoo-user] What is the best way forward? - Update 2 - SUCCESS! - CURRENT!!!
On 3/8/21 3:29 PM, Neil Bothwick wrote: With hindsight, removing firefox, thunderbird and libreoffice and replacing them with their -bin counterparts at the start of the process would have saved much time. You could switch back to the source options once the system is up to date. You're probably correct. However I don't think I would do that even if I had known / thought about doing so. Partially because changing things was questionable at / near the start and partially because this was about possibility, not efficiency. How did you manage gcc upgrades, did you run gcc-config manually whenever gcc was updated? Is "I ignored them and let emerge deal with it" count? I did see gcc upgrades along the way. I don't remember what it was at the start, probably 8. or 9.. I did see 9.3 somewhere along the way. gcc -v says that 10.2.0 is currently installed. Do you feel it was worth the effort of updating for every day of the git history? I don't know if it was worth the effort or not. I initially did one day at a time while testing the waters and going from theory to some practical experience of the method. Very quickly I used a different version of e (1 or 2) that took the date as a parameter. My command line was calling e with the date derived from the d variable and then decrementing the d variable after the e function finished. I.e. e $(date +%Y-%m-%d -d "$d days ago"); $((d=$d-1)) I would let that run, deal with any results, then hit the up arrow and enter. I just let that process continue for a while. Then at some point I optimized it into e3 and ran that for a while. Then I optimized that into the while e3; do true; done loop. But I stuck with single day steps mostly from inertia. It was working. So ... stick with it. Would a larger increment have saved time, or did you think minimising the number of issues to deal with after each emerge was more important? Maybe. If anything, it would have saved the time for emerge to process all of it's meta data. Much like an initial emerge vs an emerge --resume. But again, this was about the viability of the process, not the efficiency thereof. I probably could have gone with a week at a time. I don't know if that would have helped or not. I don't think I would go with more than a week with a largely automated process. I think one month increments probably would be pushing the envelope. I feel like some of the Python changes were 2 or maybe 3 months apart. So with two combined with how you landed, you might cross Python 3.6 w/o 3.7, to both 3.6 and 3.7, to w/o 3.6 and w/ 3.7 barrier. That probably wouldn't be pretty. Anyway, glad it worked for you - it's more or less how I would have approached it but never had to, so thanks for doing the legwork :) You're welcome. Hence the DenverCoder9 comment, for people searching ~> reading the mailing list archive in the future. -- Grant. . . . unix || die
Re: [gentoo-user] What is the best way forward? - Update 2 - SUCCESS! - CURRENT!!!
On Mon, 8 Mar 2021 15:06:01 -0700, Grant Taylor wrote: > The following packages take what seems like F O R E V E R to emerge: > > - gcc > - rust > - Firefox > - Thunderbird With hindsight, removing firefox, thunderbird and libreoffice and replacing them with their -bin counterparts at the start of the process would have saved much time. You could switch back to the source options once the system is up to date. How did you manage gcc upgrades, did you run gcc-config manually whenever gcc was updated? Do you feel it was worth the effort of updating for every day of the git history? Would a larger increment have saved time, or did you think minimising the number of issues to deal with after each emerge was more important? Anyway, glad it worked for you - it's more or less how I would have approached it but never had to, so thanks for doing the legwork :) -- Neil Bothwick WinErr 013: Unexpected error - Huh ? pgpPxSJHUiuBX.pgp Description: OpenPGP digital signature
Re: [gentoo-user] What is the best way forward? - Update 2 - SUCCESS! - CURRENT!!!
On 2/25/21 5:31 PM, Grant Taylor wrote: 10 have git switch to the next day 20 emerge -aDUN @world 30 assess / deal with masked packages 40 goto 10 It /looks/ like things are working. *TL;DR* DenverCoder9: DEAR PEOPLE FROM THE FUTURE ... This method /does/ work. I have successfully brought the problem system from ~1 year old to ~current (Gentoo portage repo < 24 hours old). *Speed Bumps* These were the four things that caused the biggest slow down in this process. 1) Source packages / ebuilds no longer available. - I found and downloaded files to DISTDIR. - I copied some ebuilds from older versions of portage to my local repo. 2) make.profile not using PORTDIR definition in make.conf. - I ran into this while working on October ~ November '20 updates. 3) PYTHON_TARGETS & PYTHON_SINGLE_TARGET - I ran into this after fixing #2. - I had to add the following to pull Python 3.6 back in so that things would work to add Python 3.7, before allowing the system to remove Python 3.6 (again). PYTHON_TARGETS="python2_7 python3_6 python3_7" PYTHON_SINGLE_TARGET="python3_7" 4) Firefox & Thunderbird 68 disliking rust ≈ 1.48. - I had to give up on retaining version 68 of Firefox and Thunderbird. - The loss of some important extensions still really hurts. *How* The high level process that I used is a very close superset of what I hypothesized. 10 have git switch to the next day 20 emerge -DUN @world 21 emerge --depclean --verbose n 22 emerge @preserved-rebuild 23 revdep-rebuild 30 assess / deal with output from steps 20-23 40 goto 10 Steps 21-23 added mid-stream to make comparison to previous message simpler. All of these steps were in a function, `e3` (see attached file), which relied on one variable, `d`, the count of how many days to go backwards and set the date (`D`) to that everything should act on. Aside: The next version of e3 would probably store `d` in a file and subsequently re-load it from said file on each invocation. Thus eliminating the reliance on the environment variable. I would probably store this file in /var/tmp as /tmp and /dev/shm are cleared on boot. After gaining enough trust in the overall process, I ended up running the following while loop: while e3; true; done This allowed the system to stay busy emerging things up to the point that something failed and needed attention. *Setup* I did a `git clone` of the Gentoo portage repo. Currently ~6 GB. I then created the branches in the git repo with the following command (from inside of the git repo directory): for ((age=1; age<1024; age++)); do eval $(printf 'git log --pretty=format:"%%H %%cd" --date=format:%%Y-%%m-%%d\ %%H:%%M:%%S --after=%s --before=%s | fgrep -m1 %s' $(date +%Y-%m-%d -d "$(($age + 1)) days ago") $(date +%Y-%m-%d -d "$(($age - 1)) days ago") $(date +%Y-%m-%d -d "$age days ago")) | read hash date time; time git checkout -b $date $hash; done Basically, this command starts at current; `stable`, and finds the first (most recent) commit for a given date and creates a branch, and works backwards for however many days configured; 1024 in the example. *Miscellany* I did `emerge -e @world` 3~5 times throughout the process just to make sure that everything was consistent. I will do this once more tomorrow after a full backup runs tonight. I did end up removing a small list of packages that were blocking emerge in one way or another. -- I decided that removing them to allow emerge to complete on it's own accord was more expedient than fighting them at the time. I will re-add them as necessary. - net-firewall/nftables - net-fs/ncpfs - media-gfx/gimp - dev-python/pycairo - dev-python/fido2 - net-analyzer/scapy - app-crypt/yubikey-manager Some of the packages were subsequently pulled back in. I did run into a bug with app-misc/pax-utils where I needed to add "-seccomp" for the package to be able to move forward. I also did the /usr/portage to /var/db/repos/gentoo et al. migration. "repo" can be ambiguous when there talking about both "Gentoo portage repo" and "git repo". Especially when the latter is managing the former. The following packages take what seems like F O R E V E R to emerge: - gcc - rust - Firefox - Thunderbird Link - xkcd - Wisdom of the Ancients (a.k.a. DenverCoder9) - https://xkcd.com/979/ *Summary* Yes, there are probably faster and / or more efficient processes to get a Gentoo system that's ~1 year behind caught up to current. But I did learn some things along the way. -- I tried to outline the toe stubbers so others can avoid them. Ultimately, I believe I have done in the last 11 days what would have been done over the course of the last ~year. Even 11 days is longer than necessary as I started with the while loop after getting to January of this year. In hindsight, I believe I could have used the while loop all along. I hope that I have
Re: [gentoo-user] What is the best way forward? - Update 1
On Sat, 27 Feb 2021 08:59:08 +, Michael wrote: > > That's right, the source files are in $DISTDIR, unless you have > > cleaned it. /var/db/pkg contains all the information portage needs > > about the installed software. > > $ emerge --info | grep DISTDIR > > will show if the source files directory is in the old > /usr/portage/distfiles, or the new default of /var/cache/distfiles, or > some custom location the user has specified in /etc/portage/make.conf. Or the somewhat faster $ portageq distdir :) -- Neil Bothwick Nixon's Principal: If 2 wrongs don't make a right, try 3. pgpa8urItgxC6.pgp Description: OpenPGP digital signature
Re: [gentoo-user] What is the best way forward? - Update 1
On Saturday, 27 February 2021 08:34:02 GMT Neil Bothwick wrote: > On Sat, 27 Feb 2021 00:47:04 -0700, Grant Taylor wrote: > > The ebuild and what looks like additional metadata files are in the > > /var/db/pkg directory tree. But the source files aren't in the tree. > > At least not for the example package I looked at. > > That's right, the source files are in $DISTDIR, unless you have cleaned > it. /var/db/pkg contains all the information portage needs about the > installed software. $ emerge --info | grep DISTDIR will show if the source files directory is in the old /usr/portage/distfiles, or the new default of /var/cache/distfiles, or some custom location the user has specified in /etc/portage/make.conf. signature.asc Description: This is a digitally signed message part.
Re: [gentoo-user] What is the best way forward? - Update 1
On Sat, 27 Feb 2021 00:47:04 -0700, Grant Taylor wrote: > The ebuild and what looks like additional metadata files are in the > /var/db/pkg directory tree. But the source files aren't in the tree. > At least not for the example package I looked at. That's right, the source files are in $DISTDIR, unless you have cleaned it. /var/db/pkg contains all the information portage needs about the installed software. -- Neil Bothwick Where do forest rangers go to "get away from it all?" pgpRKpDJOEX4i.pgp Description: OpenPGP digital signature
Re: [gentoo-user] What is the best way forward? - Update 1
On 2/26/21 11:55 PM, Arve Barsnes wrote: I'm not sure what you're saying here, but the ebuild files of the installed packages are in /var/db/pkg Hum. Today I Learned... The ebuild and what looks like additional metadata files are in the /var/db/pkg directory tree. But the source files aren't in the tree. At least not for the example package I looked at. Find across the /var/db/pkg tree does not find any tar files either. -- Grant. . . . unix || die
Re: [gentoo-user] What is the best way forward? - Update 1
On Fri, 26 Feb 2021 at 23:30, Grant Taylor wrote: > > If the packages are installed, the ebuilds are in var/db/pkg. > > The package (distribution files) for the version that is installed are > in distfiles. But that does little for an ebuild that's looking for a > newer version that's no longer on the expected mirrors. I'm not sure what you're saying here, but the ebuild files of the installed packages are in /var/db/pkg Regards, Arve
Re: [gentoo-user] What is the best way forward? - Update 1
On 2/26/21 12:50 PM, Neil Bothwick wrote: Ah yes, I hadn't thought about the mirrors being too up to date. There's also issue with older packages being installed. E.g. I have an older kernel source (4.14.127) that I'm keeping around for various reasons. I've found that the Gentoo repo / portage removes older ebuild files. So, I've had to source them and add them to my local repo. Even putting the old ebuilds into my local repo is entertaining because repoman / ebuild fail to download the source files b/c the mirrors have updated. Hence finding files and putting them in distfiles. If the packages are installed, the ebuilds are in var/db/pkg. The package (distribution files) for the version that is installed are in distfiles. But that does little for an ebuild that's looking for a newer version that's no longer on the expected mirrors. The kernel shouldn't be a problem, I would expect you to be able to jump straight to the current version, although you may prefer to recompile it once you are fully up to date on the toolchain. This system has both Open vSwitch and OpenZFS, so I'm loath to allow automation to change the kernel version. That will be a problem for future me to deal with manually. -- Grant. . . . unix || die
Re: [gentoo-user] What is the best way forward? - Update 1
On Fri, 26 Feb 2021 11:44:12 -0700, Grant Taylor wrote: > I have run into a few problems where emerge can't download files. So > I'm finding them online, downloading them, and saving them to > /usr/portage/distfiles. Ah yes, I hadn't thought about the mirrors being too up to date. > I have had a couple things (really old kernel source still installed) > that I needed to find the ebuild files for. I added them to my local > repository. If the packages are installed, the ebuilds are in var/db/pkg. The kernel shouldn't be a problem, I would expect you to be able to jump straight to the current version, although you may prefer to recompile it once you are fully up to date on the toolchain. -- Neil Bothwick Crash: (v.) to terminate a program in the usual fashion, i.e. by locking up the computer or setting fire to the printer. (n.) the process of such termination. pgp2YWCTTcJ51.pgp Description: OpenPGP digital signature
Re: [gentoo-user] What is the best way forward? - Update 1
On 2/25/21 5:31 PM, Grant Taylor wrote: 10 have git switch to the next day 20 emerge -aDUN @world 30 assess / deal with masked packages 40 goto 10 It /looks/ like things are working. This method is working. I have managed to successfully update from 2020-03-24 to 2020-05-29 in one day increments. I'm starting to see some oddities, like a given version of gimp blocking itself. Unmerging and re-emerging solved that. So I'm going to let the system spend 12 hours and do an emerge -DUNe @world && emerge --depclean --verbose n && revdep-rebuild, reboot, and continue with 2020-06-01. I have run into a few problems where emerge can't download files. So I'm finding them online, downloading them, and saving them to /usr/portage/distfiles. I have had a couple things (really old kernel source still installed) that I needed to find the ebuild files for. I added them to my local repository. I did have an ancient package, ncpfs, that was blocking things. Since I didn't need it, I have unmerged it. If / when I need it in the future, I'll deal with it then. -- Grant. . . . unix || die
Re: [gentoo-user] What is the best way forward?
On Thu, 25 Feb 2021 17:31:15 -0700, Grant Taylor wrote: > One added advantage of doing this day by day is that when I do get to > the big changes, things should be fairly clean. Thus hopefully > simplifying the big changes. You are also going to hit the annoying bugs that were fixed a few hours later. The main issue I had with the python 3.7 switch could have been dealt with by waiting until the next day. This is less of an issue with a purely stable system but it can still happen. -- Neil Bothwick "An investment in knowledge always pays the best interest." - Benjamin Franklin pgpZHj7YrYA6I.pgp Description: OpenPGP digital signature
Re: [gentoo-user] What is the best way forward?
On 2/24/21 9:29 PM, Grant Taylor wrote: I'm currently doing an "emerge -DUNe @system" on the restore of /usr/portage (typical PORTDIR) from prior to messing with things today. The system is now stable with a full -DUNe @system. emerge -DUNe @system reboot emerge -DUNe @world && emerge --depclean --verbose n && revdep-rebuild I've got multiple GB of git data. It looks like there are ~568 thousand commits between March 24th last year and now. Once that's good, and I'm back at a stable place, I'll try changing PORTDIR to be the git repo and telling git to switch to the commit that's from March 25th. Then I'll see if anything needs to be updated, doing so as necessary. Then I'll leap frog a week at a time seeing what needs to be updated, doing so as necessary. /Hopefully/ I can slowly walk forward. Time will tell. I was able to extract the last commit for every day between now and 2020-03-24 and make a branch for it. 10 have git switch to the next day 20 emerge -aDUN @world 30 assess / deal with masked packages 40 goto 10 It /looks/ like things are working. Yes, emerge is spending a LOT of time mulling over things. Many days have been "Nothing to merge; quitting." If I can slowly make my way forward in time via git commit points, I /think/ that I /should/ be able to deal with profile and / or compiler and / or glibc changes just like I would have X number of months ago. I /think/! One added advantage of doing this day by day is that when I do get to the big changes, things should be fairly clean. Thus hopefully simplifying the big changes. -- Grant. . . . unix || die
Re: [gentoo-user] What is the best way forward?
On Thursday, 25 February 2021 15:42:38 GMT Grant Taylor wrote: > On 2/25/21 2:51 AM, Michael wrote: > > A reinstall in this context is not a wholesale replace. > > ~blink~ > > > It implies obtaining the latest Stage 3 archive from a mirror, > > but retaining part of your current installation. Your /home, /etc, > > /var/lib/portage/world, plus any databases e.g. in /var/lib/mysql/ > > and your kernel config will be retained from your existing system and > > will not be replaced. Back these up first along with any particular > > customizations you have made, before you untar Stage 3, so you can > > restore them. > > Ah. You seem to be talking about what I would call an "in place > upgrade" for Windows. As in stalling n over top of n-1 or n-2. That's > definitely less disruptive than I was thinking. I was thinking that > fdisk and / or mkfs would be involved. Yes, it is an "in place upgrade", but bottom up. You upgrade the filesystem to the latest OS image, then drop in your own existing settings/configuration and finally run a system/kernel update. Unless you want to change your partitions you won't need to use fdisk. mkfs is advisable, it will clear out any old cruft and address any changes to the default system directories - e.g. where $PORTDIR, $DISTDIR may be in the latest Stage 3. A quick diff between your backup of make.conf and repos.conf against the new Stage 3 archive contents will inform you of changes to default settings. > > Then rsync portage, update all your @world packages and build a new > > kernel (make oldconfig). Spend some time merging existing application > > config files with etc-update to make them compatible with the latest > > versions of these packages, reboot and hopefully that should be all > > there is to it. > > I may end up /needing/ to go that route. For the moment, I'm going to > try the incremental updates. > > > Yes, it would have been, but what is the benefit of updating multiple > > packages many times over, instead of doing it just once? > > In some ways, this is a learning experience. As in it's a proof of concept. Yes, nowt wrong with that and a sound reason to try it out. In this case, time spent and any problem solving would be an investment in learning. > The computer in question spends 2/3 of it's life doing nothing but > idling a few programs. So, it spending time compiling and producing > heat is not a bad thing in this case. Especially when there's 10" of > snow on the ground. ;-) Neil highlighted the use case of a server/system which can't afford interruptions. I'd still reinstall as explained above, but do it offline, test, backup, and then untar over the live filesystem to keep downtime to a minimum. signature.asc Description: This is a digitally signed message part.
Re: [gentoo-user] What is the best way forward?
On 2/25/21 2:51 AM, Michael wrote: It would probably be better even with a lot of customizations. ;-) Please elaborate on what "better" means in this case. I'm thinking that you might be meaning "faster" and / or "easier" (as in less effort). At least it /should/ be better in terms of time and effort spent. Maybe. A reinstall in this context is not a wholesale replace. ~blink~ It implies obtaining the latest Stage 3 archive from a mirror, but retaining part of your current installation. Your /home, /etc, /var/lib/portage/world, plus any databases e.g. in /var/lib/mysql/ and your kernel config will be retained from your existing system and will not be replaced. Back these up first along with any particular customizations you have made, before you untar Stage 3, so you can restore them. Ah. You seem to be talking about what I would call an "in place upgrade" for Windows. As in stalling n over top of n-1 or n-2. That's definitely less disruptive than I was thinking. I was thinking that fdisk and / or mkfs would be involved. Then rsync portage, update all your @world packages and build a new kernel (make oldconfig). Spend some time merging existing application config files with etc-update to make them compatible with the latest versions of these packages, reboot and hopefully that should be all there is to it. I may end up /needing/ to go that route. For the moment, I'm going to try the incremental updates. Yes, it would have been, but what is the benefit of updating multiple packages many times over, instead of doing it just once? In some ways, this is a learning experience. As in it's a proof of concept. The computer in question spends 2/3 of it's life doing nothing but idling a few programs. So, it spending time compiling and producing heat is not a bad thing in this case. Especially when there's 10" of snow on the ground. ;-) -- Grant. . . . unix || die
Re: [gentoo-user] What is the best way forward?
On Thu, 25 Feb 2021 09:51:25 +, Michael wrote: > > Besides, wouldn't each of the incremental processes over the last year > > have been possible? ;-) > > Yes, it would have been, but what is the benefit of updating multiple > packages many times over, instead of doing it just once? One benefit, and it may be an important one depending on the usage of this system, is that you keep the system running without significant interruptions. My immediate thought when I saw this was the same as John's. Now that portage is available over git, pull in a commit from a month to a week after the last update, update the system, rinse and repeat. Another benefit is that it can be a good way of heating the house at this time of year ;-) -- Neil Bothwick PCMCIA: People Can't Memorize Computer Industry Acronyms pgpZLZu16xjOv.pgp Description: OpenPGP digital signature
Re: [gentoo-user] What is the best way forward?
On 25/02/21 08:45, hitachi303 wrote: > Am 25.02.2021 um 02:43 schrieb Grant Taylor: >> I need to update a system that hasn't been updated in 337 days (March >> 24th 2020. -- Life has been ... trying. >> >> What is the best way forward? >> >> It seems as if there have been a lot of changes in the interim; glibc, >> Python 2.7 being deprecated, default Python going to 3.7(?), other >> breaking changes >> >> Is there a way that I can sync portage to something from April, May, >> or even June of 2020, do a full update (including "-DUNe @world")? >> Iterating through multiple rounds to get current? >> >> Any help would be appreciated. >> > > > > I found it to be helpful to de-install as many programs as possible > before starting the update and the first emerge --sync. This reduces the > amount of conflicts by a considerable amount. Stuff like libreoffice or > thunderbird and so on and all of their dependencies. Everything your > system does not need to run but what you need to be productive when you > use your system. I use -av --depclean for this. > > Also there is something in the gentoo wiki about upgrading old or too > old systems. > > https://wiki.gentoo.org/wiki/Upgrading_Gentoo#Upgrading_from_older_systems > I'd do something a bit like that. The trick is to do the ABSOLUTE MINIMUM each step. Can you "emerge portage"? Given the elephant in the room that is all the python changes, if that works you're probably 90% of the way there. Do you need to update the system profile? Update it to a very basic profile and "emerge @system". NO --deep, NO --new-use, nothing like that. If portage says it can update some things but not others, then tell it to update what it can. Normally I'd "emerge -C" any blockers, but with a minimal system that's probably not wise. Once that's done, select your real profile that you want, and again do a basic "emerge @system". Followed by "emerge @world". Now do your full-blown "emerge --deep --new-use", depclean, new kernel, all that sort of stuff. Cheers, Wol
Re: [gentoo-user] What is the best way forward?
On Thursday, 25 February 2021 04:29:25 GMT Grant Taylor wrote: > On 2/24/21 9:16 PM, John Covici wrote: [snip ...] > > Unless you have a lot of customizations, reinstall would be much > > better. It would probably be better even with a lot of customizations. ;-) At least it /should/ be better in terms of time and effort spent. > I'd really rather not do that. I'm more likely to leave this system as > it is and plan on upgrading it some time in '21. There's considerably > more to it than I want to wholesale replace. A reinstall in this context is not a wholesale replace. It implies obtaining the latest Stage 3 archive from a mirror, but retaining part of your current installation. Your /home, /etc, /var/lib/portage/world, plus any databases e.g. in /var/lib/mysql/ and your kernel config will be retained from your existing system and will not be replaced. Back these up first along with any particular customizations you have made, before you untar Stage 3, so you can restore them. Then rsync portage, update all your @world packages and build a new kernel (make oldconfig). Spend some time merging existing application config files with etc-update to make them compatible with the latest versions of these packages, reboot and hopefully that should be all there is to it. > Besides, wouldn't each of the incremental processes over the last year > have been possible? ;-) Yes, it would have been, but what is the benefit of updating multiple packages many times over, instead of doing it just once? signature.asc Description: This is a digitally signed message part.
Re: [gentoo-user] What is the best way forward?
Am 25.02.2021 um 02:43 schrieb Grant Taylor: I need to update a system that hasn't been updated in 337 days (March 24th 2020. -- Life has been ... trying. What is the best way forward? It seems as if there have been a lot of changes in the interim; glibc, Python 2.7 being deprecated, default Python going to 3.7(?), other breaking changes Is there a way that I can sync portage to something from April, May, or even June of 2020, do a full update (including "-DUNe @world")? Iterating through multiple rounds to get current? Any help would be appreciated. I found it to be helpful to de-install as many programs as possible before starting the update and the first emerge --sync. This reduces the amount of conflicts by a considerable amount. Stuff like libreoffice or thunderbird and so on and all of their dependencies. Everything your system does not need to run but what you need to be productive when you use your system. I use -av --depclean for this. Also there is something in the gentoo wiki about upgrading old or too old systems. https://wiki.gentoo.org/wiki/Upgrading_Gentoo#Upgrading_from_older_systems
Re: [gentoo-user] What is the best way forward?
On 2/24/21 9:16 PM, John Covici wrote: The portdir has to be the one gotten from git, not rsync, ACK I'm currently doing an "emerge -DUNe @system" on the restore of /usr/portage (typical PORTDIR) from prior to messing with things today. I've got multiple GB of git data. It looks like there are ~568 thousand commits between March 24th last year and now. Once that's good, and I'm back at a stable place, I'll try changing PORTDIR to be the git repo and telling git to switch to the commit that's from March 25th. Then I'll see if anything needs to be updated, doing so as necessary. Then I'll leap frog a week at a time seeing what needs to be updated, doing so as necessary. /Hopefully/ I can slowly walk forward. Time will tell. and remember I think there was a major profile change during that time period along with changes in the C compiler. If I can slowly make my way forward in time via git commit points, I /think/ that I /should/ be able to deal with profile and / or compiler and / or glibc changes just like I would have X number of months ago. I /think/! Unless you have a lot of customizations, reinstall would be much better. I'd really rather not do that. I'm more likely to leave this system as it is and plan on upgrading it some time in '21. There's considerably more to it than I want to wholesale replace. Besides, wouldn't each of the incremental processes over the last year have been possible? ;-) -- Grant. . . . unix || die
Re: [gentoo-user] What is the best way forward?
On Wed, 24 Feb 2021 21:20:05 -0500, Grant Taylor wrote: > > On 2/24/21 6:48 PM, John Covici wrote: > > What you could try to do, if you are syncing using git, is to > > roll it back to those dates by checking out a commit each time > > and doing an update. I don't guarantee it would work, but its > > worth a shot, otherwise reinstall time. > > I hit send too soon. > > Thank you for the reply John. > The portdir has to be the one gotten from git, not rsync, and remember I think there was a major profile change during that time period along with changes in the C compiler. Unless you have a lot of customizations, reinstall would be much better. -- Your life is like a penny. You're going to lose it. The question is: How do you spend it? John Covici wb2una cov...@ccs.covici.com
Re: [gentoo-user] What is the best way forward?
On 2/24/21 6:48 PM, John Covici wrote: What you could try to do, if you are syncing using git, is to roll it back to those dates by checking out a commit each time and doing an update. I don't guarantee it would work, but its worth a shot, otherwise reinstall time. I hit send too soon. Thank you for the reply John. -- Grant. . . . unix || die
Re: [gentoo-user] What is the best way forward?
On 2/24/21 6:48 PM, John Covici wrote: What you could try to do, if you are syncing using git, is to roll it back to those dates by checking out a commit each time and doing an update. I don't guarantee it would work, but its worth a shot, otherwise reinstall time. And what if I was still using rsync? I'm currently doing a git clone of the gentoo-mirror/gentoo.git repository in another directory. Once that finishes, I'll see if I can list the commits in it from March and see if I can work my way forward. Does it /actually/ matter how I get the portage repository as long as it's one from a time close enough that "emerge -DUN @world" will succeed in small increments? Even if I have to automate stepping through hundreds of them. My understanding is that emerge works against the contents of the PORTDIR, usually /usr/portage. So as long as I get ... let's call it ... a compatible version That's my hope anyway. -- Grant. . . . unix || die
Re: [gentoo-user] What is the best way forward?
On Wed, 24 Feb 2021 20:43:39 -0500, Grant Taylor wrote: > > I need to update a system that hasn't been updated in 337 days > (March 24th 2020. -- Life has been ... trying. > > What is the best way forward? > > It seems as if there have been a lot of changes in the interim; > glibc, Python 2.7 being deprecated, default Python going to > 3.7(?), other breaking changes > > Is there a way that I can sync portage to something from April, > May, or even June of 2020, do a full update (including "-DUNe > @world")? Iterating through multiple rounds to get current? > > Any help would be appreciated. What you could try to do, if you are syncing using git, is to roll it back to those dates by checking out a commit each time and doing an update. I don't guarantee it would work, but its worth a shot, otherwise reinstall time. -- Your life is like a penny. You're going to lose it. The question is: How do you spend it? John Covici wb2una cov...@ccs.covici.com
