On Tue, Nov 27, 2018 at 7:20 PM Fabian Groffen <[email protected]> wrote:
> I don't want to depress this entire discussion, but it would be really > nice if we could somehow interact with special machines people have at > their company or at home. Prefix needs testing on many different > machines (non-Linux) which usually don't exist in docker images. > That's alright, we can use QEMU for some more esoteric hardware platforms, if it's an OS that runs on a normal amd64/x86 computer a Docker image can be built (I'm not an expert but there are images to learn how to do it). Or in the worst case we can create an old-school VM for those weird OS and automate the interaction with it (I did it for a robot by dumping all disk once and creating a VM from it, it worked ok). > > That said, focussing on the (usually fast) boxes like this to catch > dependency problems and more is useful. In the case below it looks like > the ld-wrapper is having issues. Would it be possible to enter the > environment for that failed run? > Glad you see the use of it :) Yeah as I mentioned in the previous mail, having docker installed in your machine, to access that exact environment after the failed bootstrap just do: # This will download the image to your machine (it may take a bit of time if its the first time you use docker its around 1GB of data I think) docker pull awesomebytes/gentoo_prefix_latest_image # This will drop you into a shell in that environment, ready to play! docker run -it awesomebytes/gentoo_prefix_latest_image /bin/bash When you are done you can just type exit. > > Thanks, > Fabian > > On 27-11-2018 17:14:19 +1100, Sam Pfeiffer wrote: > > Hello all, > > > > Another little update on my test, it took me a while to find the actual > way to > > increase the timeout to the maximum (or in other words, get out of the > default > > of 60minutes), but I finally found how. > > > > Now I have a job that just tries to bootstrap Gentoo Prefix, the last > run I made > > can be found > > here: [1] > https://dev.azure.com/12719821/12719821/_build/results?buildId=21&view=logs > > > > You can see all the log. In this case it failed after 1h35min. It failed > > compiling GCC 8.2.0-r4. > > > > The error was: > > > > 2018-11-27T03:20:31.7540250Z > > > /tmp/gentoo/var/tmp/portage/sys-devel/gcc-8.2.0-r4/work/build/./prev-gcc/xg++ > > > -B/tmp/gentoo/var/tmp/portage/sys-devel/gcc-8.2.0-r4/work/build/./prev-gcc/ > > -B/tmp/gentoo/usr/x86_64-pc-linux-gnu/bin/ -nostdinc++ > > > -B/tmp/gentoo/var/tmp/portage/sys-devel/gcc-8.2.0-r4/work/build/prev-x86_64-pc-linux-gnu/libstdc++-v3/src/.libs > > > -B/tmp/gentoo/var/tmp/portage/sys-devel/gcc-8.2.0-r4/work/build/prev-x86_64-pc-linux-gnu/libstdc++-v3/libsupc++/.libs > > -isystem > > > /tmp/gentoo/var/tmp/portage/sys-devel/gcc-8.2.0-r4/work/build/prev-x86_64-pc-linux-gnu/libstdc++-v3/include/x86_64-pc-linux-gnu > > -isystem > > > /tmp/gentoo/var/tmp/portage/sys-devel/gcc-8.2.0-r4/work/build/prev-x86_64-pc-linux-gnu/libstdc++-v3/include > > -isystem > > > /tmp/gentoo/var/tmp/portage/sys-devel/gcc-8.2.0-r4/work/gcc-8.2.0/libstdc++-v3/libsupc++ > > > -L/tmp/gentoo/var/tmp/portage/sys-devel/gcc-8.2.0-r4/work/build/prev-x86_64-pc-linux-gnu/libstdc++-v3/src/.libs > > > -L/tmp/gentoo/var/tmp/portage/sys-devel/gcc-8.2.0-r4/work/build/prev-x86_64-pc-linux-gnu/libstdc++-v3/libsupc++/.libs > > -no-pie -fno-PIE -m64 -O2 -pipe -O2 -pipe -gtoggle -DIN_GCC > > -fno-exceptions -fno-rtti -fasynchronous-unwind-tables -W -Wall > -Wno-narrowing > > -Wwrite-strings -Wcast-qual -Wmissing-format-attribute > -Woverloaded-virtual > > -pedantic -Wno-long-long -Wno-variadic-macros -Wno-overlength-strings > > -DHAVE_CONFIG_H -static-libstdc++ -static-libgcc gcov.o \ > > > > 2018-11-27T03:20:31.7545947Z hash-table.o ggc-none.o libcommon.a > > ../libcpp/libcpp.a ../libbacktrace/.libs/libbacktrace.a > > ../libiberty/libiberty.a ../libdecnumber/libdecnumber.a -o gcov > > > > 2018-11-27T03:20:31.7670930Z > > > /tmp/gentoo/var/tmp/portage/sys-devel/gcc-8.2.0-r4/work/build/./prev-gcc/collect-ld: > > 106: exec: /tmp/gentoo/usr/x86_64-pc-linux-gnu/bin/ld: Invalid argument > > > > 2018-11-27T03:20:31.8025313Z collect2: error: ld returned 2 exit status > > > > 2018-11-27T03:20:31.8026192Z Makefile:2935: recipe for target 'gcov' > failed > > > > In the end of the log we get the usual: > > > > 2018-11-27T03:23:00.8458538Z * ERROR: sys-devel/gcc-8.2.0-r4::gentoo > failed > > (compile phase): > > > > 2018-11-27T03:23:00.8480853Z * emake failed > > > > 2018-11-27T03:23:00.8501574Z * > > > > 2018-11-27T03:23:00.8524914Z * If you need support, post the output of > `emerge > > --info '=sys-devel/gcc-8.2.0-r4::gentoo'`, > > > > 2018-11-27T03:23:00.8550010Z * the complete build log and the output of > `emerge > > -pqv '=sys-devel/gcc-8.2.0-r4::gentoo'`. > > > > 2018-11-27T03:23:00.8572142Z * The complete build log is located at > > '/tmp/gentoo/var/tmp/portage/sys-devel/gcc-8.2.0-r4/temp/build.log'. > > > > 2018-11-27T03:23:00.8593188Z * The ebuild environment file is located at > > '/tmp/gentoo/var/tmp/portage/sys-devel/gcc-8.2.0-r4/temp/environment'. > > > > 2018-11-27T03:23:00.8622509Z * Working directory: > > '/tmp/gentoo/var/tmp/portage/sys-devel/gcc-8.2.0-r4/work/build' > > > > 2018-11-27T03:23:00.8642956Z * S: > > '/tmp/gentoo/var/tmp/portage/sys-devel/gcc-8.2.0-r4/work/gcc-8.2.0' > > > > 2018-11-27T03:23:02.9054285Z * > > > > 2018-11-27T03:23:02.9078678Z * Please include > > > /tmp/gentoo/var/tmp/portage/sys-devel/gcc-8.2.0-r4/work/gcc-build-logs.tar.bz2 > > in your bug report. > > > > The cool thing comes now, how do you actually execute those commands to > fill up > > a bug report? > > > > Easy, in the job I upload a Docker image with the system exactly as it > was after > > the boostrap-prefix.sh command. > > > > So, if anyone wants to debug what is going on, they just need to do > (this works > > right now): > > > > docker pull awesomebytes/gentoo_prefix_latest_image > > > > docker run -it awesomebytes/gentoo_prefix_latest_image /bin/bash > > > > And you are inside of the box: > > > > > user@b70d59ff9703:~$ ls > > > > > bootstrap-prefix.sh start_bootstrap_date.txt > > > > > user@b70d59ff9703:~$ cat start_bootstrap_date.txt > > > > > Tue Nov 27 01:52:20 UTC 2018 > > > > > user@b70d59ff9703:~$ ls /tmp/gentoo > > > > > bin etc lib lib64 run sbin stage1.log stage2.log stage3.log > tmp usr > > > var > > > > So you can do those commands suggested by emerge: > > > > /tmp/gentoo/tmp/usr/bin/emerge --info '=sys-devel/gcc-8.2.0-r4::gentoo' > --> > > [2]https://pastebin.com/LWS3Bb7S > > > > /tmp/gentoo/tmp/usr/bin/emerge -pqv '=sys-devel/gcc-8.2.0-r4::gentoo' > > --> [3]https://pastebin.com/gipspmnD > > > > etc. > > > > Or... we could even generate a new bug report automatically from this > will all > > the necessary information, including the instructions on how to get into > this > > box with exactly that state for anyone to help on fixing it. > > > > I'll try to dig further in with things that I find useful (and I hope > you also > > find useful). > > > > Have a nice day! > > > > P.S.: With this I created a bug report easily: [4] > https://bugs.gentoo.org/672042 > > > > On Tue, Nov 27, 2018 at 2:07 AM Sam Pfeiffer <[5][email protected]> > wrote: > > > > > Little update: The full build log is viewable to anyone with the link, > so here > > > you can see the progress of the current job: > > > > > [6] > https://dev.azure.com/12719821/12719821/_build/results?buildId=17&view=logs > > > > > (Or I should say, the log of it, for whenever you open the link). > > > > > On Tue, Nov 27, 2018 at 2:02 AM Sam Pfeiffer <[7] > [email protected]> > > > wrote: > > > > >> Hello everyone, > > > > >> I'm very excited to see you are interested in adding continuous > integration! > > > > >> I don't know that much about continuous integration, I've only used > it (with > > >> systems already setup for me) with in-house Jenkins servers and with > the ROS > > >> buildfarm, based on Travis CI on Github. Also a little bit of Gitlab > CI in my > > >> lab. > > > > >> I did a bit of research/testing. > > > > >> Given it's quite a hassle to maintain custom machines, I'd try to use > some of > > >> the free for opensource CI services. I've checked the conditions of a > few to > > >> see which could fit better: > > > > >> * Gitlab CI: 2000 minutes / month > > > > >> * Travis CI: Unlimited minutes / month. But only 50 minutes long per > step > > >> (like per script executed). > > > > >> * Azure pipelines: Unlimited minutes / month. 360 minutes long per > step (like > > >> per script executed). > > > > >> There are probably many more, but those are the ones I knew about. > > > > >> Given that I wanted to give a try to Azure pipelines. And I did! > > > > >> I created this repo: [8] > https://github.com/awesomebytes/gentoo_prefix_ci_test > > > > >> Where I activated Azure pipelines on it. After around 15min of > reading the > > >> docs and playing around with the web-gui I got my first pipeline > running. > > > > >> As an initial setup I thought I would create a Docker image where I > bootstrap > > >> Gentoo Prefix from a Ubuntu 16.04 (as I'm familiar with it with my > projects). > > > > >> The repo contains two important things: > > > > >> 1) The Dockerfile where I mainly trigger the > > >> bootstrap: [9] > https://github.com/awesomebytes/gentoo_prefix_ci_test/blob/master/Dockerfile > > > > >> 2) The configuration file for Azure pipelines on what to > > >> do: [10] > https://github.com/awesomebytes/gentoo_prefix_ci_test/blob/master/azure-pipelines.yml > > > > >> I've implemented here that it tries to build Gentoo Prefix, and > whatever the > > >> result, it uploads a Docker image to my DockerHub account with the > results. > > >> This implies that: > > > > >> If the bootstrap is successful, one can just [docker pull] and > [docker run] > > >> the image to play with Gentoo Prefix. > > > > >> If the bootstrap is unsuccessful, one can just [docker pull] and > [docker run] > > >> to find oneself in the exact state of the system after the bootstrap > command. > > >> And one can recover the full console log from the Azure pipelines web > > >> interface (even tho it would be nice to find out how to post it > publicly > > >> straight away). > > > > >> If all goes well in a few hours anyone will be able to find in my > DockerHub > > >> account said image (most probably the failed one), just doing: > > > > >> docker pull awesomebytes/gentoo_prefix_latest_image:latest > > > > >> docker run -it gentoo_prefix_latest_image /bin/bash > > > > >> You'll be inside of a Ubuntu 16.04 box with a user called 'user' and > with all > > >> the bootstrapped stuff in /tmp/gentoo. > > > > >> As curiosity, I checked the machines I got served as 'agents' to run > my jobs, > > >> and they were of the kind: > > > > >> CPUs: 2x Intel(R) Xeon(R) CPU E5-2673 v3 @ 2.40GHz > > > > >> RAM: 7GB > > > > >> Disk: 94GB free disk space > > > > >> More than enough to bootstrap Gentoo Prefix! > > > > >> I don't know if this is the way to go. But at least is interesting to > have it > > >> in mind. > > > > >> On Tue, Nov 27, 2018 at 12:01 AM Benda Xu <[11][email protected]> > wrote: > > > > >>> Hi Sam, > > > > >>> Sam Pfeiffer <[12][email protected]> writes: > > > > >>> > With Azure announcing unlimited minutes on CI/CD for open source > > >>> > projects: > > >>> > > > >>> [13] > https://azure.microsoft.com/en-us/blog/announcing-azure-pipelines-with-unlimited-ci-cd-minutes-for-open-source/ > > >>> > > > >>> > Even bootstrapping Gentoo prefix, with pieces of software like gcc > > >>> > taking very long to compile, is possible. > > >>> > > > >>> > The point is: I have been trying to build Gentoo Prefix during the > > >>> > last days after a few months of break since the last time I touched > > >>> > the system. And it's failing. I haven't managed yet to bootstrap it > > >>> > completely. I feel there is no CI/CD setup to catch these issues > and > > >>> > be able to offer a working version of Gentoo Prefix at any time. > > > > >>> I completely agree with you. I hope you can carry on this project to > > >>> setup proper CI for Gentoo Prefix. I am all in for help, > portage/ebuild > > >>> mentoring and coorperation. > > > > >>> A CI for Gentoo Prefix has been on my list for ages. Thank you for > > >>> triggering this. > > > > >>> Yours, > > >>> Benda > > > > >> -- > > > > >> Sammy Pfeiffer > > >> PhD Candidate at The Magic Lab within UTS. > > > > > -- > > > > > Sammy Pfeiffer > > > PhD Candidate at The Magic Lab within UTS. > > > > -- > > > > Sammy Pfeiffer > > PhD Candidate at The Magic Lab within UTS. > > > > > > > > References: > > 1. > https://dev.azure.com/12719821/12719821/_build/results?buildId=21&view=logs > > 2. https://pastebin.com/LWS3Bb7S > > 3. https://pastebin.com/gipspmnD > > 4. https://bugs.gentoo.org/672042 > > 5. mailto:[email protected] > > 6. > https://dev.azure.com/12719821/12719821/_build/results?buildId=17&view=logs > > 7. mailto:[email protected] > > 8. https://github.com/awesomebytes/gentoo_prefix_ci_test > > 9. > https://github.com/awesomebytes/gentoo_prefix_ci_test/blob/master/Dockerfile > > 10. > https://github.com/awesomebytes/gentoo_prefix_ci_test/blob/master/azure-pipelines.yml > > 11. mailto:[email protected] > > 12. mailto:[email protected] > > 13. > https://azure.microsoft.com/en-us/blog/announcing-azure-pipelines-with-unlimited-ci-cd-minutes-for-open-source/ > > > > read_char: errno==EILSEQ; invalid byte sequence for UTF-8: > -- > Fabian Groffen > Gentoo on a different level > -- *Sammy Pfeiffer* PhD Candidate at The Magic Lab within UTS.
