Re: [aur-general] pkgbuild - suggestion
Sometimes there may be a new version out, but the maintainer needs to test and make sure it's working as expected. It may need a patch to work correctly, or it may have a major flaw and every other distro is also avoiding updating to the latest version. The flagging system, I feel at least, is more so there for users to nudge a maintainer if they haven't updated it. Often people can wait a few days after the upstream release for the new version, and if they can't, they should be using the -git version anyway.
Re: [aur-general] Continuous integration of AUR packages
I was using the chroot tools for a year and a half or so before that, but had some issues occasionally with things like updating it, or using package caches inside the chroot, or other various things. I do still have the shell script I used at the time, it was called by jenkins and cloned a chroot based off a clean copy and numbered by the jenkins executor to support multiple builds at once. But I managed a few tens of thousands of docker containers for work, and figured I could use a similar build system for the arch packages, and it's been reliable for me since then. As Doug said, you need to add the packages you build to a repository for the next build to be able to install them, but that part is pretty simple and explained on the arch wiki already. So, in Jenkins I use the build generator plugin, and created a parameterized job (config.xml here: http://hastebin.com/oyuqiwokem.xml) that basically just calls `dmakepkg` on the git checkout for the aur package; you just give the build an aur package name and it will generate the job to track that package and any changes pushed to the AUR and build it. (it polls git once an hour). `dmakepkg` is in the AUR ( https://aur.archlinux.org/packages/dmakepkg-git/ ) and is what I ended up making to build the packages inside a docker container. It supports the same flags as makepkg (just passes them through) but defaults to '--force --syncdeps --noconfirm' unless other flags are specified (no point removing packages since the container is destroyed after use). Uses a slightly modified version of the official docker arch image, and is updated much more often (monthly at least instead of the 3-6 months they take). It auto-detects makepkg configs like SRCDEST/etc and copies it in to the instance and bind mounts the folders specified, as well as the pacman package cache, so all the builds follow the makepkg config on the host, but using everything else from a clean environment. If you make your own repo and add it to your host, you can use `-x` flag to use the host's pacman.conf inside the container.
Re: [aur-general] Continuous integration of AUR packages
On Thu, 18 Feb 2016 12:51:19 +0100 Baptiste Jonglez wrote: > Hi, > > The scripts in devtools [1] look like they should work just fine to > automate these kind of builds. After all, they are used to build the > official Archlinux packages. > > However, I found that the build scripts do not really handle dependencies. > When building a given package, they just install deps and makedeps from > the existing Archlinux repositories. This is an issue when AUR packages > depend on each other, because then dependencies cannot be installed from > the Archlinux repositories... ... > Thanks, > Baptiste > So create you own local repository and add the binary packages to it after they're built. pgp2gLw_2gmPF.pgp Description: OpenPGP digital signature
Re: [aur-general] Continuous integration of AUR packages
Hi, On Thu, Feb 18, 2016 at 12:25:53PM +, Justin Dray wrote: > I actually have a Jenkins setup for my packages that does almost exactly > that, I have a job generator build that I give an AUR package name to and > that's all the config for new packages, and it builds them in a docker > container. > > You can obviously set up dependencies between the jobs to build in order, > but that part isn't automatic yet. Its been pretty reliable for me over the > past year or so since I last really changed anything. > > If you're interested, let me know and I can share the scripts and stuff. > Just on my phone at the moment or I would link them here. Yes, I would be interested! Out of curiosity, why do you use docker containers instead of the "official" method with chroots? https://wiki.archlinux.org/index.php/DeveloperWiki:Building_in_a_Clean_Chroot > Cheers, > Justin > > On Thu, Feb 18, 2016, 9:51 PM Baptiste Jonglez > wrote: > > > Hi, > > > > I would like to run some kind of continuous integration of my AUR > > packages. The goal is to know when a package I maintain fails to build > > because either: > > > > 1/ its dependencies have been updated (new API, new incompatible version > >of GCC, ...) > > > > 2/ for -git packages, changes made upstream broke something (new > >dependency needed, new build system...) > > > > The scripts in devtools [1] look like they should work just fine to > > automate these kind of builds. After all, they are used to build the > > official Archlinux packages. > > > > However, I found that the build scripts do not really handle dependencies. > > When building a given package, they just install deps and makedeps from > > the existing Archlinux repositories. This is an issue when AUR packages > > depend on each other, because then dependencies cannot be installed from > > the Archlinux repositories... > > > > It *is* possible to manually pass packages to install in the chroot before > > building, but this is far from convenient. For instance, ring-daemon > > depends on opendht, so I would need to do this: > > > > cd opendht > > extra-x86_64-build > > cd ../ring-daemon > > extra-x86_64-build -- -I ../opendht/opendht-0.5.1-1-x86_64.pkg.tar.xz > > > > When there are multiple dependencies, it quickly becomes a nightmare to > > automate (especially because dependencies need to be passed in the right > > order). > > > > Is there any script that automates dependency handling when building? > > Basically, it would probably need to perform a topological sort, build > > packages in this order, and push them to a local repository to be able to > > build later packages. > > Or did I take the wrong approach entirely? > > > > Thanks, > > Baptiste > > > > [1] https://projects.archlinux.org/devtools.git/ > > > > PS : Some existing efforts I found about CI with Arch: > > > > - > > https://lists.archlinux.org/pipermail/arch-dev-public/2014-November/026757.html > > https://jenkins.arch-ci.org/ > > No script provided, site appears to be down > > > > - http://openbuildservice.org/2012/09/10/arch-linux-support/ > > Source code is unreadable (enormous Perl scripts, no modularity) > > > > signature.asc Description: PGP signature
Re: [aur-general] pkgbuild - suggestion
automatic notification sending is i believe a way to hell (10th+ notification for the same package would only make maintainer more angry). the idea is to only show it (when asked to) to user. then it's up to him/her to either notify maintainer, make a patch and do the update him/herself or even uninstall the package (if some serious problem has been found with the current version). On Wed, 17 Feb 2016, Félix Piédallu wrote: Le 17 février 2016 23:11:03 GMT+01:00, Jozef Riha a écrit : hello, this might not be a perfect place to propose new things to pkgbuild but otoh it does affect pkgbuild and does make sense for aur packages. my point is: at the moment the only way that the aur maintainers learn about upstream version bump is either by a user notification system (flags on aur) or some external (rss for instance) checks. this may work to some extent but wouldn't it be nice if the checking function would be the part of the pkgbuild (a new function alongside prepare(), build(), package()..) and this function would be invoked by an aur wrapper such as yaourt? the implementation probably would not be trivial due to security concerns but in the end, maybe having a possibility to process an html page with a regex or read rss would be a good start. j What do u mean exactly ? That when users install the package, some yaourt/makepkg tool sends a "feedback" of the version ? That would work mostly for CVS packages, when executing pkgver(). Wouldn't it better to have this kind of check executed on the AUR server ? But that would mean a (weekly ?) download of the package on the AUR server... Not the best solution i think :/ -- Envoyé de mon appareil Android avec K-9 Mail. Veuillez excuser ma brièveté.
Re: [aur-general] Continuous integration of AUR packages
I actually have a Jenkins setup for my packages that does almost exactly that, I have a job generator build that I give an AUR package name to and that's all the config for new packages, and it builds them in a docker container. You can obviously set up dependencies between the jobs to build in order, but that part isn't automatic yet. Its been pretty reliable for me over the past year or so since I last really changed anything. If you're interested, let me know and I can share the scripts and stuff. Just on my phone at the moment or I would link them here. Cheers, Justin On Thu, Feb 18, 2016, 9:51 PM Baptiste Jonglez wrote: > Hi, > > I would like to run some kind of continuous integration of my AUR > packages. The goal is to know when a package I maintain fails to build > because either: > > 1/ its dependencies have been updated (new API, new incompatible version >of GCC, ...) > > 2/ for -git packages, changes made upstream broke something (new >dependency needed, new build system...) > > The scripts in devtools [1] look like they should work just fine to > automate these kind of builds. After all, they are used to build the > official Archlinux packages. > > However, I found that the build scripts do not really handle dependencies. > When building a given package, they just install deps and makedeps from > the existing Archlinux repositories. This is an issue when AUR packages > depend on each other, because then dependencies cannot be installed from > the Archlinux repositories... > > It *is* possible to manually pass packages to install in the chroot before > building, but this is far from convenient. For instance, ring-daemon > depends on opendht, so I would need to do this: > > cd opendht > extra-x86_64-build > cd ../ring-daemon > extra-x86_64-build -- -I ../opendht/opendht-0.5.1-1-x86_64.pkg.tar.xz > > When there are multiple dependencies, it quickly becomes a nightmare to > automate (especially because dependencies need to be passed in the right > order). > > Is there any script that automates dependency handling when building? > Basically, it would probably need to perform a topological sort, build > packages in this order, and push them to a local repository to be able to > build later packages. > Or did I take the wrong approach entirely? > > Thanks, > Baptiste > > [1] https://projects.archlinux.org/devtools.git/ > > PS : Some existing efforts I found about CI with Arch: > > - > https://lists.archlinux.org/pipermail/arch-dev-public/2014-November/026757.html > https://jenkins.arch-ci.org/ > No script provided, site appears to be down > > - http://openbuildservice.org/2012/09/10/arch-linux-support/ > Source code is unreadable (enormous Perl scripts, no modularity) > >
[aur-general] Continuous integration of AUR packages
Hi, I would like to run some kind of continuous integration of my AUR packages. The goal is to know when a package I maintain fails to build because either: 1/ its dependencies have been updated (new API, new incompatible version of GCC, ...) 2/ for -git packages, changes made upstream broke something (new dependency needed, new build system...) The scripts in devtools [1] look like they should work just fine to automate these kind of builds. After all, they are used to build the official Archlinux packages. However, I found that the build scripts do not really handle dependencies. When building a given package, they just install deps and makedeps from the existing Archlinux repositories. This is an issue when AUR packages depend on each other, because then dependencies cannot be installed from the Archlinux repositories... It *is* possible to manually pass packages to install in the chroot before building, but this is far from convenient. For instance, ring-daemon depends on opendht, so I would need to do this: cd opendht extra-x86_64-build cd ../ring-daemon extra-x86_64-build -- -I ../opendht/opendht-0.5.1-1-x86_64.pkg.tar.xz When there are multiple dependencies, it quickly becomes a nightmare to automate (especially because dependencies need to be passed in the right order). Is there any script that automates dependency handling when building? Basically, it would probably need to perform a topological sort, build packages in this order, and push them to a local repository to be able to build later packages. Or did I take the wrong approach entirely? Thanks, Baptiste [1] https://projects.archlinux.org/devtools.git/ PS : Some existing efforts I found about CI with Arch: - https://lists.archlinux.org/pipermail/arch-dev-public/2014-November/026757.html https://jenkins.arch-ci.org/ No script provided, site appears to be down - http://openbuildservice.org/2012/09/10/arch-linux-support/ Source code is unreadable (enormous Perl scripts, no modularity) signature.asc Description: PGP signature
Re: [aur-general] pkgbuild - suggestion
i think it is the job of a maintainer to keep the pkgbuild updated (while
buildable at the same time). with keeping an eye on new upstream
releases.. not sure. why the "notify" button on aur webpage exists in the
first place. and then you also have pkgbuilds that lost their maintainers,
maintainer became unresponsive (is on vacation, ill, whatever). in such
cases you probably still want to be updated or at least be informed there
are updates pending.
to be honest i did not think this through - having defined this as a
function would indeed require repetitious pull from aur server which isn't
ideal. running weekly checks on aur server itself i believe really defeats
the kiss principle and picturing myself as an aur maintainer i would not
like to do it.
thank you for suggesting urlwatch - seems like a nice tool that simply
does the job. i was able to quickly cobble a oneliner to keep track of my
favourite tool: gpsprune. all i had to do was just put the gpsprune
website to urls.txt and define a hook that parses the useful information
(i certainly do not want to watch changes on the whole page).
in the end it looks like this:
def filter(url, data):
m = re.search(r'.*Runnable code(.+)\.jar.*',html2txt.html2text(data,
method='re'), re.MULTILINE)
return m.group(1)
now what if this is defined inside pkgbuild on a single line (so that it
stays in the local db), for instance like this:
UrlWatcher=('http://activityworkshop.net/software/gpsprune/download.html','.*Runnable
code(.+)\.jar.*',1)
i don't think this leaves much space for attacks on system
security/integrity - the arguments are being parsed by urlwatcher, not
executed.
so the idea is that i normally do yaourt -Syu --aur daily at present.
wouldn't it be nice if one could do (say - once a week) yaourt -Syu --aur
--check-up-stream-changes and get:
* gpsprune - possibly a new upstream version detected
?
cheers,
jose
On Wed, 17 Feb 2016, Eli Schwartz wrote:
On 02/17/2016 05:11 PM, Jozef Riha wrote:
hello,
this might not be a perfect place to propose new things to pkgbuild but
otoh it does affect pkgbuild and does make sense for aur packages. my point
is: at the moment the only way that the aur maintainers learn about
upstream version bump is either by a user notification system (flags on
aur) or some external (rss for instance) checks. this may work to some
extent but wouldn't it be nice if the checking function would be the part
of the pkgbuild (a new function alongside prepare(), build(), package()..)
and this function would be invoked by an aur wrapper such as yaourt?
the implementation probably would not be trivial due to security concerns
but in the end, maybe having a possibility to process an html page with a
regex or read rss would be a good start.
j
As you rightly pointed out, that is the job of the AUR *maintainer*.
Check out urlwatch...
--
Eli Schwartz
[aur-general] Signoff report for [community-testing]
=== Signoff report for [community-testing] === https://www.archlinux.org/packages/signoffs/ There are currently: * 1 new package in last 24 hours * 0 known bad packages * 0 packages not accepting signoffs * 0 fully signed off packages * 8 packages missing signoffs * 4 packages older than 14 days (Note: the word 'package' as used here refers to packages as grouped by pkgbase, architecture, and repository; e.g., one PKGBUILD produces one package per architecture, even if it is a split package.) == New packages in [community-testing] in last 24 hours (1 total) == * python-hypothesis-3.0.0-1 (any) == Incomplete signoffs for [community] (8 total) == * pdfsam-3.0.1-1 (any) 0/2 signoffs * python-hypothesis-3.0.0-1 (any) 0/2 signoffs * gdal-2.0.1-7 (i686) 0/1 signoffs * jack2-1.9.10-4 (i686) 0/1 signoffs * postgis-2.2.1-2 (i686) 0/1 signoffs * gdal-2.0.1-7 (x86_64) 0/2 signoffs * jack2-1.9.10-4 (x86_64) 0/2 signoffs * postgis-2.2.1-2 (x86_64) 0/2 signoffs == All packages in [community-testing] for more than 14 days (4 total) == * postgis-2.2.1-2 (i686), since 2016-01-22 * postgis-2.2.1-2 (x86_64), since 2016-01-22 * gdal-2.0.1-7 (i686), since 2016-01-23 * gdal-2.0.1-7 (x86_64), since 2016-01-23 == Top five in signoffs in last 24 hours == 1. djgera - 2 signoffs
