Re: [Lava-users] Seek Assistance: LAVA-lxc Job creation and submition
On 7 July 2017 at 07:23, Chetan Sharma wrote: > Hi Everyone > Hopefully everyone is doing well in this group. > Main intend of writing this email to seek assistance in regard to one > feature of Lava as lava-lxc which help to create a LXC instance on worker https://staging.validation.linaro.org/scheduler/job/174215/definition#defline22 > and then we can execute any script on worker and propagate its > characteristics and result in another actions of Job on board. That isn't entirely clear. What are you trying to do in the LXC? You need to deploy an LXC, start it with a boot action for the LXC and then start a test shell in the LXC where you can install the tools you need to execute. Talking to the device from the LXC can be difficult depending on how the device is configured. To use networking, you would have to have a fixed IP for the device. To use USB, you need to use the device_info support in the device dictionary to add the USB device to the LXC. > I have gone through documentation shared on Lava V2 instance for LXC > job creation, but i am not able to successfully execute job on Debian Jessie > Release. > https://validation.linaro.org/static/docs/v2/deploy-lxc.html#index-0 > >Can you assist me and share a reference process/document to proceed > further to create a job using this feature. More information is needed on exactly what you are trying to do, how the LXC is to connect to the device and what support the device offers to allow for those operations. -- Neil Williams = neil.willi...@linaro.org http://www.linux.codehelp.co.uk/ ___ linaro-dev mailing list linaro-dev@lists.linaro.org https://lists.linaro.org/mailman/listinfo/linaro-dev
Re: LAVA device type template manual
On Mon, 6 Mar 2017 15:58:07 + Ковалёв Сергей wrote: > Hello. > > Does there exists some documentation about device type template? https://staging.validation.linaro.org/static/docs/v2/development-intro.html#developing-using-device-type-templates also https://staging.validation.linaro.org/static/docs/v2/pipeline-admin.html https://staging.validation.linaro.org/static/docs/v2/devicetypes.html#device-types https://staging.validation.linaro.org/static/docs/v2/first-devices.html#adding-new-device-types > I have started to study LAVA recently. I have read some parts of > documentation already but I can't find any documentation about device > type template syntax and semantics. So I'm reading present "jinja" > files but I don't know what I can change or add. From the last link above: [Adding new device types] is the most complex part and it can be a lot of work (sometimes several months) to integrate a completely new device into LAVA. V2 offers a different and wider range of support to V1 but some devices will need new support to be written within lava-dispatcher. It is not always possible to automate a new device, depending on how the device connects to LAVA, how the device is powered and whether the software on the device allows the device to be controlled remotely. So what you can change or add depends on what you are trying to do but you should first add known devices and understand how those work before considering support for your own devices. -- Neil Williams = http://www.linux.codehelp.co.uk/ pgp1Cl0qNrulA.pgp Description: OpenPGP digital signature ___ linaro-dev mailing list linaro-dev@lists.linaro.org https://lists.linaro.org/mailman/listinfo/linaro-dev
Re: Issues with Django OpenID Authentication
On Tue, 1 Dec 2015 17:35:10 + Wookey wrote: > +++ Neil Williams [2015-12-01 16:52 +]: > > Please let us know if you are using OpenID authentication with > > LAVA. > > I use OpenID whenever I get the chance as one of the few ID protocols > where I get some control, and don't just have to handover auth to one > of the dubious web mega-corps, so I am in favour of it being an > option. If you fancy helping with https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=806352 :-) Sadly, there isn't a sane way of combining different authentication methods on a single instance as it massively complicates the admin burden of ensuring that the correct users have the correct group permissions. So when production switched from OpenID and Launchpad to LDAP run by Linaro internally, it was a switch - not an addition. It moved the authentication entirely within Linaro instead of going out to Launchpad. > I've not used lava much recently (so am rather vague on the > details without checking), but I might have to use it again soon. As far as any instance maintained by the Linaro Lab team is concerned, all logins are now using LDAP internally within Linaro and OpenId has been disabled already. The two authentication mechanisms are mutually incompatible as there is no assurance of matching one user to the one account when logging in using the other. This made the admin burden of ensuring that the correct users have the correct access to relevant devices infeasibly complex. > I think I was logging in via launchpad, which I think means I was/am > indeed using openID for the webUI? If you're thinking of validation.linaro.org, you probably did log in using Launchpad as the admin records show that you haven't logged in for some time - quite likely before production did the switch to LDAP. However, your next login will need to use LDAP. > I also recall something about > feeding a key into gnome-keyring for command-line access - is that all > old hat now? No, that is the lava-tool interface (which can be replaced by simple xmlrpc scripts if you want to avoid python-keyring). -- Neil Williams = http://www.linux.codehelp.co.uk/ pgpo24JvWA5js.pgp Description: OpenPGP digital signature ___ linaro-dev mailing list linaro-dev@lists.linaro.org https://lists.linaro.org/mailman/listinfo/linaro-dev
Issues with Django OpenID Authentication
Please let us know if you are using OpenID authentication with LAVA. Newer versions of django will make it impossible to support django-openid-auth in Debian unstable and testing. The version of django-openid-auth in Jessie can continue to be used, so we would like to know how many users want to continue with this support. OpenID as a protocol has been dying for some time and Linaro has moved over to LDAP, which is fine if LDAP is already available. The time pressure for this change is coming from the schedule to get the latest django and the latest lava packages into Ubuntu Xenial 16.04LTS which means that support needs to be implemented in the 2015.12 or 2016.1 LAVA releases. This is why this is quickly following the trusty change. We have been aware of the issues with django-openid-auth for some time, it was only when we had completed the move of the Cambridge lab to LDAP that changes involving django-openid-auth could be considered. If you are using OpenID authentication (e.g. using Launchpad or Google OpenID), please let us know. If you would like to see some other forms of authentication supported, also let us know. We can investigate Python Social Auth (http://psa.matiasaguirre.net/), if there is interest. If we don't hear from users who want django-openid-auth support for use on Debian Jessie, we will drop django-openid-auth support from all lava builds. This will leave LDAP and local Django accounts in 2015.12. If anyone has experience of other django authentication modules, also let us know. -- Neil Williams = neil.willi...@linaro.org http://www.linux.codehelp.co.uk/ ___ linaro-dev mailing list linaro-dev@lists.linaro.org https://lists.linaro.org/mailman/listinfo/linaro-dev
Freezing support for LAVA on Ubuntu Trusty 14.04LTS
See also: https://lists.linaro.org/pipermail/lava-announce/2015-November/02.html which was also sent to these lists (except linaro-dev). So far, nobody has come forward as a Trusty user. The only Trusty instance of which we are aware is already due to migrate to Debian Jessie. The LAVA software team are now applying updates which will freeze LAVA software support for Ubuntu Trusty at 2015.9 for lava-dispatcher and 2015.9.post1 for lava-server due to the complexities of supporting both django1.6 and the current django1.7 in Jessie and django1.8, possibly django1.9 by the time Debian Stretch is released. The last packages for Ubuntu Trusty 14.04LTS will be: lava-server 2015.9.post1 lava-dispatcher 2015.9 Once these changes are applied, the Debian packaging used to build future versions of LAVA packages will prevent builds against django1.6 and prevent installation if django1.6 is found, in order to prevent database corruption. This means that Trusty users will not be able to use the results of the dispatcher refactoring. Ubuntu Xenial Xerus - which is planned to be the 16.04LTS in April 2016 - is expected to pick up LAVA software releases from Debian up until the 2016.1 release (possibly 2016.2) and is also expected to be using django1.8. The next Debian stable release (Stretch), for which no date has yet been set, may use django1.9. Initial attempts at migrating a test instance from Trusty to django1.7 did not go well and the migration from Trusty to Xenial cannot be supported by the LAVA software team - the recommendation is to go directly from 2015.9 on Trusty to the same version available for Debian Jessie but there will still be work to be done to prepare and implement the migration which will be instance-dependent. Documentation is being added to assist with this migration but there will remain risks of data loss which will need to be managed for each instance. It is imperative that anyone using Trusty has an up to date backup of the postgresql database dump before considering any migration. If the existing data is to be dropped, a new install on Debian Jessie is recommended. It is not possible for the LAVA software team to support all versions of django from 1.6 to 1.9 - particular problems are known when going from django1.6 to django1.7 as the methods to migrate the lava-server database changed fundamentally in django1.7. Notes are being added to the documentation on the trusty branch based on 2015.9 to be released within lava-server 2015.9.post1 and to the documentation in the master branch (which will go into 2015.12). All future builds of LAVA software will now be made and uploaded only to Debian and releases.linaro.org. So far, nobody has come forward who is willing to maintain packaging for LAVA software on any distribution other than Debian. As the refactoring proceeds, we expect that it will become easier to package LAVA for other distributions but the migration to the refactoring must be complete first. Everyone interested in or using LAVA is encouraged to subscribe to the lava-announce mailing list which is low volume and only used for substantial changes like this. https://lists.linaro.org/mailman/listinfo/lava-announce See also https://validation.linaro.org/static/docs/support.html -- Neil Williams = neil.willi...@linaro.org http://www.linux.codehelp.co.uk/ ___ linaro-dev mailing list linaro-dev@lists.linaro.org https://lists.linaro.org/mailman/listinfo/linaro-dev
Re: Debian kernel for x86 in CI
On Thu, 8 Oct 2015 15:20:47 +0200 Krishna Garapati wrote: > Hi Steve, > > I tried the other debian package which is in Note: Be careful with your terminology. "debian package" means a .deb that you install on the system. What you are looking for is a kernel image for x86 or an x86 Debian image. > http://images.validation.linaro.org/x86/ > <http://images.validation.linaro.org/x86/debian/vmlinuz-3.16.0-4-amd64>, > since I was able to do hack session on this successfully. When I > tried the one you mentioned, I couldn't get though the hack session, > as it fails at "lava-lng-generator-01/bin/lava-test-runner: > not found". > > this is the job I had on the mentioned package, > > https://lng.validation.linaro.org/scheduler/job/895/log_file > > It went everything fine until that point. Do you know what would be > the reason? You asked the test to boot a ramdisk but that ramdisk is not a full system, it is incapable of installing packages like openssh-server. You need to use NFS to get a system which can actually run an sshd daemon (or adapt the ramdisk to have dropbear installed or similar and then change the hacking session YAML). Note that for NFS to work, the kernel you've chosen must have NFS support compiled in and the log for your job looks like it does not (which is not unexpected for a distribution kernel). Hacking a ramdisk to work on x86 isn't trivial either. https://lng.validation.linaro.org/scheduler/job/895/definition If you just want a hacking session using x86, a QEMU job is a much easier deployment. https://staging.validation.linaro.org/scheduler/job/134816/definition Booting x86 bare metal into NFS isn't straightforward, most of the kernel images you will find will be expecting to find local media and then support NFS of other parts of the filesystem once / is mounted locally. > /Krishna > > > > On 8 October 2015 at 15:07, Steve McIntyre > wrote: > > > On Thu, Oct 08, 2015 at 10:05:28AM +0200, Krishna Garapati wrote: > > >Hi, > > > > > >I am looking for a debian package for x86 target in Lava. Can > > >anyone tell > > me > > >where can I find it?. I found some at > > http://images.validation.linaro.org/x86/, > > >but i am missing Linux-headers included for these packages. I like > > >to > > deploy an > > >image which has the kernel headers included. I tried > > "linux-headers-$(uname -r) > > >" on the target, but that didn't help. > > > > Hi Krishna, > > > > If you're looking at the kernel in > > > > http://images.validation.linaro.org/x86/debian/vmlinuz-3.16.0-4-amd64 > > > > then I believe that's just a normal Debian kernel and the > > linux-headers package should definitely exist for it. I'd try the > > linux-headers-amd64 package first, described as > > > > "Header files for Linux amd64 configuration (meta-package) This > >package depends on the architecture-specific header files for the > >latest Linux kernel amd64 configuration." > > > > > > Cheers, > > -- > > Steve McIntyre > > steve.mcint...@linaro.org <http://www.linaro.org/> Linaro.org | > > Open source software for ARM SoCs > > > > -- Neil Williams = http://www.linux.codehelp.co.uk/ pgpBEdB8UsHhx.pgp Description: OpenPGP digital signature ___ linaro-dev mailing list linaro-dev@lists.linaro.org https://lists.linaro.org/mailman/listinfo/linaro-dev
Re: 'test' hangout
On Tue, 11 Feb 2014 16:17:47 + Wookey wrote: > Hangout audio and video is regularly broken for people which causes > general aggravation for people in meetings. Could we have a > 'test/corridor/watercooler/coffeemachine' hangout somewhere for people > to test if it's working for them today? > > Maybe there is one already? In LAVA we use a combination of IRC and a daily calendar event - if anyone wants to check their setup, they can ask on IRC and use the daily standup event, except during the 15 minutes each work day when it's actually in use. Any team could setup a similar daily / weekly event, if one doesn't exist already. I'm using a wheezy chroot to get around problems with the plugin on unstable/testing. The only problem I get is when trying to do screenshare, my audio clips constantly. -- Neil Williams = http://www.linux.codehelp.co.uk/ signature.asc Description: PGP signature ___ linaro-dev mailing list linaro-dev@lists.linaro.org http://lists.linaro.org/mailman/listinfo/linaro-dev
Re: GSOC results example
On Mon, 6 Jan 2014 16:58:34 + Steve McIntyre wrote: > So, who's interested in doing the GSOC this year? > > If we want to be involved, we need to start thinking about it sooner > rather than later! Me! I've got an idea for a proposal to make LAVA work with email submissions which would help integrate LAVA into some of the Debian testing strategies. -- Neil Williams = http://www.linux.codehelp.co.uk/ signature.asc Description: PGP signature ___ linaro-dev mailing list linaro-dev@lists.linaro.org http://lists.linaro.org/mailman/listinfo/linaro-dev
Re: Google hangout not working in debian sid/testing
On Thu, 10 Oct 2013 13:39:27 +0300 Serge Broslavsky wrote: > > However, with libcairo2 1.12.14-4, hangouts now work. > > > > Thanks for this investigation. I'll have to see how long the rest of > > the desktop continues running with an old version of such a critical > > library > > +1, thanks Nicolas! > > The magic formula is: > $ wget > http://snapshot.debian.org/archive/debian/20130927T214600Z/pool/main/c/cairo/libcairo2_1.12.14-4_amd64.deb > $ sudo dpkg -i libcairo2_1.12.14-4_amd64.deb While it lasts. Sooner rather than later, an updated package elsewhere in sid/testing is going to need the updated cairo which underpins all of GTK (Gnome, XFCE, LXDE, Mint etc.). The better solution will be a wheezy chroot with a browser installed and using X forwarding. If we could see the source code for google-talkplugin, it could be fixed -- Neil Williams = http://www.linux.codehelp.co.uk/ signature.asc Description: PGP signature ___ linaro-dev mailing list linaro-dev@lists.linaro.org http://lists.linaro.org/mailman/listinfo/linaro-dev
Re: Google hangout not working in debian sid/testing
On Tue, 1 Oct 2013 10:44:42 +0200 Nicolas Dechesne wrote: > hi, > > first of all, this isn't meant to be a distro trolling discussion, > there are better places for that ;-) > > several days ago google HO started to fail after updating my > debian/sid setup. as discussed on G+ [1] and on googlegroups [2], > that seemed to be impacting anyone on debian/sid. when i realized > that it was working on debian/testing (Jessie) i decided just use > 'testing'... however since last evening update on 'testing', it > started to fail too.. leaving me without any working HO at all, which > is quite inconvenient at Linaro.. > > after bisecting my system using snapshot.debian.org, i figured out > which package was the culprit: > > # apt-cache policy libcairo2 > libcairo2: > Installed: 1.12.14-4 > Candidate: 1.12.14-4 > Package pin: 1.12.14-4 Interestingly 1.12.14-5 is *not* a sufficient downgrade, it needs to be 14-4, so it's not the new upstream 1.12.16 which has broken this support. > Simply downgrading libcairo2 [3] 'fixes' the problem. I know it's not > 'fixing' it really, but i hope that this will be enough information > so that the real fix is uploaded. > > In the mean time, if any of use use Debian , you can use that as a > workaround.. I'm assuming that this is a "proprietary-blob-linked-to-the-wrong-version" bug, but I've had google-talkplugin force an upgrade on me since this started and that didn't fix it. google-talkplugin 4.6.3.0-1 However, with libcairo2 1.12.14-4, hangouts now work. Thanks for this investigation. I'll have to see how long the rest of the desktop continues running with an old version of such a critical library The change in libcairo2 at 14-5 was: Add gl/egl support back now that wayland has been multi-archified. http://ftp-master.metadata.debian.org/changelogs/main/c/cairo/unstable_changelog The change was made to fix: http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=712022 -- Neil Williams = http://www.linux.codehelp.co.uk/ signature.asc Description: PGP signature ___ linaro-dev mailing list linaro-dev@lists.linaro.org http://lists.linaro.org/mailman/listinfo/linaro-dev
Re: Cross-Compiling Qemu for Aarch64?
On Thu, 16 May 2013 14:25:07 +0200 "Mian M. Hamayun" wrote: > but even this option doesn't work for me as the apparent lacking support for > Aarch64 in "glib-2.12". > I have used the following configure command: > ./configure --prefix=/my/prefix/path --host=aarch64-linux-gnu > --cache-file=aarch64.cache Current dpkg-cross has a cross-config cache file for Aarch64 (arm64) but you also need to specify the --build triplet or some packages won't detect the cross-compilation. The dpkg-cross config file is best handled as an environment variable: http://wiki.debian.org/CrossBuildPackagingGuidelines CONFIG_SITE=/etc/dpkg-cross/cross-config.$DEB_HOST_ARCH so CONFIG_SITE=/etc/dpkg-cross/cross-config.arm64 If you think there are other values not listed in the dpkg-cross files, please file a bug against the package in Debian, but start with the existing cache values. If those do not work, try *extending* the cache list. > with the following aarch64.cache file contents: > > glib_cv_long_long_format=I64 > glib_cv_stack_grows=no That is almost certainly too short. The cache file needs to specify all necessary values for the entire cross-build. -- Neil Williams = http://www.linux.codehelp.co.uk/ pgpJQrKBQyuQ2.pgp Description: PGP signature ___ linaro-dev mailing list linaro-dev@lists.linaro.org http://lists.linaro.org/mailman/listinfo/linaro-dev
Re: Bug#616111: Bootstrapping a foreign architecture with multiarch
On Sat, 2 Apr 2011 05:47:47 +0200 Guillem Jover wrote: Adding linaro people and debian-embedded to CC: > Hi! > > On Fri, 2011-04-01 at 22:31:07 +0100, Neil Williams wrote: > > dpkg cannot be executed inside the chroot because it has not > > necessarily been unpacked at this stage, it certainly has not been > > configured. (dpkg is running from outside the chroot). > > (An Essential package does not need to be configured to be functional.) (It's quite possibly the wrong architecture inside the chroot, so this doesn't apply.) > > Yet dpkg-query won't work with --admindir to operate from outside: I'm checking on this - it could be an artefact of how the chroot was initially created. > > Anyways: > > > > > Multi-Arch: same packages will use > > > /var/lib/dpkg/info/:. > > > > > > > For Multistrap, every file in /var/lib/dpkg/ has to be created by > > > > multistrap, based on the data obtained from dpkg -e (basically the > > > > DEBIAN/ content). This data needs only to be sufficient that dpkg can > > > > correctly configure the packages once dpkg itself is able to be > > > > executed inside the new filesystem. > > > > > > It's sufficient, you just need to know the value of the Multi-Arch field. > > > > True. I've got a change now which uses dpkg -f to check the Multi-Arch > > field of the actual downloaded .deb (independent of arch) and use that > > to detect Multi-Arch: same, then lookup Architecture in the same way, > > add that to the filename of the created maintainer script etc. > > for /path/to/chroot/var/lib/dpkg/info/* > > If you are not going to install multiple instances of “Multi-Arch: same” > packages, then just keep writting the files as you are currently > doing, the next dpkg native run on that chroot image will > automatically upgrade the db layout. Which seems to be the less > onerous way to handle this. Otherwise you'll have to duplicate the > db layout handling, including format file, etc, and at that point I'd > say you are on your own. Multistrap will need to be able to create chroots containing multiple instances of Multi-Arch: same packages because it needs to be able to build cross-building chroots with libc6-dev of the build and host architectures, as well as other common dependencies like libglib2.0-0 (needed as build arch by pkg-config and as host arch by everything else). Multistrap, like debootstrap and other similar scripts, is commonly used in two roles - to create a chroot of the build architecture for building stuff in a contained environment and creating a chroot of a foreign architecture to act as a root filesystem. As with debootstrap, I see no reason to implement two different methods to creating these chroots inside a single tool, so when creating a build arch chroot multistrap uses the same processing as when creating a foreign arch chroot. That processing has to avoid a design flaw in dpkg where the normal --unpack method would attempt to run the preinst scripts. > On Fri, 2011-04-01 at 22:43:35 +0100, Neil Williams wrote: > > Further to this, --control-path doesn't include 'list' but > > $pkg:$arch.list files are created. > > This was done on purpose, .list and .conffiles are internal files, the > correct interfaces to access the information is dpkg-query. Internal files maybe, but multistrap still needs to create them to handle packages of a foreign architecture. Even once Multi-Arch is fully deployed, this will continue because the assumption with Multi-Arch is that the foreign architecture packages are being installed alongside the build architecture. When preparing a root filesystem for a foreign architecture (e.g. an embedded device), multistrap cannot put the amd64 binaries into the chroot and then add the armel, it needs to be able to unpack armel binaries into an armel chroot without running the preinst scripts or using the armel dpkg binary inside the chroot. The preinst scripts can't run on the build arch (amd64) because that would expose the chroot to the external configuration. i.e. it needs to be possible to create an armel chroot of Ubuntu natty from an i386 machine running Debian Wheezy and similarly from Ubuntu natty i386 to create a Debian Wheezy armel chroot. Currently, dpkg --unpack cannot cope with this and the existing Multi-Arch changes in dpkg do not address this problem. This has been covered before in this bug and elsewhere without resolution, but please reconsider how dpkg - even in a Multi-Arch world - needs to support unpacking foreign architecture binaries into a foreign architecture chroot and that this mandates that dpkg *must not* attempt to execute the preinst scripts inside the foreign chroot. It can *only* do so if t
Re: [PATCH] dpkg-cross: handle multi-arch packages too
On Wed, 23 Mar 2011 15:37:34 +0100 Marcin Juszkiewicz wrote: > Ubuntu 'natty' switched to multiarch recently and my cross toolchain > packages started to fail to build. Most of problems were fixed but then > dpkg-cross hit us. > > Today I looked at dpkg-cross and made attached patch to handle problem. > What patch does is converting multiarch packages into old style ones. Are you sure you want to convert Multi-Arch paths to the old paths? The idea is to get the packages concerned to use Multi-Arch paths which the toolchain should just find without needing dpkg-cross at all. The toolchain should be looking in the Multi-Arch paths first, then the old cross paths. The expected role for dpkg-cross once Multi-Arch started to be deployed was only in putting non-Multi-Arch packages into old paths, letting the toolchain find the other packages in their normal Multi-Arch paths. > There is no extra switch like "--force-even-when-multiarch" and > "--convert-anyway -A" is used instead. This is change against latest CVS > code (dpkg-cross r1.83). > > >From my tests it looks like it does proper job. It sounds like the wrong job to me. Why are the libraries IN the correct Multi-Arch paths not being found by a Multi-Arch aware toolchain? You need a Multi-Arch-Cross toolchain to work with Multi-Arch and not changes to dpkg-cross. Old toolchains need to be upgraded. If correctly Multi-Arch'd packages are not being found by a Multi-Arch aware toolchain, that's not something dpkg-cross should be expected to solve. -- Neil Williams = http://www.linux.codehelp.co.uk/ pgpwNYKIvgMPW.pgp Description: PGP signature ___ linaro-dev mailing list linaro-dev@lists.linaro.org http://lists.linaro.org/mailman/listinfo/linaro-dev
Re: [PATCH] Dpkg/Shlibs.pm: multiarch search paths
On Mon, 21 Mar 2011 14:03:37 +0100 Matthias Klose wrote: > On 21.03.2011 11:25, Neil Williams wrote: > > The people who are most likely to be doing this now are Linaro. > > Emdebian Crush stalled after Lenny (and used ARM not armel), so the > > version of gcc-4.2 in Lenny should still build with the typical > > "dpkg-buildpackage -aarm". Later versions of gcc probably have other > > fixes which would cause the reverse cross build to fail, not due to the > > failure of the original patches but due to changes in gcc itself. > > the gcc-4.4 packaging is able to build a reverse cross. I didn't check > gcc-4.5 > and gcc-4.6. > > Matthias Oops, sorry Matthias - that was meant to be "later versions of gcc *might* have other (internal) changes which could cause..." I didn't mean to infer that I'd tried and failed a reverse cross with newer versions, I haven't had time to do that test. Thanks for the info on gcc-4.4. -- Neil Williams = http://www.linux.codehelp.co.uk/ pgp6oWHSZZcBx.pgp Description: PGP signature ___ linaro-dev mailing list linaro-dev@lists.linaro.org http://lists.linaro.org/mailman/listinfo/linaro-dev
Re: [PATCH] Dpkg/Shlibs.pm: multiarch search paths
On Mon, 21 Mar 2011 09:55:26 +0100 Raphael Hertzog wrote: (Trimming the CC: feel free to drop patches@l.o if linaro-dev@l.l.o is more useful. Please keep debian-embedded in CC to reach me.) > Not knowing much about cross-compilation I had started with a sligthly > different approach than you, mimicking exactly what was done for > cross-compilation support. 09:48 < codehelp> buxy: vorlon: my original query related to not so much building a cross-compiler but building gcc-* to install and run on cross, i.e. reverse-cross where build=amd64,host=armel,target=armel 09:48 < codehelp> cross-compiler build=amd64,host=amd64,target=armel 09:48 < codehelp> doesn't give you a libgcc1 which can be used to satisfy dependencies of packages cross-built using the cross-compiler Cross-building an entire distro, like Emdebian Crush, involved a long winded process: 0: Build gcc-4.* as an arm cross-compiler (not armel in those days but that's not important to the method) 1: Build build-dependencies of gcc using the cross-compiler 2: Build gcc-4.* as a normal package using the cross-compiler The people who are most likely to be doing this now are Linaro. Emdebian Crush stalled after Lenny (and used ARM not armel), so the version of gcc-4.2 in Lenny should still build with the typical "dpkg-buildpackage -aarm". Later versions of gcc probably have other fixes which would cause the reverse cross build to fail, not due to the failure of the original patches but due to changes in gcc itself. To create a fully bootstrap-able system, gcc does need to support the reverse cross and this will need testing in Multi-arch world. Typically, gcc-4.* *should* be expected to build correctly just using: $ dpkg-buildpackage -a$arch No special target specification (except internally in debian/rules etc.), no expectation of building a cross-compiler, just building it as if it is any other package which needs to be cross-built. Of course, if libgcc1 can be made available in any other way than having to pretend that we actually want a native compiler on these systems (99% of the time we really don't), then everyone wins. > Neil (or anyone else knowledgable on the topic), can you review the > patches and tell us what's really needed and most logical according to > you? > > Steve's patch: > - adds the multi-arch path at the start of the search paths > - adds them only for stantard cross-compilation (host != build) but not > for the special cases used to detect a cross-compiler build Intuitively this does sound the correct way to do it, especially as what I wanted from the original patch was a standard cross-build, not the special case of building a cross-compiler, but: > My patch: > - adds the multi-arch path before the cross-arch patch but after the > standard search path > > - add them in all cases where the cross-compilation paths are used > > Putting the multi-arch paths after the default path should not matter as > dpkg-shlibdeps skips libraries of non-matching architecture. This is more in line with how it actually works out when trying to cross-build gcc itself. I'm not sure of the benefit of putting the multiarch lib location in front of the default lib location. I'm not sure if having some i386 libraries in a multi-arch path and the main system amd64 libraries in a non-multi-arch path would actually matter (after all, the same libraries *should* get updated together). That's a "traditional" Multi-Arch question, so I'd defer to Steve on that one. For cross-building, it's simply sufficient that the new multi-arch-cross paths precede the old dpkg-cross paths and both patches achieve this. > For the case of building a cross-compiler, Steve's experience tends to > prove that adding the multi-arch path is not needed... but I don't see why > it should be treated differently than the cross-arch paths. > > Can you enlighten us? For building a cross-compiler, Steve is correct - it doesn't matter. For using that cross-compiler to cross-compile gcc itself, it will matter. Multi-Arch paths will be necessary to build a version of libgcc1 which can be installed on the target. -- Neil Williams = http://www.linux.codehelp.co.uk/ pgpHMLHie4HvS.pgp Description: PGP signature ___ linaro-dev mailing list linaro-dev@lists.linaro.org http://lists.linaro.org/mailman/listinfo/linaro-dev
Re: tcl8.5 breaks dpkg-cross assumptions and multiarch
On Thu, 3 Feb 2011 15:39:49 + Wookey wrote: > dpkg-cross in fact only picks files out of packages by positive > identification as libraries or headers. It misses out generic 'other > stuff' in ((/usr)/lib, which generally works pretty well, but in > this tcl case it's not suffient for tcl-depending apps to cross-build. > > Bug http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=599206 discusses > this issue. ... and http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=611736 is probably going to be the root bug (renamed) to handle the clarification of exactly what files we keep and how we rationalise the legacy regular expressions in dpkg-cross. Please add further specific cases to that bug. > dpkg-cross is intended to put files needed for cross-building into > -cross packages and it's currently missing this one out. Unfortunately > because it doesn't match any of the 'standard paterns for cross-useful > files' this can only be fixed by adding a fairly specific regexp, > which is worrying close to special-casing or deciding that in fact > just fishing out specific things from /usr/lib is too conservative and we > should take the view that everything in /usr/lib is potentially useful > in cross-building and should be copied into -cross packages. Generally, from a cursory view of what is left out, the situation we have is this: 0: dpkg-cross supports autotools cross-building through a series of very detailed, very complex and potentially fragile regular expressions but, importantly, ALSO uses a series of very detailed, very complex and potentially equally fragile regular expressions on the CONTENTS of those files, some of which Debian is quietly trying to drop from packages because of other problems. (e.g .la files) 1: dpkg-cross has explicit support for pkg-config which appears to work perfectly well, principally because pkg-config is inherently less complex than the entirety of autoconf|automake|libtool|dpkg-shlibdeps etc. 2: dpkg-cross has support for CMake - although only as a direct result of 0: and 1: 3: dpkg-cross has no idea how to help SCons or any number of other build systems out there, but then it mostly doesn't have to because many of those simply don't compile stuff, (they create Arch:all packages) and the ones that do compile stuff aren't used by sufficient numbers of cross-building people for sufficient complaints to be seen for dpkg-cross to have had the support created. 4: Nobody gave a damn about Tcl until sqlite added bindings. 5: Nobody adds support to dpkg-cross until someone complains loudly enough *and* comes up with sane patches. 6: dpkg-cross has huge amounts of legacy code which someone added at somepoint because it was important but nobody has actually had the guts to unilaterally remove because we can't tell if the original package has since been fixed. (s/fixed/broken in a different way/). > In multiarch world everything in (/usr)/lib is going to end up in > /usr/lib/ or /lib/, unless packages are > re-arranged to put them elsewhere, and we expect this to work > fine so perhaps dpkg-cross should start doing the same thing, and thus > discuver any problems this does potentially create. Would > that actually do any harm? What files which are currently missed out > of -cross packages would actually cause breakage if copied over into > /usr//lib? Let's not break dpkg-cross fundamentally for everyone though. We can choose to make a different dpkg-cross which is FAR simpler (because it blindly moves files without any kind of safeguard) but as this does not involve fixing the contents of certain files, numerous autotools packages will break. So, if people want a broken dpkg-cross for testing, let's have a dpkg-multi-cross which breaks their cross-building world (using a different Provides: and conflicting with the "standard" cross packages) and leave the existing world alone. That version of dpkg-multi-cross would be trivial to write - unpack the .deb, unconditionally move all files in ./usr/lib to ./usr/$triplet, handle /usr/include and remove everything else. Repack the .deb as -cross. Break world. > I just tried a modified dpkg-cross on a pile of packages and whilst > many come out the same, you do get quite a lot more files in some > packages and some packages that were previously null now have stuff in > them. e.g apache-modules. So there is quite a lot of bloat, but > probably no breakage. Retaining the changes to the contents of the files whilst simply adding lots more *stuff* to -cross packages is the least harmful option. However, because the contents haven't changed, it doesn't actually help us identify the issues that would arise with Multiarch much. -- Neil Williams = http://www.data-freedom.org/ http://www.linux.codehelp.co.uk/ http://e-mail.is-not-s.ms/ pgpWzNT
Re: tcl8.5 breaks dpkg-cross assumptions and multiarch
On Tue, 1 Feb 2011 12:50:48 +0100 Loïc Minier wrote: > On Tue, Feb 01, 2011, Wookey wrote: > > But if something is looking for arch-independent stuff in /lib then in > > general that's wrong, and I'm not aware of any examples of > > correctly-packaged packages that need this. Any arch-independent files > > will be supplied by an arch all package that the build should depend > > on if needed. > > I might be getting your point wrong, but I certainly see a lot of files > in /lib itself which are arch-independent data used for early boot > (before /usr is available); PNG files and text files which would be > identical on all architectures. /lib vs /usr/lib is a different argument. I'm working on the basis that Wookey was meaning /usr/lib/ compared to /usr/share. The point about dpkg-cross is that it doesn't blindly take everything from /usr/lib and propagate that into /usr/$triplet/lib, it picks out stuff that it knows is useful to a cross package. Symlinks are included because there's no need to copy libfoo.so.0.0.1 as libfoo.so.0 etc. There's a complex list of regular expressions, allowing header files, static linking files, object files, .la files, anything in /usr/include/ and stuff in /usr/lib/pkg-config/ and stuff already under /usr//lib/. This list has aggregated over time. As multiarch is as far away as ever, I will discuss pruning that list significantly once Squeeze is released, leading to a dpkg-cross 3.0.0. The final list will only include stuff which dpkg-cross can reliably identify *and* which is absolutely essential to cross-builds. /usr/share won't be included, except for pkg-config files. -- Neil Williams = http://www.data-freedom.org/ http://www.linux.codehelp.co.uk/ http://e-mail.is-not-s.ms/ pgpBZmMGGvFoE.pgp Description: PGP signature ___ linaro-dev mailing list linaro-dev@lists.linaro.org http://lists.linaro.org/mailman/listinfo/linaro-dev
Re: tcl8.5 breaks dpkg-cross assumptions and multiarch
-BEGIN PGP SIGNED MESSAGE- Hash: SHA256 On 31/01/11 20:18, Steve Langasek wrote: > On Mon, Jan 31, 2011 at 05:16:15PM +, Wookey wrote: >> So, the questions is - which of these is the correct fix: >> 1) make dpkg-cross copy over symlinks even when they point into >> /usr/share >> 2) make tcl8.5 put tclConfig.sh in /usr/lib/tcl8.5 instead of >> /usr/share/tcltk/tcl8.5 >> 3) make sqlite (and similar packages) look in /usr/share/tcltk/tcl8.5 >> instead of /usr/lib/tcl8.5 when building > > 1+2: dpkg-cross should also copy over symlinks that point to /usr/share. The reason not to copy is that the target of the symlink does not exist in the -cross package and the one that exists in the native package in /usr/share contains all the wrong paths. A dangling symlink would be left if the -cross package is not installed. If the symlink is to be copied over not into /usr/share but into /usr/arm-linux-gnueabi/share then the native version of the file will still exist in /usr/share and none of the existing tools will look into /usr/arm-linux-gnueabi/share to find the correct version without a new wrapper. Nobody wins. dpkg-cross cannot reliably identify which symlinks should be preserved and how. dpkg-cross cannot copy across all symlinks which point to /usr/share because a lot of those are unwanted. The correct fix is 2. The only possible thing dpkg-cross could do is reverse the symlink and move the file from /usr/share into /usr/arm-linux-gnueabi/lib/ if a symlink to /usr/lib/ exists and not create the symlink in /usr/arm-linux-gnueabi/share at all - at which point the native version of the package still contains the symlink from /usr/share to /usr/lib/ with the native paths. Is that what is actually being requested for 1.? I don't see that it's reliable - it's masking the real bug with a horrible hack, again. How does this help when the package itself is buggy by not complying with the FHS and the cross-build looks in the wrong place? > When such symlinks exist, it's almost invariably because *something is > looking for the information in that location*. When is this not actually a bug in that package? The maintainer has agreed to fix the actual bug by implementing option 2 above, which we all seem to agree is the appropriate and optimal fix for the original issue. Under what circumstances would some other package fall into this problem *without* also being buggy in exactly the same way as tcl8.5? Yes, let's clarify policy but I don't think it's correct to expect dpkg-cross to handle packages which are simply buggy whilst risking breaking other packages which do things properly. dpkg-cross doesn't (and shouldn't) support package-specific exceptions in how -cross packages are created. > So as a general policy, they > should also be made available in a crossed package. The /usr/share/ location *must* change in the cross package so that it can be installed alongside the native, at which point any merit of retaining /usr/share is lost in the cross package because a wrapper script will still be needed to find the modified location, handle the dangling symlink or risk getting the wrong data (because the data in this case is clearly wrong for the cross build). Either the file is in the wrong place (as in this case) or the change of location is simply going to break without a package-specific wrapper script anyway because the locations of things in /usr/share should not (need to) change to support a cross-build. - -- Neil Williams = http://www.linux.codehelp.co.uk/ -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.10 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iQIcBAEBCAAGBQJNR0GdAAoJEPFn5DyBQ7aC2lcP/1K8gLDnYffSZSb9lVarDh89 Rmvk1FLIHzJQ29kWYLE29ybpLCma7ms4T+PloVv7KLWMvS36jkLiCuM+a/sP8eU3 3uPkZ5rFTflXy8q2H4MhF/CQa94QExxthpDHgeQntz9j+5dKp+Nj3uNQaHjdh6XC aY+tyMulpwaR/B0gpQXtgAKffqvKtInLQDXnNd4YWEXbRgMndm9d37GyzYdVdOsz /udBf57SpQXuLX2M4d1cX1QmbUpwMqD3P7PeZQj7gcwX0loupm6MNFb7VQJM6gKp d6o/AYBBg2X3addJMDVQZdNsNYfFm0F6JfTlk0pgNyFUOGPFmyNFxOT3QjufloY9 ZH+xmvgC9UL6gSv5voQUIezn2MSLGblP648RwW8RgdHslNG60v6I22VcZSQDkcw4 BE2CUn/Y1w+q1A74MGSAPV7lB2b9QRnxjec1xfyo11as27mudlDKwFtrGI6mzDGk oZf/SYGkCQ1cOscq1aVJGkJocfA4rlwbM67zfFybz5W9j8UZSYOVX9F1oh/OpRPF UQNaWyiOF4lf/EUf3Ct7F6Dwz9TPKhxpRgwg3kfQqTlsgjLK/a1zIf4F1h1EjP4H f0sAhgJrYQkFUW8w1LDhK92EEDZdHW7De0cA/7tx4h/8ay+NdrcpEOMWTFDoGrcy QC4k9mp+Xwnaq78yHWaz =clO0 -END PGP SIGNATURE- ___ linaro-dev mailing list linaro-dev@lists.linaro.org http://lists.linaro.org/mailman/listinfo/linaro-dev
