Re: Packaging a library that requires cross-compiled code
On Tue, 2 Oct 2007 07:01:29 -0400, Clint Adams [EMAIL PROTECTED] said: On Mon, Oct 01, 2007 at 08:09:59PM -0500, Peter Samuelson wrote: Until Debian as a whole can agree that it is important to be self-hosting, and to be confident enough in our autotools maintainers to trust their packages for automated builds, I don't think you should ask David to build a whole cross compiler. After all, that's not only a _lot_ more complex to maintain, but more build-resources-expensive. When normal practice for upstreams is to not ship any auto-generated files, and normal practice for users wishing to compile their own programs is to have autotools installed, then I might agree with you. Until then, I think that treating .ac files as if they're .c files is a canard. I don't think canard means what you think it mean: all my packages treat .ac files exactly like they treat .c files; and this factual statement blows “canard” out of the water. I have not had any issues with regenerating all autoconf files during ./debian/rules configure phase. Do people have examples of packages where this currently fails? manoj -- There is no distinctly American criminal class except Congress. Mark Twain Manoj Srivastava [EMAIL PROTECTED] http://www.debian.org/~srivasta/ 1024D/BF24424C print 4966 F272 D093 B493 410B 924B 21BA DABB BF24 424C
Re: Packaging a library that requires cross-compiled code
On Wed, Oct 03, 2007 at 01:13:34AM -0500, Manoj Srivastava wrote: I don't think canard means what you think it mean: all my packages treat .ac files exactly like they treat .c files; and this factual statement blows “canard” out of the water. Really, so your package runs ./configure and $(MAKE) and then runs .ac files through gcc to produce .o files? I have not had any issues with regenerating all autoconf files during ./debian/rules configure phase. Do people have examples of packages where this currently fails? It should fail for fakeroot, as that uses libtool features not present in unstable. -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
Re: Packaging a library that requires cross-compiled code
[Eric Dorland] s/autoconf/automake/g The same is true of automake. Good thing that isn't a problem, since you don't Build-Depends: automake, but rather Build-Depends: automake1.7 or whatever. Or did you mean that new uploads of automake1.7 break packages that worked with older uploads of automake1.7? Frankly I'd consider that a failing in your duties as maintainer. -- Peter Samuelson | org-tld!p12n!peter | http://p12n.org/ signature.asc Description: Digital signature
Re: Packaging a library that requires cross-compiled code
Felipe Sateler wrote: That defeats the purpose of autoconf, and makes much of automake's functionality redundant. If you are going to require automake, autoconf and libtool installed, why then generate the intermediate steps (configure and Makefile's)? Plus it defeats another goal of autotools: only developers should have it installed, not end users. If you go down that road, you'll end up with something like scons. .. or debhelper -- see shy jo signature.asc Description: Digital signature
Re: Packaging a library that requires cross-compiled code
* Peter Samuelson ([EMAIL PROTECTED]) wrote: [Eric Dorland] s/autoconf/automake/g The same is true of automake. Good thing that isn't a problem, since you don't Build-Depends: automake, but rather Build-Depends: automake1.7 or whatever. But this creates a problem when automake1.7 is three major revisions out of date and should be removed from the archive. Right now every time I do this I have to file hundreds of bugs and do lots of NMUs. I'm not eager to turn this into thousands. So it is already a problem. I agree with your goal in principle, but the autotools just aren't designed to be used that way, and we'll be fighting with them more than actually deriving benefit from doing this. Or did you mean that new uploads of automake1.7 break packages that worked with older uploads of automake1.7? Frankly I'd consider that a failing in your duties as maintainer. -- Eric Dorland [EMAIL PROTECTED] ICQ: #61138586, Jabber: [EMAIL PROTECTED] 1024D/16D970C6 097C 4861 9934 27A0 8E1C 2B0A 61E9 8ECF 16D9 70C6 signature.asc Description: Digital signature
Re: Packaging a library that requires cross-compiled code
Joey Hess wrote: Felipe Sateler wrote: That defeats the purpose of autoconf, and makes much of automake's functionality redundant. If you are going to require automake, autoconf and libtool installed, why then generate the intermediate steps (configure and Makefile's)? Plus it defeats another goal of autotools: only developers should have it installed, not end users. If you go down that road, you'll end up with something like scons. .. or debhelper I don't really see why. -- Felipe Sateler -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
Re: Packaging a library that requires cross-compiled code
This is exactly the point I've been trying to make for a long time, about things like autoconf and automake1.x, and why you should build-depend on them and run them every time. Because it proves that we are fully self-hosting, and the main reason _not_ to do it is the fear that we might _not_ actually be self-hosting. Which is something I believe we've promised our users, implicitly if not explicitly. Clearly, many people in Debian disagree that being self-hosting is sufficiently worthwhile, but would rather work around hypothetical autotools incompatibilities by, essentially, running them as infrequently as they can get away with, and hoping they catch whatever problems show up, before uploading. (And with a tool like autoconf, whose very _essence_ requires that it behave differently on different systems, this seems unlikely.) Until Debian as a whole can agree that it is important to be self-hosting, and to be confident enough in our autotools maintainers to trust their packages for automated builds, I don't think you should ask David to build a whole cross compiler. After all, that's not only a _lot_ more complex to maintain, but more build-resources-expensive. +1 Ondrej -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
Re: Packaging a library that requires cross-compiled code
On Mon, Oct 01, 2007 at 08:09:59PM -0500, Peter Samuelson wrote: Until Debian as a whole can agree that it is important to be self-hosting, and to be confident enough in our autotools maintainers to trust their packages for automated builds, I don't think you should ask David to build a whole cross compiler. After all, that's not only a _lot_ more complex to maintain, but more build-resources-expensive. When normal practice for upstreams is to not ship any auto-generated files, and normal practice for users wishing to compile their own programs is to have autotools installed, then I might agree with you. Until then, I think that treating .ac files as if they're .c files is a canard. -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
Re: Packaging a library that requires cross-compiled code
[Josselin Mouette] Given that especially autoconf introduces serious incompatibilities between minor releases, this is simply not feasible because it would trigger hundreds of FTBFS errors each time a new autoconf version is uploaded. So it sounds like you're saying that, although we ship source code to our users, we _assume_ that the source doesn't actually build with our tools, so we have to also ship precompiled code to work around this. To me, that's a poor attitude for a distribution that focuses on providing users with access to source. What use is source code if it can't be hacked on and rebuilt? I think we should have a little bit of faith in our autoconf maintainer not to introduce incompatible versions without changing the name. This was already done for autoconf2.13 - autoconf2.50 (the latter since renamed to autoconf). If he uploads a version of autoconf that breaks other builds, the appropriate response is to file a bug. That is what we do for gcc. (Sometimes the response is too bad, gcc is changing to be more strict, you'll have to fix your package, and yet, the world somehow fails to end.) Honestly though, I haven't seen autoconf break my packages for a very long time - I can't even remember last time it happened. (It might have been autoconf2.13 - autoconf2.50.) Were you thinking of automake, perhaps? automake changes much more frequently, which is why we have so many versions in the archive, so you can use a version that won't break your builds. -- Peter Samuelson | org-tld!p12n!peter | http://p12n.org/ signature.asc Description: Digital signature
Re: Packaging a library that requires cross-compiled code
Scribit Josselin Mouette dies 02/10/2007 hora 10:06: Because it proves that we are fully self-hosting, and the main reason _not_ to do it is the fear that we might _not_ actually be self-hosting. Which is something I believe we've promised our users, implicitly if not explicitly. Given that especially autoconf introduces serious incompatibilities between minor releases, this is simply not feasible because it would trigger hundreds of FTBFS errors each time a new autoconf version is uploaded. Shouldn't those bugs just be corrected, and would a similar number of bugs still be triggered with the next version? Curiously, Pierre -- [EMAIL PROTECTED] OpenPGP 0xD9D50D8A signature.asc Description: Digital signature
Re: Packaging a library that requires cross-compiled code
Neil Williams wrote: On Tue, 2 Oct 2007 07:01:29 -0400 Clint Adams [EMAIL PROTECTED] wrote: On Mon, Oct 01, 2007 at 08:09:59PM -0500, Peter Samuelson wrote: Until Debian as a whole can agree that it is important to be self-hosting, and to be confident enough in our autotools maintainers to trust their packages for automated builds, I don't think you should ask David to build a whole cross compiler. After all, that's not only a _lot_ more complex to maintain, but more build-resources-expensive. When normal practice for upstreams is to not ship any auto-generated files, and normal practice for users wishing to compile their own programs is to have autotools installed, then I might agree with you. That's down to the gnu automake developers - many upstream teams use 'make dist' (sadly some don't use 'make distcheck') and leave it at that. If normal practise is to change, it will have to change within automake first. That defeats the purpose of autoconf, and makes much of automake's functionality redundant. If you are going to require automake, autoconf and libtool installed, why then generate the intermediate steps (configure and Makefile's)? Plus it defeats another goal of autotools: only developers should have it installed, not end users. If you go down that road, you'll end up with something like scons. -- Felipe Sateler -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
Re: Packaging a library that requires cross-compiled code
* Josselin Mouette ([EMAIL PROTECTED]) wrote: Le lundi 01 octobre 2007 à 20:09 -0500, Peter Samuelson a écrit : This is exactly the point I've been trying to make for a long time, about things like autoconf and automake1.x, and why you should build-depend on them and run them every time. Because it proves that we are fully self-hosting, and the main reason _not_ to do it is the fear that we might _not_ actually be self-hosting. Which is something I believe we've promised our users, implicitly if not explicitly. Given that especially autoconf introduces serious incompatibilities between minor releases, this is simply not feasible because it would trigger hundreds of FTBFS errors each time a new autoconf version is uploaded. s/autoconf/automake/g The same is true of automake. -- Eric Dorland [EMAIL PROTECTED] ICQ: #61138586, Jabber: [EMAIL PROTECTED] 1024D/16D970C6 097C 4861 9934 27A0 8E1C 2B0A 61E9 8ECF 16D9 70C6 signature.asc Description: Digital signature
Re: Packaging a library that requires cross-compiled code
Simon Richter writes (Re: Packaging a library that requires cross-compiled code): David Anderson wrote: But, if there is precedent, it might not be too painful to mimick existing cross-compiler packages to build my own. I'll see what I can do. Please coordinate any such effort with [EMAIL PROTECTED] :-) I'm working on infrastructure to aid generation of cross-compiler packages (a templating mechanism that generates the necessary Package stanzas in debian/control from another file that uses a descriptive language; debian-xcontrol would be the package you are looking for, however that bit is not yet done as I concentrated on features that will allow cross-compilation of other packages first). That sounds quite freaky. I think it would be wise of David Anderson to try to avoid putting your exciting innovations into his own project plan :-). Ian. -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
Re: Packaging a library that requires cross-compiled code
David Anderson writes (Re: Packaging a library that requires cross-compiled code): This provides a way to build the entire package, using only debian-provided packages, with a 'caching' functionality to avoid having to rebuild a toolchain if a previous version of the package is installed. I saw you were having some trouble with that. I'm afraid I can't offer much beyond sympathy. It's unfortunate, but you're just tackling a difficult problem. As an alternative to the difficulties you have right now, one way perhaps would be to ship the source package containing both the source (including cross-compiler) so that it builds the binary fragment, but also contains it, and provide an easy way to allow the user to reuse the provided one without building the compiler. One case that I feel is missing from this setup is the trivial just use the blob provided by upstream. Given that we now have the entire machinery to rebuild the blob if someone really really wants to (as an aside, I'd like to repeat that the only thing I can see anyone doing with it is either breaking the flashing process, or damaging the brick by messing with the flash write timing), would it be acceptable to default to using the upstream blob, unless an explicit force flag is passed to debian/rules ? A better approach would be to compare the built binary blob and die if it's not identical to the upstream one. Obviously there would be some kind of override - easiest perhaps just to remove the existing binary blob. That means that any accidental breakage to the building machinery will be spotted and fixed, but users are still able to change it. You'd want to leave a comment in the source file warning someone who edits it that they have to hit the check over the head as well. Ian. -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
Re: Packaging a library that requires cross-compiled code
On 10/1/07, Ian Jackson [EMAIL PROTECTED] wrote: David Anderson writes (Re: Packaging a library that requires cross-compiled code): This provides a way to build the entire package, using only debian-provided packages, with a 'caching' functionality to avoid having to rebuild a toolchain if a previous version of the package is installed. I saw you were having some trouble with that. I'm afraid I can't offer much beyond sympathy. It's unfortunate, but you're just tackling a difficult problem. No worries, the folks on #debian-python have helped me figure out something. A better approach would be to compare the built binary blob and die if it's not identical to the upstream one. Obviously there would be some kind of override - easiest perhaps just to remove the existing binary blob. That means that any accidental breakage to the building machinery will be spotted and fixed, but users are still able to change it. You'd want to leave a comment in the source file warning someone who edits it that they have to hit the check over the head as well. I'm not sure what you mean by detecting accidental breakage to the build machinery, but that means building a full cross-compiler each time the package is rebuilt. Currently, we have a set of rules that use the upstream blob by default, but enable a full rebuild (including cross-compiler) when a DEB_BUILD_OPTIONS flag is passed. My objective here is simply to avoid building gcc for a 50k python package. My only interest is taking unnecessary load off all the stuff and machines that will be building my package :-). It seems to me that this solution is good, since it does enable users to manually rebuild everything if they really want to, but makes package construction in the general case nice and fast, as a platform independent python package should be. - Dave -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
Re: Packaging a library that requires cross-compiled code
David Anderson writes (Re: Packaging a library that requires cross-compiled code): I'm not sure what you mean by detecting accidental breakage to the build machinery, but that means building a full cross-compiler each time the package is rebuilt. Currently, we have a set of rules that use the upstream blob by default, but enable a full rebuild (including cross-compiler) when a DEB_BUILD_OPTIONS flag is passed. What I mean is that if you do what you describe above, you won't notice if something causes the DEB_BUILD_OPTIONS flag not to work any more. That is, your package could acquire a FTBFS bug - as in, it doesn't really recompile from its actual source, and no-one would know. Let me put it another way: people who deal specifically with your particular package are able to set a special environment variable to disable the full GCC build. People who are treating your package as just some Debian package may or may not actually need the full build, but I don't think it's safe to assume that they don't. There are quite a few automated systems that rebuild Debian packages - the buildds, my autopkgtest system, various other testers, and they are all situations where the work done to build the cross-compiler is not wasted: we repeat that processing precisely so that we know that it actually works. So I think you should make the full build the default. And if you worry that the full build might generate an incorrect binary lump, you should have the package compare the results of the full build with the prebuilt binary lump and die if they're not the same. Ian. -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
Re: Packaging a library that requires cross-compiled code
[Ian Jackson] There are quite a few automated systems that rebuild Debian packages - the buildds, my autopkgtest system, various other testers, and they are all situations where the work done to build the cross-compiler is not wasted: we repeat that processing precisely so that we know that it actually works. This is exactly the point I've been trying to make for a long time, about things like autoconf and automake1.x, and why you should build-depend on them and run them every time. Because it proves that we are fully self-hosting, and the main reason _not_ to do it is the fear that we might _not_ actually be self-hosting. Which is something I believe we've promised our users, implicitly if not explicitly. Clearly, many people in Debian disagree that being self-hosting is sufficiently worthwhile, but would rather work around hypothetical autotools incompatibilities by, essentially, running them as infrequently as they can get away with, and hoping they catch whatever problems show up, before uploading. (And with a tool like autoconf, whose very _essence_ requires that it behave differently on different systems, this seems unlikely.) Until Debian as a whole can agree that it is important to be self-hosting, and to be confident enough in our autotools maintainers to trust their packages for automated builds, I don't think you should ask David to build a whole cross compiler. After all, that's not only a _lot_ more complex to maintain, but more build-resources-expensive. -- Peter Samuelson | org-tld!p12n!peter | http://p12n.org/ signature.asc Description: Digital signature
Re: Packaging a library that requires cross-compiled code
Hi, On Sunday 30 September 2007 00:25, Bernd Zeimetz wrote: What about the following plan: It's not only pointless, but will also result in FTBFS errors once the old package is not available anymore. regards, Holger pgpZzFKDNDyYQ.pgp Description: PGP signature
Re: Packaging a library that requires cross-compiled code
Holger Levsen wrote: Hi, On Sunday 30 September 2007 00:25, Bernd Zeimetz wrote: What about the following plan: It's not only pointless, but will also result in FTBFS errors once the old package is not available anymore. not if you build-depend on oldpackge | gcc-source and just build the firmware again if the old package is not available. Building the firmware is pretty easy and takes 5 minutes on my machine. But you're right, thre's no need to rebuild Davids provided firmware blob, especially since it will not run on a user's computer. debian/rules will allow to rebuild it for the paranoids, though. -- Bernd Zeimetz [EMAIL PROTECTED] http://bzed.de/ -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
Re: Packaging a library that requires cross-compiled code
Just a quick summary of the current status of the package, following these discussions and work on #debian-python. The package python-pynxt now build-depends on python-pynxt || (gcc-4.2-source binutils-source). If python-pynxt is available, the flash_driver.bin blob is obtained by copying it out of the installed python-pynxt package. If it is not available, then a cross-compiling binutils and gcc (C compiler only) toolchain is built, which in turn rebuilds flash_driver.bin. This provides a way to build the entire package, using only debian-provided packages, with a 'caching' functionality to avoid having to rebuild a toolchain if a previous version of the package is installed. One case that I feel is missing from this setup is the trivial just use the blob provided by upstream. Given that we now have the entire machinery to rebuild the blob if someone really really wants to (as an aside, I'd like to repeat that the only thing I can see anyone doing with it is either breaking the flashing process, or damaging the brick by messing with the flash write timing), would it be acceptable to default to using the upstream blob, unless an explicit force flag is passed to debian/rules ? - Dave On 9/30/07, Bernd Zeimetz [EMAIL PROTECTED] wrote: Holger Levsen wrote: Hi, On Sunday 30 September 2007 00:25, Bernd Zeimetz wrote: What about the following plan: It's not only pointless, but will also result in FTBFS errors once the old package is not available anymore. not if you build-depend on oldpackge | gcc-source and just build the firmware again if the old package is not available. Building the firmware is pretty easy and takes 5 minutes on my machine. But you're right, thre's no need to rebuild Davids provided firmware blob, especially since it will not run on a user's computer. debian/rules will allow to rebuild it for the paranoids, though. -- Bernd Zeimetz [EMAIL PROTECTED] http://bzed.de/ -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED] -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
Re: Packaging a library that requires cross-compiled code
not if you build-depend on oldpackge | gcc-source and just build the firmware again if the old package is not available. Well, I just learned that this is broken. #403246, which is obviously not planned to be fixed soon. -- Bernd Zeimetz [EMAIL PROTECTED] http://bzed.de/ -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
Re: Packaging a library that requires cross-compiled code
On Sep 28, Ian Jackson [EMAIL PROTECTED] wrote: 1) Ship a built copy of the code in the package's .diff.gz, and DTRT at package creation time to move the .bin from debian/ to the right place in the staging tree. The source code for the .bin is in .orig.tar.gz, under a free license. Anyone is free to modify or [...] This is (unfortunately for you) not acceptable for Debian proper. Everything in Debian must be buildable from source using tools in Debian. No, you are wrong. This is the right way, and I can think about at least one precedent (yaboot). It must be possible to fully compile from source the packages in main by only using other packages in main, but there is no requirement that this is done every time the binary debian package is built[1]. Another example is documentation which is distributed in the source package in both source and compiled form and is not rebuilt because while it would be possible this would add more build dependencies, waste buildds time and would not be needed anyway. In this case the firmware image distributed as a blob in the source package can be built using the gcc-4.3-source package to create a crosscompiler (there is no requirement to have a single-step procedure). [1] In practice the release managers and/or the debian security team nowadays require this to easily allow fixing security bugs in released packages, but I expect that they will have no objections to this specific case. -- ciao, Marco signature.asc Description: Digital signature
Re: Packaging a library that requires cross-compiled code
Hi, In this case the firmware image distributed as a blob in the source package can be built using the gcc-4.3-source package to create a crosscompiler (there is no requirement to have a single-step procedure). What about the following plan: - the first version of the package will build-dep. on the gcc source, build the compiler and build the firmware. - all new versions of the package will build-depend on the old package and just copy the binary blob from it. Any objections to this? Cheers, Bernd -- Bernd Zeimetz [EMAIL PROTECTED] http://bzed.de/ -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
Re: Packaging a library that requires cross-compiled code
On Sun, Sep 30, 2007 at 12:25:51AM +0200, Bernd Zeimetz wrote: What about the following plan: - the first version of the package will build-dep. on the gcc source, build the compiler and build the firmware. - all new versions of the package will build-depend on the old package and just copy the binary blob from it. Any objections to this? That's pointless. Why bother? -- Daniel Jacobowitz CodeSourcery -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
Re: Packaging a library that requires cross-compiled code
On Sep 30, Bernd Zeimetz [EMAIL PROTECTED] wrote: - the first version of the package will build-dep. on the gcc source, build the compiler and build the firmware. - all new versions of the package will build-depend on the old package and just copy the binary blob from it. Any objections to this? It is pointless. Just add some documentation which explains how to build the firmware image (and the cross-compiler) from source. -- ciao, Marco signature.asc Description: Digital signature
Re: Packaging a library that requires cross-compiled code
Julien BLACHE wrote: Shachar Shemesh [EMAIL PROTECTED] wrote: I think we need a change in policy for handling cases where free software requires free software in order to compile which is, non the less, non buildable on the same platform. It exists already, it's called the contrib section of the archive. No. The contrib section is for free software that requires non-free software. However, all is free software in this thread. The problem does not come from the license but from the difficulty to setup a correct build environment (due to cross compilation to another architecture or another platform) JB. -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
Re: Packaging a library that requires cross-compiled code
Vincent Danjean [EMAIL PROTECTED] wrote: Julien BLACHE wrote: Shachar Shemesh [EMAIL PROTECTED] wrote: I think we need a change in policy for handling cases where free software requires free software in order to compile which is, non the less, non buildable on the same platform. It exists already, it's called the contrib section of the archive. No. The contrib section is for free software that requires non-free software. Policy 2.2.2 [...] Examples of packages which would be included in contrib are: * free packages which require contrib, non-free packages or packages which are not in our archive at all for compilation or execution, and [...] However, all is free software in this thread. The problem does not come from the license but from the difficulty to setup a correct build environment (due to cross compilation to another architecture or another platform) Yeah, right, seems to fit. JB. -- Julien BLACHE [EMAIL PROTECTED] | Debian, because code matters more Debian GNU/Linux Developer| http://www.debian.org Public key available on http://www.jblache.org - KeyID: F5D6 5169 GPG Fingerprint : 935A 79F1 C8B3 3521 FD62 7CC7 CD61 4FD7 F5D6 5169 -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
Re: Packaging a library that requires cross-compiled code
David Anderson writes (Packaging a library that requires cross-compiled code): 2) Package an arm7 cross-compiling gcc with just the right set of options, integrate that with the packaging tools, and then package with a Build-Depends on the cross-compiler. Pro: feels like the Right Way, in a perfect world. Cons: opens the floodgates of packaging cross-compilers, likely requires additions/modifications to packaging tools, and takes way more time than I'm personally ready to put into packaging my code. This is the right answer I think. Several other embedded systems already have cross-compilers in Debian, so this isn't opening any floodgates or getting into anything too unusual. I can see why it would be hard work for you to learn about packaging cross-compilers. 3) Somehow make the packaging tools properly cross-compile the .bin without having a cross-compiler package around. This was briefly suggested on #debian-mentors, but I have no idea how this would be implemented. It's not clear what you mean here. 136 bytes seems very little. Is it written in an assembly language or in C ? Pros: less time involved than 2), would make the package self-contained. Cons: building a cross-compiler in the packaging process just to build a 136 byte binary blob feels like killing a flea with a bazooka, and makes building the package much, much longer than it should be, given the amount of code and logic it actually carries. Yes, but I think that sounds like an acceptable alternative. 1) Ship a built copy of the code in the package's .diff.gz, and DTRT at package creation time to move the .bin from debian/ to the right place in the staging tree. The source code for the .bin is in .orig.tar.gz, under a free license. Anyone is free to modify or rebuild the .bin, provided they obtain an arm7 cross-compiler by non-debian means (instructions provided). People who just want to rebuild the package can do so, without caring that there is cross-compiled code involved. Pro: dead simple, the packaging problem goes away. Con: means shipping a binary blob in the source distribution, which appears to be frowned upon, regardless of whether the source is also freely available in the source distribution. This is (unfortunately for you) not acceptable for Debian proper. Everything in Debian must be buildable from source using tools in Debian. 4) Give up and stay away from the Debian main repositories, just put the package up on a private package repository. Pro: trivial solution. Cons: I'd like to do things right rather than cobbling something together. You could put the package in contrib. I think contrib would be a possibility for this. But it's far from ideal - think of the poor users who want to change the 136-byte loader program. Ian. -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
Re: Packaging a library that requires cross-compiled code
On 9/27/07, Daniel Jacobowitz [EMAIL PROTECTED] wrote: I think this is the way to go unless you get some concrete objections. There is certainly precedent - see for instance the ia32-libs / amd64-libs packages (which are frowned upon for whole different reasons). [replying here, but this is a more general reply to the whole thread] Thanks for the thoughts on this, I've decided to go with the above for the time being, with the following addendums. The upstream release (which I also author, but besides the point) also bundles the binary driver, due to the difficulty of building it directly. Given that I doubt this driver will ever change again now that it works, that the source code is provided, and that the driver is an internal implementation detail and not a pivotal component of the software, it seems like the thing to do. I also have a Debian package almost working now (lintian still has a few complaints, and I need to file an ITP iiuc), which uses the upstream binary driver to build the debian package. I don't know whether this now qualifies as main or contrib, given that the binary is provided by upstream, and therefore no special software is required to actually build the package. - Dave -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
Re: Packaging a library that requires cross-compiled code
On 9/28/07, Ian Jackson [EMAIL PROTECTED] wrote: Pro: feels like the Right Way, in a perfect world. Cons: opens the floodgates of packaging cross-compilers, likely requires additions/modifications to packaging tools, and takes way more time than I'm personally ready to put into packaging my code. This is the right answer I think. Several other embedded systems already have cross-compilers in Debian, so this isn't opening any floodgates or getting into anything too unusual. I can see why it would be hard work for you to learn about packaging cross-compilers. It would be hard work because such a package would be my second ever debian package. Packaging something as complex as a cross-compiler with multiple variants doesn't seem like the right second thing to package. But, if there is precedent, it might not be too painful to mimick existing cross-compiler packages to build my own. I'll see what I can do. 3) Somehow make the packaging tools properly cross-compile the .bin without having a cross-compiler package around. This was briefly suggested on #debian-mentors, but I have no idea how this would be implemented. It's not clear what you mean here. 136 bytes seems very little. Is it written in an assembly language or in C ? It is written in C, with an asm bootstrap. The idea was to basically download and build an arm7 cross-compiler as a part of running dpkg-buildpackage. Insane and wrong, don't say it, I know. I was just being exhaustive as to what had already been suggested. Pros: less time involved than 2), would make the package self-contained. Cons: building a cross-compiler in the packaging process just to build a 136 byte binary blob feels like killing a flea with a bazooka, and makes building the package much, much longer than it should be, given the amount of code and logic it actually carries. Yes, but I think that sounds like an acceptable alternative. If I'm going to build a cross-compiler in the process of building my package, might as well go the extra mile and make the cross-compiler into a package of its own. 4) Give up and stay away from the Debian main repositories, just put the package up on a private package repository. Pro: trivial solution. Cons: I'd like to do things right rather than cobbling something together. You could put the package in contrib. I think contrib would be a possibility for this. But it's far from ideal - think of the poor users who want to change the 136-byte loader program. I am trying as hard as possible to think of cases where you would actually want to change the program, but I'm drawing blank. All the driver does is the equivalent of memcpy(flash_addr, ram_addr, chunk_size); *FLASH_CONTROL = FLASH_WRITE_CMD;. The only thing a user could do by changing that code is damage the flash chip by messing with the timing settings. If packaging the cross-compiler is unavoidable, I'll file an ITP and try to see how that can be done. But if it goes beyond the time I'm prepared to put into it, I'll go with option 4, and host it outside the main debian archive. - Dave -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
Re: Packaging a library that requires cross-compiled code
Hi, David Anderson wrote: But, if there is precedent, it might not be too painful to mimick existing cross-compiler packages to build my own. I'll see what I can do. Please coordinate any such effort with [EMAIL PROTECTED] :-) I'm working on infrastructure to aid generation of cross-compiler packages (a templating mechanism that generates the necessary Package stanzas in debian/control from another file that uses a descriptive language; debian-xcontrol would be the package you are looking for, however that bit is not yet done as I concentrated on features that will allow cross-compilation of other packages first). Simon -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
Re: Packaging a library that requires cross-compiled code
Hi, I'm working on infrastructure to aid generation of cross-compiler packages (a templating mechanism that generates the necessary Package stanzas in debian/control from another file that uses a descriptive language; debian-xcontrol would be the package you are looking for, however that bit is not yet done as I concentrated on features that will allow cross-compilation of other packages first). Which is kinda overkill for David's package - all he needs to cross-compile is a 136 byte large blob. Also this is a piece of code which won't be changed by users ususally, as it does nothing but help to transfer the real firmware into the device (a trivial memcpy implementation according to David). All other parts of the package are arch: all, written in python, so everything needed would be a cross compiler on at least the architecture where the package is being built on. Is there anybody working on an arm7 cross-compiler yet, if so, are there plans to get it into Debian's main? I guess you want to support every arm cpu in main!? Or is there any other possible way to get this package into main? Best regards, Bernd -- Bernd Zeimetz [EMAIL PROTECTED] http://bzed.de/ -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
Re: Packaging a library that requires cross-compiled code
On 9/27/07, Bernd Zeimetz [EMAIL PROTECTED] wrote: 4) Give up and stay away from the Debian main repositories, just put the package up on a private package repository. Please don't choose this way to solve the problem. Lego Mindstorms are used a lot for education, including universities, as it is just very easy to build objects with Lego bricks, so you have more time to think about the code then about how to handle metal or other raw material. I agree. As I said, I want to do things right if I can. Unfortunately I still have no clue about cross-compiling, but please let me know if you need any help with the python parts. Feel free to join #debian-python on OFTC, all questions regarding python apps/modules are welcome there Thanks, I'll be there, since I actually do have a few questions on that part as well. Imho one very fast way to solve the problem would be to package all the python tools, including fwflash - but without the actual firmware blob. Instead you ask the people to download the file or let the tool handle this automatically - at least this would make sure people are able to use your tools with mindstorms while you have enough time to figure out how to ship the precompiled firmware in a debian package. For example you could build the firmware package on arm only and let your tools download and extract the package. If you look at the previous version of my library, which is in C and builds with SCons ( http://code.google.com/p/libnxt/ ), in C I embedded the binary image directly into the source code before compiling, using a small python script. And that python script does detect the absence of the compiled firmware image, and asks the user whether the binary should be downloaded from code.google.com. The alternative is to abort the build, make the binary image locally, and rerun. While this is a workable solution, I find it annoying because it places part of the burden of packaging on the user, for an implementation detail. I have seen this kind of thing done before, but for licensing reasons (non-free firmware that cannot be bundled in the package), whereas this seems to be a purely technical issue. Could the same kind of thing be done at package creation time, ie. prompt the packager to either build or download a working version of the binary image when the package is built (or just download it automatically if non-interactive is required) ? The end result is the same, except that end users don't have to care any more, which is always nice. If you couldn't tell btw: I think that the first solution I mentionned would work well. The flash driver is tiny, and so trivial that it is highly unlikely that it will change in the future, so I consider bundling it a debian-specific fix to the upstream build system more than anything else (so that the packaging process doesn't need to download something from the web, which is what the python setup.py is likely going to do to get around the cross-compiler issue). - Dave -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
Re: Packaging a library that requires cross-compiled code
On Wed, Sep 26, 2007 at 06:38:02PM -0700, Steve Langasek wrote: (Otherwise, you'd be building the arch: all package from the binary-arch rule on arm only; this would work, but cause brain-twistiness wrt the separation between arch: all and arch: any.) sbuild does not allow this. Bastian -- If a man had a child who'd gone anti-social, killed perhaps, he'd still tend to protect that child. -- McCoy, The Ultimate Computer, stardate 4731.3 -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
Re: Packaging a library that requires cross-compiled code
On Thu, Sep 27, 2007 at 02:59:16AM +0200, David Anderson wrote: 1) Ship a built copy of the code in the package's .diff.gz, and DTRT at package creation time to move the .bin from debian/ to the right place in the staging tree. The source code for the .bin is in .orig.tar.gz, under a free license. Anyone is free to modify or rebuild the .bin, provided they obtain an arm7 cross-compiler by non-debian means (instructions provided). People who just want to rebuild the package can do so, without caring that there is cross-compiled code involved. I think this is the way to go unless you get some concrete objections. There is certainly precedent - see for instance the ia32-libs / amd64-libs packages (which are frowned upon for whole different reasons). -- Daniel Jacobowitz CodeSourcery -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
Re: Packaging a library that requires cross-compiled code
David Anderson wrote: Therefore, question: how should I get from this situation to having a working .deb (including the cross-compiled driver), while at the same time playing nicely with Debian packaging policies? In the general case, the problem is much wider. Let me give you an example. We currently package Xen and other free virtualization solutions. Some of them can even run proprietary OSes, such as Windows. Now, suppose that one would like to improve the way Windows runs under Xen by writing, say, a custom mouse driver for Windows that para-virtualizes the mouse on Windows. Such things are quite common in todays' world. The problem is that this driver is a kernel level driver for Windows. If it were user space we could still claim it could be compiled with cross-mingw, but this is not the case here. Setting up an environment for compiling Windows kernel drivers merely so we can build this tiny blob seems out of proportion. I think we need a change in policy for handling cases where free software requires free software in order to compile which is, non the less, non buildable on the same platform. Shachar -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
Re: Packaging a library that requires cross-compiled code
Shachar Shemesh [EMAIL PROTECTED] wrote: I think we need a change in policy for handling cases where free software requires free software in order to compile which is, non the less, non buildable on the same platform. It exists already, it's called the contrib section of the archive. JB. -- Julien BLACHE - Debian GNU/Linux Developer - [EMAIL PROTECTED] Public key available on http://www.jblache.org - KeyID: F5D6 5169 GPG Fingerprint : 935A 79F1 C8B3 3521 FD62 7CC7 CD61 4FD7 F5D6 5169 -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
Packaging a library that requires cross-compiled code
[I've only just subscribed, please forgive me if I got something horribly wrong] Hello, I was redirected here for advice from #debian-mentors, following a packaging question I have. The short summary: I'd like to package a library which requires a tiny blob of cross-compiled code to run, and don't know how to achieve this in a debian-friendly way. Context: the code I'd like to package is a Python package and associated Python scripts, that enable low-level communication with a Lego Mindstorms NXT brick over USB. This is the newer Lego Mindstorms brick, which to the best of my knowledge currently has no supporting libraries available in Debian (support is available for the older, but completely different RCX model). My module enables low-level communication at two levels: a wrapper around python-pyusb to facilitate scanning for a brick and accessing it, and an implementation of the simple command protocol used by the onboard bootloader. This is a library aimed squarely at developers who want to replace Lego's firmware with something else (in my case, an open source high kernel I'm working on). One of the tools provided is called `fwflash`, which takes a binary firmware image and writes it to the brick's flash chip via the bootloader. One of the steps in the flashing procedure is to upload a tiny (136 bytes) driver for the flash controller into the brick's ram, to oversee the actual flashing process. The brick cannot be flashed without the help of this driver. (for those who care, the flash memory is single-plane, which cannot be simultaneously read and written, which requires the use of an in-ram driver for the few instructions that actually perform the write. Attempting to drive the entire process through the bootloader crashes the brick.) And, of course, as the brick is based on an arm7 architecture, this flash driver needs to be cross-compiled. From an architectural POV, the flash driver is an internal implementation detail, the end user of the library doesn't care about the technicalities that forced it into existence. Problem: the flashing tool is unusable without this tiny piece of firmware, but that code can only be compiled with a properly built arm7tdmi cross-compiling gcc, which is not available in Debian. I built my own from gnuarm.com. Therefore, question: how should I get from this situation to having a working .deb (including the cross-compiled driver), while at the same time playing nicely with Debian packaging policies? Possible solutions that we came up with on #debian-mentors: 1) Ship a built copy of the code in the package's .diff.gz, and DTRT at package creation time to move the .bin from debian/ to the right place in the staging tree. The source code for the .bin is in .orig.tar.gz, under a free license. Anyone is free to modify or rebuild the .bin, provided they obtain an arm7 cross-compiler by non-debian means (instructions provided). People who just want to rebuild the package can do so, without caring that there is cross-compiled code involved. Pro: dead simple, the packaging problem goes away. Con: means shipping a binary blob in the source distribution, which appears to be frowned upon, regardless of whether the source is also freely available in the source distribution. 2) Package an arm7 cross-compiling gcc with just the right set of options, integrate that with the packaging tools, and then package with a Build-Depends on the cross-compiler. Pro: feels like the Right Way, in a perfect world. Cons: opens the floodgates of packaging cross-compilers, likely requires additions/modifications to packaging tools, and takes way more time than I'm personally ready to put into packaging my code. 3) Somehow make the packaging tools properly cross-compile the .bin without having a cross-compiler package around. This was briefly suggested on #debian-mentors, but I have no idea how this would be implemented. Pros: less time involved than 2), would make the package self-contained. Cons: building a cross-compiler in the packaging process just to build a 136 byte binary blob feels like killing a flea with a bazooka, and makes building the package much, much longer than it should be, given the amount of code and logic it actually carries. 4) Give up and stay away from the Debian main repositories, just put the package up on a private package repository. Pro: trivial solution. Cons: I'd like to do things right rather than cobbling something together. There you have it. Ideas, thoughts? - Dave -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
Re: Packaging a library that requires cross-compiled code
On 9/27/07, David Anderson [EMAIL PROTECTED] wrote: Possible solutions that we came up with on #debian-mentors: The other possibility that was mentioned was to split the firmware out into a separate source package that produces an Arch: all package and then ensure that it is built on arm. People in the channel had no idea if this would work though :) -- bye, pabs http://wiki.debian.org/PaulWise -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
Re: Packaging a library that requires cross-compiled code
4) Give up and stay away from the Debian main repositories, just put the package up on a private package repository. Please don't choose this way to solve the problem. Lego Mindstorms are used a lot for education, including universities, as it is just very easy to build objects with Lego bricks, so you have more time to think about the code then about how to handle metal or other raw material. Unfortunately I still have no clue about cross-compiling, but please let me know if you need any help with the python parts. Feel free to join #debian-python on OFTC, all questions regarding python apps/modules are welcome there Imho one very fast way to solve the problem would be to package all the python tools, including fwflash - but without the actual firmware blob. Instead you ask the people to download the file or let the tool handle this automatically - at least this would make sure people are able to use your tools with mindstorms while you have enough time to figure out how to ship the precompiled firmware in a debian package. For example you could build the firmware package on arm only and let your tools download and extract the package. Hope I could give you some useful hints, Bernd -- Bernd Zeimetz [EMAIL PROTECTED] http://bzed.de/ -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
Re: Packaging a library that requires cross-compiled code
On Thu, Sep 27, 2007 at 11:11:40AM +1000, Paul Wise wrote: On 9/27/07, David Anderson [EMAIL PROTECTED] wrote: Possible solutions that we came up with on #debian-mentors: The other possibility that was mentioned was to split the firmware out into a separate source package that produces an Arch: all package and then ensure that it is built on arm. People in the channel had no idea if this would work though :) The only (good) way to ensure the arch: all package is built on arm is to always do the sourceful upload of the package from arm. (Otherwise, you'd be building the arch: all package from the binary-arch rule on arm only; this would work, but cause brain-twistiness wrt the separation between arch: all and arch: any.) -- Steve Langasek Give me a lever long enough and a Free OS Debian Developer to set it on, and I can move the world. [EMAIL PROTECTED] http://www.debian.org/ -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
Re: Packaging a library that requires cross-compiled code
On Wed, Sep 26, 2007 at 06:38:02PM -0700, Steve Langasek wrote: On Thu, Sep 27, 2007 at 11:11:40AM +1000, Paul Wise wrote: On 9/27/07, David Anderson [EMAIL PROTECTED] wrote: Possible solutions that we came up with on #debian-mentors: The other possibility that was mentioned was to split the firmware out into a separate source package that produces an Arch: all package and then ensure that it is built on arm. People in the channel had no idea if this would work though :) The only (good) way to ensure the arch: all package is built on arm is to always do the sourceful upload of the package from arm. (Otherwise, you'd be building the arch: all package from the binary-arch rule on arm only; this would work, but cause brain-twistiness wrt the separation between arch: all and arch: any.) I think that perhaps Paul meant to say an arch any package. If the code must always be compiled for a particular flavor of ARM processor, regardless of the host architecture of the machine which will be controlling the Lego, then it should be possible to build it is arch any, which would not get autobuilt anyways. Then, as long as the uploader (maintainer or NMU) buils on ARM, everything should be OK. Regards, -Roberto -- Roberto C. Sánchez http://people.connexer.com/~roberto http://www.connexer.com signature.asc Description: Digital signature
Re: Packaging a library that requires cross-compiled code
On Wed, Sep 26, 2007 at 09:43:17PM -0400, Roberto C. Sánchez wrote: I think that perhaps Paul meant to say an arch any package. If the code must always be compiled for a particular flavor of ARM processor, regardless of the host architecture of the machine which will be controlling the Lego, then it should be possible to build it is arch any, which would not get autobuilt anyways. Then, as long as the uploader (maintainer or NMU) buils on ARM, everything should be OK. Please completely disregard what I said. I have got it completely confused. Regards, -Roberto -- Roberto C. Sánchez http://people.connexer.com/~roberto http://www.connexer.com signature.asc Description: Digital signature
Re: Packaging a library that requires cross-compiled code
On 9/27/07, Steve Langasek [EMAIL PROTECTED] wrote: The other possibility that was mentioned was to split the firmware out into a separate source package that produces an Arch: all package and then ensure that it is built on arm. People in the channel had no idea if this would work though :) The only (good) way to ensure the arch: all package is built on arm is to always do the sourceful upload of the package from arm. Right. The issue would be whether or not the arm/armel gcc binaries support the right target characteristics: danderson the target system is an arm7, with no MMU, so no linux for starters danderson I suspect that if debian supports an arm architecture, it'd be arm9 or higher helmut -mcpu=arm7? danderson arm7tdmi * danderson digs out the arm-elf-gcc configuration danderson arm-elf, arm7 cpu, thumb interworking support, software floating point emulation. That's the combination I need. Any emdebian/other folks know if this is an option? -- bye, pabs http://wiki.debian.org/PaulWise -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
Re: Packaging a library that requires cross-compiled code
On Thu, Sep 27, 2007 at 02:59:16AM +0200, David Anderson wrote: [how best to package a stub for which the cross-compiler isn't in Debian] [possible solution 1] 1) Ship a built copy of the code in the package's .diff.gz, and DTRT at package creation time to move the .bin from debian/ to the right place in the staging tree. The source code for the .bin is in .orig.tar.gz, under a free license. Anyone is free to modify or rebuild the .bin, provided they obtain an arm7 cross-compiler by non-debian means (instructions provided). People who just want to rebuild the package can do so, without caring that there is cross-compiled code involved. Pro: dead simple, the packaging problem goes away. Con: means shipping a binary blob in the source distribution, which appears to be frowned upon, regardless of whether the source is also freely available in the source distribution. A slight variant on this would be to ship assembly source (the output of gcc -S) instead of a binary blob, and cross-assemble that. If the stub is 136 bytes long, assembly isn't too bad, and it's a step away from a binary blob. This presumes that the cross-tools in Debian can correctly assemble arm7, which I haven't verified. But the binary-blob variant is probably simplest, and I'd recommend going with that to get packaging done and the package in Debian. Alternately, check for the compiler, and use it if it's installed. (And fall back to the binary blob if it's not there.) Jon Leonard -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
