Re: [fedora-arm] ARM Primary FESCO discussion results, round 1
On 03/20/2012 08:00 PM, Brendan Conoboy wrote: On 03/20/2012 02:48 AM, Gordan Bobic wrote: Are alignment problems not considered bugs? It's not just that this will break code on ARM v7 and IIRC SPARC, but alignment issues also cause cache line straddling which has a performance impact. I took this question to be a case of knowing-just-enough-about-ARM-to-be-harmful. It's really a non-issue, particularly since later ARM chips don't have the problem. I disagree. ARMv5 is going to be around for a while yet, and the fix-up in software at least needs to be implemented either by default or set as the very first thing in rc.sysinit: https://bugzilla.redhat.com/show_bug.cgi?id=673691 Otherwise not paying attention to alignment in something like e2fsprogs could plausibly trash the file system: https://bugzilla.redhat.com/show_bug.cgi?id=680090 Even if sloppy programming is not an issue on later ARMs, IMO the alignment issues should be treated as bugs at least until ARMv5 support is completely dropped. Is there any actual reason why Anaconda cannot be used? What is to stop booting a suitable installation kernel for the target platform and having that fetch/mount an installation rootfs that takes over, same as it does in x86? Granted the amount of RAM is an issue, but that's just a case of dieting the installer back to a saner size (de-lobotomizing the text mode installer back to how it was before F11, for example). Anaconda isn't *quite* ready to go yet. And you don't actually gain much by using Anaconda for many devices- you still have to write an image to a removal storage device. Which you then run boot... and it writes a new image to a storage device. You've just made more work for yourself when you could have installed a working image directly. In that case you might as well use the same installation procedure on x86, too - there's no reason not to. On the server side, PXE abilities mean there's more to be said about Anaconda support. The only issue is, they don't exist yet :-P I found that if the kernel has support for the right SoC, it is simply a case of adding a suitable SoC merge config file and plumbing it in for the build based on a build flag. I have somewhere a 2.6.32-220.something kernel src.rpm that builds for both x86 and Marvell Kirkwood, and it was reasonably straightforward to achieve (even if it did require fixing a few bugs that were introduced by upstream vendor patches that didn't manifest on the primary arch). Yes, using one kernel source rpm is working fine for us. It's the long build time which needs to be overcome. distcc helps. Gordan ___ arm mailing list arm@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/arm
Re: [fedora-arm] ARM Primary FESCO discussion results, round 1
On 03/20/12 20:00, Brendan Conoboy wrote: On 03/20/2012 02:48 AM, Gordan Bobic wrote: Are alignment problems not considered bugs? It's not just that this will break code on ARM v7 and IIRC SPARC, but alignment issues also cause cache line straddling which has a performance impact. I took this question to be a case of knowing-just-enough-about-ARM-to-be-harmful. It's really a non-issue, particularly since later ARM chips don't have the problem. Well... It is a real (but usually small) performance issue on on some ARMv7 processors for the basic LDR/STR/LDRH/STRH instructions. However, LDM/STM/LDRD/STRD still need to be word aligned. So you're still looking at the choice between an application crashing or a risk of a two-orders-of-magnitude slowdown of some accesses due to kernel fixup in the exception handler unless the code is fixed. / Leif ___ arm mailing list arm@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/arm
Re: [fedora-arm] ARM Primary FESCO discussion results, round 1
Chris Tyler wrote: Otherwise not paying attention to alignment in something like e2fsprogs could plausibly trash the file system: https://bugzilla.redhat.com/show_bug.cgi?id=680090 Even if sloppy programming is not an issue on later ARMs, IMO the alignment issues should be treated as bugs at least until ARMv5 support is completely dropped. Alignment issues happen on x86 too, but we accept the cost of the hardware fixup and ignore them. Only because they are out of sight - out of mind and most developers (myself included until relatively recently) don't know better. x86 induced brain damage is not a good thing. Alignment issues are not necessarily bugs; sometimes they're the result of conscious decisions that the fixup is cheaper than the pages and pages of code required to avoid them in the first place (e.g., in streamed data). FWIW, of all the alignment bugs I have observed in Fedora packages, precisely 0 were in streamed data. That leads me to think that most instances are oversights rather than conscious decisions. Anaconda isn't *quite* ready to go yet. And you don't actually gain much by using Anaconda for many devices- you still have to write an image to a removal storage device. Which you then run boot... and it writes a new image to a storage device. You've just made more work for yourself when you could have installed a working image directly. In that case you might as well use the same installation procedure on x86, too - there's no reason not to. We do, after a fashion (and then wrap it in some extra layers): it's called a Live Disc :-) Indeed, but that's not an installation. Gordan ___ arm mailing list arm@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/arm
Re: [fedora-arm] ARM Primary FESCO discussion results, round 1
On Wed, 2012-03-21 at 12:28 +, Gordan Bobic wrote: Chris Tyler wrote: We do, after a fashion (and then wrap it in some extra layers): it's called a Live Disc :-) Indeed, but that's not an installation. Install to Disk on a live disc dd's the premade ext3/4 filesystem image to the hard drive and then resizes it. This is fundamentally the same installation path that we're talking about here. (In the Pi remix, for example, the dd to SD card is done by an installer script on another machine, and the resize is done on first boot.) In both cases, we're using a prebuilt filesystem image rather than the traditional RPM-by-RPM installation. -Chris ___ arm mailing list arm@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/arm
Re: [fedora-arm] ARM Primary FESCO discussion results, round 1
On 03/21/2012 05:22 AM, Chris Tyler wrote: https://bugzilla.redhat.com/show_bug.cgi?id=673691 Agreed, alignment fixups must be enabled early. Ah, I thought this was already resolved. So, er, we just need to followup on this BZ until it's resolved in rawhide? I see it's already in ARMTracker. -- Brendan Conoboy / Red Hat, Inc. / b...@redhat.com ___ arm mailing list arm@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/arm
Re: [fedora-arm] ARM Primary FESCO discussion results, round 1
Yes, using one kernel source rpm is working fine for us. It's the long build time which needs to be overcome. distcc helps. distcc helps, I've done it, but until x86 uses distcc in koji, ARM can't use it either. ___ arm mailing list arm@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/arm
Re: [fedora-arm] ARM Primary FESCO discussion results, round 1
Aloha, On Mon, 2012-03-19 at 21:00 -0400, Chris Tyler wrote: How do packagers test and resolve failures on ARM if they don't own an ARM device? There are several solutions available: 1. They can use emulation. Fedora already includes a good-quality ARM emulator (qemu-system-arm) that provides decent performance; we can package ready-to-run ARM images for that emulator. (This emulator can provide Kirkwood execution speeds on a modern x86 PC). 2. ARM computers can be bought for as little as $35 (the Raspberry Pi). 3. Packagers can use Koji for remote scratch builds (just as someone who only has 32-bit x86 can use Koji for 64-bit x86 builds). 4. We could consider setting up remote access to ARM systems (though this doesn't provide a lot of benefit over option 1 in most cases). Please excuse me if you feel I'm being somewhat cheeky with this, but I genuinely think it could be of benefit to Fedora (and also openSUSE). As some may know openSUSE uses the Open Build Service[0] for all its packaging. It is similar to Koji but not quite the same. Our current setup is to build using the public instance[1] which uses qemu for ARM emulation, similar to what Chris is proposing. What we also have is a small native ARM build farm (mostly pandaboards, with some efikas) that verifies build failures on the OBS. We have found quite a few packages fail under emulation vs native, and have subsequently submitted patches upstream for both qemu and the respective packages. So why am I suggesting Fedora consider the OBS? In a nutshell it means that more of your community can chip in with an easy(ish) tool, without having to have decent hardware to hand for running the emulator. I'm not saying it is a perfect tool, but it does work and we (the openSUSE community) would be more than happy to help where we can. Just an option for your consideration :-) Regards, Andy 0 = http://en.opensuse.org/Portal:Build_Service 1 = https://build.opensuse.org/ -- Andrew Wafaa IRC: FunkyPenguin GPG: 0x3A36312F ___ arm mailing list arm@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/arm
Re: [fedora-arm] ARM Primary FESCO discussion results, round 1
On 03/20/2012 12:30 AM, Brendan Conoboy wrote: On 03/19/2012 04:46 PM, Brendan Conoboy wrote: How do packagers test and resolve failures on ARM if they don't own an ARM device? I like Chris's suggestion of the simulator- that's good coverage. I'd also support providing login hosts to... somewhere. Seneca? Spare systems in PHX? I'm not sure, but it seems like a good idea to me. So we had discussed making some Seneca boards available. If need by, Red Hat can buy a few more boards if they can be FAS hosted there? We can also do virtual machines (let's get that setup...soon) and later, the plan is/was always to provide some login nodes in PHX AFAIK. Jon. ___ arm mailing list arm@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/arm
Re: [fedora-arm] ARM Primary FESCO discussion results, round 1
On 03/20/2012 12:30 AM, Brendan Conoboy wrote: On 03/19/2012 04:46 PM, Brendan Conoboy wrote: What does the kernel team think about the the time required to build kernels on ARM? How will it affect their workflow? I suspect Jon is going to discuss with kernel@. He is. Jon. ___ arm mailing list arm@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/arm
Re: [fedora-arm] ARM Primary FESCO discussion results, round 1
On 03/20/2012 02:48 AM, Gordan Bobic wrote: Are alignment problems not considered bugs? It's not just that this will break code on ARM v7 and IIRC SPARC, but alignment issues also cause cache line straddling which has a performance impact. I took this question to be a case of knowing-just-enough-about-ARM-to-be-harmful. It's really a non-issue, particularly since later ARM chips don't have the problem. Is there any actual reason why Anaconda cannot be used? What is to stop booting a suitable installation kernel for the target platform and having that fetch/mount an installation rootfs that takes over, same as it does in x86? Granted the amount of RAM is an issue, but that's just a case of dieting the installer back to a saner size (de-lobotomizing the text mode installer back to how it was before F11, for example). Anaconda isn't *quite* ready to go yet. And you don't actually gain much by using Anaconda for many devices- you still have to write an image to a removal storage device. Which you then run boot... and it writes a new image to a storage device. You've just made more work for yourself when you could have installed a working image directly. On the server side, PXE abilities mean there's more to be said about Anaconda support. The only issue is, they don't exist yet :-P I found that if the kernel has support for the right SoC, it is simply a case of adding a suitable SoC merge config file and plumbing it in for the build based on a build flag. I have somewhere a 2.6.32-220.something kernel src.rpm that builds for both x86 and Marvell Kirkwood, and it was reasonably straightforward to achieve (even if it did require fixing a few bugs that were introduced by upstream vendor patches that didn't manifest on the primary arch). Yes, using one kernel source rpm is working fine for us. It's the long build time which needs to be overcome. -- Brendan Conoboy / Red Hat, Inc. / b...@redhat.com ___ arm mailing list arm@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/arm
Re: [fedora-arm] ARM Primary FESCO discussion results, round 1
First stab at a few answers-- On Mon, 2012-03-19 at 16:46 -0700, Brendan Conoboy wrote: How are ARM-specific issues such as legacy alignment problems to be addressed? ARM v7 systems have hardware fixup for alignment issues, just as x86 does. ARM v5 systems can do fixup in software with a CPU trap at a small performance cost. (Note: there is a performance cost for hardware fixup, even on x86 systems) How do packagers test and resolve failures on ARM if they don't own an ARM device? There are several solutions available: 1. They can use emulation. Fedora already includes a good-quality ARM emulator (qemu-system-arm) that provides decent performance; we can package ready-to-run ARM images for that emulator. (This emulator can provide Kirkwood execution speeds on a modern x86 PC). 2. ARM computers can be bought for as little as $35 (the Raspberry Pi). 3. Packagers can use Koji for remote scratch builds (just as someone who only has 32-bit x86 can use Koji for 64-bit x86 builds). 4. We could consider setting up remote access to ARM systems (though this doesn't provide a lot of benefit over option 1 in most cases). When will server hardware be available? Real Soon Now(tm)! How about We anticipate early availability of ARM server systems in the summer of 2012. Why isn't being a secondary architecture good enough? 1. The user base is expected to grow to substantial numbers quickly. It is important to provide this user base with timely security updates and fixes, which can happen much more readily in primary arch than in secondary (which, even in the best case, lags behind PA). 2. As innovators in open source, we have an opportunity to take a leadership position by providing first-class support for emerging platforms such as the OLPC XO 1.75 and 3.0, the Raspberry Pi, and upcoming very-environmentally-friendly enterprise-class server systems, each of which supports an admirable goal. 3. The differences between ARM and x86 are not huge, and therefore cost of supporting ARM as a primary arch is not very high. ARM is little-endian (like x86), and nearly all of the libraries and languages in Fedora now support ARM (very different from even a few releases ago). (Needs strengthening) Why not wait for 64 bit ARM? 64 bit ARM is years away from general availability, let alone widespread use. Fully supporting 32-bit ARM now puts us in a solid position to support 64-bit enterprises systems as they appear. With there being so many different kernel variants, how will a kernel build complete in a reasonable period of time? The builds being done in Koji are great, but what is the plan for composes, QE and installation? Rootfs composes will be automated in much the same way that x86 spin composes are now automated. QE and installation will be different but not vastly more complex than in the x86 realm. (I think that we should explain the ARM install situation in terms of live image spin composes, because these have the greatest similarity -- the package selection is preset and not alterable during installation, for example). If Anaconda isn't used to do installations, what will be doing the things Anaconda does which just installing a bunch of packages doesn't? (I don't know what these are) These include: - setting up the partitioning - (in some cases) setting up the network configuration - setting the root password - selecting the timezone - selecting the locale and keyboard layout - (package selection, which does not apply to x86 live images) Several of these will need to move from Anaconda to firstboot (we're already working firstboot modules for a few of these for the Raspberry Pi). Will there be extra patches in the kernel to enable new vendors' ARM processors or will upstream continue to be the way? What does the kernel team think about the the time required to build kernels on ARM? How will it affect their workflow? Question: do we need to do a fully-clean build for each ARM SoC variant? Or can we do the machine and config changes for each and then make? The difference in time could be huge. The proposal suggests building just a versatile express kernel by default (to save time), then using koji flags to support alternate kernels. Is this possible? I think a better question is: Is this wise? This would mean that the majority of the ARM kernels won't be built or tested by default. (Also, do we mean Koji flags or do we mean editing the spec file to set macro values?) In the event that kernels are built separately per the above, what mechanism will be used to keep the kernels in sync? Assertions from the meeting: There must be a commitment of hardware both for build systems and test systems for PA. Being a PA carries the obligation that all packages in Fedora will be available. The proposed avenue of making broken packages temporarily excludearch is questionable and needs work. FTBFS issues should simply be fixed (That's not an ARM
Re: [fedora-arm] ARM Primary FESCO discussion results, round 1
On 03/19/2012 09:30 PM, Brendan Conoboy wrote: Being a PA carries the obligation that all packages in Fedora will be available. The proposed avenue of making broken packages temporarily excludearch is questionable and needs work. The thing that worries me about the excludearch is that there is no mechanism forcing ultimate resolution on packages which could be made to work on ARM. Right now the goal of PA is a strong motivator for fixing packages. If we make it to PA, how will we stay motivated? We're only ever going to asymptotically approach the complete x86 package set so we're going to have to strike a middle ground. What that is is open for discussion, but basically it breaks down into 3 categories: 1. Packages that are truly x86 specific do not need to be made to work on ARM. And the packages that depend upon them hopefully don't either, but there's going to be some issues there. 2. Packages that FTBFS on x86 do not need to be made to work on ARM until they're also made to work on x86. 3. Anything left can and should be made to work on ARM. How many packages are in this set? If it's down to a few dozen let's just fix them up and be done with it. If it's a few hundred we need a plan that involves getting to primary before we're at 100%. So, basically, we need data. What packages fit where in the above 3 categories? How close are we? -- Brendan Conoboy / Red Hat, Inc. / b...@redhat.com ___ arm mailing list arm@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/arm
Re: [fedora-arm] ARM Primary FESCO discussion results, round 1
On Mon, 2012-03-19 at 21:38 -0700, Brendan Conoboy wrote: 1. Packages that are truly x86 specific do not need to be made to work on ARM. And the packages that depend upon them hopefully don't either, but there's going to be some issues there. Note that this goes both ways - there are a couple of ARM packages that don't make sense on x86, too, since they're low-level, system-specific pieces. -Chris ___ arm mailing list arm@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/arm
Re: [fedora-arm] ARM Primary FESCO discussion results, round 1
On 03/19/2012 04:46 PM, Brendan Conoboy wrote: How do packagers test and resolve failures on ARM if they don't own an ARM device? I like Chris's suggestion of the simulator- that's good coverage. I'd also support providing login hosts to... somewhere. Seneca? Spare systems in PHX? I'm not sure, but it seems like a good idea to me. When will server hardware be available? During the discussion I said 2-5 months. Notting said 2 was good, 5 was bad. I'll just extrapolate that 3 is not as good and 4 isn't as bad, but ultimately it's going to be when the hardware is available. Hopefully it'll be closer to the 2 than the 5. Why isn't being a secondary architecture good enough? Greater stability in PHX, faster infrastructure, wider mirror selection, greater credibility as a distribution. Why not wait for 64 bit ARM? The mainstay of 64 bit ARM hardware won't be available for a couple years. Incredible ARM systems will be available in the interim that Fedora can run on with comparatively little effort. Additionally, much of what's needed on 64 bit ARM can be done in the 32 bit space (Virtualization support for instance). With there being so many different kernel variants, how will a kernel build complete in a reasonable period of time? Beats me- every proposal is a bit hackish. The builds being done in Koji are great, but what is the plan for composes, QE and installation? This part of the plan really needs some work. Even our discussions about how to compose images isn't quite the same as how to handle QE when it comes time to do an alpha/beta/rc/etc. If Anaconda isn't used to do installations, what will be doing the things Anaconda does which just installing a bunch of packages doesn't? (I don't know what these are) I would hope the standard image builders handle this sort of thing, but do not know. Will there be extra patches in the kernel to enable new vendors' ARM processors or will upstream continue to be the way? Jon answered that it will be upstream only. I agree. Same policy as x86. What does the kernel team think about the the time required to build kernels on ARM? How will it affect their workflow? I suspect Jon is going to discuss with kernel@. The proposal suggests building just a versatile express kernel by default (to save time), then using koji flags to support alternate kernels. Is this possible? According to Dennis: No. So, let's figure out how to get reasonable build times and good breadth on kernel builds. In the event that kernels are built separately per the above, what mechanism will be used to keep the kernels in sync? We may need to have a less ambitious list of kernels. Perhaps keep omap, tegra, highbank, but drop IMX, armada, etc. We'll see where this goes. Assertions from the meeting: There must be a commitment of hardware both for build systems and test systems for PA. In light of qemu+versatile the general test systems might be optional, but in principle if we have systems to spare and appropriate access mechanisms I don't see why not. Being a PA carries the obligation that all packages in Fedora will be available. The proposed avenue of making broken packages temporarily excludearch is questionable and needs work. The thing that worries me about the excludearch is that there is no mechanism forcing ultimate resolution on packages which could be made to work on ARM. Right now the goal of PA is a strong motivator for fixing packages. If we make it to PA, how will we stay motivated? FTBFS issues should simply be fixed (That's not an ARM problem, but we're definitely impacted by it). Suggest presenting the FTBFS list and subtracting it from the list of packages we need for ARM task list. The changes to QE, particularly because of Anaconda, will be substantial. This is not addressed in the proposal. Agree. Help! Installability doesn't necessarily mean Anaconda (See EC2), but it does mean something. A real plan is called for. We just need to formalize our plan for image generation using standard tools (Hopefully whatever was used for EC2 is valid here). All PA kernels must be derived from the same source rpm. Yes. ... Regarding mail to the 4 groups, What do we want to ask each of them? Releng: Do you have any concerns about moving ARM to PA? What are the things you need to see in place? What do you not want to own? Infrastructure: Do we have the hardware/budget ready? What do you need in place that isn't now? What do you not want to own? QE: What do you need to do proper QE on arm? How much load will this add? How can we help? Kernel: ARM will introduce build time delays, but x86 builds will still finish as fast as ever. Is this sufficient? What do you need to see to be okay with ARM on PA? -- Brendan Conoboy / Red Hat, Inc. / b...@redhat.com ___ arm mailing list arm@lists.fedoraproject.org
