Re: [fedora-arm] Koji status update
Hi John, On Thu, Nov 17, 2011 at 7:55 AM, Jon Masters wrote: > Folks, > > My take on current progress is that we have a lot of packages with bits still > needing to head upstream, and we have a number of package deltas between v5 > and v7, but the core set of packages we actually need to get a minimal build > done is about there. Minus: > > * gcc - making sure the bitfields patches are in place for v5 > * glibc - delta needs attention > > I'm not so concerned about lorax and anaconda at this stage, and I don't > think we have critical reps on python3. In reading through the package lists > again tonight on DJ's graphs, I can't help but think we could partially > import enough packages to get Koji running almost immediately tomorrow, then > pull in the rest as we get that resolved. Assuming that is the case, we > should focus on gcc and glibc reconciliation and get this going asap. > > Please reply to this mail with your input. Let's have a status sync up on IRC > later. Absolutely agree. None of the lists I've seen have core build root packages except for the two you mention. There's a number of differences between the lists I've seen and a lot of the spec patches that are in the stage4 repos were already upstreamed from the work I'd done with F-14. Once the gcc/glibc fixes are in we're good to go. I'd not looked at pushing them upstream as I'd figured the people who'd been dealing with them likely had them in hand and knew a lot more of the issues in hand than I do. I think for the F-15 build its going to a little break/fix as we go along as there's likely other bits we've missed or have changed since, as long as the people that fix those problems commit the fix to all F-15/16/rawhide branches then and there so the fixes don't get lost and have to be repeated again. Peter ___ arm mailing list arm@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/arm
Re: [fedora-arm] Koji status update
El Thu, 17 Nov 2011 09:39:58 + Peter Robinson escribió: > Hi John, > > On Thu, Nov 17, 2011 at 7:55 AM, Jon Masters > wrote: > > Folks, > > > > My take on current progress is that we have a lot of packages with > > bits still needing to head upstream, and we have a number of > > package deltas between v5 and v7, but the core set of packages we > > actually need to get a minimal build done is about there. Minus: > > > > * gcc - making sure the bitfields patches are in place for v5 > > * glibc - delta needs attention > > > > I'm not so concerned about lorax and anaconda at this stage, and I > > don't think we have critical reps on python3. In reading through > > the package lists again tonight on DJ's graphs, I can't help but > > think we could partially import enough packages to get Koji running > > almost immediately tomorrow, then pull in the rest as we get that > > resolved. Assuming that is the case, we should focus on gcc and > > glibc reconciliation and get this going asap. since we are using external repos and not using importing anything. we are going to be building the same nvrs in koji so we can not import. we need to set up the tags specially. without inheritance otherwise the f14 armv5tel builds will override all the f15 ones. but there really is no reason that we can not start building today. i am working on one patch for koji to deal with how it writes out the mock config files. > > Please reply to this mail with your input. Let's have a status sync > > up on IRC later. > > Absolutely agree. None of the lists I've seen have core build root > packages except for the two you mention. There's a number of > differences between the lists I've seen and a lot of the spec patches > that are in the stage4 repos were already upstreamed from the work I'd > done with F-14. > > Once the gcc/glibc fixes are in we're good to go. I'd not looked at > pushing them upstream as I'd figured the people who'd been dealing > with them likely had them in hand and knew a lot more of the issues in > hand than I do. even with these still to be resolved, there is zero reason right now why we cannot go ahead and build everything else. > I think for the F-15 build its going to a little break/fix as we go > along as there's likely other bits we've missed or have changed since, > as long as the people that fix those problems commit the fix to all > F-15/16/rawhide branches then and there so the fixes don't get lost > and have to be repeated again. Im sure there will be some small issues crop up, but shouldnt be too horrible. I vote that we get building today. why we make sure we get the gcc/glibc etc things resolved. we can the start mshing some repos and getting bits out there :) Dennis ___ arm mailing list arm@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/arm
Re: [fedora-arm] Koji status update
On Thu, Nov 17, 2011 at 5:39 PM, Dennis Gilmore wrote: > El Thu, 17 Nov 2011 09:39:58 + > Peter Robinson escribió: >> Hi John, >> >> On Thu, Nov 17, 2011 at 7:55 AM, Jon Masters >> wrote: >> > Folks, >> > >> > My take on current progress is that we have a lot of packages with >> > bits still needing to head upstream, and we have a number of >> > package deltas between v5 and v7, but the core set of packages we >> > actually need to get a minimal build done is about there. Minus: >> > >> > * gcc - making sure the bitfields patches are in place for v5 >> > * glibc - delta needs attention >> > >> > I'm not so concerned about lorax and anaconda at this stage, and I >> > don't think we have critical reps on python3. In reading through >> > the package lists again tonight on DJ's graphs, I can't help but >> > think we could partially import enough packages to get Koji running >> > almost immediately tomorrow, then pull in the rest as we get that >> > resolved. Assuming that is the case, we should focus on gcc and >> > glibc reconciliation and get this going asap. > > since we are using external repos and not using importing anything. we > are going to be building the same nvrs in koji so we can not import. > we need to set up the tags specially. without inheritance otherwise the > f14 armv5tel builds will override all the f15 ones. but there really > is no reason that we can not start building today. i am working on one > patch for koji to deal with how it writes out the mock config files. > >> > Please reply to this mail with your input. Let's have a status sync >> > up on IRC later. >> >> Absolutely agree. None of the lists I've seen have core build root >> packages except for the two you mention. There's a number of >> differences between the lists I've seen and a lot of the spec patches >> that are in the stage4 repos were already upstreamed from the work I'd >> done with F-14. >> >> Once the gcc/glibc fixes are in we're good to go. I'd not looked at >> pushing them upstream as I'd figured the people who'd been dealing >> with them likely had them in hand and knew a lot more of the issues in >> hand than I do. > even with these still to be resolved, there is zero reason right now > why we cannot go ahead and build everything else. > >> I think for the F-15 build its going to a little break/fix as we go >> along as there's likely other bits we've missed or have changed since, >> as long as the people that fix those problems commit the fix to all >> F-15/16/rawhide branches then and there so the fixes don't get lost >> and have to be repeated again. > Im sure there will be some small issues crop up, but shouldnt be too > horrible. > > I vote that we get building today. why we make sure we get the > gcc/glibc etc things resolved. we can the start mshing some repos and > getting bits out there :) Do we have softfp and hardfp builder ready to go? koji3 is still online for armv5tel for builds that need lots of RAM, let me know if it needs any config updates. Peter ___ arm mailing list arm@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/arm
Re: [fedora-arm] Koji status update
On Thu, Nov 17, 2011 at 3:39 AM, Peter Robinson wrote:> > Once the gcc/glibc fixes are in we're good to go. I'd not looked at > pushing them upstream as I'd figured the people who'd been dealing > with them likely had them in hand and knew a lot more of the issues in > hand than I do. I think these issues would benefit from your attention/direction. I'm a bit out of the loop but I did document the current state of glibc a couple of weeks ago: http://fedoraproject.org/wiki/Architectures/ARM/Fedora15_HardFP_Bootstrap_package_status#glibc Daniel ___ arm mailing list arm@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/arm
Re: [fedora-arm] Koji status update
On Thu, 2011-11-17 at 02:55 -0500, Jon Masters wrote: > Folks, > > My take on current progress is that we have a lot of packages with > bits still needing to head upstream, and we have a number of package > deltas between v5 and v7, but the core set of packages we actually > need to get a minimal build done is about there. Minus: > > * gcc - making sure the bitfields patches are in place for v5 > * glibc - delta needs attention > > I'm not so concerned about lorax and anaconda at this stage, and I > don't think we have critical reps on python3. In reading through the > package lists again tonight on DJ's graphs, I can't help but think we > could partially import enough packages to get Koji running almost > immediately tomorrow, then pull in the rest as we get that resolved. > Assuming that is the case, we should focus on gcc and glibc > reconciliation and get this going asap. I agree with going with what we've currently got package-wise. But saying "we should start building today" or "get Koji running...tomorrow" is missing the point. We're ready to go the moment we have finished these tasks: - the gcc/glibc issues are resolved - we've pruned the package sets back to the same core - every change/patch in those package sets has been upstreamed - the hub and builders have been successfully set up and tested with patched koji/rpm/yum etc. Work on all these pieces is proceeding in parallel (assuming that someone's on the gcc/glibc piece (who?)). We're not blocking on any one piece, but we have to finish these four before hitting 'Go'. -Chris ___ arm mailing list arm@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/arm
Re: [fedora-arm] Koji status update
On 11/17/2011 11:04 AM, Chris Tyler wrote: > I agree with going with what we've currently got package-wise. But > saying "we should start building today" or "get Koji running...tomorrow" > is missing the point. > > We're ready to go the moment we have finished these tasks: > - the gcc/glibc issues are resolved > - we've pruned the package sets back to the same core > - every change/patch in those package sets has been upstreamed > - the hub and builders have been successfully set up and tested with > patched koji/rpm/yum etc. > > Work on all these pieces is proceeding in parallel (assuming that > someone's on the gcc/glibc piece (who?)). We're not blocking on any one > piece, but we have to finish these four before hitting 'Go'. The final patch for gcc, for instance, may never make it into F15. Even the armv7hl spec file changes we pushed upstream are only in .fc16 koji builds at this point. If these changes are in FC16 or rawhide but not F15, isn't that good enough? Why not build packages in parallel too? -- Brendan Conoboy / Red Hat, Inc. / b...@redhat.com ___ arm mailing list arm@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/arm
Re: [fedora-arm] Koji status update
On Thu, 2011-11-17 at 15:03 -0800, Brendan Conoboy wrote: > On 11/17/2011 11:04 AM, Chris Tyler wrote: > > I agree with going with what we've currently got package-wise. But > > saying "we should start building today" or "get Koji running...tomorrow" > > is missing the point. :) I'm not so sure. I think the goal should be to get to rawhide as quickly as possible because it makes all of these problems go away - no rebuilding retrospectively, not being treated quite so much as a secondary, and so on (then we push for primary). Given that that is our general ambition, we should work backward and try to get to that point as quickly as we can without being too hackish in our approach. > > We're ready to go the moment we have finished these tasks: > > - the gcc/glibc issues are resolved Let's split these in two: 1). gcc. Having seen some of the discussion here with Jakub, etc. it's clear that the Fedora maintainers aren't going to accept this latest volatile bitfields patch unless it's in the upstream 4.6 branch (it is in trunk right now). So there are two upstreams to work with - Fedora, and GCC. We can try to pursue this, but it is not a quick fix (we're looking at several weeks to turn it around). At the same time, this stuff is resolved in rawhide (and I think F16). It would be more expedient to block updates to gcc being pulled in automatically (which won't happen at this stage anyway) and either in parallel fix this or undertake to keep a separate gcc. Really, by the time we release F15 we'll have only 5 months of time when there would be updates anyway. 2). glibc. Seems there's a backport that could perhaps be upstreamed, and a Java fix that we will need. Again, I favor parallelization. We build now and try to find a solution that involves either getting this into upstream Fedora 15 PA or finding an alternative. I don't see a reason not to build this now though. > > - we've pruned the package sets back to the same core I think this is straightforward. We include in the repos only the build requirements to build a buildroot and start with that. Do some closure tests on that set and make sure the same packages exist for v5 and v7. I suspect someone could turn that around tomorrow. > > - every change/patch in those package sets has been upstreamed I was thinking that way, but now I think that is a parallel activity. I am all for getting stuff upstreamed, and it will be especially important in the coming months, but I think that doesn't need to gate the initial build. Packages can keep an armX suffix in the initial build, then for the few dozen (after cutting down the initial build set) that need to be upstreamed, there will be a higher NVR from upstream soon anyway that can be rebuilt for both arches with the upstream version. IOW I think the initial build should proceed now, but that we should not begin building updates or pulling further bits from upstream until we have gotten the rest of these deltas resolved. Since the upstream version will bump once they're fixed, NVRs are preserved, everything will work. The only critical thing is that we didn't bump the R in a non-forward thinking fashion, which is why the use of the .armX. > > - the hub and builders have been successfully set up and tested with > > patched koji/rpm/yum etc. I think that should be fairly straightforward now. Dennis has patches that he has tested in a staging Koji setup so they ought to be working. > > Work on all these pieces is proceeding in parallel (assuming that > > someone's on the gcc/glibc piece (who?)). We're not blocking on any one > > piece, but we have to finish these four before hitting 'Go'. Respectfully, I disagree that we need to block on all of them. I think we can begin building, and in parallel upstream whatever can be upstreamed, ensuring that for the one or two possible exceptions we have a plan for F16/F17 along the lines of "in gcc trunk", etc. > The final patch for gcc, for instance, may never make it into F15. Even > the armv7hl spec file changes we pushed upstream are only in .fc16 koji > builds at this point. If these changes are in FC16 or rawhide but not > F15, isn't that good enough? Why not build packages in parallel too? Well, Chris is naturally going to be concerned about building updates, and there is a sense of more correctness in blocking, however I think: 1). We need to resolve a few more things before building updates, but we buy time to do that and win now by building sooner while we handle that. 2). We are past the time when we should aim for the most complete F15 release. F13 was a wonderful example of a quality ARM release that the Seneca team (in particular here) should be proud of. BUT there is an opportunity here to leave all of these problems in the dust if we can just bring F15 to a good enough close and get to rawhide soon. I know, I sound pushy. I don't mean to. I'm not calling for the rocket to be launched for a manned flight before it's ready, I'm saying F15 is
Re: [fedora-arm] Koji status update
On Thu, 2011-11-17 at 21:48 -0500, Jon Masters wrote: > On Thu, 2011-11-17 at 15:03 -0800, Brendan Conoboy wrote: > > > - we've pruned the package sets back to the same core > > I think this is straightforward. We include in the repos only the build > requirements to build a buildroot and start with that. Do some closure > tests on that set and make sure the same packages exist for v5 and v7. I > suspect someone could turn that around tomorrow. Rewording this to make more sense, what I mean is: 1). Establish correctly named mirrors in Seneca of the arm and armhfp form, in which the exact same packages are present in both. Initially form this simply by removing packages not in both v5 and v7 now. That ought to be fairly simply (DJ's script output can help there). 2). With all of those packages possibly available, start by building only a buildroot and its deps. If that works, it's a great achievement. I would ideally like us to reach this point before US Thanksgiving takes a bunch of us offline this time next week. I think it's doable. Jon. ___ arm mailing list arm@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/arm
Re: [fedora-arm] Koji status update
On Thu, 2011-11-17 at 21:55 -0500, Jon Masters wrote: > On Thu, 2011-11-17 at 21:48 -0500, Jon Masters wrote: > > On Thu, 2011-11-17 at 15:03 -0800, Brendan Conoboy wrote: > > > > > - we've pruned the package sets back to the same core > > > > I think this is straightforward. We include in the repos only the build > > requirements to build a buildroot and start with that. Do some closure > > tests on that set and make sure the same packages exist for v5 and v7. I > > suspect someone could turn that around tomorrow. > > Rewording this to make more sense, what I mean is: > > 1). Establish correctly named mirrors in Seneca of the arm and armhfp > form, in which the exact same packages are present in both. Initially > form this simply by removing packages not in both v5 and v7 now. That > ought to be fairly simply (DJ's script output can help there). > > 2). With all of those packages possibly available, start by building > only a buildroot and its deps. If that works, it's a great achievement. > I would ideally like us to reach this point before US Thanksgiving takes > a bunch of us offline this time next week. I think it's doable. 3). With a minimal buildroot (a subset of stage3, which was to get mock working but wound up also making a bootable system - though stage3 is fine too for simplicity as a set to built initially) set of packages built, set that up as a new repo and have it favored in preference to the external one for building the rest. Then a mass rebuild of packages present in v5 and v7 from the external repos can begin. Jon. ___ arm mailing list arm@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/arm
[fedora-arm] Matching SRPMs so far (was Koji status update)
I've taken a moment to ask one of my pretty chart scripts to dump a list of all SRPMs that are nvr-identical in both v5 and v7 repos-so-far. I put the list here: http://djdelorie.fedorapeople.org/arm-v5-v7-same.txt The list doesn't include a few key SRPMs we know we'll need, though, like kernel, gcc, glibc, and PackageKit. ___ arm mailing list arm@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/arm
Re: [fedora-arm] Matching SRPMs so far (was Koji status update)
On Thu, 2011-11-17 at 22:21 -0500, DJ Delorie wrote: > I've taken a moment to ask one of my pretty chart scripts to dump a > list of all SRPMs that are nvr-identical in both v5 and v7 > repos-so-far. I put the list here: > > http://djdelorie.fedorapeople.org/arm-v5-v7-same.txt > > The list doesn't include a few key SRPMs we know we'll need, though, > like kernel, gcc, glibc, and PackageKit. Thanks. I think the following would be a good procedure to use: 1). For each of kernel, gcc, glibc, PackageKit the higher numbered version from either armv5tel or armv7hl should be used and thus the one on the other architecture should be rebuilt. We could cheat with the fake kernel deps option again for kernel-headers if kernel is a pain. I don't think PackageKit can be too much trouble to unify, and gcc and glibc ought to just be build cycles - anyone got time to build them? 2). For each of the resulting packages from the list + 4 copy the original versions from the stage4 repos into "arm" and "armhfp" external repo mirrors within Seneca to use as input for Koji. 3). Build the stage3 package set. 4). Create a repo containing the rebuilt stage3 package set, now rebuilt properly and more guaranteed to be Fedora-ified. Setup a higher prio repo containing these versions and add it as an external repo. 5). Build the remainder of the packages in the list+4 that can build. 6). Resolve failures through ongoing continued integration of deltas between v5 and v7 in the background. Jon. ___ arm mailing list arm@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/arm
Re: [fedora-arm] Matching SRPMs so far (was Koji status update)
Since I love replying to myself...I'm thinking out loud here because I want us to have an agreed series of simple instructions that reduce this final phase down to a straightforward exercise. I know Dennis thinks the unified approach is a little overkill and there is an alternative based on close enough, etc. That's cool too. What I care about is what gets us to quickly build effectively stage3 again in Koji, and move from there. Can I recommend therefore that we have a discussion here on the list, come to a consensus around what list of packages where and how, etc. then write out the steps in a very straightforward fashion. Otherwise, we'll have a lot of chat on IRC but we'll run the risk of being at the same point later because we didn't make sure we're all aligned. Technically, I'm out of the office on Friday. Technically ;) Jon. ___ arm mailing list arm@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/arm
Re: [fedora-arm] Koji status update
El Thu, 17 Nov 2011 21:48:06 -0500 Jon Masters escribió: > On Thu, 2011-11-17 at 15:03 -0800, Brendan Conoboy wrote: > > On 11/17/2011 11:04 AM, Chris Tyler wrote: > > > I agree with going with what we've currently got package-wise. But > > > saying "we should start building today" or "get Koji > > > running...tomorrow" is missing the point. > > :) I'm not so sure. I think the goal should be to get to rawhide as > quickly as possible because it makes all of these problems go away - > no rebuilding retrospectively, not being treated quite so much as a > secondary, and so on (then we push for primary). Given that that is > our general ambition, we should work backward and try to get to that > point as quickly as we can without being too hackish in our approach. I agree this is the goal. we get there quickest by building the things we can today while we resolve the harder issues at the same time. > > > We're ready to go the moment we have finished these tasks: > > > - the gcc/glibc issues are resolved > > Let's split these in two: > > 1). gcc. Having seen some of the discussion here with Jakub, etc. it's > clear that the Fedora maintainers aren't going to accept this latest > volatile bitfields patch unless it's in the upstream 4.6 branch (it is > in trunk right now). So there are two upstreams to work with - Fedora, > and GCC. We can try to pursue this, but it is not a quick fix (we're > looking at several weeks to turn it around). At the same time, this > stuff is resolved in rawhide (and I think F16). It would be more > expedient to block updates to gcc being pulled in automatically (which > won't happen at this stage anyway) and either in parallel fix this or > undertake to keep a separate gcc. Really, by the time we release F15 > we'll have only 5 months of time when there would be updates anyway. > > 2). glibc. Seems there's a backport that could perhaps be upstreamed, > and a Java fix that we will need. Again, I favor parallelization. We > build now and try to find a solution that involves either getting this > into upstream Fedora 15 PA or finding an alternative. I don't see a > reason not to build this now though. > > > > - we've pruned the package sets back to the same core > > I think this is straightforward. We include in the repos only the > build requirements to build a buildroot and start with that. Do some > closure tests on that set and make sure the same packages exist for > v5 and v7. I suspect someone could turn that around tomorrow. there is no need to do any pruning of either package set. after some further examiniation of the code koji doesnt enforce the strict matching nvrs in external repos that it does in the repos for packages built in koji. so what this means is that as long as we are pretty close we are good to go. at the end we will have a matching package set on both arches because koji is really good at enforcing that. so what we need is to have a mirror of stage4 hfp at http://australia.proximity.on.ca/fedora-arm/15/armhfp/ we need to get koji updated to the scratch builds i did today. there is an issue that cropped up that im working on getting a propper fix upstream in koji, but that will not make any of the build we do today invalid. we are close enough in package set that we will not be getting wild soname differences between the two. thats the main reason we need them close enough. ive got koji setup to build things correctly in dist-f15 what we need now is 1) update koji everywhere to correctly deal with hard and soft float. 2) repo for armhfp 3) generate a newRepo for dist-f15-build 4) start building. anything that has a different nvr between hfp and sfp we should take the higher nvr. if the arm fix is already in fedora git we should just build that package from git. we should not build anything that has the 0.arm etc in it in koji we should build from git the fixed version. lets build everything in koji we have already built in stage4 and moji. at the end of doing the builds we will have a extremly solid foundation. gcc/glibc can get fixed while we build everything else. > > > - every change/patch in those package sets has been upstreamed > > I was thinking that way, but now I think that is a parallel activity. > I am all for getting stuff upstreamed, and it will be especially > important in the coming months, but I think that doesn't need to gate > the initial build. Packages can keep an armX suffix in the initial > build, then for the few dozen (after cutting down the initial build > set) that need to be upstreamed, there will be a higher NVR from > upstream soon anyway that can be rebuilt for both arches with the > upstream version. parallelise everything we can submitting the builds and filling koji is really only going to take one person. as soon as we have most everything built we can start on rawhide. i suspect we will have a mass rebuild for f17 early in the new year. we want to be ready for that. > IOW I think the initial build should proc
Re: [fedora-arm] Matching SRPMs so far (was Koji status update)
El Thu, 17 Nov 2011 22:42:07 -0500 Jon Masters escribió: > Since I love replying to myself...I'm thinking out loud here because I > want us to have an agreed series of simple instructions that reduce > this final phase down to a straightforward exercise. I know Dennis > thinks the unified approach is a little overkill and there is an > alternative based on close enough, etc. That's cool too. What I care > about is what gets us to quickly build effectively stage3 again in > Koji, and move from there. > > Can I recommend therefore that we have a discussion here on the list, > come to a consensus around what list of packages where and how, etc. > then write out the steps in a very straightforward fashion. Otherwise, > we'll have a lot of chat on IRC but we'll run the risk of being at the > same point later because we didn't make sure we're all aligned. > > Technically, I'm out of the office on Friday. Technically ;) building the packages that land in every buildroot first makes sense. I'm ok with getting that done. the beauty of the way things work is that as we build the package in koji it gets exlcuded from the external repo. so there will be no way that once we have the package built and have the unified version in koji that we accidently use the wrong version from the external repo. we already have all the f15 ga noarch rpms imported into koji and tagged into dist-f15. i will be taking all of next week off. i will be sparodically doing some work on the weekend. but next week will be pretty much completely off-line getting in some needed downtime. Dennis ___ arm mailing list arm@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/arm
[fedora-arm] FORTIFY_SOURCE
Hi Chris, There was some dialog on IRC recently about compile flags. As near as I can see it, we ought to be consistent in r-r-c between v5 and v7. Did I miss something in particular? Just tying up some loose ends here. Jon. ___ arm mailing list arm@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/arm
