Re: RFC: upstreaming debian/patches/exec_filename_* and the dde stuff
Indeed, few of the patches at http://patch-tracker.debian.org/package/hurd/1:0.5.git20140326-1 look Debian-specific. For features that are not “fully baked” yet, like DDE, wouldn’t it make sense to have a branch in the Hurd repo, instead of a set of patches in Debian? As for patches for which review “never completed”, there should really be a timeout IMO, and the patch should be either rejected for everyone (including Debian), or accepted, or amended. My 2¢, Ludo’.
Re: RFC: upstreaming debian/patches/exec_filename_* and the dde stuff
Ludovic Courtès, le Sun 06 Apr 2014 22:44:12 +0200, a écrit : Indeed, few of the patches at http://patch-tracker.debian.org/package/hurd/1:0.5.git20140326-1 look Debian-specific. For features that are not “fully baked” yet, like DDE, wouldn’t it make sense to have a branch in the Hurd repo, instead of a set of patches in Debian? Well, it's *already* a branch, in the incubator. There are some patches over that branch which lie in the Debian, which I haven't applied to the incubator because they make our DDE copy diverge even more from what was taken directly from upstream, while we'd like at some point to just push them upstream so they can profit everybody, but upstream has not really opened still. As for patches for which review “never completed”, there should really be a timeout IMO, and the patch should be either rejected for everyone (including Debian), or accepted, or amended. It's a matter of doing it, yes. Samuel
Re: Upstreaming patches [Was: RFC: upstreaming debian/patches/exec_filename_* and the dde stuff]
Put yet another way, if you ask me why isn't foo done? I'll most probably just answer because I haven't had time to do it, and nobody else took the time to do it. If you ask me could foo be done then?, I'll answer sure, patch welcome. If you ask me I've had a look, we could just apply this patch, move that, do this, and then we get foo done, then the time for me to actually do it becomes almost zero, and thus I'll happily do it. Yes, it means spending time on how to do it, but if nobody else spends it, I'll have to do it, but god knows when I'll find time to do it among all other urging matters. Samuel
Re: Upstreaming patches [Was: RFC: upstreaming debian/patches/exec_filename_* and the dde stuff]
On Mon, Apr 7, 2014 at 4:52 PM, Samuel Thibault samuel.thiba...@gnu.orgwrote: Put yet another way, if you ask me why isn't foo done? I'll most probably just answer because I haven't had time to do it, and nobody else took the time to do it. If you ask me could foo be done then?, I'll answer sure, patch welcome. If you ask me I've had a look, we could just apply this patch, move that, do this, and then we get foo done, then the time for me to actually do it becomes almost zero, and thus I'll happily do it. Yes, it means spending time on how to do it, but if nobody else spends it, I'll have to do it, but god knows when I'll find time to do it among all other urging matters. It's a time matter, and have improve space, I can understand how this is a challenging work to merge and adjust so much patches and the branches. The one who do this need have the full plan, and the big picture in the head, and done one by one carefully and leader by the initial big picture, hard as big refactor. we may need some level of frozen, to make this work easy if it's very important. The patch review mode maybe not suitable for this work, it's not minor improve. How about we make a upstream branch and do fast iterator merge and other stuff, we review this branch but not the patches, when it's OK, merge the branch to the main upstream branch or just use it as the main upstream branch? Thanks Cong Zhang
Re: Upstreaming patches [Was: RFC: upstreaming debian/patches/exec_filename_* and the dde stuff]
Zhang Cong, le Mon 07 Apr 2014 17:25:34 +0800, a écrit : The one who do this need have the full plan, I don't see why one would need a full plan. A patch deals with just part of the code, you don't need to know *all* the code to review the patch, ping people about what is wrong with it, clean it, etc. How about we make a upstream branch and do fast iterator merge and other stuff, we review this branch but not the patches, when it's OK, merge the branch to the main upstream branch or just use it as the main upstream branch? Be it a branch or patches lying in Debian package is just the same. See DDE, it *is* a branch already. But if nobody works on it, well things don't happen magically, it's as simple as that :) Samuel
Re: Upstreaming patches [Was: RFC: upstreaming debian/patches/exec_filename_* and the dde stuff]
Samuel Thibault, le Mon 07 Apr 2014 11:31:33 +0200, a écrit : Zhang Cong, le Mon 07 Apr 2014 17:25:34 +0800, a écrit : The one who do this need have the full plan, I don't see why one would need a full plan. Putting it another way, I don't think anybody has the full plan. Which is fine, all medium-to-big projects are like that. Samuel
Re: Upstreaming patches [Was: RFC: upstreaming debian/patches/exec_filename_* and the dde stuff]
On Mon, Apr 7, 2014 at 5:31 PM, Samuel Thibault samuel.thiba...@gnu.orgwrote: Zhang Cong, le Mon 07 Apr 2014 17:25:34 +0800, a écrit : The one who do this need have the full plan, I don't see why one would need a full plan. A patch deals with just part of the code, you don't need to know *all* the code to review the patch, ping people about what is wrong with it, clean it, etc. I reserve my opinion here. How about we make a upstream branch and do fast iterator merge and other stuff, we review this branch but not the patches, when it's OK, merge the branch to the main upstream branch or just use it as the main upstream branch? Be it a branch or patches lying in Debian package is just the same. See DDE, it *is* a branch already. But if nobody works on it, well things don't happen magically, it's as simple as that :) I don't think so, Debian hurd was the main hurd downstream, and debian hurd drive the hurd development. So, can I mark the dde as needed for hurd even it is experiment, and, commit it and just add some readme like info is enough? why should we make ourselves so hard to maintain our code when no other people care about that? Separate it can not help the development process, but merge the repo will, and make the build and test process easy will make us newbie friendly :)
Re: Upstreaming patches [Was: RFC: upstreaming debian/patches/exec_filename_* and the dde stuff]
On Mon, Apr 7, 2014 at 5:32 PM, Samuel Thibault samuel.thiba...@gnu.orgwrote: Samuel Thibault, le Mon 07 Apr 2014 11:31:33 +0200, a écrit : Zhang Cong, le Mon 07 Apr 2014 17:25:34 +0800, a écrit : The one who do this need have the full plan, I don't see why one would need a full plan. Putting it another way, I don't think anybody has the full plan. Which is fine, all medium-to-big projects are like that. Driver related idea may need a full plan:) Cong Zhang
Re: Upstreaming patches [Was: RFC: upstreaming debian/patches/exec_filename_* and the dde stuff]
Zhang Cong, le Mon 07 Apr 2014 19:36:02 +0800, a écrit : On Mon, Apr 7, 2014 at 5:32 PM, Samuel Thibault samuel.thiba...@gnu.org wrote: Samuel Thibault, le Mon 07 Apr 2014 11:31:33 +0200, a écrit : Zhang Cong, le Mon 07 Apr 2014 17:25:34 +0800, a écrit : The one who do this need have the full plan, I don't see why one would need a full plan. Putting it another way, I don't think anybody has the full plan. Which is fine, all medium-to-big projects are like that. Driver related idea may need a full plan:) Again, no. Drivers can work the way they prefer. The driver infrastructure itself doesn't need a bigplan, it is parts of it which need their own. For instance, the IRQ issue I mentioned has its plan by itself, and it doesn't need to interfere with the physical memory allocation issue. Samuel
Re: Upstreaming patches [Was: RFC: upstreaming debian/patches/exec_filename_* and the dde stuff]
On Mon, Apr 7, 2014 at 7:43 PM, Samuel Thibault samuel.thiba...@gnu.orgwrote: Again, no. Drivers can work the way they prefer. The driver infrastructure itself doesn't need a bigplan, it is parts of it which need their own. For instance, the IRQ issue I mentioned has its plan by itself, and it doesn't need to interfere with the physical memory allocation issue. That's not sure, unless we have a plenty of driver works, we may need adjust the infrastructure for the need or some new abstract . Although we have driver infrastructure, no enough third part driver provider now. The audio driver and video driver may be part of hurd at first ( just on repo's view), at least some high level abstract, this need a plan. Cong Zhang
Re: Upstreaming patches [Was: RFC: upstreaming debian/patches/exec_filename_* and the dde stuff]
Zhang Cong, le Mon 07 Apr 2014 20:42:04 +0800, a écrit : On Mon, Apr 7, 2014 at 7:43 PM, Samuel Thibault samuel.thiba...@gnu.org wrote: Again, no. Drivers can work the way they prefer. The driver infrastructure itself doesn't need a bigplan, it is parts of it which need their own. For instance, the IRQ issue I mentioned has its plan by itself, and it doesn't need to interfere with the physical memory allocation issue. That's not sure, unless we have a plenty of driver works, we may need adjust the infrastructure for the need or some new abstract . Yes, but that new abstract will be independant from other matters concerning drivers. Although we have driver infrastructure, no enough third part driver provider now. The audio driver and video driver may be part of hurd at first ( just on repo's view), at least some high level abstract, this need a plan. Sure. You need a plan for audio, a plan for video. But you don't need a plan for both audio video at the same time, except some general Hurdish principles, but that's not big. Samuel
Re: Upstreaming patches [Was: RFC: upstreaming debian/patches/exec_filename_* and the dde stuff]
On Mon, Apr 7, 2014 at 8:55 PM, Samuel Thibault samuel.thiba...@gnu.orgwrote: Zhang Cong, le Mon 07 Apr 2014 20:42:04 +0800, a écrit : On Mon, Apr 7, 2014 at 7:43 PM, Samuel Thibault samuel.thiba...@gnu.org wrote: Again, no. Drivers can work the way they prefer. The driver infrastructure itself doesn't need a bigplan, it is parts of it which need their own. For instance, the IRQ issue I mentioned has its plan by itself, and it doesn't need to interfere with the physical memory allocation issue. That's not sure, unless we have a plenty of driver works, we may need adjust the infrastructure for the need or some new abstract . Yes, but that new abstract will be independant from other matters concerning drivers. Although we have driver infrastructure, no enough third part driver provider now. The audio driver and video driver may be part of hurd at first ( just on repo's view), at least some high level abstract, this need a plan. Sure. You need a plan for audio, a plan for video. But you don't need a plan for both audio video at the same time, except some general Hurdish principles, but that's not big. OK, not big is a good message:) Thanks, Cong Zhang
RFC: upstreaming debian/patches/exec_filename_* and the dde stuff
Hi, I'd like to see more of the debian/patches merged, especially the exec_filename_* series. These patches cover a lot of files, and they often break and I have to amend them manually. More often than not the fix is trivial, but it is still a nuisance. I'd also like to see the dde stuff from the incubator repository merged upstream. 1/ Code that does not live in the Hurd repository receives less attention. I've been doing a lot of cleanups, static analysis and performance improvements lately. The dde stuff for example has not received any of those. 2/ The nice Guix folks surely also want all those goodies, so they either have to copy those patches, or they miss out. In either way, I feel like we (Hurd upstream) would have failed them. 3/ I feel that this situation creates an unnecessary drag for the development. I'm working on protected payloads to improve the object lookups in the Hurd servers. I've just came up with a patch that fixes some object lookups in libports. This requires changing the API of libports. I fixed all the code in the Hurd repository accordingly, unfortunately the dde stuff breaks if it isn't fixed as well. If it was living in the same repository, I could fix it up in one commit, making sure that the change is atomic. So, is the dde stuff a first class citizen? If so, I believe it should be merged. If not, may I change the API of, say, libports freely? May I merge that patch that breaks the dde servers? If not, why not, and what is the best way to proceed? Cheers, Justus
Re: RFC: upstreaming debian/patches/exec_filename_* and the dde stuff
Hello, Justus Winter, le Sun 06 Apr 2014 19:58:45 +0200, a écrit : I'd like to see more of the debian/patches merged, especially the exec_filename_* series. The discussion about it with Roland was unfortunately not finished. I'd also like to see the dde stuff from the incubator repository merged upstream. For this one, I'd like to avoid having to let userland processes disable interrupts, since this brings IRQ sharing issues. may I change the API of, say, libports freely? Well, there is not only dde, but all incubator projects, as well as hurdextras, etc. what is the best way to proceed? You can commit the dde-related changes in the incubator at the same time, both will be pulled at the same time in Debian updates, for instance. Samuel
Re: RFC: upstreaming debian/patches/exec_filename_* and the dde stuff
Samuel Thibault, le Sun 06 Apr 2014 21:25:42 +0200, a écrit : I'd also like to see the dde stuff from the incubator repository merged upstream. For this one, I'd like to avoid having to let userland processes disable interrupts, since this brings IRQ sharing issues. (and the stack needs to be polished, for now it's quite drafty in the corners). Samuel
Re: RFC: upstreaming debian/patches/exec_filename_* and the dde stuff
As I said in the irc it would be really helpfull if the debian patches got merged with the upstream. Right now whenever I get an error, I am searching in them, handpicking the part that solves it. Actually some of the debian patches are necessary to built glibc and libpthread. I will report exactly which parts I needed from which patches, as soon as possible, so I can help Justus merge them. Manolis
Re: RFC: upstreaming debian/patches/exec_filename_* and the dde stuff
Hi, Quoting Samuel Thibault (2014-04-06 21:25:42) Justus Winter, le Sun 06 Apr 2014 19:58:45 +0200, a écrit : I'd like to see more of the debian/patches merged, especially the exec_filename_* series. The discussion about it with Roland was unfortunately not finished. Please enlighten me. I wasn't around when this was discussed, and I tried to find the discussion but failed. Meanwhile, I improved the exec_filename_* series and fixed the fakeroot translator. The fakeroot translator is somewhat less useful w/o that patch series, so I'd like to see this merged. I'd also like to see the dde stuff from the incubator repository merged upstream. For this one, I'd like to avoid having to let userland processes disable interrupts, since this brings IRQ sharing issues. Why does that prevent the merge? We have plenty of stuff in the repo that doesn't work, bitrots or is otherwise in a bad shape. If someone doesn't like the idea of letting userspace processes disable interrupts, let her just not use the dde drivers. may I change the API of, say, libports freely? Well, there is not only dde, but all incubator projects, as well as hurdextras, etc. what is the best way to proceed? You can commit the dde-related changes in the incubator at the same time, both will be pulled at the same time in Debian updates, for instance. How exactly is it that it is pulled in the Debian updates? Manually by you, I presume? There's one thing that really bugs me. I thought I stated this before, or even in the original mail, but maybe it didn't came across clearly. For me, building the Hurd is like driving rusty nails into ones knee, only less fun. It has caused me so much pain, that I created a continuous integration solution that builds Debian packages. ci is nice and stuff, but ultimately I just wanted to relieve the pain. Why is it painful? Because Debian/Hurd is patched beyond recognition. Clone the Hurd repository, build some component, say one of the core servers, put it into your Debian installation and voila, your system hangs at boot time. So you clone the Debian/Hurd packaging repo, which is a weird frankenstein-repo. The upstream repo is mixed with parts of the dde branch from the incubator, on top of that lives a long list of debian/patches, some of them also carry dde stuff in them. You patch some stuff, try to build it, and surely enough you need to refresh some debian/patches. See here for the difference of the Debian packaging repo and the hurd upstream repo: http://darnassus.sceen.net/gitweb/teythoon/packaging/hurd.git/tree So I got myself a solution for this problem. I automated every step that can possibly be automated, but I still have to amend the patches in there often, like this: http://darnassus.sceen.net/gitweb/teythoon/packaging/hurd.git/commitdiff/fe9c72e60a59109bff7558224871ac5e80b9f38d Building half a Hurd package takes ~30 minutes using my solution. My workflow goes roughly like this: 1. come up with a patch against hurd upstream 2. add it to the patch series in my debian packaging repo 3. start a build 4. see my patch break b/c some other debian/patch also touches some file that my new patch touches 5. amend either mine or the other patch 6. goto 3 I do this for every patch series I propose. Once the package is built, it is uploaded to my deb repo and my test vm automatically installs the new packages and verifies that the system still works. The amount of time I still spend on just building debian packages is mind-boggling. If I wasn't the masochist I am, I would have stopped doing that a long time ago. Also, the way it is now it will never be possible to fully-automatically build Debian packages from the Hurd repository. Today it got worse. The package built fine, but surely enough all the dde components broke apart at runtime b/c I changed the API of libports. The dde components that either are in the debian package repo or in the debian/patches in the debian package repo, I don't even know for sure. Sure, I can fix those components up in the incubator repo. It was mostly an awareness problem. But then what? How exactly are they then pulled into my build solution? How am I supposed to test that I didn't break anything? If I figure out how you create the patches from the incubator repo, I could do that too. In the end, we still end up duplicating the packaging effort. By the way, you'll need fe9c72e60a59 too for the next Hurd package. This has caused me so much pain. And I imagine it is even worse for new developers. Justus
Re: RFC: upstreaming debian/patches/exec_filename_* and the dde stuff
On 07/04/14 00:26, Justus Winter wrote: Hi, Quoting Samuel Thibault (2014-04-06 21:25:42) Justus Winter, le Sun 06 Apr 2014 19:58:45 +0200, a écrit : I'd like to see more of the debian/patches merged, especially the exec_filename_* series. The discussion about it with Roland was unfortunately not finished. Please enlighten me. I wasn't around when this was discussed, and I tried to find the discussion but failed. Meanwhile, I improved the exec_filename_* series and fixed the fakeroot translator. The fakeroot translator is somewhat less useful w/o that patch series, so I'd like to see this merged. See http://marc.info/?l=hurd-bugm=127543467330197 and the rest of that thread. Regards, Emilio
Re: RFC: upstreaming debian/patches/exec_filename_* and the dde stuff
Quoting Emilio Pozuelo Monfort (2014-04-07 00:38:28) On 07/04/14 00:26, Justus Winter wrote: Hi, Quoting Samuel Thibault (2014-04-06 21:25:42) Justus Winter, le Sun 06 Apr 2014 19:58:45 +0200, a écrit : I'd like to see more of the debian/patches merged, especially the exec_filename_* series. The discussion about it with Roland was unfortunately not finished. Please enlighten me. I wasn't around when this was discussed, and I tried to find the discussion but failed. Meanwhile, I improved the exec_filename_* series and fixed the fakeroot translator. The fakeroot translator is somewhat less useful w/o that patch series, so I'd like to see this merged. See http://marc.info/?l=hurd-bugm=127543467330197 and the rest of that thread. Seriously? Emilio, Jeremy, and Samuel think it's a worthwile addition, but Roland just isn't convinced it's worth doing? That has stalled the inclusion of this for three years? Well, if there is a reason to prolong my pain, this must be it. Justus
Upstreaming patches [Was: RFC: upstreaming debian/patches/exec_filename_* and the dde stuff]
Justus Winter, le Mon 07 Apr 2014 00:26:29 +0200, a écrit : The discussion about it with Roland was unfortunately not finished. Please enlighten me. I wasn't around when this was discussed, Well, I wasn't really into the discussion either actually. Pochu did the work. See his mails called like Add a file_exec_file_name RPC around Wed, 26 May 2010. I'd also like to see the dde stuff from the incubator repository merged upstream. For this one, I'd like to avoid having to let userland processes disable interrupts, since this brings IRQ sharing issues. Why does that prevent the merge? Because that's a quite important incompatible detail in the behavior of the introduced gnumach IRQ RPC. We could perhaps have a look at having a backward-compatibility way, so we can just commit what we have, and change it later on, it's a matter of having a look at it. You can commit the dde-related changes in the incubator at the same time, both will be pulled at the same time in Debian updates, for instance. How exactly is it that it is pulled in the Debian updates? Manually by you, I presume? Just the same way as pulling the main branch. There's a README file in the README branch of the repository. This has caused me so much pain. And I imagine it is even worse for new developers. I fully understand all that, it's not fun for me either, and I'd really love to find the time to fix all that. But for now apparently only I have taken the time to dive into Zheng Da's work on DDE, so it doesn't help. We could as well just commit what is there since it's relatively split appart in the source, so it at least doesn't put ugly hacks into other directories. But it really needs some polishing in the end. As for the two other stuff in the Debian hurd package (random, procfs), yes it is a ugly hack, and I'd rather avoid it. It was just a way to get working /dev/random and /proc so people can ran a debian system at all. Of course we should think about just merging them into the main repository, or build them another way, etc. I for myself have never found the time to even just think about it. I end up spending my time mostly fixing pushing more-or-less-baked patches to Debian packages, so people at least get to try them, but the polishing+submitting work is much more work, and if nobody gives help there, well things can only go slowly for sure. There's a question of priority. For instance, should have I spent my time the other day on DDE / exec_filename / pushing some glibc patches upstream, or on fixing the pthread_condattr_setclock() so glib2.0 can continue working at all? Maybe I should just not do this kind of ugly pushing, so Debian doesn't get them, and thus people would have to really push their changes in properly, and not be happy with just seeing them applied to Debian. In the end, I'd essentially say Patch Thankfully Welcome, like they do so often on the cygwin mailing list. Of course, that means baked patches, which is quite some work, but the Hurd project is no exception to this requirement. If some discussion died just because a patch was not reviewed, then it's a matter of reviving it. I happen to have dozens of mails in my hurd inbox, I've just never found the time to revive their discussion. If nobody else helps on that, it'll continue getting dust. Put another way, please people just dive into patches, check why they are not upstream (just ask if it's not written in there), and have a look at how to fix that. I don't think there is anything strongly blocking anywhere, it's just a matter of doing things, so that things get done... Samuel
Re: RFC: upstreaming debian/patches/exec_filename_* and the dde stuff
Justus Winter, le Mon 07 Apr 2014 00:53:40 +0200, a écrit : Quoting Emilio Pozuelo Monfort (2014-04-07 00:38:28) On 07/04/14 00:26, Justus Winter wrote: Hi, Quoting Samuel Thibault (2014-04-06 21:25:42) Justus Winter, le Sun 06 Apr 2014 19:58:45 +0200, a écrit : I'd like to see more of the debian/patches merged, especially the exec_filename_* series. The discussion about it with Roland was unfortunately not finished. Please enlighten me. I wasn't around when this was discussed, and I tried to find the discussion but failed. Meanwhile, I improved the exec_filename_* series and fixed the fakeroot translator. The fakeroot translator is somewhat less useful w/o that patch series, so I'd like to see this merged. See http://marc.info/?l=hurd-bugm=127543467330197 and the rest of that thread. Seriously? Emilio, Jeremy, and Samuel think it's a worthwile addition, but Roland just isn't convinced it's worth doing? That has stalled the inclusion of this for three years? Nobody took the time to discuss with Roland, indeed. Samuel
Re: Upstreaming patches [Was: RFC: upstreaming debian/patches/exec_filename_* and the dde stuff]
Samuel Thibault, le Mon 07 Apr 2014 01:02:56 +0200, a écrit : Put another way, please people just dive into patches, Put yet another way, if something is eating time in your workflow for no apparently good reason, there was probably no good reason except lack of time for doing things better, so just spend time on saving that time, and everybody will thank you. Samuel
