Re: call for ftpmaster's transparency
> "Sean" == Sean Whitton writes: >> Also you never know how long your package will stay in the NEW >> queue and during this time lack of ITP could affect developers >> priorities. Sean> Well, I always look at the current NEW queue before packaging Sean> something. I do not. I trust the wnpp package to be reasonably accurate.
Re: call for ftpmaster's transparency
Hello, On Mon 10 Feb 2020 at 04:29PM +11, Dmitry Smirnov wrote: > No. ITPs are opportunities to team up with others, not merely for de- > duplication. For many small packages there is simply no need for teaming up at the stage of uploading to NEW. > That might be a valid situation but nevertheless how much effort it is to > file an ITP?? Not much even if filing new ITP is not mandated. > But when ITP is there, then it could be referred to as "blocked by" or > "blocking" bug, it can have "affects" relationships with other packages, etc. I find it overly effortful. > Also you never know how long your package will stay in the NEW queue and > during this time lack of ITP could affect developers priorities. Well, I always look at the current NEW queue before packaging something. -- Sean Whitton signature.asc Description: PGP signature
Re: call for ftpmaster's transparency
On Monday, 10 February 2020 1:01:05 PM AEDT Sean Whitton wrote: > AIUI, the reason REJECT comments aren't public is because it might > sometimes make people feel embarassed. Then many reviews of a packaging work that is done by mentors would be embarrassing but that's OK because everybody have a chance to learn from those reviews. (Only those who don't do anything are never embarrassed). > ITPs are great for avoiding duplicated effort in most cases. However, > there are cases in which it is possible for someone to know pretty much > for sure that there is no chance of any duplicated effort. In such > cases ITPs are busywork, which is demotivating to volunteers. No. ITPs are opportunities to team up with others, not merely for de- duplication. > For example, if I break out some mature code from a project and make its > first upstream release as an independent library, and then I want > immediately to upload it to NEW so that the next release of the project > it was broken out from can depend on the new library, there is no reason > to file an ITP. Since I am the upstream author and the code has only > just been released, I can be confident no-one else is going to try to > package it. That might be a valid situation but nevertheless how much effort it is to file an ITP?? Not much even if filing new ITP is not mandated. But when ITP is there, then it could be referred to as "blocked by" or "blocking" bug, it can have "affects" relationships with other packages, etc. Also you never know how long your package will stay in the NEW queue and during this time lack of ITP could affect developers priorities. > Another example is the Haskell team. Due to the nature of the language, > information on newly packaged libraries has to be committed to two > different git repos, and so everyone working on Haskell in Debian is > working from those two repos. So again, no real danger of duplicated > effort. I've heard something similar about Rust team. Fair enough, maybe there are some legitimate cases when ITPs could be safely avoided. Though it is nice to have an ITP anyway as a record of what is being introduced -- for example when package is rejected, ITP could capture discussion and reason for rejection, which is always useful to have for posterity. -- Best wishes, Dmitry Smirnov. --- Richard Nixon got kicked out of Washington for tapping one hotel suite. Today we're tapping every American citizen in the country, and no one has been put on trial for it or even investigated. We don't even have an inquiry into it. -- Edward Snowden signature.asc Description: This is a digitally signed message part.
Re: call for ftpmaster's transparency
On Sunday, 9 February 2020 9:04:25 PM AEDT Michael Lustfield wrote: > This is an understandable perspective, but secrecy probably isn't the best > word. Probably. If I had a better linguistic faculties then I could have find a better word. But I have had to use what I was available... > I personally think this sounds like a fantastic (and not very difficult) > idea. Thank you. > Where do you propose the bug mail be sent for NEW/binNEW packages without > an ITP? Same channels as usual. How can one comment to ITP that does not exist? > I suspect when you say, "member of ftp-masters team," what you mean is > "FTP-Masters Trainee." Correct. > I agree that it could be valuable to see comments; however, they're almost > always going to be from Trainees. Since we're not technically part of the > team, it's important that we don't speak on behalf of the team. Publishing > Trainee comments would effectively be doing that. That's perfectly fine. I don't recall a single case when a package review was not appreciated. A review or even a question asked on ITP can be useful to correct a problem or to provide more background. Whether ITP feedback if provided by Trainee or not, it could be useful anyway. > I would personally *LOVE* to see ITPs be a requirement for *ALL* new > packages. Making it a requirement and expecting ftp-masters to ignore any > upload until the ITP has existed for at least X days would be absolutely > fantastic. It would fix some redundant library uploads (see > golang/nodejs/etc.) and it would provide a mandatory level of review by > the wider community. I'm sure having it as a good practice would be enough without mandating a strong requirement to always have an ITP. There are might be legitimate cases to not file an ITP although I can think of only one such case when a source package is re-named... > Back when I tried to get gitea packaged for main, I had a number of ITPs > commented/closed mentioning the alternate library name or a reason it can't > be packaged. Makes sense. > > I'd like Debian project leader to engage in the matter of improving > > transparency of ftp-masters team operations and procedures. > > This feels a lot like starting a GR and not allowing appropriate > discussion. It's heavy-handed, isn't going to get anywhere, and is going > to hurt feelings. Project leader's duty is to facilitate communication. It is not wrong to at least make him aware of the problem. > > I want to encourage a public discussion regarding opening of the > > ftp-master mail list to the public. Currently reasons for unjustified > > secrecy of ftp- master processes is not explained... > > It's often said that emotions don't play well with productive discussions. > Adding phrases such as "where it belong", using "secrecy" over "privacy", > calling it "unjustified", and immediately jumping to demands of the DPL are > accusatory and inflammatory, and will likely just get you ignored or start > an unproductive flame war. It is my general observation that bug reporters (or those who raise concerns on mail lists) naturally tend to be perceived over-emotionally. That's understandable because they are either affected by the issue or concerned enough to report it. And others probably take it less seriously or they would have reported it themselves... Once again, "do not shoot the pianist" sort of speak... I've expressed the issue the best I could. > Why do reviews take so long? I'm not concerned about that, although it would be great to improve processing time. IMHO the bigger problem is that queue processing is unpredictable -- some packages, sometimes unimportant ones are processed very fast while others (including those that block some bug fixes) can stay in the queue for a very long time. It is very difficult to communicate urgency of a new package upload with ftpmasters. I'd probably use severity of pending ITP bug but I'm not sure if that would be effective or even the right thing to do... > - The team is tiny IMHO this is a very serious issue. There are too few ftp-masters, they are doing too much work, most certainly not delegating enough and not growing the team... > - Much of the team seems very burned out No wonder given the weight of responsibilities. I'm sure we are all much appreciate their hard work... -- Regards, Dmitry Smirnov. --- We occasionally stumble over the truth but most of us pick ourselves up and hurry off as if nothing had happened. -- Winston Churchill signature.asc Description: This is a digitally signed message part.
Re: call for ftpmaster's transparency
On Sun, Feb 09, 2020 at 07:01:05PM -0700, Sean Whitton wrote: > One key problem with the current workflow is that it makes it very > difficult to avoid reviewing identical files more than once. That would > be a big improvement. (I was just talking with Michael about this several minutes ago.) Just leaking a part of my WIP work. My core data structure looks like this {path: [hash, stamp, username, status, annotation]} The "hash " field is a salted hash, calculated like this hash(data=read(path), salt=read(neighbor_license())) This data structure is a fine-grained (per-path level) "accept/reject" record. Each path is a node. The "status" of a tree can be automatically computed form its decendant nodes. When a package enters NEW again, files with matching hashes will automatically reuse the last status assigned by human user, where status = either "accept" or "reject". There are still many other aspects from which I can reduce time consumption for human and improve efficiency.
Re: call for ftpmaster's transparency
Hello, On Sun 09 Feb 2020 at 04:04AM -06, Michael Lustfield wrote: >> To make matters worse ftp-masters rarely leave their comments in ITP >> issues. As I've recently learned that have profound effect on processing of >> new packages. > > I personally think this sounds like a fantastic (and not very difficult) idea. AIUI, the reason REJECT comments aren't public is because it might sometimes make people feel embarassed. > I would personally *LOVE* to see ITPs be a requirement for *ALL* new packages. > Making it a requirement and expecting ftp-masters to ignore any upload until > the ITP has existed for at least X days would be absolutely fantastic. It > would > fix some redundant library uploads (see golang/nodejs/etc.) and it would > provide a mandatory level of review by the wider community. ITPs are great for avoiding duplicated effort in most cases. However, there are cases in which it is possible for someone to know pretty much for sure that there is no chance of any duplicated effort. In such cases ITPs are busywork, which is demotivating to volunteers. For example, if I break out some mature code from a project and make its first upstream release as an independent library, and then I want immediately to upload it to NEW so that the next release of the project it was broken out from can depend on the new library, there is no reason to file an ITP. Since I am the upstream author and the code has only just been released, I can be confident no-one else is going to try to package it. Another example is the Haskell team. Due to the nature of the language, information on newly packaged libraries has to be committed to two different git repos, and so everyone working on Haskell in Debian is working from those two repos. So again, no real danger of duplicated effort. > Why do reviews take so long? > - The team is tiny > - Much of the team seems very burned out > - The ones that are active tend to stick to source or "unloved" tasks > - There are some very large and/or messy packages that need review > - There are a lot of redundant tasks and frequently-made mistakes > + A little more automation could help that > - (my opinion) The tools are archaic, cumbersome, and inefficient > + Fixing this would be a very (non-technically) difficult task > + An idea I have would help to bring transparency to the process... > ^ it's missing an interest requirement :( One key problem with the current workflow is that it makes it very difficult to avoid reviewing identical files more than once. That would be a big improvement. -- Sean Whitton signature.asc Description: PGP signature
Re: call for ftpmaster's transparency
Le 09/02/2020 à 18:12, gregor herrmann a écrit : > On Sun, 09 Feb 2020 04:04:25 -0600, Michael Lustfield wrote: > >> I would personally *LOVE* to see ITPs be a requirement for *ALL* new >> packages. > > Fine with me. > >> Making it a requirement and expecting ftp-masters to ignore any upload until >> the ITP has existed for at least X days would be absolutely fantastic. > > Ehm, please, no. > I would find it highly interruptive for my work if I'd have to wait > for X days. +1: don't add another delay for NEW queue! >> It would >> fix some redundant library uploads (see golang/nodejs/etc.) and it would >> provide a mandatory level of review by the wider community. >> Back when I tried to get gitea packaged for main, I had a number of ITPs >> commented/closed mentioning the alternate library name or a reason it can't >> be >> packaged. > > Maybe that's helpful for some teams, in the perl team our tools > (dh-make-perl in particular) check for existing packages and existing > wnpp bugs. Same for JS Team, our npm2deb tool shows if library already exists >> Why do reviews take so long? > > As a side note: Not all reviews take long, there's seems to be quite > some variance in the time they take. > > > Cheers, > gregor, who's usually very happy with the turnaround time of > NEW packages Same when I'm working in Perl Team, but not when I'm packaging Node.js modules :-/ Cheers, Xavier
Re: call for ftpmaster's transparency
On Sun, 09 Feb 2020 04:04:25 -0600, Michael Lustfield wrote: > I would personally *LOVE* to see ITPs be a requirement for *ALL* new packages. Fine with me. > Making it a requirement and expecting ftp-masters to ignore any upload until > the ITP has existed for at least X days would be absolutely fantastic. Ehm, please, no. I would find it highly interruptive for my work if I'd have to wait for X days. > It would > fix some redundant library uploads (see golang/nodejs/etc.) and it would > provide a mandatory level of review by the wider community. > Back when I tried to get gitea packaged for main, I had a number of ITPs > commented/closed mentioning the alternate library name or a reason it can't be > packaged. Maybe that's helpful for some teams, in the perl team our tools (dh-make-perl in particular) check for existing packages and existing wnpp bugs. > Why do reviews take so long? As a side note: Not all reviews take long, there's seems to be quite some variance in the time they take. Cheers, gregor, who's usually very happy with the turnaround time of NEW packages -- .''`. https://info.comodo.priv.at -- Debian Developer https://www.debian.org : :' : OpenPGP fingerprint D1E1 316E 93A7 60A8 104D 85FA BB3A 6801 8649 AA06 `. `' Member VIBE!AT & SPI Inc. -- Supporter Free Software Foundation Europe `- NP: Ry Cooder: Poor Man's Shangri-La signature.asc Description: Digital Signature
Re: call for ftpmaster's transparency
Hi Niels, On Sun, Feb 09, 2020 at 11:22:46AM +0100, Niels Thykier wrote: > For the parts involving tooling, are there bugs/salsa issues describing > the issue so a "non-FTP-team"-member can take a stab at fixing them? First of all the major problem we are talking about, that the reviewing process being so slow, is not well defined. Because what this is going to optimize is not the machine code, but the human's working procedure. So it's not easy for people to open a bug and describe the definite problem they have found. For instance, bugs saying "hi dak devs, I found the tool not friendly enough for human efficiency, please fix it" are very likely terrible bug reports. Something new needs to be invented (and eventually incorporated into somewhere like dak). I happen to have a bottom-up idea that is still a work-in-progress[1]. Albeit I believe what I'm doing can greatly facilitate my NEW reviewing process (as a trainee), I still suffer from the lack of time and energy to push the draft forward. I had some further private discussion with Michael and Dmitry. People's opinions on the solution differ. So I speculate that the community will have to come up with some more concrete ideas and experiment them a little bit to settle the whole NEW-SO-SLOW issue. [1] see my old post: "Idea: intermediate license review"
Re: call for ftpmaster's transparency
Quoting Michael Lustfield (2020-02-09 11:04:25) > On Thu, 06 Feb 2020 10:32:42 +1100 > Dmitry Smirnov wrote: > > > IMHO it is disturbing that one of the most essential processes in > > Debian -- acceptance of new and modified packages -- operates almost > > in secrecy. > > This is an understandable perspective, but secrecy probably isn't the > best word. > > > To make matters worse ftp-masters rarely leave their comments in ITP > > issues. As I've recently learned that have profound effect on > > processing of new packages. > > I personally think this sounds like a fantastic (and not very > difficult) idea. Me too. > Where do you propose the bug mail be sent for NEW/binNEW packages > without an ITP? I suggest (for now) to use our issue tracker only for packages with an ITP. > > One of my packages spent a year in the NEW queue at some point > > raising to position number 4. Apparently before release of Buster > > (2019-07-06) member of ftp-masters team left an internal (invisible > > to the public) comment on my package that was not communicated to me > > until 7 months later when my package was rejected based on that > > comment. The comment could have been addressed without delay if it > > was left on the corresponding ITP issue where it belong. > > I suspect when you say, "member of ftp-masters team," what you mean is > "FTP-Masters Trainee." FWIW- Trainees are not technically part of the > team. We get just enough access to be able to provide package reviews. > Those reviews are then either discussed with us or sent back in a > rejection/prod message. > > I agree that it could be valuable to see comments; however, they're > almost always going to be from Trainees. Since we're not technically > part of the team, it's important that we don't speak on behalf of the > team. Publishing Trainee comments would effectively be doing that. I suggest (for now) that ftp trainees CC an ITP (when such ITP exists) when they share their findings with ftp-masters. To help avoid misunderstandings, such messages could begin with something like this: NB! This is no FTP-Masters ruling (just suggestions from a Trainee). Is anything stopping Trainees from voluntarily changing their praxis right now to cc ITPs when available? > > A precious time was lost but more importantly one can see that > > current process requires an extra effort to communicate with > > maintainers -- a something that would not be necessary if > > ftp-masters use the official channel that exist specifically to > > discuss introduction of new packages -- ITP bug reports. [...] > > I would personally *LOVE* to see ITPs be a requirement for *ALL* new > packages. Making it a requirement and expecting ftp-masters to ignore > any upload until the ITP has existed for at least X days would be > absolutely fantastic. It would fix some redundant library uploads (see > golang/nodejs/etc.) and it would provide a mandatory level of review > by the wider community. > > Back when I tried to get gitea packaged for main, I had a number of > ITPs commented/closed mentioning the alternate library name or a > reason it can't be packaged. I think we don't need mandatory ITPs to get the ball rolling on better transparency. I suggest that (for now) we just make transparency yet another argument for voluntarily filing ITPs. - Jonas -- * Jonas Smedegaard - idealist & Internet-arkitekt * Tlf.: +45 40843136 Website: http://dr.jones.dk/ [x] quote me freely [ ] ask before reusing [ ] keep private signature.asc Description: signature
Re: call for ftpmaster's transparency
Michael Lustfield: > [...] > > I too would love to engage in a civil discussion about ways to improve the > situation. Let's start with this- > > Why do reviews take so long? > - The team is tiny > - Much of the team seems very burned out > - The ones that are active tend to stick to source or "unloved" tasks > - There are some very large and/or messy packages that need review > - There are a lot of redundant tasks and frequently-made mistakes > + A little more automation could help that > - (my opinion) The tools are archaic, cumbersome, and inefficient > + Fixing this would be a very (non-technically) difficult task > + An idea I have would help to bring transparency to the process... > ^ it's missing an interest requirement :( > For the parts involving tooling, are there bugs/salsa issues describing the issue so a "non-FTP-team"-member can take a stab at fixing them? ~Niels
Re: call for ftpmaster's transparency
On Thu, 06 Feb 2020 10:32:42 +1100 Dmitry Smirnov wrote: > IMHO it is disturbing that one of the most essential processes in Debian > -- acceptance of new and modified packages -- operates almost in secrecy. This is an understandable perspective, but secrecy probably isn't the best word. > To make matters worse ftp-masters rarely leave their comments in ITP > issues. As I've recently learned that have profound effect on processing of > new packages. I personally think this sounds like a fantastic (and not very difficult) idea. Where do you propose the bug mail be sent for NEW/binNEW packages without an ITP? > One of my packages spent a year in the NEW queue at some point raising to > position number 4. Apparently before release of Buster (2019-07-06) member > of ftp-masters team left an internal (invisible to the public) comment on > my package that was not communicated to me until 7 months later when my > package was rejected based on that comment. The comment could have been > addressed without delay if it was left on the corresponding ITP issue where > it belong. I suspect when you say, "member of ftp-masters team," what you mean is "FTP-Masters Trainee." FWIW- Trainees are not technically part of the team. We get just enough access to be able to provide package reviews. Those reviews are then either discussed with us or sent back in a rejection/prod message. I agree that it could be valuable to see comments; however, they're almost always going to be from Trainees. Since we're not technically part of the team, it's important that we don't speak on behalf of the team. Publishing Trainee comments would effectively be doing that. > A precious time was lost but more importantly one can see that current > process requires an extra effort to communicate with maintainers -- a > something that would not be necessary if ftp-masters use the official > channel that exist specifically to discuss introduction of new packages -- > ITP bug reports. > [...] I would personally *LOVE* to see ITPs be a requirement for *ALL* new packages. Making it a requirement and expecting ftp-masters to ignore any upload until the ITP has existed for at least X days would be absolutely fantastic. It would fix some redundant library uploads (see golang/nodejs/etc.) and it would provide a mandatory level of review by the wider community. Back when I tried to get gitea packaged for main, I had a number of ITPs commented/closed mentioning the alternate library name or a reason it can't be packaged. > I'd like Debian project leader to engage in the matter of improving > transparency of ftp-masters team operations and procedures. This feels a lot like starting a GR and not allowing appropriate discussion. It's heavy-handed, isn't going to get anywhere, and is going to hurt feelings. > As very minimum I recommend to change current ftp-master procedures to use > ITP bugs instead of internal comments whenever possible, for the sake of > transparency and to optimise communication. I replied to this idea above. > I want to encourage a public discussion regarding opening of the ftp-master > mail list to the public. Currently reasons for unjustified secrecy of ftp- > master processes is not explained... It's often said that emotions don't play well with productive discussions. Adding phrases such as "where it belong", using "secrecy" over "privacy", calling it "unjustified", and immediately jumping to demands of the DPL are accusatory and inflammatory, and will likely just get you ignored or start an unproductive flame war. I too would love to engage in a civil discussion about ways to improve the situation. Let's start with this- Why do reviews take so long? - The team is tiny - Much of the team seems very burned out - The ones that are active tend to stick to source or "unloved" tasks - There are some very large and/or messy packages that need review - There are a lot of redundant tasks and frequently-made mistakes + A little more automation could help that - (my opinion) The tools are archaic, cumbersome, and inefficient + Fixing this would be a very (non-technically) difficult task + An idea I have would help to bring transparency to the process... ^ it's missing an interest requirement :( -- Michael Lustfield pgpsLUD8ax93J.pgp Description: OpenPGP digital signature
Re: call for ftpmaster's transparency
Spiritually I really would like to see a transparent workflow of the FTP team. If it were a couple years ago I may stand in the same position as you. But now I'd kindly invite you to review the FTP team workflow (I joined the FTP team in order to review it), review the functionalities of dak (at least we trainees depend on it, which means what dak can do directly decides whether the process can be made transparent), and think of the possible ways we can make things better. That could provide us a more constructive start. Isn't it? Sam, IMHO DPL does not have to lead every important topics in this community, as that would be exhausting. On Thu, Feb 06, 2020 at 10:32:42AM +1100, Dmitry Smirnov wrote: > IMHO it is disturbing that one of the most essential processes in Debian > -- acceptance of new and modified packages -- operates almost in secrecy. > > Unlike most Debian teams, ftp-masters communicate in private mail list. > I understand why security team might need to operate without full public > disclosure but I see no reason for ftp-masters to avoid transparency. > Wouldn't it be easier to understand what to expect if everyone could see > how team operates? > > To make matters worse ftp-masters rarely leave their comments in ITP > issues. As I've recently learned that have profound effect on processing of > new packages. > > One of my packages spent a year in the NEW queue at some point raising to > position number 4. Apparently before release of Buster (2019-07-06) member > of ftp-masters team left an internal (invisible to the public) comment on > my package that was not communicated to me until 7 months later when my > package was rejected based on that comment. The comment could have been > addressed without delay if it was left on the corresponding ITP issue where > it belong. > > A precious time was lost but more importantly one can see that current > process requires an extra effort to communicate with maintainers -- a > something that would not be necessary if ftp-masters use the official > channel that exist specifically to discuss introduction of new packages -- > ITP bug reports. > > I'd like Debian project leader to engage in the matter of improving > transparency of ftp-masters team operations and procedures. > > As very minimum I recommend to change current ftp-master procedures to use > ITP bugs instead of internal comments whenever possible, for the sake of > transparency and to optimise communication. > > I want to encourage a public discussion regarding opening of the ftp-master > mail list to the public. Currently reasons for unjustified secrecy of ftp- > master processes is not explained... > > https://wiki.debian.org/Teams/FTPMaster > https://wiki.debian.org/NewQueue > > -- > All the best, > Dmitry Smirnov. > > --- > > Honesty is a gift we can give to others. It is also a source of power and > an engine of simplicity. Knowing that we will attempt to tell the truth, > whatever the circumstances, leaves us with little to prepare for. We can > simply be ourselves. > -- Sam Harris