Bug#840094: blends-dev: Does not recognize multiline dependencies
On Thu, Nov 24, 2016 at 08:39:54PM +0100, Petter Reinholdtsen wrote: > [Andreas Tille] > > I just want to hear confirmation from some Debian Edu developer > > whether the upload of blends-dev 0.6.95 is OK. > > As far as I can tell, these changes are intended and reflect changes > done to the source code. But I have not tracked the Debian Edu > development closely for a while. So I'd upload blends-dev 0.6.95 tomorrow if nobody else insists. Thanks for the feedback Andreas. -- http://fam-tille.de
Bug#840094: blends-dev: Does not recognize multiline dependencies
[Andreas Tille] > I just want to hear confirmation from some Debian Edu developer > whether the upload of blends-dev 0.6.95 is OK. As far as I can tell, these changes are intended and reflect changes done to the source code. But I have not tracked the Debian Edu development closely for a while. -- Happy hacking Petter Reinholdtsen
Bug#840094: blends-dev: Does not recognize multiline dependencies
Hi Ole, sorry for my longer than planned silence but today I found the time to repeat all tests with your new version. On Tue, Nov 15, 2016 at 10:12:00AM +0100, Ole Streicher wrote: > Hi Andreas, Petter and all, > > > On 15.11.2016 07:09, Andreas Tille wrote: > >> Use of uninitialized value $_ in pattern match (m//) at > >> /usr/share/blends-dev/blend-gen-control line 568, line 28. > >> [...] > > I fixed this in the git. HOWEVER, again: I have no glue what I do here. > I just assume that "next unless defined $_" does more or less what it > means, and I hope it will do so as well in the line where I put it. Same > for "last if !defined $_". I used these two phrases just because they > are already somewhere else in the script. Anyone with Perl knowledge: > CHECK IT! Our Perl knowledge seems to be quite comparable but the result seems to make sense. Debian Med, Debian Science, Debichem and Debian Junior are fine. I see some diff in Debian Edu but IMHO this makes perfectly sense: diff --git a/debian-edu-tasks.desc b/debian-edu-tasks.desc index ac0c349..a383078 100644 --- a/debian-edu-tasks.desc +++ b/debian-edu-tasks.desc @@ -1336,8 +1336,8 @@ Packages: list isc-dhcp-server-ldap pxelinux syslinux-common - debian-installer-8-netboot-i386 - debian-installer-8-netboot-amd64 + debian-installer-9-netboot-i386 + debian-installer-9-netboot-amd64 atftpd slapd openssl diff --git a/debian/control b/debian/control index 1afff19..4a9fae9 100644 --- a/debian/control +++ b/debian/control @@ -42,7 +42,7 @@ Description: Debian Edu menu reorganization Package: education-astronomy Section: metapackages Architecture: any -Depends: education-tasks (= ${binary:Version}) +Depends: education-tasks (= ${source:Version}), ${misc:Depends} Recommends: education-menus, gpredict, kstars, @@ -62,7 +62,7 @@ Description: Debian Edu astronomy related applications Package: education-chemistry Section: metapackages Architecture: any -Depends: education-tasks (= ${binary:Version}) +Depends: education-tasks (= ${source:Version}), ${misc:Depends} Recommends: chemtool, easychem, education-menus, ... The changed Depends: education-tasks (= ${source:Version}), ${misc:Depends} is for every single metapackage and IMHO that's perfectly correct. I just want to hear confirmation from some Debian Edu developer whether the upload of blends-dev 0.6.95 is OK. Kind regards Andreas. -- http://fam-tille.de
Bug#840094: blends-dev: Does not recognize multiline dependencies
[Ole] > Great! Could I in return ask you for the favour looking into the > patched blend-gen-control to see why the lines with a backslash are > not eaten up, so that one has to remove them later again? > > (see one of my previous mails) I will try to find time have a look. Could you prepare and commit a test case demonstrating the problem? It would be a great start for a test suite for blends. I had a quick glance and did not quite understand what I was looking for. -- Happy hacking Petter Reinholdtsen
Bug#840094: blends-dev: Does not recognize multiline dependencies
Hi Andreas, Petter and all, On 15.11.2016 07:09, Andreas Tille wrote: > Hi, > > I just want to announce that I'll be de-facto offline today and > tomorrow. So I can not do further testing of the "Use of uninitialized > value" testing. > > Kind regards > > Andreas. > > On Fri, Nov 11, 2016 at 12:52:46PM +0100, Andreas Tille wrote: >> Use of uninitialized value $_ in pattern match (m//) at >> /usr/share/blends-dev/blend-gen-control line 568, line 28. >> [...] I fixed this in the git. HOWEVER, again: I have no glue what I do here. I just assume that "next unless defined $_" does more or less what it means, and I hope it will do so as well in the line where I put it. Same for "last if !defined $_". I used these two phrases just because they are already somewhere else in the script. Anyone with Perl knowledge: CHECK IT! Cheers Ole
Bug#840094: blends-dev: Does not recognize multiline dependencies
Hi, I just want to announce that I'll be de-facto offline today and tomorrow. So I can not do further testing of the "Use of uninitialized value" testing. Kind regards Andreas. On Fri, Nov 11, 2016 at 12:52:46PM +0100, Andreas Tille wrote: > Hi, > > On Fri, Nov 11, 2016 at 11:50:12AM +0100, Petter Reinholdtsen wrote: > > [Ole Streicher] > > > OK, I committed another version, with backslashes everywhere. Adjusted > > > tool is attached. > > > > Very good. This one look like it should still work, and fixes a few > > mistakes in the process. Thank you very much. > > +1 > > I double checked here Debian Edu looks fine. Since there was another > code change I repeated my checks for Debian Med and Debian Science. > While Debian Med is OK for Debian Science I get in a make dist lots of > lines like: > > ... > Hit http://ftp.debian.org testing InRelease > Get:1 http://ftp.debian.org testing/main amd64 Packages/DiffIndex [27.9 kB] > Get:2 http://ftp.debian.org testing/main Translation-en/DiffIndex [27.9 kB] > Get:3 http://ftp.debian.org testing/main Translation-de/DiffIndex [27.8 kB] > Fetched 83.6 kB in 1s (83.2 kB/s) > Reading package lists... Done > Use of uninitialized value $_ in pattern match (m//) at > /usr/share/blends-dev/blend-gen-control line 568, line 28. > Use of uninitialized value $_ in pattern match (m//) at > /usr/share/blends-dev/blend-gen-control line 568, line 28. > Use of uninitialized value $_ in pattern match (m//) at > /usr/share/blends-dev/blend-gen-control line 598, line 28. > Use of uninitialized value $_ in pattern match (m//) at > /usr/share/blends-dev/blend-gen-control line 609, line 28. > Use of uninitialized value $_ in pattern match (m//) at > /usr/share/blends-dev/blend-gen-control line 568, line 72. > Use of uninitialized value $_ in pattern match (m//) at > /usr/share/blends-dev/blend-gen-control line 568, line 72. > Use of uninitialized value $_ in pattern match (m//) at > /usr/share/blends-dev/blend-gen-control line 598, line 72. > Use of uninitialized value $_ in pattern match (m//) at > /usr/share/blends-dev/blend-gen-control line 609, line 72. > Use of uninitialized value $_ in pattern match (m//) at > /usr/share/blends-dev/blend-gen-control line 598, line 248. > Use of uninitialized value $_ in pattern match (m//) at > /usr/share/blends-dev/blend-gen-control line 609, line 248. > Use of uninitialized value $_ in pattern match (m//) at > /usr/share/blends-dev/blend-gen-control line 568, line 145. > Use of uninitialized value $_ in pattern match (m//) at > /usr/share/blends-dev/blend-gen-control line 568, line 145. > Use of uninitialized value $_ in pattern match (m//) at > /usr/share/blends-dev/blend-gen-control line 598, line 145. > Use of uninitialized value $_ in pattern match (m//) at > /usr/share/blends-dev/blend-gen-control line 609, line 145. > Use of uninitialized value $_ in pattern match (m//) at > /usr/share/blends-dev/blend-gen-control line 598, line 104. > Use of uninitialized value $_ in pattern match (m//) at > /usr/share/blends-dev/blend-gen-control line 609, line 104. > Use of uninitialized value $_ in pattern match (m//) at > /usr/share/blends-dev/blend-gen-control line 568, line 108. > Use of uninitialized value $_ in pattern match (m//) at > /usr/share/blends-dev/blend-gen-control line 568, line 108. > Use of uninitialized value $_ in pattern match (m//) at > /usr/share/blends-dev/blend-gen-control line 598, line 108. > Use of uninitialized value $_ in pattern match (m//) at > /usr/share/blends-dev/blend-gen-control line 609, line 108. > Use of uninitialized value $_ in pattern match (m//) at > /usr/share/blends-dev/blend-gen-control line 568, line 77. > Use of uninitialized value $_ in pattern match (m//) at > /usr/share/blends-dev/blend-gen-control line 568, line 77. > Use of uninitialized value $_ in pattern match (m//) at > /usr/share/blends-dev/blend-gen-control line 598, line 77. > Use of uninitialized value $_ in pattern match (m//) at > /usr/share/blends-dev/blend-gen-control line 609, line 77. > Use of uninitialized value $_ in pattern match (m//) at > /usr/share/blends-dev/blend-gen-control line 598, line 62. > Use of uninitialized value $_ in pattern match (m//) at > /usr/share/blends-dev/blend-gen-control line 609, line 62. > Use of uninitialized value $_ in pattern match (m//) at > /usr/share/blends-dev/blend-gen-control line 598, line 24. > Use of uninitialized value $_ in pattern match (m//) at > /usr/share/blends-dev/blend-gen-control line 609, line 24. > ... > > > The result looks OK, but the output is suspicious. I've checked with > blends-dev=0.6.94 and here this output does not occure. > > Kind regards > >Andreas. > > -- > http://fam-tille.de > > -- http://fam-tille.de
Bug#840094: blends-dev: Does not recognize multiline dependencies
Hi Petter, On 11.11.2016 11:50, Petter Reinholdtsen wrote: > [Ole Streicher] >> OK, I committed another version, with backslashes everywhere. Adjusted >> tool is attached. > Very good. This one look like it should still work, and fixes a few > mistakes in the process. Thank you very much. > >> I tried as much as possible. However, trailing spaces are still >> removed, and the backslash now always is preceded by a space. > Sound good to me. > > We will see if I was right on the next ISO build. :) > Great! Could I in return ask you for the favour looking into the patched blend-gen-control to see why the lines with a backslash are not eaten up, so that one has to remove them later again? (see one of my previous mails) Best regards Ole
Bug#840094: blends-dev: Does not recognize multiline dependencies
Hi, On Fri, Nov 11, 2016 at 11:50:12AM +0100, Petter Reinholdtsen wrote: > [Ole Streicher] > > OK, I committed another version, with backslashes everywhere. Adjusted > > tool is attached. > > Very good. This one look like it should still work, and fixes a few > mistakes in the process. Thank you very much. +1 I double checked here Debian Edu looks fine. Since there was another code change I repeated my checks for Debian Med and Debian Science. While Debian Med is OK for Debian Science I get in a make dist lots of lines like: ... Hit http://ftp.debian.org testing InRelease Get:1 http://ftp.debian.org testing/main amd64 Packages/DiffIndex [27.9 kB] Get:2 http://ftp.debian.org testing/main Translation-en/DiffIndex [27.9 kB] Get:3 http://ftp.debian.org testing/main Translation-de/DiffIndex [27.8 kB] Fetched 83.6 kB in 1s (83.2 kB/s) Reading package lists... Done Use of uninitialized value $_ in pattern match (m//) at /usr/share/blends-dev/blend-gen-control line 568, line 28. Use of uninitialized value $_ in pattern match (m//) at /usr/share/blends-dev/blend-gen-control line 568, line 28. Use of uninitialized value $_ in pattern match (m//) at /usr/share/blends-dev/blend-gen-control line 598, line 28. Use of uninitialized value $_ in pattern match (m//) at /usr/share/blends-dev/blend-gen-control line 609, line 28. Use of uninitialized value $_ in pattern match (m//) at /usr/share/blends-dev/blend-gen-control line 568, line 72. Use of uninitialized value $_ in pattern match (m//) at /usr/share/blends-dev/blend-gen-control line 568, line 72. Use of uninitialized value $_ in pattern match (m//) at /usr/share/blends-dev/blend-gen-control line 598, line 72. Use of uninitialized value $_ in pattern match (m//) at /usr/share/blends-dev/blend-gen-control line 609, line 72. Use of uninitialized value $_ in pattern match (m//) at /usr/share/blends-dev/blend-gen-control line 598, line 248. Use of uninitialized value $_ in pattern match (m//) at /usr/share/blends-dev/blend-gen-control line 609, line 248. Use of uninitialized value $_ in pattern match (m//) at /usr/share/blends-dev/blend-gen-control line 568, line 145. Use of uninitialized value $_ in pattern match (m//) at /usr/share/blends-dev/blend-gen-control line 568, line 145. Use of uninitialized value $_ in pattern match (m//) at /usr/share/blends-dev/blend-gen-control line 598, line 145. Use of uninitialized value $_ in pattern match (m//) at /usr/share/blends-dev/blend-gen-control line 609, line 145. Use of uninitialized value $_ in pattern match (m//) at /usr/share/blends-dev/blend-gen-control line 598, line 104. Use of uninitialized value $_ in pattern match (m//) at /usr/share/blends-dev/blend-gen-control line 609, line 104. Use of uninitialized value $_ in pattern match (m//) at /usr/share/blends-dev/blend-gen-control line 568, line 108. Use of uninitialized value $_ in pattern match (m//) at /usr/share/blends-dev/blend-gen-control line 568, line 108. Use of uninitialized value $_ in pattern match (m//) at /usr/share/blends-dev/blend-gen-control line 598, line 108. Use of uninitialized value $_ in pattern match (m//) at /usr/share/blends-dev/blend-gen-control line 609, line 108. Use of uninitialized value $_ in pattern match (m//) at /usr/share/blends-dev/blend-gen-control line 568, line 77. Use of uninitialized value $_ in pattern match (m//) at /usr/share/blends-dev/blend-gen-control line 568, line 77. Use of uninitialized value $_ in pattern match (m//) at /usr/share/blends-dev/blend-gen-control line 598, line 77. Use of uninitialized value $_ in pattern match (m//) at /usr/share/blends-dev/blend-gen-control line 609, line 77. Use of uninitialized value $_ in pattern match (m//) at /usr/share/blends-dev/blend-gen-control line 598, line 62. Use of uninitialized value $_ in pattern match (m//) at /usr/share/blends-dev/blend-gen-control line 609, line 62. Use of uninitialized value $_ in pattern match (m//) at /usr/share/blends-dev/blend-gen-control line 598, line 24. Use of uninitialized value $_ in pattern match (m//) at /usr/share/blends-dev/blend-gen-control line 609, line 24. ... The result looks OK, but the output is suspicious. I've checked with blends-dev=0.6.94 and here this output does not occure. Kind regards Andreas. -- http://fam-tille.de
Bug#840094: blends-dev: Does not recognize multiline dependencies
[Ole Streicher] > OK, I committed another version, with backslashes everywhere. Adjusted > tool is attached. Very good. This one look like it should still work, and fixes a few mistakes in the process. Thank you very much. > I tried as much as possible. However, trailing spaces are still > removed, and the backslash now always is preceded by a space. Sound good to me. We will see if I was right on the next ISO build. :) -- Happy hacking Petter Reinholdtsen
Bug#840094: blends-dev: Does not recognize multiline dependencies
Hi Petter, On 11.11.2016 11:03, Petter Reinholdtsen wrote: > [Ole Streicher] >> I updated all tasks of debian-edu to be RFC834 compliant: continuation >> with indentation, no backslashes, no duplicated keywords. The diff looks >> OK for me; you should however double check. >> >> If you don't like it, revert and discuss it. > This is not going to fly. The Debian Edu task files in git are > processed automatically several times a day to build the ISOs on a > machine running Jessie, so depending on a non-released version of the > blends package is not a viable option at the moment. OK, I committed another version, with backslashes everywhere. Adjusted tool is attached. > I believe the only change we should do to the git version before the new > blend version is in stable is to drop the double > Depends/Recommends/Suggests lines. Otherwise we break the ISO builds. > > The whitespace changes make it harder to track the history of the > tasks. Can you avoid whitespace changes? > I tried as much as possible. However, trailing spaces are still removed, and the backslash now always is preceded by a space. Cheers Ole #!/usr/bin/env python3 from __future__ import print_function from glob import glob from collections import OrderedDict import re def format_section(s): return "\n".join(l[1] for l in s.values()) + "\n" def format_task(sections): return "\n".join(format_section(s) for s in sections) def get_task(f, eol=""): key = "" items = OrderedDict() sections = [] for l in f.readlines(): backslash = "\\" in l l = l.replace("\\","").rstrip() if not l and not backslash: if items: sections.append(items) items = OrderedDict() key = "" continue m = re.match(r"^(\S+):(\s*)(.*)", l) if m is not None: key = m.group(1) if key in items: if key in ("Depends", "Ignore", "Avoid", "Recommends", "Suggests"): items[key][1] += "," + eol items[key][1] += "\n " + " " * items[key][0] + m.group(3).strip() else: if key in ("Depends", "Ignore", "Avoid", "Recommends", "Suggests", "Why"): indent = len(key) + len(m.group(2)) else: indent = 0 items[key] = [indent, m.group(0).strip()] else: if items: if key in ("Depends", "Ignore", "Avoid", "Recommends", "Suggests"): items[key][1] += eol items[key][1] += "\n" + l sections.append(items) return sections for taskname in glob("tasks/*"): with open(taskname) as taskfile: s = format_task(get_task(taskfile, " \\")) with open(taskname, "w") as taskfile: taskfile.write(s)
Bug#840094: blends-dev: Does not recognize multiline dependencies
[Ole Streicher] > I updated all tasks of debian-edu to be RFC834 compliant: continuation > with indentation, no backslashes, no duplicated keywords. The diff looks > OK for me; you should however double check. > > If you don't like it, revert and discuss it. This is not going to fly. The Debian Edu task files in git are processed automatically several times a day to build the ISOs on a machine running Jessie, so depending on a non-released version of the blends package is not a viable option at the moment. I believe the only change we should do to the git version before the new blend version is in stable is to drop the double Depends/Recommends/Suggests lines. Otherwise we break the ISO builds. The whitespace changes make it harder to track the history of the tasks. Can you avoid whitespace changes? -- Happy hacking Petter Reinholdtsen
Bug#840094: blends-dev: Does not recognize multiline dependencies
Hi Andreas, Petter, and all I updated all tasks of debian-edu to be RFC834 compliant: continuation with indentation, no backslashes, no duplicated keywords. The diff looks OK for me; you should however double check. If you don't like it, revert and discuss it. I wrote a small Python script for that - nothing I would be proud of, but it did the job here. I attach it since it may help for other tasks as well, but don't blame me :-) I takes all files from the tasks/ subdir, reformats them and writes them back. No security or so. Check with "git diff" afterwards that everything is OK. Cheers Ole #!/usr/bin/env python3 from __future__ import print_function from glob import glob from collections import OrderedDict import re def format_section(s): return "\n".join(l[1] for l in s.values()) + "\n" def format_task(sections): return "\n".join(format_section(s) for s in sections) def get_task(f): key = "" items = OrderedDict() sections = [] for l in f.readlines(): backslash = "\\" in l l = l.replace("\\","").strip() if not l and not backslash: if items: sections.append(items) items = OrderedDict() key = "" continue m = re.match(r"^(\S+):(\s*)(.*)", l) if m is not None: key = m.group(1) if key in items: items[key][1] += "\n " + " " * items[key][0] + m.group(3).strip() else: if key in ("Depends", "Ignore", "Avoid", "Recommends", "Suggests", "Why"): indent = len(key) + len(m.group(2)) else: indent = 0 items[key] = [indent, m.group(0).strip()] else: items[key][1] += "\n " + " " * items[key][0] + l if items: sections.append(items) return sections for taskname in glob("tasks/*"): with open(taskname) as taskfile: s = format_task(get_task(taskfile)) with open(taskname, "w") as taskfile: taskfile.write(s)
Bug#840094: blends-dev: Does not recognize multiline dependencies
Hi all, On 11.11.2016 08:07, Andreas Tille wrote: > On Thu, Nov 10, 2016 at 10:38:32PM +0100, Ole Streicher wrote: >> --> debian-edu tasks are just broken. They don't follow any rule, and >> depending from the parser one will get always different results. Maybe >> we should fix that? remove all backslash continuation lines and >> duplicated keywords from the tasks files? > > I think it should be sufficient to add line breaks where needed. If we touch them anyway, we could make them fully RFC compliant at this time as well. Since I am currently stuck in the S-Bahn (Polizeieinsatz), I will do that for debian-edu and push. Cellphone internet is a nice thing sometimes... :-) @Petter, please review and change/revert if you disagree. Cheers Ole
Bug#840094: blends-dev: Does not recognize multiline dependencies
Hi, On Thu, Nov 10, 2016 at 10:38:32PM +0100, Ole Streicher wrote: > On 10.11.2016 21:07, Andreas Tille wrote: > > So I confirm that the first problem we detected is solved but there is > > another one breaking Debian Edu. I have again no suspicion why the '\' > > sign is not elimiunated from the list only in those few cases. > > I can also just "poke in the fog" here. I thought that the multilines > were already eaten up by lines 537-544, and should not appear again > here. However, they do, as my "print" debugger shows. Perlists, please > help!!! > > The pragmatic solution here is to just remove the backslashes from the > end of line. I can commit a patch that does this. > > However, debian-edu keeps to be broken. Reason is that the tasks contain > lines like (development) > > Depends: fp-compiler, [...multiline... ], fp-units-fv > Depends: lazarus > > so, with two "Depends" in one section. Or (electronics): That's definitely broken and not supported by other tools than blends-dev and should be fixed. Every Depends, Recommends and Suggests should be in a separate paragraph. > Depends: qucs, gpsim, oregano > Recommends: education-menus, xoscope > Suggests: kicad, kicad-doc-en, kicad-doc-de, kicad-doc-es, kicad-doc-fr > Suggests: electric, pcb, xcircuit, freehdl, gtkwave > Responsible: ? > NeedConfig: no > > two "Suggests". This does not work anymore (no idea why), but is also > IMO forbidden by RFC834. The web sentinel code does not even allow one Depends and Suggests in a common paragraph. > I have no idea why this works without RFC834 continuation, but not with > them. > > On the other hand, we *win* one more dependency: in "common", the > "firmware-ralink" dependency line was *not* preceded one with a > backslash. This shows that the backslash is just a bad idea for > continuation lines. Meanwhile I'm fully convinced that you are right here and we should fix this soon since also in Debian Science some missing backslashes caused loss of Dependencies in the resulting metapackages. > --> debian-edu tasks are just broken. They don't follow any rule, and > depending from the parser one will get always different results. Maybe > we should fix that? remove all backslash continuation lines and > duplicated keywords from the tasks files? I think it should be sufficient to add line breaks where needed. Kind regards Andreas. -- http://fam-tille.de
Bug#840094: blends-dev: Does not recognize multiline dependencies
Hi Andreas, Petter and all, On 10.11.2016 21:07, Andreas Tille wrote: > So I confirm that the first problem we detected is solved but there is > another one breaking Debian Edu. I have again no suspicion why the '\' > sign is not elimiunated from the list only in those few cases. I can also just "poke in the fog" here. I thought that the multilines were already eaten up by lines 537-544, and should not appear again here. However, they do, as my "print" debugger shows. Perlists, please help!!! The pragmatic solution here is to just remove the backslashes from the end of line. I can commit a patch that does this. However, debian-edu keeps to be broken. Reason is that the tasks contain lines like (development) Depends: fp-compiler, [...multiline... ], fp-units-fv Depends: lazarus so, with two "Depends" in one section. Or (electronics): Depends: qucs, gpsim, oregano Recommends: education-menus, xoscope Suggests: kicad, kicad-doc-en, kicad-doc-de, kicad-doc-es, kicad-doc-fr Suggests: electric, pcb, xcircuit, freehdl, gtkwave Responsible: ? NeedConfig: no two "Suggests". This does not work anymore (no idea why), but is also IMO forbidden by RFC834. I have no idea why this works without RFC834 continuation, but not with them. On the other hand, we *win* one more dependency: in "common", the "firmware-ralink" dependency line was *not* preceded one with a backslash. This shows that the backslash is just a bad idea for continuation lines. --> debian-edu tasks are just broken. They don't follow any rule, and depending from the parser one will get always different results. Maybe we should fix that? remove all backslash continuation lines and duplicated keywords from the tasks files? Best regards Ole
Bug#840094: blends-dev: Does not recognize multiline dependencies
Hi, On Thu, Nov 10, 2016 at 01:53:20PM +0100, Ole Streicher wrote: > > > > gbp clone ssh://git.debian.org/git/blends/projects/med.git > > cd med > > make dist > > grep ^, debian/control Debian Med and Debian Science are OK with the patch, however: gbp clone ssh://git.debian.org/git/debian-edu/debian-edu.git cd debian-edu make dist grep '\\' debian/control | head Suggests: \ cpqarrayd, \ isag, \ lshw, Suggests: \ gnome-accessibility-themes, Suggests: \ kde-full, \ kdeaccessibility, \ kdegraphics-thumbnailers, \ kdf, \ kfloppy, \ kinfocenter, So for some reason in the Debian Edu case '\' signs will not be ignored but added to the d/control file. > This is what I meant, and it is fixed by my last commit. Please try > again ;-) So I confirm that the first problem we detected is solved but there is another one breaking Debian Edu. I have again no suspicion why the '\' sign is not elimiunated from the list only in those few cases. Kind regards Andreas. -- http://fam-tille.de
Bug#840094: blends-dev: Does not recognize multiline dependencies
On Thu, Nov 10, 2016 at 01:53:20PM +0100, Ole Streicher wrote: > > This is what I meant, and it is fixed by my last commit. Please try > again ;-) Quick note from bad connection. Seems to work now. I'll try later more carefully. Thanks for your contribution Andreas. -- http://fam-tille.de
Bug#840094: blends-dev: Does not recognize multiline dependencies
Hi Andreas, On 10.11.2016 13:48, Andreas Tille wrote: > Hi Petter, > > On Thu, Nov 10, 2016 at 12:39:07PM +0100, Petter Reinholdtsen wrote: >> Control: tags -1 + pending >> >> [Andreas Tille] Should I commit it? >>> Yes please. Ole, you reported problems with your patch. Could you >>> please be more verbose about this problem - at best based on Petter's >>> commit to make sure we are all working on the same code? >> It is now commited. Please give it some more testing before uploading. >> Preferably also with the debian-edu git repo, where both tasks and >> control file is kept in git, allowing any changes to be easily >> discovered. > I need to admit two things: Even in Debian Med we went into the trap > criticised by Ole. In bio-cloud which is not maintained by me we were > also loosing entries. The second thing I need to admit that there are > in fact syntax errors resulting from the patch. When using the new > blends-dev package a > > gbp clone ssh://git.debian.org/git/blends/projects/med.git > cd med > make dist > grep ^, debian/control > > shows the problem. It leads to something like > > Suggests: bagpipe, > cufflinks, > embassy-phylip, > gmap, > r-cran-vegan > , > r-other-mott-happy.hbrem > , > r-other-valdar-bagphenotype.library, > soapdenovo2 > , > sra-toolkit > , > staden-io-lib-utils > , This is what I meant, and it is fixed by my last commit. Please try again ;-) Cheers Ole
Bug#840094: blends-dev: Does not recognize multiline dependencies
Hi Petter, On Thu, Nov 10, 2016 at 12:39:07PM +0100, Petter Reinholdtsen wrote: > Control: tags -1 + pending > > [Andreas Tille] > >> Should I commit it? > > > > Yes please. Ole, you reported problems with your patch. Could you > > please be more verbose about this problem - at best based on Petter's > > commit to make sure we are all working on the same code? > > It is now commited. Please give it some more testing before uploading. > Preferably also with the debian-edu git repo, where both tasks and > control file is kept in git, allowing any changes to be easily > discovered. I need to admit two things: Even in Debian Med we went into the trap criticised by Ole. In bio-cloud which is not maintained by me we were also loosing entries. The second thing I need to admit that there are in fact syntax errors resulting from the patch. When using the new blends-dev package a gbp clone ssh://git.debian.org/git/blends/projects/med.git cd med make dist grep ^, debian/control shows the problem. It leads to something like Suggests: bagpipe, cufflinks, embassy-phylip, gmap, r-cran-vegan , r-other-mott-happy.hbrem , r-other-valdar-bagphenotype.library, soapdenovo2 , sra-toolkit , staden-io-lib-utils , ... and I admit I have no idea why. The relevant entries in the tasks files are looking pretty normal. May be I need to stare a bit longer onto it ... :-( > > Thanks again Petter for your always reliable support > > No worries, as they say in Australia. :) Only in Australia? Kind regards Andreas. -- http://fam-tille.de
Bug#840094: blends-dev: Does not recognize multiline dependencies
Control: tags -1 + pending [Andreas Tille] >> Should I commit it? > > Yes please. Ole, you reported problems with your patch. Could you > please be more verbose about this problem - at best based on Petter's > commit to make sure we are all working on the same code? It is now commited. Please give it some more testing before uploading. Preferably also with the debian-edu git repo, where both tasks and control file is kept in git, allowing any changes to be easily discovered. > Thanks again Petter for your always reliable support No worries, as they say in Australia. :) -- Happy hacking Petter Reinholdtsen
Processed: Re: Bug#840094: blends-dev: Does not recognize multiline dependencies
Processing control commands: > tags -1 + pending Bug #840094 [blends-dev] blends-dev: Does not recognize multiline dependencies Added tag(s) pending. -- 840094: http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=840094 Debian Bug Tracking System Contact ow...@bugs.debian.org with problems
Bug#840094: blends-dev: Does not recognize multiline dependencies
Hi Petter, On Thu, Nov 10, 2016 at 11:07:47AM +0100, Petter Reinholdtsen wrote: > > I have not followed this issue closely, but started reading on Andreas > request. Thanks a lot for your quick response. > First of all, I believe this issue is based on incorrect documentation. Yes. > The task files have never been RFC822 compliant. I agree that they > should be RFC822 compliant, but the only handled continuation method > implemented have always been using backslash at the end of the previous > line. > > Moving to full RFC822 compliance is probably best done with a test suite > in place, to verify that existing task files still produce the expected > set of tasks after such change. Fully ACK. > [Andreas Tille] > > Petter, could you please have a look at the patch provided by Ole[1]? > > I had a look and tested a bit, and it look good to me. The old style > backslash continuation used by Debian Edu continue to work, and the new > RFC822 style continuation start working. I believe it should be applied > and uploaded to Debian. > > Should I commit it? Yes please. Ole, you reported problems with your patch. Could you please be more verbose about this problem - at best based on Petter's commit to make sure we are all working on the same code? Thanks again Petter for your always reliable support Andreas. -- http://fam-tille.de
Bug#840094: blends-dev: Does not recognize multiline dependencies
I have not followed this issue closely, but started reading on Andreas request. First of all, I believe this issue is based on incorrect documentation. The task files have never been RFC822 compliant. I agree that they should be RFC822 compliant, but the only handled continuation method implemented have always been using backslash at the end of the previous line. Moving to full RFC822 compliance is probably best done with a test suite in place, to verify that existing task files still produce the expected set of tasks after such change. [Andreas Tille] > Petter, could you please have a look at the patch provided by Ole[1]? I had a look and tested a bit, and it look good to me. The old style backslash continuation used by Debian Edu continue to work, and the new RFC822 style continuation start working. I believe it should be applied and uploaded to Debian. Should I commit it? -- Happy hacking Petter Reinholdtsen
Bug#840094: blends-dev: Does not recognize multiline dependencies
Hi Ole, On Thu, Nov 10, 2016 at 08:56:50AM +0100, Ole Streicher wrote: > > If one of our main tools is not compliant with our documented > specifications, and this may cause incomplete metapackages (which are > one central part of the blend), then I would still rate this as an RC > bug, independently of whether it is easy to fix or not. If all fails I'd considering to fix the documentation (which would be only the second best solution admittedly). > > While I fully agree that we should fix this I'm not fully convinced how > > to sensibly proceed here. The problematic thing is that we are quite > > short before a release and if we might break metapackage creation in > > some way we might get in trouble. I'm no Perl programmer myself (even > > if I think your patch looks sensible) and so IMHO staying conservative > > and add some line ending escapes could be the less invasive change. > > I checked my patch, and it does *not* work correctly, it will produce > syntax errors in the debian/control file, if RFC822 continuation lines > are used. For tasks that have all in one line, or that have > metapackages, everything seems to be OK, however. > > > If you (and Bas and other readers here) think we should fix the issue > > right now I'm fine if you apply the patch below and we should seriously > > test the metapackage creation of each Blend *before* 2016-12-05. > > > > What do you think? > > I am ready to test and also to fix; however my know-how ends here. I > don't know what is wrong with the fix. > > Just wondering, and starting to really get worried: None of the > debian-blends maintainers has enough Perl knowledge to fix this? If we > all do not know Perl, why do we use that language in one of our central > tools? That sounds to me even more RC than the bug itself... It was originally written by Petter (in CC) who is a Perl programmer but drifted a bit away from this topic. I'm really sure he will help out. Your question about the language was answered in my previous mail mentioning the rewrite. Kind regards (and thanks for pushing things in the right direction) Andreas. -- http://fam-tille.de
Bug#840094: blends-dev: Does not recognize multiline dependencies
Hi Ole, On Thu, Nov 10, 2016 at 09:06:52AM +0100, Ole Streicher wrote: > > > >> it's indeed to late in the stretch dev cycle in my opinion. > > > > That would mean to lower the severity of #840094. Ole, are you OK with > > this? > > No, I am not. The tool gives wrong results, and it does so silently. If > I would not have discovered this by chance, the Debian Astro tasks > packages were still wrong and would have stayed so in Stretch. I don't > know about other blends, whether they ever checked the metapackages -- > others still may have missing tasks as well, just because their > maintainers read the docs and trust the debian-blends package. And you > can't just change the specs in the last minute. > > And, if the package is in Stretch, people will start their new blend > with exactly this package, again running into this trouble. I see your point but I have mixed feelings. > I would really propose to fix that before Stretch. It just can't be that > difficult. Nobody knows a Perl specialist who can have a look? Petter, could you please have a look at the patch provided by Ole[1]? > (Oh, and can we have Python for the Next Generation blends-dev? This is > at least what I understand, and there is already a parser for rfc822) Did you ever checked out[2]? Its Python and it solves another major problem. It is able to create Architecture=any metapackages verifying the availability of a dependency on the said architecture. It also creates automatic changelog about differences to previous releases and adds some statistical data in json format. The price you need to pay is that you need to access UDD - but given that there is some public UDD mirror this could be turned to a non issue. And it needs testing ... since 3 years. :-( If you have a local UDD mirror you can just build the package and use it to create the metapackage source as a drop in replacement - which I'm actually doing for Debian Med to get the json data and changelog which I'm copying over. As I said: Sorry for not managing to push this into production earlier ... Kind regards Andreas. [1] https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=840094#25 [2] https://anonscm.debian.org/git/blends/blends-gsoc.git -- http://fam-tille.de
Bug#840094: blends-dev: Does not recognize multiline dependencies
Hi Andreas and Bas, On 10.11.2016 08:45, Andreas Tille wrote: > On Wed, Nov 09, 2016 at 06:27:13PM +0100, Sebastiaan Couwenberg wrote: >> On 11/09/2016 03:35 PM, Andreas Tille wrote: >>> If you (and Bas and other readers here) think we should fix the issue >>> right now I'm fine if you apply the patch below and we should seriously >>> test the metapackage creation of each Blend *before* 2016-12-05. >>> >>> What do you think? >> >> I think supporting the deb822 format should be a Blends release goal for >> buster, > > I fully agree. I admit I did way less for blends-dev than I intended to > do but other tasks that felt more urgent occupied all my Debian time. > >> it's indeed to late in the stretch dev cycle in my opinion. > > That would mean to lower the severity of #840094. Ole, are you OK with > this? No, I am not. The tool gives wrong results, and it does so silently. If I would not have discovered this by chance, the Debian Astro tasks packages were still wrong and would have stayed so in Stretch. I don't know about other blends, whether they ever checked the metapackages -- others still may have missing tasks as well, just because their maintainers read the docs and trust the debian-blends package. And you can't just change the specs in the last minute. And, if the package is in Stretch, people will start their new blend with exactly this package, again running into this trouble. I would really propose to fix that before Stretch. It just can't be that difficult. Nobody has a Perl specialist who can have a look? (Oh, and can we have Python for the Next Generation blends-dev? This is at least what I understand, and there is already a parser for rfc822) Best regards Ole
Bug#840094: blends-dev: Does not recognize multiline dependencies
Hi Andreas and Bas, On 10.11.2016 08:45, Andreas Tille wrote: > On Wed, Nov 09, 2016 at 06:27:13PM +0100, Sebastiaan Couwenberg wrote: >> On 11/09/2016 03:35 PM, Andreas Tille wrote: >>> If you (and Bas and other readers here) think we should fix the issue >>> right now I'm fine if you apply the patch below and we should seriously >>> test the metapackage creation of each Blend *before* 2016-12-05. >>> >>> What do you think? >> >> I think supporting the deb822 format should be a Blends release goal for >> buster, > > I fully agree. I admit I did way less for blends-dev than I intended to > do but other tasks that felt more urgent occupied all my Debian time. > >> it's indeed to late in the stretch dev cycle in my opinion. > > That would mean to lower the severity of #840094. Ole, are you OK with > this? No, I am not. The tool gives wrong results, and it does so silently. If I would not have discovered this by chance, the Debian Astro tasks packages were still wrong and would have stayed so in Stretch. I don't know about other blends, whether they ever checked the metapackages -- others still may have missing tasks as well, just because their maintainers read the docs and trust the debian-blends package. And you can't just change the specs in the last minute. And, if the package is in Stretch, people will start their new blend with exactly this package, again running into this trouble. I would really propose to fix that before Stretch. It just can't be that difficult. Nobody knows a Perl specialist who can have a look? (Oh, and can we have Python for the Next Generation blends-dev? This is at least what I understand, and there is already a parser for rfc822) Best regards Ole
Bug#840094: blends-dev: Does not recognize multiline dependencies
Hi Andreas & all, On 09.11.2016 15:35, Andreas Tille wrote: >> We have a clear definition of how these files should look like, namely >> RFC822, and this also defines continuation lines. > > Unfortunately in this specific feature tasks files are not RFC822 > compliant, which sucks, yes. Its even not documented (I just checked > since I intended to document it at some point in time but can't find it > :-( ) If one of our main tools is not compliant with our documented specifications, and this may cause incomplete metapackages (which are one central part of the blend), then I would still rate this as an RC bug, independently of whether it is easy to fix or not. >> I would think that there is also a quick fix for it -- the tool already >> handles continuation lines for the tasks description, so one could >> probably just take that. I have no glue of all the Perl $@^!~ special >> chars, but wouldn't do it something like the attached patch (after >> removing the obvious errors from it)? >> >> Or something else just adopted from lines 556-562 of blends-gen-control? > > While I fully agree that we should fix this I'm not fully convinced how > to sensibly proceed here. The problematic thing is that we are quite > short before a release and if we might break metapackage creation in > some way we might get in trouble. I'm no Perl programmer myself (even > if I think your patch looks sensible) and so IMHO staying conservative > and add some line ending escapes could be the less invasive change. I checked my patch, and it does *not* work correctly, it will produce syntax errors in the debian/control file, if RFC822 continuation lines are used. For tasks that have all in one line, or that have metapackages, everything seems to be OK, however. > If you (and Bas and other readers here) think we should fix the issue > right now I'm fine if you apply the patch below and we should seriously > test the metapackage creation of each Blend *before* 2016-12-05. > > What do you think? I am ready to test and also to fix; however my know-how ends here. I don't know what is wrong with the fix. Just wondering, and starting to really get worried: None of the debian-blends maintainers has enough Perl knowledge to fix this? If we all do not know Perl, why do we use that language in one of our central tools? That sounds to me even more RC than the bug itself... Cheers Ole
Bug#840094: blends-dev: Does not recognize multiline dependencies
Hi Bas, On Wed, Nov 09, 2016 at 06:27:13PM +0100, Sebastiaan Couwenberg wrote: > On 11/09/2016 03:35 PM, Andreas Tille wrote: > > If you (and Bas and other readers here) think we should fix the issue > > right now I'm fine if you apply the patch below and we should seriously > > test the metapackage creation of each Blend *before* 2016-12-05. > > > > What do you think? > > I think supporting the deb822 format should be a Blends release goal for > buster, I fully agree. I admit I did way less for blends-dev than I intended to do but other tasks that felt more urgent occupied all my Debian time. > it's indeed to late in the stretch dev cycle in my opinion. That would mean to lower the severity of #840094. Ole, are you OK with this? In addition we might mention this in the docs and publish it on the web page at least. Kind regards Andreas. -- http://fam-tille.de
Bug#840094: blends-dev: Does not recognize multiline dependencies
On Wed, Nov 09, 2016 at 01:42:37PM +0100, Ole Streicher wrote: > On 09.11.2016 12:47, Andreas Tille wrote: > > In other words: Once it was defined as syntax for these control files > > that newlines need to be escaped. I do not like it and as I said this > > is fixed in the long-term pending rewrite. However, the bug is not > > serious but at best wishlist. Would you follow this arguing? > > > > Not really. My point here is that this happens really unexpected, and > since blend-gen-control doesn't complain about the then wrong format, > one silently gets wrong dependencies. At least I did in the first > versions of debian-astro (<0.5). I fully agree that this is a bad situation. That's a problem inherited from the first days when blends-dev was part of debian-edu package where it just was implemented that way and people were aware of this. > We have a clear definition of how these files should look like, namely > RFC822, and this also defines continuation lines. Unfortunately in this specific feature tasks files are not RFC822 compliant, which sucks, yes. Its even not documented (I just checked since I intended to document it at some point in time but can't find it :-( ) > Look at > https://blends.debian.org/blends/ch08.html#edittasksfiles - it is > blends-gen-control that isn't conform to that. Yes. > I would think that there is also a quick fix for it -- the tool already > handles continuation lines for the tasks description, so one could > probably just take that. I have no glue of all the Perl $@^!~ special > chars, but wouldn't do it something like the attached patch (after > removing the obvious errors from it)? > > Or something else just adopted from lines 556-562 of blends-gen-control? While I fully agree that we should fix this I'm not fully convinced how to sensibly proceed here. The problematic thing is that we are quite short before a release and if we might break metapackage creation in some way we might get in trouble. I'm no Perl programmer myself (even if I think your patch looks sensible) and so IMHO staying conservative and add some line ending escapes could be the less invasive change. For the future I think we should start a common effort to switch to blends-dev 0.7[1] which neither has the problem above nor some other problems. If you (and Bas and other readers here) think we should fix the issue right now I'm fine if you apply the patch below and we should seriously test the metapackage creation of each Blend *before* 2016-12-05. What do you think? Kind regards Andreas. > diff --git a/devtools/blend-gen-control b/devtools/blend-gen-control > index 1aba552..cde3237 100755 > --- a/devtools/blend-gen-control > +++ b/devtools/blend-gen-control > @@ -566,9 +566,14 @@ sub load_task { > my $header; > for $header (qw(Depends Recommends Suggests)) { > if (m/^$header:\s+(.+)$/ && $1 !~ /^\s*$/) { > + my $pkgs = $1; > + while () { > + last if (m/^\S+/ || m/^\s*$/); > + $pkgs .= $_; > + } > $taskinfo{$curpkg}{$header} = () > if (! exists $taskinfo{$curpkg}{$header}); > -my ($pkglist, $missinglist) = process_pkglist($1); > +my ($pkglist, $missinglist) = process_pkglist($pkgs); > push(@{$taskinfo{$curpkg}{$header}}, @{$pkglist}); > > $haspackages += $#{$taskinfo{$curpkg}{$header}} + 1; [1] https://anonscm.debian.org/git/blends/blends-gsoc.git -- http://fam-tille.de
Bug#840094: blends-dev: Does not recognize multiline dependencies
Hi Andreas, On 09.11.2016 12:47, Andreas Tille wrote: > In other words: Once it was defined as syntax for these control files > that newlines need to be escaped. I do not like it and as I said this > is fixed in the long-term pending rewrite. However, the bug is not > serious but at best wishlist. Would you follow this arguing? > Not really. My point here is that this happens really unexpected, and since blend-gen-control doesn't complain about the then wrong format, one silently gets wrong dependencies. At least I did in the first versions of debian-astro (<0.5). We have a clear definition of how these files should look like, namely RFC822, and this also defines continuation lines. Look at https://blends.debian.org/blends/ch08.html#edittasksfiles - it is blends-gen-control that isn't conform to that. I would think that there is also a quick fix for it -- the tool already handles continuation lines for the tasks description, so one could probably just take that. I have no glue of all the Perl $@^!~ special chars, but wouldn't do it something like the attached patch (after removing the obvious errors from it)? Or something else just adopted from lines 556-562 of blends-gen-control? Best regards Ole diff --git a/devtools/blend-gen-control b/devtools/blend-gen-control index 1aba552..cde3237 100755 --- a/devtools/blend-gen-control +++ b/devtools/blend-gen-control @@ -566,9 +566,14 @@ sub load_task { my $header; for $header (qw(Depends Recommends Suggests)) { if (m/^$header:\s+(.+)$/ && $1 !~ /^\s*$/) { + my $pkgs = $1; + while () { + last if (m/^\S+/ || m/^\s*$/); + $pkgs .= $_; + } $taskinfo{$curpkg}{$header} = () if (! exists $taskinfo{$curpkg}{$header}); -my ($pkglist, $missinglist) = process_pkglist($1); +my ($pkglist, $missinglist) = process_pkglist($pkgs); push(@{$taskinfo{$curpkg}{$header}}, @{$pkglist}); $haspackages += $#{$taskinfo{$curpkg}{$header}} + 1;
Bug#840094: blends-dev: Does not recognize multiline dependencies
Hi Ole, On Sat, Oct 08, 2016 at 11:33:27AM +0200, Sebastiaan Couwenberg wrote: > On 10/08/2016 11:15 AM, Ole Streicher wrote: > > When a "Depends:" field in a task file contains more than one line, only > > the first line is used to create the dependency in debian/control. All > > others are silently ignored. > > You can have dependencies on multiple lines when you escape the newline. In other words: Once it was defined as syntax for these control files that newlines need to be escaped. I do not like it and as I said this is fixed in the long-term pending rewrite. However, the bug is not serious but at best wishlist. Would you follow this arguing? Kind regards Andreas. -- http://fam-tille.de
Bug#840094: blends-dev: Does not recognize multiline dependencies
On Sat, Oct 08, 2016 at 11:33:27AM +0200, Sebastiaan Couwenberg wrote: > On 10/08/2016 11:15 AM, Ole Streicher wrote: > > When a "Depends:" field in a task file contains more than one line, only > > the first line is used to create the dependency in debian/control. All > > others are silently ignored. > > You can have dependencies on multiple lines when you escape the newline. Correct. > See for example: > > https://anonscm.debian.org/cgit/blends/projects/gis.git/tree/tasks/devel > > That may be sufficient for your needs, although supporting the same > syntax as regular debian/control files would be nice too. BTW, this is done in the GSoC rewrite[1] which unfortunately did not made it into production yet. Probably we should try to make a common effort to switch to this version for the next release cycle. Kind regards Andreas. [1] ssh://git.debian.org/git/blends/blends-gsoc.git -- http://fam-tille.de
Bug#840094: blends-dev: Does not recognize multiline dependencies
On 10/08/2016 11:15 AM, Ole Streicher wrote: > When a "Depends:" field in a task file contains more than one line, only > the first line is used to create the dependency in debian/control. All > others are silently ignored. You can have dependencies on multiple lines when you escape the newline. See for example: https://anonscm.debian.org/cgit/blends/projects/gis.git/tree/tasks/devel That may be sufficient for your needs, although supporting the same syntax as regular debian/control files would be nice too. Kind Regards, Bas -- GPG Key ID: 4096R/6750F10AE88D4AF1 Fingerprint: 8182 DE41 7056 408D 6146 50D1 6750 F10A E88D 4AF1
Bug#840094: blends-dev: Does not recognize multiline dependencies
Package: blends-dev Version: 0.6.94 Severity: serious When a "Depends:" field in a task file contains more than one line, only the first line is used to create the dependency in debian/control. All others are silently ignored. I observed this on the "debian-astro" package which uses blends-dev. Since this is quite dangerous (it is not recognized unless one really checks the generated d/control file) and generates incorrect blends packages, I make it severity "serious". Short code inspection shows that there seem to be *two* places where the list is parsed (in blend-gen-control): once on gen_control (aroound line 250), and then in load_task (around line 570). In the first appearance, only the first line is used, while in the second one the full list is there. Unfortunately, my perl is not enough to fix this... Best regards Ole