Re: DLAs in the website: some updates and issues

[Sylvain Beucler]

Hi,

On 18/03/2019 15:56, Sylvain Beucler wrote:
> On Thu, Mar 07, 2019 at 08:02:18PM +0100, Laura Arjona Reina wrote:
>> El 5/3/19 a las 16:07, Markus Koschany escribió:
>>> thank your for your work on our website. Ideally we would like to make
>>> the whole process fully automatic without the need for any manual
>>> interaction. 
>> This is being discussed in #859123: automate import of DLAs and DSAs in
>> www.debian.org
>>
>> https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=859123
>>
>> In particular, I think this message from Lev Lamberov is relevant:
>> https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=859123#20
>>
>>> Can you tell us more about the current work flow of our DSA
>>> announcements on the front page? 
>> DSAs are manually imported by a web team member or a security team
>> member, using the parse_advisory.pl script.
>>
>>> Does someone from the webteam reviews
>>> the generation by hand? 
>> Usually yes, but also, as it is noted in Lev's message, I think the
>> format of DSA is more standard.
> I had a look at parse-dla.pl / parse-advisory.pl, and let's face it:
> it's a bunch of ad-hoc regexps that happen to work. Most of the times.
>
> I couldn't find a satisfying way to fix the trailing 
> recurring bug.

FYI I tracked down the difference ("For the (old)stable" vs. "For Debian
X") and adapted the regexp.
This confirms DLA formatting is on par with DSA's, the conversion script
is just fragile.


>>> I'm sure we can improve the current parse-dla.pl
>>> script and fix those markup bugs. We also thought about downloading the
>>> announcements from  https://lists.debian.org/debian-lts-announce/ and
>>> then create the DLA on the web page automatically. Is this a viable plan?
>>>
>> I don't know.
>>
>> I guess that if the security team does not that already it's probably
>> because of a reason (or maybe because nobody in the web team could find
>> the time+skills+motivation needed to make it possible...).
> So the core issue is taking a text mail and automagically generate a
> HTML equivalent.
>
> Lev suggested 4 months ago that LTS and DebSec work on a common
> mark-up format.  We could attempt to switch to MarkDown, but from
> experience it breaks easily, especially wrt newlines.
>
> Alternatively, a simple answer would be to keep the headers parsing
> (Package/Version/CVE ID/Debian Bug) but import the free-form
> description text verbatim as a monospace block (such as ).
> i.e. stop coping with ul/li, just auto-link https://... bits.
>
> I don't suggest merely linking the list archives, since AFAIU there is
> demand for advisories translations (if there isn't, though, a link
> would be enough IMHO).
>
> What do you think?
>
> Cheers!
> Sylvain

Re: DLAs in the website: some updates and issues

[Sylvain Beucler]

Hi,

On Thu, Mar 07, 2019 at 08:02:18PM +0100, Laura Arjona Reina wrote:
> El 5/3/19 a las 16:07, Markus Koschany escribió:
> > thank your for your work on our website. Ideally we would like to make
> > the whole process fully automatic without the need for any manual
> > interaction. 
> 
> This is being discussed in #859123: automate import of DLAs and DSAs in
> www.debian.org
> 
> https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=859123
> 
> In particular, I think this message from Lev Lamberov is relevant:
> https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=859123#20
> 
> > Can you tell us more about the current work flow of our DSA
> > announcements on the front page? 
> 
> DSAs are manually imported by a web team member or a security team
> member, using the parse_advisory.pl script.
> 
> > Does someone from the webteam reviews
> > the generation by hand? 
> 
> Usually yes, but also, as it is noted in Lev's message, I think the
> format of DSA is more standard.

I had a look at parse-dla.pl / parse-advisory.pl, and let's face it:
it's a bunch of ad-hoc regexps that happen to work. Most of the times.

I couldn't find a satisfying way to fix the trailing 
recurring bug.


> > I'm sure we can improve the current parse-dla.pl
> > script and fix those markup bugs. We also thought about downloading the
> > announcements from  https://lists.debian.org/debian-lts-announce/ and
> > then create the DLA on the web page automatically. Is this a viable plan?
> > 
> 
> I don't know.
> 
> I guess that if the security team does not that already it's probably
> because of a reason (or maybe because nobody in the web team could find
> the time+skills+motivation needed to make it possible...).

So the core issue is taking a text mail and automagically generate a
HTML equivalent.

Lev suggested 4 months ago that LTS and DebSec work on a common
mark-up format.  We could attempt to switch to MarkDown, but from
experience it breaks easily, especially wrt newlines.

Alternatively, a simple answer would be to keep the headers parsing
(Package/Version/CVE ID/Debian Bug) but import the free-form
description text verbatim as a monospace block (such as ).
i.e. stop coping with ul/li, just auto-link https://... bits.

I don't suggest merely linking the list archives, since AFAIU there is
demand for advisories translations (if there isn't, though, a link
would be enough IMHO).

What do you think?

Cheers!
Sylvain

Re: DLAs in the website: some updates and issues

[Sylvain Beucler]

Hi,

On 18/03/2019 09:55, Brian May wrote:
> Laura Arjona Reina  writes:
>
>> Other option is, instead of looking at the html code, doing
>>
>> make dla-123-1.en.html
>>
>> and open the resulting html file with a web browser.
> This command did not work for me, I had to use "make -C 2019
> dla-1716.en.html" instead.
>
> Which leads me to a 2nd point, after reading the wiki page
> 
> I was expecting a filename like:
>
> 2019/dla-1716-1.*
>
> but parse-dla.pl gave me instead:
>
> 2019/dla-1716.*
>
> I notice this seems to match the existing convention, so maybe this is
> an error in the wiki?
I confirm.

These instructions are pretty new. I made fixes a few weeks ago but I
overlooked this "-1".

Fixed the wiki page when testing for my work today :)

Cheers!
Sylvain

Re: DLAs in the website: some updates and issues

[Brian May]

Laura Arjona Reina  writes:

> Other option is, instead of looking at the html code, doing
>
> make dla-123-1.en.html
>
> and open the resulting html file with a web browser.

This command did not work for me, I had to use "make -C 2019
dla-1716.en.html" instead.

Which leads me to a 2nd point, after reading the wiki page

I was expecting a filename like:

2019/dla-1716-1.*

but parse-dla.pl gave me instead:

2019/dla-1716.*

I notice this seems to match the existing convention, so maybe this is
an error in the wiki?

Regards
-- 
Brian May 

Re: DLAs in the website: some updates and issues

[Laura Arjona Reina]

Hello
Sorry for the late reply

El 5/3/19 a las 16:07, Markus Koschany escribió:
> Hello Laura,
> 
> thank your for your work on our website. Ideally we would like to make
> the whole process fully automatic without the need for any manual
> interaction. 

This is being discussed in #859123: automate import of DLAs and DSAs in
www.debian.org

https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=859123

In particular, I think this message from Lev Lamberov is relevant:
https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=859123#20

Can you tell us more about the current work flow of our DSA
> announcements on the front page? 

DSAs are manually imported by a web team member or a security team
member, using the parse_advisory.pl script.

Does someone from the webteam reviews
> the generation by hand? 

Usually yes, but also, as it is noted in Lev's message, I think the
format of DSA is more standard.

I'm sure we can improve the current parse-dla.pl
> script and fix those markup bugs. We also thought about downloading the
> announcements from  https://lists.debian.org/debian-lts-announce/ and
> then create the DLA on the web page automatically. Is this a viable plan?
> 

I don't know.

I guess that if the security team does not that already it's probably
because of a reason (or maybe because nobody in the web team could find
the time+skills+motivation needed to make it possible...).

Kind regards
-- 
Laura Arjona Reina
https://wiki.debian.org/LauraArjona

Re: DLAs in the website: some updates and issues

[Holger Levsen]

Dear Laura,

thanks for your feedback and your help along the way!

On Tue, Mar 05, 2019 at 03:50:01PM +0100, Laura Arjona Reina wrote:
> I have created an usertag to group the bugs related to the "lts" section of
> the website:
> https://udd.debian.org/cgi-bin/bts-usertags.cgi?user=www.debian.org%40packages.debian.org=lts

very nice, thanks.

> https://wiki.debian.org/LTS/Development#Publishing_updates_on_the_website
[...]
> make dla-123-1.en.html
 
TIL & added that info to the wiki! Thanks!


-- 
tschau,
Holger

---
   holger@(debian|reproducible-builds|layer-acht).org
   PGP fingerprint: B8BF 5413 7B09 D35C F026 FE9D 091A B856 069A AA1C


signature.asc
Description: PGP signature

Re: DLAs in the website: some updates and issues

[Markus Koschany]

Hello Laura,

thank your for your work on our website. Ideally we would like to make
the whole process fully automatic without the need for any manual
interaction. Can you tell us more about the current work flow of our DSA
announcements on the front page? Does someone from the webteam reviews
the generation by hand? I'm sure we can improve the current parse-dla.pl
script and fix those markup bugs. We also thought about downloading the
announcements from  https://lists.debian.org/debian-lts-announce/ and
then create the DLA on the web page automatically. Is this a viable plan?

Regards,

Markus



signature.asc
Description: OpenPGP digital signature

DLAs in the website: some updates and issues

[Laura Arjona Reina]

Dear Debian LTS team
Thanks for your work adding the DLA advisories to the website.

I have created an usertag to group the bugs related to the "lts" section of the 
website:


https://udd.debian.org/cgi-bin/bts-usertags.cgi?user=www.debian.org%40packages.debian.org=lts

It seems that the parse-dla.pl script sometimes does not put the closing , 
 or  tags in the correct place, or introduces weird HTML tags (look for 
"HTML fix" in the commit logs from last days for some examples).


I don't know if this is a bug or just some corner cases that would require much 
effort to catch in the script, so I didn't open a bug for now.


My proposal would be that the people parsing the advisory checks the generated 
wml file(s) prior to commit. I see that the proposed workflow in 
https://wiki.debian.org/LTS/Development#Publishing_updates_on_the_website 
includes the line


$EDITOR 2019/dla-123-1* # make sure everything looks good

Other option is, instead of looking at the html code, doing

make dla-123-1.en.html

and open the resulting html file with a web browser.

If after reviewing several files you feel there is certain pattern producing the 
bad HTML, maybe we can improve the Perl script or the current format of the 
advisories to avoid the issues in the future.


Kind regards,
--
Laura Arjona Reina
https://wiki.debian.org/LauraArjona