Re: Release 1.2.1: status

[Leo Famulari]

On Sun, Mar 21, 2021 at 12:00:20AM +0100, zimoun wrote:
> We discussed that and I agree.  We also discussed some tagging and Maxim
> did some tests, IIRC.  Please go ahead and let try if it helps to
> synchronize. :-)

Here is the checklist:

https://bugs.gnu.org/47297

I've added a few items, but everyone should add the other tasks they
know about.

Re: Release 1.2.1: status

[Luis Felipe]

Thanks for the details, zimoun.


On Sunday, March 21, 2021 12:16 AM, zimoun  wrote:

[...]

> > Also, I got a backtrace when checking icecat:
> > ★★★
> > Backtrace:cecat@78.8.0-guix0-preview1 [archival]...
>
> [...]
>
> > ice-9/boot-9.scm:1667:16: In procedure raise-exception:
> > In procedure bv-length: Wrong type argument in position 1 (expecting 
> > bytevector): #f
> > ★★★
>
> Indeed, there is a bug. Because the source of ’icecat’ raises a case
> that is not handled by ’check-archival’ in (guix lint).

[...]

> Could you open
> a bug report for that? Just to not forget to fix it. :-)

Done: https://issues.guix.gnu.org/47293

Re: Release 1.2.1: status

[zimoun]

Hi Luis,

Thanks for testings and reporting.


On Fri, 19 Mar 2021 at 22:26, Luis Felipe  wrote:
> ‐‐‐ Original Message ‐‐‐
> On Thursday, March 18, 2021 2:28 PM, zimoun  wrote:
>
> [...]
>
>> We are still missing a good story to monitor what is archived on
>> Software Heritage and what is not. Because for now there is rate limit,
>> I am not able to automate… Well, it is a long WIP. :-)
>>
>> To help and avoid this rate limit, if all of us simply run:
>>
>> for pkg in $(guix package -I | cut -f1);
>> do
>> guix lint -c archival $pkg
>> done
>>
>> for all our profiles, it will ensure at least a coverage for the
>> packages using git-fetch that we individually care. Note the option
>> --manifest for “guix lint” is missing… should happen soon if no one
>> beats me. :-)
>
> I tried this with my user profile and the script hit the Software
> heritage limit after reporting the status of 33 packages (most of
> which are not archived). So, for the rest of the packages, you are
> asked to try again later (I have 95 packages in my profile).

There is 2 rate limit: one for saving and one for requesting.

Each time you do “guix lint -c archival ”, Guix requests to SWH via
their API [1] if the package is already in.  AFAIR, it is 120 requests
per hour.

Then if it is not, Guix saves to SWH via their API.  And this rate is
very low, maybe 10 per hour.  Well, if I remember correctly.


1: 

> Is "guix lint -c archival $pkg" supposed to poke Software Heritage to
> archive the $pkg if it is not archived yet? I ask because I ran the
> script later and I got the same output from the first run, i.e.,
> packages reported to be missing from Software Heritage were still
> reported as such, instead of being planned for archive.

Currently, the request/save via “guix lint -c archival” is not optimal.
For instance, the source of the package “libvirt“ is url-fetch so
requesting for it is not necessary because it cannot be saved via their
API.  And I not remember exactly how the ‘tarball’ request is counted.

BTW, the packages using ’url-fetch’ should be ingested by SWH via their
nixguix loader reading the sources.json [2].  And for example:

--8<---cut here---start->8---
$ guix lint -c archival libvirt
gnu/packages/virtualization.scm:1070:5: libvirt@5.8.0: source not archived on 
Software Heritage

$ guix download -H sha256 -f base64 
https://libvirt.org/sources/libvirt-5.8.0.tar.xz

Starting download of /tmp/guix-file.XWwdFj
>From https://libvirt.org/sources/libvirt-5.8.0.tar.xz...
 libvirt-5.8.0.tar.xz  12.5MiB678KiB/s 00:19 [##] 100.0%
/gnu/store/1pgi1bl8p7jv2mhk83kv7raak2b4k1w5-libvirt-5.8.0.tar.xz
4jMoKJsYve3B6Wb2wmQCspgxScZg7YvVLNpv6rDCDFU=
--8<---cut here---end--->8---

and in the same time, the sources.json contains: 

--8<---cut here---start->8---
{
  "type": "url",
  "urls": [
"https://libvirt.org/sources/libvirt-5.8.0.tar.xz";
  ],
  "integrity": "sha256-4jMoKJsYve3B6Wb2wmQCspgxScZg7YvVLNpv6rDCDFU="
},
--8<---cut here---end--->8---

Well, 2 possible explanations:

 1) The tarball is not in SWH; because their loader fails on it or for
 whatever else reasons
 
 2) Or the tarball is archived by SWH but they use another hashing
 (SWH-ID) than the NAR.  Well, with the information in the package, Guix
 is not able to ask to SWH with the correct hash.  That’s the main
 motivation behind disarchive [3].

Last, Guix is not able to deal with hg-fetch or svn-fetch.  In this
message [4] and the 2 follow-up in the thread, there is some explanation
to implement ‘lookup-subversion-revision’ in (guix swh). Maybe for the
next release on Nov. ;-)

Well, the dance SWH needs some love. :-)


Thanks for trying!  It really helps to have this kind of feedback.



2: 
3: 
4: 


> Also, I got a backtrace when checking icecat:
>
> ★★★
> Backtrace:cecat@78.8.0-guix0-preview1 [archival]...

[...]

> ice-9/boot-9.scm:1667:16: In procedure raise-exception:
> In procedure bv-length: Wrong type argument in position 1 (expecting 
> bytevector): #f
> ★★★

Indeed, there is a bug.  Because the source of ’icecat’ raises a case
that is not handled by ’check-archival’ in (guix lint).

Basically in the snippet:

--8<---cut here---start->8---
 (match (lookup-content (content-hash-value hash)
(symbol->string
 (content-hash-algorithm hash)))
--8<---cut here---end--->8---

’lookup-content’ expect a bytevector for ’content-hash’ and in the case
of ’icecat’, it returns #f.  Then rai

Re: Release 1.2.1: status

[Leo Famulari]

On Sat, Mar 20, 2021 at 11:56:57PM +0100, zimoun wrote:
> > From the wip-next-release branch, we should cherry-pick the tzdata
> > updates and Qt 4 removal.
> >
> > I'll rewrite the branch with those commits today, and then see about
> > getting it built on CI.
> 
> Do you mean cherry-pick and then push them to master?

We can remove Qt 4 on master, but tzdata cannot be changed on the master
branch (too many dependents). We need to build it on CI, on its own
branch, and then deploy the change on master.

Re: Release 1.2.1: status

[zimoun]

Hi Andreas,

On Fri, 19 Mar 2021 at 19:31, Andreas Enge  wrote:
> Am Thu, Mar 18, 2021 at 03:28:38PM +0100 schrieb zimoun:
>>   guix weather --display-missing
>
> I am giving it a try, but after about one hour at 100% CPU on one core it
> is still only half way through. Is this normal? I think I will stop it to
> at least redirect the output into a file...

I sympathise.  It depends the architecture from my experience.  Some are
not responding at all.  Which one are you “guix weather”ing?

If your machine is always up, you can try:

  until guix weather; do echo ok; done

even it is not really reliable.


Cheers,
simon

Re: Release 1.2.1: status

[zimoun]

Hi Leo,

On Sat, 20 Mar 2021 at 16:01, Leo Famulari  wrote:
> I suggest we use debbugs to keep track of tasks for the release.
>
> We can create a new bug called "1.2.1 release checklist".
>
> This bug can be made to depend on other bugs using the "block" feature
> of debbugs:
>
> https://debbugs.gnu.org/server-control.html
>
> Concretely, this means we send email to debbugs with messages like
> "block NNN with YYY", which means that NNN cannot be closed until YYY is
> closed.
>
> What do you think? Should I create a new bug ticket for this?

We discussed that and I agree.  We also discussed some tagging and Maxim
did some tests, IIRC.  Please go ahead and let try if it helps to
synchronize. :-)

Cheers,
simon

Re: Release 1.2.1: status

[zimoun]

Hi Leo,

On Sat, 20 Mar 2021 at 14:09, Leo Famulari  wrote:
> On Fri, Mar 19, 2021 at 09:50:55AM +0100, zimoun wrote:
>> The release work happens on master.  The branch wip-next-release
>> contains fixes, but AFAIK, it is not built by the CI, and these fixes
>> are ’core-updates’-like changes; I do not know if it is doable to merge
>> on time.
>
> I agree. The scope of ungrafting has grown way too large, and we need to
> spend the rest of the time fixing bugs.

I agree.  For instance, I have started to see if zstd could be ungrafted
and with my knowledge, my time available considering the other parts to
check, and my CPU available to build a lot, IMHO, it is not possible to
be on time.

> Maybe there are some things we could ungraft in time, or that are
> failing to work properly as grafts (ImageMagick / broken Inkscape?) and
> should thus be ungrafted or somehow adjusted.

I have not check at ImageMagick specifically.  My feeling about the
recent grafts is that they are more than security fixes because they are
often version upgrades.  And version upgrade often implies breakage here
or there.  Therefore, I agree they need to be adjusted and maybe, IMHO,
we could have a gratf freeze starting on April 1rst in order to have
time to check that everything is fine, AFAWCT.

> From the wip-next-release branch, we should cherry-pick the tzdata
> updates and Qt 4 removal.
>
> I'll rewrite the branch with those commits today, and then see about
> getting it built on CI.

Do you mean cherry-pick and then push them to master?


Cheers,
simon

Re: Release 1.2.1: status

[Leo Famulari]

I suggest we use debbugs to keep track of tasks for the release.

We can create a new bug called "1.2.1 release checklist".

This bug can be made to depend on other bugs using the "block" feature
of debbugs:

https://debbugs.gnu.org/server-control.html

Concretely, this means we send email to debbugs with messages like
"block NNN with YYY", which means that NNN cannot be closed until YYY is
closed.

What do you think? Should I create a new bug ticket for this?

Re: Release 1.2.1: status

[Leo Famulari]

On Fri, Mar 19, 2021 at 09:50:55AM +0100, zimoun wrote:
> The release work happens on master.  The branch wip-next-release
> contains fixes, but AFAIK, it is not built by the CI, and these fixes
> are ’core-updates’-like changes; I do not know if it is doable to merge
> on time.

I agree. The scope of ungrafting has grown way too large, and we need to
spend the rest of the time fixing bugs.

Maybe there are some things we could ungraft in time, or that are
failing to work properly as grafts (ImageMagick / broken Inkscape?) and
should thus be ungrafted or somehow adjusted.

>From the wip-next-release branch, we should cherry-pick the tzdata
updates and Qt 4 removal.

I'll rewrite the branch with those commits today, and then see about
getting it built on CI.

Re: Release 1.2.1: status

[Luis Felipe]

Hi zimoun :)


‐‐‐ Original Message ‐‐‐
On Thursday, March 18, 2021 2:28 PM, zimoun  wrote:

[...]

> We are still missing a good story to monitor what is archived on
> Software Heritage and what is not. Because for now there is rate limit,
> I am not able to automate… Well, it is a long WIP. :-)
>
> To help and avoid this rate limit, if all of us simply run:
>
> for pkg in $(guix package -I | cut -f1);
> do
> guix lint -c archival $pkg
> done
>
> for all our profiles, it will ensure at least a coverage for the
> packages using git-fetch that we individually care. Note the option
> --manifest for “guix lint” is missing… should happen soon if no one
> beats me. :-)

I tried this with my user profile and the script hit the Software heritage 
limit after reporting the status of 33 packages (most of which are not 
archived). So, for the rest of the packages, you are asked to try again later 
(I have 95 packages in my profile).

Is "guix lint -c archival $pkg" supposed to poke Software Heritage to archive 
the $pkg if it is not archived yet? I ask because I ran the script later and I 
got the same output from the first run, i.e., packages reported to be missing 
from Software Heritage were still reported as such, instead of being planned 
for archive.

Also, I got a backtrace when checking icecat:

★★★
gnu/packages/virtualization.scm:1070:5: libvirt@5.8.0: Software Heritage rate 
limit reached; try again later
Backtrace:cecat@78.8.0-guix0-preview1 [archival]...
  12 (primitive-load "/home/yo/.config/guix/current/bin/guix")
In guix/ui.scm:
  2164:12 11 (run-guix-command _ . _)
In ice-9/boot-9.scm:
  1736:10 10 (with-exception-handler _ _ #:unwind? _ # _)
  1731:15  9 (with-exception-handler # ?)
In srfi/srfi-1.scm:
634:9  8 (for-each # ?)
In guix/scripts/lint.scm:
 65:4  7 (run-checkers # ?)
In srfi/srfi-1.scm:
634:9  6 (for-each # ?)
In guix/scripts/lint.scm:
74:21  5 (_ _)
In guix/lint.scm:
   1225:4  4 (check-archival _)
   1092:2  3 (call-with-networking-fail-safe _ _ _)
In ice-9/boot-9.scm:
  1736:10  2 (with-exception-handler _ _ #:unwind? _ # _)
  1669:16  1 (raise-exception _ #:continuable? _)
  1667:16  0 (raise-exception _ #:continuable? _)

ice-9/boot-9.scm:1667:16: In procedure raise-exception:
In procedure bv-length: Wrong type argument in position 1 (expecting 
bytevector): #f
★★★

Re: Release 1.2.1: status

[Luis Felipe]

‐‐‐ Original Message ‐‐‐
On Friday, March 19, 2021 6:31 PM, Andreas Enge  wrote:

> Hello,
>
> Am Thu, Mar 18, 2021 at 03:28:38PM +0100 schrieb zimoun:
>
> > guix weather --display-missing
>
> I am giving it a try, but after about one hour at 100% CPU on one core it
> is still only half way through. Is this normal? I think I will stop it to
> at least redirect the output into a file...


In my case, after ~20 minutes, it progressed about 20-30%, and then the 
progress bar got reset and started again. Later, an hour and ten minutes have 
passed since I ran the command, it reached around 30% again, progressing very, 
very slowly. Then, a gnu-tls error occurred:

★★★
Backtrace:
  11 (primitive-load "/home/yo/.config/guix/current/bin/guix")
In guix/ui.scm:
  2164:12 10 (run-guix-command _ . _)
In ice-9/boot-9.scm:
  1736:10  9 (with-exception-handler _ _ #:unwind? _ # _)
  1731:15  8 (with-exception-handler # …)
In guix/scripts/weather.scm:
552:9  7 (_)
In guix/build/utils.scm:
   569:23  6 (every* # …)
In guix/scripts/weather.scm:
   553:19  5 (_ "https://ci.guix.gnu.org";)
   116:17  4 (report-server-coverage _ _ #:display-missing? _)
In unknown file:
   3 (_ # . #)
In guix/substitutes.scm:
   315:31  2 (lookup-narinfos _ _ #:open-connection _ # _)
   238:26  1 (fetch-narinfos _ _ #:open-connection _ # _)
In ice-9/boot-9.scm:
  1669:16  0 (raise-exception _ #:continuable? _)

ice-9/boot-9.scm:1669:16: In procedure raise-exception:
Throw to key `gnutls-error' with args `(# read_from_session_record_port)'.
★★★

The text in # is Spanish, 
meaning something like: Error in the push function.

This is my system information:

OS: Guix System 1ab03fb74505458e7754dce338a5da29dc754d80 x86_64
Kernel: 5.11.7-gnu
CPU: Intel i3-8100 (4) @ 3.600GHz
GPU: Intel 8th Gen Core Processor Gaussian Mixture Model
Memory: 1804MiB / 3766MiB

Re: Release 1.2.1: status

[Andreas Enge]

Hello,

Am Thu, Mar 18, 2021 at 03:28:38PM +0100 schrieb zimoun:
>   guix weather --display-missing

I am giving it a try, but after about one hour at 100% CPU on one core it
is still only half way through. Is this normal? I think I will stop it to
at least redirect the output into a file...

Andreas

Re: Release 1.2.1: status

[zimoun]

Hi Chris,

On Thu, 18 Mar 2021 at 23:37, Chris Marusich  wrote:

> More review is welcome, but unless some unforeseen issues are
> discovered, I agree that we can go ahead with adding support for
> powerpc64le-linux.

Great!  

> Where is the release work happening?  Where can I merge or apply these
> changes to ensure they get included in the next release?

The release work happens on master.  The branch wip-next-release
contains fixes, but AFAIK, it is not built by the CI, and these fixes
are ’core-updates’-like changes; I do not know if it is doable to merge
on time.

Well, maybe the patches could be directly applied to master, IMHO.

Quote the proposed Release timeline [1]:

A draft of the timeline is:

 - until April 1rst: fixes, check substitute availability, etc.
 - as soon as possible: start to build wip-next-release
 - merge branch wip-next-release when ready
 - on Monday 12th April, string freeze
 - couple of days after, branch the release and write the materials
   (ChangeLog and posts)
 - release

Cheers,
simon



PS: Sorry if the release process appears unclear.  Based on the
experiences of 1.2.0 and now this one, the idea is to propose then a
clear documented timeline for the next releases.  Basically, some
technical items are described in maintenance/doc/release.org [2] and
from my point of view, some items describing the (timeline) process
should be added.  Maybe in the manual so that we all know what we can
«expect».


1: 
2: 

Re: Release 1.2.1: status

[Christopher Baines]

Chris Marusich  writes:

> Hi simon,
>
> zimoun  writes:
>
>> First, thanks to Chris and folks, powerpc64le-linux will be [1] in the
>> next release, great! Isn’t it? :-)
>
> I'm very excited about this!  Efraim was able to rework our patches so
> they could be applied to master without rebuilding the entire world.
> The patches are currently under review here:
>
> https://debbugs.gnu.org/cgi/bugreport.cgi?bug=47182
>
> More review is welcome, but unless some unforeseen issues are
> discovered, I agree that we can go ahead with adding support for
> powerpc64le-linux.
>
> Where is the release work happening?  Where can I merge or apply these
> changes to ensure they get included in the next release?

In the past, a branch might appear for the release, but that hasn't
happened yet. Merging to master is the next step I believe, and as you
say, I think the patches are ready.


signature.asc
Description: PGP signature

Re: Release 1.2.1: status

[Chris Marusich]

Hi simon,

zimoun  writes:

> First, thanks to Chris and folks, powerpc64le-linux will be [1] in the
> next release, great! Isn’t it? :-)

I'm very excited about this!  Efraim was able to rework our patches so
they could be applied to master without rebuilding the entire world.
The patches are currently under review here:

https://debbugs.gnu.org/cgi/bugreport.cgi?bug=47182

More review is welcome, but unless some unforeseen issues are
discovered, I agree that we can go ahead with adding support for
powerpc64le-linux.

Where is the release work happening?  Where can I merge or apply these
changes to ensure they get included in the next release?

-- 
Chris


signature.asc
Description: PGP signature