Re: sha3-384 mismatch

[Seth Arnold]

On Fri, Dec 02, 2016 at 05:50:06PM +0100, Didier Roche wrote:
> However, as you can see in https://bugs.launchpad.net/bugs/1643893, I
> can still reproduce pretty easily here this kind of error (nework
> dropping and snapd disconnecting). While curl or wget can cope with the
> download as expected.

The last time I saw a Go program failing to download something from
another host, and dying in the same spot repeatably, it was because the
Go library authors didn't have EAGAIN (or was it EINTR?) errno handled
correctly everywhere in the libraries.

This was diagnosed by using 'strace' on the failing command and noticing
that a read() or recv() or recvmsg() or somoething similar wasn't retried
on EAGAIN. I don't know how the Go programmer in question found the
corresponding Go code to manipulate but I'd start by looking for read()
or recv() or recvmsg() system calls without EAGAIN error handling within
two lines or something.

Thanks


signature.asc
Description: PGP signature
-- 
Snapcraft mailing list
Snapcraft@lists.snapcraft.io
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/snapcraft

Re: sha3-384 mismatch

[Gustavo Niemeyer]

Okay, so curl is not retrying. That means the problem is specific to
something snapd is doing with the server.. might still be a problem on the
client or the server under those particular conditions.

Can I hand you a binary for you to try out?

On Mon, Dec 5, 2016 at 1:44 PM, Didier Roche  wrote:

> Le 05/12/2016 à 16:05, Gustavo Niemeyer a écrit :
>
> Xavier posted the exact URL of the failing snap in this thread:
>
> Note it's not *the* failing snap but *a* failing snap. Every "snap
> install" here is failing on my setup when they are more than a couple of
> MB. This is why I posted as such in the instructions on the bug.
>
> So, with curl -v -L, with the same snap than on the bug report, here are
> the results: http://paste.ubuntu.com/23583801/
> I did 10 successful downloads in a row. This snap is 23MB.
>
> I did retry with the new revision (49), 32MB.
> Tried 10 times with curl, 10 successful and complete downloads (one is
> http://paste.ubuntu.com/23583847/), with expected size and checksum.
> Tried 10 times with snapd, got hashsum mismatch 10 times. Download stops
> after few KBs up to few MBs.
>
> Cheers,
> Didier
>
>
>
> "[1] - https://public.apps.ubuntu.com/anon/download-snap/rFpKbTdZ
> 31LyAxWF6RpcerZov1TdtDly_24.snap (extracted from the log file)"
>
> Bret also posted another one above (thanks!).
>
>
> On Mon, Dec 5, 2016 at 12:52 PM, Didier Roche  wrote:
>
>> Le 05/12/2016 à 15:38, Gustavo Niemeyer a écrit :
>>
>>
>>
>> On Mon, Dec 5, 2016 at 12:23 PM, Didier Roche 
>> wrote:
>>
>>> I did though write on the bug: "contrary to curl or wget which both
>>> supports large downloads."
>>> The feedback thread mentioned as well "while same assets can be
>>> successfully downloaded via curl or wget".
>>>
>> I thought that was really obvious that they worked consistently and that
>>> I did rerun then multiple times or I wouldn't have opened the bug report +
>>> write this feedback. Sorry if that wasn't clear enough, let's move on :)
>>>
>>
>> If you file a bug and a developer asks for specific information that
>> wasn't provided, it means the specific information is not obvious.
>>
>> Is it the case?  Did you ever get a failure with them?  Are they retrying
>>> while they work? Do you have a verbose dumb of the process?
>>>
>>> Yes, as mentioned. I never got any failure with any of them and I did
>>> retry multiple times in loop when I saw the snapd failures.
>>> wget is in verbose mode by default and I never got any hint that it was
>>> retrying (just getting the normal download output).
>>>
>>> I did just try a verbose download in curl (here, an ubuntu 300M image).
>>> Here is the output: http://paste.ubuntu.com/23583568/. It seems that
>>> curl doesn't complain of any reconnect.
>>>
>>
>> You are downloading an image from an arbitrary server on the internet
>> unrelated to the problem we're trying to debug.
>>
>> Can you please attempt these several curl downloads while using the exact
>> same URL that failed for snapd?
>>
>>
>> As a developer asking for more debug information, can you please paste
>> the exact instructions on how to get those?
>>
>> I'm trying the https://public.apps.ubuntu.com/anon/download-snap/ based
>> url with the .snap showing up in the logs to get the exact same assets I
>> pasted snapd information on. However, curl -v returns (output stripped out):
>> *   Trying 162.213.33.92...
>> * Connected to public.apps.ubuntu.com (162.213.33.92) port 443 (#0)
>> * found 173 certificates in /etc/ssl/certs/ca-certificates.crt
>> * found 692 certificates in /etc/ssl/certs
>> * ALPN, offering http/1.1
>> * SSL connection using TLS1.2 / ECDHE_RSA_AES_128_GCM_SHA256
>> *  server certificate verification OK
>> *  server certificate status verification SKIPPED
>> *  common name: public.apps.ubuntu.com (matched)
>> *  server certificate expiration date OK
>> *  server certificate activation date OK
>> *  certificate public key: RSA
>> *  certificate version: #3
>> *  subject: C=GB,L=London,O=Canonical Group Ltd,CN=
>> public.apps.ubuntu.com
>> *  start date: Mon, 30 May 2016 00:00:00 GMT
>> *  expire date: Wed, 21 Jun 2017 12:00:00 GMT
>> *  issuer: C=US,O=DigiCert Inc,CN=DigiCert SHA2 Secure Server CA
>> *  compression: NULL
>> * ALPN, server did not agree to a protocol
>> > GET /anon/download-snap/YZ7LshLxDQQIrhAL6DMLub2yTVUA2DIK_15.snap
>> HTTP/1.1
>> > Host: public.apps.ubuntu.com
>> > User-Agent: curl/7.47.0
>> > Accept: */*
>> >
>> < HTTP/1.1 302 FOUND
>>
>> I guess that's due to the macaroon exchanged system and I'm not
>> authorized or something else?
>> Thanks for helping debugging.
>>
>> Cheers,
>> Didier
>>
>> --
>> Snapcraft mailing list
>> Snapcraft@lists.snapcraft.io
>> Modify settings or unsubscribe at: https://lists.ubuntu.com/mailm
>> an/listinfo/snapcraft
>>
>>
>
>
> --
>
> gustavo @ http://niemeyer.net
>
>
>
>
> --
> Snapcraft mailing list
> 

Re: sha3-384 mismatch

[Didier Roche]

Le 05/12/2016 à 16:05, Gustavo Niemeyer a écrit :
> Xavier posted the exact URL of the failing snap in this thread:
Note it's not *the* failing snap but *a* failing snap. Every "snap
install" here is failing on my setup when they are more than a couple of
MB. This is why I posted as such in the instructions on the bug.

So, with curl -v -L, with the same snap than on the bug report, here are
the results: http://paste.ubuntu.com/23583801/
I did 10 successful downloads in a row. This snap is 23MB.

I did retry with the new revision (49), 32MB.
Tried 10 times with curl, 10 successful and complete downloads (one is
http://paste.ubuntu.com/23583847/), with expected size and checksum.
Tried 10 times with snapd, got hashsum mismatch 10 times. Download stops
after few KBs up to few MBs.

Cheers,
Didier

>
> "[1]
> - 
> https://public.apps.ubuntu.com/anon/download-snap/rFpKbTdZ31LyAxWF6RpcerZov1TdtDly_24.snap
> 
>  (extracted
> from the log file)"
>
> Bret also posted another one above (thanks!).
>
>
> On Mon, Dec 5, 2016 at 12:52 PM, Didier Roche  > wrote:
>
> Le 05/12/2016 à 15:38, Gustavo Niemeyer a écrit :
>>
>>
>> On Mon, Dec 5, 2016 at 12:23 PM, Didier Roche
>> > wrote:
>>
>> I did though write on the bug: "contrary to curl or wget
>> which both supports large downloads."
>> The feedback thread mentioned as well "while same assets can
>> be successfully downloaded via curl or wget". 
>>
>> I thought that was really obvious that they worked
>> consistently and that I did rerun then multiple times or I
>> wouldn't have opened the bug report + write this feedback.
>> Sorry if that wasn't clear enough, let's move on :)
>>
>>
>> If you file a bug and a developer asks for specific information
>> that wasn't provided, it means the specific information is not
>> obvious.
>>
>>> Is it the case?  Did you ever get a failure with them?  Are
>>> they retrying while they work? Do you have a verbose dumb of
>>> the process?
>> Yes, as mentioned. I never got any failure with any of them
>> and I did retry multiple times in loop when I saw the snapd
>> failures.
>> wget is in verbose mode by default and I never got any hint
>> that it was retrying (just getting the normal download output).
>>
>> I did just try a verbose download in curl (here, an ubuntu
>> 300M image). Here is the output:
>> http://paste.ubuntu.com/23583568/
>> . It seems that curl
>> doesn't complain of any reconnect.
>>
>>
>> You are downloading an image from an arbitrary server on the
>> internet unrelated to the problem we're trying to debug.
>>
>> Can you please attempt these several curl downloads while using
>> the exact same URL that failed for snapd?
>
> As a developer asking for more debug information, can you please
> paste the exact instructions on how to get those?
>
> I'm trying the https://public.apps.ubuntu.com/anon/download-snap/
>  based url
> with the .snap showing up in the logs to get the exact same assets
> I pasted snapd information on. However, curl -v returns (output
> stripped out):
> *   Trying 162.213.33.92...
> * Connected to public.apps.ubuntu.com
>  (162.213.33.92) port 443 (#0)
> * found 173 certificates in /etc/ssl/certs/ca-certificates.crt
> * found 692 certificates in /etc/ssl/certs
> * ALPN, offering http/1.1
> * SSL connection using TLS1.2 / ECDHE_RSA_AES_128_GCM_SHA256
> *  server certificate verification OK
> *  server certificate status verification SKIPPED
> *  common name: public.apps.ubuntu.com
>  (matched)
> *  server certificate expiration date OK
> *  server certificate activation date OK
> *  certificate public key: RSA
> *  certificate version: #3
> *  subject: C=GB,L=London,O=Canonical Group
> Ltd,CN=public.apps.ubuntu.com 
> *  start date: Mon, 30 May 2016 00:00:00 GMT
> *  expire date: Wed, 21 Jun 2017 12:00:00 GMT
> *  issuer: C=US,O=DigiCert Inc,CN=DigiCert SHA2 Secure Server CA
> *  compression: NULL
> * ALPN, server did not agree to a protocol
> > GET /anon/download-snap/YZ7LshLxDQQIrhAL6DMLub2yTVUA2DIK_15.snap
> HTTP/1.1
> > Host: public.apps.ubuntu.com 
> > User-Agent: curl/7.47.0
> > Accept: */*
> >
> < HTTP/1.1 302 FOUND
>
> I guess that's due to the macaroon exchanged system and I'm not
> authorized or 

Re: sha3-384 mismatch

[Gustavo Niemeyer]

Yeah, -L is the flag that makes it follow the redirect, IIRC.

On Mon, Dec 5, 2016 at 1:03 PM, Bret A. Barker 
wrote:

> On Mon, Dec 05, 2016 at 03:52:32PM +0100, Didier Roche wrote:
> > Le 05/12/2016 à 15:38, Gustavo Niemeyer a écrit :
> > >
> > >
> > > On Mon, Dec 5, 2016 at 12:23 PM, Didier Roche  > > > wrote:
> > >
> > > I did though write on the bug: "contrary to curl or wget which
> > > both supports large downloads."
> > > The feedback thread mentioned as well "while same assets can be
> > > successfully downloaded via curl or wget".
> > >
> > > I thought that was really obvious that they worked consistently
> > > and that I did rerun then multiple times or I wouldn't have opened
> > > the bug report + write this feedback. Sorry if that wasn't clear
> > > enough, let's move on :)
> > >
> > >
> > > If you file a bug and a developer asks for specific information that
> > > wasn't provided, it means the specific information is not obvious.
> > >
> > >> Is it the case?  Did you ever get a failure with them?  Are they
> > >> retrying while they work? Do you have a verbose dumb of the
> process?
> > > Yes, as mentioned. I never got any failure with any of them and I
> > > did retry multiple times in loop when I saw the snapd failures.
> > > wget is in verbose mode by default and I never got any hint that
> > > it was retrying (just getting the normal download output).
> > >
> > > I did just try a verbose download in curl (here, an ubuntu 300M
> > > image). Here is the output: http://paste.ubuntu.com/23583568/
> > > . It seems that curl doesn't
> > > complain of any reconnect.
> > >
> > >
> > > You are downloading an image from an arbitrary server on the internet
> > > unrelated to the problem we're trying to debug.
> > >
> > > Can you please attempt these several curl downloads while using the
> > > exact same URL that failed for snapd?
> >
> > As a developer asking for more debug information, can you please paste
> > the exact instructions on how to get those?
> >
> > I'm trying the https://public.apps.ubuntu.com/anon/download-snap/ based
> > url with the .snap showing up in the logs to get the exact same assets I
> > pasted snapd information on. However, curl -v returns (output stripped
> out):
> > *   Trying 162.213.33.92...
> > * Connected to public.apps.ubuntu.com (162.213.33.92) port 443 (#0)
> > * found 173 certificates in /etc/ssl/certs/ca-certificates.crt
> > * found 692 certificates in /etc/ssl/certs
> > * ALPN, offering http/1.1
> > * SSL connection using TLS1.2 / ECDHE_RSA_AES_128_GCM_SHA256
> > *  server certificate verification OK
> > *  server certificate status verification SKIPPED
> > *  common name: public.apps.ubuntu.com (matched)
> > *  server certificate expiration date OK
> > *  server certificate activation date OK
> > *  certificate public key: RSA
> > *  certificate version: #3
> > *  subject: C=GB,L=London,O=Canonical Group
> > Ltd,CN=public.apps.ubuntu.com
> > *  start date: Mon, 30 May 2016 00:00:00 GMT
> > *  expire date: Wed, 21 Jun 2017 12:00:00 GMT
> > *  issuer: C=US,O=DigiCert Inc,CN=DigiCert SHA2 Secure Server CA
> > *  compression: NULL
> > * ALPN, server did not agree to a protocol
> > > GET /anon/download-snap/YZ7LshLxDQQIrhAL6DMLub2yTVUA2DIK_15.snap
> HTTP/1.1
> > > Host: public.apps.ubuntu.com
> > > User-Agent: curl/7.47.0
> > > Accept: */*
> > >
> > < HTTP/1.1 302 FOUND
> >
> > I guess that's due to the macaroon exchanged system and I'm not
> > authorized or something else?
> > Thanks for helping debugging.
> >
> > Cheers,
> > Didier
>
> snapd first hits the Ubuntu Store URL (as above) that checks ACLs and then
> redirects to the CDN url. I don't know why `curl -v` doesn't follow the
> redirect, but the URL in the resulting 'Location' header will be the CDN
> url. Note that the CDN url contains a time-limited token in the
> query-string, so after a period you will need to get a fresh one from the
> store download endpoint.
>
> -bret
>
> --
> Snapcraft mailing list
> Snapcraft@lists.snapcraft.io
> Modify settings or unsubscribe at: https://lists.ubuntu.com/
> mailman/listinfo/snapcraft
>



-- 

gustavo @ http://niemeyer.net
-- 
Snapcraft mailing list
Snapcraft@lists.snapcraft.io
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/snapcraft

Re: sha3-384 mismatch

[Gustavo Niemeyer]

Xavier posted the exact URL of the failing snap in this thread:

"[1] - https://public.apps.ubuntu.com/anon/download-snap/
rFpKbTdZ31LyAxWF6RpcerZov1TdtDly_24.snap (extracted from the log file)"

Bret also posted another one above (thanks!).


On Mon, Dec 5, 2016 at 12:52 PM, Didier Roche  wrote:

> Le 05/12/2016 à 15:38, Gustavo Niemeyer a écrit :
>
>
>
> On Mon, Dec 5, 2016 at 12:23 PM, Didier Roche  wrote:
>
>> I did though write on the bug: "contrary to curl or wget which both
>> supports large downloads."
>> The feedback thread mentioned as well "while same assets can be
>> successfully downloaded via curl or wget".
>>
> I thought that was really obvious that they worked consistently and that I
>> did rerun then multiple times or I wouldn't have opened the bug report +
>> write this feedback. Sorry if that wasn't clear enough, let's move on :)
>>
>
> If you file a bug and a developer asks for specific information that
> wasn't provided, it means the specific information is not obvious.
>
> Is it the case?  Did you ever get a failure with them?  Are they retrying
>> while they work? Do you have a verbose dumb of the process?
>>
>> Yes, as mentioned. I never got any failure with any of them and I did
>> retry multiple times in loop when I saw the snapd failures.
>> wget is in verbose mode by default and I never got any hint that it was
>> retrying (just getting the normal download output).
>>
>> I did just try a verbose download in curl (here, an ubuntu 300M image).
>> Here is the output: http://paste.ubuntu.com/23583568/. It seems that
>> curl doesn't complain of any reconnect.
>>
>
> You are downloading an image from an arbitrary server on the internet
> unrelated to the problem we're trying to debug.
>
> Can you please attempt these several curl downloads while using the exact
> same URL that failed for snapd?
>
>
> As a developer asking for more debug information, can you please paste the
> exact instructions on how to get those?
>
> I'm trying the https://public.apps.ubuntu.com/anon/download-snap/ based
> url with the .snap showing up in the logs to get the exact same assets I
> pasted snapd information on. However, curl -v returns (output stripped out):
> *   Trying 162.213.33.92...
> * Connected to public.apps.ubuntu.com (162.213.33.92) port 443 (#0)
> * found 173 certificates in /etc/ssl/certs/ca-certificates.crt
> * found 692 certificates in /etc/ssl/certs
> * ALPN, offering http/1.1
> * SSL connection using TLS1.2 / ECDHE_RSA_AES_128_GCM_SHA256
> *  server certificate verification OK
> *  server certificate status verification SKIPPED
> *  common name: public.apps.ubuntu.com (matched)
> *  server certificate expiration date OK
> *  server certificate activation date OK
> *  certificate public key: RSA
> *  certificate version: #3
> *  subject: C=GB,L=London,O=Canonical Group Ltd,CN=
> public.apps.ubuntu.com
> *  start date: Mon, 30 May 2016 00:00:00 GMT
> *  expire date: Wed, 21 Jun 2017 12:00:00 GMT
> *  issuer: C=US,O=DigiCert Inc,CN=DigiCert SHA2 Secure Server CA
> *  compression: NULL
> * ALPN, server did not agree to a protocol
> > GET /anon/download-snap/YZ7LshLxDQQIrhAL6DMLub2yTVUA2DIK_15.snap
> HTTP/1.1
> > Host: public.apps.ubuntu.com
> > User-Agent: curl/7.47.0
> > Accept: */*
> >
> < HTTP/1.1 302 FOUND
>
> I guess that's due to the macaroon exchanged system and I'm not authorized
> or something else?
> Thanks for helping debugging.
>
> Cheers,
> Didier
>
> --
> Snapcraft mailing list
> Snapcraft@lists.snapcraft.io
> Modify settings or unsubscribe at: https://lists.ubuntu.com/
> mailman/listinfo/snapcraft
>
>


-- 

gustavo @ http://niemeyer.net
-- 
Snapcraft mailing list
Snapcraft@lists.snapcraft.io
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/snapcraft

Re: sha3-384 mismatch

[Didier Roche]

Le 05/12/2016 à 09:13, Madper Xie a écrit :
> I would say the cdn that we are using correctly is not friendly to me.
> (China telecom 200M fiber.)
> Downloading slowly(20KB/s) and unstable.
>
> Hi @Didier, I noticed that you mentioned
> https://bugs.launchpad.net/bugs/1643893
> <https://bugs.launchpad.net/bugs/1643893> twice in this thread.
> However I'm not able to access it... Is the project private?

It was public at the time I published it. However, then, I had to add
private information as part of debug output (macaroons, which is private
snapd credentials) in a launchpad comment. We thus had to turn it
privately as we can't remove them (in a comment), sorry for this.
Upstream has access to it.

Here is a copy and paste of title/description: "snapd doesn't handle
well flaky network"
"

Some network can be stalled or have few disconnects. snapd doesn't
support them and dropped the connexion on cryptic message contrary to
curl or wget which both supports large downloads.

$ sudo snap install face-detection-demo --beta --devmode
-> download starts with 0, 1, 2% and suddenly jump to 100% printing:
error: cannot perform the following tasks:
- Download snap "face-detection-demo" (49) from channel "beta" (sha3-384
mismatch downloading face-detection-demo: got
4c1ddd585f69dd6e1dc19c3eb6aa2f7ecbbd5671ce9f816cbdfd0647986a59fdd9313fde013d98a00d501f8ca4b0979e
but expected
40927e42f8339fcae0ed7edfd3215ffb0abc1dadbfedce446ad4b39c06cb9ad94f30bdd0c6514bdb6477e86727b75767)

wget works on the same machine.

Easy reproducer:
- install ubuntu core on a rpi2, plug the the eth45 cable to your laptop
- switch network-manager to network sharing mode
- try installing any snap (more than a meg) from the rpi2."


Didier
> BR,
> Madper
>
> On Mon, Dec 5, 2016 at 3:23 PM, Didier Roche <didro...@ubuntu.com
> <mailto:didro...@ubuntu.com>> wrote:
>
> Le 03/12/2016 à 16:01, Gustavo Niemeyer a écrit :
>> Hi Xavier,
>>
>> There's definitely a problem interrupting the connection with the
>> server. The fact it works sometimes means it's inconsistent.
>>
>> Can you please try to download such files several times out of
>> snap and snapd, to see what the error is?
>>
>> Per my note above, we had a bug in snapd which prevents the real
>> error from being shown. We're always showing the digest mismatch
>> instead, which will of course happen if the download is
>> interrupted before its end.
>>
>>
> As written on the bug report + some part feedback emails you
> received some weeks earlier, it seems you ignored one part of the
> issue. Yes, it seems the network is failing. However, snapd
> consistenly fails in my case for downloading, while other tools,
> like wget and curl can download bigger files, many times, without
> ever getting one failure. They are more robust in difficult
> network situations it seems.
>
> So either the store connection side is fragile or snapd isn't
> robust enough while other tools cope with those conditions just
> fine (without any warning/debug message telling that they are
> retrying).
> Cheers,
> Didier
>
>> On Sat, Dec 3, 2016 at 12:49 PM, Xavier Pegenaute
>> <xpegena...@gmail.com <mailto:xpegena...@gmail.com>> wrote:
>>
>> Hi all,
>>
>> I don't know exactly where to find all the data you are
>> asking for, but this is what I found (below). Related to the
>> network, as a test, I just downloaded the linux kernel
>> .tar.xz (98MB) using python requests and worked properly. I
>> am not using any proxy at all.
>>
>> The "snap install" worked with hello-world but failed with
>> webdm, docker and nmap. I didn't try anymore.
>>
>> Let me know if you want any further test.
>> Regards
>> Xavi
>>
>>
>> -
>> root@localhost:/var/lib/snapd/snaps# ls -al webdm_24.snap
>> -rw-r--r-- 1 root root 3280896 Dec  3 14:12 webdm_24.snap
>> -
>>
>> -
>> root@localhost:/var/lib/snapd/snaps# sha384sum webdm_24.snap
>> 
>> 4ee3329efb0fc71a3217ec0fc7d92ba8f1d71db22036ddef8c4abefae6ca0dbe3e1847fde14bd420fc061c8d9b0795e8
>>  
>> webdm_24.snap
>> -
>>
>> Part of /var/lib/snapd/state.json:
>> 

Re: sha3-384 mismatch

[Madper Xie]

I would say the cdn that we are using correctly is not friendly to me.
(China telecom 200M fiber.)
Downloading slowly(20KB/s) and unstable.

Hi @Didier, I noticed that you mentioned https://bugs.launchpad.net/
bugs/1643893 twice in this thread. However I'm not able to access it... Is
the project private?

BR,
Madper

On Mon, Dec 5, 2016 at 3:23 PM, Didier Roche <didro...@ubuntu.com> wrote:

> Le 03/12/2016 à 16:01, Gustavo Niemeyer a écrit :
>
> Hi Xavier,
>
> There's definitely a problem interrupting the connection with the server.
> The fact it works sometimes means it's inconsistent.
>
> Can you please try to download such files several times out of snap and
> snapd, to see what the error is?
>
> Per my note above, we had a bug in snapd which prevents the real error
> from being shown. We're always showing the digest mismatch instead, which
> will of course happen if the download is interrupted before its end.
>
>
> As written on the bug report + some part feedback emails you received some
> weeks earlier, it seems you ignored one part of the issue. Yes, it seems
> the network is failing. However, snapd consistenly fails in my case for
> downloading, while other tools, like wget and curl can download bigger
> files, many times, without ever getting one failure. They are more robust
> in difficult network situations it seems.
>
> So either the store connection side is fragile or snapd isn't robust
> enough while other tools cope with those conditions just fine (without any
> warning/debug message telling that they are retrying).
> Cheers,
> Didier
>
> On Sat, Dec 3, 2016 at 12:49 PM, Xavier Pegenaute <xpegena...@gmail.com>
> wrote:
>
>> Hi all,
>>
>> I don't know exactly where to find all the data you are asking for, but
>> this is what I found (below). Related to the network, as a test, I just
>> downloaded the linux kernel .tar.xz (98MB) using python requests and worked
>> properly. I am not using any proxy at all.
>>
>> The "snap install" worked with hello-world but failed with webdm, docker
>> and nmap. I didn't try anymore.
>>
>> Let me know if you want any further test.
>> Regards
>> Xavi
>>
>>
>> -
>> root@localhost:/var/lib/snapd/snaps# ls -al webdm_24.snap
>> -rw-r--r-- 1 root root 3280896 Dec  3 14:12 webdm_24.snap
>> -
>>
>> -
>> root@localhost:/var/lib/snapd/snaps# sha384sum webdm_24.snap
>> 4ee3329efb0fc71a3217ec0fc7d92ba8f1d71db22036ddef8c4abefae6ca
>> 0dbe3e1847fde14bd420fc061c8d9b0795e8  webdm_24.snap
>> -
>>
>> Part of /var/lib/snapd/state.json:
>> -
>>   "tasks": {
>> "113": {
>>   "id": "113",
>>   "kind": "download-snap",
>>   "summary": "Download snap \"webdm\" (24) from channel \"stable\"",
>>   "status": 9,
>>   "clean": true,
>>   "progress": {
>> "label": "webdm",
>> "done": 3280896,
>> "total": 3280896
>>   },
>>   "data": {
>> "snap-setup": {
>>   "channel": "stable",
>>   "download-info": {
>> "anon-download-url": "https://public.apps.ubuntu.co
>> m/anon/download-snap/rFpKbTdZ31LyAxWF6RpcerZov1TdtDly_24.snap",
>> "download-url": "https://public.apps.ubuntu.co
>> m/download-snap/rFpKbTdZ31LyAxWF6RpcerZov1TdtDly_24.snap",
>> "size": 3280896,
>> "sha3-384": "d03ac82e6b4b559cc918ef581eacb
>> 933e66ad36b18f6bcb1b0ac4d91260ee14a815526d2bcecc474b215639ce2a3fcef"
>>   },
>>   "side-info": {
>> "name": "webdm",
>> "snap-id": "rFpKbTdZ31LyAxWF6RpcerZov1TdtDly",
>> "revision": "24",
>> "channel": "stable",
>> "developer-id": "canonical",
>> "developer": "canonical",
>> "summary": "# Snappyd",
>> "description": "This service allows you to manage your core
>> device from a web interface."
>>   }
>>

Re: sha3-384 mismatch

[Didier Roche]

Le 03/12/2016 à 16:01, Gustavo Niemeyer a écrit :
> Hi Xavier,
>
> There's definitely a problem interrupting the connection with the
> server. The fact it works sometimes means it's inconsistent.
>
> Can you please try to download such files several times out of snap
> and snapd, to see what the error is?
>
> Per my note above, we had a bug in snapd which prevents the real error
> from being shown. We're always showing the digest mismatch instead,
> which will of course happen if the download is interrupted before its end.
>
>
As written on the bug report + some part feedback emails you received
some weeks earlier, it seems you ignored one part of the issue. Yes, it
seems the network is failing. However, snapd consistenly fails in my
case for downloading, while other tools, like wget and curl can download
bigger files, many times, without ever getting one failure. They are
more robust in difficult network situations it seems.

So either the store connection side is fragile or snapd isn't robust
enough while other tools cope with those conditions just fine (without
any warning/debug message telling that they are retrying).
Cheers,
Didier

> On Sat, Dec 3, 2016 at 12:49 PM, Xavier Pegenaute
> <xpegena...@gmail.com <mailto:xpegena...@gmail.com>> wrote:
>
> Hi all,
>
> I don't know exactly where to find all the data you are asking
> for, but this is what I found (below). Related to the network, as
> a test, I just downloaded the linux kernel .tar.xz (98MB) using
> python requests and worked properly. I am not using any proxy at all.
>
> The "snap install" worked with hello-world but failed with webdm,
> docker and nmap. I didn't try anymore.
>
> Let me know if you want any further test.
> Regards
> Xavi
>
>
> -
> root@localhost:/var/lib/snapd/snaps# ls -al webdm_24.snap
> -rw-r--r-- 1 root root 3280896 Dec  3 14:12 webdm_24.snap
> -
>
> -
> root@localhost:/var/lib/snapd/snaps# sha384sum webdm_24.snap
> 
> 4ee3329efb0fc71a3217ec0fc7d92ba8f1d71db22036ddef8c4abefae6ca0dbe3e1847fde14bd420fc061c8d9b0795e8
>  
> webdm_24.snap
> -
>
> Part of /var/lib/snapd/state.json:
> -
>   "tasks": {
> "113": {
>   "id": "113",
>   "kind": "download-snap",
>   "summary": "Download snap \"webdm\" (24) from channel
> \"stable\"",
>   "status": 9,
>   "clean": true,
>   "progress": {
> "label": "webdm",
> "done": 3280896,
> "total": 3280896
>   },
>   "data": {
> "snap-setup": {
>   "channel": "stable",
>   "download-info": {
> "anon-download-url":
> 
> "https://public.apps.ubuntu.com/anon/download-snap/rFpKbTdZ31LyAxWF6RpcerZov1TdtDly_24.snap
> 
> <https://public.apps.ubuntu.com/anon/download-snap/rFpKbTdZ31LyAxWF6RpcerZov1TdtDly_24.snap>",
> "download-url":
> 
> "https://public.apps.ubuntu.com/download-snap/rFpKbTdZ31LyAxWF6RpcerZov1TdtDly_24.snap
> 
> <https://public.apps.ubuntu.com/download-snap/rFpKbTdZ31LyAxWF6RpcerZov1TdtDly_24.snap>",
> "size": 3280896,
> "sha3-384":
> 
> "d03ac82e6b4b559cc918ef581eacb933e66ad36b18f6bcb1b0ac4d91260ee14a815526d2bcecc474b215639ce2a3fcef"
>   },
>   "side-info": {
> "name": "webdm",
>     "snap-id": "rFpKbTdZ31LyAxWF6RpcerZov1TdtDly",
> "revision": "24",
> "channel": "stable",
> "developer-id": "canonical",
> "developer": "canonical",
> "summary": "# Snappyd",
> "description": "This service allows you to manage your
> core device from a web interface."
>   }
> }
>   },
>   "halt-tasks": [
> "114",
> "120"
>   ],
>   "log": [
> "

Re: sha3-384 mismatch

[Xavier Pegenaute]

Hi all,

I don't know exactly where to find all the data you are asking for, but
this is what I found (below). Related to the network, as a test, I just
downloaded the linux kernel .tar.xz (98MB) using python requests and worked
properly. I am not using any proxy at all.

The "snap install" worked with hello-world but failed with webdm, docker
and nmap. I didn't try anymore.

Let me know if you want any further test.
Regards
Xavi


-
root@localhost:/var/lib/snapd/snaps# ls -al webdm_24.snap
-rw-r--r-- 1 root root 3280896 Dec  3 14:12 webdm_24.snap
-

-
root@localhost:/var/lib/snapd/snaps# sha384sum webdm_24.snap
4ee3329efb0fc71a3217ec0fc7d92ba8f1d71db22036ddef8c4abefae6ca0dbe3e1847fde14bd420fc061c8d9b0795e8
webdm_24.snap
-

Part of /var/lib/snapd/state.json:
-
  "tasks": {
"113": {
  "id": "113",
  "kind": "download-snap",
  "summary": "Download snap \"webdm\" (24) from channel \"stable\"",
  "status": 9,
  "clean": true,
  "progress": {
"label": "webdm",
"done": 3280896,
"total": 3280896
  },
  "data": {
"snap-setup": {
  "channel": "stable",
  "download-info": {
"anon-download-url": "
https://public.apps.ubuntu.com/anon/download-snap/rFpKbTdZ31LyAxWF6RpcerZov1TdtDly_24.snap
",
"download-url": "
https://public.apps.ubuntu.com/download-snap/rFpKbTdZ31LyAxWF6RpcerZov1TdtDly_24.snap
",
"size": 3280896,
"sha3-384":
"d03ac82e6b4b559cc918ef581eacb933e66ad36b18f6bcb1b0ac4d91260ee14a815526d2bcecc474b215639ce2a3fcef"
  },
  "side-info": {
"name": "webdm",
"snap-id": "rFpKbTdZ31LyAxWF6RpcerZov1TdtDly",
"revision": "24",
"channel": "stable",
"developer-id": "canonical",
"developer": "canonical",
"summary": "# Snappyd",
"description": "This service allows you to manage your core
device from a web interface."
  }
}
  },
  "halt-tasks": [
"114",
"120"
  ],
  "log": [
"2016-12-03T14:08:45Z ERROR sha3-384 mismatch downloading webdm:
got
d03ac82e6b4b559cc918ef581eacb933e66ad36b18f6bcb1b0ac4d91260ee14a815526d2bcecc474b215639ce2a3fcef
but expected cea4eb570b28a3234
410cc1abaf19a8a3b0f9d80fe71f6c422334a44267ffbca454eb85684722d3a517fe7666f5b8a85"
  ],
  "change": "22",
  "spawn-time": "2016-12-03T14:08:41.26081942Z",
  "ready-time": "2016-12-03T14:08:45.426213255Z"
},
-



2016-12-02 22:24 GMT+01:00 Gustavo Niemeyer <gustavo.nieme...@canonical.com>
:

> The problem here is just that we're checking the digest mismatch before we
> check the error reading from the network. Obviously, if we fail to read
> from the network, the mismatch will always occur. If the mismatch doesn't
> occur, we might not even report the network error since we got all the data
> we needed anyway.
>
> Offending logic:
>
> https://github.com/snapcore/snapd/blob/master/store/store.go#L1388
>
>
>
>
> On Fri, Dec 2, 2016 at 4:15 PM, Gustavo Niemeyer <
> gustavo.nieme...@canonical.com> wrote:
>
>> The broken snap is a prefix of the actual snap:
>>
>> [niemeyer@nomade ~/test]% dd if=cr5pkasGhR7N3M8wKfP9DJqGxbBGeET2_25.snap
>> of=broken.snap bs=409018 count=1
>> 1+0 records in
>> 1+0 records out
>> 409018 bytes (409 kB, 399 KiB) copied, 0,0024764 s, 165 MB/s
>>
>> [niemeyer@nomade ~/test]% sha3384 broken.snap
>> broken.snap:d0e1cd6d578c8eaab13e10dac4acb
>> d7e8f6337da45b291fa7ae0358a29393059732b29bb61a65d18e693b8e6e8a53b62
>>
>>
>> Either the CDN is returning a successful code on interruption, or we're
>> mishandling the actual interruption code.
>>
>>
>>
>> On Fri, Dec 2, 2016 at 2:52 PM, Michael Vogt <michael.v...@canonical.com>
>> wrote:
>>
>>> On Fri, Dec 02, 2016 at 02:32:03PM +0100, David Barth wrote:
>>> > On Fri, Dec 2, 2016 at 1:04 PM, Pegenaute Bresme, Xavier <
>>> > xpegenaut...@iam.cat> wrote:

Re: sha3-384 mismatch

[Bret A. Barker]

Assuming you were hitting this error with a snapd prior to 2.18, then I think 
Madper is correct. Prior to 2.18 the order of the parameters for "got %s but 
expected %" was backward, see:
  https://github.com/snapcore/snapd/blob/release/2.17/store/store.go#L1155

And fixed in:
  https://github.com/snapcore/snapd/blob/release/2.18/store/store.go#L1378

-bret

On Fri, Dec 02, 2016 at 11:34:49PM +0800, Madper Xie wrote:
> Hi Pegenaute,
> 
> In fact I'm meeting this issue every day before I setup a proxy...
> May I know if you can reproduce it? And if so, did you get different
> sha3-384 every time?
> 
> I think it's more like a network issue. The downloading interrupted somehow
> so you got an incomplete snap pkg file. Which lead to a wrong sha result.
> 
> BR,
> Madper
> 
> On Fri, Dec 2, 2016 at 10:42 PM, James Tait <james.t...@canonical.com>
> wrote:
> 
> > On 2 December 2016 at 13:32, David Barth <david.ba...@canonical.com>
> > wrote:
> >
> >
> >> On Fri, Dec 2, 2016 at 1:04 PM, Pegenaute Bresme, Xavier <
> >> xpegenaut...@iam.cat> wrote:
> >>
> >>> Hi all,
> >>>
> >>> I just installed the last version [1] and when I execute these lines the
> >>> output is as follows.
> >>>
> >>> I guess the package list is not properly updated/synchronized with the
> >>> package repository. Am I wrong?, how could I fix it ?
> >>>
> >>> ----------
> >>> root@localhost:~# snap refresh
> >>> All snaps up to date.
> >>> root@localhost:~# snap install snapweb
> >>> error: cannot perform the following tasks:
> >>> - Download snap "snapweb" (25) from channel "stable" (sha3-384 mismatch
> >>> downloading snapweb: got
> >>> ​​
> >>> 8b83c8eb7f7aa306bc342fd1b424fa95ccf379c068c4735085bc81ee6b51
> >>> bb02a23b8c3c2a34f842619d7650b3152787 but expected
> >>> d0e1cd6d578c8eaab13e10dac4acbd7e8f6337da45b291fa7ae0358a2939
> >>> 3059732b29bb61a65d18e693b8e6e8a53b62)
> >>>
> >>
> >> What is surprising is the "expected" signature.
> >> Snap is downloading the correct armhf build for 0.21.2, ie #25, and the
> >> SHA3 it obtained corresponds to the published version.
> >>
> >> ​Moreover, that expected hash doesn’t seem to exist in the index​, at
> > least not for that package. Where has snapd got that value from?
> > ​
> > --
> > James Tait, BSc. | https://launchpad.net/~jamestait/
> > Software Engineer, Canonical Online Services
> > Ubuntu - Linux for human beings | www.ubuntu.com
> >
> > --
> > Snapcraft mailing list
> > Snapcraft@lists.snapcraft.io
> > Modify settings or unsubscribe at: https://lists.ubuntu.com/
> > mailman/listinfo/snapcraft
> >
> >

> -- 
> Snapcraft mailing list
> Snapcraft@lists.snapcraft.io
> Modify settings or unsubscribe at: 
> https://lists.ubuntu.com/mailman/listinfo/snapcraft


-- 
--* bret barker
--* irc: noise

-- 
Snapcraft mailing list
Snapcraft@lists.snapcraft.io
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/snapcraft

Re: sha3-384 mismatch

[James Tait]

On 2 December 2016 at 13:32, David Barth <david.ba...@canonical.com> wrote:


> On Fri, Dec 2, 2016 at 1:04 PM, Pegenaute Bresme, Xavier <
> xpegenaut...@iam.cat> wrote:
>
>> Hi all,
>>
>> I just installed the last version [1] and when I execute these lines the
>> output is as follows.
>>
>> I guess the package list is not properly updated/synchronized with the
>> package repository. Am I wrong?, how could I fix it ?
>>
>> --
>> root@localhost:~# snap refresh
>> All snaps up to date.
>> root@localhost:~# snap install snapweb
>> error: cannot perform the following tasks:
>> - Download snap "snapweb" (25) from channel "stable" (sha3-384 mismatch
>> downloading snapweb: got
>> ​​
>> 8b83c8eb7f7aa306bc342fd1b424fa95ccf379c068c4735085bc81ee6b51
>> bb02a23b8c3c2a34f842619d7650b3152787 but expected
>> d0e1cd6d578c8eaab13e10dac4acbd7e8f6337da45b291fa7ae0358a2939
>> 3059732b29bb61a65d18e693b8e6e8a53b62)
>>
>
> What is surprising is the "expected" signature.
> Snap is downloading the correct armhf build for 0.21.2, ie #25, and the
> SHA3 it obtained corresponds to the published version.
>
> ​Moreover, that expected hash doesn’t seem to exist in the index​, at
least not for that package. Where has snapd got that value from?
​
-- 
James Tait, BSc. | https://launchpad.net/~jamestait/
Software Engineer, Canonical Online Services
Ubuntu - Linux for human beings | www.ubuntu.com
-- 
Snapcraft mailing list
Snapcraft@lists.snapcraft.io
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/snapcraft