Re: [yocto] cannot re-use shared state cache between build hosts

[Andrea Galbusera]

On Mon, Sep 4, 2017 at 10:56 AM, Paul Eggleton <
paul.eggle...@linux.intel.com> wrote:

> On Monday, 4 September 2017 7:59:55 PM NZST Andrea Galbusera wrote:
> > On Mon, Sep 4, 2017 at 9:33 AM, Patrick Ohly 
> wrote:
> > > More recent bitbake should not fail like that anymore. It's still
> > > better to use an HTTP server that performs better, though.
> > >
> > > commit 6fa07752bbd3ac345cd8617da49a70e0b2dd565f
> > > Author: Patrick Ohly 
> > > Date:   Mon Jul 17 15:25:10 2017 +0200
> > >
> > > fetch2/wget.py: improve error handling during sstate check
> > >
> > > When the sstate is accessed via HTTP, the existence check can fail
> due
> > > to network issues, in which case bitbake silently continues without
> > > sstate.
> > >
> > > One such network issue is an HTTP server like Python's own
> SimpleHTTP
> > > which closes the TCP connection despite an explicit "Keep-Alive" in
> > > the HTTP request header. The server does that without a "close" in
> the
> > > HTTP response header, so the socket remains in the connection
> cache,
> > > leading to "urlopen failed:  > > descriptor>" (only visible in "bitbake -D -D" output) when trying
> to
> > > use the cached connection again.
> > >
> > > The connection might also get closed for other reasons (proxy,
> > > timeouts, etc.), so this is something that the client should be
> able
> > > to handle.
> > >
> > > This is achieved by checking for the error, removing the bad
> > > connection, and letting the check_status() method try again with a
> new
> > > connection. It is necessary to let the second attempt fail
> > > permanently, because bad proxy setups have been observed to also
> lead
> > > to such broken connections. In that case, we need to abort for real
> > > after trying twice, otherwise a build would just hang forever.
> > >
> > > [YOCTO #11782]
> > >
> >
> > Thank you Patrick for pointing that out. While debugging the issue on
> morty
> > I had some reminiscence of seeing your patch on the list, but I wasn't
> able
> > to find it back in morty's history hence inferring it landed later on.
> > Anyway doing this test on morty was a requirement... Thanks again for
> your
> > clarification. Would such a patch be suitable for eventually being
> > back-ported to morty?
>
> Doesn't look too invasive to me at least, so I'd support it.
>

I did some positive tests with Patrick's patch on top of morty and it
improves a lot the situation even with SimpleHTTPServer. Than I'd like to
propose it is backported to morty and, after further testing, reasonably
should land in pyro too. But I'm unsure how such a request should be
submitted. Since it affects bitbake, what's the proper way and which list
should I send such a request?
-- 
___
yocto mailing list
yocto@yoctoproject.org
https://lists.yoctoproject.org/listinfo/yocto

Re: [yocto] cannot re-use shared state cache between build hosts

[Paul Eggleton]

On Monday, 4 September 2017 7:59:55 PM NZST Andrea Galbusera wrote:
> On Mon, Sep 4, 2017 at 9:33 AM, Patrick Ohly  wrote:
> > More recent bitbake should not fail like that anymore. It's still
> > better to use an HTTP server that performs better, though.
> >
> > commit 6fa07752bbd3ac345cd8617da49a70e0b2dd565f
> > Author: Patrick Ohly 
> > Date:   Mon Jul 17 15:25:10 2017 +0200
> >
> > fetch2/wget.py: improve error handling during sstate check
> >
> > When the sstate is accessed via HTTP, the existence check can fail due
> > to network issues, in which case bitbake silently continues without
> > sstate.
> >
> > One such network issue is an HTTP server like Python's own SimpleHTTP
> > which closes the TCP connection despite an explicit "Keep-Alive" in
> > the HTTP request header. The server does that without a "close" in the
> > HTTP response header, so the socket remains in the connection cache,
> > leading to "urlopen failed:  > descriptor>" (only visible in "bitbake -D -D" output) when trying to
> > use the cached connection again.
> >
> > The connection might also get closed for other reasons (proxy,
> > timeouts, etc.), so this is something that the client should be able
> > to handle.
> >
> > This is achieved by checking for the error, removing the bad
> > connection, and letting the check_status() method try again with a new
> > connection. It is necessary to let the second attempt fail
> > permanently, because bad proxy setups have been observed to also lead
> > to such broken connections. In that case, we need to abort for real
> > after trying twice, otherwise a build would just hang forever.
> >
> > [YOCTO #11782]
> >
> 
> Thank you Patrick for pointing that out. While debugging the issue on morty
> I had some reminiscence of seeing your patch on the list, but I wasn't able
> to find it back in morty's history hence inferring it landed later on.
> Anyway doing this test on morty was a requirement... Thanks again for your
> clarification. Would such a patch be suitable for eventually being
> back-ported to morty?

Doesn't look too invasive to me at least, so I'd support it.

Cheers,
Paul

-- 

Paul Eggleton
Intel Open Source Technology Centre
-- 
___
yocto mailing list
yocto@yoctoproject.org
https://lists.yoctoproject.org/listinfo/yocto

Re: [yocto] cannot re-use shared state cache between build hosts

[Andrea Galbusera]

On Mon, Sep 4, 2017 at 9:33 AM, Patrick Ohly  wrote:

> On Fri, 2017-09-01 at 17:04 +0200, Andrea Galbusera wrote:
> > Hi Maciej,
> >
> > On Fri, Sep 1, 2017 at 4:08 PM, Maciej Borzęcki  > ty.com> wrote:
> > > On Fri, Sep 1, 2017 at 3:54 PM, Andrea Galbusera 
> > > wrote:
> > > > Hi!
> > > >
> > > > I was trying to share sstate between different hosts, but the
> > > consumer build
> > > > system seems to be unable to use re-use any sstate object. My
> > > scenario is
> > > > setup as follows:
> > > >
> > > > * The cache was populated by a pristine qemux86 core-image-
> > > minimal build of
> > > > morty. This was done in a crops/poky container (running in docker
> > > on Mac)
> > > > * The cache was then served via HTTP
> > >
> > > Make sure that you use a decent HTTP server. Simple `python3 -m
> > > http.server` will quickly choke when the mirror is being checked.
> > > Also
> > > running bitbake -DDD -v makes investigating this much easier.
> >
> > To be honest, the current server was indeed setup with python's
> > SimpleHTTPServer... As you suggest, I checked the verbose debug log
> > and noticed what's happening behind the apparently happy "Checking
> > sstate mirror object availability" step. After a first "SState:
> > Successful fetch test for" that I see correctly served with 200 on
> > the server side, tests for any other sstate object suddenly and
> > systematically fail with logs like this:
> ...
> > DEBUG: checkstatus() urlopen failed:  > file descriptor>
>
> More recent bitbake should not fail like that anymore. It's still
> better to use an HTTP server that performs better, though.
>
> commit 6fa07752bbd3ac345cd8617da49a70e0b2dd565f
> Author: Patrick Ohly 
> Date:   Mon Jul 17 15:25:10 2017 +0200
>
> fetch2/wget.py: improve error handling during sstate check
>
> When the sstate is accessed via HTTP, the existence check can fail due
> to network issues, in which case bitbake silently continues without
> sstate.
>
> One such network issue is an HTTP server like Python's own SimpleHTTP
> which closes the TCP connection despite an explicit "Keep-Alive" in
> the HTTP request header. The server does that without a "close" in the
> HTTP response header, so the socket remains in the connection cache,
> leading to "urlopen failed:  descriptor>" (only visible in "bitbake -D -D" output) when trying to
> use the cached connection again.
>
> The connection might also get closed for other reasons (proxy,
> timeouts, etc.), so this is something that the client should be able
> to handle.
>
> This is achieved by checking for the error, removing the bad
> connection, and letting the check_status() method try again with a new
> connection. It is necessary to let the second attempt fail
> permanently, because bad proxy setups have been observed to also lead
> to such broken connections. In that case, we need to abort for real
> after trying twice, otherwise a build would just hang forever.
>
> [YOCTO #11782]
>
> --
> Best Regards, Patrick Ohly
>
> The content of this message is my personal opinion only and although
> I am an employee of Intel, the statements I make here in no way
> represent Intel's position on the issue, nor am I authorized to speak
> on behalf of Intel on this matter.
>

Thank you Patrick for pointing that out. While debugging the issue on morty
I had some reminiscence of seeing your patch on the list, but I wasn't able
to find it back in morty's history hence inferring it landed later on.
Anyway doing this test on morty was a requirement... Thanks again for your
clarification. Would such a patch be suitable for eventually being
back-ported to morty?
-- 
___
yocto mailing list
yocto@yoctoproject.org
https://lists.yoctoproject.org/listinfo/yocto

Re: [yocto] cannot re-use shared state cache between build hosts

[Patrick Ohly]

On Fri, 2017-09-01 at 17:04 +0200, Andrea Galbusera wrote:
> Hi Maciej,
> 
> On Fri, Sep 1, 2017 at 4:08 PM, Maciej Borzęcki  ty.com> wrote:
> > On Fri, Sep 1, 2017 at 3:54 PM, Andrea Galbusera 
> > wrote:
> > > Hi!
> > >
> > > I was trying to share sstate between different hosts, but the
> > consumer build
> > > system seems to be unable to use re-use any sstate object. My
> > scenario is
> > > setup as follows:
> > >
> > > * The cache was populated by a pristine qemux86 core-image-
> > minimal build of
> > > morty. This was done in a crops/poky container (running in docker
> > on Mac)
> > > * The cache was then served via HTTP
> > 
> > Make sure that you use a decent HTTP server. Simple `python3 -m
> > http.server` will quickly choke when the mirror is being checked.
> > Also
> > running bitbake -DDD -v makes investigating this much easier.
> 
> To be honest, the current server was indeed setup with python's
> SimpleHTTPServer... As you suggest, I checked the verbose debug log
> and noticed what's happening behind the apparently happy "Checking
> sstate mirror object availability" step. After a first "SState:
> Successful fetch test for" that I see correctly served with 200 on
> the server side, tests for any other sstate object suddenly and
> systematically fail with logs like this:
...
> DEBUG: checkstatus() urlopen failed:  file descriptor>

More recent bitbake should not fail like that anymore. It's still
better to use an HTTP server that performs better, though.

commit 6fa07752bbd3ac345cd8617da49a70e0b2dd565f
Author: Patrick Ohly 
Date:   Mon Jul 17 15:25:10 2017 +0200

fetch2/wget.py: improve error handling during sstate check

When the sstate is accessed via HTTP, the existence check can fail due
to network issues, in which case bitbake silently continues without
sstate.

One such network issue is an HTTP server like Python's own SimpleHTTP
which closes the TCP connection despite an explicit "Keep-Alive" in
the HTTP request header. The server does that without a "close" in the
HTTP response header, so the socket remains in the connection cache,
leading to "urlopen failed: " (only visible in "bitbake -D -D" output) when trying to
use the cached connection again.

The connection might also get closed for other reasons (proxy,
timeouts, etc.), so this is something that the client should be able
to handle.

This is achieved by checking for the error, removing the bad
connection, and letting the check_status() method try again with a new
connection. It is necessary to let the second attempt fail
permanently, because bad proxy setups have been observed to also lead
to such broken connections. In that case, we need to abort for real
after trying twice, otherwise a build would just hang forever.

[YOCTO #11782]

-- 
Best Regards, Patrick Ohly

The content of this message is my personal opinion only and although
I am an employee of Intel, the statements I make here in no way
represent Intel's position on the issue, nor am I authorized to speak
on behalf of Intel on this matter.


-- 
___
yocto mailing list
yocto@yoctoproject.org
https://lists.yoctoproject.org/listinfo/yocto

Re: [yocto] cannot re-use shared state cache between build hosts

[Paul Eggleton]

Hi Andrea,

On Saturday, 2 September 2017 3:04:00 AM NZST Andrea Galbusera wrote:
> To be honest, the current server was indeed setup with python's
> SimpleHTTPServer... As you suggest, I checked the verbose debug log and
> noticed what's happening behind the apparently happy "Checking sstate
> mirror object availability" step. After a first "SState: Successful fetch
> test for" that I see correctly served with 200 on the server side, tests
> for any other sstate object suddenly and systematically fail with logs like
> this:
> 
> DEBUG: SState: Attempting to fetch file://7d/sstate:libxml2:i586-
> poky-linux:2.9.4:r0:i586:3:7da8fc3f7f5ed0102d23bdb86ac7ab32_package_qa.tgz
> DEBUG: Searching for 7d/sstate:libxml2:i586-poky-linux:2.9.4:r0:i586:3:
> 7da8fc3f7f5ed0102d23bdb86ac7ab32_package_qa.tgz in paths:
> /home/vagrant/koan/morty/build/sstate-cache
> DEBUG: Defaulting to /home/vagrant/koan/morty/build/sstate-cache/7d/sstate:
> libxml2:i586-poky-linux:
2.9.4:r0:i586:3:7da8fc3f7f5ed0102d23bdb86ac7ab32_package_qa.tgz
> for 7d/sstate:libxml2:i586-poky-linux:2.9.4:r0:i586:3:7da8fc3f7f5ed0102d23b
> db86ac7ab32_package_qa.tgz
> DEBUG: Testing URL file://7d/sstate:libxml2:i586-poky-linux:2.9.4:r0:i586:3:
> 7da8fc3f7f5ed0102d23bdb86ac7ab32_package_qa.tgz
> DEBUG: For url ['file', '', '7d/sstate:libxml2:i586-poky-
> linux:2.9.4:r0:i586:3:7da8fc3f7f5ed0102d23bdb86ac7ab32_package_qa.tgz', '',
> '', OrderedDict()] comparing ['file', '', '.*', '', '', OrderedDict()] to
> ['http', '192.168.33.1:8000', '
> /sstate-cache/PATH', '', '', OrderedDict([('downloadfilename', 'PATH')])]
> DEBUG: For url file://7d/sstate:libxml2:i586-poky-linux:2.9.4:r0:i586:3:
> 7da8fc3f7f5ed0102d23bdb86ac7ab32_package_qa.tgz returning
> http://192.168.33.1:8000/sstate-cache/7d/sstate%3Alibxml2%3Ai586-poky-linux%
> 3A2.9.4%3Ar0%3Ai586%3A3%3A7da8fc
> 3f7f5ed0102d23bdb86ac7ab32_package_qa.tgz;downloadfilename=7d/sstate:
> libxml2:i586-poky-linux:2.9.4:r0:i586:3:7da8fc3f7f5ed0102d23bdb86ac7ab
> 32_package_qa.tgz
> DEBUG: checkstatus: trying again
> DEBUG: checkstatus() urlopen failed:  descriptor>
> DEBUG: SState: Unsuccessful fetch test for file://7d/sstate:libxml2:i586-
> poky-linux:2.9.4:r0:i586:3:7da8fc3f7f5ed0102d23bdb86ac7ab32_package_qa.tgz
> 
> Nothing is reported server-side for any of these failures... As you
> recommend, I'll try to setup something more "decent" for the HTTP server
> and see if it helps.

Right, FYI we were using plain SimpleHTTPServer in oe-selftest, and we had to 
change it because it simply isn't reliable when you throw multiple concurrent 
requests at it as sstate does. There's a ThreadingMixIn you can inherit (see 
meta/lib/oeqa/utils/httpserver.py) but you might as well set up a proper 
server if you're doing more than just testing.

Cheers,
Paul


-- 

Paul Eggleton
Intel Open Source Technology Centre
-- 
___
yocto mailing list
yocto@yoctoproject.org
https://lists.yoctoproject.org/listinfo/yocto

Re: [yocto] cannot re-use shared state cache between build hosts

[Maciej Borzęcki]

On Fri, Sep 1, 2017 at 5:04 PM, Andrea Galbusera  wrote:
> Hi Maciej,
>
> On Fri, Sep 1, 2017 at 4:08 PM, Maciej Borzęcki 
> wrote:
>>
>> On Fri, Sep 1, 2017 at 3:54 PM, Andrea Galbusera  wrote:
>> > Hi!
>> >
>> > I was trying to share sstate between different hosts, but the consumer
>> > build
>> > system seems to be unable to use re-use any sstate object. My scenario
>> > is
>> > setup as follows:
>> >
>> > * The cache was populated by a pristine qemux86 core-image-minimal build
>> > of
>> > morty. This was done in a crops/poky container (running in docker on
>> > Mac)
>> > * The cache was then served via HTTP
>>
>> Make sure that you use a decent HTTP server. Simple `python3 -m
>> http.server` will quickly choke when the mirror is being checked. Also
>> running bitbake -DDD -v makes investigating this much easier.
>
>
> To be honest, the current server was indeed setup with python's
> SimpleHTTPServer... As you suggest, I checked the verbose debug log and
> noticed what's happening behind the apparently happy "Checking sstate mirror
> object availability" step. After a first "SState: Successful fetch test for"
> that I see correctly served with 200 on the server side, tests for any other
> sstate object suddenly and systematically fail with logs like this:
>
> DEBUG: SState: Attempting to fetch
> file://7d/sstate:libxml2:i586-poky-linux:2.9.4:r0:i586:3:7da8fc3f7f5ed0102d23bdb86ac7ab32_package_qa.tgz
> DEBUG: Searching for
> 7d/sstate:libxml2:i586-poky-linux:2.9.4:r0:i586:3:7da8fc3f7f5ed0102d23bdb86ac7ab32_package_qa.tgz
> in paths:
> /home/vagrant/koan/morty/build/sstate-cache
> DEBUG: Defaulting to
> /home/vagrant/koan/morty/build/sstate-cache/7d/sstate:libxml2:i586-poky-linux:2.9.4:r0:i586:3:7da8fc3f7f5ed0102d23bdb86ac7ab32_package_qa.tgz
> for 7d/sstate:libxml2:i586-poky-linux:2.9.4:r0:i586:3:7da8fc3f7f5ed0102d23b
> db86ac7ab32_package_qa.tgz
> DEBUG: Testing URL
> file://7d/sstate:libxml2:i586-poky-linux:2.9.4:r0:i586:3:7da8fc3f7f5ed0102d23bdb86ac7ab32_package_qa.tgz
> DEBUG: For url ['file', '',
> '7d/sstate:libxml2:i586-poky-linux:2.9.4:r0:i586:3:7da8fc3f7f5ed0102d23bdb86ac7ab32_package_qa.tgz',
> '', '', OrderedDict()] comparing ['file', '', '.*', '', '', OrderedDict()]
> to ['http', '192.168.33.1:8000', '
> /sstate-cache/PATH', '', '', OrderedDict([('downloadfilename', 'PATH')])]
> DEBUG: For url
> file://7d/sstate:libxml2:i586-poky-linux:2.9.4:r0:i586:3:7da8fc3f7f5ed0102d23bdb86ac7ab32_package_qa.tgz
> returning
> http://192.168.33.1:8000/sstate-cache/7d/sstate%3Alibxml2%3Ai586-poky-linux%3A2.9.4%3Ar0%3Ai586%3A3%3A7da8fc
> 3f7f5ed0102d23bdb86ac7ab32_package_qa.tgz;downloadfilename=7d/sstate:libxml2:i586-poky-linux:2.9.4:r0:i586:3:7da8fc3f7f5ed0102d23bdb86ac7ab32_package_qa.tgz
> DEBUG: checkstatus: trying again
> DEBUG: checkstatus() urlopen failed:  descriptor>
> DEBUG: SState: Unsuccessful fetch test for
> file://7d/sstate:libxml2:i586-poky-linux:2.9.4:r0:i586:3:7da8fc3f7f5ed0102d23bdb86ac7ab32_package_qa.tgz
>
> Nothing is reported server-side for any of these failures... As you
> recommend, I'll try to setup something more "decent" for the HTTP server and
> see if it helps.

Yeah, I think this has to do with HTTP keepalive. Anyways, caddy
worked just fine (https://caddyserver.com).

>
>
>>
>> > * The second host is a VM running Ubuntu 16.04 where I set
>> > SSTATE_MIRRORS to
>> > point to the hosted sstate cache like this:
>> >
>> > SSTATE_MIRRORS ?= "\
>> > file://.*
>> > http://192.168.33.1:8000/sstate-cache/PATH;downloadfilename=PATH";
>> >
>> > * I checked with curl that the VM can successfully get sstate objects
>> > from
>> > the server.
>> > * Then I start a new build (same metadata revisions, default
>> > configuration
>> > for core-image-minimal) and each and every task run from scratch with no
>> > sstate cache re-use.
>> >
>> > Here are the two configurations from bitbake and /etc/lsb-release files:
>> >
>> > On the container used to seed sstate cache:
>> >
>> > Build Configuration:
>> > BB_VERSION= "1.32.0"
>> > BUILD_SYS = "x86_64-linux"
>> > NATIVELSBSTRING   = "universal"
>> > TARGET_SYS= "i586-poky-linux"
>> > MACHINE   = "qemux86"
>> > DISTRO= "poky"
>> > DISTRO_VERSION= "2.2.2"
>> > TUNE_FEATURES = "m32 i586"
>> > TARGET_FPU= ""
>> > meta
>> > meta-poky
>> > meta-yocto-bsp= "morty:2a70e84643381eca0e7bf7928d4a3d56f9651128"
>> >
>> > $ cat /etc/lsb-release
>> > DISTRIB_ID=Ubuntu
>> > DISTRIB_RELEASE=16.04
>> > DISTRIB_CODENAME=xenial
>> > DISTRIB_DESCRIPTION="Ubuntu 16.04.2 LTS"
>> >
>> > On the VM that should consume the cache:
>> >
>> > Build Configuration:
>> > BB_VERSION= "1.32.0"
>> > BUILD_SYS = "x86_64-linux"
>> > NATIVELSBSTRING   = "Ubuntu-16.04"
>> > TARGET_SYS= "i586-poky-linux"
>> > MACHINE   = "qemux86"
>> > DISTRO= "poky"
>> > DISTRO_VERSION= "2.2.2"
>> > TUNE_FEATURES = "m32 i586"
>> > TARGET_FPU= ""
>> > meta
>> > meta-poky
>> > meta

Re: [yocto] cannot re-use shared state cache between build hosts

[Andrea Galbusera]

Hi Maciej,

On Fri, Sep 1, 2017 at 4:08 PM, Maciej Borzęcki 
wrote:

> On Fri, Sep 1, 2017 at 3:54 PM, Andrea Galbusera  wrote:
> > Hi!
> >
> > I was trying to share sstate between different hosts, but the consumer
> build
> > system seems to be unable to use re-use any sstate object. My scenario is
> > setup as follows:
> >
> > * The cache was populated by a pristine qemux86 core-image-minimal build
> of
> > morty. This was done in a crops/poky container (running in docker on Mac)
> > * The cache was then served via HTTP
>
> Make sure that you use a decent HTTP server. Simple `python3 -m
> http.server` will quickly choke when the mirror is being checked. Also
> running bitbake -DDD -v makes investigating this much easier.
>

To be honest, the current server was indeed setup with python's
SimpleHTTPServer... As you suggest, I checked the verbose debug log and
noticed what's happening behind the apparently happy "Checking sstate
mirror object availability" step. After a first "SState: Successful fetch
test for" that I see correctly served with 200 on the server side, tests
for any other sstate object suddenly and systematically fail with logs like
this:

DEBUG: SState: Attempting to fetch file://7d/sstate:libxml2:i586-
poky-linux:2.9.4:r0:i586:3:7da8fc3f7f5ed0102d23bdb86ac7ab32_package_qa.tgz
DEBUG: Searching for 7d/sstate:libxml2:i586-poky-linux:2.9.4:r0:i586:3:
7da8fc3f7f5ed0102d23bdb86ac7ab32_package_qa.tgz in paths:
/home/vagrant/koan/morty/build/sstate-cache
DEBUG: Defaulting to /home/vagrant/koan/morty/build/sstate-cache/7d/sstate:
libxml2:i586-poky-linux:2.9.4:r0:i586:3:7da8fc3f7f5ed0102d23bdb86ac7ab32_package_qa.tgz
for 7d/sstate:libxml2:i586-poky-linux:2.9.4:r0:i586:3:7da8fc3f7f5ed0102d23b
db86ac7ab32_package_qa.tgz
DEBUG: Testing URL file://7d/sstate:libxml2:i586-poky-linux:2.9.4:r0:i586:3:
7da8fc3f7f5ed0102d23bdb86ac7ab32_package_qa.tgz
DEBUG: For url ['file', '', '7d/sstate:libxml2:i586-poky-
linux:2.9.4:r0:i586:3:7da8fc3f7f5ed0102d23bdb86ac7ab32_package_qa.tgz', '',
'', OrderedDict()] comparing ['file', '', '.*', '', '', OrderedDict()] to
['http', '192.168.33.1:8000', '
/sstate-cache/PATH', '', '', OrderedDict([('downloadfilename', 'PATH')])]
DEBUG: For url file://7d/sstate:libxml2:i586-poky-linux:2.9.4:r0:i586:3:
7da8fc3f7f5ed0102d23bdb86ac7ab32_package_qa.tgz returning
http://192.168.33.1:8000/sstate-cache/7d/sstate%3Alibxml2%3Ai586-poky-linux%
3A2.9.4%3Ar0%3Ai586%3A3%3A7da8fc
3f7f5ed0102d23bdb86ac7ab32_package_qa.tgz;downloadfilename=7d/sstate:
libxml2:i586-poky-linux:2.9.4:r0:i586:3:7da8fc3f7f5ed0102d23bdb86ac7ab
32_package_qa.tgz
DEBUG: checkstatus: trying again
DEBUG: checkstatus() urlopen failed: 
DEBUG: SState: Unsuccessful fetch test for file://7d/sstate:libxml2:i586-
poky-linux:2.9.4:r0:i586:3:7da8fc3f7f5ed0102d23bdb86ac7ab32_package_qa.tgz

Nothing is reported server-side for any of these failures... As you
recommend, I'll try to setup something more "decent" for the HTTP server
and see if it helps.



> > * The second host is a VM running Ubuntu 16.04 where I set
> SSTATE_MIRRORS to
> > point to the hosted sstate cache like this:
> >
> > SSTATE_MIRRORS ?= "\
> > file://.* http://192.168.33.1:8000/sstate-cache/PATH;downloadfilename=
> PATH"
> >
> > * I checked with curl that the VM can successfully get sstate objects
> from
> > the server.
> > * Then I start a new build (same metadata revisions, default
> configuration
> > for core-image-minimal) and each and every task run from scratch with no
> > sstate cache re-use.
> >
> > Here are the two configurations from bitbake and /etc/lsb-release files:
> >
> > On the container used to seed sstate cache:
> >
> > Build Configuration:
> > BB_VERSION= "1.32.0"
> > BUILD_SYS = "x86_64-linux"
> > NATIVELSBSTRING   = "universal"
> > TARGET_SYS= "i586-poky-linux"
> > MACHINE   = "qemux86"
> > DISTRO= "poky"
> > DISTRO_VERSION= "2.2.2"
> > TUNE_FEATURES = "m32 i586"
> > TARGET_FPU= ""
> > meta
> > meta-poky
> > meta-yocto-bsp= "morty:2a70e84643381eca0e7bf7928d4a3d56f9651128"
> >
> > $ cat /etc/lsb-release
> > DISTRIB_ID=Ubuntu
> > DISTRIB_RELEASE=16.04
> > DISTRIB_CODENAME=xenial
> > DISTRIB_DESCRIPTION="Ubuntu 16.04.2 LTS"
> >
> > On the VM that should consume the cache:
> >
> > Build Configuration:
> > BB_VERSION= "1.32.0"
> > BUILD_SYS = "x86_64-linux"
> > NATIVELSBSTRING   = "Ubuntu-16.04"
> > TARGET_SYS= "i586-poky-linux"
> > MACHINE   = "qemux86"
> > DISTRO= "poky"
> > DISTRO_VERSION= "2.2.2"
> > TUNE_FEATURES = "m32 i586"
> > TARGET_FPU= ""
> > meta
> > meta-poky
> > meta-yocto-bsp= "morty:2a70e84643381eca0e7bf7928d4a3d56f9651128"
> >
> > $ cat /etc/lsb-release
> > DISTRIB_ID=Ubuntu
> > DISTRIB_RELEASE=16.04
> > DISTRIB_CODENAME=xenial
> > DISTRIB_DESCRIPTION="Ubuntu 16.04.3 LTS"
> >
> >
> > To me, the only differing bit that in my understanding can lead to sstate
> > cache objects invalidation is the value of

Re: [yocto] cannot re-use shared state cache between build hosts

[Maciej Borzęcki]

On Fri, Sep 1, 2017 at 3:54 PM, Andrea Galbusera  wrote:
> Hi!
>
> I was trying to share sstate between different hosts, but the consumer build
> system seems to be unable to use re-use any sstate object. My scenario is
> setup as follows:
>
> * The cache was populated by a pristine qemux86 core-image-minimal build of
> morty. This was done in a crops/poky container (running in docker on Mac)
> * The cache was then served via HTTP

Make sure that you use a decent HTTP server. Simple `python3 -m
http.server` will quickly choke when the mirror is being checked. Also
running bitbake -DDD -v makes investigating this much easier.

> * The second host is a VM running Ubuntu 16.04 where I set SSTATE_MIRRORS to
> point to the hosted sstate cache like this:
>
> SSTATE_MIRRORS ?= "\
> file://.* http://192.168.33.1:8000/sstate-cache/PATH;downloadfilename=PATH";
>
> * I checked with curl that the VM can successfully get sstate objects from
> the server.
> * Then I start a new build (same metadata revisions, default configuration
> for core-image-minimal) and each and every task run from scratch with no
> sstate cache re-use.
>
> Here are the two configurations from bitbake and /etc/lsb-release files:
>
> On the container used to seed sstate cache:
>
> Build Configuration:
> BB_VERSION= "1.32.0"
> BUILD_SYS = "x86_64-linux"
> NATIVELSBSTRING   = "universal"
> TARGET_SYS= "i586-poky-linux"
> MACHINE   = "qemux86"
> DISTRO= "poky"
> DISTRO_VERSION= "2.2.2"
> TUNE_FEATURES = "m32 i586"
> TARGET_FPU= ""
> meta
> meta-poky
> meta-yocto-bsp= "morty:2a70e84643381eca0e7bf7928d4a3d56f9651128"
>
> $ cat /etc/lsb-release
> DISTRIB_ID=Ubuntu
> DISTRIB_RELEASE=16.04
> DISTRIB_CODENAME=xenial
> DISTRIB_DESCRIPTION="Ubuntu 16.04.2 LTS"
>
> On the VM that should consume the cache:
>
> Build Configuration:
> BB_VERSION= "1.32.0"
> BUILD_SYS = "x86_64-linux"
> NATIVELSBSTRING   = "Ubuntu-16.04"
> TARGET_SYS= "i586-poky-linux"
> MACHINE   = "qemux86"
> DISTRO= "poky"
> DISTRO_VERSION= "2.2.2"
> TUNE_FEATURES = "m32 i586"
> TARGET_FPU= ""
> meta
> meta-poky
> meta-yocto-bsp= "morty:2a70e84643381eca0e7bf7928d4a3d56f9651128"
>
> $ cat /etc/lsb-release
> DISTRIB_ID=Ubuntu
> DISTRIB_RELEASE=16.04
> DISTRIB_CODENAME=xenial
> DISTRIB_DESCRIPTION="Ubuntu 16.04.3 LTS"
>
>
> To me, the only differing bit that in my understanding can lead to sstate
> cache objects invalidation is the value of NATIVELSBSTRING which is
> "universal" inside the container and "Ubuntu-16.04". This sounds strange to
> me, since both underlying systems are Ubuntu 16.04 (although not exactly the
> same dot release) as confirmed by /etc/lsb-release contents.
>
> Is the different NATIVELSBSTRING the root cause for everything being
> re-built? If so, what's causing them being different in the end and what
> does "universal" exactly mean (to me it looks like a more generic and
> incluse term than any distro label, so I'm confused...)?
>
>
> --
> ___
> yocto mailing list
> yocto@yoctoproject.org
> https://lists.yoctoproject.org/listinfo/yocto
>



-- 
Maciej Borzecki
RnDity
-- 
___
yocto mailing list
yocto@yoctoproject.org
https://lists.yoctoproject.org/listinfo/yocto

Re: [yocto] cannot re-use shared state cache between build hosts

[Andrea Galbusera]

On Fri, Sep 1, 2017 at 3:57 PM, Martin Jansa  wrote:

> Why do you use:
>
> ";downloadfilename=PATH
> "
>
> ?
>

Most likely because I copy/pasted from local.conf comments. I see the same
syntax on both "morty" [1] and "current" documentation. Shouldn't this be
part of the "keep the same two-character subdirectories layout of the
mirror" thing?

[1]
http://www.yoctoproject.org/docs/2.2/mega-manual/mega-manual.html#var-SSTATE_MIRRORS


> On Fri, Sep 1, 2017 at 3:54 PM, Andrea Galbusera  wrote:
>
>> Hi!
>>
>> I was trying to share sstate between different hosts, but the consumer
>> build system seems to be unable to use re-use any sstate object. My
>> scenario is setup as follows:
>>
>> * The cache was populated by a pristine qemux86 core-image-minimal build
>> of morty. This was done in a crops/poky container (running in docker on Mac)
>> * The cache was then served via HTTP
>> * The second host is a VM running Ubuntu 16.04 where I set SSTATE_MIRRORS
>> to point to the hosted sstate cache like this:
>>
>> SSTATE_MIRRORS ?= "\
>> file://.* http://192.168.33.1:8000/sstate-cache/PATH;downloadfilename=
>> PATH"
>>
>> * I checked with curl that the VM can successfully get sstate objects
>> from the server.
>> * Then I start a new build (same metadata revisions, default
>> configuration for core-image-minimal) and each and every task run from
>> scratch with no sstate cache re-use.
>>
>> Here are the two configurations from bitbake and /etc/lsb-release files:
>>
>> On the container used to seed sstate cache:
>>
>> Build Configuration:
>> BB_VERSION= "1.32.0"
>> BUILD_SYS = "x86_64-linux"
>> NATIVELSBSTRING   = "universal"
>> TARGET_SYS= "i586-poky-linux"
>> MACHINE   = "qemux86"
>> DISTRO= "poky"
>> DISTRO_VERSION= "2.2.2"
>> TUNE_FEATURES = "m32 i586"
>> TARGET_FPU= ""
>> meta
>> meta-poky
>> meta-yocto-bsp= "morty:2a70e84643381eca0e7bf7928d4a3d56f9651128"
>>
>> $ cat /etc/lsb-release
>> DISTRIB_ID=Ubuntu
>> DISTRIB_RELEASE=16.04
>> DISTRIB_CODENAME=xenial
>> DISTRIB_DESCRIPTION="Ubuntu 16.04.2 LTS"
>>
>> On the VM that should consume the cache:
>>
>> Build Configuration:
>> BB_VERSION= "1.32.0"
>> BUILD_SYS = "x86_64-linux"
>> NATIVELSBSTRING   = "Ubuntu-16.04"
>> TARGET_SYS= "i586-poky-linux"
>> MACHINE   = "qemux86"
>> DISTRO= "poky"
>> DISTRO_VERSION= "2.2.2"
>> TUNE_FEATURES = "m32 i586"
>> TARGET_FPU= ""
>> meta
>> meta-poky
>> meta-yocto-bsp= "morty:2a70e84643381eca0e7bf7928d4a3d56f9651128"
>>
>> $ cat /etc/lsb-release
>> DISTRIB_ID=Ubuntu
>> DISTRIB_RELEASE=16.04
>> DISTRIB_CODENAME=xenial
>> DISTRIB_DESCRIPTION="Ubuntu 16.04.3 LTS"
>>
>>
>> To me, the only differing bit that in my understanding can lead to sstate
>> cache objects invalidation is the value of NATIVELSBSTRING which is
>> "universal" inside the container and "Ubuntu-16.04". This sounds strange to
>> me, since both underlying systems are Ubuntu 16.04 (although not exactly
>> the same dot release) as confirmed by /etc/lsb-release contents.
>>
>> Is the different NATIVELSBSTRING the root cause for everything being
>> re-built? If so, what's causing them being different in the end and what
>> does "universal" exactly mean (to me it looks like a more generic and
>> incluse term than any distro label, so I'm confused...)?
>>
>>
>> --
>> ___
>> yocto mailing list
>> yocto@yoctoproject.org
>> https://lists.yoctoproject.org/listinfo/yocto
>>
>>
>
-- 
___
yocto mailing list
yocto@yoctoproject.org
https://lists.yoctoproject.org/listinfo/yocto

Re: [yocto] cannot re-use shared state cache between build hosts

[Martin Jansa]

Why do you use:

";downloadfilename=PATH
"

?

On Fri, Sep 1, 2017 at 3:54 PM, Andrea Galbusera  wrote:

> Hi!
>
> I was trying to share sstate between different hosts, but the consumer
> build system seems to be unable to use re-use any sstate object. My
> scenario is setup as follows:
>
> * The cache was populated by a pristine qemux86 core-image-minimal build
> of morty. This was done in a crops/poky container (running in docker on Mac)
> * The cache was then served via HTTP
> * The second host is a VM running Ubuntu 16.04 where I set SSTATE_MIRRORS
> to point to the hosted sstate cache like this:
>
> SSTATE_MIRRORS ?= "\
> file://.* http://192.168.33.1:8000/sstate-cache/PATH;downloadfilename=PATH
> "
>
> * I checked with curl that the VM can successfully get sstate objects from
> the server.
> * Then I start a new build (same metadata revisions, default configuration
> for core-image-minimal) and each and every task run from scratch with no
> sstate cache re-use.
>
> Here are the two configurations from bitbake and /etc/lsb-release files:
>
> On the container used to seed sstate cache:
>
> Build Configuration:
> BB_VERSION= "1.32.0"
> BUILD_SYS = "x86_64-linux"
> NATIVELSBSTRING   = "universal"
> TARGET_SYS= "i586-poky-linux"
> MACHINE   = "qemux86"
> DISTRO= "poky"
> DISTRO_VERSION= "2.2.2"
> TUNE_FEATURES = "m32 i586"
> TARGET_FPU= ""
> meta
> meta-poky
> meta-yocto-bsp= "morty:2a70e84643381eca0e7bf7928d4a3d56f9651128"
>
> $ cat /etc/lsb-release
> DISTRIB_ID=Ubuntu
> DISTRIB_RELEASE=16.04
> DISTRIB_CODENAME=xenial
> DISTRIB_DESCRIPTION="Ubuntu 16.04.2 LTS"
>
> On the VM that should consume the cache:
>
> Build Configuration:
> BB_VERSION= "1.32.0"
> BUILD_SYS = "x86_64-linux"
> NATIVELSBSTRING   = "Ubuntu-16.04"
> TARGET_SYS= "i586-poky-linux"
> MACHINE   = "qemux86"
> DISTRO= "poky"
> DISTRO_VERSION= "2.2.2"
> TUNE_FEATURES = "m32 i586"
> TARGET_FPU= ""
> meta
> meta-poky
> meta-yocto-bsp= "morty:2a70e84643381eca0e7bf7928d4a3d56f9651128"
>
> $ cat /etc/lsb-release
> DISTRIB_ID=Ubuntu
> DISTRIB_RELEASE=16.04
> DISTRIB_CODENAME=xenial
> DISTRIB_DESCRIPTION="Ubuntu 16.04.3 LTS"
>
>
> To me, the only differing bit that in my understanding can lead to sstate
> cache objects invalidation is the value of NATIVELSBSTRING which is
> "universal" inside the container and "Ubuntu-16.04". This sounds strange to
> me, since both underlying systems are Ubuntu 16.04 (although not exactly
> the same dot release) as confirmed by /etc/lsb-release contents.
>
> Is the different NATIVELSBSTRING the root cause for everything being
> re-built? If so, what's causing them being different in the end and what
> does "universal" exactly mean (to me it looks like a more generic and
> incluse term than any distro label, so I'm confused...)?
>
>
> --
> ___
> yocto mailing list
> yocto@yoctoproject.org
> https://lists.yoctoproject.org/listinfo/yocto
>
>
-- 
___
yocto mailing list
yocto@yoctoproject.org
https://lists.yoctoproject.org/listinfo/yocto

[yocto] cannot re-use shared state cache between build hosts

[Andrea Galbusera]

Hi!

I was trying to share sstate between different hosts, but the consumer
build system seems to be unable to use re-use any sstate object. My
scenario is setup as follows:

* The cache was populated by a pristine qemux86 core-image-minimal build of
morty. This was done in a crops/poky container (running in docker on Mac)
* The cache was then served via HTTP
* The second host is a VM running Ubuntu 16.04 where I set SSTATE_MIRRORS
to point to the hosted sstate cache like this:

SSTATE_MIRRORS ?= "\
file://.* http://192.168.33.1:8000/sstate-cache/PATH;downloadfilename=PATH";

* I checked with curl that the VM can successfully get sstate objects from
the server.
* Then I start a new build (same metadata revisions, default configuration
for core-image-minimal) and each and every task run from scratch with no
sstate cache re-use.

Here are the two configurations from bitbake and /etc/lsb-release files:

On the container used to seed sstate cache:

Build Configuration:
BB_VERSION= "1.32.0"
BUILD_SYS = "x86_64-linux"
NATIVELSBSTRING   = "universal"
TARGET_SYS= "i586-poky-linux"
MACHINE   = "qemux86"
DISTRO= "poky"
DISTRO_VERSION= "2.2.2"
TUNE_FEATURES = "m32 i586"
TARGET_FPU= ""
meta
meta-poky
meta-yocto-bsp= "morty:2a70e84643381eca0e7bf7928d4a3d56f9651128"

$ cat /etc/lsb-release
DISTRIB_ID=Ubuntu
DISTRIB_RELEASE=16.04
DISTRIB_CODENAME=xenial
DISTRIB_DESCRIPTION="Ubuntu 16.04.2 LTS"

On the VM that should consume the cache:

Build Configuration:
BB_VERSION= "1.32.0"
BUILD_SYS = "x86_64-linux"
NATIVELSBSTRING   = "Ubuntu-16.04"
TARGET_SYS= "i586-poky-linux"
MACHINE   = "qemux86"
DISTRO= "poky"
DISTRO_VERSION= "2.2.2"
TUNE_FEATURES = "m32 i586"
TARGET_FPU= ""
meta
meta-poky
meta-yocto-bsp= "morty:2a70e84643381eca0e7bf7928d4a3d56f9651128"

$ cat /etc/lsb-release
DISTRIB_ID=Ubuntu
DISTRIB_RELEASE=16.04
DISTRIB_CODENAME=xenial
DISTRIB_DESCRIPTION="Ubuntu 16.04.3 LTS"


To me, the only differing bit that in my understanding can lead to sstate
cache objects invalidation is the value of NATIVELSBSTRING which is
"universal" inside the container and "Ubuntu-16.04". This sounds strange to
me, since both underlying systems are Ubuntu 16.04 (although not exactly
the same dot release) as confirmed by /etc/lsb-release contents.

Is the different NATIVELSBSTRING the root cause for everything being
re-built? If so, what's causing them being different in the end and what
does "universal" exactly mean (to me it looks like a more generic and
incluse term than any distro label, so I'm confused...)?
-- 
___
yocto mailing list
yocto@yoctoproject.org
https://lists.yoctoproject.org/listinfo/yocto