Re: [OE-core] [PATCH 1/2] sstate: don't limit the thread pool size when checking mirrors

2021-12-03 Thread Jose Quaresma 
Richard Purdie  escreveu no dia quinta,
2/12/2021 à(s) 23:16:

> On Thu, 2021-12-02 at 22:05 +, Jose Quaresma wrote:
> > This improves the time needed to check the mirrors,
> > reducing it from 50s to 8s when building the core-image-minimal
> > using the yocto upstream sstate and hashequivlance servers
> > on a machine with 8 cpu cores.
> >
> > Tested with:
> >
> >  SSTATE_MIRRORS = "file://.*
> http://sstate.yoctoproject.org/dev/PATH;downloadfilename=PATH;
> >  BB_HASHSERVE_UPSTREAM = "typhoon.yocto.io:8687"
> >  bitbake core-image-minimal
> >
> > Signed-off-by: Jose Quaresma 
> > ---
> >  meta/classes/sstate.bbclass | 2 +-
> >  1 file changed, 1 insertion(+), 1 deletion(-)
> >
> > diff --git a/meta/classes/sstate.bbclass b/meta/classes/sstate.bbclass
> > index 0326d27c74..5e404d7cd8 100644
> > --- a/meta/classes/sstate.bbclass
> > +++ b/meta/classes/sstate.bbclass
> > @@ -1005,7 +1005,7 @@ def sstate_checkhashes(sq_data, d, siginfo=False,
> currentcount=0, summary=True,
> >  tasklist.append((tid, sstatefile))
> >
> >  if tasklist:
> > -nproc = min(int(d.getVar("BB_NUMBER_THREADS")),
> len(tasklist))
> > +nproc = len(tasklist)
> >
> >  progress = len(tasklist) >= 100
> >  if progress:
>
> No, we're not doing that. That opens a ton (thousands) of parallel
> connections
> to the servers and isn't fair on the systems hosting the data. In many
> cases
> that would look like a DoS attack.
>
> Cheers,
>
> Richard
>
>
>
>
Got it and I have to agree that it will launch a DoS to the server.
However there are some cases around that have the mirror hosted locally on
the file system.
I'll try to find a solution that doesn't affect the http mirror.

-- 
Best regards,

José Quaresma

-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#159111): 
https://lists.openembedded.org/g/openembedded-core/message/159111
Mute This Topic: https://lists.openembedded.org/mt/87464090/21656
Group Owner: openembedded-core+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub 
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-

Re: [OE-core] [PATCH 1/2] sstate: don't limit the thread pool size when checking mirrors

2021-12-02 Thread Richard Purdie 
On Thu, 2021-12-02 at 22:05 +, Jose Quaresma wrote:
> This improves the time needed to check the mirrors,
> reducing it from 50s to 8s when building the core-image-minimal
> using the yocto upstream sstate and hashequivlance servers
> on a machine with 8 cpu cores.
> 
> Tested with:
> 
>  SSTATE_MIRRORS = "file://.* 
> http://sstate.yoctoproject.org/dev/PATH;downloadfilename=PATH;
>  BB_HASHSERVE_UPSTREAM = "typhoon.yocto.io:8687"
>  bitbake core-image-minimal
> 
> Signed-off-by: Jose Quaresma 
> ---
>  meta/classes/sstate.bbclass | 2 +-
>  1 file changed, 1 insertion(+), 1 deletion(-)
> 
> diff --git a/meta/classes/sstate.bbclass b/meta/classes/sstate.bbclass
> index 0326d27c74..5e404d7cd8 100644
> --- a/meta/classes/sstate.bbclass
> +++ b/meta/classes/sstate.bbclass
> @@ -1005,7 +1005,7 @@ def sstate_checkhashes(sq_data, d, siginfo=False, 
> currentcount=0, summary=True,
>  tasklist.append((tid, sstatefile))
>  
>  if tasklist:
> -nproc = min(int(d.getVar("BB_NUMBER_THREADS")), len(tasklist))
> +nproc = len(tasklist)
>  
>  progress = len(tasklist) >= 100
>  if progress:

No, we're not doing that. That opens a ton (thousands) of parallel connections
to the servers and isn't fair on the systems hosting the data. In many cases
that would look like a DoS attack.

Cheers,

Richard




-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#159109): 
https://lists.openembedded.org/g/openembedded-core/message/159109
Mute This Topic: https://lists.openembedded.org/mt/87464090/21656
Group Owner: openembedded-core+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub 
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-

[OE-core] [PATCH 1/2] sstate: don't limit the thread pool size when checking mirrors

2021-12-02 Thread Jose Quaresma 
This improves the time needed to check the mirrors,
reducing it from 50s to 8s when building the core-image-minimal
using the yocto upstream sstate and hashequivlance servers
on a machine with 8 cpu cores.

Tested with:

 SSTATE_MIRRORS = "file://.* 
http://sstate.yoctoproject.org/dev/PATH;downloadfilename=PATH;
 BB_HASHSERVE_UPSTREAM = "typhoon.yocto.io:8687"
 bitbake core-image-minimal

Signed-off-by: Jose Quaresma 
---
 meta/classes/sstate.bbclass | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/meta/classes/sstate.bbclass b/meta/classes/sstate.bbclass
index 0326d27c74..5e404d7cd8 100644
--- a/meta/classes/sstate.bbclass
+++ b/meta/classes/sstate.bbclass
@@ -1005,7 +1005,7 @@ def sstate_checkhashes(sq_data, d, siginfo=False, 
currentcount=0, summary=True,
 tasklist.append((tid, sstatefile))
 
 if tasklist:
-nproc = min(int(d.getVar("BB_NUMBER_THREADS")), len(tasklist))
+nproc = len(tasklist)
 
 progress = len(tasklist) >= 100
 if progress:
-- 
2.34.1


-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#159106): 
https://lists.openembedded.org/g/openembedded-core/message/159106
Mute This Topic: https://lists.openembedded.org/mt/87464090/21656
Group Owner: openembedded-core+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub 
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-