Re: More efficient build farm animal wakeup?

2022-11-23 Thread Andrew Dunstan
On 2022-11-23 We 16:59, Tom Lane wrote: > Thomas Munro writes: >> On Thu, Nov 24, 2022 at 10:00 AM Magnus Hagander wrote: >>> On Wed, Nov 23, 2022 at 9:15 AM Thomas Munro wrote: >>> Are you saying you still think it's worth pursuing longpoll or similar >>> methods for it, or that this is

Re: More efficient build farm animal wakeup?

2022-11-23 Thread Tom Lane
Thomas Munro writes: > On Thu, Nov 24, 2022 at 10:00 AM Magnus Hagander wrote: >> On Wed, Nov 23, 2022 at 9:15 AM Thomas Munro wrote: >> Are you saying you still think it's worth pursuing longpoll or similar >> methods for it, or that this is good enough? > I personally think it'd be pretty

Re: More efficient build farm animal wakeup?

2022-11-23 Thread Thomas Munro
On Thu, Nov 24, 2022 at 10:00 AM Magnus Hagander wrote: > On Wed, Nov 23, 2022 at 9:15 AM Thomas Munro wrote: > Are you saying you still think it's worth pursuing longpoll or similar > methods for it, or that this is good enough? I personally think it'd be pretty neat, to squeeze out that last

Re: More efficient build farm animal wakeup?

2022-11-23 Thread Magnus Hagander
On Wed, Nov 23, 2022 at 9:15 AM Thomas Munro wrote: > On Wed, Nov 23, 2022 at 2:09 PM Andres Freund wrote: > > It's a huge improvement here. > > Same here. eelpout + elver looking good, just a fraction of a second > hitting that web server each minute. Long polling will be better and > shave

Re: More efficient build farm animal wakeup?

2022-11-23 Thread Thomas Munro
On Wed, Nov 23, 2022 at 2:09 PM Andres Freund wrote: > It's a huge improvement here. Same here. eelpout + elver looking good, just a fraction of a second hitting that web server each minute. Long polling will be better and shave off 30 seconds (+/- 30) on start time, but this avoids a lot of

Re: More efficient build farm animal wakeup?

2022-11-22 Thread Andres Freund
Hi, On 2022-11-22 17:35:12 -0500, Andrew Dunstan wrote: > The server side appears to be working well. > > The new client side code is being tested on crake and working fine - the > all-up-to-date case takes just a second or two, almost all of which is > taken with getting the json file from the

Re: More efficient build farm animal wakeup?

2022-11-22 Thread Tom Lane
Andrew Dunstan writes: > The new client side code is being tested on crake and working fine - the > all-up-to-date case takes just a second or two, almost all of which is > taken with getting the json file from the server. No git calls at all > are done on the client in this case. Nice! I

Re: More efficient build farm animal wakeup?

2022-11-22 Thread Andrew Dunstan
On 2022-11-22 Tu 13:04, Magnus Hagander wrote: > > > On Tue, Nov 22, 2022 at 12:10 AM Magnus Hagander > wrote: > > > > On Mon, Nov 21, 2022 at 11:42 PM Andrew Dunstan > wrote: > > > On 2022-11-21 Mo 16:20, Magnus Hagander wrote: > > n Mon, Nov 21, 2022 at 9:58 PM Tom

Re: More efficient build farm animal wakeup?

2022-11-22 Thread Magnus Hagander
On Tue, Nov 22, 2022 at 12:10 AM Magnus Hagander wrote: > > > On Mon, Nov 21, 2022 at 11:42 PM Andrew Dunstan > wrote: > >> >> On 2022-11-21 Mo 16:20, Magnus Hagander wrote: >> > n Mon, Nov 21, 2022 at 9:58 PM Tom Lane wrote: >> > >> > Andrew Dunstan writes: >> b> > The buildfarm

Re: More efficient build farm animal wakeup?

2022-11-21 Thread Magnus Hagander
On Mon, Nov 21, 2022 at 11:42 PM Andrew Dunstan wrote: > > On 2022-11-21 Mo 16:20, Magnus Hagander wrote: > > n Mon, Nov 21, 2022 at 9:58 PM Tom Lane wrote: > > > > Andrew Dunstan writes: > > > The buildfarm server now creates a companion to > > branches_of_interest.txt > > >

Re: More efficient build farm animal wakeup?

2022-11-21 Thread Magnus Hagander
On Mon, Nov 21, 2022 at 11:41 PM Tom Lane wrote: > Andrew Dunstan writes: > > On 2022-11-21 Mo 15:58, Tom Lane wrote: > >> But if we're trying to improve matters in this area, this doesn't seem > >> like quite the way to go. > > > Well, 5 minutes was originally chosen because it was sufficient

Re: More efficient build farm animal wakeup?

2022-11-21 Thread Andrew Dunstan
On 2022-11-21 Mo 16:20, Magnus Hagander wrote: > n Mon, Nov 21, 2022 at 9:58 PM Tom Lane wrote: > > Andrew Dunstan writes: > > The buildfarm server now creates a companion to > branches_of_interest.txt > > called branches_of_interest.json which looks like this: > > ... okay

Re: More efficient build farm animal wakeup?

2022-11-21 Thread Tom Lane
Andrew Dunstan writes: > On 2022-11-21 Mo 15:58, Tom Lane wrote: >> But if we're trying to improve matters in this area, this doesn't seem >> like quite the way to go. > Well, 5 minutes was originally chosen because it was sufficient for the > purpose for which up to now the server used its

Re: More efficient build farm animal wakeup?

2022-11-21 Thread Andrew Dunstan
On 2022-11-21 Mo 15:58, Tom Lane wrote: > Andrew Dunstan writes: >> The buildfarm server now creates a companion to branches_of_interest.txt >> called branches_of_interest.json which looks like this: > ... okay ... > >> It updates this every time it does a git fetch, currently every 5 minutes.

Re: More efficient build farm animal wakeup?

2022-11-21 Thread Magnus Hagander
On Mon, Nov 21, 2022 at 11:27 PM Andrew Dunstan wrote: > > On 2022-11-21 Mo 16:26, Magnus Hagander wrote: > > > > > Is there a reason this file is a list of hashes each hash with a > > single value in it? Would it make more sense if it was: > > { > > "REL_11_STABLE": "140c803723", > >

Re: More efficient build farm animal wakeup?

2022-11-21 Thread Andrew Dunstan
On 2022-11-21 Mo 16:26, Magnus Hagander wrote: > > Is there a reason this file is a list of hashes each hash with a > single value in it? Would it make more sense if it was: > { >   "REL_11_STABLE": "140c803723", >   "REL_12_STABLE": "4cbcb7ed85", >   "REL_13_STABLE": "c13667b518", >  

Re: More efficient build farm animal wakeup?

2022-11-21 Thread Magnus Hagander
On Mon, Nov 21, 2022 at 9:51 PM Andrew Dunstan wrote: > > On 2022-11-20 Su 17:32, Thomas Munro wrote: > > On Sun, Nov 20, 2022 at 2:44 AM Andrew Dunstan > wrote: > >> It might not suit your use case, but one of the things I do to reduce > >> fetch load is to run a local mirror which runs > >> >

Re: More efficient build farm animal wakeup?

2022-11-21 Thread Magnus Hagander
n Mon, Nov 21, 2022 at 9:58 PM Tom Lane wrote: > Andrew Dunstan writes: > > The buildfarm server now creates a companion to branches_of_interest.txt > > called branches_of_interest.json which looks like this: > > ... okay ... > Yeah, it's not as efficient as something like long polling or web

Re: More efficient build farm animal wakeup?

2022-11-21 Thread Tom Lane
Andrew Dunstan writes: > The buildfarm server now creates a companion to branches_of_interest.txt > called branches_of_interest.json which looks like this: ... okay ... > It updates this every time it does a git fetch, currently every 5 minutes. That up-to-five-minute delay, on top of whatever

Re: More efficient build farm animal wakeup?

2022-11-21 Thread Andrew Dunstan
On 2022-11-20 Su 17:32, Thomas Munro wrote: > On Sun, Nov 20, 2022 at 2:44 AM Andrew Dunstan wrote: >> It might not suit your use case, but one of the things I do to reduce >> fetch load is to run a local mirror which runs >> >>git fetch -q --prune >> >> every 5 minutes. It also runs a git

Re: More efficient build farm animal wakeup?

2022-11-20 Thread Thomas Munro
On Sun, Nov 20, 2022 at 2:44 AM Andrew Dunstan wrote: > It might not suit your use case, but one of the things I do to reduce > fetch load is to run a local mirror which runs > >git fetch -q --prune > > every 5 minutes. It also runs a git daemon, and several of my animals > point at that.

Re: More efficient build farm animal wakeup?

2022-11-20 Thread Thomas Munro
On Mon, Nov 21, 2022 at 10:31 AM Magnus Hagander wrote: > Um, branches of interest will only pick up when it gets a new *branch*, not a > new *commit*, so I think that would be a very different problem to solve. And > I don't think we have new branche *that* often... Sure, could be done with

Re: More efficient build farm animal wakeup?

2022-11-20 Thread Magnus Hagander
On Sun, Nov 20, 2022 at 4:56 AM Thomas Munro wrote: > On Sun, Nov 20, 2022 at 1:35 AM Magnus Hagander > wrote: > > tl,tr; it's not there now, but yes if we can find a smart way for th ebf > clients to consume it, it is something we could build and deploy fairly > easily. > > Cool -- it sounds a

Re: More efficient build farm animal wakeup?

2022-11-19 Thread Andres Freund
Hi, On 2022-11-19 16:12:24 +1300, Thomas Munro wrote: > Is there a way to find out about new git commits that is more > efficient and timely than running N git fetches or whatever every > minute in a cron job? Maybe some kind of long polling where you send > an HTTP request that says "I think

Re: More efficient build farm animal wakeup?

2022-11-19 Thread Thomas Munro
On Sun, Nov 20, 2022 at 1:35 AM Magnus Hagander wrote: > tl,tr; it's not there now, but yes if we can find a smart way for th ebf > clients to consume it, it is something we could build and deploy fairly > easily. Cool -- it sounds a lot like you've thought about this already :-) About the

Re: More efficient build farm animal wakeup?

2022-11-19 Thread Andrew Dunstan
On 2022-11-18 Fr 22:12, Thomas Munro wrote: > Hi, > > Is there a way to find out about new git commits that is more > efficient and timely than running N git fetches or whatever every > minute in a cron job? Maybe some kind of long polling where you send > an HTTP request that says "I think the

Re: More efficient build farm animal wakeup?

2022-11-19 Thread Magnus Hagander
On Sat, Nov 19, 2022 at 4:13 AM Thomas Munro wrote: > Hi, > > Is there a way to find out about new git commits that is more > efficient and timely than running N git fetches or whatever every > minute in a cron job? Maybe some kind of long polling where you send > an HTTP request that says "I

More efficient build farm animal wakeup?

2022-11-18 Thread Thomas Munro
Hi, Is there a way to find out about new git commits that is more efficient and timely than running N git fetches or whatever every minute in a cron job? Maybe some kind of long polling where you send an HTTP request that says "I think the tips of branches x, y, z are at 111, 222, 333" and the