Re: [aur-general] AUR package metadata dump

2018-11-16 Thread Eli Schwartz via aur-general
On 11/16/18 2:11 PM, Thore Bödecker via aur-general wrote: > Anywho, I just wanted to put this out there and gather some thoughts, feedback > and opinions on this. This discussion is at this point, no longer "hey, do you know what happened and why this doesn't work?" I think at this point (we're

Re: [aur-general] AUR package metadata dump

2018-11-16 Thread Thore Bödecker via aur-general
On 16.11.18 - 18:27, Florian Pritz via aur-general wrote: > My idea is to either generate the results on demand or cache them in the > code. If cached in the code, there would be no database load. It would > just pass through the code so we can perform rate limiting. Granted, if > we can implemente

Re: [aur-general] AUR package metadata dump

2018-11-16 Thread Florian Pritz via aur-general
On Fri, Nov 16, 2018 at 05:35:28PM +0300, Dmitry Marakasov wrote: > - Much less load on the server. > > I've looked through API code and it does an extra SQL query per a > package to get extended data such as dependencies and licenses, which > consists of multiple unions and joins involvin

Re: [aur-general] AUR package metadata dump

2018-11-16 Thread Dmitry Marakasov
* brent s. (b...@square-r00t.net) wrote: > (SNIP) > >> While fetching data from API, Repology does a 1 second pause between > >> requests to not create excess load on the server, but there are still > >> frequent 429 errors. I've tried 2 second delays, but the 429s are still > >> there, and fetch

Re: [aur-general] AUR package metadata dump

2018-11-16 Thread Dmitry Marakasov
* Florian Pritz via aur-general (aur-general@archlinux.org) wrote: > > The way Repology currently fetches AUR package data is as follows: > > - fetch https://aur.archlinux.org/packages.gz > > - split packages into 100 item packs > > - fetch JSON data for packages in each pack from > > https://aur

Re: [aur-general] AUR package metadata dump

2018-11-16 Thread Dmitry Marakasov
* Eli Schwartz via aur-general (aur-general@archlinux.org) wrote: > > I'm maintainer of Repology.org, a service which monitors, aggregates > > and compares package vesion accross 200+ package repositories with > > a purpose of simplifying package maintainers work by discovering > > new versions fa

Re: [aur-general] AUR package metadata dump

2018-11-15 Thread Jiachen YANG via aur-general
On 2018/11/16 7:31, Uwe Koloska wrote: > Hi Eli, > > Am 15.11.18 um 20:26 schrieb Eli Schwartz via aur-general: >> The source code running the website is here: >> https://git.archlinux.org/aurweb.git/about/ >> >> We currently provide the url, but not the sources for download, since >> the use case

Re: [aur-general] AUR package metadata dump

2018-11-15 Thread Uwe Koloska
Hi Eli, Am 15.11.18 um 20:26 schrieb Eli Schwartz via aur-general: > The source code running the website is here: > https://git.archlinux.org/aurweb.git/about/ > > We currently provide the url, but not the sources for download, since > the use case for our community has not (yet?) proposed that t

Re: [aur-general] AUR package metadata dump

2018-11-15 Thread brent s.
On 11/15/18 2:58 PM, Eli Schwartz via aur-general wrote: > > It's a pity that I forgot to reply with the exact same link and almost > the exact same caveat in the very next paragraph, isn't it? > > The paragraph which you quoted as "(SNIP)". > it most likely would have been more noticeable if y

Re: [aur-general] AUR package metadata dump

2018-11-15 Thread Eli Schwartz via aur-general
On 11/15/18 2:50 PM, brent s. wrote: > On 11/15/18 14:26, Eli Schwartz via aur-general wrote: >> On 11/15/18 1:25 PM, Dmitry Marakasov wrote: >>> Hi! >>> > (SNIP) >>> While fetching data from API, Repology does a 1 second pause between >>> requests to not create excess load on the server, but there

Re: [aur-general] AUR package metadata dump

2018-11-15 Thread brent s.
On 11/15/18 14:26, Eli Schwartz via aur-general wrote: > On 11/15/18 1:25 PM, Dmitry Marakasov wrote: >> Hi! >> (SNIP) >> While fetching data from API, Repology does a 1 second pause between >> requests to not create excess load on the server, but there are still >> frequent 429 errors. I've tried

Re: [aur-general] AUR package metadata dump

2018-11-15 Thread Eli Schwartz via aur-general
On 11/15/18 1:25 PM, Dmitry Marakasov wrote: > Hi! > > I'm maintainer of Repology.org, a service which monitors, aggregates > and compares package vesion accross 200+ package repositories with > a purpose of simplifying package maintainers work by discovering > new versions faster, improving colla

Re: [aur-general] AUR package metadata dump

2018-11-15 Thread Florian Pritz via aur-general
On Thu, Nov 15, 2018 at 09:25:02PM +0300, Dmitry Marakasov wrote: > The way Repology currently fetches AUR package data is as follows: > - fetch https://aur.archlinux.org/packages.gz > - split packages into 100 item packs > - fetch JSON data for packages in each pack from > https://aur.archlinux

[aur-general] AUR package metadata dump

2018-11-15 Thread Dmitry Marakasov
Hi! I'm maintainer of Repology.org, a service which monitors, aggregates and compares package vesion accross 200+ package repositories with a purpose of simplifying package maintainers work by discovering new versions faster, improving collaboration between maintainers and giving software authhors