> However, for a read-only API key, how does one know if it's working? > I set 'Authorization': 'Api-Key foo-bar-1234-4312' for a GET, and I > got results back vs a 4xx error code. So from an error handling > perspective it seems hard to gauge if I am using a valid api key > getting premium service vs an invalid api key quietly lumped into > the anonymous rate-limit bucket.
This is something we should fix. I've filed https://github.com/peeringdb/peeringdb/issues/1220 to get it addressed -Steve > On Aug 9, 2022, at 1:56 PM, Dale W. Carder <[email protected]> wrote: > > Thus spake Chris Caputo ([email protected]) on Mon, Aug 08, 2022 at 04:41:17PM > +0000: >> Per the below plan, this change was just implemented: >> >> --- >> On August 8th, adjust and watch for feedback from the community: >> >> - anonymous queries limited to 20/minute per IP address >> - authenticated queries limited to 60/minute per user/org >> --- >> >> Please advise if you run into any issues. > > This is about where I start to get concerned. First off, I'm not > sure how well communicated this was. I'd like to think that I'm > generally aware of what's happening in our ecosystem, but someone > (thankfully) had to point this out to me. > > So, our provisioning code is perhaps naive... jobs are dispatched > into a task queue where they are run to completion, one per ASN. > At present it would be non-trivial to implement a bulk query to > cache ahead of time (making peeringdb lookups asynchronous), but > that absolutely is on our longer-term roadmap. It's also not the > easiest to rate-limit the queue as only some of them actually need > a peeringdb lookup (a huge amount of our peers are private asn > and/or in a non-dfz l3vpn's), but we have limited the concurrency > and can count on the general case that our code is reassuringly > slow. > > Luckily, some of the other things suggested below are easy, and I > was testing it out today. We'll set a custom user-agent, limit > our query to only the fields we care about, and use an api key. > > However, for a read-only API key, how does one know if it's working? > I set 'Authorization': 'Api-Key foo-bar-1234-4312' for a GET, and I > got results back vs a 4xx error code. So from an error handling > perspective it seems hard to gauge if I am using a valid api key > getting premium service vs an invalid api key quietly lumped into > the anonymous rate-limit bucket. > > Dale > > >> On Tue, 31 May 2022, Chris Caputo wrote: >>> After the initial introduction of PeeringDB API throttling, some software >>> both open source and private, has been identified and updated. (open >>> source details are below; please upgrade and encourage others to do so) >>> >>> This API throttling is being implemented to control costs by encouraging >>> efficient software design while making sure the PeeringDB resource is >>> shared well. The use of API keys is being encouraged so that admins can >>> reach out to users/orgs with runaway or inefficient software, and because >>> it is more secure than user/pass. In addition, org API keys ease employee >>> transitions. >>> >>> Some tips for coders is below. >>> >>> API throttling in place today: >>> >>> - repeated anonymous identical requests with a response size above 100k >>> are being limited to 1/hour >>> - repeated anonymous identical requests of any size are being limited to >>> 2/minute >>> - anonymous queries are being limited to 400/minute per IP address >>> - authenticated queries are being limited to 500/minute per user/org >>> >>> Here is the current schedule of throttling changes. The schedule may >>> adjust as needed as new packages that need update are discovered, so as to >>> minimize disruption to the community... >>> >>> On June 27th, adjust and watch for feedback from the community: >>> >>> - anonymous queries limited to 300/minute per IP address >>> - authenticated queries limited to 400/minute per user/org >>> >>> On July 11th, adjust and watch for feedback from the community: >>> >>> - anonymous queries limited to 200/minute per IP address >>> - authenticated queries limited to 300/minute per user/org >>> >>> On July 18th, adjust and watch for feedback from the community: >>> >>> - anonymous queries limited to 100/minute per IP address >>> - authenticated queries limited to 200/minute per user/org >>> >>> On July 25th, adjust and watch for feedback from the community: >>> >>> - anonymous queries limited to 50/minute per IP address >>> - authenticated queries limited to 100/minute per user/org >>> >>> On August 1st, adjust and watch for feedback from the community: >>> >>> - anonymous queries limited to 30/minute per IP address >>> - authenticated queries limited to 80/minute per user/org >>> >>> On August 8th, adjust and watch for feedback from the community: >>> >>> - anonymous queries limited to 20/minute per IP address >>> - authenticated queries limited to 60/minute per user/org >>> >>> On August 15th, adjust and watch for feedback from the community: >>> >>> - anonymous queries limited to 10/minute per IP address >>> - authenticated queries limited to 40/minute per user/org >>> >>> Feedback/questions/concerns welcome. >>> >>> Thanks, >>> Chris >>> >>> Software: >>> >>> - arouteserver v1.16.0: has many updates including API key support along >>> with more efficient querying. >>> >>> - PeerFinder: API key & efficient querying patches at >>> https://github.com/rucarrol/PeerFinder/pull/17 will hopefully be >>> integrated. >>> >>> Coding tips: >>> >>> - Begin using a PeeringDB API key for all requests: >>> >>> https://docs.peeringdb.com/howto/api_keys/ >>> >>> - Begin performing actual caching, such as by using peeringdb-py. >>> >>> http://peeringdb.github.io/peeringdb-py/ >>> >>> - If unable to use a caching agent such as peeringdb-py: >>> >>> - Use an API key. >>> >>> - Set a User-Agent: header. >>> >>> - Use bulk queries (asn__in=$list_of_ASN_separated_by_comma) by >>> querying 30 to 150 ASNs at a time (tune as appropriate). >>> >>> - Add a delay in between queries that is randomly between 2 and 2.5 >>> seconds, to reduce thundering herd. >> _______________________________________________ >> Pdb-tech mailing list >> [email protected] >> https://lists.peeringdb.com/cgi-bin/mailman/listinfo/pdb-tech > _______________________________________________ > Pdb-tech mailing list > [email protected] > https://lists.peeringdb.com/cgi-bin/mailman/listinfo/pdb-tech _______________________________________________ Pdb-tech mailing list [email protected] https://lists.peeringdb.com/cgi-bin/mailman/listinfo/pdb-tech
