Hi Chris,
Is there a basis for calculating why there should only be 40 requests
for authorized participants at the end? Also, is the Query_String
limited to some maximum size?
When I benchmark it, even with the maximum utilization of 150 ASN
numbers in the query list for a large IX like DE-CIX, I see about ten
queries with ASN_LIST, including the IX and NetIX queries. With that, we
would have already exhausted 25% of the volume.
My general suggestion would be that we leave a bit more headroom for
requests in the same period without a self-throttling penalty. The
target value should conclude at 10% of the queries for the largest IX as
a variable; therefore, in 2022, at least 100 ~ 120 requests per minute
shall be allowed.
I wrote https://github.com/ipcjk/ixgen half a decade ago (god, how time
flies). I patched in the API keys yesterday; ASN_LIST will also be
included in the next release. However, there is another significant
advantage; the thing works with a local cache of the JSON files from
PeeringDB. It can be used as a simple API server directly as a binary
with compatible queries. So you can quickly get rid of 1000+ queries in
a few seconds without SQL, other dependencies, and bugging the original
peeringDB-source.
BR Jörg
On 31 May 2022, at 21:31, Chris Caputo wrote:
After the initial introduction of PeeringDB API throttling, some
software
both open source and private, has been identified and updated. (open
source details are below; please upgrade and encourage others to do
so)
This API throttling is being implemented to control costs by
encouraging
efficient software design while making sure the PeeringDB resource is
shared well. The use of API keys is being encouraged so that admins
can
reach out to users/orgs with runaway or inefficient software, and
because
it is more secure than user/pass. In addition, org API keys ease
employee
transitions.
Some tips for coders is below.
API throttling in place today:
- repeated anonymous identical requests with a response size above
100k
are being limited to 1/hour
- repeated anonymous identical requests of any size are being
limited to
2/minute
- anonymous queries are being limited to 400/minute per IP address
- authenticated queries are being limited to 500/minute per user/org
Here is the current schedule of throttling changes. The schedule may
adjust as needed as new packages that need update are discovered, so
as to
minimize disruption to the community...
On June 27th, adjust and watch for feedback from the community:
- anonymous queries limited to 300/minute per IP address
- authenticated queries limited to 400/minute per user/org
On July 11th, adjust and watch for feedback from the community:
- anonymous queries limited to 200/minute per IP address
- authenticated queries limited to 300/minute per user/org
On July 18th, adjust and watch for feedback from the community:
- anonymous queries limited to 100/minute per IP address
- authenticated queries limited to 200/minute per user/org
On July 25th, adjust and watch for feedback from the community:
- anonymous queries limited to 50/minute per IP address
- authenticated queries limited to 100/minute per user/org
On August 1st, adjust and watch for feedback from the community:
- anonymous queries limited to 30/minute per IP address
- authenticated queries limited to 80/minute per user/org
On August 8th, adjust and watch for feedback from the community:
- anonymous queries limited to 20/minute per IP address
- authenticated queries limited to 60/minute per user/org
On August 15th, adjust and watch for feedback from the community:
- anonymous queries limited to 10/minute per IP address
- authenticated queries limited to 40/minute per user/org
Feedback/questions/concerns welcome.
Thanks,
Chris
Software:
- arouteserver v1.16.0: has many updates including API key support
along
with more efficient querying.
- PeerFinder: API key & efficient querying patches at
https://github.com/rucarrol/PeerFinder/pull/17 will hopefully be
integrated.
Coding tips:
- Begin using a PeeringDB API key for all requests:
https://docs.peeringdb.com/howto/api_keys/
- Begin performing actual caching, such as by using peeringdb-py.
http://peeringdb.github.io/peeringdb-py/
- If unable to use a caching agent such as peeringdb-py:
- Use an API key.
- Set a User-Agent: header.
- Use bulk queries (asn__in=$list_of_ASN_separated_by_comma) by
querying 30 to 150 ASNs at a time (tune as appropriate).
- Add a delay in between queries that is randomly between 2 and 2.5
seconds, to reduce thundering herd.
_______________________________________________
Pdb-tech mailing list
[email protected]
https://lists.peeringdb.com/cgi-bin/mailman/listinfo/pdb-tech
_______________________________________________
Pdb-tech mailing list
[email protected]
https://lists.peeringdb.com/cgi-bin/mailman/listinfo/pdb-tech
