Hi there, I was actually going to comment on this. I had no trouble executing the query. I think James is correct in that you are getting blocked by the API. However, I may have a solution for you.
It looks like you are trying to pull back multiple pages from the same API. Drill can actually do this for you without you having to resort to complex queries. Take a look at the docs below. You should be able to configure this so that Drill will do the query generation for you. I thought I had implemented throttling or at least a configurable delay between queries but I can't find it if I did. Also, you shouldn't need to do all the casting. If you're using the latest version of Drill and the incoming data is in JSON format (and well formed) Drill should be able to figure all that out. Best, -- C https://github.com/apache/drill/blob/master/contrib/storage-http/Pagination.md drill/contrib/storage-http/Pagination.md at master · apache/drill github.com > On Jan 10, 2024, at 02:54, jnturton (via GitHub) <g...@apache.org> wrote: > > > jnturton commented on issue #2869: > URL: https://github.com/apache/drill/issues/2869#issuecomment-1884350260 > > I think this is likely to be due to rate limiting on the API you're > querying. Each `FROM http.feed.____` clause you've got is going to result in > a separate HTTP request to the API, and probably too many of those are being > sent in too short a space of time for the server's liking. I can't remember > if @cgivre added a throttling feature to the HTTP plugin but another > possibility might be to split the query into a few pieces using CTAS to keep > retrieved data around locally for later queries that rely on it. > > > -- > This is an automated message from the Apache Git Service. > To respond to the message, please log on to GitHub and use the > URL above to go to the specific comment. > > To unsubscribe, e-mail: dev-unsubscr...@drill.apache.org > > For queries about this service, please contact Infrastructure at: > us...@infra.apache.org >
