Hi Folks,
Great discussion! I will take into account rate-limiting and make it
configurable for the http request as well as all
I was wondering if there is anything I might have missed that would make it
technically impossible to do or at least difficult enough to not warrant
the effort
Is
Well, the difference is, a technical user writes the UDF and a
non-technical user may use this built-in thing (misconfigure it) and shoot
themselves in the foot.
On Wed, Jul 1, 2020, 6:40 PM Andrew Melo wrote:
> On Wed, Jul 1, 2020 at 8:13 PM Burak Yavuz wrote:
> >
> > I'm not sure having a
On Wed, Jul 1, 2020 at 8:13 PM Burak Yavuz wrote:
>
> I'm not sure having a built-in sink that allows you to DDOS servers is the
> best idea either. foreachWriter is typically used for such use cases, not
> foreachBatch. It's also pretty hard to guarantee exactly-once, rate limiting,
> etc.
On Wed, Jul 1, 2020 at 6:13 PM Burak Yavuz wrote:
> I'm not sure having a built-in sink that allows you to DDOS servers is the
> best idea either
>
Do you think it would be used accidentally? If so we could have it with
default per server rate limits that people would have to explicitly tune.
>
I'm not sure having a built-in sink that allows you to DDOS servers is the
best idea either. foreachWriter is typically used for such use cases, not
foreachBatch. It's also pretty hard to guarantee exactly-once, rate
limiting, etc.
Best,
Burak
On Wed, Jul 1, 2020 at 5:54 PM Holden Karau wrote:
I think adding something like this (if it doesn't already exist) could help
make structured streaming easier to use, foreachBatch is not the best API.
On Wed, Jul 1, 2020 at 2:21 PM Jungtaek Lim
wrote:
> I guess the method, query parameter, header, and the payload would be all
> different for
I guess the method, query parameter, header, and the payload would be all
different for almost every use case - that makes it hard to generalize and
requires implementation to be pretty much complicated to be flexible enough.
I'm not aware of any custom sink implementing REST so your best bet
Hi All,
We ingest alot of restful APIs into our lake and I'm wondering if it is at
all possible to created a rest sink in structured streaming?
For now I'm only focusing on restful services that have an incremental ID
so my sink can just poll for new data then ingest.
I can't seem to find a
