Hi All,
We ingest alot of restful APIs into our lake and I'm wondering if it is at all possible to created a rest sink in structured streaming? For now I'm only focusing on restful services that have an incremental ID so my sink can just poll for new data then ingest. I can't seem to find a connector that does this and my gut instinct tells me it's probably because it isn't possible due to something completely obvious that I am missing I know some RESTful API obfuscate the IDs to a hash of strings and that could be a problem but since I'm planning on focusing on just numerical IDs that just get incremented I think I won't be facing that issue Can anyone let me know if this sounds like a daft idea? Will I need something like Kafka or kinesis as a buffer and redundancy or am I overthinking this? I would love to bounce ideas with people who runs structured streaming jobs in production Kind regards San
