I have a scenario similar to yours, but we are using udf to do exactly
that. But you need to get the value of a broadcast variable from the udf.
But it's not clear how to achieve it, does anyone know?
Burak Yavuz 于2019年11月19日周二 下午12:23写道：
> If you store the data that you're going to broadcast
If you store the data that you're going to broadcast as a Delta table (see
delta.io) and perform a stream-batch (where your Delta table is the batch)
join, it will auto-update once the table receives any updates.
Best,
Burak
On Mon, Nov 18, 2019, 6:21 AM Bryan Jeffrey wrote:
> Hello.
>
> We're
Hello.
We're running applications using Spark Streaming. We're going to begin
work to move to using Structured Streaming. One of our key scenarios is to
lookup values from an external data source for each record in an incoming
stream. In Spark Streaming we currently read the external data,
