Hi Reza, Rui,
Can we use [slowly changing lookup cache] approach if the source is [HDFS
(or) HIVE] (data is changing), where the PCollection cannot fit into Memory
in BeamSQL?
This PCollection will be JOINED with Windowed PCollection Created from
Reading data in Kafka in BeamSQL.
Thanks and
+1
On Tue, 16 Jul 2019 at 20:36, Rui Wang wrote:
> Another approach is to let BeamSQL support it natively, as the title of
> this thread says: "as a Table in BeamSQL".
>
> We might be able to define a table with properties that says this table
> return a PCollectionView. By doing so we will
Another approach is to let BeamSQL support it natively, as the title of
this thread says: "as a Table in BeamSQL".
We might be able to define a table with properties that says this table
return a PCollectionView. By doing so we will have a trigger based
PCollectionView available in SQL rel nodes,
Hi Rahul,
FYI, that patterns is also available in the Beam docs ( with updated code
example )
https://beam.apache.org/documentation/patterns/side-input-patterns/.
Please note in the DoFn that feeds the View.asSingleton() you will need to
manually call BigQuery using the BigQuery client.
