To the first issue your are facing:
In BeamSQL, we tried to solve the similar requirement.
BeamSQL supports reading JSON format message from Pubsub, writing to
Bigquery and writing messages that fail to parse in another Pubsub topic.
BeamSQL uses the pre-processing transform to parse JSON
I'm trying to write a robust pipeline that takes input from PubSub and
writes to BigQuery. For every PubsubMessage that is not successfully
written to BigQuery, I'd like to get the original PubsubMessage back and be
able to write to an error output collection. I'm not sure this is quite
possible,
Hi Jozef,
Short answer is yes, Beam SQL provides a JDBC driver[1] that UI tools
should in theory be able to consume.
Longer answer:
Primary use case for the JDBC driver at the moment is to expose Beam SQL as
shell REPL environment, see [2]. Beam SQL shell wraps[3] the sqlline tool
[4] to
Hello,
does anyone use or is aware some kind of editor integration options for
BeamSQL? It would be to enable less technical people to execute SQL or do
data analysis queries conveniently. E.g. like HUE integration for SparkSQL
or similar
Thanks,
Jozef
