Hi,
I'm trying to write to Postgres database from Python using portable runner.
The API [1] has a "statement" parameter to let us specify a SQL statement:
driver_class_name – name of the jdbc driver class
jdbc_url – full jdbc url to the database.
username – database username
password – database
Hi,
I want to use Presto/Trino via Beam SQL. If so a Beam SQL statement can
support all Batch/Stream/OLAP/ad-hoc situations.
LuNing Wang
Evan Galpin 于2022年3月31日周四 04:30写道：
> Though this might not be what's being asked, might it be possible to
> integrate Presto using the JDBC driver[1] along w
Though this might not be what's being asked, might it be possible to
integrate Presto using the JDBC driver[1] along with Beam's JdbcIO[2] ?
[1] https://prestodb.io/docs/current/installation/jdbc.html
[2]
https://beam.apache.org/releases/javadoc/2.37.0/org/apache/beam/sdk/io/jdbc/JdbcIO.html
- Ev
> Are you referring to worker crashes? Or do you mean stopping
onepipeline and then starting a new one?
Both actually.
> Depends on the runner, but generally state is not all stored in memory.
I'm using Flink. At my knowledge they suggest a state less than 5Mb in size
(I have to double check)
Dear all,
I have a usecase where the main pipeline uses additional side input that is
changing from time to time. I get a notification of a change in a stream,
then I can emit the new state of the side input. Any idea on how to emit
the state (that I can read from an external source easily) when t
I figured out that the problem is with the local runner. Once I deploy the
pipeline to Dataflow everything works as expected. Does anyone have an idea
what might be the issue when running locally? It's tedious to debug a
complicated pipeline if I can't do it locally.
Thanks,
Matyas
On Tue, 29 Mar
Hi everyone,
About the Distinct [1] transformation. I couldn't find what precautions I
need to take when using it in terms of memory consumption and performance.
Furthermore, how does it behave if the pipeline crashes/restarted from
state, is its state restored on rerun (hence removes duplicates t
Hi,
Have any plans to support Trino/Pesto runner?
Best,
LuNing Wang
