I have a python pipeline that uses the bigquery storage write method (cross
language with Java). I’m building launcher and worker docker images and
then launching the flex template. The launcher fails due to the following
runtime error in dataflow.
I’m using runner v2 and it is a streaming pipelin
You might need to pass the --save_main_session pipeline option. See:
https://beam.apache.org/documentation/sdks/python-pipeline-dependencies/#pickling-and-managing-the-main-session
Alternatively, structure your pipeline as a package:
https://beam.apache.org/documentation/sdks/python-pipeline-depe
I did not find BEAM-13846 but this suggests String is never supported:
https://github.com/apache/beam/blob/master/sdks/java/io/jdbc/src/test/java/org/apache/beam/sdk/io/jdbc/JdbcUtilTest.java#L59
However, you could use the code from the test to create yours.
On Thu, Feb 22, 2024 at 11:20 AM Vard
Does your code work without the launcher? Better check this step by step to
figure out which part causes this error.
On Sat, Feb 24, 2024 at 3:25 AM George Dekermenjian
wrote:
> I have a python pipeline that uses the bigquery storage write method
> (cross language with Java). I’m building launch
Thanks!!’
On Sat, 24 Feb 2024 at 12:23 PM Valentyn Tymofieiev via user <
user@beam.apache.org> wrote:
> You might need to pass the --save_main_session pipeline option. See:
> https://beam.apache.org/documentation/sdks/python-pipeline-dependencies/#pickling-and-managing-the-main-session
>
> Altern
Adding the following to both Dockerfile.launcher and Dockerfile.worker did
the trick for me.
COPY --from=apache/beam_java11_sdk:latest /opt/apache/beam/jars
/opt/apache/beam/jars COPY --from=apache/beam_java11_sdk:latest
/opt/java/openjdk /opt/java/openjdk ENV JAVA_HOME=/opt/java/openjdk ENV
PAT
Here is what I did:
CREATE TABLE IF NOT EXISTS test2 (id bigint DEFAULT
nextval('sap_tm_customer_id_seq'::regclass) NOT NULL, name VARCHAR(10),
load_date_time TIMESTAMP)
make sure id cannot be NULL (you might not need this).
I tried this for my data without using the id field:
class ExampleRow(
Great, you figured it out. Thanks for posting this back to the list.
On Sat, Feb 24, 2024 at 5:23 PM George Dekermenjian
wrote:
> Adding the following to both Dockerfile.launcher and Dockerfile.worker did
> the trick for me.
>
>
>
> COPY --from=apache/beam_java11_sdk:latest /opt/apache/beam/jars
Thanks!! It was useful!!!
On Sat, 24 Feb 2024 at 6:13 PM XQ Hu via user wrote:
> Here is what I did:
>
> CREATE TABLE IF NOT EXISTS test2 (id bigint DEFAULT
> nextval('sap_tm_customer_id_seq'::regclass) NOT NULL, name VARCHAR(10),
> load_date_time TIMESTAMP)
>
> make sure id cannot be NULL (you
