Interactive runner that uses flink runner can read kafka messages?

Hello, I'm playing with the interactive runner on a notebook and the flink runner is used as the underlying runner. I wonder if it can read messages from Kafka. I checked the example notebook

Re: Fails to run two multi-language pipelines locally?

It may be a configuration issue. It works if I don't specify *flink_master*, which uses an embedded flink cluster. On Thu, 7 Mar 2024 at 12:47, Jaehyeon Kim wrote: > I works fine if I only use Kafka read/write as I only see a single > container - two transforms (read and write) but a single cont

Re: Fails to run two multi-language pipelines locally?

I works fine if I only use Kafka read/write as I only see a single container - two transforms (read and write) but a single container. If I add SqlTransform, however, another container is created and it begins to create an error. My speculation is the containers don't recognise each other and get

Re: Fails to run two multi-language pipelines locally?

Oh, sorry, I didn't see that. I would look earlier in the logs and see why it failed to bring up the containers (or, if they started, look in the container logs to see why they died). On Wed, Mar 6, 2024 at 5:28 PM Jaehyeon Kim wrote: > > I am not using the python local runner but the flink runn

Re: Fails to run two multi-language pipelines locally?

I am not using the python local runner but the flink runner. A flink cluster is started locally. On Thu, 7 Mar 2024 at 12:13, Robert Bradshaw via user wrote: > Streaming portable pipelines are not yet supported on the Python local > runner. > > On Wed, Mar 6, 2024 at 5:03 PM Jaehyeon Kim wrote:

Re: Fails to run two multi-language pipelines locally?

Streaming portable pipelines are not yet supported on the Python local runner. On Wed, Mar 6, 2024 at 5:03 PM Jaehyeon Kim wrote: > > Hello, > > I use the python SDK and my pipeline reads messages from Kafka and transforms > via SQL. I see two containers are created but it seems that they don't

Fails to run two multi-language pipelines locally?

Hello, I use the python SDK and my pipeline reads messages from Kafka and transforms via SQL. I see two containers are created but it seems that they don't communicate with each other so that the Flink task manager keeps killing the containers. The Flink cluster runs locally. Is there a way to run

Re: [Question] Does Apache SnowflakeIO support Apache Parquet?

Hello, The reason why is that Parquet can help user reduce the storage requirement, please refer to below and here is the link https://www.databricks.com/glossary/what-is-parquet. Any response would be appreciated. Thank you. [image: image.png] Regards, Xinmin On Tue, Mar 5, 2024 at 1:08 PM Wi

Updating streaming DataFlow pipeline

Hi I have a question. I do have a long running streaming dataflow jobs done with python sdk. I did notice that when I do update a job with exactly same code that created it, it may randomly fail (~50/50) with error like `The Coder or type for step event_stats/join/dewindow.None/FromValue has