Re: Pyspark UDF as a data source for streaming

Hi, Have you come back with some ideas for implementing this? Specifically integrating Spark Structured Streaming with REST API? FYI, I did some work on it as it can have potential wider use cases, i.e. the seamless integration of Spark Structured Streaming with Flask REST API for real-time data

Re: Pyspark UDF as a data source for streaming

c 28, 2023 at 4:53 PM Поротиков Станислав Вячеславович > wrote: > >> Yes, it's actual data. >> >> >> >> Best regards, >> >> Stanislav Porotikov >> >> >> >> *From:* Mich Talebzadeh >> *Sent:* Wednesday, December 27, 2023

Re: Pyspark UDF as a data source for streaming

; *From:* Mich Talebzadeh > *Sent:* Thursday, December 28, 2023 5:14 PM > *To:* Hyukjin Kwon > *Cc:* Поротиков Станислав Вячеславович ; > user@spark.apache.org > *Subject:* Re: Pyspark UDF as a data source for streaming > > > > You can work around this issue by trying to write

RE: Pyspark UDF as a data source for streaming

Ok. Thank you very much! Best regards, Stanislav Porotikov From: Mich Talebzadeh Sent: Thursday, December 28, 2023 5:14 PM To: Hyukjin Kwon Cc: Поротиков Станислав Вячеславович ; user@spark.apache.org Subject: Re: Pyspark UDF as a data source for streaming You can work around this issue

Re: Pyspark UDF as a data source for streaming

танислав Вячеславович > wrote: > >> Yes, it's actual data. >> >> >> >> Best regards, >> >> Stanislav Porotikov >> >> >> >> *From:* Mich Talebzadeh >> *Sent:* Wednesday, December 27, 2023 9:43 PM >> *Cc:* user@spark.a

Re: Pyspark UDF as a data source for streaming

rotikov > > > > *From:* Mich Talebzadeh > *Sent:* Wednesday, December 27, 2023 9:43 PM > *Cc:* user@spark.apache.org > *Subject:* Re: Pyspark UDF as a data source for streaming > > > > Is this generated data actual data or you are testing the application? > >

RE: Pyspark UDF as a data source for streaming

Yes, it's actual data. Best regards, Stanislav Porotikov From: Mich Talebzadeh Sent: Wednesday, December 27, 2023 9:43 PM Cc: user@spark.apache.org Subject: Re: Pyspark UDF as a data source for streaming Is this generated data actual data or you are testing the application? Sounds like a form

RE: Pyspark UDF as a data source for streaming

From: Mich Talebzadeh Sent: Wednesday, December 27, 2023 6:17 PM To: Поротиков Станислав Вячеславович Cc: user@spark.apache.org Subject: Re: Pyspark UDF as a data source for streaming Ok so you want to generate some random data and load it into Kafka on a regular interval and the rest

RE: Pyspark UDF as a data source for streaming

otikov From: Mich Talebzadeh Sent: Wednesday, December 27, 2023 6:17 PM To: Поротиков Станислав Вячеславович Cc: user@spark.apache.org Subject: Re: Pyspark UDF as a data source for streaming Ok so you want to generate some random data and load it into Kafka on a regular interval and the rest

Re: Pyspark UDF as a data source for streaming

Ok so you want to generate some random data and load it into Kafka on a regular interval and the rest? HTH Mich Talebzadeh, Dad | Technologist | Solutions Architect | Engineer London United Kingdom view my Linkedin profile

Pyspark UDF as a data source for streaming

Hello! Is it possible to write pyspark UDF, generated data to streaming dataframe? I want to get some data from REST API requests in real time and consider to save this data to dataframe. And then put it to Kafka. I can't realise how to create streaming dataframe from generated data. I am new in