evolving. So if anyone is interested,
> please support the project.
>
> --
> Lingzhe Sun
> Hirain Technologies
>
>
> *From:* Mich Talebzadeh
> *Date:* 2023-04-11 02:06
> *To:* Rajesh Katkar
> *CC:* user
> *Subject:* Re: spark streami
support
the project.
Lingzhe Sun
Hirain Technologies
From: Mich Talebzadeh
Date: 2023-04-11 02:06
To: Rajesh Katkar
CC: user
Subject: Re: spark streaming and kinesis integration
What I said was this"In so far as I know k8s does not support spark structured
streaming?"
So it is an ope
;>
>>
>> *From:* Mich Talebzadeh
>> *Date:* 2023-04-11 02:06
>> *To:* Rajesh Katkar
>> *CC:* user
>> *Subject:* Re: spark streaming and kinesis integration
>> What I said was this
>> "In so far as I know k8s does not support spark structured stre
s interested,
> please support the project.
>
> --
> Lingzhe Sun
> Hirain Technologies
>
>
> *From:* Mich Talebzadeh
> *Date:* 2023-04-11 02:06
> *To:* Rajesh Katkar
> *CC:* user
> *Subject:* Re: spark streaming and kinesis integration
for quite long time. Kind of worried that this project might finally
become outdated as k8s is evolving. So if anyone is interested, please support
the project.
Lingzhe Sun
Hirain Technologies
From: Mich Talebzadeh
Date: 2023-04-11 02:06
To: Rajesh Katkar
CC: user
Subject: Re: spark streaming
Just to clarify, a major benefit of k8s in this case is to host your Spark
applications in the form of containers in an automated fashion so that one
can easily deploy as many instances of the application as required
(autoscaling). From below:
What I said was this
"In so far as I know k8s does not support spark structured streaming?"
So it is an open question. I just recalled it. I have not tested myself. I
know structured streaming works on Google Dataproc cluster but I have not
seen any official link that says Spark Structured
Do you have any link or ticket which justifies that k8s does not support
spark streaming ?
On Thu, 6 Apr, 2023, 9:15 pm Mich Talebzadeh,
wrote:
> Do you have a high level diagram of the proposed solution?
>
> In so far as I know k8s does not support spark structured streaming?
>
> Mich
Use case is , we want to read/write to kinesis streams using k8s
Officially I could not find the connector or reader for kinesis from spark
like it has for kafka.
Checking here if anyone used kinesis and spark streaming combination ?
On Thu, 6 Apr, 2023, 7:23 pm Mich Talebzadeh,
wrote:
> Hi
kar
Cc: u...@spark.incubator.apache.org
Subject: Re: spark streaming and kinesis integration
⚠ [EXTERNAL EMAIL]: Use Caution
Do you have a high level diagram of the proposed solution?
In so far as I know k8s does not support spark structured streaming?
Mich Talebzadeh,
Lead Solutions
Do you have a high level diagram of the proposed solution?
In so far as I know k8s does not support spark structured streaming?
Mich Talebzadeh,
Lead Solutions Architect/Engineering Lead
Palantir Technologies
London
United Kingdom
view my Linkedin profile
Hi Rajesh,
What is the use case for Kinesis here? I have not used it personally, Which
use case it concerns
https://aws.amazon.com/kinesis/
Can you use something else instead?
HTH
Mich Talebzadeh,
Lead Solutions Architect/Engineering Lead
Palantir Technologies
London
United Kingdom
view
Hi Spark Team,
We need to read/write the kinesis streams using spark streaming.
We checked the official documentation -
https://spark.apache.org/docs/latest/streaming-kinesis-integration.html
It does not mention kinesis connector. Alternative is -
https://github.com/qubole/kinesis-sql which is
13 matches
Mail list logo
