Re: Re: spark streaming and kinesis integration

outdated as k8s is evolving. So if anyone is interested, > please support the project. > > -- > Lingzhe Sun > Hirain Technologies > > > *From:* Mich Talebzadeh > *Date:* 2023-04-11 02:06 > *To:* Rajesh Katkar > *CC:* user > *Subject

Re: Re: spark streaming and kinesis integration

s interested, please support the project. Lingzhe Sun Hirain Technologies From: Mich Talebzadeh Date: 2023-04-11 02:06 To: Rajesh Katkar CC: user Subject: Re: spark streaming and kinesis integration What I said was this"In so far as I know k8s does not support spark structured streaming?&quo

Re: Re: spark streaming and kinesis integration

logies >> >> >> *From:* Mich Talebzadeh >> *Date:* 2023-04-11 02:06 >> *To:* Rajesh Katkar >> *CC:* user >> *Subject:* Re: spark streaming and kinesis integration >> What I said was this >> "In so far as I know k8s does not support spark struc

Re: Re: spark streaming and kinesis integration

f anyone is interested, > please support the project. > > -- > Lingzhe Sun > Hirain Technologies > > > *From:* Mich Talebzadeh > *Date:* 2023-04-11 02:06 > *To:* Rajesh Katkar > *CC:* user > *Subject:* Re: spark streaming and kinesis in

Re: Re: spark streaming and kinesis integration

active for quite long time. Kind of worried that this project might finally become outdated as k8s is evolving. So if anyone is interested, please support the project. Lingzhe Sun Hirain Technologies From: Mich Talebzadeh Date: 2023-04-11 02:06 To: Rajesh Katkar CC: user Subject: Re: spark str

Re: spark streaming and kinesis integration

Just to clarify, a major benefit of k8s in this case is to host your Spark applications in the form of containers in an automated fashion so that one can easily deploy as many instances of the application as required (autoscaling). From below: https://price2meet.com/gcp/docs/dataproc_docs_concepts

Re: spark streaming and kinesis integration

What I said was this "In so far as I know k8s does not support spark structured streaming?" So it is an open question. I just recalled it. I have not tested myself. I know structured streaming works on Google Dataproc cluster but I have not seen any official link that says Spark Structured Streami

Re: spark streaming and kinesis integration

Do you have any link or ticket which justifies that k8s does not support spark streaming ? On Thu, 6 Apr, 2023, 9:15 pm Mich Talebzadeh, wrote: > Do you have a high level diagram of the proposed solution? > > In so far as I know k8s does not support spark structured streaming? > > Mich Talebzade

Re: spark streaming and kinesis integration

Use case is , we want to read/write to kinesis streams using k8s Officially I could not find the connector or reader for kinesis from spark like it has for kafka. Checking here if anyone used kinesis and spark streaming combination ? On Thu, 6 Apr, 2023, 7:23 pm Mich Talebzadeh, wrote: > Hi Raj

RE: spark streaming and kinesis integration

kar Cc: u...@spark.incubator.apache.org Subject: Re: spark streaming and kinesis integration ⚠ [EXTERNAL EMAIL]: Use Caution Do you have a high level diagram of the proposed solution? In so far as I know k8s does not support spark structured streaming? Mich Talebzadeh, Lead Solutions

Re: spark streaming and kinesis integration

Do you have a high level diagram of the proposed solution? In so far as I know k8s does not support spark structured streaming? Mich Talebzadeh, Lead Solutions Architect/Engineering Lead Palantir Technologies London United Kingdom view my Linkedin profile

Re: spark streaming and kinesis integration

Hi Rajesh, What is the use case for Kinesis here? I have not used it personally, Which use case it concerns https://aws.amazon.com/kinesis/ Can you use something else instead? HTH Mich Talebzadeh, Lead Solutions Architect/Engineering Lead Palantir Technologies London United Kingdom view m