Re: [EXTERNAL] Re: Spark streaming
Thanks, I'll try it out. On Fri, 19 Aug 2022, 6:12 pm Saurabh Gulati, wrote: > You can also try out > https://debezium.io/documentation/reference/0.10/connectors/mysql.html > -- > *From:* Ajit Kumar Amit > *Sent:* 19 August 2022 14:30 > *To:* sandra sukumaran > *Cc:* user@spark.apache.org > *Subject:* [EXTERNAL] Re: Spark streaming > > *Caution! This email originated outside of FedEx. Please do not open > attachments or click links from an unknown or suspicious origin*. > https://github.com/allwefantasy/spark-binlog > <https://urldefense.com/v3/__https://github.com/allwefantasy/spark-binlog__;!!BL9GA0TyTA!actkW_0ZhxHTImtHaWtPiApgCD3oISkeTDvhezK35qIcodH-PjmcOxZ9BCpVfZxXdhNi9ciMkltEEY7C5_QtFQ$> > > Sent from my iPhone > > On 19 Aug 2022, at 5:45 PM, sandra sukumaran > wrote: > > > Dear Sir, > > > > Is there any possible method to fetch MySQL database bin log, with > the help of spark streaming. > Kafka streaming is not applicable in this case. > > > > Thanks and regards > Sandra > >
Re: [EXTERNAL] Re: Spark streaming
You can also try out https://debezium.io/documentation/reference/0.10/connectors/mysql.html From: Ajit Kumar Amit Sent: 19 August 2022 14:30 To: sandra sukumaran Cc: user@spark.apache.org Subject: [EXTERNAL] Re: Spark streaming Caution! This email originated outside of FedEx. Please do not open attachments or click links from an unknown or suspicious origin. https://github.com/allwefantasy/spark-binlog<https://urldefense.com/v3/__https://github.com/allwefantasy/spark-binlog__;!!BL9GA0TyTA!actkW_0ZhxHTImtHaWtPiApgCD3oISkeTDvhezK35qIcodH-PjmcOxZ9BCpVfZxXdhNi9ciMkltEEY7C5_QtFQ$> Sent from my iPhone On 19 Aug 2022, at 5:45 PM, sandra sukumaran wrote: Dear Sir, Is there any possible method to fetch MySQL database bin log, with the help of spark streaming. Kafka streaming is not applicable in this case. Thanks and regards Sandra
Re: [EXTERNAL] Re: Spark streaming - Data Ingestion
I am beginner with spark may , also know how to connect MySQL database with spark streaming Thanks and regards Akash P On Wed, 17 Aug, 2022, 8:28 pm Saurabh Gulati, wrote: > Another take: > >- Debezium ><https://debezium.io/documentation/reference/stable/connectors/mysql.html> >to read Write Ahead logs(WAL) and send to Kafka >- Kafka connect to write to cloud storage -> Hive > - OR > > >- Spark streaming to parse WAL -> Storage -> Hive > > Regards > -- > *From:* Gibson > *Sent:* 17 August 2022 16:53 > *To:* Akash Vellukai > *Cc:* user@spark.apache.org > *Subject:* [EXTERNAL] Re: Spark streaming - Data Ingestion > > *Caution! This email originated outside of FedEx. Please do not open > attachments or click links from an unknown or suspicious origin*. > If you have space for a message log like, then you should try: > > MySQL -> Kafka (via CDC) -> Spark (Structured Streaming) -> HDFS/S3/ADLS > -> Hive > > On Wed, Aug 17, 2022 at 5:40 PM Akash Vellukai > wrote: > > Dear sir > > I have tried a lot on this could you help me with this? > > Data ingestion from MySql to Hive with spark- streaming? > > Could you give me an overview. > > > Thanks and regards > Akash P > >
Re: [EXTERNAL] Re: Spark streaming - Data Ingestion
The idea behind spark-streaming is to process change events as they occur, hence the suggestions above that require capturing change events using Debezium. But you can use jdbc drivers to connect Spark to relational databases On Wed, Aug 17, 2022 at 6:21 PM Akash Vellukai wrote: > I am beginner with spark may , also know how to connect MySQL database > with spark streaming > > Thanks and regards > Akash P > > On Wed, 17 Aug, 2022, 8:28 pm Saurabh Gulati, > wrote: > >> Another take: >> >>- Debezium >><https://debezium.io/documentation/reference/stable/connectors/mysql.html> >>to read Write Ahead logs(WAL) and send to Kafka >>- Kafka connect to write to cloud storage -> Hive >> - OR >> >> >>- Spark streaming to parse WAL -> Storage -> Hive >> >> Regards >> -- >> *From:* Gibson >> *Sent:* 17 August 2022 16:53 >> *To:* Akash Vellukai >> *Cc:* user@spark.apache.org >> *Subject:* [EXTERNAL] Re: Spark streaming - Data Ingestion >> >> *Caution! This email originated outside of FedEx. Please do not open >> attachments or click links from an unknown or suspicious origin*. >> If you have space for a message log like, then you should try: >> >> MySQL -> Kafka (via CDC) -> Spark (Structured Streaming) -> HDFS/S3/ADLS >> -> Hive >> >> On Wed, Aug 17, 2022 at 5:40 PM Akash Vellukai < >> akashvellukai...@gmail.com> wrote: >> >> Dear sir >> >> I have tried a lot on this could you help me with this? >> >> Data ingestion from MySql to Hive with spark- streaming? >> >> Could you give me an overview. >> >> >> Thanks and regards >> Akash P >> >>
Re: [EXTERNAL] Re: Spark streaming - Data Ingestion
Another take: * Debezium<https://debezium.io/documentation/reference/stable/connectors/mysql.html> to read Write Ahead logs(WAL) and send to Kafka * Kafka connect to write to cloud storage -> Hive * OR * Spark streaming to parse WAL -> Storage -> Hive Regards From: Gibson Sent: 17 August 2022 16:53 To: Akash Vellukai Cc: user@spark.apache.org Subject: [EXTERNAL] Re: Spark streaming - Data Ingestion Caution! This email originated outside of FedEx. Please do not open attachments or click links from an unknown or suspicious origin. If you have space for a message log like, then you should try: MySQL -> Kafka (via CDC) -> Spark (Structured Streaming) -> HDFS/S3/ADLS -> Hive On Wed, Aug 17, 2022 at 5:40 PM Akash Vellukai mailto:akashvellukai...@gmail.com>> wrote: Dear sir I have tried a lot on this could you help me with this? Data ingestion from MySql to Hive with spark- streaming? Could you give me an overview. Thanks and regards Akash P