Spark streaming

Dear sir, I want to check the logs of MySQL database using spark streaming, can someone help me with those listening queries. Thanks and regards Akash P

Re: Data ingestion

from my experience, spark can read/write from/to both mysql and hive fluently. regards. Akash Vellukai wrote: How we could do data ingestion from MySQL to Hive with the help of Spark streaming and not with Kafka - To

Re: Data ingestion

If you are on aws, you can use RDS + AWS DMS to save data to s3 and then read streaming data with spark structured streaming from s3 into hive Best regards > On 17 Aug 2022, at 20:51, Akash Vellukai wrote: > >  > Dear Sir, > > > How we could do data ingestion from MySQL to Hive with the

Re: Data ingestion

Hello Spark does not have any built-in solution for this problem. Most probably you will want to use Debezium+Kafka and read with Spark from Kafka [image: facebook] [image: twitter] [image: linkedin]

Data ingestion

Dear Sir, How we could do data ingestion from MySQL to Hive with the help of Spark streaming and not with Kafka Thanks and regards Akash

Re: [EXTERNAL] Re: Spark streaming - Data Ingestion

I am beginner with spark may , also know how to connect MySQL database with spark streaming Thanks and regards Akash P On Wed, 17 Aug, 2022, 8:28 pm Saurabh Gulati, wrote: > Another take: > >- Debezium > >to

Re: [EXTERNAL] Re: Spark streaming - Data Ingestion

The idea behind spark-streaming is to process change events as they occur, hence the suggestions above that require capturing change events using Debezium. But you can use jdbc drivers to connect Spark to relational databases On Wed, Aug 17, 2022 at 6:21 PM Akash Vellukai wrote: > I am

Re: [EXTERNAL] Re: Spark streaming - Data Ingestion

Another take: * Debezium to read Write Ahead logs(WAL) and send to Kafka * Kafka connect to write to cloud storage -> Hive * OR * Spark streaming to parse WAL -> Storage -> Hive Regards

Re: Spark streaming - Data Ingestion

If you have space for a message log like, then you should try: MySQL -> Kafka (via CDC) -> Spark (Structured Streaming) -> HDFS/S3/ADLS -> Hive On Wed, Aug 17, 2022 at 5:40 PM Akash Vellukai wrote: > Dear sir > > I have tried a lot on this could you help me with this? > > Data ingestion from

Spark streaming - Data Ingestion

Dear sir I have tried a lot on this could you help me with this? Data ingestion from MySql to Hive with spark- streaming? Could you give me an overview. Thanks and regards Akash P