[GitHub] [hudi] nsivabalan commented on issue #5952: [SUPPORT] HudiDeltaStreamer S3EventSource SQS optimize for reading large number of files in parallel fashion

2022-11-03 Thread GitBox
nsivabalan commented on issue #5952: URL: https://github.com/apache/hudi/issues/5952#issuecomment-1302873977 Since we have a patch addressing the proposed fix, closing out the issue. Feel free to reach out to us if you need any further assistance. -- This is an automated message from

[GitHub] [hudi] nsivabalan commented on issue #5952: [SUPPORT] HudiDeltaStreamer S3EventSource SQS optimize for reading large number of files in parallel fashion

2022-09-13 Thread GitBox
nsivabalan commented on issue #5952: URL: https://github.com/apache/hudi/issues/5952#issuecomment-1244960649 again, just to clarify. the code you have referenced ``` sparkSession.read().json(eventRecords) ``` is actually reading data from SQS and does not involve reading from

[GitHub] [hudi] nsivabalan commented on issue #5952: [SUPPORT] HudiDeltaStreamer S3EventSource SQS optimize for reading large number of files in parallel fashion

2022-09-13 Thread GitBox
nsivabalan commented on issue #5952: URL: https://github.com/apache/hudi/issues/5952#issuecomment-1244959102 @glory9211 : got it. can you check if this is what you suggested https://github.com/apache/hudi/pull/6661/files -- This is an automated message from the Apache Git Service.

[GitHub] [hudi] nsivabalan commented on issue #5952: [SUPPORT] HudiDeltaStreamer S3EventSource SQS optimize for reading large number of files in parallel fashion

2022-08-27 Thread GitBox
nsivabalan commented on issue #5952: URL: https://github.com/apache/hudi/issues/5952#issuecomment-1229347594 looks like there is some confusion here. S3EventsSource actually reads events from SQS and populates the data into Hudi metadata table. this is the first stage which does not look

[GitHub] [hudi] nsivabalan commented on issue #5952: [SUPPORT] HudiDeltaStreamer S3EventSource SQS optimize for reading large number of files in parallel fashion

2022-08-16 Thread GitBox
nsivabalan commented on issue #5952: URL: https://github.com/apache/hudi/issues/5952#issuecomment-1216258288 @rmahindra123 : can you follow up when you can . -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL