Re: Flink s3 streaming performance

2020-06-06 Thread venkata sateesh` kolluru
ache.org/docs/current/hadoop-aws/tools/hadoop-aws/index.html#General_S3A_Client_configuration > > On Sat, Jun 6, 2020 at 1:26 AM venkata sateesh` kolluru < > vkollur...@gmail.com> wrote: > >> Hi Kostas and Arvid, >> >> Thanks for your suggestions. >> >> Th

Re: Flink s3 streaming performance

2020-06-05 Thread venkata sateesh` kolluru
;> messages. >>> Try to use a NoSQL database or write multiple messages into one file in >>> S3 (1 or 10) >>> >>> If you still want to go with your scenario then try a network optimized >>> instance and use s3a in Flink and configure s3 entropy

Re: Flink s3 streaming performance

2020-05-31 Thread venkata sateesh` kolluru
ink, because I've ended up > using a custom S3Sink, but I did have some issues writing to S3 because the > request wasn't parallelised. Check this thread, > http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Parallisation-of-S3-write-sink-td34064.html#a3407

Flink s3 streaming performance

2020-05-30 Thread venkata sateesh` kolluru
Hello, I have posted the same in stackoverflow but didnt get any response. So posting it here for help. https://stackoverflow.com/questions/62068787/flink-s3-write-performance-optimization?noredirect=1#comment109814428_62068787 Details: I am working on a flink application on kubernetes(eks) whi