Re: Batch Flink Job S3 write performance vs Spark

Hey, I don't think there will be a big performance difference. Both systems have many users writing data to S3, so this will be optimized for both. On Tue, Feb 25, 2020 at 6:03 PM sri hari kali charan Tummala < kali.tumm...@gmail.com> wrote: > Hi All, > > have a question did anyone compared the

Re: Batch Flink Job S3 write performance vs Spark

sorry for being lazy I would have gone through flink source code. On Wed, Feb 26, 2020 at 9:35 AM sri hari kali charan Tummala < kali.tumm...@gmail.com> wrote: > Ok, thanks for the clarification. > > On Wed, Feb 26, 2020 at 9:22 AM Arvid Heise wrote: > >> Exactly. We use the hadoop-fs as an indi

Re: Batch Flink Job S3 write performance vs Spark

Ok, thanks for the clarification. On Wed, Feb 26, 2020 at 9:22 AM Arvid Heise wrote: > Exactly. We use the hadoop-fs as an indirection on top of that, but Spark > probably does the same. > > On Wed, Feb 26, 2020 at 3:52 PM sri hari kali charan Tummala < > kali.tumm...@gmail.com> wrote: > >> Than

Re: Batch Flink Job S3 write performance vs Spark

Exactly. We use the hadoop-fs as an indirection on top of that, but Spark probably does the same. On Wed, Feb 26, 2020 at 3:52 PM sri hari kali charan Tummala < kali.tumm...@gmail.com> wrote: > Thank you (the two systems running on Java and using the same set of > libraries), so from my understa

Re: Batch Flink Job S3 write performance vs Spark

Thank you (the two systems running on Java and using the same set of libraries), so from my understanding, Flink uses AWS SDK behind the scenes same as spark. On Wed, Feb 26, 2020 at 8:49 AM Arvid Heise wrote: > Fair benchmarks are notoriously difficult to setup. > > Usually, it's easy to find

Re: Batch Flink Job S3 write performance vs Spark

Fair benchmarks are notoriously difficult to setup. Usually, it's easy to find a workload where one system shines and as its vendor you report that. Then, the competitor benchmarks a different use case where his system outperforms ours. In the end, customers are more confused than before. You sho

Batch Flink Job S3 write performance vs Spark

Hi All, have a question did anyone compared the performance of Flink batch job writing to s3 vs spark writing to s3? -- Thanks & Regards Sri Tummala

Batch Flink Job S3 write performance vs Spark

Hi All, have a question did anyone compared the performance of Flink batch job writing to s3 vs spark writing to s3? -- Thanks & Regards Sri Tummala