Use-case: I am trying to see how to use flink with s3, where we use our own client libraries or things like AWS firehose to put data into S3, then process it in batch using flink. This clients are putting data into S3 with out HDFS - Aka we aren't using HDFS on top of S3.
Most of what I can find referenced [1] is using HDFS backed by S3 ( S3AFileSystem, NativeS3FileSystem) I find one reference [2] that using S3 Filesystem (S3FileSystem) doesn't wo rk. Can anyone with Flink experience help give any insight on this? References: - [1] - https://ci.apache.org/projects/flink/flink-docs-release-1.0/setup/aws.html - [2] - http://stackoverflow.com/questions/32959790/run-apache-flink-with-amazon-s3 -- *Steve Morin | Managing Partner - CTO* *Nvent* O 800-407-1156 ext 803 <800-407-1156;803> | M 347-453-5579 smo...@nventdata.com <smo...@nventdata.com> *Enabling the Data Driven Enterprise* *(Ask us how we can setup scalable open source realtime billion+ event/data collection/analytics infrastructure in weeks)* Service Areas: Management & Strategy Consulting | Data Engineering | Data Science & Visualization
