Hi, I got s3a working on localstack. The missing piece of information from Flink documentation seems to be that the system requires a HADOOP_HOME and core-site.xml.
Flink documentation states that s3p (presto) should be used for file checkpointing into s3. I am using RocksDB, which I assume also means that I should use s3p (the documentation was not specific about that). Is that assumption correct? However, I cannot get s3p working now. I did the following so far: I created the s3-fs-presto plugin directory and copied the jar from the opt directory there. I am not sure where to put the configuration keys though. The documentation states that I can just put in my flink-conf.yaml, but I had no success. Where do I put the presto configuration keys? Are there any other missing steps? Is this something that would only work on an EMR environment with a real HIVE running? # The S3 storage endpoint server. This can be used to connect to an S3-compatible storage # system instead of AWS. When using v4 signatures, it is recommended to set this to the # AWS region-specific endpoint (e.g., http[s]://<bucket>.s3-<AWS-region>. amazonaws.com). hive.s3.endpoint: http://aws:4566 # Use HTTPS to communicate with the S3 API (defaults to true). hive.s3.ssl.enabled: false # Use path-style access for all requests to the S3-compatible storage. # This is for S3-compatible storage that doesn’t support virtual-hosted-style access. (defaults to false) hive.s3.path-style-access: true But that also did not work. Any advice would be appreciated. -Marco Villalobos
