1)You can use the application cluster mode you can find how to configure in the
official flink documentation
https://ci.apache.org/projects/flink/flink-docs-release-1.12/deployment/resource-providers/standalone/kubernetes.html#deploy-application-cluster
2)for HA you can use kubernetes HA:
http
Hi,
I'm using the official docker image: apache/flink:1.12.1-scala_2.11-java11
I'm trying to run a Flink job on an EKS cluster. The job is running under a
k8s service account that is tied to an IAM role. If I'm not using s3 as
RocksDB checkpoint backend, everything works just fine. However, when
Hi Thomas,
I think you might be looking for this:
https://github.com/apache/flink/pull/16717
Best
Ingo
On Sat, Sep 25, 2021, 20:46 Thomas Wang wrote:
> Hi,
>
> I'm using the official docker image: apache/flink:1.12.1-scala_2.11-java11
>
> I'm trying to run a Flink job on an EKS cluster. The j
Hi Thomas,
did you try to login to EKS node and run some aws command like : aws s3 ls
?
It sounds like EKS issue, but not 100% sure.
Best
On Sat, 25 Sept 2021 at 22:12, Ingo Bürk wrote:
> Hi Thomas,
>
> I think you might be looking for this:
> https://github.com/apache/flink/pull/16717
>
>
> Be
We need to overwrite using
WebIdentityTokenFileCredentialsProviderhttps://github.com/aws/aws-sdk-java-v2/issues/1470#issuecomment-543601232.
otherwise java takes presidency to secret key and access keys than SA
On Saturday, September 25, 2021, 04:37:22 PM EDT, Xiangyu Su
wrote:
Hi
Thanks Ingo. Adding the following setting worked.
fs.s3a.aws.credentials.provider:
com.amazonaws.auth.WebIdentityTokenCredentialsProvider
Thomas
On Sat, Sep 25, 2021 at 1:12 PM Ingo Bürk wrote:
> Hi Thomas,
>
> I think you might be looking for this:
> https://github.com/apache/flink/pull/16717
Hi!
Try the PARTITIONED BY clause. See
https://ci.apache.org/projects/flink/flink-docs-master/docs/connectors/table/formats/parquet/
Harshvardhan Shinde 于2021年9月24日周五
下午5:52写道:
> Hi,
> I wanted to know if we can write streaming data to S3 in parquet format
> with partitioning.
> Here's what I w
Hi, Puneet
Could you share whether you are using Flink's session mode or application
mode?
>From the log, you are using `StandaloneDispatcher`, but you will use it in
both session and application mode.
If you use application mode, this might be in line with expectations.
Best,
Guowei
On Fri, Se
Hi, Harshvardhan
I think CaiZhi is right.
I only have a small addition. Because I see that you want to convert Table
to DataStream, you can look at FileSink (ParquetWriterFactory)[1].
[1]
https://ci.apache.org/projects/flink/flink-docs-master/docs/connectors/datastream/file_sink/#bulk-encoded-for
Ingo,
It looks like I'm now seeing "Caused by: java.lang.NullPointerException:
You must specify a value for roleArn and roleSessionName". I assume I would
also need to specify that through the configuration file. Could you suggest
the key for this configuration? Thanks.
Thomas
On Sat, Sep 25, 20
Ingo,
I dig into the Flink code a little bit. It looks like the key for
specifying the roleArn and roleSessionName are
fs.s3a.aws.credentials.provider:
com.amazonaws.auth.WebIdentityTokenCredentialsProvider
fs.s3a.aws.credentials.provider.role.arn: arn:aws:iam::...:role/...
fs.s3a.aws.credentials.
11 matches
Mail list logo