There are multiple ways of passing configuration parameters to your user
defined code in Flink
a) use getRuntimeContext().getUserCodeClassLoader().getResource() to load
a config file from your user code jar or the classpath.
b) use getRuntimeContext().getExecutionConfig().getGlobalJobParameters() to
access a configuration object serialized from the main method.
you can pass a custom object to the job parameters, or use Flink's
"Configuration" object in your main method:
final StreamExecutionEnvironment env =
StreamExecutionEnvironment.getExecutionEnvironment();
Configuration config = new Configuration();
config.setString("foo", "bar");
env.getConfig().setGlobalJobParameters(config);
c) Load the flink-conf.yaml:
Configuration conf = GlobalConfiguration.loadConfiguration();
I'm not 100% sure if this approach works, as it is not intended to be used
in user code (I believe).
Let me know if this helps!
Best,
Robert
On Thu, Feb 20, 2020 at 1:50 PM Chesnay Schepler <ches...@apache.org> wrote:
> First things first, we do not intend for users to use anything in the S3
> filesystem modules except the filesystems itself,
> meaning that you're somewhat treading on unsupported ground here.
>
> Nevertheless, the S3 modules contain a large variety of AWS-provided
> CerentialsProvider implementations,
> that can derive credentials from environment variables, system properties,
> files on the classpath and many more.
>
> Ultimately though, you're kind of asking us how to use AWS APIs, for which
> I would direct you to the AWS documentation.
>
> On 20/02/2020 13:16, David Magalhães wrote:
>
> I'm using
> org.apache.flink.fs.s3base.shaded.com.amazonaws.client.builder.AwsClientBuilder
> to create a S3 client to copy objects and delete object inside
> a TwoPhaseCommitSinkFunction.
>
> Shouldn't be another way to set up configurations without put them
> hardcoded ? Something like core-site.xml or flink-conf.yaml ?
>
> Right now I need to have them hardcoded like this.
>
> AmazonS3ClientBuilder.standard
> .withPathStyleAccessEnabled(true)
> .withEndpointConfiguration(
> new EndpointConfiguration("http://minio:9000";, "us-east-1")
> )
> .withCredentials(
> new AWSStaticCredentialsProvider(new BasicAWSCredentials("minio",
> "minio123"))
> )
> .build
>
> Thanks
>
>
>
