Re: Flink SQL and createRemoteEnvironment

[sangram reddy]

Hi,

createRemoteEnvironment(...) methods have some obscure documentation.

createRemoteEnvironment(String

host,int port,String
...
jarFiles)

What will be the port here? and which jatFiles to supply?

API also mentions "The remote environment sends (parts of) the program
to a cluster for execution. "

What exactly does that mean?

Thanks,

Sangram




On Tue, Nov 28, 2023 at 11:12 AM Xuyang  wrote:

> Hi, Dan.
> Can you provide more details?
>
> > I'm seeing unexpected behavior where it appears like the sql is
> executed locally.
> Did you find a minicluster started locally running you program?
>
> > In my case the remote environment is inside AWS and it doesn't appear
> to pick up the region and credentials unless I set the environment
> variables locally
> I think you need make sure your local machin can connect to AWS
> environment firstly.
>
> Overall, I think `*StreamExecutionEnvironment#**createRemoteEnvironment *`
> can meet your requirements.
>
>
> --
> Best！
> Xuyang
>
>
> At 2023-11-28 03:49:44, "Oxlade, Dan"  wrote:
>
> Hi,
>
> If I use *StreamExecutionEnvironment.createRemoteEnvironment* and then
> *var tEnv = StreamTableEnvironment.create(env)* from the resulting remote 
> *StreamExecutionEvironment
> *will any sql executed using *tEnv.executeSql* be executed remotely
> inside the flink cluster?
>
> I'm seeing unexpected behavior where it appears like the sql is executed
> locally. In my case the remote environment is inside AWS and it doesn't
> appear to pick up the region and credentials unless I set the environment
> variables locally - I want the job to run inside the cluster and use the
> remote AWS context, including things like the auth credentials of the AWS
> task.
>
> I feel like I might be fundamentally misunderstanding.
>
> Thanks
> Dan
>
> T. Rowe Price International Ltd (registered number 3957748) is registered
> in England and Wales with its registered office at Warwick Court, 5
> Paternoster Square, London EC4M 7DX. T. Rowe Price International Ltd is
> authorised and regulated by the Financial Conduct Authority. The company
> has a branch in Dubai International Financial Centre (regulated by the DFSA
> as a Representative Office).
>
> T. Rowe Price (including T. Rowe Price International Ltd and its
> affiliates) and its associates do not provide legal or tax advice. Any
> tax-related discussion contained in this e-mail, including any attachments,
> is not intended or written to be used, and cannot be used, for the purpose
> of (i) avoiding any tax penalties or (ii) promoting, marketing, or
> recommending to any other party any transaction or matter addressed herein.
> Please consult your independent legal counsel and/or professional tax
> advisor regarding any legal or tax issues raised in this e-mail.
>
> The contents of this e-mail and any attachments are intended solely for
> the use of the named addressee(s) and may contain confidential and/or
> privileged information. Any unauthorized use, copying, disclosure, or
> distribution of the contents of this e-mail is strictly prohibited by the
> sender and may be unlawful. If you are not the intended recipient, please
> notify the sender immediately and delete this e-mail.
>
>

Flink SQL and createRemoteEnvironment

[Oxlade, Dan]

Hi,

If I use StreamExecutionEnvironment.createRemoteEnvironment and then
var tEnv = StreamTableEnvironment.create(env) from the resulting remote 
StreamExecutionEvironment will any sql executed using tEnv.executeSql be 
executed remotely inside the flink cluster?

I'm seeing unexpected behavior where it appears like the sql is executed 
locally. In my case the remote environment is inside AWS and it doesn't appear 
to pick up the region and credentials unless I set the environment variables 
locally - I want the job to run inside the cluster and use the remote AWS 
context, including things like the auth credentials of the AWS task.

I feel like I might be fundamentally misunderstanding.

Thanks
Dan
T. Rowe Price International Ltd (registered number 3957748) is registered in 
England and Wales with its registered office at Warwick Court, 5 Paternoster 
Square, London EC4M 7DX. T. Rowe Price International Ltd is authorised and 
regulated by the Financial Conduct Authority. The company has a branch in Dubai 
International Financial Centre (regulated by the DFSA as a Representative 
Office).

T. Rowe Price (including T. Rowe Price International Ltd and its affiliates) 
and its associates do not provide legal or tax advice. Any tax-related 
discussion contained in this e-mail, including any attachments, is not intended 
or written to be used, and cannot be used, for the purpose of (i) avoiding any 
tax penalties or (ii) promoting, marketing, or recommending to any other party 
any transaction or matter addressed herein. Please consult your independent 
legal counsel and/or professional tax advisor regarding any legal or tax issues 
raised in this e-mail.

The contents of this e-mail and any attachments are intended solely for the use 
of the named addressee(s) and may contain confidential and/or privileged 
information. Any unauthorized use, copying, disclosure, or distribution of the 
contents of this e-mail is strictly prohibited by the sender and may be 
unlawful. If you are not the intended recipient, please notify the sender 
immediately and delete this e-mail.