Re: Where to hide credentials when submitting a beam pipeline

2017-07-07 Thread Parker Coleman
I asked this same question a few weeks ago, we needed a way to have our pipeline connect to an Azure instance (I think the ticket actually came from that discussion). We decided to put the credentials in Google Cloud Storage, It can be locked down with restrictive permissions and its easy to access

Re: Where to hide credentials when submitting a beam pipeline

2017-07-07 Thread Lukasz Cwik
I think your looking for this feature: https://issues.apache.org/jira/browse/BEAM-2492 On Fri, Jul 7, 2017 at 11:40 AM, Randal Moore wrote: > Maybe this is more of a question for DataFlow - but I'm submitting a > pipeline that needs to access a rest service running in a GKE kubernetes > instance

Where to hide credentials when submitting a beam pipeline

2017-07-07 Thread Randal Moore
Maybe this is more of a question for DataFlow - but I'm submitting a pipeline that needs to access a rest service running in a GKE kubernetes instance. I need to pass in creds. I started with pipeline-options which work but all options get exposed on the DataFlow web pages. Is there a way to pas