Good to hear that it's working, thanks for the update! On Sat, Dec 1, 2018, 4:29 AM Vijay Balakrishnan <bvija...@gmail.com wrote:
> Hi Gordon, > Finally figured out my issue.Do not need to add http:// in proxyHost name. > String proxyHost= "proxy-chain....com";//not http://proxy-chain...com > kinesisConsumerConfig.setProperty(AWSUtil.AWS_CLIENT_CONFIG_PREFIX + > "proxyHost", proxyHost);//<== mo http:// in proxyHost name > > TIA, > Vijay > > > On Wed, Nov 14, 2018 at 12:50 AM Tzu-Li (Gordon) Tai <tzuli...@apache.org> > wrote: > >> Hi Vijay, >> >> I’m pretty sure that this should work with the properties that you >> provided, unless the AWS Kinesis SDK isn’t working as expected. >> >> What I’ve tested is that with those properties, the ClientConfiguration >> used to build the Kinesis client has the proxy domain / host / ports etc. >> properly set. >> And according to [1], this should be enough to configure the constructed >> Kinesis client to connect via the proxy. >> >> Cheers, >> Gordon >> >> [1] >> https://docs.aws.amazon.com/sdk-for-java/v1/developer-guide/section-client-configuration.html >> >> >> On 7 November 2018 at 1:19:02 AM, Vijay Balakrishnan (bvija...@gmail.com) >> wrote: >> >> Hi Gordon, >> This still didn't work :( >> >> Tried a few combinations with: >> kinesisConsumerConfig.setProperty(AWSUtil.AWS_CLIENT_CONFIG_PREFIX + >> "proxyDomain", "..."); >> >> inesisConsumerConfig.setProperty(AWSUtil.AWS_CLIENT_CONFIG_PREFIX + >> "proxyHost", "http://.....com"); >> >> kinesisConsumerConfig.setProperty(AWSUtil.AWS_CLIENT_CONFIG_PREFIX + >> "proxyPort", "911"); >> >> kinesisConsumerConfig.setProperty(AWSUtil.AWS_CLIENT_CONFIG_PREFIX + >> "proxyUsername", "..."); >> >> kinesisConsumerConfig.setProperty(AWSUtil.AWS_CLIENT_CONFIG_PREFIX + >> "proxyPassword", ".."); >> >> kinesisConsumerConfig.setProperty(AWSUtil.AWS_CLIENT_CONFIG_PREFIX + >> "nonProxyHosts", " >> >> >> How does the FlinkKinesisProducer work so seamlessly through a proxy ? >> TIA, >> Vijay >> >> On Thu, Oct 4, 2018 at 6:41 AM Tzu-Li (Gordon) Tai <tzuli...@apache.org> >> wrote: >> >>> Hi, >>> >>> Since Flink 1.5, you should be able to set all available configurations >>> on the ClientConfiguration through the consumer Properties (see FLINK-9188 >>> [1]). >>> >>> The way to do that would be to prefix the configuration you want to set >>> with "aws.clientconfig" and add that to the properties, as such: >>> >>> ``` >>> Properties kinesisConsumerProps = new Properties(); >>> kinesisConsumerProps.setProperty("aws.clientconfig.proxyHost", ...); >>> kinesisConsumerProps.setProperty("aws.clientconfig.proxyPort", ...); >>> kinesisConsumerProps.setProperty("aws.clientconfig.proxyUsert", ...); >>> ... >>> ``` >>> >>> Could you try that out and see if it works for you? >>> >>> I've also realized that this feature isn't documented very well, and >>> have opened a ticket for that [2]. >>> >>> Cheers, >>> Gordon >>> >>> [1] https://issues.apache.org/jira/browse/FLINK-9188 >>> [2] https://issues.apache.org/jira/browse/FLINK-10492 >>> >>> On Thu, Oct 4, 2018, 7:57 PM Aljoscha Krettek <aljos...@apache.org> >>> wrote: >>> >>>> Hi, >>>> >>>> I'm looping in Gordon and Thomas, they might have some idea about how >>>> to resolve this. >>>> >>>> Best, >>>> Aljoscha >>>> >>>> On 3. Oct 2018, at 17:29, Vijay Balakrishnan <bvija...@gmail.com> >>>> wrote: >>>> >>>> I have been trying with all variations to no avail of java >>>> -Dhttp.nonProxyHosts=.. -Dhttps.proxyHost=http://... >>>> -Dhttps.proxyPort=911 -Dhttps.proxyUser= -Dhttps.proxyPassword=.. >>>> -Dhttp.proxyHost=http://.. -Dhttp.proxyPort=911 -Dhttp.proxyUser=... >>>> -Dhttp.proxyPassword=... -jar .. after looking at the code in >>>> com.amazonaws.ClientConfiguration >>>> >>>> On Tue, Oct 2, 2018 at 3:49 PM Vijay Balakrishnan <bvija...@gmail.com> >>>> wrote: >>>> >>>>> HI, >>>>> How do I use FlinkKinesisConsumer using the Properties through a proxy >>>>> ? Getting a Connection issue through the proxy. >>>>> Works outside the proxy. >>>>> >>>>> Properties kinesisConsumerConfig = new Properties(); >>>>> >>>>> kinesisConsumerConfig.setProperty(AWSConfigConstants.AWS_REGION, region); >>>>> >>>>> if (local) { >>>>> >>>>> kinesisConsumerConfig.setProperty(AWSConfigConstants.AWS_ACCESS_KEY_ID, >>>>> accessKey); >>>>> >>>>> kinesisConsumerConfig.setProperty(AWSConfigConstants.AWS_SECRET_ACCESS_KEY, >>>>> secretKey); >>>>> } else { >>>>> >>>>> kinesisConsumerConfig.setProperty(AWSConfigConstants.AWS_CREDENTIALS_PROVIDER, >>>>> "AUTO"); >>>>> } >>>>> >>>>> //only for Consumer >>>>> >>>>> kinesisConsumerConfig.setProperty(ConsumerConfigConstants.SHARD_GETRECORDS_MAX, >>>>> "10000"); >>>>> >>>>> kinesisConsumerConfig.setProperty(ConsumerConfigConstants.SHARD_GETRECORDS_INTERVAL_MILLIS, >>>>> "2000"); >>>>> FlinkKinesisConsumer<Tuple2<InputMetadata, CameraWithCube>> >>>>> kinesisConsumer = new FlinkKinesisConsumer<>( >>>>> "kinesisTopicRead", new Tuple2KinesisSchema(), >>>>> kinesisConsumerConfig); >>>>> TIA >>>>> >>>> >>>>