[
https://issues.apache.org/jira/browse/FLINK-8190?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Tzu-Li (Gordon) Tai updated FLINK-8190:
---------------------------------------
Description:
The required internals for pattern-based topic discovery was implemented as
part of FLINK-6022 (along with partition discovery). However, the functionality
for pattern-based topic discovery was not yet exposed via any visible user API
on the version-specific subclasses of {{FlinkKafkaConsumerBase}}.
I propose to add two more constructors for this:
{code}
public FlinkKafkaConsumerXX(java.util.regex.Pattern subscriptionPattern,
DeserializationSchema<T> schema, Properties props);
public FlinkKafkaConsumerXX(java.util.regex.Pattern subscriptionPattern,
KeyedDeserializationSchema<T> schema, Properties props);
{code}
This allows the consumer to pick up all matching topics on startup.
To continuously pick up matching topics on the fly when they are created after
the job has already started running, users should additionally set the
{{KEY_PARTITION_DISCOVERY_INTERVAL_MILLIS}} property, as they would do for
partition discovery.
was:
The required internals for pattern-based topic discovery was implemented as
part of FLINK-6022 (along with partition discovery). However, the functionality
for pattern-based topic discovery was not yet exposed via any visible user API
on the version-specific subclasses of {{FlinkKafkaConsumerBase}}.
I propose to add two more constructors for this:
{code}
public FlinkKafkaConsumerXX(Pattern subscriptionPattern,
DeserializationSchema<T> schema, Properties props);
public FlinkKafkaConsumerXX(Pattern subscriptionPattern,
KeyedDeserializationSchema<T> schema, Properties props);
{code}
This allows the consumer to pick up all matching topics on startup.
To continuously pick up matching topics on the fly when they are created after
the job has already started running, users should additionally set the
{{KEY_PARTITION_DISCOVERY_INTERVAL_MILLIS}} property, as they would do for
partition discovery.
> Add extra FlinkKafkaConsumer constructors to expose pattern-based topic
> subscription functionality
> --------------------------------------------------------------------------------------------------
>
> Key: FLINK-8190
> URL: https://issues.apache.org/jira/browse/FLINK-8190
> Project: Flink
> Issue Type: New Feature
> Components: Kafka Connector
> Reporter: Tzu-Li (Gordon) Tai
> Assignee: Tzu-Li (Gordon) Tai
> Fix For: 1.4.0, 1.5.0
>
>
> The required internals for pattern-based topic discovery was implemented as
> part of FLINK-6022 (along with partition discovery). However, the
> functionality for pattern-based topic discovery was not yet exposed via any
> visible user API on the version-specific subclasses of
> {{FlinkKafkaConsumerBase}}.
> I propose to add two more constructors for this:
> {code}
> public FlinkKafkaConsumerXX(java.util.regex.Pattern subscriptionPattern,
> DeserializationSchema<T> schema, Properties props);
> public FlinkKafkaConsumerXX(java.util.regex.Pattern subscriptionPattern,
> KeyedDeserializationSchema<T> schema, Properties props);
> {code}
> This allows the consumer to pick up all matching topics on startup.
> To continuously pick up matching topics on the fly when they are created
> after the job has already started running, users should additionally set the
> {{KEY_PARTITION_DISCOVERY_INTERVAL_MILLIS}} property, as they would do for
> partition discovery.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
