Hanghang Liu created GOBBLIN-1922:
-------------------------------------
Summary: Create work units for selected topics partitions in Kafka
source
Key: GOBBLIN-1922
URL: https://issues.apache.org/jira/browse/GOBBLIN-1922
Project: Apache Gobblin
Issue Type: New Feature
Components: gobblin-kafka
Reporter: Hanghang Liu
Assignee: Shirshanka Das
Add a new feature in Kafka source to create workUnit for selected topics
partitions. This feature can provide functionality for:
# Recompute and split tasks(workUnit) during runtime for selected topics
partitiions, instead of recompute for all Kafka topics
# Make topic level replan feasible
# One step closer to dynamic work unit allocation
Need to have a followup to make work unit packer able to pack the recomputed
workunits into a desired number of containers
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
