[
https://issues.apache.org/jira/browse/KAFKA-12755?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Colin McCabe updated KAFKA-12755:
---------------------------------
Description:
*Problems*
The core module takes a long time to compile. There are several reasons for
this. One is that it’s too big -- it would be better as several gradle
modules. Gradle is good about compiling multiple modules in parallel, but if
you have one really big module, you lose that parallelism. Another issue with
the core module is that it’s written in Scala, and the Scala compiler takes
longer than the Java one.
A lot of server-side code is in the “clients” module. From there, it ends up
on the CLASSPATH of producers, consumers, and admin clients. This has a lot of
bad effects: it bloats the size of the clients jar, and allows downstream
projects to peek at code that should be isolated to the broker.
A lot of tools can’t be put into the “tools” module because they depend on
classes that are in “core”. And tools can’t have a core dependency, because
that would impose a core dependency on connect as well.
One example of this problem is StorageTool and ClusterTool. These tools ended
up getting written in Scala and put in the “core” module, rather than the
“tools” module.
Our long-term goal is to migrate from Scala to Java, and the monolithic core
module is an obstacle to that.
*Proposed Fixes*
Rename the “metadata” module to “controller” to reflect the fact that it
contains the controller
Make the "controller" module depend on "raft" rather than the other way around
("raft" used to depend on "metadata") This reflects the fact that the
controller consumes the API provided by the raft module. (There is a separate
PR to do this.)
Create a new “server-common” module for common code which is shared by several
server modules, but not needed for clients.
Remove the dependency between "connect" and "tools"
Create a new “server-tools“ module which depends on “core”
*The Server-Common Module*
The server-common module should contain:
* Pluggable APIs that are used only in the server (not in any client)
* The KIP-405 tiered storage APIs
* Authorizer APIs
* CreateTopicPolicy, AlterConfigPolicy, etc.
* Common Java utility code that is used in the server, but not used in the
client, such as ApiMessageAndVersion, KafkaEventQueue, VersionRange, etc.
was:
*Problems*
The core module takes a long time to compile. There are several reasons for
this. One is that it’s too big -- it would be better as several gradle
modules. Gradle is good about compiling multiple modules in parallel, but if
you have one really big module, you lose that parallelism. Another issue with
the core module is that it’s written in Scala, and the Scala compiler takes
longer than the Java one.
A lot of server-side code is in the “clients” module. From there, it ends up
on the CLASSPATH of producers, consumers, and admin clients. This has a lot of
bad effects: it bloats the size of the clients jar, and allows downstream
projects to peek at code that should be isolated to the broker.
A lot of tools can’t be put into the “tools” module because they depend on
classes that are in “core”. And tools can’t have a core dependency, because
that would impose a core dependency on connect as well.
One example of this problem is StorageTool and ClusterTool. These tools ended
up getting written in Scala and put in the “core” module, rather than the
“tools” module.
*Proposed Fixes*
Rename the “metadata” module to “controller” to reflect the fact that it
contains the controller
Make the "controller" module depend on "raft" rather than the other way around
("raft" used to depend on "metadata") This reflects the fact that the
controller consumes the API provided by the raft module. (There is a separate
PR to do this.)
Create a new “server-common” module for common code which is shared by several
server modules, but not needed for clients.
Remove the dependency between "connect" and "tools"
Create a new “server-tools“ module which depends on “core”
*The Server-Common Module*
The server-common module should contain:
* Pluggable APIs that are used only in the server (not in any client)
* The KIP-405 tiered storage APIs
* Authorizer APIs
* CreateTopicPolicy, AlterConfigPolicy, etc.
* Common Java utility code that is used in the server, but not used in the
client, such as ApiMessageAndVersion, KafkaEventQueue, VersionRange, etc.
> Add server-common, server-tools gradle modules
> ----------------------------------------------
>
> Key: KAFKA-12755
> URL: https://issues.apache.org/jira/browse/KAFKA-12755
> Project: Kafka
> Issue Type: Improvement
> Reporter: Colin McCabe
> Priority: Major
> Attachments: out.jpg, out2.jpg
>
>
> *Problems*
> The core module takes a long time to compile. There are several reasons for
> this. One is that it’s too big -- it would be better as several gradle
> modules. Gradle is good about compiling multiple modules in parallel, but if
> you have one really big module, you lose that parallelism. Another issue
> with the core module is that it’s written in Scala, and the Scala compiler
> takes longer than the Java one.
> A lot of server-side code is in the “clients” module. From there, it ends up
> on the CLASSPATH of producers, consumers, and admin clients. This has a lot
> of bad effects: it bloats the size of the clients jar, and allows downstream
> projects to peek at code that should be isolated to the broker.
> A lot of tools can’t be put into the “tools” module because they depend on
> classes that are in “core”. And tools can’t have a core dependency, because
> that would impose a core dependency on connect as well.
> One example of this problem is StorageTool and ClusterTool. These tools
> ended up getting written in Scala and put in the “core” module, rather than
> the “tools” module.
> Our long-term goal is to migrate from Scala to Java, and the monolithic core
> module is an obstacle to that.
> *Proposed Fixes*
> Rename the “metadata” module to “controller” to reflect the fact that it
> contains the controller
> Make the "controller" module depend on "raft" rather than the other way
> around ("raft" used to depend on "metadata") This reflects the fact that the
> controller consumes the API provided by the raft module. (There is a
> separate PR to do this.)
> Create a new “server-common” module for common code which is shared by
> several server modules, but not needed for clients.
> Remove the dependency between "connect" and "tools"
> Create a new “server-tools“ module which depends on “core”
> *The Server-Common Module*
> The server-common module should contain:
> * Pluggable APIs that are used only in the server (not in any client)
> * The KIP-405 tiered storage APIs
> * Authorizer APIs
> * CreateTopicPolicy, AlterConfigPolicy, etc.
> * Common Java utility code that is used in the server, but not used in the
> client, such as ApiMessageAndVersion, KafkaEventQueue, VersionRange, etc.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
