Hi David! A pretty long and interesting email!
On 17/05/2020 13:01, David Leangen wrote: > > Hi! > > As I just mentioned in a different email: > >> As part of my documentation project, I am starting to turn my attention >> towards the code itself. I am trying to understand how the system works. […] >> To understand the system better, I need to take a step back and understand >> Guice. I am learning about Guice now. So far, guice looks quite easy to >> understand and use (at least for me because as a long-time OSGi user I >> understand very well the concepts of DI, api/impl separation, services, >> etc.). I can understand very well the motivation for using a framework like >> Guice, and I think (hope!) it should help me to understand what is going on >> in the system. > > You’ll have to excuse my inexperience with Guice. However, to understand > James, it is imperative (ok, actually “declarative” hahahaha I’m so funny) to > understand its organization into Guice Modules. I have not read through all > of the Guice documentation, but I think I get the idea. > > In this context, I have started looking at the James code base. There are > many very nice things about it. Despite its size and complexity, I was able > to understand quite a lot thanks to the good organization and naming > conventions. That said, I think we could still do a lot better. > > Organizing into Guice Modules is very nice. One of the important benefits is > that it helps a lot to break the system down into reasonable chunks that are > more understandable. It is simply impossible for a normal human to understand > the system all at once. It is necessary to have several levels of > abstraction. The use of Guice Modules to provide a “just right" level of > abstraction (not too high so as to be useless when trying to run it, but not > so low as to be much too detailed) is essential. I would wager that in most > situations, this is the level of abstraction that is most useful for a > developer or system operator (and definitely for an application assembler). > The only time a package or class level is necessary is when actually making > changes to the code. > > I would even double down on my statement and say that the organization of the > Guice Modules is perhaps THE most important abstraction available to allow > people to understand the system. I had a write at "why we chose Guice" and your statements perfectly complete it, I will add it in https://github.com/apache/james-project/blob/master/src/adr/0036-against-use-of-conditional-statements-in-guice-modules.md > > Ideally, to help provide a better understanding of the system and its > compile-time (and even to some extent its runtime) organization, I think it > is important to: > > * Match the Guice Modules with the Maven modules, matching them exactly if > possible > —> At first glance, this seems pretty good That is what we tried to do, however we also wanted to limit the cardinality of maven modules, and split guice-maven-modules when it was used by different profiles. > > * Ensure that each Module is well-contained (i.e. no “leaks” or coupling to > other implementations) > —> I found this part to be quite problematic (topic for a different > email) > > * Understand the API surface area of each Module > —> I am having a lot of trouble with this (again, a topic for a > different email) > > > Below is a list of all the Guice Modules I was able to find in the system. I > simply did a search and manually extracted these. There are 144!! > > Although I could guess a little bit as to what they do, I was otherwise > unable to understand many things: > > * What is the purpose? I don't understand this question. If you take into account the cardinality (profile x components) and the fact that profiles don't enable the same administration features (that thus needs to be split), a high cardinality is in my opinion understandable. > * How does it relate to other Modules? > * Which Modules are implementing the same API, so are “swappable”? > * What is the hierarchy? (Assuming that there are Modules of Modules)? Modules can "install" other modules, we use that to better split the logic, for instance for the JMAP protocol. > > > It is a great start! However, to be able to put more order into all of this, > these are my initial thoughts: > > 1. Document each module individually (even if only a line or two of text) > 2. Understand if there is any hierarchy (i.e. Modules of Modules) > —> Refactor / Rename the Modules to make the hierarchy immediately clear > 3. Validate that each Guice Module is perfectly aligned with its Maven module > 4. With this clearer picture, evaluate whether or not the current project > structure is still appropriate > 5. Update the documentation of each Maven module to show all of its Guice > Modules This documentation effort would be a great move toward James as a toolkit to write your own email server. We of course had it in mind during Guice adoption, but not yet had feedback on the topic. I am not aware of community usage of James Guice modules to build their own mail server. Such a documentation effort could of course help! * Maybe marker interfaces to 'tag' modules as 'ComponentModule' or 'ProtocolModule' could help the module discovery? That would allow to quickly identify top-level modules. Also, some JavaDoc in the modules themselves would of course help. We have for the mailet some documentation page generated from the javadoc, writing a documentation page for guice modules based on this model can be doable. > > With this clearer understanding, it should be much easier to wire together a > “product” without having to rely on a “supported product”. I think the system > appears to be well-designed. It’s just very difficult to understand right > now. If we can make it easier to grok, it will be much easier to use. For sure! Until now we had only main developers contributing to the part of the code and cruelly lacked feedback. Benoit > > > Thoughts? > > Cheers, > =David > > > BlobStoreAPIModule > BlobExportMechanismModule > LinshareBlobExportMechanismModule > LocalFileBlobExportMechanismModule > BlobMemoryModule > ObjectStorageBlobStoreModule > ObjectStorageDependenciesModule > MyExtensionModule > CassandraDLPConfigurationStoreModule > CassandraDomainListModule > CassandraJmapModule > CassandraMailRepositoryModule > CassandraRecipientRewriteTableModule > CassandraSieveRepositoryModule > CassandraUsersRepositoryModule > CassandraEventStoreModule > CassandraBlobStoreModule > CassandraCacheSessionModule > CassandraDeadLetterModule > CassandraDeletedMessageVaultModule > CassandraMailboxModule > CassandraQuotaMailingModule > CassandraQuotaModule > CassandraSessionModule > ElasticSearchClientModule > ElasticSearchMailboxModule > ElasticSearchQuotaSearcherModule > TikaMailboxModule > CassandraMetricsModule > CassandraRoutesModule > InconsistencySolvingRoutesModule > SolveMailboxInconsistenciesModules > SolveMessageInconsistenciesModules > TestDockerElasticSearchModule > TestDockerESMetricReporterModule > TestTikaModule > LdapUsersRepositoryModule > BlobStoreChoosingModule > RabbitMQEventBusModule > DistributedTaskManagerModule > TaskSerializationModule > TestAwsS3BlobStoreModule > TestRabbitMQModule > TestSwiftBlobStoreModule > ActiveMQQueueModule > ProtocolHandlerModule > DefaultProcessorsConfigurationProviderModule > DNSServiceModule > DropWizardMetricsModule > HostnameModule > LoggingMetricsModule > MailStoreRepositoryModule > RawPostDequeueDecoratorModule > TaskManagerModule > CleanupTaskModule > ClockModule > CommonServicesModule > IsStartedProbeModule > MailetProcessingModule > MimeMessageModule > PeriodicalHealthChecksModule > StartablesModule > StartUpChecksModule > ElasticSearchMetricReporterModule > IMAPServerModule > JMAPCommonModule > JMAPModule > MethodsModule > JMAPDraftServerModule > TestJMAPServerModule > SearchModule > JMXServerModule > LMTPServerModule > DefaultEventModule > FastRetryBackoffModule > MemoryDeadLetterModule > PreDeletionHookModule > MailboxModule > SpamAssassinListenerModule > CamelMailetContainerModule > DKIMMailetModule > ManageSieveServerModule > SieveModule > NettyServerModule > POP3ServerModule > RabbitMQModule > SieveFileRepositoryModule > SieveJPARepositoryModules > JSPFModule > SMTPServerModule > MyExtensionModule > ExtensionModule > HealthCheckRoutesModule > NoJwtModule > TaskRoutesModule > WebAdminServerModule > DataRoutesModules > DLPRoutesModule > SieveRoutesModule > JmapTasksModule > InconsistencyQuotasSolvingRoutesModule > MailboxesBackupModule > MailboxesExportRoutesModule > MailboxesRoutesModule > MailboxRoutesModule > MessagesRoutesModule > ReIndexingModule > MailQueueRoutesModule > MailRepositoriesRoutesModule > SwaggerRoutesModule > SpamAssassinModule > JPADataModule > JPADomainListModule > JPAEntityManagerModule > JPAMailRepositoryModule > JPARecipientRewriteTableModule > JPAUsersRepositoryModule > TestJPAConfigurationModule > TestJPAConfigurationModuleWithSqlValidation > NoDatabaseAuthentication > WithDatabaseAuthentication > JPAMailboxModule > JpaQuotaModule > JPAQuotaSearchModule > LuceneSearchMailboxModule > TestJPAConfigurationModule > DeletedMessageVaultModule > DeletedMessageVaultRetentionModule > DeletedMessageVaultRoutesModule > TestDeleteMessageVaultPreDeletionHookModule > MemoryDataJmapModule > MemoryDataModule > MemoryEventStoreModule > MemoryMailboxModule > MemoryQuotaModule > MemoryQuotaSearchModule > MemoryMailQueueModule > FakeSearchMailboxModule > LifeCycleModule > MultiLifeCycleTestCase > UnauthorizedModule > WebadminIntegrationTestModule > SpamAssassinModule > TestingSessionModule > > > --------------------------------------------------------------------- > To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org > For additional commands, e-mail: server-dev-h...@james.apache.org > --------------------------------------------------------------------- To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org For additional commands, e-mail: server-dev-h...@james.apache.org
