Hi everyone, SDAP should be able to support many users' needs out of the box, but there have been projects I've seen interested in utilizing SDAP that would require specialized endpoints to deal with, for example, certain niche data types or algorithms that are tailored to specific datasets or that produce specially-formatted outputs.
Rather than add new algorithms to SDAP's releases for these specific use cases, we should support these projects to easily write their own using SDAP's data-access and existing analysis mechanisms and have these algorithms easily addable to SDAP at runtime. Thomas Loubrieu pointed me towards where SDAP reads in the algorithm modules [1] which is configurable by an .ini file [2], we could likely utilize this as our entrypoint to this plugin system. The user would just have to ensure that their plugins are packaged up and importable to SDAP's environment, and that the nexus_handler classes they're using are loaded on import. I have tested this successfully on my local machine. We should also develop a guide to help users do this and provide a sample in our main repo. Some questions: 1. The plugins should be developed on the same environment SDAP runs in as defined in pyproject.toml, but what if additional dependencies are required? What if they require updates to existing dependencies? What if their requirements are incompatible or conflict with SDAP's requirements (but are not necessarily breaking)? 2. How should we handle building images with the plugins for Docker and Kubernetes deployments? 3. Should we work to extend this support beyond user-provided algorithms? For example, to data-access backends? The SDAP PMC could also release and maintain a set of "official" algorithm plugins whose functionalities we deem might be applicable to a subset of the user base, but are specific enough that inclusion in the base set of SDAP algorithms might be excessive. I'd love to hear your thoughts on this. -Riley [1] https://github.com/apache/sdap-nexus/blob/c920ea39c44cf8d581aa94c9d7cdbc9ef9b2ad67/analysis/webservice/webapp.py#L122 [2] https://github.com/apache/sdap-nexus/blob/c920ea39c44cf8d581aa94c9d7cdbc9ef9b2ad67/analysis/webservice/config/web.ini#L32
