Few comments: Fully support switching to poetry as the dependency management / build tool. The main sticking point we may run into is whether or not all of our dependencies are pure-python libraries. I don't recall specifics but I thought there might be at least one dependency (shapely maybe?) that requires specific c libraries be installed; which is why conda was chosen in the first place. It's possible this is no longer the case and the benefits of switching to poetry IMO justify the cost of looking at this again. Additionally, I know there has been improved support between conda/poetry in that poetry will recognize if it is within a conda environment and use that environment instead of creating a new virtualenv. So if we were able to identify which specific dependencies require installation by conda, we could limit the conda dependencies to just those libraries but have everything else managed by poetry.
With respect to combining the packages. The intent here was to move towards a micro-services architecture where the various algorithms were independent deployables with data-access maintained as a shared library used by all of them. In reality, this has not played out and it seems installations of sdap continue to use all the algorithms all the time. So, in my mind it is a question of what the desired architecture for sdap is. I'm still of the belief that a microservices style architecture is more flexible and scalable; however, it does increase the maintenance burden required to maintain all of the services. In the end, an uber-app is easier to manage from an installation perspective. - Frank On Thu, Feb 29, 2024 at 8:39 AM Stepheny Perez <skpe...@apache.org> wrote: > Riley, > > I think this is a good suggestion -- we need an easier way to manage SDAP > dependencies and transitive dependencies, especially given the license > issues we've come across multiple times now. This would also make building > the software locally a lot easier. I'm a big fan of poetry in particular > and think it would suit SDAP well. > > I'm also supportive of combining the analysis and data-access packages > into a single package - the current organization makes development more > tedious without much benefit -- especially because they are packaged > together in the Docker container anyways. > > Stepheny > > > > On 2024/02/28 22:07:34 Riley Kuttruff wrote: > > We currently use conda/mamba to install dependencies for the granule > ingester and Nexus components in both local dev environments and for > building Docker images. This seems to have unwanted consequences as Docker > appears to be forcing inclusion of some GPL-licensed tools and libraries > that we don't seem to use but likely block releases under ASF policy. This > issue was brought up in our IPMC graduation discussion as needing to be > solved. > > > > I'd like to propose replacing conda with another package manager like > poetry or simply using pip. We could also take this opportunity to combine > the two top-level Nexus modules so that they're not > individual-yet-interdependent modules with their own dependencies, but > rather two submodules of a single project with a single, shared list of > dependencies. > > >
