That sounds about right. I just want to be able to exclude super heavy things like the beam engine, and include other things I use more frequently like cpython etc. Grouping things together will probably make them easier to understand, and hopefully a desire to standardize the interfaces a bit more for them over time. Great idea overall.
On Tue, Mar 18, 2025 at 11:07 AM Bart Maertens <[email protected]> wrote: > absolutely! Once we have these plugins grouped in modules, it will be a lot > easier to clean them up and reduce code duplication. > > On Tue, Mar 18, 2025 at 4:49 PM Hans Van Akelyen < > [email protected]> > wrote: > > > Hi All, > > > > I have been pondering a bit and we have quite some modules that that > > actually need each other to work properly. > > We might have been a bit overzealous when splitting everything in > separate > > modules in the project. > > I would like to propose combining following actions/transforms and love > to > > hear your thoughts. > > > > Move Actions/Transforms related to a specific database type to the > > plugins/database. > > eg. > > hop-transform-pgbulkloader -> hop-databases-postgresql > > > > hop-action-mysqlbulkfile -> hop-databases-mysql > > hop-action-mysqlbulkload -> hop-databases-mysql > > hop-transform-mysqlbulkloader -> hop-databases-mysql > > ... > > > > Combine File related transforms into a single module: > > > > hop-transform-fileexists -> hop-transform-files > > hop-transform-filelocked -> hop-transform-files > > hop-transform-filemetadata -> hop-transform-files > > hop-transform-filesfromresult -> hop-transform-files > > hop-transform-filestoresult -> hop-transform-files > > hop-transform-getfilenames -> hop-transform-files > > hop-transform-getfilesrowcount -> hop-transform-files > > hop-transform-getsubfolders -> hop-transform-files > > hop-transform-processfiles -> hop-transform-files > > > > Same for actions > > > > We also have some in/outputs of the same type split into separate modules > > and might also make more sense to combine these: > > eg. > > hop-transform-cubeinput -> hop-transform-cube > > hop-transform-cubeoutput -> hop-transform-cube > > > > I would like to make more logical groups and shared code in the > > transforms/actions/databases. > > This will make our build and deployment more simple and faster as we are > > losing a lot of time on all these very small modules. > > It also has the benefit of users being able to remove "groups" of > > functionality. > > > > Let me know what you think, if we have consensus I'll draw up a full list > > which can be used for further discussion. > > > > Hans > > >
