Re: [DISCUSS] Drill 2 and plug-in organisation

2022-01-18 Thread Paul Rogers
Hi James, My experience might be a bit old. I seem to recall, way back when, we did try to build some plugins outside of Drill itself and that there were issues. Maybe it was just the inconvenience of debugging? Perhaps the test libraries were not available? Development is fastest when you can wri

Re: [DISCUSS] Drill 2 and plug-in organisation

2022-01-18 Thread James Turton
For my part, I'd forgotten that GitHub does give users the opportunity to attach binary distributables to releases.  So my first thought of "GitHub would mean using Git repositories to host Jar files" was off the mark. Paul, setting aside the hosting and distribution for a moment, may I ask a

Re: [DISCUSS] Drill 2 and plug-in organisation

2022-01-17 Thread Paul Rogers
Hi Ted, Thanks for the explanation, makes sense. Ideally, the client side would be somewhat agnostic about the repo it pulls from. In a corporate setting, it should pull from the "JFrog Repository" that everyone seems to use (but which I know basically nothing.) Oh, lord, a plugin architecture fo

Re: [DISCUSS] Drill 2 and plug-in organisation

2022-01-17 Thread Ted Dunning
Paul, I understood your suggestion. My point is that publishing to Maven central is a bit of a pain while publishing by posting to Github is nearly painless. In particular, because Github inherently produces a relatively difficult to fake hash for each commit, referring to a dependency using tha

Re: [DISCUSS] Drill 2 and plug-in organisation

2022-01-17 Thread Paul Rogers
Hi Ted, Well said. Just to be clear, I wasn't suggesting that we use Maven-the-build-tool to distribute plugins. Rather, I was simply observing that building a global repo is a bit of a project and asked, "what could we use that already exists?" The Python repo? No. The Ubuntu/RedHat/whatever Linu

Re: [DISCUSS] Drill 2 and plug-in organisation

2022-01-17 Thread Ted Dunning
I don't think that Maven is a forced move just because Drill is in Java. It may be a good move, but it isn't a forgone conclusion. For one thing, the conventions that Maven uses are pretty hard-wired and it may be difficult to have a reliable deny-list of known problematic plugins. Publishing to Ma

Re: [DISCUSS] Drill 2 and plug-in organisation

2022-01-17 Thread James Turton
Thank you Ted and Paul for the feedback.  Since Java is compiled, Maven is probably better fit than GitHub for distribution?  If Drillbits can write to their jars/3rdparty directory then I can imagine Drill gaining the ability to fetch and install plugins itself without too much trouble, at lea

Re: [DISCUSS] Drill 2 and plug-in organisation

2022-01-14 Thread Paul Rogers
Hi All, James raises an important issue, I've noticed that it used to be easy to build and test Drill, now it is a struggle, because of the many odd external dependencies we have introduced. That acts as a big damper on contributions: none of us get paid enough to spend more time fighting builds t

Re: [DISCUSS] Drill 2 and plug-in organisation

2022-01-13 Thread Ted Dunning
The bigger reason for a separate plug-in world is the enhancement of community. I would recommend looking at the Julia community for examples of effective ways to drive plug in structure. At the core, for any pure julia package, you can simply add a package by referring to the github repository w

[DISCUSS] Drill 2 and plug-in organisation

2022-01-13 Thread James Turton
Hello dev community Discussions about reorganising the Drill source code to better position the project to support plug-ins for the "long tail" of weird and wonderful systems and data formats have been coming up here and there for a few months, e.g. in https://github.com/apache/drill/pull/2359