I definitely see the problem here: we don't currently provide a way for something like a Microsoft Excel or PowerBI or Tableau to use ADBC drivers without bundling all of the ones they want to support or requiring/embedding Python or R. I also see how this is a particular problem for Windows and MacOS: on Linux we already provide system packages, although I don't know the extent to which they are used in the wild. I can also see that while it's "easy" to bundle/redistribute the drivers in arrow-adbc for multiple package managers, some proprietary database vendor might not want to do that (and I think that we *do* want proprietary database vendors to distribute ADBC drivers).
I am not sure that the behaviour should be baked into the driver manager itself, but it does seem reasonable to establish a convention, even if that convention is "wherever the system looks for shared libraries" (e.g., Matt's LD_LIBRARY_PATH comment). I can't envision us providing Windows-specific installers for drivers in the arrow-adbc directory, but if somebody else is going to do this it might make sense to establish that convention ourselves. -dewey On Tue, Apr 23, 2024 at 10:50 PM Ian Cook <ianmc...@apache.org> wrote: > > Ha—no, I was thinking of a special ADBC-specific environment variable, > which would work irrespective of the OS. > > On Tue, Apr 23, 2024 at 21:38 Matt Topol <zotthewiz...@gmail.com> wrote: > > > An environment variable like LD_LIBRARY_PATH perhaps? =p > > > > On Tue, Apr 23, 2024, 8:40 PM Ian Cook <ianmc...@apache.org> wrote: > > > > > What if the driver managers respected an environment variable containing > > a > > > delimited list of driver search paths? I think that would get us closer > > to > > > having true system-level configurability while mostly avoiding surprises > > > and inflexibility. > > > > > > Ian > > > > > > On Tue, Apr 23, 2024 at 8:22 PM David Li <lidav...@apache.org> wrote: > > > > > > > I'd rather not hard code it directly into the manager, both because > > this > > > > may surprise applications that don't want it and would be inflexible > > for > > > > applications who are looking to use it, but providing an additional > > list > > > of > > > > search paths that (say) Excel can configure + some platform-specific > > > > guidance on a standard list seems reasonable. > > > > > > > > On Wed, Apr 24, 2024, at 02:45, Ian Cook wrote: > > > > > I wonder if there is a relatively simple way to solve this problem. > > The > > > > > ADBC driver manager libraries already make it possible to dynamically > > > > load > > > > > drivers, and I believe these libraries already allow the user to > > > specify > > > > > which driver to use by passing either a bare filename or a full file > > > > path. > > > > > > > > > > So perhaps we could simply establish an ordered list of standard > > > > directory > > > > > locations in which the ADBC driver manager will look for drivers when > > > > they > > > > > are specified by bare filename. We would have to specify this > > > differently > > > > > for each mainstream type of OS, but I think that is doable. This > > could > > > be > > > > > codified in the ADBC docs and implemented in the ADBC driver > > managers. > > > > > Anyone looking to achieve system-wide ADBC driver "registration" > > could > > > > take > > > > > advantage of this, whereas anyone who prefers application-specific > > > > > implementation could safely ignore it. > > > > > > > > > > I suspect that we would want the driver manager to look first in > > > > > application-specific directories (which might vary depending on which > > > > ADBC > > > > > driver language library one is using), then fall back on user-level > > > > config > > > > > directories, then finally fall back on system-level config > > directories. > > > > > > > > > > I believe that Windows, macOS, and Linux distros all have standard > > > > > user-level and system-level config directories that are often used > > for > > > > this > > > > > type of thing. > > > > > > > > > > Does this seem reasonable? Are there any gotchas that would prevent > > an > > > > > approach like this from working? > > > > > > > > > > Ian > > > > > > > > > > On Mon, Apr 1, 2024 at 5:44 PM Curt Hagenlocher < > > c...@hagenlocher.org> > > > > > wrote: > > > > > > > > > >> The advantage to system-wide registration of drivers (however that's > > > > >> accomplished) is of course that it allows driver authors to provide > > a > > > > >> single installer or set of instructions for the driver to be > > installed > > > > >> without regard for different usage scenarios. So if Tableau and > > Excel > > > > can > > > > >> both use ODBC drivers, then I (as a hypothetical author of a niche > > > > driver) > > > > >> don't have to solve N installation problems for N possible use > > cases. > > > > And > > > > >> my spouse (as a non-developer finance user) can just run one > > installer > > > > and > > > > >> know that the data source will be available in multiple tools. Or at > > > > least > > > > >> that's the principle. > > > > >> > > > > >> For a real-world example, compare the instructions for installing > > ODBC > > > > >> drivers into Tableau ( > > > > >> > > > > >> > > > > > > > > > https://help.tableau.com/current/pro/desktop/en-us/examples_otherdatabases.htm > > > > >> ) with those for installing JDBC drivers ( > > > > >> > > > > >> > > > > > > > > > https://help.tableau.com/current/pro/desktop/en-us/examples_otherdatabases_jdbc.htm > > > > >> ). The JDBC instructions include copying or installing files to a > > > > specific > > > > >> directory which possibly needs to be created. The ODBC instructions > > > ... > > > > >> don't. > > > > >> > > > > >> With what I'm most immediately invested in -- database drivers for > > > > >> Microsoft Power BI -- part of the problem actually ends up being > > that > > > > many > > > > >> drivers are closed source and/or not freely redistributable. So for > > > > someone > > > > >> to use Power BI with Oracle, they either need a way to install > > Oracle > > > > >> drivers onto their machine in a standard way which lets us find them > > > or > > > > we > > > > >> need to go through a painful and sometimes expensive "biz dev" > > effort > > > to > > > > >> get the right to redistribute those drivers and install them > > > ourselves. > > > > >> > > > > >> I am of course aware that there can also be significant downsides to > > > > such > > > > >> system-wide registration. > > > > >> > > > > >> -Curt > > > > >> > > > > >> On Wed, Mar 20, 2024 at 7:23 AM Antoine Pitrou <anto...@python.org> > > > > wrote: > > > > >> > > > > >> > > > > > >> > Also, with ADBC driver implementations currently in flux (none of > > > them > > > > >> > has reached the "stable" status in > > > > >> > https://arrow.apache.org/adbc/main/driver/status.html), it might > > > be a > > > > >> > disservice to users to implicitly fetch drivers from potentially > > > > >> > outdated DLLs on the current system. > > > > >> > > > > > >> > Regards > > > > >> > > > > > >> > Antoine. > > > > >> > > > > > >> > > > > > >> > Le 20/03/2024 à 15:08, Matt Topol a écrit : > > > > >> > >> it seems like the current driver manager work has been largely > > > > >> targeting > > > > >> > > an app-specific implementation. > > > > >> > > > > > > >> > > Yup, that was the intention. So far discussions of ADBC having a > > > > >> > > system-wide driver registration paradigm like ODBC have mostly > > > been > > > > to > > > > >> > > discuss how much we dislike that paradigm and would prefer ADBC > > to > > > > stay > > > > >> > > with the app-specific approach that we currently have. :) > > > > >> > > > > > > >> > > As of yet, no one has requested such a paradigm so the > > discussions > > > > >> > haven't > > > > >> > > gotten revived. > > > > >> > > > > > > >> > > On Wed, Mar 20, 2024 at 9:22 AM David Coe < > > > david....@microsoft.com > > > > >> > .invalid> > > > > >> > > wrote: > > > > >> > > > > > > >> > >> ODBC has different OS-level driver managers available on their > > > > >> > respective > > > > >> > >> systems. It seems like the current driver manager< > > > > >> > >> https://arrow.apache.org/adbc/main/cpp/driver_manager.html> > > work > > > > has > > > > >> > been > > > > >> > >> largely targeting an app-specific implementation. Have there > > been > > > > any > > > > >> > >> discussions of ADBC having a similar system-wide driver > > > > registration > > > > >> > >> paradigm like ODBC does? > > > > >> > >> > > > > >> > > > > > > >> > > > > > >> > > > > > > > > >
