Did you mean to name BDB in the second sentence, or is that a limitation of DBM as compared to GDBM. Either way, thanks. I'll start with BDB and compare it to SQLite.
On Fri, 2014-07-11 at 13:07 +0800, Elias Mårtenson wrote: > BDB is much heavier than DBM (of which GDBM is an implementation). DBM > only allows a single process to open the database for writing. > > > I'd like to start with DBM support, as it is simple and efficient, and > also because I have more direct experience with it. BDB is cool > though, and I can add that as a second phase. > > > Regards, > Elias > > > On 11 July 2014 13:04, David B. Lamkins <dlamk...@gmail.com> wrote: > Could you also do BDB? I'd like to try both GDBM and BDB. > > I've quickly skimmed both manuals; it looks like GDBM provides > no > support for transactions or journals. I'd like to be able to > keep that > functionality without having to write it myself (at least in > the initial > implementation). > > SQL is optional to BDB on Linux builds. I can live without the > SQL > support... > > > On Fri, 2014-07-11 at 12:24 +0800, Elias Mårtenson wrote: > > > Would you be willing to move the CF stuff on top of GDBM? > It's a much > > better fit for it, and will give you everything you do with > SQLite > > today, without the extra overhead (SQLite is also not very > efficient > > if you open multiple databases). > > > > > > Regards, > > Elias > > > > > > On 11 July 2014 12:21, David B. Lamkins <dlamk...@gmail.com> > wrote: > > On Thu, 2014-07-10 at 18:52 -0500, Blake McBride > wrote: > > > > > > > > > > > The component file system is for APL-only > applications. The > > databse > > > API is to communicate with the rest of the world. > It > > depends on what > > > you need. > > > > > > > > > and... > > > > > > > > The problem is that you are creating a component > file system > > that > > > _depends_ on features of SQLite and is > _needlessly_ > > dependent on those > > > features. I understand it works on other > databases, but it > > is an > > > unreasonable burden on those systems. You could > just as > > easily create > > > the same system that doesn't depend on particulars > of > > SQLite. That > > > would work everywhere in a way that is reasonable > to any of > > the > > > systems. > > > > > > > > > Ahhh... OK. Perhaps I've had an "aha!" moment. :) > > > > If I understand you correctly, generalizing the > current > > component file > > implementation to work with databases other than > SQLite is > > problematic > > for all of the reasons you've discussed: poor > generality, poor > > use of > > resources, and - in general - a poor fit to SQL. > > > > Therefore, here's my proposal: > > > > I'll drop all further work on generalizing the > current > > component file > > API to work with SQL databases. IOW, the component > file system > > will > > remain - literally - a component *file* system. The > quick > > patch that I > > worked up last night will never become part of my > code. > > > > Programmers who want a way to persist data for an > APL-only > > single-user > > application and a shallow learning curve and no > sysadmin or > > dbadmin > > concerns will (I hope) appreciate and use the simple > > component-file API. > > Programmers who need the functionality of a real > database - > > and all that > > it offers in terms of access control, concurrency > and remote > > access - > > can use lib_sql with a PostgreSQL backend. > > > > The presence of SQLite will remain an incidental > > implementation detail > > of the component file library. All further concerns > specific > > to the use > > of SQLite as an implementation vehicle will be > treated as > > out-of-scope > > for the project. > > > > The current component file code can serve as a > reference > > implementation > > to validate the design choices not implicitly > required by the > > ISO APL > > Annex. There are a number of such choices that I was > forced to > > make in > > the absence of detailed guidance from the spec. For > example: > > the choice > > to ensure that component IDs assigned by CF_APPEND > are always > > nondecreasing for a given component file. There are > many other > > detailed > > choices that had to be made. If you have a concern > with how > > I've > > implemented the API to meet the spec, we can work > out the > > details > > together. > > > > Meanwhile, if someone feels that it's important to > layer a > > component > > file API on top of lib_sql *and* allow a choice of > database > > backends, > > they may either start from scratch or fork the > existing code. > > Personally, I don't see the need for this particular > layering > > of > > abstractions; for anything more complex than an > > integer-indexed file of > > APL objects, I'd prefer to work directly with > lib_sql. But if > > someone > > else sees value in the layered approach, by all > means pursue > > it as a > > separate project. > > > > Meanwhile, I'll continue working on design sketches > for a > > component file > > system that doesn't build upon SQLite. The primary > reason for > > doing so > > will be to increase performance while not requiring > use of > > explicit > > transactions to reduce single-record insert times > below the > > 100ms mark. > > Of course the content of the component file will > need to > > remain reliable > > in the face of software and hardware failures; > that's one of > > the > > implementation attributes that I got "for free" by > choosing > > SQLite as a > > short-term implementation tactic. An alternative > > implementation will be > > challenging, no doubt, but worth the effort in the > long run. > > Whether > > such an implementation should be done as a native > library or > > as a hybrid > > is not clear to me at this time. > > > > The take-away from all of this should be the > following: if you > > use the > > existing component file library (and I certainly > hope that you > > will do > > so in cases where it best serves your needs), you > can be > > assured that > > any future work that I do with the component file > library will > > preserve > > your data. Any change in the file format will be > accompanied > > by a > > utility workspace to migrate your existing component > files. > > > > > > > > This is the most I've been involved with a > community > > project. Fun! > > > > > > > > > Indeed. :) > > > > Also, it's worth once again acknowledging Jüergen's > work in > > making all > > of this possible. All of these little libraries and > utilities > > are > > penny-ante stuff compared to the core interpreter. I > just want > > to say, > > again, that I am grateful for the ability to work > with a > > full-featured > > open-source APL interpreter for the first time ever. > If you > > look at the > > history of free APL systems, they've traditionally > been thin > > on the > > ground, undermaintained and not very interesting. > > > > A lot of technologies have developed or matured in > the past 30 > > years to > > get to the point where we are today with free APL: > Unicode, > > Linux, > > Emacs, gcc, fast hires video and the x86 hardware > platform > > being > > foremost among a plethora of candidates. None of us > would be > > here on > > this mailing list today, however, had Jüergen not > put in an > > immense > > amount of design and implementation effort to create > GNU APL. > > I > > sincerely appreciate his continuing support and > involvement. > > > > > > > > > > >