I'm certainly willing to give GDBM a try as a replacement for SQLite. Does SQLite bog down with multiple databases? Any idea why? A global lock, perhaps...?
On Fri, 2014-07-11 at 12:24 +0800, Elias Mårtenson wrote: > Would you be willing to move the CF stuff on top of GDBM? It's a much > better fit for it, and will give you everything you do with SQLite > today, without the extra overhead (SQLite is also not very efficient > if you open multiple databases). > > > Regards, > Elias > > > On 11 July 2014 12:21, David B. Lamkins <dlamk...@gmail.com> wrote: > On Thu, 2014-07-10 at 18:52 -0500, Blake McBride wrote: > > > > > > > The component file system is for APL-only applications. The > databse > > API is to communicate with the rest of the world. It > depends on what > > you need. > > > > > and... > > > > > The problem is that you are creating a component file system > that > > _depends_ on features of SQLite and is _needlessly_ > dependent on those > > features. I understand it works on other databases, but it > is an > > unreasonable burden on those systems. You could just as > easily create > > the same system that doesn't depend on particulars of > SQLite. That > > would work everywhere in a way that is reasonable to any of > the > > systems. > > > > > Ahhh... OK. Perhaps I've had an "aha!" moment. :) > > If I understand you correctly, generalizing the current > component file > implementation to work with databases other than SQLite is > problematic > for all of the reasons you've discussed: poor generality, poor > use of > resources, and - in general - a poor fit to SQL. > > Therefore, here's my proposal: > > I'll drop all further work on generalizing the current > component file > API to work with SQL databases. IOW, the component file system > will > remain - literally - a component *file* system. The quick > patch that I > worked up last night will never become part of my code. > > Programmers who want a way to persist data for an APL-only > single-user > application and a shallow learning curve and no sysadmin or > dbadmin > concerns will (I hope) appreciate and use the simple > component-file API. > Programmers who need the functionality of a real database - > and all that > it offers in terms of access control, concurrency and remote > access - > can use lib_sql with a PostgreSQL backend. > > The presence of SQLite will remain an incidental > implementation detail > of the component file library. All further concerns specific > to the use > of SQLite as an implementation vehicle will be treated as > out-of-scope > for the project. > > The current component file code can serve as a reference > implementation > to validate the design choices not implicitly required by the > ISO APL > Annex. There are a number of such choices that I was forced to > make in > the absence of detailed guidance from the spec. For example: > the choice > to ensure that component IDs assigned by CF_APPEND are always > nondecreasing for a given component file. There are many other > detailed > choices that had to be made. If you have a concern with how > I've > implemented the API to meet the spec, we can work out the > details > together. > > Meanwhile, if someone feels that it's important to layer a > component > file API on top of lib_sql *and* allow a choice of database > backends, > they may either start from scratch or fork the existing code. > Personally, I don't see the need for this particular layering > of > abstractions; for anything more complex than an > integer-indexed file of > APL objects, I'd prefer to work directly with lib_sql. But if > someone > else sees value in the layered approach, by all means pursue > it as a > separate project. > > Meanwhile, I'll continue working on design sketches for a > component file > system that doesn't build upon SQLite. The primary reason for > doing so > will be to increase performance while not requiring use of > explicit > transactions to reduce single-record insert times below the > 100ms mark. > Of course the content of the component file will need to > remain reliable > in the face of software and hardware failures; that's one of > the > implementation attributes that I got "for free" by choosing > SQLite as a > short-term implementation tactic. An alternative > implementation will be > challenging, no doubt, but worth the effort in the long run. > Whether > such an implementation should be done as a native library or > as a hybrid > is not clear to me at this time. > > The take-away from all of this should be the following: if you > use the > existing component file library (and I certainly hope that you > will do > so in cases where it best serves your needs), you can be > assured that > any future work that I do with the component file library will > preserve > your data. Any change in the file format will be accompanied > by a > utility workspace to migrate your existing component files. > > > > > This is the most I've been involved with a community > project. Fun! > > > > > Indeed. :) > > Also, it's worth once again acknowledging Jüergen's work in > making all > of this possible. All of these little libraries and utilities > are > penny-ante stuff compared to the core interpreter. I just want > to say, > again, that I am grateful for the ability to work with a > full-featured > open-source APL interpreter for the first time ever. If you > look at the > history of free APL systems, they've traditionally been thin > on the > ground, undermaintained and not very interesting. > > A lot of technologies have developed or matured in the past 30 > years to > get to the point where we are today with free APL: Unicode, > Linux, > Emacs, gcc, fast hires video and the x86 hardware platform > being > foremost among a plethora of candidates. None of us would be > here on > this mailing list today, however, had Jüergen not put in an > immense > amount of design and implementation effort to create GNU APL. > I > sincerely appreciate his continuing support and involvement. > > >