On Jun 5, 2018, at 1:04 PM, Stephen Chrzanowski <pontia...@gmail.com> wrote: > > None of these applications describe a hook for my Delphi code
This feels like an instance of the XY problem: https://meta.stackexchange.com/questions/66377/what-is-the-xy-problem You’re focused on your predetermined solution and are rejecting a new idea because it doesn’t fit your preconception of the needed solution. Dan’s suggestion solves your stated problem, but it will be more work than integrating one of the distributed SQLite extensions I pointed you to. The question then is, will these distributed SQLite variants solve your *actual* problem, or are you going to be forced to DIY? I do not rule out that possibility in any of the commentary below. I just want to make sure you consider all of the facets before starting work. > my application would hook into through an external call ranging from a > socket (Needs to be online) or make a shell call to do work. To top it > off, most of what was proposed does not have a 'native' Win32 interface but > relies on getting a completely different language to play in my game. Bedrock runs out-of-process relative to your Delphi application. Your app would start and stop the local instance, and it would communicate with it over a localhost HTTP connection. You don’t need shell calls, and the wish for a native Windows calling interface is a distraction, as we will see below. (You’ll see why I’ve dropped consideration of dqlite and rqlite later on.) You can choose to make the lifetime of Bedrock’s DB server roughly identical with that of your app, or you could instead allow the DB daemon to run in the background after your app quits so that it can continue to establish and maintain synchronization with the master server. When deciding, keep in mind that other clients on other computers might also be updating the master while your local app instance is down. You might prefer to let the local Bedrock instance run in the background to pull those updates down so it doesn’t delay your local app’s next launch. To communicate with the local Bedrock daemon, you’d use the HTTP client feature that I assume must be built into Delphi’s standard library these days. This should be no more difficult to program to than any other DB access API. You’d configure the local Bedrock instance as a slave, pointing it at the central server configured as a master, and let the two sync up as and when the connection between them appears and disappears. This is one of the advantages of running this service out of process, in the background. > - The machine I'm working on will need to be always online I do not believe that to be true. All of these distributed SQLite variants should cope with temporary connection losses. If not, they’d be useless in practice, because network failure is a thing! Indeed, it’s pretty much the central problem these offerings solve. The only thing I worry about is that there may be default timeouts based on assumptions that are incorrect for your application. If so, fixing that would be much easier than either implementing proper distributed consensus protocols or reinventing them poorly: https://en.wikipedia.org/wiki/Paxos_(computer_science) https://en.wikipedia.org/wiki/Raft_(computer_science) Synchronization is a Hard Problem®. You really really really want to leave it to the experts, if you can. > Sometimes, if I'm remote, no > cell signal, no Wifi, I'm skunked on this requirement. I believe when the connection comes back, Bedrock will automatically sync up with the master. Distributed synchronization is a field of expertise all on its own. The history of computing is littered with the shells of products and even companies that did this poorly. > additional 'stuff' to be installed on the client machine to go forward, That’s what your application’s installer is for. Or were you hoping to distribute a single statically linked executable? Bedrock is open source, so it may be possible to repackage it as a static library exposing a C interface, but I think the background synchronization feature is worth leaving it out-of-process. > - requires at least 3 machines to maintain a quorum There are strong theoretical reasons to require a minimum of 3 machines, basically being that that is the smallest number that allows you to form a majority, without which you cannot achieve Consensus, that being one of the three-choose-two options from the CAP theorem: https://en.wikipedia.org/wiki/CAP_theorem If you don’t need “C”, then you’re in the AP regime: you prefer to have availability and partition tolerance instead. Availability means your local DB is always there, and partition tolerance means you can live without being able to talk to the master for arbitrarily long amounts of time. The Raft protocol is strictly CP, so that rules out dqlite and rqlite if you were wanting an AP solution instead. Bedrock, however, has the ASYNC consistency mode: https://github.com/Expensify/Bedrock/blob/master/sqlitecluster/SQLiteNode.cpp#L718 This sets the transaction to allow eventual consistency, meaning that no other peer must concur with the transaction for it to go through. Thus, the ASYNC mode gives you an AP solution. Bedrock also supports a CP solution with its ONE mode and 2-3 peers. You could choose to shift into CP mode if your application detects that the remote master is currently reachable, then shift back to AP mode when the connection drops. This means setting up two cloud masters, each in a separate data center, geographically-separated. That would get you protection against one of the masters being temporarily unreachable even though the Internet connection is up. > ...needs to make…socket connections, or some other funky jazz. You were already going to have to do network I/O in your proposed hand-rolled synchronization mechanism. What’s the practical difference between communicating between two Windows EXEs over a custom protocol vs. communicating with the local distributed DB instance over HTTP and letting it maintain synchronization with the master in the background? If anything, the localhost-only HTTP communication will be far easier, being a standard protocol over a reliable network with a battle-tested implementation, which is presumably provided by your language’s standard library. > - has reliance on languages other than the two languages I'm dealing with > (Pascal and SQL) Bedrock accepts SQL over its HTTP interface. If you’re using a library-provided HTTP implementation, you may need to deal very little with the details of the HTTP protocol, depending on how much it hides from you. > - way overboard and way over complicated for a very simple and small > exchange of data. If you hand-roll it, you will certainly miss subtleties which will result in you hitting well-known hard problems in distributed systems. > - requires a 3rd party application to sneak-a-peak at my data, which > introduces problems where that 3rd party app does something I don't want or > don't expect. That’s also true of SQLite, Delphi, Windows, the network interface’s and HDD drivers, the mainboard firmware, and the processor’s microcode. > I don't think I'm reinventing the wheel. Sure, conceptually what I'm doing > has already been done, but, I've got the desire to save on tire wear and go > down the path that gets me to where I want without having to dodge > roadblocks. Going around the immovable roadblocks will just send you out through the boonies where your tires will be punctured, your car sunk in mud, and the skeetos that show up on the ATC radars will eat you alive. Bring sunblock. :) _______________________________________________ sqlite-users mailing list sqlite-users@mailinglists.sqlite.org http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users
