David, Thanks for the explanation. Your second point (no reason to have the data in 4D) strikes me as more theological than practical, but the rest are certainly excellent reasons. I wasn’t aware that 4D didn’t cope well with thrashing tables, so that’s something I’ll bear in mind in future.
Live and learn. Jeremy Jeremy Roussak j...@mac.com > On 25 Sep 2017, at 19:03, David Adams via 4D_Tech <4d_tech@lists.4d.com> > wrote: > > On Mon, Sep 25, 2017 at 6:06 PM, Jeremy Roussak via 4D_Tech < > 4d_tech@lists.4d.com> wrote: > >> Please forgive me if I’m being dim, but isn’t a solution (maybe not the >> best, but a solution) to maintain the log as records in a table, which is >> periodically emptied into a file by a process which opens, writes and >> closes that file, then deletes the records in the table? >> >> It’s not an approach which I’ve tried, and you may have good reasons for >> rejecting it. I’m just curious to know what they are. >> >> > Jeremy, > > Using a 4D table as a sort of cache for log lines is a perfectly sensible > idea, but also one I'd like to avoid. I did this some years back in a very > high-volume system and ended up killing a lot of performance as a > consequence. (Just ask Justin Leavens.) Now, you could take steps to reduce > the network activity involved, to be sure, but there are still a few points > that come to mind: > > * 4D doesn't do well (or at least hasn't historically done well) with > tables that get thrashed a lot. Add-delete-add-delete. Gets really slow and > horrible. So then you have to do compacts, or hijinks with reusing records, > or all kinds of special magic with never deleting records. Life is too > short for any of that, if at all avoidable. > > * There's no _real_ reason to have the data in 4D at any point. 4D isn't an > analysis platform and it isn't something that I'd use for > high-volume/low-data-content material like log entries. So, there's no good > reason for it to be in 4D other than as a cache. I would consider using a > small table as a cache for storing lines that didn't upload/output to a log > file, but only if the % of failure was pretty low. > > * With a post-to-table-log-and-clear system, you now need a process to poll > the table. You can do this in a way that is relatively inexpensive, but > it's not free. > > * Potentially lots of network traffic for zero gain. > > But there are tons of ways to think about this sort of thing, and lots of > different setups and requirements. My judgment calls and current > constraints are likely different to what other people are dealing with. So, > there are plenty of good arguments for any particular design. I'm just > trying to be clear that while I'm not keen on the approach you suggest in > this case, it's not because I think it's inherently bad - I've used it > before. It's just that in this case I don't want the log data in 4D, if at > all possible. I see that as pure cost with no gain. > > Hopefully, someone will find something silly in what I'm doing as the CALL > WORKER idea is really ver nice. You have n processes doing their thing - > running code, serving Web requests, handling windows, etc. Then they can > send log data (events, behaviors, errors, etc.) over to a central log > worker. Using CALL WORKER, this is quite fast. (4D can stack up tons of > these little requests quite well.) From there, you've got one process > handling all of the log data. A single external file for a particular log > is perfect at this point. It's fast and super easy to use. The whole point > of using a single process for this is to avoid contention on the file > amongst processes. That can otherwise be a *huge* and totally unworthy > bottleneck. So, one process for logging which has the file open in > read-write all the time. So good. Such a standard design. Simple, clean, > cheap, efficient...doesn't work in 4D. Unless there's a bug in my bug > report ;-) ********************************************************************** 4D Internet Users Group (4D iNUG) FAQ: http://lists.4d.com/faqnug.html Archive: http://lists.4d.com/archives.html Options: http://lists.4d.com/mailman/options/4d_tech Unsub: mailto:4d_tech-unsubscr...@lists.4d.com **********************************************************************
