On Thu, Sep 14, 2017 at 9:56 AM, Tim Nevels via 4D_Tech < 4d_tech@lists.4d.com> wrote:
> Wow… this turned out to be a lot longer post than I expected. But it might be helpful to less experienced 4D developers. Dude, you've been hiding your light under a bushel...you should write long posts more often! Great summary of options for improving/managing stored procedures. I just want to add or emphasize a couple of things about pre-emptive (thread safe/multi-core) processes. First though, a caveat: I don't know what I'm talking about. I mean that what I'm going to say is not well field-tested in 4D and that I would _very_ much appreciate hearing from anyone that has real-world experience running pre-emptive threads in 4D. Okay, now the points: * You don't need to use workers for pre-emptive threads, you can use Execute on server or New process too. * You can't use plug-ins at all. Not sure when/if that will change, but it's a constraint we have to live with. Someone at 4D or Rob L. could probably give us a guess if there will ever be a chance for support for plug-ins in thread-safe processes. * The original source code for 4DIC is super, super, super old. Not sure how much it's changed down the years but, yeah.... * You can't use LAUNCH EXTERNAL PROCESS at all. Not sure when/if that will change, but it's a constraint we have to live with. (I doubt that this one will every change, but what do I know.) So, how _can_ we safely communicate with the outside world from a thread-safe process? I can think of at least four ways: 1) The database itself. 2) External files. 3) CALL FORM / CALL WORKER. 4) The HTTP Get and HTTP Request commands. Comments below. --------------------- The database itself --------------------- Yes, absolutely, this is a great way to go. --------------------- External files --------------------- Hmmm. Well, it's got a lot more merit in theory than you might think, but there are problems. As I've mentioned before, I accidentally found a fatal bug using CALL WORKER and a worker to stream log data to a worker that would in turn stream the data to disk. Bog standard log processing design. Kills 4D (stacks up redundant processes on Mac until you have to quit, crashes Win) - in my case in about a minute. I sent in a sample database to illustrate the bug (and some crazy runtime errors to match in the Runtime Explorer) during V16.0. I've retested in .1 and 2 and I think through R3 on that branch. Still not working. For the record, this is true of cooperative and pre-emptive threads...although they behave a bit differently. The issue seems to be around a file lock not being released properly and so you get an "impossible" race condition. So I don't trust external file support for the moment. Bummer. ------------------------ CALL FORM / CALL WORKER ------------------------ I guess. I like the commands, but they're not designed to run without losing messages when you kill a worker or shut down. That can be fine, or it can make them a non-starter, it depends on what you're doing. I think that instead of pushing messages at a pre-emptive process, I would have a pre-emptive thread call out for tasks or what have you. Kind of an instinct, I'd appreciate hearing other people's real-world results - good or bad. ------------------------ HTTP Get / HTTP Request ------------------------ Right, this looks like a very promising option. Welsh Harris tells me that he's been using these commands successfully in pre-emptive threads. It sure looks like 4D went to trouble to make them work in pre-emptive mode, so good on them. The thing about these commands is that they give you a whole lot more than the ability to call out for a Web page, they give you the ability to communicate with *any* kind of service that allows for messaging over HTTP. Frankly, that's nearly everything these days. You can push/pull data from mail servers, logging services, etc. this way. It isn't that hard to find intermediary services to do that kind of service proxying for you. Partly this is because 80/443 tend to be open on all networks, so you find many clever services build to run over them....and now you see another reason I'm so intent on having a full JSON parser natively in 4D...can't use NTK's parser in a pre-emptive process. Oh. Yeah, a lack of native JSON parsing can make this HTTP-based approach a non-starter in some cases. Also a few comments about pre-emptive processing generally: * They're not magic. * It looks really challenging to implement, so well done 4D for getting even this far. * It sounds like a bunch of the limits 4D imposes come from the OS level. So, it's not 4D, it's reality. I feel like giving them the benefit of the doubt on this for sure. * Doesn't support dot notation someone said the other day? I'd like to know the story on that, maybe they're using an external library that isn't thread-safe for their parsing? * Many tasks can't readily or productively be parallelized. Nature of things. * The easiest (pure math) parallelizations are already being done for you...in your GPU. * The limiting factor in parallelization is the portion of the job that are inherently unparallelizable. Apart from made-up examples and tasks better sent to the GPU, most jobs have components that have to be done in a specific sequence and therefore can't be split up. http://tutorials.jenkov.com/java-concurrency/amdahls-law.html * There are also a lot of cases where there's extra complexity and time involved in dividing up a job and then gathering up the results. * With the previous point in mind, pick tasks that stand alone and don't really need to interact with other processes to work out timing. No messaging at all to get the job done? Perfect. * Want more cores? Use more machines. 4D has this beautiful little tool called 4D Engine. You can build up special versions of your app that can do all sorts of stuff on a different box - they can even use pre-emptive threads (Remote can't). Use HTTP to call back to your server for work or with results. This architecture is pretty awesome and, after you get over the hassle of setting up the HTTP communications layer, makes it _way_ easier to keep a large system up and running. Have to restart the server? The stand-alones can just wait until it's back, you don't have to restart them. So good. I'm planning to try out a log writer design shortly that will run a pre-emptive worker that receives CALL WORKER blocks with log data that will then be forwarded over HTTP to a cloud-based log service, so I may get some better feel for real-world results. ********************************************************************** 4D Internet Users Group (4D iNUG) FAQ: http://lists.4d.com/faqnug.html Archive: http://lists.4d.com/archives.html Options: http://lists.4d.com/mailman/options/4d_tech Unsub: mailto:4d_tech-unsubscr...@lists.4d.com **********************************************************************
