On Mon, May 8, 2017 at 11:03 AM, Keisuke Miyako via 4D_Tech < 4d_tech@lists.4d.com> wrote:
Another great post, thanks. I really like the way you've broken out the various scenarios, how they should behave and how you've observed them behaving. Just to make it really clear, all I'm talking about is raising errors consistently and not killing/restarting workers as a side-effect. I in no way have _any_ problem with _any_ of 4D's restrictions on preemptive processing. None. I looked into it a bit, and it's no simple task to make a language run on multiple cores. So, they've got my complete respect on that. Likewise, some of the restrictions in the language are imposed by the underlying OS. So, it's reality imposing a constraint, not some arbitrary choice by 4D. Likewise, I appreciate what the compiler does for detecting errors in advance. I'm always all in on the compiler doing more and, in this case, it's a good effort. But I am concerned (as you figured) about what comes up as #3 in your taxonomy: "CALL WORKER is used to "run cooperative code in an existing preemptive process" Personally, I think that 4D should *not* run the code. It should throw an error that we can catch. Every time. It's not 4D's bug, it's my bug. But 4D needs to tell me about it. This is very realistically the sort of flaw that could creep into a system of almost any level of complexity from simple on up. Bad call: My fault Scolding me: 4D's job > and you can invoke that method by passing an empty string. > http://doc.4d.com/4Dv16R2/4D/16-R2.1620/CALL-WORKER.301-3111543.en.html > if we followed that rule and always passed an empty string, we would never have this problem. just saying... I guess I haven't said clearly, but I'm not primarily thinking of myself. There is zero chance I'll start tossing unregulated code around for execution. That's just stupid/insane/unsafe. It's a weak design, although I can see why 4D did it. I've got a quite involved system of message passing I'm working on that goes like this: CallMyMagicThingy($object;{$result_callback_object}) Any method that is going to get a callback has one parameter: C_OBJECT($1;$message) That's it. There are formalisms and validators for the contents of the message. (A header with routing details and action flags, a message payload, and so on.) This, in fact, is was arguing for comprehensive JSON scanning features a few months back. Without that, you can't write any sort of generic validation code. Instead, you need a lot of code that embeds the rules in the code. Ugh. I'd rather build my own meta-data/schema to generate, validate, document, and test the formats. Can't. So, I scaled back the system to code with only much simpler formats and had to write individual validators. Wasteful and disappointing, but the native command set doesn't support anything more. Anyway. > now your feature request, if I understand correctly, talks about detecting, on the receiving end, > if a request to execute cooperative code has been received in a preemptive process. > I think it is not a problem, since there is nothing to receive since CALL WORKER does make such requests. I'm not parsing out what you're saying here, so I'm probably missing your point. What I'm asking for is that the *recipient* throws an error. The worker throws an error that I can catch. From there, I've got a lot of options as how to proceed. There's no call-response with CALL WORKER so there is not automatic way to report back the flaw to the code that caused the flaw. But that's another matter: C_BOOLEN($okay) $okay:=Call worker("PreemptiveWorker";"BadCode") If($okay=false) ALERT("Aiuto! Something went wrong!") End if I could go on about alternatives here all day, but that's not the point. At the bare minimum, the preemptive worker should not terminate and should report an error. > * I don't get an error. > this must be CALL WORKER asked to do something impossible. (issue #3 above) Could be. Hard to know. Impossible to count on. The docs don't say. It has been drilled into us for decades that when something is not in the docs *it is not a reliable behavior.* There is absolutely no way to distinguish between "Oh, yeah, we just didn't put that in the docs, good idea" from "that's an undocumented behavior, don't count on it." They look the same from the outside since there's nothing to look at! > * The worker may be restarted in cooperative mode. Which may end up clearing existing variables and so on (?) > the worker must have been killed or aborted. Yes, exactly - but not by my code - by 4D. This is a bug or design flaw. It's not something I think is easy to defend as a good idea. "Oh, we will keep this process running until you KILL it or quit. And that's a guarantee! (+)" (+) Not an actual guarantee, see the back of your ticket. > * Other people report getting an error. > some "eval" code must have been executed in the worker (issue #2 above) Yeah, I'll buy that. > * Sometimes I get an error. > * Sometimes an error handler catches, sometimes it doesn't. > a mix of all the other problems, just presenting themselves in random order? I cant' say for sure as I haven't studied it systemically enough. It was enough of a dog's breakfast that it struck me as a waste of my time to try to dig in. (I've poured hundreds of hours into CALL WORKER and CALL FORM already, but I draw the line at features that just don't look like they should have come out of the oven yet.) There's no promised behavior from 4D about errors in case #3, it's easy to get weird results, and the side-effects are potentially catastrophic. So, yeah, I'm not seeing putting a bunch more time into it as smart move for me. I really appreciate you offering a solid, well thought-out technical answer. You've given me some food for thought and I think given everyone a broader perspective on the system. I want to emphasize again that I like CALL WORKER a lot. It's easy to use and has a classic good 4D economy. Big power, small package. But, it needs to be used correctly. Personally, I'd much prefer a true messaging system rather than remote and unregulated code execution scheme. That's a great approach (and time tested), and would make all of the following better: * None of the nonesense I've been complaining about. * Saving the queue for later processing. * Load-balancing in real-time. * Message distribution and routing. Easily. * Message security policies. * Message/service actions that can be extended to any JSON-capable system out there. Very easy. * Publish/subscribe: Easy. * Networked: Easy MQs (Message Queues) come out of the middleware world. It's always been all about interchange amongst heterogeneous systems. That's one reason messages are normally passed as, well, messages. Passing code locks your system into a specific interpreter. Personally, I don't care about preemptive much. I've noticed it seems to be the new "For loops are faster" and that people get pretty excited. I figured that I should check it out. I expect it to be suitable for only a small range of problems. (There's a reason all of the demos show a bunch of random calculations in a loop...it's because it's hard to find real-world examples of discrete, time-consuming, CPU-bound, low-dependency tasks.) When I have a suitable problem, I'll do a very, very, very specific and tight implementation. Using the feature casually in this context could end in tears too easily. ********************************************************************** 4D Internet Users Group (4D iNUG) FAQ: http://lists.4d.com/faqnug.html Archive: http://lists.4d.com/archives.html Options: http://lists.4d.com/mailman/options/4d_tech Unsub: mailto:4d_tech-unsubscr...@lists.4d.com **********************************************************************
