> On Sun, May 7, 2017 at 1:29 PM, Keisuke Miyako via 4D_Tech <
4d_tech@lists.4d.com> wrote:
Hey, thanks for writing! I think the world of your contributions and work,
but we disagree pretty completely on this. I'll offer my thoughts and point
of view on all of this and, hopefully, you can give me a new way of looking
things. (You did manage to shift my thinking on ON ERR CALL in general
recently.) And if anyone else has real-world comments or observations, I'm
all in on hearing them, even if they make me change my opinion. My opinions
are *always* right, I just try to change them a lot ;-)
> is "code injection" even a problem?
Absolutely.
> preemptive mode is all about multi-tasking.
If by that you mean "taking advantage of idle cores", then I agree.
> I would expect minimal communication between those tasks,
> literally, they should be "preemptive".
> it's true you can start a preemptive process with CALL WORKER,
> but that is not required,
> you can also use New process or Execute on server.
> it's totally fine to have a runaway preemptive task.
Good point, thanks for pointing it out. I wasn't thinking about New
process/Execute on server, excellent point.
Just to make it clear, I have very little interest in preemptive
processing. I expect it to be a powerful and useful technique in a *very*
narrow range of problems. To justify the effort of using preemptive
threads, you need a task that is 1) taking a long time, 2) CPU-bound, and
3) doesn't require so much time to prepare and marshall that you don't get
much speed gain, and 4) has a sufficiently large non-cooperative component
to make it work the trouble. That's a lot of hurdles to clear and,
honestly, I don't have a single task in day-to-day use that makes it clear
of all of them. I've worked on systems (and am likely to work on systems
again soon) where such tasks exists, but they're not so commonplace.
> workers, on the other hand, is all about messaging.
Sadly, we *don't* have messages. We have blocks of code that are run.
That's not messaging. It only seems like messaging *if you're 4D.* As a 4D
developer, there's no queue, there's no message. If you're inside of 4D
itself, you can see a queue, etc. But as a 4D developer? You can't. All we
have is a pipe that things pop out of one-at-a-time and then execute. We
don't know if anything is in the pipe, we don't know when something
arrives, we don't have any ability to view/screen/filter/reject/trap that.
There's no event, nothing. Code is being tacked onto the end of a thread
and run blindly. It's an inherently dangerous design, but it _can_ be used
well. It just isn't built to run safely out of the box. Don't get me wrong,
I like the feature - I just don't use it without quite a bit of scaffolding
in place to minimize and/or eliminate risks.
As a practical example a very common, obvious, and good use of a preemptive
thread with CALL PROCESS, consider a log writer. It's a perfect setup.
Let's say you're logging errors/tasks/event/requests/whatever in 8
processes. Should each have its own log file? If not, it's a huge,
time-sucking waste to manage the contention between processes fighting for
a lock on the file. So, the better solution is to pipe all log entries to a
single process for writing. And since writing to disk is a relatively slow
operation, often of low value for logs, why suck up time on the main CPU?
CALL WORKER("LogWriter_Errors";"LogWriter_Start";"Errors")
LogWriter_Errors opens/appends/whatever a log file on disk *and keeps it
open*. Then any number of processes can stream log lines in for writing:
CALL WORKER("LogWriter_Errors";"LogWriter_Write";$error_details)
LogWriter_Errors catches the line, does a SEND PACKET and you're done and
dusted. You could set it up to handle different types of logs (say have
errors and requests both open), use different workers for different types
of logs, etc. None of the data has to go through 4D's data file, you've
just got a pass-through to disk. Sweet. Speed improvement over fighting
over the log file, etc.? 20x is *easy*. That's a huge optimization -
largely because of the design, the preemptive part - I don't even know how
much that helps. But the critical detail is that *the log file is opened
once at the start of the worker and closed only when the worker finishes.*
If that fails, you've got a big fat problem. (As it turns out, 4D currently
has a bug that makes race conditions on files super easy to create. It's in
the bug system somewhere but 16.2 and 16R2 make no difference.) So, yeah,
CALL WORKER is 100% appropriate, a preemptive worker is 100%
appropriate...and the ability to break the whole think with a simple
mistake makes it too dangerous to use.
In fact, that's my conclusion about preemptive mode now: Not ready for
prime time. My hat is off to 4D for trying on this as making a language
multi-threaded isn't a small thing. But I've found it just way too easy to
break so far. (I've tried a few times and ran into problems quickly in each
case.)
I expect that you *can* use preemptive mode today for specific tasks quite
successfully and to great benefit. I'd just be super careful, pick
well-defined stand-alone tasks, and be darn sure you make zero mistakes.
**********************************************************************
4D Internet Users Group (4D iNUG)
FAQ: http://lists.4d.com/faqnug.html
Archive: http://lists.4d.com/archives.html
Options: http://lists.4d.com/mailman/options/4d_tech
Unsub: mailto:4d_tech-unsubscr...@lists.4d.com
**********************************************************************
