Re: Preemptive mode flaw

[David Adams via 4D_Tech]

On Sun, May 7, 2017 at 3:07 PM, Keisuke Miyako via 4D_Tech <
4d_tech@lists.4d.com> wrote:

> first up, I myself am still in the process of learning how workers work,
> and I totally agree one should probably start with small POCs or
> relatively minor tasks before going full on.
>

I think many of us here will be very interested in how you end up using
workers, so please share when you find applications that you like!


> keep in mind a CALL WORKER can create a new worker if the previous one was
> killed.
> for instance. imagine a recursive worker method, using process variables
> for context,
> aborted for whatever reason from the debugger or runtime explorer.
> the worker is killed, but a new "call" comes in from the previous context,
> starting a new worker that has no idea where the previous worker left off.
> that can be a nightmare.
>

Yes, exactly! There is no On Worker Start or On Worker Stop event - so it's
*very* important that they not be terminated accidentally.

For what it's worth, I've been working with workers, call message, and call
form intensively for a few months and am pretty deep into the details of
the mechanics. I like the commands and I think 4D did an admirable job of
providing power without complexity. I never would have come up with the
design they did as there are a lot of standard queuing features I would
never have considered dropping the way they did. I've tried to be open
minded about their design choices. But the more I get into it, the more I'm
seeing how different designs wouldn't be much more complex (if any more
complex) without taking away any of the power.

As far as preemptive mode goes, I think that CALL WORKER and preemptive
processing are wedded in people's minds because that's how 4D has been
presenting the subjects. I guess that they added workers to solve the
problem of dropping IP variables for IP communication. I didn't use IP
variables that way (too hard and expensive to use safely, too dangerous and
messy to use any other way)...I just use them for pseudo-constants/cached
items. So, CALL WORKER didn't solve a problem I had...but I love message
queues and have been trying to convert CALL WORKER into an actual message
queue system. (Callbacks and Publish/Subscribe.)

keep in mind a client process can never be preemptive.
>

Yes, I've noticed that. I don't know what the reasoning is there or if it
will change. It's just server-side processes, in compiled mode, using legal
commands, on a multi-core machine, where you've launched the process in
preemptive mode. Whew! Or stand-alone...
**
4D Internet Users Group (4D iNUG)
FAQ:  http://lists.4d.com/faqnug.html
Archive:  http://lists.4d.com/archives.html
Options: http://lists.4d.com/mailman/options/4d_tech
Unsub:  mailto:4d_tech-unsubscr...@lists.4d.com
**

Re: Preemptive mode flaw

[Keisuke Miyako via 4D_Tech]

first up, I myself am still in the process of learning how workers work,
and I totally agree one should probably start with small POCs or relatively 
minor tasks before going full on.

what I think I leaned so far:

keep in mind a CALL WORKER can create a new worker if the previous one was 
killed.
for instance. imagine a recursive worker method, using process variables for 
context,
aborted for whatever reason from the debugger or runtime explorer.
the worker is killed, but a new "call" comes in from the previous context,
starting a new worker that has no idea where the previous worker left off.
that can be a nightmare.

keep in mind a client process can never be preemptive.
most (if not all) published examples are designed for single user,
where some background processes are spawned simultaneously and report to a UI 
process.
they explain how workers work,
but in client server all the spawned workers will be cooperative,
sharing the same core as the UI process,
not really a show care for preemptive multi-processing.

2017/05/07 12:59、David Adams via 4D_Tech 
<4d_tech@lists.4d.com> のメール：
So, yeah, CALL WORKER is 100% appropriate, a preemptive worker is 100%
appropriate...and the ability to break the who



**
4D Internet Users Group (4D iNUG)
FAQ:  http://lists.4d.com/faqnug.html
Archive:  http://lists.4d.com/archives.html
Options: http://lists.4d.com/mailman/options/4d_tech
Unsub:  mailto:4d_tech-unsubscr...@lists.4d.com
**

Re: Tip: Code to get all thread-safe commands

[David Adams via 4D_Tech]

On Sun, May 7, 2017 at 1:10 PM, Keisuke Miyako via 4D_Tech <
4d_tech@lists.4d.com> wrote:

> also check out the "How Do I?" example
>
> http://blog.4d.com/preemptive-processes-for-4d-language/
> http://download.4d.com//Demos/4D_v15_R5/HDI_Command_name.zip


Hey, that's sweet! Thanks for the link. I really should read the blog more
carefully and check out more demos, there's lots of great stuff.


> another important piece of the puzzle is PROCESS PROPERTIES
> http://doc.4d.com/4Dv16R2/4D/16-R2.1620/PROCESS-PROPERTIES.
> 301-3111842.en.html


Thanks, and I always appreciate the examples that deal with the bit math
for me. Because lazy.
**
4D Internet Users Group (4D iNUG)
FAQ:  http://lists.4d.com/faqnug.html
Archive:  http://lists.4d.com/archives.html
Options: http://lists.4d.com/mailman/options/4d_tech
Unsub:  mailto:4d_tech-unsubscr...@lists.4d.com
**

Re: Preemptive mode flaw

[David Adams via 4D_Tech]

> On Sun, May 7, 2017 at 1:29 PM, Keisuke Miyako via 4D_Tech <
4d_tech@lists.4d.com> wrote:

Hey, thanks for writing! I think the world of your contributions and work,
but we disagree pretty completely on this. I'll offer my thoughts and point
of view on all of this and, hopefully, you can give me a new way of looking
things. (You did manage to shift my thinking on ON ERR CALL in general
recently.) And if anyone else has real-world comments or observations, I'm
all in on hearing them, even if they make me change my opinion. My opinions
are *always* right, I just try to change them a lot ;-)

> is "code injection" even a problem?

Absolutely.

> preemptive mode is all about multi-tasking.
If by that you mean "taking advantage of idle cores", then I agree.

> I would expect minimal communication between those tasks,
> literally, they should be "preemptive".
> it's true you can start a preemptive process with CALL WORKER,
> but that is not required,
> you can also use New process or Execute on server.
> it's totally fine to have a runaway preemptive task.

Good point, thanks for pointing it out. I wasn't thinking about New
process/Execute on server, excellent point.

Just to make it clear, I have very little interest in preemptive
processing. I expect it to be a powerful and useful technique in a *very*
narrow range of problems. To justify the effort of using preemptive
threads, you need a task that is 1) taking a long time, 2) CPU-bound, and
3) doesn't require so much time to prepare and marshall that you don't get
much speed gain, and 4) has a sufficiently large non-cooperative component
to make it work the trouble. That's a lot of hurdles to clear and,
honestly, I don't have a single task in day-to-day use that makes it clear
of all of them. I've worked on systems (and am likely to work on systems
again soon) where such tasks exists, but they're not so commonplace.

> workers, on the other hand, is all about messaging.
Sadly, we *don't* have messages. We have blocks of code that are run.
That's not messaging. It only seems like messaging *if you're 4D.* As a 4D
developer, there's no queue, there's no message. If you're inside of 4D
itself, you can see a queue, etc. But as a 4D developer? You can't. All we
have is a pipe that things pop out of one-at-a-time and then execute. We
don't know if anything is in the pipe, we don't know when something
arrives, we don't have any ability to view/screen/filter/reject/trap that.
There's no event, nothing. Code is being tacked onto the end of a thread
and run blindly. It's an inherently dangerous design, but it _can_ be used
well. It just isn't built to run safely out of the box. Don't get me wrong,
I like the feature - I just don't use it without quite a bit of scaffolding
in place to minimize and/or eliminate risks.

As a practical example a very common, obvious, and good use of a preemptive
thread with CALL PROCESS, consider a log writer. It's a perfect setup.
Let's say you're logging errors/tasks/event/requests/whatever in 8
processes. Should each have its own log file? If not, it's a huge,
time-sucking waste to manage the contention between processes fighting for
a lock on the file. So, the better solution is to pipe all log entries to a
single process for writing. And since writing to disk is a relatively slow
operation, often of low value for logs, why suck up time on the main CPU?

CALL WORKER("LogWriter_Errors";"LogWriter_Start";"Errors")
LogWriter_Errors opens/appends/whatever a log file on disk *and keeps it
open*. Then any number of processes can stream log lines in for writing:

CALL WORKER("LogWriter_Errors";"LogWriter_Write";$error_details)
LogWriter_Errors catches the line, does a SEND PACKET and you're done and
dusted. You could set it up to handle different types of logs (say have
errors and requests both open), use different workers for different types
of logs, etc. None of the data has to go through 4D's data file, you've
just got a pass-through to disk. Sweet. Speed improvement over fighting
over the log file, etc.? 20x is *easy*. That's a huge optimization -
largely because of the design, the preemptive part - I don't even know how
much that helps. But the critical detail is that *the log file is opened
once at the start of the worker and closed only when the worker finishes.*
If that fails, you've got a big fat problem. (As it turns out, 4D currently
has a bug that makes race conditions on files super easy to create. It's in
the bug system somewhere but 16.2 and 16R2 make no difference.) So, yeah,
CALL WORKER is 100% appropriate, a preemptive worker is 100%
appropriate...and the ability to break the whole think with a simple
mistake makes it too dangerous to use.

In fact, that's my conclusion about preemptive mode now: Not ready for
prime time. My hat is off to 4D for trying on this as making a language
multi-threaded isn't a small thing. But I've found it just way too easy to
break so far. (I've tried a few times and ran into problems quickly i

Re: Tip: Code to get all thread-safe commands

[Douglas von Roeder via 4D_Tech]

I saw the Gestalt command and figured it was a new command (with Macintosh
roots).

So, what other things were "leaked"?


On Sat, May 6, 2017 at 8:09 PM Keisuke Miyako via 4D_Tech <
4d_tech@lists.4d.com> wrote:

> also check out the "How Do I?" example
>
> http://blog.4d.com/preemptive-processes-for-4d-language/
> http://download.4d.com//Demos/4D_v15_R5/HDI_Command_name.zip
>
> another important piece of the puzzle is PROCESS PROPERTIES
>
> http://doc.4d.com/4Dv16R2/4D/16-R2.1620/PROCESS-PROPERTIES.301-3111842.en.html
>
> if you've attended JPR's training session you would have seen an
> undocumented feature
> Gestalt ("4thr") but PROCESS PROPERTIES is the official way to do it;
> http://doc.4d.com/4Dv16R2/4D/16-R2.1620/Gestalt.301-3111463.en.html
>
> > 2017/05/07 9:47、David Adams via 4D_Tech <4d_tech@lists.4d.com> のメール：
> >  // GetCommandList
>
>
>
> **
> 4D Internet Users Group (4D iNUG)
> FAQ:  http://lists.4d.com/faqnug.html
> Archive:  http://lists.4d.com/archives.html
> Options: http://lists.4d.com/mailman/options/4d_tech
> Unsub:  mailto:4d_tech-unsubscr...@lists.4d.com
> **

-- 
--
Douglas von Roeder
949-336-2902
**
4D Internet Users Group (4D iNUG)
FAQ:  http://lists.4d.com/faqnug.html
Archive:  http://lists.4d.com/archives.html
Options: http://lists.4d.com/mailman/options/4d_tech
Unsub:  mailto:4d_tech-unsubscr...@lists.4d.com
**

Re: Preemptive mode flaw

[David Adams via 4D_Tech]

> Not a flaw… don’t do this… you know you are not supposed to,

I did it on purpose to see what happens. I'm fanatical about checking
inputs and preconditions and wanted to see what sort of error to trap for.
In many cases, there is no error trapped - it just screws up.

This scenario is *way* to easy to produce in the wild to simply leave it up
to developers to screen their own code. I objected to EXECUTE for a very
long time because of this very reason. Unsafe at any speed. I only started
using it under tight regulation once I could check that a method exists
before running the code.

This situation would be 100% avoidable with as lightly different design for
CALL WORKER. Instead of appending code to the workers chain of execution,
pass a message. (CALL WORKER calls are not messages in any normal use of
that word in computing.) Pass a message, let the developer collect it and
decide what to do. At that point, it's on us. As it stands, it's on them as
they haven't provided 1) proper intercept tools to screen the call before
it is executed, 2) a reliable or consistent error handling mechanism to
trap the error.

> I was told it will produce a runtime error

Sometimes it produces a runtime error, sometimes it doesn't. Sometimes it
kills the process that then restarts.

On Sun, May 7, 2017 at 12:37 PM, npdennis  wrote:

> CALL WORKER("Preeemptive_Worker";"Display_Message";"Hello world!”)
>
>
> Not a flaw… don’t do this… you know you are not supposed to, I was told it
> will produce a runtime error
>
> This is the same as
>
> C_LONGINT(myVar)
> EXECUTE FORMULA(“myVar:=\”Hello\””)
>
> It will compile just fine, but throw a runtime error.
>
>
> --
> Neil Dennis
> 4D Developer since 1990
>
> GreaText - Designing Software for the Way You Work
> 716 East 1850 N
> North Ogden, UT 84414
>
> mailto:npden...@greatext.com 
> http://www.greatext.com/
>
>
>
**
4D Internet Users Group (4D iNUG)
FAQ:  http://lists.4d.com/faqnug.html
Archive:  http://lists.4d.com/archives.html
Options: http://lists.4d.com/mailman/options/4d_tech
Unsub:  mailto:4d_tech-unsubscr...@lists.4d.com
**

Re: Preemptive mode flaw

[Keisuke Miyako via 4D_Tech]

is "code injection" even a problem?

preemptive mode is all about multi-tasking.

I would expect minimal communication between those tasks,
literally, they should be "preemptive".
it's true you can start a preemptive process with CALL WORKER,
but that is not required,
you can also use New process or Execute on server.
it's totally fine to have a runaway preemptive task.

workers, on the other hand, is all about messaging.

it's a different way of cooperating compared to cooperative processes,
that is, they is not "manager (scheduler)" or concept of "yield (IDLE)",
nevertheless, process are expected to cooperate and not preempt.

maybe it's better not to consider the two features as a set.

2017/05/07 11:20、David Adams via 4D_Tech 
<4d_tech@lists.4d.com> のメール：

I'm looking at preemptive mode again and, again, tried something I wondered
about. "What happens when you inject bad code?" Say you've got a preemptive
thread, it can't use MESSAGE. If you put MESSAGE in the call chain, the
Compiler complains. Go the Compiler! But what happens if you have a method
that calls MESSAGE:



**
4D Internet Users Group (4D iNUG)
FAQ:  http://lists.4d.com/faqnug.html
Archive:  http://lists.4d.com/archives.html
Options: http://lists.4d.com/mailman/options/4d_tech
Unsub:  mailto:4d_tech-unsubscr...@lists.4d.com
**

Re: Tip: Code to get all thread-safe commands

[Keisuke Miyako via 4D_Tech]

also check out the "How Do I?" example

http://blog.4d.com/preemptive-processes-for-4d-language/
http://download.4d.com//Demos/4D_v15_R5/HDI_Command_name.zip

another important piece of the puzzle is PROCESS PROPERTIES
http://doc.4d.com/4Dv16R2/4D/16-R2.1620/PROCESS-PROPERTIES.301-3111842.en.html

if you've attended JPR's training session you would have seen an undocumented 
feature
Gestalt ("4thr") but PROCESS PROPERTIES is the official way to do it;
http://doc.4d.com/4Dv16R2/4D/16-R2.1620/Gestalt.301-3111463.en.html

> 2017/05/07 9:47、David Adams via 4D_Tech <4d_tech@lists.4d.com> のメール：
>  // GetCommandList



**
4D Internet Users Group (4D iNUG)
FAQ:  http://lists.4d.com/faqnug.html
Archive:  http://lists.4d.com/archives.html
Options: http://lists.4d.com/mailman/options/4d_tech
Unsub:  mailto:4d_tech-unsubscr...@lists.4d.com
**

Re: Preemptive mode flaw

[npdennis via 4D_Tech]

> CALL WORKER("Preeemptive_Worker";"Display_Message";"Hello world!”)

Not a flaw… don’t do this… you know you are not supposed to, I was told it will 
produce a runtime error

This is the same as

C_LONGINT(myVar)
EXECUTE FORMULA(“myVar:=\”Hello\””)

It will compile just fine, but throw a runtime error.


--
Neil Dennis
4D Developer since 1990

GreaText - Designing Software for the Way You Work
716 East 1850 N
North Ogden, UT 84414

mailto:npden...@greatext.com
http://www.greatext.com/


**
4D Internet Users Group (4D iNUG)
FAQ:  http://lists.4d.com/faqnug.html
Archive:  http://lists.4d.com/archives.html
Options: http://lists.4d.com/mailman/options/4d_tech
Unsub:  mailto:4d_tech-unsubscr...@lists.4d.com
**

Preemptive mode flaw

[David Adams via 4D_Tech]

I'm looking at preemptive mode again and, again, tried something I wondered
about. "What happens when you inject bad code?" Say you've got a preemptive
thread, it can't use MESSAGE. If you put MESSAGE in the call chain, the
Compiler complains. Go the Compiler! But what happens if you have a method
that calls MESSAGE:

Display_Message

The Compiler complains. Go the Compiler!

Now what if you have this:

CALL WORKER("Preeemptive_Worker";"Display_Message";"Hello world!")

It's illegal - you're injecting code into the preemptive worker that
requires cooperative mode to work. What happens? It really depends. It
isn't consistent, it isn't documented, and it isn't good.

I've put more details into a feature request here:

Provide developer control when a preemptive worker receives non-preemptive
code
http://forums.4d.fr/Post//19391591/1/

Given how many people seem excited about preemptive threads for some
reason, I'm surprised that this hasn't come up. That also makes me
entertain the possibility that I'm being stupid or, more likely, trapped in
some kind of tunnel vision. So, if I've missed the obvious. I'll be
embarrassed, and then I'll get over it. So, don't be shy, feel free to make
me look dumb.

Or, if not, feel free to jump in and vote for my feature requests. I think
we need an incoming-code-for-execution inspection phase anyway.

Note: I'm working from a little scratch database that I've built today. No
frills, very simple code. I'm using 16.1 on OS X and have tried various
things in stand-alone and under 4D Server.
**
4D Internet Users Group (4D iNUG)
FAQ:  http://lists.4d.com/faqnug.html
Archive:  http://lists.4d.com/archives.html
Options: http://lists.4d.com/mailman/options/4d_tech
Unsub:  mailto:4d_tech-unsubscr...@lists.4d.com
**

Tip: Code to get all thread-safe commands

[David Adams via 4D_Tech]

I was just looking up a couple of details about 4D's thread-safe commands
and noticed a cross-ref to the Command name docs:

http://doc.4d.com/4Dv16/4D/16.1/Command-name.301-3375914.en.html

I'm always the last to know. 4D's boosted this command's features pretty
substantially. You can now get the thread-safety of the command and what
theme it belongs to. Nice! (The property bits only have one value now - but
perhaps more will be added later?) I combined a couple of the examples and
rewrote them in my own idiom...posted below. All this does is extract the
name, thread-safety, and theme of every command in the language into a
series of arrays. That's it. It only takes a few minutes to write this
code, but I thought I'd post it to draw attention to the new(ish?) powers
of Command name...in case I'm *not* the last to know. In your own work you
might want filtering, to put the data into a C_OBJECT, etc., etc.

  // GetCommandList

ARRAY LONGINT(Command_Number_al;0)
ARRAY TEXT(Command_Names_at;0)
ARRAY TEXT(Command_Themes_at;0)
ARRAY BOOLEAN(Command_ThreadSafe_ab;0)

Repeat
C_LONGINT($command_number)
C_LONGINT($flags_l)
c_text($theme_name)
C_TEXT($command_name)
$command_number:=$command_number+1
$command_name:=Command name($command_number;$flags_l;$theme_name)
If (OK=1)  //command number exists
If (Length($command_name)>0)  //command is not disabled
C_BOOLEAN($thread_safe)
$thread_safe:=$flags_l ?? 0
APPEND TO ARRAY(Command_Names_at;$command_name)
APPEND TO ARRAY(Command_Number_al;$command_number)
APPEND TO ARRAY(Command_Themes_at;$theme_name)
APPEND TO ARRAY(Command_ThreadSafe_ab;$thread_safe)
End if
End if
 Until(OK=0) //end of existing commands
**
4D Internet Users Group (4D iNUG)
FAQ:  http://lists.4d.com/faqnug.html
Archive:  http://lists.4d.com/archives.html
Options: http://lists.4d.com/mailman/options/4d_tech
Unsub:  mailto:4d_tech-unsubscr...@lists.4d.com
**

Re: Counting # of specific characters in string/text

[Alan Chan via 4D_Tech]

Hi Chip

We have been talking about * all along. Without * has never been in my test.

My test always consider total length of text "and" total count of occurrence. 
My test is for 4,300,000 length text and 100,000 occurrence. It took 14ms on my 
machine.

Alan Chan

4D iNug Technical <4d_tech@lists.4d.com> writes:
>v13 testing
>position - no * find character (u + umlaut) end of 2,000,000 length text
>56ms +/- ms
>
>Position (*) - find same character at end of 2,000,000 length text
>3 ms +/- 1ms
>
>so... when you know that diacritcallity, or capitalization, is important 
>Position with * is MUCH faster -- at least in v13

**
4D Internet Users Group (4D iNUG)
FAQ:  http://lists.4d.com/faqnug.html
Archive:  http://lists.4d.com/archives.html
Options: http://lists.4d.com/mailman/options/4d_tech
Unsub:  mailto:4d_tech-unsubscr...@lists.4d.com
**

Re: Counting # of specific characters in string/text

[Chip Scheide via 4D_Tech]

v13 testing
position - no * find character (u + umlaut) end of 2,000,000 length text
56ms +/- ms

Position (*) - find same character at end of 2,000,000 length text
3 ms +/- 1ms

so... when you know that diacritcallity, or capitalization, is important 
Position with * is MUCH faster -- at least in v13


> Oops... my bad. Reading too fast. Please ignore this.
> 
> Alan Chan
> 
> 4D iNug Technical <4d_tech@lists.4d.com> writes:
>> I found out why your result is quite different from mine. In your 
>> code, you evaluate the time once per execution of position. However, 
>> the replace string was evaluated once per 10,000 replacement (per 
>> execution of replace string).
> 
> 
> **
> 4D Internet Users Group (4D iNUG)
> FAQ:  http://lists.4d.com/faqnug.html
> Archive:  http://lists.4d.com/archives.html
> Options: http://lists.4d.com/mailman/options/4d_tech
> Unsub:  mailto:4d_tech-unsubscr...@lists.4d.com
> **

Hell is other people 
 Jean-Paul Sartre
**
4D Internet Users Group (4D iNUG)
FAQ:  http://lists.4d.com/faqnug.html
Archive:  http://lists.4d.com/archives.html
Options: http://lists.4d.com/mailman/options/4d_tech
Unsub:  mailto:4d_tech-unsubscr...@lists.4d.com
**

Re: Counting # of specific characters in string/text

[Alan Chan via 4D_Tech]

Oops... my bad. Reading too fast. Please ignore this.

Alan Chan

4D iNug Technical <4d_tech@lists.4d.com> writes:
>I found out why your result is quite different from mine. In your code, you 
>evaluate the time once per execution of position. However, the replace string 
>was evaluated once per 10,000 replacement (per execution of replace string).


**
4D Internet Users Group (4D iNUG)
FAQ:  http://lists.4d.com/faqnug.html
Archive:  http://lists.4d.com/archives.html
Options: http://lists.4d.com/mailman/options/4d_tech
Unsub:  mailto:4d_tech-unsubscr...@lists.4d.com
**

Re: Counting # of specific characters in string/text

[Alan Chan via 4D_Tech]

Hi Arnaud,

I found out why your result is quite different from mine. In your code, you 
evaluate the time once per execution of position. However, the replace string 
was evaluated once per 10,000 replacement (per execution of replace string).

Alan Chan

4D iNug Technical <4d_tech@lists.4d.com> writes:
>since v15R3, using 'Replace string' for count seems always faster to me than 
>'Position' - even if [compiled+strict comparison] it's quite the same. Could 
>be a detail, but instead of measuring time to loop $i times, I often prefer to 
>count how much
>iterations are executed during a given time, it makes interpreted/compiled 
>switch easier. All results under are from compiled, all 32b versions except 
>v16 in 64b. 
>
>The little v12 test base I've used is here:
>

**
4D Internet Users Group (4D iNUG)
FAQ:  http://lists.4d.com/faqnug.html
Archive:  http://lists.4d.com/archives.html
Options: http://lists.4d.com/mailman/options/4d_tech
Unsub:  mailto:4d_tech-unsubscr...@lists.4d.com
**

Re: Counting # of specific characters in string/text

[Alan Chan via 4D_Tech]

Hi Arnaud,

First of all, it seems we have reverse definition of Diacritical. To avoid 
confusion, let's use (*) to represent the operation.

I'm interested how you could count execution in "given time". In this time, we 
execute 100,000 times of position(*) in 14ms and Replace String(*) to replace 
100,000 occurance of same length of replacement string in 16ms.

In your test, you got 684 execution of position in 180 ticks (3 seconds). In my 
test, I got 100,000 execution of position in less than 1 tick. Even with 
replace string(*), it replaced 100,000 occurence of string of same length in 
16ms (just under 1
tick which is 16.67ms). Due to high overhead of your testing code, it might 
compromise the result.

OK. I have confessed that I tested only 13.6 (not 16) since I don't want to 
install 16 on my development machine. However, based on 4D, Replace String(*) 
in v16 perform as fast as v13.6 if the replacement string is of same length 
(very fast indeed -
100,000 replacement in 16ms). For replacement of string of longer length, v16 
will do "pre-scan" number of occurance and create an appropriate block of 
memory for the result. It means, it should be taking a little more time than 
replacement string
of same length.

My test code in ES_Count

C_LONGINT($0;$c;$start;$pos;$len)
C_TEXT($1;$2;$source;$find)
$c:=0
$source:=$1
$find:=$2
$len:=Length($find)
$start:=1
If ((Length($source)>0) & ($len>0))
Repeat 
$pos:=Position($find;$source;$start;*) 
If ($pos>0)
$start:=$pos+$len
$c:=$c+1
End if 
Until ($pos=0)
End if 
$0:=$c

Alan Chan

4D iNug Technical <4d_tech@lists.4d.com> writes:
>since v15R3, using 'Replace string' for count seems always faster to me than 
>'Position' - even if [compiled+strict comparison] it's quite the same. Could 
>be a detail, but instead of measuring time to loop $i times, I often prefer to 
>count how much
>iterations are executed during a given time, it makes interpreted/compiled 
>switch easier. All results under are from compiled, all 32b versions except 
>v16 in 64b. 
>
>The little v12 test base I've used is here:
>

**
4D Internet Users Group (4D iNUG)
FAQ:  http://lists.4d.com/faqnug.html
Archive:  http://lists.4d.com/archives.html
Options: http://lists.4d.com/mailman/options/4d_tech
Unsub:  mailto:4d_tech-unsubscr...@lists.4d.com
**

Long Beach - 4D World Tour

[npdennis via 4D_Tech]

I’m returning home this morning from the Long Beach 4D World Tour…

The trip was well worth the cost even with Air, Food, Lodging and the class 
cost.

Some training highlights:

 - Workers and Processes
 - Message communication
 - In depth look at ListBoxes
 - Code style hints to make programming easier to day
 - Tips to make programming transitions easy to future version
 - Tools to convert to v16 (Including an impressive picture convert utility)
 - Much more...

There was much, much more. This was my second tour, I’m definitely coming back 
to the next one. There is only one more in the US, if you haven’t gone to one 
already I don’t think it is too late.

Neil



**
4D Internet Users Group (4D iNUG)
FAQ:  http://lists.4d.com/faqnug.html
Archive:  http://lists.4d.com/archives.html
Options: http://lists.4d.com/mailman/options/4d_tech
Unsub:  mailto:4d_tech-unsubscr...@lists.4d.com
**

Re: Counting # of specific characters in string/text

[Arnaud de Montard via 4D_Tech]

> Le 6 mai 2017 à 12:07, Alan Chan via 4D_Tech <4d_tech@lists.4d.com> a écrit :
> 
> I believe it's true only for non-diacritical operation. However, for 
> diacritical operation - replace string(*) and position(*), Position(*) is 
> still faster than replace string.

Hi Alan, 
since v15R3, using 'Replace string' for count seems always faster to me than 
'Position' - even if [compiled+strict comparison] it's quite the same. Could be 
a detail, but instead of measuring time to loop $i times, I often prefer to 
count how much iterations are executed during a given time, it makes 
interpreted/compiled switch easier. All results under are from compiled, all 
32b versions except v16 in 64b. 

The little v12 test base I've used is here:


-- 
Arnaud de Montard 



**
4D Internet Users Group (4D iNUG)
FAQ:  http://lists.4d.com/faqnug.html
Archive:  http://lists.4d.com/archives.html
Options: http://lists.4d.com/mailman/options/4d_tech
Unsub:  mailto:4d_tech-unsubscr...@lists.4d.com
**

Re: Counting # of specific characters in string/text

[Alan Chan via 4D_Tech]

I believe it's true only for non-diacritical operation. However, for 
diacritical operation - replace string(*) and position(*), Position(*) is still 
faster than replace string.

In order to provide BEST scenario for replace string(*), I use same length of 
old and new string for the test (this should be fastest for replace string 
operation).

C_TEXT($source;$find;$rtext)
C_LONGINT($count;$time;$rcount)
$count:=10
$find:=Char(13)
$source:=("this is to test how fast replace string is"+$find)*$count

$time:=0
ES_TestTime (0;->$time)
$rtext:=Replace string($source;$find;$find;*)
ES_TestTime (1;->$time;"total time for replace string:")

$time:=0
ES_TestTime (0;->$time)
$rcount:=ES_Count ($source;$find)  //this is simple position operation
ES_TestTime (1;->$time;"total time for position:")

Replace string took 16ms while position took 14ms consistently.

Alan Chan

4D iNug Technical <4d_tech@lists.4d.com> writes:
>About using Replace string and "big" text (example: count lines in a csv):
>
> 
>With 4D v16 compiled, Replace string is now always faster that the loop with 
>Position, more specially when diacritical (e=é…). But before it was the wheel 
>of death…

**
4D Internet Users Group (4D iNUG)
FAQ:  http://lists.4d.com/faqnug.html
Archive:  http://lists.4d.com/archives.html
Options: http://lists.4d.com/mailman/options/4d_tech
Unsub:  mailto:4d_tech-unsubscr...@lists.4d.com
**