Re: Survey on 4D’s strategy for implementing Preemptive Processing / worker processes

[Robert ListMail via 4D_Tech]

James, when 4D is keeping the Postgres DB updated what mechanism is used? On 
the same machine, LAN or using a SOAP call that goes over the internet?

Thanks,

R

Sent from my iPhone

> On Feb 22, 2017, at 10:23 AM, James Crate via 4D_Tech <4d_tech@lists.4d.com> 
> wrote:
> 
> 4D pushes data to PostgreSQL, or retrieves data from PostgreSQL, based on 
> either user-triggered actions or periodic task execution.

**
4D Internet Users Group (4D iNUG)
FAQ:  http://lists.4d.com/faqnug.html
Archive:  http://lists.4d.com/archives.html
Options: http://lists.4d.com/mailman/options/4d_tech
Unsub:  mailto:4d_tech-unsubscr...@lists.4d.com
**

Re: Survey on 4D’s strategy for implementing Preemptive Processing / worker processes

[James Crate via 4D_Tech]

It’s not very sophisticated. 4D pushes data to PostgreSQL, or retrieves data 
from PostgreSQL, based on either user-triggered actions or periodic task 
execution. At no point does the Rails/PostgreSQL system access 4D.  The fact 
that 4D uses SQL to access the PostgreSQL database has specific disadvantages, 
such as updates involving schema changes needing to have coordinated release. 

Jim Crate

> On Feb 22, 2017, at 1:17 AM, Robert ListMail via 4D_Tech 
> <4d_tech@lists.4d.com> wrote:
> 
> Jim, that sound sophisticated. :). So, when Rails is handling a web request 
> in what ways can Rails get to 4D data? Does all data need to be in 
> PostgresSQL database Rails knows about?
> 
> Thanks,
> 
> Robert
> 
> Sent from my iPhone
> 
>> On Feb 21, 2017, at 9:33 AM, James Crate via 4D_Tech <4d_tech@lists.4d.com> 
>> wrote:
>> 
>> I am, both using 4D to directly access the PostgreSQL database with Rob 
>> Laveaux’s plugin and using LEP+curl to access APIs I’ve built. Going 
>> forward, I am moving away from direct database access to exclusively using 
>> APIs. Rails is used for all web access, with no web serving built in 4D at 
>> all.
>> 
>> Jim Crate
> 
> **
> 4D Internet Users Group (4D iNUG)
> FAQ:  http://lists.4d.com/faqnug.html
> Archive:  http://lists.4d.com/archives.html
> Options: http://lists.4d.com/mailman/options/4d_tech
> Unsub:  mailto:4d_tech-unsubscr...@lists.4d.com
> **

**
4D Internet Users Group (4D iNUG)
FAQ:  http://lists.4d.com/faqnug.html
Archive:  http://lists.4d.com/archives.html
Options: http://lists.4d.com/mailman/options/4d_tech
Unsub:  mailto:4d_tech-unsubscr...@lists.4d.com
**

Re: Survey on 4D’s strategy for implementing Preemptive Processing / worker processes

[Robert ListMail via 4D_Tech]

Jim, that sound sophisticated. :). So, when Rails is handling a web request in 
what ways can Rails get to 4D data? Does all data need to be in PostgresSQL 
database Rails knows about?

Thanks,

Robert

Sent from my iPhone

> On Feb 21, 2017, at 9:33 AM, James Crate via 4D_Tech <4d_tech@lists.4d.com> 
> wrote:
> 
> I am, both using 4D to directly access the PostgreSQL database with Rob 
> Laveaux’s plugin and using LEP+curl to access APIs I’ve built. Going forward, 
> I am moving away from direct database access to exclusively using APIs. Rails 
> is used for all web access, with no web serving built in 4D at all.
> 
> Jim Crate

**
4D Internet Users Group (4D iNUG)
FAQ:  http://lists.4d.com/faqnug.html
Archive:  http://lists.4d.com/archives.html
Options: http://lists.4d.com/mailman/options/4d_tech
Unsub:  mailto:4d_tech-unsubscr...@lists.4d.com
**

Re: Survey on 4D’s strategy for implementing Preemptive Processing / worker processes

[James Crate via 4D_Tech]

>> On Nov 3, 2016, at 8:11 PM, James Crate  wrote:
>> 
>> Ruby/Rails apps - processing large numbers of files; performing slow fuzzy 
>> search on slices of data; 

> On Feb 20, 2017, at 7:13 PM, Robert ListMail via 4D_Tech 
> <4d_tech@lists.4d.com> wrote:
> 
> Jim, are you using Ruby/Rails with 4D in any way…?

I am, both using 4D to directly access the PostgreSQL database with Rob 
Laveaux’s plugin and using LEP+curl to access APIs I’ve built. Going forward, I 
am moving away from direct database access to exclusively using APIs. Rails is 
used for all web access, with no web serving built in 4D at all.

Jim Crate

**
4D Internet Users Group (4D iNUG)
FAQ:  http://lists.4d.com/faqnug.html
Archive:  http://lists.4d.com/archives.html
Options: http://lists.4d.com/mailman/options/4d_tech
Unsub:  mailto:4d_tech-unsubscr...@lists.4d.com
**

Re: Survey on 4D’s strategy for implementing Preemptive Processing / worker processes

[Robert ListMail via 4D_Tech]

Jim, are you using Ruby/Rails with 4D in any way…?

Thanks,

Robert

> On Nov 3, 2016, at 8:11 PM, James Crate  wrote:
> 
> Ruby/Rails apps - processing large numbers of files; performing slow fuzzy 
> search on slices of data; 

**
4D Internet Users Group (4D iNUG)
FAQ:  http://lists.4d.com/faqnug.html
Archive:  http://lists.4d.com/archives.html
Options: http://lists.4d.com/mailman/options/4d_tech
Unsub:  mailto:4d_tech-unsubscr...@lists.4d.com
**

Re: Survey on 4D’s strategy for implementing Preemptive Processing / worker processes

[Tony Ringsmuth]

Hey Keith

>Sorry Tony, I also dislike the idea of IP variables being copied into new 
> preemptive processes.

It seems pretty un-popular with the community.  I’m at peace with that.
My big beef (as stated before) is that excluding entire methods, based on any 
content, and any sub-content just makes this a terrible implementation in my 
view.

Think of it like this:
There are some commands that we cannot call during the redraw of a record 
listing:  like ALERT.
Why didn’t 4D make it such that a record listing cannot call any 4D method that 
contains the command ALERT?  They didn’t.  
I have ALERT is some methods that are called from record listings:  but it’s 
behind conditions that never invoke the ALERT if I’m in the context of a record 
listing.

Regards,
Tony




On 11/4/16, 7:54 AM, "4D_Tech on behalf of Keith White" 
<4d_tech-boun...@lists.4d.com on behalf of kwh...@synergist.co.uk> wrote:

Hi

Given the current restrictions, we're currently not in a position to 
contemplate Preemptive.  Of course we'll be working on our code to move towards 
it.

But what I just don't get is why 4D thinks that putting compiler _errors_ 
in for this feature instead of just warnings, is a great idea.  4D has made the 
compiler check and error on a lot of things where it _can_ check, but why 
bother stopping you from compiling when the compiler can't check everything 
anyway?

Runtime errors will be generated if offending code happens to execute.  
These runtime checks are already in place for cases like dereferencing pointers 
to IP variables, surely.

Also consider EXECUTE METHOD (a thread safe command, apparently) - the 
compiler obviously can't check what code this command might execute.

The main issue we'll face with attempting to get our code to be thread safe 
will be IP Arrays.  Our IP arrays are static (in that they are populated on 
server startup) and without them performance would be very badly degraded.  I'm 
encouraged by the discussion here of a new type of semi-static array concept 
that could be populated at startup - as long as:-

1. Static arrays could be accessed (but not updated) from cooperative 
processes
2. Static arrays could be updated at times other than startup during 
development when running in interpreted mode (and hence there are no 
pre-emptive processes)

Sorry Tony, I also dislike the idea of IP variables being copied into new 
preemptive processes.

Best regards

Keith White
Synergist Express Ltd, UK.
**
4D Internet Users Group (4D iNUG)
FAQ:  http://lists.4d.com/faqnug.html
Archive:  http://lists.4d.com/archives.html
Options: http://lists.4d.com/mailman/options/4d_tech
Unsub:  mailto:4d_tech-unsubscr...@lists.4d.com
**


**
4D Internet Users Group (4D iNUG)
FAQ:  http://lists.4d.com/faqnug.html
Archive:  http://lists.4d.com/archives.html
Options: http://lists.4d.com/mailman/options/4d_tech
Unsub:  mailto:4d_tech-unsubscr...@lists.4d.com
**

Re: Survey on 4D’s strategy for implementing Preemptive Processing / worker processes

[Chip Scheide]

my 2 cents

If i am to understand correctly, Tony, you and I use IP variables in a 
similar manner - we use them (primarily) as 'Konstants'.

I think a possible solution to at least this part of the IP issue is:
- some mechanism that allows us to either easily (as part of 4D not a 
plugin/component) either create 'Konstants' (doesn't exist now), or a 
means to flag IP vars as constants (basically write once, read many 
values).  To my thinking this last part could be done via a pair of 
database setting/preferences.
  set PreFix __  Post Fix __  :: string ___ for IP vars to be 
considered as constants.
  set IP Constant Method name ___  to run during startup

This would allow us to use a method (which may call many others) from 
which IP constants are given a value. Allow us to flag both for 4D, and 
for ourselves which IP vars are contents and should not be changed 
(after initialization) during execution.  The pre fix/Postfix option is 
to allow for personal preference in naming conventions (i.e. add the 
var designator to the front or back of the name).



On Tue, 01 Nov 2016 15:36:00 -0500, Tony Ringsmuth wrote:
>>(Peter said) 2. Allowing preemptive code to use interprocess 
>> variables with caveat that
> they are not in fact interprocess would be confusing and misleading (and
> lead to situation that code will behave differently in preemptive and
> standard mode - see next point.)
> 
> Tony’s reply:
> In the implementation that I am suggesting, the worker process would 
> behave more as if it was running in a batch workstation.
> As a worker process starts up, it could call a method that would see 
> that the ~interprocess varialbes~ are not populated, and then 
> populate them, if desired.
> Given a choice between absolutely no IP var usage whatsoever, and, 
> using IP vars in a different way:  I think “using IP vars in a 
> different way” would be preffered.
**
4D Internet Users Group (4D iNUG)
FAQ:  http://lists.4d.com/faqnug.html
Archive:  http://lists.4d.com/archives.html
Options: http://lists.4d.com/mailman/options/4d_tech
Unsub:  mailto:4d_tech-unsubscr...@lists.4d.com
**

Re: Survey on 4D’s strategy for implementing Preemptive Processing / worker processes

[Keith White]

Hi

Given the current restrictions, we're currently not in a position to 
contemplate Preemptive.  Of course we'll be working on our code to move towards 
it.

But what I just don't get is why 4D thinks that putting compiler _errors_ in 
for this feature instead of just warnings, is a great idea.  4D has made the 
compiler check and error on a lot of things where it _can_ check, but why 
bother stopping you from compiling when the compiler can't check everything 
anyway?

Runtime errors will be generated if offending code happens to execute.  These 
runtime checks are already in place for cases like dereferencing pointers to IP 
variables, surely.

Also consider EXECUTE METHOD (a thread safe command, apparently) - the compiler 
obviously can't check what code this command might execute.

The main issue we'll face with attempting to get our code to be thread safe 
will be IP Arrays.  Our IP arrays are static (in that they are populated on 
server startup) and without them performance would be very badly degraded.  I'm 
encouraged by the discussion here of a new type of semi-static array concept 
that could be populated at startup - as long as:-

1. Static arrays could be accessed (but not updated) from cooperative processes
2. Static arrays could be updated at times other than startup during 
development when running in interpreted mode (and hence there are no 
pre-emptive processes)

Sorry Tony, I also dislike the idea of IP variables being copied into new 
preemptive processes.

Best regards

Keith White
Synergist Express Ltd, UK.
**
4D Internet Users Group (4D iNUG)
FAQ:  http://lists.4d.com/faqnug.html
Archive:  http://lists.4d.com/archives.html
Options: http://lists.4d.com/mailman/options/4d_tech
Unsub:  mailto:4d_tech-unsubscr...@lists.4d.com
**

Re: Survey on 4D’s strategy for implementing Preemptive Processing / worker processes

[David Adams]

Fuzzy matching...now I really won't be able to sleep tonight. I was pretty
obsessive for some time. There are a bunch of good papers about using
n-gram tables (2, 3, or 4-grams) in SQL databases to perform highly
optimized comparisons. It takes that little bit of extra setup, but then
you can get solid coverage of huge data sets with excellent performance. I
love Levenshtein, but n-grams are better at detecting names entered out of
order, like "Adam David" instead of "David Adams." LCS and Levenshtein
won't find this as a match, but even a 3-gram will flag it.

Let me know if you're interested and I'll dig up papers or references.
**
4D Internet Users Group (4D iNUG)
FAQ:  http://lists.4d.com/faqnug.html
Archive:  http://lists.4d.com/archives.html
Options: http://lists.4d.com/mailman/options/4d_tech
Unsub:  mailto:4d_tech-unsubscr...@lists.4d.com
**

Re: Survey on 4D’s strategy for implementing Preemptive Processing / worker processes

[David Adams]

I'm not sure how well it works with PostgreSQL, but for Levenshtein, you
can screen out strings that are longer/shorter than your starting string by
greater than your target threshold. So, if you have a 7 character string
and want an edit distance of 2, your candidate pool can be limited to
strings with lengths 5-9. (Off the top of my head.) If the PostgreSQL
character_length() function is fast enough, this can help. For bigram
comparisons, you could potentially apply something of a similar pre-slicing.

Not sure if this pays off in your case, but if you can do fast filters
before applying the more intensive coefficient calculation, maybe it would
be of some help?
**
4D Internet Users Group (4D iNUG)
FAQ:  http://lists.4d.com/faqnug.html
Archive:  http://lists.4d.com/archives.html
Options: http://lists.4d.com/mailman/options/4d_tech
Unsub:  mailto:4d_tech-unsubscr...@lists.4d.com
**

Re: Survey on 4D’s strategy for implementing Preemptive Processing / worker processes

[David Adams]

> It is slow. However, being able to utilize all 16 logical cores on the
dual-processor xserve takes it from being “too slow”
> to “ok with a warning that this is slow”. Even still, it takes 30 seconds
or so to find possible duplicates in a set of 1000 names.

What algorithm(s) are you using and against what RDBMS? Several engines
have some fuzzy comparators baked in. If you could move the fuzzy matches
onto the server, it might work faster.
**
4D Internet Users Group (4D iNUG)
FAQ:  http://lists.4d.com/faqnug.html
Archive:  http://lists.4d.com/archives.html
Options: http://lists.4d.com/mailman/options/4d_tech
Unsub:  mailto:4d_tech-unsubscr...@lists.4d.com
**

Re: Survey on 4D’s strategy for implementing Preemptive Processing / worker processes

[James Crate]

On Nov 3, 2016, at 9:30 PM, David Adams  wrote:
> 
>> processing large numbers of files; performing slow fuzzy search on slices
> of data;
> 
> You had me at fuzzy ;-) There are so many fantastic fuzzy algorithms that
> are too slow to run in real-time or on normal gear. Any speed improvement
> would be welcome. I've pitched 4D about integrating various fuzzy
> algorithms into the database engine itself (not too hard), but it's not
> happened.

It is slow. However, being able to utilize all 16 logical cores on the 
dual-processor xserve takes it from being “too slow” to “ok with a warning that 
this is slow”. Even still, it takes 30 seconds or so to find possible 
duplicates in a set of 1000 names. 

If 4D’s preemptive threading works well enough, it might be possible to run 
your fuzzy matching utilizing cpu across all currently connected clients. 

Jim Crate

**
4D Internet Users Group (4D iNUG)
FAQ:  http://lists.4d.com/faqnug.html
Archive:  http://lists.4d.com/archives.html
Options: http://lists.4d.com/mailman/options/4d_tech
Unsub:  mailto:4d_tech-unsubscr...@lists.4d.com
**

Re: Survey on 4D’s strategy for implementing Preemptive Processing / worker processes

[David Adams]

> processing large numbers of files; performing slow fuzzy search on slices
of data;

You had me at fuzzy ;-) There are so many fantastic fuzzy algorithms that
are too slow to run in real-time or on normal gear. Any speed improvement
would be welcome. I've pitched 4D about integrating various fuzzy
algorithms into the database engine itself (not too hard), but it's not
happened.
**
4D Internet Users Group (4D iNUG)
FAQ:  http://lists.4d.com/faqnug.html
Archive:  http://lists.4d.com/archives.html
Options: http://lists.4d.com/mailman/options/4d_tech
Unsub:  mailto:4d_tech-unsubscr...@lists.4d.com
**

Re: Survey on 4D’s strategy for implementing Preemptive Processing / worker processes

[James Crate]

On Nov 1, 2016, at 5:49 PM, David Adams  wrote:
> 
> I'm not going to answer either of Tony's questions as I don't feel
> qualified to say one way or another. I'll admit that I've paid almost no
> attention to the preemptive/cooperative subject in upcoming versions of 4D.
> Can those of you who have tried it all out and gotten some experience
> provide a bit of background? Specifically: What problems does this help
> with in your systems. I can easily see why 4D has put a lot of effort into
> improving threading in their server code, what I'm not clear on is which
> problem(s) or classes of problems this will help with meaningfully that
> fall into our hands. I'm not suggesting there aren't problems like this, I
> just don't know what they are.

I haven’t tried out 4D’s implementation and am not likely to do so in the near 
future.

However, things I have used multi-threading for:

Cocoa apps - cropping/resizing/exporting/importing images (huge performance 
gain), performing http requests, loading thumbnails to cache, etc.

Ruby/Rails apps - processing large numbers of files; performing slow fuzzy 
search on slices of data; 

There is a certain satisfaction from converting a lengthy single-core operation 
to a short one that lights up all cores at 100%.

Jim Crate

**
4D Internet Users Group (4D iNUG)
FAQ:  http://lists.4d.com/faqnug.html
Archive:  http://lists.4d.com/archives.html
Options: http://lists.4d.com/mailman/options/4d_tech
Unsub:  mailto:4d_tech-unsubscr...@lists.4d.com
**

Re: Survey on 4D’s strategy for implementing Preemptive Processing / worker processes

[David Adams]

> Does that appear “complex” to you?
Not at all. It also doesn't look like it uses any of the commands that are
forbidden in preemptive mode.

> In this case it would be difficult to summarize the data in all the
possible query options users have..
> It could be done but you’d have to store so many different totals for so
many different option combinations s
>  it just seemed like more work than it was worth for the time saved. I’m
not talking about saving hours to do a
> report, I’m down to trying to shave off a few minutes.
Yeah, summary data can be more trouble than it's worth, for sure. It
depends entirely on circumstances. And it's a generally a total nightmare
when people combine extract (reporting) data with relational data in a
single table.
**
4D Internet Users Group (4D iNUG)
FAQ:  http://lists.4d.com/faqnug.html
Archive:  http://lists.4d.com/archives.html
Options: http://lists.4d.com/mailman/options/4d_tech
Unsub:  mailto:4d_tech-unsubscr...@lists.4d.com
**

Re: Survey on 4D’s strategy for implementing Preemptive Processing / worker processes

[Tony Ringsmuth]

David,

Tony wrote:
> Currently, 4D is trying to get around range-checking type overhead within
Preemptive
> processes by doing their checking at compile time.

David Adams wrote:
Where did you hear this? I didn't see any mention of this in the docs you
mentioned and don't see how what the compiler scans for replaces runtime
range checks. 

This is my assumption.

> Also, runtime checks aren't particularly slow and either you
or 4D needs to do them.
Yes, I agree.

Tony



On Wed, Nov 2, 2016 at 11:27 AM, Tony Ringsmuth  wrote:

> OK:  I can see that my idea of Interprocess variables getting a separate
> instance in Preemptive processes is not popular with several of you.
>
> Consider then, a middle ground:
> Currently, 4D is trying to get around range-checking type overhead within
> Preemptive processes by doing their checking at compile time.  Problem 
here
> is that WHOLE METHODS are then restricted based on any restricted content,
> or sub-content. THIS IS WHAT REALY IRKS ME.
>
> Here’s PROPOSAL #2:
> Restrict only the code-lines that contain restricted items, including
> inter-process variables, instead of entire methods.
>
> So, your method could not contain:
> If($IsPreemptive)
>   $GetSomeValue:=vMyValue
> ELSE
>   $GetSomeValue:=<>MyIpVar
> END IF
>
> Internally, 4D compiler would have to put cod around that line, similar to
> range checking;  but, unlike Range checking; as long as your code never
> hits that line (within a preemptive process), execution is never slowed
> down.  If you do hit that line, 4D generates an error, and does NOT run 
the
> offending line.
>
> The code should really not be any slower, however, it may be somewhat
> larger, when compiled.
>
> Tony
>
>
> **
> 4D Internet Users Group (4D iNUG)
> FAQ:  http://lists.4d.com/faqnug.html
> Archive:  http://lists.4d.com/archives.html
> Options: http://lists.4d.com/mailman/options/4d_tech
> Unsub:  mailto:4d_tech-unsubscr...@lists.4d.com
> **
>
**
4D Internet Users Group (4D iNUG)
FAQ:  http://lists.4d.com/faqnug.html
Archive:  http://lists.4d.com/archives.html
Options: http://lists.4d.com/mailman/options/4d_tech
Unsub:  mailto:4d_tech-unsubscr...@lists.4d.com
**


**
4D Internet Users Group (4D iNUG)
FAQ:  http://lists.4d.com/faqnug.html
Archive:  http://lists.4d.com/archives.html
Options: http://lists.4d.com/mailman/options/4d_tech
Unsub:  mailto:4d_tech-unsubscr...@lists.4d.com
**

Re: Survey on 4D’s strategy for implementing Preemptive Processing / worker processes

[Tim Nevels]

On Nov 1, 2016, at 7:23 PM, David Adams wrote:

> Who knew? Great trick. A good alternative (but more complicated) is to use
> a memory-based message queue and send all of the log data to a single
> process that has the file open. This is a perfect use of CALL WORKER.

Yes, I agree. 

> I gave up on complex triggers in 4D years and versions ago ;-(

Most are less than 10 lines of code. And the few that call methods the called 
methods are under 100 lines total. 

They mostly just assign values to fields like:

Case of 
   : (Trigger event=On Saving New Record Event)
  If ([CaseActivityNotes]CreateDate=!00/00/!)
 [CaseActivityNotes]CreateDate:=Current date
  End if 
  If ([CaseActivityNotes]CreateTime=?00:00:00?)
 [CaseActivityNotes]CreateTime:=Current time
  End if 
  If ([CaseActivityNotes]CreatedBy="")
 [CaseActivityNotes]CreatedBy:=Current user
  End if 

   : (Trigger event=On Saving Existing Record Event)
  [CaseActivityNotes]UpdateDate:=Current date
  [CaseActivityNotes]UpdateTime:=Current time
  [CaseActivityNotes]UpdatedBy:=Current user
End case 

AddRecordChangesLogEntry2 (Trigger 
event;->[CaseActivityNotes]CaseNumber;->[Patients])

Does that appear “complex” to you? 

I like to guarantee that some field get exactly what I want in them regardless 
of who or what changes a record. Seems like a proper use of triggers.

> Not necessarily applicable in your case, but do you have a way of storing
> summary data for reporting purposes that is accurate (enough) for you
> needs? I'm big on well-normalized structures (ever more so)...but I don't
> see all tables as relational. Lots of data analysis tools use very
> different topologies and rules for the purposes of reporting. I don't mix
> extract data and live data in the same tables, but I definitely have tables
> that are pre-renderings of complex calculations. That's about the only way
> to make some reports feasible in, for example, a Web environments. Lots of
> us have data that is 100% static - like past quarterly data. It isn't even
> allowed to change. Grind it up and boil it down!

In this case it would be difficult to summarize the data in all the possible 
query options users have.. It could be done but you’d have to store so many 
different totals for so many different option combinations s it just seemed 
like more work than it was worth for the time saved. I’m not talking about 
saving hours to do a report, I’m down to trying to shave off a few minutes. 

And with normalized data that’s one less stored procedure that has to run every 
night to update the totals. I’m sure E.F. Codd is smiling right now. :)

> Cool, please report back on what you find. There will also be a break-even
> point for the extra hassle of marshalling all of those services, so  you
> may get some real-world sensitivity for that.

The good news these are regularly run reports so we know how long they take on 
average. I can easily put in some debug code to let me run the same report 
calling the old method and the new method that is preemptive. And this is 
really just a way for me to try out this new preemptive stuff. I always like to 
program with a purpose. 

Tim


Tim Nevels
Innovative Solutions
785-749-3444
timnev...@mac.com


**
4D Internet Users Group (4D iNUG)
FAQ:  http://lists.4d.com/faqnug.html
Archive:  http://lists.4d.com/archives.html
Options: http://lists.4d.com/mailman/options/4d_tech
Unsub:  mailto:4d_tech-unsubscr...@lists.4d.com
**

Re: Survey on 4D’s strategy for implementing Preemptive Processing / worker processes

[Neil Dennis]

So if you are going to write it twice (once for preemptive and once for not) 
separate the methods and you are there….


Method1

$Find_i:=find in array(<>aPerson_ID;$PersonID)
$TheName:=<>aPerson_Name{$Find_i}

Method1Preemptive
Query([person];[Person]id=$PersonID)
$TheName:=[Person]Name

Of course you would need to duplicate the surrounding code as well if any. But 
if you are planning on maintaining two code bases, your worker can just use the 
query and do whatever else it needs. Your regular process code can use your 
interprocess cache. Also if you don’t let your worker die, the worker can load 
the cache once and store them as process vars. Process vars wouldn’t need to be 
loaded each time as with regular processes. There are lots of easy ways to make 
this work other than allowing interprocess vars into a method you want to call 
preemptively.

My survey opinion is still unchanged… for what workers are designed to do the 
limitations are not a huge burden. Most uses I see for preemptive processes are 
for server jobs, report writing, imports, exports, syncing between databases 
etc.


Neil
--
Neil Dennis
4D Developer since 1990

GreaText - Designing Software for the Way You Work
716 East 1850 N
North Ogden, UT 84414

mailto:npden...@greatext.com
http://www.greatext.com/



**
4D Internet Users Group (4D iNUG)
FAQ:  http://lists.4d.com/faqnug.html
Archive:  http://lists.4d.com/archives.html
Options: http://lists.4d.com/mailman/options/4d_tech
Unsub:  mailto:4d_tech-unsubscr...@lists.4d.com
**

Re: Survey on 4D’s strategy for implementing Preemptive Processing / worker processes

[David Adams]

Memory-based caches are great and it is frustrating to have to resort to
building them scratch. How much speed difference will it make? I have no
idea, and it depends on what you're caching, where you're caching & how
much it costs to load the cache. I was recently working with PHP (on its
own, not via 4D) for a spell and got a bit frustrated at the CGI-like
processing model:

* Invoke PHP script.
* It runs with a blank slate.
* Set things up.
* Go die now.

This approach makes sense, given PHP's roots, and it has some
advantages...or can be seen to have advantages. Anyway, I have some static
lookup arrays that I would like to cache. They let me translated
user-friendly labels into stable record IDs for various callbacks. So,
imagine a pair of arrays and a simple lookup on a unique string that
returns a unique number. Options:

1) Cache them in a disk file and reload from disk when the PHP process
launches.
2) Build the values right into an array in the script so that it loads at
startup.
3) Make a database connection and fetch the necessary data.
4) Use an in-memory cache, like Memcached.

I tried testing out a bunch of solutions that were workable in my case and
was surprised to find that #2 was very competitive. I pre-build the array
declarations into the script and then they're right there. (I used 4D to
build the script.)  #1 brings hassles and going to the file system is
typically "slow". Setting up a DB connection is, likewise, a bit slow and
expensive for a quick lookup. I didn't try Memcached, but would if I had a
good production reason for it.

The moral of the story is that performance depends on context and the only
way to find out what something costs is to measure it. In your case, it may
be that reloading the data isn't as expensive as you fear. Or it might be
bad, there's no way of knowing a priori.

Note: The PHP discussion above wasn't just me wandering around and not
getting to the point. I think ;-) Going back to the #2 solution, I've got a
PHP script that does some work - that's all written as normal PHP. Then
I've got some lookup values that belong in the script. They're static once
the script is built, but they need to be injected from 4D. So there's a
template:

* Load template.
* Inject dynamic data from 4D into the PHP script as array definitions.
* Copy script to target folder. Now it's stable until rebuilt.

Not every situation allows for such an approach, but we're already talking
about ones that do. The very same idea is totally practical in 4D:

* Define a method template or use a special comment marker in the method
you want to inject static values into.
* During your build cycle, include a step to update whatever methods you
need with METHOD SET CODE.
* Build and deploy.

I use a screen to do a bunch of stuff before and after building and it
would be easy to add 'inject static assignments' as a step. Why not? METHOD
SET CODE is great, for those that haven't used it. So, your binding is
right before compile time, but definitely before runtime. Earlier binding =
good news, for the most part.
**
4D Internet Users Group (4D iNUG)
FAQ:  http://lists.4d.com/faqnug.html
Archive:  http://lists.4d.com/archives.html
Options: http://lists.4d.com/mailman/options/4d_tech
Unsub:  mailto:4d_tech-unsubscr...@lists.4d.com
**

Re: Survey on 4D’s strategy for implementing Preemptive Processing / worker processes

[John DeSoi]

> On Nov 1, 2016, at 2:03 PM, Tony Ringsmuth  wrote:
> 
> QUESTION #1
>   In light of the current restrictions, how likely are you to leverage 
> the use of Preemptive Processing in the near future? 
>   NOT LIKELY / SOMEWHAT LIKELY / VERY LIKELY


Not likely because a method and all dependents must only use only preemptive 
commands. It seems this feature is only going to be useful in cases where you 
can write something from scratch to be preemptive and also be willing to 
maintain separate methods if you need other features. For example, I have a 
NET_Send method which can send a blob via a plugin call or HTTP. In order to be 
preemptive, I now have to move the HTTP client call to a different method and 
then rewrite all of the callers down from the top level process (likely 
duplicating many methods), just to have a process that works in preemptive mode.

> 3: 4D should NOT prohibit the use of methods in Preemptive processes based on 
> any content therein; however, at runtime, if a method actually executes a 
> prohibited command or pluggin: 4D would generate an error.
>   So, you could code like:
>   IF($InPreemptive_b)
>   `Do something safe for Preemptive
>   ELSE
>   `Use some command that is NOT Preemptive compliant
>   END IF
> 
> QUESTION #2
> If this strategy was implemented, how likely would you be to leverage the use 
> of Preemptive Processing in your applications?

I would be much more likely to use preemption if the above was possible. I 
think the proposal about changing the nature of IP variables is problematic and 
should be addressed in some other way. 


John DeSoi, Ph.D.
**
4D Internet Users Group (4D iNUG)
FAQ:  http://lists.4d.com/faqnug.html
Archive:  http://lists.4d.com/archives.html
Options: http://lists.4d.com/mailman/options/4d_tech
Unsub:  mailto:4d_tech-unsubscr...@lists.4d.com
**

Re: Survey on 4D’s strategy for implementing Preemptive Processing / worker processes

[Tony Ringsmuth]

Neil

>  If it works for preemptive then it should work in all cases. Two cases means 
> double maintenance, and double testing
Currently, I have a lot of places that get cached values out of IP variables 
that I load on startup.  I want my cooperative process to be able to continue 
to take advantage of that speed.
However, my Preemptive processes, instead, would have to query and load every 
time.
My example was very simplistic:  so think this:

IF($IsPreemptive)
Query([person];[Person]id=$PersonID)
$TheName:=[Person]Name
ELSE
$Find_i:=find in array(<>aPerson_ID;$PersonID)
$TheName:=<>aPerson_Name{$Find_i}
END IF

This way, I don’t have to kill the speed of my Cooperative processes to benefit 
my Preemptive processes.

Tony





On 11/1/16, 7:31 PM, "4D_Tech on behalf of Neil Dennis" 
<4d_tech-boun...@lists.4d.com on behalf of npden...@greatext.com> wrote:

> If($IsPreemptive)
>  $GetSomeValue:=vMyValue
> ELSE
>  $GetSomeValue:=<>MyIpVar
> END IF

So if you have to visit the code to make it work both ways why not just 
simply use

 $GetSomeValue:=vMyValue

If it works for preemptive then it should work in all cases. Two cases 
means double maintenance, and double testing… I’m missing the advantage of the 
extra code.


Neil
--
Neil Dennis
4D Developer since 1990

GreaText - Designing Software for the Way You Work
716 East 1850 N
North Ogden, UT 84414

mailto:npden...@greatext.com
http://www.greatext.com/

**
4D Internet Users Group (4D iNUG)
FAQ:  http://lists.4d.com/faqnug.html
Archive:  http://lists.4d.com/archives.html
Options: http://lists.4d.com/mailman/options/4d_tech
Unsub:  mailto:4d_tech-unsubscr...@lists.4d.com
**


**
4D Internet Users Group (4D iNUG)
FAQ:  http://lists.4d.com/faqnug.html
Archive:  http://lists.4d.com/archives.html
Options: http://lists.4d.com/mailman/options/4d_tech
Unsub:  mailto:4d_tech-unsubscr...@lists.4d.com
**

Re: Survey on 4D’s strategy for implementing Preemptive Processing / worker processes

[Neil Dennis]

> If($IsPreemptive)
>  $GetSomeValue:=vMyValue
> ELSE
>  $GetSomeValue:=<>MyIpVar
> END IF

So if you have to visit the code to make it work both ways why not just simply 
use

 $GetSomeValue:=vMyValue

If it works for preemptive then it should work in all cases. Two cases means 
double maintenance, and double testing… I’m missing the advantage of the extra 
code.


Neil
--
Neil Dennis
4D Developer since 1990

GreaText - Designing Software for the Way You Work
716 East 1850 N
North Ogden, UT 84414

mailto:npden...@greatext.com
http://www.greatext.com/

**
4D Internet Users Group (4D iNUG)
FAQ:  http://lists.4d.com/faqnug.html
Archive:  http://lists.4d.com/archives.html
Options: http://lists.4d.com/mailman/options/4d_tech
Unsub:  mailto:4d_tech-unsubscr...@lists.4d.com
**

Re: Survey on 4D’s strategy for implementing Preemptive Processing / worker processes

[Tony Ringsmuth]

OK:  I can see that my idea of Interprocess variables getting a separate 
instance in Preemptive processes is not popular with several of you.

Consider then, a middle ground:
Currently, 4D is trying to get around range-checking type overhead within 
Preemptive processes by doing their checking at compile time.  Problem here is 
that WHOLE METHODS are then restricted based on any restricted content, or 
sub-content. THIS IS WHAT REALY IRKS ME.

Here’s PROPOSAL #2:
Restrict only the code-lines that contain restricted items, including 
inter-process variables, instead of entire methods.

So, your method could not contain:
If($IsPreemptive)
  $GetSomeValue:=vMyValue
ELSE
  $GetSomeValue:=<>MyIpVar
END IF

Internally, 4D compiler would have to put cod around that line, similar to 
range checking;  but, unlike Range checking; as long as your code never hits 
that line (within a preemptive process), execution is never slowed down.  If 
you do hit that line, 4D generates an error, and does NOT run the offending 
line.

The code should really not be any slower, however, it may be somewhat larger, 
when compiled.

Tony


**
4D Internet Users Group (4D iNUG)
FAQ:  http://lists.4d.com/faqnug.html
Archive:  http://lists.4d.com/archives.html
Options: http://lists.4d.com/mailman/options/4d_tech
Unsub:  mailto:4d_tech-unsubscr...@lists.4d.com
**

Re: Survey on 4D’s strategy for implementing Preemptive Processing / worker processes

[David Adams]

On Wed, Nov 2, 2016 at 10:00 AM, Tim Nevels  wrote:
On Nov 1, 2016, at 4:49 PM, David Adams wrote:

> There's a note about docrefs being shared in cooperative threads and not
> shared in preemptive threads. I'm not at all clear what that means. I
would
> expect you would pass paths anyway...isn't that what everyone does now
> anyway?

>> I think this means you do something like:

>> <>docRef_h:=Open document(“logfile.txt”)

>> in one process. Then in another process you can do:

>> SEND PACKET(<>docRef_h;”a message for the log file”)

>> So you open the log file once in your On Startup method.
>> Then from every process that is running you can just do a
>> SEND PACKET to the already open file.

Who knew? Great trick. A good alternative (but more complicated) is to use
a memory-based message queue and send all of the log data to a single
process that has the file open. This is a perfect use of CALL WORKER.

> Of course the change is easy. Just use APPEND TO DOCUMENT command and,
> as you said, pass a file path instead of a DocRef to the worker.

Hmm. That might cause a lot of thrashing in the file system. You could just
pass the message and let the CALL WORKER keep the file open exclusively
during it's lifetime.

> I can see how each preemptive worker would not have access to open files
in other processes, cooperative or preemptive.

4D's delegating/using the OS so there are limits on what it can do. Real
limits that they can't work around. It sounds like they're sheltering us
from a whole lot of hassles in a good way.

> I have a client that I know for sure will upgrade to v16 early next year.
The very first thing I will do when I upgrade to
> v16 is make sure that all the triggers are running preemptive. That’s
step #1. Get those triggers to run as fast as possible
>  on the server. Everyone should benefit from that.

I gave up on complex triggers in 4D years and versions ago ;-(

> Step #2 will be to try and redesign some of the queries and reports that
now take many seconds to run. I’ll break the
> queries and report calculations into pieces and do each piece in a worker
and then at the end combine the results. Now it
> all runs in 1 core. Maybe breaking it into 4 parts and running it in 4
cores simultaneously will result in faster execution
> time.
Not necessarily applicable in your case, but do you have a way of storing
summary data for reporting purposes that is accurate (enough) for you
needs? I'm big on well-normalized structures (ever more so)...but I don't
see all tables as relational. Lots of data analysis tools use very
different topologies and rules for the purposes of reporting. I don't mix
extract data and live data in the same tables, but I definitely have tables
that are pre-renderings of complex calculations. That's about the only way
to make some reports feasible in, for example, a Web environments. Lots of
us have data that is 100% static - like past quarterly data. It isn't even
allowed to change. Grind it up and boil it down!

> So the idea is the “CalculateTotals” method at the beginning of the
report method that builds a bunch of arrays of totals
> will be rewritten to be preemptive. The rest of the report method will
not need to be changed. “CalculateTotals” is where
> all the work will be done. I’ll set it to “execute on server”. Then split
things into 4 pieces, call 4 workers, wait for
> them all the finish and then combine everything into a single set of
arrays.

Cool, please report back on what you find. There will also be a break-even
point for the extra hassle of marshalling all of those services, so  you
may get some real-world sensitivity for that.
**
4D Internet Users Group (4D iNUG)
FAQ:  http://lists.4d.com/faqnug.html
Archive:  http://lists.4d.com/archives.html
Options: http://lists.4d.com/mailman/options/4d_tech
Unsub:  mailto:4d_tech-unsubscr...@lists.4d.com
**

Re: Survey on 4D’s strategy for implementing Preemptive Processing / worker processes

[Tim Nevels]

On Nov 1, 2016, at 4:49 PM, David Adams wrote:

> There's a note about docrefs being shared in cooperative threads and not
> shared in preemptive threads. I'm not at all clear what that means. I would
> expect you would pass paths anyway...isn't that what everyone does now
> anyway?

I think this means you do something like:

<>docRef_h:=Open document(“logfile.txt”)

in one process. Then in another process you can do:

SEND PACKET(<>docRef_h;”a message for the log file”)

So you open the log file once in your On Startup method. Then from every 
process that is running you can just do a SEND PACKET to the already open file. 

Of course the change is easy. Just use APPEND TO DOCUMENT command and, as you 
said, pass a file path instead of a DocRef to the worker.

I can see how each preemptive worker would not have access to open files in 
other processes, cooperative or preemptive. The preemptive thread is almost 
like a separate application and if it tried to write to that <>docRef_h it 
would fail because it would not have context. The data structure behind 
<>docRef_h for a file on disk is unknown. The worker process is living in its 
own world and has no access to 4D things that are shared among processes. 

Other examples of a shared things that any process — and even components — can 
access are hierarchical list references and menu references. (I actually didn’t 
know that document references were shared among processes. I’ve never utilized 
that “feature”.)

> Anyway, that's all pretty deep in the weeds so I'll go back to my original
> question: What problems are people having that server-side, preemptive
> threads going to solve? I'll be very curious to hear about examples. And
> I'll also immediately wonder if those problems can't be solved with a
> different architecture using any version of 4D. Maybe not a better
> solution, but good to have in the mix. (Multiple machine redundant
> processing to good results has been around as long as 4D Server itself.) If
> you are served better by a different architecture, it makes a big
> difference in how you approach interprocess/inter-machine messaging, so
> it's worth thinking through.


I have a client that I know for sure will upgrade to v16 early next year. The 
very first thing I will do when I upgrade to v16 is make sure that all the 
triggers are running preemptive. That’s step #1. Get those triggers to run as 
fast as possible on the server. Everyone should benefit from that. 

Step #2 will be to try and redesign some of the queries and reports that now 
take many seconds to run. I’ll break the queries and report calculations into 
pieces and do each piece in a worker and then at the end combine the results. 
Now it all runs in 1 core. Maybe breaking it into 4 parts and running it in 4 
cores simultaneously will result in faster execution time. 

So the idea is the “CalculateTotals” method at the beginning of the report 
method that builds a bunch of arrays of totals will be rewritten to be 
preemptive. The rest of the report method will not need to be changed. 
“CalculateTotals” is where all the work will be done. I’ll set it to “execute 
on server”. Then split things into 4 pieces, call 4 workers, wait for them all 
the finish and then combine everything into a single set of arrays. 

Tim


Tim Nevels
Innovative Solutions
785-749-3444
timnev...@mac.com


**
4D Internet Users Group (4D iNUG)
FAQ:  http://lists.4d.com/faqnug.html
Archive:  http://lists.4d.com/archives.html
Options: http://lists.4d.com/mailman/options/4d_tech
Unsub:  mailto:4d_tech-unsubscr...@lists.4d.com
**

Re: Survey on 4D’s strategy for implementing Preemptive Processing / worker processes

[Cannon Smith]

I’m sorry, Tony, but I’m not a fan of the idea of IP variables behaving 
differently in different contexts. I haven’t actually used preemptive processes 
yet, but I’ve been studying them as I plan to use them in a few different 
contexts in a new project I’m on once I get to that point.

I’ve been looking at where I would like to use IP variables in preemptive 
processes. In pretty much every case I have some information (usually parallel 
arrays or a c_object) that I fill once at startup from somewhere (a file, 
tables, the internet, calculated, etc.) that I need access to in other 
processes. But these IP variables are never changed after startup. Does this 
align with what other people do?

I’ve thought of two possible solutions. The first is simply to cache this 
information in the datafile somewhere and pull from there when needed. 
Hopefully it is in 4D's cache and will be quick to access. This assumes we are 
on the server which is true for now, but won’t be true if/when 4D allows 
preemptive processes on the client.

The other solution would be to have 4D introduce the idea of a static variable. 
So if I declared an interprocess variable the normal way, it would not be 
allowed in preemptive processes. But if it was declared as static, then it is 
only read only for any process once it is set so I assume that a preemptive 
process could read this variable without issue. This would solve most of my 
issues with interprocess variables in preemptive processes.

I do agree that it would be nice if 4D threw an error at runtime instead of 
relying on compile time checks if an unsafe command was used in a preemptive 
process. At least, I agree with that as long as doing it this way isn’t slower. 
I guess that depends on how the internal checks are made. But that would make 
it possible to have different code paths in the same method.

Finally, I actually wish that it was possible to surround thread unsafe code in 
a semaphore in a preemptive process. This should be discouraged, but I don’t 
see why it should be completely disallowed. There are exceptions to every rule, 
it seems, and I think there will be cases where it is worth the slight slow 
down in code to run some thread unsafe code.

But my biggest hope is that we can have the concept of static-ly defined 
variables, especially arrays and objects. Would this solve similar issues for 
other people?

--
Cannon Smith
Synergy Farm Solutions Inc.
Hill Spring, AB Canada
403-626-3236



**
4D Internet Users Group (4D iNUG)
FAQ:  http://lists.4d.com/faqnug.html
Archive:  http://lists.4d.com/archives.html
Options: http://lists.4d.com/mailman/options/4d_tech
Unsub:  mailto:4d_tech-unsubscr...@lists.4d.com
**

Re: Survey on 4D’s strategy for implementing Preemptive Processing / worker processes

[David Adams]

I'm not going to answer either of Tony's questions as I don't feel
qualified to say one way or another. I'll admit that I've paid almost no
attention to the preemptive/cooperative subject in upcoming versions of 4D.
Can those of you who have tried it all out and gotten some experience
provide a bit of background? Specifically: What problems does this help
with in your systems. I can easily see why 4D has put a lot of effort into
improving threading in their server code, what I'm not clear on is which
problem(s) or classes of problems this will help with meaningfully that
fall into our hands. I'm not suggesting there aren't problems like this, I
just don't know what they are.

And, after admitting my complete ignorance, I'll boldly go forward with a
couple of opinions from Tony's detailed suggestions. First off, the
IP-variables-that-are-process variables idea: I hate it. For the very
simple reason that it's fundamentally confusing, unless you understand it
already. There are or have been a few areas in 4D where you had to
carefully think through your execution context to understand how the could
would and should operate:

* Triggers
* Stored procedures
* Web of any kind
* Web contextual mode in particular
* Methods marked as 'execute on server'

4D normally does a masterful job of sheltering us from the difference
amongst various platforms and contexts. It's a wonderful thing to be able
to take for granted. Kudos. But when the context matters, *you can't see it
from the code.* You need prior knowledge and experience. That's expensive.
Also, if history is anything to go by, the documentation has always been
poor or absent on these subjects. The exceptional nature of triggers was
never well documented by 4D, and then it changed dramatically, the whole
dual-threads-for-client-connetions business has *never* been documented by
4D (outside of a conference) as far as I know, and so on. Even if there are
some docs on some of these subjects (and there are), they're pretty thin
for such important subjects. As a result, people tend to get pretty messed
up by these various parts of the language and consequently avoid using
them. Not a great outcome. If IP variables mean "IP, unless you're in this
one special kind of process, where they are process vars"...ummm. That's
confusing. Think about the scope of sets, another example of weak
documentation and misleading/confusing naming prefixes. Not helpful. And
while an experienced, dedicated 4D programmer might well learn all of the
kinks of these various exceptional parts of the language, a new or
occasional programmer would have no chance at all. Welcome to the new
person messing up your code! It's easy to forget, but younger programmers
tend to use a blend of languages for various tasks and may not learn or
retain the nuances of each one.

I also don't get the obsession with IP variables as I don't need a lot of
them. I don't know why they seem so important. I also never missed having
client variables in triggers. I strongly suspect that people have become
used to using IP variables for inter-process communications because 4D
didn't have any native alternative. IP variables
are not a messaging system and they're a terrible way to pass data
routinely between processes. Seriously, the
only thing worse is SET PROCESS VARIABLE which is just pathological. If you
need to pass messages, pass messages! NTK has given us a nice way to do
this for, oh, forever? And CALL WORKER also provides a very nice-looking
native method. You get a sequential message queue with no fuss. Sweet. No
semaphores, not IP locking, no nonesense. Easy, safe, lovely. It's only for
one-machine communications, but that's a huge start.

Okay, maybe I'm lying - maybe I do have a lot of IP vars. I just checked
and in a system with just shy of 6,000 methods I've got 152 IP arrays and
100 IP variables. A lot of those could be replaced very easily, a few I
would miss. Is that a lot? It doesn't feel like a lot, but "a lot" is a
relative term.

[Later] Thanks to Tony for linking to the relevant docs:
http://livedoc.4d.com/4D-Language-Reference-16/Processes/Preemptive-4D-processes.300-3036340.en.html

Hmm. No support on client machines. Hmmm. So, we're talking compiled,
server-side/single-user execution. That makes the restrictions make a whole
lot more sense to me. If this is so, and the tasks you need to get more
cores on are client-side in nature, there's a pretty strong argument for
using compiled-and-merged single-user apps that communicate with the server
using HTTP calls (etc.) At that point, you aren't going to use IP vars or
CALL WORKER from the server in any case. For what it's worth, that model
gets you more 'cores' because you're using a distributed architecture with
more machines.

Anyway, back to the docs. Right, no plug-ins makes sense, no IP vars makes
sense. You can use the network (HTTP Get) or file system (Create document,
TEXT TO DOCUMENT, etc.), so that's how you message the outside world. And

Re: Survey on 4D’s strategy for implementing Preemptive Processing / worker processes

[Peter Bozek]

On Tue, Nov 1, 2016 at 10:22 PM, Tony Ringsmuth  wrote:

> > Are you saying that will not be allowed and thus the StandardTriggerCode
> method cannot run preemptively?
> Yes, If you have ANY trigger in the whole database that references ANY
> restricted item (IP vars, Plugins, Various 4D commands or any method that
> does), then correct: StandardTriggerCode cannot run preemptively.
> (pretty horrible;  isn’t it)


As I understand it, if StandardTriggerCode runs inside a trigger, it can
run preemptively (and trigger and code that invokes trigger with SAVE
RECORD can run preemptively as well.)

Problem would be if you had a wrapper around SAVE RECORD and some trigger
cannot run preemptively - then wrapper cannot run preemptively (as 4D
cannot know record in which table will be saved inside the wrapper.)

But there is no way around this.

-- 

Peter Bozek
**
4D Internet Users Group (4D iNUG)
FAQ:  http://lists.4d.com/faqnug.html
Archive:  http://lists.4d.com/archives.html
Options: http://lists.4d.com/mailman/options/4d_tech
Unsub:  mailto:4d_tech-unsubscr...@lists.4d.com
**

Re: Survey on 4D’s strategy for implementing Preemptive Processing / worker processes

[Tony Ringsmuth]

Tim,

Here’s the section where you can read up on SAVE RECORD and pointers:
http://livedoc.4d.com/4D-Language-Reference-16/Processes/Preemptive-4D-processes.300-3036340.en.html

Read the section under triggers.

> Are you saying that will not be allowed and thus the StandardTriggerCode 
> method cannot run preemptively?  
Yes, If you have ANY trigger in the whole database that references ANY 
restricted item (IP vars, Plugins, Various 4D commands or any method that 
does), then correct: StandardTriggerCode cannot run preemptively.
(pretty horrible;  isn’t it)

Tony


On 11/1/16, 4:15 PM, "4D_Tech on behalf of Tim Nevels" 
<4d_tech-boun...@lists.4d.com on behalf of timnev...@mac.com> wrote:

On Nov 1, 2016, at 3:35 PM, Tony Ringsmuth wrote:

> CURRENT RESTRICTIONS to using preemptive processes (up through the 
upcoming release of v16):
> 1: Any method that uses any restricted items cannot be called 
preemptively.  Restricted items include:
>   A: Interprocess variables
>   B: Plugins
>   C: Various 4D commands
>   D: Any component method that is not Preemptive save
> 2: No method that calls any other method that has restricted items can be 
called preemptively
> 3: Any method that saves records via pointers to a table is restricted 
(If you have any Table Trigger in the whole database that has any restricted 
items (as listed in #1 or #2))
>   Example: SAVE RECORD($MyTable->)
> 
> QUESTION #1
>   In light of the current restrictions, how likely are you to 
leverage the use of Preemptive Processing in the near future? 

NOT LIKELY because of table pointer usage. I was not aware of this 
restriction. Sounds like a serious limitation. I was not aware of pointer 
restrictions. 

Can you reference documentation talking about this? I’d like to read more 
about the restrictions on using pointers in preemptive code.

I have trigger code like “StandardTriggerCode( ->[Table1]UpdatedByUserName 
; … )”  Are you saying that will not be allowed and thus the 
StandardTriggerCode method cannot run preemptively?  


>   NOT LIKELY / SOMEWHAT LIKELY / VERY LIKELY
> 
>   YOUR COMMENTS:
> 
> 
> IF your answer to #1 is “NO” , then consider this alternate 
implementation. 
> 
> TONY’S PROPOSED ALTERNATE STRATEGY:
> 1: Each Preemptive worker process should have it’s own complete 
instantiation of IP Variables; similar to it’s set of Process variables.  The 
scope and lifespan of those IP variables would be limited to that worker 
process, and that worker process only.  It would be up to the developer to 
populate those IP variables, or do with them as he wishes.
>   As a ~possible~ option to this:  perhaps we could have an option 
parameter to indicate to 4D to copy all IP variables from the current process 
to the new worker process
> 
> 2: IP variables would then become a non-restricted item, and can be 
freely used in Preemptive processes.

So your idea is the make IP variables behave differently in preemptive 
processes. They look and appear to be IP variables but in fact they are not. 
They are really process variables, but they have an IP variable name. I don’t 
like that at all. I think we need a better solution than this. This is really 
an ugly “hack” that will make reading 4D code difficult to understand. Is this 
an IP variable that only my worker process can change, or this is an IP 
variable that all other processes can also access and change.

And since a method can run as “indifferent” meaning it can run as either 
either “normal” or as “preemptive” you would need to actually have 2 different 
versions of the method: compiled like now for “normal” execution and another 
copy compiled that does all this weird stuff to make IP variable still work. I 
don’t like this either. 

I can understand the strong desire to reuse existing code and make as few 
changes as possible to get it to run preemptively. You have existing code that 
is filled with IP variables. Now you want to make it run without rewriting it 
to not use IP variables. 

I have an idea. Have a compatibility setting to allow iP variables in 
preemptive processes. 

If you turn this on then 4D will do all kinds of low level magic when it 
compiles. If a method it is running preemptively it does all kinds of locking 
and semaphore stuff needed to access IP variables. It will slow down your 
preemptive code but at least you can still run with IP variables. Another big 
problem is deadlock conditions with multiple workers/processes trying to access 
and update the same IP variable. 

> 3: 4D should NOT prohibit the use of methods in Preemptive processes 
based on any content therein; however, at runtime, if a method actually 
executes a prohibited command or pluggin: 4D would generate an error.
>   So, you could code like:
>   IF($InPreemptive_b)
>   

Re: Survey on 4D’s strategy for implementing Preemptive Processing / worker processes

[Peter Bozek]

On Tue, Nov 1, 2016 at 9:36 PM, Tony Ringsmuth  wrote:

> As 4D is implementing it;  your are correct.
> But the goal should be to Provide a path for
> - The best possible performance gain
> - With the least required amount of re-development
> I think they may be succeeding on #1, but totally failing on #2.  I think
> it can be much better.
>

Tony,

Yes, I understand you want to simplify migration of existing code into
preemptive processes. From my point of view, developer will have to do
quite a lot of changes, so helping with some particular issues may not help
much - may be even counterproductive.

Just an example -  interprocess variables: they are used across processes,
that's why they are interprocess. They cannot be used for the same purpose
in preemptive process. Developer should check each code that use them
because that code may need to be rewritten - so he can as well change them
to process variables during that process.

Regarding your question - I plan to use preemptive processes, but am not
sure if I will convert any existing code to preemptive process, so your
question #2 may not apply.

-- 

Peter Bozek
**
4D Internet Users Group (4D iNUG)
FAQ:  http://lists.4d.com/faqnug.html
Archive:  http://lists.4d.com/archives.html
Options: http://lists.4d.com/mailman/options/4d_tech
Unsub:  mailto:4d_tech-unsubscr...@lists.4d.com
**

Re: Survey on 4D’s strategy for implementing Preemptive Processing / worker processes

[Tony Ringsmuth]

Jody,

Regarding #3:  Pointers in SAVE RECORD…

This ½ of my biggest frustration:  The other half is that if a method contains 
any restricted item, then the whole method cannot be used. These are what kill 
it for me.
This “All or nothing” of a method (and any sub-dependency) is the killer.  It 
eliminates range-checking speed loss, but makes the re-coding effort really 
really bad.

I would be OK if we simply could not execute any lines of code that reference 
an IP variable.  But that, again, would require 4D to do more range-checking 
type stuff.

So, my suggestion becomes: Just make a separate instance of IP Vars.  Then we 
don’t lose any speed on runtime-checking, AND, we have FAR FAR less re-coding 
to do.

Regards,
Tony


On 11/1/16, 3:34 PM, "4D_Tech on behalf of G-Mail" 
<4d_tech-boun...@lists.4d.com on behalf of jody.be...@gmail.com> wrote:

Tony:

> QUESTION #1

 VERY LIKELY  (We will)

With the understanding of some frustration we would have, and understanding 
of what may be their reasons, we would use them because of the advantages of 
them being preemptive.

The item I do not like is item #3. How can I code without pointers ;-)

Thus far, I have not spent much time on how we will change our coding to 
handle multiple methods in a worker process accessing an object that we would 
normally give an interprocess scope. I know we could use process variables for 
the objects to achieve what we need. I just hate ‘polluting’ the rest of the 
processes with these process variables needed for the Worker process. At the 
same time, I understand their need to not have to deal with IP variables being 
accessed from other non-worker processes.

We have not run into the need of a plugin for a worker process, but I could 
see where it could be nice to have. I do understand that at this point they may 
want to just not permit them as an easy solution at this time.

I can see your point you make about writing a method If(Preemptive)…. Else 
Do This. It makes maintaining our code easier.

Of course this is just where we are at with our code and clients.


> On 11/01/2016, at 1:03 PM, Tony Ringsmuth  wrote:
> 
> Greetings
> 
> I’m on a quest to persuade 4D to adopt a more useful implementation of 
preemptive processing than they are currently adopting. 
> Please answer questions #1 and #2 below if you can.
> 
> CURRENT RESTRICTIONS to using preemptive processes (up through the 
upcoming release of v16):
> 1: Any method that uses any restricted items cannot be called 
preemptively.  Restricted items include:
>   A: Interprocess variables
>   B: Plugins
>   C: Various 4D commands
>   D: Any component method that is not Preemptive save
> 2: No method that calls any other method that has restricted items can be 
called preemptively
> 3: Any method that saves records via pointers to a table is restricted 
(If you have any Table Trigger in the whole database that has any restricted 
items (as listed in #1 or #2))
>   Example: SAVE RECORD($MyTable->)
> 
> QUESTION #1
>   In light of the current restrictions, how likely are you to 
leverage the use of Preemptive Processing in the near future? 
>   NOT LIKELY / SOMEWHAT LIKELY / VERY LIKELY
> 
>   YOUR COMMENTS:
> 
> 
> IF your answer to #1 is “NO” , then consider this alternate 
implementation. 
> 
> TONY’S PROPOSED ALTERNATE STRATEGY:
> 1: Each Preemptive worker process should have it’s own complete 
instantiation of IP Variables; similar to it’s set of Process variables.  The 
scope and lifespan of those IP variables would be limited to that worker 
process, and that worker process only.  It would be up to the developer to 
populate those IP variables, or do with them as he wishes.
>   As a ~possible~ option to this:  perhaps we could have an option 
parameter to indicate to 4D to copy all IP variables from the current process 
to the new worker process
> 
> 2: IP variables would then become a non-restricted item, and can be 
freely used in Preemptive processes.
> 
> 3: 4D should NOT prohibit the use of methods in Preemptive processes 
based on any content therein; however, at runtime, if a method actually 
executes a prohibited command or pluggin: 4D would generate an error.
>   So, you could code like:
>   IF($InPreemptive_b)
>   `Do something safe for Preemptive
>   ELSE
>   `Use some command that is NOT Preemptive compliant
>   END IF
> 
> QUESTION #2
> If this strategy was implemented, how likely would you be to leverage the 
use of Preemptive Processing in your applications?
> 
> 
> I appreciate your feedback in this
> Thanks!
> Tony Ringsmuth
> 
> 
> **
> 4D Internet Users 

Re: Survey on 4D’s strategy for implementing Preemptive Processing / worker processes

[Tony Ringsmuth]

Dennis,

You didn’t answer question #1.

>Dennis said: Before getting too cavalier on requesting a change in the 
>implementation... make sure you understand why the restrictions are in place.

Tony’s response:
yes: I understand the reason for the restriction; and I agree, the speed loss 
for preemptive processes trying to share interprocess variables would be 
terrible.
That’s why I’m suggesting that 4D just give every Preemptive Process it’s own 
set:  that way, there’s NO SPEED LOSS.


  >Dennis said:   1 - Then why have inter process variables... just use process 
variables, to copy them would require semaphores thus cutting down the 
performance... I want workers to be faster not slower for my convenience.

Tony’s response:
In the current implementation, I can’t use an entire method, just because that 
method references one IP variable, or a sub-method, or a plugin.
In my proposed method, I just code around (within my method) the lines of code 
that use IP variables or plugins.

>... I want workers to be faster not slower for my convenience

Tony’s response:
I also want them faster.  Creating an isolated instance of IP variables will 
not cost any speed whatsoever (except for the initial time to instantiate them: 
similar to the time to instantiate process vars).


> Dennis said:   2 - What is the purpose of an interposes variable with 
> access to only one process... isn't that just a process variable?
Tony’s response: Basically, yes: it becomes a process variable.
But then we can choose HOW or whether to use it, and can write a simple 
condition around it; rather than having to create an entirely new code base, 
free of any dependency that contains an IP var.

> Dennis said:   3 - I like the checks at compile time to keep 
> multi-threaded code faster.
Great.  So, in what ways have you leveraged it already?


Regards,
Tony


On 11/1/16, 3:05 PM, "4D_Tech on behalf of Dennis, Neil" 
<4d_tech-boun...@lists.4d.com on behalf of neil.den...@umb.com> wrote:

Before getting too cavalier on requesting a change in the implementation... 
make sure you understand why the restrictions are in place. In most places the 
implementation would be too slow to offset the benefit (e.g. putting semaphores 
around inter-process variables), in some cases to protect integrity (e.g. most 
plugins are not thread safe), lastly some cases are impossible for any 
application (e.g. UI)

Anyway, to your survey... I would be much, much more likely to use them if 
they were quick and more efficient vs. slower and easier to implement.

Tony's proposed strategy...
1 - Then why have inter process variables... just use process variables, to 
copy them would require semaphores thus cutting down the performance... I want 
workers to be faster not slower for my convenience.
2 - What is the purpose of an interposes variable with access to only one 
process... isn't that just a process variable?
3 - I like the checks at compile time to keep multi-threaded code faster.

In short I do understand the restrictions and agree with them. I hope they 
don't ever sacrifice performance just to make things a bit easier to code. If I 
want easy I can just use a regular process, if I want quick and scalable I will 
put in a little more effort and use a worker.

Neil






---

Privacy Disclaimer: This message contains confidential information and is 
intended only for the named addressee. If you are not the named addressee you 
should not disseminate, distribute or copy this email. Please delete this email 
from your system and notify the sender immediately by replying to this email.  
If you are not the intended recipient you are notified that disclosing, 
copying, distributing or taking any action in reliance on the contents of this 
information is strictly prohibited.

The Alternative Investments division of UMB Fund Services provides a full 
range of services to hedge funds, funds of funds and private equity funds.  Any 
tax advice in this communication is not intended to be used, and cannot be 
used, by a client or any other person or entity for the purpose of (a) avoiding 
penalties that may be imposed on any taxpayer or (b) promoting, marketing, or 
recommending to another party any matter addressed herein.
**
4D Internet Users Group (4D iNUG)
FAQ:  http://lists.4d.com/faqnug.html
Archive:  http://lists.4d.com/archives.html
Options: http://lists.4d.com/mailman/options/4d_tech
Unsub:  mailto:4d_tech-unsubscr...@lists.4d.com
**


**
4D Internet Users Group (4D iNUG)
FAQ:  http://lists.4d.com/faqnug.html
Archive:  http://lists.4d.com/archives.html
Options: http://lists.4d.com/mailman/options/4d_tech
Unsub:  

RE: Survey on 4D’s strategy for implementing Preemptive Processing / worker processes

[Dennis, Neil]

Before getting too cavalier on requesting a change in the implementation... 
make sure you understand why the restrictions are in place. In most places the 
implementation would be too slow to offset the benefit (e.g. putting semaphores 
around inter-process variables), in some cases to protect integrity (e.g. most 
plugins are not thread safe), lastly some cases are impossible for any 
application (e.g. UI)

Anyway, to your survey... I would be much, much more likely to use them if they 
were quick and more efficient vs. slower and easier to implement.

Tony's proposed strategy...
1 - Then why have inter process variables... just use process variables, to 
copy them would require semaphores thus cutting down the performance... I want 
workers to be faster not slower for my convenience.
2 - What is the purpose of an interposes variable with access to only one 
process... isn't that just a process variable?
3 - I like the checks at compile time to keep multi-threaded code faster.

In short I do understand the restrictions and agree with them. I hope they 
don't ever sacrifice performance just to make things a bit easier to code. If I 
want easy I can just use a regular process, if I want quick and scalable I will 
put in a little more effort and use a worker.

Neil






---


Privacy Disclaimer: This message contains confidential information and is 
intended only for the named addressee. If you are not the named addressee you 
should not disseminate, distribute or copy this email. Please delete this email 
from your system and notify the sender immediately by replying to this email.  
If you are not the intended recipient you are notified that disclosing, 
copying, distributing or taking any action in reliance on the contents of this 
information is strictly prohibited.

The Alternative Investments division of UMB Fund Services provides a full range 
of services to hedge funds, funds of funds and private equity funds.  Any tax 
advice in this communication is not intended to be used, and cannot be used, by 
a client or any other person or entity for the purpose of (a) avoiding 
penalties that may be imposed on any taxpayer or (b) promoting, marketing, or 
recommending to another party any matter addressed herein.
**
4D Internet Users Group (4D iNUG)
FAQ:  http://lists.4d.com/faqnug.html
Archive:  http://lists.4d.com/archives.html
Options: http://lists.4d.com/mailman/options/4d_tech
Unsub:  mailto:4d_tech-unsubscr...@lists.4d.com
**

Re: Survey on 4D’s strategy for implementing Preemptive Processing / worker processes

[Spencer Hinsdale]

Hi Tony,

I love your preemptive enthusiasm!  But I also want you to update to watchOS 3 
so you have the Breath App at hand.

IMO 4D has always, always done a terrific job of migrating us, be it to 
unicode, or out of subtables, or to that new OS.

Do you remember Windows NT?  Microsoft totally abstracted the hardware.  Except 
developers weren't thrilled with the video, so they hardwired that back in.  
Oops.


-spencer

> On Nov 1, 2016, at 12:03 PM, Tony Ringsmuth  wrote:
> 
> 
> I appreciate your feedback in this
> Thanks!
> Tony Ringsmuth

**
4D Internet Users Group (4D iNUG)
FAQ:  http://lists.4d.com/faqnug.html
Archive:  http://lists.4d.com/archives.html
Options: http://lists.4d.com/mailman/options/4d_tech
Unsub:  mailto:4d_tech-unsubscr...@lists.4d.com
**

Re: Survey on 4D’s strategy for implementing Preemptive Processing / worker processes

[Kenneth Geiger]

Hi Tony,

Question 1: Not Likely
Question 2: Likely

Regards,

Ken Geiger

On Tue, Nov 1, 2016 at 1:03 PM, Tony Ringsmuth  wrote:

> Greetings
>
> I’m on a quest to persuade 4D to adopt a more useful implementation of
> preemptive processing than they are currently adopting.
> Please answer questions #1 and #2 below if you can.
>
> CURRENT RESTRICTIONS to using preemptive processes (up through the
> upcoming release of v16):
> 1: Any method that uses any restricted items cannot be called
> preemptively.  Restricted items include:
> A: Interprocess variables
> B: Plugins
> C: Various 4D commands
> D: Any component method that is not Preemptive save
> 2: No method that calls any other method that has restricted items can be
> called preemptively
> 3: Any method that saves records via pointers to a table is restricted (If
> you have any Table Trigger in the whole database that has any restricted
> items (as listed in #1 or #2))
> Example: SAVE RECORD($MyTable->)
>
> QUESTION #1
>In light of the current restrictions, how likely are you to
> leverage the use of Preemptive Processing in the near future?
>NOT LIKELY / SOMEWHAT LIKELY / VERY LIKELY
>
> YOUR COMMENTS:
>
>
> IF your answer to #1 is “NO” , then consider this alternate implementation.
>
> TONY’S PROPOSED ALTERNATE STRATEGY:
> 1: Each Preemptive worker process should have it’s own complete
> instantiation of IP Variables; similar to it’s set of Process variables.
> The scope and lifespan of those IP variables would be limited to that
> worker process, and that worker process only.  It would be up to the
> developer to populate those IP variables, or do with them as he wishes.
> As a ~possible~ option to this:  perhaps we could have an option
> parameter to indicate to 4D to copy all IP variables from the current
> process to the new worker process
>
> 2: IP variables would then become a non-restricted item, and can be freely
> used in Preemptive processes.
>
> 3: 4D should NOT prohibit the use of methods in Preemptive processes based
> on any content therein; however, at runtime, if a method actually executes
> a prohibited command or pluggin: 4D would generate an error.
> So, you could code like:
> IF($InPreemptive_b)
> `Do something safe for Preemptive
> ELSE
> `Use some command that is NOT Preemptive compliant
> END IF
>
> QUESTION #2
> If this strategy was implemented, how likely would you be to leverage the
> use of Preemptive Processing in your applications?
>
>
> I appreciate your feedback in this
> Thanks!
> Tony Ringsmuth
>
>
> **
> 4D Internet Users Group (4D iNUG)
> FAQ:  http://lists.4d.com/faqnug.html
> Archive:  http://lists.4d.com/archives.html
> Options: http://lists.4d.com/mailman/options/4d_tech
> Unsub:  mailto:4d_tech-unsubscr...@lists.4d.com
> **
**
4D Internet Users Group (4D iNUG)
FAQ:  http://lists.4d.com/faqnug.html
Archive:  http://lists.4d.com/archives.html
Options: http://lists.4d.com/mailman/options/4d_tech
Unsub:  mailto:4d_tech-unsubscr...@lists.4d.com
**