Re: 4D Server Hosting - The Future ?

[Two Way Communications via 4D_Tech]

Hi Kirk,

Very interesting.

Here are my experiences on running a 4D v17R5 server, client on Mac and windows:

If you remember my presentation on the 2018 summit (with the SVG charts for 
temperature mapping), that is the one running on it.

First setup: t2.micro on AWS, smallest possible and free ;-)
Pros:
Very reliable
good for prototyping
handling: quite ok, which has surprised me since it has only 1GB RAM ( ! )
Selection to array / array to selection: tested with + 100.000 records, not to 
bad (seconds rather than minutes)
Lists are displayed in listboxes, using fields. Display is rather immediate.
Detail forms load within 2 to 10 seconds, depending on how many queries in it

Cons:
bandwidth not always sufficient, which drops handling to a crawl, and also 
causes big delays when accessing the instance with RDP

2nd setup: m5.large, 8GB RAM, 2 cores. 160$/month
Pros:
Very reliable
much better handling due to guaranteed bandwidth
a *lot* faster than the t2.micro, very workable

I also put 2 RDP users on it, because the server license includes 2 CALS. Works 
even better, even on a Mac!

I studied what it would take to have more RDP users, but there, I get lost a 
little bit.
From what I understood, I would need an RDP gateway, so another AWS instance, 
and then buy additional CALS for RDP.
That sounds quite expensive…

The other thing I tried recently is S3, because one of my applications manages 
a lot (+ 1,000,000) of documents.

There is a 4Dsummit example that manages all the S3 stuff: very impressive!
Storing and retrieving documents has never been so easy and fast. Cost is 
affordable (by EU standards ;-) )

I would love to here other AWS experiences.

Regards,

Rudy Mortier
Two Way Communications bvba 



> On 13 Oct 2019, at 00:51, Kirk Brooks via 4D_Tech <4d_tech@lists.4d.com> 
> wrote:
> 
> Hey Rudy,
> I made a post yesterday on the thread about preemptive processes and how it
> works on VM. Ping me if you can't find it. You may find it useful because I
> posted some actual data from running a demo you can download. I ran it on
> my laptop and an AWS instance. The general specs of the AWS instance are
> there too. It shows the sort of differences I've been seeing. It seems the
> issue with AWS is partly the sheer horsepower you sign up for in terms of
> cores and such but equally important is the amount of bandwidth you commit
> to. Impacts the performance and cost quickly.
> 
> I'm hoping someone with more expertise in this might join the conversation.
> (Balinder? you out there?) It's almost good for directly connecting. I
> think if you are running 4D to power a web server it's quite good. Also, I
> think if you deploy there optimizations you can make in code to
> accommodate the network will become apparent. I quickly saw that places
> where I move data from the server to the client and manipulate it on the
> client are very speedy. ORDA is probably going to help with that. I'm also
> curious if you can set up a situation that cost effective where you run an
> instance for the server and then some others to support clients using RDP
> or the like.
> 
> It is certainly appealing.
> 
> On Sat, Oct 12, 2019 at 2:28 PM Two Way Communications via 4D_Tech <
> 4d_tech@lists.4d.com> wrote:
> 
>> Hi Kirk,
>> 
>> I am very interested on your progress with AWS. Can you tell me a little
>> bit more about the kind of 4D DB you are running on it?
>> Are you running client/server or using SQL? Do you notice speed
>> differences between ORDA / Classic 4D?
>> 
>> 
>> I’m asking because I am currently using AWS myself. The response is quite
>> ok, but still way slower than LAN client server.
>> I am thinking of putting more 4D databases on AWS.
>> 
>> 
>> Regards,
>> 
>> Rudy Mortier
>> Two Way Communications bvba
>> 
>> 
>> 
>>> On 10 Oct 2019, at 17:09, Kirk Brooks via 4D_Tech <4d_tech@lists.4d.com>
>> wrote:
>>> 
>>> Peter,
>>> I am in the process of moving a database from our own hardware to and AWS
>>> instance. It's true that the most expensive part of setting it up, at
>> this
>>> point, is getting the appropriate amount of band width and throughput
>> speed.
>>> 
>>> The other thing about VM vs metal is the whole pre-emptive process
>>> benefit basically goes away. Thomas Maul has shown this at the Summit.
>>> Having n+ virtual cores doesn't do anything to actually increase
>> processing
>>> speed because the VM is running on whatever is allocated to it.
>>> Theoretically you could have a VM with 4 cores running an instance with
>> 32
>>> cores. So preemptive threading is looking to be mainly a benefit for
>>> companies that run their own hardware and for desktop apps.
>>> 
>>> On Thu, Oct 10, 2019 at 5:25 AM Peter Jakobsson via 4D_Tech <
>>> 4d_tech@lists.4d.com> wrote:
>>> 
 Hi !
 
 I just finished a 1.5 hour phonecall with a support services manager at
 the technical services company who supply one of my customers with all
 their hardware/software/maintenance services.
 
>>>

Re: 4D Server Hosting - The Future ?

[Kirk Brooks via 4D_Tech]

Hey Rudy,
I made a post yesterday on the thread about preemptive processes and how it
works on VM. Ping me if you can't find it. You may find it useful because I
posted some actual data from running a demo you can download. I ran it on
my laptop and an AWS instance. The general specs of the AWS instance are
there too. It shows the sort of differences I've been seeing. It seems the
issue with AWS is partly the sheer horsepower you sign up for in terms of
cores and such but equally important is the amount of bandwidth you commit
to. Impacts the performance and cost quickly.

I'm hoping someone with more expertise in this might join the conversation.
(Balinder? you out there?) It's almost good for directly connecting. I
think if you are running 4D to power a web server it's quite good. Also, I
think if you deploy there optimizations you can make in code to
accommodate the network will become apparent. I quickly saw that places
where I move data from the server to the client and manipulate it on the
client are very speedy. ORDA is probably going to help with that. I'm also
curious if you can set up a situation that cost effective where you run an
instance for the server and then some others to support clients using RDP
or the like.

It is certainly appealing.

On Sat, Oct 12, 2019 at 2:28 PM Two Way Communications via 4D_Tech <
4d_tech@lists.4d.com> wrote:

> Hi Kirk,
>
> I am very interested on your progress with AWS. Can you tell me a little
> bit more about the kind of 4D DB you are running on it?
> Are you running client/server or using SQL? Do you notice speed
> differences between ORDA / Classic 4D?
>
>
> I’m asking because I am currently using AWS myself. The response is quite
> ok, but still way slower than LAN client server.
> I am thinking of putting more 4D databases on AWS.
>
>
> Regards,
>
> Rudy Mortier
> Two Way Communications bvba
>
>
>
> > On 10 Oct 2019, at 17:09, Kirk Brooks via 4D_Tech <4d_tech@lists.4d.com>
> wrote:
> >
> > Peter,
> > I am in the process of moving a database from our own hardware to and AWS
> > instance. It's true that the most expensive part of setting it up, at
> this
> > point, is getting the appropriate amount of band width and throughput
> speed.
> >
> > The other thing about VM vs metal is the whole pre-emptive process
> > benefit basically goes away. Thomas Maul has shown this at the Summit.
> > Having n+ virtual cores doesn't do anything to actually increase
> processing
> > speed because the VM is running on whatever is allocated to it.
> > Theoretically you could have a VM with 4 cores running an instance with
> 32
> > cores. So preemptive threading is looking to be mainly a benefit for
> > companies that run their own hardware and for desktop apps.
> >
> > On Thu, Oct 10, 2019 at 5:25 AM Peter Jakobsson via 4D_Tech <
> > 4d_tech@lists.4d.com> wrote:
> >
> >> Hi !
> >>
> >> I just finished a 1.5 hour phonecall with a support services manager at
> >> the technical services company who supply one of my customers with all
> >> their hardware/software/maintenance services.
> >>
> >> He basically brought me up to date on “how things work” today which is
> >> essentially that everything to do with platforms is now virtualised to
> >> allow them to ‘tune’ resources to demand in realtime and provide
> seamless,
> >> no downtime backup. Basically, my 4d Server is now a “cloud service”
> >> without me even being aware of it, it’s just that the hardware involved
> >> happens to be located on the preises.
> >>
> >> In particular we discussed backup configurations for 4D server and this
> >> was interesting because, while I requested independent drives for
> logfile
> >> (“journal”) and datafile purposes, he essentially told me to just stick
> >> everything on the same drive because it was virtual anyway and had
> multiple
> >> redundancy protection via raid, 15-minute snapshotting etc. He offered
> to
> >> “create” a C: and a D: drive to make me feel better, but pointed out
> that
> >> they’re not much more independent than 2 folders would have been.
> >>
> >> CONCLUSON
> >> I now realise that the “WAN” / “LAN” distinction is disappearing. He
> said
> >> the only reason the “cloud” solution wasn’t hosted off-site was that
> they
> >> had measured the bandwidth that the customer used and calculated that
> the
> >> cost would be astronomical if it was on AWS or something like that, but
> in
> >> all other respects it was a cloud solution.
> >>
> >> I was wondering, how do other major 4D server deployers optimise their
> >> deployment strategies to take advantage of this ? It seems a great thing
> >> that we are being “floated out to the cloud” without actually having to
> do
> >> extra significant work, but what about things like the backup strategy
> ? I
> >> don’t really like the idea that the log file has the same redundancy
> system
> >> as the main datafile because the whole idea is that the corruption
> doesn’t
> >> get replicated (which is what a RAID system does) and i

Re: 4D Server Hosting - The Future ?

[Two Way Communications via 4D_Tech]

Hi Kirk,

I am very interested on your progress with AWS. Can you tell me a little bit 
more about the kind of 4D DB you are running on it?
Are you running client/server or using SQL? Do you notice speed differences 
between ORDA / Classic 4D?


I’m asking because I am currently using AWS myself. The response is quite ok, 
but still way slower than LAN client server.
I am thinking of putting more 4D databases on AWS.


Regards,

Rudy Mortier
Two Way Communications bvba 



> On 10 Oct 2019, at 17:09, Kirk Brooks via 4D_Tech <4d_tech@lists.4d.com> 
> wrote:
> 
> Peter,
> I am in the process of moving a database from our own hardware to and AWS
> instance. It's true that the most expensive part of setting it up, at this
> point, is getting the appropriate amount of band width and throughput speed.
> 
> The other thing about VM vs metal is the whole pre-emptive process
> benefit basically goes away. Thomas Maul has shown this at the Summit.
> Having n+ virtual cores doesn't do anything to actually increase processing
> speed because the VM is running on whatever is allocated to it.
> Theoretically you could have a VM with 4 cores running an instance with 32
> cores. So preemptive threading is looking to be mainly a benefit for
> companies that run their own hardware and for desktop apps.
> 
> On Thu, Oct 10, 2019 at 5:25 AM Peter Jakobsson via 4D_Tech <
> 4d_tech@lists.4d.com> wrote:
> 
>> Hi !
>> 
>> I just finished a 1.5 hour phonecall with a support services manager at
>> the technical services company who supply one of my customers with all
>> their hardware/software/maintenance services.
>> 
>> He basically brought me up to date on “how things work” today which is
>> essentially that everything to do with platforms is now virtualised to
>> allow them to ‘tune’ resources to demand in realtime and provide seamless,
>> no downtime backup. Basically, my 4d Server is now a “cloud service”
>> without me even being aware of it, it’s just that the hardware involved
>> happens to be located on the preises.
>> 
>> In particular we discussed backup configurations for 4D server and this
>> was interesting because, while I requested independent drives for logfile
>> (“journal”) and datafile purposes, he essentially told me to just stick
>> everything on the same drive because it was virtual anyway and had multiple
>> redundancy protection via raid, 15-minute snapshotting etc. He offered to
>> “create” a C: and a D: drive to make me feel better, but pointed out that
>> they’re not much more independent than 2 folders would have been.
>> 
>> CONCLUSON
>> I now realise that the “WAN” / “LAN” distinction is disappearing. He said
>> the only reason the “cloud” solution wasn’t hosted off-site was that they
>> had measured the bandwidth that the customer used and calculated that the
>> cost would be astronomical if it was on AWS or something like that, but in
>> all other respects it was a cloud solution.
>> 
>> I was wondering, how do other major 4D server deployers optimise their
>> deployment strategies to take advantage of this ? It seems a great thing
>> that we are being “floated out to the cloud” without actually having to do
>> extra significant work, but what about things like the backup strategy ? I
>> don’t really like the idea that the log file has the same redundancy system
>> as the main datafile because the whole idea is that the corruption doesn’t
>> get replicated (which is what a RAID system does) and it’s independent at
>> the logical level.
>> 
>> We seem one step away from being able to supply server solutions where
>> “our” customer doesn’t have to host the database server on premises. Is
>> anybody doing this at an advanced level ? (e.g. connecting with 4D client
>> native to a 4D server that’s 3rd-party hosted).
>> 
>> Regards
>> 
>> Peter
>> 
>> **
>> 4D Internet Users Group (4D iNUG)
>> Archive:  http://lists.4d.com/archives.html
>> Options: https://lists.4d.com/mailman/options/4d_tech
>> Unsub:  mailto:4d_tech-unsubscr...@lists.4d.com
>> **
> 
> 
> 
> -- 
> Kirk Brooks
> San Francisco, CA
> ===
> 
> What can be said, can be said clearly,
> and what you can’t say, you should shut up about
> 
> *Wittgenstein and the Computer *
> **
> 4D Internet Users Group (4D iNUG)
> Archive:  http://lists.4d.com/archives.html
> Options: https://lists.4d.com/mailman/options/4d_tech
> Unsub:  mailto:4d_tech-unsubscr...@lists.4d.com
> **

**
4D Internet Users Group (4D iNUG)
Archive:  http://lists.4d.com/archives.html
Options: https://lists.4d.com/mailman/options/4d_tech
Unsub:  mailto:4d_tech-unsubscr...@lists.4d.com
*

Re: 4D Server Hosting - The Future ? & Using Github to Manage Distributed Development

[Kirk Brooks via 4D_Tech]

Hi JPR,
i'm glad the 'Bat Signal' still works.

On Fri, Oct 11, 2019 at 1:34 AM JPR via 4D_Tech <4d_tech@lists.4d.com>
wrote:

> Do you like Fairy Tales?
>
I thought that was the demo...


> > So what you are saying is that if you are deploying 4D Server in a VM
> there is not need to try and use preemptive mode. You get no benefit from
> doing that. It’s a waste of time?
> Wrong. It will be a benefit in terms of speed, even not as large as you
> dream, because the system will optimise the use of each possible core. And
> a benefit in term of smoothness of execution, because no process will
> freeze the others.
>
OK - but assuming I'm running on a VM isn't that speed and usability
benefit coming from moving the processing into a Worker?


> > The whole benefit of preemptive mode is to allow 4D to assign a process
> to a core.
> Wrong. What I explained is that 4D does NOT assign a process to a core,
> but assigns a process to a THREAD, and the System assigns cores to Threads.

And once more I display my sketchy understanding of how the hardware works.


> > But in the case of a VM the actual, physical cores are managed by the VM.
> Wrong. The VM talks to the System, and the System manages cores. In fact,
> just imagine that a machine can run several VM at the same time.
>
Ibid.

If you have been at the 2019 Tour, remember J_OtherObjects example, where I
> show that if you use cooperative processes, with one executing an
> excruciating method, then the Forms on use are kind of frozen. Then, by
> switching to preemptive mode, the same awfully blocking method doesn't
> impact on interface while running.
>
True (in fact I just pulled it out to look at again). But we are running
that on our machines where the _threads_ can be assigned by the _system_ to
various cores and I can see that in the activity viewer. In fact running on
'bare metal' shows a lot of variation in the activity of the various cores.

I went rummaging around in the knowledgebase to see if I could find what
I'm remembering. I think it was one of the keynotes from the '16 Summit and
I couldn't find it. But I did find Laurent Ensault's session on the
subject: https://kb.4d.com/assetid=77547

I rewatched it and ran the example database. The demo spawns 4 cooperative
and 4 preemptive processes then runs an intensive method 1mil times in
each.

I made a small change to his demo - I added the start and end ticks to the
process record in addition to the duration. Why? Because I wanted to see
the total time spent on processing the 'job'. I do this by taking the end
of the last process to finish and subtracting the start of the first
process to begin. I also moved the demo db to v17.3.

As luck would have it I have an AWS instance with v17.3 installed so I am
able to run the demo locally and on AWS.
On my laptop:

Cooperative 1: 376 ticks
Cooperative 2: 376 ticks
Cooperative 3: 375 ticks
Cooperative 4: 367 ticks

Total run time:  395 ticks


Preemp 1: 118 ticks
Preemp 2: 119 ticks
Preemp 3: 117 ticks
Preemp 4: 119 ticks

Total run time:  119 ticks  //  that's what we want to see!


On AWS:

Cooperative 1: 740 ticks
Cooperative 2: 706 ticks
Cooperative 3: 793 ticks
Cooperative 4: 731 ticks

Total run time:  821 ticks


Preemp 1: 881 ticks
Preemp 2: 824 ticks
Preemp 3: 857 ticks
Preemp 4: 848 ticks

Total run time:  881 ticks

Your first thought might be, "hold on, why is the total run time almost the
same as each process' run time?" The very first line of the method sets a
variable with the tick count. Because Laurent included the IDLE command in
the demo method those methods yield the processor back to 4D on each
iteration. (The video talks about the details on this.) So Coop1 starts and
then yields back to 4D which starts Coop2 and so on. In contrast Preemptive
processes simply start and run without having to 'yield back' to 4D.

So what we see is that each them gets started pretty much at the same time
and complete pretty close together as well. This probably wouldn't be the
case if they weren't doing exactly the same work and if 4D were actually
working instead of doing a demo. Total run time, as I said, is the tick
count for the last process to finish minus the tick count of the first one
to start. Or the total amount of time spent processing by all 4 processes.

This illustrates the point I recalled from back then: when you run on a VM
the advantage of the the OS to distribute threads to the actual cores is
abstracted away. They may be distributed to virtual cores but that doesn't
impact the total processing time. In the case of AWS apparently this VM
overhead actually causes it to run slower but this is a small sample set. I
have no idea why and frankly don't have the expertise to figure it out. But
it's there.

FYI - the AWS instance is an m5ad.xlarge: 4 cores, 16gb RAM. It sorta
sucks. I'd love to hear from anyone actually running 4D server on AWS with
good performance.

-- 
Kirk Brooks
San Francisco, CA
===

What can be said

Re: 4D Server Hosting - The Future ? & Using Github to Manage Distributed Development

[Narinder Chandi via 4D_Tech]

JPR,

Hi. Ah, that's great, thanks for confirming that. I should really find some 
time to experiment with Project mode then ;)

Regards,
 
Narinder Chandi,
ToolBox Systems Ltd.
 
I am available for new consulting opportunities…
http://4d.1045681.n5.nabble.com/ANN-4D-Developer-Available-td5765443.html
-- 

-Original Message-
From: 4D_Tech <4d_tech-boun...@lists.4d.com> on behalf of 4D Tech Mailing List 
<4d_tech@lists.4d.com>
Reply-To: 4D Tech Mailing List <4d_tech@lists.4d.com>
Date: Friday, 11 October 2019 at 09:33
To: 4D Tech Mailing List <4d_tech@lists.4d.com>
Cc: JPR 
Subject: Re: 4D Server Hosting - The Future ? & Using Github to Manage 
Distributed Development

AFAIK, if you use the Project mode, you don't need any binary, for 
everything is in text files (.json, .xml, .txt, .4DProject, .css, . 4DCatalog , 
.4dm, .4DForm, etc.) The .4DB is not necessary.


**
4D Internet Users Group (4D iNUG)
Archive:  http://lists.4d.com/archives.html
Options: https://lists.4d.com/mailman/options/4d_tech
Unsub:  mailto:4d_tech-unsubscr...@lists.4d.com
**

Re: 4D Server Hosting - The Future ? & Using Github to Manage Distributed Development

[JPR via 4D_Tech]

[JPR]

Hi Guys,

Once upon a time, in a magic computer, a lovely virtual princess was waiting in 
a cooperative process for a strong and handsome virtual prince to awake her... 
Unfortunately, the prince was a cooperative one and he was waiting for the door 
to open, and the key was held by the princess... 

Do you like Fairy Tales?

Here is another one, from Goethe: Once upon a time, an old sorcerer departed 
his workshop, leaving his apprentice with programs to execute. Tired of waiting 
tasks waiting in queue to access one of the 4 CPUs of his computer, the 
apprentice enchanted a virtual machine to do the work, using preemptive 
programming in which he was not fully trained. The apprentice soon realised 
that he couldn't control the tasks because he did not know the magic required 
to do so. The apprentice began to split the CPU in two with a magic mouse, but 
each of the pieces becomes a whole CPU that was splitting again and again in 
the virtual space, until the power supply became red hot!  When all seemed 
lost, the old sorcerer returned and quickly reduced the number of preemptive 
tasks to a more reasonable number! 

Do you believe it? If you have been assisting to the training day I gave during 
the 2019 Tour, you remember that we have discussed about that, and we have seen 
examples on the practical use of preemptive workers, the use of the Signal 
object, how to handle this typical Catch-22 situation, etc. We have also seen 
that the preemptive multithreading was useful to gain speed only in some very 
very specific cases (Remember the demo K_PreemptiveVsCooperative)

Sorry for those of you who still believe in him, but Santa doesn't exist. 
Virtual cores are NOT actual cores. And even if you get zillions of cores, 
there is still only ONE bus to access to memory. So, don't dream, multiplying 
the number of cores will NOT multiply the speed of every database operations. I 
mean, it will not be slower, but (cores * 8) ≠ (speed * 8). And this is even 
much more true in case of hyper threading. And obviously even much much moe 
true in case of a VM.

So let's try to be crystal clear:

> So what you are saying is that if you are deploying 4D Server in a VM there 
> is not need to try and use preemptive mode. You get no benefit from doing 
> that. It’s a waste of time?
Wrong. It will be a benefit in terms of speed, even not as large as you dream, 
because the system will optimise the use of each possible core. And a benefit 
in term of smoothness of execution, because no process will freeze the others.

> The whole benefit of preemptive mode is to allow 4D to assign a process to a 
> core.
Wrong. What I explained is that 4D does NOT assign a process to a core, but 
assigns a process to a THREAD, and the System assigns cores to Threads. The 
same Thread can be switched from core to core by the System, you don't even 
have to know.

> But in the case of a VM the actual, physical cores are managed by the VM.
Wrong. The VM talks to the System, and the System manages cores. In fact, just 
imagine that a machine can run several VM at the same time.

If you have been at the 2019 Tour, remember J_OtherObjects example, where I 
show that if you use cooperative processes, with one executing an excruciating 
method, then the Forms on use are kind of frozen. Then, by switching to 
preemptive mode, the same awfully blocking method doesn't impact on interface 
while running. 

So please, don't mix Processes, Threads, and Cores. Preemptive programming is a 
real plus from many points of view, including it helps 4D to optimise it's own 
internal optimisation processing. And also, remember that Preemptive 
programming is like food, it's necessary, but stay reasonable...

Regarding the other question:
> Yes, you're correct, currently the move from binary to Project format is one 
> way, whether this will become bi-directional I don't know.

> This was asked to JPR at the last 4D World Tour show and his response was 
> “why would you want to do that?” And then he proceeded to explain that to 
> implement a feature like that would require many hours of engineering time, 
> and QA time and then it would have to be maintained into the future. 
> So he said are no plans for a “project folder back to a single .4DB file” 
> feature. Only way to have the happen is to convince 4D it is a needed and 
> necessary feature. 
Thank you, Tim, it's exactly what I said (plus the accent) and it's still the 
current status.

> But, I still need to fully understand exactly why 4D is not eliminating the 
> interpreted structure binary .4DB file completely at this time? Maybe 4D plan 
> to do so eventually? Was this question asked at the last tour? If we are 
> given the Project feature then we should be all in and have no option but to 
> embrace it and source code repos such as Git, since, pretty much every other 
> programming language is doing it this way.
AFAIK, if you use the Project mode, you don't need any binary, for everything

Re: 4D Server Hosting - The Future ?

[Kirk Brooks via 4D_Tech]

Hey Tim,

On Thu, Oct 10, 2019 at 10:43 AM Tim Nevels via 4D_Tech <
4d_tech@lists.4d.com> wrote:

> So what you are saying is that if you are deploying 4D Server in a VM
> there is not need to try and use preemptive mode. You get no benefit from
> doing that. It’s a waste of time?

That's what I got. Thomas showed a demo app he wrote deployed on AWS or the
like and using a lot of cores (32?). But it did not improve the overall
processing time. The reason being, as I understand it, the whole benefit of
preemptive mode is to allow 4D to assign a process to a core. But in the
case of a VM the actual, physical cores are managed by the VM. It really
couldn't be otherwise and support all the cool stuff that makes the VM
attractive.

I don't know that makes preemptive a waste of time but it does mean you
need to think about where your db will be running to decide if it's worth
the effort to implement it.

On the other hand this means that in a VM environment CALL WORKER is pretty
much as useful and a lot less hassle.

It would be interesting to know if preemptive is something that can be
called for a local process on a client machine. Now _that_ would be pretty
handy. Ask JPR when you chat with him.

-- 
Kirk Brooks
San Francisco, CA
===

What can be said, can be said clearly,
and what you can’t say, you should shut up about

*Wittgenstein and the Computer *
**
4D Internet Users Group (4D iNUG)
Archive:  http://lists.4d.com/archives.html
Options: https://lists.4d.com/mailman/options/4d_tech
Unsub:  mailto:4d_tech-unsubscr...@lists.4d.com
**

RE: 4D Server Hosting - The Future ?

[Dennis, Neil via 4D_Tech]

> The other thing about VM vs metal is the whole pre-emptive process 
> benefit basically goes away.

... can go away...

If you have a physical machine with 32 cores and your VM run 8 shared with 4 
other VMs running 8 cores, then you may really get 8 cores. If you run 32 VMs 
all wanting 8 virtual, then of course you are not getting it. I think the point 
was that virtual cores do not always mean physical cores. It will depend on the 
number of physical resources allocated to the VM.

Neil








Privacy Disclaimer: This message contains confidential information and is 
intended only for the named addressee. If you are not the named addressee you 
should not disseminate, distribute or copy this email. Please delete this email 
from your system and notify the sender immediately by replying to this email.  
If you are not the intended recipient you are notified that disclosing, 
copying, distributing or taking any action in reliance on the contents of this 
information is strictly prohibited.

The Alternative Investments division of UMB Fund Services provides a full range 
of services to hedge funds, funds of funds and private equity funds.  Any tax 
advice in this communication is not intended to be used, and cannot be used, by 
a client or any other person or entity for the purpose of (a) avoiding 
penalties that may be imposed on any taxpayer or (b) promoting, marketing, or 
recommending to another party any matter addressed herein.
**
4D Internet Users Group (4D iNUG)
Archive:  http://lists.4d.com/archives.html
Options: https://lists.4d.com/mailman/options/4d_tech
Unsub:  mailto:4d_tech-unsubscr...@lists.4d.com
**

Re: 4D Server Hosting - The Future ?

[Tim Nevels via 4D_Tech]

On Oct 10, 2019, at 10:25 AM, Kirk Brooks wrote:

> The other thing about VM vs metal is the whole pre-emptive process
> benefit basically goes away. Thomas Maul has shown this at the Summit.
> Having n+ virtual cores doesn't do anything to actually increase processing
> speed because the VM is running on whatever is allocated to it.
> Theoretically you could have a VM with 4 cores running an instance with 32
> cores. So preemptive threading is looking to be mainly a benefit for
> companies that run their own hardware and for desktop apps.

I was not aware of this. So what you are saying is that if you are deploying 4D 
Server in a VM there is not need to try and use preemptive mode. You get no 
benefit from doing that. It’s a waste of time? Sounds like a giant bummer. Hope 
you are wrong. I’ll put in the secret distress call code “JPR" in this message 
and see if JPR catches this and knows anything about this and can comment. 

If not, I’m going to make it a point to ask one of the Laurents about this at 
4D Summit so I’m clear on when you get preemptive benefits. Do you get any when 
running 4D Server or 4D Client in a VM? 

As you know I love running 4D Server in a VM environment that is properly 
configured on great hardware. Performance is great for 4D Server and for 4D 
Client instances. And when you need more disk space you can get it in a few 
minutes by just altering some VM parameters. Same for adding more RAM, just 
takes a few minutes and a reboot. 

And since the disks are all allocated from a SAN that is RAID “whatever” you 
don’t have to worry about a disk failure and losing data. RAID will protect you 
from that. And putting the .4dd on a separate physical disk than the .journal 
file can’t be done and is not a consideration for providing additional 
protection from hardware failure. 

The last part of fault tolerance and recover from failures is to implement 
“Volume Shadow Copy” and get a snapshot of the machine. I have another client 
that does this every hour. So if the VM goes completely bad you can restore it 
back to what it was up to 1 hour ago. That brings drive “C:\” back to where it 
was. And if your data file is on “D:\” it is stored on the SAN “hard drives” so 
it is suppose be accurate and good up to the last operation. This is what I am 
told. 

That last thing I tell myself when running 4D in a VM environment is that 
server hardware failure and disaster recover is not longer my problem. It’s the 
IT department’s problem. I’m just providing a 4D Server application that runs 
on their “machine”. I didn’t set up or configure the “machine”. I didn’t 
implement the backup plan or strategy. Everything depends on IT doing it right 
and providing an environment that can recover from hardware failures. 

In the past I was very involved with server hardware setup and configuration 
and backups, and so this was another of my “worries” or considerations. Did we 
buy enough RAM. Did we get enough hard drive space and are we using the drives 
the right way. Is the backup software running and working. And if there was a 
problem or failure I was the first one called to help fix it. That’s all gone 
when 4D Server is running in a VM. It’s all someone else’s problem. All I have 
to say is “when you get the VM restored and online again, let me know. I’ll 
check 4D Server and see if we lost any data.” I like it that way. 

Tim

*
Tim Nevels
Innovative Solutions
785-749-3444
timnev...@mac.com
*

**
4D Internet Users Group (4D iNUG)
Archive:  http://lists.4d.com/archives.html
Options: https://lists.4d.com/mailman/options/4d_tech
Unsub:  mailto:4d_tech-unsubscr...@lists.4d.com
**

Re: 4D Server Hosting - The Future ?

[Robb Detlefs via 4D_Tech]

We host both the Server and the Client in the cloud. End users connect into the 
"client cloud" using Remote Desktop.

Robb


--
Robb Detlefs | Director, West Coast Operations & Strategic Initiatives
robb.detl...@gallerysystems.com | 
510.652.8950 x233 or 646.733.2239 x233
GallerySystems | www.gallerysystems.com
3200 College Ave. Suite 6; Berkeley, CA 94705




On Oct 10, 2019, at 10:02 AM, John DeSoi via 4D_Tech 
<4d_tech@lists.4d.com> wrote:

EXTERNAL: Do not click links or open attachments if you do not recognize the 
sender.

I don't see how "cloud" hosting can be the future of 4D when LAN execution is 
great but WAN execution is horribly slow for the same application. Yes, you can 
rewrite in various ways (harder to write, understand, and maintain) but it is 
not an easy task.

John DeSoi, Ph.D.


On Oct 10, 2019, at 6:24 AM, Peter Jakobsson via 4D_Tech 
<4d_tech@lists.4d.com> wrote:

CONCLUSON
I now realise that the “WAN” / “LAN” distinction is disappearing. He said the 
only reason the “cloud” solution wasn’t hosted off-site was that they had 
measured the bandwidth that the customer used and calculated that the cost 
would be astronomical if it was on AWS or something like that, but in all other 
respects it was a cloud solution.

**
4D Internet Users Group (4D iNUG)
Archive:  
https://urldefense.com/v3/__http://lists.4d.com/archives.html__;!09EJrJkydGw!Xytu_B2U0mQcyAmuZQJ4z6tA7e_ofPTL3OqMZxKUuOz4k9HZHFqY7PGYVS6H8-KlhuLYcF6BBQ$
Options: 
https://urldefense.com/v3/__https://lists.4d.com/mailman/options/4d_tech__;!09EJrJkydGw!Xytu_B2U0mQcyAmuZQJ4z6tA7e_ofPTL3OqMZxKUuOz4k9HZHFqY7PGYVS6H8-KlhuLpyeegqA$
Unsub:  mailto:4d_tech-unsubscr...@lists.4d.com
**

**
4D Internet Users Group (4D iNUG)
Archive:  http://lists.4d.com/archives.html
Options: https://lists.4d.com/mailman/options/4d_tech
Unsub:  mailto:4d_tech-unsubscr...@lists.4d.com
**

Re: 4D Server Hosting - The Future ?

[John DeSoi via 4D_Tech]

I don't see how "cloud" hosting can be the future of 4D when LAN execution is 
great but WAN execution is horribly slow for the same application. Yes, you can 
rewrite in various ways (harder to write, understand, and maintain) but it is 
not an easy task.

John DeSoi, Ph.D.


> On Oct 10, 2019, at 6:24 AM, Peter Jakobsson via 4D_Tech 
> <4d_tech@lists.4d.com> wrote:
> 
> CONCLUSON
> I now realise that the “WAN” / “LAN” distinction is disappearing. He said the 
> only reason the “cloud” solution wasn’t hosted off-site was that they had 
> measured the bandwidth that the customer used and calculated that the cost 
> would be astronomical if it was on AWS or something like that, but in all 
> other respects it was a cloud solution.

**
4D Internet Users Group (4D iNUG)
Archive:  http://lists.4d.com/archives.html
Options: https://lists.4d.com/mailman/options/4d_tech
Unsub:  mailto:4d_tech-unsubscr...@lists.4d.com
**

Re: 4D Server Hosting - The Future ?

[Tom Benedict via 4D_Tech]

>BTW - when using a 'cloud' service, depending on your personal/business 
> 
> outlook you might want to think about who has access to your data, and 
> what are they doing with it. This applies not just to colo and 4D but 
> to everything in 'the cloud’.

That’s why you encrypt your data. 
https://blog.4d.com/introduction-to-data-encryption-in-4d/ 


Tom Benedict
**
4D Internet Users Group (4D iNUG)
Archive:  http://lists.4d.com/archives.html
Options: https://lists.4d.com/mailman/options/4d_tech
Unsub:  mailto:4d_tech-unsubscr...@lists.4d.com
**

Re: 4D Server Hosting - The Future ?

[Tom Benedict via 4D_Tech]

Indeed, that is the magic of VM and SAN. They can change the truck’s engine 
while its traveling down the road at 60mph. I was a skeptic until I saw it in 
action. Hot swapping of disks, memory, even ‘blades’ with CPUs on them happens 
transparently.

Keep in mind, though, that this is not done to save money. It might save money, 
but what it really does is make systems more reliable and faster to deploy.

Tom Benedict

> On Oct 10, 2019, at 09:12, Peter Jakobsson via 4D_Tech <4d_tech@lists.4d.com> 
> wrote:
> 
>> On 10 Oct 2019, at 16:58, Chip Scheide <4d_o...@pghrepository.org> wrote:
>> 
>> that all is great until the hardware running the 4,382,619 VMs crashes
> 
> According to “the guy”, that’s all taken care of. Even if the “metal” melts 
> down, the VM’s just “seamlessly” migrate themselves onto other metal without 
> the users even knowing !

**
4D Internet Users Group (4D iNUG)
Archive:  http://lists.4d.com/archives.html
Options: https://lists.4d.com/mailman/options/4d_tech
Unsub:  mailto:4d_tech-unsubscr...@lists.4d.com
**

Re: 4D Server Hosting - The Future ?

[Tom Benedict via 4D_Tech]

The key to getting good 4D performance out of VMs is dedicated resources. Once 
we got the VM ‘locked’ so it wouldn’t be ’smart’ about reallocating ‘idle’ 
resources (disk space, cache memory etc) to other VMs performance was very 
good. I don’t know much about VMs, but I’m surprised that you can’t dedicate 
cores to a specific VM like you can other resources.

Tom Benedict

> On Oct 10, 2019, at 08:23, Peter Jakobsson via 4D_Tech <4d_tech@lists.4d.com> 
> wrote:
> 
> Thanks for your post Kirk.
> 
> The guy I spoke to seemed to have it the other way around - the VM’s had 4-12 
> cores and the “metal” about 32.
> 
> He also was of the categorical opinion that the only way to really keep 
> applications “isolated” from each other (i.e. not bring everything else down 
> when they crashed) was to give each mission critical application or service 
> its own VM.
> 
> When I put to him “what about the natural O/S level multi-threading” he felt 
> there were too many vulnerabilities and mentioned especially the “crypto 
> viruses” and the Intel multi-threading bug. His approach was basically - if 
> your VM needs ore resources then we can simply allocated more. He wasn’t 
> really bothered by the idea of applications that were multi-threaded 
> internally because it’s all the one big bucket and if something inside the 
> bucket needs more power then just make the bucket bigger.
> 
> Peter
> 
> 
>> On 10 Oct 2019, at 16:09, Kirk Brooks via 4D_Tech <4d_tech@lists.4d.com>
>> wrote:
>> 
>> The other thing about VM vs metal is the whole pre-emptive process
>> benefit basically goes away. Thomas Maul has shown this at the Summit.
>> Having n+ virtual cores doesn't do anything to actually increase processing
>> speed because the VM is running on whatever is allocated to it.
>> Theoretically you could have a VM with 4 cores running an instance with 32
>> cores

**
4D Internet Users Group (4D iNUG)
Archive:  http://lists.4d.com/archives.html
Options: https://lists.4d.com/mailman/options/4d_tech
Unsub:  mailto:4d_tech-unsubscr...@lists.4d.com
**

Re: 4D Server Hosting - The Future ?

[Chip Scheide via 4D_Tech]

it is a repeating cycle.

change the paradigm to make more money.
Once everyone that needs one has a mainframe switch to desktops, now 
everyone has a desktop
switch to the 'cloud'.
eventually it will swing back to personal/business owned devices as 
data security becomes a (bigger) issue

BTW - when using a 'cloud' service, depending on your personal/business 
outlook you might want to think about who has access to your data, and 
what are they doing with it. This applies not just to colo and 4D but 
to everything in 'the cloud'.

What is Apple doing with your playlists and other purchasing habits 
(and what does that say about you), who else is looking at the files in 
your dropbox account, and nobody should have to wonder about what 
google, amazon, Facebook twitter etc are doing with all the information 
you maybe giving them.


Just because your are paranoid does NOT mean that they are not *really* 
out there...  :)

On Thu, 10 Oct 2019 08:59:48 -0700, Kirk Brooks via 4D_Tech wrote:
> It is kind of
> interesting that we are moving back to the topology the industry started
> with - what's the difference between a 'main frame computer' and 'cloud
> computing'?
We have done so much, with so little, for so long;
We are now qualified to anything with nothing 
  - unknown
**
4D Internet Users Group (4D iNUG)
Archive:  http://lists.4d.com/archives.html
Options: https://lists.4d.com/mailman/options/4d_tech
Unsub:  mailto:4d_tech-unsubscr...@lists.4d.com
**

Re: 4D Server Hosting - The Future ?

[Peter Jakobsson via 4D_Tech]

> On 10 Oct 2019, at 16:58, Chip Scheide <4d_o...@pghrepository.org> wrote:
> 
> that all is great until the hardware running the 4,382,619 VMs crashes

According to “the guy”, that’s all taken care of. Even if the “metal” melts 
down, the VM’s just “seamlessly” migrate themselves onto other metal without 
the users even knowing !

Don’t ask me quite how that works but he kept saying it wasn't a problem.

-P

**
4D Internet Users Group (4D iNUG)
Archive:  http://lists.4d.com/archives.html
Options: https://lists.4d.com/mailman/options/4d_tech
Unsub:  mailto:4d_tech-unsubscr...@lists.4d.com
**

Re: 4D Server Hosting - The Future ?

[Kirk Brooks via 4D_Tech]

Peter,
I had a discussion with Tim Nevels on another channel about this last week
too. He's a big fan and maybe will weigh in here. I will not miss having to
make a trip to the colo to change an SSD that died. (Hint: mirrored SSDs so
you don't have to do that in the middle of the night.) It is kind of
interesting that we are moving back to the topology the industry started
with - what's the difference between a 'main frame computer' and 'cloud
computing'?

On Thu, Oct 10, 2019 at 8:23 AM Peter Jakobsson via 4D_Tech <
4d_tech@lists.4d.com> wrote:

> Thanks for your post Kirk.
>
> The guy I spoke to seemed to have it the other way around - the VM’s had
> 4-12 cores and the “metal” about 32.
>
> He also was of the categorical opinion that the only way to really keep
> applications “isolated” from each other (i.e. not bring everything else
> down when they crashed) was to give each mission critical application or
> service its own VM.
>
> When I put to him “what about the natural O/S level multi-threading” he
> felt there were too many vulnerabilities and mentioned especially the
> “crypto viruses” and the Intel multi-threading bug. His approach was
> basically - if your VM needs ore resources then we can simply allocated
> more. He wasn’t really bothered by the idea of applications that were
> multi-threaded internally because it’s all the one big bucket and if
> something inside the bucket needs more power then just make the bucket
> bigger.
>
> Peter
>
>
> > On 10 Oct 2019, at 16:09, Kirk Brooks via 4D_Tech <4d_tech@lists.4d.com>
> > wrote:
> >
> > The other thing about VM vs metal is the whole pre-emptive process
> > benefit basically goes away. Thomas Maul has shown this at the Summit.
> > Having n+ virtual cores doesn't do anything to actually increase
> processing
> > speed because the VM is running on whatever is allocated to it.
> > Theoretically you could have a VM with 4 cores running an instance with
> 32
> > cores
>
> **
> 4D Internet Users Group (4D iNUG)
> Archive:  http://lists.4d.com/archives.html
> Options: https://lists.4d.com/mailman/options/4d_tech
> Unsub:  mailto:4d_tech-unsubscr...@lists.4d.com
> **



-- 
Kirk Brooks
San Francisco, CA
===

What can be said, can be said clearly,
and what you can’t say, you should shut up about

*Wittgenstein and the Computer *
**
4D Internet Users Group (4D iNUG)
Archive:  http://lists.4d.com/archives.html
Options: https://lists.4d.com/mailman/options/4d_tech
Unsub:  mailto:4d_tech-unsubscr...@lists.4d.com
**

Re: 4D Server Hosting - The Future ?

[Chip Scheide via 4D_Tech]

that all is great until the hardware running the 4,382,619 VMs crashes
:/

Chip
On Thu, 10 Oct 2019 16:23:18 +0100, Peter Jakobsson via 4D_Tech wrote:
> 
> He also was of the categorical opinion that the only way to really 
> keep applications “isolated” from each other (i.e. not bring 
> everything else down when they crashed) was to give each mission 
> critical application or service its own VM.
We have done so much, with so little, for so long;
We are now qualified to anything with nothing 
  - unknown
**
4D Internet Users Group (4D iNUG)
Archive:  http://lists.4d.com/archives.html
Options: https://lists.4d.com/mailman/options/4d_tech
Unsub:  mailto:4d_tech-unsubscr...@lists.4d.com
**

Re: 4D Server Hosting - The Future ?

[Peter Jakobsson via 4D_Tech]

Thanks for your post Kirk.

The guy I spoke to seemed to have it the other way around - the VM’s had 4-12 
cores and the “metal” about 32.

He also was of the categorical opinion that the only way to really keep 
applications “isolated” from each other (i.e. not bring everything else down 
when they crashed) was to give each mission critical application or service its 
own VM.

When I put to him “what about the natural O/S level multi-threading” he felt 
there were too many vulnerabilities and mentioned especially the “crypto 
viruses” and the Intel multi-threading bug. His approach was basically - if 
your VM needs ore resources then we can simply allocated more. He wasn’t really 
bothered by the idea of applications that were multi-threaded internally 
because it’s all the one big bucket and if something inside the bucket needs 
more power then just make the bucket bigger.

Peter


> On 10 Oct 2019, at 16:09, Kirk Brooks via 4D_Tech <4d_tech@lists.4d.com>
> wrote:
> 
> The other thing about VM vs metal is the whole pre-emptive process
> benefit basically goes away. Thomas Maul has shown this at the Summit.
> Having n+ virtual cores doesn't do anything to actually increase processing
> speed because the VM is running on whatever is allocated to it.
> Theoretically you could have a VM with 4 cores running an instance with 32
> cores

**
4D Internet Users Group (4D iNUG)
Archive:  http://lists.4d.com/archives.html
Options: https://lists.4d.com/mailman/options/4d_tech
Unsub:  mailto:4d_tech-unsubscr...@lists.4d.com
**

Re: 4D Server Hosting - The Future ?

[Kirk Brooks via 4D_Tech]

Peter,
I am in the process of moving a database from our own hardware to and AWS
instance. It's true that the most expensive part of setting it up, at this
point, is getting the appropriate amount of band width and throughput speed.

The other thing about VM vs metal is the whole pre-emptive process
benefit basically goes away. Thomas Maul has shown this at the Summit.
Having n+ virtual cores doesn't do anything to actually increase processing
speed because the VM is running on whatever is allocated to it.
Theoretically you could have a VM with 4 cores running an instance with 32
cores. So preemptive threading is looking to be mainly a benefit for
companies that run their own hardware and for desktop apps.

On Thu, Oct 10, 2019 at 5:25 AM Peter Jakobsson via 4D_Tech <
4d_tech@lists.4d.com> wrote:

> Hi !
>
> I just finished a 1.5 hour phonecall with a support services manager at
> the technical services company who supply one of my customers with all
> their hardware/software/maintenance services.
>
> He basically brought me up to date on “how things work” today which is
> essentially that everything to do with platforms is now virtualised to
> allow them to ‘tune’ resources to demand in realtime and provide seamless,
> no downtime backup. Basically, my 4d Server is now a “cloud service”
> without me even being aware of it, it’s just that the hardware involved
> happens to be located on the preises.
>
> In particular we discussed backup configurations for 4D server and this
> was interesting because, while I requested independent drives for logfile
> (“journal”) and datafile purposes, he essentially told me to just stick
> everything on the same drive because it was virtual anyway and had multiple
> redundancy protection via raid, 15-minute snapshotting etc. He offered to
> “create” a C: and a D: drive to make me feel better, but pointed out that
> they’re not much more independent than 2 folders would have been.
>
> CONCLUSON
> I now realise that the “WAN” / “LAN” distinction is disappearing. He said
> the only reason the “cloud” solution wasn’t hosted off-site was that they
> had measured the bandwidth that the customer used and calculated that the
> cost would be astronomical if it was on AWS or something like that, but in
> all other respects it was a cloud solution.
>
> I was wondering, how do other major 4D server deployers optimise their
> deployment strategies to take advantage of this ? It seems a great thing
> that we are being “floated out to the cloud” without actually having to do
> extra significant work, but what about things like the backup strategy ? I
> don’t really like the idea that the log file has the same redundancy system
> as the main datafile because the whole idea is that the corruption doesn’t
> get replicated (which is what a RAID system does) and it’s independent at
> the logical level.
>
> We seem one step away from being able to supply server solutions where
> “our” customer doesn’t have to host the database server on premises. Is
> anybody doing this at an advanced level ? (e.g. connecting with 4D client
> native to a 4D server that’s 3rd-party hosted).
>
> Regards
>
> Peter
>
> **
> 4D Internet Users Group (4D iNUG)
> Archive:  http://lists.4d.com/archives.html
> Options: https://lists.4d.com/mailman/options/4d_tech
> Unsub:  mailto:4d_tech-unsubscr...@lists.4d.com
> **



-- 
Kirk Brooks
San Francisco, CA
===

What can be said, can be said clearly,
and what you can’t say, you should shut up about

*Wittgenstein and the Computer *
**
4D Internet Users Group (4D iNUG)
Archive:  http://lists.4d.com/archives.html
Options: https://lists.4d.com/mailman/options/4d_tech
Unsub:  mailto:4d_tech-unsubscr...@lists.4d.com
**

Re: 4D Server Hosting - The Future ?

[Tom Benedict via 4D_Tech]

Hi Peter,

Indeed, it is a brave new world, isn’t it? And like you said, it all happened 
transparently.

At my last employer everything was virtualized. The SAN ‘guarenteed’ data 
uptime and redundancy. “Disks” (HDD or SSD) were ‘auto-healing’ and hardware 
failover was automatic. It is really quite amazing. And it worked very well. 
Once the VMs were optimized it was as fast or faster than dedicated hardware. 
Moving 4D Server to the “external cloud" is another question. For optimum (or 
maybe even just tolerable) performance, your app will need to be designed with 
a very lightweight front end. Or abandon 4D client and use a web front end.

>I don’t really like the idea that the log file has the same redundancy system 
>as the main datafile because the whole idea is that the corruption doesn’t get 
>replicated (which is what a RAID system does) and it’s independent at the 
>logical level.
> 
A SAN is really a lot more that just a RAID. As for a backup strategy, at this 
point we’re really talking about a disaster recovery plan, since the SAN is 
‘guarenteed'. You might consider mirroring to another ‘cloud service'. Since 
"spinning up" a VM is so easy now, all you need is another 4D server license to 
set up a 4D mirror server. Have it integrate the log file every minute or two 
and you have a near real time backup, ready for (manual) failover. Since only 
the log files sent, and they are tiny, the bandwidth costs would be extremely 
small.

Tom benedict

**
4D Internet Users Group (4D iNUG)
Archive:  http://lists.4d.com/archives.html
Options: https://lists.4d.com/mailman/options/4d_tech
Unsub:  mailto:4d_tech-unsubscr...@lists.4d.com
**