Go watch Thomas Maul's presentation. Now.
I wasn't able to tune into this 4DMethod user group presentation live but caught it later. First off, thanks to Brent Raymond for doing such a great job with 4DMethod. I don't know if everyone is taking advantage of this great resource. Given the various time zones we all live in, having presentations available for playback at any time is *massively* helpful. There are lots of good presentations up there so check it out. Anyway, for anyone expecting to continue working with 4D down the versions, go watch this excellent presentation now! There's an embedded link to the presentation here at the moment: https://4dmethod.com/ The meeting starts with the usual helpful reviews of 4D news and business, NUG threads and pointers to KB of interest. There's a bit of a technical glitch in the middle and if you want to skip through that, move to between 36 and 37 minutes. (36:45 when I watched,) I haven't been in 4D fanboi mode on this list much lately, so excuse me while I gush a bit. You know how sometimes 4D makes you scratch your head and then other times it takes your breath away because it does something so elegantly? This is one of those "check it out, it's so elegant" times. So check it out! Many old-timers will recall that I've said "SET PROCESS VARIABLE is evil" more than once. Without belaboring my entirely reasonable and balanced position ;-), 15 R5/16.0 (ish) bring a new and fundamentally improved native interprocess/inter-window communication systems. Thomas' demos are clear and persuasive. The upcoming features aren't going to allow for every type/pattern of message queue that you might want, but they do allow for some very helpful and common ones. * Passing messages between windows in one process or across processes on one machine with CALL FORM (perhaps it should be named CALL WINDOW?). This makes complex UI code way simpler to implement and more efficient to run. * Passing messages between processes with CALL WORKER. * In both cases, 'messages' are a method call with whatever parameters you need. Thomas doesn't discuss it, but an obvious strategy is to pass JSON objects. Then again, you can pass simple parameters as well, which can be even easier to read and write. Your choice. * Thread-safe, pre-emptive, avoids globals and semaphores, very efficient polling, etc. Yeah, nicely done 4D. I've been writing message queues in 4D for a really long time...the first one was at 4D years ago to generate plug-in serial numbers. (There was a special plug-in to do it that was tied to a single machine...and there was no sort of RPC system in 4D Server at the time.) I figured out SET PROCESS VARIABLE & friends enough to see their downsides and never used them again. (I also never used subtables in productions after the first attempt, for that matter.) Instead, I use record-based queues (which I still like) and IPC communication with NTK (which I also still like.) The new commands looks extremely well thought out and deceptively easy to use. (Meaning, you could be forgiven for not even knowing how much complexity 4D is sheltering you from, and how well they're doing it.) If you've done any sort of queuing work yourself, you'll quickly appreciate that the new approach is one of those areas where 4D is hitting it out of the park. Like I said early, excuse my gushing...but this is an awesome set of features. Note that the demo is showing features in unreleased versions so things may change (improve). For example, it appears that the message queues die if 4D shuts down or crashes. This is absolutely fine for some uses and a deal-breaker for others. Perhaps CALL WORKER will gain an option to retain unprocessed messages? P.S. If anyone is interested in multi-platform queuing solutions, ping me. I'm diving into RabbitMQ these days and it's sweet. Welsh Harris turned me onto IronMQ and it also looks pretty great. It's more similar architecturally to AWS's SQS than RabbitMQ but any of these three are solid-looking choices for scalable, distributed systems. ** 4D Internet Users Group (4D iNUG) FAQ: http://lists.4d.com/faqnug.html Archive: http://lists.4d.com/archives.html Options: http://lists.4d.com/mailman/options/4d_tech Unsub: mailto:4d_tech-unsubscr...@lists.4d.com **
Re: Go watch Thomas Maul's presentation. Now.
David, thank you for this writeup. :) I also want to thank Brent and Ed for putting so much effort in to making a active, growing, online usergroup. Plus, thank you for inviting 4D to to participate. (including the 'hey 4D' hot seat) You should watch the entire meeting. It's is worth the time. But, if you would like a direct link to skip the technical difficulties, you can jump to here: https://youtu.be/eEjrHR2BelE?t=36m57s -Brian On Sep 2, 2016, at 9:35 AM, David Adams mailto:dpad...@gmail.com>> wrote: I wasn't able to tune into this 4DMethod user group presentation live but caught it later. First off, thanks to Brent Raymond for doing such a great job with 4DMethod. I don't know if everyone is taking advantage of this great resource. Given the various time zones we all live in, having presentations available for playback at any time is *massively* helpful. There are lots of good presentations up there so check it out. Anyway, for anyone expecting to continue working with 4D down the versions, go watch this excellent presentation now! There's an embedded link to the presentation here at the moment: https://4dmethod.com/ The meeting starts with the usual helpful reviews of 4D news and business, NUG threads and pointers to KB of interest. There's a bit of a technical glitch in the middle and if you want to skip through that, move to between 36 and 37 minutes. (36:45 when I watched,) I haven't been in 4D fanboi mode on this list much lately, so excuse me while I gush a bit. You know how sometimes 4D makes you scratch your head and then other times it takes your breath away because it does something so elegantly? This is one of those "check it out, it's so elegant" times. So check it out! Many old-timers will recall that I've said "SET PROCESS VARIABLE is evil" more than once. Without belaboring my entirely reasonable and balanced position ;-), 15 R5/16.0 (ish) bring a new and fundamentally improved native interprocess/inter-window communication systems. Thomas' demos are clear and persuasive. The upcoming features aren't going to allow for every type/pattern of message queue that you might want, but they do allow for some very helpful and common ones. * Passing messages between windows in one process or across processes on one machine with CALL FORM (perhaps it should be named CALL WINDOW?). This makes complex UI code way simpler to implement and more efficient to run. * Passing messages between processes with CALL WORKER. * In both cases, 'messages' are a method call with whatever parameters you need. Thomas doesn't discuss it, but an obvious strategy is to pass JSON objects. Then again, you can pass simple parameters as well, which can be even easier to read and write. Your choice. * Thread-safe, pre-emptive, avoids globals and semaphores, very efficient polling, etc. Yeah, nicely done 4D. I've been writing message queues in 4D for a really long time...the first one was at 4D years ago to generate plug-in serial numbers. (There was a special plug-in to do it that was tied to a single machine...and there was no sort of RPC system in 4D Server at the time.) I figured out SET PROCESS VARIABLE & friends enough to see their downsides and never used them again. (I also never used subtables in productions after the first attempt, for that matter.) Instead, I use record-based queues (which I still like) and IPC communication with NTK (which I also still like.) The new commands looks extremely well thought out and deceptively easy to use. (Meaning, you could be forgiven for not even knowing how much complexity 4D is sheltering you from, and how well they're doing it.) If you've done any sort of queuing work yourself, you'll quickly appreciate that the new approach is one of those areas where 4D is hitting it out of the park. Like I said early, excuse my gushing...but this is an awesome set of features. Note that the demo is showing features in unreleased versions so things may change (improve). For example, it appears that the message queues die if 4D shuts down or crashes. This is absolutely fine for some uses and a deal-breaker for others. Perhaps CALL WORKER will gain an option to retain unprocessed messages? P.S. If anyone is interested in multi-platform queuing solutions, ping me. I'm diving into RabbitMQ these days and it's sweet. Welsh Harris turned me onto IronMQ and it also looks pretty great. It's more similar architecturally to AWS's SQS than RabbitMQ but any of these three are solid-looking choices for scalable, distributed systems. ** 4D Internet Users Group (4D iNUG) FAQ: http://lists.4d.com/faqnug.html Archive: http://lists.4d.com/archives.html Options: http://lists.4d.com/mailman/options/4d_tech Unsub: mailto:4d_tech-unsubscr...@lists.4d.com ** ** 4D Internet Users Group
Re: Go watch Thomas Maul's presentation. Now
On Sep 2, 2016, at 1:36 PM, Brian Young wrote: > I also want to thank Brent and Ed for putting so much effort in to making a > active, growing, online usergroup. Plus, thank you for inviting 4D to to > participate. (including the 'hey 4D' hot seat) > > You should watch the entire meeting. It's is worth the time. But, if you > would like a direct link to skip the technical difficulties, you can jump to > here: I agree with Brian and David. I was at the Thomas Maul 4D Method presentation. Is that the right way to say I was watching on Google Hangout? Thomas gave a great presentation with a lot of detail, and examples. Everyone should see it and you will learn something new. I asked a question about advice on how to use the new preemptive system and rewriting old code to use the new stuff. The surprising answer from Thomas was that he had done this for a project and would advise to not go rewrite old code. If it is running now and working, don’t go rewrite it. Leave it alone. He did and it introduced bugs that didn’t exist in the old code that was working just fine that he then had to go track down and fix. His advice is you are better off using the new v16 stuff for new things you need to implement, or to redo something that you need to speed up. Also another bit of info is on the new v16 data cache system that was announced in April at 4D Summit 2016 in Portland. I asked if it would be available in a v15R version before v16. Answer was it’s not ready yet. They had hoped it would be done and ready to be released in v16.0, but they have found a few issues that need to be fixed. So no new v16 cache system in v15R versions and probably not in v16.0. But it will be released some time during v16’s life. No need to rush in getting the new cache system out. The current one is working just fine. Better to make the new version rock solid first and then release it. I think that’s the right approach. Tim Tim Nevels Innovative Solutions 785-749-3444 timnev...@mac.com ** 4D Internet Users Group (4D iNUG) FAQ: http://lists.4d.com/faqnug.html Archive: http://lists.4d.com/archives.html Options: http://lists.4d.com/mailman/options/4d_tech Unsub: mailto:4d_tech-unsubscr...@lists.4d.com **
RE: Go watch Thomas Maul's presentation. Now
Upfront - thanks for your time watching - and the friendly comments. > I asked if it would be available in a v15R version before v16. Answer was > it’s not ready yet. They had hoped it would be done and ready to be released > in v16.0, but they have found a few issues that need to be fixed. So no new > v16 cache system in v15R versions and probably not in v16.0. But it will be > released some time during v16’s life. Sorry, it seems I was not clear on this point. The engine for the new cache manager is done. It was started last Xmas, tested intensively and is even already in production usage in some internal systems. It was NOT ready for 4D v15 R5 through, as reminder: Every 3 month, usually around the time we publish a final R-Release, we fork the next one. This means, we take the latest version of our internal source (including all features and bug fixes done at that time) and start to REMOVE features. Every feature not fully tested is removed (=delayed for a following version). R5 was forked end of May - at that time we did not finished all testing for the cache manager. As the cache manager is a very important part of the database server, we had a very long testing planed. So, it is not in R5. But it will be in v16. In v16 the cache manager will run in fully automatic mode. Only thing you can set/modify is cache size, similar as in current versions. What's not in v16.x, is a set of features to fine tune the cache usage, to give different priorities to specific tables or index, which will improve performance even more for very large databases. And this will come with a following v16 R Release (development is finished, testing already started, but will need more time). Thomas 4D Deutschland GmbH Geschäftsführer Laurent Ribardière Telefon: +49-89-856 33 43-0 Amtsgericht München HRB 92258 Fax: +49-89-856 33 43-99 Obere Hauptstrasse 2 E-Mail: thomas.m...@4d.com 85386 Eching - Deutschland Web: www.4D.com ** 4D Internet Users Group (4D iNUG) FAQ: http://lists.4d.com/faqnug.html Archive: http://lists.4d.com/archives.html Options: http://lists.4d.com/mailman/options/4d_tech Unsub: mailto:4d_tech-unsubscr...@lists.4d.com **
Re: Go watch Thomas Maul's presentation. Now.
I also really enjoyed watching the video. It is a subject that I was really excited about at the Summit and I appreciate the 4DMethod user group making this possible. In the video, Thomas talked about a new (to 4D) paradigm where all UI related code could run on the main thread (the Application process) and background processes (workers) could be used for potentially long-running code that shouldn't block the main thread. This is exactly what many other environments do and would be accomplished by opening all windows using DIALOG(*) from within the Application process context and then communicating between threads using CALL FORM (I really wish this was named CALL WINDOW) and CALL WORKER. I've been mulling this over and am excited about some possibilities, but also see several pitfalls that I don't know how to overcome with the current 4D language (at least not without slow workarounds). I'm hoping to receive some guidance from 4D on these thoughts as well as spark discussion from everyone about these new features. (To be clear, Thomas did not recommend converting existing applications to this approach. In fact, he recommended _not_ doing so. I have the luxury of having just started a new project, so I'm really interested in how this approach might work in real life.) - There has been worry about not being able to access interprocess variables from a pre-emptive worker process. If all the windows are in the same process, I would expect that quite a few interprocess variables could be converted to process variables. These variables could be passed to a worker process when it needed to do some work so it had the most recent values. For example, some of us keep a list of currently open windows for building a "Windows" menu. Probably most of us keep this in an interprocess array right now and protect access to it using a semaphore. If all windows (and menus) were in the same process, this could become a process array and would not need to be protected by a semaphore anymore. Having said that, I still think there is a need for workers to be able to occasionally access shared resources. I hope these features evolve so that shared resources can be accessed by protecting with a semaphore or some other locking mechanism. I understand that semaphores are somewhat expensive and, if used a lot, would negate the usefulness of pre-emptive processes since the underlying threads would be blocking so often. But I think they are still useful when used judiciously and are really quite fast when needed (just a small portion of a millisecond in most cases). - The documentation says the Application process is automatically a worker process. I thought this was interesting because it means that if all an application's windows are in this process, we can use CALL FORM to execute code within a window's context, but we can also use CALL WORKER to execute code in that process, but not tied to a specific window. This "smells" important to me, but I can't quite put my finger on why. Something to keep in mind, though. - I think this paradigm is interesting for progress indicators. I like the idea of showing progress right in the window with a spinner or thermometer. My current way to show progress is to open a progress window in another process since it is a bit tricky to update the UI of a window during the run loop. But if a worker process is doing all the work, the progress indicator could be shown in the window that started the work and the worker process could use CALL FORM periodically to tell it how much work was done. However, I see two problems I'm not sure how to overcome: 1. How can the user cancel some work that is being done? If I send a message using CALL WORKER to cancel the work, I don't think it gets the message until the work is done. Is it possible for a worker process to periodically check for new "messages" while in the middle of work? Or would the window itself have to chop up the work in some manner and only send the worker a bit at a time? That seems much slower as the two processes would have to have back and forth communication for each “block” of work. 2. By shifting the work to a worker process, the UI on the window stays active. In some cases this is desirable, but in others you may not want the user to be able to click or type anything in that window until the work is done. Something to consider. - In my current code, I tend to let almost every window have its own process. One reason for this is that the window has full control over selections and current records in every table. Even when the widows are array and variable based, this is very helpful. It seems like things could get very complicated if you had several windows open in the same process. For example, what if two of the windows were list windows for the same table and a third table was editing a record in that table? I suppose I could use named selections, but they
Re: Go watch Thomas Maul's presentation. Now.
1. has already been covered quite comprehensively by JPR; you got one half of the solution (chunking the task), the other half is that the worker calls itself recursively, instead of waiting instruction to drip. on a related point, when calling a worker (other than yourself), it is important to take into account that workers may finish their task in any order, so you need to coordinate for than when delegating a task to multiple workers. 2. I guess is will have to be managed logically by disabling the UI, perhaps by display a (translucent) image over the entire form. > 2016/09/06 6:53、Cannon Smith のメール： > > 1. How can the user cancel some work that is being done? If I send a > message using CALL WORKER to cancel the work, I don't think it gets the > message until the work is done. Is it possible for a worker process to > periodically check for new "messages" while in the middle of work? Or would > the window itself have to chop up the work in some manner and only send the > worker a bit at a time? That seems much slower as the two processes would > have to have back and forth communication for each “block” of work. > 2. By shifting the work to a worker process, the UI on the window stays > active. In some cases this is desirable, but in others you may not want the > user to be able to click or type anything in that window until the work is > done. Something to consider. 宮古 啓介 セールス・エンジニア 株式会社フォーディー・ジャパン 〒150-0043 東京都渋谷区道玄坂1-10-2 渋谷THビル6F Tel: 03-6427-8441 Fax: 03-6427-8449 keisuke.miy...@4d.com www.4D.com/JP ** 4D Internet Users Group (4D iNUG) FAQ: http://lists.4d.com/faqnug.html Archive: http://lists.4d.com/archives.html Options: http://lists.4d.com/mailman/options/4d_tech Unsub: mailto:4d_tech-unsubscr...@lists.4d.com **
Re: Go watch Thomas Maul's presentation. Now.
Hi Miyako, I think that makes sense, but let me make sure I understand. Let’s say a window needed a worker to process 1000 items in a loop. You are saying that the worker’s code should be constructed so that the window can pass the 1000 items to it in one message and then the worker will process, say, 10 items and then send a message to itself (CALL WORKER?). In most cases the worker will receive this message and know to process the next 10 items. But if the window had sent a cancel message in the mean time, that will be the message that comes off the stack first. So the worker will cancel the job. And I guess it has to be smart enough to ignore the last message it sent to itself. Is that correct? Thanks for the insight. -- Cannon Smith Synergy Farm Solutions Inc. Hill Spring, AB Canada 403-626-3236 > On Sep 5, 2016, at 6:43 PM, Keisuke Miyako wrote: > > 1. has already been covered quite comprehensively by JPR; you got one half of > the solution (chunking the task), the other half is that the worker calls > itself recursively, instead of waiting instruction to drip. on a related > point, when calling a worker (other than yourself), it is important to take > into account that workers may finish their task in any order, so you need to > coordinate for than when delegating a task to multiple workers. ** 4D Internet Users Group (4D iNUG) FAQ: http://lists.4d.com/faqnug.html Archive: http://lists.4d.com/archives.html Options: http://lists.4d.com/mailman/options/4d_tech Unsub: mailto:4d_tech-unsubscr...@lists.4d.com **
Re: Go watch Thomas Maul's presentation. Now.
On Mon, Sep 5, 2016 at 2:53 PM, Cannon Smith < can...@synergyfarmsolutions.com> > wrote: I also really enjoyed watching the video. It is a subject that I was > really excited about at the Summit and I appreciate the 4DMethod user group > making this possible. > Thanks for any thoughts! I'll jump in with a few thoughts...but they don't necessarily address your problems directly. I've got a different perspective on queues because I always think of them as multi-machine systems, which totally colors how I look at queuing and message features. First off, I'd say there's merit in thinking about CALL WINDOW (let's fact it, they'll have to rename it CALL WINDOW or CALL FORM WINDOW eventually...so why don't we pretend it's happened already) and CALL WORKER as discrete features. They can interact, but they don't need to. Personally, I'm a lot more interested in CALL WORKER, but CALL WINDOW is probably vastly more important for UI work in 4D. In any case, I'm just more interested in CALL WORKER as it relates 4D to the larger world of messaging more obviously. (And because I really, really like headless tasks and they're part of my core programming style.) So, skipping CALL WINDOW for now and thinking about CALL WORKER, here are some random thoughts: * Think of tasks that can be processed without thinking how or where. * If you really want to scale, you want to bring in resources past 4D server. A generalized message bus can help with this. Instead of everything having to happen on 4D Server or it's connected clients, you can have processing happen over a range of clients. They can be written as stand-alone 4D's, PHP, Python, Ruby, etc. How nice is that? That is very nice. Very. Put another way, want a great way to get more cores/thread/optimize 4D Server? Push work out onto other machines running code outside of 4D Server. For a lot of systems, there are _heaps_ of tasks that fit well with this. If you don't need to write data back to the 4D database, you can often farm the work out. (Sending messages, composing/preparing/building Web pages, generating SMS, performing various calculations, etc.) * If you want a generalized message queue...or if you think that you might want one...pass one parameter as your message, and make that message a JSON block. This should work fine with CALL WORKER or any other message queue. The point is that the payload of the message is JSON with whatever data you need...the mechanics of message transfer can be modified, as needed. For example, you might start with CALL WORKER, then move to SQS and 4D stand-alones that read the messages. This transition could be pretty tidy. * Yeah, push rather than pull-polling. The wrong polling model for messages can really kill performance and throughput for no gain. Don't think of client-server, think of publisher & consumer. And don't think of them as bits of software think of them as roles in a message exchange. Thomas has a nice sort of example of this as an aside in his talk. When you're implementing an RPC pattern with EXECUTE ON SERVER, he says not to poll the server from the client but, instead, to use EXECUTE ON CLIENT from the server to the client to deliver finished results. Lots more efficient because there isn't any extra checking for incomplete results. In this example, the client calls the server and the server calls the client. So, "client" and "server" don't mean anything - you've got one piece of software sending a message to another. More like email than a Web server. * IP variables, semaphores, etc. Forget it. Brittle, not going to persist, 4D specific, can't scale, a real headache. I've written code to do all of that, it can work perfectly...but it's easy to break. Use records. Just to be entirely clear, I'm not saying that IP variables (etc.) can't be programmed to work perfectly, I'm saying that it's a dead-end. It hasn't made sense in server-side programing for a long time. (Read: Since day 1.) If it doesn't work in triggers, it probably doesn't make sense in other server side code. Just my opinion, but this time I'm right ;-) * Speaking of records, you can have the fanciest message bus on earth and there are _still_ times when you're going to need a database to track state/condition of a task, make message persist, log things, etc. Not strange...very normal. Given that, if you've got very complex or large data to transfer, you can stash it in a record and then your message JSON should include a reference to the record so that the worker can retrieve all of the required details. This strategy will prepare you to deal with cloud-based message queues which often have fairly restrictive message payload limits. * But what if I'm in a transaction! Not a problem for me that often, but CALL WORKER gets around that quite beautifully in most scenarios that I can think of - you just threw the problem over the wall into another process. * Regarding passing selections, etc. A traditional (non-4D) way of dealing with this is to
Re: Go watch Thomas Maul's presentation. Now.
Great presentation, I'm also excited about the upcoming features. Here are some thoughts about the implementation details. CALL WORKER and CALL FORM are adding more 4D commands that take a method name as string. 4D does not have anything like closures or procedure pointers, so it introduces even more ways to break your code when you change a method name that is embedded somewhere as a string (or even a generated string). Don't forget about ON ERR CALL, New Process, and EXECUTE METHOD. It seems trivial to fix - just add a command, e.g. "Method name", that takes the *tokenized* method name and returns the method name as a string for use in any of the above commands. Now the design environment can really tell you where the method is referenced. If you agree, vote for the future request here: http://forums.4d.fr/Post/EN/17984703/1/17984704 My second concern is that a method cannot be preemptive if it calls a non-preemptive command. Any any callers of that method also cannot be preemptive. A compile time error is generated in all cases. This seems like an extreme limitation when it would be easy to arrange for the method to conditionally avoid calling the non-preemptive commands when in preemptive mode. Developers do this all the time for generic methods that need to be run in different contexts. A method that runs on the server can't show a user interface and it is easy to adjust the method to do something different when it runs on the server. The compile time error means that separate methods for the same functionality will often need to be created. For example, I have a "NET_Send" method which can transport a blob using a plugin or HTTP Post. Assuming the HTTP Post method allows preemption but the plugin does not, I now have to split that into two separate methods. If there are hundreds of callers to the original method, now I have to change all of that code creating duplicate methods just so one option can work in preemptive mode. Why not provide an option to get compile warnings instead of errors? Would it be that difficult for the compiler just to emit a runtime error if a non-preemptive command was called in a preemptive process? Looking forward to v16! John DeSoi, Ph.D. > On Sep 5, 2016, at 4:53 PM, Cannon Smith > wrote: > > I also really enjoyed watching the video. It is a subject that I was really > excited about at the Summit and I appreciate the 4DMethod user group making > this possible. ** 4D Internet Users Group (4D iNUG) FAQ: http://lists.4d.com/faqnug.html Archive: http://lists.4d.com/archives.html Options: http://lists.4d.com/mailman/options/4d_tech Unsub: mailto:4d_tech-unsubscr...@lists.4d.com **
Re: Go watch Thomas Maul's presentation. Now.
in reality, "preemptive or not" is a property of the thread, not the method. the compiler needs to now how to start a thread before your code reaches a conditional statement. a warning won't work, you will simply get a runtime error even if you find a way to "fool" the compiler. (for example, by conjuring a method name passed to "New process" by string concatenation) the idea is to leave all method properties to "agnostic", and only set "preemptive" for methods that specifically start a process (=thread); via New process, Execute on server, or a menu item. all generic subroutines will simply inherit the property of the thread in which it was called. no need for duplicates. > 2016/09/08 10:25、John DeSoi のメール： > > The compile time error means that separate methods for the same functionality > will often need to be created. For example, I have a "NET_Send" method which > can transport a blob using a plugin or HTTP Post. Assuming the HTTP Post > method allows preemption but the plugin does not, I now have to split that > into two separate methods. If there are hundreds of callers to the original > method, now I have to change all of that code creating duplicate methods just > so one option can work in preemptive mode. > > Why not provide an option to get compile warnings instead of errors? Would it > be that difficult for the compiler just to emit a runtime error if a > non-preemptive command was called in a preemptive process? 宮古 啓介 セールス・エンジニア 株式会社フォーディー・ジャパン 〒150-0043 東京都渋谷区道玄坂1-10-2 渋谷THビル6F Tel: 03-6427-8441 Fax: 03-6427-8449 keisuke.miy...@4d.com www.4D.com/JP ** 4D Internet Users Group (4D iNUG) FAQ: http://lists.4d.com/faqnug.html Archive: http://lists.4d.com/archives.html Options: http://lists.4d.com/mailman/options/4d_tech Unsub: mailto:4d_tech-unsubscr...@lists.4d.com **
Re: Go watch Thomas Maul's presentation. Now.
No need for duplicates as long as I don't care about having any pre-emptive processes, right? In the current design, the method that starts the process and every method in it's call chain must have only pre-emptive commands if I want it to run in pre-emptive mode. Correct? John DeSoi, Ph.D. > On Sep 7, 2016, at 10:46 PM, Keisuke Miyako wrote: > > the idea is to leave all method properties to "agnostic", > and only set "preemptive" for methods that specifically start a process > (=thread); > via New process, Execute on server, or a menu item. > > all generic subroutines will simply inherit the property of the thread in > which it was called. > > no need for duplicates. ** 4D Internet Users Group (4D iNUG) FAQ: http://lists.4d.com/faqnug.html Archive: http://lists.4d.com/archives.html Options: http://lists.4d.com/mailman/options/4d_tech Unsub: mailto:4d_tech-unsubscr...@lists.4d.com **
Re: Go watch Thomas Maul's presentation. Now.
I got to wondering how expensive it really is to used named selections everywhere and then move them to the current selection to do some work on them and then back again. It turns out that it is really fast if you use CUT NAMED SELECTION and USE NAMED SELECTION to move it back and forth and it doesn’t seem to matter how many records are in the named selection. I suspect that when using these two commands, internally 4D just updates a pointer or something like that. To test I created selections of varying sizes and then moved them into a named selection and then back into the current selection thousands of times. Each cycle averaged 3-8 microseconds (not milliseconds)! Maybe there are other gotchas I’m not thinking of, but based on these tests it seems like it would be possible to have all an application’s windows in the main process as long as they all used named selections and just moved them temporarily into the current selection to work on it before moving it back into the named selection. -- Cannon Smith Synergy Farm Solutions Inc. Hill Spring, AB Canada 403-626-3236 > On Sep 5, 2016, at 3:53 PM, Cannon Smith > wrote: > > - In my current code, I tend to let almost every window have its own process. > One reason for this is that the window has full control over selections and > current records in every table. Even when the widows are array and variable > based, this is very helpful. It seems like things could get very complicated > if you had several windows open in the same process. For example, what if two > of the windows were list windows for the same table and a third table was > editing a record in that table? I suppose I could use named selections, but > they seem limited. For example, I don't know how to programmatically ask a > named selection to sort itself a certain way without making it the current > selection and then pushing the current selection back into the named > selection. Is this an expensive operation? I don't know as I've not tested > it. Intuitively it seems like it would be. It is certainly expensive code > wise in the sense that I'd have to remember to never accidentally use the > current selection except to do temporary work for a named selection. ** 4D Internet Users Group (4D iNUG) FAQ: http://lists.4d.com/faqnug.html Archive: http://lists.4d.com/archives.html Options: http://lists.4d.com/mailman/options/4d_tech Unsub: mailto:4d_tech-unsubscr...@lists.4d.com **
Re: Go watch Thomas Maul's presentation. Now.
On Sep 13, 2016, at 2:00 PM, Cannon Smith wrote: > I got to wondering how expensive it really is to used named selections > everywhere and then move them to the current selection to do some work on > them and then back again. It turns out that it is really fast if you use CUT > NAMED SELECTION and USE NAMED SELECTION to move it back and forth and it > doesn’t seem to matter how many records are in the named selection. I suspect > that when using these two commands, internally 4D just updates a pointer or > something like that. You are exactly correct. I remember many years ago at a DEVCON, that was before the 4D Summits, Laurent speaking about named selections and how doing a CUT NAMED SELECTION was super efficient because it just copied a 4 byte pointer to the current selection to another location with a name assigned to it. Same for the USE SELECTION. Internally it is just a quick pointer assignment. This was back with 4D Server v1 and Macintosh 68K CPUs. A great 4D Server machine back then was a Quadra 900 with a 25MHz 68040 processor and 16MB or RAM. Yeah, 16 megabytes. The Quadra 900 had a list price of $8,500 in 1991. Tim Tim Nevels Innovative Solutions 785-749-3444 timnev...@mac.com ** 4D Internet Users Group (4D iNUG) FAQ: http://lists.4d.com/faqnug.html Archive: http://lists.4d.com/archives.html Options: http://lists.4d.com/mailman/options/4d_tech Unsub: mailto:4d_tech-unsubscr...@lists.4d.com **
Re: Go watch Thomas Maul's presentation. Now.
Hi Tim, I suspected as much. Thanks for the confirmation! -- Cannon Smith Synergy Farm Solutions Inc. Hill Spring, AB Canada 403-626-3236 > On Sep 13, 2016, at 2:27 PM, Tim Nevels wrote: > > You are exactly correct. I remember many years ago at a DEVCON, that was > before the 4D Summits, Laurent speaking about named selections and how doing > a CUT NAMED SELECTION was super efficient because it just copied a 4 byte > pointer to the current selection to another location with a name assigned to > it. Same for the USE SELECTION. Internally it is just a quick pointer > assignment. > ** 4D Internet Users Group (4D iNUG) FAQ: http://lists.4d.com/faqnug.html Archive: http://lists.4d.com/archives.html Options: http://lists.4d.com/mailman/options/4d_tech Unsub: mailto:4d_tech-unsubscr...@lists.4d.com **
Re: Go watch Thomas Maul's presentation. Now.
2 tips:
1.
CUT NAMED SELECTION doesn't unload the current record;
the current selection will be empty but you keep an out-of-selection current
record.
2.
you need to touch the highlight set after a CUT/COPY sequence on a listbox
e.g.
CUT NAMED SELECTION([Invoice];"$temp")
USE SET("$List1Highlight")
RELATE MANY SELECTION([InvoiceDetail]invoice)
USE NAMED SELECTION("$temp")
//release the current record
UNLOAD RECORD([Invoice])
//touch to force redraw
COPY SET("$List1Highlight";"$List1Highlight")
> 2016/09/13 23:57、Cannon Smith のメール：
>
> Maybe there are other gotchas I’m not thinking of
宮古 啓介
セールス・エンジニア
株式会社フォーディー・ジャパン
〒150-0043
東京都渋谷区道玄坂1-10-2 渋谷THビル6F
Tel: 03-6427-8441
Fax: 03-6427-8449
keisuke.miy...@4d.com
www.4D.com/JP
**
4D Internet Users Group (4D iNUG)
FAQ: http://lists.4d.com/faqnug.html
Archive: http://lists.4d.com/archives.html
Options: http://lists.4d.com/mailman/options/4d_tech
Unsub: mailto:4d_tech-unsubscr...@lists.4d.com
**
Re: Go watch Thomas Maul's presentation. Now.
Good to know. Thanks, Miyako! -- Cannon Smith Synergy Farm Solutions Inc. Hill Spring, AB Canada 403-626-3236 > On Sep 13, 2016, at 3:21 PM, Keisuke Miyako wrote: > > 2 tips: ** 4D Internet Users Group (4D iNUG) FAQ: http://lists.4d.com/faqnug.html Archive: http://lists.4d.com/archives.html Options: http://lists.4d.com/mailman/options/4d_tech Unsub: mailto:4d_tech-unsubscr...@lists.4d.com **
Re: Go watch Thomas Maul's presentation. Now.
Cannon, What's the benefit of doing that? In my typical work I deal with things that want to be open in multiple windows, like sales orders, items, trips, etc. So I haven't based anything on the 'traditional' output form/input form / drill-down architecture for - a really long time. Is that the kind of situation this would be a benefit for? It is cool to know these things are so speedy and have such low overhead. On Tue, Sep 13, 2016 at 7:57 AM, Cannon Smith < can...@synergyfarmsolutions.com> wrote: > it seems like it would be possible to have all an application’s windows > in the main process as long as they all used named selections and just > moved them temporarily into the current selection to work on it before > moving it back into the named selection. > -- Kirk Brooks San Francisco, CA === ** 4D Internet Users Group (4D iNUG) FAQ: http://lists.4d.com/faqnug.html Archive: http://lists.4d.com/archives.html Options: http://lists.4d.com/mailman/options/4d_tech Unsub: mailto:4d_tech-unsubscr...@lists.4d.com **
Re: Go watch Thomas Maul's presentation. Now.
maybe a bit late -
the biggest problem with named selections
they are static.
unlike a set, you can simply add or remove records from a named
selection.
Of course adding or removing records from a named selection would also
require resorting.
So the cost of using a named selection increases - possible a lot - if
you are trying to use them in a context where you need to add/delete
from the selection.
Also - depending on the sorting complexity, the sort could be
'expensive'
a simple sort on a field of the table in the named selection is not too
expensive,
however, a cross relational sort, or worse a sort by formula could get
very expensive.
And having to apply the sort criteria repeatedly could introduce delays
the user would find annoying to unacceptable.
Sets:
create set([table];"Myset")
Add to set([table];"Myset")
Create Set([table];"To_be_Unioned")
Union("Myset;"To_be_Unioned";"Myset")
Intersection("Myset;"To_be_Unioned";"Myset")
Difference("Myset;"To_be_Unioned";"Myset")
Use Set("Myset")
clear Set("Myset")
Order By([table];[table]Field;>)
** Note this only needs to be done once - just before display
Named selections
Copy Named Selection([table];"MySelection") (alternatively Cut Named
Selection)
Adding to it?
create record([table])
save record([table]
create set([table];"ToAdd")
use named selection("MySelection")
create set([table];"Add_to")
Union("Add_to";"ToAdd";"Add_to")
Order By([table];[table]Field;>)
Copy Named Selection([table];"MySelection") (alternatively Cut Named
Selection)
** Note this would need to be done every time a record was added
Deleting from?
see above, replace Create/save with a Query
I think you can see my point -
Yes Named selections have many uses, but I do not see them as a
replacement for sets when working with displayed lists of records where
users can/are adding removing records from the display/table
On Tue, 13 Sep 2016 08:57:39 -0600, Cannon Smith wrote:
> I got to wondering how expensive it really is to used named
> selections everywhere and then move them to the current selection to
> do some work on them and then back again. It turns out that it is
> really fast if you use CUT NAMED SELECTION and USE NAMED SELECTION to
> move it back and forth and it doesn’t seem to matter how many records
> are in the named selection. I suspect that when using these two
> commands, internally 4D just updates a pointer or something like that.
>
> To test I created selections of varying sizes and then moved them
> into a named selection and then back into the current selection
> thousands of times. Each cycle averaged 3-8 microseconds (not
> milliseconds)!
>
> Maybe there are other gotchas I’m not thinking of, but based on these
> tests it seems like it would be possible to have all an application’s
> windows in the main process as long as they all used named selections
> and just moved them temporarily into the current selection to work on
> it before moving it back into the named selection.
>
> --
> Cannon Smith
> Synergy Farm Solutions Inc.
> Hill Spring, AB Canada
> 403-626-3236
>
>
>
>
>> On Sep 5, 2016, at 3:53 PM, Cannon Smith
>> wrote:
>>
>> - In my current code, I tend to let almost every window have its own
>> process. One reason for this is that the window has full control
>> over selections and current records in every table. Even when the
>> widows are array and variable based, this is very helpful. It seems
>> like things could get very complicated if you had several windows
>> open in the same process. For example, what if two of the windows
>> were list windows for the same table and a third table was editing a
>> record in that table? I suppose I could use named selections, but
>> they seem limited. For example, I don't know how to programmatically
>> ask a named selection to sort itself a certain way without making it
>> the current selection and then pushing the current selection back
>> into the named selection. Is this an expensive operation? I don't
>> know as I've not tested it. Intuitively it seems like it would be.
>> It is certainly expensive code wise in the sense that I'd have to
>> remember to never accidentally use the current selection except to
>> do temporary work for a named selection.
>
> **
> 4D Internet Users Group (4D iNUG)
> FAQ: http://lists.4d.com/faqnug.html
> Archive: http://lists.4d.com/archives.html
> Options: http://lists.4d.com/mailman/options/4d_tech
> Unsub: mailto:4d_tech-unsubscr...@lists.4d.com
> **
**
4D Internet Users Group (4D iNUG)
FAQ: http://lists.4d.com/faqnug.html
Archive: http://lists.4d.com/archives.html
Options: http://lists.4d.com/mailman/options/4d_tech
Unsub: mailto:4d_tech-unsubscr...@lists.4d.com
*
Re: Go watch Thomas Maul's presentation. Now.
I don’t use the traditional output form/input form drill down either. But your question is exactly the one I’m trying to explore. I think most of us have traditionally programmed in 4D with one process for each window. Up until now all these “processes” were really on the same thread as far as the OS was concerned and 4D was doing the time slicing. With the introduction of preemptive processes, we now have the ability to have our code run on different OS threads. All the other environments I know of use one thread for all the UI and other threads as needed for background or potentially long running tasks. This is akin to putting all our windows in the Application process and using preemptive processes for background tasks. If I heard right, Thomas alluded to this as an idea if you had the chance to start an app from scratch. I suppose if we continue to use normal processes for windows and 4D continues to time-slice on the UI thread, it amounts to the same thing. So, is there a benefit to keeping all UI in the Application process? Does it out-weigh the costs of doing it? I’m really not sure, but that is what I’m trying to think through and understand. -- Cannon Smith Synergy Farm Solutions Inc. Hill Spring, AB Canada 403-626-3236 > On Sep 14, 2016, at 2:26 AM, Kirk Brooks wrote: > > What's the benefit of doing that? In my typical work I deal with things > that want to be open in multiple windows, like sales orders, items, trips, > etc. So I haven't based anything on the 'traditional' output form/input > form / drill-down architecture for - a really long time. Is that the kind > of situation this would be a benefit for? > ** 4D Internet Users Group (4D iNUG) FAQ: http://lists.4d.com/faqnug.html Archive: http://lists.4d.com/archives.html Options: http://lists.4d.com/mailman/options/4d_tech Unsub: mailto:4d_tech-unsubscr...@lists.4d.com **
Re: Go watch Thomas Maul's presentation. Now.
Hi Chip, Thanks for chiming in. I totally agree that named selections shouldn’t be used where sets can be used. I was only considering named selections in places where the current selection would normally be used. Intuitively I thought that moving a named selection into the current selection to work on it (sort, new query, add records, etc.) and then back into the named selection would be costly, but it turns out that it has almost zero cost when used in the specific way mentioned. Thanks. -- Cannon Smith Synergy Farm Solutions Inc. Hill Spring, AB Canada 403-626-3236 > On Sep 14, 2016, at 8:00 AM, Chip Scheide <4d_o...@pghrepository.org> wrote: > > I think you can see my point - > Yes Named selections have many uses, but I do not see them as a > replacement for sets when working with displayed lists of records where > users can/are adding removing records from the display/table > ** 4D Internet Users Group (4D iNUG) FAQ: http://lists.4d.com/faqnug.html Archive: http://lists.4d.com/archives.html Options: http://lists.4d.com/mailman/options/4d_tech Unsub: mailto:4d_tech-unsubscr...@lists.4d.com **
Re: Go watch Thomas Maul's presentation. Now.
Cannon, OK, I see what your talking about. Frankly I have no idea what the best approach is. This is the sort of thing I'll wait to hear what some of the 4D luminaries suggest. From what I am understanding about the benefits of multi-threading I'm not at all clear on how many actual applications will be able to actually take advantage of it. People who know a lot more about this sort of thing than I do seem to be saying it's actually a small subset of apps that can actually benefit from it. Plus to do so will require significant re-writes of code. And I'm hearing a pretty consistent message about that - namely don't do it. Be it taking advantage of multi-threading or fully exploiting c-objects. Though I have found the benefits of judicial refactoring to use c-objects worthwhile on balance. I will say that I have been moving in the direction of condensing, or modularizing, forms to work within the scope of themselves, as it were. The old habits of having forms directly manipulating IP vars and even process vars is something I'm reducing or eliminating. Form objects and dynamic variables allow this. This makes it possible to think about the window as its own scope, I suppose, though that hasn't really been my motivation. So to your point it facilitates containing the UI in a single, or fewer, processes. Which still begs your original question - does it really matter? On Wed, Sep 14, 2016 at 8:44 AM, Cannon Smith < can...@synergyfarmsolutions.com> wrote: > With the introduction of preemptive processes, we now have the ability to > have our code run on different OS threads. All the other environments I > know of use one thread for all the UI and other threads as needed for > background or potentially long running tasks. This is akin to putting all > our windows in the Application process and using preemptive processes for > background tasks. If I heard right, Thomas alluded to this as an idea if > you had the chance to start an app from scratch. > > I suppose if we continue to use normal processes for windows and 4D > continues to time-slice on the UI thread, it amounts to the same thing. So, > is there a benefit to keeping all UI in the Application process? Does it > out-weigh the costs of doing it? I’m really not sure, but that is what I’m > trying to think through and understand. > -- Kirk Brooks San Francisco, CA === ** 4D Internet Users Group (4D iNUG) FAQ: http://lists.4d.com/faqnug.html Archive: http://lists.4d.com/archives.html Options: http://lists.4d.com/mailman/options/4d_tech Unsub: mailto:4d_tech-unsubscr...@lists.4d.com **
Re: Go watch Thomas Maul's presentation. Now.
Kirk Brooks wrote: >What's the benefit of doing that? In my typical work I deal with things >that want to be open in multiple windows, like sales orders, items, trips, >etc. So I haven't based anything on the 'traditional' output form/input >form / drill-down architecture for - a really long time. Is that the kind >of situation this would be a benefit for? How different is your interface from the traditional one? -- -- Tom Dillon 825 N. 500 W. DataCraft Moab, UT 84532 tomdil...@datacraft-inc.com 720/209-6502 -- Imagination is more important than knowledge, because everything we know, first had to be imagined. --- Albert Einstein -- ** 4D Internet Users Group (4D iNUG) FAQ: http://lists.4d.com/faqnug.html Archive: http://lists.4d.com/archives.html Options: http://lists.4d.com/mailman/options/4d_tech Unsub: mailto:4d_tech-unsubscr...@lists.4d.com **
Re: Go watch Thomas Maul's presentation. Now.
Hey Tom, I get the sense a lot of 4D dbs are built around the idea you choose a table by starting with an output list form, do some sort of query to narrow down the selection, double click on a record to open it into an input form which may show lists of related records and then you drill down into those. The key thing is all this is taking place in the single window. My approach has been to make an interface for some aspect of the total 'job' and build lists that relate to that. I work with Orders a lot so I've got several interfaces that provide ways to look at lists of them. 'My List' let's users choose lists of order relevant to them, like in process, quotes, etc. There's also an Order Finder that let's 'em find orders by simple criteria: number, name, address, etc. I even build a popup list of the most recent 10 orders when you right click on a customer name in the 'Partner Picker'. Regardless of where you ask to look at an order it opens a new window, in a new process and displays the order. Unlike the drill-down approach this let's you open several orders at the same time and work with them. Having said all that I don't have any real data on this - just perceptions from conversations. On Wed, Sep 14, 2016 at 12:16 PM, Tom Dillon wrote: > How different is your interface from the traditional one? > -- Kirk Brooks San Francisco, CA === ** 4D Internet Users Group (4D iNUG) FAQ: http://lists.4d.com/faqnug.html Archive: http://lists.4d.com/archives.html Options: http://lists.4d.com/mailman/options/4d_tech Unsub: mailto:4d_tech-unsubscr...@lists.4d.com **
Re: Go watch Thomas Maul's presentation. Now.
> On Sep 14, 2016, at 10:44 AM, Cannon Smith > wrote: > > All the other environments I know of use one thread for all the UI and other > threads as needed for background or potentially long running tasks. This is > akin to putting all our windows in the Application process and using > preemptive processes for background tasks. If I heard right, Thomas alluded > to this as an idea if you had the chance to start an app from scratch. It is a terrible limitation and one of the reasons I stopped using some other development tools. For example, if a process wants to get a user confirmation it must call the main process to show the dialog. Then the dialog has to make *callbacks* back to the process to continue (CALL WORKER?, CALL FORM?). Things that can be handled very simply in 4D a few lines of conditional statements turn in to multiple procedures having to deal with interprocess communication. John DeSoi, Ph.D. ** 4D Internet Users Group (4D iNUG) FAQ: http://lists.4d.com/faqnug.html Archive: http://lists.4d.com/archives.html Options: http://lists.4d.com/mailman/options/4d_tech Unsub: mailto:4d_tech-unsubscr...@lists.4d.com **
Re: Go watch Thomas Maul's presentation. Now.
Yep, for sure. I’d love to hear more about this aspect from 4D itself. There is lots of information about messaging between processes, but not much on this part yet. Thanks. -- Cannon Smith Synergy Farm Solutions Inc. Hill Spring, AB Canada 403-626-3236 > On Sep 14, 2016, at 11:32 AM, Kirk Brooks wrote: > > Which still begs your original question - does it really matter? ** 4D Internet Users Group (4D iNUG) FAQ: http://lists.4d.com/faqnug.html Archive: http://lists.4d.com/archives.html Options: http://lists.4d.com/mailman/options/4d_tech Unsub: mailto:4d_tech-unsubscr...@lists.4d.com **
Re: Go watch Thomas Maul's presentation. Now.
Hi John, I wondered about that. I would have thought that would be pretty rare if the code was structured right, but it sounds like you ran into it quite often? Thanks for the feedback. -- Cannon Smith Synergy Farm Solutions Inc. Hill Spring, AB Canada 403-626-3236 > On Sep 14, 2016, at 2:56 PM, John DeSoi wrote: > > It is a terrible limitation and one of the reasons I stopped using some other > development tools. For example, if a process wants to get a user confirmation > it must call the main process to show the dialog. Then the dialog has to make > *callbacks* back to the process to continue (CALL WORKER?, CALL FORM?). > Things that can be handled very simply in 4D a few lines of conditional > statements turn in to multiple procedures having to deal with interprocess > communication. ** 4D Internet Users Group (4D iNUG) FAQ: http://lists.4d.com/faqnug.html Archive: http://lists.4d.com/archives.html Options: http://lists.4d.com/mailman/options/4d_tech Unsub: mailto:4d_tech-unsubscr...@lists.4d.com **
Re: Go watch Thomas Maul's presentation. Now.
Hi Cannon, > On Sep 14, 2016, at 4:35 PM, Cannon Smith > wrote: > > I wondered about that. I would have thought that would be pretty rare if the > code was structured right, but it sounds like you ran into it quite often? Yes, I ran into it all the time and it has nothing to do with problem code structure. It has to do with two rules that are common in Mac environments that support multi-processing. 1. All user interface and interaction must occur on the main thread. 2. No I/O or long execution can occur on the main thread -- otherwise the interface is locked up. So everything ends up being done with callbacks. It is not just the Mac, consider any modern Javascript environment. If you make a I/O request, you can't just wait for the result and write an if/then statement to handle it. Everything is done by passing callbacks and closures around. In my opinion, it is more difficult to read, write, and maintain. John DeSoi, Ph.D. ** 4D Internet Users Group (4D iNUG) FAQ: http://lists.4d.com/faqnug.html Archive: http://lists.4d.com/archives.html Options: http://lists.4d.com/mailman/options/4d_tech Unsub: mailto:4d_tech-unsubscr...@lists.4d.com **
Re: Go watch Thomas Maul's presentation. Now.
On Sep 14, 2016, at 4:35 PM, Cannon Smith wrote: > Yep, for sure. I’d love to hear more about this aspect from 4D itself. There > is lots of information about messaging between processes, but not much on > this part yet. Here’s my perspective on new 4D features. They are great and they can be useful. Can be useful to you. May be useful to you. But maybe not. Just because 4D adds a new feature to 4D does not mean all developers should and/or need to use that new feature. New features are a new tool you may find useful. But you may find for your particular application you have no need for new feature. 4D commands and features are like tools in your toolbox. Example: Say you are a carpenter and you build typical wood frame houses. Someone releases a MIG welder that uses a new technology to automatically select the proper type of shielding gas and mixture. You don’t have to choose argon, oxygen, helium, carbon dioxide or the mixture. This is a great feature… if you are a welder. But if you work 99.9% of the time with wood, you’ll find it very hard to find a use for this new MIG welder. Here is a 4D example: v3 and multiple processes and multiple windows. Fantastic 4D feature I use all the time in almost all my 4D projects, except for one. I wanted to update that app to have multiple windows. I thought it would be so much more useful to the client. So I did a little work and made a demo with this app of how it could work with multiple windows and showed it to the client hoping to get approval to do the upgrade work on the app. I just knew they were going to love it. They hated it! I was shocked. They complained this made things too complicated. They liked only having one window open at a time. They didn’t want to have all this “power”. They wanted to do only one thing at a time. They had no need or desire to have several windows open at once to do comparison of information or any of the other reasons I could come up with. So this great feature of multiple windows open at once in 4D was a feature this client did not want any part of. I’m going to continue to evaluate the new worker processes and preemptive features in v16. Right now I don’t have any existing problems or situations where I can immediately see it would be valuable. Maybe there will be in the future. Maybe not. I’m glad I will have this new tool in my toolbox so that if I do run into a situation where I need it, I can use it. So what would be a good use or a “need” for worker processes? From what I understand so far, the primary value of worker processes is the ability speed of a time consuming process by breaking it into pieces and spreading the load across multiple processors/cores. Say you have a super complex query that currently takes 10 seconds to run. You’d like it to run in 2 seconds. You figure out a way to break the query into 5 parts. Each part runs in 2 seconds and you can combine the results of all 5 parts easily. Now having 5 worker processes that are pre-emptive, can all run simultaneously and can use all the cores in your CPU is exactly the tool you need. In a short time you can improved the speed of this operation by a factor of 5. Before we had a tool like this you would have probably tried to do some kind of caching of query results or hashing or some other scheme to speed up that slow query. Maybe a lot more work. More tables, updating data in triggers and testing, testing, testing. Using worker processes avoids all this. You just use brute force and the full power of your computer to get the job done faster. I’m not sure there is going to be a lot of value in trying to use worker processes to do all the processing and let the UI process be stupid and just sit there waiting for a message from a worker that he’s got the work done that you need unless your UI is currently sluggish due to excessive processing going on. Is that the problem you are trying to resolve using worker processes? Or do you want to use worker processes just for the sake of using them and the pure enjoyment of doing it. That can also be a valid reason. :) Tim Tim Nevels Innovative Solutions 785-749-3444 timnev...@mac.com ** 4D Internet Users Group (4D iNUG) FAQ: http://lists.4d.com/faqnug.html Archive: http://lists.4d.com/archives.html Options: http://lists.4d.com/mailman/options/4d_tech Unsub: mailto:4d_tech-unsubscr...@lists.4d.com **
Re: Go watch Thomas Maul's presentation. Now.
> On Sep 14, 2016, at 6:32 PM, John DeSoi wrote: > > Hi Cannon, > >> On Sep 14, 2016, at 4:35 PM, Cannon Smith >> wrote: >> >> I wondered about that. I would have thought that would be pretty rare if the >> code was structured right, but it sounds like you ran into it quite often? > > Yes, I ran into it all the time and it has nothing to do with problem code > structure. It has to do with two rules that are common in Mac environments > that support multi-processing. > > 1. All user interface and interaction must occur on the main thread. > > 2. No I/O or long execution can occur on the main thread -- otherwise the > interface is locked up. > > So everything ends up being done with callbacks. I only started writing Cocoa apps about 10 years ago, but I haven’t run into any real problems with the Cocoa way of doing things. Especially now with NSOperationQueue/NSOperation or GCD and blocks, it has become very easy to write multi-threaded code. Back in the days of using NSThread it was probably more work and more annoying, but even then there was performSelector:onMainThread:waitUntilDone:. Nowadays there are so many more options. I think the only time I’ve ever even had to implement a callback is for file system event notifications. However, I’d be hard-pressed to think of anything to use 4D’s implementation of worker threads (as I’ve heard them discussed) that wouldn’t be easier to accomplish some other way. For example, will CALL FORM be able to be implemented as elegantly as Cocoa’s NSNotification system? Or will it be the same level of terrible as CALL PROCESS and On Outside Call? Jim Crate ** 4D Internet Users Group (4D iNUG) FAQ: http://lists.4d.com/faqnug.html Archive: http://lists.4d.com/archives.html Options: http://lists.4d.com/mailman/options/4d_tech Unsub: mailto:4d_tech-unsubscr...@lists.4d.com **
Re: Go watch Thomas Maul's presentation. Now.
> On Thu, Sep 15, 2016 at 8:36 AM, Tim Nevels wrote: > Here’s my perspective on new 4D features. They are great and they can be > useful. Can be useful to you. May be useful to you. But maybe not. > So I did a little work and made a demo with this app of how it could work with > multiple windows and showed it to the client hoping to get approval to do the > upgrade work on the app. I just knew they were going to love it. > > They hated it! Thanks for posting this, I've seen it too. Yeah, it's true - what we like often isn't representative of what end users want. Single-window apps are becoming more and more the norm so what you found is only more likely going down the road. > Or do you want to use worker processes just for the sake of using them and the > pure enjoyment of doing it. That can also be a valid reason. :) I'm not that worked up about CALL FORM WINDOW, but I can think of a few places where it would be very nice. Then again, I'm *way* more into headless code than UI work. (I do plenty of UI work, but I like headless code.) Getting CALL WORKER to support a UI? I haven't really thought about it that much...that's not the most obvious case for me. A lot of UI-bound tasks aren't straightforward to parallelize and then there's a lot of threshing/overhead potential in managing the messaging. On the other hand, I'm really, really, really into background tasks. Don't think of CALL WORKER as a subordinate tool for CALL FORM WINDOW. They're totally different, at least in my mind. As I probably said at the top of this thread, I've been writing task queues for about 30 years (!) and can't imagine life without them. So many uses. If you want to scale (really scale), you need distributed processing. If you want to manage distributed processing, you need a task/message-oriented architecture. Does CALL WORKER fit into this? Kind of. Not 100% since it seems to be single-machine oriented, but it's a great first step and, if you set your code up right, you can pretty seamlessly extend it to a multi-machine or multi-language solution based on records (which also adds persistence, if you need it, and traceability when you want it) and/or an external queuing system (IronMQ looks pretty good, I have to say.) A few other notes: * If anyone is thinking about how to use process/IP vars in CALL WORKER, you're thinking about it wrong. Not because you're a bad person, and not primarily for technical reasons. A task should be *atomic* and not bound to the original calling context. Anything else is a huge, snarly, evil box of goo. Filled with rusty razorblades. Forget variables, they're completely unhelpful in this context. Unless you want to stand on a bunch of gooey, rusty razorblades. * Pre-emptive vs. cooperative threads. Is this the new "For loops are faster" of the 4D world? Seriously, most of us and most of our apps absolutely won't know or care. The guys in France that write 4D and 4D Server? Yes, they should care. A lot. Us? Not so much. I'm filing under "worry about it when I've got a reason to". Apart from that, I'll go with the words of Bill Atkinson (he wrote QuickDraw and HyperCard, back in the day): "How do you get a 400% optimization? Stop doing something stupid." So, I'll just try not to do obviously stupid things ;-) * Don't think of messaging as client-server. Think of it as sender and recipient. Then a lot of the confusion/complexity about callbacks looks different. A callback is a return address. The state of the art has changed in the field a lot in the past decades...queuing has gone from being a arcane, back-office/mainframe subject to a core component of the software we use every day. The right or wrong polling strategy can make or break you. This isn't news...but now it's common knowledge. (Long-polling or push instead of constant repolling. Constant repolling is the naive approach...I've used it...it's obvious...I now hand my head in shame. It really does saturate resources way too fast for no reason. I like push everywhere now.) CALL WORKER has a lot of potential and I think will be a gentle introduction to task queues for a lot of 4D developers. They look dead easy to use. Impressively so...way to go 4D. Given their limits (as I understand them), I'll repeat a few recommendations or anyone that thinks that they _might_ move towards a multi-machine or multi-platform system in the future: 1) Ignore process and IP variables. Period. (In fact, I'd recommend this no matter what - mingling variables this way is kind of a crappy way to code anyway.) 2) Make your payload one parameter, and make that JSON. (Real JSON that any language can parse.) 3) If you need larger/more complex data, store it in a record/file/whatever and include a reference in the JSON that enables the worker/task process to load the details. That's it. If you don't ever plan to go beyond the built-in feature set, then I think that point #2 can be ignored. It's pretty sweet that 4D lets you pass raw parameters in -
Re: Go watch Thomas Maul's presentation. Now.
I could possibly agree with this if objects were more powerful and complete, but in the current implementation it just invites mistakes and maintenance issues. Object property names are just strings. Unless you build your own system where every property is an object that can provide meta-data, you can't even determine the type of the property. 4D objects are not self-describing as in other object systems. Having variables (and parameters) allows type checking and cross referencing to understand where things are used in the code base. If I have 100 instances of "name" as a property in my code, there is no way to safely make a change to that if I only want it to impact a subset of my objects. If you could declare constants in code without restarting 4D, it might change my mind. John DeSoi, Ph.D. > On Sep 14, 2016, at 6:44 PM, David Adams wrote: > > 1) Ignore process and IP variables. Period. (In fact, I'd recommend this no > matter what - mingling variables this way is kind of a crappy way to code > anyway.) > > 2) Make your payload one parameter, and make that JSON. (Real JSON that any > language can parse.) ** 4D Internet Users Group (4D iNUG) FAQ: http://lists.4d.com/faqnug.html Archive: http://lists.4d.com/archives.html Options: http://lists.4d.com/mailman/options/4d_tech Unsub: mailto:4d_tech-unsubscr...@lists.4d.com **
Re: Go watch Thomas Maul's presentation. Now.
On Thu, Sep 15, 2016 at 11:11 AM, John DeSoi wrote: > I could possibly agree with this if objects were more powerful and complete, > but in the current implementation it just invites mistakes and maintenance > issues. Object property names are just strings. Unless you build your own > system where every property is an object that can provide meta-data, you can't > even determine the type of the property. 4D objects are not self-describing as > in other object systems. Having variables (and parameters) allows type > checking and cross referencing to understand where things are used in the code > base. If I have 100 instances of "name" as a property in my code, there is no > way to safely make a change to that if I only want it to impact a subset of my > objects. All fair criticisms and observations. For sure, 4D's lack of understanding of types past the built-in simples is a long-standing frustration. C_OBJECT is kind of misleading -they're not structs/records/objects... they're more like C_STRUCTURED_TEXT, as you point out. For anyone planning to use CALL WORKER solely as designed, then the direct parameter passing looks great. This is what Thomas demonstrated and it makes for some very simple+powerful+readable code. Alternatively, the just-one-JSON-block probably adds complexity with no benefit to most examples, in this case. But you have to pick your fights in 4D (or anything else)...If you think that you'll want to move to a larger system with custom features, you'll need records or documents and, at that point, the direct primitive parameters are likely to be a pain. They don't translate outside of 4D and don't map easily to anything _in_ 4D, apart from a big BLOB. In case it's not clear, I've been down this particular road before...I was going with specific fieldsa big BLOB was ultimately easier to work with.JSON is the new BLOB ;-) Anyway with just one data container, your transport and data storage code can be written once and then forgotten about. For what it's worth, the task queues that I've written or been involved with in the past didn't have a huge number of different message types. So, it wasn't a matter of hundreds of different messages - more like a dozen or two. At that point, the code for composing and then parsing a message isn't that hard to verify. For every composition (sender) you've got a reader (receiver.) They need to be kept in sync. Not entirely generic, but error-free once you're done. (Entirely generic is overrated, IMO...and I've been guiltier than many of going too far in that direction.) Put another way, once the message format is stable and the code to build/consume the message is verified, you're done. Nothing needs to change in that area again. Message formats *should* end up being very stable over time or else something else is going wrong in the system and/or it's early days and your'e still in discovery mode. Oh, reminder: Put version numbers in messages as part of your standard format so that you know if you're getting an obsolete format, want to switch up your parser code to support multiple versions, etc. ** 4D Internet Users Group (4D iNUG) FAQ: http://lists.4d.com/faqnug.html Archive: http://lists.4d.com/archives.html Options: http://lists.4d.com/mailman/options/4d_tech Unsub: mailto:4d_tech-unsubscr...@lists.4d.com **
Re: Go watch Thomas Maul's presentation. Now.
John, On Wed, Sep 14, 2016 at 6:11 PM, John DeSoi wrote: > If you could declare constants in code without restarting 4D, it might > change my mind. > I'm using Vincent's excellent 4DPOP constants module to do just this. Yeah you have to do some restarts while you're developing but if you set up the constant names well (and this could be another thread unto itself) you have a nice popup set of key names to use for constructing c-objs. I just completed a big refactoring of some code along David's description. The idea is to have 4D assemble some complex data into a JSON which can then be handed off to a 4D form for display and editing or to a web page. The edits go into the JSON and that gets handed back to 4D - and 4D knows what to do with it. This issue of consistently naming the keys quickly appeared - even with many accessor methods. I found the constants a great help. -- Kirk Brooks San Francisco, CA === ** 4D Internet Users Group (4D iNUG) FAQ: http://lists.4d.com/faqnug.html Archive: http://lists.4d.com/archives.html Options: http://lists.4d.com/mailman/options/4d_tech Unsub: mailto:4d_tech-unsubscr...@lists.4d.com **
Re: Go watch Thomas Maul's presentation. Now.
actually, I beg to differ on this.
inter-process variables, yes, I suppose you can work around them.
but I have not yet found a good way to write worker code without using process
variables.
true, you can run a worker recursively, passing local/contextual information as
arguments packed as an object,
but what about large data or more specifically, arrays?
of course you can insert the array in an object too,
but as I understand, an object passed to a worker is copied, not referenced,
so I would imagine there is going to be a huge over-head
when, for example, a worker is progressively building an array in the
background.
a process variable, to me, seems to be the ideal data buffer for such tasks.
given that a preemptive worker has no UI,
the variable is insulated from other processes,
unlike classic process variables that are bound to a form object.
but then, any process can run a method in that worker's context,
so the real gem of preemptive coding in 4D could well be in
how the methods are logically organised.
perhaps one can ASSERT that a method can only be called from a certain worker.
so the standard signature could be something like
MyWorker (Current process name;object{;object}...)
so that the worker always know who called it.
finally, in development,
a worker can be aborted (and the process variables destroyed),
but a caller might keep calling it in a loop and thus create a new instance,
so it might be useful to protect the recursive code in the callee method that
uses process variables.
> 2016/09/15 8:44、David Adams のメール：
>
> 1) Ignore process and IP variables. Period. (In fact, I'd recommend this no
> matter what - mingling variables this way is kind of a crappy way to code
> anyway.)
宮古 啓介
セールス・エンジニア
株式会社フォーディー・ジャパン
〒150-0043
東京都渋谷区道玄坂1-10-2 渋谷THビル6F
Tel: 03-6427-8441
Fax: 03-6427-8449
keisuke.miy...@4d.com
www.4D.com/JP
**
4D Internet Users Group (4D iNUG)
FAQ: http://lists.4d.com/faqnug.html
Archive: http://lists.4d.com/archives.html
Options: http://lists.4d.com/mailman/options/4d_tech
Unsub: mailto:4d_tech-unsubscr...@lists.4d.com
**
Re: Go watch Thomas Maul's presentation. Now.
On Thursday, September 15, 2016, Keisuke Miyako wrote: > actually, I beg to differ on this. > > inter-process variables, yes, I suppose you can work around them. > > but I have not yet found a good way to write worker code without using > process variables. > > Right, I said something stupid. I think that's the technical term ;-) I was only thinking of process vars used for IP communication with the set/get commands. Within a worker? Whatever works. If you scale out to a larger system, you have to consider what works there. If you're in another language, the code will be a complete rewrite anyway. Put more succinctly, I'm saying * limit messages to parameters * if you expect to move beyond the built-in feature set, consolidate on a JSON block (real JSON for interopability) * if you need to pass something large or unwieldy, stick it in a resource and pass the resource reference in the JSON. If you do all of that and later find you want to run jobs in Python using SQS, you'll be in good shape. For 'SQS', insert your queue name here and for 'Python', insert your language. Everything speaks HTTP, SQL, and JSON these days. ** 4D Internet Users Group (4D iNUG) FAQ: http://lists.4d.com/faqnug.html Archive: http://lists.4d.com/archives.html Options: http://lists.4d.com/mailman/options/4d_tech Unsub: mailto:4d_tech-unsubscr...@lists.4d.com **
Re: Go watch Thomas Maul's presentation. Now.
Great presentation and such an awesome new feature!
Really like your example for logging David, CALL WORKER makes this so easy/fast
without messing with semaphores and such. Huge deal for job queues.
I've been working on some front end stuff and have come up with least 3 really
nice use cases for using CALL WORKER with CALL FORM. All of these examples can
be done without CALL WORKER/FORM, but they seem to make it easier...
1. Async data fetching
If you want async data fetching just like you have with ajax in the web
browser, now it really easy to setup. And if you are fetching data from a 3rd
party it is probably a good idea. For instance, while 4D is running an HTTP
GET, the user interface is locked. If you are getting data from a 3rd party
you can't be sure how long it will take. So you might lock up the interface
for a while as HTTP GET is waiting for a response. That is a bad experience
for the user. Now you can just call a worker to get the data and your UI is
still responsive. When the worker is done it can call the form with the
retrieved data.
- CALL WORKER("HttpGetAsync1";"HttpGetAsync";$Input_o;"CallbackMethod";Current
form window)
- When the worker is done it calls CallbackMethod($Output_o)
2. Lots of async data fetching
What if you want to pull a bunch of stuff from a 3rd party to display in the
UI, how about a bunch of pictures from Amazon S3? Don't just grab one at a
time, do what a web browser would do and fire up 8 workers and make 8 http
requests at a time to pull them faster. Same concept as above but just cycle
through calling workers named HttpGetAsync1, HttpGetAsync2, HttpGetAsync3, etc.
Once you get to 8, cycle back and call HttpGetAsync1 again. What's so great
about CALL WORKER is you don't have to worry about whether HttpGetAsync1 is
already busy making a request, just keep calling it and 4D will handle queuing
all the requests for you.
3. Modal screen example
What if you want a modal screen behind a modal dialog like in this example:
http://www.w3schools.com/bootstrap/bootstrap_modal.asp
Now it is really easy:
MainForm_WinRef_l:=Current form window
DIALOG("MyForm";*) //note the asterisk, learned from Thomas's presentation
OBJECT SET VISIBLE(*;"ModalScreen";True)
Now when you close the dialog just call the window to tell it to hide the
ModalScreen object:
CALL FORM(MainForm_WinRef_l;"HideModalScreen")
Code in HideModalScreen is just:
OBJECT SET VISIBLE(*;"ModalScreen";False)
Fun stuff,
Welsh
**
4D Internet Users Group (4D iNUG)
FAQ: http://lists.4d.com/faqnug.html
Archive: http://lists.4d.com/archives.html
Options: http://lists.4d.com/mailman/options/4d_tech
Unsub: mailto:4d_tech-unsubscr...@lists.4d.com
**
Re: Go watch Thomas Maul's presentation. Now.
Thanks James and John. I appreciate your perspectives. -- Cannon Smith Synergy Farm Solutions Inc. Hill Spring, AB Canada 403-626-3236 > On Sep 14, 2016, at 5:27 PM, James Crate wrote: > > I only started writing Cocoa apps about 10 years ago, but I haven’t run into > any real problems with the Cocoa way of doing things. Especially now with > NSOperationQueue/NSOperation or GCD and blocks, it has become very easy to > write multi-threaded code. Back in the days of using NSThread it was probably > more work and more annoying, but even then there was > performSelector:onMainThread:waitUntilDone:. Nowadays there are so many more > options. I think the only time I’ve ever even had to implement a callback is > for file system event notifications. ** 4D Internet Users Group (4D iNUG) FAQ: http://lists.4d.com/faqnug.html Archive: http://lists.4d.com/archives.html Options: http://lists.4d.com/mailman/options/4d_tech Unsub: mailto:4d_tech-unsubscr...@lists.4d.com **
Re: Go watch Thomas Maul's presentation. Now.
Just to poke at Dave a bit :) On Wed, 14 Sep 2016 20:11:44 -0500, John DeSoi wrote: > If you could declare constants in code without restarting 4D, it > might change my mind. Isn't this what IP variables are for? Chip ** 4D Internet Users Group (4D iNUG) FAQ: http://lists.4d.com/faqnug.html Archive: http://lists.4d.com/archives.html Options: http://lists.4d.com/mailman/options/4d_tech Unsub: mailto:4d_tech-unsubscr...@lists.4d.com **
Re: Go watch Thomas Maul's presentation. Now.
On Thu, Sep 15, 2016 at 11:34 PM, Welsh Harris wrote: > Great presentation and such an awesome new feature! > > Really like your example for logging David, CALL WORKER makes this so > easy/fast without messing with semaphores and such. Huge deal for job queues. > > I've been working on some front end stuff and have come up with least 3 really > nice use cases for using CALL WORKER with CALL FORM. All of these examples > can be done without CALL WORKER/FORM, but they seem to make it easier... Hey Welsh, good post. You're right, CALL WORKER + CALL FORM could be really great for any sort of real-time/streaming data. From the logging example, perhaps the log worker posts critical errors to a display form. So, only important stuff is showing up in real-time in the UI while everything is still getting logged normally. I guess that you could filter and selectively display all sorts of data: graphs, lists filtered by user/machine/etc. That all fits into what Thomas showed, too. I liked the way his demo used two instances of the same form to show different information in different windows, for example. And, yeah, you could do all of the above in native 4D now but...so much uglier and harder. ** 4D Internet Users Group (4D iNUG) FAQ: http://lists.4d.com/faqnug.html Archive: http://lists.4d.com/archives.html Options: http://lists.4d.com/mailman/options/4d_tech Unsub: mailto:4d_tech-unsubscr...@lists.4d.com **
Re: Go watch Thomas Maul's presentation. Now.
aOn Thu, Sep 15, 2016 at 11:11 AM, John DeSoi wrote: > I could possibly agree with this if objects were more powerful and complete, > but in the current implementation it just invites mistakes and maintenance > issues. Object property names are just strings. Unless you build your own > system where every property is an object that can provide meta-data, you can't > even determine the type of the property. 4D objects are not self-describing as > in other object systems. Having variables (and parameters) allows type > checking and cross referencing to understand where things are used in the code > base. If I have 100 instances of "name" as a property in my code, there is no > way to safely make a change to that if I only want it to impact a subset of my > objects. I was just thinking about John's post again and think it's worth more comment. John's points are all right. If you look at Thomas' presentation, you'll see how clear and readable calls to CALL WORKER on when you pass parameters. In contrast, passing through a JSON block or reference of some kind is completely opaque. You can't see what's going on from the call, you have to figure out what's going into the JSON. This is potentially a quick road to hard to debug, hard to trace, sloppy code. I truly detest the habit in JavaScript to pass around these ginormous JSON manifests. Ugh. (Particularly in JavaScript where closures are super nice...we just don't have anything like them in 4D.) Anyway, I can clearly complain about anything ;-) I've been recommending a JSON-messaging strategy for a *minority* of cases. Namely, the situations where CALL WORKER is going to be extended to a multi-machine setup of any kind. Say, for example, that you want to pass message to a compiled-and-merged app. That's a sweet setup. You can get very stable little 'clients' using pure 4D code. In fact, you can set up build parameters so that you main system and the compiled-and-merged versions come out of the same code base. In that situation, you'll be happy to have JSON blocks. But how often is this going to be the case? I don't know. I'd guess the minority - maybe never for some people. (For me it's 'always' because I'm used to distributed processing and that's how I think about it...but that's me.) Is it worth overbuilding something for a minority case you might not have? Usually not. Sometimes it is and then it's a bit+ painful to do a rewrite...but what about the times you overwrite Just In Case and never need the extra features? I always feel foolish for having wasted the time and effort to make a *more* complex system for zero gain. So potentially, a painful rewrite or potentially hanging your head in shame over pointlessly complex code. Like so many things, judgement call. So: * If you're only ever going to use CALL WORKER on one machine, as designed, use raw parameters. This makes for much better and more readable 4D code that a new programmer can pick up right away. * If you're going to extend or replace CALL WORKER _for sure_, I'd suggest starting with JSON blocks and working out some generalized tools for naming/documentation/parsing/building/validating along the way. * If you _might_ extend or replace CALL WORKER, it's a tougher call. Go with your gut. In any case, you could always write a CALL WORKER adapter where your normal CALL WORKER handlers write reformatted messages as JSON blocks for your extended system. File under "it depends." Judgement calls. That's why we get the medium bucks. ** 4D Internet Users Group (4D iNUG) FAQ: http://lists.4d.com/faqnug.html Archive: http://lists.4d.com/archives.html Options: http://lists.4d.com/mailman/options/4d_tech Unsub: mailto:4d_tech-unsubscr...@lists.4d.com **
