Re: Native 4D code to convert numbers into text?
Thanks! On Wed, Jun 17, 2020 at 1:08 AM Keith Culotta wrote: > This was written for v6 a while back. The app is still being used believe > it or not. > > = > // METHOD: DtoS - Dollar to String - kc 3/1/00 > // INPUT: real - dollar to convert to string > // was -> $0:=String($1;"$###,###,###,##0.00") > > String(200;$0) > _O_C_INTEGER($i;$sec) > C_REAL($partVal) > > _O_ARRAY STRING(12;mag1;20) // this should be be declared elsewhere > _O_ARRAY STRING(12;mag2;8) > _O_ARRAY STRING(12;mag3;4) > mag1{0}:="" > mag1{1}:=" One" > mag1{2}:=" Two" > mag1{3}:=" Three" > mag1{4}:=" Four" > mag1{5}:=" Five" > mag1{6}:=" Six" > mag1{7}:=" Seven" > mag1{8}:=" Eight" > mag1{9}:=" Nine" > mag1{10}:=" Ten" > mag1{11}:=" Eleven" > mag1{12}:=" Twelve" > mag1{13}:=" Thirteen" > mag1{14}:=" Fourteen" > mag1{15}:=" Fifteen" > mag1{16}:=" Sixteen" > mag1{17}:=" Seventeen" > mag1{18}:=" Eighteen" > mag1{19}:=" Nineteen" > > mag2{1}:=" Twenty" > mag2{2}:=" Thirty" > mag2{3}:=" Forty" > mag2{4}:=" Fifty" > mag2{5}:=" Sixty" > mag2{6}:=" Seventy" > mag2{7}:=" Eighty" > mag2{8}:=" Ninety" > > mag3{1}:=" " > mag3{2}:=" Thousand" > mag3{3}:=" Million" > mag3{4}:=" Billion" > > $theNum:=$1 // Num(Request("a Number")) > $Target:=String(Trunc($theNum;0)) > $TLength:=Length($Target) > $sec:=1 > $theOutput:="" > While (Length($Target)>0) > $partVal:=Num(Substring($target;Length($target)-3+1;3);$sec) > If ($partVal=0) > $theOutput:=" "+$theOutput > Else > $theOutput:=DtoS_Get3 > (Substring($target;Length($target)-3+1;3);$sec)+mag3{$sec}+$theOutput > End if > $target:=Substring($Target;1;Length($target)-3) > $sec:=$sec+1 > End while > $cents:=String(Round(Round($theNum;2)-Trunc($theNum;0);2);"#.00") > $cents:=Substring($cents;2;2) > $theOutput:=$theOutput+"Dollars and "+$cents+"/100" > > > $theOutput:=Replace string($theOutput;" ";" ") // extra spaces may > appear for "000" > $0:=Replace string($theOutput;" ";" ") > > > = > // METHOD: DtoS_Get3 - return words for a 3 digit section xxx,XXX,xxx > // INPUT1: string - 0..999 > // INPUT2: integer - what 3 digit section are we in ...3,2,1 > > _O_C_STRING(3;$1) > _O_C_STRING(200;$0) > _O_C_INTEGER($numValueR2;$numValueR1;$numValueM2;$2) > > While (Length($1)<3) > $1:="0"+$1 > End while > > $numValueR2:=Num(Substring($1;2;2)) > $numValueR1:=Num(Substring($1;3;1)) > $numValueM2:=Num(Substring($1;2;1)) > $numValueL1:=Num(Substring($1;1;1)) > > If ($numValueR2<20) // handle the rightmost 2 digits > $0:=mag1{$numValueR2} > Else > $0:=mag2{$numValueM2-1}+mag1{$numValueR1} > End if > > If ($numValueL1>0) > $0:=mag1{$numValueL1}+" Hundred"+$0 > End if > > Keith - CDI > > > On Jun 16, 2020, at 4:48 AM, David Adams via 4D_Tech < > 4d_tech@lists.4d.com> wrote: > > > > I'm looking for code that converts something like this: > > > > 24120.10 into something like this: > > > > twenty-four thousand, one hundred and twenty point one > > > > The hope is to emulate Python's num2words. > > > > I figure anyone who had to write a check printing feature years ago might > > have something along these lines, and may be willing to share. I'm not > > trying to print checks, that's just an example. > > > > Definitely after native code, hoping to avoid a PHP call. > > > > For reference, Dani Beaubien sent me a link to an old post from Garri > Ogata > > with something like the code I'm asking about: > > > > > > > http://4d.1045681.n5.nabble.com/Number-To-Text-native-4D-code-td4864780.html > > > > I also found what looks like a version of that code here: > > > > https://kb.4d.com/assetid=76423 > > > > This version has a bunch of extra ) characters, but runs once you fix > them. > > However, this code does *not* include decimals. I'd like to keep the > > decimals part. > > > > Thanks for any help. > > ** > > 4D Internet Users Group (4D iNUG) > > New Forum: https://discuss.4D.com > > Archive: http://lists.4d.com/archives.html > > Options: https://lists.4d.com/mailman/options/4d_tech > > Unsub: mailto:4d_tech-unsubscr...@lists.4d.com > > ** > > ** 4D Internet Users Group (4D iNUG) New Forum: https://discuss.4D.com Archive: http://lists.4d.com/archives.html Options: https://lists.4d.com/mailman/options/4d_tech Unsub: mailto:4d_tech-unsubscr...@lists.4d.com **
Native 4D code to convert numbers into text?
I'm looking for code that converts something like this: 24120.10 into something like this: twenty-four thousand, one hundred and twenty point one The hope is to emulate Python's num2words. I figure anyone who had to write a check printing feature years ago might have something along these lines, and may be willing to share. I'm not trying to print checks, that's just an example. Definitely after native code, hoping to avoid a PHP call. For reference, Dani Beaubien sent me a link to an old post from Garri Ogata with something like the code I'm asking about: http://4d.1045681.n5.nabble.com/Number-To-Text-native-4D-code-td4864780.html I also found what looks like a version of that code here: https://kb.4d.com/assetid=76423 This version has a bunch of extra ) characters, but runs once you fix them. However, this code does *not* include decimals. I'd like to keep the decimals part. Thanks for any help. ** 4D Internet Users Group (4D iNUG) New Forum: https://discuss.4D.com Archive: http://lists.4d.com/archives.html Options: https://lists.4d.com/mailman/options/4d_tech Unsub: mailto:4d_tech-unsubscr...@lists.4d.com **
Re: Defining and documenting Objects
Just went back and looked at the thread. And, again, I see that there's huge amount of confusion and murk about what constitutes a "global" variable. Flow of control, type, and scope are all basic language design features. Scope in 4D ispeculiar. Scope is typically tied to a collection of methods that share access to a variable. 4D absolutely, positively does not have anything like that. It doesn't have any real notion of data hiding. That's the problem with "globals". 4D has a *runtime* concept of scope where the scope boundary is a process. That's hard to explain to most people. It's also not something that can easily be tested in code at rest since it depends on the runtime context and exact path of execution. It's an inherently unsafe approach compared with well known alternatives. "Modular" programming is a concept from the 1970s, it's nothing new. And it came out of both best practice, the pain of project failures, and some really systematic thinking and research. The ideal goal of scope is to limit the number of methods that can directly touch an item and the lifetime of a variable. Heck, there are languages without even the possibility of global variables. One of the thing that drives people crazy about JavaScript historically is the lack of block level scope. (Not a concept we have in 4D, nor that really makes sense in 4D.) 4D's feature set is actively antagonistic to anti-global scoping: * This is why *SET PROCESS VARIABLE* and *VARIABLE TO VARIABLE* are evil. * With pointers to locals that can escape outside of the current method, you've now got a hard to figure out level of scope. * Double for 4D's reference-based dictionaries (*C_OBJECT*.) The notion that process variables aren't globals and that inter-process variables are globals is wrong. They're both global, but with a specific runtime-only boundary called a process. It's a semi-porous boundary that you can violate without any control at any time with *CALL WORKER/CALL FORM*, *VARIABLE TO VARIABLE*, and *SET PROCESS VARIABLE. *Alternative design include benign messaging and function calls. Also nothing new, decades old. And, as Tim was saying, don't worry about the tiny memory and speed issues, if there are any. I couldn't measure them in V3 or V6 and doubt that you can now. Programmers are always guided by their own worst experiences, and the team developing the server-side of 4D *has different experiences than we do*. They worry about edge cases. A lot. A whole lot. I'm entirely sympathetic to this, they really do need to worry about edge cases. Do we need to worry about the rare cases? Only when we have them. I have them at the expected frequency. Namely, almost never. It's just silly to design to edge cases when you don't have to. With that said, the new *Form* function is *excellent*. Really outstanding, it's easily one of the greatest things I've seen in V17. I'm also happy to see 4D finally come around on this. A day nearly 30 years ago now has always stuck in my mind because it was so awful. Let's just say that the Engineering team back then was *really* hostile to the idea and lost the most knowledgeable programmer I ever worked with as a result. Glad they've seen the light. And thanks to Cannon Smith for the feature request that seems to have been the genesis of the feature. Oh Canada! ** 4D Internet Users Group (4D iNUG) Archive: http://lists.4d.com/archives.html Options: https://lists.4d.com/mailman/options/4d_tech Unsub: mailto:4d_tech-unsubscr...@lists.4d.com **
Re: Defining and documenting Objects
Keisuke, Thanks for the quick response, it's always great to hear from you. Yes, I think you've described exactly the situation. I've got a collection which is tied to a listbox. I've got an automatic selected object. I've got an object display bound to the selected object. I change the source value in the collection and the binding doesn't fire all the way through. Here's where the master data lives: collection.selected_object.element The item I've modified via code isn't displayed in the listbox, it's displayed in a bound variable. The bound variable isn't updating, that's my problem. Is that the bug you're talking about being fixed? For the listboxes themselves, I've been calling *REDRAW* and it works well. I've got a multi-page form and if I switch to a different page and back, the bound variable redraw is done. I guess I could update the contents of my bound variable directly but, honestly, that kind of subverts the whole binding system. Then it's really unclear what the real data is...plus I don't want my code to have to know about binding subscribers. On Fri, Oct 12, 2018 at 3:43 PM Keisuke Miyako via 4D_Tech < 4d_tech@lists.4d.com> wrote: > Is this about a cell in a collection listbox? > > the reason why a redraw needed to be forced (Form.col:=Form.col) > was related to how ref. counting works, > in particular, > when you have an expression defined as the "current item" property. > > the current item object can be referenced from another object on the same > form, > in addition to the listbox cell itself. > so when you update that object it by code, > there was an uncertainty as to whether the original object is meant to be > updated to retained. > > anyway, it should be fixed in ACI0098483 > > https://bugs.4d.fr/fixedbugslist?branch=17 > > 2018/10/12 13:35、David Adams via 4D_Tech <4d_tech@lists.4d.com 4d_tech@lists.4d.com>>のメール: > I was looking for the an answer to "why don't my object variables on > fields automatically redraw when changed?" > > > > ** > 4D Internet Users Group (4D iNUG) > Archive: http://lists.4d.com/archives.html > Options: https://lists.4d.com/mailman/options/4d_tech > Unsub: mailto:4d_tech-unsubscr...@lists.4d.com > ** ** 4D Internet Users Group (4D iNUG) Archive: http://lists.4d.com/archives.html Options: https://lists.4d.com/mailman/options/4d_tech Unsub: mailto:4d_tech-unsubscr...@lists.4d.com **
Re: SEND PACKET in a multi-process environment
I've written a lot of multi-process log writing code in my career. A lot. Enough to run into edge cases and performance issues of various sorts that are irrelevant to most. The standard way to implement a multi-process log goes like this: * Have one log writer process/thread. * That process is the only thread to touch the file, so zero contention. * Use an interprocess message bus (of whatever design) to pass log messages from your various threads to the log. So the file IO is limited to one location and you don't have to open/close the file all the time. The performance difference is ridiculously large, but arguably the main benefit is avoiding file contention hassles. There are several ways to to the interprocess communications in 4D: * NTK IPC channels. Works great, has always worked great. Not multi-core because it's a plug-in. If anyone cares. * IP arrays with a semaphore or mutex. Can be 100% reliable but, potentially, kind of expensive. * *CALL WORKER*, if you don't care about losing a few entries. That final one seems like a great solution, and it might work perfectly in V17 In V16, it wasn't thread-safe and you could blow up your server just by rolling a log file. I submitted this bug with a small sample database early in the V16 product line and it's reportedly fixed in V17.0, but I haven't confirmed that. This might be the same bug David is hitting? Or a closely related one? If so, V17 is worth a try, even if just as an experiment. ** 4D Internet Users Group (4D iNUG) Archive: http://lists.4d.com/archives.html Options: https://lists.4d.com/mailman/options/4d_tech Unsub: mailto:4d_tech-unsubscr...@lists.4d.com **
Re: Defining and documenting Objects
I don't follow the list any longer, but I do check in to look for answers. Today I was looking for the an answer to "why don't my object variables on fields automatically redraw when changed?" Didn't find it, does anyone have a workaround? While I was checking on my question, I ran into this massive thread on objects. As it turns out, I've been working in this area for a couple of weeks and have a few thoughts to share. I haven't read the whole thread, so apologies for repetitions. I don't have time for this post...but here it is anyway. If you care about this issue enough to work on it, you'll find a bunch of solid ideas in the TL;DR version. I don't have the time to edit it into something shorter and better, but here is a very short version: * 4D's new system leads to very easy to break, brittle, hard to correct coding problems. By default. * You can use tables instead of objects. But do you want to? * 4D dropped the ball on supporting complex types, so we each have to do all of that work on our own. * *JSON Validate* can help. Some. * It takes quite a bit of work to write a custom type definition and validation system but, if you need quality code, you don't have a better alternative. * Use constructors! TL;DR version: First off, some background about types. There are only a few very core issues in language design...you could have spare fingers on one handtype is one of those core questions. What we've got in 4D: * Primitive types. * Arrays of primitives. * Dictionaries/associative arrays. We do *not* have compound types nor do we have objects. *C_OBJECT* is a dictionary, not an object. We need dictionaries, so that's great, but the name is misleading to many. Anyway, what difference does it make? Well, a compound type has many, many advantages. It's a collection of elements into a distinguishable format with *rules*. 4D's tables are the closest thing to this we've got. But there are tons of times when you really don't want a table, you want a pure, in-memory structure. Or at least I do. And every other programmer in history does. Just saying. What difference does it make that we don't have compound types (structs in C, records in Pascal, whatever you prefer to call them?) It makes a huge, huge difference. Here's some of what compound/complex types are good for: * They're *types *(formally defined data structures) so you can pass them around and get an interpreter, linter, or compiler to complain if you pass the wrong type. You know how 4D uses longints for a zillion things, like window refs, process ids numbers, etc. With a custom type, you could have windowRef, or processId as distinguishable types. Pass in some rando number to a method requiring a window reference and get an error from the interpreter or, depending on how it's all coded, the compiler. I used a simple type (longint) here as an example, but the same applies to more complex structures. * Type can embed rules right into the structured format. In a sense, objects are complex types with behavior bound in. In a simpler case, you can add ranges to elements - like "only allow numbers from 1-3" or "only allow the strings 'Mon','Tue', or 'Wed'", force reals to round to a certain number of decimal places, etc., etc. Again, 4D tables are the closest thing a pure-4D programmer will be familiar with. And the field-level constraints in 4D are pretty painful to use (required, etc.) because of how error handling is implemented in 4D...so many people don't use field-level constraints. I don't. Okay, so we don't have types...how much difference does it make in V17. It makes a huge difference, as people have clearly noticed already and are exploring on this thread. I like the new dot notation, but there are some massive issues with names. Such as: * If you add a name to an object, the element is added. What if it's a typo? You get a new element. No error. This is described as a "feature". Well, sometimes it is a feature, sometimes it's a bug. There's no "strict mode" choice. That's a missing feature in 4D. Or a design flaw, call it what you will. * Case-sensitive. I just can't say how stupid I find this. Has anyone in history *ever* found this useful? If so, get them away from the keyboard and put them into management or something where they can do less harm. Pet peeve. Tabs not spaces people! Spaces, who does that? But I digress from my digression. 4D says that this is to be "more compatible with JavaScript." Why? I can't think of any reason. I've spent enough time with JavaScript to really enjoy some of its features (closures!, some of the messaging/delegation, the range of normal operators...man 4D needs ++, etc.), but I've never heard *anyone* outside of 4D point to JavaScript as an example of what a language should look like. Quite the opposite. * So, an "object" can have anything in it. Or not. It's a completely unfixed format. Sometimes that's what you want. Most of the time? I can hardly think of *any* time I want a completely
Re: Do I want to use SELECTION TO ARRAY instead of GOTO SELECTED RECORD server-side in V17
Just for the record, my 160x speed improvement came with pretty naive code. I loop to a record, build the text I need for that row, append the block to the BLOB, clear the text and hit the next row. All-in-all, only a couple of lines of extra code. After what everyone has contributed and suggested, along with my own local testing, my summary goes like this: *Build a text var* Yeah, don't do that. In my case, the final block size is ~3MB, not huge...but the resizing is pretty aggressive. *Build a BLOB var* Works great. The downside is that if you ultimately need text, you (even if briefly) need to double the amount of RAM you're consuming. I'm defaulting to a max BLOB of 5MB and change, so this should never be a problem. *Write to a text file* This should never slow down, no matter how big the file gets. This has always been true, the limiting factor being the speed of the drive. Drives are fast now. Memory is also fine because you take no RAM to store the document and can load it into RAM in one go very easily. I'm going with a BLOB instead of documents. Why? Old prejudice, and I like avoiding the file system. Permissions, file contention issues in 4D (been there not long ago...and after *more than a year*, there's reportedly a fix), data leakage. Plus, old prejudice. In my specific case, a BLOB should be fine as I'm not pushing the envelope on memory usage. Under other circumstances, I can easily see documents as being the best bet. Particularly if you have other reasons to want the data in documents for dispatching, archiving, etc. Thanks for the help! ** 4D Internet Users Group (4D iNUG) Archive: http://lists.4d.com/archives.html Options: https://lists.4d.com/mailman/options/4d_tech Unsub: mailto:4d_tech-unsubscr...@lists.4d.com **
Re: Do I want to use SELECTION TO ARRAY instead of GOTO SELECTED RECORD server-side in V17
Thanks to John for adding writing to disk to Dani's test set. You make some good points about documents in the modern world. You can probably append to a document all day long and never see any change in speed. Years ago, the legend was that you didn't have to buffer before calling *SEND PACKET* because 4D was doing it already. Meaning also that you had to close/open the document it you were writing a log to track down crashes. No idea if this is true now. And, yeah, then you can *DOCUMENT TO TEXT* in one go, it that's what you're after. But, and I don't know if this is just me or not, but I hate writing scratch files. Just...don't like it. I can't back that up with any good reasons, I just have a bit of an allergy to the file system. Permissions issues, etc. Maybe I should rethink that. Another nice feature of files is that you can use them in so many ways...they can then be read by another program for dispatch, etc. So, hmm. ** 4D Internet Users Group (4D iNUG) Archive: http://lists.4d.com/archives.html Options: https://lists.4d.com/mailman/options/4d_tech Unsub: mailto:4d_tech-unsubscr...@lists.4d.com **
Re: Do I want to use SELECTION TO ARRAY instead of GOTO SELECTED RECORD server-side in V17
Dani, Thank you, for a very large value of "thank you." I woke up this morning all excited to write some tests, and you had already done the work. It's really great that you took the time to do the tests, to share your conclusions and your test method. Thank you! In my case, I'm not so kind or generous ;-) I've got a specific method that I'm using for testing. I pass in a method and it does the work. This is a fixed data set, so the results are always identical. The only thing I'm changing is how I'm building up the block of text. I posted about the four method I tired and they were all the same, about 6 minutes. I went back, added a new test for *SET TEXT TO BLOB* in the loop and then one call to *BLOB to text* at the end. The results? What otherwise take about 6 minutes now takes just over 2 seconds. I had to run it a few times to convince myself I wasn't imagining things. I wasn't. I thought that I was past the point where I'd ever see a 100x (well, x160 in this case) improvement from a small code change. So thank you all for this information, it's a game-changer for me! Onto the other point, memory. A Dani commented, if you need text then you have to briefly double the RAM consumed. In my case, I need text. (The text is passed to a plugin that doesn't accept BLOBs in this situation.) What do people thing about RAM these days? I'm still kind of conservative. Tim Nevels (off-line) said to me the other day, "David, why are you worrying about memory? Everyone has tons of memory." I'm already not worrying about performance in advance, and I *don't* feel like spending more time worrying about security, so can't I worry about memory a bit? Where do you guys stand on this on modern gear? When was the last time you bumped into harmfull memory constraints? Thanks ** 4D Internet Users Group (4D iNUG) Archive: http://lists.4d.com/archives.html Options: https://lists.4d.com/mailman/options/4d_tech Unsub: mailto:4d_tech-unsubscr...@lists.4d.com **
Re: Do I want to use SELECTION TO ARRAY instead of GOTO SELECTED RECORD server-side in V17?
Bernd, Thanks for the report, it's interesting and believable. In this case, I have to get the data into memory to send it... I guess I could write to a file, load that and then send it. Interesting concept, thanks! On Mon, Sep 17, 2018 at 4:53 PM Bernd Fröhlich via 4D_Tech < 4d_tech@lists.4d.com> wrote: > David Adams: > > > I'm checking for the simplest, least difficult default as that should > work in the vast majority of cases. > > > Here are my two Eurocents: > it does not matter how you collect the data, but concatenating the text in > memory vs. writing to a file makes a HUGE difference when you have lots of > records. > > I have a method in my toolbox that writes out a selection or a whole > table into a file. > Looping through the records and appending each record to a file. > Works great. > > Then I did the same but instead of writing to a file, I appended to a > textvariable. > Works well for small amounts of data but is unusable for more than about > 2 records as it slows down ever more. > (I am lazy and did not do any pre-sizing, chunking, whatever.) > > So if you want to export the data anyway and look for the simplest way, > then just loop through the data and write into a file. > > Greetings from Germany, > Bernd Fröhlich > ** > 4D Internet Users Group (4D iNUG) > Archive: http://lists.4d.com/archives.html > Options: https://lists.4d.com/mailman/options/4d_tech > Unsub: mailto:4d_tech-unsubscr...@lists.4d.com > ** ** 4D Internet Users Group (4D iNUG) Archive: http://lists.4d.com/archives.html Options: https://lists.4d.com/mailman/options/4d_tech Unsub: mailto:4d_tech-unsubscr...@lists.4d.com **
Re: Do I want to use SELECTION TO ARRAY instead of GOTO SELECTED RECORD server-side in V17?
On Sun, Sep 16, 2018 at 9:09 PM Olivier Flury via 4D_Tech < 4d_tech@lists.4d.com> wrote: > Not really an answer to your question but a more general observation: > > Very often, when I have to do an export function towards other systems, I > end up in doing a bulk export. > > Either because I am lazy or the other side is lazy or both sides are lazy. > It's easy, just send everything. But sometimes it is worth, doing more > homework, on both sides, and export only what has changed since the last > export. Sometimes this is simply not possible (too complex, the other side > wants all data etc.), but very often it would be. And it would - most of > the time - result in a drastic reduction of amount of data to exchange. > Well observed. In this case, I have the good fortune to be working with source tables that include modification counters. (Think "sequence numbers" on each change.) This makes it *incredibly* easy to find all records that have been modified or added since the last push. For deletion, records in 4D are marked with a deleted date. I've always found this practice annoying and weird...and boy am I glad it's being used in this. With the delete date field in place (soft deleted records), it's simple to handle delete reconciliation. I don't push the delete date field, just a Boolean based on it. So, you push everything up including the delete flag, then a simple DELETE FROM table WHERE marked_for_deletion = true takes care of it. Amazingly easy. Welsh Harris and Tim Nevels get all the credit for setting things up this way in advance. Even still, there are sometimes still millions of records to move, particularly when we're first pushing a table over. In most cases, pushes are quite manageable in size on a day-to-day basis. These tables are getting pushed largely for aggregation and analysis, so it's not one-off operations. Push and keep pushing as more data flows through 4D. ** 4D Internet Users Group (4D iNUG) Archive: http://lists.4d.com/archives.html Options: https://lists.4d.com/mailman/options/4d_tech Unsub: mailto:4d_tech-unsubscr...@lists.4d.com **
Re: Do I want to use SELECTION TO ARRAY instead of GOTO SELECTED RECORD server-side in V17?
Hello people, thanks for all of the suggestions and feedback. Several of you pointed out that my thinking was in the fuzzy-to-nonsense range. I hate you all. So much. Not really, the comments were completely on the mark. I'm trying to find a decent default option that will work over a wide range of conditions. Some tables are tiny, some are very large. Some pushes have a lot of data, some have vey little. Most machines are up on AWS, a few are not...but have GB fiber connections. So, there are variablestoo may for a best-fit solution for every case. I'm checking for the simplest, least difficult default as that should work in the vast majority of cases. If something does need help, then I can get in an optimize it, as required. I have one case like that already and it was a total bear to finish. Sometimes you get the bear, sometimes the bear gets you... I was doing some chores today which gave me a chance to run tests and leave my computer alone while things ran. What I wanted to test out was nothing but the behavior of creating my big text object. What happens with it after that isn't relevant. The text block built is identical on each run. So, I did milliseconds before & after and memory stats before & after. Then looked at the results. I ran compiled, single-user, macOS 10.12.6 with 16GB or RAM and a 3GB cache. This is a MacBook Pro from the past couple of years...one of the ones with the shitty keyboard. Man, that thing really makes me less productive. Back to the tests. I ran compiled only. I launched, ran a test, got the results, ran the test again, got the requests and quit. Just because that's how I normally test stuff, at a bare minimum. I didn't reboot. My test sample pulls five smallish fields from about 30K records in a table with a lot more fields. The methods for navigating the data are what differ. The four versions that I checked are: * *GOTO SELECTED RECORD* * *SELECTION TO ARRAY* + *For* loop * Entity selection + *For each* loop * Entity selection + *For* loop I'll start with the speed report as it's short: It doesn't matter. The performance range for the four solutions on their two runs was so small that I chalked it up to normal variability. So, *speed is not an issue*. This leaves memory, ease of development, and stability/brittleness. For memory, I'm not terribly confident in the numbers. The first run values are confounded by everything else that might be happening with the first run against that data, so I've ended up ignoring those results. The second run results confirm what Keith said. Namely, that *SELECTION TO ARRAY* takes more memory than *GOTO SELECTED RECORD*. Kind of has to. On the two entity selection results, they seemed to use more memory than *GOTO SELECTED RECORD* and less than *SELECTION TO ARRAY*. Or maybe not, I'm not testing more. I don't want to start any superstitions here. I will say again though that if you care and want to test it, forget first principles. Even if you had and understood the 4D source code, you would have very poor chances of guessing how things would behave by the time you're running your code. If anyone does some testing, I'm sure there's an appetite for the results. Okay, that's it. Not the kind of results that you can broadly apply. There are just so many factors involved. For example, the memory profile on *SELECTION TO ARRAY* grows as you add more and more arrays. It has to, and you can even calculate the costs pretty well. The difference for *GOTO SELECTED RECORD* is likely very little. I only tested on a machine that runs the engine, Client would likely show dramatically different behavior. Regarding the entity selections, that was just out of curiosity. I won't be using them here. They don't offer a speed or memory advantage over *GOTO SELECTED RECORD* and they make the code vulnerable to breakage. Someone changes the capitalization of a field, and my code breaks. My server side code with no UI...Not worth the risk. The code clarity on *GOTO SELECTED RECORD* and then using field references is *excellent*. It's at least as good as the readability of the entity-dot versions. But with regular old field references, there's no risk of weird breakage and the code is instantly understandable to any existing 4D programmer. ** 4D Internet Users Group (4D iNUG) Archive: http://lists.4d.com/archives.html Options: https://lists.4d.com/mailman/options/4d_tech Unsub: mailto:4d_tech-unsubscr...@lists.4d.com **
Re: Do I want to use SELECTION TO ARRAY instead of GOTO SELECTED RECORD server-side in V17?
Thanks Keith, that all sounds pretty right to me too. And, yes, I'm chunking the pushes into batches to avoid a truly massive text object, when necessary. I don't know about with SQL Server and don't remember about MySQL, but with Postgres it's important to avoid single row inserts where possible. Postgres is absolutely fanatical about integrity, so each operation is automatically wrapped in a transaction. Pushing rows one-by-one can, therefore, be surprisingly slow. So I bundle multiple rows into a mutli-value insert, and then several of those into a transaction. It goes pretty quickly and for smaller pushes, it probably doesn't matter how I build up the text, it just isn't that big. (Note for the PG people in the crowd, I'm using an UPSERT, so COPY TO isn't an option.) Regarding the ever-resizing text block, I don't know how that behaves today. I used to know...and knew the sort of crazy stuff you could do to improve performance. Like, pre-size a giant block, keep track of the last byte written and then write into bytes by hand. So, you didn't resize the block as you went along. You might need a final trim or an expansion...but you pretty much kept the block as a set size and then write into it by direct character reference, etc. It's a lot more work and I have no idea how modern versions of 4D on modern versions of Mac/Win handle all of this. I have a notion in the back of my mind that all of that kind of byte tracking stuff isn't too useful any more. I notice that both 4D and Postgres don't have VARCHAR types internally, it's all variable length text. Haven't testing anything. Oh, we don't have any double-byte data to worry about. Thanks again! ** 4D Internet Users Group (4D iNUG) Archive: http://lists.4d.com/archives.html Options: https://lists.4d.com/mailman/options/4d_tech Unsub: mailto:4d_tech-unsubscr...@lists.4d.com **
Do I want to use SELECTION TO ARRAY instead of GOTO SELECTED RECORD server-side in V17?
Short version: I need to load some fields from records into a big text thingy. The code runs on the server-side only. I'm keen to preserve RAM. What are the trade-offs in V17 between *GOTO SELECTED* record and *SELECTION TO ARRAY*? I've been using *SELECTION TO ARRAY*, but it's hard to read, write, and maintain. And, I realized, might be de-optimized for memory because you have to load all of the data you're processing into arrays. (Yes, you can chunk it, but that doesn't change the fundamental point that you pre-load a lot of data.) Any test results or thoughts? I considered a fair range of option and did comparison tests on none. The long version below includes more details on the two solutions I'm down to, plus the ideas that I discarded. TL;DR version I'm working in V17 and I'm hoping that someone has done some real-world tests already that could help me out with a question. Here's the setup: I need to load up some fields from lots of records and push them into an external system. It's going to Postgres, but that's not an important detail, the result is a ginormous text object. The result could just as well be a text or JSON file dump. The main constraint is available memory. Performance matters when there are millions of records but, typically, the only important consideration is memory. As far as the final solution goes, it's ideally code that's easy to write, read, and maintain. As a plus, we can position the code to run server side, so client-server optimization isn't an issue. And, for the record, in lots of cases there isn't enough data to make memory an issue at all, so readable reliable code is definitely a preference. Note: Yes, I can chunk data in ranges, etc. to keep things within my memory footprint. I'm doing thatbut the question still remains Here are the solutions I've come up with: *QUERY* and a *For* loop with *GOTO SELECTED RECORD*. Easy to read, write and maintain. But when you use *GOTO SELECTED RECORD*, do you get the whole record in V17? Without fat fields? Since this is server-side or stand-alone, should I care? On the upside, you're only loading one record at a time, so only burning through memory for that record while you use it. *SELECTION TO ARRAY* and a *For* loop This is what I have been doingbased on old habits as much as anything. Yes, you only get the columns you want, but it gets _all_ of the rows at once. So, you burn up a lot of memory with the arrays and then duplicate++ that memory when building up the output. On the code side, that kind of *SELECTION TO ARRAY*-loop-read by index code is ugly, tedious to write, and tedious to maintain. It's clear(ish) and reliable, but only worth it if it pays for itself somehow. In other words, it has to be a good deal better than *GOTO SELECTED RECORD* to be worth it. Says the guy who has been doing all *SELECTION TO ARRAY* forever. Entity Selection and a *For* or *For each* loop I have no clue why an entity selection is *C_OBJECT* instead of *C_COLLECTION*, to give you a sense of how much I know about this stuff. I was happy to discover that you can easily create an entity selection from a current selection, so old style queries work fine: *C_OBJECT*($stuff_es) *QUERY*([Stuff];[Stuff]Counter>=1) $stuff_es:=*Create entity selection*([Stuff]) The resulting *For*/*For each* loop code is very readable, it's == *GOTO SELECTED RECORD*, but with a different syntax. Otherwise, same same. I *suspect* that the memory use here is excellent. I'm guessing that as you navigate through the entity selection, you're only really pulling the data you use. But maybe not. If you do a For each, you get an object (entity) with all of the fields. So, possibly this is approach is even worse than *GOTO SELECTED RECORD* which, I'm guessing, doesn't load as many fields. I haven't tested these points out in any way. If anyone has dug into this, it would be great to know about the difference (if any) in what 4D loads when you: -- Use *GOTO SELECTED RECORD* -- Use a *For each* loop on an entity selection, which builds an $entity_object which you can then read/write to/from like $entity_object.ID -- Use a *For* loop on an entity selection and then reference $specifc_es[0].ID It's pretty easy to imagine different ways that 4D might have implemented things that are more or less efficient in each of these days. I have no idea what they actually did.I'm kind of curious about this behavior in V17, but have already talked myself out of using entity selections. Why? Because the table and field references are brittle and *case-sensitive*. Man, I truly hate case-sensitive names. When do I want them? Never. Not once, and I never will. This isn't all on 4D, many languages are case-sensitive. It makes sense if you're a computer. I'm not a computer, I'm a person...to me its just horrible. Anyway, not exclusively a 4D problem...because in 4D you can avoid it altogether. For those that haven't been following along at home, here's a hello world le
Re: v17 PDF-Manuals
Thanks Wayne! ** 4D Internet Users Group (4D iNUG) Archive: http://lists.4d.com/archives.html Options: https://lists.4d.com/mailman/options/4d_tech Unsub: mailto:4d_tech-unsubscr...@lists.4d.com **
Re: v17 PDF-Manuals
I was just at the 4D site looking for PDF manualsand didn't find them. So I searched here and found this thread. I'm going to be away from the Internet and want to take documentation with me. I expect that I'll have some down time, and figured it might be a nice time to look at recoding some things as collections. (I'm coming from R3 where object access is still through the OB SET/OB GET...which make me not use C_OBJECT all that much.) To be clear, I won't have the Internet. At all. So, is there any way to get an off-line version of the V17? I don't see a way to download the HTML versions and don't see any PDF versions. If not, I'll just skip what I was planning and do something else with the time. Thanks! ** 4D Internet Users Group (4D iNUG) Archive: http://lists.4d.com/archives.html Options: https://lists.4d.com/mailman/options/4d_tech Unsub: mailto:4d_tech-unsubscr...@lists.4d.com **
Re: Who’s the Gestapo?
Robert: * This list is run by 4D. * I've never heard that about the censoring, and it hasn't happened to me. Which it probably would have, if it were happening at all. Are you 100% certain that there's not technical glitch on your people' side? * I'd suggest *not* comparing people to the Gestapo in general but, in particular, when they're old enough that their parents can remember the Gestapo. On Sat, Jul 14, 2018 at 2:34 PM Robert ListMail via 4D_Tech < 4d_tech@lists.4d.com> wrote: > I tried to turn on a few young developers to 4D and suggested they join > the list. They were never able to post. I take it the list is highly > censored based on the feedback I’ve received. I didn’t know the bar was > that high. So, who’s the gatekeeper...? > > Thanks, > > Robert > > Sent from my iPhone > ** > 4D Internet Users Group (4D iNUG) > FAQ: http://lists.4d.com/faqnug.html > Archive: http://lists.4d.com/archives.html > Options: https://lists.4d.com/mailman/options/4d_tech > Unsub: mailto:4d_tech-unsubscr...@lists.4d.com > ** ** 4D Internet Users Group (4D iNUG) FAQ: http://lists.4d.com/faqnug.html Archive: http://lists.4d.com/archives.html Options: https://lists.4d.com/mailman/options/4d_tech Unsub: mailto:4d_tech-unsubscr...@lists.4d.com **
Re: Chromium Embedded Framework in built applications question
Thanks for the answers, I wondered if it might be to do with Blink...seems pretty heavy if you're not using it. Actually, we might have a couple of Web areas in there. I wonder if 4D detects that and includes/excludes the framework accordingly. Thanks for the R6 suggestion, we can try that out and see what happens. ** 4D Internet Users Group (4D iNUG) FAQ: http://lists.4d.com/faqnug.html Archive: http://lists.4d.com/archives.html Options: https://lists.4d.com/mailman/options/4d_tech Unsub: mailto:4d_tech-unsubscr...@lists.4d.com **
Chromium Embedded Framework in built applications question
A friend recently pointed out that built apps had unexpectedly increased in sizea lot. We used to come in at something like 150MB and are now at over 600MB. macOS, 16.3 R3 Digging into the ressoures in the built app, this package appears: Chromium Embedded Framework.framework ...at about 260MB. The compiled database is less than 55MB. Does anyone know what this Chromium Embedded Framework.framework file is and how to get rid of it? You can't just strip it out after the build as 4D complains on launch. Thanks! ** 4D Internet Users Group (4D iNUG) FAQ: http://lists.4d.com/faqnug.html Archive: http://lists.4d.com/archives.html Options: https://lists.4d.com/mailman/options/4d_tech Unsub: mailto:4d_tech-unsubscr...@lists.4d.com **
Re: 7-zip from within 4D
On Wed, Jan 9, 2013 at 4:30 AM, Tony Ringsmuth wrote: > Wayne, > > I use 7zip extensively, driving it from 4D. > > What I do is this: After a 4D backup completes, from the On Backup Shutdown > database event, I initiate an encrypted 7-zip of the backup file, and some > info files. After they're zipped, then 4D server ftp's them to several > remote locations. > > It works rock solid. > > If you need, I can forward you the crucial code samples. It's basically > done through LEP and batch files. > Tony, If your offer of code hasn't expired after more than five years, I'm looking for exactly this functionality on Windows right now. Hope you're well, -- David Adams ** 4D Internet Users Group (4D iNUG) FAQ: http://lists.4d.com/faqnug.html Archive: http://lists.4d.com/archives.html Options: https://lists.4d.com/mailman/options/4d_tech Unsub: mailto:4d_tech-unsubscr...@lists.4d.com **
Re: Bug or feature?
Hey, long thread - I was even part of it at some point. I just wanted to send out a big *thank you* to Jeff Kain for starting it. (And to Chip for the awesome feature bug picture.) I'm writing now because I've been banging my head against the desk for a couple of days trying to get past a problemthe very problem Jeff reported. I don't want to go through or rehash the whole thread, it's there in the archives for anyone that cares. But my current issue (16R3 macOS) is a bit the same, and a bit different. The big difference is that *there is nothing wrong with the JSON*. It's valid JSON. But 4D blows up on some of it, some of the time and the *ON ERR CALL* does not check. Jeff's trick with declaring Is object has at least stopped the blocking error. Now I'll have to see if the system is actually working. This has all been a bit slow for me as the error only appears in compiled mode and only in one method (more on that in a moment.) And there are nearly 6M records in my test file and over 8M in production. When you blow up, the process is dead, you're compiled and you can't get anything back to identify *which* record blew up. So I grafted in some logging code to log activity before it happens so that I could figure out what the problem is. I didn't ;-) But at least I can look and see the record. Tim Penner made a solid point earlier about Booleans in JSON being all lower-case. I'm wondering if that has something to do with it? Here's a clause from the JSON: { "name": "is Name Finalized", "old": "False", "new": "True" } Except here's the thing, those are formatted as quoted JSON strings, not booleans. We don't want booleans, we want strings. These are valid strings. If we wanted bools, it would look like this: { "name": "is Name Finalized", "old": false, "new": true } Not what we want, but Tim's point is worth keeping in mind. I'm wondering if without the Is object hint, 4D is getting overly-aggressive about interpreting "True" and "False" (or "true" and "false") as Booleans? No idea. Regarding the "but my JSON is okay!" claim, here's how I checked. The 4D bits are all in compiled mode: -- My main big method with an *ON ERR CALL* that doesn't have any callbacks, current method name, etc. It's either empty or Error:=Error. This fails, and on the same record(s). --> This method loads the data into a text array and the value is then parsed from there, it is not directly parsed as a field -- A big loop that scans over the same records but only copies the data into a variable and parse it. *I don't get any errors from the same records* . -- Checking an individual record in an input and clicking a button that has a little parser script. *I don't get any errors from the same records*. -- Copy the JSON out and putting it into a JSON format window I've got (two text vars, one in and one out.) The JSON stringify stuff works fine to pretty print or flatten the JSON. -- Copy the JSON out and into any JSON tool you like. No problem. So, it's very weird. 4D blows up on the JSON in one case and no other. If anyone can suggest why this might be happening, I'd love to hear about it. More to the point, should we explicitly set Is object pretty much everywhere in our code. We have a ton of calls to *JSON Parse* that would need updates, if this is the case. Is there any downside to specifying Is Object to be aware of? Thanks! P.S. If possible, please CC me directly in responses on dpad...@gmail.com as I do not monitor this list. ** 4D Internet Users Group (4D iNUG) FAQ: http://lists.4d.com/faqnug.html Archive: http://lists.4d.com/archives.html Options: https://lists.4d.com/mailman/options/4d_tech Unsub: mailto:4d_tech-unsubscr...@lists.4d.com **
Re: [Warning] Settings properties values on object field by object notation
Thanks to TIm and Brian for responding here, always appreciated. There is no lack of clarity from 4D that the official channel is to post on the forums. Before that, it was bugs.4d.fr. That's always been clear. Heck, I can't even count the number of times I said that on this list over the years. Of course when I said it, I didn't really think much good would come from it. I can open up 4D today and point out tons of bugs (cosmetic, usability, functional, and crashing) that have been known about for years and versions. So, yeah, reporting bugs doesn't have a lot to do with bugs getting fixed. Pardon me for being so simple minded, but what remains completely unclear is why customer information is ignored. See, that's the issue: You've got customers sending you information, you know what they say, and you say "nope, doesn't count." That's about 4D's relationship with your customers, it's not about your customers doing things wrong. "Posting on the NUG doesn't count" just feels like a weak cop-out. That 4D doesn't see this, I just don't understand. If you guys used any sort of normal, commercial bug-tracking or forum platform, I think the position "do everything there" would be a lot more tenable. But a hidden bug system (which I still have no access to because of my hemisphere) and a home-made forum system in 2017? You just have to accept that that's not going to work for some people. For good reason. I've tried the Forums. While 4D Engineers participate there a little, I heard nothing form the Product Team. Ever. And from Engineering, I mostly heard or observed that posts were being put in the "wrong" forum, and that any sort of design critique wasn't welcome. You can say that a feature is not working correctly, but that a feature or feature set has a design flaw? Not a good reception. At all. The question is, do you want to make 4D better? Do you want to make 4D more reliable? Yes or no. Blowing off customer input = "no". Making it harder for customers to provide input = "no". Not providing feedback on bug status = "no." Not documenting the tools properly = "no". Calling bad design "standard behavior" = "no." Telling people that its' down to us to track down a bug that freezes the server and that 4D isn't actively investigating it on their own? Yeah, that's most definitely "no." Giving people a hard time for pointing out flaws in the *design* of features, also "no". Your business, your choice. ** 4D Internet Users Group (4D iNUG) FAQ: http://lists.4d.com/faqnug.html Archive: http://lists.4d.com/archives.html Options: http://lists.4d.com/mailman/options/4d_tech Unsub: mailto:4d_tech-unsubscr...@lists.4d.com **
Re: [Warning] Settings properties values on object field by object notation
> So, if you want to officially request such a change, it needs to go on the forums. Does 4D realize that *officially* ignoring their customers isn't exactly an example of great customer service. I tried the forums for about six months, I don't think I got _any_ information from 4D Engineering, but I sure did get told I was wrong a lot. About matters of fact and opinion. And, to date, I have seen zero evidence that participating on the Forums does anything at all to forward the cause of bugs or features getting fixed. This whole business that somehow it's our fault that we're not "doing it right" just doesn't sell. I mean, 4D can say that - but how many people does it convince? I doubt it's more than a handful, if even that.The only obvious interpretation is that 4D doesn't want to be bothered with feature requests and bug reports (even on the Forums, frankly). Hey, your company, your rules...but it doesn't make 4D come across as customer-focused. But, again, it's entirely up to 4D how to deal with their business. But, please, quit saying that the problems with customer service are the customers. ** 4D Internet Users Group (4D iNUG) FAQ: http://lists.4d.com/faqnug.html Archive: http://lists.4d.com/archives.html Options: http://lists.4d.com/mailman/options/4d_tech Unsub: mailto:4d_tech-unsubscr...@lists.4d.com **
Re: Working with a 64 bit Integer in an object
Yeah, if 4D isn't working, try NTK. In either case, if you can get at the value as raw text then you'll be able to find text --> 64-bit conversion code. The question in 4D at that point is how to store it. I know it's a field type, but can you touch it with anything other than SQL? ** 4D Internet Users Group (4D iNUG) FAQ: http://lists.4d.com/faqnug.html Archive: http://lists.4d.com/archives.html Options: http://lists.4d.com/mailman/options/4d_tech Unsub: mailto:4d_tech-unsubscr...@lists.4d.com **
Re: [Warning] Settings properties values on object field by object notation
> it is an interesting point of view, > to frame it as an issue of the assignment operator being different, > as opposed to how objects and their properties are different "because they are references". I think that the real issue for a 4D programmer is that a record is a persistent data store and that field modifications are supposed to be stored reliably. How 4D handles all of that internally is, literally, none of our business. (4D refuses to discuss any of this.) The data should be stored reliably 100% of the time. If that can be done in a more optimized way, great! ** 4D Internet Users Group (4D iNUG) FAQ: http://lists.4d.com/faqnug.html Archive: http://lists.4d.com/archives.html Options: http://lists.4d.com/mailman/options/4d_tech Unsub: mailto:4d_tech-unsubscr...@lists.4d.com **
Re: [Warning] Settings properties values on object field by object notation
On Wed, Nov 1, 2017 at 1:32 PM, Keisuke Miyako via 4D_Tech < 4d_tech@lists.4d.com> wrote: > I hate to repeat myself, > > but it seems the problem with object notation > is that it's not as straightforward to know if a field has been changed or not. > but that's not how references work. I almost said earlier "it would be like saying that when you use pointers, you have to know and remember to do an explicit save." > there is literally no way of knowing that an object is a field, just by parsing the code. Could be. But why should I know or care? 4D has made it *very* clear that we are not supposed to know about its internals (or even ask), nor are we to count on them remaining stable between versions. Fine, but that position cuts both ways: You can't then use internals as an excuse. Problems with internals aren't an excuse. They're 4D's problem, so 4D should fix the problem. (We obviously can't.) It's not good enough for Engineering to throw up their hands and say it's going in the too hard basket because of the internals. Since we're not supposed to know about the, telling us about them as an excuse for a bug isn't okay. Particularly when the bug *guarantees* that 4D users and customers will screw up their data. It *will* happen. Imagine this thought exercise: * Ask 100 people "What should happen when you change a field and save the record." 100 people will say "It saves the change." * Ask 100 people "What happens when you change a field and save the record." 100 people will say "It saves the change." No one will say "It depends!" Or "Well, the first thing you have to understand is..." It's 4D's bug, 4D needs to fix it, not expect us to remember this obscure and counter-intuitive "behavior." It's a bug, however it is documented. It's just dead wrong. Hey! I found a 100% reliable workaround: Don't use object fields. ** 4D Internet Users Group (4D iNUG) FAQ: http://lists.4d.com/faqnug.html Archive: http://lists.4d.com/archives.html Options: http://lists.4d.com/mailman/options/4d_tech Unsub: mailto:4d_tech-unsubscr...@lists.4d.com **
Re: [Warning] Settings properties values on object field by object notation
On Wed, Nov 1, 2017 at 1:05 PM, Keisuke Miyako via 4D_Tech < 4d_tech@lists.4d.com> wrote: > > I thought it was also quite clear in Tim's reply which was reposted here, > that we failed you on this, and that we are in the process of updating the > documentation. > If changing a field and then saving the record doesn't save the field change, that's a bug. Adding a footnote to the documentation isn't a fix, it's an admission of failure. ** 4D Internet Users Group (4D iNUG) FAQ: http://lists.4d.com/faqnug.html Archive: http://lists.4d.com/archives.html Options: http://lists.4d.com/mailman/options/4d_tech Unsub: mailto:4d_tech-unsubscr...@lists.4d.com **
Re: [Warning] Settings properties values on object field by object notation
>I am going to post here a response I got from Tim Penner in the TAOW a bit earlier today. While it does not change my mind that this issue should be characterized as a bug, it does clarify what the issue really is. It's a bug. If 4D wants to document it as standard behavior, that's their business. But it's still a bug and will lead to real-world problems. This is the kind of bug I'd rank *extremely* serious. Why? Well, imagine any number of data change scenarios some of which change the field on one way and some in another. Depending on the exact sequence of steps the same modification will result in putting the data into a different state. This is exactly the sort of situation that can run undetected for days/weeks/months in the field. And then what do you do? There is literally no way to unscramble the eggs. You have *no idea* what quality your data is at that point. It has any number of undetectable, subtle flaws. The number may be zero, the number may be .0005% of the candidate fields, it may be 7.23% of the candidate fields. Unless you've got source data to completely rebuild everything from scratch, you'll never know and never be able to fix it. Object fields sound better and better to me all the time ;-) They're non-relational, accessed through a "novel" and incomplete JSON serialization scheme, take a lot of space (the same data in primitives takes less room), have a bunch of adjacent/extra commands devoted to working with themand, I'm told, are *not* designed to be used for the majority of cases where JSON fields are commonly used in ORDBS and NoSQL systems. I just don't get it. Anyway, modifying and saving a field of this sort should be *deterministic*. Making it functionally non-deterministic isn't defensible for a RDMS in any way. Input --> [Function] ---> It depends That's fine if you want something random, it's not fine in any other case. It's a bug. But, again, 4D is free to say it isn't. Their business, their rules. > Am I the only one that was clueless that there is a field dirty field flag that controls whether or not a field gets updated or not during a save record. Guess that does make sense. For a long time you would bump up against this on large data types and the Old() function. That's about the only practical way I remember it being an issue for us. ** 4D Internet Users Group (4D iNUG) FAQ: http://lists.4d.com/faqnug.html Archive: http://lists.4d.com/archives.html Options: http://lists.4d.com/mailman/options/4d_tech Unsub: mailto:4d_tech-unsubscr...@lists.4d.com **
Re: [Warning] Settings properties values on object field by object notation
On Wed, Nov 1, 2017 at 10:23 AM, Tim Nevels via 4D_Tech < 4d_tech@lists.4d.com> wrote: > I can guess the answer to that. The engineer responsible looked at the > code and said, “yeah I can fix that, but it will be a lot of work”. And > somebody said, "OK then don’t bother. You’ve got more important things to > do. We’ll just call it ‘Standard Behavior’”. Pitiful! :( > We'll never know. It could be that it is the result of a carefully thought out design. Then again, you could have a function like Add(2;2) that returns 5 and claim that it's "standard behavior" because it's consistent. No matter what, it's still a bug (except for sufficiently large values of 2.) ** 4D Internet Users Group (4D iNUG) FAQ: http://lists.4d.com/faqnug.html Archive: http://lists.4d.com/archives.html Options: http://lists.4d.com/mailman/options/4d_tech Unsub: mailto:4d_tech-unsubscr...@lists.4d.com **
Re: [Warning] Settings properties values on object field by object notation
> It looks like this bug ACI0097454 was marked as Standard Behavior on October 24th. Smells like a bug. Looks like a bug. Sounds like a bug. Walks like a bug. I mean, if you have a light switch that gives you a shock every time you touch it, you wouldn't say "Oh, it always does that. It's standard behavior." I mean, that's an accurate description (ouch!), but it's not a reasonable reaction. ** 4D Internet Users Group (4D iNUG) FAQ: http://lists.4d.com/faqnug.html Archive: http://lists.4d.com/archives.html Options: http://lists.4d.com/mailman/options/4d_tech Unsub: mailto:4d_tech-unsubscr...@lists.4d.com **
Re: [Warning] Settings properties values on object field by object notation
On Tue, Oct 31, 2017 at 8:21 AM, Alberto Bachler via 4D_Tech < 4d_tech@lists.4d.com> wrote: Just a quick note of thanks for posting this behavior to the list. On Tue, Oct 31, 2017 at 3:49 PM, John Baughman via 4D_Tech < 4d_tech@lists.4d.com> wrote: ...and to you too John for posting that you also got an official response. ** 4D Internet Users Group (4D iNUG) FAQ: http://lists.4d.com/faqnug.html Archive: http://lists.4d.com/archives.html Options: http://lists.4d.com/mailman/options/4d_tech Unsub: mailto:4d_tech-unsubscr...@lists.4d.com **
Re: [Warning] Settings properties values on object field by object notation
On Tue, Oct 31, 2017 at 3:49 PM, John Baughman via 4D_Tech < 4d_tech@lists.4d.com> wrote: > And, yes, I too was told that the engineers were leaning to declaring this > as standard behavior. Why? I have not a clue. > I'm failing to come up with any good arguments in support of such a posture. If you make an assignment to a field and save the record, the assignment should stick. Nothing else makes sense from a DBMS perspective. That's exactly the sort of behavior that should be absolute and unquestionable. (The only exception being that you have to be in a position to write any changes to the record.) I hope that they come in off the ledge and don't create a buggy feature that will lead to some really, really hard to detect bugs in the field for years to come. ** 4D Internet Users Group (4D iNUG) FAQ: http://lists.4d.com/faqnug.html Archive: http://lists.4d.com/archives.html Options: http://lists.4d.com/mailman/options/4d_tech Unsub: mailto:4d_tech-unsubscr...@lists.4d.com **
Re: Method to put a c-obj (or JSON) into a hierarchical list
Jeff, Now that's a smart idea. I'm interested. ** 4D Internet Users Group (4D iNUG) FAQ: http://lists.4d.com/faqnug.html Archive: http://lists.4d.com/archives.html Options: http://lists.4d.com/mailman/options/4d_tech Unsub: mailto:4d_tech-unsubscr...@lists.4d.com **
Re: info.plist
I've been working with BUILD APPLICATION a lot lately, so a couple of notes: * 4D _will_ record an error in the log if the path to the license files is wrong or, for a built single-user app, the path to the engine is wrong. * 4D will _not_ record an error if the path to your custom icons is wrong. * It's not that hard to clear out extra resources after a build, like .xlf files and so on. * 4D has a tech tip about clearing out the SVG component help file which has a helpful description of the file paths for compiled databases and merged apps. * Go ahead and clear out the /Cache folders from within Components, Plugins, and Resources after the build. They can contain 100s of MB of stuff and you don't need it. Check the archives for some past tips and code from Tim Nevels on tweaking the info/about details. There are several different build keys you can use: Tip: I also like to onboard a small build_info.json file after the build. It's got details that I might want to read out about the build itself for error logs, about boxes, error reports, etc. This isn't part of the standard build project file but it's easy to implement. I'd be very interested to hear what other people clear out post-build. ** 4D Internet Users Group (4D iNUG) FAQ: http://lists.4d.com/faqnug.html Archive: http://lists.4d.com/archives.html Options: http://lists.4d.com/mailman/options/4d_tech Unsub: mailto:4d_tech-unsubscr...@lists.4d.com **
Re: [Snippet] Process_IsPreemptive
Hello! The 1980s called, they want their naming conventions back! ;-) ** 4D Internet Users Group (4D iNUG) FAQ: http://lists.4d.com/faqnug.html Archive: http://lists.4d.com/archives.html Options: http://lists.4d.com/mailman/options/4d_tech Unsub: mailto:4d_tech-unsubscr...@lists.4d.com **
PROCESS PROPERTIES origin = -29
I've just discovered an unexpected process origin value of -29. It's not any any version of the docs that I can find. Although hunting for a -term is a challenge. Anyway, it's coming up for the Compiler window process. Has anyone else seen any other use of this origin code? Thanks ** 4D Internet Users Group (4D iNUG) FAQ: http://lists.4d.com/faqnug.html Archive: http://lists.4d.com/archives.html Options: http://lists.4d.com/mailman/options/4d_tech Unsub: mailto:4d_tech-unsubscr...@lists.4d.com **
Re: [Snippet] Process_IsPreemptive
Yeah, well, the code above doesn't work because of what the kids call a "bug". PROCESS PROPERTIES(Current process;$name;$state;$time;$mode) Should (obviously) be PROCESS PROPERTIES($process_id;$name;$state;$time;$mode) And that's what I get for *not* writing unit tests for this "simple" function. I tend to screw up simple routines at a way higher rate than more complex routines. Weird, but that's how it is for me. I also noticed that 4D late published a Tech Tip for the same functionality: http://kb.4d.com/assetid=77860 Their code looks right. Grab theirs, if you prefer their variable naming style. ** 4D Internet Users Group (4D iNUG) FAQ: http://lists.4d.com/faqnug.html Archive: http://lists.4d.com/archives.html Options: http://lists.4d.com/mailman/options/4d_tech Unsub: mailto:4d_tech-unsubscr...@lists.4d.com **
Re: POP3 Email Processing (and ramblings by Tim Nevels on 4D)
On Sat, Oct 28, 2017 at 3:21 PM, Tim Nevels via 4D_Tech < 4d_tech@lists.4d.com> wrote: > On Oct 27, 2017, at 8:11 PM, David Adams wrote: > > Tim, I wasn't really making a broad comment of any kind about 4D or criticizing - just citing a bit of the history I was there for. The original programmer was Phil Weiss, nice guy. But I do remember how shocked 4D was when they got the code and discovered that it was not re-entrant. Then there was ITK, which was really kind of great (again, a single programmer - Christian Quest?- the same person that did some of the network layer pieces back when TCP/IP was just an option). But that disappeared...or sort of withered on the vine. And then NTK, which I like a lot. All three of these tools were written by individual programmers and they didn't spend years on them. Rob could say something about how long the 4DIC-like features of NTK took to write, not that I expect him to. (No reason he should.). And he's got a cross-platform C++ framework as well, I believe. Paul Carnine wrote a multi-platform framewor too - but with support for even more platforms - for his award-winning video software. (That's where he went, if I remember correctly, it had nothing to do with 4D.) If 4D comes up with a native replacement 4DIC, great! For example, I'm really happy that they've got HTTP Get and HTTP Request in the language, very helpful. But waiting for a future feature that hasn't been announced, shown, put into a beta or released? Something that might or might not happen, and who knows when? And who knows how it will work or what it will do when/if it arrives? That's not something useful to me today or something that I can plan around. If I need a feature, I already need it. So I have to work with what we've got. Once there's a native and non-blocking way to do email in 4D, great! If they've got it and I need it, I'll check it out. Until then, there are stacks of could-based mail services with nice APIs. That's where I'd be looking, particularly since HTTP Get and HTTP Request seem to do a decent job with no real downsides. ** 4D Internet Users Group (4D iNUG) FAQ: http://lists.4d.com/faqnug.html Archive: http://lists.4d.com/archives.html Options: http://lists.4d.com/mailman/options/4d_tech Unsub: mailto:4d_tech-unsubscr...@lists.4d.com **
Re: POP3 Email Processing
> It blocks the whole application while waiting for and downloading messages from the server. This is because the plugin interface of 4D is designed > to be in cooperative mode. Even switching to v16 and using preemptive threads will not help, the plugin interface isn't thread-safe. Plug-ins can yield time (in two flavors) to the scheduler, so they don't *have* to be completely blocking. The original PDM code was not re-entrant and that's legacy seems to have been haunting us for the 25 years since 4D took it on. Plug-ins are not thread-safe in 4D, period. Will that change? *Can* that even change? That's above my pay grade...not a clue. Maybe Rob, Miyako or someone from 4D could comment. So far, I've heard nothing on this important question. What Kirk says. He's using a watch folder system which, now that I think about it, is a fantastic idea in his setup. You can use HTTP Get/Request in a non-blocking way, so that's *always* a solid possibility for external and internal communications. ** 4D Internet Users Group (4D iNUG) FAQ: http://lists.4d.com/faqnug.html Archive: http://lists.4d.com/archives.html Options: http://lists.4d.com/mailman/options/4d_tech Unsub: mailto:4d_tech-unsubscr...@lists.4d.com **
Re: Tab bar in methods
Nice! Thanks John. Thanks Tim. Much appreciated. ** 4D Internet Users Group (4D iNUG) FAQ: http://lists.4d.com/faqnug.html Archive: http://lists.4d.com/archives.html Options: http://lists.4d.com/mailman/options/4d_tech Unsub: mailto:4d_tech-unsubscr...@lists.4d.com **
Tab bar in methods
I've been noticing a "bug" for months in the Method Editor. This might just be an R-track feature, not sure. Anyway, if you open a method named "My_Method", you see the name twice. It's up in the window title and then again inside the window. It looks weird and I assumed it to be a cosmetic bug that would get caught and go away. Then I somehow opened up another method in the same window and realize that it's a tab bar in the method window. That's not a feature I'm likely to use much, but it seems like a decent feature. I use that in browsers constantly. Here's the question: Is there a way to hide the tab bar when there's only one tab? Browsers tend to have this option because a one-tab tab bar kind of makes no sense and wastes space for no purpose. I had a look at the prefs that I could find and didn't find one that seemed to apply. Any suggestions? ** 4D Internet Users Group (4D iNUG) FAQ: http://lists.4d.com/faqnug.html Archive: http://lists.4d.com/archives.html Options: http://lists.4d.com/mailman/options/4d_tech Unsub: mailto:4d_tech-unsubscr...@lists.4d.com **
Re: Server Process "Frozen"-ish
On Thu, Oct 26, 2017 at 7:45 AM, Timothy Penner via 4D_Tech < 4d_tech@lists.4d.com> wrote: Thanks for chiming in. Given the number and nature of the people reporting this bug, it's clearly a real bug. Can you give us some insight into what efforts 4D is making internally to track this down and fix it? The way I interpret your remarks is "We're waiting for customers to figure out what the problem is." Is that a mis-characterization or even a totally off-base reading of your message? Is there an *active* effort within to track down this particular problem? Thanks. ** 4D Internet Users Group (4D iNUG) FAQ: http://lists.4d.com/faqnug.html Archive: http://lists.4d.com/archives.html Options: http://lists.4d.com/mailman/options/4d_tech Unsub: mailto:4d_tech-unsubscr...@lists.4d.com **
Re: Tip: Exporting to a target folder in a package
Have you tried Select document? It lets you navigate inside of packages. As far as code goes, you can do whatever you like with a package. I've just been doing that week, doing post-build cleanup. If I remember correctly, the path is something like: ...your long path:BuildAppName.app:Contents:Database:Resources: On macOS, the application name is used like a folder name to get down inside of the package. If you're doing a build, you can insert help files, delete .xlf files, etc. On Tue, Oct 24, 2017 at 9:29 PM, Peter Jakobsson via 4D_Tech < 4d_tech@lists.4d.com> wrote: > Hi > > I have XLIFF applications that generate stuff like custom constants and > language resources. > > Unfortunately, if you use 4D’s desktop navigation commands like “Create > Document(“”)” or Select Folder and the like, to allow the user (me !) to > locate the target folder for the export, it doesn’t let you navigate inside > a package (which is where all my destination folders are since they are in > the Resource folder for the structure). > > The only way I could do it was: > > 1. open the package in the MacOS desktop using the right-click contextual > “Open Package” command > 2. drag the resources folder to the sidebar shortcuts list > 3. then use the sidebar shortcut as the target for the export path > > This seems to work. The question is, is it a feature or a bug ? Will 4D > patch this up once they discover it so that I can no longer use this > workaround ? > > Any comments, feel free ! > > Regards > > Peter > > ** > 4D Internet Users Group (4D iNUG) > FAQ: http://lists.4d.com/faqnug.html > Archive: http://lists.4d.com/archives.html > Options: http://lists.4d.com/mailman/options/4d_tech > Unsub: mailto:4d_tech-unsubscr...@lists.4d.com > ** ** 4D Internet Users Group (4D iNUG) FAQ: http://lists.4d.com/faqnug.html Archive: http://lists.4d.com/archives.html Options: http://lists.4d.com/mailman/options/4d_tech Unsub: mailto:4d_tech-unsubscr...@lists.4d.com **
Re: QUERY into variable vs. Records in Table
> And all it's doing is waiting for a relatively rare condition to happen (and a record to appear in this table). It might > happen at most once or twice per day, maybe only once every few days, but when it happens it has to be dealt with > immediately. It's such a waste to be constantly querying for a record that 9 times out of 10 won't exist. Can this rare condition that creates a record do anything else? Something that would make it into a push/notification instead of record/semaphore that you have to constantly poll for? And, the last time I tested, Records in table takes advantage of a stored statistic so it's speed is very fast and also very consistent. It doesn't change based on table size, etc. As you know, InnoDB doesn't store that stat, so SELECT COUNT(*) on a big table can kill you. Postgres, on the other hand, stores tons of stats and has a sample-based frequency table on board or the query analyzer to read from. ** 4D Internet Users Group (4D iNUG) FAQ: http://lists.4d.com/faqnug.html Archive: http://lists.4d.com/archives.html Options: http://lists.4d.com/mailman/options/4d_tech Unsub: mailto:4d_tech-unsubscr...@lists.4d.com **
Re: PDF in Web Area not displaying on a Mac
John, This all sounds very familiar & is why I went with Rob's viewer plug-in. http://www.pluggers.nl/product/pdf-viewer/ This plug-in isn't crazy expensive like some of the PDF generating tools. (I think Rob faces high licensing costs of his own on those.) ** 4D Internet Users Group (4D iNUG) FAQ: http://lists.4d.com/faqnug.html Archive: http://lists.4d.com/archives.html Options: http://lists.4d.com/mailman/options/4d_tech Unsub: mailto:4d_tech-unsubscr...@lists.4d.com **
Re: PDF in Web Area not displaying on a Mac
On Thu, Oct 19, 2017 at 10:51 AM, Wayne Stewart via 4D_Tech < 4d_tech@lists.4d.com> wrote: > John, > > I use Rob Laveaux's pdf viewer plugin rather than the web area.I use Rob > Laveaux's pdf viewer plugin rather than the web area. +1 ** 4D Internet Users Group (4D iNUG) FAQ: http://lists.4d.com/faqnug.html Archive: http://lists.4d.com/archives.html Options: http://lists.4d.com/mailman/options/4d_tech Unsub: mailto:4d_tech-unsubscr...@lists.4d.com **
Re: v13+ unti testing UI
Chip, Your idea is vastly simpler to implement in versions of 4D with CALL FORM and CALL WORKER. That's somewhere from the late 15 R series onwards and V16.0+. ** 4D Internet Users Group (4D iNUG) FAQ: http://lists.4d.com/faqnug.html Archive: http://lists.4d.com/archives.html Options: http://lists.4d.com/mailman/options/4d_tech Unsub: mailto:4d_tech-unsubscr...@lists.4d.com **
Re: Server Process "Frozen"-ish
Cannon, Thanks for being so systematic about tracking down this problem and keeping the list informed. ** 4D Internet Users Group (4D iNUG) FAQ: http://lists.4d.com/faqnug.html Archive: http://lists.4d.com/archives.html Options: http://lists.4d.com/mailman/options/4d_tech Unsub: mailto:4d_tech-unsubscr...@lists.4d.com **
Re: 4D v16.2 false record locks
> But at the risk of dating myself John was a *very* precocious 8 year old. On Mon, Oct 16, 2017 at 5:22 AM, John Baughman via 4D_Tech < 4d_tech@lists.4d.com> wrote: > Hey Chuck your right of course. But at the risk of dating myself, when > I first came to Hawaii in 1969 for my first tour in the Navy I was given a > choice, fly or boat. A first class booking on the SS Lurline sure was > tempting, but this sailor got sea sick far too easily so I opted to fly. > > John > > > Sent from my iPad > John Baughman > Kailua,Hawaii > john...@hawaii.rr.com > > > On Oct 15, 2017, at 6:32 AM, Chuck Miller via 4D_Tech < > 4d_tech@lists.4d.com> wrote: > > > > I assume he flew > > > > Regards > > > > Chuck > > > > > Chuck Miller Voice: (617) 739-0306 > > Informed Solutions, Inc. Fax: (617) 232-1064 > > mailto:cjmillerinformed-solutions.com > > Brookline, MA 02446 USA Registered 4D Developer > > Providers of 4D and Sybase connectivity > > http://www.informed-solutions.com > > > > > This message and any attached documents contain information which may be > confidential, subject to privilege or exempt from disclosure under > applicable law. These materials are intended only for the use of the > intended recipient. If you are not the intended recipient of this > transmission, you are hereby notified that any distribution, disclosure, > printing, copying, storage, modification or the taking of any action in > reliance upon this transmission is strictly prohibited. Delivery of this > message to any person other than the intended recipient shall not > compromise or waive such confidentiality, privilege or exemption from > disclosure as to this communication. > > > >> On Oct 11, 2017, at 11:09 PM, Alan Tilson via 4D_Tech < > 4d_tech@lists.4d.com> wrote: > >> > >> Hey John, > >> How did you end up in Hawaii? > >> Alan > >> > >> On Wed, Oct 4, 2017 at 10:02 PM, John Baughman via 4D_Tech < > >> 4d_tech@lists.4d.com> wrote: > >> > >>> I am going to chime in here with a recent real world experience. > >>> > >>> A while back I posted to the NUG a problem that I was having with > external > >>> data storage using custom mode. When the data was updated and properly > >>> saved to the external data folder, the field continued to display the > old > >>> version of the data until the server or standalone 4D was restarted. > This > >>> was a show stopper for me upgrading a client to v16. > >>> > >>> I called tech support and after uploading a sample db demonstrating the > >>> problem was told that it was in fact a bug and a bug case was opened > for > >>> it. This bug existed in 16.1 and R4 beta. > >>> > >>> I then called Tracy, my sales rep, and asked her to expedite the bug. > >>> > >>> A couple of weeks ago I received a message from 4D TS saying the bug > was > >>> fixed in the latest nightly build of 16.1. I confirmed the fix, but > noted > >>> that it was still broken in R4 beta. TS acknowledged this and said > that the > >>> fix would not show up in the R releases until R5. > >>> > >>> 16.2 was released and as expected no bug. > >>> > >>> Today R4 was released and voila the bug has been fixed. > >>> > >>> Now in the context of this thread, this appears to be an anomaly? It > does > >>> say to me that 4D does heavily prioritize the bugs in their bug list. > For > >>> example, you might have noticed that in 16.2 the issue with the Design > >>> Environment not remembering the open windows from the last session has > been > >>> fixed. Not so in R4. Interesting, yes? > >>> > >>> For me personally, I am not one to complain. I may be naive, but I > trust > >>> that the folks working on 4D are doing the best they can. Sure there > could > >>> be improvements, but I am one to live with what I got and be patient > for > >>> what is yet to come. > >>> > >>> I understand the frustrations, but that is my 2 cents worth. > >>> > >>> John > >>> > >>> > >>> > >>> > >>> > >>>> On Oct 4, 2017
Re: v13 - odd compiler error
Check if there is a typo and you've declared $43 instead of $3. The errors you describe are almost exactly what I would expect in this case. Not that I've ever done anything like that...but it happened to a friend once or twice. ** 4D Internet Users Group (4D iNUG) FAQ: http://lists.4d.com/faqnug.html Archive: http://lists.4d.com/archives.html Options: http://lists.4d.com/mailman/options/4d_tech Unsub: mailto:4d_tech-unsubscr...@lists.4d.com **
Re: NB: Pre-emptive mode hassle
Right, about the custom code scanner/pre-processor/compiler macro idea, I forgot to mention that you could use such a tool to remedy problems in 4D. For background, some of this takes a syntax in your code that enables you to identify the following: -- Required parameters -- Optional parameters -- Parameters that are of magic types, such as strings that are actually method names. There are a ton of ways to implement that, I've settled on one I'm happy enough with...but it involves work no matter what. (4D's compiler declaration system isn't very expressive.) Anyway, with that in place and a custom code scanner, here are some very handy things that you can do: * Count parameters and see if optional parameters are missing or if there are too many parameters in place. (The compiler doesn't do this reliably and has no concept of an optional parameter.) * Interpret parameters of magical types. For example, check that a call to ON ERR CALL is taking a proper method name, or that a call to CALL WORKER is taking a preemptively safe method. (This second one is harder to do, but doable.) * Strip out calls to TRACE as they (mysteriously) are thread-unsafe. Why? Why are they even in the compiled output? It makes no sense, they can't run anyway. And more...I'm sure everyone can come up with a few ideas for what you can do by rewriting code just before compile and build. ** 4D Internet Users Group (4D iNUG) FAQ: http://lists.4d.com/faqnug.html Archive: http://lists.4d.com/archives.html Options: http://lists.4d.com/mailman/options/4d_tech Unsub: mailto:4d_tech-unsubscr...@lists.4d.com **
Re: NB: Pre-emptive mode hassle
> Also trying to justify reducing the size of the process variable table. > Supposedly having 1 - 10 object variables was far superior to 100 process > variables. Who the hell cares if your process variable table is 1MB in size > when everyone has over 4,000MB to work with? And don’t try to say “it will > save time starting new processes”. Really? By how much real world time? By > 4ms? Can any human notice a decrease in time by that much? That whole business with the process variable table was and is nonsense. It makes no measurable difference to process startup and never has. > And if you really think about it, a C_OBJECT variable most likely takes up > more space in memory than a C_TEXT or C_LONGINT or C_BOOLEAN. You know, with > all the extra overhead that a C_OBJECT has to manage to provide the fast > lookup and access to properties. Don't know, can't know, 4D refuses to say. Absolutely refuses. "Because it might change." ? Such an absurd and unconvincing position to take. Here is where I *don't* talk about the Postgres documentation other than to say "how it's done." > I do a lot of UI work with 4D. I love doing UI work. Windows and dialog boxes > that users deal with all day every day. So 4D is great for a lot of the work > that I do. Right on! I wasn't saying 4D is necessarily bad for that - just that it's not my favorite thing to do. I actually really like 4D's form editor and think that they should have patented the field entry UI. That's genius. The Properly List's buggy text handling has been a pain for 20 years or sobut, overall, I like multiple pages and other stuff in there. > Isn’t it interesting that 4D became so useful and popular because it provided > a great way to build very nice user interfaces to database solutions. I never > realized you don’t care about UI stuff and actively avoid it. It explains a > lot. You are a code monster and backend database monster. If 4D is not > advancing the language and the database you think it is sitting still. Too bad > they can’t seem to advance both with equal speed. Yeah, I'm happiest in a pure code situation, for sure. I very much think about code as a series of modules with well-defined interactions. UI tends to gum that up...plus there are those slow, erratic, inefficient peripherals attached - people. Hey don't get me wrong! I'm a total people personwell, apart from the people part. > And by the way, your 4D Method presentation example database had a very nice > UI. Beautiful. So you do have talent or good taste there, in my opinion. Dude, you need to raise your standards, but thanks. All I've ever really done is steal the 4D graphic design charter from the 1990s, grab the FatCow icon set, and make sure that things tend to be aligned. I did just pick up "Information Dashboard Design" by Stephen Few, which I'm pretty excited about. But that's about the visual display of statistical data and is for my work with Domo, Loggly/SumoLogic not 4D. ** 4D Internet Users Group (4D iNUG) FAQ: http://lists.4d.com/faqnug.html Archive: http://lists.4d.com/archives.html Options: http://lists.4d.com/mailman/options/4d_tech Unsub: mailto:4d_tech-unsubscr...@lists.4d.com **
Re: Min/Max for Date Fields in 4D
On Thu, Oct 12, 2017 at 6:57 AM, Arnaud de Montard via 4D_Tech < 4d_tech@lists.4d.com> wrote: > > It's a solution if the selection is huge (is it?). But sort selection > (I've always found 4d sort is slow), then 2 selection to array, that's 3 > server calls, at end. I'd prefer selection to array (1 call), sort array, > read first and last items in array. > Agreed. I'd want to know that there is any optimization needed past FIRST RECORD and LAST RECORD before diving into anything else. And, past that, you would want to balance the cost of pulling more data over the network in once call with pulling tiny bits of data over the network with multiple calls. I suspect this is getting a bit silly, but another idea is to have an method set to "Execute on server" that does the work and packs the results in an object and returns it. Yeah, that might work. ** 4D Internet Users Group (4D iNUG) FAQ: http://lists.4d.com/faqnug.html Archive: http://lists.4d.com/archives.html Options: http://lists.4d.com/mailman/options/4d_tech Unsub: mailto:4d_tech-unsubscr...@lists.4d.com **
Re: Min/Max for Date Fields in 4D
Alan, I haven't followed this thread closely, so apologies if I say something silly. * Is it important to optimize this operation past the sensible solution you've already tried? * Since you already have a sorted selection, why no use that and SELECTION RANGE TO ARRAY? So, use SELECTION RANGE TO ARRAY to get a single value from the first record, that's your min. Then use SELECTION RANGE ARRAY again on the last record to get the max. I can't think of anything else better. You would have to test it out to see if the double call is even worth the trouble. On Wed, Oct 11, 2017 at 11:30 AM, Alan Tilson via 4D_Tech < 4d_tech@lists.4d.com> wrote: > Greetings everyone, > > Is there a way to get the earliest and latest dates from a selection of > records similar to using Min/Max for numerical data? > > I have sorted the data and captured the first and last dates but this seems > rather a lot of work to get this info. > > I believe that I have also used Selection To Array to capture the dates in > the array and then sorted the array. > > Surely there is something I've overlooked? > > Currently using v13.6. > > Thanks, > Alan Tilson > ** > 4D Internet Users Group (4D iNUG) > FAQ: http://lists.4d.com/faqnug.html > Archive: http://lists.4d.com/archives.html > Options: http://lists.4d.com/mailman/options/4d_tech > Unsub: mailto:4d_tech-unsubscr...@lists.4d.com > ** ** 4D Internet Users Group (4D iNUG) FAQ: http://lists.4d.com/faqnug.html Archive: http://lists.4d.com/archives.html Options: http://lists.4d.com/mailman/options/4d_tech Unsub: mailto:4d_tech-unsubscr...@lists.4d.com **
Re: Making LOG EVENT thread-safe
Hey John, we ended up testing different things, I didn't ever get too far into the memory side. (Thanks for that, good to know.) I retested again last night and I can kill 4D from entirely legal code. I'll show you at dinner, if you like. There is a *concurrency bug*. I'm not convinced that everyone at 4D fully grasps the nature of a concurrency bug and they're definitely not grasping their importance. It's an existential bug. What's going on should be *impossible*. I don't mean rare (it is rare, but not that rare.) There's a _reason_ I've reported this bug and have been going on about it. When do I normally do that? I don't. And it is extremely unlikely that I ever will again. There's a margin in doing so, but it's pretty much all negative margin ;-) They should email Laurent E., he understands concurrency perfectly and should appreciate the situation. Plus, I think this feature set comes from him. It's possible Engineering will discover that it's a highly specific and localized bug, not a more generalized bug - let's hope so. But there is *no* way for us to know that on the outside. I could keep probing to get a better sense of the shape of the bug, but that's a massive misuse of my time. I've got real work to do. So, no logging to files via a worker in 4D for me :( I think that I can still use normal processes to do a record-based passthrough, but that's a very different performance proposition entirely. (There's no way that I can use that approach for the Web logs, I've killed 4D before that way...not doing it again.) ** 4D Internet Users Group (4D iNUG) FAQ: http://lists.4d.com/faqnug.html Archive: http://lists.4d.com/archives.html Options: http://lists.4d.com/mailman/options/4d_tech Unsub: mailto:4d_tech-unsubscr...@lists.4d.com **
Re: NB: Pre-emptive mode hassle
> This would be super useful!. Great feature that would make it much easier to adapt methods so that they would run in > preemptive mode. The 4D engineers probably never even thought about doing something like this. If you don't mind and are on the forums, would you mind putting in a feature request? And, for what it's worth, it would be super weird if they *hadn't* thought of this. The languages they write in have all kinds of compiler pre-processing instrucxtions and compiler macros. > I would suggest something like this: > //%C- to turn off compiling code > SET TEXT TO PASTEBOARD($stuff_t) > //%C+ to turn on compiling code > That’s how you do it with range checking using //%R- and //%R+. Nice! I was thinking of how you turn off warnings (look at Compiler_Suppress_Warnings in our source if you don't know that one), but your way is vastly better. It's actually a really exciting feature request the way you have it, submit it! > We have to remember to bring this up at 4D Summit next year in the Q&A session so it is publicly directed at Laurent. You > get him to publicly say “yeah, that is a good idea, we’ll do it”. That’s the best way to cut through the red tape and get a > new feature implemented. Hmm. Yes. That sounds like an efficient way to manage software development. Cough-cough. Hopeully, the feature request system has some value. (Time will tell.) > That is exactly the advice JPR gave at the 4D World Tour. If it ain’t broke, don’t rewrite it! > Use it for new code or if you have to redo things to get a performance boost. Fair point, but my point is a bit different: It is suprisingly hard to reuse *any* code. Error handlers, utility methods, tons of stuff that we all have around. And not for obvious reasons like showing UI. So then what? You've got two different versions of your code to make one preemptively safe? When the only difference is, say, the TRACE command. Duplicating routines and making the drift apart is a bad sort of code decay. That's the hardship, not the idea of writing fresh code for fresh problems. > Too many developers go crazy when 4D adds some new features and get it in their head they MUST > rewrite everything to use the new features. That’s not perspective. I think “nice, another tool > in the toolbox that I can use for something new when I get the chance." Honestly, I can't remember anyone I've met like that. Plenty of people dive in to check out a new featureabout 6-12 months after they're introduced. I doubt a lot of people are trying to use preemptive mode. Or, if they are, they're not saying anything here. > I'd love to hear from other people that are using preemptive processes > regarding things they found that are good, bad, surprising, exciting, and > me'h. > CALL FORM. CALL FORM. CALL FORM. That’s the most useful new command I’ve found. Yes, it solves a lot of problems that I don't have ;-) I'm just not into complicated 4D GUI work by choice. If I were doing that sort of thing, I'd use CALL FORM for sure. (EXECUTE METHOD IN WINDOW.) The old Outside Call/CALL PROCESS system was dangerous garbage. And don't get me started on GET/SET PROCESS VARIABLE. Rob's IPC channels in NTK work great. The design is better than CALL FORM and would be familiar to anyone that's used conventially message queues. (You send messages, you don't push code into another thread and expand it.) But plug-ins aren't thread-safe. ** 4D Internet Users Group (4D iNUG) FAQ: http://lists.4d.com/faqnug.html Archive: http://lists.4d.com/archives.html Options: http://lists.4d.com/mailman/options/4d_tech Unsub: mailto:4d_tech-unsubscr...@lists.4d.com **
Re: NB: Pre-emptive mode hassle
Thanks Tim! More findings: * I don't get it. I really am unclear how exactly things are supposed to be set up. It's confusing, at least to me. I've tried diagrams and writing it out...but I'm still not clear. * I want to use the 'indifferent' execution mode everywhere I can, but that's not working out. * You cannot set execution mode in batch mode the way you can with the invisible, etc. attributes. * Dang, it's seriously time-consuming and hard to nail it down. Hopefully, the tools will improve a lot in future versions. I have some optimism that they will. * TRACE is not thread-safe. It's also meaningless compiled. And yet, it blocks you. Why? * SET TEXT TO PASTEBOARD is not thread-safe. I only use it for debug stuff, it would be nice to have an option to have a compiler declaration syntax to say something like: // %i C:123 ... or whateverlike the range check compiler comments. The idea being you can say "just compile this out, don't include it when you see it" and then the command number. (Made up number above.) * If you're building a component and you want the shared methods to be accessible to preemptive methods in the host, it looks like you *must* declare the component method as 'can be run in a preemptive', etc. The 'indifferent' option doesn't seem to work. That could be documented, I have no idea. And, as a reminder, no IP variables. For existing pseudo-constants, use real custom constants instead: Managing Custom Constants with Code http://kb.4d.com/assetid=77806 All thanks and praise to Cannon! With all of the above said, my biggest advice is *wait*. This stuff is going to have to get better. (The compiler in particular.) If you have some tasks where preemptive threads are going to be a big help (workers or regular processes), fine, go for it. But it's going to be easier to write fresh code than rewrite or reuse old code in a lot of real-world databases, so try and start with something this is either 1) super simple or 2) mostly based on new code. I'd love to hear from other people that are using preemptive processes regarding things they found that are good, bad, surprising, exciting, and me'h. ** 4D Internet Users Group (4D iNUG) FAQ: http://lists.4d.com/faqnug.html Archive: http://lists.4d.com/archives.html Options: http://lists.4d.com/mailman/options/4d_tech Unsub: mailto:4d_tech-unsubscr...@lists.4d.com **
Re: NB: Pre-emptive mode hassle
Right, I just scanned a big code base for methods passed to CALL WORKER (some my code, some not) and found that 5 out of 7 won't run in preemptive mode. Oh crap. ** 4D Internet Users Group (4D iNUG) FAQ: http://lists.4d.com/faqnug.html Archive: http://lists.4d.com/archives.html Options: http://lists.4d.com/mailman/options/4d_tech Unsub: mailto:4d_tech-unsubscr...@lists.4d.com **
Re: Post Key command not working in converted 4Dv16 database
Hey John, Yeah, I pretty well guessed something like that was going on in your case...I mostly answered for the sake of the archives. I often ask questions and am offered well-meaning and sensible answers...but there's almost as often a good reason why I'm bearing down on the particular point raised. So, understood. ** 4D Internet Users Group (4D iNUG) FAQ: http://lists.4d.com/faqnug.html Archive: http://lists.4d.com/archives.html Options: http://lists.4d.com/mailman/options/4d_tech Unsub: mailto:4d_tech-unsubscr...@lists.4d.com **
Re: NB: Pre-emptive mode hassle
Typical, my "summary" is longer than the original. ** 4D Internet Users Group (4D iNUG) FAQ: http://lists.4d.com/faqnug.html Archive: http://lists.4d.com/archives.html Options: http://lists.4d.com/mailman/options/4d_tech Unsub: mailto:4d_tech-unsubscr...@lists.4d.com **
Re: NB: Pre-emptive mode hassle
To summarize the bit above: -- If you pass non-preemptive code to a pre-emptive worker, it won't work. -- That's not a bug, that's a limitation. -- 4D doesn't automatically handle this situation well, but they've said that they'll make ON ERR CALL work, which seems like a reasonable response. -- The Compiler should check methods passed to workers. Maybe it will, maybe it won't. I think that it should (obviously), but I don't get to decide. -- For now, what you need to do is remember to *manually* check code passed to the second parameter of CALL WORKER. It's on us. -- The easiest way that I've found to take this step is as follows: * Create a method marked to run in preemptive mode. You don't run it, it's just a place to expose code to the compiler. * Use Find in Design to locate all CALL WORKER calls. * Copy and paste in whatever methods you pass in the second parameter. These are what get run through EXECUTE by the worker, once received. * Use the compiler to check syntax. Now you should see errors on the methods that are going to fail. A bonus from doing things this way is that you get an error the *specific* methods that are causing trouble. For those that don't know, when you compile 4D scans the visible call tree for each method that _might_ be called from within a preemptive process. However, it only reports specific errors at the top level. It's pretty frustrating trying to figure out what the problem is (don't forget triggers). The docs say that this is the way the Compiler works, so it's not technically a bug (unexpected behavior.) It is, however, a giant, time-sucking hole that makes getting code into preemptive mode harder than it needs to be. Seems like a good feature request - the compiler gathers the necessary information during its scan and then throws it away and gives us a report that says "Something seems not okay?" Anyway, that's the state of the art today. P.S. Because of the silent failures in preemptive mode, your code may not be executing at all. Or some of it might not be. Orit might be causing problems. There's just no way to know without looking very, very carefully. It's early days for preemptive mode, so I'd suggest you consider any work with it experimental. ** 4D Internet Users Group (4D iNUG) FAQ: http://lists.4d.com/faqnug.html Archive: http://lists.4d.com/archives.html Options: http://lists.4d.com/mailman/options/4d_tech Unsub: mailto:4d_tech-unsubscr...@lists.4d.com **
NB: Pre-emptive mode hassle
I've finally tracked down a weird problem. I have some code that I'm (still) trying to run in a worker. I've got the system set up so that I can launch in cooperative or preemptive mode based on a setting. This way, if there's a problem with preemptive mode, we can tweak a config setting and try again in cooperative mode. This is all working in tests. But my overall system was failing to deliver in preemptive mode. The preemptive worker launches just fine...by my subsequent calls to the work fail, but only in preemptive (multi-thread/multi-core) mode. I've got an ON ERR CALL installed that's thread-safe...but it never fires. I think this is part of a bug I reported months ago regarding passing non-thread safe code to a thread-safe worker. The compiler tries to stop us from *compiling* such code, but the compiled mode runtime interpreter does not. You can pass anything into a worker, even unsafe code. The exact results at that point areundefined. I think that Thomas Maul said on the forums that getting ON ERR working reliably in this situation would be done. (I'm in R3, maybe it has been done.) Anyway, the warning is this: You have to take extra steps to vet your code because *the compiler does not inspect the statements that you submit through CALL WORKER.* I think it's recently started testing that ON ERR CALL methods exist in one situation, but that's the first and only time I'm aware of that 4D sees arguments to EXECUTE (under whatever name) as anything bug raw strings. So, lesson relearned. Anyway, what you need to do is look through your code and see any that might be Unclean. The compiler is said to be able to dump out this attribute for all methods. The other way is to create a method that is marked to run in preemptive mode and then put calls to your other methods in there. In this case, the compiler will flag any that are unsafe for you. Kind of a hassle, but there it is. ** 4D Internet Users Group (4D iNUG) FAQ: http://lists.4d.com/faqnug.html Archive: http://lists.4d.com/archives.html Options: http://lists.4d.com/mailman/options/4d_tech Unsub: mailto:4d_tech-unsubscr...@lists.4d.com **
Re: Post Key command not working in converted 4Dv16 database
Hey John, if you still need to fix what you described, CALL FORM ought to work. Just past what method you want to run and the target window's form will run it through EXECUTE for you. doesn't mean that POST KEY shouldn't be fixed if it's broken, of course. ** 4D Internet Users Group (4D iNUG) FAQ: http://lists.4d.com/faqnug.html Archive: http://lists.4d.com/archives.html Options: http://lists.4d.com/mailman/options/4d_tech Unsub: mailto:4d_tech-unsubscr...@lists.4d.com **
Re: Query By Attribute vs Map NULL values to blank values
> I still do not get why the R4 documentation for Query by attribute discusses how a # query will function if the "Map null > to blank” checkbox is or is not checked it the checkbox does not exist for an object field. John, random thought: Object fields aren't supported in 4D's SQL. So any SQL-specific features and such are completely meaningless for ojject fields. And by "not supported" I don't mean ignored. If you do SELECT * FROM MyTableWithAnObjectField; 4D freaks out because there's an object field. At least as late as R3. This has come up a few times here and other people have looked at this in detail. (I think that Jeff Kain, at thel least, has the details.) In my case, I had to rewrite my ugly little SQL browser to scan the field list for object fields and remove them from statements. That's okay for my needs, but I'm not using 4D's SQL for anything real any longer. ** 4D Internet Users Group (4D iNUG) FAQ: http://lists.4d.com/faqnug.html Archive: http://lists.4d.com/archives.html Options: http://lists.4d.com/mailman/options/4d_tech Unsub: mailto:4d_tech-unsubscr...@lists.4d.com **
Re: Making LOG EVENT thread-safe
> As a side note, I am sure you are aware of this, but by killing the worker every 100,000 calls, out of the 2 million calls > nearly all of the log entries are missed as they are in the queue when the worker is killed. I am getting at most 24,000 > log entries in the text file. I've lost track, but I think that this was the very thing I was testing - the %/# of lost log lines I would end up getting. The more I think about it, the more I'm convinced that what I'm trying to do simply isn't part of the designed capacity of 4D's system. I can live with that, I'll just move the logging out of 4D. I may lose something that way (a bit of data that would come from a 4D looking that could inject some extra data) and it's a pity, but 4D never promised me a log writer. It's just not sufficient to requirement. So, checking what NGNIX offers. On the log enrichment side, Loggly offers, well, very little but I'll be checking out SumoLogic down the road and that looks a whole lot better. (It's a Splunk clone, as far as I can see, but cheaper.) ** 4D Internet Users Group (4D iNUG) FAQ: http://lists.4d.com/faqnug.html Archive: http://lists.4d.com/archives.html Options: http://lists.4d.com/mailman/options/4d_tech Unsub: mailto:4d_tech-unsubscr...@lists.4d.com **
Re: Making LOG EVENT thread-safe
Add, On the entirely likely chance that my first answer sounded insufferably arrogant - just wanted to addWell, I'm sleep deprived like I can't remember, and massively jet lagged. I was trying to say that this whole area is one of my *few* areas of genuine expertise. I don't have many. Just ask anyone ;-) It's just an accident of personal history that this is one of my areas of interest and experience - there are *tons* of equally important things that I'm clueless about. So, not trying to be a jerk about it, but I figure it was worth giving you the background that I'm not figuring any of these things out from scratch at this point...only figuring out how to implement things within 4D's feature set. ** 4D Internet Users Group (4D iNUG) FAQ: http://lists.4d.com/faqnug.html Archive: http://lists.4d.com/archives.html Options: http://lists.4d.com/mailman/options/4d_tech Unsub: mailto:4d_tech-unsubscr...@lists.4d.com **
Re: Making LOG EVENT thread-safe
> I understand your thinking that 4D may not the way to go for event logging, > but your concern about CLOSE DOCUMENT is where we do not agree. I'am sure you > have seen CLOSE DOCUMENT fail, but I suspect that what you are seeing is not > an issue with CLOSE DOCUMENT, but something else not readily apparent. I think > you said at some point that 4D has recognized this as a bug, but I still > question whether in fact it is. Yes, you're absolutely right on this point. I don't know what the bug is. It *manifests* as an impossible file lock error, but I think it's very, very likely that this is a symptom of something else. Either way, it's a show-stopper for me and I can't fix it. > Perhaps you can show me in the sample database how to make CLOSE DOCUMENT fail > or have another database that demonstrates the problem? That one already shows the problem. I've reproduced it, as has 4D and a couple of other people. Once it happens, you get all kinds of weird stuff - duplicate workers with "unique" names, the runtime monitor gets super buggy and starts throwing range errors...until you're dead. If you *don't* find this result on your gear, please post what version of 4D you're testing with. >> When I saw your initial post in this regard, I jumped in because I do a lot of > work that involves the opening and closing of documents. If a problem exists I > really need to know about it. Thus far my testing based on your posts has not > demonstrated anything that has caused me to doubt that CLOSE DOCUMENT will in > fact close a file. Just try out a compiled version of the original database I sent without any of your modifications and let it run for a couple of minutes. Oh, I forgot to say earlier about John's finding of ~2.1M messages being a kind of breaking point...that number may not be replicable in other tests. You might find a different number. The payloads in my scratch database are quite small. For all I know, if you made the payloads 10x bigger, you would crash with ~210K messages. I won't be testing this myself. On Mon, Oct 9, 2017 at 3:33 AM, John Baughman via 4D_Tech < 4d_tech@lists.4d.com> wrote: > > > On Oct 8, 2017, at 4:42 PM, David Adams via 4D_Tech < > 4d_tech@lists.4d.com> wrote: > > > > What I'm concerned about is that CLOSE DOCUMENT may be > > returning without the file *actually* being closed. That's really > > problematic. > > > I understand your thinking that 4D may not the way to go for event > logging, but your concern about CLOSE DOCUMENT is where we do not agree. > I'am sure you have seen CLOSE DOCUMENT fail, but I suspect that what you > are seeing is not an issue with CLOSE DOCUMENT, but something else not > readily apparent. I think you said at some point that 4D has recognized > this as a bug, but I still question whether in fact it is. > > Perhaps you can show me in the sample database how to make CLOSE DOCUMENT > fail or have another database that demonstrates the problem? > > When I saw your initial post in this regard, I jumped in because I do a > lot of work that involves the opening and closing of documents. If a > problem exists I really need to know about it. Thus far my testing based on > your posts has not demonstrated anything that has caused me to doubt that > CLOSE DOCUMENT will in fact close a file. > > John > > > John Baughman > Kailua, Hawaii > (808) 262-0328 > john...@hawaii.rr.com > > > ** > 4D Internet Users Group (4D iNUG) > FAQ: http://lists.4d.com/faqnug.html > Archive: http://lists.4d.com/archives.html > Options: http://lists.4d.com/mailman/options/4d_tech > Unsub: mailto:4d_tech-unsubscr...@lists.4d.com > ** > ** 4D Internet Users Group (4D iNUG) FAQ: http://lists.4d.com/faqnug.html Archive: http://lists.4d.com/archives.html Options: http://lists.4d.com/mailman/options/4d_tech Unsub: mailto:4d_tech-unsubscr...@lists.4d.com **
Re: Making LOG EVENT thread-safe
Hey John, thanks for the great post and all the work. There's no reason for other people to know this, but John and I have *totally* different coding styles. So I was really pleased he went through my example and reworked it in his own way. That he turned up some problems is instructive because he, for sure, approaches things form a different angle than me. > Open/Close Document: I think we all agree, even David, that it is now and has always been our responsibility to close > documents as quickly as possible after opening them. Sorry I gave the impression at some point I ever thought otherwise. I now and always believe that we're responsible for closing our own files. That's not a question. What I'm concerned about is that CLOSE DOCUMENT may be returning without the file *actually* being closed. That's really problematic. > Worker processes in 4D are ready for prime time. I suspect that this is true for many uses of workers. For UI stuff, they seem like they'll be able to keep up with what even a complex UI in 4D can push at it. And workers and CALL FORM are *massively* better than any previous native solution. All of that stuff with SET PROCESS VARIABLE or shared arrays with semaphores was just terrible - real garbage. I've said that for years, so I was quite excited about the new features. On the other hand, I was looking forward to taking a standard high-volume task and leverage 4D's preemptive processing ability. John came up with a solution, but it involves Semaphores and quite a bit of infrastructure that either makes preemptive unavailable, or so expensive to get that it's pointless. For reliability, I'm going to look at sending the logs from NGNIX and cut 4D out of the process entirely. ** 4D Internet Users Group (4D iNUG) FAQ: http://lists.4d.com/faqnug.html Archive: http://lists.4d.com/archives.html Options: http://lists.4d.com/mailman/options/4d_tech Unsub: mailto:4d_tech-unsubscr...@lists.4d.com **
Re: Making LOG EVENT thread-safe
Add, Thanks a lot for chiming in! It's always good to hear from you guys in the US office. I'm not sure what problem your solution is solving. I'm not trying to be flip, I just want to make sure to focus on the meat of the issue. A really common approach to log writing is as follows: Have one process manage the log disk file. So. when that thread starts, it opens the file, keeps it open, writes to it and can then close it neatly. You can then have any number of other processes message the "writer" to stream information to the disk file. Lots of OS-level files, Web servers, etc. follow this model. Why? Because it is *substantially* faster and easier than anything else. You completely sidestep race conditions on the file lock - there's no chance of a race condition because only one process *ever* opens the file. It's a nice, tidy solution. That's what I wanted to do with 4D's worker when I first saw them - I assumed that would be pretty much anyone's impression because it's such a classic use for a message+worker setup. I was excited because with a preemptive worker, you could push all of this low-value logging over onto a secondary core. It's a satisfyingly complete, tidy, and naturally low-conflict, high-performance solution. That doesn't work in 4D. Unfortunately, this doesn't seem to work. I can easily generate race conditions from opening and close the file and restarting the worker. This should be *impossible*. Something is wrong in how 4D is managing file locks. I don't know what. I start a worker, open a file, write to it, close the file, kill the worker. The file should be closed. Period. Usually it is, sometimes it isn't. I have no explanation for this, but it definitively blows up any plan to use workers for logging via a preemptive process. John Baughman in his tests thinks that the problem is flooding the worker and blowing up 4D's via memory exception. Could be. I briefly did a bit of load testing like that, but not a lot. So, either way, it doesn't look like 4D's CALL WORKER and worker system is designed to scale or work safely/reliably under heavy load. I've come to suspect that it was never intended for such tasks.The lack of a pause mechanism or a way to count 'queued' entries are telling. They're very standard (universal) features of a queuing/messaging system, but 4D doesn't have them or seem to appreciate why you would need them. I've got feature requests in for this stuff from months ago. Note that if you do any kind of heavy load, you need to be able to measure the state of a queue to see if you need to spin up another one. That 4D lacks this feature (or seemingly interest in such a feature) suggests that I'm off-base to even consider 4D workers for heavy loads. Just not built for it. 4D never said that their workers were meant for heavy lifting. I think that they were designed to handle various kinds of UI-related tasks in 4D. I haven't got much interest in that (I did a bit of a publish-subscribe system that I showed at 4DMETHOD, that was fun to work through), and I'll guess that CALL WORKER and CALL FORM are just fine for that. But for what I need? I have to conclude that 4D isn't the right tool ** 4D Internet Users Group (4D iNUG) FAQ: http://lists.4d.com/faqnug.html Archive: http://lists.4d.com/archives.html Options: http://lists.4d.com/mailman/options/4d_tech Unsub: mailto:4d_tech-unsubscr...@lists.4d.com **
Re: Making LOG EVENT thread-safe
I was never focused on LOG EVENT (?) I've never used the command and would prefer not to in this case. I only brought it up because it would potentially well with Loggly. (I'm not committed to Loggy and may not stick with that platform, for what that's worth. But it's worth trying for now.) I have no access to the forums, so I don't know what your link says. But if the title is safe to go by then, yes, that might help. I'll have to test it out to see how well it works before I know. In theory, what I was trying to do in the first place is a completely safe idea - even better than LEP. But it isn't safe in practice. ** 4D Internet Users Group (4D iNUG) FAQ: http://lists.4d.com/faqnug.html Archive: http://lists.4d.com/archives.html Options: http://lists.4d.com/mailman/options/4d_tech Unsub: mailto:4d_tech-unsubscr...@lists.4d.com **
Re: Making LOG EVENT thread-safe
I've posted about my various worker log problems here and on some other threads and even posted my sample database for review. Lately, John Baughman has really been sinking his teeth into this issue and has spent a *ton* of time working on it. Thank you John! The news is bad, but I figured I'd post it here as part of this thread. I won't say a lot, but John might want to explain his point of view. He's an optimistic person, with a cheerful nature and he likes people. Despite that, I still like him ;-) Anyway, John took a copy of the sample database (including the correction contributed by David Porter) and tried to figure out what he could. My summary: * John is able to crash 4D just by flooding the queue. * He came up with some ideas to help manage the situation, but they involved either or semaphores or IP variables. * John also came to see (very quickly) that you need a function to report current queue size. Now, it's entirely possible that 4D never intended their CALL WORKER system to be used in a high-transaction setting. They have offered no guidance on the systems limitations and scope. I briefly tried crashing it with a message flood months back and thought that 4D preformed well. In fact, I was looking to see if it created disk scratch files, but I couldn't find any. John sounds like he's able to flood and crash the system pretty reliably, but not every time. One of my primary goals in using CALL WORKER was to avoid writing records and take some work off of core 1. There appears to be no safe and reliable way to meet those goals. This is a total bummer as even my HTTP push solution won't be safe. My next good idea is to see about pushing as much of the logging as I can onto NGNIX which will absolutely, definitely have zero problems with whatever we can throw at it. John my have more to say, but I think I'm likely done with this subject. ** 4D Internet Users Group (4D iNUG) FAQ: http://lists.4d.com/faqnug.html Archive: http://lists.4d.com/archives.html Options: http://lists.4d.com/mailman/options/4d_tech Unsub: mailto:4d_tech-unsubscr...@lists.4d.com **
Re: 4D v16.2 false record locks
Hey John, Thanks again for trying. I kill the worker from *within* the worker *after* closing the file. It *usually* works. But "usually" is worthless when the consequences of it not working are a dead server. 4D has accepted it as a bug in, I think 16.0. they just haven't fixed it. For now, I'm not using files from workers. I'm streaming C_OBJECTS over CALL WORKER, invoking a method that transforms the results, bundles it up and then pushes the data in batches to Loggly. We'll see how that goes. ** 4D Internet Users Group (4D iNUG) FAQ: http://lists.4d.com/faqnug.html Archive: http://lists.4d.com/archives.html Options: http://lists.4d.com/mailman/options/4d_tech Unsub: mailto:4d_tech-unsubscr...@lists.4d.com **
Re: How do workers die?
> > Perhaps I am misunderstanding your question. Do you mean after a worker > has been killed with KILL WORKER? > Clearly, I asked my question poorly as you and Julio both found my question unclear in the same way. (Or perhaps it's just that it's unclear to people who can speak Portuguese?) In any case, yes, I mean when a worker has been killed explicitly. I've tested out workers extensively - I've spent months on CALL WORKER, etc. and drafted a few hundred pages about them. None of that will ever see the light of day, but I've spent a fair amount of time studying them. I'm now trying to put them into production, working with the limitations they have based on the design choices that were made for this part of 4D. And bugs. The way I like to kill workers is to have a method that does it that is called from *within* the worker. This allows for a more organized shutdown. Closing files, etc. You can call the worker to execute the method in the context of the worker very easily. Tip: It's also easy to forbid method to run outside of the context of a worker with a specific name. If (Current process name # "YourWorkerNameHere") // Deal with the illegal call. Else // Do your thing End if That's a solution to a problem not everyone will run into, but it's a tidy little trick when it is appropriate. ** 4D Internet Users Group (4D iNUG) FAQ: http://lists.4d.com/faqnug.html Archive: http://lists.4d.com/archives.html Options: http://lists.4d.com/mailman/options/4d_tech Unsub: mailto:4d_tech-unsubscr...@lists.4d.com **
Re: How do workers die?
Julio, Thanks for taking a stab at it. I should have been more clear, I mean when workers are killed. I do this from within the worker to try and have an orderly shutdown. Your reading of the docs is correct though. Workers are a thread of execution, like any other process, but 4D has its own magic loop waiting for incoming 'messages'. These messages aren't really messages, they're blocks of code that are appended to the worker's thread of execution and then run through EXECUTE. At least, that's how they behave. ** 4D Internet Users Group (4D iNUG) FAQ: http://lists.4d.com/faqnug.html Archive: http://lists.4d.com/archives.html Options: http://lists.4d.com/mailman/options/4d_tech Unsub: mailto:4d_tech-unsubscr...@lists.4d.com **
Re: 4D v16.2 false record locks
Again: Start worker Open file. Use file Close file. Kill worker Start worker again *** FIle lock conflict *** 4D's accepted it as a bug as of some months back. ** 4D Internet Users Group (4D iNUG) FAQ: http://lists.4d.com/faqnug.html Archive: http://lists.4d.com/archives.html Options: http://lists.4d.com/mailman/options/4d_tech Unsub: mailto:4d_tech-unsubscr...@lists.4d.com **
Re: 4D v16.2 false record locks
Hey Chip, thanks or chiming in. I've always been disciplined about opening and closing files on my own, I have no idea what 4D's defaults are on this...or if there is even a documented (promised) behavior. The traces of old, dead workers I was referring to *might* not actually exist and that wasn't what I was talking about. I'm talking closing the file explicitly, killing the worker explicitly, and then the worker is restarted. And that's when you can run into a conflict. Which should never happen, but it does. Regarding <>whatever, you can't use that kind of variable in a preemptive process of any kind. That's one of the more painful costs of admission, but it also makes sense. ** 4D Internet Users Group (4D iNUG) FAQ: http://lists.4d.com/faqnug.html Archive: http://lists.4d.com/archives.html Options: http://lists.4d.com/mailman/options/4d_tech Unsub: mailto:4d_tech-unsubscr...@lists.4d.com **
How do workers die?
Does anyone know or can they find out the following? What happens when workers die? 4D has at least two different approaches to this: * HTTP processes wait before dying, but then die if there are no new requests after some time. (Or at least they used to.) Meaning, an HTTP process may or may not have old variables and so forth in place. * New process () / Execute on server () When they finish, they're dead. They're supposed to release file locks, record locks, etc. If you start a new copy of the process with the same name, it starts off in a "virginal" state. Which is it for workers? I thought it was like New process(), but am seeing evidence that it's not always the case. The docs aren't explicit. I take the implication to be that they should be fully dead, even if started almost instantly. But this doesn't seem to be true. The difference is pretty important. Is there a way to find out authoritatively? I've sent it to Wayne for tech support, so maybe that will work in time...but perhaps someone knows or has a shortcut to finding out? ** 4D Internet Users Group (4D iNUG) FAQ: http://lists.4d.com/faqnug.html Archive: http://lists.4d.com/archives.html Options: http://lists.4d.com/mailman/options/4d_tech Unsub: mailto:4d_tech-unsubscr...@lists.4d.com **
Re: 4D v16.2 false record locks
> Not an anomaly. John & Neil, thanks for the reports, I find them encouraging. We're dealing with somewhat different situations. You're going through US Tech Support, which I have no contact with and you're using the TAOW, which we don't currently have access to in AU/NZ. The business case for 4D devoting resources to rolling their own bulletin boards and help desk software instead of renting tools from Atlassian (Aussie! Aussie! Aussie! Oi! Oi Oi) or similar escapes mebut that's a different discussion. Anyway, our tech support is Wayne Stewart. Everyone loves Wayne. If I made anyone think that I was criticizing Wayne, I am sorry and also sad. Wayne is a helpful *to a fault*. Even now he's trying to come up with workaround for my particular bug. But here's the thing, it's a *concurrency problem* and it is *inside of 4D*. That's not a problem we can fix. The only thing Wayne can do (and has done months ago) is kick it to France. Its their bug and only they can fix it. Wayne told me that they accepted it as a bug months ago, and that's the last we heard. Why do I care about this bug? No one here will have noticed, but I don't talk about bugs a whole lot. For the most part, I don't care. If 4D can't do it one way and I can find another, I'll do that. If 4D can't do it, I'll find another tool. In this case I *do* care. Here's why: It's an important bug that blocks certain designs and leaves me mistrusting workers and preemptive mode quite fundamentally. My problem boils down to how workers deal with file locks. In theory, a worker processes requests in strict order. (They're not requests, they're EXECUTE statements run in the context of the worker, but nevermind). So, if you have a worker and close a file, it should be closed completely. If you kill the worker and it restarts, the file close done explicitly in the worker should finish before the worker dies. Managing file locks across/amongst process is the sort of basic concurrency problem that was worked out over 50 years ago. And when I say "worked out", I don't mean "dude wrote some code" I mean the basic reality and mathematics of concurrency were worked out. In that process, the semaphore was invented. Why? Because it is necessary in many situations. Are semaphores hard to get your head around? Yes. But what's _really_ at issue are the issues of concurrent computing themselves, that's what's hard to grasp. I remember how many smart people just could not accept that semaphores were necessary to lock shared resources when V3 introduced semaphores. Some people _still_ don't seem to grasp them. Here are the options that you've got for a situation where a race condition might occur: 1) You develop a system that is *provably* never going to have the problem, ever. 2) You are inevitably going to encounter the problem. Because science. There is no third option other than "it doesn't seem likely, so I'll risk it." See outcome 2. In my case, I'm architecting some stuff that is going to be very high volume, so option 2 isn't a choice. So I need a truly safe solution. I stumbled across the file lock problem *by accident*. I don't want this bug to exist. But here's the thing, if what we've been told about workers and preemptive process is all correct, then this bug is *impossible*. But it's there. So there's something most definitely wrong. Is the problem very narrow? Is it widespread? I have no way of knowing and 4D has told me exactly nothing. (I asked on the Forums but was a) told nothing or b) accused of trying to "sabotage" the command, whatever that means.) So, I have to go with what I can see: It is not safe to rely on file open/close in workers. I tried a delay and a bunch of other things and they don't solve the problem. I guess never closing the file might work, but that's not an option. (I don't want a 2GB HTTPD log file, thanks.) And here's the thing, there is no workaround to a concurrency bug. It's 4D's bug, only they can fix it. So, in my case, to get my custom log data out I have to avoid files entirely. Again, high volume - race condition possible = race condition inevitable. I've got two different architectures: Source process ---> CALL WORKER ---> HTTP Request --> Logging platform Or [Log_Record] Read with a standard process Write to a log file There is so much data involved that log records aren't a sustainable solution. (The last time we went down this road we killed 4D and then InnoDB in MySQL. Postgres, you're next!) As I've been unclear: The lock problem manifests differently in cooperative and preemptive processes, but occurs in both. There is no workaround, only alternative architectures and an actual fix to 4D. My frustration is not with tech support - Wayne's great - it's with France not fixing their bugs. And, as it has _always_ been for me, they're an informational black hole. I don't get any information from them. A few times over the years, ever. That's it. So, how am I to know if they're working on this?
Re: What to do about users who won't follow instructions?
On Thu, Oct 5, 2017 at 3:13 PM, Stephen J. Orth via 4D_Tech < 4d_tech@lists.4d.com> wrote: > > Let me know what you think... > Awesome! Thanks for sharing the pictures, they really get your idea across. ** 4D Internet Users Group (4D iNUG) FAQ: http://lists.4d.com/faqnug.html Archive: http://lists.4d.com/archives.html Options: http://lists.4d.com/mailman/options/4d_tech Unsub: mailto:4d_tech-unsubscr...@lists.4d.com **
Re: What to do about users who won't follow instructions?
Steve, That looks great! Thanks for the screen shots, they really show what you were talking about. Curious: Do the pictures show up on the NUG? If we can post pictures there...I'd like to know. ** 4D Internet Users Group (4D iNUG) FAQ: http://lists.4d.com/faqnug.html Archive: http://lists.4d.com/archives.html Options: http://lists.4d.com/mailman/options/4d_tech Unsub: mailto:4d_tech-unsubscr...@lists.4d.com **
Re: What to do about users who won't follow instructions?
On Wed, Oct 4, 2017 at 5 :06 PM, Stephen J. Orth via 4D_Tech < 4d_tech@lists.4d.com> wrote: > Pat, > > Completely agree on your comment about video! > > > > Let me know if you would like to see a screen shot or anything... > That sounds *awesome*. Any chance that you would do a 4DMETHOD or 4D Summit presentation on this? ** 4D Internet Users Group (4D iNUG) FAQ: http://lists.4d.com/faqnug.html Archive: http://lists.4d.com/archives.html Options: http://lists.4d.com/mailman/options/4d_tech Unsub: mailto:4d_tech-unsubscr...@lists.4d.com **
Re: 4D v16.2 false record locks
> If you have a bug that you really want fixes: you need to work with Tech Support. If you don’t: then it’s just wishful thinking. I have done this. It's a bug that 4D should care about *way* more than me. It crashes the server. Easily Submitted to tech support with a demo database, etc. It is NOT MY FAULT that 4D doesn't fix properly reported bugs. It is NOT OUR FAULT. It is their fault. I'm getting tired of hearing from people that someone bugs aren't being fixed because we're somehow mysteriously doing it wrong. I also appreciate that I'm one of the very people that spit out that line over, and over again down the years on this list. Here's a list of positions that I can no longer accept on face value: "We aren't documenting that because it might change." This is silly in the extreme. Every piece of software might change, lots of them say "This might change in a future version." Even 4D says this clearly in their docs. So when you hear that line, it's an excuse for something else - who knows what. "It isn't Tech Support's job to report bugs." Of course it is. "You have to <> to get you bug fixed." 1. It's not my bug, it's THEIR bug. They're not doing me a favor. 2. Jumping through the hoops makes no particular impact that I can see. "Feature requests should be submitted through the appropriate section of the forums where there is open voting." True. Does this get feature requests implemented? Not as far as I can tell. (Well, I did see one - but it was from Rob and he's got a special relationship with 4D.) My cynical interpretation is that the feature request forum exists to stop people complaining about missing features. Your mileage may vary. ** 4D Internet Users Group (4D iNUG) FAQ: http://lists.4d.com/faqnug.html Archive: http://lists.4d.com/archives.html Options: http://lists.4d.com/mailman/options/4d_tech Unsub: mailto:4d_tech-unsubscr...@lists.4d.com **
Re: 4D v16.2 false record locks
> It is what it is. What’s more important, getting the bug fixed or bitching about customer service? > I always vote for doing whatever it takes to get bugs fixed. Agreed! But I've found _nothing_ that works. Ever. I've jumped through all of the hoops down the year, it hasn't worked. My crashing bug with V16.0+ where I supplied a database to demonstrate the problem in about a minute? Nothing. No response, no action, no fix. I take exception to the notion that somehow our bugs aren't getting fixed because we're "not doing them right." Sure, that explains some of it. All of it? Not a chance. *It's not our fault.* You mentioned that it's not Tim Penner's job to take comments here and report bugs. Why not? In fact, Tim does just that a lot of times - which seems like good service from Tim and the US crew. I mean, it's crazy to think that somehow it's *our* fault that a bug isn't getting fixed even though 4D knows about it. It's their bug, not ours. I can't wait on bug fixes anyway. With their response times and release cycles, I need to fix a problem in an hour or a day - I can't wait six months to forever. If I can't use 4D for a task, I just don't. When I reported my crashing bug (with demo database), I did it for *their* sake. I figured "Hey, they're proud of preemptive mode and workers. Guess they'd like to know it can bring down the server." Not so much. I don't get that. In my case, I'm using a different architecture (an inferior one) as I can't rely on logging to disk from a worker. I used to report bugs and make feature requests a lot. I was one of the top submitters for some time. Then one day I realized out of hundreds (probably thousands) of reports, I'd gotten a couple of emails. Ever. I knew how to do a "proper" submission because I'd worked at 4D for four years in Tech support and then running IT in the USA. I couldn't get support for crashing servers *then*. And it was their business. But I definitely knew how to format a bug report. Down the years I've used bugs.4d.fr and more lately the Forums. Still, no change. No feedback, no fixes. Maybe it's just me? I'm not kidding about that. Neil reports a very different, and satisfactory-sounding, experience. Part of the problem is doubtlessly that as an Aussie developer, I don't have access to the bug system. But here is a little thought experiment. Imagine two different company's with different attitudes to bugs and quality. Let's call them Huey and Dewey. Huey is *passionate* about quality and customer service. They take is as a core part of the business to detect bugs before they reach customers, to improve their processes when a bug does slip through, to release fixes as soon as possible - even before customers know that there is a bug, and to do everything reasonable to help customers reporting a problem. They're constantly trying to improve their already substantial documentation. (I saw a company just like this last year, it was a thing of true beauty.) Dewy, well Dewey just doesn't know. They don't keep customers up to date, they add a lot of hassle between customers and problems, they don't actively look for non-critical bugs, they have no roadmap, they don't share their priorities or even logic, and they ignore problems that they are fully aware of. Many key features and behaviors are undocumented or underdocumented. What does 4D look more like? Huey or Dewy? ** 4D Internet Users Group (4D iNUG) FAQ: http://lists.4d.com/faqnug.html Archive: http://lists.4d.com/archives.html Options: http://lists.4d.com/mailman/options/4d_tech Unsub: mailto:4d_tech-unsubscr...@lists.4d.com **
Re: 4D v16.2 false record locks
Dennis, That's great to hear. This is definitely not my experience...perhaps they don't want to hear from me. I'm glad that you're getting good support, that's encouraging. ** 4D Internet Users Group (4D iNUG) FAQ: http://lists.4d.com/faqnug.html Archive: http://lists.4d.com/archives.html Options: http://lists.4d.com/mailman/options/4d_tech Unsub: mailto:4d_tech-unsubscr...@lists.4d.com **
Re: 4D v16.2 false record locks
Tim, I'm with John on this one. The payoff for sending in bug reports and feature requests is incredibly low. Zero feedback and results are rare. *No matter what hoops are jumped.* 4D the company gives the _very_ clear impression that they just don't want to know. If they do want to know, they're doing a terrible job of making that clear, making it possible, making it work, or making it worth out time to even try. IMO ** 4D Internet Users Group (4D iNUG) FAQ: http://lists.4d.com/faqnug.html Archive: http://lists.4d.com/archives.html Options: http://lists.4d.com/mailman/options/4d_tech Unsub: mailto:4d_tech-unsubscr...@lists.4d.com **
Re: 4D v16.2 false record locks
...but seriously, what I'd like to see 4D do is an *entire release* that consists of *nothing* but fixing bugs - as many as they can find, as many as we can find and longstanding annoyances. No fancy new features. Just cleaning up crashes, usability bugs (I see lingering 2004 or V11 usability bugs in the Design environment every single day in V16) and other weird stuff. From what I've ever heard, this is what the bulk of developers *actually* want. On Tue, Oct 3, 2017 at 9:43 PM, David Adams wrote: > > That’s not his job. It is the job of every 4D developer to report bugs > to 4D by opening a tech support case. > > And to refuse to close that tech support case, or allow it to auto > close, until the bug is fixed. > > Sigh. Brining a fresh new flavor to the concept of "world class" customer > service... > ** 4D Internet Users Group (4D iNUG) FAQ: http://lists.4d.com/faqnug.html Archive: http://lists.4d.com/archives.html Options: http://lists.4d.com/mailman/options/4d_tech Unsub: mailto:4d_tech-unsubscr...@lists.4d.com **
Re: 4D v16.2 false record locks
> That’s not his job. It is the job of every 4D developer to report bugs to 4D by opening a tech support case. > And to refuse to close that tech support case, or allow it to auto close, until the bug is fixed. Sigh. Brining a fresh new flavor to the concept of "world class" customer service... ** 4D Internet Users Group (4D iNUG) FAQ: http://lists.4d.com/faqnug.html Archive: http://lists.4d.com/archives.html Options: http://lists.4d.com/mailman/options/4d_tech Unsub: mailto:4d_tech-unsubscr...@lists.4d.com **
[Snippet] Process_IsPreemptive
I realized that I wanted to know for some error logs if the current process is preemptive (thread-safe) or not. So I checked PROCESS PROPERTIES and, yes, it will tell you. But it uses bit math. Oh no! I always run away from bit math...and I decided that I was being silly to do so today. The "math" in this case is nothing but an operator, not even slightly hard. So, here's the code which I've tested out and seems to work: // Process_IsPreemptive // DESCRIPTION: Tests is process is running preemptively. C_BOOLEAN($0;$is_preemptive) C_LONGINT($1;$process_id) If (Count parameters>=1) $process_id:=$1 Else $process_id:=Current process End if C_TEXT($name) C_LONGINT($state;$time;$mode) PROCESS PROPERTIES(Current process;$name;$state;$time;$mode) C_BOOLEAN($preemptive) $is_preemptive:=$mode ?? 1 $0:=$is_preemptive This is handy for error logs, as mentioned, and also just to verify what's going on. It's surprisingly tricky to launch a process in preemptive mode...or least I find it so. You can use New process, Execute on server or CALL WORKER, or so the docs say. I tried this code out with a worker and a standard process, both in single-user. (Preemptive mode isn't supported on 4D Remote becausewe don't know, 4D hasn't said. Presumably something in the network layer?) Tip: The order of arguments for method to run and process name are the same for New process and Execute on server but are reversed on CALL WORKER. Tip: The easy way to see the state of a process is from the Runtime Explorer. There are new icons since late in the V15R series for each sort of process. Oddly enough, a preemptive process' icon changes to a cooperative process icon when it dies. So, don't necessarily trust the icons for dead processes. ** 4D Internet Users Group (4D iNUG) FAQ: http://lists.4d.com/faqnug.html Archive: http://lists.4d.com/archives.html Options: http://lists.4d.com/mailman/options/4d_tech Unsub: mailto:4d_tech-unsubscr...@lists.4d.com **
Re: Love 4D's math interpretation...
Crap, it's documented. And, as we all know, 4D doesn't document or discuss things that might change in future versions for fear of frightening the children. But this is documented, so I guess we're safe relying on it as is ;-) ** 4D Internet Users Group (4D iNUG) FAQ: http://lists.4d.com/faqnug.html Archive: http://lists.4d.com/archives.html Options: http://lists.4d.com/mailman/options/4d_tech Unsub: mailto:4d_tech-unsubscr...@lists.4d.com **
Re: Making LOG EVENT thread-safe
Related wrinkleI'm now wondering about something else: Does KILL WORKER completely kill a worker, or does it sometimes just hurt it really, really badly? I have been working under the assumption that when KILL WORKER completes, that the worker is *totally* gone. Meaning, no process-level values (variables, etc.) remain anywhere.A worker can be restarted very quickly after being killed. It gets the same name (workers have unique, case-sensitive names, outside of buggy situations) - but the contents should be blank. At least, that's what I think and what would be best. Web processes, as an example, do *not* work like this - they stay alive for a few seconds and may get reused. New process *does* work like this. Let a process finish and restart a fresh version and you get a blank slate. I suspect with workers that they normally die completely when killed, but not always. I've just reread the docs and they don't address this point explicitly. This is the sort of question I tried using the Forums to get answered and, frankly, I didn't get answers. So, I'm going to have to go with the assumption that a new worker *may not be virginal.* There may be some gunk hanging around form an earlier incarnation of the worker. This is only relevant if you use KILL WORKER, but that's a standard command. Does anyone know anything more about this that could help change some of my speculations into information, one wary or another? ** 4D Internet Users Group (4D iNUG) FAQ: http://lists.4d.com/faqnug.html Archive: http://lists.4d.com/archives.html Options: http://lists.4d.com/mailman/options/4d_tech Unsub: mailto:4d_tech-unsubscr...@lists.4d.com **
Re: Making LOG EVENT thread-safe
Hey, David Porter spotted a bug in my code and I wanted to push this part of the thread up to the list for the archives, etc.: On Tue, Sep 26, 2017 at 5:05 PM, David Porter wrote: > Thanks for posting the LogWorker code, it has been fun experimenting with it. > > I have been testing only the preemptive test code. > Running on a Mac Pro (Late 2013) with 6-Core Intel Xeon E5 , Mac OS X 10.12.6 > 4D 16.1 hotfix 2 , 64 bit app. ( Not server ) > > Yes, there is a bug in the code. > Method: LogWorker_Kill ,need to add the line > Log_Path:=“” // This is required to get the LogWorker_Start code to open the log again > > Even so, your code would crash interpreted before the Kill was ever sent. > Putting in delay process in LogWorker_CallABunch_Preemptive, would get it to run interpreted. > So, clearly there is a 4D bug here. > > Compiled, I am able to remove all the delay processes, and run it to completion. > I do get some runtime errors about broken communication. > Again, I can make them go away with some delay processes, something I was trying to avoid. > > So, I encourage you to take another look at this code. > My first reply, before checking the code based on what David said: Hey! Thanks for checking the code, you're awesome! Thanks for the bug catch on my code. I'm sure that wasn't from me. Darn kids, getting into my code and messing things up How long did you have to delay process compiled to get it to work? I tried up to 3 seconds and it still didn't work. In theory (as I understand it), DELAY PROCESS should *not* be needed. Meanin, CLOSE DOCUMENT should not return until the document is actually closed. I think I even tried it with SET CHANNEL and found the same problem, but don't quote me on that. If my understanding about the synchronous nature of the file/channel commands is correct, and a process delay is needed, it points to an underlying problem that would leave me not trusting the solution. Even if it only shows up after a few hours/days/weeks...that's not good enough :( My second reply now: Okay, I've gone back and updated the code, just as David described. That fix makes the system *way* better. Unfortunately, even with that fix and the 3 second delay, the problem still crops up. It takes longer, but it's still there. I guess it's an (un)happy coincidence that my bug amplified an underlying bug in 4D. For a logging situation, a race condition rate of 1:1,000,000 isn't even close to good enough. It needs to be zero. Always. In this case, i was able to provoke the bug in a few minutes :( I'm told that 4D has accepted my report but Partners in Aus don't have direct access to the bug system. And I've never heard anything from France so I don't have a bug report number or anything else to do on. ** 4D Internet Users Group (4D iNUG) FAQ: http://lists.4d.com/faqnug.html Archive: http://lists.4d.com/archives.html Options: http://lists.4d.com/mailman/options/4d_tech Unsub: mailto:4d_tech-unsubscr...@lists.4d.com **
Re: Making LOG EVENT thread-safe
Hey Tim, > Consider the situation where you have a mission critical system that is up 24/7 and you need logging to always work every second of the day. > How do you handle the situation where the stand-alone log writer app crashes, is not running due to a machine restart, or for whatever reason > goes deaf and stops responding to the HTTP calls? You certainly don’t want any possibility of losing some logging information. Again, talking about a workaround. I want to log to disk out of 4D via SEND PACKET. As to your point, it's true. However, I'm seeing is a *far* more likely that 4D Server crashes than a 4D stand-alone. But your point is good. Perhaps it's better to find a small, stable service (Windows) that can run in the background to handle the HTTP-push-to-disk task and leave 4D out of it entirely. ** 4D Internet Users Group (4D iNUG) FAQ: http://lists.4d.com/faqnug.html Archive: http://lists.4d.com/archives.html Options: http://lists.4d.com/mailman/options/4d_tech Unsub: mailto:4d_tech-unsubscr...@lists.4d.com **
Re: Making LOG EVENT thread-safe
Keisuke, Thanks for the background information on LOG EVENT, your details are *always* interesting. In my case, I don't think that it has anything to do with what I'm dealing with. I'm not talking about 4D events or 4D logs, I'm talking about custom logs with data I'm sending out. LOG EVENT isn't my first choice, it isn't even my secondbut my first and second choices aren't doable because of this fatal bug in 4D. On Tue, Sep 26, 2017 at 4:26 PM, Keisuke Miyako via 4D_Tech < 4d_tech@lists.4d.com> wrote: > sometimes you will see a kind of "comment" line in the request log, > in particular "TTF" or time to forget, > which describes a change of state in the process: > > a: automatic relation > b: record stack > c: current record > d: current selection > e: transaction > > these are not real "requests", > in that the information is not sent by itself. > rather, it has an impact on future requests. > > "INFO" is a pseudo module name > (as opposed to srv4, cli4, dbmg, etc.) > for writing such entries in the request log. > > in v2004, 4D was cooperative, > so you could trust that the logs are recorded sequentially. > > however, since v11, > a different process could write interject between "srv4" and "INFO", for > example, > so you can not assume that the two subsequent lines make a pair. > > "process info index" and "task id" is used to link such lines. > > it's in the documentation. > > http://doc.4d.com/4Dv16/4D/16.1/Appendix-E-Description-of- > log-files.300-3373556.en.html > > > 2017/09/26 23:07、David Adams via 4D_Tech <4d_tech@lists.4d.com> のメール: > > Not sure which ID that is > > > > > ** > 4D Internet Users Group (4D iNUG) > FAQ: http://lists.4d.com/faqnug.html > Archive: http://lists.4d.com/archives.html > Options: http://lists.4d.com/mailman/options/4d_tech > Unsub: mailto:4d_tech-unsubscr...@lists.4d.com > ** > ** 4D Internet Users Group (4D iNUG) FAQ: http://lists.4d.com/faqnug.html Archive: http://lists.4d.com/archives.html Options: http://lists.4d.com/mailman/options/4d_tech Unsub: mailto:4d_tech-unsubscr...@lists.4d.com **
Re: Making LOG EVENT thread-safe
Chip's right about server-side code not needing network bandwidth. I was jumbling a few of my stories unclearly. Yes, we're starting with server-side only process getting logged, but that will change. Once you want to keep error, access, and event logs and tie them together, it's nice to be able to log from the client side as well. I'm planning ahead a bit here. > doesn't have to be that way; you can have a local external database on the client side. If I'm going to use an external database, it's going to be Postgres. 4D has given every appearance of abandoning further work on their SQL "support", so I'm not going to base any designs on what amounts to a dead-end feature. Sorry :( I liked the idea of SQL in 4D and loved the idea of external database. But, for what it's worth, they're no usable from preemptive processes anyway, so... > but I am slightly puzzled how LOG EVENT is in the spotlight. I brought it up because that's one way to get events to Loggly. Loggly describes itself as an "agentless" service. Which just means "we use system agents and don't provide our own." So, LOG EVENT should work. File that under "commands I've never used". So I wasn't even clear if it was a sane idea to ask for a preemptive-safe version. Seems like it should be. If it isn't, I'll just put that idea out of my mind forever. > preemptively logging can make things more complicated, My basic design with a preemptive worker collecting log events in memory centrally and pushing them out to a disk file is *entirely* normal. This is what high-volume Web servers do. Probably anyone that's done a lot of working thought of the same design *instantly* when they heard about CALL WORKER. It's just that obvious. The only reason I can't do this is because of a *crashing server bug* in 4D. As far as I can tell. So, good design, proven design, sensible design...doesn't work. Doesn't workwith extreme prejudice. > a new ID was introduced because preemptive multi-line logs can be interwoven. Not sure which ID that is, but I'm planning on using the unique ID from PROCESS PROPERTIES and a unified timeline to tie together events from different logs related to the same process. ** 4D Internet Users Group (4D iNUG) FAQ: http://lists.4d.com/faqnug.html Archive: http://lists.4d.com/archives.html Options: http://lists.4d.com/mailman/options/4d_tech Unsub: mailto:4d_tech-unsubscr...@lists.4d.com **
Re: Making LOG EVENT thread-safe
On Tue, Sep 26, 2017 at 10:38 AM, Jeremy Roussak via 4D_Tech < 4d_tech@lists.4d.com> wrote: > David, > > Thanks for the explanation. Your second point (no reason to have the data > in 4D) strikes me as more theological than practical, Depends on your situation. I nearly killed a system once from pushing the logging data over the network. Not philosophical, existential in the contemporary sense of "life and death threat." So, that happened. But in a small system? Everything is easier in a small system. So, yeah, unless you're going to have tons of data in the logs and they're very busy, I'd say that it's a matter of taste. > but the rest are certainly excellent reasons. I wasn’t aware that 4D > didn’t cope well with thrashing tables, so that’s something I’ll bear in > mind in future. > Well, that's been my experience, perhaps things have changed? I usually mange to slip out of the back of the room when they're asking for volunteers to do server admin. Many years of suffering...don't like server admin. ** 4D Internet Users Group (4D iNUG) FAQ: http://lists.4d.com/faqnug.html Archive: http://lists.4d.com/archives.html Options: http://lists.4d.com/mailman/options/4d_tech Unsub: mailto:4d_tech-unsubscr...@lists.4d.com **
Re: Making LOG EVENT thread-safe
> As stupid as it sounds, another option is to push the log entries to something else locally - even a stand-alone built 4D app - and let it write the disk files. > Then you could use a service with a custom agent, like SumoLogic or Splunk. Actually, that's not a terrible idea at all...who cares if that data file gets bloaty? > Clear it out, throw it out, start a new one. Now I'm loving this idea. Do a special build to generate a stand-alone log writer app and message it with HTTP. This tool is functioning as a logging "agent." There are a lot of advantages to this approach in some environments: * You can put it on the same machine as the server(s), or not. * You can send in all sorts of different logging data. * You can do something smart with the data, like aggregates, if you want to. (I don't want to, but someone else might like to do that.) * You completely divide up stashing something in the log "send it to the log writer" from what happens next. Want to change formats? Services? Transport mechanisms? It's all right there. So, you could use disk files one day and switch to HTTP upload the next...or pushing it into Postgres, or whatever. * Since the target app is a stand-alone, it can run preemptive processes. Nice when you're generating log data on 4D Remote which _cannot_ run preemptive processes. Yeah, I think this has gone from Plan B to Plan A. On Tue, Sep 26, 2017 at 8:28 AM, David Adams wrote: > > Not having my server crash sounds like a very good reason to use records. > > At least until a better solution comes along. > > Plan B in this case is to push the data to Loggly via HTTP. > > As stupid as it sounds, another option is to push the log entries to > something else locally - even a stand-alone built 4D app - and let it write > the disk files. Then you could use a service with a custom agent, like > SumoLogic or Splunk. Actually, that's not a terrible idea at all...who > cares if that data file gets bloaty? Clear it out, throw it out, start a > new one. > > ** 4D Internet Users Group (4D iNUG) FAQ: http://lists.4d.com/faqnug.html Archive: http://lists.4d.com/archives.html Options: http://lists.4d.com/mailman/options/4d_tech Unsub: mailto:4d_tech-unsubscr...@lists.4d.com **
Re: Making LOG EVENT thread-safe
> At least until a better solution comes along. I have zero reason to assume that this bug will ever be fixed. ** 4D Internet Users Group (4D iNUG) FAQ: http://lists.4d.com/faqnug.html Archive: http://lists.4d.com/archives.html Options: http://lists.4d.com/mailman/options/4d_tech Unsub: mailto:4d_tech-unsubscr...@lists.4d.com **
Re: Making LOG EVENT thread-safe
> Not having my server crash sounds like a very good reason to use records. > At least until a better solution comes along. Plan B in this case is to push the data to Loggly via HTTP. As stupid as it sounds, another option is to push the log entries to something else locally - even a stand-alone built 4D app - and let it write the disk files. Then you could use a service with a custom agent, like SumoLogic or Splunk. Actually, that's not a terrible idea at all...who cares if that data file gets bloaty? Clear it out, throw it out, start a new one. ** 4D Internet Users Group (4D iNUG) FAQ: http://lists.4d.com/faqnug.html Archive: http://lists.4d.com/archives.html Options: http://lists.4d.com/mailman/options/4d_tech Unsub: mailto:4d_tech-unsubscr...@lists.4d.com **
Re: Making LOG EVENT thread-safe
On Mon, Sep 25, 2017 at 6:06 PM, Jeremy Roussak via 4D_Tech < 4d_tech@lists.4d.com> wrote: > Please forgive me if I’m being dim, but isn’t a solution (maybe not the > best, but a solution) to maintain the log as records in a table, which is > periodically emptied into a file by a process which opens, writes and > closes that file, then deletes the records in the table? > > It’s not an approach which I’ve tried, and you may have good reasons for > rejecting it. I’m just curious to know what they are. > > Jeremy, Using a 4D table as a sort of cache for log lines is a perfectly sensible idea, but also one I'd like to avoid. I did this some years back in a very high-volume system and ended up killing a lot of performance as a consequence. (Just ask Justin Leavens.) Now, you could take steps to reduce the network activity involved, to be sure, but there are still a few points that come to mind: * 4D doesn't do well (or at least hasn't historically done well) with tables that get thrashed a lot. Add-delete-add-delete. Gets really slow and horrible. So then you have to do compacts, or hijinks with reusing records, or all kinds of special magic with never deleting records. Life is too short for any of that, if at all avoidable. * There's no _real_ reason to have the data in 4D at any point. 4D isn't an analysis platform and it isn't something that I'd use for high-volume/low-data-content material like log entries. So, there's no good reason for it to be in 4D other than as a cache. I would consider using a small table as a cache for storing lines that didn't upload/output to a log file, but only if the % of failure was pretty low. * With a post-to-table-log-and-clear system, you now need a process to poll the table. You can do this in a way that is relatively inexpensive, but it's not free. * Potentially lots of network traffic for zero gain. But there are tons of ways to think about this sort of thing, and lots of different setups and requirements. My judgment calls and current constraints are likely different to what other people are dealing with. So, there are plenty of good arguments for any particular design. I'm just trying to be clear that while I'm not keen on the approach you suggest in this case, it's not because I think it's inherently bad - I've used it before. It's just that in this case I don't want the log data in 4D, if at all possible. I see that as pure cost with no gain. Hopefully, someone will find something silly in what I'm doing as the CALL WORKER idea is really ver nice. You have n processes doing their thing - running code, serving Web requests, handling windows, etc. Then they can send log data (events, behaviors, errors, etc.) over to a central log worker. Using CALL WORKER, this is quite fast. (4D can stack up tons of these little requests quite well.) From there, you've got one process handling all of the log data. A single external file for a particular log is perfect at this point. It's fast and super easy to use. The whole point of using a single process for this is to avoid contention on the file amongst processes. That can otherwise be a *huge* and totally unworthy bottleneck. So, one process for logging which has the file open in read-write all the time. So good. Such a standard design. Simple, clean, cheap, efficient...doesn't work in 4D. Unless there's a bug in my bug report ;-) ** 4D Internet Users Group (4D iNUG) FAQ: http://lists.4d.com/faqnug.html Archive: http://lists.4d.com/archives.html Options: http://lists.4d.com/mailman/options/4d_tech Unsub: mailto:4d_tech-unsubscr...@lists.4d.com **
Re: Making LOG EVENT thread-safe
On Mon, Sep 25, 2017 at 12:04 PM, Epperlein, Lutz (agendo) via 4D_Tech < 4d_tech@lists.4d.com> wrote: I asked because we use a little own implementation of writing log files. This is all before V16. We use semaphores to avoid race conditions while writing to the file. And one of our first projects after updating to V16 should be the rewriting of this code using workers. So I'm a bit disappointed to hear that. Yes, centralizing a log writer seems like the most obvious use of a worker, so I also was disappointed. But perhaps I'm wrong? It would not be the first time that I've done something stupid. Just in case I'm missing something obvious, I've posted a copy of the test database I've used to demonstrate the bug here: http://www.4dcompanion.com/LogWorker.zip It's a crude test database. What I was doing is compiling it and running LogWorker_CallABunch_Coop or LogWorker_Test_Preemptive. The test isn't to see if you can use CALL WORKER to execute code that passes a log entry to a worker. You can. The test isn't to see if you can then keep the log file open just in the worker to avoid contention on the file. You can. The test is to see what happens when you need to "roll" the log. Namely, close the current log, rename it and open a new log. This is where things go sideways. If I remember correctly, I've retested this as far as 16.2 and R3 but not R4. Perhaps a fresh set of eyes would help here. I do see that I tried putting in a 3 second delay when rolling the log in case that gave 4D and/or the OS a chance to catch up. Didn't help. So, if I'm doing something foolish and wrong *please* tell me. I hate being wrong, but I'd also rather know when I'm wrong. So, yeah, let me know. If there's another way to approach this that's reliable, also please say. If that way requires semaphores then, well, nothat kind of defeats the whole purpose. At that point, I'd use non-preemptive code with NTK IPC channels. That's been working flawlessly for years. One thing: Any solution has to be 100% reliable. For server-side code like this, a success rate of 99.9% isn't even close. When an error can bring the server down, it's a non-starter. Particularly an error that's super easy to generate. The truth is that I'd really like to use disk files for logs. They're simple, lightweight, cheap and robust. You can feed them into your aggregator of choice in a million different ways, they take up no room in the database, you can resume posting of entries after a crash, they're just a solid choice. Hence, their near-universal popularity. P.S. I know that using CALL WORKER inevitably means some log entries will be lost during restart. That's an acceptable cost in this particular case. Crashing/freezing the server is not. ** 4D Internet Users Group (4D iNUG) FAQ: http://lists.4d.com/faqnug.html Archive: http://lists.4d.com/archives.html Options: http://lists.4d.com/mailman/options/4d_tech Unsub: mailto:4d_tech-unsubscr...@lists.4d.com **