Re: [fpc-pascal] Dynamic arrays, yet another pitfall
On 02/17/2014 09:10 PM, Florian Klämpfl wrote: Am 16.02.2014 17:16, schrieb Michael Van Canneyt: 2) SetLength enforces unique ref. count. I'am against stating this in the docs. It is an implementation detail. Moreover this would ask for a long winding explanation what "unique ref count" exactly means for the user. The proper way to do a deep copy is using copy. Again an in depth explanation of "deep copy" is necessary in that docs. -Michael ___ fpc-pascal maillist - fpc-pascal@lists.freepascal.org http://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-pascal
Re: [fpc-pascal] Dynamic arrays, yet another pitfall
Am 17.02.2014 21:09, schrieb Florian Klämpfl: Am 16.02.2014 14:07, schrieb Jürgen Hestermann: When using unknown features of a programming language for the first time then the documentaion should tell all aspects in detail and describe the exact behaviour. No. Details might be implementation specific behaviour which might change. So you mean it can change in that SetLength may *not* create a copy some day? Or can we rely on that this will always happen? If the latter, why not document it (it's realy important to know because you cannot go without setlength). All details which are *not* object to change should be documented so we know how data structures like dynamic arrays behave. ___ fpc-pascal maillist - fpc-pascal@lists.freepascal.org http://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-pascal
Re: [fpc-pascal] Dynamic arrays, yet another pitfall
In our previous episode, Martin Frb said: > >> the array), does not mean there is copy on write. > > So basically, if for a given size x, setlength(x,1) always reallocates, then > > it is COW, otherwise not :-) That came out wrong wrong. Should've been for given x, setlength(p,x+1); p (after) is always not equal to p(before). > Not sure what you mean? I meant to say that if the implementation can increase the dyn array with 1 without reallocation, and it does nevertheless, it is COW not mere reallocation. But since when to reallocate is implementation dependent, that is a hard thing to prove. > > (and even if it does, one could still argue it is an implementation detail) > > > > But if you want to use the term, IMHO it would be: > "Dynamic arrays have no copy-on-write. But SetLength perform a > copy-on-write on dynamic arrays." ... depending on allocation, Setlength might reallocate and copy ... ___ fpc-pascal maillist - fpc-pascal@lists.freepascal.org http://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-pascal
Re: [fpc-pascal] Dynamic arrays, yet another pitfall
On 17/02/2014 20:10, Florian Klämpfl wrote: Am 16.02.2014 17:16, schrieb Michael Van Canneyt: 2) SetLength enforces unique ref. count. I'am against stating this in the docs. It is an implementation detail. The proper way to do a deep copy is using copy. IMHO, it must be documented, even if it is not meant to be used as a way to make a copy. Because it has the side effect, that started this thread. A:=b; a[1]:=1; // affects b Setlength(a,length(a)+1)); a[1]:=2; // does NOT affects b This is behaviour that affect every user app. If it is not documented, the "a:=b,a[1]:=1;" is no longer a documented use case of arrays. And if not documented it is not an allowed usage either? -- Btw: How smart is Copy() a:=nil; Setlength(2); a:=copy(a,0,length(a)); Since the source is no longer going to be used, it could be re-used instead of being copied to new memory. ___ fpc-pascal maillist - fpc-pascal@lists.freepascal.org http://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-pascal
Re: [fpc-pascal] Dynamic arrays, yet another pitfall
Am 16.02.2014 17:16, schrieb Michael Van Canneyt: > 2) SetLength enforces unique ref. count. I'am against stating this in the docs. It is an implementation detail. The proper way to do a deep copy is using copy. ___ fpc-pascal maillist - fpc-pascal@lists.freepascal.org http://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-pascal
Re: [fpc-pascal] Dynamic arrays, yet another pitfall
Am 16.02.2014 14:07, schrieb Jürgen Hestermann: > When using unknown features of a programming language for the first time > then the documentaion should tell all aspects in detail and describe the > exact behaviour. No. Details might be implementation specific behaviour which might change. ___ fpc-pascal maillist - fpc-pascal@lists.freepascal.org http://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-pascal
Re: [fpc-pascal] Dynamic arrays, yet another pitfall
On 17/02/2014 19:50, Jonas Maebe wrote: On 17 Feb 2014, at 20:41, Martin Frb wrote: "Dynamic arrays have no copy-on-write. But SetLength perform a copy-on-write on dynamic arrays." I am not sure, if that makes to much sense either It doesn't, because even if you don't change the length it makes a unique copy. Right of course, it is an unconditional copy always (if refcount > 1). (same as unique instance) But even **assuming** the following: - setlength (with no length) to be considered a write - copy only if length changes That would still not mean that dyn array would have COW. Because this would still be only one function acting on this type, where as other write operations on the type clearly show that: The data type dyn array has no COW ___ fpc-pascal maillist - fpc-pascal@lists.freepascal.org http://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-pascal
Re: [fpc-pascal] Dynamic arrays, yet another pitfall
On 17 Feb 2014, at 20:41, Martin Frb wrote: > "Dynamic arrays have no copy-on-write. But SetLength perform a copy-on-write > on dynamic arrays." > > I am not sure, if that makes to much sense either It doesn't, because even if you don't change the length it makes a unique copy. Jonas ___ fpc-pascal maillist - fpc-pascal@lists.freepascal.org http://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-pascal
Re: [fpc-pascal] Dynamic arrays, yet another pitfall
On 17/02/2014 19:34, Marco van de Voort wrote: In our previous episode, Martin Frb said: Just because there is a function, that includes copying (when modifying the array), does not mean there is copy on write. So basically, if for a given size x, setlength(x,1) always reallocates, then it is COW, otherwise not :-) Not sure what you mean? (and even if it does, one could still argue it is an implementation detail) But if you want to use the term, IMHO it would be: "Dynamic arrays have no copy-on-write. But SetLength perform a copy-on-write on dynamic arrays." I am not sure, if that makes to much sense either. copy-on-write should apply to the data type, not to the/a function that acts on data of that type. And the data type "dyn array" does not have copy-on-write. If it had, the *ALL* [1] write operation would have to trigger the COW. [1] this excludes access via pointer, that is designed to bypass the type specific behaviour. ___ fpc-pascal maillist - fpc-pascal@lists.freepascal.org http://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-pascal
Re: [fpc-pascal] Dynamic arrays, yet another pitfall
In our previous episode, Martin Frb said: > Just because there is a function, that includes copying (when modifying > the array), does not mean there is copy on write. So basically, if for a given size x, setlength(x,1) always reallocates, then it is COW, otherwise not :-) (and even if it does, one could still argue it is an implementation detail) ___ fpc-pascal maillist - fpc-pascal@lists.freepascal.org http://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-pascal
Re: [fpc-pascal] Dynamic arrays, yet another pitfall
On 17/02/2014 18:19, Jürgen Hestermann wrote: Am 2014-02-17 18:49, schrieb Martin Frb: Then why does this discussion continue, now that it is documented? Does it matter what other people consider a write and what not? (Unless that part made it into the docs, but there was no mention it did)? Of course it would not matter if it wasn't part of the documentation. But here http://www.freepascal.org/docs-html/ref/refsu18.html it still says: "there is no copy-on-write". This sentence is wrong IMO. No this is right. As I said in my other mail: In the same way I can write functions for an object that performs a (modifying) operation on the object, and copies the object before. Yet if I wrote such code, no one would say, that the object I use, has COW. Just because there is a function, that includes copying (when modifying the array), does not mean there is copy on write. Otherwise all referenced data, would always be copy on write, because for any referenced data, such a function can be written. ___ fpc-pascal maillist - fpc-pascal@lists.freepascal.org http://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-pascal
Re: [fpc-pascal] Dynamic arrays, yet another pitfall
On 17 Feb 2014, at 19:19, Jürgen Hestermann wrote: > And I have not seen what change was made to the documentation. svn co http://svn.freepascal.org/svn/fpcdocs/trunk It's the same as when something is fixed in FPC: this does not change anything to the latest release, which is what's available on the website. > At the above link and also at http://wiki.freepascal.org/Dynamic_array still > shows me the old entry I still find no words about setlength making a copy. The wiki is a collaborative effort that is unrelated to the official documentation. You can even update it yourself, if you want. Jonas ___ fpc-pascal maillist - fpc-pascal@lists.freepascal.org http://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-pascal
Re: [fpc-pascal] Dynamic arrays, yet another pitfall
Am 2014-02-17 18:49, schrieb Martin Frb: Then why does this discussion continue, now that it is documented? Does it matter what other people consider a write and what not? (Unless that part made it into the docs, but there was no mention it did)? Of course it would not matter if it wasn't part of the documentation. But here http://www.freepascal.org/docs-html/ref/refsu18.html it still says: "there is no copy-on-write". This sentence is wrong IMO. And many people here were trying to convince me that this is information enough because everybody of course knows that SetLength is not a write and therefore a copy is made. And I have not seen what change was made to the documentation. At the above link and also at http://wiki.freepascal.org/Dynamic_array still shows me the old entry I still find no words about setlength making a copy. ___ fpc-pascal maillist - fpc-pascal@lists.freepascal.org http://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-pascal
Re: [fpc-pascal] Dynamic arrays, yet another pitfall
On 17/02/2014 17:28, Jürgen Hestermann wrote: Am 2014-02-17 18:01, schrieb Martin Frb: > On 17/02/2014 16:40, Jürgen Hestermann wrote: > It appears, that the copy-on-setlength is intended. So in that the documentation is right. It is intended but is was not mentioned in the documentation. Therefore this whole thread exists at all. Then why does this discussion continue, now that it is documented? Does it matter what other people consider a write and what not? (Unless that part made it into the docs, but there was no mention it did)? ___ fpc-pascal maillist - fpc-pascal@lists.freepascal.org http://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-pascal
Re: [fpc-pascal] Dynamic arrays, yet another pitfall
Am 2014-02-17 18:01, schrieb Martin Frb: > On 17/02/2014 16:40, Jürgen Hestermann wrote: > It appears, that the copy-on-setlength is intended. So in that the documentation is right. It is intended but is was not mentioned in the documentation. Therefore this whole thread exists at all. > As for this being a good or bad idea, is a different question. (There are pro and con apparently). But even if we concluded it bad, changing it now would be by far worse. > Besides I believe it is not possible > setlength(a,10) > b := a; > setlength(a,1200) > The data pointed to by both a and b (before the final setlength) has a refcount of 2. But it has no info where a and b are. So in SetLength(a, 1200) it is unknown where the other reference (b) is. And that means b can not follow this change. You can only say this because you know how dynamic arrays are implemented. But someone who only has to rely on the documentation does not know where both pointers point to (he not even knows they are pointers). So what should he assume? There could be an intermediate data structure that holds the pointer to the data array in the same way as it holds the reference counter. So 2 pointers pointing to the same data structure and only this structure is changed by SetLength. Then there would be no copy on write with SetLength. But how can we know? ___ fpc-pascal maillist - fpc-pascal@lists.freepascal.org http://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-pascal
Re: [fpc-pascal] Dynamic arrays, yet another pitfall
On 17/02/2014 16:40, Jürgen Hestermann wrote: But changing the size it is a modification! When an array can be resized then the resizing means that I have to write to it (in contrast to reading, where no data is modified). I am astonished that everybody here seems not to see this. Maybe because they already got so accustomed to the behaviour that now far-fetched explanations are fabricated to make the documentation look right in some sense only because it *has* to be right (like a dogma). In the question really, if the documentation is right? The documentation (if right) expresses the expected behaviour that the person who designed the function had intended. It appears, that the copy-on-setlength is intended. So in that the documentation is right. As for this being a good or bad idea, is a different question. (There are pro and con apparently). But even if we concluded it bad, changing it now would be by far worse. Besides I believe it is not possible setlength(a,10) b := a; setlength(a,1200) The data pointed to by both a and b (before the final setlength) has a refcount of 2. But it has no info where a and b are. So in SetLength(a, 1200) it is unknown where the other reference (b) is. And that means b can not follow this change. Other theoretical solutions would be: - do not allow setlength, if refcount > 1 - do another level of pointer/referencing - do a full COW None of those appeal to me. ___ fpc-pascal maillist - fpc-pascal@lists.freepascal.org http://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-pascal
Re: [fpc-pascal] Dynamic arrays, yet another pitfall
On 17/02/2014 16:40, Jürgen Hestermann wrote: Am 2014-02-17 10:38, schrieb Lukasz Sokol: > To 'write' usually means : changing DATA element(s). That's it. What else is it when adding or removing elements? The array is modified. That can only mean it was written to it. > SetLength is /not/ a write in this sense : because you're resizing the paper you're > going to write on (e.g. adding another page or cutting it to size) But changing the size it is a modification! It does not matter. Who says that it is an effect of the (implicit) write that triggers the copy? For strings there is UniqueInstance(), and even though it does not write to the string, it makes a copy, if needed. SetLength is a function too. SetLength includes making the array unique as part of its function description. So this "making unique" does not happen because of COW (which does not apply), but because of the (now) documented effects of a function used on the array. In the same way I can write functions for an object that performs a (modifying) operation on the object, and copies the object before. Yet if I wrote such code, no one would say, that the object I use, has COW. ___ fpc-pascal maillist - fpc-pascal@lists.freepascal.org http://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-pascal
Re: [fpc-pascal] Dynamic arrays, yet another pitfall
Am 2014-02-17 10:38, schrieb Lukasz Sokol: > To 'write' usually means : changing DATA element(s). That's it. What else is it when adding or removing elements? The array is modified. That can only mean it was written to it. > SetLength is /not/ a write in this sense : because you're resizing the paper you're > going to write on (e.g. adding another page or cutting it to size) But changing the size it is a modification! When an array can be resized then the resizing means that I have to write to it (in contrast to reading, where no data is modified). I am astonished that everybody here seems not to see this. Maybe because they already got so accustomed to the behaviour that now far-fetched explanations are fabricated to make the documentation look right in some sense only because it *has* to be right (like a dogma). Just ask a pupil who started programming recently about what he thinks in this case. Whether he would consider adding or removing elements to an array to be a write operation or not. I suppose you would be astonished about the answers. > As far I understood, SetLength is changing the /description/ of the array (meta-data?) but not the data itself : > in this sense, it is not a 'write data' operation. Why is a change not a write? I don't get it. I can only change a variable if I can write to it. > It does 'write' as in: change parameters - of the array DESCRIPTION area, but that area is not ARRAY DATA; That's not true. Before there where 10 elements, now there are only 7. So 3 DATA elements where removed! Everybody always says that these managed types should be handled like black boxes. But suddenly it is quite important to know how exactly these types are handled by the compiler. But when this is so important then it needs to be documented! > and enlarging/shrinking the memory allocation - but that does not change the content of the already > written-to array elements: it may add some blank ones or remove some written-to ones, but the rest > remains UNCHANGED, i.e. they (the just-added or remaining-after-removal ones) are not being written-to > by doing SetLength.) I think you are kidding if you try to make this sound like a clear and fundamental explanation. With this logic changing only 1 element is not a write either because "the rest remains UNCHANGED". ___ fpc-pascal maillist - fpc-pascal@lists.freepascal.org http://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-pascal
Re: [fpc-pascal] Dynamic arrays, yet another pitfall
On 17/02/14 09:51, Michael Van Canneyt wrote: > > > On Mon, 17 Feb 2014, Lukasz Sokol wrote: > >> On 17/02/14 06:02, Jürgen Hestermann wrote: >>> >>> Am 2014-02-16 17:16, schrieb Michael Van Canneyt: It does exactly that, it says: 1) No COW >>> >>> As said already: SetLength *is* a write! >>> >>> >> >> I'm not of compiler background - but I think you're confusing things here. >> >> To 'write' usually means : changing DATA element(s). That's it. You have a >> sheet of paper of >> known size and you write on it/erase/change the text. But the size of the >> array remains the same. >> >> SetLength is /not/ a write in this sense : because you're resizing the paper >> you're >> going to write on (e.g. adding another page or cutting it to size) >> >> As far I understood, SetLength is changing the /description/ of the array >> (meta-data?) but not the data itself : >> in this sense, it is not a 'write data' operation. >> >> It does 'write' as in: change parameters - of the array DESCRIPTION area, >> but that area is not ARRAY DATA; >> and enlarging/shrinking the memory allocation - but that does not change the >> content of the already >> written-to array elements: it may add some blank ones or remove some >> written-to ones, but the rest >> remains UNCHANGED, i.e. they (the just-added or remaining-after-removal >> ones) are not being written-to >> by doing SetLength.) > > That is the long version of what I was trying to say. Thank you :) > > Michael. > > *grin* *wink* so many redundant words ... some need them ;) Can you add the long(er) description to the help text ? :) -L. ___ fpc-pascal maillist - fpc-pascal@lists.freepascal.org http://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-pascal
Re: [fpc-pascal] Dynamic arrays, yet another pitfall
On Mon, 17 Feb 2014, Lukasz Sokol wrote: On 17/02/14 06:02, Jürgen Hestermann wrote: Am 2014-02-16 17:16, schrieb Michael Van Canneyt: It does exactly that, it says: 1) No COW As said already: SetLength *is* a write! I'm not of compiler background - but I think you're confusing things here. To 'write' usually means : changing DATA element(s). That's it. You have a sheet of paper of known size and you write on it/erase/change the text. But the size of the array remains the same. SetLength is /not/ a write in this sense : because you're resizing the paper you're going to write on (e.g. adding another page or cutting it to size) As far I understood, SetLength is changing the /description/ of the array (meta-data?) but not the data itself : in this sense, it is not a 'write data' operation. It does 'write' as in: change parameters - of the array DESCRIPTION area, but that area is not ARRAY DATA; and enlarging/shrinking the memory allocation - but that does not change the content of the already written-to array elements: it may add some blank ones or remove some written-to ones, but the rest remains UNCHANGED, i.e. they (the just-added or remaining-after-removal ones) are not being written-to by doing SetLength.) That is the long version of what I was trying to say. Thank you :) Michael.___ fpc-pascal maillist - fpc-pascal@lists.freepascal.org http://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-pascal
Re: [fpc-pascal] Dynamic arrays, yet another pitfall
On 17/02/14 06:02, Jürgen Hestermann wrote: > > Am 2014-02-16 17:16, schrieb Michael Van Canneyt: >> It does exactly that, it says: >> 1) No COW > > As said already: SetLength *is* a write! > > I'm not of compiler background - but I think you're confusing things here. To 'write' usually means : changing DATA element(s). That's it. You have a sheet of paper of known size and you write on it/erase/change the text. But the size of the array remains the same. SetLength is /not/ a write in this sense : because you're resizing the paper you're going to write on (e.g. adding another page or cutting it to size) As far I understood, SetLength is changing the /description/ of the array (meta-data?) but not the data itself : in this sense, it is not a 'write data' operation. It does 'write' as in: change parameters - of the array DESCRIPTION area, but that area is not ARRAY DATA; and enlarging/shrinking the memory allocation - but that does not change the content of the already written-to array elements: it may add some blank ones or remove some written-to ones, but the rest remains UNCHANGED, i.e. they (the just-added or remaining-after-removal ones) are not being written-to by doing SetLength.) That being said, maybe a mention of what it means to the user of the compiler, could actually be beneficial to overall understanding... like " SetLength: * Enforces unique refcount (refcount := 1). * If refcount was more than 1 (i.e. 2 or more pointers to the array existed before calling SetLength) then it copies array data to new data area, trims or enlarges copied data accordingly and returns new pointer to the resized array. * otherwise (if refcount was 1) just trims/enlarges the array data area accordingly. " Hope this helps, -L. ___ fpc-pascal maillist - fpc-pascal@lists.freepascal.org http://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-pascal
Re: [fpc-pascal] Dynamic arrays, yet another pitfall
Am 2014-02-16 17:16, schrieb Michael Van Canneyt: It does exactly that, it says: 1) No COW As said already: SetLength *is* a write! ___ fpc-pascal maillist - fpc-pascal@lists.freepascal.org http://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-pascal
Re: [fpc-pascal] Dynamic arrays, yet another pitfall
On Sun, 16 Feb 2014, Jürgen Hestermann wrote: Am 2014-02-16 13:47, schrieb Florian Klämpfl: setlength does not behave freaky but its behaviour is well designed. The reason why setlength does a deep copy is simple: multithreading. If setlength had no deep copy semantics, it would need locking of the whole array data, not only locked access to the ref. counter. There may be good reasons for doing it the way it has been done. But for a programmer (who has not written the compiler) this behaviour is totaly unexpected. When using unknown features of a programming language for the first time then the documentaion should tell all aspects in detail and describe the exact behaviour. It does exactly that, it says: 1) No COW and I added: 2) SetLength enforces unique ref. count. Michael. ___ fpc-pascal maillist - fpc-pascal@lists.freepascal.org http://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-pascal ___ fpc-pascal maillist - fpc-pascal@lists.freepascal.org http://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-pascal
Re: [fpc-pascal] Dynamic arrays, yet another pitfall
Am 2014-02-16 13:47, schrieb Florian Klämpfl: > setlength does not behave freaky but its behaviour is well designed. The > reason why setlength does a deep copy is simple: multithreading. If > setlength had no deep copy semantics, it would need locking of the whole > array data, not only locked access to the ref. counter. There may be good reasons for doing it the way it has been done. But for a programmer (who has not written the compiler) this behaviour is totaly unexpected. When using unknown features of a programming language for the first time then the documentaion should tell all aspects in detail and describe the exact behaviour. ___ fpc-pascal maillist - fpc-pascal@lists.freepascal.org http://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-pascal
Re: [fpc-pascal] Dynamic arrays, yet another pitfall
Am 2014-02-16 13:20, schrieb Michael Van Canneyt: > Depends on how you define 'write to the array'. > "Writing to the array" = "Changing the value of one of the elements in the array." > In this sense, setlength does not write to the array, which would mean changing the value of > one or more of the elements in the array. It does not do that. It resizes the array and > the result is by definition a different array. But what else than writing is it to add or remove elements? It's definitely not reading. Changing the length of an ansistring is writing too IMO. > But taking the above definition, with "element" = "character", you see that ansistrings do have copy-on-write. Nobody doubt that. But dynamic arrays claim to *not* have copy-on-write although SetLength *does* a copy-on-write. When comparing ansistrings to dynamic arrays: Setlength on ansistrings does copy-on-write but the documentation says that copy-on-write does not exist for dynamic arrays. Still Setlength on dynamic arrays does a copy-on-write in the same way as it does for ansistrings. > It's largely a question of semantics, I suppose. Well, IMO "writing" is doing changes to the stored data. In opposite to reading it writes bytes to memory. Reading the variable before and after a SetLength show different results (length() reports different values). If Setlength should *not* be considered "writing" then the documenation needs to make this very clear. It should not leave any doubts nor allow different interpretation of the behaviour. Especially when such illogical and schizophrenic behaviour exists. ___ fpc-pascal maillist - fpc-pascal@lists.freepascal.org http://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-pascal
Re: [fpc-pascal] Dynamic arrays, yet another pitfall
Am 15.02.2014 19:41, schrieb Michael Van Canneyt: > > That setlength behaves rather freakish for dynamic arrays, does not > mean we have copy-on-write. setlength does not behave freaky but its behaviour is well designed. The reason why setlength does a deep copy is simple: multithreading. If setlength had no deep copy semantics, it would need locking of the whole array data, not only locked access to the ref. counter. ___ fpc-pascal maillist - fpc-pascal@lists.freepascal.org http://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-pascal
Re: [fpc-pascal] Dynamic arrays, yet another pitfall
On Sun, 16 Feb 2014, Jürgen Hestermann wrote: Am 2014-02-15 19:41, schrieb Michael Van Canneyt: That setlength behaves rather freakish for dynamic arrays, does not mean we have copy-on-write. Well, setlength *is* a write. What else is it? Doesn't it write to the array? Depends on how you define 'write to the array'. "Writing to the array" = "Changing the value of one of the elements in the array." In this sense, setlength does not write to the array, which would mean changing the value of one or more of the elements in the array. It does not do that. It resizes the array and the result is by definition a different array. But taking the above definition, with "element" = "character", you see that ansistrings do have copy-on-write. It's largely a question of semantics, I suppose. Michael.___ fpc-pascal maillist - fpc-pascal@lists.freepascal.org http://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-pascal
Re: [fpc-pascal] Dynamic arrays, yet another pitfall
Am 2014-02-15 19:41, schrieb Michael Van Canneyt: That setlength behaves rather freakish for dynamic arrays, does not mean we have copy-on-write. Well, setlength *is* a write. What else is it? Doesn't it write to the array? ___ fpc-pascal maillist - fpc-pascal@lists.freepascal.org http://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-pascal
Re: [fpc-pascal] Dynamic arrays, yet another pitfall
On Sat, 15 Feb 2014, Jürgen Hestermann wrote: Am 2014-02-10 09:13, schrieb Michael Van Canneyt: No. There is no COW, only ref. counting. SetLength just forces an unique instance of the array if needed. The documentation explicitly mentions that: 'Dynamic arrays are reference counted: assignment of one dynamic array-type variable to another will let both variables point to the same array. Contrary to ansistrings, an assignment to an element of one array will be reflected in the other: there is no copy-on-write.' But what do these sentences say? They say that there is *no* copy-on-write. Correct. So why does it suddenly do copy-on-write when using SetLength? Copy-On-Write means that if you write to *an element* of the array, the array is duplicated if it has a ref count>0. As with strings. Since this does not happen for dynamic arrays, there is no copy on write. That setlength behaves rather freakish for dynamic arrays, does not mean we have copy-on-write. Michael.___ fpc-pascal maillist - fpc-pascal@lists.freepascal.org http://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-pascal
Re: [fpc-pascal] Dynamic arrays, yet another pitfall
Am 2014-02-10 09:13, schrieb Michael Van Canneyt: >> No. There is no COW, only ref. counting. SetLength just forces an unique >> instance of the array if needed. > The documentation explicitly mentions that: > 'Dynamic arrays are reference counted: assignment of one dynamic array-type > variable to another will let both variables point to the same array. > Contrary to ansistrings, an assignment to an element of one array will > be reflected in the other: there is no copy-on-write.' But what do these sentences say? They say that there is *no* copy-on-write. So why does it suddenly do copy-on-write when using SetLength? SetLength is a write to the array. It adds (or removes) elements. So why do I get a copy? The documentation says that I will *not* get a copy (no COW). So the documentation is wrong (or at least totaly misleading/incomplete). > But I have added a paragraph about the "reset ref. count" behaviour of setlength. What has a reference counter to do with the behaviour I see? If SetLength would just simply change the number of elements it would not change a reference counter. So nothing else happens, just elements are added or removed. The same as when I change an element. At least that's what the documentation says (to me). ___ fpc-pascal maillist - fpc-pascal@lists.freepascal.org http://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-pascal
Re: [fpc-pascal] Dynamic arrays, yet another pitfall
Am 2014-02-09 19:29, schrieb Florian Klämpfl: > Am 09.02.2014 18:41, schrieb Jürgen Hestermann: >> So it seems there is a copy-on-write *but* only when using SetLength. > No. There is no COW, only ref. counting. SetLength just forces an unique > instance of the array if needed. But this *is* Copy-On-Write! When I set the length it is not different to changing elements. In both cases I write to the array (modify it). But suddenly I have a duplicate. I would not expect such a behaviour as I don't get a dublicate when changing elements either. >> What a very consistent design! > There is a very good reason for this design: speed. COW would mean that > at every write to a dyn. array element, ref. count must be checked. This > would render dyn. arrays useless for a lot of applications due to poor > speed. So dyn. arrays with COW simply make no sense. Yes. Therefore I was expecting that *never* a copy is made. While Ansistrings *do* have COW dynamic arrays do *not* have it. If that would be true then variables pointing to the same dynamic array should do this forever (unless the programmer assigns a new array with := ). Setlength should *not* be considered different to modifying elements. For what reason? Both are writes to the array. As this seems not to be true it should be mentinoned in the documentation with big exclamation marks!! Otherwise *everybody* who is using dynamic arrays for the first time will assume Setlength beeing treated the same as element changes. Why should someone consider these things as different? This is a source of errors and a huge confusion as discussed here too: http://www.delphitools.info/2011/06/15/poll-dynamic-arrays-as-reference-or-value-type ___ fpc-pascal maillist - fpc-pascal@lists.freepascal.org http://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-pascal
Re: [fpc-pascal] Dynamic arrays, yet another pitfall
On Sun, 9 Feb 2014, Florian Klämpfl wrote: Am 09.02.2014 18:41, schrieb Jürgen Hestermann: So it seems there is a copy-on-write *but* only when using SetLength. No. There is no COW, only ref. counting. SetLength just forces an unique instance of the array if needed. The documentation explicitly mentions that: 'Dynamic arrays are reference counted: assignment of one dynamic array-type variable to another will let both variables point to the same array. Contrary to ansistrings, an assignment to an element of one array will be reflected in the other: there is no copy-on-write.' But I have added a paragraph about the "reset ref. count" behaviour of setlength. Michael.___ fpc-pascal maillist - fpc-pascal@lists.freepascal.org http://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-pascal
Re: [fpc-pascal] Dynamic arrays, yet another pitfall
On 09.02.2014 19:10, patspiper wrote: On 09/02/14 18:47, Sven Barth wrote: On 09.02.2014 16:34, Flávio Etrusco wrote: In other words: dynamic arrays are like AnsiStrings without the copy-on-write semantics. I'd certainly wish Borland copied the COW semantics :-/ Dynamic arrays have full COW semantics. It seems not: SetLength(A,10); A[0]:=33; B:=A; A[0]:=31; b[0]:=9; WriteLn(a[0], b[0]); // prints 9 and 9 (fpc 2.6.3) Ehm yes... seems that I was mistaken a bit by the dynamic array implementation. *blush* Regards, Sven ___ fpc-pascal maillist - fpc-pascal@lists.freepascal.org http://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-pascal
Re: [fpc-pascal] Dynamic arrays, yet another pitfall
Am 09.02.2014 18:41, schrieb Jürgen Hestermann: > So it seems there is a copy-on-write *but* only when using SetLength. No. There is no COW, only ref. counting. SetLength just forces an unique instance of the array if needed. > What a very consistent design! There is a very good reason for this design: speed. COW would mean that at every write to a dyn. array element, ref. count must be checked. This would render dyn. arrays useless for a lot of applications due to poor speed. So dyn. arrays with COW simply make no sense. ___ fpc-pascal maillist - fpc-pascal@lists.freepascal.org http://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-pascal
Re: [fpc-pascal] Dynamic arrays, yet another pitfall
On 09/02/14 18:47, Sven Barth wrote: On 09.02.2014 16:34, Flávio Etrusco wrote: In other words: dynamic arrays are like AnsiStrings without the copy-on-write semantics. I'd certainly wish Borland copied the COW semantics :-/ Dynamic arrays have full COW semantics. It seems not: SetLength(A,10); A[0]:=33; B:=A; A[0]:=31; b[0]:=9; WriteLn(a[0], b[0]); // prints 9 and 9 (fpc 2.6.3) If Jürgen would have provided a full compilable example we could check whether he has a bug in his own code or there is a bug in the compiler as certainly the result of his first code snipped must be length 10 for array "A" and length 20 for array "B". His second example is as expected and as designed. True on both counts. Regards, Stephano ___ fpc-pascal maillist - fpc-pascal@lists.freepascal.org http://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-pascal
Re: [fpc-pascal] Dynamic arrays, yet another pitfall
Am 2014-02-09 17:47, schrieb Sven Barth: On 09.02.2014 16:34, Flávio Etrusco wrote: In other words: dynamic arrays are like AnsiStrings without the copy-on-write semantics. I'd certainly wish Borland copied the COW semantics :-/ Dynamic arrays have full COW semantics. Now I am fully confused. It's hard to find out how things work in Free Pascal given the poor documentation. This sentence from http://www.freepascal.org/docs-html/ref/refsu18.html is *very* misleading: "Dynamic arrays are reference counted: assignment of one dynamic array-type variable to another will let both variables point to the same array. Contrary to ansistrings, an assignment to an element of one array will be reflected in the other: there is no copy-on-write." Reference counted? Point to the same array? No copy-on-write? It seems you cannot believe anything from the documentation. You must question every statement. *Everything* can only be determined by writing test programs. I now found a similar posting 5 years ago regarding the same issue: http://lists.freepascal.org/lists/fpc-pascal/2009-February/020198.html Why ist this not part of the documentation? So it seems there is a copy-on-write *but* only when using SetLength. What a very consistent design! ___ fpc-pascal maillist - fpc-pascal@lists.freepascal.org http://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-pascal
Re: [fpc-pascal] Dynamic arrays, yet another pitfall
Am 2014-02-09 16:34, schrieb Flávio Etrusco: > In other words: dynamic arrays are like AnsiStrings without the > copy-on-write semantics. I'd certainly wish Borland copied the COW > semantics :-/ Yes. Therefore the issue that I described for dynamic arrays cannot occur for ansistrings. But COW can also be a drawback. Imagine an ansistring within a record of a heavily pointer-connected deeply nested data structure and I want to write a function that gives me back a reference to this string: - function FindMyStringToModify : AnsiString; ... var A : AnsiString; A := FindMyStringToModify; A := 'New String'; - then the change to A will not affect the ansistring in the nested data structure (because of COW). But this is possible for dynamic arrays *if* they are not empty. ___ fpc-pascal maillist - fpc-pascal@lists.freepascal.org http://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-pascal
Re: [fpc-pascal] Dynamic arrays, yet another pitfall
On 09.02.2014 16:34, Flávio Etrusco wrote: In other words: dynamic arrays are like AnsiStrings without the copy-on-write semantics. I'd certainly wish Borland copied the COW semantics :-/ Dynamic arrays have full COW semantics. If Jürgen would have provided a full compilable example we could check whether he has a bug in his own code or there is a bug in the compiler as certainly the result of his first code snipped must be length 10 for array "A" and length 20 for array "B". His second example is as expected and as designed. Regards, Sven ___ fpc-pascal maillist - fpc-pascal@lists.freepascal.org http://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-pascal
Re: [fpc-pascal] Dynamic arrays, yet another pitfall
On 09.02.2014 15:34, Jürgen Hestermann wrote: With the following declaration and code: --- var A,B: array of integer; ... SetLength(A,10); B := A; SetLength(B,20); --- both variables A and B point to the same array with 20 Elements. Changing A changes B and vice versa. And again you did not provide a full compilable example. The following code === code begin === program tdynarrtest; var a, b: array of LongInt; begin SetLength(a, 10); b := a; SetLength(b, 20); Writeln(Length(a)); Writeln(Length(b)); end. === code end === results in === output begin === 10 20 === output end === So exactly what I would expect from the COW mechanism. Regards, Sven ___ fpc-pascal maillist - fpc-pascal@lists.freepascal.org http://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-pascal
Re: [fpc-pascal] Dynamic arrays, yet another pitfall
On Sun, Feb 9, 2014 at 12:34 PM, Jürgen Hestermann wrote: > (...) > With the following declaration and code: > > --- > var A,B: array of integer; > ... > SetLength(A,10); > B := A; > SetLength(B,20); > --- > > both variables A and B point to the same array with 20 Elements. > Changing A changes B and vice versa. > But a slight modification > > --- > SetLength(A,0); > B := A; > SetLength(B,20); > --- > > makes both variables A and B totaly decoupled! Although B is still assigned > to be the same as A each variable is now a separate array with individual > lengths and elements. Variable A has the length 0 and variable B is of > length 20. Changing the length for one of them does no longer change the > length of the other. If someone thinks about dynamic arrays as black boxes > without the need to know the details because they are handled in the > background then he will certainly be baffled by this. > (...) In other words: dynamic arrays are like AnsiStrings without the copy-on-write semantics. I'd certainly wish Borland copied the COW semantics :-/ -Flávio ___ fpc-pascal maillist - fpc-pascal@lists.freepascal.org http://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-pascal
[fpc-pascal] Dynamic arrays, yet another pitfall
There is another possible pitfall with dynamic arrays I trapped into which I would like to share. Those who handle dynamic arrays in a virtuoso manner daily may skip this posting of course. But I think for those who never used them before it may be of use. Maybe it can be added to the Wiki (although again I had to make assumptions which may be wrong): With the following declaration and code: --- var A,B: array of integer; ... SetLength(A,10); B := A; SetLength(B,20); --- both variables A and B point to the same array with 20 Elements. Changing A changes B and vice versa. But a slight modification --- SetLength(A,0); B := A; SetLength(B,20); --- makes both variables A and B totaly decoupled! Although B is still assigned to be the same as A each variable is now a separate array with individual lengths and elements. Variable A has the length 0 and variable B is of length 20. Changing the length for one of them does no longer change the length of the other. If someone thinks about dynamic arrays as black boxes without the need to know the details because they are handled in the background then he will certainly be baffled by this. Why is this so? Actually the variables A and B are just (4 byte) pointers to a structure which holds further information of the array (reference counter, length, pointer to the elements...). Setting the length of such a dynamic array to zero means that all data it pointed to will be freed and the pointer is set to NIL. Assigning this NIL-value to another variable of the same type sets this one to NIL too (after removing any potentially existing data of it). Changes for each of these variables no longer affect the other one because they do not point to the same structure anymore. Each has its own structure when adding elements with SetLength. This is especially a pitfall when the dynamic array is part of a complex structure and the reference to it should be given back by function or procedure parameters. It will work as long as the array has elements but fail when all elements have been (temporarily) removed! Under the hood dynamic arrays are similar to pointers to records: --- type MyType = record end; var A,B : ^MyType; --- After --- new(A); B := A; --- B points to the same structure as A and changing A^ changed B^ too. But after --- dispose(A); A := nil; B := A; --- A and B are both nil and --- new(A); new(B); --- would create two different structures with each variable pointing to one of them. The same happens for dynamic arrays. Each length-command changes the structure where the pointer points to but when set to length zero it does not point to anything anymore (is NIL). Afterwards both variables are independend from another. I think this is an important information about dynamic arrays which I was missing from the documentation. Without this knowledge it can be hard to find subtle errors. ___ fpc-pascal maillist - fpc-pascal@lists.freepascal.org http://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-pascal
