Hi Holger, Very cool analysis. Good to see people spend quality time on reviewing internals like this.
I agree with Bastien's remarks though. And I want to add this (as coder of the system): The 'treestore' is for Outliner only. It saves the state as the UI shows outliner data. It's not saving anything related to the scene description or relationships. It's always been a weak design issue here. I really want UIs to be able to save state, but the way we do it could probably be done much better. -Ton- -------------------------------------------------------- Ton Roosendaal - t...@blender.org - www.blender.org Chairman Blender Foundation, Director Blender Institute Entrepotdok 57A, 1018 AD, Amsterdam, the Netherlands > On 18 Nov 2016, at 17:08, Bastien Montagne <montagn...@wanadoo.fr> wrote: > > Well… I’d say your 'nice trick' to read .blend file was actually already > wrong, outliner tree is not the only guy needing special pointer magic > during writefile time, other areas (aware at least about some mesh data) > need it too, so you could still have overlapping 'virtual memory areas' > across whole file. > > Further more, there is absolutely nothing saying .blend file should > ensure a virtual non-overlapping memory map, we are only using pointers > because it is (in most case) the easiest and cheapest way to get uuids > for our data blocks, that’s all. > > Never a good idea to rely on an implementation-specific behavior in file > format area, only rely on written specs (though .blend format is not > that well documented either). > > My two cents anyway, > Cheers, > Bastien > > Le 18/11/2016 à 16:55, homac a écrit : >> Hello blender devs! >> >> >> I have noticed a change in blender file format, which has some >> consequences, which I would like to discuss. Because the topic is a bit >> difficult to explain, I wrote this email. I'm also sitting in >> #blendercoders. >> >> This change has no consequences for blender now but it may have for >> future versions of blender or thirdparty applications using .blend files. >> >> In terms of LoC, this change is tiny, and therefore I thought there is a >> realistic chance to convince you to change it again or accept a patch, I >> could write. >> >> >> The Change >> ========== >> Each block in a .blend file contains a copy of a unique area from heap >> memory. The start address stored in its block header is used to identify >> the block (and thereby the memory area) referenced by a pointer. >> >> Thus, a .blend file represented parts of the heap from the virtual >> address space of a previous run of Blender. This implicitly meant, that >> memory areas in the file could not overlap each other (they where disjunct). >> >> Some data structures such as trees or node networks, which have elements >> spread over several different areas in the heap, need special care: >> Traditionally, those structures where for example copied (packed) into >> one continuous memory area (e.g. an array) which is temporarily >> allocated on the heap before written to the block. To prevent malloc >> from assigning the same address to another temporarily allocated buffer, >> the buffer is deallocated after the entire file was written. That way, >> the fundamental assumption, that blocks are disjunct from each other, >> still holds. >> >> Actually, Blender just needs the start address of a block to be unique, >> because pointers, which have to be resolved while reading a file, always >> reference start addresses only. Thus, recently (see [1]) the code for >> writing a tree structure was changed in a way which guarantees just that: >> >> The start address of a block is unique, but its content may >> (virtually) overlap other blocks. >> >> In this case, the start address is taken from the area inside the >> preceding block, and thus guaranteed to be unique. This is a neat trick >> to circumvent the deferred deallocation of the temporary buffer after >> writing the file but it changes the concept. >> >> >> Consequence >> =========== >> >> A pointer, referencing an element inside of a block, such as a member of >> a struct, cannot be resolved without knowledge about the structure which >> contains the member. >> >> For example, in the old model it was possible to map the file in memory >> and fill an address table with the start addresses and length of the >> blocks and its new address in heap. Based on this address table, any >> pointer could be mapped to its new address by identifying the block >> which contains this address. >> >> This concept allowed for a very lazy method to read a file, just picking >> the data needed. It also allowed for a generic method which is almost >> independent from changes in blenders read and write functions. >> >> With the new model, this is not possible, because blocks can overlap and >> addresses may be ambiguous. >> >> >> Regards >> Holger Machens >> >> >> [1] https://developer.blender.org/rBb98b331d04fe3a335cc0656809b4b09196507500 >> _______________________________________________ >> Bf-committers mailing list >> Bf-committers@blender.org >> https://lists.blender.org/mailman/listinfo/bf-committers >> > > _______________________________________________ > Bf-committers mailing list > Bf-committers@blender.org > https://lists.blender.org/mailman/listinfo/bf-committers _______________________________________________ Bf-committers mailing list Bf-committers@blender.org https://lists.blender.org/mailman/listinfo/bf-committers