Hi Paul, I'm pretty satisfied with the performance of my importers right now, and specially compared to max-script. Turned out the major bottleneck was from UV data, since I wasn't using foreach_set which is a much efficient method than adding uv data in a loop. check http://blenderartists.org/forum/showthread.php?321210-Importing-UV-coordinates This is for video game data that usually isn't very heavy though (although I'm also importing whole levels with hundreds of assets, all in ~40 seconds), and also not using an efficient method for unpacking half-floats (used a lot in video games for uvs).
As I said before, I heard users complaining .obj importer performance vs zbrush's for example in >500 mb data (sculpts mainly). There might be room for improvement there, but consider the topics discussed (would be good to rewrite it? is C an better option? can small tweaks improve performance considerably?) Regards On Thu, Mar 6, 2014 at 12:26 PM, Paul Melis <paul.me...@surfsara.nl> wrote: > Hi Sebastian, > > I read your interesting thread on blender import improvements. Did you > make any progress on this topic? Would this be something that a > summer-of-code student can work on as well? > > Regards, > Paul > > > > On 01/05/14 23:00, Sebastian A. Brachi wrote: > >> Hi all, >> this is the first time writing to the list, I'd like to start learning a >> lot and hopefully help to improve blender in some areas. >> My main interest right now is Blender's import/export pipeline. >> Currently I'm making an addon to import a lot of different formats from >> different game engines and also rewriting/improving some existing ones >> like >> PSK/PSA and MD5. >> >> I'd like to research the best possible ways to import to blender; my first >> concern besides code style is speed in large or medium files, and I have a >> couple questions. I've been reading >> http://wiki.blender.org/index.php/User:Ideasman42/ImportExport_TODO , and >> the idea of using C/C++ modules is very interesting. Here are some >> observations and questions for importing, >> >> (expect some misconceptions): >> >> >> 1) Python file reading/parsing. >> >> *Seems fast enough to me in binary data, even with processing many >> GB's. I >> haven't tested XML/text data though (seems many users complain of the obj >> importer, but where are the main bottlenecks is unknown to me) >> Also best practices for reading seems to improve the speed (see 2)) >> >> Q: C + Ctypes doesn't seem very good since we don't want to rewrite the >> reading part in C if it's done in python, and compile dlls or OS with the >> addons, right? But if someone >> like me would do it, does it seem like the best option because we can >> still keep the mesh building or other data handling in more readable >> python? >> >> Q: Is it worth to investigate Cython/pypy for speed improvements here >> and >> was it used in past addons pre 2.5? >> I haven't done any tests so far and I'd like to know opinions first, >> haven't found more than a couple threads in the list that mention it, and >> a >> few in BA: >> e.g., >> http://blenderartists.org/forum/showthread.php?278213- >> TEST-Cython-performances-test >> >> >> >> 2) Python binary data unpack (struct module). >> >> * This seems to add a lot of overhead, specially in big files. Also >> best >> practices allow for some speedups (like the use of struct.Struct or >> list.extend). >> Besides the main document in the API for best practices with a few tips >> in string handling when importing, I couldn't fine much info. >> >> Q: Is it worth to start/modify a document for best practices, and also >> add benchmarks? Who could I ask to review it? >> >> * What if Blender could accept/interpret python bytes objects to build >> geometry, avoiding the unpacking in python? E.g., reading a face index >> buffer all at once, and just passing the count, >> type (short in most cases) and bytes object to Blender. In case of >> Vertex >> buffer seems more complicated, since many parameters need to be defined, >> such as stride, type of each element, >> ignore vertex normals if included, etc. >> >> Q: Would this be a reasonable option to investigate, or even possible >> to >> do? >> >> * Another bottleneck is when binary data uses half-floats (used >> extensively in game formats for UV data). >> The struct module doesn't have support for them (there is a patch >> though: http://bugs.python.org/issue11734), so a python >> function must be used instead. I'm using this one: >> http://davidejones.com/blog/1413-python-precision-floating-point/ which >> is >> pretty slow. >> >> Q: Is the python function optimal? I couldn't find better ones. >> Q: If not, is it feasible to do one of the following? : >> a) Implement the patch mentioned above in our blender python >> b) Create the function in C and expose it to the python API >> Q: If b) is the best option, do you think is an OK task for me, as a >> first approach to Blender's C code? (no much experience in C besides >> college first year) >> >> >> 3) Python data to blender data (converting lists to C arrays with >> mesh.vertices.add, mesh.polygons.add, etc): >> >> *I've been doing a lot of tests but not much digging into C code. I >> don't >> seem to understand the process very well. >> Using ctypes arrays, or array.array don't seem to have any performance >> improvement when used in reading the file (it's actually a little slower) >> nor building the mesh. >> >> Q: When using python data to import geometry, the python objects need >> to >> be converted to C arrays and that's the overhead right? >> Q: When using c-like objects in python like ctypes array or >> array.array, >> they still need to be converted to C arrays, so that's why performance is >> not improved? >> Q: What could be done to avoid the conversion without using C? >> Something >> like passing a pointer to a ctypes array instead? >> >> >> Thanks! >> Regards >> _______________________________________________ >> Bf-committers mailing list >> Bf-committers@blender.org >> http://lists.blender.org/mailman/listinfo/bf-committers >> > > _______________________________________________ Bf-committers mailing list Bf-committers@blender.org http://lists.blender.org/mailman/listinfo/bf-committers