On Fri, 29 Jul 2016 16:17:06 +1000 David Seikel <[email protected]> said:
> On Fri, 29 Jul 2016 12:52:45 +0900 Carsten Haitzler (The Rasterman) > <[email protected]> wrote: > > > On Fri, 29 Jul 2016 12:40:05 +1000 David Seikel <[email protected]> > > said: > > > > > On Fri, 29 Jul 2016 10:40:35 +0900 Carsten Haitzler (The Rasterman) > > > <[email protected]> wrote: > > > > > > > On Fri, 29 Jul 2016 10:41:14 +1000 David Seikel > > > > <[email protected]> said: > > > > > > > > > On Fri, 29 Jul 2016 08:22:51 +0900 Carsten Haitzler (The > > > > > Rasterman) <[email protected]> wrote: > > > > > > > > > > > On Thu, 28 Jul 2016 14:55:46 -0700 Cedric BAIL > > > > > > <[email protected]> said: > > > > > > > > > > > > > Hello, > > > > > > > > > > > > > > I came to realize that splitting this 3 components don't > > > > > > > really make any more sense. If you want to build anything > > > > > > > on top of efl, you are > > > > > > > > > > > > i totally agree. in fact if we are going to merge... we > > > > > > should do a lot more merging. we need to decide on a future > > > > > > much smaller set of libraries > > > > > > > > > > > > e.g. (lib* assumed) > > > > > > > > > > > > efl: > > > > > > eina, eo, ecore, efl, eio > > > > > > > > > > > > eflgui: > > > > > > evas, edje, emotion, elementary, ector, ecore_audio, > > > > > > ecore_imf, ecore_imf_evas, ecore_input > > > > > > > > > > > > eflnet: > > > > > > ecore_con, ecore_ipc, eldbus, ecore_avahi > > > > > > > > > > > > eflsys: > > > > > > ecore_file, efreet, efreet_mime, efreet_trash, elocation > > > > > > > > > > > > others left over to figure out... > > > > > > ephysics, eet, elua, embryo, eolian, ethumb, ethumb_client > > > > > > > > > > > > and definitely unportable or dubiously universal lib api's: > > > > > > eeze, ecore_drm, ecore_drm2, ecore_buffer, ecore_cocoa, > > > > > > ecore_fb, ecore_psl1ght, ecore_sdl, ecore_wayland, ecore_wl2, > > > > > > ecore_win32, ecore_x, elput, escape, evil > > > > > > > > > > I think you left out evas_3D, though I'm beginning to suspect > > > > > I'm the only major user of that outside of Samsung. Cedric > > > > > mentioned that I might be. > > > > > > > > it is part of evas already. so included under evas. > > > > > > Cool then. > > > > > > > > Eet should be part of the efl library, it's a basic part of the > > > > > system, the data format used by most everything else. Eolian is > > > > > basically part of eo, it could join it in efl. On the other > > > > > hand, it's more a tool than a library? > > > > > > > > eolian tho is not used at runtime. only for code and doc > > > > generation... thus i left it out. eet is indeed core... today. > > > > but it has some issues. we still end up having to allocate masses > > > > of ram to decode data. we can mmap strings directly BUT > > > > realistically i never imagined our edje objects and files would > > > > become so big. they are huge. we eat up multiple MB of data in an > > > > app just for this. at least 100's of Kb for sure. and every app > > > > loads/decodes the same stuff into ram from the eet file. > > > > > > > > what I need to do is work out something new. something that: > > > > > > > > 1. actually isn't a flat key space. the keys in eet files are > > > > actually using a lot of space as they redundantly encode a parent > > > > namespace, so make it a tree to avoid this. > > > > 2. all of this data should be mmaped directly from the file and > > > > accessed directly without decoding anything to ram at all. all > > > > fields used on the fly. for the above directory structure etc. > > > > 3. actual data structures should be accessed directly as mmaped > > > > data form the file and NOT decoded into ram first. mmap them in > > > > place and come up with some magic to be able to access them as if > > > > they were data structures somehow (macros of static inline funcs > > > > of some sort). we'd then not have pointers but OFFSETS. every > > > > piece of data is "it's located as offset X in this file". this on > > > > 64bit will drop memory needed as now we don't use 64bit anymore > > > > (well unless we want eet files > 4GB...). I was mulling some kind > > > > of variable sized encoding of values so as long as the data only > > > > needs 1 bit, it can become part of a flags region of the mmaped > > > > mem with 1 bit masked out. same for other numbers (chars, int's, > > > > shorts, longs, long longs etc.) - encode things in some way where > > > > it'll only use the bits needed. etc. etc. - i haven't thought it > > > > out fully yet, BUt this needs a redesign to really address the > > > > scaling issues we have now. eet was great when things were small. > > > > not anymore. :( > > > > > > Rewriting eet is a whole other subject, might be better to start a > > > new thread about it. > > > > > > For my (not small) SledjHamr virtual world project, I want to use > > > some sort of fast and compact representation of all the world > > > data. My initial thought was to try eet, a custom binary protocol > > > I have half designed, and Lua scripts that have been compressed. > > > Actually, I'm using Lua scripts as protocol / file format for > > > prototyping things, but Lua is going to be a big part anyway. I'll > > > benchmark them all, see which works best. I want to store all this > > > data in whichever format works best, and use the same format for > > > the network protocol. > > > > > > My custom binary protocol could be used as this eet rewrite. It's > > > fairly generic, since virtual worlds deal with a world full of > > > random data types and structures. I haven't gotten far enough yet > > > with anything that it's set in stone, so I could easily re-aim it's > > > design a little to be new eet. > > > > > > I'm gonna write my custom binary protocol anyway, based on EFL > > > goodness, so might as well coordinate with you and have it do both. > > > Alternatively, I'll add new eet to the list and benchmark it as > > > well, if someone else writes it. > > > > it sounds like what i'm thinking is not very lua-ish unless you are > > happy to have luajit ffi to deal directly with mmaped memory for you. > > it'd need custom lua ffi to mimic what i'm thinking of for data > > struct access. it wouldn't be about network at all. it'd be about > > treating file as memory and accessing it in-place "zero > > copy/allocation". yes i'd want it to be a portable mmapable file so i > > think for those on bigendian every field access will also need a > > byteswap too every time it's used, not just on load. :( sorry. that's > > the downside, but bigendian is close to dead so i am not really too > > concerned. > > I can't recall off the top of my head, ARM can be both big and little > endian, or was that PowerPC? ARM is one of my targets, I want to run > this virtual world on smartphones. PowerPC isn't interesting for me as > a target, I doubt any of my users will use one of those. 68k is big endian. everything else CAN do both (except x86 which is little). by CONVENTION arm is little. mips, ppc and sparc are big by convention. but i do know of at least one mips linux port that chose to be little instead of big. it's just a processor mode you choose at boot. > Not sure if I explained myself well enough there. This is a big > project, gonna take a looong time. For now it's all entirely > experimental, which means I write some of it in Lua for the speed of > development, the rest in C. I fully intend to translate anything > that needs the speed into C (possibly with bits of assembler). Also, > Lua is the official scripting language for SledjHamr, so there's gotta > be lots of Lua API for users to use. > > For me I'm designing both a file format and a network protocol. > Basically when you first connect to the world, the raw files themselves > are thrown at you across the network. After that, any changes are sent > as the same protocol, but only the changes. So on disk, and on wire, > exact same format / protocol. > > So in these early days, I have whipped up a quick and dirty Lua format > to describe in world 3D objects. I'll play around with that until I'm > happy with it, and then it becomes the Lua way of creating in world > objects. THEN I planned to write the eet version, based on the data > structure I worked out in the Lua experiments. THEN I planned to write > my binary protocol. THEN benchmark the lot, see which works best. > > The point is to have both tight binary, and human readable protocols. > Binary for speed and low disk / memory, Lua for the scripters. > > The other major thing is to be compatible with Second Life and OpenSim, > (SL & OS) especially OS's hypergrid system, that lets people visit > other servers by teleporting from their home server. So I'll need to > write wrappers around those existing protocols to map to mine. Even > worse, other people want horrid bloated formats / protocols like JSON > and XML, so more wrappers to cater for them. > > But first I gotta start with the Lua protocol experiments. The data > files are Lua tables, which another Lua script just loops through and > passes to C. The internal data structures are mostly Evas_3D objects. ok so you're a ways off. > I've already written a "wrapper" that translates the Second Life > scripting language (LSL - Linden Scripting Language) into Lua, then > runs the Lua under LuaJIT, using EFL worker threads. It manages to > do the translation and compilation many orders of magnitude faster than > SL or OS. Proof that my major design decisions are likely to provide > the serious speed boost I'm aiming for. Not hard really, SL and OS > code bases are both truly crap, one of the main reasons I'm doing this > project. > > In the end, after the experiments, Lua will mostly be used for user > scripting. but you are using luajit which allows ffi which means you could actually direct access maed data structs with pretty much the same overhead as c. > Soooo, to reply.... > > I wasn't intending to implement the binary protocol in Lua, it can be > entirely handled by C. It needs to be as fast as possible, which in > general means minimal overhead in the protocol. Mmapping files, zero > copy, and all the other tricksy C stuff can be done in C. If Lua needs > any of that info (and it will), C massages it into Lua data then it may end up the files are bigger than you'd like because they won't be compressed. the best i can do is come up with really compact layout with little gaps/padding, but it will not be ULTRA compact. it'll be optimized for mapping into mem directly and direct access without decode. > structures. That's the price you pay for scripting languages, they are > a bit slow. Initial network stuff will purely be HTTP transferring the > files, the files cached, then I can mmap the files. Later changes, will > update the cached files. I'm not sure yet if it should just change the > world straight away, then update cached files, or update cached files, > and re process them. Logging off and logging back on, or going away > and coming back again, just has to deal with the changed files, HTTP can > deal with that fine. > > So no need for LuaJIT FFI for reading the files. Though I do intend to > use LuaJIT all the way, coz speeeeeed! For what it is worth, I don't > think LuaJIT FFI is ready for large projects yet. You have to strip > things that FFI can't cope with out of C header files. The next > version of LuaJIT promises to be able to deal with raw C headers, > that's when it will be ready. i was actually expecting hand-crafted ffi c api's. > > > I also intend to write wrappers for my fast, compact, binary > > > protocol, coz every one else wants to use bloated human readable > > > protocols. My main goals are to be faster, smaller, and better > > > than Second Life / OpenSim, yet still be backwards compatible. I > > > could throw in an eet wrapper as well, for EFL backwards > > > compatibility. B-) > > > > my goal is to have them be accessed with NO copies or allocation. > > data structs are accessed directly from mmaped file space. the > > accessing of that data will need to know like base address of file > > and some surrounding metadata to know where to find the real data > > probably. i'm thinking things like (edb == efl data blob) > > As I mentioned, the network protocol downloads or updates files, then > I can mmap them etc. The wrappers translate the legacy protocols into > the new file format. The only reason for the wrappers is legacy > protocols, and bloated human readable protocols. People that want to > use the bloat, they can damn well deal with the resource usage needed > to translate to the binary protocol, and not slow every one else down. > As for legacy protocols, given the expected speed ups, I would not be > surprised if wrappers translating to my binary protocol ends up being > faster than the pure legacy protocol handling in SL and OS. Yes, their > code is that bad. > > > typedef struct { > > const char *title; > > int val; > > My_Data *child; > > } My_Data; > > > > Edb_File *edb_file_open("blah.blob", EDB_READ"); > > Edb_Data *edb_file_path_get(file, "/path/to.key"); > > char *str = edb_blob_field(file, blob, title); > > int val = edb_blob_field(file, blob, value); > > double valf = edb_blob_field(file, blob, value); > > Edb_Data *childblob = edb_blob_field(file, blob, child); > > > > with some macro magic, offsetof(), typeof(), static inline funcs i > > think i can do this to the point where an access basically compiles > > down to something like > > > > char *str = file->baseptr + offset; > > int val = *(file->baseptr + offset); > > > > etc. (bigendian would be doing some byteswapping there). an access > > will not be as fast as a native in-memory struct but it'd save a lot > > of ram. > > Sounds good to me. > > > > > ethumb i think needs a redesign from scratch and so i was > > > > separating it out like we did with e_dbus after 1.7. anything i > > > > think may need a replacement in the future i was separating out. > > > > > > Can we do more than just images this time? Thumbnailing large 3D > > > objects into a tiny copy that rotates in a file requester is > > > something I want to do, or an ordinary 2D thumbnail at least. B-) > > > > that is not a thumbnail. that's interactive unless you mean to > > generate like 10 or 20 low res images with the object slightly > > rotated back and forth...? that'd make sense. as long as someone > > WRITES a thumbnailer for that format/file i suspect it'd work. using > > evas_3d + osemesa installed + buffer engine and you could work fine. > > I said "file requester". Actually it's your avatars inventory, which but you wanted the thumbnail IN the requester. > might include 3D objects, textures, scripts, sounds, text files, and > other things. Since I want to store inventory as files, I've been > playing with the Elementary file requester widget for this task. So > the inventory requester can show thumbnails of the textures, but not of > the 3D objects. One big problem with SL and OS is that names can be > identical, sometimes. They are full of inconsistencies like that. So > how do you tell the difference between "tree", "tree", and "tree" if > all you can see in your inventory is the standard icon for "3D model"? > Well, you could drag each one into the world, pick the one you want, > delete the other two, which sucks. So I want to show a thumbnail for > the 3D objects. A tiny icon sized image would work, but a slowly > rotating tiny version of the 3D model would be better. if you want an actual 3d view of it continually rotating you'll be wanting to DIY then. this has nothing to do with thumbnailing. it's a full actual load of the model into memory and a full render of it spinning. > Now SL and OS include the ability to "take photos", create an image > file of the current view of the world. So yes, I'll need to write > software that scrapes the 3D scene and dumps it as an image file. So > I'll have to write code to turn 3D objects into 2D image files anyway. > Shrink it for the thumbnail, simple. > > I'll also need code to generate smaller LOD versions of the model (I > might use an existing library for that). Shrink that all the way down > to a tiny LOD that fits nicely into thumbnail size. The model itself > doesn't have to be interactive, the interaction is to click on the > thumbnail and drag it elsewhere, or just select it. Rotating slowly, > perhaps only rotating the one that is currently hovered over. > > > > > > could be part of eflgui, since ecore_audio is. Evas_3D probably > > > > > belongs in eflgui as well, and I understand there may be some > > > > > linking of evas_3D and ephysics at some stage, so they might go > > > > > together. > > > > > > > > ephysics has some memory issues atm and it really needs to > > > > probably become a loabable module or simple efl (edje > > > > specifically here) will dlopen libephysics on demand only when > > > > needed. > > > > > > > > https://phab.enlightenment.org/T4227 > > > > > > > > about 72Kb there just to link it in and never actually use it (for > > > > ephysics and it's dependencies). per process. thats private dirty > > > > pages. thus things that are used rarely shoduld be split out or > > > > somehow made to only dynamically link on demand when used. > > > > > > Ah, I forgot about the Bullet library. 3D applications might be > > > needing physics, certainly mine will, and Bullet works for me. > > > There will need to be some integration between some sort of physics > > > system and Evas_3D. I can see two separate physics engines though, > > > a full one like Bullet, and a simple one for basic stuff like not > > > falling through the ground or walking through walls. The simple > > > one we could merge, the full one leave separate. > > > > simple one still atm would be using bullet. ephysics does anyway > > unless you are saying to write a simple physics engine ourselves to > > avoid bullet overhead for simple stuff? either way for now until then > > it makes sense to keep ephysics separate and dlopen() it only when > > needed as a hello world or 99% of ui's don't need it. > > Yep, I was saying to write our own small physics engine for simple > stuff. We already have bounding boxes. In my case, just stopping > peoples avatars from falling through terrain and walking through > walls. SL and OS do that I believe, and that is most of the physics > requirements a lot of the time. Sure people can script stuff that > needs a full physics engine, but that's not common in most places > I have seen. Considering how badly that simple level of physics works > in SL and OS, we don't need a lot to at least be as good. I'd be happy > with that. > > > > > > Elua I haven't played with yet, it was lacking docs and examples > > > > > last time I looked, and I didn't have the time to grok the > > > > > source code. So not sure where it belongs. > > > > > > > > > > I don't even know what elput is. > > > > > > > > for running against drm/kms framebuffer and getting input devices > > > > directly. e uses it. its like the input part of ecore_fb. needed > > > > if you want to run terminology or efl apps directly in the fb but > > > > in drm/kms mode not fbdev (ecore_fb) > > > > > > Ah, I've not used drm/kms, though I have used fb, so that's why I > > > wasn't familiar with it. > > > > > > > > You are probably well aware of my thoughts about efreet, so I > > > > > wont mention where I think it belongs. lol > > > > > > > > it likely needs to also get abstractions BUt it does actually work > > > > for open source tools/code across linux's, bsd's and even osx for > > > > the open source stuff. but windows is a different issue and > > > > ultimately i think this needs a higher abstraction layer and a > > > > new design that is at the api and functionality level abstracted > > > > from FDO but has an FDO data backend to import/update FDO data > > > > into a neutral data store we can use. then add back-ends for > > > > windows and osx and whatever other OS needed too. > > > > > > I'm happy to resurrect my old FDO code, and beat it into that sort > > > of shape. I also wanted to have a higher abstraction, and actually > > > use it in E, importing FDO or other stuff. We are on the same page > > > here. B-) > > > > good. if your old code works then its a good place to start. i'd > > probably start by making a neutral datastore at least a fairly > > barebones one with api's to access it THEN the importers/watchers > > like efreetd does. and of course.. efreet data also consumes memory > > like edje... the mmapable data structs like above with the data blobs > > is something i'd want to build on. > > Well, it's been many years, it probably bit rotted. I recall it wasn't > far off basically working. I had planned on rewriting parts of it to > tidy things up, a concept Sebastian didn't seem to understand. There's > probably even some more infrastructure in EFL these days that would be > useful. I could likely get rid of the tree implementation I rolled > just for FDO. > > Writing it on top of these new eet mmaped files we discuss above, sounds > great. Gives us a priority as well, eet replacement first, efreet next. yes. :) -- ------------- Codito, ergo sum - "I code, therefore I am" -------------- The Rasterman (Carsten Haitzler) [email protected] ------------------------------------------------------------------------------ _______________________________________________ enlightenment-devel mailing list [email protected] https://lists.sourceforge.net/lists/listinfo/enlightenment-devel
