/me trims the old stuff we are no longer talking about. Told you we needed a new thread. B-)
On Fri, 29 Jul 2016 15:35:01 +0900 Carsten Haitzler (The Rasterman)
<ras...@rasterman.com> wrote:
> On Fri, 29 Jul 2016 16:17:06 +1000 David Seikel <onef...@gmail.com>
> said:
>
> > On Fri, 29 Jul 2016 12:52:45 +0900 Carsten Haitzler (The Rasterman)
> > <ras...@rasterman.com> wrote:
> >
> > > On Fri, 29 Jul 2016 12:40:05 +1000 David Seikel
> > > <onef...@gmail.com> said:
> > >
> > > > On Fri, 29 Jul 2016 10:40:35 +0900 Carsten Haitzler (The
> > > > Rasterman) <ras...@rasterman.com> wrote:
> > > >
> > > > > On Fri, 29 Jul 2016 10:41:14 +1000 David Seikel
> > > > > <onef...@gmail.com> said:
> > > > >
<dodgy snip>
> > > > > > Eet should be part of the efl library, it's a basic part of
> > > > > > the system, the data format used by most everything else.
> > > > > > Eolian is basically part of eo, it could join it in efl.
> > > > > > On the other hand, it's more a tool than a library?
> > > > >
> > > > > eolian tho is not used at runtime. only for code and doc
> > > > > generation... thus i left it out. eet is indeed core... today.
> > > > > but it has some issues. we still end up having to allocate
> > > > > masses of ram to decode data. we can mmap strings directly BUT
> > > > > realistically i never imagined our edje objects and files
> > > > > would become so big. they are huge. we eat up multiple MB of
> > > > > data in an app just for this. at least 100's of Kb for sure.
> > > > > and every app loads/decodes the same stuff into ram from the
> > > > > eet file.
> > > > >
> > > > > what I need to do is work out something new. something that:
> > > > >
> > > > > 1. actually isn't a flat key space. the keys in eet files are
> > > > > actually using a lot of space as they redundantly encode a
> > > > > parent namespace, so make it a tree to avoid this.
> > > > > 2. all of this data should be mmaped directly from the file
> > > > > and accessed directly without decoding anything to ram at
> > > > > all. all fields used on the fly. for the above directory
> > > > > structure etc. 3. actual data structures should be accessed
> > > > > directly as mmaped data form the file and NOT decoded into
> > > > > ram first. mmap them in place and come up with some magic to
> > > > > be able to access them as if they were data structures
> > > > > somehow (macros of static inline funcs of some sort). we'd
> > > > > then not have pointers but OFFSETS. every piece of data is
> > > > > "it's located as offset X in this file". this on 64bit will
> > > > > drop memory needed as now we don't use 64bit anymore (well
> > > > > unless we want eet files > 4GB...). I was mulling some kind
> > > > > of variable sized encoding of values so as long as the data
> > > > > only needs 1 bit, it can become part of a flags region of the
> > > > > mmaped mem with 1 bit masked out. same for other numbers
> > > > > (chars, int's, shorts, longs, long longs etc.) - encode
> > > > > things in some way where it'll only use the bits needed. etc.
> > > > > etc. - i haven't thought it out fully yet, BUt this needs a
> > > > > redesign to really address the scaling issues we have now.
> > > > > eet was great when things were small. not anymore. :(
> > > >
> > > > Rewriting eet is a whole other subject, might be better to
> > > > start a new thread about it.
> > > >
> > > > For my (not small) SledjHamr virtual world project, I want to
> > > > use some sort of fast and compact representation of all the
> > > > world data. My initial thought was to try eet, a custom binary
> > > > protocol I have half designed, and Lua scripts that have been
> > > > compressed. Actually, I'm using Lua scripts as protocol / file
> > > > format for prototyping things, but Lua is going to be a big
> > > > part anyway. I'll benchmark them all, see which works best. I
> > > > want to store all this data in whichever format works best, and
> > > > use the same format for the network protocol.
> > > >
> > > > My custom binary protocol could be used as this eet rewrite.
> > > > It's fairly generic, since virtual worlds deal with a world
> > > > full of random data types and structures. I haven't gotten far
> > > > enough yet with anything that it's set in stone, so I could
> > > > easily re-aim it's design a little to be new eet.
> > > >
> > > > I'm gonna write my custom binary protocol anyway, based on EFL
> > > > goodness, so might as well coordinate with you and have it do
> > > > both. Alternatively, I'll add new eet to the list and benchmark
> > > > it as well, if someone else writes it.
> > >
> > > it sounds like what i'm thinking is not very lua-ish unless you
> > > are happy to have luajit ffi to deal directly with mmaped memory
> > > for you. it'd need custom lua ffi to mimic what i'm thinking of
> > > for data struct access. it wouldn't be about network at all. it'd
> > > be about treating file as memory and accessing it in-place "zero
> > > copy/allocation". yes i'd want it to be a portable mmapable file
> > > so i think for those on bigendian every field access will also
> > > need a byteswap too every time it's used, not just on
> > > load. :( sorry. that's the downside, but bigendian is close to
> > > dead so i am not really too concerned.
> >
> > I can't recall off the top of my head, ARM can be both big and
> > little endian, or was that PowerPC? ARM is one of my targets, I
> > want to run this virtual world on smartphones. PowerPC isn't
> > interesting for me as a target, I doubt any of my users will use
> > one of those.
>
> 68k is big endian. everything else CAN do both (except x86 which is
> little). by CONVENTION arm is little. mips, ppc and sparc are big by
> convention. but i do know of at least one mips linux port that chose
> to be little instead of big. it's just a processor mode you choose at
> boot.
How about an endian bit in the file? If it's the wrong endianness,
flip the file once, then store forever. I suspect that apart from the
odd squirt across a network, most files are gonna stay on the same
computer for their lifetime. Even though my case involves mostly
Internet squirting. I can live with an endian flip once per downloaded
file, before storing it in the cache. The point of the cache is to not
have to download stuff you saw last time.
Linux on PCs, MacOS, Windows on crappy business / student laptops
(you'd be surprised how many people use those for virtual worlds), and
Android (ARM and x86) are my targets. I don't have the hardware to test
anything else, and they are very rare amongst people using virtual
worlds. I will try testing it on Linux running on a x486, but that's
not a real target, just to see if I can. B-)
OK, I have three or four Amiga's in Melbourne, which I last saw last
millennia. No idea if they still work. Oh wait, I have a PPC in my
PS3, and Cells.
> > Not sure if I explained myself well enough there. This is a big
> > project, gonna take a looong time. For now it's all entirely
> > experimental, which means I write some of it in Lua for the speed of
> > development, the rest in C. I fully intend to translate anything
> > that needs the speed into C (possibly with bits of assembler).
> > Also, Lua is the official scripting language for SledjHamr, so
> > there's gotta be lots of Lua API for users to use.
> >
> > For me I'm designing both a file format and a network protocol.
> > Basically when you first connect to the world, the raw files
> > themselves are thrown at you across the network. After that, any
> > changes are sent as the same protocol, but only the changes. So on
> > disk, and on wire, exact same format / protocol.
> >
> > So in these early days, I have whipped up a quick and dirty Lua
> > format to describe in world 3D objects. I'll play around with that
> > until I'm happy with it, and then it becomes the Lua way of
> > creating in world objects. THEN I planned to write the eet
> > version, based on the data structure I worked out in the Lua
> > experiments. THEN I planned to write my binary protocol. THEN
> > benchmark the lot, see which works best.
> >
> > The point is to have both tight binary, and human readable
> > protocols. Binary for speed and low disk / memory, Lua for the
> > scripters.
> >
> > The other major thing is to be compatible with Second Life and
> > OpenSim, (SL & OS) especially OS's hypergrid system, that lets
> > people visit other servers by teleporting from their home server.
> > So I'll need to write wrappers around those existing protocols to
> > map to mine. Even worse, other people want horrid bloated
> > formats / protocols like JSON and XML, so more wrappers to cater
> > for them.
> >
> > But first I gotta start with the Lua protocol experiments. The data
> > files are Lua tables, which another Lua script just loops through
> > and passes to C. The internal data structures are mostly Evas_3D
> > objects.
>
> ok so you're a ways off.
Yep. Writing an entire world simulator is a biiiiig job, and I'm just
one coder. The good thing is that there's so many independent bits, I
can pick and choose what to work on next. It's lots of fun. B-)
> > I've already written a "wrapper" that translates the Second Life
> > scripting language (LSL - Linden Scripting Language) into Lua, then
> > runs the Lua under LuaJIT, using EFL worker threads. It manages to
> > do the translation and compilation many orders of magnitude faster
> > than SL or OS. Proof that my major design decisions are likely to
> > provide the serious speed boost I'm aiming for. Not hard really,
> > SL and OS code bases are both truly crap, one of the main reasons
> > I'm doing this project.
> >
> > In the end, after the experiments, Lua will mostly be used for user
> > scripting.
>
> but you are using luajit which allows ffi which means you could
> actually direct access maed data structs with pretty much the same
> overhead as c.
Why? I'd have to do it in C anyway, coz Lua isn't always gonna be
involved. Do it once in C, no need to translate the code into Lua when
C is perfectly capable of handing data to Lua. Duplicated code in two
different languages, that's asking for trouble. Remember, the initial
Lua code during the experimental phase is just to speed up writing the
experimental code. Lots of it is likely to get rewritten in C when I
stop experimenting.
> > Soooo, to reply....
> >
> > I wasn't intending to implement the binary protocol in Lua, it can
> > be entirely handled by C. It needs to be as fast as possible,
> > which in general means minimal overhead in the protocol. Mmapping
> > files, zero copy, and all the other tricksy C stuff can be done in
> > C. If Lua needs any of that info (and it will), C massages it into
> > Lua data
>
> then it may end up the files are bigger than you'd like because they
> won't be compressed. the best i can do is come up with really compact
> layout with little gaps/padding, but it will not be ULTRA compact.
> it'll be optimized for mapping into mem directly and direct access
> without decode.
I'm not sure how that's related to what you replied to. I don't see
how doing it in C or Lua affects compression.
I picked HTTP for the basic file transfer, so I can leverage all the
existing HTTP infrastructure in the world. Proxies, encryption, caches,
CDN's, load balancing, last modified date, etc. And automatic
compression of the data over the wire.
This stuff is already written by others, and well optimized / tested.
Exposing the assets to web servers like this also allows fancy things
by other developers, like managing their assets with their favourite
CMS, writing in world scripts in what ever web capable language they
want, and bizarre mashups I never thought of.
If I remember eet already has signing and encryption, which are
optional. Encryption will need decoding anyway. Compression on the
disk could merely be an option. Let the coder decide to compress large
files or not, depending on their needs. Any tiny files probably wont
compress well anyway.
In my case, parts of some world might be visited often, like your home,
and they could be left uncompressed for speedy direct acces. Places you
rarely visit could be compressed most of the time, but uncompressed
while you visit. A place might only be visited once a week, but be
quite large, so you want it compressed for the other six days.
> > structures. That's the price you pay for scripting languages, they
> > are a bit slow. Initial network stuff will purely be HTTP
> > transferring the files, the files cached, then I can mmap the
> > files. Later changes, will update the cached files. I'm not sure
> > yet if it should just change the world straight away, then update
> > cached files, or update cached files, and re process them. Logging
> > off and logging back on, or going away and coming back again, just
> > has to deal with the changed files, HTTP can deal with that fine.
> >
> > So no need for LuaJIT FFI for reading the files. Though I do
> > intend to use LuaJIT all the way, coz speeeeeed! For what it is
> > worth, I don't think LuaJIT FFI is ready for large projects yet.
> > You have to strip things that FFI can't cope with out of C header
> > files. The next version of LuaJIT promises to be able to deal with
> > raw C headers, that's when it will be ready.
>
> i was actually expecting hand-crafted ffi c api's.
I added that last bit about C headers with respect to elua, which is
basically a replacement for edje_lua2, using FFI, if I have that
correct. Hand-crafted FFI C APIs leave us with the same problem
edje_lua and edje_lua2 had, too many APIs to hand craft means it
doesn't get done quickly. We should both know that, you are the one
that didn't finish edje_lua, I'm the one that didn't finish edje_lua2.
Sure, hand crafted API's work for a small number of APIs, but for the
huge amount we have in EFL, I think waiting for the next version of
LuaJIT, and just feeding it standard EFL C headers gets a lot more done
quickly.
Yes I know you wanted bidirectional data transfer in each function
call, which EFL C codo doesn't do, but I actually found that slowed
things down, rarely did I actually need that.
> > > > I also intend to write wrappers for my fast, compact, binary
> > > > protocol, coz every one else wants to use bloated human readable
> > > > protocols. My main goals are to be faster, smaller, and better
> > > > than Second Life / OpenSim, yet still be backwards compatible.
> > > > I could throw in an eet wrapper as well, for EFL backwards
> > > > compatibility. B-)
> > >
> > > my goal is to have them be accessed with NO copies or allocation.
> > > data structs are accessed directly from mmaped file space. the
> > > accessing of that data will need to know like base address of file
> > > and some surrounding metadata to know where to find the real data
> > > probably. i'm thinking things like (edb == efl data blob)
> >
> > As I mentioned, the network protocol downloads or updates files,
> > then I can mmap them etc. The wrappers translate the legacy
> > protocols into the new file format. The only reason for the
> > wrappers is legacy protocols, and bloated human readable
> > protocols. People that want to use the bloat, they can damn well
> > deal with the resource usage needed to translate to the binary
> > protocol, and not slow every one else down. As for legacy
> > protocols, given the expected speed ups, I would not be surprised
> > if wrappers translating to my binary protocol ends up being faster
> > than the pure legacy protocol handling in SL and OS. Yes, their
> > code is that bad.
> >
> > > typedef struct {
> > > const char *title;
> > > int val;
> > > My_Data *child;
> > > } My_Data;
> > >
> > > Edb_File *edb_file_open("blah.blob", EDB_READ");
> > > Edb_Data *edb_file_path_get(file, "/path/to.key");
> > > char *str = edb_blob_field(file, blob, title);
> > > int val = edb_blob_field(file, blob, value);
> > > double valf = edb_blob_field(file, blob, value);
> > > Edb_Data *childblob = edb_blob_field(file, blob, child);
> > >
> > > with some macro magic, offsetof(), typeof(), static inline funcs i
> > > think i can do this to the point where an access basically
> > > compiles down to something like
> > >
> > > char *str = file->baseptr + offset;
> > > int val = *(file->baseptr + offset);
> > >
> > > etc. (bigendian would be doing some byteswapping there). an access
> > > will not be as fast as a native in-memory struct but it'd save a
> > > lot of ram.
> >
> > Sounds good to me.
> >
> > > > > ethumb i think needs a redesign from scratch and so i was
> > > > > separating it out like we did with e_dbus after 1.7. anything
> > > > > i think may need a replacement in the future i was separating
> > > > > out.
> > > >
> > > > Can we do more than just images this time? Thumbnailing large
> > > > 3D objects into a tiny copy that rotates in a file requester is
> > > > something I want to do, or an ordinary 2D thumbnail at least.
> > > > B-)
> > >
> > > that is not a thumbnail. that's interactive unless you mean to
> > > generate like 10 or 20 low res images with the object slightly
> > > rotated back and forth...? that'd make sense. as long as someone
> > > WRITES a thumbnailer for that format/file i suspect it'd work.
> > > using evas_3d + osemesa installed + buffer engine and you could
> > > work fine.
> >
> > I said "file requester". Actually it's your avatars inventory,
> > which
>
> but you wanted the thumbnail IN the requester.
Yep. I can't recall off the top of my head, the requester already shows
thumbnails of images? If not, it should. So why not add the ability to
add other types of thumbnail? You wrote a terminal proggy that shows
textures and videos, if it's good enough for a terminal, it's even
better for a file requester. We could even add that to terminology,
show 3D models when hovering over them. B-)
If the standard file requester can't do it, I'll have to write my own.
You know that Evas_3D can put buttons on 3D objects, and 3D objects on
buttons? It's doable, easily.
> > might include 3D objects, textures, scripts, sounds, text files, and
> > other things. Since I want to store inventory as files, I've been
> > playing with the Elementary file requester widget for this task. So
> > the inventory requester can show thumbnails of the textures, but
> > not of the 3D objects. One big problem with SL and OS is that
> > names can be identical, sometimes. They are full of
> > inconsistencies like that. So how do you tell the difference
> > between "tree", "tree", and "tree" if all you can see in your
> > inventory is the standard icon for "3D model"? Well, you could drag
> > each one into the world, pick the one you want, delete the other
> > two, which sucks. So I want to show a thumbnail for the 3D
> > objects. A tiny icon sized image would work, but a slowly rotating
> > tiny version of the 3D model would be better.
>
> if you want an actual 3d view of it continually rotating you'll be
> wanting to DIY then. this has nothing to do with thumbnailing. it's a
> full actual load of the model into memory and a full render of it
> spinning.
Yes, but the thumbnail engine is responsible for doing that sort of
thing in another process, possibly running through a directory building
them before they are needed. The result as a tiny version of the
file, that is used as an icon.
I recall we had a similar discussion before, I wanted to use ethumb to
do the FDO icon search. Again, the result is the same, a small image
is returned after some processing.
The task is the same, building multiple icons in a deamon, to leave the
main app or apps to get on with other things in the mean time. Make it
modular, so I can write my own "produce a 3D icon", "a 2D icon from a
3D model", and "do the complex FDO icon search" modules.
It's thumbnails Jim, but not as we know it. The result is the same,
some lengthy process produces a small icon. That the icon might be an
animation, a 3D model, or even an edje file (file requesters for edje
editors), shouldn't matter. Sometimes you even want to generate a
whole bunch, and store them on disk for later, so you only need to do
them once.
> > Now SL and OS include the ability to "take photos", create an image
> > file of the current view of the world. So yes, I'll need to write
> > software that scrapes the 3D scene and dumps it as an image file.
> > So I'll have to write code to turn 3D objects into 2D image files
> > anyway. Shrink it for the thumbnail, simple.
> >
> > I'll also need code to generate smaller LOD versions of the model (I
> > might use an existing library for that). Shrink that all the way
> > down to a tiny LOD that fits nicely into thumbnail size. The model
> > itself doesn't have to be interactive, the interaction is to click
> > on the thumbnail and drag it elsewhere, or just select it.
> > Rotating slowly, perhaps only rotating the one that is currently
> > hovered over.
<snip>
> > > > > > You are probably well aware of my thoughts about efreet, so
> > > > > > I wont mention where I think it belongs. lol
> > > > >
> > > > > it likely needs to also get abstractions BUt it does actually
> > > > > work for open source tools/code across linux's, bsd's and
> > > > > even osx for the open source stuff. but windows is a
> > > > > different issue and ultimately i think this needs a higher
> > > > > abstraction layer and a new design that is at the api and
> > > > > functionality level abstracted from FDO but has an FDO data
> > > > > backend to import/update FDO data into a neutral data store
> > > > > we can use. then add back-ends for windows and osx and
> > > > > whatever other OS needed too.
> > > >
> > > > I'm happy to resurrect my old FDO code, and beat it into that
> > > > sort of shape. I also wanted to have a higher abstraction, and
> > > > actually use it in E, importing FDO or other stuff. We are on
> > > > the same page here. B-)
> > >
> > > good. if your old code works then its a good place to start. i'd
> > > probably start by making a neutral datastore at least a fairly
> > > barebones one with api's to access it THEN the importers/watchers
> > > like efreetd does. and of course.. efreet data also consumes
> > > memory like edje... the mmapable data structs like above with the
> > > data blobs is something i'd want to build on.
> >
> > Well, it's been many years, it probably bit rotted. I recall it
> > wasn't far off basically working. I had planned on rewriting parts
> > of it to tidy things up, a concept Sebastian didn't seem to
> > understand. There's probably even some more infrastructure in EFL
> > these days that would be useful. I could likely get rid of the
> > tree implementation I rolled just for FDO.
> >
> > Writing it on top of these new eet mmaped files we discuss above,
> > sounds great. Gives us a priority as well, eet replacement first,
> > efreet next.
>
> yes. :)
In this case, I already wrote much of the FDO code before, so I
volunteer to do this efreet "rewrite". "Rewrite" in quotes, coz efreet
was essentially a rewrite of my version. lol
As for the eet rewrite, you are talking as if you'll do it. Now I've
already said that I was planning writing something similar anyway, and
benchmarking it against eet. Though I'll take your word for it that
eet isn't suitable for the sort of large data I'll be dealing with.
(For example, my own little OpenSim world is stored in about 70 GB of
database, though that's over 400 users worth of inventory and in world
assets covering about 25 square kms of virtual land, to a height of 16
kms. I expect I could severely reduce that database size, mostly from
the automated deduplication I'll write. The exact same texture could
be in dozens of inventories, and used in dozens of 3D objects, SL and
OS will store that single texture dozens + dozens of times. Then it
caches the assets on the server as individual files for a bit of speed,
so (dozens + dozens) * 2 copies of the one large image. I said their
code was crap.)
Now if you rewrite eet, I might still write my version, and benchmark
them. Dunno yet. With so much to write for this project, I have a
tendency to hope others write some bits for me. So normally I'd just
drop it and wait for your version. However, I'm still waiting for you
to make terminology a widget (yep, I want to use that in SledjHamr
to, don't ask). I might just get around to writing my original plan
first, before you get around to eet, I did put some effort into half way
designing it. While we have almost identical goals, I have a bit extra
I want to add on. I could easily bend my version to suit you.
Timing of either of us getting this done is completely unknown. The
race is on? B-)
--
A big old stinking pile of genius that no one wants
coz there are too many silver coated monkeys in the world.
signature.asc
Description: PGP signature
------------------------------------------------------------------------------
_______________________________________________ enlightenment-devel mailing list enlightenment-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/enlightenment-devel
