On Tue, 20 Sep 2016 11:59:45 -0300 Gustavo Sverzut Barbieri <barbi...@gmail.com> said:
> On Tue, Sep 20, 2016 at 1:54 AM, Carsten Haitzler <ras...@rasterman.com> > wrote: > > On Tue, 20 Sep 2016 01:08:48 -0300 Gustavo Sverzut Barbieri > > <barbi...@gmail.com> said: > > > >> On Mon, Sep 19, 2016 at 8:44 PM, Carsten Haitzler <ras...@rasterman.com> > >> wrote: > >> > On Mon, 19 Sep 2016 11:31:58 -0300 Gustavo Sverzut Barbieri > >> > <barbi...@gmail.com> said: > >> > > >> >> Hi all, > >> >> > >> >> Today I did commit support for libproxy in ecore_con's new API, > >> >> Efl.Net.Dialer.Tcp, Efl.Net.Dialer.Http and Efl.Net.Dialer.Websocket > >> >> (implicit from HTTP). > >> > > >> > ummm i just am looking now (i have a lot of catching up to do). can you > >> > undo the changes to configure.ac and make this a dlopen/dlsym approach? > >> > well use eina_module. look a the old curl eina_module approach. look at > >> > what i did in ecore_audio too. in fact i need to look over all the efl > >> > net code, but you want to use emile for ssl stuff and eina_module for > >> > curl too there too (i haven't looked yet). there are very good reasons > >> > to do all of this. > >> > >> I've checked what you did for curl, since I use that and need more symbols. > >> > >> my question and concern are just: > >> - not being able to compile without libproxy. If we always detect, > >> should we remove --enable-libproxy from configure.ac? > > > > yes. it moves to runtime not compile-time. several reasons: > > > > 1. makes compiling simpler. people dont have to find dependencies and those > > -dev/devel pkgs too. if they add libproxy later after building efl magically > > the feature works without a rebuild. > > 2. speed up startup time with less linking going on for apps especially for > > features they may never use > > 3. saves memory - symbol tables and dirtying private pages to do the symbol > > fixups is expensive. i nuked like 320k or so before 1.18 release of actual > > real memory usage that was dirty pages from rarely used libraries.features. > > it's expensive and so only load in if/when needed to save this cost. this > > cost is private pages PER PROCESS using efl so it multiples and is not a > > one-off cost. > > 4. the idea of isolating such costs into a proxy daemon/process probably is > > a good thing and that saves adding the above cost for any process then > > needing to do proxy pac file parsing etc. and isolates the cost into a > > single daemon. > > I was asking if we should offer a way to not even try runtime detection. and do what instead? you mean just link libproxy into the efl daemon proxy binary? > >> - I ask the above because libproxy is a monster, it will pull in a JS > >> loader, glib, C++... all of that, per process. If using pacrunner > >> (from connman guys) you can just link with libdbus, less bad. > > > > in GENERAL go for a dlopen(eina_module) style approach, BUT if a daemon > > makes sense, then do that. > > It's not something that excludes the other. Actually the opposite if > we do the proper way. > > - efl does libproxy dlopen, so doesn't link to that library (simple > API but heavy in dependency) but the moment we do any network stuff we bloat out memory usage then. have 10 apps needing to do network stuff. 10x bloatage. :) > - efl could recommend pacrunner's libproxy until we do our own, a big > improvement - zero work doing our own is less work. just use libproxy IN an efl binary - just like is there right now but in the lib... move to binary but connect to it (if connect fails, exec then keep trying to connect every 0.1 sec until there). > - efl could provide its own libproxy.so, like > /usr/lib/enlightenment/libproxy.so which would talk to our own server > (which can be built to use system's libproxy, which in turn can be the > official one or pacrunner's drop-in replacement). Our dlopen() can > prefer that version. just the above. no systemd. no dbus. nothing but a basic local unix socket which efl.net already does. it's all there. efreetd works this way already. > - we can employ LD_PRELOAD tricks in Enlightenment to force loading > our libproxy.so instead of the system. Our daemon must be started > without it, of course :-) > > > >it makes more sense for us to do our own daemon for efl > > like efreet. dbus is not a good solution for this. > > agreed, the API is simple so a pure unix socket can do, protocol is: > > send: <int:url_length><char[]:url> > recv: <int:count><int:proxy1_len><char[]:proxy1><int:proxy2_len><char > []:proxy2>.... > > as libproxy IS blocking and they explicitly recommend to be called > from a thread, our code will be simple send/recv calls, no other > libraries to pollute our users. yup. just re-use efl.net internally to connect to an efl.net.proxy binary. if connect fails, launch binary and set up timer to retry connect until connected. when connected, send req just like above, with reply coming back. perfect. :) re-use that connection for every proxy request and the daemon will deal with it. really simple. LATER we can also add pacrunner support too. we can at runtime USE this if it is there, if not, use our own as above. > > supporting pacrunner later > > of course is doable, but i would not do this out of the box because > > pacrunner is not everywhere, dbus session buses do not exist when you do > > things like use the console (terminology using fbcon is a major problem > > when it comes to ethumb and was for efreet because no session bus is active > > thus everything bound to a session bus is broken). i will eventually > > rewrite ethumb features likely in elm and drop ethumb because of such > > issues. also ethumb design is broken for SMACK systems so we really do need > > a "fork off a slave process that is custom for your app only" method of > > working. > > you know my opinion on this: this is a nasty approach. The dbus is > simple to achieve if you start that, as you do in E if there was no > session. Or use systemd, it should be doing that from pam_systemd. or just use ecore_exe and have it launch whatever thumbnailer slave in the bg is needed and chat to it with ecore_exe_send() and handle exe data coming back as results. now the exe inherits the parent process smack label and all is well. > SMACK is a different issue and the design of Ethumb came from a time > where we expected all processes to share thumbnails, like efm, your > elm file selector, etc. Private thumbnails as you mentioned started to > gain traction later... application isolation as well, it began with > iOS/Android, linux desktop still is not there. i know. thus needing a redesign. perhaps like above. :) > > for proxies i see the value of a single efl.net.proxy daemon service to > > serve as a proxy handler - send relevant info to this daemon, let it decide > > and then answer back with what connection to make. so basically take the > > libproxy stuff you just committed, move it to a daemon (like efreet does > > now), and connect to it (execute then connect if connect fails). if you > > don't, then i'll eventually need to do this. > > see my plan above, sounds progressive and we don't need to redo > anything. It will work right now, if pacrunner is there it will work > and if we ever create a daemon, we can provide a libproxy.so to be > LD_PRELOAD'ed on apps (pure libC shim). make daemon now. daemon uses libproxy. code is already there minus the daemon "efl.net client handling" and the client side connect+launch logic. > > keep this in mind as a design point. for data that is "global" (efreet style > > stuff, proxy info etc.) then a single daemon doing the work will be the best > > solution. maybe in future we might/can merge these daemons into a single > > "efl daemon". > > indeed makes sense. > > > >> > emile too needs to switch to using this approach internally as well. > >> > > >> >> Libproxy (https://libproxy.github.io) tries to dynamically find a > >> >> proxy for a given URL, unlike plain $http_proxy, $no_proxy... which > >> >> needs a mess to get right in complex environments such as big > >> >> corporations. It will also use PAC - ProxyAutoConfiguration, those > >> >> JavaScript like files named wpad.dat, as well as the wpad host on the > >> >> local network, where it tries to find the JS file. > >> >> > >> >> More than that, libproxy uses the configuration from Desktop > >> >> Environments. There are none for E (some volunteer?), but we can use > >> >> the one from Gnome3/Gnome or KDE. It will also handle envvars and some > >> >> /etc/sysconf configuration. > >> > > >> > what kind of config? you mean like a dbus interface exposed from e to > >> > provide proxy info? i haven't looked... > >> > >> whatever you want, like > >> https://github.com/libproxy/libproxy/tree/master/libproxy/modules > >> > >> they have config_envvars that check http_proxy, no_proxy... but they > >> also have one for gnome that uses gsettings. > > > > err so you mean write code into libproxy to get proxy settings from e? a > > quick look at the above implies that... is there a way of hooking into this > > e.g. from the above "efl proxy daemon"? actually i'm sure there is... > > from what I understand, you create a module ("mm_info_" symbol), see > https://github.com/libproxy/libproxy/blob/master/libproxy/modules/config_gnome.cpp > then you put it in /usr/lib/libproxy/${VERSION}/modules/. That module > can provide a function to say if the module should be used, like based > on envvars. oooh aaah. ok. so it has a module system to load something from another desktop env. we dont need to patch libproxy itself. cool. this might be worth looking into specifically for other toolkit/de apps using libproxy directly. > > either way a single daemon would solve a lot of the above issues of libproxy > > pulling in half the universe into every app needing to do network stuff. > > agreed. BUT looking at ArchLinux packaging, well-known software does > link to that... glib-networking qt5-base vlc. Definitely we can and > should improve, but it's not a blocker. it is a blocker for things like tizen where mem footprint really matters much more and you cant dismiss like the above :) > >> - push distros to package pacrunner (glib + dbus + curl, very simple) > > > > we pushed connman years ago. not again. distros just don't want to. we need > > to DIY and provide our own "pacrunner". that's easy enough for us as we do > > it with efreetd already. we have lots of infra to do this easily. if we > > just have an eflnet "proxy handler" and efl.net spawns and talks to it we > > have a solution that doesn't require pacrunner to be packaged and so on > > (going by history of connman i just dont want to go here again), and it's > > TRIVIAL for us to support. you have there all the relevant code already. > > just needs to be put into its own binary, with an ipc/whatever conn for the > > user and some code to connect and/or launch it. > > doing this way will not help, just creates another technology just our > apps will use... and frankly that would benefit what? just terminology > to get previews, E to check updates? already do it with efreet. it's less problematic on installs - no service files to install in special dirs for dbus/socket ectivation, and its very simple. yes - it's just for efl.net and no one else. it's just moving code around so instead of in the lib it's in a binary the lib connects to/runs then connects to. > most of the other software doing network uses that libproxy, like I > said, qt/glib, vlc... thus all of those would be missed. We need a > solution that works for all of them, so the libproxy drop-in repl. that's a pain as we now have to have a conflicting .so that is in the ld.so lookup path/LD_LIBRARY_PATH and we're going to fight with the real libproxy to replace it. what matters is the mem footprint of efl. someone writes their hello world app and looks they shouldnt see it gobble up multiple mb's of ram. if they use a network we should try minimize the impact. we can do nothing about other toolkits/apps, but we CAN do something about our own footprint, and we should. > also, doing the pacrunner ourselves brings what to the table? Almost just USE libproxy in the binary. > nothing other than work. libproxy already talks to neworkmanager. > pacrunner talks to connman. We'd have to also monitor these. We'd need > to pick a JS, expose functions, do testing... It's not as simple as it > looks like, having a main loop and few C helpers won't help much :-/ just use libproxy in the binary. EXACTLy like you have right now but in the efl net .so. dont REPLACE all of this and DIY. just use it. this is easy and solves the dlopen/dlsym issue by isolating into a single executable shared between all network using efl processes. NEXt step is to avoid doing this connect+exe to our own daemon by connecting to pacrunner if there (and falling back). > >> AFAIU (still trying to get more details on the working), pacrunner > >> will interact with connman's proxy configuration... I need to stop and > >> read/ask which one is the controller. But that's the ideal situation, > >> we already do connman, set property there and it spreads to the whole > >> system. > > > > the problem is 99% of users dont want or use connman. after years i've seen > > that and i don't see that binding ourselves even deeper is a good idea. > > connman works fine. i use it personally. but it's just not palatable for > > distros or users. providing our own little proxy daemon process like > > efreetd would be a self-sustained solution that is GUARANTEED to be > > installed and work in every environment. > > as per above, it's not solving any problems, just adding more. see above. it is. solves the mem footprint problem across all efl apps needing to do network access. it is a trivial amount of work (a day at best) as you just USE libproxy IN that binary. just like is there right now. it's moving the libproxy code from in-process via efl .so libs, into a "daemon" binary efl spawns and talks to. -- ------------- Codito, ergo sum - "I code, therefore I am" -------------- The Rasterman (Carsten Haitzler) ras...@rasterman.com ------------------------------------------------------------------------------ _______________________________________________ enlightenment-devel mailing list enlightenment-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/enlightenment-devel
