Re: [E-devel] Efl.Net and libproxy

[The Rasterman]

On Mon, 26 Sep 2016 13:52:55 -0300 Gustavo Sverzut Barbieri
 said:

> On Fri, Sep 23, 2016 at 11:39 PM, Carsten Haitzler 
> wrote:
> > On Wed, 21 Sep 2016 23:47:15 -0300 Gustavo Sverzut Barbieri
> >  said:
> >> >> looks good?
> >> >
> >> > no. no idle die. see above. :) it's portable. it's simple. it works
> >> > everywhere until you have oddball smack/selinux etc. envs and then the
> >> > env can take care of it via launch or socket actvation. first make the
> >> > "works everywhere" version. :)
> >>
> >> it's almost done, the libproxy.so needs to fork-exec on errors... that's
> >> it.
> >
> > why does it need to forke+exec on error?
> 
> to spawn the daemon?
> 
> error here = connect error, if that fails, spawn a new daemon

oh the "connect to daemon failed" error. ok. i was thinking other errors like
"proxy lookup failed" or "dns lookup failed" or whatever. :)

-- 
- Codito, ergo sum - "I code, therefore I am" --
The Rasterman (Carsten Haitzler)ras...@rasterman.com


--
___
enlightenment-devel mailing list
enlightenment-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/enlightenment-devel

Re: [E-devel] Efl.Net and libproxy

[Gustavo Sverzut Barbieri]

On Fri, Sep 23, 2016 at 11:39 PM, Carsten Haitzler  wrote:
> On Wed, 21 Sep 2016 23:47:15 -0300 Gustavo Sverzut Barbieri
>  said:
>> >> looks good?
>> >
>> > no. no idle die. see above. :) it's portable. it's simple. it works
>> > everywhere until you have oddball smack/selinux etc. envs and then the env
>> > can take care of it via launch or socket actvation. first make the "works
>> > everywhere" version. :)
>>
>> it's almost done, the libproxy.so needs to fork-exec on errors... that's it.
>
> why does it need to forke+exec on error?

to spawn the daemon?

error here = connect error, if that fails, spawn a new daemon

--
___
enlightenment-devel mailing list
enlightenment-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/enlightenment-devel

Re: [E-devel] Efl.Net and libproxy

[The Rasterman]

On Wed, 21 Sep 2016 23:47:15 -0300 Gustavo Sverzut Barbieri
 said:

> On Wed, Sep 21, 2016 at 8:13 PM, Carsten Haitzler 
> wrote:
> > On Wed, 21 Sep 2016 11:29:00 -0300 Gustavo Sverzut Barbieri
> >  said:
> >
> >> On Wed, Sep 21, 2016 at 9:08 AM, Carsten Haitzler 
> >> wrote:
> >> > On Tue, 20 Sep 2016 19:12:57 -0300 Gustavo Sverzut Barbieri
> >> >  said:
> >> >
> >> >> damn raster... I had to do so I could check.
> >> >>
> >> >> dlopen -> in git.
> >> >>
> >> >> server and libproxy.so wrapper, attached with the basics, I'm not
> >> >> doing all the cumbersome details to get a single process running and
> >> >> spawn it from libproxy.so wrapper without a race condition.
> >> >
> >> > there is no race.
> >> >
> >> > connect. if connect fail, spawn, set timer to connect every 0.1 sec until
> >> > successful.
> >> >
> >> > there is no race. first daemon spawned that binds to the socket wins.
> >> > every other one will fail and exit. there is no race to deal with as the
> >> > bind does the job for the daemon - first one in wins and the rest fail.
> >>
> >> there is a problem with stale sockets. If daemon dies and leaves the
> >> socket file, then next daemons will try to bind and fail... then
> >> nobody wins. If we first unlink(), then there is a race.
> >>
> >> Or should we use an abstract socket?
> >
> > or ... never die. ever. once up, stay up (of course until session shuts
> > down). unless of course it's a crash/bug/segv in which case there is no
> > unlink. :)
> 
> we never crash :-D
> 
> seriously, this is not possible.

my point is - if we crash, there is no race. the new proc just takes over the
same socket file. either way we could use abstract sockets too on linux and
real file ones everywhere else.

> >> >> I'd simplify all of that by using dbus with acquiring a name in the
> >> >> session bus and let that entity control it. Also would let the dbus
> >> >> daemon set isolation, like not inheriting current processes limits,
> >> >> namespaces and all.
> >> >
> >> > no it isn't any easier with dbus. see above. this is what efreetd does
> >> > not. its REALLyYsimple and race-free.
> >>
> >> still doesn't address inheriting caps and other access controls...
> >
> > efreetd doesnt need such caps. neither does this efl.proxy. remember how i
> > differentiated between "global resources" vs things like thumbnails where
> > thumbs are not global resources - they are directly tied tot he src file and
> > permissions on it and the permissions of the app etc.
> >
> > a "proxy lookup" daemon is going to provide a global resource - the ability
> > to take an input target machine and give a resulting proxy server to use,
> > and that global resource may be as simple as an $http_proxy env var, or as
> > complex as a bunch of js in a pac file to execute to figure out which proxy
> > to use.
> 
> that's the case, you may need to talk to some configuration daemon
> (ie: gsettings) or even the network (ie: http://wpad/wpad.dat or
> http://wpad.${searchdomain}/wpad.dat) and then you may be limited by
> some inherited capability.
> 
> then you're right the daemon and its output is global, however it may
> be launched from a specific scope and thus not delivering what it
> should.

what it looks like is you want to do systemd socket stuff FIRST then dot he
spawning. just saying that spawning is simple and universal so do it first, and
if then socket files are provided the code nicely works to avoid the spawn code
anyway.

> >> say constrained process A spawned the daemon. Then unconstrained
> >> process B will get the constraints.
> >
> > in a constrained environment like one with SMACK this daemon would launch as
> > part of your login session. such constrained environments have to deal with
> > this. they could also set up systemd socket activation too. efreetd - same
> > story. outside of such env's the "i'll launch if i can't connect" just works
> > (tm).
> 
> this is what I suggest, to launch it elsewhere. I mentioned systemd,
> you also mention systemd here... but then, right below you bash that
> option :-/ note that is only an example of a possibility, it's not
> mandated.

our FIRSt port of call should be to spawn from library THEN allow for others to
spawn (as part of session or socket activated etc.). those in such envs will
likely have systemd anyway for this or know to set up such spawning (ie the
system makers themselves not the users).

> >> more concrete description is: for some reason "A" was started without
> >> network access. The daemon inherits that and thus libproxy won't work.
> >> That's okay, expected.
> >
> > actually it'd still work because proxy daemon is doing a lookup locally as
> > to what to use. the app itself still does the connect and thus gets
> > blocked. the proxy daemon isnt meant to be doing dns lookups or anything
> > else - just return the $http_proxy to use or runt he js 

Re: [E-devel] Efl.Net and libproxy

[Gustavo Sverzut Barbieri]

On Wed, Sep 21, 2016 at 8:13 PM, Carsten Haitzler  wrote:
> On Wed, 21 Sep 2016 11:29:00 -0300 Gustavo Sverzut Barbieri
>  said:
>
>> On Wed, Sep 21, 2016 at 9:08 AM, Carsten Haitzler 
>> wrote:
>> > On Tue, 20 Sep 2016 19:12:57 -0300 Gustavo Sverzut Barbieri
>> >  said:
>> >
>> >> damn raster... I had to do so I could check.
>> >>
>> >> dlopen -> in git.
>> >>
>> >> server and libproxy.so wrapper, attached with the basics, I'm not
>> >> doing all the cumbersome details to get a single process running and
>> >> spawn it from libproxy.so wrapper without a race condition.
>> >
>> > there is no race.
>> >
>> > connect. if connect fail, spawn, set timer to connect every 0.1 sec until
>> > successful.
>> >
>> > there is no race. first daemon spawned that binds to the socket wins. every
>> > other one will fail and exit. there is no race to deal with as the bind 
>> > does
>> > the job for the daemon - first one in wins and the rest fail.
>>
>> there is a problem with stale sockets. If daemon dies and leaves the
>> socket file, then next daemons will try to bind and fail... then
>> nobody wins. If we first unlink(), then there is a race.
>>
>> Or should we use an abstract socket?
>
> or ... never die. ever. once up, stay up (of course until session shuts down).
> unless of course it's a crash/bug/segv in which case there is no unlink. :)

we never crash :-D

seriously, this is not possible.


>> >> I'd simplify all of that by using dbus with acquiring a name in the
>> >> session bus and let that entity control it. Also would let the dbus
>> >> daemon set isolation, like not inheriting current processes limits,
>> >> namespaces and all.
>> >
>> > no it isn't any easier with dbus. see above. this is what efreetd does not.
>> > its REALLyYsimple and race-free.
>>
>> still doesn't address inheriting caps and other access controls...
>
> efreetd doesnt need such caps. neither does this efl.proxy. remember how i
> differentiated between "global resources" vs things like thumbnails where
> thumbs are not global resources - they are directly tied tot he src file and
> permissions on it and the permissions of the app etc.
>
> a "proxy lookup" daemon is going to provide a global resource - the ability to
> take an input target machine and give a resulting proxy server to use, and 
> that
> global resource may be as simple as an $http_proxy env var, or as complex as a
> bunch of js in a pac file to execute to figure out which proxy to use.

that's the case, you may need to talk to some configuration daemon
(ie: gsettings) or even the network (ie: http://wpad/wpad.dat or
http://wpad.${searchdomain}/wpad.dat) and then you may be limited by
some inherited capability.

then you're right the daemon and its output is global, however it may
be launched from a specific scope and thus not delivering what it
should.



>> say constrained process A spawned the daemon. Then unconstrained
>> process B will get the constraints.
>
> in a constrained environment like one with SMACK this daemon would launch as
> part of your login session. such constrained environments have to deal with
> this. they could also set up systemd socket activation too. efreetd - same
> story. outside of such env's the "i'll launch if i can't connect" just works
> (tm).

this is what I suggest, to launch it elsewhere. I mentioned systemd,
you also mention systemd here... but then, right below you bash that
option :-/ note that is only an example of a possibility, it's not
mandated.


>> more concrete description is: for some reason "A" was started without
>> network access. The daemon inherits that and thus libproxy won't work.
>> That's okay, expected.
>
> actually it'd still work because proxy daemon is doing a lookup locally as to
> what to use. the app itself still does the connect and thus gets blocked. the
> proxy daemon isnt meant to be doing dns lookups or anything else - just return
> the $http_proxy to use or runt he js to figure out what proxy to use given a
> nice fat bit of data in the pac file to choose what may be internal and/or
> external sites by name/ip etc. and what proxy to use.

no, that may be accessing an URL in the case user choose PAC (Proxy
Auto Configuration), which is recommended in most setups.

it will then need HTTP and DNS access.

some configuration input, like for official libproxy, will need
gsettings access. Others will need dbus (ie: pacrunner). If we go
crazy and reimplement that ourselves in EFL, we still need file access
-- as you know that comes with constraints.


>> However "B" has network access and is started afterwards, it will
>> check and the daemon is there. It will use the daemon with libproxy
>> that doesn't work due lack of network access. This is not expected.
>>
>> I know not all systems employ fine grained capabilities and
>> smack/selinux, but some do and we need to be careful.
>
> such constrained environments have to 

Re: [E-devel] Efl.Net and libproxy

[The Rasterman]

On Wed, 21 Sep 2016 11:29:00 -0300 Gustavo Sverzut Barbieri
 said:

> On Wed, Sep 21, 2016 at 9:08 AM, Carsten Haitzler 
> wrote:
> > On Tue, 20 Sep 2016 19:12:57 -0300 Gustavo Sverzut Barbieri
> >  said:
> >
> >> damn raster... I had to do so I could check.
> >>
> >> dlopen -> in git.
> >>
> >> server and libproxy.so wrapper, attached with the basics, I'm not
> >> doing all the cumbersome details to get a single process running and
> >> spawn it from libproxy.so wrapper without a race condition.
> >
> > there is no race.
> >
> > connect. if connect fail, spawn, set timer to connect every 0.1 sec until
> > successful.
> >
> > there is no race. first daemon spawned that binds to the socket wins. every
> > other one will fail and exit. there is no race to deal with as the bind does
> > the job for the daemon - first one in wins and the rest fail.
> 
> there is a problem with stale sockets. If daemon dies and leaves the
> socket file, then next daemons will try to bind and fail... then
> nobody wins. If we first unlink(), then there is a race.
> 
> Or should we use an abstract socket?

or ... never die. ever. once up, stay up (of course until session shuts down).
unless of course it's a crash/bug/segv in which case there is no unlink. :)

> >> I'd simplify all of that by using dbus with acquiring a name in the
> >> session bus and let that entity control it. Also would let the dbus
> >> daemon set isolation, like not inheriting current processes limits,
> >> namespaces and all.
> >
> > no it isn't any easier with dbus. see above. this is what efreetd does not.
> > its REALLyYsimple and race-free.
> 
> still doesn't address inheriting caps and other access controls...

efreetd doesnt need such caps. neither does this efl.proxy. remember how i
differentiated between "global resources" vs things like thumbnails where
thumbs are not global resources - they are directly tied tot he src file and
permissions on it and the permissions of the app etc.

a "proxy lookup" daemon is going to provide a global resource - the ability to
take an input target machine and give a resulting proxy server to use, and that
global resource may be as simple as an $http_proxy env var, or as complex as a
bunch of js in a pac file to execute to figure out which proxy to use.

> say constrained process A spawned the daemon. Then unconstrained
> process B will get the constraints.

in a constrained environment like one with SMACK this daemon would launch as
part of your login session. such constrained environments have to deal with
this. they could also set up systemd socket activation too. efreetd - same
story. outside of such env's the "i'll launch if i can't connect" just works
(tm).

> more concrete description is: for some reason "A" was started without
> network access. The daemon inherits that and thus libproxy won't work.
> That's okay, expected.

actually it'd still work because proxy daemon is doing a lookup locally as to
what to use. the app itself still does the connect and thus gets blocked. the
proxy daemon isnt meant to be doing dns lookups or anything else - just return
the $http_proxy to use or runt he js to figure out what proxy to use given a
nice fat bit of data in the pac file to choose what may be internal and/or
external sites by name/ip etc. and what proxy to use.

> However "B" has network access and is started afterwards, it will
> check and the daemon is there. It will use the daemon with libproxy
> that doesn't work due lack of network access. This is not expected.
> 
> I know not all systems employ fine grained capabilities and
> smack/selinux, but some do and we need to be careful.

such constrained environments have to launch the proxy daemon separately. as
above. as long as they either set up socket activation OR just launch it on
login everything works as normal.

> What I can do to solve this is to allow the server to be started from
> socket activation like systemd (also in --user variant). In that case,
> if the user cares about isolation, he uses the
> efl-proxy-resolver.socket and efl-proxy-resolver.service to start
> those on demand. The client will try to connect and it will work, thus
> it won't spawn any daemon on its own. Systemd will spawn the service
> with proper caps.

and everyone on bsd, windows, osx will hate you for doing this. tying to
systemd as the ONLY and DEFAULT method is stupid. you shot portability in the
foot.

the PORTABLE way that works on osx, windows (assuming local sockets work),
bsd's, solaris, linux and EVERY *nix is what i described. connect, if connect
fails, launch, then keep connecting until success.

"odd" constrained env's can simply avoid ever needing to trigger the "launch"
code by ensuring the daemon is already started OR use socket activation.
ecore_con/ipc already supported socket activation systemd stuff if the systemd
env vars were set to pass in the socket fd. all of that is/was already done...

Re: [E-devel] Efl.Net and libproxy

[Simon Lees]

On 09/22/2016 12:02 AM, Gustavo Sverzut Barbieri wrote:
> On Wed, Sep 21, 2016 at 9:27 AM, Simon Lees  wrote:
>>
>>
>> On 09/21/2016 12:33 AM, Gustavo Sverzut Barbieri wrote:
>>> On Tue, Sep 20, 2016 at 11:43 AM, Simon Lees  wrote:


 On 09/20/2016 02:24 PM, Carsten Haitzler (The Rasterman) wrote:
> On Tue, 20 Sep 2016 01:08:48 -0300 Gustavo Sverzut Barbieri
>  said:
>
>> On Mon, Sep 19, 2016 at 8:44 PM, Carsten Haitzler 
>> wrote:
>>> On Mon, 19 Sep 2016 11:31:58 -0300 Gustavo Sverzut Barbieri
>>>  said:
>>>
 Hi all,

 Today I did commit support for libproxy in ecore_con's new API,
 Efl.Net.Dialer.Tcp, Efl.Net.Dialer.Http and Efl.Net.Dialer.Websocket
 (implicit from HTTP).
>>>
>>> ummm i just am looking now (i have a lot of catching up to do). can you 
>>> undo
>>> the changes to configure.ac and make this a dlopen/dlsym approach? well 
>>> use
>>> eina_module. look a the old curl eina_module approach. look at what i 
>>> did in
>>> ecore_audio too. in fact i need to look over all the efl net code, but 
>>> you
>>> want to use emile for ssl stuff and eina_module for curl too there too 
>>> (i
>>> haven't looked yet). there are very good reasons to do all of this.
>>
>> I've checked what you did for curl, since I use that and need more 
>> symbols.
>>
>> my question and concern are just:
>>  - not being able to compile without libproxy. If we always detect,
>> should we remove --enable-libproxy from configure.ac?
>
> yes. it moves to runtime not compile-time. several reasons:
>
> 1. makes compiling simpler. people dont have to find dependencies and 
> those
> -dev/devel pkgs too. if they add libproxy later after building efl 
> magically
> the feature works without a rebuild.

 No no no, just Never that is about the most evil thing you can do if
 there is no configure check distro's won't know that it should be pulled
 in as a dependency and you will end up with different behavior on
 different machines because one has a completely unrelated package
 installed which also installed the dependency.

 Keep the ./configure check, add a --disable-libproxy flag for those that
 don't want to go find it and sure after that dlopen it or whatever. But
 the #1 priority should be making sure that distro packagers don't screw
 it up because after all in a ideal world thats how most people should
 get efl, sure make it easy enough for anyone to compile but don't make
 it easier for distros to shoot there users in the foot by not realising
 theres a extra dependency that probably should be pulled in. Unless of
 course libproxy is not useful on a normal Linux distro and will only
 ever be used in embedded contexts.
>>>
>>> Simon, although I do understand your PoV -- packaging -- Raster is
>>> right and is concerned about runtime impact on non-users.
>>>
>>> If you link to a library, particularly one that has a huge dep graph
>>> like libproxy, you end pulling too much and impacting everyone... even
>>> if that's not used.
>>>
>>
>> So how big in filesize are we talking about? if the distro ships
>> libproxy and isn't slackware it will take care of the dependency graph
>> on its own, if the distro doesn't have it the packager / user can then
>> just pass the --disable flag when building. "Large dependency chain"
>> shouldn't really be an argument distro's handle this fine.
> 
> it's not about disk footprint, rather memory footprint.
> 
> say you use terminology without the URL preview... then why should you
> be linking with libproxy, which in turn will link with glib, mozjs...
> 
> once, if you click an URL, it will dlopen() libproxy and pull those.
> 

Ahh i'm not talking about not dlopening, I just want there to be a
configure check for the library as well as dlopening so that people are
atleast aware they should install it on there system.

-- 

Simon Lees (Simotek)http://simotek.net

Emergency Update Team   keybase.io/simotek
SUSE LinuxAdeliade Australia, UTC+9:30
GPG Fingerprint: 5B87 DB9D 88DC F606 E489 CEC5 0922 C246 02F0 014B



signature.asc
Description: OpenPGP digital signature
--
___
enlightenment-devel mailing list
enlightenment-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/enlightenment-devel

Re: [E-devel] Efl.Net and libproxy

[Gustavo Sverzut Barbieri]

On Wed, Sep 21, 2016 at 9:27 AM, Simon Lees  wrote:
>
>
> On 09/21/2016 12:33 AM, Gustavo Sverzut Barbieri wrote:
>> On Tue, Sep 20, 2016 at 11:43 AM, Simon Lees  wrote:
>>>
>>>
>>> On 09/20/2016 02:24 PM, Carsten Haitzler (The Rasterman) wrote:
 On Tue, 20 Sep 2016 01:08:48 -0300 Gustavo Sverzut Barbieri
  said:

> On Mon, Sep 19, 2016 at 8:44 PM, Carsten Haitzler 
> wrote:
>> On Mon, 19 Sep 2016 11:31:58 -0300 Gustavo Sverzut Barbieri
>>  said:
>>
>>> Hi all,
>>>
>>> Today I did commit support for libproxy in ecore_con's new API,
>>> Efl.Net.Dialer.Tcp, Efl.Net.Dialer.Http and Efl.Net.Dialer.Websocket
>>> (implicit from HTTP).
>>
>> ummm i just am looking now (i have a lot of catching up to do). can you 
>> undo
>> the changes to configure.ac and make this a dlopen/dlsym approach? well 
>> use
>> eina_module. look a the old curl eina_module approach. look at what i 
>> did in
>> ecore_audio too. in fact i need to look over all the efl net code, but 
>> you
>> want to use emile for ssl stuff and eina_module for curl too there too (i
>> haven't looked yet). there are very good reasons to do all of this.
>
> I've checked what you did for curl, since I use that and need more 
> symbols.
>
> my question and concern are just:
>  - not being able to compile without libproxy. If we always detect,
> should we remove --enable-libproxy from configure.ac?

 yes. it moves to runtime not compile-time. several reasons:

 1. makes compiling simpler. people dont have to find dependencies and those
 -dev/devel pkgs too. if they add libproxy later after building efl 
 magically
 the feature works without a rebuild.
>>>
>>> No no no, just Never that is about the most evil thing you can do if
>>> there is no configure check distro's won't know that it should be pulled
>>> in as a dependency and you will end up with different behavior on
>>> different machines because one has a completely unrelated package
>>> installed which also installed the dependency.
>>>
>>> Keep the ./configure check, add a --disable-libproxy flag for those that
>>> don't want to go find it and sure after that dlopen it or whatever. But
>>> the #1 priority should be making sure that distro packagers don't screw
>>> it up because after all in a ideal world thats how most people should
>>> get efl, sure make it easy enough for anyone to compile but don't make
>>> it easier for distros to shoot there users in the foot by not realising
>>> theres a extra dependency that probably should be pulled in. Unless of
>>> course libproxy is not useful on a normal Linux distro and will only
>>> ever be used in embedded contexts.
>>
>> Simon, although I do understand your PoV -- packaging -- Raster is
>> right and is concerned about runtime impact on non-users.
>>
>> If you link to a library, particularly one that has a huge dep graph
>> like libproxy, you end pulling too much and impacting everyone... even
>> if that's not used.
>>
>
> So how big in filesize are we talking about? if the distro ships
> libproxy and isn't slackware it will take care of the dependency graph
> on its own, if the distro doesn't have it the packager / user can then
> just pass the --disable flag when building. "Large dependency chain"
> shouldn't really be an argument distro's handle this fine.

it's not about disk footprint, rather memory footprint.

say you use terminology without the URL preview... then why should you
be linking with libproxy, which in turn will link with glib, mozjs...

once, if you click an URL, it will dlopen() libproxy and pull those.

-- 
Gustavo Sverzut Barbieri
--
Mobile: +55 (16) 99354-9890

--
___
enlightenment-devel mailing list
enlightenment-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/enlightenment-devel

Re: [E-devel] Efl.Net and libproxy

[Gustavo Sverzut Barbieri]

On Wed, Sep 21, 2016 at 9:08 AM, Carsten Haitzler  wrote:
> On Tue, 20 Sep 2016 19:12:57 -0300 Gustavo Sverzut Barbieri
>  said:
>
>> damn raster... I had to do so I could check.
>>
>> dlopen -> in git.
>>
>> server and libproxy.so wrapper, attached with the basics, I'm not
>> doing all the cumbersome details to get a single process running and
>> spawn it from libproxy.so wrapper without a race condition.
>
> there is no race.
>
> connect. if connect fail, spawn, set timer to connect every 0.1 sec until
> successful.
>
> there is no race. first daemon spawned that binds to the socket wins. every
> other one will fail and exit. there is no race to deal with as the bind does
> the job for the daemon - first one in wins and the rest fail.

there is a problem with stale sockets. If daemon dies and leaves the
socket file, then next daemons will try to bind and fail... then
nobody wins. If we first unlink(), then there is a race.

Or should we use an abstract socket?



>> I'd simplify all of that by using dbus with acquiring a name in the
>> session bus and let that entity control it. Also would let the dbus
>> daemon set isolation, like not inheriting current processes limits,
>> namespaces and all.
>
> no it isn't any easier with dbus. see above. this is what efreetd does not. 
> its
> REALLyYsimple and race-free.

still doesn't address inheriting caps and other access controls...

say constrained process A spawned the daemon. Then unconstrained
process B will get the constraints.

more concrete description is: for some reason "A" was started without
network access. The daemon inherits that and thus libproxy won't work.
That's okay, expected.

However "B" has network access and is started afterwards, it will
check and the daemon is there. It will use the daemon with libproxy
that doesn't work due lack of network access. This is not expected.

I know not all systems employ fine grained capabilities and
smack/selinux, but some do and we need to be careful.

What I can do to solve this is to allow the server to be started from
socket activation like systemd (also in --user variant). In that case,
if the user cares about isolation, he uses the
efl-proxy-resolver.socket and efl-proxy-resolver.service to start
those on demand. The client will try to connect and it will work, thus
it won't spawn any daemon on its own. Systemd will spawn the service
with proper caps.

If systemd is not used, then the client will fail to connect, will
fork-exec the daemon and it will create and bind the socket on its
own, inheriting parent's caps. A possible workaround for this is to
die after some idle timeout, then it would "auto-fix" if problems like
that happens.

looks good?



>> however as you dislike that, be my guest :-D
>>
>>  gcc -o efl-proxy-resolver libproxy-efl-server.c `pkg-config --cflags
>> --libs ecore eina efl eo libproxy-1.0`
>>  gcc -shared -o libproxy-efl.so libproxy-efl.c `pkg-config --cflags
>> eina` -lpthread
>
> why did you do both a client exe and a server? just need the server... the
> client copnnect/talk is in efl.net :) if you use efl.net like ecore_con it'd 
> do
> the XDG_RUNTIME_DIR for you for user local sockets (unless they are full 
> path).
> yuou just should have re-use efl.net inside there.

client is not an exe, it's a libproxy.so drop-in replacement so you
can LD_PRELOAD, like from E... after all you want all your processes
to use the same proxy configuration and benefits, right? VLC, glib,
qt...

-- 
Gustavo Sverzut Barbieri
--
Mobile: +55 (16) 99354-9890

--
___
enlightenment-devel mailing list
enlightenment-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/enlightenment-devel

Re: [E-devel] Efl.Net and libproxy

[Simon Lees]

On 09/21/2016 12:33 AM, Gustavo Sverzut Barbieri wrote:
> On Tue, Sep 20, 2016 at 11:43 AM, Simon Lees  wrote:
>>
>>
>> On 09/20/2016 02:24 PM, Carsten Haitzler (The Rasterman) wrote:
>>> On Tue, 20 Sep 2016 01:08:48 -0300 Gustavo Sverzut Barbieri
>>>  said:
>>>
 On Mon, Sep 19, 2016 at 8:44 PM, Carsten Haitzler 
 wrote:
> On Mon, 19 Sep 2016 11:31:58 -0300 Gustavo Sverzut Barbieri
>  said:
>
>> Hi all,
>>
>> Today I did commit support for libproxy in ecore_con's new API,
>> Efl.Net.Dialer.Tcp, Efl.Net.Dialer.Http and Efl.Net.Dialer.Websocket
>> (implicit from HTTP).
>
> ummm i just am looking now (i have a lot of catching up to do). can you 
> undo
> the changes to configure.ac and make this a dlopen/dlsym approach? well 
> use
> eina_module. look a the old curl eina_module approach. look at what i did 
> in
> ecore_audio too. in fact i need to look over all the efl net code, but you
> want to use emile for ssl stuff and eina_module for curl too there too (i
> haven't looked yet). there are very good reasons to do all of this.

 I've checked what you did for curl, since I use that and need more symbols.

 my question and concern are just:
  - not being able to compile without libproxy. If we always detect,
 should we remove --enable-libproxy from configure.ac?
>>>
>>> yes. it moves to runtime not compile-time. several reasons:
>>>
>>> 1. makes compiling simpler. people dont have to find dependencies and those
>>> -dev/devel pkgs too. if they add libproxy later after building efl magically
>>> the feature works without a rebuild.
>>
>> No no no, just Never that is about the most evil thing you can do if
>> there is no configure check distro's won't know that it should be pulled
>> in as a dependency and you will end up with different behavior on
>> different machines because one has a completely unrelated package
>> installed which also installed the dependency.
>>
>> Keep the ./configure check, add a --disable-libproxy flag for those that
>> don't want to go find it and sure after that dlopen it or whatever. But
>> the #1 priority should be making sure that distro packagers don't screw
>> it up because after all in a ideal world thats how most people should
>> get efl, sure make it easy enough for anyone to compile but don't make
>> it easier for distros to shoot there users in the foot by not realising
>> theres a extra dependency that probably should be pulled in. Unless of
>> course libproxy is not useful on a normal Linux distro and will only
>> ever be used in embedded contexts.
> 
> Simon, although I do understand your PoV -- packaging -- Raster is
> right and is concerned about runtime impact on non-users.
> 
> If you link to a library, particularly one that has a huge dep graph
> like libproxy, you end pulling too much and impacting everyone... even
> if that's not used.
> 

So how big in filesize are we talking about? if the distro ships
libproxy and isn't slackware it will take care of the dependency graph
on its own, if the distro doesn't have it the packager / user can then
just pass the --disable flag when building. "Large dependency chain"
shouldn't really be an argument distro's handle this fine.

-- 

Simon Lees (Simotek)http://simotek.net

Emergency Update Team   keybase.io/simotek
SUSE LinuxAdeliade Australia, UTC+9:30
GPG Fingerprint: 5B87 DB9D 88DC F606 E489 CEC5 0922 C246 02F0 014B



signature.asc
Description: OpenPGP digital signature
--
___
enlightenment-devel mailing list
enlightenment-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/enlightenment-devel

Re: [E-devel] Efl.Net and libproxy

[The Rasterman]

On Tue, 20 Sep 2016 19:12:57 -0300 Gustavo Sverzut Barbieri
 said:

> damn raster... I had to do so I could check.
> 
> dlopen -> in git.
> 
> server and libproxy.so wrapper, attached with the basics, I'm not
> doing all the cumbersome details to get a single process running and
> spawn it from libproxy.so wrapper without a race condition.

there is no race.

connect. if connect fail, spawn, set timer to connect every 0.1 sec until
successful.

there is no race. first daemon spawned that binds to the socket wins. every
other one will fail and exit. there is no race to deal with as the bind does
the job for the daemon - first one in wins and the rest fail.

> I'd simplify all of that by using dbus with acquiring a name in the
> session bus and let that entity control it. Also would let the dbus
> daemon set isolation, like not inheriting current processes limits,
> namespaces and all.

no it isn't any easier with dbus. see above. this is what efreetd does not. its
REALLyYsimple and race-free.

> however as you dislike that, be my guest :-D
> 
>  gcc -o efl-proxy-resolver libproxy-efl-server.c `pkg-config --cflags
> --libs ecore eina efl eo libproxy-1.0`
>  gcc -shared -o libproxy-efl.so libproxy-efl.c `pkg-config --cflags
> eina` -lpthread

why did you do both a client exe and a server? just need the server... the
client copnnect/talk is in efl.net :) if you use efl.net like ecore_con it'd do
the XDG_RUNTIME_DIR for you for user local sockets (unless they are full path).
yuou just should have re-use efl.net inside there.

> (note libproxy.so doesn't link with eina)
> 
> EINA_LOG_LEVELS=efl-proxy-resolver:4 ./efl-proxy-resolver &
> LD_PRELOAD=libproxy-efl.so proxy http://google.com
> 
> /usr/bin/proxy is a test tool provided by libproxy. the server should
> print it's serving requests.
> 
> 
> On Tue, Sep 20, 2016 at 11:59 AM, Gustavo Sverzut Barbieri
>  wrote:
> > On Tue, Sep 20, 2016 at 1:54 AM, Carsten Haitzler 
> > wrote:
> >> On Tue, 20 Sep 2016 01:08:48 -0300 Gustavo Sverzut Barbieri
> >>  said:
> >>
> >>> On Mon, Sep 19, 2016 at 8:44 PM, Carsten Haitzler 
> >>> wrote:
> >>> > On Mon, 19 Sep 2016 11:31:58 -0300 Gustavo Sverzut Barbieri
> >>> >  said:
> >>> >
> >>> >> Hi all,
> >>> >>
> >>> >> Today I did commit support for libproxy in ecore_con's new API,
> >>> >> Efl.Net.Dialer.Tcp, Efl.Net.Dialer.Http and Efl.Net.Dialer.Websocket
> >>> >> (implicit from HTTP).
> >>> >
> >>> > ummm i just am looking now (i have a lot of catching up to do). can you
> >>> > undo the changes to configure.ac and make this a dlopen/dlsym approach?
> >>> > well use eina_module. look a the old curl eina_module approach. look at
> >>> > what i did in ecore_audio too. in fact i need to look over all the efl
> >>> > net code, but you want to use emile for ssl stuff and eina_module for
> >>> > curl too there too (i haven't looked yet). there are very good reasons
> >>> > to do all of this.
> >>>
> >>> I've checked what you did for curl, since I use that and need more
> >>> symbols.
> >>>
> >>> my question and concern are just:
> >>>  - not being able to compile without libproxy. If we always detect,
> >>> should we remove --enable-libproxy from configure.ac?
> >>
> >> yes. it moves to runtime not compile-time. several reasons:
> >>
> >> 1. makes compiling simpler. people dont have to find dependencies and those
> >> -dev/devel pkgs too. if they add libproxy later after building efl
> >> magically the feature works without a rebuild.
> >> 2. speed up startup time with less linking going on for apps especially for
> >> features they may never use
> >> 3. saves memory - symbol tables and dirtying private pages to do the symbol
> >> fixups is expensive. i nuked like 320k or so before 1.18 release of actual
> >> real memory usage that was dirty pages from rarely used libraries.features.
> >> it's expensive and so only load in if/when needed to save this cost. this
> >> cost is private pages PER PROCESS using efl so it multiples and is not a
> >> one-off cost.
> >> 4. the idea of isolating such costs into a proxy daemon/process probably
> >> is a good thing and that saves adding the above cost for any process then
> >> needing to do proxy pac file parsing etc. and isolates the cost into a
> >> single daemon.
> >
> > I was asking if we should offer a way to not even try runtime detection.
> >
> >
> >>>  - I ask the above because libproxy is a monster, it will pull in a JS
> >>> loader, glib, C++... all of that, per process. If using pacrunner
> >>> (from connman guys) you can just link with libdbus, less bad.
> >>
> >> in GENERAL go for a dlopen(eina_module) style approach, BUT if a daemon
> >> makes sense, then do that.
> >
> > It's not something that excludes the other. Actually the opposite if
> > we do the proper way.
> >
> >  - efl does libproxy dlopen, so doesn't link to that library (simple
> > API but 

Re: [E-devel] Efl.Net and libproxy

[The Rasterman]

On Tue, 20 Sep 2016 11:59:45 -0300 Gustavo Sverzut Barbieri
 said:

> On Tue, Sep 20, 2016 at 1:54 AM, Carsten Haitzler 
> wrote:
> > On Tue, 20 Sep 2016 01:08:48 -0300 Gustavo Sverzut Barbieri
> >  said:
> >
> >> On Mon, Sep 19, 2016 at 8:44 PM, Carsten Haitzler 
> >> wrote:
> >> > On Mon, 19 Sep 2016 11:31:58 -0300 Gustavo Sverzut Barbieri
> >> >  said:
> >> >
> >> >> Hi all,
> >> >>
> >> >> Today I did commit support for libproxy in ecore_con's new API,
> >> >> Efl.Net.Dialer.Tcp, Efl.Net.Dialer.Http and Efl.Net.Dialer.Websocket
> >> >> (implicit from HTTP).
> >> >
> >> > ummm i just am looking now (i have a lot of catching up to do). can you
> >> > undo the changes to configure.ac and make this a dlopen/dlsym approach?
> >> > well use eina_module. look a the old curl eina_module approach. look at
> >> > what i did in ecore_audio too. in fact i need to look over all the efl
> >> > net code, but you want to use emile for ssl stuff and eina_module for
> >> > curl too there too (i haven't looked yet). there are very good reasons
> >> > to do all of this.
> >>
> >> I've checked what you did for curl, since I use that and need more symbols.
> >>
> >> my question and concern are just:
> >>  - not being able to compile without libproxy. If we always detect,
> >> should we remove --enable-libproxy from configure.ac?
> >
> > yes. it moves to runtime not compile-time. several reasons:
> >
> > 1. makes compiling simpler. people dont have to find dependencies and those
> > -dev/devel pkgs too. if they add libproxy later after building efl magically
> > the feature works without a rebuild.
> > 2. speed up startup time with less linking going on for apps especially for
> > features they may never use
> > 3. saves memory - symbol tables and dirtying private pages to do the symbol
> > fixups is expensive. i nuked like 320k or so before 1.18 release of actual
> > real memory usage that was dirty pages from rarely used libraries.features.
> > it's expensive and so only load in if/when needed to save this cost. this
> > cost is private pages PER PROCESS using efl so it multiples and is not a
> > one-off cost.
> > 4. the idea of isolating such costs into a proxy daemon/process probably is
> > a good thing and that saves adding the above cost for any process then
> > needing to do proxy pac file parsing etc. and isolates the cost into a
> > single daemon.
> 
> I was asking if we should offer a way to not even try runtime detection.

and do what instead? you mean just link libproxy into the efl daemon proxy
binary?

> >>  - I ask the above because libproxy is a monster, it will pull in a JS
> >> loader, glib, C++... all of that, per process. If using pacrunner
> >> (from connman guys) you can just link with libdbus, less bad.
> >
> > in GENERAL go for a dlopen(eina_module) style approach, BUT if a daemon
> > makes sense, then do that.
> 
> It's not something that excludes the other. Actually the opposite if
> we do the proper way.
> 
>  - efl does libproxy dlopen, so doesn't link to that library (simple
> API but heavy in dependency)

but the moment we do any network stuff we bloat out memory usage then. have 10
apps needing to do network stuff. 10x bloatage. :)

>  - efl could recommend pacrunner's libproxy until we do our own, a big
> improvement - zero work

doing our own is less work. just use libproxy IN an efl binary - just like is
there right now but in the lib... move to binary but connect to it (if connect
fails, exec then keep trying to connect every 0.1 sec until there).

>  - efl could provide its own libproxy.so, like
> /usr/lib/enlightenment/libproxy.so which would talk to our own server
> (which can be built to use system's libproxy, which in turn can be the
> official one or pacrunner's drop-in replacement). Our dlopen() can
> prefer that version.

just the above. no systemd. no dbus. nothing but a basic local unix socket
which efl.net already does. it's all there. efreetd works this way already.

>  - we can employ LD_PRELOAD tricks in Enlightenment to force loading
> our libproxy.so instead of the system. Our daemon must be started
> without it, of course :-)
> 
> 
> >it makes more sense for us to do our own daemon for efl
> > like efreet. dbus is not a good solution for this.
> 
> agreed, the API is simple so a pure unix socket can do, protocol is:
> 
> send: 
> recv:  []:proxy2>
> 
> as libproxy IS blocking and they explicitly recommend to be called
> from a thread, our code will be simple send/recv calls, no other
> libraries to pollute our users.

yup. just re-use efl.net internally to connect to an efl.net.proxy binary. if
connect fails, launch binary and set up timer to retry connect until connected.
when connected, send req just like above, with reply coming back. perfect. :)
re-use that connection for every proxy request and the daemon will deal with
it. 

Re: [E-devel] Efl.Net and libproxy

[Gustavo Sverzut Barbieri]

damn raster... I had to do so I could check.

dlopen -> in git.

server and libproxy.so wrapper, attached with the basics, I'm not
doing all the cumbersome details to get a single process running and
spawn it from libproxy.so wrapper without a race condition.

I'd simplify all of that by using dbus with acquiring a name in the
session bus and let that entity control it. Also would let the dbus
daemon set isolation, like not inheriting current processes limits,
namespaces and all.

however as you dislike that, be my guest :-D

 gcc -o efl-proxy-resolver libproxy-efl-server.c `pkg-config --cflags
--libs ecore eina efl eo libproxy-1.0`
 gcc -shared -o libproxy-efl.so libproxy-efl.c `pkg-config --cflags
eina` -lpthread

(note libproxy.so doesn't link with eina)

EINA_LOG_LEVELS=efl-proxy-resolver:4 ./efl-proxy-resolver &
LD_PRELOAD=libproxy-efl.so proxy http://google.com

/usr/bin/proxy is a test tool provided by libproxy. the server should
print it's serving requests.


On Tue, Sep 20, 2016 at 11:59 AM, Gustavo Sverzut Barbieri
 wrote:
> On Tue, Sep 20, 2016 at 1:54 AM, Carsten Haitzler  
> wrote:
>> On Tue, 20 Sep 2016 01:08:48 -0300 Gustavo Sverzut Barbieri
>>  said:
>>
>>> On Mon, Sep 19, 2016 at 8:44 PM, Carsten Haitzler 
>>> wrote:
>>> > On Mon, 19 Sep 2016 11:31:58 -0300 Gustavo Sverzut Barbieri
>>> >  said:
>>> >
>>> >> Hi all,
>>> >>
>>> >> Today I did commit support for libproxy in ecore_con's new API,
>>> >> Efl.Net.Dialer.Tcp, Efl.Net.Dialer.Http and Efl.Net.Dialer.Websocket
>>> >> (implicit from HTTP).
>>> >
>>> > ummm i just am looking now (i have a lot of catching up to do). can you 
>>> > undo
>>> > the changes to configure.ac and make this a dlopen/dlsym approach? well 
>>> > use
>>> > eina_module. look a the old curl eina_module approach. look at what i did 
>>> > in
>>> > ecore_audio too. in fact i need to look over all the efl net code, but you
>>> > want to use emile for ssl stuff and eina_module for curl too there too (i
>>> > haven't looked yet). there are very good reasons to do all of this.
>>>
>>> I've checked what you did for curl, since I use that and need more symbols.
>>>
>>> my question and concern are just:
>>>  - not being able to compile without libproxy. If we always detect,
>>> should we remove --enable-libproxy from configure.ac?
>>
>> yes. it moves to runtime not compile-time. several reasons:
>>
>> 1. makes compiling simpler. people dont have to find dependencies and those
>> -dev/devel pkgs too. if they add libproxy later after building efl magically
>> the feature works without a rebuild.
>> 2. speed up startup time with less linking going on for apps especially for
>> features they may never use
>> 3. saves memory - symbol tables and dirtying private pages to do the symbol
>> fixups is expensive. i nuked like 320k or so before 1.18 release of actual
>> real memory usage that was dirty pages from rarely used libraries.features.
>> it's expensive and so only load in if/when needed to save this cost. this 
>> cost
>> is private pages PER PROCESS using efl so it multiples and is not a one-off
>> cost.
>> 4. the idea of isolating such costs into a proxy daemon/process probably is a
>> good thing and that saves adding the above cost for any process then needing 
>> to
>> do proxy pac file parsing etc. and isolates the cost into a single daemon.
>
> I was asking if we should offer a way to not even try runtime detection.
>
>
>>>  - I ask the above because libproxy is a monster, it will pull in a JS
>>> loader, glib, C++... all of that, per process. If using pacrunner
>>> (from connman guys) you can just link with libdbus, less bad.
>>
>> in GENERAL go for a dlopen(eina_module) style approach, BUT if a daemon makes
>> sense, then do that.
>
> It's not something that excludes the other. Actually the opposite if
> we do the proper way.
>
>  - efl does libproxy dlopen, so doesn't link to that library (simple
> API but heavy in dependency)
>
>  - efl could recommend pacrunner's libproxy until we do our own, a big
> improvement - zero work
>
>  - efl could provide its own libproxy.so, like
> /usr/lib/enlightenment/libproxy.so which would talk to our own server
> (which can be built to use system's libproxy, which in turn can be the
> official one or pacrunner's drop-in replacement). Our dlopen() can
> prefer that version.
>
>  - we can employ LD_PRELOAD tricks in Enlightenment to force loading
> our libproxy.so instead of the system. Our daemon must be started
> without it, of course :-)
>
>
>>it makes more sense for us to do our own daemon for efl
>> like efreet. dbus is not a good solution for this.
>
> agreed, the API is simple so a pure unix socket can do, protocol is:
>
> send: 
> recv: 
> 
>
> as libproxy IS blocking and they explicitly recommend to be called
> from a thread, our code will be simple send/recv calls, no 

Re: [E-devel] Efl.Net and libproxy

[Gustavo Sverzut Barbieri]

On Tue, Sep 20, 2016 at 11:43 AM, Simon Lees  wrote:
>
>
> On 09/20/2016 02:24 PM, Carsten Haitzler (The Rasterman) wrote:
>> On Tue, 20 Sep 2016 01:08:48 -0300 Gustavo Sverzut Barbieri
>>  said:
>>
>>> On Mon, Sep 19, 2016 at 8:44 PM, Carsten Haitzler 
>>> wrote:
 On Mon, 19 Sep 2016 11:31:58 -0300 Gustavo Sverzut Barbieri
  said:

> Hi all,
>
> Today I did commit support for libproxy in ecore_con's new API,
> Efl.Net.Dialer.Tcp, Efl.Net.Dialer.Http and Efl.Net.Dialer.Websocket
> (implicit from HTTP).

 ummm i just am looking now (i have a lot of catching up to do). can you 
 undo
 the changes to configure.ac and make this a dlopen/dlsym approach? well use
 eina_module. look a the old curl eina_module approach. look at what i did 
 in
 ecore_audio too. in fact i need to look over all the efl net code, but you
 want to use emile for ssl stuff and eina_module for curl too there too (i
 haven't looked yet). there are very good reasons to do all of this.
>>>
>>> I've checked what you did for curl, since I use that and need more symbols.
>>>
>>> my question and concern are just:
>>>  - not being able to compile without libproxy. If we always detect,
>>> should we remove --enable-libproxy from configure.ac?
>>
>> yes. it moves to runtime not compile-time. several reasons:
>>
>> 1. makes compiling simpler. people dont have to find dependencies and those
>> -dev/devel pkgs too. if they add libproxy later after building efl magically
>> the feature works without a rebuild.
>
> No no no, just Never that is about the most evil thing you can do if
> there is no configure check distro's won't know that it should be pulled
> in as a dependency and you will end up with different behavior on
> different machines because one has a completely unrelated package
> installed which also installed the dependency.
>
> Keep the ./configure check, add a --disable-libproxy flag for those that
> don't want to go find it and sure after that dlopen it or whatever. But
> the #1 priority should be making sure that distro packagers don't screw
> it up because after all in a ideal world thats how most people should
> get efl, sure make it easy enough for anyone to compile but don't make
> it easier for distros to shoot there users in the foot by not realising
> theres a extra dependency that probably should be pulled in. Unless of
> course libproxy is not useful on a normal Linux distro and will only
> ever be used in embedded contexts.

Simon, although I do understand your PoV -- packaging -- Raster is
right and is concerned about runtime impact on non-users.

If you link to a library, particularly one that has a huge dep graph
like libproxy, you end pulling too much and impacting everyone... even
if that's not used.

Raster is approaching those as runtime optional dependencies with
manual linking. Currently this is done with libcurl and some other
audio stuff. Once needed they will be dlopen()'ed, if they are found
the feature is enabled, otherwise it will remain working without that
feature.

To solve packaging issues my suggestion is that we should document
these guys and ask the packages to add some "Recommends: libcurl,
libproxy..." instead of "Depends: libcurl, libproxy".


-- 
Gustavo Sverzut Barbieri
--
Mobile: +55 (16) 99354-9890

--
___
enlightenment-devel mailing list
enlightenment-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/enlightenment-devel

Re: [E-devel] Efl.Net and libproxy

[Gustavo Sverzut Barbieri]

On Tue, Sep 20, 2016 at 1:54 AM, Carsten Haitzler  wrote:
> On Tue, 20 Sep 2016 01:08:48 -0300 Gustavo Sverzut Barbieri
>  said:
>
>> On Mon, Sep 19, 2016 at 8:44 PM, Carsten Haitzler 
>> wrote:
>> > On Mon, 19 Sep 2016 11:31:58 -0300 Gustavo Sverzut Barbieri
>> >  said:
>> >
>> >> Hi all,
>> >>
>> >> Today I did commit support for libproxy in ecore_con's new API,
>> >> Efl.Net.Dialer.Tcp, Efl.Net.Dialer.Http and Efl.Net.Dialer.Websocket
>> >> (implicit from HTTP).
>> >
>> > ummm i just am looking now (i have a lot of catching up to do). can you 
>> > undo
>> > the changes to configure.ac and make this a dlopen/dlsym approach? well use
>> > eina_module. look a the old curl eina_module approach. look at what i did 
>> > in
>> > ecore_audio too. in fact i need to look over all the efl net code, but you
>> > want to use emile for ssl stuff and eina_module for curl too there too (i
>> > haven't looked yet). there are very good reasons to do all of this.
>>
>> I've checked what you did for curl, since I use that and need more symbols.
>>
>> my question and concern are just:
>>  - not being able to compile without libproxy. If we always detect,
>> should we remove --enable-libproxy from configure.ac?
>
> yes. it moves to runtime not compile-time. several reasons:
>
> 1. makes compiling simpler. people dont have to find dependencies and those
> -dev/devel pkgs too. if they add libproxy later after building efl magically
> the feature works without a rebuild.
> 2. speed up startup time with less linking going on for apps especially for
> features they may never use
> 3. saves memory - symbol tables and dirtying private pages to do the symbol
> fixups is expensive. i nuked like 320k or so before 1.18 release of actual
> real memory usage that was dirty pages from rarely used libraries.features.
> it's expensive and so only load in if/when needed to save this cost. this cost
> is private pages PER PROCESS using efl so it multiples and is not a one-off
> cost.
> 4. the idea of isolating such costs into a proxy daemon/process probably is a
> good thing and that saves adding the above cost for any process then needing 
> to
> do proxy pac file parsing etc. and isolates the cost into a single daemon.

I was asking if we should offer a way to not even try runtime detection.


>>  - I ask the above because libproxy is a monster, it will pull in a JS
>> loader, glib, C++... all of that, per process. If using pacrunner
>> (from connman guys) you can just link with libdbus, less bad.
>
> in GENERAL go for a dlopen(eina_module) style approach, BUT if a daemon makes
> sense, then do that.

It's not something that excludes the other. Actually the opposite if
we do the proper way.

 - efl does libproxy dlopen, so doesn't link to that library (simple
API but heavy in dependency)

 - efl could recommend pacrunner's libproxy until we do our own, a big
improvement - zero work

 - efl could provide its own libproxy.so, like
/usr/lib/enlightenment/libproxy.so which would talk to our own server
(which can be built to use system's libproxy, which in turn can be the
official one or pacrunner's drop-in replacement). Our dlopen() can
prefer that version.

 - we can employ LD_PRELOAD tricks in Enlightenment to force loading
our libproxy.so instead of the system. Our daemon must be started
without it, of course :-)


>it makes more sense for us to do our own daemon for efl
> like efreet. dbus is not a good solution for this.

agreed, the API is simple so a pure unix socket can do, protocol is:

send: 
recv: 


as libproxy IS blocking and they explicitly recommend to be called
from a thread, our code will be simple send/recv calls, no other
libraries to pollute our users.



> supporting pacrunner later
> of course is doable, but i would not do this out of the box because pacrunner
> is not everywhere, dbus session buses do not exist when you do things like use
> the console (terminology using fbcon is a major problem when it comes to 
> ethumb
> and was for efreet because no session bus is active thus everything bound to a
> session bus is broken). i will eventually rewrite ethumb features likely in 
> elm
> and drop ethumb because of such issues. also ethumb design is broken for SMACK
> systems so we really do need a "fork off a slave process that is custom for
> your app only" method of working.

you know my opinion on this: this is a nasty approach. The dbus is
simple to achieve if you start that, as you do in E if there was no
session. Or use systemd, it should be doing that from pam_systemd.

SMACK is a different issue and the design of Ethumb came from a time
where we expected all processes to share thumbnails, like efm, your
elm file selector, etc. Private thumbnails as you mentioned started to
gain traction later... application isolation as well, it began with
iOS/Android, linux desktop still is 

Re: [E-devel] Efl.Net and libproxy

[Simon Lees]

On 09/20/2016 02:24 PM, Carsten Haitzler (The Rasterman) wrote:
> On Tue, 20 Sep 2016 01:08:48 -0300 Gustavo Sverzut Barbieri
>  said:
> 
>> On Mon, Sep 19, 2016 at 8:44 PM, Carsten Haitzler 
>> wrote:
>>> On Mon, 19 Sep 2016 11:31:58 -0300 Gustavo Sverzut Barbieri
>>>  said:
>>>
 Hi all,

 Today I did commit support for libproxy in ecore_con's new API,
 Efl.Net.Dialer.Tcp, Efl.Net.Dialer.Http and Efl.Net.Dialer.Websocket
 (implicit from HTTP).
>>>
>>> ummm i just am looking now (i have a lot of catching up to do). can you undo
>>> the changes to configure.ac and make this a dlopen/dlsym approach? well use
>>> eina_module. look a the old curl eina_module approach. look at what i did in
>>> ecore_audio too. in fact i need to look over all the efl net code, but you
>>> want to use emile for ssl stuff and eina_module for curl too there too (i
>>> haven't looked yet). there are very good reasons to do all of this.
>>
>> I've checked what you did for curl, since I use that and need more symbols.
>>
>> my question and concern are just:
>>  - not being able to compile without libproxy. If we always detect,
>> should we remove --enable-libproxy from configure.ac?
> 
> yes. it moves to runtime not compile-time. several reasons:
> 
> 1. makes compiling simpler. people dont have to find dependencies and those
> -dev/devel pkgs too. if they add libproxy later after building efl magically
> the feature works without a rebuild.

No no no, just Never that is about the most evil thing you can do if
there is no configure check distro's won't know that it should be pulled
in as a dependency and you will end up with different behavior on
different machines because one has a completely unrelated package
installed which also installed the dependency.

Keep the ./configure check, add a --disable-libproxy flag for those that
don't want to go find it and sure after that dlopen it or whatever. But
the #1 priority should be making sure that distro packagers don't screw
it up because after all in a ideal world thats how most people should
get efl, sure make it easy enough for anyone to compile but don't make
it easier for distros to shoot there users in the foot by not realising
theres a extra dependency that probably should be pulled in. Unless of
course libproxy is not useful on a normal Linux distro and will only
ever be used in embedded contexts.

-- 

Simon Lees (Simotek)http://simotek.net

Emergency Update Team   keybase.io/simotek
SUSE LinuxAdeliade Australia, UTC+9:30
GPG Fingerprint: 5B87 DB9D 88DC F606 E489 CEC5 0922 C246 02F0 014B



signature.asc
Description: OpenPGP digital signature
--
___
enlightenment-devel mailing list
enlightenment-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/enlightenment-devel

Re: [E-devel] Efl.Net and libproxy

[The Rasterman]

On Tue, 20 Sep 2016 01:08:48 -0300 Gustavo Sverzut Barbieri
 said:

> On Mon, Sep 19, 2016 at 8:44 PM, Carsten Haitzler 
> wrote:
> > On Mon, 19 Sep 2016 11:31:58 -0300 Gustavo Sverzut Barbieri
> >  said:
> >
> >> Hi all,
> >>
> >> Today I did commit support for libproxy in ecore_con's new API,
> >> Efl.Net.Dialer.Tcp, Efl.Net.Dialer.Http and Efl.Net.Dialer.Websocket
> >> (implicit from HTTP).
> >
> > ummm i just am looking now (i have a lot of catching up to do). can you undo
> > the changes to configure.ac and make this a dlopen/dlsym approach? well use
> > eina_module. look a the old curl eina_module approach. look at what i did in
> > ecore_audio too. in fact i need to look over all the efl net code, but you
> > want to use emile for ssl stuff and eina_module for curl too there too (i
> > haven't looked yet). there are very good reasons to do all of this.
> 
> I've checked what you did for curl, since I use that and need more symbols.
> 
> my question and concern are just:
>  - not being able to compile without libproxy. If we always detect,
> should we remove --enable-libproxy from configure.ac?

yes. it moves to runtime not compile-time. several reasons:

1. makes compiling simpler. people dont have to find dependencies and those
-dev/devel pkgs too. if they add libproxy later after building efl magically
the feature works without a rebuild.
2. speed up startup time with less linking going on for apps especially for
features they may never use
3. saves memory - symbol tables and dirtying private pages to do the symbol
fixups is expensive. i nuked like 320k or so before 1.18 release of actual
real memory usage that was dirty pages from rarely used libraries.features.
it's expensive and so only load in if/when needed to save this cost. this cost
is private pages PER PROCESS using efl so it multiples and is not a one-off
cost.
4. the idea of isolating such costs into a proxy daemon/process probably is a
good thing and that saves adding the above cost for any process then needing to
do proxy pac file parsing etc. and isolates the cost into a single daemon.

>  - I ask the above because libproxy is a monster, it will pull in a JS
> loader, glib, C++... all of that, per process. If using pacrunner
> (from connman guys) you can just link with libdbus, less bad.

in GENERAL go for a dlopen(eina_module) style approach, BUT if a daemon makes
sense, then do that. it makes more sense for us to do our own daemon for efl
like efreet. dbus is not a good solution for this. supporting pacrunner later
of course is doable, but i would not do this out of the box because pacrunner
is not everywhere, dbus session buses do not exist when you do things like use
the console (terminology using fbcon is a major problem when it comes to ethumb
and was for efreet because no session bus is active thus everything bound to a
session bus is broken). i will eventually rewrite ethumb features likely in elm
and drop ethumb because of such issues. also ethumb design is broken for SMACK
systems so we really do need a "fork off a slave process that is custom for
your app only" method of working.

for proxies i see the value of a single efl.net.proxy daemon service to serve
as a proxy handler - send relevant info to this daemon, let it decide and then
answer back with what connection to make. so basically take the libproxy stuff
you just committed, move it to a daemon (like efreet does now), and connect to
it (execute then connect if connect fails). if you don't, then i'll eventually
need to do this.

keep this in mind as a design point. for data that is "global" (efreet style
stuff, proxy info etc.) then a single daemon doing the work will be the best
solution. maybe in future we might/can merge these daemons into a single "efl
daemon".

> > emile too needs to switch to using this approach internally as well.
> >
> >> Libproxy (https://libproxy.github.io) tries to dynamically find a
> >> proxy for a given URL, unlike plain $http_proxy, $no_proxy... which
> >> needs a mess to get right in complex environments such as big
> >> corporations. It will also use PAC - ProxyAutoConfiguration, those
> >> JavaScript like files named wpad.dat, as well as the wpad host on the
> >> local network, where it tries to find the JS file.
> >>
> >> More than that, libproxy uses the configuration from Desktop
> >> Environments. There are none for E (some volunteer?), but we can use
> >> the one from Gnome3/Gnome or KDE. It will also handle envvars and some
> >> /etc/sysconf configuration.
> >
> > what kind of config? you mean like a dbus interface exposed from e to
> > provide proxy info? i haven't looked...
> 
> whatever you want, like
> https://github.com/libproxy/libproxy/tree/master/libproxy/modules
> 
> they have config_envvars that check http_proxy, no_proxy... but they
> also have one for gnome that uses gsettings.

err so you mean write code into libproxy to get proxy settings from e? a 

Re: [E-devel] Efl.Net and libproxy

[Gustavo Sverzut Barbieri]

On Mon, Sep 19, 2016 at 8:44 PM, Carsten Haitzler  wrote:
> On Mon, 19 Sep 2016 11:31:58 -0300 Gustavo Sverzut Barbieri
>  said:
>
>> Hi all,
>>
>> Today I did commit support for libproxy in ecore_con's new API,
>> Efl.Net.Dialer.Tcp, Efl.Net.Dialer.Http and Efl.Net.Dialer.Websocket
>> (implicit from HTTP).
>
> ummm i just am looking now (i have a lot of catching up to do). can you undo
> the changes to configure.ac and make this a dlopen/dlsym approach? well use
> eina_module. look a the old curl eina_module approach. look at what i did in
> ecore_audio too. in fact i need to look over all the efl net code, but you 
> want
> to use emile for ssl stuff and eina_module for curl too there too (i haven't
> looked yet). there are very good reasons to do all of this.

I've checked what you did for curl, since I use that and need more symbols.

my question and concern are just:
 - not being able to compile without libproxy. If we always detect,
should we remove --enable-libproxy from configure.ac?

 - I ask the above because libproxy is a monster, it will pull in a JS
loader, glib, C++... all of that, per process. If using pacrunner
(from connman guys) you can just link with libdbus, less bad.





> emile too needs to switch to using this approach internally as well.
>
>> Libproxy (https://libproxy.github.io) tries to dynamically find a
>> proxy for a given URL, unlike plain $http_proxy, $no_proxy... which
>> needs a mess to get right in complex environments such as big
>> corporations. It will also use PAC - ProxyAutoConfiguration, those
>> JavaScript like files named wpad.dat, as well as the wpad host on the
>> local network, where it tries to find the JS file.
>>
>> More than that, libproxy uses the configuration from Desktop
>> Environments. There are none for E (some volunteer?), but we can use
>> the one from Gnome3/Gnome or KDE. It will also handle envvars and some
>> /etc/sysconf configuration.
>
> what kind of config? you mean like a dbus interface exposed from e to provide
> proxy info? i haven't looked...

whatever you want, like
https://github.com/libproxy/libproxy/tree/master/libproxy/modules

they have config_envvars that check http_proxy, no_proxy... but they
also have one for gnome that uses gsettings.


>> Just be aware that libproxy is nasty and will load many crap in your
>> application binary, including mozjs to interpret JS mentioned above.
>
> ugh. i know. pac files. js in there. ugh. it's times like this that all the
> "web tech crowd" makes me want to cry. it's like they never cared about
> performance... ever. just stuff some js anywhere because we have a browser
> using it already and hey whats an extra few mb to something already needing 
> 1gb
> to run... :( oh wait1 we already have js spinning so it's no cost to us! argh!

indeed. At least pacrunner isolates this in their daemon, not every
process. And the file is shared... not to say they picked a light JS
engine (duktape) as the wpad.dat is super-simple.


>> A better solution is to use libproxy replacement from pacrunner
>> (http://git.kernel.org/cgit/network/connman/pacrunner.git/), that one
>> will only pull in libdbus and the JS is done at the daemon, which also
>> keeps shared cache not replicating it in all processes.
>
> much better, OR we even spin off a process - let it execute as a child, read
> back results in parent (and cache them until proxy data changes).

less bad, but still one per process doing networking.


>> Be traditional libproxy or pacrunner's alternative, the current code
>> should be more dynamic, however needs more testing. If you work inside
>> a proxy, give it a try and report problems, you can use the
>> src/examples/ecore/efl_io_copier or efl_net_dialer_http_example to
>> test.
>
> we do indeed live inside one at work. and there is the PAC file from hell...
>
>> It's important to get more testing as in near future I'm converting
>> parts of E/EFL that used ecore_con to use this and I don't want to
>> break things :-)
>
> please please... don't race! you bring up very important things above. also
> dlopen the sucker. don't make it a configure option + link option. runtime 
> link
> as needed. duplicate whatever elements of the libproxy headers needed to talk
> to it.

I can do this very easily, 3 functions, no public struct, just returns
a char **.


> we need to do much more of this for "rarely used dependencies". unless it's
> needed by like 80%+ of apps, then it should be dlopen'd instead of linked.
> linking is EXPENSIVE.
>
> also the issue running the PAC files... it makes sense to ecore_exe() a slave
> and have it figure it out once and store the result (ok ok PAC files
> technically have to be re-executed per connect because they can switch per
> target... urgh). we COULd do it like efreet with a daemon approach for now. we
> can support pacrunner as a daemon later too and avoid running our own. in fact
> this is likely the best approach - do it the 

Re: [E-devel] Efl.Net and libproxy

[The Rasterman]

On Mon, 19 Sep 2016 11:31:58 -0300 Gustavo Sverzut Barbieri
 said:

> Hi all,
> 
> Today I did commit support for libproxy in ecore_con's new API,
> Efl.Net.Dialer.Tcp, Efl.Net.Dialer.Http and Efl.Net.Dialer.Websocket
> (implicit from HTTP).

ummm i just am looking now (i have a lot of catching up to do). can you undo
the changes to configure.ac and make this a dlopen/dlsym approach? well use
eina_module. look a the old curl eina_module approach. look at what i did in
ecore_audio too. in fact i need to look over all the efl net code, but you want
to use emile for ssl stuff and eina_module for curl too there too (i haven't
looked yet). there are very good reasons to do all of this.

emile too needs to switch to using this approach internally as well.

> Libproxy (https://libproxy.github.io) tries to dynamically find a
> proxy for a given URL, unlike plain $http_proxy, $no_proxy... which
> needs a mess to get right in complex environments such as big
> corporations. It will also use PAC - ProxyAutoConfiguration, those
> JavaScript like files named wpad.dat, as well as the wpad host on the
> local network, where it tries to find the JS file.
> 
> More than that, libproxy uses the configuration from Desktop
> Environments. There are none for E (some volunteer?), but we can use
> the one from Gnome3/Gnome or KDE. It will also handle envvars and some
> /etc/sysconf configuration.

what kind of config? you mean like a dbus interface exposed from e to provide
proxy info? i haven't looked...

> Just be aware that libproxy is nasty and will load many crap in your
> application binary, including mozjs to interpret JS mentioned above.

ugh. i know. pac files. js in there. ugh. it's times like this that all the
"web tech crowd" makes me want to cry. it's like they never cared about
performance... ever. just stuff some js anywhere because we have a browser
using it already and hey whats an extra few mb to something already needing 1gb
to run... :( oh wait1 we already have js spinning so it's no cost to us! argh!

> A better solution is to use libproxy replacement from pacrunner
> (http://git.kernel.org/cgit/network/connman/pacrunner.git/), that one
> will only pull in libdbus and the JS is done at the daemon, which also
> keeps shared cache not replicating it in all processes.

much better, OR we even spin off a process - let it execute as a child, read
back results in parent (and cache them until proxy data changes).

> Be traditional libproxy or pacrunner's alternative, the current code
> should be more dynamic, however needs more testing. If you work inside
> a proxy, give it a try and report problems, you can use the
> src/examples/ecore/efl_io_copier or efl_net_dialer_http_example to
> test.

we do indeed live inside one at work. and there is the PAC file from hell...

> It's important to get more testing as in near future I'm converting
> parts of E/EFL that used ecore_con to use this and I don't want to
> break things :-)

please please... don't race! you bring up very important things above. also
dlopen the sucker. don't make it a configure option + link option. runtime link
as needed. duplicate whatever elements of the libproxy headers needed to talk
to it.

we need to do much more of this for "rarely used dependencies". unless it's
needed by like 80%+ of apps, then it should be dlopen'd instead of linked.
linking is EXPENSIVE.

also the issue running the PAC files... it makes sense to ecore_exe() a slave
and have it figure it out once and store the result (ok ok PAC files
technically have to be re-executed per connect because they can switch per
target... urgh). we COULd do it like efreet with a daemon approach for now. we
can support pacrunner as a daemon later too and avoid running our own. in fact
this is likely the best approach - do it the efreetd way. no no. do not use
dbus. yes i know it does activation for us. no. this is far more trouble, but i
won't get into that here. just connect to a local user ipc service of our own,
if not there, spawn the daemon and try connecting again a bit later. repeat
until connect succeeded. do it in background async.

> -- 
> Gustavo Sverzut Barbieri
> --
> Mobile: +55 (16) 99354-9890
> 
> --
> ___
> enlightenment-devel mailing list
> enlightenment-devel@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/enlightenment-devel
> 


-- 
- Codito, ergo sum - "I code, therefore I am" --
The Rasterman (Carsten Haitzler)ras...@rasterman.com


--
___
enlightenment-devel mailing list
enlightenment-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/enlightenment-devel

Re: [E-devel] Efl.Net and libproxy

[Gustavo Sverzut Barbieri]

On Mon, Sep 19, 2016 at 11:31 AM, Gustavo Sverzut Barbieri
 wrote:
> Hi all,
>
> Today I did commit support for libproxy in ecore_con's new API,
> Efl.Net.Dialer.Tcp, Efl.Net.Dialer.Http and Efl.Net.Dialer.Websocket
> (implicit from HTTP).
>
> Libproxy (https://libproxy.github.io) tries to dynamically find a
> proxy for a given URL, unlike plain $http_proxy, $no_proxy... which
> needs a mess to get right in complex environments such as big
> corporations. It will also use PAC - ProxyAutoConfiguration, those
> JavaScript like files named wpad.dat, as well as the wpad host on the
> local network, where it tries to find the JS file.
>
> More than that, libproxy uses the configuration from Desktop
> Environments. There are none for E (some volunteer?), but we can use
> the one from Gnome3/Gnome or KDE. It will also handle envvars and some
> /etc/sysconf configuration.
>
> Just be aware that libproxy is nasty and will load many crap in your
> application binary, including mozjs to interpret JS mentioned above.
>
> A better solution is to use libproxy replacement from pacrunner
> (http://git.kernel.org/cgit/network/connman/pacrunner.git/), that one
> will only pull in libdbus and the JS is done at the daemon, which also
> keeps shared cache not replicating it in all processes.
>
> Be traditional libproxy or pacrunner's alternative, the current code
> should be more dynamic, however needs more testing. If you work inside
> a proxy, give it a try and report problems, you can use the
> src/examples/ecore/efl_io_copier or efl_net_dialer_http_example to
> test.
>
> It's important to get more testing as in near future I'm converting
> parts of E/EFL that used ecore_con to use this and I don't want to
> break things :-)

Ah, and I forgot to mention that pacrunner is developed by the same
team as connman, so uses its configuration... all nicely integrated
:-)


-- 
Gustavo Sverzut Barbieri
--
Mobile: +55 (16) 99354-9890

--
___
enlightenment-devel mailing list
enlightenment-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/enlightenment-devel

[E-devel] Efl.Net and libproxy

[Gustavo Sverzut Barbieri]

Hi all,

Today I did commit support for libproxy in ecore_con's new API,
Efl.Net.Dialer.Tcp, Efl.Net.Dialer.Http and Efl.Net.Dialer.Websocket
(implicit from HTTP).

Libproxy (https://libproxy.github.io) tries to dynamically find a
proxy for a given URL, unlike plain $http_proxy, $no_proxy... which
needs a mess to get right in complex environments such as big
corporations. It will also use PAC - ProxyAutoConfiguration, those
JavaScript like files named wpad.dat, as well as the wpad host on the
local network, where it tries to find the JS file.

More than that, libproxy uses the configuration from Desktop
Environments. There are none for E (some volunteer?), but we can use
the one from Gnome3/Gnome or KDE. It will also handle envvars and some
/etc/sysconf configuration.

Just be aware that libproxy is nasty and will load many crap in your
application binary, including mozjs to interpret JS mentioned above.

A better solution is to use libproxy replacement from pacrunner
(http://git.kernel.org/cgit/network/connman/pacrunner.git/), that one
will only pull in libdbus and the JS is done at the daemon, which also
keeps shared cache not replicating it in all processes.

Be traditional libproxy or pacrunner's alternative, the current code
should be more dynamic, however needs more testing. If you work inside
a proxy, give it a try and report problems, you can use the
src/examples/ecore/efl_io_copier or efl_net_dialer_http_example to
test.

It's important to get more testing as in near future I'm converting
parts of E/EFL that used ecore_con to use this and I don't want to
break things :-)


-- 
Gustavo Sverzut Barbieri
--
Mobile: +55 (16) 99354-9890

--
___
enlightenment-devel mailing list
enlightenment-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/enlightenment-devel