Re: [systemd-devel] [RFC] Initial libsystemd-asyncns commit

[Daniel Buch]

Yes im still working on it (Should have been finished days ago. 24 hours a
day is again not enough -.-). So far i went for the threaded version, and
cleaned up stuff according to this mail thread.

And i updated it to match systemd with c99 null initialisation of structs
etc. That might be wrong since you want it to be public now?


2014/1/3 David Timothy Strauss 

> Just to consider what other folks are doing, I know Fedora builds
> libcurl with a thread-isolated, NSS-based resolver.
>
> On a less-related note, at Pantheon improve DNS performance on servers
> by setting resolv.conf to localhost and running Unbound there. Unbound
> then uses the datacenter's recursive DNS servers for things that miss
> the local cache. This minimizes the time spent in blocked threads --
> and waiting for lookups even with async libraries. As a bonus, you
> also get DNSSec validation when possible.
> ___
> systemd-devel mailing list
> systemd-devel@lists.freedesktop.org
> http://lists.freedesktop.org/mailman/listinfo/systemd-devel
>
___
systemd-devel mailing list
systemd-devel@lists.freedesktop.org
http://lists.freedesktop.org/mailman/listinfo/systemd-devel

Re: [systemd-devel] [RFC] Initial libsystemd-asyncns commit

[David Timothy Strauss]

Just to consider what other folks are doing, I know Fedora builds
libcurl with a thread-isolated, NSS-based resolver.

On a less-related note, at Pantheon improve DNS performance on servers
by setting resolv.conf to localhost and running Unbound there. Unbound
then uses the datacenter's recursive DNS servers for things that miss
the local cache. This minimizes the time spent in blocked threads --
and waiting for lookups even with async libraries. As a bonus, you
also get DNSSec validation when possible.
___
systemd-devel mailing list
systemd-devel@lists.freedesktop.org
http://lists.freedesktop.org/mailman/listinfo/systemd-devel

Re: [systemd-devel] [RFC] Initial libsystemd-asyncns commit

[Kay Sievers]

On Thu, Jan 2, 2014 at 10:53 PM, Tom Gundersen  wrote:
> On Wed, Dec 11, 2013 at 2:41 AM, Lennart Poettering  
> wrote:

>> To follow the naming scheme of the other libs we should probably call
>> this "sd-asyncs.c", and the header file should be in src/systemd/
>> together with the other public APIs.
>
> Hm, the fact that this is async is sort of a detail, is it not? I
> mean, all our libs are async... How about sd-dns? It is slightly
> wrong, but sd-ns could mean anything...

libsystemd-dns sounds good to me.

>> Right now I am more leaning towards the thread option. But maybe that
>> changes tomorrow again. Other opinions? Kay?
>>
>> glib is embedding a copy of libasyncns currently, they opted for the
>> thread solution. Maybe we should follow suit.
>
> Doesn't seem to be any strong arguments either way... So shall we just
> go with pthreads, if that's what's used elsewhere...?

Yeah, a thread to call into the nss stuff sound like the better option
than a worker process.

Kay
___
systemd-devel mailing list
systemd-devel@lists.freedesktop.org
http://lists.freedesktop.org/mailman/listinfo/systemd-devel

Re: [systemd-devel] [RFC] Initial libsystemd-asyncns commit

[Tom Gundersen]

On Thu, Jan 2, 2014 at 10:53 PM, Tom Gundersen  wrote:
> On Wed, Dec 11, 2013 at 2:41 AM, Lennart Poettering
>  wrote:
>> On Wed, 11.12.13 02:13, Daniel Buch (boogiewasth...@gmail.com) wrote:
>>
>> Heya,
>>
>> Hmm, so thinking about it I have the suspicion this should probably be
>> linked into libsystemd-bus, and thus live in src/libsystemd-bus/. The
>> reason for this is cyclic deps: libsystemd-bus really should make use of
>> this to resolve host names when used in tcp mode. However, this stuff
>> really should come with integration into sd-event out of the box too
>> which is already part of libsystemd-bus (for similar reasons). So we'd
>> have asyncns both as consumer of APIs of libsystemd-bus and as
>> provider of APIs to it.
>>
>> Of course, one could argue that DNS is hardly part of bus access but
>> then again, sd-event is neither. Maybe we should just start treating
>> libsystemd-bus as that library where everything we expose ends up in,
>> except when it is really clear that it's only consumer, never provider
>> to stuff in libsystemd-bus.
>>
>>> Reindentation is done to fit systemd
>>> ---
>>>  Makefile.am   |   23 +
>>>  src/libsystemd-asyncns/asyncns.c  | 1513 
>>> +
>>>  src/libsystemd-asyncns/asyncns.h  |  163 
>>>  src/libsystemd-asyncns/test-asyncns.c |  178 
>>>  4 files changed, 1877 insertions(+)
>>>  create mode 100644 src/libsystemd-asyncns/asyncns.c
>>>  create mode 100644 src/libsystemd-asyncns/asyncns.h
>>>  create mode 100644 src/libsystemd-asyncns/test-asyncns.c
>>>
>>> diff --git a/Makefile.am b/Makefile.am
>>> index 19da6ea..a0564b5 100644
>>> --- a/Makefile.am
>>> +++ b/Makefile.am
>>> @@ -659,6 +659,29 @@ tests += test-rtnl
>>>
>>>  # 
>>> --
>>>  noinst_LTLIBRARIES += \
>>> + libsystemd-asyncns.la
>>> +
>>> +libsystemd_asyncns_la_SOURCES = \
>>> + src/libsystemd-asyncns/asyncns.c \
>>> + src/libsystemd-asyncns/asyncns.h
>>
>> To follow the naming scheme of the other libs we should probably call
>> this "sd-asyncs.c", and the header file should be in src/systemd/
>> together with the other public APIs.
>
> Hm, the fact that this is async is sort of a detail, is it not? I
> mean, all our libs are async... How about sd-dns? It is slightly
> wrong, but sd-ns could mean anything...
>
>>> +
>>> +libsystemd_asyncns_la_CFLAGS = \
>>> + -pthread
>>> +
>>> +test_asyncns_SOURCES = \
>>> + src/libsystemd-asyncns/test-asyncns.c
>>> +
>>> +test_asyncns_LDADD = \
>>> + libsystemd-asyncns.la
>>> +
>>> +test_asyncns_LDFLAGS = \
>>> + -lresolv \
>>> + -pthread
>>
>> I figure the -lresolv thing will eventually need a configure.ac check,
>> but for now I'd just leave it like this, when this breaks for people I
>> am sure they will cook up a patch quickly...
>>
>>> +#ifdef HAVE_CONFIG_H
>>> +#include 
>>> +#endif
>>
>> Unnecessary, we do this on the gcc cmdline implicitly far all our files...
>>
>>> +
>>> +/* #undef HAVE_PTHREAD */
>>> +
>>> +#include 
>>> +#include 
>>> +#include 
>>> +#include 
>>> +#include 
>>> +#include 
>>> +#include 
>>> +#include 
>>> +#include 
>>> +#include 
>>> +#include 
>>> +#include 
>>> +#include 
>>> +#include 
>>> +#include 
>>> +#include 
>>> +#include 
>>> +#include 
>>> +#include 
>>> +
>>> +#ifdef HAVE_SYS_PRCTL_H
>>> +#include 
>>> +#endif
>>
>> This file isn't optional for us...
>>
>>> +
>>> +#if HAVE_PTHREAD
>>> +#include 
>>> +#endif
>>> +
>>> +#include "asyncns.h"
>>> +
>>> +#ifndef MSG_NOSIGNAL
>>> +#define MSG_NOSIGNAL 0
>>> +#endif
>>
>> This is non-Linux compat and should go i figure...
>>
>>> +struct asyncns {
>>
>> probably should be called sd_asyncns now...
>>
>>> +int fds[MESSAGE_FD_MAX];
>>> +
>>> +#ifndef HAVE_PTHREAD
>>> +pid_t workers[MAX_WORKERS];
>>> +#elseso, you just suggested pthreads is the way to go. any strong reasons?
>>> +pthread_t workers[MAX_WORKERS];
>>> +#endif
>>
>> We should probably stick to one implementation here... Either with
>> process or with threads, but not chicken out anymore like I did
>> before...
>>
>> Every other day I come to different conclusions which one would be the
>> one and only mode we should support.
>>
>> The big advantage of the process option is that things are neatly
>> isolated from the parent. THe big disadvantages are that it pollutes the
>> "ps" output and might generate page faults in the parent all the time,
>> since we fork and don't exec().
>>
>> Right now I am more leaning towards the thread option. But maybe that
>> changes tomorrow again. Other opinions? Kay?
>>
>> glib is embedding a copy of libasyncns currently, they opted for the
>> thread solution. Maybe we should follow suit.
>
> Doesn't seem to be any strong arguments either way... So shall we just
> go with pthreads, if that's what's used elsewhere...?

Kay says:
> yeah, a thread. seems natural, we do not share data with the main

Re: [systemd-devel] [RFC] Initial libsystemd-asyncns commit

[Tom Gundersen]

On Wed, Dec 11, 2013 at 2:41 AM, Lennart Poettering
 wrote:
> On Wed, 11.12.13 02:13, Daniel Buch (boogiewasth...@gmail.com) wrote:
>
> Heya,
>
> Hmm, so thinking about it I have the suspicion this should probably be
> linked into libsystemd-bus, and thus live in src/libsystemd-bus/. The
> reason for this is cyclic deps: libsystemd-bus really should make use of
> this to resolve host names when used in tcp mode. However, this stuff
> really should come with integration into sd-event out of the box too
> which is already part of libsystemd-bus (for similar reasons). So we'd
> have asyncns both as consumer of APIs of libsystemd-bus and as
> provider of APIs to it.
>
> Of course, one could argue that DNS is hardly part of bus access but
> then again, sd-event is neither. Maybe we should just start treating
> libsystemd-bus as that library where everything we expose ends up in,
> except when it is really clear that it's only consumer, never provider
> to stuff in libsystemd-bus.
>
>> Reindentation is done to fit systemd
>> ---
>>  Makefile.am   |   23 +
>>  src/libsystemd-asyncns/asyncns.c  | 1513 
>> +
>>  src/libsystemd-asyncns/asyncns.h  |  163 
>>  src/libsystemd-asyncns/test-asyncns.c |  178 
>>  4 files changed, 1877 insertions(+)
>>  create mode 100644 src/libsystemd-asyncns/asyncns.c
>>  create mode 100644 src/libsystemd-asyncns/asyncns.h
>>  create mode 100644 src/libsystemd-asyncns/test-asyncns.c
>>
>> diff --git a/Makefile.am b/Makefile.am
>> index 19da6ea..a0564b5 100644
>> --- a/Makefile.am
>> +++ b/Makefile.am
>> @@ -659,6 +659,29 @@ tests += test-rtnl
>>
>>  # 
>> --
>>  noinst_LTLIBRARIES += \
>> + libsystemd-asyncns.la
>> +
>> +libsystemd_asyncns_la_SOURCES = \
>> + src/libsystemd-asyncns/asyncns.c \
>> + src/libsystemd-asyncns/asyncns.h
>
> To follow the naming scheme of the other libs we should probably call
> this "sd-asyncs.c", and the header file should be in src/systemd/
> together with the other public APIs.

Hm, the fact that this is async is sort of a detail, is it not? I
mean, all our libs are async... How about sd-dns? It is slightly
wrong, but sd-ns could mean anything...

>> +
>> +libsystemd_asyncns_la_CFLAGS = \
>> + -pthread
>> +
>> +test_asyncns_SOURCES = \
>> + src/libsystemd-asyncns/test-asyncns.c
>> +
>> +test_asyncns_LDADD = \
>> + libsystemd-asyncns.la
>> +
>> +test_asyncns_LDFLAGS = \
>> + -lresolv \
>> + -pthread
>
> I figure the -lresolv thing will eventually need a configure.ac check,
> but for now I'd just leave it like this, when this breaks for people I
> am sure they will cook up a patch quickly...
>
>> +#ifdef HAVE_CONFIG_H
>> +#include 
>> +#endif
>
> Unnecessary, we do this on the gcc cmdline implicitly far all our files...
>
>> +
>> +/* #undef HAVE_PTHREAD */
>> +
>> +#include 
>> +#include 
>> +#include 
>> +#include 
>> +#include 
>> +#include 
>> +#include 
>> +#include 
>> +#include 
>> +#include 
>> +#include 
>> +#include 
>> +#include 
>> +#include 
>> +#include 
>> +#include 
>> +#include 
>> +#include 
>> +#include 
>> +
>> +#ifdef HAVE_SYS_PRCTL_H
>> +#include 
>> +#endif
>
> This file isn't optional for us...
>
>> +
>> +#if HAVE_PTHREAD
>> +#include 
>> +#endif
>> +
>> +#include "asyncns.h"
>> +
>> +#ifndef MSG_NOSIGNAL
>> +#define MSG_NOSIGNAL 0
>> +#endif
>
> This is non-Linux compat and should go i figure...
>
>> +struct asyncns {
>
> probably should be called sd_asyncns now...
>
>> +int fds[MESSAGE_FD_MAX];
>> +
>> +#ifndef HAVE_PTHREAD
>> +pid_t workers[MAX_WORKERS];
>> +#elseso, you just suggested pthreads is the way to go. any strong reasons?
>> +pthread_t workers[MAX_WORKERS];
>> +#endif
>
> We should probably stick to one implementation here... Either with
> process or with threads, but not chicken out anymore like I did
> before...
>
> Every other day I come to different conclusions which one would be the
> one and only mode we should support.
>
> The big advantage of the process option is that things are neatly
> isolated from the parent. THe big disadvantages are that it pollutes the
> "ps" output and might generate page faults in the parent all the time,
> since we fork and don't exec().
>
> Right now I am more leaning towards the thread option. But maybe that
> changes tomorrow again. Other opinions? Kay?
>
> glib is embedding a copy of libasyncns currently, they opted for the
> thread solution. Maybe we should follow suit.

Doesn't seem to be any strong arguments either way... So shall we just
go with pthreads, if that's what's used elsewhere...?

> (Actually, it might be worth checking the glib copy, and see if they
> made any changes to the sources, which we might want to steal back for us.
>
>> +#ifndef HAVE_PTHREAD
>> +
>> +static int close_allv(const int except_fds[]) {
>
> We already have this in close_many() in util.h. This func

Re: [systemd-devel] [RFC] Initial libsystemd-asyncns commit

[Lennart Poettering]

On Thu, 12.12.13 16:28, Marcel Holtmann (mar...@holtmann.org) wrote:

> 
> Hi Lennart,
> 
> >> why do we have to spawn threads or do forks for DNS. This looks all
> >> pretty expensive. In ConnMan for example we just wrote our own async
> >> DNS using a mainloop. Works perfectly fine and is dirt cheap.
> > 
> > Well, we don't fork threads/processes for each call but reuse them.
> > 
> > What libasyncns does what your solution doesn't do is go via NSS. This
> > means /etc/hosts, nss-myhostname, nss-ldap, nss-mdns and so on just
> > work, while that all is lost when doing DNS natively.
> > 
> > I am pretty sure we should not bypass NSS for this. 
>  
>  actually NSS for DNS is pretty nasty stuff. I am pretty sure we should 
>  bypass it and create a proper implementation. Is anybody actually using 
>  NIS or LDAP for domain name resolution?
>  
> >>> 
> >>> Yes, there are solutions that are using LDAP for hostname resolution
> >>> quite heavily - actually are based around LDAP without any
> >>> local /etc/hosts.
> >> 
> >> that is extremely heavy and must suck form a latency point of
> >> view. Then again, nothing that a DNS<->LDAP bridge couldn’t easily
> >> support. Since dragging LDAP dependencies into every program that has
> >> to load NSS modules is not a good idea either.
> > 
> > Note that "nscd" was created to deal with the performance of the LDAP
> > setups and also doesn't require loading everything into the same process.
> 
> and that is an excuse to keep using NSS modules? We took a rather
> shitty idea and hacked it up a little and it is now a little bit less
> shitty.

Well, I agree that NSS is a bit aged, given its synchronous
behaviour. But it's the only name resolving API that is somewhat
accepted these days. With systemd we try to keep the focus on the
general purpose, which means we cannot just decide that NSS is shit and
bypass all NSS modules. Or, maybe we can do that one day, but then we
have to have something that is at least is powerful, and supports
loadable modules and shit -- but we don't have that.

In the meantime, going via NSS is the only viable option I think. And
the API is generic enough to allow other backends too. It's a 1:1
translation of the classic getaddrinfo() interface to an async way of
things. Whether there's actually getaddrinfo() behind it doesn't really
matter, we can totally swap that out.

For example, if eventuallly we have some kind of local DNS resolver
daemon that can do split horizon and whatnot, and also supports an
interface for LDAP, mdns, ... then we can certainly talk to that
directly from the same lib, nobody needs to know that...

Lennart

-- 
Lennart Poettering, Red Hat
___
systemd-devel mailing list
systemd-devel@lists.freedesktop.org
http://lists.freedesktop.org/mailman/listinfo/systemd-devel

Re: [systemd-devel] [RFC] Initial libsystemd-asyncns commit

[Marcel Holtmann]

Hi Lennart,

>> why do we have to spawn threads or do forks for DNS. This looks all
>> pretty expensive. In ConnMan for example we just wrote our own async
>> DNS using a mainloop. Works perfectly fine and is dirt cheap.
> 
> Well, we don't fork threads/processes for each call but reuse them.
> 
> What libasyncns does what your solution doesn't do is go via NSS. This
> means /etc/hosts, nss-myhostname, nss-ldap, nss-mdns and so on just
> work, while that all is lost when doing DNS natively.
> 
> I am pretty sure we should not bypass NSS for this. 
 
 actually NSS for DNS is pretty nasty stuff. I am pretty sure we should 
 bypass it and create a proper implementation. Is anybody actually using 
 NIS or LDAP for domain name resolution?
 
>>> 
>>> Yes, there are solutions that are using LDAP for hostname resolution
>>> quite heavily - actually are based around LDAP without any
>>> local /etc/hosts.
>> 
>> that is extremely heavy and must suck form a latency point of
>> view. Then again, nothing that a DNS<->LDAP bridge couldn’t easily
>> support. Since dragging LDAP dependencies into every program that has
>> to load NSS modules is not a good idea either.
> 
> Note that "nscd" was created to deal with the performance of the LDAP
> setups and also doesn't require loading everything into the same process.

and that is an excuse to keep using NSS modules? We took a rather shitty idea 
and hacked it up a little and it is now a little bit less shitty.

Just to be clear, I have nothing against the general idea, I am just saying 
that NSS itself with its current design is pretty limited. And keep working 
around it is not helping either.

Regards

Marcel

___
systemd-devel mailing list
systemd-devel@lists.freedesktop.org
http://lists.freedesktop.org/mailman/listinfo/systemd-devel

Re: [systemd-devel] [RFC] Initial libsystemd-asyncns commit

[Lennart Poettering]

On Thu, 12.12.13 15:20, Marcel Holtmann (mar...@holtmann.org) wrote:

> 
> Hi Andrey,
> 
>  why do we have to spawn threads or do forks for DNS. This looks all
>  pretty expensive. In ConnMan for example we just wrote our own async
>  DNS using a mainloop. Works perfectly fine and is dirt cheap.
> >>> 
> >>> Well, we don't fork threads/processes for each call but reuse them.
> >>> 
> >>> What libasyncns does what your solution doesn't do is go via NSS. This
> >>> means /etc/hosts, nss-myhostname, nss-ldap, nss-mdns and so on just
> >>> work, while that all is lost when doing DNS natively.
> >>> 
> >>> I am pretty sure we should not bypass NSS for this. 
> >> 
> >> actually NSS for DNS is pretty nasty stuff. I am pretty sure we should 
> >> bypass it and create a proper implementation. Is anybody actually using 
> >> NIS or LDAP for domain name resolution?
> >> 
> > 
> > Yes, there are solutions that are using LDAP for hostname resolution
> > quite heavily - actually are based around LDAP without any
> > local /etc/hosts.
> 
> that is extremely heavy and must suck form a latency point of
> view. Then again, nothing that a DNS<->LDAP bridge couldn’t easily
> support. Since dragging LDAP dependencies into every program that has
> to load NSS modules is not a good idea either.

Note that "nscd" was created to deal with the performance of the LDAP
setups and also doesn't require loading everything into the same process.

Lennart

-- 
Lennart Poettering, Red Hat
___
systemd-devel mailing list
systemd-devel@lists.freedesktop.org
http://lists.freedesktop.org/mailman/listinfo/systemd-devel

Re: [systemd-devel] [RFC] Initial libsystemd-asyncns commit

[Marcel Holtmann]

Hi Andrey,

 why do we have to spawn threads or do forks for DNS. This looks all
 pretty expensive. In ConnMan for example we just wrote our own async
 DNS using a mainloop. Works perfectly fine and is dirt cheap.
>>> 
>>> Well, we don't fork threads/processes for each call but reuse them.
>>> 
>>> What libasyncns does what your solution doesn't do is go via NSS. This
>>> means /etc/hosts, nss-myhostname, nss-ldap, nss-mdns and so on just
>>> work, while that all is lost when doing DNS natively.
>>> 
>>> I am pretty sure we should not bypass NSS for this. 
>> 
>> actually NSS for DNS is pretty nasty stuff. I am pretty sure we should 
>> bypass it and create a proper implementation. Is anybody actually using NIS 
>> or LDAP for domain name resolution?
>> 
> 
> Yes, there are solutions that are using LDAP for hostname resolution
> quite heavily - actually are based around LDAP without any
> local /etc/hosts.

that is extremely heavy and must suck form a latency point of view. Then again, 
nothing that a DNS<->LDAP bridge couldn’t easily support. Since dragging LDAP 
dependencies into every program that has to load NSS modules is not a good idea 
either.

Regards

Marcel

___
systemd-devel mailing list
systemd-devel@lists.freedesktop.org
http://lists.freedesktop.org/mailman/listinfo/systemd-devel

Re: [systemd-devel] [RFC] Initial libsystemd-asyncns commit

[Lennart Poettering]

On Thu, 12.12.13 14:09, Marcel Holtmann (mar...@holtmann.org) wrote:

> 
> Hi Lennart,
> 
> >> why do we have to spawn threads or do forks for DNS. This looks all
> >> pretty expensive. In ConnMan for example we just wrote our own async
> >> DNS using a mainloop. Works perfectly fine and is dirt cheap.
> > 
> > Well, we don't fork threads/processes for each call but reuse them.
> > 
> > What libasyncns does what your solution doesn't do is go via NSS. This
> > means /etc/hosts, nss-myhostname, nss-ldap, nss-mdns and so on just
> > work, while that all is lost when doing DNS natively.
> > 
> > I am pretty sure we should not bypass NSS for this. 
> 
> actually NSS for DNS is pretty nasty stuff. I am pretty sure we should
> bypass it and create a proper implementation. Is anybody actually
> using NIS or LDAP for domain name resolution?

Well, I am not opposed to having a different DNS implementation that is
smarter than the one in glibc (i.e. can do split hoirzon and stuff), but
I am sure that should be behind NSS, not bypassing it.

> The problem with NSS is that it still does not fix the fundamental
> problem. It is sequential and blocking. You can spawn as many threads
> as you want and it will stay sequential.

Are you suggesting that there was a global lock around NSS that
serializes all NSS requests? The getaddrinfo() RFC makes clear that the
call needs to be reentrant, this would hence be news to me?

> I fully agree with you that we need to consider /etc/hosts, the local
> hostname and also mDNS, but I get the feeling it would be more
> beneficial to just build that in. ConnMan ships a DNS proxy cache to
> actually be able to be smart and parallel. We did not bother with the
> details like /etc/hosts so far since our use cases always had an empty
> /etc/hosts with just localhost only anyway. However that is easy to
> fix.

Well, note that the API libasyncns exposes is actually abstract enough
to support a different backend instead of NSS. It is also close enough
to glibc to not be surprising. I am pretty sure that NSS should be kept
in the loop, but even if we decide that we shouldn't need that then
there's no reason to not do that behing the same API...

Lennart

-- 
Lennart Poettering, Red Hat
___
systemd-devel mailing list
systemd-devel@lists.freedesktop.org
http://lists.freedesktop.org/mailman/listinfo/systemd-devel

Re: [systemd-devel] [RFC] Initial libsystemd-asyncns commit

[Andrey Borzenkov]

В Thu, 12 Dec 2013 14:09:42 +0100
Marcel Holtmann  пишет:

> Hi Lennart,
> 
> >> why do we have to spawn threads or do forks for DNS. This looks all
> >> pretty expensive. In ConnMan for example we just wrote our own async
> >> DNS using a mainloop. Works perfectly fine and is dirt cheap.
> > 
> > Well, we don't fork threads/processes for each call but reuse them.
> > 
> > What libasyncns does what your solution doesn't do is go via NSS. This
> > means /etc/hosts, nss-myhostname, nss-ldap, nss-mdns and so on just
> > work, while that all is lost when doing DNS natively.
> > 
> > I am pretty sure we should not bypass NSS for this. 
> 
> actually NSS for DNS is pretty nasty stuff. I am pretty sure we should bypass 
> it and create a proper implementation. Is anybody actually using NIS or LDAP 
> for domain name resolution?
> 

Yes, there are solutions that are using LDAP for hostname resolution
quite heavily - actually are based around LDAP without any
local /etc/hosts.

> The problem with NSS is that it still does not fix the fundamental problem. 
> It is sequential and blocking. You can spawn as many threads as you want and 
> it will stay sequential.
> 
> I fully agree with you that we need to consider /etc/hosts, the local 
> hostname and also mDNS, but I get the feeling it would be more beneficial to 
> just build that in. ConnMan ships a DNS proxy cache to actually be able to be 
> smart and parallel. We did not bother with the details like /etc/hosts so far 
> since our use cases always had an empty /etc/hosts with just localhost only 
> anyway. However that is easy to fix.
> 
> Spawning threads or forking is expensive (even if you reuse it) comparing to 
> what you actually do here. You just want to resolve a host. It is especially 
> expensive since you can not control on what is actually used. And there are 
> clearly use cases where you have to ask a specific nameserver or set of 
> nameservers.
> 
> Regards
> 
> Marcel
> 
> ___
> systemd-devel mailing list
> systemd-devel@lists.freedesktop.org
> http://lists.freedesktop.org/mailman/listinfo/systemd-devel

___
systemd-devel mailing list
systemd-devel@lists.freedesktop.org
http://lists.freedesktop.org/mailman/listinfo/systemd-devel

Re: [systemd-devel] [RFC] Initial libsystemd-asyncns commit

[Marcel Holtmann]

Hi Lennart,

>> why do we have to spawn threads or do forks for DNS. This looks all
>> pretty expensive. In ConnMan for example we just wrote our own async
>> DNS using a mainloop. Works perfectly fine and is dirt cheap.
> 
> Well, we don't fork threads/processes for each call but reuse them.
> 
> What libasyncns does what your solution doesn't do is go via NSS. This
> means /etc/hosts, nss-myhostname, nss-ldap, nss-mdns and so on just
> work, while that all is lost when doing DNS natively.
> 
> I am pretty sure we should not bypass NSS for this. 

actually NSS for DNS is pretty nasty stuff. I am pretty sure we should bypass 
it and create a proper implementation. Is anybody actually using NIS or LDAP 
for domain name resolution?

The problem with NSS is that it still does not fix the fundamental problem. It 
is sequential and blocking. You can spawn as many threads as you want and it 
will stay sequential.

I fully agree with you that we need to consider /etc/hosts, the local hostname 
and also mDNS, but I get the feeling it would be more beneficial to just build 
that in. ConnMan ships a DNS proxy cache to actually be able to be smart and 
parallel. We did not bother with the details like /etc/hosts so far since our 
use cases always had an empty /etc/hosts with just localhost only anyway. 
However that is easy to fix.

Spawning threads or forking is expensive (even if you reuse it) comparing to 
what you actually do here. You just want to resolve a host. It is especially 
expensive since you can not control on what is actually used. And there are 
clearly use cases where you have to ask a specific nameserver or set of 
nameservers.

Regards

Marcel

___
systemd-devel mailing list
systemd-devel@lists.freedesktop.org
http://lists.freedesktop.org/mailman/listinfo/systemd-devel

Re: [systemd-devel] [RFC] Initial libsystemd-asyncns commit

[Thomas H.P. Andersen]

On Wed, Dec 11, 2013 at 2:41 AM, Lennart Poettering
 wrote:
> On Wed, 11.12.13 02:13, Daniel Buch (boogiewasth...@gmail.com) wrote:
>
> Heya,
>
> Hmm, so thinking about it I have the suspicion this should probably be
> linked into libsystemd-bus, and thus live in src/libsystemd-bus/. The
> reason for this is cyclic deps: libsystemd-bus really should make use of
> this to resolve host names when used in tcp mode. However, this stuff
> really should come with integration into sd-event out of the box too
> which is already part of libsystemd-bus (for similar reasons). So we'd
> have asyncns both as consumer of APIs of libsystemd-bus and as
> provider of APIs to it.
>
> Of course, one could argue that DNS is hardly part of bus access but
> then again, sd-event is neither. Maybe we should just start treating
> libsystemd-bus as that library where everything we expose ends up in,
> except when it is really clear that it's only consumer, never provider
> to stuff in libsystemd-bus.
>
>> Reindentation is done to fit systemd
>> ---
>>  Makefile.am   |   23 +
>>  src/libsystemd-asyncns/asyncns.c  | 1513 
>> +
>>  src/libsystemd-asyncns/asyncns.h  |  163 
>>  src/libsystemd-asyncns/test-asyncns.c |  178 
>>  4 files changed, 1877 insertions(+)
>>  create mode 100644 src/libsystemd-asyncns/asyncns.c
>>  create mode 100644 src/libsystemd-asyncns/asyncns.h
>>  create mode 100644 src/libsystemd-asyncns/test-asyncns.c
>>
>> diff --git a/Makefile.am b/Makefile.am
>> index 19da6ea..a0564b5 100644
>> --- a/Makefile.am
>> +++ b/Makefile.am
>> @@ -659,6 +659,29 @@ tests += test-rtnl
>>
>>  # 
>> --
>>  noinst_LTLIBRARIES += \
>> + libsystemd-asyncns.la
>> +
>> +libsystemd_asyncns_la_SOURCES = \
>> + src/libsystemd-asyncns/asyncns.c \
>> + src/libsystemd-asyncns/asyncns.h
>
> To follow the naming scheme of the other libs we should probably call
> this "sd-asyncs.c", and the header file should be in src/systemd/
> together with the other public APIs.
>
>> +
>> +libsystemd_asyncns_la_CFLAGS = \
>> + -pthread
>> +
>> +test_asyncns_SOURCES = \
>> + src/libsystemd-asyncns/test-asyncns.c
>> +
>> +test_asyncns_LDADD = \
>> + libsystemd-asyncns.la
>> +
>> +test_asyncns_LDFLAGS = \
>> + -lresolv \
>> + -pthread
>
> I figure the -lresolv thing will eventually need a configure.ac check,
> but for now I'd just leave it like this, when this breaks for people I
> am sure they will cook up a patch quickly...
>
>> +#ifdef HAVE_CONFIG_H
>> +#include 
>> +#endif
>
> Unnecessary, we do this on the gcc cmdline implicitly far all our files...
>
>> +
>> +/* #undef HAVE_PTHREAD */
>> +
>> +#include 
>> +#include 
>> +#include 
>> +#include 
>> +#include 
>> +#include 
>> +#include 
>> +#include 
>> +#include 
>> +#include 
>> +#include 
>> +#include 
>> +#include 
>> +#include 
>> +#include 
>> +#include 
>> +#include 
>> +#include 
>> +#include 
>> +
>> +#ifdef HAVE_SYS_PRCTL_H
>> +#include 
>> +#endif
>
> This file isn't optional for us...
>
>> +
>> +#if HAVE_PTHREAD
>> +#include 
>> +#endif
>> +
>> +#include "asyncns.h"
>> +
>> +#ifndef MSG_NOSIGNAL
>> +#define MSG_NOSIGNAL 0
>> +#endif
>
> This is non-Linux compat and should go i figure...
>
>> +struct asyncns {
>
> probably should be called sd_asyncns now...
>
>> +int fds[MESSAGE_FD_MAX];
>> +
>> +#ifndef HAVE_PTHREAD
>> +pid_t workers[MAX_WORKERS];
>> +#else
>> +pthread_t workers[MAX_WORKERS];
>> +#endif
>
> We should probably stick to one implementation here... Either with
> process or with threads, but not chicken out anymore like I did
> before...
>
> Every other day I come to different conclusions which one would be the
> one and only mode we should support.
>
> The big advantage of the process option is that things are neatly
> isolated from the parent. THe big disadvantages are that it pollutes the
> "ps" output and might generate page faults in the parent all the time,
> since we fork and don't exec().
>
> Right now I am more leaning towards the thread option. But maybe that
> changes tomorrow again. Other opinions? Kay?
>
> glib is embedding a copy of libasyncns currently, they opted for the
> thread solution. Maybe we should follow suit.
>
> (Actually, it might be worth checking the glib copy, and see if they
> made any changes to the sources, which we might want to steal back for us.
It was removed from glib again (see
https://bugzilla.gnome.org/show_bug.cgi?id=616754)

Just before the removal the glib version had this history:
https://git.gnome.org/browse/glib/log/gio/libasyncns/asyncns.c?id=aa586f63543fd584b782dbc1f90bcfae2c96502e
So just non-linux fixes.

>> +#ifndef HAVE_PTHREAD
>> +
>> +static int close_allv(const int except_fds[]) {
>
> We already have this in close_many() in util.h. This function can go away 
> without replacement.
>
>> +static int reset_sigsv(const int except[

Re: [systemd-devel] [RFC] Initial libsystemd-asyncns commit

[Lennart Poettering]

On Wed, 11.12.13 02:36, Marcel Holtmann (mar...@holtmann.org) wrote:

> why do we have to spawn threads or do forks for DNS. This looks all
> pretty expensive. In ConnMan for example we just wrote our own async
> DNS using a mainloop. Works perfectly fine and is dirt cheap.

Well, we don't fork threads/processes for each call but reuse them.

What libasyncns does what your solution doesn't do is go via NSS. This
means /etc/hosts, nss-myhostname, nss-ldap, nss-mdns and so on just
work, while that all is lost when doing DNS natively.

I am pretty sure we should not bypass NSS for this. 

Lennart

-- 
Lennart Poettering, Red Hat
___
systemd-devel mailing list
systemd-devel@lists.freedesktop.org
http://lists.freedesktop.org/mailman/listinfo/systemd-devel

Re: [systemd-devel] [RFC] Initial libsystemd-asyncns commit

[Lennart Poettering]

On Wed, 11.12.13 02:13, Daniel Buch (boogiewasth...@gmail.com) wrote:

Heya,

Hmm, so thinking about it I have the suspicion this should probably be
linked into libsystemd-bus, and thus live in src/libsystemd-bus/. The
reason for this is cyclic deps: libsystemd-bus really should make use of
this to resolve host names when used in tcp mode. However, this stuff
really should come with integration into sd-event out of the box too
which is already part of libsystemd-bus (for similar reasons). So we'd
have asyncns both as consumer of APIs of libsystemd-bus and as
provider of APIs to it. 

Of course, one could argue that DNS is hardly part of bus access but
then again, sd-event is neither. Maybe we should just start treating
libsystemd-bus as that library where everything we expose ends up in,
except when it is really clear that it's only consumer, never provider
to stuff in libsystemd-bus.

> Reindentation is done to fit systemd
> ---
>  Makefile.am   |   23 +
>  src/libsystemd-asyncns/asyncns.c  | 1513 
> +
>  src/libsystemd-asyncns/asyncns.h  |  163 
>  src/libsystemd-asyncns/test-asyncns.c |  178 
>  4 files changed, 1877 insertions(+)
>  create mode 100644 src/libsystemd-asyncns/asyncns.c
>  create mode 100644 src/libsystemd-asyncns/asyncns.h
>  create mode 100644 src/libsystemd-asyncns/test-asyncns.c
> 
> diff --git a/Makefile.am b/Makefile.am
> index 19da6ea..a0564b5 100644
> --- a/Makefile.am
> +++ b/Makefile.am
> @@ -659,6 +659,29 @@ tests += test-rtnl
>  
>  # 
> --
>  noinst_LTLIBRARIES += \
> + libsystemd-asyncns.la
> +
> +libsystemd_asyncns_la_SOURCES = \
> + src/libsystemd-asyncns/asyncns.c \
> + src/libsystemd-asyncns/asyncns.h

To follow the naming scheme of the other libs we should probably call
this "sd-asyncs.c", and the header file should be in src/systemd/
together with the other public APIs.

> +
> +libsystemd_asyncns_la_CFLAGS = \
> + -pthread
> +
> +test_asyncns_SOURCES = \
> + src/libsystemd-asyncns/test-asyncns.c
> +
> +test_asyncns_LDADD = \
> + libsystemd-asyncns.la
> +
> +test_asyncns_LDFLAGS = \
> + -lresolv \
> + -pthread

I figure the -lresolv thing will eventually need a configure.ac check,
but for now I'd just leave it like this, when this breaks for people I
am sure they will cook up a patch quickly...

> +#ifdef HAVE_CONFIG_H
> +#include 
> +#endif

Unnecessary, we do this on the gcc cmdline implicitly far all our files...

> +
> +/* #undef HAVE_PTHREAD */
> +
> +#include 
> +#include 
> +#include 
> +#include 
> +#include 
> +#include 
> +#include 
> +#include 
> +#include 
> +#include 
> +#include 
> +#include 
> +#include 
> +#include 
> +#include 
> +#include 
> +#include 
> +#include 
> +#include 
> +
> +#ifdef HAVE_SYS_PRCTL_H
> +#include 
> +#endif

This file isn't optional for us...

> +
> +#if HAVE_PTHREAD
> +#include 
> +#endif
> +
> +#include "asyncns.h"
> +
> +#ifndef MSG_NOSIGNAL
> +#define MSG_NOSIGNAL 0
> +#endif

This is non-Linux compat and should go i figure...

> +struct asyncns {

probably should be called sd_asyncns now...

> +int fds[MESSAGE_FD_MAX];
> +
> +#ifndef HAVE_PTHREAD
> +pid_t workers[MAX_WORKERS];
> +#else
> +pthread_t workers[MAX_WORKERS];
> +#endif

We should probably stick to one implementation here... Either with
process or with threads, but not chicken out anymore like I did
before...

Every other day I come to different conclusions which one would be the
one and only mode we should support. 

The big advantage of the process option is that things are neatly
isolated from the parent. THe big disadvantages are that it pollutes the
"ps" output and might generate page faults in the parent all the time,
since we fork and don't exec().

Right now I am more leaning towards the thread option. But maybe that
changes tomorrow again. Other opinions? Kay?

glib is embedding a copy of libasyncns currently, they opted for the
thread solution. Maybe we should follow suit.

(Actually, it might be worth checking the glib copy, and see if they
made any changes to the sources, which we might want to steal back for us.

> +#ifndef HAVE_PTHREAD
> +
> +static int close_allv(const int except_fds[]) {

We already have this in close_many() in util.h. This function can go away 
without replacement.

> +static int reset_sigsv(const int except[]) {

This too in reset_all_signal_handlers().

> +static int fd_nonblock(int fd) {

And this we have tooo.

> +static int fd_cloexec(int fd) {

And this too...

> +
> +switch (req->type) {
> +
> +case REQUEST_ADDRINFO: {
> +   struct addrinfo ai, *result = 
> NULL;
> +   const addrinfo_request_t 
> *ai_req = &packet->addrinfo_request;
> +   const char *node, *service;
> + 

Re: [systemd-devel] [RFC] Initial libsystemd-asyncns commit

[Marcel Holtmann]

Hi Daniel,

> Reindentation is done to fit systemd
> ---
> Makefile.am   |   23 +
> src/libsystemd-asyncns/asyncns.c  | 1513 +
> src/libsystemd-asyncns/asyncns.h  |  163 
> src/libsystemd-asyncns/test-asyncns.c |  178 
> 4 files changed, 1877 insertions(+)
> create mode 100644 src/libsystemd-asyncns/asyncns.c
> create mode 100644 src/libsystemd-asyncns/asyncns.h
> create mode 100644 src/libsystemd-asyncns/test-asyncns.c
> 
> diff --git a/Makefile.am b/Makefile.am
> index 19da6ea..a0564b5 100644
> --- a/Makefile.am
> +++ b/Makefile.am
> @@ -659,6 +659,29 @@ tests += test-rtnl
> 
> # 
> --
> noinst_LTLIBRARIES += \
> + libsystemd-asyncns.la
> +
> +libsystemd_asyncns_la_SOURCES = \
> + src/libsystemd-asyncns/asyncns.c \
> + src/libsystemd-asyncns/asyncns.h
> +
> +libsystemd_asyncns_la_CFLAGS = \
> + -pthread
> +
> +test_asyncns_SOURCES = \
> + src/libsystemd-asyncns/test-asyncns.c
> +
> +test_asyncns_LDADD = \
> + libsystemd-asyncns.la
> +
> +test_asyncns_LDFLAGS = \
> + -lresolv \
> + -pthread
> +
> +tests += test-asyncns
> +
> +# 
> --
> +noinst_LTLIBRARIES += \
>   libsystemd-shared.la
> 
> libsystemd_shared_la_SOURCES = \
> diff --git a/src/libsystemd-asyncns/asyncns.c 
> b/src/libsystemd-asyncns/asyncns.c
> new file mode 100644
> index 000..f1c2602
> --- /dev/null
> +++ b/src/libsystemd-asyncns/asyncns.c
> @@ -0,0 +1,1513 @@
> +/***
> +  This file is part of libasyncns.
> +
> +  Copyright 2005-2008 Lennart Poettering
> +
> +  libasyncns is free software; you can redistribute it and/or modify
> +  it under the terms of the GNU Lesser General Public License as
> +  published by the Free Software Foundation, either version 2.1 of the
> +  License, or (at your option) any later version.
> +
> +  libasyncns is distributed in the hope that it will be useful, but
> +  WITHOUT ANY WARRANTY; without even the implied warranty of
> +  MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
> +  Lesser General Public License for more details.
> +
> +  You should have received a copy of the GNU Lesser General Public
> +  License along with libasyncns. If not, see
> +  .
> + ***/
> +
> +#ifdef HAVE_CONFIG_H
> +#include 
> +#endif
> +
> +/* #undef HAVE_PTHREAD */
> +
> +#include 
> +#include 
> +#include 
> +#include 
> +#include 
> +#include 
> +#include 
> +#include 
> +#include 
> +#include 
> +#include 
> +#include 
> +#include 
> +#include 
> +#include 
> +#include 
> +#include 
> +#include 
> +#include 
> +
> +#ifdef HAVE_SYS_PRCTL_H
> +#include 
> +#endif
> +
> +#if HAVE_PTHREAD
> +#include 
> +#endif
> +
> +#include "asyncns.h"
> +
> +#ifndef MSG_NOSIGNAL
> +#define MSG_NOSIGNAL 0
> +#endif
> +
> +#define MAX_WORKERS 16
> +#define MAX_QUERIES 256
> +#define BUFSIZE (10240)
> +
> +typedef enum {
> +REQUEST_ADDRINFO,
> +RESPONSE_ADDRINFO,
> +REQUEST_NAMEINFO,
> +RESPONSE_NAMEINFO,
> +REQUEST_RES_QUERY,
> +REQUEST_RES_SEARCH,
> +RESPONSE_RES,
> +REQUEST_TERMINATE,
> +RESPONSE_DIED
> +} query_type_t;
> +
> +enum {
> +REQUEST_RECV_FD = 0,
> +REQUEST_SEND_FD = 1,
> +RESPONSE_RECV_FD = 2,
> +RESPONSE_SEND_FD = 3,
> +MESSAGE_FD_MAX = 4
> +};
> +
> +struct asyncns {
> +int fds[MESSAGE_FD_MAX];
> +
> +#ifndef HAVE_PTHREAD
> +pid_t workers[MAX_WORKERS];
> +#else
> +pthread_t workers[MAX_WORKERS];
> +#endif
> +unsigned valid_workers;

why do we have to spawn threads or do forks for DNS. This looks all pretty 
expensive. In ConnMan for example we just wrote our own async DNS using a 
mainloop. Works perfectly fine and is dirt cheap.

Regards

Marcel

___
systemd-devel mailing list
systemd-devel@lists.freedesktop.org
http://lists.freedesktop.org/mailman/listinfo/systemd-devel