[systemd-devel] sd-bus: get size of array container in D-Bus message
Hi folks, I have a quick question: is there a way to get container size when deserializing an array from a D-Bus message (be it an array of trivial or non-trivial D-Bus types)? Say I enter a container with sd_bus_message_enter_container, and then, before reading individual elements, I'd like to get the number of elements in that array in that message (so I can reserve storage on my side, etc.). If not, could such an API function be added? Thanks in advance. Stanislav.
[systemd-devel] Extend sd_bus_message_append_basic to take length of a string
Hi folks,
Does sd-bus API provide a way to append a char string of a basic type (be it
SD_BUS_TYPE_STRING, SD_BUS_TYPE_OBJECT_PATH, or SD_BUS_TYPE_SIGNATURE for
that matter) to a D-Bus message while allowing the caller to explicitly specify
the length of that char string?
sd_bus_message_append_basic() function takes a pointer only, and calls
strlen() on that pointer.
If there is no such API, could we add a new function with string size as an
additional parameter. It would make the API more usable and general and, as a
specific use case, help C++ code using sd-bus API to append char strings of
type std::string_view to the D-Bus message without having to convert a
std::string_view to a std::string just to get null-terminated string which
can then be passed safely to sd_bus_message_append_basic(). Right now we have
to do that conversion which defeats the fundamental purpose of
std::string_view type in C++ standard library.
For example something along these lines:
int sd_bus_message_append_basic_string_n(sd_bus_message *m, char type, const
void *p, size_t len);
Which would work only for the above-mentioned three D-Bus string types.
And in C++ code (sdbus-c++ wrapper, for example), we could call it simply like
this:
std::string_view view{...};
sd_bus_message_append_basic_string_n(msg, SD_BUS_TYPE_STRING, view.data(),
view.length());
What do you think of this proposal? Could we extend sd-bus API along these
lines? Would you do that or shall I create a PR?
Kind regards,
Stanislav.
Re: [systemd-devel] sd_bus_get_timeout returns absolute time despite what's documented
Hey Lennart, Yes, I was referring to man pages I read a few years ago when I used sd_bus_get_timeout() for the first time. And, before posting this mail, I also checked on my current Linux development machine, which has systemd v247. So I was not that far :-) The fixed wording in v250 is correct, understandable and explicit to me. All good now. Thanks a lot. Stanislav. --- Original Message --- On Monday, January 9th, 2023 at 1:02 PM, Lennart Poettering wrote: > On Mo, 09.01.23 12:53, Lennart Poettering (lenn...@poettering.net) wrote: > > > https://www.freedesktop.org/software/systemd/man/sd_bus_get_fd.html#Description > > > > Note that the returned time-value is absolute, based of > > CLOCK_MONOTONIC and specified in microseconds. When converting > > this value in order to pass it as third argument to poll() (which > > expects relative milliseconds), care should be taken to convert to > > a relative time and use a division that rounds up to ensure the > > I/O polling operation doesn't sleep for shorter than necessary, > > which might result in unintended busy looping (alternatively, use > > ppoll(2) instead of plain poll(), which understands timeouts with > > nano-second granularity). > > > > That's pretty explicit already, no? > > > This was fixed in 2021 btw, 25060a570c106cf5a14a3268bb0d38d9feb7fdab > i.e. systemd 250. Upgrade! > > > (I mean, you have half a point, the first sentence of the explanation > > might people think this was a relative timeout, but we all read the > > full documentation, no, before actually using this API, no? ;-)) > > > > Anyway, will prep a fix that rewords the first sentence to make this > > clearer right away. > > > https://github.com/systemd/systemd/pull/25985 > > Lennart > > -- > Lennart Poettering, Berlin
[systemd-devel] sd_bus_get_timeout returns absolute time despite what's documented
Hi everyone, According to sd_bus_get_timeout(3) man page, sd_bus_get_timeout() returns "timeout in us to pass to poll()" and it also states that "the returned time-value is relative". However, in reality, this function returns an absolute time point (a-ka relative to the monotonic clock epoch). Hence, the returned value cannot be simply converted to milliseconds and given to poll(). It must first be subtracted from the current monotonic clock time. I followed the documentation and had a bug in my code for some time :) Shall the documentation be updated to mention the necessity of conversion from absolute to relative? Or does sd_bus_get_timeout() have a bug and shall this function do the subtraction for the user and return relative time? Thanks, kind regards, Stanislav.
[systemd-devel] Exception safety od sd-bus
Hi guys! Assuming sd-bus is used in a C++ application, is sd-bus safe against exceptions flying from e.g. a sd-bus vtable callback handler (provided by the C++ application) and catching them in the caller of sd_bus_process() (which is the same C++ app)? Or this is not supported (so leaks or whatever obscure situations may happen then)? Thanks, Stanislav Angelovič. ___ systemd-devel mailing list systemd-devel@lists.freedesktop.org https://lists.freedesktop.org/mailman/listinfo/systemd-devel
Re: [systemd-devel] sd-bus change that broke sdbus-c++
Done: https://github.com/systemd/systemd/issues/16193 On Tue, Jun 16, 2020 at 10:50 AM Lennart Poettering wrote: > On Mi, 10.06.20 18:24, Stanislav Angelovič (angelovi...@gmail.com) wrote: > > > Hi folks, > > > > sdbus-c++, a C++ library on top of sd-bus, is based on plain sd-bus > > messages to implement the concept of Variant type. Under plain message > here > > I mean one created via `sd_bus_message_new` with the type being > > _SD_BUS_MESSAGE_TYPE_INVALID. We use such a message as a generic > container > > of serialized D-Bus data, from which we read and to which we write. More > of > > this, including an example, is explained in the mailing list thread > spawned > > a year ago: > > > https://lists.freedesktop.org/archives/systemd-devel/2019-May/042748.html. > > > > That all of sudden stopped working with systemd 245.6. The reason is > highly > > likely this commit in systemd: > > > https://github.com/systemd/systemd/commit/a2dd991d0fde59dc0574bd4d0c1438f01dc0b8ff > . > > This commit changed `sd_bus_message_new` to be more strict and > immediately > > reject creating messages of type _SD_BUS_MESSAGE_TYPE_INVALID, whereas > > before that case was allowed, as were also allowed a few serialization > > operations upon such a message (the rest like enqueuing checked the type > > and failed, naturally). That original behavior was a perfect match for > our > > need for plain sd-bus messages. I understand that this was based on > > internal implementation of sd-bus, since as I checked the documentation > > now, it instructs clients to use one of the method call, method return, > > method error and signal types. > > > > What would you suggest as a solution in this situation? Would it be > > possible to get back to the original behavior? Or would you suggest > > creating a new API method for creating plain messages? Any is fine for > > sdbus-c++, but the latter is a little more difficult because we cannot > > conditionally decide upon the behavior in our code based on the systemd > > version, as 245 exhibits various behaviors depending on the minor > version... > > Please file a bug on github. > > Lennart > > -- > Lennart Poettering, Berlin > ___ systemd-devel mailing list systemd-devel@lists.freedesktop.org https://lists.freedesktop.org/mailman/listinfo/systemd-devel
[systemd-devel] sd-bus change that broke sdbus-c++
Hi folks, sdbus-c++, a C++ library on top of sd-bus, is based on plain sd-bus messages to implement the concept of Variant type. Under plain message here I mean one created via `sd_bus_message_new` with the type being _SD_BUS_MESSAGE_TYPE_INVALID. We use such a message as a generic container of serialized D-Bus data, from which we read and to which we write. More of this, including an example, is explained in the mailing list thread spawned a year ago: https://lists.freedesktop.org/archives/systemd-devel/2019-May/042748.html. That all of sudden stopped working with systemd 245.6. The reason is highly likely this commit in systemd: https://github.com/systemd/systemd/commit/a2dd991d0fde59dc0574bd4d0c1438f01dc0b8ff. This commit changed `sd_bus_message_new` to be more strict and immediately reject creating messages of type _SD_BUS_MESSAGE_TYPE_INVALID, whereas before that case was allowed, as were also allowed a few serialization operations upon such a message (the rest like enqueuing checked the type and failed, naturally). That original behavior was a perfect match for our need for plain sd-bus messages. I understand that this was based on internal implementation of sd-bus, since as I checked the documentation now, it instructs clients to use one of the method call, method return, method error and signal types. What would you suggest as a solution in this situation? Would it be possible to get back to the original behavior? Or would you suggest creating a new API method for creating plain messages? Any is fine for sdbus-c++, but the latter is a little more difficult because we cannot conditionally decide upon the behavior in our code based on the systemd version, as 245 exhibits various behaviors depending on the minor version... Thanks in advance for your help. Stanislav. ___ systemd-devel mailing list systemd-devel@lists.freedesktop.org https://lists.freedesktop.org/mailman/listinfo/systemd-devel
[systemd-devel] sd-bus signal callback return value
Hi folks, Please, what is the difference in sd-bus behavior when my D-Bus signal callback returns a zero return value versus a positive return value? And what's the difference between those return values in D-Bus method callback, D-Bus async method reply callback, D-Bus property get callback, and D-Bus property set callback? Which shall I return in normal cases? Zero? Thanks a lot in advance for your response... Take care & stay healthy, Stanislav. ___ systemd-devel mailing list systemd-devel@lists.freedesktop.org https://lists.freedesktop.org/mailman/listinfo/systemd-devel
[systemd-devel] GetMachineId implementation in libsystemd
Hi guys, I have two questions regarding implementation of org.freedesktop.DBus.Peer.GetMachineId() in libsystemd. 1. The method only checks the '/etc/machine-id' file. If libsystemd, which contains sd-bus implementation, which is quite self-contained and systemd-independent, is used in a non-systemd linux environment, the file does not exist. Hence, GetMachineId fails. But there is '/var/lib/dbus/machine-id' available, created by dbus package itself. Shouldn't the method also check the presence of the latter file if the former one doesn't exist? I can prepare the patch if it's ok. 2. Error handling issue. If the file is not there, the corresponding C function returns error value, which winds all the way up to sd_bus_process() which returns error. This seems to me like mixing levels of errors. I thought that when sd_bus_process returns error, we have serious problem because something bus infrastructure-related failed. Shouldn't the method just return a D-Bus error reply and in C function return OK (because the method body was found and executed, so from sd-bus view things are fine, but on the application level the method could not finish properly, so it sends error message to the caller)? Also here I can prepare a patch, if it's ok. Thanks, and cheers. Stanislav. ___ systemd-devel mailing list systemd-devel@lists.freedesktop.org https://lists.freedesktop.org/mailman/listinfo/systemd-devel
Re: [systemd-devel] sd-bus: Enabling free-standing, bus-independent plain messages
Hi Lennart, Sorry for a bit late reply, I've been quite busy recently. > So we could readd the ability to create a bus message with a NULL bus, > but I am a bit concerned that people then always pass NULL which might > ultimately result in constant remarshalling if the message is > eventually enqueued on a real bus. When you put together a message it > actually matters which bus connection you do that for... My proposal is that we would allow passing NULL bus ptr only for `sd_bus_message_new` function. And such a message would be disallowed from enqueuing. In functions that need real bus ptr we might place an assert that bus != NULL. In a limited set of functions, which I mentioned in previous e-mail (`sd_bus_message_seal` for example), we would allow bus == NULL. > Note that message behaviour and so on depends in sometimes subtle and sometimes not > so subtle ways on the bus connection used, i.e. whether we are talking > to a real bus or not, and so on. I am not to keen of making this > completely independent I must admit... > > What's the usecase for using bus message as general marshalling > storage anyway? Can you elaborate? I provided a use case example a few years ago in https://lists.freedesktop.org/archives/systemd-devel/2016- November/037929.html, but I'll elaborate more. sdbus-c++ provides such a generic layer of abstraction that its clients don't work with messages but call D-Bus methods as-if it was a local operation, using native types. For that, all D-Bus types have their native representation. A D-Bus array is represented as std::vector. A D-Bus string is std::string. And for D-Bus variant there is custom sdbus::Variant class that needs to be able to store whatever D-Bus types. So for example, for a method `doSomething` that takes a D-Bus string and an int, and returns an array of variants, sdbus-c++ generates something like `std::vector doSomething(std::string str, int i);`. Clients call this method, and sdbus-c++ does all the job -- it creates a method call message, serializes data into it, sends the message, receives the reply message, and automatically deserializes its contents into std::vector. But variant is special. When deserializing, sdbus-c++ is unaware what real type is under each variant, but it still must be able to get a given variant out of the message and store it in the vector to be returned. This is done by `sd_bus_message_copy` of just that variant message part. It cannot do `sd_bus_message_enter_container` and `sd_bus_message_read` directly, because sd-bus API requires signature of underlying variant contents, and this is what sdbus-c++ has no idea about at this point. It's the client who knows, so the client then at a later point, in his own code, asks for an object of specific type from that Variant, like so `double d = variant.get()`. Here is when `sd_bus_message_enter_container` and `sd_bus_message_read` on variant message are called, so the deserialization from variant message takes place. It's something like 2-step deserialization -- but only with variants, they are special in that. Of course, our first idea was to use C++ type-erasure technique employed by e.g. std::any to implement the Variant class. This works fine when we construct Variant from a native type. But when we need to create Variant out of a part of incoming D-Bus message, like in the example of method reply message above, we are stuck. The code would be very complex, practically impossible to write, due to conversion of run-time combinations of D-Bus types to static, compile-time type representations (vectors, maps of various keys and values that could maps again etc...). Hence, using sd-bus message as an underlying implementation of Variant class is the simplest and most reasonable solution; we are simply using the concept that is implemented in sd-bus already. We can easily create Variant out of a native type (by serializing the value into underlying message), create Variant out of existing message (by copying the message part into underlying message), and read data back from Variant (by reading the underlying message). > A compromise might be that we readd a concept of allowing > bus-independent messages to be generated again (i.e. pass NULL as bus > object when creating a bus message), with the most reduced feature set > possible, and at the same time refuse to enqueue such messages on any > bus, thus forcing people to use sd_bus_message_copy() to explicitly > remarshal for the bus, instead of doing implicitly so. Fully agree, that is also what I meant. (The reduced feature set would be the functions I mentioned in the previous e-mail.) > if you want to prep such a patch I think we should merge it. OK, I will try. So we don't introduce any new message factory function without bus parameter, but make use of existing `sd_bus_message_new`, right? We will modify it to allow NULL bus ptr. However, this function needs three things from the bus when initializing message: `message_version`, `c
[systemd-devel] sd-bus: Enabling free-standing, bus-independent plain messages
Hi (Lennart :) Quick question: Would it be possible to extend sd-bus to also allow creating messages without requiring a bus? Let me explain: Currently, if we want to create any message in sd-bus, we need a valid bus pointer. That might make perfect sense for messages that come from or will eventually go to bus. But sd-bus also supports plain messages. Do we also need that hard msg->bus dependency for plain messages which we use only as a local, temporary storage of serialized data? Some years ago we had a little discussion about a few tweaks in sd-bus API to allow modelling the concept of Variant type in higher-level sd-bus bindings ( https://lists.freedesktop.org/archives/systemd-devel/2016-November/037929.html). A Variant is essentially implemented as a class around plain sd-bus message which is only used a storage of the underlying type. To create a Variant instance, we simply call `sd_bus_message_new` factory with type _SD_BUS_MESSAGE_TYPE_INVALID. Nice and simple. But that factory requires real bus ptr. This leads to a more complicated, less efficient and not-that-nice solution that must take hold of some bus (if there is none it must create one), must cache the bus (so that it's not created and destroyed at every creation/deletion of Variant instance). Yes, we have `sd_bus_default_system`, but even with that there is some effort that has some pitfalls (e.g. we can't `std::move` a Variant to a different thread, because it might potentially outlive the current one). All that nomenclature is needed just to get a plain message to read/write data from. It looks like the bus is not really needed when the message is used as a local storage only, i.e. when only these API functions are used: * creating plain, empty message (`sd_bus_message_new`) * writing/reading to/from the message (`sd_bus_message_append*`, `sd_bus_message_read*`, ...) * sealing the message (`sd_bus_message_seal` * copying to/from the message (`sd_bus_message_copy`) * rewinding the message (`sd_bus_message_rewind`) * peeking message type (`sd_bus_message_peek_type`) * reffing/unreffing the message (sd_bus_message_ref/unref`) Even today, vast majority of these functions doesn't seem to need the bus. The exception are `sd_bus_message_seal` and `sd_bus_message_new` itself. We'd need to either modify `sd_bus_message_new` to allow taking in NULL bus ptr, or creating a new function for creating such plain local messages. We'd need to modify `sd_bus_message_seal` to simply consider the fact that m->bus can be NULL. And we'd probably need to add asserts for m->bus to functions which truly require real bus ptr presence. (We might even provide the option to link the message to a real bus later, and introduce new `sd_bus_message_set_bus` function, but I'm not sure whether that is good.) To summarize: sd-bus already offers API for creating plain message. To make it fully true, let's just not require the bus there. The proposed change would IMHO make sd-bus message more flexible, and would make it easier, more intuitive and more robust to work with and model Variants around such free-standing, bus-independent sd-bus messages in higher-level languages. What do you think? Thank you, and kind regards, Stanislav. ___ systemd-devel mailing list systemd-devel@lists.freedesktop.org https://lists.freedesktop.org/mailman/listinfo/systemd-devel
Re: [systemd-devel] refcount of bus returned from sd_bus_default_system
Thank you Lennart for detailed explanation. I realized a few things in sd-bus behavior (and in my wrong approach to default bus -- yes I forgot about hello message) when I debugged the thing earlier today, and now your explanation cleared that up pretty nicely to me. I see again that sd-bus design is well thought-out, I'm glad to work with it, and I hope to keep up to high design quality in c++ binding on top of it (the quality is also increasing the more I understand some details of sd-bus). Perhaps more detailed docs on sd-bus would help a lot.. I will fix the approach to default busses on my side. This is a very special corner case in sdbus-c++ used in order to be able to create plain empty messages, which are then used as storage of Variant class, without user having to provide an existing bus connection themselves. Perhaps that could be refactored in future -- but disadvantage is that users won't be able to create self-contained Variants (they will have to provide their bus connection explicitly to create a Variant). Thanks a lot! Take care. On Thu, May 9, 2019, 17:02 Lennart Poettering wrote: > On Mi, 08.05.19 22:50, Stanislav Angelovič (angelovi...@gmail.com) wrote: > > > Heya, > > > > when writing sdbus-c++, we've observed that sd_bus_default_system > function > > called in a fresh new thread returns a bus with initial ref count 2. We > > built our code upon that assumption -- we had to unref the bus twice when > > the thread local storage got freed, otherwise we'd have gotten > > memory leaks. > > Uh, no, this was never the right thing to do. > > > Now it broke, however, because in systemd v242, the initial ref count is > 1. > > Is that a conscious change? Can we build upon the guarantee that it will > > always be 1? (1 actually seems reasonable, unlike 2). > > Originally, in sd-bus each bus message (i.e. each sd_bus_message > object) keeps a ref on the bus connection (i.e. each sd_bus object) it > belongs to. Moreover, if a message is queued into a bus connection it > is referenced by it. Thus you have a ref cycle, as long as a mesage is > queued into a bus and not processed yet. The intention then was that > whenever you want to get rid of a bus at the end of your > thread/process you'd flush out the bus anyway (because otherwise you > lose queued but unwritten messages), at which point the cycle would be > cleared up as soon as all messages are written. > > Also note, that when a bus connection is set up a "Hello" message is > immediately enqueued into it (this is required by the spec). Thus, by > default there'd be two references onto the bus connection object: the > one you own yourself as the caller, and the one from the Hello bus > message queued onto it. > > To flush out messages you should use the sd_bus_flush() call. it's > blocking, and all it does is force out all queued outgoing messages > that are not written yet, thus emptying the write queue. Then, call > sd_bus_close(), to explicitly terminate the connection. This will > empty the read queue too. At this point there are no messages queued > anymore, i.e. all refs on the bus object must be held by you and not > internally. Finally, get rid of the whole thing by doing > sd_bus_unref(). Since these three calls are often combined, there's > sd_bus_flush_close_unref() to combine them into one. > > Calling sd_bus_flush() is entirely optional btw, it's sufficient to just > call sd_bus_close(). However in that case and pending unwritten > messages are just forgotten, and not written to the connection socket, > and never seen by the receiving side. > > Now you might wonder, why doesn't sd_bus_unref() always implicitly > call sd_bus_flush()? Reffing/unreffing should not be blocking, that's > why. Moreover, the concept of "default" busses of a thread is that > various bits and pieces running in the thread could quickly ask for > the default bus connection, do some stuff with it, enqueue a message > or two, then unref it again. Other code might then pick it up again, > enqueue some more messages on them, unref it again, and so on. Then, > when the thread is about to end, there might be a number of messages > queued: before exiting from the thread they should be flushed out, but > only then, there's no need to do so earlier. Thus, in the earlier > uses of the bus connection your'd just unref the bus at the end, but > when you are finally done with the default connection altogether youd' > use the more powerful sd_bus_flush_close_unref() instead, and do all > the work synchronously, that the earlier users didn't bother to wait > for. > > Note that when an sd_bus connection is attached to an sd_event event > lo
[systemd-devel] refcount of bus returned from sd_bus_default_system
Heya, when writing sdbus-c++, we've observed that sd_bus_default_system function called in a fresh new thread returns a bus with initial ref count 2. We built our code upon that assumption -- we had to unref the bus twice when the thread local storage got freed, otherwise we'd have gotten memory leaks. Now it broke, however, because in systemd v242, the initial ref count is 1. Is that a conscious change? Can we build upon the guarantee that it will always be 1? (1 actually seems reasonable, unlike 2). Thanks, Stan. ___ systemd-devel mailing list systemd-devel@lists.freedesktop.org https://lists.freedesktop.org/mailman/listinfo/systemd-devel
[systemd-devel] Build only libsystemd as a shared library
Hi systemd-ers, Having recent systemd sources, how can I build libsystemd.so only? I was able to build the static version with this: meson build/ ninja -C build version.h ninja -C build libsystemd.a But how can I build the shared one? Is there a configuration flag? (I'm not familiar with meson.) In older, makefile-based systemd versions, I would just run: make built-sources make libsystemd.la I want to build libsystemd only because it contains sd-bus implementation and sd-bus (and our sdbus-c++ layer on top of it) can thus be used also in non-systemd environments (which is great). Thank you, have a nice day, Stanislav. ___ systemd-devel mailing list systemd-devel@lists.freedesktop.org https://lists.freedesktop.org/mailman/listinfo/systemd-devel
[systemd-devel] sd-bus: "cancelling" async method call
Hi, Is there a way in sd-bus to "cancel" an async call issued through sd_bus_call_async() or sd_bus_call_method_async()? Under cancelling here I mean that the call issued can proceed to its end, but I'm just no more interested in results and don't want sd-bus to invoke provided callback when the reply comes. sd_bus_call_async() and sd_bus_call_method_async() functions provide sd_bus_slot* as an output argument, so if I applied sd_bus_slot_unref() on that slot ptr would sd-bus ignore the async call reply then? Or is there another way? Thanks a lot :) Stanislav. ___ systemd-devel mailing list systemd-devel@lists.freedesktop.org https://lists.freedesktop.org/mailman/listinfo/systemd-devel
Re: [systemd-devel] sd-bus: serving method call message in a separate thread
Hi Lennart, Thanks a lot for your elaborate reply. See inline replies... On Tue, Mar 5, 2019 at 11:27 AM Lennart Poettering wrote: > > On Mo, 04.03.19 21:56, Stanislav Angelovič (angelovi...@gmail.com) wrote: > > > sd-bus doesn't natively care for threads. However it's written in kind > of a threads-aware style, i.e. it keeps only local state, no global > variables. This means you can do locking around it yourself and thus > make it usable from multiple threads. Good design, IMO. > > To do that, allocate a single mutex of some form covering the sd_bus > object and all sd_bus_message associated with it. You need to take the > lock when you enter any sd_bus_xyz call, and release it after. To > ensure that you can use sd-bus APIs from one thread while another one > poll()s on the connection make sure that you do not use sd-bus' own > polling (i.e. do not use sd_bus_wait(), since that would mean we'd > have to wait for IO with the lock taken). Instead, use your own poll() > loop that uses sd_bus_get_fd(), sd_bus_get_events(), > sd_bus_get_timeout() to figure out what to wait for, and then unlock > the mutex before poll() and lock it again after. > Yes, we have played with an approach along these lines in sdbus-c++ recently. But, looking at sd-bus code, perhaps not all operations on both bus and messages need a mutual lock. If in our library design, if we ensure that the bus is running properly, then it seems we can e.g. write output arguments to a method reply message (provided that the message is accessed only in this one thread) without locking access to the entire bus and other messages. > Note that DBus puts an emphasis on message ordering, which means > worker-thread-pool based dispatching is kinda incompatible with what > it is built for (as thread pool stuff usually means out-of-order > dispatching). Hence, whether doing threaded sd-bus is a good idea is a > different question to ask. The only specific thing in our worker-thread approach here is that when a D-Bus object receives a call of its method, it doesn't guarantee that it sends the reply before accepting another call of that method. If it receives calls c1 and c2 that order, it might e.,g, reply to c2 first, then to c1. On the caller side, the caller abstracts away from this and it still normally waits for the reply to its issued call. Is that against D-Bus message ordering rules? > > sd-bus itself tries to push people towards a different model btw: that > you have a separate bus connection for each thread > (i.e. sd_bus_default() and friends tells you the default connection > for your specific thread), and that within that thread you have perfect > ordering. I see. However, each connection then must have a different bus name. They cannot share the same bus name... > > > 1. In sd_bus_message_handler, create a deep copy of the method call > > sd_bus_message via sd_bus_message_copy(). And pass a pointer to this copy > > to a worker thread. This is very straight-forward and simple, it solves the > > race condition, but introduces a performance cost of copying (I don't know > > how big this cost is, perhaps it's rather negligible). > > It's fine to pass the sd_bus_message to worker threads, as long as you > make sure you always have a reference to it. While passing the > reference you don't have to lock the message/bus, but when accessing > the message you have to take the lock however. Taking the lock on our side is not sufficient, still. Because sd-bus accesses the message, too -- it unrefs it after method callback, without taking any lock. So we still have race condition. Creating a message copy (to which sd-bus no more keeps reference, just we) and then passing this copy to a different thread seems a solution to me. With one drawback -- making the copy. > > > 3. Solution on sd-bus side :-) which is to make sd_bus_message ref count > > handling thread-safe (atomic), just like one for sd_bus. This avoids the > > need for deep copy. What do you think? Anyway, we'd still have to come up > > with a solution now, based on already releases sd-bus versions. > > Given the fact that DBus as it stands now isn't really meant for > processing msgs in worker thread pools I don't think we want to add > more thread support to sd-bus. Doing worker thread pools for DBus can > work in special cases but not in the general case (example: if you > create an object in another service dbus gives you the guarantee that > the method reply and the signal announcing the object arrive in a > specific order, which is a property that cannot be held up if worker > thread pools are used), hence I am very conservative with supporting > this beyond how we already do (i.e that we maintain no global state, > a
[systemd-devel] sd-bus: serving method call message in a separate thread
Hi sd-bus-ers! Quick question: How can I process a method call on service side in a different thread, without creating race condition? Longer version: In sdbus-c++, we are working on server-side asynchronous method call support. In sd-bus, a service handles D-Bus method calls via sd_bus_message_handler_t callback that is registered during object vtable registration. The callback receives sd_bus_message* as the first parameter (among others), which is the method call message. In a typical implementation, this method call is served synchronously (D-Bus method parameters are deserialized from the message, an operation is executed, and either method reply or method error, depending on the condition, is sent back) and the callback returns execution to sd-bus. Standard stuff so far. Now, I would like to handle the method call asynchronously, in a thread pool. So a typical approach would be to increment ref count of sd_bus_message and push the sd_bus_message* in a queue. It would then be popped by one of worker threads and processed -- method parameters are deserialized, method logic is executed, and either method reply or method error is created and sent back, and sd_bus_message* ref count is decremented. Now, sending back the reply or an error is thread-safe in my implementation, I don't have an issue with that. The problem that I want to discuss is the method call sd_bus_message instance. I cannot simply forward a pointer to it to a separate thread, because of it's non-atomic ref count handling, which is the source of race condition (hopefully this is the only data race condition here). It's manipulated by both the D-Bus loop thread (which decrements ref count after sd_bus_message_handler_t returns) and the worker thread concurrently (which also decrements ref count after it has handled the method call). I see three potential solutions now: 1. In sd_bus_message_handler, create a deep copy of the method call sd_bus_message via sd_bus_message_copy(). And pass a pointer to this copy to a worker thread. This is very straight-forward and simple, it solves the race condition, but introduces a performance cost of copying (I don't know how big this cost is, perhaps it's rather negligible). 2. Don't pass the sd_bus_message* to the worker thread. In sd_bus_message_handler, rather deserialize all arguments, create (empty) method reply message, and move these to the worker thread. The worker thread executes the logic and serializes results to that reply message, and sends it back. The problem here is that we have to create a method reply or method error before the fact (before executing method logic), which in case of method error is impossible because we don't know possible error name and message beforehand. 3. Solution on sd-bus side :-) which is to make sd_bus_message ref count handling thread-safe (atomic), just like one for sd_bus. This avoids the need for deep copy. What do you think? Anyway, we'd still have to come up with a solution now, based on already releases sd-bus versions. So the only feasible solution for now seems to be #1 (while #3 could also be a sd-bus improvement for the future). Do you agree that this is the good way along the lines of sd-bus design principles, or are there more options I'm unaware of? Thank you for your constructive comments, and sorry for long elaboration :) I wanted to be clear... Cheers, Stanislav. ___ systemd-devel mailing list systemd-devel@lists.freedesktop.org https://lists.freedesktop.org/mailman/listinfo/systemd-devel
[systemd-devel] D-Bus method parameter names in introspection XML
Hi guys, sd-bus implements and automatically provides org.freedesktop.DBus.Introspectable interface for each object vtable. Xml output of Introspect method of that interface provides all necessary information but one -- names of method parameters. It's quite convenient that when one introspects a service and its interfaces, he wants very much to see names of method parameters. Naming of parameters adds a whole different level of self-expressiveness and clarity to interface description. D-Bus specification ( https://dbus.freedesktop.org/doc/dbus-specification.html#introspection-format) mentions names of method parameters as well. Are there any ways to solve this on sd-bus user side? Either way, would it be possible to extend sd-bus vtable registration and introspection facilities to incorporate names of method parameters? Thank you. Stanislav, sdbus-c++ maintainer. ___ systemd-devel mailing list systemd-devel@lists.freedesktop.org https://lists.freedesktop.org/mailman/listinfo/systemd-devel
[systemd-devel] sd-bus: calling D-Bus method from a D-Bus method callback upon the same D-Bus connection
Hi guys, in sd-bus, when D-Bus method callback of my service is invoked (someone is calling me), is it possible to issue another synchronous call to another service upon *the same sd_bus connection*? I.e. does sd_bus instance allow that while it is processing incoming requests and invoking callbacks, a callback in its body uses this sd_bus instance to quickly issue another request-reply call before returning? Or in such a case, the proxy needs to create a connection of its own for that quick request-reply round? Thanks :) ___ systemd-devel mailing list systemd-devel@lists.freedesktop.org https://lists.freedesktop.org/mailman/listinfo/systemd-devel
[systemd-devel] Direct connection between peers in sd-bus
Hi, In sd-bus, I guess it is possible to have a one-to-one connection between a service and a client, i.e. connection without D-bus daemon), am I right? If yes, is there any example (in systemd source tree or elsewhere) of sd-bus one-to-one communication usage that I could look at for inspiration and learning on how is that done? Thanks a lot. ___ systemd-devel mailing list systemd-devel@lists.freedesktop.org https://lists.freedesktop.org/mailman/listinfo/systemd-devel
[systemd-devel] Async server with sd-bus
Hi folks, Suppose we have a server whose methods may take relatively long time and I would like to process them asynchronously within the server, so while a client is waiting for reply to his time-consuming call, other clients (which could be served quickly, for example) are not starved because of this. I suppose sd-bus supports such an approach, since it is message-based. Also, implementation-wise, I suppose these async replies must be sent back by the processing thread that handles requests upon the bus connection (since sd_bus and sd_bus_message seem to no be thread-safe). So I need to sent the reply messages from the worker thread back to the processing thread, to send them out. I am right with my assumptions and statements? Have you already solved such a problem somewhere in systemd? Thanks for help and cheers! Stanislav. ___ systemd-devel mailing list systemd-devel@lists.freedesktop.org https://lists.freedesktop.org/mailman/listinfo/systemd-devel
Re: [systemd-devel] sd-bus connections & authentication timeout
Hi Jan, thanks for quick response. Instead of sd_bus_process(), we could perhaps use sd_bus_flush() after creating the connection, as that one actually processes the requests until the connection changes state to 'running'. I tried this and it helped. Regarding polling, I understand, but I fear this way we could get race condition on the connection instance, since we'd have a thread doing the polling and processing in a loop on the connection, and some other thread (say handling the UI, or an external system event, or incoming DBus call from another connection or whatever else) issuing a DBus call on the same connection. I think the connection instance is not thread-safe, as once I experienced such data race related problems. Maybe there is another way of doing that? One that comes to my mind is having just one single thread handling the connection and communication through it, but then it would have to poll on other fd's as well, to be able to communicate with other threads, for example to get DBus call request from the UI thread and perform it on the connection. This involves asynchronous programming paradigm for quite a simple thing, IMHO. What do you think? Thank you, Stanislav. On Mon, Mar 20, 2017 at 5:19 PM, Jan Alexander Steffens < jan.steff...@gmail.com> wrote: > On Mon, Mar 20, 2017, 17:14 Jan Alexander Steffens > wrote: > >> >> You could try calling sd_bus_process(bus, NULL) in a loop while it >> returns >0 so that the initial hello is handled. >> > > Actually, never mind, this is not reliable. IIRC the initial handshake has > multiple steps so this can return zero before everything is done. Can't get > around polling to do this properly. > ___ systemd-devel mailing list systemd-devel@lists.freedesktop.org https://lists.freedesktop.org/mailman/listinfo/systemd-devel
[systemd-devel] sd-bus connections & authentication timeout
Hi, We use sd-bus for DBus IPC in our own applications. Some applications of ours cause the dbus daemon to issue "Connection has not authenticated soon enough, closing it" message, leading to the org.freedesktop.DBus.Error.Timeout error at the peer side when the connection is used for an IPC call. The situation is the following: 1. We have a DBus client application. 2. A connection is created using sd_bus_open_system() on application startup. 3. The connection is kept open and unused (i.e. no other sd-bus functions invoked on it). 4. After 30 seconds, we get the above-mentioned authentication timeout message from the dbus daemon, and any attempt to issue calls against a service via this connection fails with immediate timeout error as mentioned above. However, if the call is made before the 30 seconds authentication timeout, the dbus daemon prints nothing and the call succeeds, just like any subsequent call, even if issued after 1 hour from the first one. Is that correct behavior? We would like to open dbus connections at application startup, and use it for communication at any time later, depending on user actions. Is there something we are missing? And a side suggestion: If we want to close a connection that has not been used for any IPC call at all, after some struggling we have realized that sd_bus_unref() is not enough, as there is a lingering DBus.Hello reply which keeps the reference count on that connection, leading to lingering connections. sd_bus_flush_close_unref() is needed instead. That behavior could be mentioned in the documentation, e.g. at http://0pointer.net/blog/the-new-sd-bus-api-of-systemd.html in the reference client implementation, to save others from pain. Thanks a lot, Stanislav. ___ systemd-devel mailing list systemd-devel@lists.freedesktop.org https://lists.freedesktop.org/mailman/listinfo/systemd-devel
Re: [systemd-devel] sd-bus: Implementing Variant concept: Copying sd-bus message contents
On Mon, Nov 28, 2016 at 6:45 PM, Lennart Poettering wrote: > On Sat, 26.11.16 16:01, Stanislav Angelovič (angelovi...@gmail.com) wrote: > > > Hi, > > > > A quick question: Is it possible to append contents (the serialized data) > > of an sd-bus message to another sd-bus message? Conversely, is it > possible > > to extract a part of sd-bus message contents into a separate > > message? > > There's sd_bus_message_copy() which can copy the contents (or a part > of it) of a message into another message. > > There's currently no API that exposes the raw binary bits, and they > aren't particularly useful anyway on dbus1 as they are not safe > regarding alignment and you cannot copy them byte-by-byte to arbitrary > memory locations. > > Details: > > > > In the process of creating a method call message, I would like to do > this: > > 1. Create a temporary, empty message A. > > 2. Serialize data into the message A. > > 3. Create a method call message B. > > 4. Serialize some user data into the message B, plus copy the data of the > > message A into the message B. And send the message B. > > sd_bus_message_copy() should work fine for that. > > Analogously, when deserializing data from the message B upon its > reception > > on the server side, I would like to do this: > > 1. Extract user data from the message B, plus copy selected data into a > > separate, empty message A. > > 2. Destroy the message B. > > 3. In some other context, deserialize real data from the message A. > > > > Why am I asking is because we like sd-bus and are building a C++ binding > on > > top of it. We stumbled upon a problem when modelling Variant. The typical > > flow in such a case is that upon e.g. deserialization, first the Variant > > instances are created from the message contents, and then at some later > > phase, users want to extract real data from the Variant. In this phase, > our > > C++ binding would perform the deserialization from that temporry message > > (message A) that the Variant would hold. > > > > For example, if the method call message carries an array of Variants, we > > first deserialize the message into std::vector, where each > Variant > > instance keeps the temporary sd-bus message with the variant data, and > when > > the user asks for a concrete data from that Variant, the deserialization > > from that temporary message takes place. > > > > This is the way it's done in e.g. dbus-c++, the C++ binding for libdbus. > > libdbus API allows creating "empty" messages, and allows copying the data > > among messages with the possibility of reading from them at the same > time. > > Is it possible to create an empty message using sd-bus? Regarding > copying, > > we know there's the sd_bus_message_copy function, but once we create the > > copy message, we cannot read from it later because of the assertion that > > the message is sealed, which in reality is not. > > Hmm, so internally there's bus_message_seal() which I figure we could > make publically available, so that you can create a new msg, copy the > data to it, seal it, and read it back. > Yes, exactly, I need to create a message copy, but also be able to read from it afterwards. If invoking bus_message_seal() is sufficient to make the message readable after creating it, it would be so welcome to have it in the public API. Additionally, we'd need a simple factory function for a message that won't have a context of a method call, a signal or an error message. Just plain message that will be used for storing some data and later, via sd_bus_message_copy(), copying the data into a 'real' message. There is message_new function in sd-bus, but it's internal. Would it be possible to make it public, just like lidbbus provides dbus_message_new()? > > > Is there a way to solve this in sd-bus? Perhaps another way than I > > described? Many thanks in advance for your response! > > There's no nicer way right now. But I think we could add a > sd_bus_message_dup() or so that duplicates a message, but does so > efficiently, i.e. shares the memory for the actual payload. This way > you could duplicate a message to simply get a new read ptr on the > same, immutable message. I think that would suit your usecase best? > sd_bus_message_dup() as you describe it could perfectly be an option, but -- similarly to sd_bus_message_copy() -- it would have to allow not only duplication of the entire message, but also a part of it. This could perhaps be handled -- consistently to sd_bus_message_copy() -- with an additional 'complete' boolean parameter
[systemd-devel] sd-bus: Implementing Variant concept: Copying sd-bus message contents
Hi, A quick question: Is it possible to append contents (the serialized data) of an sd-bus message to another sd-bus message? Conversely, is it possible to extract a part of sd-bus message contents into a separate message? Details: In the process of creating a method call message, I would like to do this: 1. Create a temporary, empty message A. 2. Serialize data into the message A. 3. Create a method call message B. 4. Serialize some user data into the message B, plus copy the data of the message A into the message B. And send the message B. Analogously, when deserializing data from the message B upon its reception on the server side, I would like to do this: 1. Extract user data from the message B, plus copy selected data into a separate, empty message A. 2. Destroy the message B. 3. In some other context, deserialize real data from the message A. Why am I asking is because we like sd-bus and are building a C++ binding on top of it. We stumbled upon a problem when modelling Variant. The typical flow in such a case is that upon e.g. deserialization, first the Variant instances are created from the message contents, and then at some later phase, users want to extract real data from the Variant. In this phase, our C++ binding would perform the deserialization from that temporry message (message A) that the Variant would hold. For example, if the method call message carries an array of Variants, we first deserialize the message into std::vector, where each Variant instance keeps the temporary sd-bus message with the variant data, and when the user asks for a concrete data from that Variant, the deserialization from that temporary message takes place. This is the way it's done in e.g. dbus-c++, the C++ binding for libdbus. libdbus API allows creating "empty" messages, and allows copying the data among messages with the possibility of reading from them at the same time. Is it possible to create an empty message using sd-bus? Regarding copying, we know there's the sd_bus_message_copy function, but once we create the copy message, we cannot read from it later because of the assertion that the message is sealed, which in reality is not. Is there a way to solve this in sd-bus? Perhaps another way than I described? Many thanks in advance for your response! With kind regards, Stan. ___ systemd-devel mailing list systemd-devel@lists.freedesktop.org https://lists.freedesktop.org/mailman/listinfo/systemd-devel
