Re: Deadlock in kded whith Qt 5.6

2016-01-22 Thread David Faure 
On Thursday 21 January 2016 13:54:32 Thiago Macieira wrote:
> Can I limit the spy hook to method calls? That is, exclude signals.

Yes, for the purpose of kded/kiod that is fine
A not-loaded-yet-module cannot possibly connect to a dbus signal.

> Another attempt, also untested.

It works !

Even the proxyscout module with its usage of system bus in the module ctor gets 
autostarted+autoloaded just fine.

Thanks Thiago.

-- 
David Faure, fa...@kde.org, http://www.davidfaure.fr
Working on KDE Frameworks 5

___
Kde-frameworks-devel mailing list
Kde-frameworks-devel@kde.org
https://mail.kde.org/mailman/listinfo/kde-frameworks-devel

Re: Deadlock in kded whith Qt 5.6

2016-01-22 Thread Thiago Macieira 
On Friday 22 January 2016 09:21:12 David Faure wrote:
> On Thursday 21 January 2016 13:54:32 Thiago Macieira wrote:
> > Can I limit the spy hook to method calls? That is, exclude signals.
> 
> Yes, for the purpose of kded/kiod that is fine
> A not-loaded-yet-module cannot possibly connect to a dbus signal.
> 
> > Another attempt, also untested.
> 
> It works !
> 
> Even the proxyscout module with its usage of system bus in the module ctor
> gets autostarted+autoloaded just fine.
> 
> Thanks Thiago.

You're welcome. This version is simpler too.

I've just uploaded PS 4 with this version and an updated commit message.

-- 
Thiago Macieira - thiago (AT) macieira.info - thiago (AT) kde.org
   Software Architect - Intel Open Source Technology Center
  PGP/GPG: 0x6EF45358; fingerprint:
  E067 918B B660 DBD1 105C  966C 33F5 F005 6EF4 5358

___
Kde-frameworks-devel mailing list
Kde-frameworks-devel@kde.org
https://mail.kde.org/mailman/listinfo/kde-frameworks-devel

Re: Deadlock in kded whith Qt 5.6

2016-01-21 Thread Thiago Macieira 
On Thursday 21 January 2016 08:37:38 David Faure wrote:
> > I didn't see any "invoking message spies" of the debug I left. Are you
> > sure
> > this had the hook installed?
> 
> You mean the messageFilter? That's exactly the issue, that it's not getting
> called. Therefore autoload doesn't happen.
> 
> It might be a consequence of your previous commit (1f6fa1f37a14742dd),
> not necessarily a problem in the current patch under test.

I don't see how. The spy hooks are called before the queue would get 
suspended. We can invert the order to see if it helps, but I don't see how it 
would help or hinder. It would make the code more complex.

Can I limit the spy hook to method calls? That is, exclude signals.

> > For that matter, can you apply the attached patch so we also get feedback
> > on the queued delivery?
> 
> With this patch applied, here's one log for the first call
> (autostart+autoload, doesn't work)
> 
> http://www.davidfaure.fr/2016/first_call.txt

This one says "delivery is suspended" a couple of times. But as I said, the 
spy processing happens *before* this, with conditional:

if (Q_UNLIKELY(qDBusSpyHookList.exists()) && qApp) {

The list exists if you've installed a hook. Is it possible that this is 
happening before the QCoreApplication is created?

Note the race condition: you should install the hook *before* connecting to 
the bus. But even if that were the case, we should still be seeing the spy 
later.

> and one log for the second call (kded already started, autoload works)
> 
> http://www.davidfaure.fr/2016/second_call.txt

This one shows "invoking message spies"
-- 
Thiago Macieira - thiago (AT) macieira.info - thiago (AT) kde.org
   Software Architect - Intel Open Source Technology Center
  PGP/GPG: 0x6EF45358; fingerprint:
  E067 918B B660 DBD1 105C  966C 33F5 F005 6EF4 5358

___
Kde-frameworks-devel mailing list
Kde-frameworks-devel@kde.org
https://mail.kde.org/mailman/listinfo/kde-frameworks-devel

Re: Deadlock in kded whith Qt 5.6

2016-01-21 Thread Ralf Habacker 
Am 21.01.2016 um 08:37 schrieb David Faure:

>  
> With this patch applied, here's one log for the first call 
> (autostart+autoload, doesn't work)
>
> http://www.davidfaure.fr/2016/first_call.txt
>
> and one log for the second call (kded already started, autoload works)
>
> http://www.davidfaure.fr/2016/second_call.txt
>
This patches https://codereview.qt-project.org/#/c/146767/1 and 
https://codereview.qt-project.org/#/c/146768/, which prints out message
'serial' and 'reply serial to' on qdbus side, may help to understand the
message flow more easier.

Ralf
___
Kde-frameworks-devel mailing list
Kde-frameworks-devel@kde.org
https://mail.kde.org/mailman/listinfo/kde-frameworks-devel

Re: Deadlock in kded whith Qt 5.6

2016-01-21 Thread Thiago Macieira 
On Thursday 21 January 2016 08:37:38 David Faure wrote:
> On Wednesday 20 January 2016 08:41:31 Thiago Macieira wrote:
> > On Wednesday 20 January 2016 09:08:38 David Faure wrote:
> > > > Can you run with QDBUS_DEBUG=1 and tell me the messages it prints?
> > 
> > [cut]
> > 
> > I didn't see any "invoking message spies" of the debug I left. Are you
> > sure
> > this had the hook installed?
> 
> You mean the messageFilter? That's exactly the issue, that it's not getting
> called. Therefore autoload doesn't happen.

Another attempt, also untested.
-- 
Thiago Macieira - thiago (AT) macieira.info - thiago (AT) kde.org
   Software Architect - Intel Open Source Technology Center
  PGP/GPG: 0x6EF45358; fingerprint:
  E067 918B B660 DBD1 105C  966C 33F5 F005 6EF4 5358
>From 10191ce9c6d31225442fde85c6d4470681d4f511 Mon Sep 17 00:00:00 2001
From: Thiago Macieira 
Date: Tue, 19 Jan 2016 22:29:33 -0800
Subject: [PATCH 1/1] Call out to QtDBus message spies in the main thread

Whenever there are spies installed, we call out to the main thread to
call to the kded/kiod message spies. This allows the spy code to do just
about anything, where previously it was restricted in what it could do
to avoid deadlocking or triggering assertions if it recursed back into
QDBusConnection code in the manager thread.

After the spies are done with, the message is re-inserted into the
QDBusConnection processing pipeline. That means it may be delayed again
if the dispatching is disabled.

Change-Id: I3d11545be52c43119f0f142b0e9d447415c2
---
 src/dbus/qdbusconnection_p.h |  3 ++-
 src/dbus/qdbusintegrator_p.h | 19 ++
 src/dbus/qdbusintegrator.cpp | 48 +---
 3 files changed, 57 insertions(+), 13 deletions(-)

diff --git a/src/dbus/qdbusconnection_p.h b/src/dbus/qdbusconnection_p.h
index f030a3f..c77daf7 100644
--- a/src/dbus/qdbusconnection_p.h
+++ b/src/dbus/qdbusconnection_p.h
@@ -1,7 +1,7 @@
 /
 **
 ** Copyright (C) 2015 The Qt Company Ltd.
-** Copyright (C) 2015 Intel Corporation.
+** Copyright (C) 2016 Intel Corporation.
 ** Contact: http://www.qt.io/licensing/
 **
 ** This file is part of the QtDBus module of the Qt Toolkit.
@@ -281,6 +281,7 @@ private slots:
 
 signals:
 void dispatchStatusChanged();
+void spyHooksFinished(const QDBusMessage );
 void messageNeedsSending(QDBusPendingCallPrivate *pcall, void *msg, int timeout = -1);
 void signalNeedsConnecting(const QString , const QDBusConnectionPrivate::SignalHook );
 bool signalNeedsDisconnecting(const QString , const QDBusConnectionPrivate::SignalHook );
diff --git a/src/dbus/qdbusintegrator_p.h b/src/dbus/qdbusintegrator_p.h
index 2038ac9..e266512 100644
--- a/src/dbus/qdbusintegrator_p.h
+++ b/src/dbus/qdbusintegrator_p.h
@@ -1,6 +1,7 @@
 /
 **
 ** Copyright (C) 2015 The Qt Company Ltd.
+** Copyright (C) 2016 Intel Corporation.
 ** Contact: http://www.qt.io/licensing/
 **
 ** This file is part of the QtDBus module of the Qt Toolkit.
@@ -65,6 +66,7 @@
 QT_BEGIN_NAMESPACE
 
 class QDBusConnectionPrivate;
+class QDBusMessage;
 
 // Really private structs used by qdbusintegrator.cpp
 // Things that aren't used by any other file
@@ -133,6 +135,23 @@ private:
 bool handled;
 };
 
+class QDBusSpyCallEvent : public QMetaCallEvent
+{
+public:
+typedef void (*Hook)(const QDBusMessage&);
+QDBusSpyCallEvent(QDBusConnectionPrivate *cp, const QDBusConnection , const QDBusMessage ,
+  const Hook *hooks, int count)
+: QMetaCallEvent(0, 0, Q_NULLPTR, cp, 0), conn(c), msg(msg), hooks(hooks), hookCount(count)
+{}
+~QDBusSpyCallEvent();
+void placeMetaCall(QObject *) Q_DECL_OVERRIDE;
+
+QDBusConnection conn;   // keeps the refcount in QDBusConnectionPrivate up
+QDBusMessage msg;
+const Hook *hooks;
+int hookCount;
+};
+
 QT_END_NAMESPACE
 
 Q_DECLARE_METATYPE(QDBusSlotCache)
diff --git a/src/dbus/qdbusintegrator.cpp b/src/dbus/qdbusintegrator.cpp
index dffdfe2..ebf42b0 100644
--- a/src/dbus/qdbusintegrator.cpp
+++ b/src/dbus/qdbusintegrator.cpp
@@ -1,7 +1,7 @@
 /
 **
 ** Copyright (C) 2015 The Qt Company Ltd.
-** Copyright (C) 2015 Intel Corporation.
+** Copyright (C) 2016 Intel Corporation.
 ** Contact: http://www.qt.io/licensing/
 **
 ** This file is part of the QtDBus module of the Qt Toolkit.
@@ -120,8 +120,7 @@ void qdbusDefaultThreadDebug(int action, int condition, QDBusConnectionPrivate *
 qdbusThreadDebugFunc qdbusThreadDebug = 0;
 #endif
 
-typedef void (*QDBusSpyHook)(const QDBusMessage&);
-typedef QVarLengthArray QDBusSpyHookList;
+typedef QVarLengthArray QDBusSpyHookList;
 Q_GLOBAL_STATIC(QDBusSpyHookList, qDBusSpyHookList)
 
 extern "C" {

Re: Deadlock in kded whith Qt 5.6

2016-01-20 Thread David Faure 
On Wednesday 20 January 2016 08:41:31 Thiago Macieira wrote:
> On Wednesday 20 January 2016 09:08:38 David Faure wrote:
> > > Can you run with QDBUS_DEBUG=1 and tell me the messages it prints?
> > 
> [cut]
> 
> I didn't see any "invoking message spies" of the debug I left. Are you sure 
> this had the hook installed?

You mean the messageFilter? That's exactly the issue, that it's not getting 
called.
Therefore autoload doesn't happen.

It might be a consequence of your previous commit (1f6fa1f37a14742dd),
not necessarily a problem in the current patch under test.

> For that matter, can you apply the attached patch so we also get feedback on 
> the queued delivery?
 
With this patch applied, here's one log for the first call (autostart+autoload, 
doesn't work)

http://www.davidfaure.fr/2016/first_call.txt

and one log for the second call (kded already started, autoload works)

http://www.davidfaure.fr/2016/second_call.txt

-- 
David Faure, fa...@kde.org, http://www.davidfaure.fr
Working on KDE Frameworks 5

___
Kde-frameworks-devel mailing list
Kde-frameworks-devel@kde.org
https://mail.kde.org/mailman/listinfo/kde-frameworks-devel

Re: Deadlock in kded whith Qt 5.6

2016-01-20 Thread Thiago Macieira 
On Wednesday 20 January 2016 09:08:38 David Faure wrote:
> > Can you run with QDBUS_DEBUG=1 and tell me the messages it prints?
> 
[cut]

I didn't see any "invoking message spies" of the debug I left. Are you sure 
this had the hook installed?

For that matter, can you apply the attached patch so we also get feedback on 
the queued delivery?

-- 
Thiago Macieira - thiago (AT) macieira.info - thiago (AT) kde.org
   Software Architect - Intel Open Source Technology Center
  PGP/GPG: 0x6EF45358; fingerprint:
  E067 918B B660 DBD1 105C  966C 33F5 F005 6EF4 5358
--- src/dbus/qdbusintegrator.cpp
+++ src/dbus/qdbusintegrator.cpp
@@ -525,6 +525,7 @@ bool QDBusConnectionPrivate::handleMessage(const QDBusMessage )
 return false;
 if (!dispatchEnabled && !QDBusMessagePrivate::isLocal(amsg)) {
 // queue messages only, we'll handle them later
+qDBusDebug() << this << "delivery is suspended";
 pendingMessages << amsg;
 return amsg.type() == QDBusMessage::MethodCallMessage;
 }
@@ -1119,8 +1120,10 @@ void QDBusConnectionPrivate::doDispatch()
 // dispatch previously queued messages
 PendingMessageList::Iterator it = pendingMessages.begin();
 PendingMessageList::Iterator end = pendingMessages.end();
-for ( ; it != end; ++it)
+for ( ; it != end; ++it) {
+qDBusDebug() << this << "got queued message" << *it;
 handleMessage(qMove(*it));
+}
 pendingMessages.clear();
 }
 }
___
Kde-frameworks-devel mailing list
Kde-frameworks-devel@kde.org
https://mail.kde.org/mailman/listinfo/kde-frameworks-devel

Re: Deadlock in kded whith Qt 5.6

2016-01-20 Thread Thiago Macieira 
On Wednesday 20 January 2016 08:55:17 David Faure wrote:
> Doesn't seem to support service autostart + module autoload together.

> but if I kill kded5 again and do this right away:
> $ qdbus org.kde.kcookiejar5  /modules/kcookiejar
> then it doesn't work: kded5 gets started, but the call gets this reply
> Error: org.freedesktop.DBus.Error.UnknownObject
> No such object path '/modules/kcookiejar'
> 

> The messageFilter doesn't run at all, when doing autostart+autoload
> together.

I need a little more debugging. The limited debugging I did here shows that 
the spy hooks do get called and in the right (main) thread. And since the 
hooks are processed before the main dispatcher queue, it shouldn't affect the 
previous code.

Can you run with QDBUS_DEBUG=1 and tell me the messages it prints?

-- 
Thiago Macieira - thiago (AT) macieira.info - thiago (AT) kde.org
   Software Architect - Intel Open Source Technology Center
  PGP/GPG: 0x6EF45358; fingerprint:
  E067 918B B660 DBD1 105C  966C 33F5 F005 6EF4 5358
___
Kde-frameworks-devel mailing list
Kde-frameworks-devel@kde.org
https://mail.kde.org/mailman/listinfo/kde-frameworks-devel

Re: Deadlock in kded whith Qt 5.6

2016-01-20 Thread David Faure 
On Tuesday 19 January 2016 23:59:31 Thiago Macieira wrote:
> On Wednesday 20 January 2016 08:55:17 David Faure wrote:
> > Doesn't seem to support service autostart + module autoload together.
> 
> > but if I kill kded5 again and do this right away:
> > $ qdbus org.kde.kcookiejar5  /modules/kcookiejar
> > then it doesn't work: kded5 gets started, but the call gets this reply
> > Error: org.freedesktop.DBus.Error.UnknownObject
> > No such object path '/modules/kcookiejar'
> > 
> 
> > The messageFilter doesn't run at all, when doing autostart+autoload
> > together.
> 
> I need a little more debugging. The limited debugging I did here shows that 
> the spy hooks do get called and in the right (main) thread. And since the 
> hooks are processed before the main dispatcher queue, it shouldn't affect the 
> previous code.
> 
> Can you run with QDBUS_DEBUG=1 and tell me the messages it prints?
 
Client side:
$ qdbus org.kde.kcookiejar5  /modules/kcookiejar
qdbus(31279)/default QDBusConnectionPrivate::setConnection: 
QDBusConnection(ptr=0x7fa1340032f0, name="qt_default_session_bus", 
baseService=":1.0") : connected successfully
qdbus(31279)/default qDBusSignalFilter: QDBusConnection(ptr=0x7fa1340032f0, 
name="qt_default_session_bus", baseService=":1.0") got message (signal): 
QDBusMessage(type=Signal, service="org.freedesktop.DBus", 
path="/org/freedesktop/DBus", interface="org.freedesktop.DBus", 
member="NameAcquired", signature="s", contents=(":1.0") )
qdbus(31279)/default QDBusConnectionPrivate::sendWithReplyAsync: 
QDBusConnection(ptr=0x7fa1340032f0, name="qt_default_session_bus", 
baseService=":1.0") sending message: QDBusMessage(type=MethodCall, 
service="org.kde.kcookiejar5", path="/modules/kcookiejar", 
interface="org.freedesktop.DBus.Introspectable", member="Introspect", 
signature="", contents=() )
qdbus(31279)/default QDBusConnectionPrivate::processFinishedCall: 
QDBusConnection(ptr=0x7fa1340032f0, name="qt_default_session_bus", 
baseService=":1.0") got message reply: QDBusMessage(type=Error, service=":1.1", 
error name="org.freedesktop.DBus.Error.UnknownObject", error message="No such 
object path '/modules/kcookiejar'", signature="s", contents=("No such object 
path '/modules/kcookiejar'") )
qdbus(31279)/default QDBusBlockingCallWatcher::~QDBusBlockingCallWatcher: 
QDBusConnection: warning: blocking call took a long time (1604 ms, max for this 
thread is 200 ms) to service "org.kde.kcookiejar5" path "/modules/kcookiejar" 
interface "org.freedesktop.DBus.Introspectable" member "Introspect"
Error: org.freedesktop.DBus.Error.UnknownObject
No such object path '/modules/kcookiejar'
QDBusConnection(ptr=0x7fa1340032f0, name="qt_default_session_bus", 
baseService=":1.0") Disconnected

Server side:
Jan 20 09:06:56 asterix org.kde.kcookiejar5[31231]: kded5(31282)/default 
KCrash::initialize: INITIALIZED
Jan 20 09:06:56 asterix org.kde.kcookiejar5[31231]: kded5(31282)/default 
QDBusConnectionPrivate::setConnection: QDBusConnection(ptr=0x7f4b18003120, 
name="qt_default_session_bus", baseService=":1.1") : connected successfully
Jan 20 09:06:56 asterix org.kde.kcookiejar5[31231]: kded5(31282)/default 
qDBusSignalFilter: QDBusConnection(ptr=0x7f4b18003120, 
name="qt_default_session_bus", baseService=":1.1") got message (signal): 
QDBusMessage(type=Signal, service="org.freedesktop.DBus", 
path="/org/freedesktop/DBus", interface="org.freedesktop.DBus", 
member="NameAcquired", signature="s", contents=(":1.1") )
Jan 20 09:06:56 asterix org.kde.kcookiejar5[31231]: kded5(31282)/default 
QDBusConnectionPrivate::addSignalHook: QDBusConnection(ptr=0x7f4b18003120, 
name="qt_default_session_bus", baseService=":1.1") Adding rule: 
"type='signal',path='/BreezeStyle',interface='org.kde.Breeze.Style',member='reparseConfiguration'"
Jan 20 09:06:56 asterix org.kde.kcookiejar5[31231]: kded5(31282)/default 
QDBusConnectionPrivate::addSignalHook: QDBusConnection(ptr=0x7f4b18003120, 
name="qt_default_session_bus", baseService=":1.1") Adding rule: 
"type='signal',path='/BreezeDecoration',interface='org.kde.Breeze.Style',member='reparseConfiguration'"
Jan 20 09:06:56 asterix org.kde.kcookiejar5[31231]: kded5(31282)/default 
QDBusConnectionPrivate::sendWithReplyAsync: QDBusConnection(ptr=0x7f4b18003120, 
name="qt_default_session_bus", baseService=":1.1") sending message: 
QDBusMessage(type=MethodCall, service="org.freedesktop.DBus", 
path="/org/freedesktop/DBus", interface="org.freedesktop.DBus", 
member="RequestName", signature="", contents=("org.kde.kded5", 4) )
Jan 20 09:06:56 asterix org.kde.kcookiejar5[31231]: kded5(31282)/default 
qDBusSignalFilter: QDBusConnection(ptr=0x7f4b18003120, 
name="qt_default_session_bus", baseService=":1.1") got message (signal): 
QDBusMessage(type=Signal, service="org.freedesktop.DBus", 
path="/org/freedesktop/DBus", interface="org.freedesktop.DBus", 
member="NameAcquired", signature="s", contents=("org.kde.kded5") )
Jan 20 09:06:56 asterix org.kde.kcookiejar5[31231]: kded5(31282)/default

Re: Deadlock in kded whith Qt 5.6

2016-01-19 Thread Aleix Pol 
On Mon, Jan 18, 2016 at 9:23 AM, David Faure  wrote:
> On Friday 15 January 2016 13:48:14 Aleix Pol wrote:
>> On Fri, Jan 15, 2016 at 1:02 PM, Aleix Pol  wrote:
>> > Hi,
>> > I realized earlier last week that some of my applications were
>> > freezing, it turns out it's because my kded is freezing and any dbus
>> > calls there need to time out (they take 30s to time out btw).
>> >
>> > Anyway, here's the backtrace: https://paste.kde.org/pajbfmoh8
>> >
>> > I saw David did something in that direction earlier but I couldn't
>> > spot exactly where's the problem, so I decided to report it here,
>> > hoping he'd know what's going on. :D
>> >
>> > Thanks!!
>> >
>> > Aleix
>>
>> Now that I know how to report a bug to kded5, I reported it. Here it
>> is, for completion:
>> https://bugs.kde.org/show_bug.cgi?id=358017
>
> Thiago: indeed the BlockingQueuedConnection call from messageFilter to the
> main thread occasionnally leads to deadlocks, we need your
> "postpone incoming messages" idea.

It's good to know there's a plan, would it maybe be possible to expand
a bit? I'd be interested in looking into it as my system is quite
unusable and I can expect other people will become soon enough.

Aleix
___
Kde-frameworks-devel mailing list
Kde-frameworks-devel@kde.org
https://mail.kde.org/mailman/listinfo/kde-frameworks-devel

Re: Deadlock in kded whith Qt 5.6

2016-01-19 Thread Thiago Macieira 
On Monday 18 January 2016 09:23:51 David Faure wrote:
> Thiago: indeed the BlockingQueuedConnection call from messageFilter to the
> main thread occasionnally leads to deadlocks, we need your
> "postpone incoming messages" idea.

Please try the attached patch, also found at
https://codereview.qt-project.org/146639
-- 
Thiago Macieira - thiago (AT) macieira.info - thiago (AT) kde.org
   Software Architect - Intel Open Source Technology Center
  PGP/GPG: 0x6EF45358; fingerprint:
  E067 918B B660 DBD1 105C  966C 33F5 F005 6EF4 5358
>From fafa4b3b89a02e136f8f4af8455ee2fd8a7cf9f4 Mon Sep 17 00:00:00 2001
From: Thiago Macieira 
Date: Tue, 19 Jan 2016 22:29:33 -0800
Subject: [PATCH 1/1] Call out to QtDBus message spies in the main thread

Whenever there are spies installed, we call out to the main thread to
call to the kded/kiod message spies. This allows the spy code to do just
about anything, where previously it was restricted in what it could do
to avoid deadlocking or triggering assertions if it recursed back into
QDBusConnection code in the manager thread.

After the spies are done with, the message is re-inserted into the
QDBusConnection processing pipeline. That means it may be delayed again
if the dispatching is disabled.

Change-Id: I3d11545be52c43119f0f142b0e9d447415c2
---
 src/dbus/qdbusconnection_p.h |  4 +++-
 src/dbus/qdbusintegrator_p.h | 19 +
 src/dbus/qdbusintegrator.cpp | 51 ++--
 3 files changed, 62 insertions(+), 12 deletions(-)

diff --git a/src/dbus/qdbusconnection_p.h b/src/dbus/qdbusconnection_p.h
index f030a3f..d7787fa 100644
--- a/src/dbus/qdbusconnection_p.h
+++ b/src/dbus/qdbusconnection_p.h
@@ -1,7 +1,7 @@
 /
 **
 ** Copyright (C) 2015 The Qt Company Ltd.
-** Copyright (C) 2015 Intel Corporation.
+** Copyright (C) 2016 Intel Corporation.
 ** Contact: http://www.qt.io/licensing/
 **
 ** This file is part of the QtDBus module of the Qt Toolkit.
@@ -228,6 +228,7 @@ public:
 void registerService(const QString );
 void unregisterService(const QString );
 
+bool preHandleMessage(const QDBusMessage );
 bool handleMessage(const QDBusMessage );
 
 QDBusMetaObject *findMetaObject(const QString , const QString ,
@@ -281,6 +282,7 @@ private slots:
 
 signals:
 void dispatchStatusChanged();
+void messageNeedsHandling(const QDBusMessage );
 void messageNeedsSending(QDBusPendingCallPrivate *pcall, void *msg, int timeout = -1);
 void signalNeedsConnecting(const QString , const QDBusConnectionPrivate::SignalHook );
 bool signalNeedsDisconnecting(const QString , const QDBusConnectionPrivate::SignalHook );
diff --git a/src/dbus/qdbusintegrator_p.h b/src/dbus/qdbusintegrator_p.h
index 2038ac9..e266512 100644
--- a/src/dbus/qdbusintegrator_p.h
+++ b/src/dbus/qdbusintegrator_p.h
@@ -1,6 +1,7 @@
 /
 **
 ** Copyright (C) 2015 The Qt Company Ltd.
+** Copyright (C) 2016 Intel Corporation.
 ** Contact: http://www.qt.io/licensing/
 **
 ** This file is part of the QtDBus module of the Qt Toolkit.
@@ -65,6 +66,7 @@
 QT_BEGIN_NAMESPACE
 
 class QDBusConnectionPrivate;
+class QDBusMessage;
 
 // Really private structs used by qdbusintegrator.cpp
 // Things that aren't used by any other file
@@ -133,6 +135,23 @@ private:
 bool handled;
 };
 
+class QDBusSpyCallEvent : public QMetaCallEvent
+{
+public:
+typedef void (*Hook)(const QDBusMessage&);
+QDBusSpyCallEvent(QDBusConnectionPrivate *cp, const QDBusConnection , const QDBusMessage ,
+  const Hook *hooks, int count)
+: QMetaCallEvent(0, 0, Q_NULLPTR, cp, 0), conn(c), msg(msg), hooks(hooks), hookCount(count)
+{}
+~QDBusSpyCallEvent();
+void placeMetaCall(QObject *) Q_DECL_OVERRIDE;
+
+QDBusConnection conn;   // keeps the refcount in QDBusConnectionPrivate up
+QDBusMessage msg;
+const Hook *hooks;
+int hookCount;
+};
+
 QT_END_NAMESPACE
 
 Q_DECLARE_METATYPE(QDBusSlotCache)
diff --git a/src/dbus/qdbusintegrator.cpp b/src/dbus/qdbusintegrator.cpp
index dffdfe2..652924f 100644
--- a/src/dbus/qdbusintegrator.cpp
+++ b/src/dbus/qdbusintegrator.cpp
@@ -1,7 +1,7 @@
 /
 **
 ** Copyright (C) 2015 The Qt Company Ltd.
-** Copyright (C) 2015 Intel Corporation.
+** Copyright (C) 2016 Intel Corporation.
 ** Contact: http://www.qt.io/licensing/
 **
 ** This file is part of the QtDBus module of the Qt Toolkit.
@@ -120,8 +120,7 @@ void qdbusDefaultThreadDebug(int action, int condition, QDBusConnectionPrivate *
 qdbusThreadDebugFunc qdbusThreadDebug = 0;
 #endif
 
-typedef void (*QDBusSpyHook)(const QDBusMessage&);
-typedef QVarLengthArray QDBusSpyHookList;
+typedef QVarLengthArray QDBusSpyHookList;

Re: Deadlock in kded whith Qt 5.6

2016-01-19 Thread David Faure 
On Tuesday 19 January 2016 22:54:35 Thiago Macieira wrote:
> On Monday 18 January 2016 09:23:51 David Faure wrote:
> > Thiago: indeed the BlockingQueuedConnection call from messageFilter to the
> > main thread occasionnally leads to deadlocks, we need your
> > "postpone incoming messages" idea.
> 
> Please try the attached patch, also found at
>   https://codereview.qt-project.org/146639

Doesn't seem to support service autostart + module autoload together.

If kded5 isn't running, then
$ qdbus org.kde.kcookiejar5  
works (autostart OK)
and then
$ qdbus org.kde.kcookiejar5  /modules/kcookiejar
works (module autoload OK)

but if I kill kded5 again and do this right away:
$ qdbus org.kde.kcookiejar5  /modules/kcookiejar
then it doesn't work: kded5 gets started, but the call gets this reply
Error: org.freedesktop.DBus.Error.UnknownObject
No such object path '/modules/kcookiejar'

This is obviously after reverting my BlockingQueuedConnection hacks in kded and 
kiod.


Same issue with kiod (who has the same logic, but starts much faster).
$ qdbus org.kde.kpasswdserver /modules/kpasswdserver
Error: org.freedesktop.DBus.Error.UnknownObject
No such object path '/modules/kpasswdserver'
$ qdbus org.kde.kpasswdserver /modules/kpasswdserver
works the second time.

The messageFilter doesn't run at all, when doing autostart+autoload together.

-- 
David Faure, fa...@kde.org, http://www.davidfaure.fr
Working on KDE Frameworks 5

___
Kde-frameworks-devel mailing list
Kde-frameworks-devel@kde.org
https://mail.kde.org/mailman/listinfo/kde-frameworks-devel

Re: Deadlock in kded whith Qt 5.6

2016-01-19 Thread Andreas Hartmetz 
On Dienstag, 19. Januar 2016 10:02:33 CET Aleix Pol wrote:
> On Mon, Jan 18, 2016 at 9:23 AM, David Faure  wrote:
> > On Friday 15 January 2016 13:48:14 Aleix Pol wrote:
> >> On Fri, Jan 15, 2016 at 1:02 PM, Aleix Pol  
wrote:
> >> > Hi,
> >> > I realized earlier last week that some of my applications were
> >> > freezing, it turns out it's because my kded is freezing and any
> >> > dbus
> >> > calls there need to time out (they take 30s to time out btw).
> >> > 
> >> > Anyway, here's the backtrace: https://paste.kde.org/pajbfmoh8
> >> > 
> >> > I saw David did something in that direction earlier but I
> >> > couldn't
> >> > spot exactly where's the problem, so I decided to report it here,
> >> > hoping he'd know what's going on. :D
> >> > 
> >> > Thanks!!
> >> > 
> >> > Aleix
> >> 
> >> Now that I know how to report a bug to kded5, I reported it. Here
> >> it
> >> is, for completion:
> >> https://bugs.kde.org/show_bug.cgi?id=358017
> > 
> > Thiago: indeed the BlockingQueuedConnection call from messageFilter
> > to the main thread occasionnally leads to deadlocks, we need your
> > "postpone incoming messages" idea.
> 
> It's good to know there's a plan, would it maybe be possible to expand
> a bit? I'd be interested in looking into it as my system is quite
> unusable and I can expect other people will become soon enough.
> 
I briefly talked about this with David yesterday and looked at the code.
What happened previously in kded5:
- it installs a message filter using libdbus API
- when a message to a not-yet-loaded module comes along, the module
  is loaded on the fly and directly from the message filter function 
  before the message is allowed to continue on its way

What happens now:
- the D-Bus connection lives in its own thread, in order to facilitate 
  using the connection from arbitrary threads in the application
- this puts the message filter in the D-Bus thread
- multithreading issues ensue

At the moment, you can pick between various poisons and eventually you 
have two threads waiting for each other and kded5 is mostly deadlocked.
The suggested solution is probably to *queue* messages to not-yet-loaded 
modules and to continue processing other D-Bus message while the 
required modules for the queued message are loading. That would break 
reasonable expectations about message ordering and blocking the least, I 
guess.

Ceteri censeo...
If you do anything blocking in D-Bus, you are going to have a bad time 
unless you have made sure that it is not so. "What could possibly go 
wrong?" is not good enough. Exceptions are calls to quickly replying 
services that are guaranteed to never make D-Bus calls of their own (*), 
and messages to the D-Bus daemon itself.
Please do not use blocking D-Bus calls unless you REALLY know what you 
are doing.
Now I've shamed myself into writing that documentation patch for QtDBus 
that I have wanted to write and submit since a year or three. It should 
have a big fat warning.
Asynchronous code is more tedious but it is in the general case the only 
correct way to make D-Bus calls.

Regarding the current situation, the synchronous module loading was okay 
before the Qt change unless modules somehow made D-Bus calls to 
something else than the daemon (to register their address) just by being 
loaded.
We have had lots of kded5 issues due to blocking calls over the years 
but I don'r remember seeing module loading in any backtraces.

(*) That is where deadlocks happen - directly or indirectly calling out 
to something that is in turn waiting for you.
___
Kde-frameworks-devel mailing list
Kde-frameworks-devel@kde.org
https://mail.kde.org/mailman/listinfo/kde-frameworks-devel

Re: Deadlock in kded whith Qt 5.6

2016-01-19 Thread Thiago Macieira 
On Tuesday 19 January 2016 15:56:47 you wrote:
> I briefly talked about this with David yesterday and looked at the code.
> What happened previously in kded5:
> - it installs a message filter using libdbus API

QtDBus API. In fact, it's QtDBus API that exists and has existed only for kded 
(and now kiod).

> - when a message to a not-yet-loaded module comes along, the module
>   is loaded on the fly and directly from the message filter function
>   before the message is allowed to continue on its way
> 
> What happens now:
> - the D-Bus connection lives in its own thread, in order to facilitate
>   using the connection from arbitrary threads in the application
> - this puts the message filter in the D-Bus thread
> - multithreading issues ensue

Because the new QtDBus code assumes that certain functions will always be 
called from any-thread-but-QDBusConnectionManager's, as the code is very 
careful to not ever call to user code in the manager's thread. The spy 
function breaks that promise and I didn't catch it when designing the threaded 
model.

> At the moment, you can pick between various poisons and eventually you
> have two threads waiting for each other and kded5 is mostly deadlocked.
> The suggested solution is probably to *queue* messages to not-yet-loaded
> modules and to continue processing other D-Bus message while the
> required modules for the queued message are loading. That would break
> reasonable expectations about message ordering and blocking the least, I
> guess.

Actually, we'll queue them all, since we don't know which messages trigger 
loading and which ones don't. So for every incoming message, we'll call out to 
the spy function in the main thread and, after it returns, we'll reinject the 
message into the normal QtDBus processing.

> Regarding the current situation, the synchronous module loading was okay
> before the Qt change unless modules somehow made D-Bus calls to
> something else than the daemon (to register their address) just by being
> loaded.
> We have had lots of kded5 issues due to blocking calls over the years
> but I don'r remember seeing module loading in any backtraces.

-- 
Thiago Macieira - thiago (AT) macieira.info - thiago (AT) kde.org
   Software Architect - Intel Open Source Technology Center
  PGP/GPG: 0x6EF45358; fingerprint:
  E067 918B B660 DBD1 105C  966C 33F5 F005 6EF4 5358

___
Kde-frameworks-devel mailing list
Kde-frameworks-devel@kde.org
https://mail.kde.org/mailman/listinfo/kde-frameworks-devel

Re: Deadlock in kded whith Qt 5.6

2016-01-18 Thread David Faure 
On Friday 15 January 2016 13:48:14 Aleix Pol wrote:
> On Fri, Jan 15, 2016 at 1:02 PM, Aleix Pol  wrote:
> > Hi,
> > I realized earlier last week that some of my applications were
> > freezing, it turns out it's because my kded is freezing and any dbus
> > calls there need to time out (they take 30s to time out btw).
> >
> > Anyway, here's the backtrace: https://paste.kde.org/pajbfmoh8
> >
> > I saw David did something in that direction earlier but I couldn't
> > spot exactly where's the problem, so I decided to report it here,
> > hoping he'd know what's going on. :D
> >
> > Thanks!!
> >
> > Aleix
> 
> Now that I know how to report a bug to kded5, I reported it. Here it
> is, for completion:
> https://bugs.kde.org/show_bug.cgi?id=358017

Thiago: indeed the BlockingQueuedConnection call from messageFilter to the
main thread occasionnally leads to deadlocks, we need your
"postpone incoming messages" idea.

-- 
David Faure, fa...@kde.org, http://www.davidfaure.fr
Working on KDE Frameworks 5

___
Kde-frameworks-devel mailing list
Kde-frameworks-devel@kde.org
https://mail.kde.org/mailman/listinfo/kde-frameworks-devel

Re: Deadlock in kded whith Qt 5.6

2016-01-15 Thread Aleix Pol 
On Fri, Jan 15, 2016 at 1:02 PM, Aleix Pol  wrote:
> Hi,
> I realized earlier last week that some of my applications were
> freezing, it turns out it's because my kded is freezing and any dbus
> calls there need to time out (they take 30s to time out btw).
>
> Anyway, here's the backtrace: https://paste.kde.org/pajbfmoh8
>
> I saw David did something in that direction earlier but I couldn't
> spot exactly where's the problem, so I decided to report it here,
> hoping he'd know what's going on. :D
>
> Thanks!!
>
> Aleix

Now that I know how to report a bug to kded5, I reported it. Here it
is, for completion:
https://bugs.kde.org/show_bug.cgi?id=358017

Aleix
___
Kde-frameworks-devel mailing list
Kde-frameworks-devel@kde.org
https://mail.kde.org/mailman/listinfo/kde-frameworks-devel

Deadlock in kded whith Qt 5.6

2016-01-15 Thread Aleix Pol 
Hi,
I realized earlier last week that some of my applications were
freezing, it turns out it's because my kded is freezing and any dbus
calls there need to time out (they take 30s to time out btw).

Anyway, here's the backtrace: https://paste.kde.org/pajbfmoh8

I saw David did something in that direction earlier but I couldn't
spot exactly where's the problem, so I decided to report it here,
hoping he'd know what's going on. :D

Thanks!!

Aleix
___
Kde-frameworks-devel mailing list
Kde-frameworks-devel@kde.org
https://mail.kde.org/mailman/listinfo/kde-frameworks-devel