Hey, I spent today to improve the searching situation a bit, but couldn't fix all problems yet. It's a more of a workaround than a real fix as you can see but at least basic searching works this way. A better solution will need changes in some more places and probably nepomuk-core available (As I don't really feel like copying more stuff from the nepomuk repository).
Additionally to nepomuksearchengine I also fixed nepomuksearch, although I don't know if it is used at all (I also haven't tested this part). I also added on commit to kdepim (cbd7a8777a4bfd89125be4cf58e276b51388adc5) which quotes the search string, because otherwise a searchpattern consisting of several words is not interpreted as one pattern but rather the AND- conjunction of each individual word. Apart from the current solution being a rather ugly hack, there also exist two other problems: - The search collection is indexed and then searched which is probably not what we want. If somebody can tell me how to determine the search collection (by some attribute or so) I could exclude it from the feeder. Another option would be to exclude it by adding a IndexPolicyAttribute to the search collection. - Each query gets somehow executed twice. I can't see the call twice on the kmail side, but on the akonadi-server there are always two queries, no idea why. Apart from those issues and the feeder not having indexed all emails here it works pretty well. The search itself seems pretty fast: ~1-3 seconds for a query resulting in a small result set, which searches through fulltext/and headers in ~80000 indexed emails. If the result set is large filling the table seems to be much more of a bottleneck than the actual query. If the attached fixes are applied to akonadi and we manage to exclude the search collection, we'll have at least half-way decent searching in 4.8. Cheers, Christian
>From 16bfabe51581517321179c6db0a36aad97a25ea9 Mon Sep 17 00:00:00 2001 From: Christian Mollekopf <[email protected]> Date: Thu, 19 Jan 2012 01:05:25 +0100 Subject: [PATCH] Use the akonadiItemId property to get the item id. The assumption that the nepomuk uri is the same as the akonadi uri holds no longer true. The requested property would ideally be added to the existing query, instead we rely on the caller to request the property for now or use a second query. --- server/src/nepomuk/org.kde.nepomuk.Query.xml | 5 ++- server/src/nepomuk/queryserviceclient.cpp | 25 ++++---------- server/src/nepomuk/queryserviceclient.h | 8 ++-- server/src/nepomuksearch.cpp | 37 ++++++++++++++------- server/src/nepomuksearch.h | 2 + server/src/search/nepomuksearchengine.cpp | 46 ++++++++++++++----------- server/src/search/nepomuksearchengine.h | 2 +- 7 files changed, 68 insertions(+), 57 deletions(-) diff --git a/server/src/nepomuk/org.kde.nepomuk.Query.xml b/server/src/nepomuk/org.kde.nepomuk.Query.xml index 96e131a..24b6c67 100644 --- a/server/src/nepomuk/org.kde.nepomuk.Query.xml +++ b/server/src/nepomuk/org.kde.nepomuk.Query.xml @@ -12,11 +12,12 @@ <arg type="s" direction="out" /> </method> <signal name="newEntries"> - <arg name="entries" type="a(sda{s(isss)}s)" /> + <arg name="entries" type="a(sda{s(isss)})" /> <annotation name="com.trolltech.QtDBus.QtTypeName.In0" value="QList<Nepomuk::Query::Result>" /> </signal> <signal name="entriesRemoved"> - <arg name="entries" type="as" /> + <arg name="entries" type="a(sda{s(isss)}s)" /> + <annotation name="com.trolltech.QtDBus.QtTypeName.In0" value="QList<Nepomuk::Query::Result>" /> </signal> <signal name="resultCount"> <arg name="count" type="i" /> diff --git a/server/src/nepomuk/queryserviceclient.cpp b/server/src/nepomuk/queryserviceclient.cpp index 9ddda4e..74daa48 100644 --- a/server/src/nepomuk/queryserviceclient.cpp +++ b/server/src/nepomuk/queryserviceclient.cpp @@ -30,6 +30,7 @@ #include <QtCore/QEventLoop> #include <QtCore/QTimer> +#include <KDE/KUrl> namespace { @@ -83,7 +84,6 @@ public: loop( 0 ) { } - void _k_entriesRemoved( const QStringList& ); void _k_finishedListing(); bool handleQueryReply( QDBusReply<QDBusObjectPath> reply ); @@ -97,17 +97,6 @@ public: QEventLoop* loop; }; - -void Nepomuk::Query::QueryServiceClient::Private::_k_entriesRemoved( const QStringList& uris ) -{ - QList<QUrl> ul; - foreach( const QString& s, uris ) { - ul.append( QUrl( s ) ); - } - emit q->entriesRemoved( ul ); -} - - void Nepomuk::Query::QueryServiceClient::Private::_k_finishedListing() { emit q->finishedListing(); @@ -124,8 +113,8 @@ bool Nepomuk::Query::QueryServiceClient::Private::handleQueryReply( QDBusReply<Q dbusConnection ); connect( queryInterface, SIGNAL(newEntries(QList<Nepomuk::Query::Result>)), q, SIGNAL(newEntries(QList<Nepomuk::Query::Result>)) ); - connect( queryInterface, SIGNAL(entriesRemoved(QStringList)), - q, SLOT(_k_entriesRemoved(QStringList)) ); + connect( queryInterface, SIGNAL(entriesRemoved(QList<Nepomuk::Query::Result>)), + q, SIGNAL(entriesRemoved(QList<Nepomuk::Query::Result>)) ); connect( queryInterface, SIGNAL(finishedListing()), q, SLOT(_k_finishedListing()) ); // run the listing async in case the event loop below is the only one we have @@ -163,12 +152,12 @@ Nepomuk::Query::QueryServiceClient::~QueryServiceClient() } -bool Nepomuk::Query::QueryServiceClient::query( const QString& query ) +bool Nepomuk::Query::QueryServiceClient::query( const QString& query, const QHash<QString, QString> &encodedRps ) { close(); if ( d->queryServiceInterface->isValid() ) { - return d->handleQueryReply( d->queryServiceInterface->sparqlQuery( query, QHash<QString, QString>() ) ); + return d->handleQueryReply( d->queryServiceInterface->sparqlQuery( query, encodedRps ) ); } else { akDebug() << "Could not contact query service."; @@ -177,9 +166,9 @@ bool Nepomuk::Query::QueryServiceClient::query( const QString& query ) } -bool Nepomuk::Query::QueryServiceClient::blockingQuery( const QString& q ) +bool Nepomuk::Query::QueryServiceClient::blockingQuery( const QString& q, const QHash<QString, QString> &encodedRps ) { - if( query( q ) ) { + if( query( q, encodedRps ) ) { QEventLoop loop; d->loop = &loop; loop.exec(); diff --git a/server/src/nepomuk/queryserviceclient.h b/server/src/nepomuk/queryserviceclient.h index c55c0eb..a0e7517 100644 --- a/server/src/nepomuk/queryserviceclient.h +++ b/server/src/nepomuk/queryserviceclient.h @@ -20,6 +20,7 @@ #define _NEPOMUK_QUERY_SERVICE_CLIENT_H_ #include <QtCore/QObject> +#include <QtCore/QHash> class QUrl; @@ -77,7 +78,7 @@ namespace Nepomuk { * * \sa QueryParser */ - bool query( const QString& query ); + bool query( const QString& query, const QHash<QString, QString> &encodedRps = QHash<QString, QString>() ); /** * Start a query using the Nepomuk user query language. @@ -95,7 +96,7 @@ namespace Nepomuk { * * \sa query(const QString&), close() */ - bool blockingQuery( const QString& query ); + bool blockingQuery( const QString& query, const QHash<QString, QString> &encodedRps = QHash<QString, QString>() ); /** * Close the client, thus stop to monitor the query @@ -118,7 +119,7 @@ namespace Nepomuk { * \param entries A list of resource URIs identifying the resources * that dropped out of the query results. */ - void entriesRemoved( const QList<QUrl>& entries ); + void entriesRemoved( const QList<Nepomuk::Query::Result>& entries ); /** * Emitted when the initial listing has been finished, ie. if all @@ -131,7 +132,6 @@ namespace Nepomuk { class Private; Private* const d; - Q_PRIVATE_SLOT( d, void _k_entriesRemoved( const QStringList& ) ) Q_PRIVATE_SLOT( d, void _k_finishedListing() ) }; } diff --git a/server/src/nepomuksearch.cpp b/server/src/nepomuksearch.cpp index 5b12829..578e926 100644 --- a/server/src/nepomuksearch.cpp +++ b/server/src/nepomuksearch.cpp @@ -27,19 +27,16 @@ using namespace Akonadi; -static qint64 uriToItemId( const QUrl &url ) +static qint64 resultToId( const Nepomuk::Query::Result &result ) { - bool ok = false; - - const qint64 id = url.queryItemValue( QLatin1String( "item" ) ).toLongLong( &ok ); - - // We don't want attachments - ok = ok && !url.hasFragment(); - - if ( !ok ) + const Soprano::Node &property = result.requestProperty(QUrl( QLatin1String( "http://akonadi-project.org/ontologies/aneo#akonadiItemId"; ) )); + if (!(property.isValid() && property.isLiteral() && property.literal().isString())) { + qWarning() << "Failed to get requested akonadiItemId property"; + qDebug() << "AkonadiItemId missing in query results!" << result.resourceUri() << property.isValid() << property.isLiteral() << property.literal().isString() << property.literal().type() << result.requestProperties().size(); + qDebug() << result.requestProperties().values().first().toString(); return -1; - else - return id; + } + return property.literal().toString().toLongLong(); } NepomukSearch::NepomukSearch( QObject* parent ) @@ -47,6 +44,9 @@ NepomukSearch::NepomukSearch( QObject* parent ) { mSearchService = new Nepomuk::Query::QueryServiceClient( this ); connect( mSearchService, SIGNAL(newEntries(QList<Nepomuk::Query::Result>)), + this, SLOT(idJitsAdded(QList<Nepomuk::Query::Result>)) ); + mIdSearchService = new Nepomuk::Query::QueryServiceClient( this ); + connect( mSearchService, SIGNAL(newEntries(QList<Nepomuk::Query::Result>)), this, SLOT(hitsAdded(QList<Nepomuk::Query::Result>)) ); } @@ -81,13 +81,26 @@ void NepomukSearch::hitsAdded( const QList<Nepomuk::Query::Result>& entries ) } Q_FOREACH( const Nepomuk::Query::Result &result, entries ) { - const qint64 itemId = uriToItemId( result.resourceUri() ); + QHash<QString, QString> encodedRps; + //We do another query to get the akonadiItemId attribute (since it may not have been added to the original query) + encodedRps.insert( QString::fromLatin1( "reqProp1" ), QUrl(QString::fromLatin1("http://akonadi-project.org/ontologies/aneo#akonadiItemId";)).toString() ); + mIdSearchService->blockingQuery(QString::fromLatin1("select distinct ?r ?reqProp1 where { { ?r a ?v2 . FILTER(?r=<%1>) . { ?r <http://akonadi-project.org/ontologies/aneo#akonadiItemId> ?reqProp1 . } } . ?r <http://www.semanticdesktop.org/ontologies/2007/08/15/nao#userVisible> ?v1 . FILTER(?v1>0) . } LIMIT 1").arg(result.resourceUri().toString()), encodedRps); + } +} + +void NepomukSearch::idHitsAdded(const QList< Nepomuk::Query::Result >& entries) +{ + + Q_FOREACH( const Nepomuk::Query::Result &result, entries ) { + const qint64 itemId = resultToId( result ); if ( itemId == -1 ) continue; mMatchingUIDs.insert( QString::number( itemId ) ); } + } + #include "nepomuksearch.moc" diff --git a/server/src/nepomuksearch.h b/server/src/nepomuksearch.h index 25c331f..a596db3 100644 --- a/server/src/nepomuksearch.h +++ b/server/src/nepomuksearch.h @@ -40,10 +40,12 @@ class NepomukSearch : public QObject private Q_SLOTS: void hitsAdded( const QList<Nepomuk::Query::Result>& entries ); + void idHitsAdded( const QList<Nepomuk::Query::Result>& entries ); private: QSet<QString> mMatchingUIDs; Nepomuk::Query::QueryServiceClient* mSearchService; + Nepomuk::Query::QueryServiceClient* mIdSearchService; }; } diff --git a/server/src/search/nepomuksearchengine.cpp b/server/src/search/nepomuksearchengine.cpp index bf7199e..58d562e 100644 --- a/server/src/search/nepomuksearchengine.cpp +++ b/server/src/search/nepomuksearchengine.cpp @@ -36,19 +36,16 @@ using namespace Akonadi; -static qint64 uriToItemId( const QUrl &url ) +static qint64 resultToId( const Nepomuk::Query::Result &result ) { - bool ok = false; - - const qint64 id = url.queryItemValue( QLatin1String( "item" ) ).toLongLong( &ok ); - - // We don't want attachments - ok = ok && !url.hasFragment(); - - if ( !ok ) + const Soprano::Node &property = result.requestProperty(QUrl( QLatin1String( "http://akonadi-project.org/ontologies/aneo#akonadiItemId"; ) )); + if (!(property.isValid() && property.isLiteral() && property.literal().isString())) { + qWarning() << "Failed to get requested akonadiItemId property"; + qDebug() << "AkonadiItemId missing in query results!" << result.resourceUri() << property.isValid() << property.isLiteral() << property.literal().isString() << property.literal().type() << result.requestProperties().size(); + qDebug() << result.requestProperties().values().first().toString(); return -1; - else - return id; + } + return property.literal().toString().toLongLong(); } NepomukSearchEngine::NepomukSearchEngine( QObject* parent ) @@ -80,21 +77,31 @@ void NepomukSearchEngine::addSearch( const Collection &collection ) { if ( collection.queryLanguage() != QLatin1String( "SPARQL" ) ) return; + const QString &q = collection.queryString(); + + //FIXME the requested property must be passed to here from the calling code + //Ideally the Query is passed as object so we can check here for the akonadiItemId property, and add it if missing + if (!q.contains(QString::fromLatin1("reqProp1")) || !q.contains(QString::fromLatin1("http://akonadi-project.org/ontologies/aneo#akonadiItemId";))) { + qWarning() << "The query MUST contain exactly one required property (http://akonadi-project.org/ontologies/aneo#akonadiItemId), if another property is additionally requested or the akonadiItemId is missing the search will fail (due to this hack)"; + qWarning() << q; + return; + } Nepomuk::Query::QueryServiceClient *query = new Nepomuk::Query::QueryServiceClient( this ); connect( query, SIGNAL(newEntries(QList<Nepomuk::Query::Result>)), this, SLOT(hitsAdded(QList<Nepomuk::Query::Result>)) ); - connect( query, SIGNAL(entriesRemoved(QList<QUrl>)), - this, SLOT(hitsRemoved(QList<QUrl>)) ); + connect( query, SIGNAL(entriesRemoved(QList<Nepomuk::Query::Result>)), + this, SLOT(hitsRemoved(QList<Nepomuk::Query::Result>)) ); mMutex.lock(); mQueryMap.insert( query, collection.id() ); mQueryInvMap.insert( collection.id(), query ); // needed for fast lookup in removeSearch() mMutex.unlock(); - // query with SPARQL statement - query->query( collection.queryString() ); + QHash<QString, QString> encodedRps; + encodedRps.insert( QString::fromLatin1( "reqProp1" ), QUrl(QString::fromLatin1("http://akonadi-project.org/ontologies/aneo#akonadiItemId";)).toString() ); //FIXME hack because the reqProp is not passed to here by the caller + query->query( collection.queryString(), encodedRps ); } void NepomukSearchEngine::removeSearch( qint64 collectionId ) @@ -169,7 +176,7 @@ void NepomukSearchEngine::hitsAdded( const QList<Nepomuk::Query::Result>& entrie const Collection collection = Collection::retrieveById( collectionId ); Q_FOREACH( const Nepomuk::Query::Result &result, entries ) { - const qint64 itemId = uriToItemId( result.resourceUri() ); + const qint64 itemId = resultToId( result ); if ( itemId == -1 ) continue; @@ -181,7 +188,7 @@ void NepomukSearchEngine::hitsAdded( const QList<Nepomuk::Query::Result>& entrie mCollector->dispatchNotifications(); } -void NepomukSearchEngine::hitsRemoved( const QList<QUrl> &entries ) +void NepomukSearchEngine::hitsRemoved( const QList<Nepomuk::Query::Result>& entries ) { Nepomuk::Query::QueryServiceClient *query = qobject_cast<Nepomuk::Query::QueryServiceClient*>( sender() ); if ( !query ) { @@ -193,9 +200,8 @@ void NepomukSearchEngine::hitsRemoved( const QList<QUrl> &entries ) qint64 collectionId = mQueryMap.value( query ); mMutex.unlock(); const Collection collection = Collection::retrieveById( collectionId ); - - Q_FOREACH( const QUrl &uri, entries ) { - const qint64 itemId = uriToItemId( uri ); + Q_FOREACH( const Nepomuk::Query::Result &result, entries ) { + const qint64 itemId = resultToId( result ); if ( itemId == -1 ) continue; diff --git a/server/src/search/nepomuksearchengine.h b/server/src/search/nepomuksearchengine.h index 491012f..40e6060 100644 --- a/server/src/search/nepomuksearchengine.h +++ b/server/src/search/nepomuksearchengine.h @@ -50,7 +50,7 @@ class NepomukSearchEngine : public QObject, public AbstractSearchEngine private Q_SLOTS: void reloadSearches(); void hitsAdded( const QList<Nepomuk::Query::Result>& entries ); - void hitsRemoved( const QList<QUrl> &entries ); + void hitsRemoved( const QList<Nepomuk::Query::Result>& entries ); private: QMutex mMutex; -- 1.7.8.3
_______________________________________________ Nepomuk mailing list [email protected] https://mail.kde.org/mailman/listinfo/nepomuk
