Bug#761867: Webapp: Add conjunctive search for ctags

2015-03-12 Thread Orestis Ioannou
Hello,

I have attached the patch for the conjunctive search.
Raw sql commands are the following:

First I find files with the ctags. (there are 2 tags in the following case)

SELECT ctags.file_id AS file_id
FROM ctags JOIN files ON files.id = ctags.file_id
WHERE ctags.tag IN (:tag_1, :tag_2) GROUP BY ctags.file_id
HAVING count(DISTINCT ctags.tag) = 2

then i use the ids i recovered to find the package names etc. (there are
4 ids as the first command had 4 results.)

SELECT files.id AS file_id,
package_names.name AS package,
packages.version AS version, files.path AS path
FROM files, package_names, packages
WHERE files.package_id = packages.id
AND files.id IN (:id_1, :id_2, :id_3, :id_4)
AND packages.name_id = package_names.id

Cheers
From 34e6c018afeba3e633afebe80a77ca4e9a777a59 Mon Sep 17 00:00:00 2001
From: Orestis Ioannou 
Date: Thu, 12 Mar 2015 13:53:34 +0100
Subject: [PATCH] Webapp: Add conjunctive search for ctags

Closes: #761867
---
 debsources/app/sources/routes.py   |  9 +
 .../app/sources/templates/sources/test_ctag.html   | 43 ++
 debsources/app/views.py| 16 ++--
 debsources/models.py   | 32 
 debsources/tests/test_webapp.py|  7 
 5 files changed, 104 insertions(+), 3 deletions(-)
 create mode 100644 debsources/app/sources/templates/sources/test_ctag.html

diff --git a/debsources/app/sources/routes.py b/debsources/app/sources/routes.py
index 1ac8282..eec69c3 100644
--- a/debsources/app/sources/routes.py
+++ b/debsources/app/sources/routes.py
@@ -198,6 +198,15 @@ bp_sources.add_url_rule(
 err_func=ErrorHandler('sources'),
 pagination=True))
 
+# CtagView
+bp_sources.add_url_rule(
+'/test_ctag/',
+view_func=CtagView.as_view(
+'test_ctag',
+render_func=bind_render('sources/test_ctag.html'),
+err_func=ErrorHandler('sources'),
+pagination=True))
+
 
 # api
 bp_sources.add_url_rule(
diff --git a/debsources/app/sources/templates/sources/test_ctag.html b/debsources/app/sources/templates/sources/test_ctag.html
new file mode 100644
index 000..fa5e41e
--- /dev/null
+++ b/debsources/app/sources/templates/sources/test_ctag.html
@@ -0,0 +1,43 @@
+{#
+  Copyright (C) 2013  Matthieu Caneill 
+  License: GNU Affero General Public License, version 3 or above.
+#}
+{# copied from templates/ctag.html #}
+
+{% extends "sources/base.html" %}
+n
+{% block title %}Ctag: {{ ctag }}
+{% if package %}(in package {{ package }}){% endif %}
+(page {{ page }}){% endblock %}
+
+{% block breadcrumbs %}ctag / {{ sha256 }}{% endblock %}
+
+{% block content %}
+
+{{ self.title() }}
+
+{{ count }} result{% if count >= 2 %}s{% endif %}:
+
+
+  {% for result in results %}
+
+  {% if conjunctive %}
+  
+  {{ result.package }}/{{ result.version }}/{{ result.path }}
+  {% else %}
+   
+   {{ result.package }}/{{ result.version }}/{{ result.path }}
+  {% endif %}
+ 
+  {% endfor %}
+
+
+{{ macros.render_pagination(pagination) }}
+
+{% endblock %}
diff --git a/debsources/app/views.py b/debsources/app/views.py
index ba27494..99bc2b8 100644
--- a/debsources/app/views.py
+++ b/debsources/app/views.py
@@ -425,8 +425,17 @@ class CtagView(GeneralView):
 else:
 pagination = None
 slice_ = None
-count, results = Ctag.find_ctag(session, ctag, slice_=slice_,
-package=package)
+
+conjunctive = False
+if ',' in ctag:
+ctags = ctag.split(',')
+conjunctive = True
+(count, results) = Ctag.conjunctive_search(session,
+   ctags, slice_=slice_,
+   package=package)
+else:
+(count, results) = Ctag.find_ctag(session, ctag, slice_=slice_,
+  package=package)
 if self.d.get('pagination'):
 pagination = Pagination(page, offset, count)
 else:
@@ -437,7 +446,8 @@ class CtagView(GeneralView):
 count=count,
 page=page,
 package=package,
-pagination=pagination)
+pagination=pagination,
+conjunctive=conjunctive)
 
 
 class PrefixView(GeneralView):
diff --git a/debsources/models.py b/debsources/models.py
index e4d05fc..94d4f42 100644
--- a/debsources/models.py
+++ b/debsources/models.py
@@ -386,6 +386,38 @@ class Ctag(Base):
for res in results.all()]
 return (count, results)
 
+@staticmethod
+def conjunctive_search(session, ctags, package=None, slice_=None):
+find_files = (session.query(Ctag.file_id.label("file_id"))
+  .join(File)
+  .filter(Ctag.tag.in_(ctags))
+  .group_by(Ctag.file_id)
+   

Bug#761867: Webapp: Add conjunctive search for ctags

2015-03-13 Thread Stefano Zacchiroli
[ just documenting here a quick summary of IRC discussions ]

On Thu, Mar 12, 2015 at 02:33:03PM +0100, Orestis Ioannou wrote:
> I have attached the patch for the conjunctive search.
> Raw sql commands are the following:

The patch looks good, but there are concerns that this feature will
allow to DoS the DB, given the ctags table currently contains ~380
million tuples.

We've pinged friendly neighborhood Postgres experts (Myon, mnencia) who
are looking into this to give feedback about: whether the feature is a
good idea at all and/or whether there are better ways to implement this
query.

I duly notice that we already have non-conjunctive ctags-based search.
And it's not given us problems thus far. If we can make the conjunctive
query, performance-wise, "as bad as" the non-conjunctive version, then
it's probably good to do.

Cheers.
-- 
Stefano Zacchiroli  . . . . . . .  z...@upsilon.cc . . . . o . . . o . o
Maître de conférences . . . . . http://upsilon.cc/zack . . . o . . . o o
Former Debian Project Leader  . . @zack on identi.ca . . o o o . . . o .
« the first rule of tautology club is the first rule of tautology club »


-- 
To UNSUBSCRIBE, email to debian-qa-requ...@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org
Archive: https://lists.debian.org/20150313104753.ga27...@upsilon.cc