Bug#622117: [PATCH 2/2] Fill field builder in pkg_history
On Sun, Jun 19, 2011 at 06:23:02PM +0200, Kurt Roeckx wrote: On Sun, Jun 19, 2011 at 06:09:07PM +0200, Joachim Breitner wrote: Hi, I was under the impression that Kurt already changed the schema (20110611165217.ga21...@roeckx.be), otherwise I would not have suggested to apply the patch. Sorry for the trouble it was causing. But in the mean time the schema was updated to not have an schema per arch, but 1 big table for all arches. And there were view and rules created for the old tables. But the new views (and rules) don't know anything about builder. They do so now. Kind regards, Philipp Kern -- .''`. Philipp KernDebian Developer : :' : http://philkern.de Stable Release Manager `. `' xmpp:p...@0x539.de Wanna-Build Admin `-finger pkern/k...@db.debian.org signature.asc Description: Digital signature
Bug#622117: [PATCH 2/2] Fill field builder in pkg_history
Hi, Am Donnerstag, den 21.07.2011, 11:27 +0200 schrieb Philipp Kern: On Sun, Jun 19, 2011 at 06:23:02PM +0200, Kurt Roeckx wrote: On Sun, Jun 19, 2011 at 06:09:07PM +0200, Joachim Breitner wrote: Hi, I was under the impression that Kurt already changed the schema (20110611165217.ga21...@roeckx.be), otherwise I would not have suggested to apply the patch. Sorry for the trouble it was causing. But in the mean time the schema was updated to not have an schema per arch, but 1 big table for all arches. And there were view and rules created for the old tables. But the new views (and rules) don't know anything about builder. They do so now. nice. I have extended log.php to show the column (see https://buildd.debian.org/~nomeata/pgstatus/logs.php?pkg=lftp), you can merge it from grieg:~nomeata/pgstatus. I guess the next step is to try to extract the buildds from old logs, to make views like this more useful: https://buildd.debian.org/~nomeata/pgstatus/logs.php?pkg=haskell-regex-tdfaarch=armel I’ll have a look at this. I guess a one-shot script that generates the SQL instructions, which you can then review, would be enough? Greetings, Joachim -- Joachim nomeata Breitner Debian Developer nome...@debian.org | ICQ# 74513189 | GPG-Keyid: 4743206C JID: nome...@joachim-breitner.de | http://people.debian.org/~nomeata signature.asc Description: This is a digitally signed message part
Bug#622117: [PATCH 2/2] Fill field builder in pkg_history
On 07/21/2011 02:21 PM, Joachim Breitner wrote: nice. I have extended log.php to show the column (see https://buildd.debian.org/~nomeata/pgstatus/logs.php?pkg=lftp), you can merge it from grieg:~nomeata/pgstatus. I already made a change earlier today, and is visible on the live status pages. Did you check it? I’ll have a look at this. I guess a one-shot script that generates the SQL instructions, which you can then review, would be enough? That would be nice, indeed. Regards, -- Mehdi Dogguy مهدي الدڤي http://dogguy.org/ -- To UNSUBSCRIBE, email to debian-bugs-dist-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org
Bug#622117: [PATCH 2/2] Fill field builder in pkg_history
Hi, Am Donnerstag, den 21.07.2011, 15:15 +0200 schrieb Mehdi Dogguy: On 07/21/2011 02:21 PM, Joachim Breitner wrote: nice. I have extended log.php to show the column (see https://buildd.debian.org/~nomeata/pgstatus/logs.php?pkg=lftp), you can merge it from grieg:~nomeata/pgstatus. I already made a change earlier today, and is visible on the live status pages. Did you check it? No, I didn’t, my bad. I removed my commit. You forgot to change the colspan from =6 to =7 in logs.php, though. I’ll have a look at this. I guess a one-shot script that generates the SQL instructions, which you can then review, would be enough? That would be nice, indeed. Already done, reviewed with phil, and currently running. Greetings, Joachim -- Joachim nomeata Breitner Debian Developer nome...@debian.org | ICQ# 74513189 | GPG-Keyid: 4743206C JID: nome...@joachim-breitner.de | http://people.debian.org/~nomeata signature.asc Description: This is a digitally signed message part
Bug#622117: [PATCH 2/2] Fill field builder in pkg_history
Hi,
Am Samstag, den 11.06.2011, 23:13 +0200 schrieb Joachim Breitner:
no idea what’s wrong. Maybe the test cases in
lib/debian/wannabuild/logs_tests.py don’t quite match the real world.
Since for the test suite, by print-debugging, I know that when the line
sql = tbl.insert().values(package=self.package,
distribution=self.suite,
version=self.version,
timestamp=datetime.utcfromtimestamp(self.timestamp),
result=self.state,
builder=self.builder,
build_time=self.build_time,
disk_space=self.disk_space)
is executed, self.builder has a sensible value.
I first suspected that maybe the format of the first line of the log
file has changes (which is parsed in _parse_arch), which seems to be the
case when I compare recent build logs with what’s in the test suite.
But then, the build is only extracted from the Sender field, in
_parse_sender. And even if the Sender is not in the expected format, it
should log invalid sender F:'%s' S:'%s' and set the builder to the
full content of the Sender line.
Per /org/buildd.debian.org/log/inject I see that this is not the case. I
also see the current format for the From lines ('buildd on antheil
bui...@antheil.debian.org'). If I put that in the test suite, it still
works. (Patch attached)
That’s all for now, I have no more ideas than that.
Greetings,
Joachim
--
Joachim nomeata Breitner
Debian Developer
nome...@debian.org | ICQ# 74513189 | GPG-Keyid: 4743206C
JID: nome...@joachim-breitner.de | http://people.debian.org/~nomeata
From 3bdca7edbd2f2b8b88dbdf9c14024e96bbc21177 Mon Sep 17 00:00:00 2001
From: Joachim Breitner m...@joachim-breitner.de
Date: Sun, 19 Jun 2011 14:44:56 +0200
Subject: [PATCH] Use a more current From line in test cases
---
lib/debian/wannabuild/logs_tests.py | 14 +++---
1 files changed, 7 insertions(+), 7 deletions(-)
diff --git a/lib/debian/wannabuild/logs_tests.py b/lib/debian/wannabuild/logs_tests.py
index 94d729b..86744f0 100644
--- a/lib/debian/wannabuild/logs_tests.py
+++ b/lib/debian/wannabuild/logs_tests.py
@@ -175,7 +175,7 @@ Received: from world by raff.debian.org with esmtp (Exim 4.63) (envelope-f
Subject: Log for successful build of gobby_0.4.9-2+b2 on i386 (debian/unstable)
Message-ID: 20090122115647.4A47DB8413701@buildd.something.local
Date: Thu, 22 Jan 2009 03:56:47 -0800 (PST)
-From: buildd@something.local (Debian/i386 Build Daemon)
+From: buildd on antheil bui...@antheil.debian.org
To: undisclosed-recipients:;
Delivered-To: logs@buildd.somewhere
@@ -190,7 +190,7 @@ Received: from world by raff.debian.org with esmtp (Exim 4.63) (envelope-f
Subject: Log for successful build of gobby_0.4.9-2+b2 on i386 (debian/unstable)
Message-ID: 20090122115647.4A47DB8413701@buildd.something.local
Date: Thu, 22 Jan 2009 03:56:47 -0800 (PST)
-From: buildd@something.local (Debian/i386 Build Daemon)
+From: buildd on antheil bui...@antheil.debian.org
To: undisclosed-recipients:;
Delivered-To: logs@buildd.somewhere
Content-Transfer-Encoding: base64
@@ -205,7 +205,7 @@ Received: from world by raff.debian.org with esmtp (Exim 4.63) (envelope-f
Subject: Log for successful build of gobby_0.4.9-2+b2 on i386 (debian/unstable)
Message-ID: 20090122115647.4A47DB8413701@buildd.something.local
Date: Thu, 22 Jan 2009 03:56:47 -0800 (PST)
-From: buildd@something.local (Debian/i386 Build Daemon)
+From: buildd on antheil bui...@antheil.debian.org
To: undisclosed-recipients:;
Delivered-To: logs@buildd.somewhere
Content-Transfer-Encoding: quoted-printable
@@ -243,7 +243,7 @@ sbuild (Debian sbuild) 0.58.2 (18 Mar 2009) on buildd.net.philkern.de
self.assertEqual(buildlog.version, '0.4.9-2+b2')
self.assertEqual(buildlog.arch, 'i386')
self.assertEqual(buildlog.suite, 'unstable')
-self.assertEqual(buildlog.builder, 'something.local')
+self.assertEqual(buildlog.builder, 'antheil.debian.org')
# Commit the log to the database. This should not raise any exception.
filename = buildlog.store()
self.assertEqual(buildlog.summary()[0],
@@ -409,7 +409,7 @@ Received: from world by raff.debian.org with esmtp (Exim 4.63) (envelope-f
Subject: Log for successful build of gobby_0.4.9-2 (dist=experimental)
Message-ID: 20090122115647.4A47DB8413701@buildd.something.local
Date: Thu, 22 Jan 2009 03:56:47 -0800 (PST)
-From: buildd@something.local (Debian/i386 Build Daemon)
+From: buildd on antheil bui...@antheil.debian.org
To: undisclosed-recipients:;
Delivered-To: logs@buildd.somewhere
@@ -428,7 +428,7 @@ Received: from world by raff.debian.org with esmtp (Exim 4.63) (envelope-f
Subject: Log for attempted build of gobby_0.4.9-2 on i386 (debian/unstable)
Message-ID:
Bug#622117: [PATCH 2/2] Fill field builder in pkg_history
On Sun, Jun 19, 2011 at 02:48:28PM +0200, Joachim Breitner wrote: Hi, I first suspected that maybe the format of the first line of the log file has changes (which is parsed in _parse_arch), which seems to be the case when I compare recent build logs with what's in the test suite. The first line looks like: sbuild (Debian sbuild) 0.61.0 (23 Feb 2011) on barber.debian.org But then, the build is only extracted from the Sender field, in _parse_sender. And even if the Sender is not in the expected format, it should log invalid sender F:'%s' S:'%s' and set the builder to the full content of the Sender line. The config has something like this in it: $mailfrom = 'buildd on lebrun bui...@lebrun.debian.org'; The email address should be right, or things won't work (when you need to manually sign). So the question is which part of the of that field you want to parse? And should we than standardise our config files? Kurt -- To UNSUBSCRIBE, email to debian-bugs-dist-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org
Bug#622117: [PATCH 2/2] Fill field builder in pkg_history
Hi,
Am Sonntag, den 19.06.2011, 15:09 +0200 schrieb Kurt Roeckx:
On Sun, Jun 19, 2011 at 02:48:28PM +0200, Joachim Breitner wrote:
Hi,
I first suspected that maybe the format of the first line of the log
file has changes (which is parsed in _parse_arch), which seems to be the
case when I compare recent build logs with what's in the test suite.
The first line looks like:
sbuild (Debian sbuild) 0.61.0 (23 Feb 2011) on barber.debian.org
after more code consulting: The first line is only considered if the
Subject line does not look like expected. So this is irrelevant for our
problem.
But then, the build is only extracted from the Sender field, in
_parse_sender. And even if the Sender is not in the expected format, it
should log invalid sender F:'%s' S:'%s' and set the builder to the
full content of the Sender line.
The config has something like this in it:
$mailfrom = 'buildd on lebrun bui...@lebrun.debian.org';
The email address should be right, or things won't work (when you
need to manually sign). So the question is which part of the of
that field you want to parse? And should we than standardise our
config files?
The parsing of the field was already in place:
def _parse_sender(self):
Generate a builder name from the From field of the header. It
assumes that the machine name is sufficiently unique. This
avoids leaking the complete email address.
pre:
self._sender is not None
post:
self.builder is not None
# Let's assume that there are two possibilities for email
# address encodings: bui...@host.org and buildd on host
# bui...@host.org. For other insane things this regular
# expression should also return something sane.
m = re.match('(.*)?(\S*)@(\S+\.[^\s]*)(.*)?', self._sender)
if m:
# group 3 is the hostname part of the mail address
self.builder = m.group(3)
else:
# It did not match: just use the sender verbatim and log
# the strange From line.
self.factory.logger.info(invalid sender F:'%s' S:'%s',
self._sender, self._subject)
self.builder = self._sender
so in any case, self.build should be set. Yet you say that the field is
always null... So it looks like it is somehow not put into the
Database...
ok, found it: There is a file lib/debian/wannabuild/db.py which I have
overlooked and where columns have to be registered. So the value that we
have passed to values() was silently ignored... please try again with
the attached patch.
Greetings,
Joachim
--
Joachim nomeata Breitner
Debian Developer
nome...@debian.org | ICQ# 74513189 | GPG-Keyid: 4743206C
JID: nome...@joachim-breitner.de | http://people.debian.org/~nomeata
From 8a37e6fc7ec4889b456efa7e7811c243d1758c41 Mon Sep 17 00:00:00 2001
From: Joachim Breitner m...@joachim-breitner.de
Date: Sun, 19 Jun 2011 16:09:05 +0200
Subject: [PATCH] Make builder column known to DBFactory
---
lib/debian/wannabuild/db.py |1 +
1 files changed, 1 insertions(+), 0 deletions(-)
diff --git a/lib/debian/wannabuild/db.py b/lib/debian/wannabuild/db.py
index b3c53c9..769b5ed 100644
--- a/lib/debian/wannabuild/db.py
+++ b/lib/debian/wannabuild/db.py
@@ -51,6 +51,7 @@ class DBFactory(object):
Column('version', String, primary_key=True),
Column('timestamp', DateTime, primary_key=True),
Column('result', String),
+Column('builder', String),
Column('build_time', Integer),
Column('disk_space', Integer),
schema=arch)
--
1.7.5.4
signature.asc
Description: This is a digitally signed message part
Bug#622117: [PATCH 2/2] Fill field builder in pkg_history
Hi,
Am Sonntag, den 19.06.2011, 17:34 +0200 schrieb Philipp Kern:
ok, found it: There is a file lib/debian/wannabuild/db.py which I have
overlooked and where columns have to be registered. So the value that we
have passed to values() was silently ignored... please try again with
the attached patch.
That patch breaks things. I don't know exactly what the error is,
I just see this in the exim log:
Child process of address_pipe transport returned 75 (could mean
temporary error) from command: /usr/bin/nice
That's why you get detailled logging
in /org/buildd.debian.org/log/inject
ProgrammingError: (ProgrammingError) column builder of relation
pkg_history does not exist
LINE 1: ...ackage, distribution, version, timestamp, result, builder,
b...
'INSERT INTO powerpc.pkg_history (package, distribution, version,
timestamp, result, builder, build_time, disk_space) VALUES
(%(package)s, %(distribution)s, %(version)s, %(timestamp)s,
%(result)s, %(builder)s, %(build_time)s, %(disk_space)s)' {'package':
'libburn', 'timestamp': datetime.datetime(2011, 6, 19, 15, 1, 27),
'builder': 'poulenc.debian.org', 'build_time': None, 'version':
'1.1.0-1', 'result': 'successful', 'distribution': 'sid',
'disk_space': None}
But changing the schema will be lots of fun due to the rules you'd
need to modify now.
I was under the impression that Kurt already changed the schema
(20110611165217.ga21...@roeckx.be), otherwise I would not have
suggested to apply the patch. Sorry for the trouble it was causing.
Greetings,
Joachim
--
Joachim nomeata Breitner
Debian Developer
nome...@debian.org | ICQ# 74513189 | GPG-Keyid: 4743206C
JID: nome...@joachim-breitner.de | http://people.debian.org/~nomeata
signature.asc
Description: This is a digitally signed message part
Bug#622117: [PATCH 2/2] Fill field builder in pkg_history
On Sun, Jun 19, 2011 at 05:09:00PM +0200, Kurt Roeckx wrote:
On Sun, Jun 19, 2011 at 04:11:03PM +0200, Joachim Breitner wrote:
Hi,
ok, found it: There is a file lib/debian/wannabuild/db.py which I have
overlooked and where columns have to be registered. So the value that we
have passed to values() was silently ignored... please try again with
the attached patch.
That patch breaks things. I don't know exactly what the error is,
I just see this in the exim log:
Child process of address_pipe transport returned 75 (could mean temporary
error) from command: /usr/bin/nice
That's why you get detailled logging in /org/buildd.debian.org/log/inject
ProgrammingError: (ProgrammingError) column builder of relation pkg_history
does not exist
LINE 1: ...ackage, distribution, version, timestamp, result, builder, b...
'INSERT INTO powerpc.pkg_history (package, distribution, version, timestamp,
result, builder, build_time, disk_space) VALUES (%(package)s, %(distribution)s,
%(version)s, %(timestamp)s, %(result)s, %(builder)s, %(build_time)s,
%(disk_space)s)' {'package': 'libburn', 'timestamp': datetime.datetime(2011, 6,
19, 15, 1, 27), 'builder': 'poulenc.debian.org', 'build_time': None, 'version':
'1.1.0-1', 'result': 'successful', 'distribution': 'sid', 'disk_space': None}
But changing the schema will be lots of fun due to the rules you'd need to
modify now.
Kind regards,
Philipp Kern
--
.''`. Philipp KernDebian Developer
: :' : http://philkern.de Stable Release Manager
`. `' xmpp:p...@0x539.de Wanna-Build Admin
`-finger pkern/k...@db.debian.org
signature.asc
Description: Digital signature
Bug#622117: [PATCH 2/2] Fill field builder in pkg_history
On Sun, Jun 19, 2011 at 06:09:07PM +0200, Joachim Breitner wrote: Hi, I was under the impression that Kurt already changed the schema (20110611165217.ga21...@roeckx.be), otherwise I would not have suggested to apply the patch. Sorry for the trouble it was causing. But in the mean time the schema was updated to not have an schema per arch, but 1 big table for all arches. And there were view and rules created for the old tables. But the new views (and rules) don't know anything about builder. Kurt -- To UNSUBSCRIBE, email to debian-bugs-dist-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org
Bug#622117: [PATCH 2/2] Fill field builder in pkg_history
On Sun, Jun 19, 2011 at 04:11:03PM +0200, Joachim Breitner wrote: Hi, ok, found it: There is a file lib/debian/wannabuild/db.py which I have overlooked and where columns have to be registered. So the value that we have passed to values() was silently ignored... please try again with the attached patch. That patch breaks things. I don't know exactly what the error is, I just see this in the exim log: Child process of address_pipe transport returned 75 (could mean temporary error) from command: /usr/bin/nice Kurt -- To UNSUBSCRIBE, email to debian-bugs-dist-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org
Bug#622117: [PATCH 2/2] Fill field builder in pkg_history
On Wed, Jun 08, 2011 at 10:10:09AM +0200, nome...@debian.org wrote: From: Joachim Breitner m...@joachim-breitner.de This needs according adjustments to the SQL layout. This change is also not covered by unit tests (which seem not to cover changes to pkg_history at all) Your patches got applied, and the schema is updated, but builder is always set to NULL. The public schema wasn't updated yet, the views need to be recreated. But I'm waiting with that to first get it working. Kurt -- To UNSUBSCRIBE, email to debian-bugs-dist-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org
Bug#622117: [PATCH 2/2] Fill field builder in pkg_history
Hi Kurt, Am Samstag, den 11.06.2011, 18:52 +0200 schrieb Kurt Roeckx: On Wed, Jun 08, 2011 at 10:10:09AM +0200, nome...@debian.org wrote: From: Joachim Breitner m...@joachim-breitner.de This needs according adjustments to the SQL layout. This change is also not covered by unit tests (which seem not to cover changes to pkg_history at all) Your patches got applied, and the schema is updated, but builder is always set to NULL. The public schema wasn't updated yet, the views need to be recreated. But I'm waiting with that to first get it working. no idea what’s wrong. Maybe the test cases in lib/debian/wannabuild/logs_tests.py don’t quite match the real world. Since for the test suite, by print-debugging, I know that when the line sql = tbl.insert().values(package=self.package, distribution=self.suite, version=self.version, timestamp=datetime.utcfromtimestamp(self.timestamp), result=self.state, builder=self.builder, build_time=self.build_time, disk_space=self.disk_space) is executed, self.builder has a sensible value. Greetings, Joachim -- Joachim nomeata Breitner Debian Developer nome...@debian.org | ICQ# 74513189 | GPG-Keyid: 4743206C JID: nome...@joachim-breitner.de | http://people.debian.org/~nomeata signature.asc Description: This is a digitally signed message part
Bug#622117: [PATCH 2/2] Fill field builder in pkg_history
From: Joachim Breitner m...@joachim-breitner.de This needs according adjustments to the SQL layout. This change is also not covered by unit tests (which seem not to cover changes to pkg_history at all) --- lib/debian/wannabuild/logs.py |2 ++ 1 files changed, 2 insertions(+), 0 deletions(-) diff --git a/lib/debian/wannabuild/logs.py b/lib/debian/wannabuild/logs.py index b70a045..aeca4d7 100644 --- a/lib/debian/wannabuild/logs.py +++ b/lib/debian/wannabuild/logs.py @@ -669,6 +669,7 @@ class MailedBuildLog(BuildLog): self.version is not None self.timestamp is not None self.state is not None +self.builder is not None 'build_time' in self.__dict__ 'disk_space' in self.__dict__ @@ -683,6 +684,7 @@ class MailedBuildLog(BuildLog): version=self.version, timestamp=datetime.utcfromtimestamp(self.timestamp), result=self.state, + builder=self.builder, build_time=self.build_time, disk_space=self.disk_space) self.factory.dbfactory.conn.execute(sql) -- 1.7.5.3 -- To UNSUBSCRIBE, email to debian-bugs-dist-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org
