On Mon, Dec 09, 2002 at 10:39:43PM +0000, Roger Leigh wrote: > There is some documentation for using it, but is there any for the > initial setup? > > I certainly have read the docs from CVS, but I'm stuck at the > installation stage. I can see what installs where from the debian/* > packaging, but what I need to do to setup the archive and database > isn't clear.
I have got it going! It is pretty straight forward once you get to know what all these names do. Even now though I still forget the correct name for the correct purpose. What I did: - Edit katie.conf to suit. Make sure the path to the templates directory is correct. - Edit vars. - Edit apt.conf so it has the same information as kate.conf. If there are areas in apt.conf but not in katie.conf, you will get errors when generating index files. One day in the future I would like to see vars and apt.conf autogenereted from katie.conf; I am not sure if this is possible or not. - Change paths to config files in utils.py. - Configure postgresql to allow access and ability to create database. - psql -f init_pool.sql template1 - alyson imports katie.conf into database. - rose creates initial directories (actually some where missing; I can't remember which onces were missing and which ones were misconfigured though now). - cron.unchecked checks files in queue/unchecked and moves them to queue/new or queue/accepted. - To manually accept queue/new or queue/byhand files, cd to that directory, and type "lisa *.changes". - Anything else I forgot to include. - melanie marks packages for deletion. This doesn't happen straight away, by default there is a delay (in katie.conf) before files are actually removed. (melanie complains that it cannot delete the files from buildd/*, but I can't see why, I deleted the same files mentioned manually without any problems). - look at database with pgaccess often to try and find out what is happening. These files need customization: - cron.daily installs accepted files and rebuilds the database. (do not use udpate-* for private archives). Check apt-ftparchive doesn't produce any errors, this might be because you have areas defined in apt.conf but not in katie.conf. - cron.buildd is for buildd, I don't know where to get wanna-build from so I don't use it. - cron.hourly doesn't appear to be required. - cron.weekly might be important, but I don't understand what it does. - cron.monthly doesn't appear to be required. - mkmaintainers adds NON-US maintainers and base maintainers. Potentially dangerous features: - turn off BXA notifications in katie.conf. Or change the template. - turn off auto closing bugs in katie.conf. Or at least change the template. - when new package is accepted it will inform uploader and package maintainer; the maintainer might confuse this E-Mail as an unauthorised upload to the Debian archive. I am not sure what the best solution is here. - some templates in templates/* have E-Mail addresses hardcoded (From and To E-Mail addresses). - cron.* have E-Mail addresses hardcoded. Features I really dislike: - cron.daily has: kelly -pa *.changes | tee REPORT | \ mail -s "Install for $(date +%D)" [EMAIL PROTECTED] This comes up with lots of errors if *.changes doesn't exist. - No way to override component eg. (main,contrib,non-free) without changing source; I have a patch that will let you override it in lisa. I have also patched/hacked jennifer to use existing overide component if one already exists. This patch has the side effect that you can only have one version of a package in all components combined; I don't see this as being a problem. See patch attached. It seems to work fine, but I haven't tested it since replacing several tabs with spaces... It might be better if the changes to jennifer were made optional by a config switch. lisa already has the ability to specify the component in the section, but this didn't seem to be used. - if you don't have testing, some of the scripts will fail. Testing is hardcoded and required. - *NO* E-Mail addresses should get hardcoded in non-config files anywhere! - cron.daily is pretty verbose. - cron.daily generates override.potato.all3 and override.sid.all3, but I can't see what use these files are. What I don't like but can't really be fixed: - GPG key for signing Release files can't be password protected, it needs to work in a Cron job. What I do like: - good checking of packages, eg. to ensure that full source code could be found. Then again a large number of my packages failed to install because I forgot to use -sa when compiling :-(. Still to learn: - How to move packages to stable. - How to add/remove distributions/components. It appears that you must modify three things, 1. katie.conf, 2. apt.conf, 3. database. I really don't like this, it is prone to error. alyson wont update a database that isn't blank. - How to best layout a private archive. At the moment I am thinking of doing something similar to Debian, and initially uploading files to unstable before I test them, and then move them to stable after I am sure it works. This way other people can try out my packages even though known bugs might exist. I probably will not use testing, ever. I can't really see a good reason for it, and there is no way a script could even tell automatically if the version is buggy or not (no - I do not want to setup my own BTS ;-) ). - My idea all along was to create three components, one main, one ACL (for files required to get access control lists going) and one for SE-Linux (should be obvious). This means that if you don't want SE-Linux files, you don't have to get them. However, debian-installer will only check that main component for udeb files. I am not sure if I should just abandon the other two components and put everything in main anyway, or what. - Not sure how to move files between components, this might require a delete operation followed by a new upload. - Where is the check to ensure files aren't uploaded to stable? - nothing except mkmaintainers seems to use the ftpdir/indices directory??? If anybody wants any patches and/or config files, please let me know. Hope this helps somebody. -- Brian May <[EMAIL PROTECTED]>
Index: katie.py =================================================================== RCS file: /cvs/dak/dak/katie.py,v retrieving revision 1.27 diff -u -r1.27 katie.py --- katie.py 16 Oct 2002 02:47:32 -0000 1.27 +++ katie.py 10 Dec 2002 23:39:34 -0000 @@ -668,6 +668,42 @@ ################################################################################ + def get_override_component (self, package, suite, binary_type, file): + files = self.pkg.files; + + if binary_type == "": # must be source + type = "dsc"; + else: + type = binary_type; + + # Override suite name; used for example with proposed-updates + if self.Cnf.Find("Suite::%s::OverrideSuite" % (suite)) != "": + suite = self.Cnf["Suite::%s::OverrideSuite" % (suite)]; + + # Avoid <undef> on unknown distributions + suite_id = db_access.get_suite_id(suite); + if suite_id == -1: + return None; + type_id = db_access.get_override_type_id(type); + + q = self.projectB.query("SELECT c.name FROM override o, component c WHERE package = '%s' AND suite = %s AND type = %s AND o.component = c.id" + % (package, suite_id, type_id)); + result = q.getresult(); + # If checking for a source package fall back on the binary override type + if type == "dsc" and not result: + deb_type_id = db_access.get_override_type_id("deb"); + udeb_type_id = db_access.get_override_type_id("udeb"); + q = self.projectB.query("SELECT c.name FROM override o, component c WHERE package = '%s' AND suite = %s AND (type = %s OR type = %s) AND o.component = c.id" + % (package, suite_id, deb_type_id, udeb_type_id)); + result = q.getresult(); + + if result: + return result[0][0]; + else: + return None; + + ################################################################################ + def reject (self, str, prefix="Rejected: "): if str: # Unlike other rejects we add new lines first to avoid trailing Index: lisa =================================================================== RCS file: /cvs/dak/dak/lisa,v retrieving revision 1.20 diff -u -r1.20 lisa --- lisa 26 Nov 2002 16:15:56 -0000 1.20 +++ lisa 10 Dec 2002 23:39:43 -0000 @@ -51,6 +51,7 @@ Logger = None; Priorities = None; +Components = None; Sections = None; reject_message = ""; @@ -271,6 +272,27 @@ ################################################################################ +class Component_Completer: + def __init__ (self): + self.components = []; + q = projectB.query("SELECT name FROM component"); + for i in q.getresult(): + self.components.append(i[0]); + + def complete(self, text, state): + if state == 0: + self.matches = []; + n = len(text); + for word in self.components: + if word[:n] == text: + self.matches.append(word); + try: + return self.matches[state] + except IndexError: + return None + +############################################################ + class Section_Completer: def __init__ (self): self.sections = []; @@ -315,9 +337,11 @@ def check_valid (new): for pkg in new.keys(): + component = new[pkg]["component"]; section = new[pkg]["section"]; priority = new[pkg]["priority"]; type = new[pkg]["type"]; + new[pkg]["component id"] = db_access.get_component_id(component); new[pkg]["section id"] = db_access.get_section_id(section); new[pkg]["priority id"] = db_access.get_priority_id(new[pkg]["priority"]); # Sanity checks @@ -338,6 +362,10 @@ index += 1; section = new[pkg]["section"]; priority = new[pkg]["priority"]; + component = new[pkg]["component"]; + if new[pkg]["component id"] == -1: + component += "[!]"; + broken = 1; if new[pkg]["section id"] == -1: section += "[!]"; broken = 1; @@ -345,9 +373,9 @@ priority += "[!]"; broken = 1; if indexed: - line = "(%s): %-20s %-20s %-20s" % (index, pkg, priority, section); + line = "(%s): %-20s %-20s %-20s %-20s" % (index, pkg, priority, component, section); else: - line = "%-20s %-20s %-20s" % (pkg, priority, section); + line = "%-20s %-20s %-20s %-20s" % (pkg, priority, component, section); line = line.strip()+'\n'; file.write(line); note = Katie.pkg.changes.get("lisa note"); @@ -411,19 +439,23 @@ continue; s = line.split(); # Pad the list if necessary - s[len(s):3] = [None] * (3-len(s)); - (pkg, priority, section) = s[:3]; + s[len(s):4] = [None] * (4-len(s)); + (pkg, priority, component, section) = s[:4]; if not new.has_key(pkg): utils.warn("Ignoring unknown package '%s'" % (pkg)); else: # Strip off any invalid markers, print_new will readd them. + if component.endswith("[!]"): + component = component[:-3]; if section.endswith("[!]"): section = section[:-3]; if priority.endswith("[!]"): priority = priority[:-3]; for file in new[pkg]["files"]: + Katie.pkg.files[file]["component"] = component; Katie.pkg.files[file]["section"] = section; Katie.pkg.files[file]["priority"] = priority; + new[pkg]["component"] = component; new[pkg]["section"] = section; new[pkg]["priority"] = priority; @@ -431,6 +463,7 @@ def edit_index (new, index): priority = new[index]["priority"] + component = new[index]["component"] section = new[index]["section"] type = new[index]["type"]; done = 0 @@ -439,10 +472,10 @@ answer = "XXX"; if type != "dsc": - prompt = "[B]oth, Priority, Section, Done ? "; + prompt = "[B]oth, Priority, Component, Section, Done ? "; else: prompt = "[S]ection, Done ? "; - edit_priority = edit_section = 0; + edit_priority = edit_component = edit_section = 0; while prompt.find(answer) == -1: answer = utils.our_raw_input(prompt); @@ -453,6 +486,8 @@ if answer == 'P': edit_priority = 1; + elif answer == 'C': + edit_component = 1; elif answer == 'S': edit_section = 1; elif answer == 'B': @@ -472,6 +507,17 @@ got_priority = 1; priority = new_priority; + if edit_component: + readline.set_completer(Components.complete); + got_component = 0; + while not got_component: + new_component = utils.our_raw_input("New component: ").strip(); + if Components.components.count(new_component) == 0: + print "E: '%s' is not a valid component, try again." % (new_component); + else: + got_component = 1; + component = new_component; + # Edit the section if edit_section: readline.set_completer(Sections.complete); @@ -489,8 +535,10 @@ for file in new[index]["files"]: Katie.pkg.files[file]["section"] = section; + Katie.pkg.files[file]["component"] = component; Katie.pkg.files[file]["priority"] = priority; new[index]["priority"] = priority; + new[index]["component"] = component; new[index]["section"] = section; return new; @@ -507,7 +555,7 @@ index += 1; new_index[index] = i; - prompt = "(%s) edit override <n>, Editor, Done ? " % (index_range(index)); + prompt = "(%s) edit override <n>, Component change for all, Editor, Done ? " % (index_range(index)); got_answer = 0 while not got_answer: @@ -515,6 +563,21 @@ answer = answer[:1].upper(); if answer == "E" or answer == "D": got_answer = 1; + + elif answer == "C": + readline.set_completer(Components.complete); + got_component = 0; + while not got_component: + new_component = utils.our_raw_input("New component: ").strip(); + if Components.components.count(new_component) == 0: + print "E: '%s' is not a valid component, try again." % (new_component); + else: + got_component = 1; + for pkg in new.keys(): + new[pkg]["component"] = new_component; + for file in new[pkg]["files"]: + Katie.pkg.files[file]["component"] = new_component; + elif katie.re_isanum.match (answer): answer = int(answer); if (answer < 1) or (answer > index): @@ -622,9 +685,9 @@ for suite in changes["suite"].keys(): suite_id = db_access.get_suite_id(suite); for pkg in new.keys(): - component_id = db_access.get_component_id(new[pkg]["component"]); type_id = db_access.get_override_type_id(new[pkg]["type"]); priority_id = new[pkg]["priority id"]; + component_id = new[pkg]["component id"]; section_id = new[pkg]["section id"]; projectB.query("INSERT INTO override (suite, component, type, package, priority, section) VALUES (%s, %s, %s, '%s', %s, %s)" % (suite_id, component_id, type_id, pkg, priority_id, section_id)); for file in new[pkg]["files"]: @@ -730,7 +793,7 @@ ################################################################################ def init(): - global Cnf, Options, Logger, Katie, projectB, Sections, Priorities; + global Cnf, Options, Logger, Katie, projectB, Components, Sections, Priorities; Cnf = utils.get_conf(); @@ -761,6 +824,7 @@ projectB = Katie.projectB; + Components = Component_Completer(); Sections = Section_Completer(); Priorities = Priority_Completer(); readline.parse_and_bind("tab: complete"); Index: jennifer =================================================================== RCS file: /cvs/dak/dak/jennifer,v retrieving revision 1.28 diff -u -r1.28 jennifer --- jennifer 21 Oct 2002 13:54:53 -0000 1.28 +++ jennifer 10 Dec 2002 23:39:52 -0000 @@ -663,6 +663,12 @@ reject("unknown component `%s' for suite `%s'." % (files[file]["component"], suite)); continue + #hack to use existing override component, if one exists + override_component = Katie.get_override_component(files[file]["package"], suite, files[file].get("dbtype",""), file); + + if override_component: + files[file]["component"] = override_component; + # See if the package is NEW if not Katie.in_override_p(files[file]["package"], files[file]["component"], suite, files[file].get("dbtype",""), file): files[file]["new"] = 1; @@ -1137,7 +1143,7 @@ raise; except: print "ERROR"; - traceback.print_exc(file=sys.stderr); + traceback.print_exc(file=sys.stderr); pass; # Restore previous WD