Re: [Spacewalk-devel] [PATCH] Filters on reposync
Hi Michael, Thanks a lot for your answer. Regards. Baptiste - Mail original - De: Michael Mraka michael.mr...@redhat.com À: spacewalk-devel@redhat.com Envoyé: Mardi 18 Octobre 2011 13:51:16 Objet: Re: [Spacewalk-devel] [PATCH] Filters on reposync Baptiste AGASSE wrote: % Hi, % % I've just looked at reposync.py code a I don't see any part related to % getting include/exclude filters from database as properties of % repositories that Jan asked me when i've submited some code for % filters (he said that he didn't like that filters are specified at % runtime). % % Any news on it ? If, you decide to store filters in database, these % will be accessible through repositories webui ? Hi Baptiste, yesterday and today I finally saved some time to look into it again. I added a database table to hold include/exclude filters and spacewalk-repo-sync code which is able to use it. Unfortunately there's not webUI part yet so if you want to check it you have to insert filter data directy into database. E.g. insert into rhncontentsourcefilter (id, source_id, sort_order, flag, filter) values (sequence_nextval('rhn_csf_id_seq') , 536, 1, '+', 'samba-client'); where source_id is repository id (that's the end part as in https://spacewalk/rhn/channels/manage/repos/RepoEdit.do?id=536 url), sort_order specifies the order of filters which belong to the same repo, flag is either '+' or '-' (include / exclude) and filter is the list of packages to be filtered. Regards. -- Michael Mráka Satellite Engineering, Red Hat ___ Spacewalk-devel mailing list Spacewalk-devel@redhat.com https://www.redhat.com/mailman/listinfo/spacewalk-devel ___ Spacewalk-devel mailing list Spacewalk-devel@redhat.com https://www.redhat.com/mailman/listinfo/spacewalk-devel
Re: [Spacewalk-devel] [PATCH] Filters on reposync
Hi Michael, my original idea was to be able to stack includes / excludes on top of each other, e.g. include=openoffice.org-* exclude=openoffice.org-draw,openoffice.org-langpack* include=openoffice.org-langpack-en,openoffice.org-langpack-de which in the end will sync openoffice.org packages except *-draw and *-langpacks but will also include *langpack-en and *langpack-de. Ok, i know that my solution don't care about include/exclude order and can't do this, but i can take a look on it The filtering and dependecy resolver parts look good. I'm not keen on inheriting ContentSource from yum.YumBase because it leads to implicit import of all yum repos defined in the server's /etc/yum.repo.d/ which have nothing to do with repos defined in Spacewalk (the application). Morever you have to delete them so they not mess up with the valid spacewalk repos. Yes, i haven't found many resources on it, i did the best i could to use yum dependency resolver % - All versions of the packages excluded by a filter is now deleted % from DB and filesystem Well, this changes current behavior of spacewalk-repo-sync because we never remove packages which are already in the channel during sync process. E.g. you might have packages pushed manually (via rhnpush) into channel, should they also be removed? I don't think so. I decided skip this part of patch for now. Yes, in the patch that I've submitted, include/exclude is in repository scope: only excluded packages from current repository are deleted Regards. ___ Spacewalk-devel mailing list Spacewalk-devel@redhat.com https://www.redhat.com/mailman/listinfo/spacewalk-devel ___ Spacewalk-devel mailing list Spacewalk-devel@redhat.com https://www.redhat.com/mailman/listinfo/spacewalk-devel
Re: [Spacewalk-devel] [PATCH] Filters on reposync
Hi Jan, Following your advices, i modified my code: Now, it get include and exclude filters from database: I ran the following SQL query on my Oracle XE Database: ALTER TABLE rhnContentSource ADD (include_filter VARCHAR(255), exclude_filter VARCHAR(255)); Regards. Baptiste - Mail original - De: Baptiste AGASSE baptiste.aga...@lyra-network.com À: spacewalk-devel@redhat.com Envoyé: Jeudi 18 Août 2011 22:50:34 Objet: Re: [Spacewalk-devel] [PATCH] Filters on reposync Hi Jan, Ok, i can take a look on this next week. Regards. Baptiste - Mail original - De: Jan Pazdziora jpazdzi...@redhat.com À: spacewalk-devel@redhat.com Envoyé: Jeudi 18 Août 2011 13:39:57 Objet: Re: [Spacewalk-devel] [PATCH] Filters on reposync On Wed, Aug 17, 2011 at 08:23:36PM +0200, Baptiste AGASSE wrote: Following your advices I have modified my code: - You can now include and / or exclude packages (with --include and / or --exclude options) - Include filter takes priority over exclude filter: if one package meet 'include' and 'exclude' rules, it will be included eg: exclude = [ 'openoffice.org-langpack-*', ...] include = [ 'openoffice.org-langpack-en-*', ...] - Package filtering is in yum_src.py - Yum dependencies resolver is now used to find selected packages dependencies - All versions of the packages excluded by a filter is now deleted from DB and filesystem - Print elapsed time at end of sync Any comments are welcome. I don't really like the fact that the exclude/include options are specified on the spacewalk-repo-sync runtime, rather than being properties of the repository. In other words -- these exclude/include lists should be specified in the database, so that they would be used any time spacewalk-repo-sync is run, no matter if it is run from the command line or via scheduled event by taskomatic. Now that the core functionality is in place, could you amend it some more and have the lists stored in some database table and used from there? -- Jan Pazdziora Principal Software Engineer, Satellite Engineering, Red Hat ___ Spacewalk-devel mailing list Spacewalk-devel@redhat.com https://www.redhat.com/mailman/listinfo/spacewalk-devel diff --git a/backend/satellite_tools/repo_plugins/yum_src.py b/backend/satellite_tools/repo_plugins/yum_src.py index bfc6161..9871a32 100644 --- a/backend/satellite_tools/repo_plugins/yum_src.py +++ b/backend/satellite_tools/repo_plugins/yum_src.py @@ -74,14 +74,15 @@ class YumUpdateMetadata(UpdateMetadata): no = self._no_cache.setdefault(file['name'], set()) no.add(un) -class ContentSource: +class ContentSource(yum.YumBase): url = None name = None -repo = None cache_dir = '/var/cache/rhn/reposync/' -def __init__(self, url, name): -self.url = url -self.name = name +repo_id = None +filters = {'include': [], 'exclude': []} + +def __init__(self, url, name, filters = None ): +yum.YumBase.__init__(self) self._clean_cache(self.cache_dir + name) # read the proxy configuration in /etc/rhn/rhn.conf @@ -97,14 +98,14 @@ class ContentSource: else: self.proxy_url = None -def list_packages(self): - list packages -repo = yum.yumRepo.YumRepository(self.name) -self.repo = repo +if filters: +self.filters = filters + +repo = yum.yumRepo.YumRepository(name) repo.cache = 0 repo.metadata_expire = 0 -repo.mirrorlist = self.url -repo.baseurl = [self.url] +repo.mirrorlist = url +repo.baseurl = [url] repo.basecachedir = self.cache_dir if self.proxy_url is not None: repo.proxy = self.proxy_url @@ -113,13 +114,36 @@ class ContentSource: warnings.disable() repo.baseurlSetup() warnings.restore() - repo.setup(False) -sack = repo.getPackageSack() -sack.populate(repo, 'metadata', None, 0) -list = sack.returnPackages() -to_return = [] -for pack in list: +repos = self.repos.findRepos('*') +for rep in repos: +self.repos.disableRepo(rep.id) +self.repos.delete(rep.id) + +self.repo_id = repo.id +self.repos.add(repo) +self.pkgSack = self.repos.getRepo(self.repo_id).getPackageSack() +self.pkgSack.populate(self.repos.getRepo(self.repo_id), 'metadata', None, 0) + +def getName(self): +return self.repos.getRepo(self.repo_id).id + +def getUrl(self): +return self.repos.getRepo(self.repo_id).mirrorlist + +def getFilters(self): +return self.filters + +def getRepo(self): +return self.repos.getRepo(self.repo_id) + +def list_packages(self): + list packages +return self._list_packages(self.pkgSack.returnPackages()) + +def
Re: [Spacewalk-devel] [PATCH] Filters on reposync
Hi Michael, Following your advices I have modified my code: - You can now include and / or exclude packages (with --include and / or --exclude options) - Include filter takes priority over exclude filter: if one package meet 'include' and 'exclude' rules, it will be included eg: exclude = [ 'openoffice.org-langpack-*', ...] include = [ 'openoffice.org-langpack-en-*', ...] - Package filtering is in yum_src.py - Yum dependencies resolver is now used to find selected packages dependencies - All versions of the packages excluded by a filter is now deleted from DB and filesystem - Print elapsed time at end of sync Any comments are welcome. Regards. Baptiste - Mail original - De: Michael Mraka michael.mr...@redhat.com À: spacewalk-devel@redhat.com Envoyé: Lundi 8 Août 2011 11:45:41 Objet: Re: [Spacewalk-devel] [PATCH] Filters on reposync Baptiste AGASSE wrote: % Hi all, % % I've modified /backend/satellite_tools/reposync.py to add filters % support with dependencies solving (like the --rpm-list option of % cobbler) to spacewalk reposync command. Hi Baptiste, this is good feature, thanks for sharing it. % It add the --filters option to reposync eg: --filters osad rhncfg-* % foo* bar So the --filters 'xyz*' means to include xyz* packages only; It would be great to have also the opposite option to exclude some packages which can even stack on each other. E.g. spacewalk-repo-sync --exclude=openoffice.org-langpack-* \ --include=openoffice.org-langpack-en-* ... % It allow to download only selected packages from a repository in order % to save disk space, mainly if you use only few packages from it, and % deletes packages already present on your system (in spacewalk server % and on the filesystem) that don't meet any filters (only packages with % the same NVREA) % % TODO: - Search in spacewalk server older versions of packages % downloaded previously that don't match filters and remove them (I will % work on it). - Make it available from web UI in repositories % management part. % % Maybe someone can work on the web UI and the database schema to make % this option available directly from web UI ? (i'm not familiar with % java and oracle DB) % % Any comments are welcome, it's the first time that i'm programming in % python :). % diff --git a/backend/satellite_tools/reposync.py b/backend/satellite_tools/reposync.py % index 6834adb..0428bb9 100644 % --- a/backend/satellite_tools/reposync.py % +++ b/backend/satellite_tools/reposync.py ... % @@ -132,6 +137,7 @@ class RepoSync: % self.parser.add_option('-t', '--type', action='store', dest='type', help='The type of repo, currently only yum is supported', default='yum') % self.parser.add_option('-f', '--fail', action='store_true', dest='fail', default=False , help=If a package import fails, fail the entire operation) % self.parser.add_option('-q', '--quiet', action='store_true', dest='quiet', default=False, help=Print no output, still logs output) % +self.parser.add_option('-p', '--filters', action='store_true', dest='filters', help=Synchronize only the packets that meet the filter and their dependencies) action='store_true' means it's True/False option but you likely want return a string (list of patterns) % return self.parser.parse_args() % % def load_plugin(self): % @@ -308,9 +314,43 @@ class RepoSync: % self.regen = True % ... % +def filter_packages(self, filters, packages): % +# Returns 3 lists : selected packages, dependencies, and others % +selected = [] % +dependencies = [] % +others = [] % +for pack in packages: % +# Select all packages that match one filter % +match = False % +for filter_str in filters: % +reg = re.compile(^ + filter_str.replace(*,.*) + $) % +if reg.match(pack.name): Wouldn't be fnmatch.filter() better/easier to use here? % +match = True % +break % +if match: ... % + % +def package_deps(self, package, packages_list): ... This is yum repo plugin specific code and it would be better to implement it in the plugin itself, i.e. repo_plugins/yum_src.py in this case. Moreover it would be better to call yum's internal depsolver and not reinvent it again. Regards, -- Michael Mráka Satellite Engineering, Red Hat ___ Spacewalk-devel mailing list Spacewalk-devel@redhat.com https://www.redhat.com/mailman/listinfo/spacewalk-develdiff --git a/backend/satellite_tools/repo_plugins/yum_src.py b/backend/satellite_tools/repo_plugins/yum_src.py index bfc6161..db6ee3b 100644 --- a/backend/satellite_tools/repo_plugins/yum_src.py +++ b/backend/satellite_tools/repo_plugins/yum_src.py @@ -74,14 +74,14 @@ class YumUpdateMetadata(UpdateMetadata): no = self._no_cache.setdefault(file['name'], set
[Spacewalk-devel] [PATCH] Filters on reposync
Hi all, I've modified /backend/satellite_tools/reposync.py to add filters support with dependencies solving (like the --rpm-list option of cobbler) to spacewalk reposync command. It add the --filters option to reposync eg: --filters osad rhncfg-* foo* bar It allow to download only selected packages from a repository in order to save disk space, mainly if you use only few packages from it, and deletes packages already present on your system (in spacewalk server and on the filesystem) that don't meet any filters (only packages with the same NVREA) TODO: - Search in spacewalk server older versions of packages downloaded previously that don't match filters and remove them (I will work on it). - Make it available from web UI in repositories management part. Maybe someone can work on the web UI and the database schema to make this option available directly from web UI ? (i'm not familiar with java and oracle DB) Any comments are welcome, it's the first time that i'm programming in python :). Regards. Baptiste. diff --git a/backend/satellite_tools/reposync.py b/backend/satellite_tools/reposync.py index 6834adb..0428bb9 100644 --- a/backend/satellite_tools/reposync.py +++ b/backend/satellite_tools/reposync.py @@ -15,6 +15,7 @@ # import sys, os, time import hashlib +import re from datetime import datetime import traceback from optparse import OptionParser @@ -44,6 +45,7 @@ class RepoSync: fail = False quiet = False regen = False +filters = [] def main(self): initCFG('server') @@ -89,6 +91,9 @@ class RepoSync: quit = True self.error_msg(--channel must be specified) +if options.filters: +self.filters = options.filters.split() + self.log_msg(\nSync started: %s % (time.asctime(time.localtime( self.log_msg(str(sys.argv)) @@ -132,6 +137,7 @@ class RepoSync: self.parser.add_option('-t', '--type', action='store', dest='type', help='The type of repo, currently only yum is supported', default='yum') self.parser.add_option('-f', '--fail', action='store_true', dest='fail', default=False , help=If a package import fails, fail the entire operation) self.parser.add_option('-q', '--quiet', action='store_true', dest='quiet', default=False, help=Print no output, still logs output) +self.parser.add_option('-p', '--filters', action='store_true', dest='filters', help=Synchronize only the packets that meet the filter and their dependencies) return self.parser.parse_args() def load_plugin(self): @@ -308,9 +314,43 @@ class RepoSync: self.regen = True def import_packages(self, plug, url): -packages = plug.list_packages() +repo_packages = plug.list_packages() to_process = [] -self.print_msg(Repo + url + has + str(len(packages)) + packages.) +to_delete = [] +self.print_msg(Repo + url + has + str(len(repo_packages)) + packages.) + +if len(self.filters) 0: +selected, dependencies, others = self.filter_packages(self.filters, repo_packages) +packages = selected +packages.extend(dependencies) +self.print_msg(Repo + url + has + str(len(selected)) + packages selected by filters.) +self.print_msg(Repo + url + has + str(len(dependencies)) + packages dependencies.) +self.print_msg(Repo + url + has + str(len(others)) + packages excluded.) + +for pack in others: +# TODO: search for older versions of package and drop them too +db_pack = rhnPackage.get_info_for_package( + [pack.name, pack.version, pack.release, pack.epoch, pack.arch], + self.channel_label) + +to_remove = False +to_unlink = False +if db_pack['path']: +pack.path = os.path.join(CFG.MOUNT_POINT, db_pack['path']) +if self.match_package_checksum(pack.path, +pack.checksum_type, pack.checksum): +# package is already on disk +to_remove = True +if db_pack['channel_label'] == self.channel_label or db_pack['channel_label'] == self.channel_label: +# package is already in the channel +to_unlink = True + +if to_remove or to_unlink: +to_delete.append((pack, to_remove, to_unlink)) +else: +self.print_msg(Repo + url + has no filters set.) +packages = repo_packages + for pack in packages: db_pack = rhnPackage.get_info_for_package( [pack.name, pack.version, pack.release, pack.epoch, pack.arch], @@ -327,15 +367,16 @@ class RepoSync: if db_pack['channel_label'] == self.channel_label:
[Spacewalk-devel] spacewalk-proxy and osad
Hi all, I've already posted that on spacewalk-list but, maybe my problem is just a bug in the default jabber server configuration on spacewalk proxy. We have some problems with osad and spacewalk proxy. All boxes behind the proxy are seen as offline in osad status on spacewalk server, and no action assigned to these machines are picked up. Also, on spacewalk web ui, in details = connection section of host behind proxy, i can see spacewalk proxy host listed. We can see boxes connected on spacewalk proxy (on port 5222), the proxy is connected on spacewalk server (on port 5222), but we don't see any trafic on 5269 port from spacewalk-proxy to spacewalk server. we have iptables firewal between the spacewalk server and proxy and we opened 80,443,5222,5269 ports from proxy to spacewalk server. spacewalk server: CentOS 5.6 x86_64 spacewalk 1.4 SElinux : Permissive spacewalk proxy: CentOS 5.6 x86_64 spacewalk proxy installed from spacewalk 1.4 repo SELinux: Enforcing, with spacewalk-proxy-selinux installed Regards. ___ Spacewalk-devel mailing list Spacewalk-devel@redhat.com https://www.redhat.com/mailman/listinfo/spacewalk-devel