Thank you for the suggestion. Sorry for the long delay in replying. We did try and re-publish the repo with the force full command but it made no difference.
We have found a workaround for our repository management and we now do not need to use pulp for it in this case. So I don’t have a need to pursue this request here, but thank you for the help. As far as we got before trying another idea is that in /var/lib/pulp/published/yum/master/yum_distributor/ <channel name>/ We saw this for a failing channel: rhel-6-server-rpms]# ls -l */repodata/* -rw-r--r--. 1 apache apache 1188540 Jul 30 06:11 1564465100.45/repodata/245ba6fe28e4a180b78cdd87f479af63cee47f4e-comps.xml -rw-r--r--. 1 apache apache 45078695 Jul 30 06:03 1564465100.45/repodata/7fac8b0fefc059c50f4cc311f158eef6a9b34fd5-filelists.xml.gz -rw-r--r--. 1 apache apache 3878183 Jul 30 06:10 1564465100.45/repodata/c45aa9b5d1d34e37f2e3797a3c73936d7c4db8be-updateinfo.xml.gz -rw-r--r--. 1 apache apache 57733120 Jul 30 06:03 1564465100.45/repodata/other.xml.gz -rw-r--r--. 1 apache apache 40497152 Jul 30 06:03 1564465100.45/repodata/primary.xml.gz -rw-r--r--. 1 apache apache 2159 Jul 30 03:25 1564465100.45/repodata/productid -rw-r--r--. 1 apache apache 1129 Jul 30 06:29 1564465100.45/repodata/repomd.xml And for a working one: rhel-6-server-extras-rpms]# ls -l */repodata/* -rw-r--r--. 1 apache apache 24475 Jul 29 13:05 1564405504.35/repodata/48c121111734b4da32d6404a178195b1aff90ace-primary.xml.gz -rw-r--r--. 1 apache apache 124 Jul 29 13:05 1564405504.35/repodata/7a747e6dc73c3ccef57c839995d90b6ef4c76092-comps.xml -rw-r--r--. 1 apache apache 45585 Jul 29 13:05 1564405504.35/repodata/97b983d84b21cca6f485389cb81856474502924a-other.xml.gz -rw-r--r--. 1 apache apache 674921 Jul 29 13:05 1564405504.35/repodata/bbf9e6b2768d5f57f2eb4c29b850aa3f33f40ac2-filelists.xml.gz -rw-r--r--. 1 apache apache 37759 Jul 29 13:05 1564405504.35/repodata/db8a96a2d3d6c8a9d38a79e77fd1a2e81161cea6-updateinfo.xml.gz -rw-r--r--. 1 apache apache 1813 Jul 29 13:05 1564405504.35/repodata/repomd.xml But for the broken channel repomd.xml contains no reference to the primary or other xml files: <?xml version="1.0" encoding="UTF-8"?> <repomd xmlns="http://linux.duke.edu/metadata/repo <http://linux.duke.edu/metadata/repo>" xmlns:rpm="http://linux.duke.edu/metadata/rpm <http://linux.duke.edu/metadata/rpm>"><revision>1564465100</revision> <data type="updateinfo"><location href="repodata/c45aa9b5d1d34e37f2e3797a3c73936d7c4db8be-updateinfo.xml.gz" /><timestamp>1564467055</timestamp><size>3878183</size><checksum type="sha1">c45aa9b5d1d34e37f2e3797a3c73936d7c4db8be</checksum> <open-size>19705102</open-size><open-checksum type="sha1">bbe90a0ff665d33c0a3e9302a397fd7f39ad6654</open-checksum></data> <data type="group"><location href="repodata/245ba6fe28e4a180b78cdd87f479af63cee47f4e-comps.xml" /><timestamp>1564467062</timestamp><size>1188540</size><checksum type="sha1">245ba6fe28e4a180b78cdd87f479af63cee47f4e</checksum></data> <data type="productid"><location href="repodata/productid" /><timestamp>1564457113</timestamp><size>2159</size><checksum type="sha1">625cf1b205833fcfb38a99c5ed94c3c25257379f</checksum></data> </repomd> While the working one has the repomd data with the correct references: <?xml version="1.0" encoding="UTF-8"?> <repomd xmlns="http://linux.duke.edu/metadata/repo <http://linux.duke.edu/metadata/repo>" xmlns:rpm="http://linux.duke.edu/metadata/rpm <http://linux.duke.edu/metadata/rpm>"><revision>1564405504</revision> <data type="filelists"><location href="repodata/bbf9e6b2768d5f57f2eb4c29b850aa3f33f40ac2-filelists.xml.gz" /><timestamp>1564405505</timestamp><size>674921</size><checksum type="sha1">bbf9e6b2768d5f57f2eb4c29b850aa3f33f40ac2</checksum><op en-size>17892914</open-size><open-checksum type="sha1">15526bf2873fe3e6cac20664cfb7b011b2ad1541</open-checksum></data> <data type="other"><location href="repodata/97b983d84b21cca6f485389cb81856474502924a-other.xml.gz" /><timestamp>1564405506</timestamp><size>45585</size><checksum type="sha1">97b983d84b21cca6f485389cb81856474502924a</checksum><open-size>2 42048</open-size><open-checksum type="sha1">6bc00f3aec0458704584b5b771271a22b4848de9</open-checksum></data> <data type="primary"><location href="repodata/48c121111734b4da32d6404a178195b1aff90ace-primary.xml.gz" /><timestamp>1564405506</timestamp><size>24475</size><checksum type="sha1">48c121111734b4da32d6404a178195b1aff90ace</checksum><open-si ze>244509</open-size><open-checksum type="sha1">23cb6df373205139fb3bf474f1ea3acb34007347</open-checksum></data> <data type="updateinfo"><location href="repodata/db8a96a2d3d6c8a9d38a79e77fd1a2e81161cea6-updateinfo.xml.gz" /><timestamp>1564405508</timestamp><size>37759</size><checksum type="sha1">db8a96a2d3d6c8a9d38a79e77fd1a2e81161cea6</checksum><o pen-size>217172</open-size><open-checksum type="sha1">49a1157c12b1764b387e3b83f0a2ee5413d63fc8</open-checksum></data> <data type="group"><location href="repodata/7a747e6dc73c3ccef57c839995d90b6ef4c76092-comps.xml" /><timestamp>1564405508</timestamp><size>124</size><checksum type="sha1">7a747e6dc73c3ccef57c839995d90b6ef4c76092</checksum></data> </repomd> We are seeing the following error when some of the sync jobs are running: pulp: pulp.plugins.util.publish_step:ERROR: [f22fccad] (12990-15904) Finalizing failed Jul 30 02:19:36 our-host pulp: pulp.plugins.util.publish_step:ERROR: [f22fccad] (12990-15904) Traceback (most recent call last): Jul 30 02:19:37 our-host pulp: pulp.plugins.util.publish_step:ERROR: [f22fccad] (12990-15904) File "/usr/lib/python2.7/site-packages/pulp/plugins/util/publish_step.py", line 265, in process Jul 30 02:19:37 our-host pulp: pulp.plugins.util.publish_step:ERROR: [f22fccad] (12990-15904) self.finalize() Jul 30 02:19:37 our-host pulp: pulp.plugins.util.publish_step:ERROR: [f22fccad] (12990-15904) File "/usr/lib/python2.7/site-packages/pulp_rpm/plugins/distributors/yum/publish.py", line 472, in finalize Jul 30 02:19:37 our-host pulp: pulp.plugins.util.publish_step:ERROR: [f22fccad] (12990-15904) self.file_lists_context.checksum) Jul 30 02:19:37 our-host pulp: pulp.plugins.util.publish_step:ERROR: [f22fccad] (12990-15904) File "/usr/lib/python2.7/site-packages/pulp_rpm/plugins/distributors/yum/metadata/repomd.py", line 132, in add_metadata_file_metadata Jul 30 02:19:37 our-host pulp: pulp.plugins.util.publish_step:ERROR: [f22fccad] (12990-15904) content = file_handle.read() Jul 30 02:19:37 our-host pulp: pulp.plugins.util.publish_step:ERROR: [f22fccad] (12990-15904) File "/usr/lib64/python2.7/gzip.py", line 254, in read Jul 30 02:19:37 our-host pulp: pulp.plugins.util.publish_step:ERROR: [f22fccad] (12990-15904) self._read(readsize) Jul 30 02:19:37 our-host pulp: pulp.plugins.util.publish_step:ERROR: [f22fccad] (12990-15904) File "/usr/lib64/python2.7/gzip.py", line 313, in _read Jul 30 02:19:37 our-host pulp: pulp.plugins.util.publish_step:ERROR: [f22fccad] (12990-15904) self._add_read_data( uncompress ) Jul 30 02:19:37 our-host pulp: pulp.plugins.util.publish_step:ERROR: [f22fccad] (12990-15904) File "/usr/lib64/python2.7/gzip.py", line 331, in _add_read_data Jul 30 02:19:37 our-host pulp: pulp.plugins.util.publish_step:ERROR: [f22fccad] (12990-15904) self.extrabuf = self.extrabuf[offset:] + data Jul 30 02:19:37 our-host pulp: pulp.plugins.util.publish_step:ERROR: [f22fccad] (12990-15904) MemoryError As I say we’ve solved our issue another way but am including the investigations as far as we got in the event it is useful elsewhere. Regards r. > On 31 Jul 2019, at 2:03 am, Ina Panova <[email protected]> wrote: > > Try to re-publish the repo with foce-full flag and see if the symlinks got > fixed. > > Also check /var/lib/pulp/published, there you can find created repodata and > symlinks pointing to the content. > > > -------- > Regards, > > Ina Panova > Senior Software Engineer| Pulp| Red Hat Inc. > > "Do not go where the path may lead, > go instead where there is no path and leave a trail." > > > On Sun, Jul 28, 2019 at 11:20 PM Rolf Loudon <[email protected] > <mailto:[email protected]>> wrote: > > Hello > > Thanks for the assistance. The problem is more confusing than at first > thought. > > If I go to http://xxxxxx/pulp/repos/rhel-6-server-rpms > <http://xxxxxx/pulp/repos/rhel-6-server-rpms> in a browser I see the two > links for ‘Packages’ and ‘repodata’. > > What I did not previously notice is that if I examine either link, or click > on them, the URL used is not correct. It is > http://xxxxxxx/pulp/repos/repodata <http://xxxxxxx/pulp/repos/repodata> which > returns a 404. > I tried manually entering > http://xxxxxx/pulp/repos/rhel-6-server-rpms/repodata > <http://xxxxxx/pulp/repos/rhel-6-server-rpms/repodata> and I get sensible > information: > > 21dc6f5f63a58ea0c1f5afc1089b954a840234a6-comps.xml > 5c8c97d6bbe41e935c933967a2a5f652ac138442-filelists.xml.gz > 8c5ebae207e15589b4dd0362930f6986337be2d7-updateinfo.xml.gz > other.xml.gz > primary.xml.gz > productid > repomd.xml > > Correspondingly if I use http://xxxxxx/pulp/repos/rhel-6-server-rpms/Packages > <http://xxxxxx/pulp/repos/rhel-6-server-rpms/Packages> I get a similar > sensible output. > > I tested further and find that this behaviour is the same for all the > categories, rhel-6-server-extras-rpms, rhel-6-server-optional-rpms, and > rhel-6-server-supp-rpms and so on: the URL for ‘Packages’ and ‘repodata’ is > not correct. > > What is further confusing is that using the spacewalk application to sync > packages from my pulp server does in fact work for the other categories, just > not for the Base packages: > > On the spacewalk host using the sync command: > > 1. For example the ‘Extras’ category: > > # spacewalk-repo-sync -c rhel6-server-extras-x86_64-pulp > 20:48:54 ====================================== > 20:48:54 | Channel: rhel6-server-extras-x86_64-pulp > 20:48:54 ====================================== > 20:48:54 Sync of channel started. > 20:48:54 Repo URL: http://xxxxxxxxxxxxx/pulp/repos/rhel-6-server-extras-rpms > <http://xxxxxxxxxxxxx/pulp/repos/rhel-6-server-extras-rpms> > 20:48:55 Packages in repo: 97 > 20:48:59 No new packages to sync. > 20:48:59 Repo http://xxxxxxxxxx/pulp/repos/rhel-6-server-extras-rpms > <http://xxxxxxxxxx/pulp/repos/rhel-6-server-extras-rpms> has comps file > 7a747e6dc73c3ccef57c839995d90b6ef4c76092-comps.xml. > 20:48:59 Repo http://xxxxxxxxxx/pulp/repos/rhel-6-server-extras-rpms > <http://xxxxxxxxxx/pulp/repos/rhel-6-server-extras-rpms> has 54 errata. > 20:49:02 Sync of channel completed in 0:00:07. > 20:49:02 Total time: 0:00:07 > > There are no new packages in this category but as is shown there is no error > returned > > 2. Using the base package: > > # spacewalk-repo-sync -c rhel6-x86_64-pulp > 20:49:23 ====================================== > 20:49:23 | Channel: rhel6-x86_64-pulp > 20:49:23 ====================================== > 20:49:23 Sync of channel started. > 20:49:23 Repo URL: http://xxxxxxxxxxx/pulp/repos/rhel-6-server-rpms > <http://xxxxxxxxxxx/pulp/repos/rhel-6-server-rpms> > 20:49:24 ERROR: requested datatype primary not available > 20:49:24 Sync of channel completed in 0:00:00. > 20:49:24 Total time: 0:00:00 > > The different behaviour between the Base set and the Extras, Supp, Optional, > etc is further confusing and unexplained. > > So the essence of the problem now is that the URL provided for ‘Packages’ and > ‘repodata’ in each category is wrong and does not include the category name. > > Any ideas? > > Many thanks > > r. > > On 26 Jul 2019, at 6:46 pm, Ina Panova <[email protected] > <mailto:[email protected]>> wrote: > >> Hi Rolf, >> >> if you go to http://xxxxxxxxxx/pulp/repos/rhel-6-server-rpms >> <http://xxxxxxxxxx/pulp/repos/rhel-6-server-rpms> can you share the >> repodata with us? >> The client seems to complain because of missing primary metadata file. >> >> -------- >> Regards, >> >> Ina Panova >> Senior Software Engineer| Pulp| Red Hat Inc. >> >> "Do not go where the path may lead, >> go instead where there is no path and leave a trail." >> >> >> On Wed, Jul 24, 2019 at 1:29 PM Rolf Loudon <[email protected] >> <mailto:[email protected]>> wrote: >> Thanks >> >> Pulp install is via RPM and is version 2.14.0-1 >> >> Yes, I am pointing the pulp create repo command to >> https://cdn.redhat.com/content/dist/rhel/server/6/6server/x86_64/os >> <https://cdn.redhat.com/content/dist/rhel/server/6/6server/x86_64/os> >> I have seen the downloads being transferred and seemingly via Akamai (which >> I presume RedHat uses). >> >> R. >> >>> On 24 Jul 2019, at 9:17 pm, Dennis Kliban <[email protected] >>> <mailto:[email protected]>> wrote: >>> >>> What version of Pulp are you using? Is the repository you are syncing >>> publicly available? >>> >>> On Wed, Jul 24, 2019 at 2:52 AM Rolf Loudon <[email protected] >>> <mailto:[email protected]>> wrote: >>> Hello >>> >>> I have created and synced a repository in pulp. >>> >>> pulp-admin repo list shows it as follows: >>> >>> Id: rhel-6-server-rpms >>> >>> Display Name: None >>> >>> Description: None >>> >>> Content Unit Counts: >>> >>> Erratum: 4324 >>> >>> Package Category: 10 >>> >>> Package Group: 202 >>> >>> Rpm: 20785 >>> >>> Yum Repo Metadata File: 1 >>> >>> >>> >>> I have a (old) spacewalk server on which I defined the above repo as a >>> repository source. However when I try to sync the repo into spacewalk I get >>> this: >>> >>> 06:44:11 Sync of channel started. >>> >>> 06:44:11 Repo URL: http://xxxxxxxxxx/pulp/repos/rhel-6-server-rpms >>> <http://xxxxxxxxxx/pulp/repos/rhel-6-server-rpms> >>> 06:44:11 ERROR: requested datatype primary not available >>> >>> 06:44:11 Sync of channel completed in 0:00:00. >>> >>> 06:44:11 Total time: 0:00:00 >>> >>> >>> >>> To eliminate spacewalk as the source of the problem I configured a server >>> that has never before used the pulp repo in /etc/yum.repos.d/pulp.repo : >>> >>> >>> >>> [rhel-base] >>> >>> name=Red Hat Enterprise Linux Pulp $releasever - $basearch >>> >>> baseurl=http://xxxxxxxxx/pulp/repos/rhel-6-server-rpms >>> <http://xxxxxxxxx/pulp/repos/rhel-6-server-rpms> >>> enabled=1 >>> >>> >>> >>> When I run ‘yum repolist’ it fails in the same way: >>> >>> >>> >>> # yum repolist >>> >>> Loaded plugins: priorities, protectbase, refresh-packagekit, rhnplugin >>> >>> This system is receiving updates from RHN Classic or Red Hat Satellite. >>> >>> Error: requested datatype primary not available >>> >>> >>> >>> I have tried numerous times to recreate the repo in pulp and resync and so >>> on. >>> >>> Any assistance much appreciated. >>> >>> r. >>> >>> _______________________________________________ >>> Pulp-list mailing list >>> [email protected] <mailto:[email protected]> >>> https://www.redhat.com/mailman/listinfo/pulp-list >>> <https://www.redhat.com/mailman/listinfo/pulp-list> >> _______________________________________________ >> Pulp-list mailing list >> [email protected] <mailto:[email protected]> >> https://www.redhat.com/mailman/listinfo/pulp-list >> <https://www.redhat.com/mailman/listinfo/pulp-list>_______________________________________________ > Pulp-list mailing list > [email protected] <mailto:[email protected]> > https://www.redhat.com/mailman/listinfo/pulp-list > <https://www.redhat.com/mailman/listinfo/pulp-list>
_______________________________________________ Pulp-list mailing list [email protected] https://www.redhat.com/mailman/listinfo/pulp-list
