Tried again for both rhel7 and rhel6 repo. The download_concurrency is always
set to "1". We tried to sync rhel6 repo first and task was completed. Rerun
sync on rhel7 repo failed again. Our network team didn't find any issues on the
network and proxy.
This is our remote setting:
{
"_created": "2019-06-21T16:10:05.493082Z",
"_href":
"/pulp/api/v3/remotes/rpm/rpm/b308a912-1dc9-46cb-b340-450a96e5ee93/",
"_last_updated": "2019-06-21T16:10:05.493107Z",
"_type": "rpm.rpm",
"download_concurrency": 1,
"name": "rhel-x86_64-server-7",
"policy": "immediate",
"proxy_url": "http://proxyhost.bloomberg.com:8000";,
"ssl_ca_certificate":
"c77dce7447d4862f2b5fbd794cf6fcdc6f01e3c313cc9a12d2bf47cfd6b8db21",
"ssl_client_certificate":
"c5c5dfac3e16e93b394af137ebf8f0c5eaa37277699e72dc8042a53f6133b999",
"ssl_client_key":
"3f8bde4c2baf94196a9e40d0834f86a2890b3df08053eef9283b3c136aae487f",
"ssl_validation": true,
"url":
"https://cdn.redhat.com/content/dist/rhel/server/7/7Server/x86_64/os/";,
"validate": true
}
Task status:
Allow: GET, PATCH, DELETE, HEAD, OPTIONS
Connection: close
Content-Length: 3515
Content-Type: application/json
Date: Thu, 27 Jun 2019 21:32:27 GMT
Server: gunicorn/19.9.0
Vary: Accept, Cookie
X-Frame-Options: SAMEORIGIN
{
"_created": "2019-06-27T18:15:28.484561Z",
"_href": "/pulp/api/v3/tasks/72da60f0-1beb-4447-b8d2-718a40b5fccc/",
"created_resources": [],
"error": {
"code": null,
"description": "Timeout on reading data from socket",
"traceback": " File
\"/opt/python/3.6.5/lib/python3.6/site-packages/rq/worker.py\", line 812, in
perform_job\n rv = job.perform()\n File
\"/opt/python/3.6.5/lib/python3.6/site-packages/rq/job.py\", line 588, in
perform\n self._result = self._execute()\n File
\"/opt/python/3.6.5/lib/python3.6/site-packages/rq/job.py\", line 594, in
_execute\n return self.func(*self.args, **self.kwargs)\n File
\"/opt/python/3.6.5/lib/python3.6/site-packages/pulp_rpm/app/tasks/synchronizing.py\",
line 68, in synchronize\n dv.create()\n File
\"/opt/python/3.6.5/lib/python3.6/site-packages/pulpcore/plugin/stages/declarative_version.py\",
line 169, in create\n loop.run_until_complete(pipeline)\n File
\"/opt/python/3.6.5/lib64/python3.6/asyncio/base_events.py\", line 468, in
run_until_complete\n return future.result()\n File
\"/opt/python/3.6.5/lib/python3.6/site-packages/pulpcore/plugin/stages/api.py\",
line 209, in create_pipeline\n await asyncio.gather(*futures)\n File
\"/opt/python/3.6.5/lib/python3.6/site-packages/pulpcore/plugin/stages/api.py\",
line 43, in __call__\n await self.run()\n File
\"/opt/python/3.6.5/lib/python3.6/site-packages/pulpcore/plugin/stages/artifact_stages.py\",
line 132, in run\n pb.done += task.result() # download_count\n File
\"/opt/python/3.6.5/lib/python3.6/site-packages/pulpcore/plugin/stages/artifact_stages.py\",
line 155, in _handle_content_unit\n await
asyncio.gather(*downloaders_for_content)\n File
\"/opt/python/3.6.5/lib/python3.6/site-packages/pulpcore/plugin/stages/models.py\",
line 78, in download\n download_result = await
downloader.run(extra_data=self.extra_data)\n File
\"/opt/python/3.6.5/lib/python3.6/site-packages/pulpcore/plugin/download/base.py\",
line 212, in run\n return await self._run(extra_data=extra_data)\n File
\"/opt/python/3.6.5/lib/python3.6/site-packages/backoff/_async.py\", line 131,
in retry\n ret = await target(*args, **kwargs)\n File
\"/opt/python/3.6.5/lib/python3.6/site-packages/pulpcore/plugin/download/http.py\",
line 184, in _run\n to_return = await self._handle_response(response)\n
File
\"/opt/python/3.6.5/lib/python3.6/site-packages/pulpcore/plugin/download/http.py\",
line 158, in _handle_response\n chunk = await
response.content.read(1048576) # 1 megabyte\n File
\"/opt/python/3.6.5/lib64/python3.6/site-packages/aiohttp/streams.py\", line
369, in read\n await self._wait('read')\n File
\"/opt/python/3.6.5/lib64/python3.6/site-packages/aiohttp/streams.py\", line
297, in _wait\n await waiter\n"
},
"finished_at": "2019-06-27T21:19:18.675004Z",
"name": "pulp_rpm.app.tasks.synchronizing.synchronize",
"non_fatal_errors": [],
"parent": null,
"progress_reports": [
{
"done": 20201,
"message": "Parsed Packages",
"state": "running",
"suffix": null,
"total": 24397
},
{
"done": 11966,
"message": "Downloading Artifacts",
"state": "failed",
"suffix": null,
"total": 11966
},
{
"done": 5,
"message": "Downloading Metadata Files",
"state": "canceled",
"suffix": null,
"total": 5
},
{
"done": 23300,
"message": "Associating Content",
"state": "canceled",
"suffix": null,
"total": 23300
},
{
"done": 3583,
"message": "Parsed Erratum",
"state": "running",
"suffix": null,
"total": 3583
}
],
"spawned_tasks": [],
"started_at": "2019-06-27T18:15:28.618096Z",
"state": "failed",
"worker": "/pulp/api/v3/workers/d1db2594-52b6-402e-8ef1-7c0a5635c3c4/"
}
From: [email protected] At: 06/27/19 10:54:05To: Bin Li (BLOOMBERG/ 120 PARK
)
Cc: [email protected], [email protected]
Subject: Re: [Pulp-list] create remote to redhat on pulp 3
It says "Timeout on reading data from socket" so either this is a bug, or the
remote server is not delivering you data over the socket connection, perhaps
due to some rate limiting. The remote has a download_concurrency option [0] so
perhaps reducing that will cause the server to give you all of the data. You
can also open an issue with reproducer details and we can try to reproduce it.
[0]:
https://pulp-rpm.readthedocs.io/en/latest/restapi.html#operation/remotes_rpm_rpm_create
On Thu, Jun 27, 2019 at 10:48 AM Bin Li (BLOOMBERG/ 120 PARK)
<[email protected]> wrote:
I am now able to sync the redhat repo through proxy but the task failed
eventually on a timeout after sync most of contents. I tried sync both rhel7
and rhel6 os repo and both gave the same error. Any idea what could cause this?
# ./get /pulp/api/v3/tasks/738fc1bc-463f-4d7e-9ac9-ebb1e629b6f7/
HTTP/1.1 200 OK
Allow: GET, PATCH, DELETE, HEAD, OPTIONS
Connection: close
Content-Length: 3512
Content-Type: application/json
Date: Thu, 27 Jun 2019 14:38:12 GMT
Server: gunicorn/19.9.0
Vary: Accept, Cookie
X-Frame-Options: SAMEORIGIN
{
"_created": "2019-06-26T20:35:38.514762Z",
"_href": "/pulp/api/v3/tasks/738fc1bc-463f-4d7e-9ac9-ebb1e629b6f7/",
"created_resources": [],
"error": {
"code": null,
"description": "Timeout on reading data from socket",
"traceback": " File
\"/opt/python/3.6.5/lib/python3.6/site-packages/rq/worker.py\", line 812, in
perform_job\n rv = job.perform()\n File
\"/opt/python/3.6.5/lib/python3.6/site-packages/rq/job.py\", line 588, in
perform\n self._result = self._execute()\n File
\"/opt/python/3.6.5/lib/python3.6/site-packages/rq/job.py\", line 594, in
_execute\n return self.func(*self.args, **self.kwargs)\n File
\"/opt/python/3.6.5/lib/python3.6/site-packages/pulp_rpm/app/tasks/synchronizing.py\",
line 68, in synchronize\n dv.create()\n File
\"/opt/python/3.6.5/lib/python3.6/site-packages/pulpcore/plugin/stages/declarative_version.py\",
line 169, in create\n loop.run_until_complete(pipeline)\n File
\"/opt/python/3.6.5/lib64/python3.6/asyncio/base_events.py\", line 468, in
run_until_complete\n return future.result()\n File
\"/opt/python/3.6.5/lib/python3.6/site-packages/pulpcore/plugin/stages/api.py\",
line 209, in create_pipeline\n await asyncio.gather(*futures)\n File
\"/opt/python/3.6.5/lib/python3.6/site-packages/pulpcore/plugin/stages/api.py\",
line 43, in __call__\n await self.run()\n File
\"/opt/python/3.6.5/lib/python3.6/site-packages/pulpcore/plugin/stages/artifact_stages.py\",
line 132, in run\n pb.done += task.result() # download_count\n File
\"/opt/python/3.6.5/lib/python3.6/site-packages/pulpcore/plugin/stages/artifact_stages.py\",
line 155, in _handle_content_unit\n await
asyncio.gather(*downloaders_for_content)\n File
\"/opt/python/3.6.5/lib/python3.6/site-packages/pulpcore/plugin/stages/models.py\",
line 78, in download\n download_result = await
downloader.run(extra_data=self.extra_data)\n File
\"/opt/python/3.6.5/lib/python3.6/site-packages/pulpcore/plugin/download/base.py\",
line 212, in run\n return await self._run(extra_data=extra_data)\n File
\"/opt/python/3.6.5/lib/python3.6/site-packages/backoff/_async.py\", line 131,
in retry\n ret = await target(*args, **kwargs)\n File
\"/opt/python/3.6.5/lib/python3.6/site-packages/pulpcore/plugin/download/http.py\",
line 184, in _run\n to_return = await self._handle_response(response)\n
File
\"/opt/python/3.6.5/lib/python3.6/site-packages/pulpcore/plugin/download/http.py\",
line 158, in _handle_response\n chunk = await
response.content.read(1048576) # 1 megabyte\n File
\"/opt/python/3.6.5/lib64/python3.6/site-packages/aiohttp/streams.py\", line
369, in read\n await self._wait('read')\n File
\"/opt/python/3.6.5/lib64/python3.6/site-packages/aiohttp/streams.py\", line
297, in _wait\n await waiter\n"
},
"finished_at": "2019-06-26T21:57:33.499085Z",
"name": "pulp_rpm.app.tasks.synchronizing.synchronize",
"non_fatal_errors": [],
"parent": null,
"progress_reports": [
{
"done": 3757,
"message": "Downloading Artifacts",
"state": "failed",
"suffix": null,
"total": 3757
},
{
"done": 5,
"message": "Downloading Metadata Files",
"state": "canceled",
"suffix": null,
"total": 5
},
{
"done": 11350,
"message": "Associating Content",
"state": "canceled",
"suffix": null,
"total": 11350
},
{
"done": 3582,
"message": "Parsed Erratum",
"state": "running",
"suffix": null,
"total": 3582
},
{
"done": 8201,
"message": "Parsed Packages",
"state": "running",
"suffix": null,
"total": 24392
}
],
"spawned_tasks": [],
"started_at": "2019-06-26T20:35:38.650635Z",
"state": "failed",
"worker": "/pulp/api/v3/workers/d1db2594-52b6-402e-8ef1-7c0a5635c3c4/"
}
From: [email protected] At: 06/23/19 07:14:50To: Bin Li (BLOOMBERG/ 120 PARK
)
Cc: [email protected]
Subject: Re: [Pulp-list] create remote to redhat on pulp 3
The problem is that the proxy setting is not being respected by Pulp. I filed
an issue about this[0] and also made a patch to fix it[1]. This patch should be
merged in the next 24 hours.
If you cannot access cdn.redhat.com without a proxy, the error you got makes
sense. I was only able to reproduce the error message when I added an incorrect
entry for cdn.redhat.com in /etc/hosts.
[0] https://pulp.plan.io/issues/5011
[1] https://github.com/pulp/pulpcore-plugin/pull/107
On Fri, Jun 21, 2019 at 12:26 PM Bin Li (BLOOMBERG/ 120 PARK)
<[email protected]> wrote:
Thanks. I am now getting a different error. Please advise
"_href": "/pulp/api/v3/tasks/fcc679b8-1ad3-4a16-834b-47b946caaeed/",
"created_resources": [],
"error": {
"code": null,
"description": "Cannot connect to host cdn.redhat.com:443 ssl:None
[Connect call failed ('23.65.16.251', 443)]",
"traceback": " File
\"/opt/python/3.6.5/lib/python3.6/site-packages/rq/worker.py\", line 812, in
perform_job\n rv = job.perform()\n File
\"/opt/python/3.6.5/lib/python3.6/site-packages/rq/job.py\", line 588, in
perform\n self._result = self._execute()\n File
\"/opt/python/3.6.5/lib/python3.6/site-packages/rq/job.py\", line 594, in
_execute\n return self.func(*self.args, **self.kwargs)\n File
\"/tmp/pulp_rpm/pulp_rpm/app/tasks/synchronizing.py\", line 67, in
synchronize\n dv.create()\n File
\"/tmp/pulpcore-plugin/pulpcore/plugin/stages/declarative_version.py\", line
169, in create\n loop.run_until_complete(pipeline)\n File
\"/opt/python/3.6.5/lib64/python3.6/asyncio/base_events.py\", line 468, in
run_until_complete\n return future.result()\n File
\"/tmp/pulpcore-plugin/pulpcore/plugin/stages/api.py\", line 209, in
create_pipeline\n await asyncio.gather(*futures)\n File
\"/tmp/pulpcore-plugin/pulpcore/plugin/stages/api.py\", line 43, in __call__\n
await self.run()\n File
\"/tmp/pulp_rpm/pulp_rpm/app/tasks/synchronizing.py\", line 231, in run\n
result = await downloader.run()\n File
\"/tmp/pulpcore-plugin/pulpcore/plugin/download/base.py\", line 212, in run\n
return await self._run(extra_data=extra_data)\n File
\"/opt/python/3.6.5/lib/python3.6/site-packages/backoff/_async.py\", line 131,
in retry\n ret = await target(*args, **kwargs)\n File
\"/tmp/pulpcore-plugin/pulpcore/plugin/download/http.py\", line 182, in _run\n
async with self.session.get(self.url) as response:\n File
\"/opt/python/3.6.5/lib64/python3.6/site-packages/aiohttp/client.py\", line
1005, in __aenter__\n self._resp = await self._coro\n File
\"/opt/python/3.6.5/lib64/python3.6/site-packages/aiohttp/client.py\", line
476, in _request\n timeout=real_timeout\n File
\"/opt/python/3.6.5/lib64/python3.6/site-packages/aiohttp/connector.py\", line
522, in connect\n proto = await self._create_connection(req, traces,
timeout)\n File
\"/opt/python/3.6.5/lib64/python3.6/site-packages/aiohttp/connector.py\", line
854, in _create_connection\n req, traces, timeout)\n File
\"/opt/python/3.6.5/lib64/python3.6/site-packages/aiohttp/connector.py\", line
992, in _create_direct_connection\n raise last_exc\n File
\"/opt/python/3.6.5/lib64/python3.6/site-packages/aiohttp/connector.py\", line
974, in _create_direct_connection\n req=req, client_error=client_error)\n
File \"/opt/python/3.6.5/lib64/python3.6/site-packages/aiohttp/connector.py\",
line 931, in _wrap_create_connection\n raise
client_error(req.connection_key, exc) from exc\n"
},
I was able to connect using client cert through proxy
# curl -Iv --cacert /etc/rhsm/ca/redhat-uep.pem --key
/etc/pki/entitlement/8521692907269500331-key.pem --cert
/etc/pki/entitlement/8521692907269500331.pem
https://cdn.redhat.com/content/dist/rhel/server/7/7Server/x86_64/os/
* About to connect() to proxy ... port ... (#0)
* Trying ...
* Connected to ... () port ... (#0)
* Establish HTTP proxy tunnel to cdn.redhat.com:443
> CONNECT cdn.redhat.com:443 HTTP/1.1
> Host: cdn.redhat.com:443
> User-Agent: curl/7.29.0
> Proxy-Connection: Keep-Alive
>
< HTTP/1.1 200 Connection established
HTTP/1.1 200 Connection established
<
* Proxy replied OK to CONNECT request
* Initializing NSS with certpath: sql:/etc/pki/nssdb
From: [email protected] At: 06/20/19 16:34:40To: Bin Li (BLOOMBERG/ 120 PARK
)
Cc: [email protected]
Subject: Re: [Pulp-list] create remote to redhat on pulp 3
You need to make sure that the following services are running:
pulp-resource-manager, pulp-worker@0, pulp-worker@1 (at least 1 worker is
needed).
On Thu, Jun 20, 2019 at 4:11 PM Bin Li (BLOOMBERG/ 120 PARK)
<[email protected]> wrote:
I installed the latest version of component. No more errors when try to sync
the repo but the task is in waiting state forever. Any idea what I should check?
"versions": [
{
"component": "pulpcore",
"version": "3.0.0rc3.dev0"
},
{
"component": "pulpcore-plugin",
"version": "0.1.0rc3.dev0"
},
{
"component": "pulp_rpm",
"version": "3.0.0b4.dev0"
}
From: [email protected] At: 06/18/19 13:42:01To: Bin Li (BLOOMBERG/ 120 PARK
)
Cc: [email protected]
Subject: Re: [Pulp-list] create remote to redhat on pulp 3
This issue has now been resolved and a fix is available on master branch of
pulpcore-plugin[0].
[0] https://github.com/pulp/pulpcore-plugin/tree/master/
On Wed, Jun 12, 2019 at 1:51 PM Dennis Kliban <[email protected]> wrote:
There is an open issue about this[0]. I recommend clicking the watch button so
you get notifications about it. We should be fixing this soon.
[0] https://pulp.plan.io/issues/4825
On Wed, Jun 12, 2019 at 1:42 PM Bin Li (BLOOMBERG/ 120 PARK)
<[email protected]> wrote:
I was not able to sync a protected repo from redhat. The host running pulp 3
subscribes to redhat directly.
I first tried to use the following to create remote:
{
"name": "rhel-x86_64-server-7",
"url": "https://cdn.redhat.com/content/dist/rhel/server/7/7Server/x86_64/os";,
"ssl_ca_certificate": "/etc/rhsm/ca/redhat-uep.pem",
"ssl_client_certificate": "/etc/pki/consumer/cert.pem",
"ssl_client_key": "/etc/pki/consumer/key.pem",
"ssl_validation": true,
"proxy_url": "http://proxy:80";,
"download_concurrency": 1,
"policy": "immediate"
}
Remote was created but I got errors "Jun 12 12:51:43 ip-1-76-158-244 rq[19389]:
pulp: rq.worker:ERROR: ssl.SSLError: [PEM: NO_START_LINE] no start line
(_ssl.c:3626)" when I tried to sync with http POST :24817${REMOTE_HREF}sync/
repository=$REPO_HREF
Then I tried to converted the pem file to a json format
{
"name": "rhel-x86_64-server-7",
"url": "https://cdn.redhat.com/content/dist/rhel/server/7/7Server/x86_64/os";,
"ssl_ca_certificate": "-----BEGIN
CERTIFICATE-----\nMIIG/TCCBOWgAwIBAgIBNzANBgkqhkiG9w0BAQUFADCBsTELMAkGA1UEBhMCVVMx\nFzAVBg
...
"ssl_client_certificate": "-----BEGIN
CERTIFICATE-----\nMIIMojCCCoqgAwIBAgIISqmnKnJ9sEowDQ
...
"ssl_client_key": "-----BEGIN RSA PRIVATE
KEY-----\nMIIJKQIBAAKCAgEAuujl5HxnIDTSOemZOyH3Jr5xxgrc1rCee
...
"ssl_validation": true,
"proxy_url": "http://proxy:80";,
"download_concurrency": 1,
"policy": "immediate"
}
I got this error when I tried to sync repo with the remote:
Jun 12 12:07:57 ip-1-76-158-244 rq[19389]: ssl.SSLError: [SSL] PEM lib
(_ssl.c:3503)
What is the correct json input for creating a remote to redhat? How do I
troubleshoot the sync issue? Thanks for your
help._______________________________________________
Pulp-list mailing list
[email protected]
https://www.redhat.com/mailman/listinfo/pulp-list
_______________________________________________
Pulp-list mailing list
[email protected]
https://www.redhat.com/mailman/listinfo/pulp-list
_______________________________________________
Pulp-list mailing list
[email protected]
https://www.redhat.com/mailman/listinfo/pulp-list
