+1 to what @Daniel Alley <[email protected]> said. I'd encourage to use temporary workarounds mentioned by Daniel to mitigate the issue and once 3.14 is out to upgrade if possible.
-------- Regards, Ina Panova Senior Software Engineer| Pulp| Red Hat Inc. "Do not go where the path may lead, go instead where there is no path and leave a trail." On Fri, Jun 18, 2021 at 9:55 PM Daniel Alley <[email protected]> wrote: > Well.. the specific fix we need here *might* be backportable, although > the full patch is not. I can look into it next week, if the issues > continue. > > On Fri, Jun 18, 2021 at 3:46 PM Daniel Alley <[email protected]> wrote: > >> Yup, that's the issue. I've seen all 3 of those errors before while I >> was testing my fix for this issue (retriable downloads). The >> cdn.redhat.com seems to be having a very bad day, today. >> >> It's a feature though, and not backportable, because the changes touch >> the database and several different areas of code. My only suggestions are >> to either reduce the download_concurrency further, use on_demand sync so >> that it doesn't immediately download packages at all, or wait (because the >> CDN issue is likely temporary). And upgrade once you get the chance (once >> Katello packages pulpcore 3.14). >> >> On Fri, Jun 18, 2021 at 3:03 PM Bin Li (BLOOMBERG/ 120 PARK) < >> [email protected]> wrote: >> >>> Still have errors on this repo >>> https://cdn.redhat.com/content/dist/rhel8/8.4/aarch64/appstream/os >>> >>> >>> Got 3 different errors for last 3 runs. >>> Jun 18 14:17:22 pulpserver rq: pulp: rq.worker:ERROR: >>> aiohttp.client_exceptions.ClientPayloadError: Response payload is not >>> completed >>> Jun 18 14:17:22 pulpserver rq: Traceback (most recent call last): >>> Jun 18 14:17:22 pulpserver rq: File >>> "/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/rq/worker.py", >>> line 886, in perform_job >>> Jun 18 14:17:22 pulpserver rq: rv = job.perform() >>> Jun 18 14:17:22 pulpserver rq: File >>> "/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/rq/job.py", line >>> 664, in perform >>> Jun 18 14:17:22 pulpserver rq: self._result = self._execute() >>> Jun 18 14:17:22 pulpserver rq: File >>> "/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/rq/job.py", line >>> 670, in _execute >>> Jun 18 14:17:22 pulpserver rq: return self.func(*self.args, >>> **self.kwargs) >>> Jun 18 14:17:22 pulpserver rq: File >>> "/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/pulp_rpm/app/tasks/synchronizing.py", >>> line 266, in synchronize >>> Jun 18 14:17:22 pulpserver rq: dv.create() >>> Jun 18 14:17:22 pulpserver rq: File >>> "/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/pulpcore/plugin/stages/declarative_version.py", >>> line 148, in create >>> Jun 18 14:17:22 pulpserver rq: loop.run_until_complete(pipeline) >>> Jun 18 14:17:22 pulpserver rq: File >>> "/opt/python/3.7.3/lib64/python3.7/asyncio/base_events.py", line 584, in >>> run_until_complete >>> Jun 18 14:17:22 pulpserver rq: return future.result() >>> Jun 18 14:17:22 pulpserver rq: File >>> "/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/pulpcore/plugin/stages/api.py", >>> line 225, in create_pipeline >>> Jun 18 14:17:22 pulpserver rq: await asyncio.gather(*futures) >>> Jun 18 14:17:22 pulpserver rq: File >>> "/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/pulpcore/plugin/stages/api.py", >>> line 43, in __call__ >>> Jun 18 14:17:22 pulpserver rq: await self.run() >>> Jun 18 14:17:22 pulpserver rq: File >>> "/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/pulpcore/plugin/stages/artifact_stages.py", >>> line 152, in run >>> Jun 18 14:17:22 pulpserver rq: pb.done += task.result() # download_count >>> Jun 18 14:17:22 pulpserver rq: File >>> "/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/pulpcore/plugin/stages/artifact_stages.py", >>> line 178, in _handle_content_unit >>> Jun 18 14:17:22 pulpserver rq: await >>> asyncio.gather(*downloaders_for_content) >>> Jun 18 14:17:22 pulpserver rq: File >>> "/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/pulpcore/plugin/stages/models.py", >>> line 88, in download >>> Jun 18 14:17:22 pulpserver rq: download_result = await >>> downloader.run(extra_data=self.extra_data) >>> Jun 18 14:17:22 pulpserver rq: File >>> "/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/pulpcore/download/base.py", >>> line 227, in run >>> Jun 18 14:17:22 pulpserver rq: return await >>> self._run(extra_data=extra_data) >>> Jun 18 14:17:22 pulpserver rq: File >>> "/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/pulp_rpm/app/downloaders.py", >>> line 89, in _run >>> Jun 18 14:17:22 pulpserver rq: to_return = await >>> self._handle_response(response) >>> Jun 18 14:17:22 pulpserver rq: File >>> "/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/pulpcore/download/http.py", >>> line 180, in _handle_response >>> Jun 18 14:17:22 pulpserver rq: chunk = await >>> response.content.read(1048576) # 1 megabyte >>> Jun 18 14:17:22 pulpserver rq: File >>> "/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/aiohttp/streams.py", >>> line 368, in read >>> Jun 18 14:17:22 pulpserver rq: await self._wait('read') >>> Jun 18 14:17:22 pulpserver rq: File >>> "/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/aiohttp/streams.py", >>> line 296, in _wait >>> Jun 18 14:17:22 pulpserver rq: await waiter >>> Jun 18 14:17:22 pulpserver rq: >>> aiohttp.client_exceptions.ClientPayloadError: Response payload is not >>> completed >>> Jun 18 14:17:22 pulpserver rq: Traceback (most recent call last): >>> Jun 18 14:17:22 pulpserver rq: File >>> "/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/rq/worker.py", >>> line 886, in perform_job >>> Jun 18 14:17:22 pulpserver rq: rv = job.perform() >>> Jun 18 14:17:22 pulpserver rq: File >>> "/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/rq/job.py", line >>> 664, in perform >>> Jun 18 14:17:22 pulpserver rq: self._result = self._execute() >>> Jun 18 14:17:22 pulpserver rq: File >>> "/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/rq/job.py", line >>> 670, in _execute >>> Jun 18 14:17:22 pulpserver rq: return self.func(*self.args, >>> **self.kwargs) >>> Jun 18 14:17:22 pulpserver rq: File >>> "/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/pulp_rpm/app/tasks/synchronizing.py", >>> line 266, in synchronize >>> Jun 18 14:17:22 pulpserver rq: dv.create() >>> Jun 18 14:17:22 pulpserver rq: File >>> "/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/pulpcore/plugin/stages/declarative_version.py", >>> line 148, in create >>> Jun 18 14:17:22 pulpserver rq: loop.run_until_complete(pipeline) >>> Jun 18 14:17:22 pulpserver rq: File >>> "/opt/python/3.7.3/lib64/python3.7/asyncio/base_events.py", line 584, in >>> run_until_complete >>> Jun 18 14:17:22 pulpserver rq: return future.result() >>> Jun 18 14:17:22 pulpserver rq: File >>> "/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/pulpcore/plugin/stages/api.py", >>> line 225, in create_pipeline >>> Jun 18 14:17:22 pulpserver rq: await asyncio.gather(*futures) >>> Jun 18 14:17:22 pulpserver rq: File >>> "/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/pulpcore/plugin/stages/api.py", >>> line 43, in __call__ >>> Jun 18 14:17:22 pulpserver rq: await self.run() >>> Jun 18 14:17:22 pulpserver rq: File >>> "/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/pulpcore/plugin/stages/artifact_stages.py", >>> line 152, in run >>> Jun 18 14:17:22 pulpserver rq: pb.done += task.result() # download_count >>> Jun 18 14:17:22 pulpserver rq: File >>> "/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/pulpcore/plugin/stages/artifact_stages.py", >>> line 178, in _handle_content_unit >>> Jun 18 14:17:22 pulpserver rq: await >>> asyncio.gather(*downloaders_for_content) >>> Jun 18 14:17:22 pulpserver rq: File >>> "/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/pulpcore/plugin/stages/models.py", >>> line 88, in download >>> Jun 18 14:17:22 pulpserver rq: download_result = await >>> downloader.run(extra_data=self.extra_data) >>> Jun 18 14:17:22 pulpserver rq: File >>> "/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/pulpcore/download/base.py", >>> line 227, in run >>> Jun 18 14:17:22 pulpserver rq: return await >>> self._run(extra_data=extra_data) >>> Jun 18 14:17:22 pulpserver rq: File >>> "/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/pulp_rpm/app/downloaders.py", >>> line 89, in _run >>> Jun 18 14:17:22 pulpserver rq: to_return = await >>> self._handle_response(response) >>> Jun 18 14:17:22 pulpserver rq: File >>> "/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/pulpcore/download/http.py", >>> line 180, in _handle_response >>> Jun 18 14:17:22 pulpserver rq: chunk = await >>> response.content.read(1048576) # 1 megabyte >>> Jun 18 14:17:22 pulpserver rq: File >>> "/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/aiohttp/streams.py", >>> line 368, in read >>> Jun 18 14:17:22 pulpserver rq: await self._wait('read') >>> Jun 18 14:17:22 pulpserver rq: File >>> "/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/aiohttp/streams.py", >>> line 296, in _wait >>> Jun 18 14:17:22 pulpserver rq: await waiter >>> Jun 18 14:17:22 pulpserver rq: >>> aiohttp.client_exceptions.ClientPayloadError: Response payload is not >>> completed >>> ================================== >>> Jun 18 14:34:09 pulpserver rq: Traceback (most recent call last): >>> Jun 18 14:34:09 pulpserver rq: File >>> "/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/aiohttp/connector.py", >>> line 936, in _wrap_create_connection >>> Jun 18 14:34:09 pulpserver rq: return await >>> self._loop.create_connection(*args, **kwargs) # type: ignore # noqa >>> Jun 18 14:34:09 pulpserver rq: File >>> "/opt/python/3.7.3/lib64/python3.7/asyncio/base_events.py", line 986, in >>> create_connection >>> Jun 18 14:34:09 pulpserver rq: >>> ssl_handshake_timeout=ssl_handshake_timeout) >>> Jun 18 14:34:09 pulpserver rq: File >>> "/opt/python/3.7.3/lib64/python3.7/asyncio/base_events.py", line 1014, in >>> _create_connection_transport >>> Jun 18 14:34:09 pulpserver rq: await waiter >>> Jun 18 14:34:09 pulpserver rq: File >>> "/opt/python/3.7.3/lib64/python3.7/asyncio/sslproto.py", line 526, in >>> data_received >>> Jun 18 14:34:09 pulpserver rq: ssldata, appdata = >>> self._sslpipe.feed_ssldata(data) >>> Jun 18 14:34:09 pulpserver rq: File >>> "/opt/python/3.7.3/lib64/python3.7/asyncio/sslproto.py", line 189, in >>> feed_ssldata >>> Jun 18 14:34:09 pulpserver rq: self._sslobj.do_handshake() >>> Jun 18 14:34:09 pulpserver rq: File >>> "/opt/python/3.7.3/lib64/python3.7/ssl.py", line 763, in do_handshake >>> Jun 18 14:34:09 pulpserver rq: self._sslobj.do_handshake() >>> Jun 18 14:34:09 pulpserver rq: ssl.SSLError: [SSL: >>> SSLV3_ALERT_UNEXPECTED_MESSAGE] sslv3 alert unexpected message (_ssl.c:1056) >>> Jun 18 14:34:09 pulpserver rq: The above exception was the direct cause >>> of the following exception: >>> Jun 18 14:34:09 pulpserver rq: Traceback (most recent call last): >>> Jun 18 14:34:09 pulpserver rq: File >>> "/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/rq/worker.py", >>> line 886, in perform_job >>> Jun 18 14:34:09 pulpserver rq: rv = job.perform() >>> Jun 18 14:34:09 pulpserver rq: File >>> "/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/rq/job.py", line >>> 664, in perform >>> Jun 18 14:34:09 pulpserver rq: self._result = self._execute() >>> Jun 18 14:34:09 pulpserver rq: File >>> "/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/rq/job.py", line >>> 670, in _execute >>> Jun 18 14:34:09 pulpserver rq: return self.func(*self.args, >>> **self.kwargs) >>> Jun 18 14:34:09 pulpserver rq: File >>> "/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/pulp_rpm/app/tasks/synchronizing.py", >>> line 266, in synchronize >>> Jun 18 14:34:09 pulpserver rq: dv.create() >>> Jun 18 14:34:09 pulpserver rq: File >>> "/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/pulpcore/plugin/stages/declarative_version.py", >>> line 148, in create >>> Jun 18 14:34:09 pulpserver rq: loop.run_until_complete(pipeline) >>> Jun 18 14:34:09 pulpserver rq: File >>> "/opt/python/3.7.3/lib64/python3.7/asyncio/base_events.py", line 584, in >>> run_until_complete >>> Jun 18 14:34:09 pulpserver rq: return future.result() >>> Jun 18 14:34:09 pulpserver rq: File >>> "/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/pulpcore/plugin/stages/api.py", >>> line 225, in create_pipeline >>> Jun 18 14:34:09 pulpserver rq: await asyncio.gather(*futures) >>> Jun 18 14:34:09 pulpserver rq: File >>> "/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/pulpcore/plugin/stages/api.py", >>> line 43, in __call__ >>> Jun 18 14:34:09 pulpserver rq: await self.run() >>> Jun 18 14:34:09 pulpserver rq: File >>> "/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/pulpcore/plugin/stages/artifact_stages.py", >>> line 152, in run >>> Jun 18 14:34:09 pulpserver rq: pb.done += task.result() # download_count >>> Jun 18 14:34:09 pulpserver rq: File >>> "/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/pulpcore/plugin/stages/artifact_stages.py", >>> line 178, in _handle_content_unit >>> Jun 18 14:34:09 pulpserver rq: await >>> asyncio.gather(*downloaders_for_content) >>> Jun 18 14:34:09 pulpserver rq: File >>> "/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/pulpcore/plugin/stages/models.py", >>> line 88, in download >>> Jun 18 14:34:09 pulpserver rq: download_result = await >>> downloader.run(extra_data=self.extra_data) >>> Jun 18 14:34:09 pulpserver rq: File >>> "/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/pulpcore/download/base.py", >>> line 227, in run >>> Jun 18 14:34:09 pulpserver rq: return await >>> self._run(extra_data=extra_data) >>> Jun 18 14:34:09 pulpserver rq: File >>> "/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/pulp_rpm/app/downloaders.py", >>> line 87, in _run >>> Jun 18 14:34:09 pulpserver rq: async with self.session.get(url, >>> proxy=self.proxy, auth=self.auth) as response: >>> Jun 18 14:34:09 pulpserver rq: File >>> "/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/aiohttp/client.py", >>> line 1012, in __aenter__ >>> Jun 18 14:34:09 pulpserver rq: self._resp = await self._coro >>> Jun 18 14:34:09 pulpserver rq: File >>> "/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/aiohttp/client.py", >>> line 483, in _request >>> Jun 18 14:34:09 pulpserver rq: timeout=real_timeout >>> Jun 18 14:34:09 pulpserver rq: File >>> "/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/aiohttp/connector.py", >>> line 523, in connect >>> Jun 18 14:34:09 pulpserver rq: proto = await >>> self._create_connection(req, traces, timeout) >>> Jun 18 14:34:09 pulpserver rq: File >>> "/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/aiohttp/connector.py", >>> line 856, in _create_connection >>> Jun 18 14:34:09 pulpserver rq: req, traces, timeout) >>> Jun 18 14:34:09 pulpserver rq: File >>> "/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/aiohttp/connector.py", >>> line 1097, in _create_proxy_connection >>> Jun 18 14:34:09 pulpserver rq: req=req) >>> Jun 18 14:34:09 pulpserver rq: File >>> "/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/aiohttp/connector.py", >>> line 941, in _wrap_create_connection >>> Jun 18 14:34:09 pulpserver rq: raise >>> ClientConnectorSSLError(req.connection_key, exc) from exc >>> Jun 18 14:34:09 pulpserver rq: >>> aiohttp.client_exceptions.ClientConnectorSSLError: Cannot connect to host >>> cdn.redhat.com:443 ssl:default [[SSL: SSLV3_ALERT_UNEXPECTED_MESSAGE] >>> sslv3 alert unexpected message (_ssl.c:1056)] >>> Jun 18 14:34:09 pulpserver rq: Traceback (most recent call last): >>> Jun 18 14:34:09 pulpserver rq: File >>> "/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/aiohttp/connector.py", >>> line 936, in _wrap_create_connection >>> Jun 18 14:34:09 pulpserver rq: return await >>> self._loop.create_connection(*args, **kwargs) # type: ignore # noqa >>> Jun 18 14:34:09 pulpserver rq: File >>> "/opt/python/3.7.3/lib64/python3.7/asyncio/base_events.py", line 986, in >>> create_connection >>> Jun 18 14:34:09 pulpserver rq: >>> ssl_handshake_timeout=ssl_handshake_timeout) >>> Jun 18 14:34:09 pulpserver rq: File >>> "/opt/python/3.7.3/lib64/python3.7/asyncio/base_events.py", line 1014, in >>> _create_connection_transport >>> Jun 18 14:34:09 pulpserver rq: await waiter >>> Jun 18 14:34:09 pulpserver rq: File >>> "/opt/python/3.7.3/lib64/python3.7/asyncio/sslproto.py", line 526, in >>> data_received >>> Jun 18 14:34:09 pulpserver rq: ssldata, appdata = >>> self._sslpipe.feed_ssldata(data) >>> Jun 18 14:34:09 pulpserver rq: File >>> "/opt/python/3.7.3/lib64/python3.7/asyncio/sslproto.py", line 189, in >>> feed_ssldata >>> Jun 18 14:34:09 pulpserver rq: self._sslobj.do_handshake() >>> Jun 18 14:34:09 pulpserver rq: File >>> "/opt/python/3.7.3/lib64/python3.7/ssl.py", line 763, in do_handshake >>> Jun 18 14:34:09 pulpserver rq: self._sslobj.do_handshake() >>> Jun 18 14:34:09 pulpserver rq: ssl.SSLError: [SSL: >>> SSLV3_ALERT_UNEXPECTED_MESSAGE] sslv3 alert unexpected message (_ssl.c:1056) >>> Jun 18 14:34:09 pulpserver rq: The above exception was the direct cause >>> of the following exception: >>> Jun 18 14:34:09 pulpserver rq: Traceback (most recent call last): >>> Jun 18 14:34:09 pulpserver rq: File >>> "/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/rq/worker.py", >>> line 886, in perform_job >>> Jun 18 14:34:09 pulpserver rq: rv = job.perform() >>> Jun 18 14:34:09 pulpserver rq: File >>> "/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/rq/job.py", line >>> 664, in perform >>> Jun 18 14:34:09 pulpserver rq: self._result = self._execute() >>> Jun 18 14:34:09 pulpserver rq: File >>> "/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/rq/job.py", line >>> 670, in _execute >>> Jun 18 14:34:09 pulpserver rq: return self.func(*self.args, >>> **self.kwargs) >>> Jun 18 14:34:09 pulpserver rq: File >>> "/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/pulp_rpm/app/tasks/synchronizing.py", >>> line 266, in synchronize >>> Jun 18 14:34:09 pulpserver rq: dv.create() >>> Jun 18 14:34:09 pulpserver rq: File >>> "/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/pulpcore/plugin/stages/declarative_version.py", >>> line 148, in create >>> Jun 18 14:34:09 pulpserver rq: loop.run_until_complete(pipeline) >>> Jun 18 14:34:09 pulpserver rq: File >>> "/opt/python/3.7.3/lib64/python3.7/asyncio/base_events.py", line 584, in >>> run_until_complete >>> Jun 18 14:34:09 pulpserver rq: return future.result() >>> Jun 18 14:34:09 pulpserver rq: File >>> "/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/pulpcore/plugin/stages/api.py", >>> line 225, in create_pipeline >>> Jun 18 14:34:09 pulpserver rq: await asyncio.gather(*futures) >>> Jun 18 14:34:09 pulpserver rq: File >>> "/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/pulpcore/plugin/stages/api.py", >>> line 43, in __call__ >>> Jun 18 14:34:09 pulpserver rq: await self.run() >>> Jun 18 14:34:09 pulpserver rq: File >>> "/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/pulpcore/plugin/stages/artifact_stages.py", >>> line 152, in run >>> Jun 18 14:34:09 pulpserver rq: pb.done += task.result() # download_count >>> Jun 18 14:34:09 pulpserver rq: File >>> "/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/pulpcore/plugin/stages/artifact_stages.py", >>> line 178, in _handle_content_unit >>> Jun 18 14:34:09 pulpserver rq: await >>> asyncio.gather(*downloaders_for_content) >>> Jun 18 14:34:09 pulpserver rq: File >>> "/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/pulpcore/plugin/stages/models.py", >>> line 88, in download >>> Jun 18 14:34:09 pulpserver rq: download_result = await >>> downloader.run(extra_data=self.extra_data) >>> Jun 18 14:34:09 pulpserver rq: File >>> "/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/pulpcore/download/base.py", >>> line 227, in run >>> Jun 18 14:34:09 pulpserver rq: return await >>> self._run(extra_data=extra_data) >>> Jun 18 14:34:09 pulpserver rq: File >>> "/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/pulp_rpm/app/downloaders.py", >>> line 87, in _run >>> Jun 18 14:34:09 pulpserver rq: async with self.session.get(url, >>> proxy=self.proxy, auth=self.auth) as response: >>> Jun 18 14:34:09 pulpserver rq: File >>> "/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/aiohttp/client.py", >>> line 1012, in __aenter__ >>> Jun 18 14:34:09 pulpserver rq: self._resp = await self._coro >>> Jun 18 14:34:09 pulpserver rq: File >>> "/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/aiohttp/client.py", >>> line 483, in _request >>> Jun 18 14:34:09 pulpserver rq: timeout=real_timeout >>> Jun 18 14:34:09 pulpserver rq: File >>> "/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/aiohttp/connector.py", >>> line 523, in connect >>> Jun 18 14:34:09 pulpserver rq: proto = await >>> self._create_connection(req, traces, timeout) >>> Jun 18 14:34:09 pulpserver rq: File >>> "/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/aiohttp/connector.py", >>> line 856, in _create_connection >>> Jun 18 14:34:09 pulpserver rq: req, traces, timeout) >>> Jun 18 14:34:09 pulpserver rq: File >>> "/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/aiohttp/connector.py", >>> line 1097, in _create_proxy_connection >>> Jun 18 14:34:09 pulpserver rq: req=req) >>> Jun 18 14:34:09 pulpserver rq: File >>> "/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/aiohttp/connector.py", >>> line 941, in _wrap_create_connection >>> Jun 18 14:34:09 pulpserver rq: raise >>> ClientConnectorSSLError(req.connection_key, exc) from exc >>> Jun 18 14:34:09 pulpserver rq: >>> aiohttp.client_exceptions.ClientConnectorSSLError: Cannot connect to host >>> cdn.redhat.com:443 ssl:default [[SSL: SSLV3_ALERT_UNEXPECTED_MESSAGE] >>> sslv3 alert unexpected message (_ssl.c:1056)] >>> >>> ================================= >>> Jun 18 14:47:28 pulpserver rq: pulp: rq.worker:ERROR: >>> aiohttp.client_exceptions.ServerDisconnectedError >>> Jun 18 14:47:28 pulpserver rq: Traceback (most recent call last): >>> Jun 18 14:47:28 pulpserver rq: File >>> "/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/rq/worker.py", >>> line 886, in perform_job >>> Jun 18 14:47:28 pulpserver rq: rv = job.perform() >>> Jun 18 14:47:28 pulpserver rq: File >>> "/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/rq/job.py", line >>> 664, in perform >>> Jun 18 14:47:28 pulpserver rq: self._result = self._execute() >>> Jun 18 14:47:28 pulpserver rq: File >>> "/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/rq/job.py", line >>> 670, in _execute >>> Jun 18 14:47:28 pulpserver rq: return self.func(*self.args, >>> **self.kwargs) >>> Jun 18 14:47:28 pulpserver rq: File >>> "/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/pulp_rpm/app/tasks/synchronizing.py", >>> line 266, in synchronize >>> Jun 18 14:47:28 pulpserver rq: dv.create() >>> Jun 18 14:47:28 pulpserver rq: File >>> "/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/pulpcore/plugin/stages/declarative_version.py", >>> line 148, in create >>> Jun 18 14:47:28 pulpserver rq: loop.run_until_complete(pipeline) >>> Jun 18 14:47:28 pulpserver rq: File >>> "/opt/python/3.7.3/lib64/python3.7/asyncio/base_events.py", line 584, in >>> run_until_complete >>> Jun 18 14:47:28 pulpserver rq: return future.result() >>> Jun 18 14:47:28 pulpserver rq: File >>> "/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/pulpcore/plugin/stages/api.py", >>> line 225, in create_pipeline >>> Jun 18 14:47:28 pulpserver rq: await asyncio.gather(*futures) >>> Jun 18 14:47:28 pulpserver rq: File >>> "/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/pulpcore/plugin/stages/api.py", >>> line 43, in __call__ >>> Jun 18 14:47:28 pulpserver rq: await self.run() >>> Jun 18 14:47:28 pulpserver rq: File >>> "/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/pulpcore/plugin/stages/artifact_stages.py", >>> line 152, in run >>> Jun 18 14:47:28 pulpserver rq: pb.done += task.result() # download_count >>> Jun 18 14:47:28 pulpserver rq: File >>> "/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/pulpcore/plugin/stages/artifact_stages.py", >>> line 178, in _handle_content_unit >>> Jun 18 14:47:28 pulpserver rq: await >>> asyncio.gather(*downloaders_for_content) >>> Jun 18 14:47:28 pulpserver rq: File >>> "/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/pulpcore/plugin/stages/models.py", >>> line 88, in download >>> Jun 18 14:47:28 pulpserver rq: download_result = await >>> downloader.run(extra_data=self.extra_data) >>> Jun 18 14:47:28 pulpserver rq: File >>> "/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/pulpcore/download/base.py", >>> line 227, in run >>> Jun 18 14:47:28 pulpserver rq: return await >>> self._run(extra_data=extra_data) >>> Jun 18 14:47:28 pulpserver rq: File >>> "/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/pulp_rpm/app/downloaders.py", >>> line 87, in _run >>> Jun 18 14:47:28 pulpserver rq: async with self.session.get(url, >>> proxy=self.proxy, auth=self.auth) as response: >>> Jun 18 14:47:28 pulpserver rq: File >>> "/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/aiohttp/client.py", >>> line 1012, in __aenter__ >>> Jun 18 14:47:28 pulpserver rq: self._resp = await self._coro >>> Jun 18 14:47:28 pulpserver rq: File >>> "/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/aiohttp/client.py", >>> line 504, in _request >>> Jun 18 14:47:28 pulpserver rq: await resp.start(conn) >>> Jun 18 14:47:28 pulpserver rq: File >>> "/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/aiohttp/client_reqrep.py", >>> line 847, in start >>> Jun 18 14:47:28 pulpserver rq: message, payload = await >>> self._protocol.read() # type: ignore # noqa >>> Jun 18 14:47:28 pulpserver rq: File >>> "/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/aiohttp/streams.py", >>> line 591, in read >>> Jun 18 14:47:28 pulpserver rq: await self._waiter >>> Jun 18 14:47:28 pulpserver rq: >>> aiohttp.client_exceptions.ServerDisconnectedError >>> Jun 18 14:47:28 pulpserver rq: Traceback (most recent call last): >>> Jun 18 14:47:28 pulpserver rq: File >>> "/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/rq/worker.py", >>> line 886, in perform_job >>> Jun 18 14:47:28 pulpserver rq: rv = job.perform() >>> Jun 18 14:47:28 pulpserver rq: File >>> "/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/rq/job.py", line >>> 664, in perform >>> Jun 18 14:47:28 pulpserver rq: self._result = self._execute() >>> Jun 18 14:47:28 pulpserver rq: File >>> "/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/rq/job.py", line >>> 670, in _execute >>> Jun 18 14:47:28 pulpserver rq: return self.func(*self.args, >>> **self.kwargs) >>> Jun 18 14:47:28 pulpserver rq: File >>> "/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/pulp_rpm/app/tasks/synchronizing.py", >>> line 266, in synchronize >>> Jun 18 14:47:28 pulpserver rq: dv.create() >>> Jun 18 14:47:28 pulpserver rq: File >>> "/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/pulpcore/plugin/stages/declarative_version.py", >>> line 148, in create >>> Jun 18 14:47:28 pulpserver rq: loop.run_until_complete(pipeline) >>> Jun 18 14:47:28 pulpserver rq: File >>> "/opt/python/3.7.3/lib64/python3.7/asyncio/base_events.py", line 584, in >>> run_until_complete >>> Jun 18 14:47:28 pulpserver rq: return future.result() >>> Jun 18 14:47:28 pulpserver rq: File >>> "/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/pulpcore/plugin/stages/api.py", >>> line 225, in create_pipeline >>> Jun 18 14:47:28 pulpserver rq: await asyncio.gather(*futures) >>> Jun 18 14:47:28 pulpserver rq: File >>> "/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/pulpcore/plugin/stages/api.py", >>> line 43, in __call__ >>> Jun 18 14:47:28 pulpserver rq: await self.run() >>> Jun 18 14:47:28 pulpserver rq: File >>> "/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/pulpcore/plugin/stages/artifact_stages.py", >>> line 152, in run >>> Jun 18 14:47:28 pulpserver rq: pb.done += task.result() # download_count >>> Jun 18 14:47:28 pulpserver rq: File >>> "/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/pulpcore/plugin/stages/artifact_stages.py", >>> line 178, in _handle_content_unit >>> Jun 18 14:47:28 pulpserver rq: await >>> asyncio.gather(*downloaders_for_content) >>> Jun 18 14:47:28 pulpserver rq: File >>> "/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/pulpcore/plugin/stages/models.py", >>> line 88, in download >>> Jun 18 14:47:28 pulpserver rq: download_result = await >>> downloader.run(extra_data=self.extra_data) >>> Jun 18 14:47:28 pulpserver rq: File >>> "/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/pulpcore/download/base.py", >>> line 227, in run >>> Jun 18 14:47:28 pulpserver rq: return await >>> self._run(extra_data=extra_data) >>> Jun 18 14:47:28 pulpserver rq: File >>> "/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/pulp_rpm/app/downloaders.py", >>> line 87, in _run >>> Jun 18 14:47:28 pulpserver rq: async with self.session.get(url, >>> proxy=self.proxy, auth=self.auth) as response: >>> Jun 18 14:47:28 pulpserver rq: File >>> "/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/aiohttp/client.py", >>> line 1012, in __aenter__ >>> Jun 18 14:47:28 pulpserver rq: self._resp = await self._coro >>> Jun 18 14:47:28 pulpserver rq: File >>> "/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/aiohttp/client.py", >>> line 504, in _request >>> Jun 18 14:47:28 pulpserver rq: await resp.start(conn) >>> Jun 18 14:47:28 pulpserver rq: File >>> "/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/aiohttp/client_reqrep.py", >>> line 847, in start >>> Jun 18 14:47:28 pulpserver rq: message, payload = await >>> self._protocol.read() # type: ignore # noqa >>> Jun 18 14:47:28 pulpserver rq: File >>> "/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/aiohttp/streams.py", >>> line 591, in read >>> Jun 18 14:47:28 pulpserver rq: await self._waiter >>> Jun 18 14:47:28 pulpserver rq: >>> aiohttp.client_exceptions.ServerDisconnectedError >>> >>> >>> The aarch64 baseos repo completed successfully. >>> >>> >>> >>> >>> From: [email protected] At: 06/18/21 10:55:56 UTC-4:00 >>> To: [email protected] >>> Cc: Bin Li (BLOOMBERG/ 120 PARK ) <[email protected]>, >>> [email protected] >>> Subject: Re: [Pulp-list] pulp 3.7.3 aarch64 repo >>> >>> replace "but only with the errors" with "but only with the normal types >>> of errors" >>> >>> On Fri, Jun 18, 2021 at 10:52 AM Daniel Alley <[email protected]> wrote: >>> >>>> @Bin Li <[email protected]> >>>> Could you post the error that was logged? I think this traceback is >>>> missing some of the relevant details. >>>> >>>> I can tell you what I expect that problem *might* be, because we've >>>> been hearing about it a lot recently. Again, this is just a guess, without >>>> more information. >>>> >>>> We have found that some content delivery networks (including the Red >>>> Hat one) start returning seemingly irrelevant error messages when they get >>>> 'angry' about too much traffic coming from one source. Rather than >>>> returning HTTP 429 or some other standard response, sometimes it will just >>>> terminate the connection, or give you SSL errors, or return an incomplete >>>> payload. This might be some type of DDoS prevention / flood control >>>> mechanism, I'm not entirely sure. The mitigation is to either reduce the >>>> "download_concurrency" on your remote from 10 to 5, or using "on_demand" >>>> syncs rather than "immediate". Both of those are just to lessen the load on >>>> the CDN and make it less likely to start dropping connections. >>>> >>>> But the "proper" fix is being worked on for pulpcore 3.14. We want to >>>> automatically retry downloads that fail using a backoff algorithm to >>>> prevent the entire sync from failing. We were already doing this, but only >>>> with the errors, rather than the weird ones that seem to come up in >>>> practice. And we're making the number of retry attempts configurable. >>>> >>>> On Fri, Jun 18, 2021 at 9:04 AM Dennis Kliban <[email protected]> >>>> wrote: >>>> >>>>> I don't see anything special about these repos. Are you still >>>>> experiencing this problem? >>>>> >>>>> >>>>> On Wed, Jun 16, 2021 at 5:00 PM Bin Li (BLOOMBERG/ 120 PARK) < >>>>> [email protected]> wrote: >>>>> >>>>>> Tried to download aarch64 repo and got the errors below: >>>>>> >>>>>> "description": "", >>>>>> "traceback": " File >>>>>> \"/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/rq/worker.py\", >>>>>> line 886, in perform_job\n rv = job.perform()\n File >>>>>> \"/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/rq/job.py\", >>>>>> line 664, in perform\n self._result = self._execute()\n File >>>>>> \"/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/rq/job.py\", >>>>>> line 670, in _execute\n return self.func(*self.args, **self.kwargs)\n >>>>>> File >>>>>> \"/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/pulp_rpm/app/tasks/synchronizing.py\", >>>>>> line 266, in synchronize\n dv.create()\n File >>>>>> \"/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/pulpcore/plugin/stages/declarative_version.py\", >>>>>> line 148, in create\n loop.run_until_complete(pipeline)\n File >>>>>> \"/opt/python/3.7.3/lib64/python3.7/asyncio/base_events.py\", line 584, >>>>>> in >>>>>> run_until_complete\n return future.result()\n File >>>>>> \"/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/pulpcore/plugin/stages/api.py\", >>>>>> line 225, in create_pipeline\n await asyncio.gather(*futures)\n File >>>>>> \"/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/pulpcore/plugin/stages/api.py\", >>>>>> line 43, in __call__\n await self.run()\n File >>>>>> \"/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/pulpcore/plugin/stages/artifact_stages.py\", >>>>>> line 152, in run\n pb.done += task.result() # download_count\n File >>>>>> \"/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/pulpcore/plugin/stages/artifact_stages.py\", >>>>>> line 178, in _handle_content_unit\n await >>>>>> asyncio.gather(*downloaders_for_content)\n File >>>>>> \"/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/pulpcore/plugin/stages/models.py\", >>>>>> line 88, in download\n download_result = await >>>>>> downloader.run(extra_data=self.extra_data)\n File >>>>>> \"/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/pulpcore/download/base.py\", >>>>>> line 227, in run\n return await self._run(extra_data=extra_data)\n File >>>>>> \"/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/pulp_rpm/app/downloaders.py\", >>>>>> line 87, in _run\n async with self.session.get(url, proxy=self.proxy, >>>>>> auth=self.auth) as response:\n File >>>>>> \"/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/aiohttp/client.py\", >>>>>> line 1012, in __aenter__\n self._resp = await self._coro\n File >>>>>> \"/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/aiohttp/client.py\", >>>>>> line 504, in _request\n await resp.start(conn)\n File >>>>>> \"/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/aiohttp/client_reqrep.py\", >>>>>> line 847, in start\n message, payload = await self._protocol.read() # >>>>>> type: >>>>>> ignore # noqa\n File >>>>>> \"/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/aiohttp/streams.py\", >>>>>> line 591, in read\n await self._waiter\n >>>>>> >>>>>> The same error happened on both remote >>>>>> https://cdn.redhat.com/content/dist/rhel8/8.4/aarch64/baseos/os and >>>>>> https://cdn.redhat.com/content/dist/rhel8/8.4/aarch64/appstream/os >>>>>> >>>>>> Please advise. >>>>>> >>>>>> Thanks >>>>>> _______________________________________________ >>>>>> Pulp-list mailing list >>>>>> [email protected] >>>>>> https://listman.redhat.com/mailman/listinfo/pulp-list >>>>> >>>>> _______________________________________________ >>>>> Pulp-list mailing list >>>>> [email protected] >>>>> https://listman.redhat.com/mailman/listinfo/pulp-list >>>> >>>> >>> _______________________________________________ > Pulp-list mailing list > [email protected] > https://listman.redhat.com/mailman/listinfo/pulp-list
_______________________________________________ Pulp-list mailing list [email protected] https://listman.redhat.com/mailman/listinfo/pulp-list
