On Sat, 10 Dec 2022 12:30:40 -0500 Mansour Al Akeel <mansour.alak...@gmail.com> wrote:
> I am using Openrc > > This was my initial /etc/conf.d/docker > DOCKER_OPTS="--storage-driver overlay2 --data-root /srv/var/lib/docker" > > when I try: > > $ docker pull hello-world > > Error response from daemon: Get "https://registry-1.docker.io/v2/": > net/http: request canceled while waiting for connection (Client.Timeout > exceeded while awaiting headers) > > Trying to troubleshoot: > > localhost /home/mansour # tail -n 20 /var/log/docker.log > time="2022-12-10T12:17:03.473550705-05:00" level=info msg="scheme \"unix\" > not registered, fallback to default scheme" module=grpc > time="2022-12-10T12:17:03.473566413-05:00" level=info > msg="ccResolverWrapper: sending update to cc: > {[{unix:///run/containerd/containerd.sock <nil> 0 <nil>}] <nil> <nil>}" > module=grpc > time="2022-12-10T12:17:03.473573787-05:00" level=info msg="ClientConn > switching balancer to \"pick_first\"" module=grpc > time="2022-12-10T12:17:03.474530993-05:00" level=info msg="parsed scheme: > \"unix\"" module=grpc > time="2022-12-10T12:17:03.474545549-05:00" level=info msg="scheme \"unix\" > not registered, fallback to default scheme" module=grpc > time="2022-12-10T12:17:03.474563752-05:00" level=info > msg="ccResolverWrapper: sending update to cc: > {[{unix:///run/containerd/containerd.sock <nil> 0 <nil>}] <nil> <nil>}" > module=grpc > time="2022-12-10T12:17:03.474571186-05:00" level=info msg="ClientConn > switching balancer to \"pick_first\"" module=grpc > time="2022-12-10T12:17:03.478908716-05:00" level=warning msg="Your kernel > does not support cgroup blkio weight" > time="2022-12-10T12:17:03.478927115-05:00" level=warning msg="Your kernel > does not support cgroup blkio weight_device" > time="2022-12-10T12:17:03.479037897-05:00" level=info msg="Loading > containers: start." > time="2022-12-10T12:17:03.495743563-05:00" level=info msg="failed to read > ipv6 net.ipv6.conf.<bridge>.accept_ra" bridge=docker0 > syspath=/proc/sys/net/ipv6/conf/docker0/accept_ra > time="2022-12-10T12:17:03.518761943-05:00" level=info msg="Default bridge > (docker0) is assigned with an IP address 172.17.0.0/16. Daemon option --bip > can be used to set a preferred IP address" > time="2022-12-10T12:17:03.518886881-05:00" level=info msg="failed to read > ipv6 net.ipv6.conf.<bridge>.accept_ra" bridge=docker0 > syspath=/proc/sys/net/ipv6/conf/docker0/accept_ra > time="2022-12-10T12:17:03.534616741-05:00" level=info msg="Loading > containers: done." > time="2022-12-10T12:17:03.541080189-05:00" level=info msg="Docker daemon" > commit=a89b84221c graphdriver(s)=overlay2 version=20.10.17 > time="2022-12-10T12:17:03.541122352-05:00" level=info msg="Daemon has > completed initialization" > time="2022-12-10T12:17:03.549888103-05:00" level=info msg="API listen on > /var/run/docker.sock" > time="2022-12-10T12:17:27.025622231-05:00" level=warning msg="Error getting > v2 registry: Get \"https://registry-1.docker.io/v2/\": net/http: request > canceled while waiting for connection (Client.Timeout exceeded while > awaiting headers)" > time="2022-12-10T12:17:27.025667054-05:00" level=info msg="Attempting next > endpoint for pull after error: Get \"https://registry-1.docker.io/v2/\": > net/http: request canceled while waiting for connection (Client.Timeout > exceeded while awaiting headers)" > time="2022-12-10T12:17:27.026851821-05:00" level=error msg="Handler for > POST /v1.41/images/create returned error: Get \" > https://registry-1.docker.io/v2/\": net/http: request canceled while > waiting for connection (Client.Timeout exceeded while awaiting headers)" > > time="2022-12-10T12:15:42.036053086-05:00" level=info msg="loading plugin > \"io.containerd.internal.v1.tracing\"..." type=io.containerd.internal.v1 > time="2022-12-10T12:15:42.036068412-05:00" level=error msg="failed to > initialize a tracing processor \"otlp\"" error="no OpenTelemetry endpoint: > skip plugin" > time="2022-12-10T12:15:42.036100189-05:00" level=info msg="loading plugin > \"io.containerd.grpc.v1.cri\"..." type=io.containerd.grpc.v1 > time="2022-12-10T12:15:42.036390695-05:00" level=info msg="Start cri plugin > with config {PluginConfig:{ContainerdConfig:{Snapshotter:overlayfs > DefaultRuntimeName:runc DefaultRuntime:{Type: Path: Engine: > PodAnnotations:[] ContainerAnnotations:[] Root: Options:map[] > PrivilegedWithoutHostDevices:false BaseRuntimeSpec: NetworkPluginConfDir: > NetworkPluginMaxConfNum:0} UntrustedWorkloadRuntime:{Type: Path: Engine: > PodAnnotations:[] ContainerAnnotations:[] Root: Options:map[] > PrivilegedWithoutHostDevices:false BaseRuntimeSpec: NetworkPluginConfDir: > NetworkPluginMaxConfNum:0} Runtimes:map[runc:{Type:io.containerd.runc.v2 > Path: Engine: PodAnnotations:[] ContainerAnnotations:[] Root: > Options:map[BinaryName: CriuImagePath: CriuPath: CriuWorkPath: IoGid:0 > IoUid:0 NoNewKeyring:false NoPivotRoot:false Root: ShimCgroup: > SystemdCgroup:false] PrivilegedWithoutHostDevices:false BaseRuntimeSpec: > NetworkPluginConfDir: NetworkPluginMaxConfNum:0}] NoPivot:false > DisableSnapshotAnnotations:true DiscardUnpackedLayers:false > IgnoreRdtNotEnabledErrors:false} > CniConfig:{NetworkPluginBinDir:/opt/cni/bin > NetworkPluginConfDir:/etc/cni/net.d NetworkPluginMaxConfNum:1 > NetworkPluginConfTemplate: IPPreference:} Registry:{ConfigPath: > Mirrors:map[] Configs:map[] Auths:map[] Headers:map[]} > ImageDecryption:{KeyModel:node} DisableTCPService:true > StreamServerAddress:127.0.0.1 StreamServerPort:0 StreamIdleTimeout:4h0m0s > EnableSelinux:false SelinuxCategoryRange:1024 SandboxImage: > k8s.gcr.io/pause:3.6 StatsCollectPeriod:10 SystemdCgroup:false > EnableTLSStreaming:false X509KeyPairStreaming:{TLSCertFile: TLSKeyFile:} > MaxContainerLogLineSize:16384 DisableCgroup:false DisableApparmor:false > RestrictOOMScoreAdj:false MaxConcurrentDownloads:3 DisableProcMount:false > UnsetSeccompProfile: TolerateMissingHugetlbController:true > DisableHugetlbController:true DeviceOwnershipFromSecurityContext:false > IgnoreImageDefinedVolumes:false NetNSMountsUnderStateDir:false > EnableUnprivilegedPorts:false EnableUnprivilegedICMP:false} > ContainerdRootDir:/var/lib/containerd > ContainerdEndpoint:/run/containerd/containerd.sock > RootDir:/var/lib/containerd/io.containerd.grpc.v1.cri > StateDir:/run/containerd/io.containerd.grpc.v1.cri}" > time="2022-12-10T12:15:42.036530498-05:00" level=info msg="Connect > containerd service" > time="2022-12-10T12:15:42.036627681-05:00" level=info msg="Get image > filesystem path > \"/var/lib/containerd/io.containerd.snapshotter.v1.overlayfs\"" > time="2022-12-10T12:15:42.036896077-05:00" level=error msg="failed to load > cni during init, please check CRI plugin status before setting up network > for pods" error="cni config load failed: no network config found in > /etc/cni/net.d: cni plugin not initialized: failed to load cni config" > time="2022-12-10T12:15:42.036986623-05:00" level=info msg="Start > subscribing containerd event" > time="2022-12-10T12:15:42.037028034-05:00" level=info msg="Start recovering > state" > time="2022-12-10T12:15:42.037035874-05:00" level=info msg=serving... > address=/run/containerd/containerd.sock.ttrpc > time="2022-12-10T12:15:42.037068181-05:00" level=info msg=serving... > address=/run/containerd/containerd.sock > time="2022-12-10T12:15:42.037085346-05:00" level=info msg="containerd > successfully booted in 0.015889s" > time="2022-12-10T12:15:42.037072303-05:00" level=info msg="Start event > monitor" > time="2022-12-10T12:15:42.037107987-05:00" level=info msg="Start snapshots > syncer" > time="2022-12-10T12:15:42.037119126-05:00" level=info msg="Start cni > network conf syncer for default" > time="2022-12-10T12:15:42.037127266-05:00" level=info msg="Start streaming > server" > > > Looking at resolv.conf > > localhost /home/mansour # cat /etc/resolv.conf > # Generated by dhcpcd from wlp3s0.dhcp > # /etc/resolv.conf.head can replace this line > domain Home > nameserver 192.168.1.1 > # /etc/resolv.conf.tail can replace this line > > So, I added the following to my /etc/dhcpcd.conf > static domain_name_servers=8.8.8.8 8.8.4.4 192.168.1.1 > > And restarted dhcpcd: > > localhost /home/mansour # cat /etc/resolv.conf > # Generated by dhcpcd from wlp3s0.dhcp > # /etc/resolv.conf.head can replace this line > domain Home > nameserver 8.8.8.8 > nameserver 8.8.4.4 > nameserver 192.168.1.1 > # /etc/resolv.conf.tail can replace this line > > > docker.log still shows the same error: > time="2022-12-10T12:28:16.519126445-05:00" level=warning msg="Error getting > v2 registry: Get \"https://registry-1.docker.io/v2/\": net/http: request > canceled while waiting for connection (Client.Timeout exceeded while > awaiting headers)" > time="2022-12-10T12:28:16.519173944-05:00" level=info msg="Attempting next > endpoint for pull after error: Get \"https://registry-1.docker.io/v2/\": > net/http: request canceled while waiting for connection (Client.Timeout > exceeded while awaiting headers)" > time="2022-12-10T12:28:16.520489790-05:00" level=error msg="Handler for > POST /v1.41/images/create returned error: Get \" > https://registry-1.docker.io/v2/\": net/http: request canceled while > waiting for connection (Client.Timeout exceeded while awaiting headers)" > > I am not able to login with "docker login" > > Where do I look further for additional troubleshooting hints? Google didn't > help a lot. > > Thank you. Hello, my first step would be checking if there is some inherent network problem reaching registry-1.docker.io/v2, e.g. via curl -v https://registry-1.docker.io/v2/ If that runs successfully, you could try if there is an MTU problem. I remember that I had to set MTU to 1450 in some virtualized environment, otherwise docker would not run correctly, i.e. I had in /etc/conf.d/docer DOCKER_OPTS="--mtu=1450" Hope that gives you an idea what to look for. If both does not help, there is wireshark, to watch your network, very low-level, but might give some hints. Cheers Andreas
