Hi Anthony, Can you please share your docker compose file? The file in my repo works fine for me https://github.com/anilkulkarni87/airflow-docker
It was getting stuck owing to the user permission and was resolved. On Tue, Aug 17, 2021 at 5:58 AM Anthony Joyce < [email protected]> wrote: > Thanks Jarek. > > The commands below can grab the IP from the hostname, but nc -zvvn > definitely hangs when trying to connect to my host IP on the Postgres port. > This helps me narrow it down a bit. Thank you for that suggestion! > > Anthony > > On Aug 16, 2021, at 1:56 PM, Jarek Potiuk <[email protected]> wrote: > > Another thing to try - just run airflow image with `bash` parameter > (`docker run -it <image> bash`) and see if you can connect to your > host/port (there is `nc` tool installed in the production image). I believe > when you try it you will hang on trying to connect to your database. > > I think there is one reason why it could fail - in case your host name is > wrongly parsed from the URL (because it contains restricted characters) - > this could happen. > > This is what the connectivity check does (this is where it seems to hang) > - copied it with explanation on what is happening (gethosbyname will also > work for IP addresses BTW). You might try to reproduce this. > > # Checks if it is possible to connect to the host using netcat. > # > # We want to avoid misleading messages and perform only forward lookup > of the service IP address. > # Netcat when run without -n performs both forward and reverse lookup > and fails if the reverse > # lookup name does not match the original name even if the host is > reachable via IP. This happens > # randomly with docker-compose in GitHub Actions. > # Since we are not using reverse lookup elsewhere, we can perform > forward lookup in python > # And use the IP in NC and add '-n' switch to disable any DNS use. > # Even if this message might be harmless, it might hide the real > reason for the problem > # Which is the long time needed to start some services, seeing this > message might be totally misleading > # when you try to analyse the problem, that's why it's best to avoid > it, > ip=$(python -c "import socket; print(socket.gethostbyname('${host}'))") > nc -zvvn "${ip}" "${port}" > > > J. > > > On Mon, Aug 16, 2021 at 7:31 PM Anthony Joyce < > [email protected]> wrote: > >> Thanks for the suggestions. Will do. And yes, I am using the official >> apache-airflow image. >> >> Anthony >> >> On Aug 16, 2021, at 1:29 PM, Daniel Standish <[email protected]> >> wrote: >> >> Let me offer a suggestion. >> >> I think it's helpful to simplify things and isolate components. To that >> end what you could do is scrap docker for a moment. Create a >> python virtualenv in your host. Install airflow into this virtualenv. Set >> AIRFLOW__CORE__SQL_ALCHEMY_CONN as appropriate and run airflow in the >> virtualenv. Then you'll be able to confirm the issue is docker and not >> some other connectivity issue. >> >> Next move on to docker, perhaps without compose. One thing you could do >> is get a terminal in a running container and try to connect to this server >> e.g. using telnet or psql (you might have to install these). Then you'll >> figure out what connection string will actually work and it's just a matter >> of setting up the env vars correctly for airflow to connect in the right >> way. >> >> Also, are you using the official apache-airflow image? If not, I would >> recommend that. >> >> >> >> >> On Mon, Aug 16, 2021 at 10:22 AM Anthony Joyce < >> [email protected]> wrote: >> >>> Hi Daniel, >>> >>> Thanks for the suggestion! I’m new at Docker so apologies if some of the >>> configs are incorrect. Still learning. >>> >>> I removed service Postgres since I am using my local db. Still stuck on >>> *airflow-init*. There are no log entries that explain what it is doing >>> or what it is waiting for, but obviously *airflow-init *is waiting for >>> something. I’ve since restarted docker service, flushed iptables chains >>> relating to Docker, restarted again. Still stuck. >>> >>> Anthony >>> >>> On Aug 16, 2021, at 1:11 PM, Daniel Standish <[email protected]> >>> wrote: >>> >>> If you are using an already existing postgres, then why are you running >>> postgres service in compose? >>> >>> If that's *not* the database you are connecting to, then you should >>> remove it from your compose file. (conversely if you *are* trying to >>> use compose postgres, then we might have some configuration suggestions to >>> offer) >>> --- >>> Separately, on mac compose, for the container to access host i think you >>> might need to add svc.localhost or something like that... there are SO >>> posts on this topic eg >>> https://stackoverflow.com/questions/24319662/from-inside-of-a-docker-container-how-do-i-connect-to-the-localhost-of-the-mac >>> >>> >>> >> > > -- > +48 660 796 129 > > > -- Cheers, Anil Kulkarni https://anilkulkarni.com/
