We haven’t had that behavior, we use gdal.Open on many large files inside docker and this isn’t an issue we’ve had. We haven’t specifically used EFS, we use NFS and an S3 fuse layer with docker and have had very good performance with the python bindings.
Note that we haven’t used privileged mode though, we aren’t using selinux, and we use the same uid/gid inside the docker container as owns the files. So that might be something to look at. Michael Smith Remote Sensing/GIS Center US Army Corps of Engineers > On Mar 18, 2018, at 8:09 AM, Ouwen Huang <ouwen...@gmail.com> wrote: > > I feel quite silly, but it seems that the `gdal.Open` was actually opening > the file from the docker mounted NFS, just very slowly. I left it to run for > 2 minutes or so and it returned. > > But this is weird behavior because on the host a `*Dataset` was returned > instantly, and within the docker container running `cp /mnt/efs/<my_geo.tiff> > <docker_local_dir` is also instant. The 2min latency seems to only occur > within a docker mounted network filesystem for the `gdal.Open` command. > > Some other findings, the `/mnt/efs/` directory i'm using has several files > (100k+). When I move the geotiff under a new directory within EFS > (`mnt/efs/test`), the `gdal.Open` command is instant within the docker > container. > > Is there a reason why `gdal.Open` is slow for a cluttered directory but unix > `cp` is not? > >> On Sun, Mar 18, 2018 at 2:51 AM Ouwen Huang <ouwen...@gmail.com> wrote: >> Hey Michael, >> >> Thanks for the quick response, I've been bashing my head against this for a >> while. >> Unfortunately restarting the docker daemon did not work for me. I can access >> the EFS from within docker using regular unix commands like `cp`, and `cat`, >> even with GDAL `VSIFOpenL` and `VSIFReadL`. However, `gdal.Open` will just >> hang without giving an error or returning. >> >> I'm using the following mount command from amazon EFS if this helps debug >> anything: >> `sudo mount -t nfs4 -o >> nfsvers=4.1,rsize=1048576,wsize=1048576,hard,timeo=600,retrans=2 >> <fs-id>.efs.us-west-2.amazonaws.com:/ /mnt/efs` >> >> I'm glad that you have a working version of this though, this had worked for >> me in the past but I did not keep track of my versioning. >> >> What host os version, docker version and gdal version are you using? >> >> - Ouwen >> >>> On Sun, Mar 18, 2018 at 2:31 AM Michael Smith >>> <michael.smith.e...@gmail.com> wrote: >>> Have you restarted the docker daemon after mounting the EFS drive to the >>> local OS? If not, do that and then map the volume with -v. That worked for >>> me. >>> >>> Michael Smith >>> Remote Sensing/GIS Center >>> US Army Corps of Engineers >>> >>> > On Mar 18, 2018, at 7:15 AM, Ouwen Huang <ouwen...@gmail.com> wrote: >>> > >>> > Hello, >>> > >>> > I am having some trouble loading a GeoTiff into GDAL from a docker >>> > container that is mounting a NFS (specifically amazon EFS). When mounting >>> > a volume with the `docker -v` command or mounting the NFS directly into >>> > docker under `--privileged` mode. The `gdal.Open` command hangs. >>> > >>> > I am able to run os level commands to open the GeoTiff of interest from >>> > within the docker container. In fact, if I copy from the NFS to a local >>> > directory, gdal works fine. >>> > >>> > I'm trying to understand what might be causing the `gdal.Open` command to >>> > hang. I've tested that `VSIFOpenL` and `VSIFReadL` work with the GeoTiff >>> > within the docker container so I'm very confused what is happening. >>> > >>> > I've tried different versions of GDAL (1.10 - 2.3) and differing versions >>> > of docker with no luck. >>> > >>> > Any help would be greatly appreciated! >>> > >>> > Best, >>> > Ouwen Huang >>> > _______________________________________________ >>> > gdal-dev mailing list >>> > gdal-dev@lists.osgeo.org >>> > https://lists.osgeo.org/mailman/listinfo/gdal-dev >> -- >> Duke MD/PhD Candidate >> Ycombinator Fellow >> cell: 210.860.3915 >> http://ouwen.io/ > -- > Duke MD/PhD Candidate > Ycombinator Fellow > cell: 210.860.3915 > http://ouwen.io/
_______________________________________________ gdal-dev mailing list gdal-dev@lists.osgeo.org https://lists.osgeo.org/mailman/listinfo/gdal-dev
