Thanks Even, Disabling reading the directory is another work around for my use case as well: (GDAL_DISABLE_READDIR_ON_OPEN=TRUE)
On Wed, Jun 21, 2017 at 5:02 AM, Even Rouault <even.roua...@spatialys.com> wrote: > Matt, > > > > > My actual problem is a bit more specific then being unable to open S3 > files > > > after upload. The actual problem is that within the same Python session, > I > > > can open a file off S3 with the vsis3 driver, but then if I upload a new > > > file that previously did not exist (using boto3), gdal does not see it > as a > > > valid file. > > > > Yes I'm aware of that issue. There's indeed metadata (file size & date, > directory listing) and data (chunks of files) cached by /vsicurl/ and > related file systems like /vsis3/ . /vsicurl/ was designed at a time where > web resources didn't change that much and it was unlikely during a same > GDAL session to see changes, but with cloud offerings, this is no longer > the case. > > > > A few weeks ago I've added in trunk a CPL_VSIL_CURL_NON_CACHED config > option that can be set to disable caching on a file or set of files. > > See https://trac.osgeo.org/gdal/wiki/ConfigOptions#CPL_VSIL_ > CURL_NON_CACHED > > > > So in your example, if you set > > CPL_VSIL_CURL_NON_CACHED=/vsis3/put_here_the_bucket_name , that will work. > > > > I've also just added per https://trac.osgeo.org/gdal/ticket/6937 a new > function VSICurlClearCache() function (bound to SWIG as > gdal.VSICurlClearCache()). So if you add gdal.VSICurlClearCache() just > after the s3.meta.client.upload_file() call, that will work too. > > > > Both mechanisms are complementary. > > > > CPL_VSIL_CURL_NON_CACHED is useful in scenarios where you don't know when > the server content can change (some other processes or machines do that > behind your back). Its advantage is that it doesn't require modification of > code (it was designed for MapServer use case typically). The drawback of it > is that you loose all caching when a same file is opened, close, opened, > closed, ... several times during the process. > > > > VSICurlClearCache() will give you more control if you master when uploads > happen. > > > > I've also backported VSICurlClearCache() to 2.2 branch. > > > > As far as VSI_CACHE=TRUE is concerned, its scope of caching is restricted > to a same VSI file handle instance. Can be useful if the global 16 MB > vsicurl cache isn't big enough for very large files. > > > > Even > > > > -- > > Spatialys - Geospatial professional services > > http://www.spatialys.com >
_______________________________________________ gdal-dev mailing list gdal-dev@lists.osgeo.org https://lists.osgeo.org/mailman/listinfo/gdal-dev