[mapserver-users] Potential bug when using Layer-Tile-Index where indexed files are GeoTiff files

[Sommer, Ashley (L, Dutton Park)]

Hi Everyone,
I'm trying to get WCS service with TIME parameter support working on latest 
Mapserver.
My data is a timeseries of large geotiff files, in a directory, with the date 
encoded in the filename.
Eg:
 - /datasets/mydataset/v1/layer1/raster_2020-08-01.tif
 - /datasets/mydataset/v1/layer1/raster_2020-08-02.tif
 - etc.
I'm had it working on the WMS side of things, simply using Runtime 
Substitution, and embedding the TIME parameter into the DATA directive.
eg:
  DATA "/datasets/mydataset/v1/layer1/raster_%time%.tif"
That worked for WMS, but not for WCS. When Mapserver's WCS service encounters a 
TIME parameter, it assumes I'm using a tile index. If there is no tile index, 
it errors out.

So I create a tile index for this dataset. Using gdaltindex for a record of 
"location" for each raster, then adding a temporal attribute to the dbf file, 
and assigning dates to each of the records.
I then tried to use that in place of DATA:
LAYER
  NAME "mylayer"
  TILEINDEX "/datasets/mydataset/v1/layer1/index.shp"
  TILEITEM "location"
END
Unfortunately, that doesn't work, because Mapserver now requires the use of 
new-style Layer-Tile-Index directives in order for WCS to work properly.
So I changed it to:
LAYER
  NAME "MyIndex"
  TILEINDEX "/datasets/mydataset/v1/layer1/index.shp"
  TILEITEM "location"
END
LAYER
  TILEINDEX "MyIndex"
  TILEITEM "location"
END

This now works to some extent, but then I get an error:
msTiledSHPTryOpen(): Unable to access file. Unable to open shapefile 
'raster_2020-08-02.tif' for layer 'MyIndex' ... fatal error. msShapefileOpen(): 
Unable to access file. (/tmp/raster_2020-08-02.tif) msShapefileOpen(): Unable 
to access file. (/datasets/mydataset/v1/layer1/raster_2020-08-01.tif)

I believe I have tracked down why this is happening, and I believe it is a bug.
1) Mapserver _can_ open the .shp and .dbf file at the location. It must be able 
to to get the file location from the dbf index.
2) msTiledSHPTryOpen() is a helper fn that runs _after_ opening a tile index, 
to test if we can open the first-indexed file (to template some parts of the 
layer).
See comment here: 
https://github.com/MapServer/MapServer/blob/0d156a72893b1df712cb49c1b3ddaeed5365a68b/mapshape.c#L2028
See also msTileSHPTryOpen fn here: 
https://github.com/MapServer/MapServer/blob/0d156a72893b1df712cb49c1b3ddaeed5365a68b/mapshape.c#L1868
3) So msTiledSHPTryOpen() tries to open the first-indexed file: 
"/datasets/mydataset/v1/layer1/raster_2020-08-02.tif" but returns an error, 
even though it _can_ open that file.
The problem is, it is using msShapefileOpen() to do this, which in-turn uses 
msSHPOpen() to open the tif file:
See msShapefileOpen fn: 
https://github.com/MapServer/MapServer/blob/0d156a72893b1df712cb49c1b3ddaeed5365a68b/mapshape.c#L1665
See opener: 
https://github.com/MapServer/MapServer/blob/0d156a72893b1df712cb49c1b3ddaeed5365a68b/mapshape.c#L1686
4) msSHPOpen() assumes whatever file its opening will have a .shp file and a 
.shx file.
In this case, it's trying to open a .tif file. The function tries to open a 
non-existent /datasets/mydataset/v1/layer1/raster_2020-08-02.shp file.
See the implementation here: 
https://github.com/MapServer/MapServer/blob/0d156a72893b1df712cb49c1b3ddaeed5365a68b/mapshape.c#L194
And where it opens a file here: 
https://github.com/MapServer/MapServer/blob/0d156a72893b1df712cb49c1b3ddaeed5365a68b/mapshape.c#L252

So I believe it is a bug to use msShapefileOpen() to test if a tileindex can 
open a file at its first-indexed location, because that file may not be a .shp 
file.

I might be able to put in a PR to fix the bug, but I don't know the codebase 
well enough to not potentially introduce regressions.
Let me know what you think?

- Ashley Sommer
___
MapServer-users mailing list
MapServer-users@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/mapserver-users

[mapserver-users] [Feature Request] Ability to extract capture group contents from regex validation

[Sommer, Ashley (L, Dutton Park)]

Hi All,
This is slightly related to my previous thread today, but more general.

I want to gauge interested in this idea.
I want to be able to use Runtime Substitution mechanisms to do something like 
this:

MAP
  ...
  VALIDATION
"datetime" "^([0-9]{4})[-_]([0-9]{2})[-_]([0-9]{2})$" # <- 3 capture groups
  END

  LAYER
DATA "/mnt/dataset/%datetime_1%/%datetime_2%/%datetime_3%.tif"
...
  END
  ...
END

Where the query with =2018-01-29 is split into parts using capture 
groups 1, 2, and 3 from the validation regex, and stored as new runtime 
substitution variables: %datetime_1%, %datetime_2%, and %datetime_3%.
Then using these values, substitute them into the DATA string (or any other 
valid substitution point).
Substituted:
DATA "/mnt/dataset/2018/01/29.tif"

The regex mechanism to validate the incoming parameter value is already in 
place, and it seems like just one more little step is needed to save the 
captured groups into new runtime substitution variables.

- Ashley Sommer
___
mapserver-users mailing list
mapserver-users@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/mapserver-users

[mapserver-users] Extracting Year, Month, Day from WMS time query

[Sommer, Ashley (L, Dutton Park)]

Hi All,

I'm trying to get a very specific configuration working in my mapfile.

I have geotiff files in a directory structure that looks like:
http://example.org/v1/AUTH_xyz/my-dataset//_MM_DD/product_name__MM_DD.vrt
Where  is a year (eg 2018) and _MM_DD is a date (eg 2018_08_01)

EG:
DATA 
"/vsicurl/http://example.org/v1/AUTH_xyz/my-dataset/2018/2018_08_01/product_name_2018_08_01.vrt;

I'm trying to get this working with a dynamic mapfile, with runtime 
substitution, and ideally, I'd also like it to work with WMS Time queries.

I've found that the WMS TIME query parameter doesn't automatically work with 
runtime substitution. It can be used as an attribute [TIME] in a FILTER, or in 
an EXPRESSION, but nowhere else.
To work around that, I've set up a Runtime Substitution variable also named 
time:

VALIDATION
"time" "^[0-9]{4}-[0-9]{2}-[0-9]{2}$"
END

That works, it captures the right variable and allows me to use the 
WMS-provided TIME param in the DATA directive like:
LAYER
  DATA 
"/vsicurl/http://example.org/v1/AUTH_xyz/my-dataset/2018/%time%/product_name_%time%.vrt;

But that doesn't quite match the naming scheme of the dataset.
In order for this to work properly, I need to be able to extract the year, 
month and day portions of the datestring, and use them individually.
Like:
DATA 
"/vsicurl/http://example.org/v1/AUTH_xyz/my-dataset/%year%/%year%_%month%_%day%/product_name_%year%_%month%_%day%.vrt;

I've read the documentation around WMS_Time and Runtime Substitution, but I 
can't quite work out if this is even possible.
Note, I understand I could probably do this at the apache level, by 
transforming the query with a rewrite eg:
"=2018-08-01" -> "=2018-08-01=2018=08=01"
But I'd prefer to do it at the Mapfile level.

- Ashley Sommer

___
mapserver-users mailing list
mapserver-users@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/mapserver-users