Re: [Viking-devel] A shared tile cache
Robert Norris writes: > For the tile servers currently offered on the main osm.org website I would > name them: > > MapQuest_Open > OSM_Cycle > OSM_Humanitarian > OSM_Mapnik > OSM_Transport sounds fine. > Of course one could use spaces instead of underscores to make things > interesting... I hope you're joking. My view is that spaces are highly unfriendly to using command-line tools, and IMHO have no place in file or directory names on a unix/posix system. I do know how to write something like find . -type f -mtime +7 | xargs ls -ltr so that it works with spaces. But there's no reason to make people go through that. pgpWfumx9xUNX.pgp Description: PGP signature -- ___ Viking-devel mailing list Viking-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/viking-devel Viking home page: http://viking.sf.net/
Re: [Viking-devel] A shared tile cache
>> I'd like any of the developers to read this draft about a common shared >> cache that I'd like to implement in JOSM: >> >> https://josm.openstreetmap.de/wiki/SharedTileCache >Generally this seems sensible. A few comments: >maybe the min free space in the fs, or both. Min free space in the FS is a good idea. >There's an "OSM", but really this isn't about OSM. It's about TMS. >Bing isn't about OSM (even though they graciously let us edit based on >it). >So I'd use "TMS" rather than "osm/tiles". Agreed. There are various other free/Open/OpenData sources that offer Tiles eg: BlueMarble, CalTopo, OS NPE, Other various historical maps. I don't understand the limit namespace 'squatting' request by the Marble developer. Could the reasoning be explained somewhere? Instead I would list the specific short names used so that applications can *create* and use the cache even without adhering to the cache.ini specification. For the tile servers currently offered on the main osm.org website I would name them: MapQuest_Open OSM_Cycle OSM_Humanitarian OSM_Mapnik OSM_Transport Of course one could use spaces instead of underscores to make things interesting... >I think zoom/x/y is good. I really don't like the way viking does it >now, which I can never easily remember. I totally agree! >A conversion program to read an old cache and move it to the new format >would be nice. But for me not a big deal Note there is cache conversion tool (in python) in the source repository tools directory - 'viking-cache-mbtile.py' ATM it only converts a Viking cache into an MBTiles file. Then you can use the mbutil script (https://github.com/mapbox/mbutil) to convert the MBTiles files into a TMS layout. I suspect I will expand this tool to convert directly from a Viking cache layout to a TMS layout. Renaming it 'viking-cache.py' might be an idea... I further notice these 'tools' weren't distributed in build output, which I fixed in the last couple of days. >A big issue is caching lifetime, in terms of when to recheck the server With the age=X It might be an idea to the have an absolute minimum tile age value - such as at least 24 hours, so any lower values in the age should be ignored and 24 hours would be used (86400 seconds). -- ___ Viking-devel mailing list Viking-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/viking-devel Viking home page: http://viking.sf.net/
Re: [Viking-devel] A shared tile cache
Yuri D'Elia writes: > I'd like any of the developers to read this draft about a common shared > cache that I'd like to implement in JOSM: > > https://josm.openstreetmap.de/wiki/SharedTileCache Generally this seems sensible. A few comments: There should be a way to configure the max size used by the cache, or maybe the min free space in the fs, or both. There's an "OSM", but really this isn't about OSM. It's about TMS. Bing isn't about OSM (even though they graciously let us edit based on it). So I'd use "TMS" rather than "osm/tiles". I think zoom/x/y is good. I really don't like the way viking does it now, which I can never easily remember. A conversion program to read an old cache and move it to the new format would be nice. But for me not a big deal A big issue is caching lifetime, in terms of when to recheck the server vs just accessing without checking. Your description of "minimum tile age" seems to be this, but really the question is: what's the parameter that says "if the difference between now and the time this tile was last verified to be valid is < X, then just use it". what is the mod date on the tile file? - Origin server's generation timestamp (I like this)? - When Fetched the first? - When validated? when a tile is old enough to check, and the origin server says it's unmodified, how is this recorded? Is there a second file? In the .ini for the tile? sqlite? Your point about storing view count (and last view time) is a good one. Probably LRU pruning would be good, maybe in a background process. I don't mean to sound negative - my comments are pretty minor. pgpNnXTwZMMo1.pgp Description: PGP signature -- ___ Viking-devel mailing list Viking-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/viking-devel Viking home page: http://viking.sf.net/
Re: [Viking-devel] A shared tile cache
On 03/24/2014 03:01 AM, Robert Norris wrote: > Location on Windows: Include Windows 8.1 (I think it follows the Win7 > layout) and 'C:\UsersPath\UserName' is better as %UserProfile% > (indeed see http://en.wikipedia.org/wiki/Home_directory for other > OSs) Thanks, updated. > There are many other applications that could use such a shared cache > layout (they must have there own schemes ATM, even if they at least > use the standard Z/X/Y.ext structure somewhere. Immediately can I > think of Marble, Navit, GPS Prune and libchamplain which would all > benefit from using this scheme. I would be good to contact as many > interested parties as possible. I'll try. > Should there be way to say never cache the tiles? e.g. size=-1 Agree, added. > May be specify what is 'short' for the directory name, e.g. up to 20 > characters. No opinion here, but I could see how it could be helpful for defining ID sizes. Added. > I have 9.0Gb in my viking-maps cache. If you've been using any such > map program for a few years you may have got quite a lot of data. > Transforming between old and new layouts *must be seamless*. Hence I > won't be applying your patch, but thank you for it - indeed for a > new user as yourself - it will be easier to keep that cache layout. Honestly, I migrated mine. I had roughly 2gb of data. I invoked "rename" and mv with an arithmetic expression a couple of times. As I wrote in the bug report, my inclination would be to provide a script to interested users, but I definitely see how on Windows and Mac this is not a "nice" the way to go. > I don't really understand the structure behind Viking's original > cache layout. I guess the omission of the file extension was just > because the original author didn't think to add it! The top level dir > names all seem to end in z0, I believe was in place for UTM > positioning where z is a zone, as there are repeating UTM zones > across the earth with Northings & Eastings within them. However I > think UTM things get translated into Mercator Lat/Longs so the zone > part is superfluous anyway. I see, I was wondering that myself. > I'm at least glad you're thinking of simple .ini files rather than > .json files that becoming all the rage by some people ;) I'm not fond of the "ini" extension, since I would *not* advocate a real INI parser, but just a simple k=v text file, without comments even. I wasn't sure about using ".conf" or not, though. The idea of using the "tile.ext.ini" (.ini appended as opposed to a [tile.ini]) comes also for simplicity when generating filenames. > Separately, could you explain how you think will do pruning in JOSM > in more detail? It can be quite complicated: e.g. How often and when > it occurs, how will it find out the size of the cache (and how long > that will take), does it delete highest zoom, lowest zoom or by > 'however the files are on disk' first? Do you worry about creating > new files if the size is already reached? How any feedback to the > user will work, if it is required at all? If you do switch to a cache > new layout - are you going to auto delete the old layout? You would be surprised to know that the second reason JOSM becomes slower is that it *doesn't* do any pruning. There's an option to prune the cache, but AFAIK it's broken. I think that maybe this is the reason of using /tmp/ on linux? But anyway. Cache pruning is complex, and there's no unique way to do it, so this is the reason a deliberately specify how to perform it. In JOSM, I would perform pruning only on program exit. This is due to the fact that assessing the cache size requires a full tree traversal, and will not be fast enough for interactive work. Josm has a list of rects that it keeps for "downloaded" areas, and I was planning to do something very simple at first: - get cache size / file list - sort by size/mtime/zoom respectively - while cache size > limit: - pick first tile in the sorted list that's not visible and remove it - if cache has not shrunk, disable visibility check - if cache has not shrunk, break Of course, this is very crude, it doesn't take into account how often areas a "looked at" at all, which would actually be much better. This is where those "metadata" files come in handy. JOSM already stores the ETag (for servers that don't support if-modified-since). Though I don't believe there are any... this is actually a perfect place to store the tile "hit counts" to perform usage-based pruning. -- Learn Graph Databases - Download FREE O'Reilly Book "Graph Databases" is the definitive new guide to graph databases and their applications. Written by three acclaimed leaders in the field, this first edition is now available. Download your free book today! http://p.sf.net/sfu/13534_NeoTech ___ Viking-devel mailing list Viking-devel@lists.sourceforge.net https://lists.sourceforge
Re: [Viking-devel] A shared tile cache
On 03/24/2014 03:01 AM, Robert Norris wrote: > Location on Windows: Include Windows 8.1 (I think it follows the Win7 > layout) and 'C:\UsersPath\UserName' is better as %UserProfile% > (indeed see http://en.wikipedia.org/wiki/Home_directory for other > OSs) Thanks, updated. > There are many other applications that could use such a shared cache > layout (they must have there own schemes ATM, even if they at least > use the standard Z/X/Y.ext structure somewhere. Immediately can I > think of Marble, Navit, GPS Prune and libchamplain which would all > benefit from using this scheme. I would be good to contact as many > interested parties as possible. I'll try. > Should there be way to say never cache the tiles? e.g. size=-1 Agree, added. > May be specify what is 'short' for the directory name, e.g. up to 20 > characters. No opinion here, but I could see how it could be helpful for defining ID sizes. Added. > I have 9.0Gb in my viking-maps cache. If you've been using any such > map program for a few years you may have got quite a lot of data. > Transforming between old and new layouts *must be seamless*. Hence I > won't be applying your patch, but thank you for it - indeed for a > new user as yourself - it will be easier to keep that cache layout. Honestly, I migrated mine. I had roughly 2gb of data. I invoked "rename" and mv with an arithmetic expression a couple of times. As I wrote in the bug report, my inclination would be to provide a script to interested users, but I definitely see how on Windows and Mac this is not a "nice" the way to go. > I don't really understand the structure behind Viking's original > cache layout. I guess the omission of the file extension was just > because the original author didn't think to add it! The top level dir > names all seem to end in z0, I believe was in place for UTM > positioning where z is a zone, as there are repeating UTM zones > across the earth with Northings & Eastings within them. However I > think UTM things get translated into Mercator Lat/Longs so the zone > part is superfluous anyway. I see, I was wondering that myself. > I'm at least glad you're thinking of simple .ini files rather than > .json files that becoming all the rage by some people ;) I'm not fond of the "ini" extension, since I would *not* advocate a real INI parser, but just a simple k=v text file, without comments even. I wasn't sure about using ".conf" or not, though. The idea of using the "tile.ext.ini" (.ini appended as opposed to a [tile.ini]) comes also for simplicity when generating filenames. > Separately, could you explain how you think will do pruning in JOSM > in more detail? It can be quite complicated: e.g. How often and when > it occurs, how will it find out the size of the cache (and how long > that will take), does it delete highest zoom, lowest zoom or by > 'however the files are on disk' first? Do you worry about creating > new files if the size is already reached? How any feedback to the > user will work, if it is required at all? If you do switch to a cache > new layout - are you going to auto delete the old layout? You would be surprised to know that the second reason JOSM becomes slower is that it *doesn't* do any pruning. There's an option to prune the cache, but AFAIK it's broken. I think that maybe this is the reason of using /tmp/ on linux? But anyway. Cache pruning is complex, and there's no unique way to do it, so this is the reason a deliberately specify how to perform it. In JOSM, I would perform pruning only on program exit. This is due to the fact that assessing the cache size requires a full tree traversal, and will not be fast enough for interactive work. Josm has a list of rects that it keeps for "downloaded" areas, and I was planning to do something very simple at first: - get cache size / file list - sort by size/mtime/zoom respectively - while cache size > limit: - pick first tile in the sorted list that's not visible and remove it - if cache has not shrunk, disable visibility check - if cache has not shrunk, break Of course, this is very crude, it doesn't take into account how often areas a "looked at" at all, which would actually be much better. This is where those "metadata" files come in handy. JOSM already stores the ETag (for servers that don't support if-modified-since). Though I don't believe there are any... this is actually a perfect place to store the tile "hit counts" to perform usage-based pruning. -- Learn Graph Databases - Download FREE O'Reilly Book "Graph Databases" is the definitive new guide to graph databases and their applications. Written by three acclaimed leaders in the field, this first edition is now available. Download your free book today! http://p.sf.net/sfu/13534_NeoTech ___ Viking-devel mailing list Viking-devel@lists.sourceforge.net https://lists.sourceforge
Re: [Viking-devel] A shared tile cache
> > I'd like any of the developers to read this draft about a common shared > cache that I'd like to implement in JOSM: > > https://josm.openstreetmap.de/wiki/SharedTileCache > > Do you see anything missing or that you don't like? > There is nothing "magical" about it, I'd just like to follow some common > rules that would enable cache sharing. > Thanks for taking positive steps to resolve this cache issue. This has been mentioned several times in the past in various places or I've at least thought it in my head :) Here are some thoughts, which got a bit longer than I initially intended... Location on Windows: Include Windows 8.1 (I think it follows the Win7 layout) and 'C:\UsersPath\UserName' is better as %UserProfile% (indeed see http://en.wikipedia.org/wiki/Home_directory for other OSs) There are many other applications that could use such a shared cache layout (they must have there own schemes ATM, even if they at least use the standard Z/X/Y.ext structure somewhere. Immediately can I think of Marble, Navit, GPS Prune and libchamplain which would all benefit from using this scheme. I would be good to contact as many interested parties as possible. Should there be way to say never cache the tiles? e.g. size=-1 May be specify what is 'short' for the directory name, e.g. up to 20 characters. I have 9.0Gb in my viking-maps cache. If you've been using any such map program for a few years you may have got quite a lot of data. Transforming between old and new layouts *must be seamless*. Hence I won't be applying your patch, but thank you for it - indeed for a new user as yourself - it will be easier to keep that cache layout. I suspect JOSM can make a more clean cut jump from one lay out to another. Depends how vocal the Windows users are :), as they're the ones that would notice the change. But some people do use JOSM for general purpose GPX / shapefile / etc... usage, not just OSM editing. I still need to think how best to go about doing this in Viking, may be reading from the old first and saving to new - although that does make the code more complicated - and of course my time commitments and general interest in writing it... I added the 'DIRECTDIRACCESS' method into Viking, as part of starting to building some support for the standard layout. However the current implementation is aimed at pointing a directory source where you generate the tiles yourself (such as through an OSM rendering stack - not that I've ever gotten around to doing this...) which is why it deliberately doesn't download anything. I never realized JOSM used a flat dir layout. I can see how this would get slow under Windows. Not such a good idea to potentially have many thousands of files in one directory in any OS. I don't really understand the structure behind Viking's original cache layout. I guess the omission of the file extension was just because the original author didn't think to add it! The top level dir names all seem to end in z0, I believe was in place for UTM positioning where z is a zone, as there are repeating UTM zones across the earth with Northings & Eastings within them. However I think UTM things get translated into Mercator Lat/Longs so the zone part is superfluous anyway. I'm at least glad you're thinking of simple .ini files rather than .json files that becoming all the rage by some people ;) Separately, could you explain how you think will do pruning in JOSM in more detail? It can be quite complicated: e.g. How often and when it occurs, how will it find out the size of the cache (and how long that will take), does it delete highest zoom, lowest zoom or by 'however the files are on disk' first? Do you worry about creating new files if the size is already reached? How any feedback to the user will work, if it is required at all? If you do switch to a cache new layout - are you going to auto delete the old layout? NB In the Viking source code repo, in the tools directory there is a python program - viking-cache-mbtile.py - that converts a Viking cache into a MB Tiles files. You can then convert the MBTiles file into a normal TMS layout using http://github.com/mapbox/mbutil mbutil script. Be Seeing You - Rob. If at first you don't succeed, then skydiving isn't for you. -- Learn Graph Databases - Download FREE O'Reilly Book "Graph Databases" is the definitive new guide to graph databases and their applications. Written by three acclaimed leaders in the field, this first edition is now available. Download your free book today! http://p.sf.net/sfu/13534_NeoTech ___ Viking-devel mailing list Viking-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/viking-devel Viking home page: http://viking.sf.net/
[Viking-devel] A shared tile cache
I'd like any of the developers to read this draft about a common shared cache that I'd like to implement in JOSM: https://josm.openstreetmap.de/wiki/SharedTileCache Do you see anything missing or that you don't like? There is nothing "magical" about it, I'd just like to follow some common rules that would enable cache sharing. I'd ask interested parties to use the relevant JOSM's bug report for proposals/revisions if possible: https://josm.openstreetmap.de/ticket/9813 but writing here is fine as well (I'll update the document myself). Thanks a lot. -- Learn Graph Databases - Download FREE O'Reilly Book "Graph Databases" is the definitive new guide to graph databases and their applications. Written by three acclaimed leaders in the field, this first edition is now available. Download your free book today! http://p.sf.net/sfu/13534_NeoTech ___ Viking-devel mailing list Viking-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/viking-devel Viking home page: http://viking.sf.net/
