[mapserver-users] Expire tiles and update new after Postgres DB update using mapcache

2019-07-26 Thread parsis presswala
I have developed the map using MapServer and pre-generated tile using
mapcache. Now I have an idea to update the database every day. As changes
apply in the database only those parts should be re-render and new tile
should be generated. As changes are very small and rendering all ties from
start is difficult to process.

I got expiry list of tiles by following osm2pgsql:

osm2pgsql --append --slim --cache 1000 --multi-geometry
/home/parsis/diff_file_minute.xml --number-processes 2 --hstore -d testdemo
-r xml -e6-11 -o /home/parsis/expire.list

Got list something like this:

 - 11/329/713
 - 8/40/91
 - 10/164/357
 - 7/29/47

I find another approach using render_expired
,
but don't know how to use it with mapcache.

How is it possible using mapcache_seed?
___
mapserver-users mailing list
mapserver-users@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/mapserver-users

[mapserver-users] Expire tiles and update new after Postgres DB update using mapcache

2019-07-23 Thread parsis presswala
Hello,
I have configure the mapserver and generated tiles using
mapcahing(mapcache_seed) for whole world till 200 m scaling (zoom level
16). Now I want to update the database using changeset provided by OSM and
regenerate only those tiles which data is updated in database. How should I
achieve this?
I want to configure this process automatically. As some new chnages updated
in database, old tiles shoud expire and new tiles should be generated
itself.


Regards,
Parsis Presswala
___
mapserver-users mailing list
mapserver-users@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/mapserver-users

[mapserver-users] Sqlite error in generated tiles

2019-05-14 Thread parsis presswala
Hello,
I am seeding tiles using multiple sqlite database.I am generating tiles of
15 zoom level but there are some potion in which I am not able load my
tiles even after seeding is completed. When I check what's wrong I got  *sqlite
backend failed on set: SQL logic error (1) *error.

Here is a url in which I am getting error.
http://localhost/worldcache/gmaps/world-tileset@grid2/15/23007/14603.png


I have checked the sqlite file version which is also not specific. I am
getting this.
*1000-11000.sqlite3: SQLite 3.x database, last written using SQLite version
3022000*

I tried to delete that area and again regenerate it but had not luck,
what should I do?

Thanks.
___
mapserver-users mailing list
mapserver-users@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/mapserver-users

[mapserver-users] Sqlite error in generated tiles

2019-05-14 Thread parsis presswala
Hello,
I am seeding tiles using multiple sqlite database.I am generating tiles of
15 zoom level but there are some potion in which I am not able load my
tiles even after seeding is completed. When I check what's wrong I got  *sqlite
backend failed on set: SQL logic error (1) *error.

Here is a url in which I am getting error.
http://localhost/worldcache/gmaps/world-tileset@grid2/15/23007/14603.png


I have checked the sqlite file version which is also not specific. I am
getting this.
*1000-11000.sqlite3: SQLite 3.x database, last written using SQLite version
3022000*

I tried to delete that area and again regenerate it but had not luck,
what should I do?

Thanks.
___
mapserver-users mailing list
mapserver-users@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/mapserver-users

[mapserver-users] changeset have no data

2019-03-26 Thread parsis presswala
Hello List,
I have imported data of world in database and now I want to update changes
in database.
For that I have download a change set of 2.5 GB from planet osm and created
sample table (changeDB) to check the changes.

Command Used: osm2pgsql -d changeDB --create --slim --cache 1000
--number-processes 2 --hstore --multi-geometry
path/osmdata/changesets-190318.osm.bz2

Terminal during processing shows this:
*Reading in file: path/osmdata/changesets-190318.osm.bz2*
*Using XML parser.*
*  parse time: 959s*
*Node stats: total(0), max(0) in 0s*
*Way stats: total(0), max(0) in 0s*
*Relation stats: total(0), max(0) in 0s*

No data has been updated in table.
___
mapserver-users mailing list
mapserver-users@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/mapserver-users

Re: [mapserver-users] Error While Generating Tiles

2019-03-14 Thread parsis presswala
Hello, I am getting following result by explain . It has been
running for an hour. Can you help me with this?
*Bitmap Heap Scan on planet_osm_polygon  (cost=182016.93..13405835.58
rows=16599 width=138)*
*   Recheck Cond: (name IS NOT NULL)*
*   Filter: (((leisure = ANY ('{park,nature_reserve}'::text[])) OR (tourism
IS NOT NULL) OR (historic IS NOT NULL)) AND (way &&
'01032031BF0D0001000500E8C60D48207933C1E5BD415EE1174A41E8C60D48207933C1FD8D6E9DBD245F41A17A94B809754A41FD8D6E9DBD245F41A17A94B809754A41E5BD415EE1174A41E8C60D48207933C1E5BD415EE1174A41'::geometry))*
*   ->  Bitmap Index Scan on planet_osm_polygon_name_index
(cost=0.00..182012.78 rows=9749894 width=0)*
* Index Cond: (name IS NOT NULL)*
*(5 rows)*


On Wed, Mar 13, 2019 at 10:55 PM parsis presswala 
wrote:

> Thanks, I'll check it out.
>
> On Wed, 13 Mar 2019 at 8:53 PM, Erik Gustafson 
> wrote:
>
>> Hi,
>>
>> seems like you're missing a spatial index on the geometry column.
>> CREATE INDEX planet_osm_polygon_sidx ON planet_osm_polygon USING
>> GIST(geom);
>>
>> If that doesn't help, do: explain analyze verbose ; and send the
>> results.
>>
>>
>>
>> On Wed, Mar 6, 2019 at 2:56 PM parsis presswala <
>> parsispressw...@gmail.com> wrote:
>>
>>> Hello,
>>>
>>> Yes I am using OSM data.
>>> I have imported it using osm2pgsql. Database: asiadb and tables
>>> are planet_osm_line, planet_osm_point, planet_osm_polygon, planet_osm_roads.
>>> I have imported data of whole planet using 40 GB .osm.pbf file.
>>> First I have only imported data of Asia and created map using
>>> map-server, which is working fine but as soon as import data of whole
>>> world, this problem occurred.
>>> I have 61 GB RAM and 8 Core processor as i am working on Amazon Elastic
>>> Compute Cloud (Amazon EC2) type r4.2xlarge.
>>>
>>> Here is my mapfile.
>>>  http://13.126.201.198/asiamap.map
>>>
>>> I have run EXPLAIN  and got following result:
>>> "Seq Scan on planet_osm_polygon  (cost=0.00..18878787.88 rows=36617920
>>> width=277)"
>>> "  Filter: ((leisure IS NOT NULL) OR (landuse IS NOT NULL) OR (amenity
>>> IS NOT NULL))"
>>>
>>>
>>> On Wed, Mar 6, 2019 at 7:04 PM Yves Jacolin 
>>> wrote:
>>>
>>>> Some questions that can help you :)
>>>>
>>>> is it OSM data? How do you import them?
>>>> What is the extent of the data? Could you check if spatial indexes
>>>> exist? Could you run EXPLAIN  and analyze it?
>>>>
>>>> Y.
>>>>
>>>> Le mer. 6 mars 2019 à 14:29, parsis presswala <
>>>> parsispressw...@gmail.com> a écrit :
>>>>
>>>>> Hello,
>>>>>
>>>>> Here is a log of PostgreSQL. I consider that PostgreSQL takes very
>>>>> long time to execute query on polygon. What should I do fot it?
>>>>>
>>>>> 2019-03-06 11:23:49.620 UTC [2302] parsis@asiadb STATEMENT:  select
>>>>> "highway"::text,"name"::text,"reflen"::text,"ref"::text,ST_AsBinary(ST_Force2D("way"),'NDR')
>>>>> as geom,"nullid"::text from (select *, NULL as nullid from ( SELECT way,
>>>>> highway, name, ref, oneway, CHAR_LENGTH(ref) AS reflen FROM 
>>>>> planet_osm_line
>>>>> WHERE highway IN ('motorway', 'trunk','motorway_link','trunk_link') AND
>>>>> (name IS NOT NULL OR ref IS NOT NULL)) AS data) as nullidq where "way" &&
>>>>> ST_GeomFromText('POLYGON((-9473834.25396072
>>>>> 2853929.66787328,-9473834.25396072 3446928.13330971,-8880835.78852428
>>>>> 3446928.13330971,-8880835.78852428 2853929.66787328,-9473834.25396072
>>>>> 2853929.66787328))',900913)
>>>>> 2019-03-06 11:24:14.911 UTC [2592] parsis@asiadb LOG:  could not send
>>>>> data to client: Broken pipe
>>>>> 2019-03-06 11:24:14.911 UTC [2592] parsis@asiadb STATEMENT:  select
>>>>> "stylegroup"::text,ST_AsBinary(ST_Force2D("way"),'NDR') as
>>>>> geom,"nullid"::text from (select *, NULL as nullid from ( SELECT way, CASE
>>>>> WHEN leisure in
>>>>> ('park','garden','playground','golf_course','sports_centre','pitch','stadium','nature_reserve','common')
>>>>> THEN 'leisuree' WHEN landuse
>>>>> IN('park','forest','grass','farmyard','farm','farmland','wood','meadow','village_green','rec

Re: [mapserver-users] Error While Generating Tiles

2019-03-13 Thread parsis presswala
Thanks, I'll check it out.

On Wed, 13 Mar 2019 at 8:53 PM, Erik Gustafson 
wrote:

> Hi,
>
> seems like you're missing a spatial index on the geometry column.
> CREATE INDEX planet_osm_polygon_sidx ON planet_osm_polygon USING
> GIST(geom);
>
> If that doesn't help, do: explain analyze verbose ; and send the
> results.
>
>
>
> On Wed, Mar 6, 2019 at 2:56 PM parsis presswala 
> wrote:
>
>> Hello,
>>
>> Yes I am using OSM data.
>> I have imported it using osm2pgsql. Database: asiadb and tables
>> are planet_osm_line, planet_osm_point, planet_osm_polygon, planet_osm_roads.
>> I have imported data of whole planet using 40 GB .osm.pbf file.
>> First I have only imported data of Asia and created map using map-server,
>> which is working fine but as soon as import data of whole world, this
>> problem occurred.
>> I have 61 GB RAM and 8 Core processor as i am working on Amazon Elastic
>> Compute Cloud (Amazon EC2) type r4.2xlarge.
>>
>> Here is my mapfile.
>>  http://13.126.201.198/asiamap.map
>>
>> I have run EXPLAIN  and got following result:
>> "Seq Scan on planet_osm_polygon  (cost=0.00..18878787.88 rows=36617920
>> width=277)"
>> "  Filter: ((leisure IS NOT NULL) OR (landuse IS NOT NULL) OR (amenity IS
>> NOT NULL))"
>>
>>
>> On Wed, Mar 6, 2019 at 7:04 PM Yves Jacolin 
>> wrote:
>>
>>> Some questions that can help you :)
>>>
>>> is it OSM data? How do you import them?
>>> What is the extent of the data? Could you check if spatial indexes
>>> exist? Could you run EXPLAIN  and analyze it?
>>>
>>> Y.
>>>
>>> Le mer. 6 mars 2019 à 14:29, parsis presswala 
>>> a écrit :
>>>
>>>> Hello,
>>>>
>>>> Here is a log of PostgreSQL. I consider that PostgreSQL takes very long
>>>> time to execute query on polygon. What should I do fot it?
>>>>
>>>> 2019-03-06 11:23:49.620 UTC [2302] parsis@asiadb STATEMENT:  select
>>>> "highway"::text,"name"::text,"reflen"::text,"ref"::text,ST_AsBinary(ST_Force2D("way"),'NDR')
>>>> as geom,"nullid"::text from (select *, NULL as nullid from ( SELECT way,
>>>> highway, name, ref, oneway, CHAR_LENGTH(ref) AS reflen FROM planet_osm_line
>>>> WHERE highway IN ('motorway', 'trunk','motorway_link','trunk_link') AND
>>>> (name IS NOT NULL OR ref IS NOT NULL)) AS data) as nullidq where "way" &&
>>>> ST_GeomFromText('POLYGON((-9473834.25396072
>>>> 2853929.66787328,-9473834.25396072 3446928.13330971,-8880835.78852428
>>>> 3446928.13330971,-8880835.78852428 2853929.66787328,-9473834.25396072
>>>> 2853929.66787328))',900913)
>>>> 2019-03-06 11:24:14.911 UTC [2592] parsis@asiadb LOG:  could not send
>>>> data to client: Broken pipe
>>>> 2019-03-06 11:24:14.911 UTC [2592] parsis@asiadb STATEMENT:  select
>>>> "stylegroup"::text,ST_AsBinary(ST_Force2D("way"),'NDR') as
>>>> geom,"nullid"::text from (select *, NULL as nullid from ( SELECT way, CASE
>>>> WHEN leisure in
>>>> ('park','garden','playground','golf_course','sports_centre','pitch','stadium','nature_reserve','common')
>>>> THEN 'leisuree' WHEN landuse
>>>> IN('park','forest','grass','farmyard','farm','farmland','wood','meadow','village_green','recreation_ground')
>>>> THEN 'landusee' WHEN amenity in
>>>> ('university','school','college','library','fuel','parking','cinema','theatre','place_of_worship','hospital')
>>>> THEN 'amenityy' END AS stylegroup FROM planet_osm_polygon where (leisure is
>>>> not null) or (landuse is not null) or (amenity is not null)) AS data) as
>>>> nullidq where "way" && ST_GeomFromText('POLYGON((8116389.7548
>>>> -1276192.28146022,8116389.7548 3467795.44203122,12860377.4777137
>>>> 3467795.44203122,12860377.4777137 -1276192.28146022,8116389.7548
>>>> -1276192.28146022))',900913)
>>>> 2019-03-06 11:24:14.911 UTC [2617] parsis@asiadb LOG:  could not send
>>>> data to client: Broken pipe
>>>> 2019-03-06 11:24:14.911 UTC [2617] parsis@asiadb STATEMENT:  select
>>>> "stylegroup"::text,ST_AsBinary(ST_Force2D("way"),'NDR') as
>>>> geom,"nullid"::text from (select *, NULL as nullid from ( SELECT way, CASE
>>>> WHEN leisure in
>>>> ('park','garden','playground','golf_course','sports_centre','pit

Re: [mapserver-users] Error While Generating Tiles

2019-03-06 Thread parsis presswala
I have improve Postgresql configuration shared_buffer = 2 MB and
work_mem = 500 MB.
Is this configuration fine or I need to improve more?

On Wed, Mar 6, 2019 at 7:53 PM parsis presswala 
wrote:

> No, not yet.
>
> On Wed, Mar 6, 2019 at 7:49 PM Yves Jacolin 
> wrote:
>
>> Did you improve Postgresql configuration (shared_buffer and work_mem
>> parameters) ?
>>
>> Y.
>>
>> Le mer. 6 mars 2019 à 15:15, parsis presswala 
>> a écrit :
>>
>>> Hello,
>>>
>>> I have already created indexing on some fields. Here is my indexed
>>> fields.
>>> CREATE INDEX planet_osm_polygon_idx
>>>   ON public.planet_osm_polygon
>>>   USING btree
>>>   (waterway COLLATE pg_catalog."default", "natural" COLLATE
>>> pg_catalog."default", landuse COLLATE pg_catalog."default", building
>>> COLLATE pg_catalog."default", aeroway COLLATE pg_catalog."default", leisure
>>> COLLATE pg_catalog."default", amenity COLLATE pg_catalog."default");
>>>
>>> Even I have experiment with r4.8xlarge with 32 CPU and 244 GB RAM. But
>>> got same result.
>>> What should I do?
>>>
>>> On Wed, Mar 6, 2019 at 7:31 PM Yves Jacolin 
>>> wrote:
>>>
>>>> You probablty needs to create index(es) on leisure, landuse and
>>>> amenity. Probably both of all but you need to do some test.
>>>>
>>>> Second, you probably needs to improve PostgreSQL configuration to let
>>>> PostgreSQL use your 61 GB of RAM :) shared_buffers and work_mem are good
>>>> parameters to look for.
>>>>
>>>> Y.
>>>>
>>>> Le mer. 6 mars 2019 à 14:56, parsis presswala <
>>>> parsispressw...@gmail.com> a écrit :
>>>>
>>>>> Hello,
>>>>>
>>>>> Yes I am using OSM data.
>>>>> I have imported it using osm2pgsql. Database: asiadb and tables
>>>>> are planet_osm_line, planet_osm_point, planet_osm_polygon, 
>>>>> planet_osm_roads.
>>>>> I have imported data of whole planet using 40 GB .osm.pbf file.
>>>>> First I have only imported data of Asia and created map using
>>>>> map-server, which is working fine but as soon as import data of whole
>>>>> world, this problem occurred.
>>>>> I have 61 GB RAM and 8 Core processor as i am working on Amazon
>>>>> Elastic Compute Cloud (Amazon EC2) type r4.2xlarge.
>>>>>
>>>>> Here is my mapfile.
>>>>>  http://13.126.201.198/asiamap.map
>>>>>
>>>>> I have run EXPLAIN  and got following result:
>>>>> "Seq Scan on planet_osm_polygon  (cost=0.00..18878787.88 rows=36617920
>>>>> width=277)"
>>>>> "  Filter: ((leisure IS NOT NULL) OR (landuse IS NOT NULL) OR (amenity
>>>>> IS NOT NULL))"
>>>>>
>>>>>
>>>>> On Wed, Mar 6, 2019 at 7:04 PM Yves Jacolin <
>>>>> yves.jaco...@camptocamp.com> wrote:
>>>>>
>>>>>> Some questions that can help you :)
>>>>>>
>>>>>> is it OSM data? How do you import them?
>>>>>> What is the extent of the data? Could you check if spatial indexes
>>>>>> exist? Could you run EXPLAIN  and analyze it?
>>>>>>
>>>>>> Y.
>>>>>>
>>>>>> Le mer. 6 mars 2019 à 14:29, parsis presswala <
>>>>>> parsispressw...@gmail.com> a écrit :
>>>>>>
>>>>>>> Hello,
>>>>>>>
>>>>>>> Here is a log of PostgreSQL. I consider that PostgreSQL takes very
>>>>>>> long time to execute query on polygon. What should I do fot it?
>>>>>>>
>>>>>>> 2019-03-06 11:23:49.620 UTC [2302] parsis@asiadb STATEMENT:  select
>>>>>>> "highway"::text,"name"::text,"reflen"::text,"ref"::text,ST_AsBinary(ST_Force2D("way"),'NDR')
>>>>>>> as geom,"nullid"::text from (select *, NULL as nullid from ( SELECT way,
>>>>>>> highway, name, ref, oneway, CHAR_LENGTH(ref) AS reflen FROM 
>>>>>>> planet_osm_line
>>>>>>> WHERE highway IN ('motorway', 'trunk','motorway_link','trunk_link') AND
>>>>>>> (name IS NOT NULL OR ref IS NOT NULL)) AS data) as nu

Re: [mapserver-users] Error While Generating Tiles

2019-03-06 Thread parsis presswala
No, not yet.

On Wed, Mar 6, 2019 at 7:49 PM Yves Jacolin 
wrote:

> Did you improve Postgresql configuration (shared_buffer and work_mem
> parameters) ?
>
> Y.
>
> Le mer. 6 mars 2019 à 15:15, parsis presswala 
> a écrit :
>
>> Hello,
>>
>> I have already created indexing on some fields. Here is my indexed fields.
>> CREATE INDEX planet_osm_polygon_idx
>>   ON public.planet_osm_polygon
>>   USING btree
>>   (waterway COLLATE pg_catalog."default", "natural" COLLATE
>> pg_catalog."default", landuse COLLATE pg_catalog."default", building
>> COLLATE pg_catalog."default", aeroway COLLATE pg_catalog."default", leisure
>> COLLATE pg_catalog."default", amenity COLLATE pg_catalog."default");
>>
>> Even I have experiment with r4.8xlarge with 32 CPU and 244 GB RAM. But
>> got same result.
>> What should I do?
>>
>> On Wed, Mar 6, 2019 at 7:31 PM Yves Jacolin 
>> wrote:
>>
>>> You probablty needs to create index(es) on leisure, landuse and amenity.
>>> Probably both of all but you need to do some test.
>>>
>>> Second, you probably needs to improve PostgreSQL configuration to let
>>> PostgreSQL use your 61 GB of RAM :) shared_buffers and work_mem are good
>>> parameters to look for.
>>>
>>> Y.
>>>
>>> Le mer. 6 mars 2019 à 14:56, parsis presswala 
>>> a écrit :
>>>
>>>> Hello,
>>>>
>>>> Yes I am using OSM data.
>>>> I have imported it using osm2pgsql. Database: asiadb and tables
>>>> are planet_osm_line, planet_osm_point, planet_osm_polygon, 
>>>> planet_osm_roads.
>>>> I have imported data of whole planet using 40 GB .osm.pbf file.
>>>> First I have only imported data of Asia and created map using
>>>> map-server, which is working fine but as soon as import data of whole
>>>> world, this problem occurred.
>>>> I have 61 GB RAM and 8 Core processor as i am working on Amazon
>>>> Elastic Compute Cloud (Amazon EC2) type r4.2xlarge.
>>>>
>>>> Here is my mapfile.
>>>>  http://13.126.201.198/asiamap.map
>>>>
>>>> I have run EXPLAIN  and got following result:
>>>> "Seq Scan on planet_osm_polygon  (cost=0.00..18878787.88 rows=36617920
>>>> width=277)"
>>>> "  Filter: ((leisure IS NOT NULL) OR (landuse IS NOT NULL) OR (amenity
>>>> IS NOT NULL))"
>>>>
>>>>
>>>> On Wed, Mar 6, 2019 at 7:04 PM Yves Jacolin <
>>>> yves.jaco...@camptocamp.com> wrote:
>>>>
>>>>> Some questions that can help you :)
>>>>>
>>>>> is it OSM data? How do you import them?
>>>>> What is the extent of the data? Could you check if spatial indexes
>>>>> exist? Could you run EXPLAIN  and analyze it?
>>>>>
>>>>> Y.
>>>>>
>>>>> Le mer. 6 mars 2019 à 14:29, parsis presswala <
>>>>> parsispressw...@gmail.com> a écrit :
>>>>>
>>>>>> Hello,
>>>>>>
>>>>>> Here is a log of PostgreSQL. I consider that PostgreSQL takes very
>>>>>> long time to execute query on polygon. What should I do fot it?
>>>>>>
>>>>>> 2019-03-06 11:23:49.620 UTC [2302] parsis@asiadb STATEMENT:  select
>>>>>> "highway"::text,"name"::text,"reflen"::text,"ref"::text,ST_AsBinary(ST_Force2D("way"),'NDR')
>>>>>> as geom,"nullid"::text from (select *, NULL as nullid from ( SELECT way,
>>>>>> highway, name, ref, oneway, CHAR_LENGTH(ref) AS reflen FROM 
>>>>>> planet_osm_line
>>>>>> WHERE highway IN ('motorway', 'trunk','motorway_link','trunk_link') AND
>>>>>> (name IS NOT NULL OR ref IS NOT NULL)) AS data) as nullidq where "way" &&
>>>>>> ST_GeomFromText('POLYGON((-9473834.25396072
>>>>>> 2853929.66787328,-9473834.25396072 3446928.13330971,-8880835.78852428
>>>>>> 3446928.13330971,-8880835.78852428 2853929.66787328,-9473834.25396072
>>>>>> 2853929.66787328))',900913)
>>>>>> 2019-03-06 11:24:14.911 UTC [2592] parsis@asiadb LOG:  could not
>>>>>> send data to client: Broken pipe
>>>>>> 2019-03-06 11:24:14.911 UTC [2592] parsis@asiadb STATEMENT:  select
>>>>>&g

Re: [mapserver-users] Error While Generating Tiles

2019-03-06 Thread parsis presswala
Hello,

I have already created indexing on some fields. Here is my indexed fields.
CREATE INDEX planet_osm_polygon_idx
  ON public.planet_osm_polygon
  USING btree
  (waterway COLLATE pg_catalog."default", "natural" COLLATE
pg_catalog."default", landuse COLLATE pg_catalog."default", building
COLLATE pg_catalog."default", aeroway COLLATE pg_catalog."default", leisure
COLLATE pg_catalog."default", amenity COLLATE pg_catalog."default");

Even I have experiment with r4.8xlarge with 32 CPU and 244 GB RAM. But got
same result.
What should I do?

On Wed, Mar 6, 2019 at 7:31 PM Yves Jacolin 
wrote:

> You probablty needs to create index(es) on leisure, landuse and amenity.
> Probably both of all but you need to do some test.
>
> Second, you probably needs to improve PostgreSQL configuration to let
> PostgreSQL use your 61 GB of RAM :) shared_buffers and work_mem are good
> parameters to look for.
>
> Y.
>
> Le mer. 6 mars 2019 à 14:56, parsis presswala 
> a écrit :
>
>> Hello,
>>
>> Yes I am using OSM data.
>> I have imported it using osm2pgsql. Database: asiadb and tables
>> are planet_osm_line, planet_osm_point, planet_osm_polygon, planet_osm_roads.
>> I have imported data of whole planet using 40 GB .osm.pbf file.
>> First I have only imported data of Asia and created map using map-server,
>> which is working fine but as soon as import data of whole world, this
>> problem occurred.
>> I have 61 GB RAM and 8 Core processor as i am working on Amazon Elastic
>> Compute Cloud (Amazon EC2) type r4.2xlarge.
>>
>> Here is my mapfile.
>>  http://13.126.201.198/asiamap.map
>>
>> I have run EXPLAIN  and got following result:
>> "Seq Scan on planet_osm_polygon  (cost=0.00..18878787.88 rows=36617920
>> width=277)"
>> "  Filter: ((leisure IS NOT NULL) OR (landuse IS NOT NULL) OR (amenity IS
>> NOT NULL))"
>>
>>
>> On Wed, Mar 6, 2019 at 7:04 PM Yves Jacolin 
>> wrote:
>>
>>> Some questions that can help you :)
>>>
>>> is it OSM data? How do you import them?
>>> What is the extent of the data? Could you check if spatial indexes
>>> exist? Could you run EXPLAIN  and analyze it?
>>>
>>> Y.
>>>
>>> Le mer. 6 mars 2019 à 14:29, parsis presswala 
>>> a écrit :
>>>
>>>> Hello,
>>>>
>>>> Here is a log of PostgreSQL. I consider that PostgreSQL takes very long
>>>> time to execute query on polygon. What should I do fot it?
>>>>
>>>> 2019-03-06 11:23:49.620 UTC [2302] parsis@asiadb STATEMENT:  select
>>>> "highway"::text,"name"::text,"reflen"::text,"ref"::text,ST_AsBinary(ST_Force2D("way"),'NDR')
>>>> as geom,"nullid"::text from (select *, NULL as nullid from ( SELECT way,
>>>> highway, name, ref, oneway, CHAR_LENGTH(ref) AS reflen FROM planet_osm_line
>>>> WHERE highway IN ('motorway', 'trunk','motorway_link','trunk_link') AND
>>>> (name IS NOT NULL OR ref IS NOT NULL)) AS data) as nullidq where "way" &&
>>>> ST_GeomFromText('POLYGON((-9473834.25396072
>>>> 2853929.66787328,-9473834.25396072 3446928.13330971,-8880835.78852428
>>>> 3446928.13330971,-8880835.78852428 2853929.66787328,-9473834.25396072
>>>> 2853929.66787328))',900913)
>>>> 2019-03-06 11:24:14.911 UTC [2592] parsis@asiadb LOG:  could not send
>>>> data to client: Broken pipe
>>>> 2019-03-06 11:24:14.911 UTC [2592] parsis@asiadb STATEMENT:  select
>>>> "stylegroup"::text,ST_AsBinary(ST_Force2D("way"),'NDR') as
>>>> geom,"nullid"::text from (select *, NULL as nullid from ( SELECT way, CASE
>>>> WHEN leisure in
>>>> ('park','garden','playground','golf_course','sports_centre','pitch','stadium','nature_reserve','common')
>>>> THEN 'leisuree' WHEN landuse
>>>> IN('park','forest','grass','farmyard','farm','farmland','wood','meadow','village_green','recreation_ground')
>>>> THEN 'landusee' WHEN amenity in
>>>> ('university','school','college','library','fuel','parking','cinema','theatre','place_of_worship','hospital')
>>>> THEN 'amenityy' END AS stylegroup FROM planet_osm_polygon where (leisure is
>>>> not null) or (landuse is not null) or (amenity is not null)) AS data) as
>>>> nullidq where "way" && ST_GeomFromText('POLYGON((8116389.7548
>>>> -1276192.28146022,8116389.7548 3467795.44203122,1286037

Re: [mapserver-users] Error While Generating Tiles

2019-03-06 Thread parsis presswala
Hello,

Yes I am using OSM data.
I have imported it using osm2pgsql. Database: asiadb and tables
are planet_osm_line, planet_osm_point, planet_osm_polygon, planet_osm_roads.
I have imported data of whole planet using 40 GB .osm.pbf file.
First I have only imported data of Asia and created map using map-server,
which is working fine but as soon as import data of whole world, this
problem occurred.
I have 61 GB RAM and 8 Core processor as i am working on Amazon Elastic
Compute Cloud (Amazon EC2) type r4.2xlarge.

Here is my mapfile.
 http://13.126.201.198/asiamap.map

I have run EXPLAIN  and got following result:
"Seq Scan on planet_osm_polygon  (cost=0.00..18878787.88 rows=36617920
width=277)"
"  Filter: ((leisure IS NOT NULL) OR (landuse IS NOT NULL) OR (amenity IS
NOT NULL))"


On Wed, Mar 6, 2019 at 7:04 PM Yves Jacolin 
wrote:

> Some questions that can help you :)
>
> is it OSM data? How do you import them?
> What is the extent of the data? Could you check if spatial indexes exist?
> Could you run EXPLAIN  and analyze it?
>
> Y.
>
> Le mer. 6 mars 2019 à 14:29, parsis presswala 
> a écrit :
>
>> Hello,
>>
>> Here is a log of PostgreSQL. I consider that PostgreSQL takes very long
>> time to execute query on polygon. What should I do fot it?
>>
>> 2019-03-06 11:23:49.620 UTC [2302] parsis@asiadb STATEMENT:  select
>> "highway"::text,"name"::text,"reflen"::text,"ref"::text,ST_AsBinary(ST_Force2D("way"),'NDR')
>> as geom,"nullid"::text from (select *, NULL as nullid from ( SELECT way,
>> highway, name, ref, oneway, CHAR_LENGTH(ref) AS reflen FROM planet_osm_line
>> WHERE highway IN ('motorway', 'trunk','motorway_link','trunk_link') AND
>> (name IS NOT NULL OR ref IS NOT NULL)) AS data) as nullidq where "way" &&
>> ST_GeomFromText('POLYGON((-9473834.25396072
>> 2853929.66787328,-9473834.25396072 3446928.13330971,-8880835.78852428
>> 3446928.13330971,-8880835.78852428 2853929.66787328,-9473834.25396072
>> 2853929.66787328))',900913)
>> 2019-03-06 11:24:14.911 UTC [2592] parsis@asiadb LOG:  could not send
>> data to client: Broken pipe
>> 2019-03-06 11:24:14.911 UTC [2592] parsis@asiadb STATEMENT:  select
>> "stylegroup"::text,ST_AsBinary(ST_Force2D("way"),'NDR') as
>> geom,"nullid"::text from (select *, NULL as nullid from ( SELECT way, CASE
>> WHEN leisure in
>> ('park','garden','playground','golf_course','sports_centre','pitch','stadium','nature_reserve','common')
>> THEN 'leisuree' WHEN landuse
>> IN('park','forest','grass','farmyard','farm','farmland','wood','meadow','village_green','recreation_ground')
>> THEN 'landusee' WHEN amenity in
>> ('university','school','college','library','fuel','parking','cinema','theatre','place_of_worship','hospital')
>> THEN 'amenityy' END AS stylegroup FROM planet_osm_polygon where (leisure is
>> not null) or (landuse is not null) or (amenity is not null)) AS data) as
>> nullidq where "way" && ST_GeomFromText('POLYGON((8116389.7548
>> -1276192.28146022,8116389.7548 3467795.44203122,12860377.4777137
>> 3467795.44203122,12860377.4777137 -1276192.28146022,8116389.7548
>> -1276192.28146022))',900913)
>> 2019-03-06 11:24:14.911 UTC [2617] parsis@asiadb LOG:  could not send
>> data to client: Broken pipe
>> 2019-03-06 11:24:14.911 UTC [2617] parsis@asiadb STATEMENT:  select
>> "stylegroup"::text,ST_AsBinary(ST_Force2D("way"),'NDR') as
>> geom,"nullid"::text from (select *, NULL as nullid from ( SELECT way, CASE
>> WHEN leisure in
>> ('park','garden','playground','golf_course','sports_centre','pitch','stadium','nature_reserve','common')
>> THEN 'leisuree' WHEN landuse
>> IN('park','forest','grass','farmyard','farm','farmland','wood','meadow','village_green','recreation_ground')
>> THEN 'landusee' WHEN amenity in
>> ('university','school','college','library','fuel','parking','cinema','theatre','place_of_worship','hospital')
>> THEN 'amenityy' END AS stylegroup FROM planet_osm_polygon where (leisure is
>> not null) or (landuse is not null) or (amenity is not null)) AS data) as
>> nullidq where "way" && ST_GeomFromText('POLYGON((8116389.7548
>> -1276192.28146022,8116389.7548 3467795.44203122,12860377.4777137
>> 3467795.44203122,12860377.4777137 -1276192.28146022,8116389.7548
>> -1276192.28146022))',900913)
>> 2019-03-06 11:24:14.911 UTC [2592] parsis@asiadb FATAL:  connection to
>> client lost
>> 2019-03-06 12:08:25.564 UTC [3362] parsis@asiadb STATEMENT:  select
>> ST_AsBinary(ST_Force2D("way"),'NDR') as ge

Re: [mapserver-users] Error While Generating Tiles

2019-03-06 Thread parsis presswala
POLYGON((-20228294.8225998
-20228294.8225998,-20228294.8225998 17723606.9653297,17723606.9653297
17723606.9653297,17723606.9653297 -20228294.8225998,-20228294.8225998
-20228294.8225998))',900913)
2019-03-06 12:08:25.564 UTC [3363] FATAL:  terminating connection due to
administrator command
2019-03-06 12:08:25.564 UTC [3363] STATEMENT:  select
ST_AsBinary(ST_Force2D("way"),'NDR') as geom,"nullid"::text from (select *,
NULL as nullid from ( SELECT way FROM planet_osm_polygon WHERE (waterway IN
('riverbank')) OR ("natural" IN ('water')) OR (landuse IN ('basin',
'reservoir')) ) AS data) as nullidq where "way" &&
ST_GeomFromText('POLYGON((-20228294.8225998
-20228294.8225998,-20228294.8225998 17723606.9653297,17723606.9653297
17723606.9653297,17723606.9653297 -20228294.8225998,-20228294.8225998
-20228294.8225998))',900913)
2019-03-06 12:08:25.570 UTC [1535] LOG:  worker process: parallel worker
for PID 3255 (PID 3261) exited with exit code 1
2019-03-06 12:08:25.570 UTC [1535] LOG:  worker process: parallel worker
for PID 3362 (PID 3363) exited with exit code 1
2019-03-06 12:08:25.570 UTC [1535] LOG:  worker process: parallel worker
for PID 3255 (PID 3260) exited with exit code 1
2019-03-06 12:08:25.609 UTC [3426] parsis@asiadb LOG:  could not send data
to client: Broken pipe
2019-03-06 12:08:25.609 UTC [3426] parsis@asiadb STATEMENT:  select
"admin_level"::text,ST_AsBinary(ST_Force2D("way"),'NDR') as
geom,"nullid"::text from (select *, NULL as nullid from (select * from
planet_osm_line where admin_level IN('2') and boundary IN
('administrative') and way NOT IN (SELECT way FROM planet_osm_line where
admin_level IN('2') and boundary like 'administrative' and (tags->'dispute'
= 'yes' or tags->'ISO3166-1' IN ('IN','CN','PK') or way IN(select way from
in_cn_ignored_border AS data) as nullidq where "way" &&
ST_GeomFromText('POLYGON((-20228294.8225998
-20228294.8225998,-20228294.8225998 17723606.9653297,17723606.9653297
17723606.9653297,17723606.9653297 -20228294.8225998,-20228294.8225998
-20228294.8225998))',900913)
2019-03-06 12:08:25.609 UTC [3426] parsis@asiadb FATAL:  connection to
client lost

Regards,
Parsis Presswala

On Wed, Mar 6, 2019 at 6:35 PM Yves Jacolin 
wrote:

> Hello,
>
> Did you try to curl the request and see what happens: timeout, etc. If so,
> try to get some log (mapserver or postgresql log) and see where is the
> bottleneck.
>
> Y.
>
> Le mer. 6 mars 2019 à 13:58, parsis presswala 
> a écrit :
>
>> Hello,
>> I am getting following error while generating tiles. I can't get the
>> reason of this error. I an using mapcache to generate tiles.
>>
>> failed to seed tile z10,x645,y510:
>> curl failed to request url
>> http://13.126.201.198/cgi-bin/mapserv?VERSION=1.1.1=GetMap=WMS==5201998.739765%2c-81328.655306%2c5795150.079258%2c511822.684187=3880=3880=image%2fpng=EPSG%3a3857=osm_new_admin%2cosm_new_adminocean%2cosm_new_landusages%2cosm_new_water%2cbuildings%2caeroway%2cosm_new_waterways%2cosm_new_railways%2cplanet_osm_roads%2cplanet_osm_roads_line%2cosm_new_railways1%2ccountry_line%2ccountry_line_claim%2cstate_line%2cmotorway_label%2cmainroad_label%2cminorroad_label%2cosm_new_railways_symbol%2cplace-country%2cplace-state%2cplace-city%2cplace-town%2cplace-Suburb%2cplace-village%2cplace-locality%2carea_label%2caerodrome_label%2cwaterway_label%2cwaterarea_label%2cpoi_pointlabel=%2fhome%2fubuntu%2fworldmap%2fasiamap.map
>> : Operation timed out after 60 milliseconds with 0 bytes received
>>
>> aborting seed as 100.0% of the last 1000 requests failed
>>
>> seeded 225 tiles, now at z10 x660 y510
>> seeded 2 metatiles (450 total tiles, 450 non-empty tiles) in 1201.8
>> seconds at 0.4 tiles/sec (0.4 non-empty tiles/sec)
>>
>> If anyone could help me with this, it would be greatly appreciated.
>>
>> Regards,
>> Parsis Presswala
>> ___
>> mapserver-users mailing list
>> mapserver-users@lists.osgeo.org
>> https://lists.osgeo.org/mailman/listinfo/mapserver-users
>
>
>
> --
> Yves Jacolin
> Training and support manager - Team Manager
> Camptocamp
>
> Tel (France) : +33 4 58 48 20 43
> Tel (Switzerland) : +41 21 619 10 43
> Mob. : +33 6 18 75 42 21
>
> email : yves.jaco...@camptocamp.com
> http://www.camptocamp.com
>
___
mapserver-users mailing list
mapserver-users@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/mapserver-users

[mapserver-users] How much memory is required to seed tiles?

2019-01-28 Thread parsis presswala
Hello,
I am running mapserver on 61 GB RAM and cpu of 8 core.
When I am trying to seed the tiles it is showing me following error.

I am seeding tiles with following command:
mapcache_seed -c /etc/apache2/mapcache.xml -t asia-tileset -g grid2 -n 4 -z
3,3

failed to seed tile z3,x0,y0:
curl failed to request url
http://13.126.201.198/cgi-bin/mapserv?VERSION=1.1.1=GetMap=WMS==-20428865.584820%2c-20428865.584820%2c20428866.270399%2c20428866.270399=2088=2088=image%2fpng=EPSG%3a3857=osm_new_admin%2cosm_new_adminocean%2cosm_new_landusages%2cosm_new_water%2cbuildings%2caeroway%2cosm_new_waterways%2cosm_new_railways%2cplanet_osm_roads%2cplanet_osm_roads_line%2cosm_new_railways1%2ccountry_line%2ccountry_line_claim%2cstate_line%2cmotorway_label%2cmainroad_label%2cminorroad_label%2cosm_new_railways_symbol%2cplace-country%2cplace-state%2cplace-city%2cplace-town%2cplace-Suburb%2cplace-village%2cplace-locality%2carea_label%2caerodrome_label%2cwaterway_label%2cwaterarea_label%2cpoi_pointlabel=%2fhome%2fubuntu%2fworldmap%2fasiamap.map
:
The requested URL returned error: 504 Gateway Timeout

aborting seed as 100.0% of the last 1000 requests failed

I was seeding tiles successfully when I have only two continents (asia +
central america) in database.But as I insert other continents in database I
am getting error.


Here is my mapfile.

http://13.126.201.198/asiamap.map

I have used MAXSCALEDENOM and MINSCALEDENOM in every layer. Here is my
mapfile. What should I change in it?

I haven't use  https://github.com/mapserver/basemaps , I have created on my
own.


Can I generate tiles with same configurations?


Thanks in advance.
___
mapserver-users mailing list
mapserver-users@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/mapserver-users