Re: [QGIS-Developer] [Qgis-user] Multi users environment: From geopackage to a database?
QGIS-Devs
I'm forwarding this thread from QGIS-users mailinglist to this list.
This is about a crash report (aka severe bug but also a show stopper)
regarding GeoPackage - even read-only.
Goal should IMHO be that 1. QGIS does not crash, 2. QGIS supports
read-only multi-user access, and 3. QGIS prevents users to open
GeoPackage as write access unless it's sure the resource is locked.
I did not find any open issue on this topic except [1] and [2] which
are from 2017 and 2018. But perhaps I'm missing something.
=> Is there any dev working on this long known and outstanding issue?
:Stefan
[1] https://github.com/qgis/QGIS/issues/23991
[2] https://github.com/qgis/QGIS/issues/27899
Am Mi., 6. Mai 2020 um 12:22 Uhr schrieb Robert Nuske
:
>
> Hi Bruno,
>
> another list of known issues with SQLite/GeoPackage and SAMBA in particular
> https://trac.osgeo.org/gdal/wiki/UserDocs/SQLite
>
>
> regards
> robert
>
> Am 06.05.20 um 12:00 schrieb Stefan Keller:
> > Hi,
> >
> > This should be IMHO an issue of highest priority for QGIS.
> >
> > Jésahel wrote
> >> I've created a script to deactivate wal on samba shares, because samba is
> >> the main issue.
> >> It contains this only line : QSettings().setValue("/qgis/walForSqlite3",
> >> False)
> > That's interesting. Are you sure it's Samba?
> > SQLite FAQ [2] confirms that NFS has problems.
> > Did not find issues on Samba, but file locking with fcntl is mentioned here
> > [3].
> >
> > Goal should IMHO be that 1. No crash, 2. read-only multi-user access
> > should be possible and 3. writing users must be warned if QGIS isn't
> > able to exclusively open the file.
> > And Jésahel script suggests, that at least point 1 and 2 are possible.
> >
> > Andreas wrote:
> >> Unfortunately it is a known issue that Geopackages and QGIS are bad at
> >> handling simultaneous write and even read requests
> >> and developers are working on finding solutions (hopefully soon).
> > The only issue I find on QGIS tracker is [1]. Am I missing something?
> >
> > :Stefan
> >
> > [1] https://github.com/qgis/QGIS/issues/23991
> > [2] https://www.sqlite.org/faq.html#q5
> > [3] https://www.samba.org/samba/docs/old/Samba3-HOWTO/locking.html
> >
> > Am Mi., 6. Mai 2020 um 10:42 Uhr schrieb Jésahel Benoist
> > :
> >> Hi Bruno,
> >>
> >> I've worked a lot on this issue for two years now. I've started with
> >> sqlite (that was faster and better) but finally I decided to migrate to
> >> geopackage. It was painful at first but now it's OK. But it needs some
> >> rules.
> >>
> >> First, I've broken the original big geopackage files into smaller ones.
> >> The users are advised to work only one by one on a specific thematic/file
> >> (read access is possible).
> >> I've created a script to deactivate wal on samba shares, because samba is
> >> the main issue. It contains this only line :
> >> QSettings().setValue("/qgis/walForSqlite3", False)
> >> You should download some tools like Spatialite_GUI and
> >> SQLiteDatabaseBrowser so that you can really work on the db and have
> >> better control. QGIS's dbmanager is a bit limited and sometimes buggy for
> >> sqlite/geopackage, especially with special chars.
> >> Try to work with QGIS internal explorer, it's a tool that has good
> >> export/import functions.
> >> Backup files at least one time a day !
> >>
> >> Now with the covid I have created my own online postgis server so that we
> >> can work at home without "help" of IT department and it's GREAT.
> >> Multi-user, faster, smaller, stronger (lol). And some nice capabilities
> >> like spatial filtering ! I highly recommend this.
> >>
> >> Questions for all : I've recently discovered this
> >> https://gdal.org/user/virtual_file_systems.html#virtual-file-systems
> >> Wouldn't it be possible to avoid samba's multi-users bug with vsi ?
> >>
> >> Jésahel
> >>
> >>
> >>
> >> Le mer. 6 mai 2020 à 09:12, Andreas Neumann a écrit :
> >>> Hi Bruno,
> >>>
> >>> Unfortunately it is a known issue that Geopackages and QGIS are bad at
> >>> handling simultaneous write and even read requests and developers are
> >>> working on finding solutions (hopefully soon). In single user scenarios
> >>> Geopackages are fine. But Multiuser must be avoided, even for reading
> >>> only, unfortunately.
> >>>
> >>> As to PostgreSQL vs. Postgis: Postgis is the spatial extension of
> >>> PostgreSQL. If you need geometries (which I assume) than you will need
> >>> Postgis.
> >>>
> >>> Andreas
> >>>
> >>> Am 06.05.20 um 08:58 schrieb bru...@mailbox.org:
> >>>
> >>> Hi
> >>>
> >>> In my company we are five people who sometimes work with GIS, and we are
> >>> moving from ArcGIS to QGIS (how cool is QGIS and SLYR!) and in this
> >>> context from ESRI file geodatabase to geopackage. Today we run into
> >>> serious issues when two people worked with the same geopackage. I was
> >>> aware, that it is dangerous and unwise to edit the same geopackage from
> >>> two different computers at the same time. But I did n
Re: [QGIS-Developer] [Qgis-user] Multi users environment: From geopackage to a database?
Hi Stefan,
Thanks for the forward.
We are definitely aware of these issues and you can expect further
discussion on the QGIS and GDAL developer lists in the near future.
Several key QGIS users already identified this as a major hurdle when
migrating data from Shp to gpkg.
Hopefully a good solution can be found.
Andreas
Am 06.05.20 um 14:18 schrieb Stefan Keller:
QGIS-Devs
I'm forwarding this thread from QGIS-users mailinglist to this list.
This is about a crash report (aka severe bug but also a show stopper)
regarding GeoPackage - even read-only.
Goal should IMHO be that 1. QGIS does not crash, 2. QGIS supports
read-only multi-user access, and 3. QGIS prevents users to open
GeoPackage as write access unless it's sure the resource is locked.
I did not find any open issue on this topic except [1] and [2] which
are from 2017 and 2018. But perhaps I'm missing something.
=> Is there any dev working on this long known and outstanding issue?
:Stefan
[1] https://github.com/qgis/QGIS/issues/23991
[2] https://github.com/qgis/QGIS/issues/27899
Am Mi., 6. Mai 2020 um 12:22 Uhr schrieb Robert Nuske
:
Hi Bruno,
another list of known issues with SQLite/GeoPackage and SAMBA in particular
https://trac.osgeo.org/gdal/wiki/UserDocs/SQLite
regards
robert
Am 06.05.20 um 12:00 schrieb Stefan Keller:
Hi,
This should be IMHO an issue of highest priority for QGIS.
Jésahel wrote
I've created a script to deactivate wal on samba shares, because samba is the
main issue.
It contains this only line : QSettings().setValue("/qgis/walForSqlite3", False)
That's interesting. Are you sure it's Samba?
SQLite FAQ [2] confirms that NFS has problems.
Did not find issues on Samba, but file locking with fcntl is mentioned here [3].
Goal should IMHO be that 1. No crash, 2. read-only multi-user access
should be possible and 3. writing users must be warned if QGIS isn't
able to exclusively open the file.
And Jésahel script suggests, that at least point 1 and 2 are possible.
Andreas wrote:
Unfortunately it is a known issue that Geopackages and QGIS are bad at handling
simultaneous write and even read requests
and developers are working on finding solutions (hopefully soon).
The only issue I find on QGIS tracker is [1]. Am I missing something?
:Stefan
[1] https://github.com/qgis/QGIS/issues/23991
[2] https://www.sqlite.org/faq.html#q5
[3] https://www.samba.org/samba/docs/old/Samba3-HOWTO/locking.html
Am Mi., 6. Mai 2020 um 10:42 Uhr schrieb Jésahel Benoist :
Hi Bruno,
I've worked a lot on this issue for two years now. I've started with sqlite
(that was faster and better) but finally I decided to migrate to geopackage. It
was painful at first but now it's OK. But it needs some rules.
First, I've broken the original big geopackage files into smaller ones. The
users are advised to work only one by one on a specific thematic/file (read
access is possible).
I've created a script to deactivate wal on samba shares, because samba is the main issue.
It contains this only line : QSettings().setValue("/qgis/walForSqlite3", False)
You should download some tools like Spatialite_GUI and SQLiteDatabaseBrowser so
that you can really work on the db and have better control. QGIS's dbmanager is
a bit limited and sometimes buggy for sqlite/geopackage, especially with
special chars.
Try to work with QGIS internal explorer, it's a tool that has good
export/import functions.
Backup files at least one time a day !
Now with the covid I have created my own online postgis server so that we can work at
home without "help" of IT department and it's GREAT. Multi-user, faster,
smaller, stronger (lol). And some nice capabilities like spatial filtering ! I highly
recommend this.
Questions for all : I've recently discovered this
https://gdal.org/user/virtual_file_systems.html#virtual-file-systems
Wouldn't it be possible to avoid samba's multi-users bug with vsi ?
Jésahel
Le mer. 6 mai 2020 à 09:12, Andreas Neumann a écrit :
Hi Bruno,
Unfortunately it is a known issue that Geopackages and QGIS are bad at handling
simultaneous write and even read requests and developers are working on finding
solutions (hopefully soon). In single user scenarios Geopackages are fine. But
Multiuser must be avoided, even for reading only, unfortunately.
As to PostgreSQL vs. Postgis: Postgis is the spatial extension of PostgreSQL.
If you need geometries (which I assume) than you will need Postgis.
Andreas
Am 06.05.20 um 08:58 schrieb bru...@mailbox.org:
Hi
In my company we are five people who sometimes work with GIS, and we are moving
from ArcGIS to QGIS (how cool is QGIS and SLYR!) and in this context from ESRI
file geodatabase to geopackage. Today we run into serious issues when two
people worked with the same geopackage. I was aware, that it is dangerous and
unwise to edit the same geopackage from two different computers at the same
time. But I did not expect both QGIS applications to crash immediately and the
geopackage to
