Hello,
Thank you for the thorough explanation Harleen, very helpful. It turned out
that also our roles.xml was corrupted in production. It did keep most of
the role_ids, however it removed nearly all of the role-user associations,
including the admin. Hence the lock out. We ware able to restore most of
the accounts by re-adding the associations. Also the Geofence data rules
were missing for most of the workspaces/accounts.
I believe the corrupt file occurs when doing to much I/O to the xml files.
The first error before the 403 was a xml.etree.ElementTree.ParseError.
After that, the roles.xml was mostly lost. I am able to reproduce the error
on our staging environment by making a lot of API calls to add users and
check the xml for existing users. A solution that worked for me is leaving
enough time between the calls. Last night I've added 450+ users, roles and
workspaces without any problems. There were however a few minutes between
each user creation as there was a data processing and uploading step in
between. I am now backing up the users.xml and roles.xml regularly, losing
those account associations in production had a huge impact.
Andrea, I hope to find some time to make a vanilla installation with some
descriptions and the script.
Thanks again for the great help!
Kind regards,
Darell
On Wed, Jan 24, 2018 at 9:26 AM, Andrea Aime <andrea.a...@geo-solutions.it>
wrote:
> Hi all,
> thanks for moving the discussion forward. Just to be clear, the only way
> a developer may look into this
> is that you make it easily reproducable on a vanilla installation that
> anyone can run on their own.
> If GeoFence has to be plugged in to reproduce, that is not a problem, just
> describe it, and provide
> that and the script to make it happen in a ticket at https://osgeo-org.
> atlassian.net/projects/GEOS/summary
>
> Cheers
> Andrea
>
>
> On Tue, Jan 23, 2018 at 9:25 PM, Harleen Lappano <hlapp...@cssflorida.net>
> wrote:
>
>> Hey again,
>>
>> Just to add I hit the snag again after making those above changes too.
>> However, I isolated the situation more and how to sort of recover back.
>>
>> The roles.xml file is getting corrupted somehow. It is removing some of
>> the users that were already in place including admin. Which is why the
>> forbidden 403 status is occurring. By syncing up the roles.xml file again
>> with admin, you will be able to get access again. But you do need the
>> interval check tag set in the config in order for geoserver to read in the
>> external changes. Or I think you can restart geoserver, not sure about that
>> though.
>>
>> However, just don't know what is causing it to get corrupted. The users
>> are getting added correctly to the users.xml file, but the roles on the
>> other hand is not becoming off sync.
>>
>>
>>
>> On Tue, Jan 23, 2018 at 10:47 AM, Harleen Lappano <
>> hlapp...@cssflorida.net> wrote:
>>
>>> Hey,
>>>
>>> Andrea, I also don't know if we will be able to run a test here without
>>> Geofence. It would be on my own time right now since we battled through the
>>> issue for now. However, I can say we tried to dump a users.xml and
>>> roles.xml while it was running to add the users that way instead of the
>>> REST api and that did however cause an issue when the Authentication Cache
>>> went to run. We also attempted to add one user that way too. Same issue.
>>> I'm not sure if we just hit it with luck on that one user, but it did have
>>> the same effect.
>>>
>>> Darell as far as the steps go I will list what we do. Since we are just
>>> pushing GeoFence up for the first time to production our redeploy envolves
>>> a bit and it uses ansible. However, I'll try to list it hope it makes
>>> sense.
>>> Also by doing this the existing users will be lost, so make sure you
>>> have a backup or a way to get them back in.
>>> We currently have two user group services. The default one to keep admin
>>> credentials on. And a second one for our security authentication. It is the
>>> second one that holds all the users.
>>>
>>> Steps we took:
>>>
>>> 1. Stop Tomcat Service
>>> 2. Install GeoFence
>>> 1. Unarchive geoserver-{{geofence_version}}
>>> -SNAPSHOT-geofence-server-plugin.zip (This is the embedded
>>> version)
>>> 2. Drop them in the tomcat/webapps/geoserver/WEB-INF/lib/
>>> 3. Change permissions, owner and group. Which ever one is set for
>>> your tomcat. Permissions 0644
>>> 3. Copy Security Files (These are the ones we made modifications for
>>> our setup).
>>> - List of files Under /tomcat/webapps/geoserver/data/security/
>>> - config.xml (This contained our filter chains)
>>> - rest.properties (we made a change here for access on a
>>> specific role)
>>> - /filter/{{our_authentication}}/config.xml
>>> - /role/default/
>>> - /config.xml (this has the change I mentioned, however, we
>>> changed it after deploy because we were using our deploy
>>> package)
>>> - /roles.xml
>>> - (the roles.xsd gets generated after first start up)
>>> - /usergroup/{{our_usergroup_name}}/ (this is a new one we
>>> created that is not default.)
>>> - /config.xml
>>> - /users.xml
>>> - /users.xsd
>>> 4. Start Tomcat Service
>>> 5. Wait For Tomcat to Start
>>> 6. Add Data Rules
>>> 1. These are our data rules using Geofence REST.
>>> 7. After this we go to the admin page and see we have control again.
>>> 8. We also ran a test with our admin user, so this might have
>>> triggered the authentication cache. Mabe best to wait 10 minutes till
>>> your
>>> cache refreshes before adding the users.
>>> 9. Stop Tomcat Service
>>> 10. Modify users.xml and roles.xml as mentioned in the last email.
>>> 11. Start Tomcat Service
>>> 12. Ran our Users script
>>> 1. Loop through all users
>>> 1. curl REST API to add user
>>> 2. curl REST API to associate user with role
>>> 3. sleep 1 second (Not sure if we need this. Just noticed
>>> without it, it will chug and possibly run into a stack overflow
>>> exception)
>>>
>>>
>>> And those my friend our the steps we took. Like I said we use ansible,
>>> so we never isolated what exactly caused it to unlock. Actually listing
>>> this out, I am wondering if it was the refresh of users.xml and roles.xml
>>> to a blank state.
>>>
>>> Hope that helps.
>>>
>>>
>>>
>>> On Tue, Jan 23, 2018 at 6:45 AM, Darell van der Voort <
>>> darell...@gmail.com> wrote:
>>>
>>>> Hello Harleen, Andrea,
>>>>
>>>> Thank you very much for the input! Harleen, your problem is exactly the
>>>> same as we are facing. I am also adding authentication with geofence and
>>>> pull the exisiting users from the users.xml file. Normally we only add just
>>>> a few users without any problems. But indeed with more than 200 users the
>>>> lockdown happens.
>>>>
>>>> I have just pushed a new Geoserver build to production and changed both
>>>> config files to checkInterval 0 and validating to false. However the
>>>> problem still persists. I have rebooted the instances and restarted all the
>>>> task definitions. What do you mean exactly with 'redeploy Geofence and
>>>> security settings including the users.xml and roles.xml'? We have those
>>>> settings on a shared mount so that each Geoserver instance uses the same
>>>> settings. What steps did you do to redeploy, without creating new user and
>>>> role xmls?
>>>>
>>>> Andrea, I do not thing it can be reproduced without Geofence in the
>>>> mix, as this is the only way to add users via the REST API. I can however
>>>> reproduce it with Geofence. Also thanks for the support link.
>>>>
>>>> Kind regards,
>>>> Darell
>>>>
>>>> On Tue, Jan 23, 2018 at 10:47 AM, Andrea Aime <
>>>> andrea.a...@geo-solutions.it> wrote:
>>>>
>>>>> Hi Darell and Harleen,
>>>>> it seems you're on to something. Wondering, can it be reproduced
>>>>> without GeoFence in the mix?
>>>>> It would be useful to have a script that reproduces the problem on a
>>>>> vanilla GeoServer installation, if that can be done
>>>>> please open a ticket with it at: https://osgeo-org.atlassia
>>>>> n.net/projects/GEOS/summary
>>>>>
>>>>> Darell, this list offers volountary support, which does not do
>>>>> "urgent" (well, not with guarantees at least, it can happen of course).
>>>>> If you're in a hurry I'd suggest you also check
>>>>> http://geoserver.org/support/
>>>>>
>>>>> Cheers
>>>>> Andrea
>>>>>
>>>>>
>>>>> On Tue, Jan 23, 2018 at 1:50 AM, Harleen Lappano <
>>>>> hlapp...@cssflorida.net> wrote:
>>>>>
>>>>>> Hey Darell,
>>>>>>
>>>>>> I'm not a expert on Geoserver. However, funny story is that I ran
>>>>>> into this same issue last Friday as we were pushing our changes to our
>>>>>> production.
>>>>>> We are using Geoserver 2.9.1 for our product. I don't remember the
>>>>>> tomcat version.
>>>>>>
>>>>>> If it wasn't for your email, we would have been still trying to trace
>>>>>> down all our changes and deltas we made for this push window, so I
>>>>>> appreciate that you posted this it helped us narrow down and push pass
>>>>>> this
>>>>>> issue.
>>>>>>
>>>>>> We were adding authentication to our geoserver with geofence so we
>>>>>> can add users using the REST api. I did a similar script where I pulled
>>>>>> our
>>>>>> existing users in our system to also add them to the geoserver's user
>>>>>> list
>>>>>> using REST api. At some point in our process we had a total lock down on
>>>>>> our geoserver in production. So just to see I ran a script on our dev box
>>>>>> to add 200 random users to the geoserver and was able to reproduce what
>>>>>> happened in our production.
>>>>>>
>>>>>> This is what we did based on our observations of the problem and it
>>>>>> seemed to get pass this issue for now. We are still running tests this
>>>>>> week
>>>>>> on our product, however, mabe it will help you get along as well.
>>>>>> We are thinking that there might be an issue in geoserver regarding
>>>>>> the users.xml or roles.xml. It seems like the 3 components that might be
>>>>>> the cause are 1. Authentication Cache that runs every 10 minutes, 2. Our
>>>>>> curl commands in our script that adds the user. and 3. The interval check
>>>>>> for modifications on the users.xml or roles.xml. (these are the names we
>>>>>> chose, but it is the xml that holds the user group service and the role
>>>>>> server).
>>>>>>
>>>>>> In order for us to get back to an unlock state, I had to redeploy our
>>>>>> GeoFence and security settings including the users.xml and roles.xml. I'm
>>>>>> not sure what is in the set that causes the geoserver to unlock because
>>>>>> we
>>>>>> didn't trace it down that far.
>>>>>> Then we went ahead and changed our config.xml for our user group and
>>>>>> roles has (two config files):
>>>>>> <checkInterval>0</checkInterval>
>>>>>> <validating>false</validating>
>>>>>>
>>>>>> We weren't sure about validating, but we turned it off anyways. After
>>>>>> that and a restart, we were able to run our script to add the users
>>>>>> without
>>>>>> any problems. I will add that in our script we added a second delay after
>>>>>> each add. I noticed in the original time we did our script we ran into a
>>>>>> stack overflow issue.
>>>>>>
>>>>>> After all this I think we are concluding that there might have been a
>>>>>> lock on the file resource when reading and writing. However, this is
>>>>>> just a
>>>>>> hunch without further testing.
>>>>>> Hopefully someone on this mailing list knows exactly what is
>>>>>> happening.
>>>>>>
>>>>>>
>>>>>>
>>>>>> On Mon, Jan 22, 2018 at 5:02 AM, Darell van der Voort <
>>>>>> darell...@gmail.com> wrote:
>>>>>>
>>>>>>> Hello everybody,
>>>>>>>
>>>>>>> I'm running Geoserver 2.11.2 on Java 1.8.0_144 32 bits on Amazon
>>>>>>> Linux together with Tomcat 9. I have a very urgent problem.
>>>>>>>
>>>>>>> I have a Python script that uses the REST API to automatically
>>>>>>> create users, set read/write permissions and create workspaces. I have
>>>>>>> been
>>>>>>> using this for the last months without any problems. However today I
>>>>>>> had to
>>>>>>> create 450+ accounts and workspace so I wrote a loop to generate the
>>>>>>> accounts. This went flawless till about account 137. After that I got a
>>>>>>> access denied:
>>>>>>>
>>>>>>> HTTP 403 status Forbidden
>>>>>>> Access is denied
>>>>>>> The server understood the request but refuses to authorize it.
>>>>>>> Apache Tomcat/9.01
>>>>>>>
>>>>>>> Also when I log into the web interface using the admin account I
>>>>>>> cannot access anything. The data is however still served. This leads me
>>>>>>> to
>>>>>>> believe it is some kind of protection from Geoserver against DDoS or
>>>>>>> brute
>>>>>>> force attacks. I do not have another account with admin rights.
>>>>>>> How do I re-enable access to the Geoserver? I already tried
>>>>>>> restarting the server instance, but no luck.
>>>>>>>
>>>>>>> Thanks!
>>>>>>> Darell
>>>>>>>
>>>>>>> ------------------------------------------------------------
>>>>>>> ------------------
>>>>>>> Check out the vibrant tech community on one of the world's most
>>>>>>> engaging tech sites, Slashdot.org! http://sdm.link/slashdot
>>>>>>> _______________________________________________
>>>>>>> Geoserver-users mailing list
>>>>>>>
>>>>>>> Please make sure you read the following two resources before posting
>>>>>>> to this list:
>>>>>>> - Earning your support instead of buying it, but Ian Turton:
>>>>>>> http://www.ianturton.com/talks/foss4g.html#/
>>>>>>> - The GeoServer user list posting guidelines:
>>>>>>> http://geoserver.org/comm/userlist-guidelines.html
>>>>>>>
>>>>>>> If you want to request a feature or an improvement, also see this:
>>>>>>> https://github.com/geoserver/geoserver/wiki/Successfully-req
>>>>>>> uesting-and-integrating-new-features-and-improvements-in-GeoServer
>>>>>>>
>>>>>>>
>>>>>>> Geoserver-users@lists.sourceforge.net
>>>>>>> https://lists.sourceforge.net/lists/listinfo/geoserver-users
>>>>>>>
>>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>> Regards,
>>>>>> Harleen
>>>>>>
>>>>>>
>>>>>> ------------------------------------------------------------
>>>>>> ------------------
>>>>>> Check out the vibrant tech community on one of the world's most
>>>>>> engaging tech sites, Slashdot.org! http://sdm.link/slashdot
>>>>>> _______________________________________________
>>>>>> Geoserver-users mailing list
>>>>>>
>>>>>> Please make sure you read the following two resources before posting
>>>>>> to this list:
>>>>>> - Earning your support instead of buying it, but Ian Turton:
>>>>>> http://www.ianturton.com/talks/foss4g.html#/
>>>>>> - The GeoServer user list posting guidelines:
>>>>>> http://geoserver.org/comm/userlist-guidelines.html
>>>>>>
>>>>>> If you want to request a feature or an improvement, also see this:
>>>>>> https://github.com/geoserver/geoserver/wiki/Successfully-req
>>>>>> uesting-and-integrating-new-features-and-improvements-in-GeoServer
>>>>>>
>>>>>>
>>>>>> Geoserver-users@lists.sourceforge.net
>>>>>> https://lists.sourceforge.net/lists/listinfo/geoserver-users
>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>> --
>>>>>
>>>>> Regards,
>>>>>
>>>>> Andrea Aime
>>>>>
>>>>> ==
>>>>> GeoServer Professional Services from the experts! Visit
>>>>> http://goo.gl/it488V for more information.
>>>>> ==
>>>>>
>>>>> Ing. Andrea Aime
>>>>> @geowolf
>>>>> Technical Lead
>>>>>
>>>>> GeoSolutions S.A.S.
>>>>> Via di Montramito 3/A
>>>>> <https://maps.google.com/?q=Via+di+Montramito+3/A+55054+%C2%A0Massarosa&entry=gmail&source=g>
>>>>> 55054 Massarosa
>>>>> <https://maps.google.com/?q=Via+di+Montramito+3/A+55054+%C2%A0Massarosa&entry=gmail&source=g>
>>>>> (LU)
>>>>> phone: +39 0584 962313 <+39%200584%20962313>
>>>>> fax: +39 0584 1660272 <+39%200584%20166%200272>
>>>>> mob: +39 339 8844549 <+39%20339%20884%204549>
>>>>>
>>>>> http://www.geo-solutions.it
>>>>> http://twitter.com/geosolutions_it
>>>>>
>>>>> AVVERTENZE AI SENSI DEL D.Lgs. 196/2003
>>>>>
>>>>> Le informazioni contenute in questo messaggio di posta elettronica e/o
>>>>> nel/i file/s allegato/i sono da considerarsi strettamente riservate. Il
>>>>> loro utilizzo è consentito esclusivamente al destinatario del messaggio,
>>>>> per le finalità indicate nel messaggio stesso. Qualora riceviate questo
>>>>> messaggio senza esserne il destinatario, Vi preghiamo cortesemente di
>>>>> darcene notizia via e-mail e di procedere alla distruzione del messaggio
>>>>> stesso, cancellandolo dal Vostro sistema. Conservare il messaggio stesso,
>>>>> divulgarlo anche in parte, distribuirlo ad altri soggetti, copiarlo, od
>>>>> utilizzarlo per finalità diverse, costituisce comportamento contrario ai
>>>>> principi dettati dal D.Lgs. 196/2003.
>>>>>
>>>>> The information in this message and/or attachments, is intended solely
>>>>> for the attention and use of the named addressee(s) and may be
>>>>> confidential
>>>>> or proprietary in nature or covered by the provisions of privacy act
>>>>> (Legislative Decree June, 30 2003, no.196 - Italy's New Data Protection
>>>>> Code).Any use not in accord with its purpose, any disclosure,
>>>>> reproduction,
>>>>> copying, distribution, or either dissemination, either whole or partial,
>>>>> is
>>>>> strictly forbidden except previous formal approval of the named
>>>>> addressee(s). If you are not the intended recipient, please contact
>>>>> immediately the sender by telephone, fax or e-mail and delete the
>>>>> information in this message that has been received in error. The sender
>>>>> does not give any warranty or accept liability as the content, accuracy or
>>>>> completeness of sent messages and accepts no responsibility for changes
>>>>> made after they were sent or for other risks which arise as a result of
>>>>> e-mail transmission, viruses, etc.
>>>>>
>>>>>
>>>>
>>>
>>>
>>> --
>>> Regards,
>>> Harleen Lappano
>>> 407-473-7157 <(407)%20473-7157>
>>>
>>
>>
>>
>> --
>> Regards,
>> Harleen Lappano
>> 407-473-7157 <(407)%20473-7157>
>>
>
>
>
> --
>
> Regards,
>
> Andrea Aime
>
> ==
> GeoServer Professional Services from the experts! Visit
> http://goo.gl/it488V for more information.
> ==
>
> Ing. Andrea Aime
> @geowolf
> Technical Lead
>
> GeoSolutions S.A.S.
> Via di Montramito 3/A
> <https://maps.google.com/?q=Via+di+Montramito+3/A+55054+%C2%A0Massarosa+%3Chttps://maps.google.com/?q%3DVia%2Bdi%2BMontramito%2B3/A%2B55054%2B%25C2%25A0Massarosa%26entry%3Dgmail%26source%3Dg%3E&entry=gmail&source=g>
> 55054 Massarosa
> <https://maps.google.com/?q=Via+di+Montramito+3/A+55054+%C2%A0Massarosa+%3Chttps://maps.google.com/?q%3DVia%2Bdi%2BMontramito%2B3/A%2B55054%2B%25C2%25A0Massarosa%26entry%3Dgmail%26source%3Dg%3E&entry=gmail&source=g>
> (LU)
> phone: +39 0584 962313 <+39%200584%20962313>
> fax: +39 0584 1660272 <+39%200584%20166%200272>
> mob: +39 339 8844549 <+39%20339%20884%204549>
>
> http://www.geo-solutions.it
> http://twitter.com/geosolutions_it
>
> AVVERTENZE AI SENSI DEL D.Lgs. 196/2003
>
> Le informazioni contenute in questo messaggio di posta elettronica e/o
> nel/i file/s allegato/i sono da considerarsi strettamente riservate. Il
> loro utilizzo è consentito esclusivamente al destinatario del messaggio,
> per le finalità indicate nel messaggio stesso. Qualora riceviate questo
> messaggio senza esserne il destinatario, Vi preghiamo cortesemente di
> darcene notizia via e-mail e di procedere alla distruzione del messaggio
> stesso, cancellandolo dal Vostro sistema. Conservare il messaggio stesso,
> divulgarlo anche in parte, distribuirlo ad altri soggetti, copiarlo, od
> utilizzarlo per finalità diverse, costituisce comportamento contrario ai
> principi dettati dal D.Lgs. 196/2003.
>
> The information in this message and/or attachments, is intended solely for
> the attention and use of the named addressee(s) and may be confidential or
> proprietary in nature or covered by the provisions of privacy act
> (Legislative Decree June, 30 2003, no.196 - Italy's New Data Protection
> Code).Any use not in accord with its purpose, any disclosure, reproduction,
> copying, distribution, or either dissemination, either whole or partial, is
> strictly forbidden except previous formal approval of the named
> addressee(s). If you are not the intended recipient, please contact
> immediately the sender by telephone, fax or e-mail and delete the
> information in this message that has been received in error. The sender
> does not give any warranty or accept liability as the content, accuracy or
> completeness of sent messages and accepts no responsibility for changes
> made after they were sent or for other risks which arise as a result of
> e-mail transmission, viruses, etc.
>
>
------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
_______________________________________________
Geoserver-users mailing list
Please make sure you read the following two resources before posting to this
list:
- Earning your support instead of buying it, but Ian Turton:
http://www.ianturton.com/talks/foss4g.html#/
- The GeoServer user list posting guidelines:
http://geoserver.org/comm/userlist-guidelines.html
If you want to request a feature or an improvement, also see this:
https://github.com/geoserver/geoserver/wiki/Successfully-requesting-and-integrating-new-features-and-improvements-in-GeoServer
Geoserver-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/geoserver-users
