subject:"Re\: Sqlite 3.12 breaks the Django test suite"

Re: Sqlite 3.12 breaks the Django test suite

2016-04-07 Thread Aymeric Augustin

This other ticket about a SQLite segfault may or may not be related:
https://code.djangoproject.com/ticket/24080

-- 
Aymeric.

> On 07 Apr 2016, at 13:00, Raphael Hertzog  wrote:
> 
> [ CCing an upstream developer of SQlite too ]
> 
> Hello,
> 
> I did not want to open a ticket as I'm not sure if the problem
> is in SQLite or in Django but the Django test suite fails really badly
> with SQLite 3.12.0 that got recently released (and which is already
> in Debian Unstable).
> 
> When using the default --parallel run, I get many weird
> errors and sometimes even a deadlock. When I run it with --parallel=1,
> I got a segfault.
> 
> You can see a log of the failure here (this is a parallel run):
> https://ci.debian.net/data/packages/unstable/amd64/p/python-django/20160406_191352.autopkgtest.log.gz
> 
> There's a backtrace of the segfault (with --parallel=1):
> https://bugs.debian.org/cgi-bin/bugreport.cgi?filename=gdb.txt;bug=820225;msg=27;att=1
> 
> Cheers,
> 
> PS: Tracked here in Debian too:
> https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=820225
> -- 
> Raphaël Hertzog ◈ Writer/Consultant ◈ Debian Developer
> 
> Discover the Debian Administrator's Handbook:
> → http://debian-handbook.info/get/
> 
> -- 
> You received this message because you are subscribed to the Google Groups 
> "Django developers  (Contributions to Django itself)" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to django-developers+unsubscr...@googlegroups.com.
> To post to this group, send email to django-developers@googlegroups.com.
> Visit this group at https://groups.google.com/group/django-developers.
> To view this discussion on the web visit 
> https://groups.google.com/d/msgid/django-developers/20160407110012.GA18876%40home.ouaza.com.
> For more options, visit https://groups.google.com/d/optout.

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/ADA26AA4-2560-44E0-8EBD-CB275DCA51B6%40polytechnique.org.
For more options, visit https://groups.google.com/d/optout.

Re: Sqlite 3.12 breaks the Django test suite

2016-04-07 Thread Florian Apolloner

Perfect, I can reproduce it. This also happens if the default/other 
databases are not in memory -- I'll see what happens if I switch to 
postgres for the second database (though I am getting an error there 
currently -- might have to fix master :D)

On Thursday, April 7, 2016 at 1:00:25 PM UTC+2, Raphael Hertzog wrote:
>
> [ CCing an upstream developer of SQlite too ] 
>
> Hello, 
>
> I did not want to open a ticket as I'm not sure if the problem 
> is in SQLite or in Django but the Django test suite fails really badly 
> with SQLite 3.12.0 that got recently released (and which is already 
> in Debian Unstable). 
>
> When using the default --parallel run, I get many weird 
> errors and sometimes even a deadlock. When I run it with --parallel=1, 
> I got a segfault. 
>
> You can see a log of the failure here (this is a parallel run): 
>
> https://ci.debian.net/data/packages/unstable/amd64/p/python-django/20160406_191352.autopkgtest.log.gz
>  
>
> There's a backtrace of the segfault (with --parallel=1): 
>
> https://bugs.debian.org/cgi-bin/bugreport.cgi?filename=gdb.txt;bug=820225;msg=27;att=1
>  
>
> Cheers, 
>
> PS: Tracked here in Debian too: 
> https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=820225 
> -- 
> Raphaël Hertzog ◈ Writer/Consultant ◈ Debian Developer 
>
> Discover the Debian Administrator's Handbook: 
> → http://debian-handbook.info/get/ 
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/d74755bd-b281-4d29-88c1-884719ed26a2%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Sqlite 3.12 breaks the Django test suite

2016-04-07 Thread Raphael Hertzog

Le jeudi 07 avril 2016, Aymeric Augustin a écrit :
> This other ticket about a SQLite segfault may or may not be related:
> https://code.djangoproject.com/ticket/24080

Given how old it is, I was tempted to say it's entirely unrelated. But
maybe not. I discovered faulthandler thanks to this and the stacktrace
appears in the same place (rollback handling):

$ cd tests
$ LANG=C PYTHONPATH=.. python3 -X faulthandler ./runtests.py --verbosity=2 
--parallel=1 admin_views.tests.AdminViewBasicTest
[...]
test_popup_dismiss_related (admin_views.tests.AdminViewBasicTest) ... ok
Fatal Python error: Segmentation fault

Current thread 0x7fab40575700 (most recent call first):
  File 
"/home/rhertzog/deb/pkg/python-django/django/db/backends/sqlite3/base.py", line 
321 in execute
  File "/home/rhertzog/deb/pkg/python-django/django/db/backends/utils.py", line 
62 in execute
  File "/home/rhertzog/deb/pkg/python-django/django/db/backends/base/base.py", 
line 288 in _savepoint_rollback
  File "/home/rhertzog/deb/pkg/python-django/django/db/backends/base/base.py", 
line 328 in savepoint_rollback
  File "/home/rhertzog/deb/pkg/python-django/django/db/transaction.py", line 
243 in __exit__
  File "/home/rhertzog/deb/pkg/python-django/django/test/testcases.py", line 
1016 in _rollback_atomics
  File "/home/rhertzog/deb/pkg/python-django/django/test/testcases.py", line 
1073 in _fixture_teardown
  File "/home/rhertzog/deb/pkg/python-django/django/test/testcases.py", line 
919 in _post_teardown
  File "/home/rhertzog/deb/pkg/python-django/django/test/testcases.py", line 
217 in __call__
  File "/usr/lib/python3.5/unittest/suite.py", line 122 in run
  File "/usr/lib/python3.5/unittest/suite.py", line 84 in __call__
  File "/usr/lib/python3.5/unittest/runner.py", line 176 in run
  File "/home/rhertzog/deb/pkg/python-django/django/test/runner.py", line 494 
in run_suite
  File "/home/rhertzog/deb/pkg/python-django/django/test/runner.py", line 533 
in run_tests
  File "./runtests.py", line 275 in django_tests
  File "./runtests.py", line 458 in 

Maybe SQLite 3.12 makes it more likely to trigger to the problem or something
like that. I can reproduce the crash with Python 2 and Python 3.

Cheers,
-- 
Raphaël Hertzog ◈ Writer/Consultant ◈ Debian Developer

Discover the Debian Administrator's Handbook:
→ http://debian-handbook.info/get/

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/20160407131413.GA26492%40home.ouaza.com.
For more options, visit https://groups.google.com/d/optout.

Re: Sqlite 3.12 breaks the Django test suite

2016-04-07 Thread Florian Apolloner

This is the list of SQL statements we are executing: 
https://dpaste.de/Mqu1/raw first column is thread id, second is cursor id 
and third is the statement. I am a little bit confused about why the cursor 
ids seem to be "jumpy" like that -- but in general this should not cause 
issues I think

On Thursday, April 7, 2016 at 3:06:14 PM UTC+2, Florian Apolloner wrote:
>
> Perfect, I can reproduce it. This also happens if the default/other 
> databases are not in memory -- I'll see what happens if I switch to 
> postgres for the second database (though I am getting an error there 
> currently -- might have to fix master :D)
>
> On Thursday, April 7, 2016 at 1:00:25 PM UTC+2, Raphael Hertzog wrote:
>>
>> [ CCing an upstream developer of SQlite too ] 
>>
>> Hello, 
>>
>> I did not want to open a ticket as I'm not sure if the problem 
>> is in SQLite or in Django but the Django test suite fails really badly 
>> with SQLite 3.12.0 that got recently released (and which is already 
>> in Debian Unstable). 
>>
>> When using the default --parallel run, I get many weird 
>> errors and sometimes even a deadlock. When I run it with --parallel=1, 
>> I got a segfault. 
>>
>> You can see a log of the failure here (this is a parallel run): 
>>
>> https://ci.debian.net/data/packages/unstable/amd64/p/python-django/20160406_191352.autopkgtest.log.gz
>>  
>>
>> There's a backtrace of the segfault (with --parallel=1): 
>>
>> https://bugs.debian.org/cgi-bin/bugreport.cgi?filename=gdb.txt;bug=820225;msg=27;att=1
>>  
>>
>> Cheers, 
>>
>> PS: Tracked here in Debian too: 
>> https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=820225 
>> -- 
>> Raphaël Hertzog ◈ Writer/Consultant ◈ Debian Developer 
>>
>> Discover the Debian Administrator's Handbook: 
>> → http://debian-handbook.info/get/ 
>>
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/813967fd-3a1f-4b2a-93cc-c14f320f920f%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Sqlite 3.12 breaks the Django test suite

2016-04-07 Thread Sam Cooke

We ran into this issue a couple of months ago - I'm not sure how helpful
this is but just in case extra information is useful I'll share our
workarounds. The faulthandler output was roughly the same - the error
happened during a rollback. This also happened on multiple python and
sqlite versions.

It started when we removed a model and the one foreign key to that model.
The quickest workaround we found was to simply add the model and foreign
key back when running tests - which implies to me that the structure of the
database was related to whether or not it segfaulted (the test where the
segfault occurred was unrelated to the model that was removed).

The second option was we could remove the creation of a model during a test
(i.e. something that meant a rollback was required).

The third option was to use an on-disk sqlite database rather than in
memory.

I hope some of that is useful,
Sam

On Thu, 7 Apr 2016 at 14:37 Florian Apolloner  wrote:

> This is the list of SQL statements we are executing:
> https://dpaste.de/Mqu1/raw first column is thread id, second is cursor id
> and third is the statement. I am a little bit confused about why the cursor
> ids seem to be "jumpy" like that -- but in general this should not cause
> issues I think
>
>
> On Thursday, April 7, 2016 at 3:06:14 PM UTC+2, Florian Apolloner wrote:
>>
>> Perfect, I can reproduce it. This also happens if the default/other
>> databases are not in memory -- I'll see what happens if I switch to
>> postgres for the second database (though I am getting an error there
>> currently -- might have to fix master :D)
>>
>> On Thursday, April 7, 2016 at 1:00:25 PM UTC+2, Raphael Hertzog wrote:
>>>
>>> [ CCing an upstream developer of SQlite too ]
>>>
>>> Hello,
>>>
>>> I did not want to open a ticket as I'm not sure if the problem
>>> is in SQLite or in Django but the Django test suite fails really badly
>>> with SQLite 3.12.0 that got recently released (and which is already
>>> in Debian Unstable).
>>>
>>> When using the default --parallel run, I get many weird
>>> errors and sometimes even a deadlock. When I run it with --parallel=1,
>>> I got a segfault.
>>>
>>> You can see a log of the failure here (this is a parallel run):
>>>
>>> https://ci.debian.net/data/packages/unstable/amd64/p/python-django/20160406_191352.autopkgtest.log.gz
>>>
>>> There's a backtrace of the segfault (with --parallel=1):
>>>
>>> https://bugs.debian.org/cgi-bin/bugreport.cgi?filename=gdb.txt;bug=820225;msg=27;att=1
>>>
>>> Cheers,
>>>
>>> PS: Tracked here in Debian too:
>>> https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=820225
>>> --
>>> Raphaël Hertzog ◈ Writer/Consultant ◈ Debian Developer
>>>
>>> Discover the Debian Administrator's Handbook:
>>> → http://debian-handbook.info/get/
>>>
>> --
> You received this message because you are subscribed to the Google Groups
> "Django developers (Contributions to Django itself)" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to django-developers+unsubscr...@googlegroups.com.
> To post to this group, send email to django-developers@googlegroups.com.
> Visit this group at https://groups.google.com/group/django-developers.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/django-developers/813967fd-3a1f-4b2a-93cc-c14f320f920f%40googlegroups.com
> 
> .
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/CAMyx3A1YcqPkzUNboc%3DntngpAyLjLJ_9gy5vfZ_b5pxyqGjZrQ%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: Sqlite 3.12 breaks the Django test suite

2016-04-07 Thread Florian Apolloner

I was able to reduce the admin_view testcases to 
https://github.com/apollo13/django/blob/f086934f02efaca272e7e9b29921a31fd47821aa/tests/admin_views/tests.py
 
(see my sqlite_segfault branch)

On Thursday, April 7, 2016 at 3:06:14 PM UTC+2, Florian Apolloner wrote:
>
> Perfect, I can reproduce it. This also happens if the default/other 
> databases are not in memory -- I'll see what happens if I switch to 
> postgres for the second database (though I am getting an error there 
> currently -- might have to fix master :D)
>
> On Thursday, April 7, 2016 at 1:00:25 PM UTC+2, Raphael Hertzog wrote:
>>
>> [ CCing an upstream developer of SQlite too ] 
>>
>> Hello, 
>>
>> I did not want to open a ticket as I'm not sure if the problem 
>> is in SQLite or in Django but the Django test suite fails really badly 
>> with SQLite 3.12.0 that got recently released (and which is already 
>> in Debian Unstable). 
>>
>> When using the default --parallel run, I get many weird 
>> errors and sometimes even a deadlock. When I run it with --parallel=1, 
>> I got a segfault. 
>>
>> You can see a log of the failure here (this is a parallel run): 
>>
>> https://ci.debian.net/data/packages/unstable/amd64/p/python-django/20160406_191352.autopkgtest.log.gz
>>  
>>
>> There's a backtrace of the segfault (with --parallel=1): 
>>
>> https://bugs.debian.org/cgi-bin/bugreport.cgi?filename=gdb.txt;bug=820225;msg=27;att=1
>>  
>>
>> Cheers, 
>>
>> PS: Tracked here in Debian too: 
>> https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=820225 
>> -- 
>> Raphaël Hertzog ◈ Writer/Consultant ◈ Debian Developer 
>>
>> Discover the Debian Administrator's Handbook: 
>> → http://debian-handbook.info/get/ 
>>
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/e4a298a0-e873-4578-9b55-4583bb3faa3a%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Sqlite 3.12 breaks the Django test suite

2016-04-07 Thread Florian Apolloner

I've created an application which shows the behavior: 
https://github.com/apollo13/sqlite_bug -- I cannot yet reproduce it without 
running it through the testing process -- ./manage.py test admin_views 
triggers the bug in this application!

On Thursday, April 7, 2016 at 3:37:14 PM UTC+2, Florian Apolloner wrote:
>
> This is the list of SQL statements we are executing: 
> https://dpaste.de/Mqu1/raw first column is thread id, second is cursor id 
> and third is the statement. I am a little bit confused about why the cursor 
> ids seem to be "jumpy" like that -- but in general this should not cause 
> issues I think
>
> On Thursday, April 7, 2016 at 3:06:14 PM UTC+2, Florian Apolloner wrote:
>>
>> Perfect, I can reproduce it. This also happens if the default/other 
>> databases are not in memory -- I'll see what happens if I switch to 
>> postgres for the second database (though I am getting an error there 
>> currently -- might have to fix master :D)
>>
>> On Thursday, April 7, 2016 at 1:00:25 PM UTC+2, Raphael Hertzog wrote:
>>>
>>> [ CCing an upstream developer of SQlite too ] 
>>>
>>> Hello, 
>>>
>>> I did not want to open a ticket as I'm not sure if the problem 
>>> is in SQLite or in Django but the Django test suite fails really badly 
>>> with SQLite 3.12.0 that got recently released (and which is already 
>>> in Debian Unstable). 
>>>
>>> When using the default --parallel run, I get many weird 
>>> errors and sometimes even a deadlock. When I run it with --parallel=1, 
>>> I got a segfault. 
>>>
>>> You can see a log of the failure here (this is a parallel run): 
>>>
>>> https://ci.debian.net/data/packages/unstable/amd64/p/python-django/20160406_191352.autopkgtest.log.gz
>>>  
>>>
>>> There's a backtrace of the segfault (with --parallel=1): 
>>>
>>> https://bugs.debian.org/cgi-bin/bugreport.cgi?filename=gdb.txt;bug=820225;msg=27;att=1
>>>  
>>>
>>> Cheers, 
>>>
>>> PS: Tracked here in Debian too: 
>>> https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=820225 
>>> -- 
>>> Raphaël Hertzog ◈ Writer/Consultant ◈ Debian Developer 
>>>
>>> Discover the Debian Administrator's Handbook: 
>>> → http://debian-handbook.info/get/ 
>>>
>>

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/3bc96d8b-5004-4412-9802-c8532182fec2%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Sqlite 3.12 breaks the Django test suite

2016-04-07 Thread Florian Apolloner

mkdir ~/sqlite
cd ~/sqlite
git clone g...@github.com:django/django.git
git clone g...@github.com:apollo13/sqlite_bug.git
export PYTHONPATH=`pwd`/django
cd sqlite_bug
gdb python
(in gdb shell) r ./manage.py test admin_views

Please tell me if this does not cause a segfault for you.

Cheers,
Florian

On Thu, Apr 7, 2016, at 06:07 PM, Richard Hipp wrote:
> On 4/7/16, Florian Apolloner  wrote:
> > I've created an application which shows the behavior:
> > https://github.com/apollo13/sqlite_bug -- I cannot yet reproduce it without
> >
> > running it through the testing process -- ./manage.py test admin_views
> > triggers the bug in this application!
> 
> I have not touched Python in over a decade.  Please help me track down
> the problem by giving me step-by-step instructions on how to get this
> test running, inside of gdb, on Ubuntu 14.04.3 LTS.
> -- 
> D. Richard Hipp
> d...@sqlite.org

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/1460046444.1161220.572000497.6B596361%40webmail.messagingengine.com.
For more options, visit https://groups.google.com/d/optout.

Re: Sqlite 3.12 breaks the Django test suite

2016-04-08 Thread Aymeric Augustin

For the record, the discussion continues here:
https://www.sqlite.org/src/info/7f7f8026eda38

-- 
Aymeric.

> On 07 Apr 2016, at 18:27, Florian Apolloner  wrote:
> 
> mkdir ~/sqlite
> cd ~/sqlite
> git clone g...@github.com:django/django.git
> git clone g...@github.com:apollo13/sqlite_bug.git
> export PYTHONPATH=`pwd`/django
> cd sqlite_bug
> gdb python
> (in gdb shell) r ./manage.py test admin_views
> 
> Please tell me if this does not cause a segfault for you.
> 
> Cheers,
> Florian
> 
> On Thu, Apr 7, 2016, at 06:07 PM, Richard Hipp wrote:
>> On 4/7/16, Florian Apolloner  wrote:
>>> I've created an application which shows the behavior:
>>> https://github.com/apollo13/sqlite_bug -- I cannot yet reproduce it without
>>> 
>>> running it through the testing process -- ./manage.py test admin_views
>>> triggers the bug in this application!
>> 
>> I have not touched Python in over a decade.  Please help me track down
>> the problem by giving me step-by-step instructions on how to get this
>> test running, inside of gdb, on Ubuntu 14.04.3 LTS.
>> -- 
>> D. Richard Hipp
>> d...@sqlite.org
> 
> -- 
> You received this message because you are subscribed to the Google Groups 
> "Django developers  (Contributions to Django itself)" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to django-developers+unsubscr...@googlegroups.com.
> To post to this group, send email to django-developers@googlegroups.com.
> Visit this group at https://groups.google.com/group/django-developers.
> To view this discussion on the web visit 
> https://groups.google.com/d/msgid/django-developers/1460046444.1161220.572000497.6B596361%40webmail.messagingengine.com.
> For more options, visit https://groups.google.com/d/optout.

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/AADCC3D9-8A11-4F69-9320-21511885042A%40polytechnique.org.
For more options, visit https://groups.google.com/d/optout.

Re: Sqlite 3.12 breaks the Django test suite

Re: Sqlite 3.12 breaks the Django test suite

Re: Sqlite 3.12 breaks the Django test suite

Re: Sqlite 3.12 breaks the Django test suite

Re: Sqlite 3.12 breaks the Django test suite

Re: Sqlite 3.12 breaks the Django test suite

Re: Sqlite 3.12 breaks the Django test suite

Re: Sqlite 3.12 breaks the Django test suite

Re: Sqlite 3.12 breaks the Django test suite

9 matches

Site Navigation

Mail list logo

Footer information