Adam, I've gone sideways and made a little progress, so I'm back. In terms of our versions, we are decently in sync. We have everything you listed and more. Most of the the version numbers match. We have 18 that are marginally newer than what you are running and you have 4 marginally newer than ours. I don't think the problem lies there. I've attached a spreadsheet if you want to have a look.Our extras are:
GDAL==2.4.2 keyring==10.6.0 keyrings.alt==3.0 numpy==1.13.3 pycrypto==2.6.1 pygobject==3.26.1 pyxdg==0.25 virtualenv==13.1.2 I'm a little suspicious of the keyring* files, as I noticed that psql has its own. I haven't tried uninstalling them yet, though. Our versions of mod-wsgi are the same, so I don't plan to mess with that. I'd be interested in knowing what version of OpenSSL you are running, though. Ours is 1.1.1 (no letter revision). This may be something I need to upgrade. I very carefully removed psql 11.4, testing every step of the way to be sure the HTTP version still ran. I then uninstalled psycopg2 and re-installed version 2.8.3. I checked and there is no binary. With this, the error log now shows some timeout messages instead of the truncated or oversized response headers messages. I'm not going into depth here because I realized I have one more thing I need to try before running down the list of psycopg2 revisions. In my research, I read that the preferred way to reference the site-packages in the WSGIDaemonProcess was to use python-home with a path to the virtual environment root, instead of concatenating the full site-packages path to the python-path option. This earned me a stackdump, because wsgi couldn't find arches in the settings.py import arches line. I just have to try changing that path in settings.py to see whether that just takes me back to the truncated or oversized response headers messages or opens new doors. Some adventure. Martha On Tuesday, January 7, 2020 at 3:33:05 PM UTC-8, Martha S wrote: > > I appreciate this, Adam. > > I may end up uninstalling the newer version of postgreSQL anyway, but will > de4finitely start here. > > Thanks, > Martha > > On Tuesday, January 7, 2020 at 2:50:27 PM UTC-8, Adam Cox wrote: >> >> Hm, well I hope this helps. Here's one v4 instance I currently have >> running on HTTPS without any trouble. https://csh.legiongis.com. A >> little examination of that server shows: >> >> (env)$ pip freeze >> >> asn1crypto==0.24.0 >> backports-abc==0.5 >> backports.functools-lru-cache==1.5 >> bcrypt==3.1.7 >> beautifulsoup4==4.7.1 >> certifi==2019.6.16 >> cffi==1.12.3 >> chardet==3.0.4 >> CouchDB==1.2 >> coverage==4.5.3 >> cryptography==2.7 >> Django==1.11.15 >> django-cors-headers==2.2.0 >> django-guardian==1.4.9 >> django-nose==1.4.6 >> django-oauth-toolkit==1.1.2 >> django-recaptcha==1.4.0 >> django-revproxy==0.9.15 >> django-webtest==1.9.6 >> ecdsa==0.13.2 >> edtf==2.7.0 >> elasticsearch==5.5.3 >> enum34==1.1.6 >> extras==1.0.0 >> fabric==2.4.0 >> fixtures==3.0.0 >> funcsigs==1.0.2 >> future==0.17.1 >> futures==3.2.0 >> idna==2.8 >> invoke==1.2.0 >> ipaddress==1.0.22 >> isodate==0.6.0 >> junitxml==0.7 >> linecache2==1.0.0 >> livereload==2.6.1 >> mapbox-vector-tile==1.2.0 >> mapnik==0.1 >> mock==2.0.0 >> ModestMaps==1.4.7 >> nose==1.3.7 >> oauthlib==3.0.1 >> paramiko==2.5.0 >> pbr==5.3.1 >> Pillow==6.0.0 >> protobuf==3.8.0 >> psycopg2-binary==2.7.4 >> pyclipper==1.1.0.post1 >> pycparser==2.19 >> pycryptodome==3.8.2 >> PyLD==1.0.4 >> PyNaCl==1.3.0 >> pyOpenSSL==19.0.0 >> pyparsing==2.4.0 >> PyPrind==2.11.2 >> python-dateutil==2.8.0 >> python-jose==2.0.2 >> python-memcached==1.59 >> python-mimeparse==1.6.0 >> python-subunit==1.3.0 >> pytz==2019.1 >> PyYAML==3.12 >> rdflib==4.2.2 >> requests==2.22.0 >> sauceclient==1.0.0 >> selenium==3.141.0 >> Shapely==1.6.4.post2 >> simplejson==3.16.0 >> singledispatch==3.4.0.3 >> six==1.12.0 >> soupsieve==1.9.1 >> SPARQLWrapper==1.8.2 >> sst==0.2.4 >> testtools==2.3.0 >> TileStache==1.51.13 >> tornado==5.1.1 >> traceback2==1.4.0 >> unicodecsv==0.14.1 >> unittest2==1.1.0 >> urllib3==1.21.1 >> waitress==1.3.0 >> WebOb==1.8.5 >> WebTest==2.0.33 >> Werkzeug==0.15.4 >> >> --- >> >> (env)$ dpkg -l | grep wsgi >> ii libapache2-mod-wsgi 4.5.17-1 >> amd64 Python WSGI adapter module for Apache >> >> --- >> >> (env)$ apache2 -v >> Server version: Apache/2.4.29 (Ubuntu) >> Server built: 2019-09-16T12:58:48 >> >> --- >> >> (I'm not sure if you're using certbot/Let's Encrypt, but for what it's >> worth) >> >> $ certbot --version >> certbot 0.31.0 >> >> --- >> >> On that server, postgres is not installed, but psql is. >> >> $ psql --version >> psql (PostgreSQL) 10.11 (Ubuntu 10.11-1.pgdg18.04+1) >> >> Hope that all helps, hope I'm not missing anything. >> >> Adam >> >> On Mon, Jan 6, 2020 at 5:07 PM Martha S <marth...@lacity.org> wrote: >> >>> Adam, >>> >>> I worked my way down the psycopg2 version list with no success. I made >>> sure to restart Apache after every uninstall/reinstall.. I finally had to >>> quit at version 2.6.2, which could not be installed. Here's the quirk: I >>> have both postgres 9.6.15 and 11.4-1 on my server. The arches database is >>> 9.6.15 and I can't find anything in the forum ever suggesting that the >>> database be upgraded. I imagine the server was initially provisioned with >>> 11.5, but we added 9.6 during the arches install. >>> >>> This has not previously been an issue, but it is now that we want to >>> implement SSL. While I couldn't get relief with any of the higher psycopg2 >>> versions, version 2.6.2 complains Error: could not determine PostgreSQL >>> version from '11.5'. That's not a typo, even though the version claims to >>> be 11.4-1; psycopg2 won't install. Too bad, because it's the last version >>> that worked for some folks when this issue first came to light. >>> >>> So, I know from our earlier conversation you thought I should perhaps >>> reinstall mod_wsgi at this point, but now I'm wondering if I should >>> uninstall postgres 11.4-1 instead and try again to replace psycopg2 from >>> the top of the release list or if there's some other path you'd recommend. >>> >>> I truly appreciate you hanging in there with me, >>> Martha >>> >>> >>> On Monday, January 6, 2020 at 11:52:38 AM UTC-8, Martha S wrote: >>>> >>>> Thanks again, Adam, >>>> >>>> I'll report back when I've taken this next step, in case others have to >>>> deal with this issue. >>>> >>>> Martha >>>> >>>> On Monday, January 6, 2020 at 11:48:15 AM UTC-8, Adam Cox wrote: >>>>> >>>>> Ok, well the key when I fixed this for myself was finding the release >>>>> of psycopg2 that comes with the correct OpenSSL version: >>>>> http://initd.org/psycopg/docs/news.html >>>>> >>>>> Hopefully if you just uninstall psycopg2 (and psycopg2-binary) from >>>>> your virtual environment and then look at that release list and try >>>>> installing/uninstalling releases that have different OpenSSL versions, >>>>> then >>>>> one of them should work. At least, that's the easiest next thing to try. >>>>> >>>>> (env)$ pip install psycopg2==x.x.x >>>>> >>>>> If none of the changes to that python library work then maybe you >>>>> should go the route of reinstalling mod_wsgi. >>>>> >>>>> Adam >>>>> >>>>> On Mon, Jan 6, 2020 at 12:51 PM Martha S <marth...@lacity.org> wrote: >>>>> >>>>>> Yes, Adam, >>>>>> >>>>>> I can confirm I restarted Apache after the initial psycopg2 >>>>>> uninstall/install, after uninstalling the binary, after redoing the >>>>>> uninstall/install, and after adding the wsgi parameter to my config >>>>>> file. >>>>>> Nope at every turn. >>>>>> >>>>>> If you don't feel wsgi is the problem, I'll proceed with installing >>>>>> psycopg2 from GitHub, unless you feel that is inadvisable. I'm thinking >>>>>> that there must be some difference in our server setups that made the >>>>>> cure >>>>>> for you not work for me. >>>>>> >>>>>> Thanks, >>>>>> Martha >>>>>> >>>>>> On Monday, January 6, 2020 at 10:41:30 AM UTC-8, Adam Cox wrote: >>>>>>> >>>>>>> Hi Martha, these libssl libraries are the root of the problem (at >>>>>>> least, when I tracked it down this is exactly where I ended up). As to >>>>>>> your >>>>>>> steps above, you do need to restart apache after every change to the >>>>>>> virtual environment. Can you confirm that you did that right after you >>>>>>> made >>>>>>> the first change to the psycopg2 library? I'm not able to look at my >>>>>>> own >>>>>>> installations at the moment, but I have never had to use that >>>>>>> WSGIApplicationGroup parameter. I was able to fix it by just updating >>>>>>> psycopg2 (I can't remember of -binary was in use for me or not). I'll >>>>>>> let >>>>>>> you know when I get a chance to look. >>>>>>> >>>>>>> Adam >>>>>>> >>>>>>> On Sun, Jan 5, 2020 at 7:44 PM Martha S <marth...@lacity.org> wrote: >>>>>>> >>>>>>>> I've found one more item in the psycopg 2.8.4 docs. Should this be >>>>>>>> my next step? >>>>>>>> >>>>>>>> Warning >>>>>>>> >>>>>>>> >>>>>>>> The psycopg2 wheel package comes packaged, among the others, with >>>>>>>> its own libssl binary. This may create conflicts with other >>>>>>>> extension modules binding with libssl as well, for instance with >>>>>>>> the Python ssl >>>>>>>> <https://docs.python.org/3/library/ssl.html#module-ssl> module: in >>>>>>>> some cases, under concurrency, the interaction between the two >>>>>>>> libraries >>>>>>>> may result in a segfault. In case of doubts you are advised to use a >>>>>>>> package built from source. >>>>>>>> >>>>>>>> >>>>>>>> On Sunday, January 5, 2020 at 1:30:56 PM UTC-8, Martha S wrote: >>>>>>>>> >>>>>>>>> Adam, >>>>>>>>> >>>>>>>>> Unfortunately, this did not repair the startup error. Once I >>>>>>>>> created a new virtual environment, installed arches, >>>>>>>>> uninstalled/re-installed psycopg2, corrected a directory oversight in >>>>>>>>> my >>>>>>>>> config file, and pointed to the new environment in apache2, I still >>>>>>>>> got the >>>>>>>>> following in my apache2 error.log and no coredump: >>>>>>>>> >>>>>>>>> [Sun Jan 05 19:58:35.485867 2020] [mpm_event:notice] [pid >>>>>>>>> 37579:tid 139672454474688] AH00491: caught SIGTERM, shutting down >>>>>>>>> >>>>>>>>> [Sun Jan 05 19:58:35.636147 2020] [mpm_event:notice] [pid 1993:tid >>>>>>>>> 140438837296064] AH00489: Apache/2.4.29 (Ubuntu) OpenSSL/1.1.1 >>>>>>>>> mod_wsgi/4.5.17 Python/2.7 configured -- resuming normal operations >>>>>>>>> >>>>>>>>> [Sun Jan 05 19:58:35.636244 2020] [core:notice] [pid 1993:tid >>>>>>>>> 140438837296064] AH00094: Command line: '/usr/sbin/apache2' >>>>>>>>> >>>>>>>>> [Sun Jan 05 19:58:49.346899 2020] [wsgi:error] [pid 2006:tid >>>>>>>>> 140438713235200] [client 161.149.102.22:2459] >>>>>>>>> wsgi:error 'arches': /Projects/ssl/hpla/hpla/wsgi.py >>>>>>>>> >>>>>>>>> [Sun Jan 05 19:58:49.650548 2020] [core:notice] [pid 1993:tid >>>>>>>>> 140438837296064] AH00051: child pid 2002 exit signal Segmentation >>>>>>>>> fault >>>>>>>>> (11), possible coredump in /etc/apache2 >>>>>>>>> >>>>>>>>> [Sun Jan 05 19:58:50.731314 2020] [wsgi:error] [pid 2004:tid >>>>>>>>> 140438585472768] [client 161.149.102.22:2460] Truncated or >>>>>>>>> oversized response headers received from daemon process 'arches': >>>>>>>>> /Projects/ssl/hpla/hpla/wsgi.py, referer: >>>>>>>>> https://historicplacesla.lacity.org/ >>>>>>>>> >>>>>>>>> >>>>>>>>> I went back to my new environment to double-check psycopg2 and >>>>>>>>> found: >>>>>>>>> >>>>>>>>> (ENV)hpla:/Projects/ssl$ pip freeze | grep psycopg2 >>>>>>>>> >>>>>>>>> DEPRECATION: Python 2.7 nag >>>>>>>>> >>>>>>>>> psycopg2==2.8.3 >>>>>>>>> >>>>>>>>> psycopg2-binary==2.7.4 >>>>>>>>> >>>>>>>>> I figured the binary was the problem and uninstalled that. Nope. >>>>>>>>> So I uninstalled/reinstalled psycopg2 and see that psycopg2==2.8.3 >>>>>>>>> and no binary is listed. Nope. >>>>>>>>> Just because, I restarted apache2. Nope. The error regarding >>>>>>>>> truncated or oversized response headers remains and there is no >>>>>>>>> coredump. >>>>>>>>> I found a suggestion online that I should add >>>>>>>>> WSGIApplicationGroup %{GLOBAL} to my config file. Nope. >>>>>>>>> >>>>>>>>> I see other suggestions, like switching from the system mod-wsgi >>>>>>>>> to the Python package and the psycopg2-binary package, but still hope >>>>>>>>> for >>>>>>>>> further guidance here from anyone successfully serving arches with >>>>>>>>> HTTPS on >>>>>>>>> Ubuntu 18.04. Is mod-wsgi 4.5.17 a problem? >>>>>>>>> >>>>>>>>> Thanks, >>>>>>>>> Martha >>>>>>>>> >>>>>>>>> >>>>>>>>> On Saturday, December 28, 2019 at 6:38:02 AM UTC-8, Adam Cox wrote: >>>>>>>>>> >>>>>>>>>> Hi Martha, the daemon process error has to do with your config >>>>>>>>>> file, you'll just need to make sure the arches process is created >>>>>>>>>> before >>>>>>>>>> you reference it. >>>>>>>>>> >>>>>>>>>> The other issue, the truncated........ Is much trickier, but has >>>>>>>>>> to do specifically with the version of psycopg2 (a python library in >>>>>>>>>> your >>>>>>>>>> virtual environment) that you have installed. I put this comment on >>>>>>>>>> a PR a >>>>>>>>>> long time ago, which gives a little context: >>>>>>>>>> https://github.com/archesproject/arches/pull/4965#issuecomment-521806491 >>>>>>>>>> >>>>>>>>>> Ultimately, I recommend creating a new virtual environment, pip >>>>>>>>>> installing everything as normal, then >>>>>>>>>> >>>>>>>>>> Pip uninstall psycopg2 >>>>>>>>>> Pip install psycopg2==2.8.3 >>>>>>>>>> >>>>>>>>>> You can then point the apache configs at the new environment and >>>>>>>>>> it should solve this problem. >>>>>>>>>> >>>>>>>>>> Good luck, >>>>>>>>>> Adam >>>>>>>>>> >>>>>>>>>> On Fri, Dec 27, 2019, 6:38 PM Martha S <marth...@lacity.org> >>>>>>>>>> wrote: >>>>>>>>>> >>>>>>>>>>> I feel I'm sooo close to implementing secured access to Arches >>>>>>>>>>> and need a little help to get over the top, please. I'm running >>>>>>>>>>> Ubuntu >>>>>>>>>>> 18.04. My certificate is in and working. >>>>>>>>>>> >>>>>>>>>>> I tried making a copy of 000-default.conf (with --preserve=all), >>>>>>>>>>> changing the port to 443, and adding these lines as directed by the >>>>>>>>>>> CA: >>>>>>>>>>> >>>>>>>>>>> ServerName historicplacesla.lacity.org >>>>>>>>>>> SSLEngine on >>>>>>>>>>> >>>>>>>>>>> SSLCertificateFile XXX.crt >>>>>>>>>>> SSLCertificateKeyFile XXX.key >>>>>>>>>>> >>>>>>>>>>> When I tried to restart Apache2 after enabling SSL, the >>>>>>>>>>> complaint was 'Name duplicates previous WSGI daemon definition', so >>>>>>>>>>> I >>>>>>>>>>> commented it out in default-ssl.conf and tried again. Apache >>>>>>>>>>> started, but I >>>>>>>>>>> still can't launch Arches. The browser gets an error 500 and the >>>>>>>>>>> Arches >>>>>>>>>>> error.log has the entry: >>>>>>>>>>> >>>>>>>>>>> 'Daemon process called 'arches' cannot be accessed by this WSGI >>>>>>>>>>> application: /Projects/prod/hpla/hpla/wsgi.py, referer: >>>>>>>>>>> https://historicplacesla.lacity.org/arches.' >>>>>>>>>>> >>>>>>>>>>> I get the same error if I try with http. My next effort was to >>>>>>>>>>> rename my 000-default.conf and uncomment the line in >>>>>>>>>>> default-ssl.conf. Now, >>>>>>>>>>> in addition to the error above, I get >>>>>>>>>>> >>>>>>>>>>> '[wsgi:error] Truncated or oversized response headers received >>>>>>>>>>> from daemon process 'arches': /Projects/prod/hpla/hpla/wsgi.py >>>>>>>>>>> [core:notice] AH00051: child pid 83357 exit signal Segmentation >>>>>>>>>>> fault (11), possible coredump in /etc/apache2 >>>>>>>>>>> >>>>>>>>>>> There is no new file, dump or otherwise in /etc/apache2. >>>>>>>>>>> >>>>>>>>>>> For the moment, I've disabled SSL, renamed the two configuration >>>>>>>>>>> files so only 000-default.conf is loaded, restarted Apache2 and >>>>>>>>>>> things are >>>>>>>>>>> running as usual. I get that it is at least partly an access rights >>>>>>>>>>> issue. >>>>>>>>>>> historicplacesla.lacity.org has been added to ALLOWED_HOSTS in >>>>>>>>>>> settings.py. >>>>>>>>>>> >>>>>>>>>>> What do I still need to do to what to deal with these errors? >>>>>>>>>>> >>>>>>>>>>> Thanks so much and happy new year to all, >>>>>>>>>>> Martha >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> -- >>>>>>>>>>> -- To post, send email to arches...@googlegroups.com. To >>>>>>>>>>> unsubscribe, send email to arches...@googlegroups.com. For more >>>>>>>>>>> information, visit >>>>>>>>>>> https://groups.google.com/d/forum/archesproject?hl=en >>>>>>>>>>> --- >>>>>>>>>>> You received this message because you are subscribed to the >>>>>>>>>>> Google Groups "Arches Project" group. >>>>>>>>>>> To unsubscribe from this group and stop receiving emails from >>>>>>>>>>> it, send an email to arches...@googlegroups.com. >>>>>>>>>>> To view this discussion on the web visit >>>>>>>>>>> https://groups.google.com/d/msgid/archesproject/d5bebd17-14c3-4925-b5e7-ba5128d446d0%40googlegroups.com >>>>>>>>>>> >>>>>>>>>>> <https://groups.google.com/d/msgid/archesproject/d5bebd17-14c3-4925-b5e7-ba5128d446d0%40googlegroups.com?utm_medium=email&utm_source=footer> >>>>>>>>>>> . >>>>>>>>>>> >>>>>>>>>> -- >>>>>>>> -- To post, send email to arches...@googlegroups.com. To >>>>>>>> unsubscribe, send email to arches...@googlegroups.com. For more >>>>>>>> information, visit >>>>>>>> https://groups.google.com/d/forum/archesproject?hl=en >>>>>>>> --- >>>>>>>> You received this message because you are subscribed to the Google >>>>>>>> Groups "Arches Project" group. >>>>>>>> To unsubscribe from this group and stop receiving emails from it, >>>>>>>> send an email to arches...@googlegroups.com. >>>>>>>> To view this discussion on the web visit >>>>>>>> https://groups.google.com/d/msgid/archesproject/e14b6b6c-b617-4279-b059-bc3c941643c8%40googlegroups.com >>>>>>>> >>>>>>>> <https://groups.google.com/d/msgid/archesproject/e14b6b6c-b617-4279-b059-bc3c941643c8%40googlegroups.com?utm_medium=email&utm_source=footer> >>>>>>>> . >>>>>>>> >>>>>>> -- >>>>>> -- To post, send email to arches...@googlegroups.com. To >>>>>> unsubscribe, send email to arches...@googlegroups.com. For more >>>>>> information, visit >>>>>> https://groups.google.com/d/forum/archesproject?hl=en >>>>>> --- >>>>>> You received this message because you are subscribed to the Google >>>>>> Groups "Arches Project" group. >>>>>> To unsubscribe from this group and stop receiving emails from it, >>>>>> send an email to arches...@googlegroups.com. >>>>>> To view this discussion on the web visit >>>>>> https://groups.google.com/d/msgid/archesproject/d888b6f9-47f2-42ad-8f7b-ed6aa98724d2%40googlegroups.com >>>>>> >>>>>> <https://groups.google.com/d/msgid/archesproject/d888b6f9-47f2-42ad-8f7b-ed6aa98724d2%40googlegroups.com?utm_medium=email&utm_source=footer> >>>>>> . >>>>>> >>>>> -- >>> -- To post, send email to arches...@googlegroups.com. To unsubscribe, >>> send email to arches...@googlegroups.com. For more information, visit >>> https://groups.google.com/d/forum/archesproject?hl=en >>> --- >>> You received this message because you are subscribed to the Google >>> Groups "Arches Project" group. >>> To unsubscribe from this group and stop receiving emails from it, send >>> an email to arches...@googlegroups.com. >>> To view this discussion on the web visit >>> https://groups.google.com/d/msgid/archesproject/4c7b9b8e-857c-48b3-8bc0-fa09f848dd42%40googlegroups.com >>> >>> <https://groups.google.com/d/msgid/archesproject/4c7b9b8e-857c-48b3-8bc0-fa09f848dd42%40googlegroups.com?utm_medium=email&utm_source=footer> >>> . >>> >> -- -- To post, send email to archesproject@googlegroups.com. To unsubscribe, send email to archesproject+unsubscr...@googlegroups.com. For more information, visit https://groups.google.com/d/forum/archesproject?hl=en --- You received this message because you are subscribed to the Google Groups "Arches Project" group. To unsubscribe from this group and stop receiving emails from it, send an email to archesproject+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/archesproject/856c1a11-72b7-4558-96e4-6d177ca3979f%40googlegroups.com.
HPLA vs Adam version comparison.xlsx
Description: MS-Excel 2007 spreadsheet
