Re: Questions on Release Management Process
On Mon, Mar 21, 2022 at 9:18 PM Jun Omae wrote: > > On Tue, Mar 22, 2022 at 9:10 AM Mark Phippard wrote: > > > > This seems to be the main thing I am stuck on. I used Debian Buster so > > that it would just be Python 2.7. It also occurred to me I was seeing > > this using the release tarball so SWIG would not even be involved any > > more .. right? The bindings were generated when the last RM generated > > the tarball. > > > > Does this ring any bells to anyone about why this is invalid? > > > > File > > "/home/svnrm/subversion-1.14.2/subversion/bindings/swig/python/libsvn/core.py", > > line 144 > > def apr_initialize() -> "apr_status_t": > > ^ > > SyntaxError: invalid syntax > > > > I have tried searching for "->" but not coming up with anything. It > > does seem vaguely familiar but I think I am remembering the similar > > Ruby operator that was deprecated. It seems like we are missing > > something obvious else this would have shown up when people signed the > > last release. > > > > Mark > > The swig-py for Python 2 depends "swig -python -classic". However, > -classic option is no longer supported since SWIG 4. SWIG 3 should be > used for 1.14.x releases to support Python 2.7 (swig3.0 package is > able to use in Debian Buster). I went looking for this information in subversion/bindings/swig/INSTALL and didn't see it the first time around, but it turns out it was there. Hopefully r1899118 and r1899120 make things a bit more clear... Nathan
Re: Questions on Release Management Process
Hi, On Tue, Mar 22, 2022 at 9:10 AM Mark Phippard wrote: > > On Mon, Mar 21, 2022 at 4:59 PM Stefan Sperling wrote: > > > > On Mon, Mar 21, 2022 at 04:46:55PM -0400, Mark Phippard wrote: > > > On Mon, Mar 21, 2022 at 4:31 PM Stefan Sperling wrote: > > > > This might be a swig problem? Perhaps the version of swig and the > > > > version of python don't work well together? Not sure. > > > > > > HACKING implies I should use the version that the clean room > > > environment installs and not what the OS installs so I ran: > > > > > > ./configure --with-swig=/opt/svnrm/prefix > > > > > > This could be an example where the docs are just not explicit enough > > > and I was overthinking. Now that I think about it maybe the problem is > > > that this is not found when I run the tests. Perhaps I should use the > > > OS SWIG for building and testing? Makes me wonder how the clean room > > > version is ever used though? > > > > While the clean-room env includes swig, it does not include python. > > So python comes from the system. And, as far as I understand, python > > is complaining about a syntax error in the code generated by swig. > > > > I would try to find out whether the clean-room swig expects a particular > > range of python versions. Maybe you can use a corresponding version of > > python in your docker image. > > > > What might also help is upgrading swig to a newer version. If upgrading > > tools in the clean-room env fixes such problems then we should upgrade them. > > This seems to be the main thing I am stuck on. I used Debian Buster so > that it would just be Python 2.7. It also occurred to me I was seeing > this using the release tarball so SWIG would not even be involved any > more .. right? The bindings were generated when the last RM generated > the tarball. > > Does this ring any bells to anyone about why this is invalid? > > File > "/home/svnrm/subversion-1.14.2/subversion/bindings/swig/python/libsvn/core.py", > line 144 > def apr_initialize() -> "apr_status_t": > ^ > SyntaxError: invalid syntax > > I have tried searching for "->" but not coming up with anything. It > does seem vaguely familiar but I think I am remembering the similar > Ruby operator that was deprecated. It seems like we are missing > something obvious else this would have shown up when people signed the > last release. > > Mark The swig-py for Python 2 depends "swig -python -classic". However, -classic option is no longer supported since SWIG 4. SWIG 3 should be used for 1.14.x releases to support Python 2.7 (swig3.0 package is able to use in Debian Buster). In fact, source files bundled in subversion 1.14.1 archive are generated by SWIG 3.0.12. [[[ $ tar xaf ~/arc/subversion-1.14.1.tar.bz2 -O subversion-1.14.1/subversion/bindings/swig/python/core.py | head -5 # This file was automatically generated by SWIG (http://www.swig.org). # Version 3.0.12 # # Do not make changes to this file unless you know what you are doing--modify # the SWIG interface file instead. $ tar xaf ~/arc/subversion-1.14.1.tar.bz2 -O subversion-1.14.1/subversion/bindings/swig/python/core.c | head -5 /* * This file was automatically generated by SWIG (http://www.swig.org). * Version 3.0.12 * * This file is not intended to be easily readable and contains a number of ]]] -- Jun Omae (大前 潤)
Re: Questions on Release Management Process
> On Mar 21, 2022, at 8:10 PM, Mark Phippard wrote: > > On Mon, Mar 21, 2022 at 4:59 PM Stefan Sperling wrote: >> >>> On Mon, Mar 21, 2022 at 04:46:55PM -0400, Mark Phippard wrote: >>> On Mon, Mar 21, 2022 at 4:31 PM Stefan Sperling wrote: This might be a swig problem? Perhaps the version of swig and the version of python don't work well together? Not sure. >>> >>> HACKING implies I should use the version that the clean room >>> environment installs and not what the OS installs so I ran: >>> >>> ./configure --with-swig=/opt/svnrm/prefix >>> >>> This could be an example where the docs are just not explicit enough >>> and I was overthinking. Now that I think about it maybe the problem is >>> that this is not found when I run the tests. Perhaps I should use the >>> OS SWIG for building and testing? Makes me wonder how the clean room >>> version is ever used though? >> >> While the clean-room env includes swig, it does not include python. >> So python comes from the system. And, as far as I understand, python >> is complaining about a syntax error in the code generated by swig. >> >> I would try to find out whether the clean-room swig expects a particular >> range of python versions. Maybe you can use a corresponding version of >> python in your docker image. >> >> What might also help is upgrading swig to a newer version. If upgrading >> tools in the clean-room env fixes such problems then we should upgrade them. > > This seems to be the main thing I am stuck on. I used Debian Buster so > that it would just be Python 2.7. It also occurred to me I was seeing > this using the release tarball so SWIG would not even be involved any > more .. right? The bindings were generated when the last RM generated > the tarball. > > Does this ring any bells to anyone about why this is invalid? > > File > "/home/svnrm/subversion-1.14.2/subversion/bindings/swig/python/libsvn/core.py", > line 144 >def apr_initialize() -> "apr_status_t": > ^ > SyntaxError: invalid syntax > > I have tried searching for "->" but not coming up with anything. It > does seem vaguely familiar but I think I am remembering the similar > Ruby operator that was deprecated. It seems like we are missing > something obvious else this would have shown up when people signed the > last release. Found this explanation https://svn.haxx.se/users/archive-2020-06/0005.shtml Mark
Re: Questions on Release Management Process
On Mon, Mar 21, 2022 at 4:59 PM Stefan Sperling wrote: > > On Mon, Mar 21, 2022 at 04:46:55PM -0400, Mark Phippard wrote: > > On Mon, Mar 21, 2022 at 4:31 PM Stefan Sperling wrote: > > > This might be a swig problem? Perhaps the version of swig and the > > > version of python don't work well together? Not sure. > > > > HACKING implies I should use the version that the clean room > > environment installs and not what the OS installs so I ran: > > > > ./configure --with-swig=/opt/svnrm/prefix > > > > This could be an example where the docs are just not explicit enough > > and I was overthinking. Now that I think about it maybe the problem is > > that this is not found when I run the tests. Perhaps I should use the > > OS SWIG for building and testing? Makes me wonder how the clean room > > version is ever used though? > > While the clean-room env includes swig, it does not include python. > So python comes from the system. And, as far as I understand, python > is complaining about a syntax error in the code generated by swig. > > I would try to find out whether the clean-room swig expects a particular > range of python versions. Maybe you can use a corresponding version of > python in your docker image. > > What might also help is upgrading swig to a newer version. If upgrading > tools in the clean-room env fixes such problems then we should upgrade them. This seems to be the main thing I am stuck on. I used Debian Buster so that it would just be Python 2.7. It also occurred to me I was seeing this using the release tarball so SWIG would not even be involved any more .. right? The bindings were generated when the last RM generated the tarball. Does this ring any bells to anyone about why this is invalid? File "/home/svnrm/subversion-1.14.2/subversion/bindings/swig/python/libsvn/core.py", line 144 def apr_initialize() -> "apr_status_t": ^ SyntaxError: invalid syntax I have tried searching for "->" but not coming up with anything. It does seem vaguely familiar but I think I am remembering the similar Ruby operator that was deprecated. It seems like we are missing something obvious else this would have shown up when people signed the last release. Mark
Re: Questions on Release Management Process
On Mon, Mar 21, 2022 at 04:46:55PM -0400, Mark Phippard wrote: > On Mon, Mar 21, 2022 at 4:31 PM Stefan Sperling wrote: > > This might be a swig problem? Perhaps the version of swig and the > > version of python don't work well together? Not sure. > > HACKING implies I should use the version that the clean room > environment installs and not what the OS installs so I ran: > > ./configure --with-swig=/opt/svnrm/prefix > > This could be an example where the docs are just not explicit enough > and I was overthinking. Now that I think about it maybe the problem is > that this is not found when I run the tests. Perhaps I should use the > OS SWIG for building and testing? Makes me wonder how the clean room > version is ever used though? While the clean-room env includes swig, it does not include python. So python comes from the system. And, as far as I understand, python is complaining about a syntax error in the code generated by swig. I would try to find out whether the clean-room swig expects a particular range of python versions. Maybe you can use a corresponding version of python in your docker image. What might also help is upgrading swig to a newer version. If upgrading tools in the clean-room env fixes such problems then we should upgrade them.
Re: Questions on Release Management Process
On Mon, Mar 21, 2022 at 4:31 PM Stefan Sperling wrote: > > On Mon, Mar 21, 2022 at 12:44:44PM -0400, Mark Phippard wrote: > > I have been toying with the idea of volunteering to be Release > > Manager. I am not ready to do so yet as I have run into numerous > > problems. > > > > The docs are decent but they are definitely written to an audience > > that knows a lot of things that I do not and it feels like a lot of > > knowledge is just assumed. Keep in mind I am a MacOS/Windows user with > > no Python experience. My main fear is running into a problem during > > the process where the script needs to be fixed and not being able to > > do it. I do not have Mike/Paul on speed dial to help me any more. > > This is good news, thank you Mark! > > I will try to help as best as I can. Thanks > > It would probably be helpful if someone saved a start to finish log of > > all of the commands they ran and from what directory when they were > > RM. > > I do not have such a transcript. However, I have been following the > docs almost to the letter for the last few releases. I have also > tried to keep the docs updated if something was no longer working. > > > The release.py script implies you can/should run the process from > > people.apache.org. I have no idea what that is, how I would access it, > > or what the experience would be like once I was there. I imagine you > > SSH into the box and I would assume you would do everything in your > > home folder if you did this, but the docs use /opt/svnrm as the > > example and it seems weird you would use a folder like this on a > > shared system. > > I believe people.apache.org no longer offers shell access. That is sort of good news in that it simplifies the options > > At the moment, I am not considering this option. Since I do not have a > > Linux machine available, I am trying to use Docker to build a Debian > > image to run the process. > > Using a virtual machine or container sounds like a good solution to me. > > > I am running everything as root. I could change it to a user account > > if it would fix my problems. > > I was always running commands as my own user. There should be no need > to use the root account for anything release.py is doing. I was only using root because that is where you end up by default when using Docker. I am glad I mentioned it though since it explains the test failure. I will do the extra work in my Dockerfile to create a user account and run with that. > > I should note up front that I do not have a GPG key so I would need to > > create a new one and "do something" before I begin. The key I used to > > use when signing releases was lost about 4 computers ago. I do note > > that HACKING mentions if you use people.a.o you do not do the signing > > from there. So I guess you use scp or something to pull the files down > > to sign them locally and then push your signatures back up to > > people.a.o after? > > > > Problem 1: Rolling the tarballs > > > > The process creates the tarballs but fails near the end. It looks GPG > > related? > > >INFO:root:Building Unix tarballs > >INFO:root:Moving artifacts and calculating checksums > >Traceback (most recent call last): > > File "trunk/tools/dist/release.py", line 1916, in > >main() > > File "trunk/tools/dist/release.py", line 1912, in main > >args.func(args) > > File "trunk/tools/dist/release.py", line 983, in roll_tarballs > >download_file(KEYS, filepath, None) > > File "trunk/tools/dist/release.py", line 289, in download_file > >response = urlopen(url) > > File "/usr/lib/python2.7/urllib2.py", line 154, in urlopen > >return opener.open(url, data, timeout) > > File "/usr/lib/python2.7/urllib2.py", line 435, in open > >response = meth(req, response) > > File "/usr/lib/python2.7/urllib2.py", line 548, in http_response > >'http', request, response, code, msg, hdrs) > > File "/usr/lib/python2.7/urllib2.py", line 473, in error > >return self._call_chain(*args) > > File "/usr/lib/python2.7/urllib2.py", line 407, in _call_chain > >result = func(*args) > > File "/usr/lib/python2.7/urllib2.py", line 556, in http_error_default > >raise HTTPError(req.get_full_url(), code, msg, hdrs, fp) > >urllib2.HTTPError: HTTP Error 404: Not Found > > It seems ASF have removed the KEYS file our script is trying to fetch. > See http://people.apache.org/keys/ where it says "Project group files are > no longer created." > > It looks like what the script wants to do here is obtain a copy of > the Subversion project's KEYS file and store it along with release > artifacts. If we want to keep doing this we will have to maintain > our own KEYS file on the website, I suppose. Otherwise, we could > decide to no longer provide such a file and remove relevant code > from the script. Not sure what is better. Since I do not know what this was used for, maybe someone can help ge
Re: Questions on Release Management Process
On Mon, Mar 21, 2022 at 12:44:44PM -0400, Mark Phippard wrote: > I have been toying with the idea of volunteering to be Release > Manager. I am not ready to do so yet as I have run into numerous > problems. > > The docs are decent but they are definitely written to an audience > that knows a lot of things that I do not and it feels like a lot of > knowledge is just assumed. Keep in mind I am a MacOS/Windows user with > no Python experience. My main fear is running into a problem during > the process where the script needs to be fixed and not being able to > do it. I do not have Mike/Paul on speed dial to help me any more. This is good news, thank you Mark! I will try to help as best as I can. > It would probably be helpful if someone saved a start to finish log of > all of the commands they ran and from what directory when they were > RM. I do not have such a transcript. However, I have been following the docs almost to the letter for the last few releases. I have also tried to keep the docs updated if something was no longer working. > The release.py script implies you can/should run the process from > people.apache.org. I have no idea what that is, how I would access it, > or what the experience would be like once I was there. I imagine you > SSH into the box and I would assume you would do everything in your > home folder if you did this, but the docs use /opt/svnrm as the > example and it seems weird you would use a folder like this on a > shared system. I believe people.apache.org no longer offers shell access. > At the moment, I am not considering this option. Since I do not have a > Linux machine available, I am trying to use Docker to build a Debian > image to run the process. Using a virtual machine or container sounds like a good solution to me. > I am running everything as root. I could change it to a user account > if it would fix my problems. I was always running commands as my own user. There should be no need to use the root account for anything release.py is doing. > I should note up front that I do not have a GPG key so I would need to > create a new one and "do something" before I begin. The key I used to > use when signing releases was lost about 4 computers ago. I do note > that HACKING mentions if you use people.a.o you do not do the signing > from there. So I guess you use scp or something to pull the files down > to sign them locally and then push your signatures back up to > people.a.o after? > > Problem 1: Rolling the tarballs > > The process creates the tarballs but fails near the end. It looks GPG related? >INFO:root:Building Unix tarballs >INFO:root:Moving artifacts and calculating checksums >Traceback (most recent call last): > File "trunk/tools/dist/release.py", line 1916, in >main() > File "trunk/tools/dist/release.py", line 1912, in main >args.func(args) > File "trunk/tools/dist/release.py", line 983, in roll_tarballs >download_file(KEYS, filepath, None) > File "trunk/tools/dist/release.py", line 289, in download_file >response = urlopen(url) > File "/usr/lib/python2.7/urllib2.py", line 154, in urlopen >return opener.open(url, data, timeout) > File "/usr/lib/python2.7/urllib2.py", line 435, in open >response = meth(req, response) > File "/usr/lib/python2.7/urllib2.py", line 548, in http_response >'http', request, response, code, msg, hdrs) > File "/usr/lib/python2.7/urllib2.py", line 473, in error >return self._call_chain(*args) > File "/usr/lib/python2.7/urllib2.py", line 407, in _call_chain >result = func(*args) > File "/usr/lib/python2.7/urllib2.py", line 556, in http_error_default >raise HTTPError(req.get_full_url(), code, msg, hdrs, fp) >urllib2.HTTPError: HTTP Error 404: Not Found It seems ASF have removed the KEYS file our script is trying to fetch. See http://people.apache.org/keys/ where it says "Project group files are no longer created." It looks like what the script wants to do here is obtain a copy of the Subversion project's KEYS file and store it along with release artifacts. If we want to keep doing this we will have to maintain our own KEYS file on the website, I suppose. Otherwise, we could decide to no longer provide such a file and remove relevant code from the script. Not sure what is better. > Problem 2: Test failure > > One of the lock_tests consistently fails. > > I have confirmed I get this failure using the 1.14.1 tarball so I > assume it is related to running the tests inside Docker. If this ends > up being the only problem, I could probably just safely ignore it and > let the others that sign the release verify the tests all work for > them. The failing test has a comment on top which explains why :) /* XXX NOTE: This test will fail on most Unix-like systems when run as the root user, because flock() will ignore file permissions. */ > Problem 3: Python bindings tests fail
Questions on Release Management Process
I have been toying with the idea of volunteering to be Release Manager. I am not ready to do so yet as I have run into numerous problems. The docs are decent but they are definitely written to an audience that knows a lot of things that I do not and it feels like a lot of knowledge is just assumed. Keep in mind I am a MacOS/Windows user with no Python experience. My main fear is running into a problem during the process where the script needs to be fixed and not being able to do it. I do not have Mike/Paul on speed dial to help me any more. It would probably be helpful if someone saved a start to finish log of all of the commands they ran and from what directory when they were RM. The release.py script implies you can/should run the process from people.apache.org. I have no idea what that is, how I would access it, or what the experience would be like once I was there. I imagine you SSH into the box and I would assume you would do everything in your home folder if you did this, but the docs use /opt/svnrm as the example and it seems weird you would use a folder like this on a shared system. At the moment, I am not considering this option. Since I do not have a Linux machine available, I am trying to use Docker to build a Debian image to run the process. I am running everything as root. I could change it to a user account if it would fix my problems. I should note up front that I do not have a GPG key so I would need to create a new one and "do something" before I begin. The key I used to use when signing releases was lost about 4 computers ago. I do note that HACKING mentions if you use people.a.o you do not do the signing from there. So I guess you use scp or something to pull the files down to sign them locally and then push your signatures back up to people.a.o after? Problem 1: Rolling the tarballs The process creates the tarballs but fails near the end. It looks GPG related? /opt/svnrm# python trunk/tools/dist/release.py build-env 1.14.0 INFO:root:Creating release environment INFO:root:Fetching autoconf-2.69 INFO:root:Building autoconf INFO:root:Fetching libtool-2.4.6 INFO:root:Building libtool ar: `u' modifier ignored since `D' is the default (see `U') ar: `u' modifier ignored since `D' is the default (see `U') INFO:root:Fetching swig-3.0.12 INFO:root:Building swig root@40beb55e988a:/opt/svnrm# python trunk/tools/dist/release.py roll 1.14.2 1899108 INFO:root:Rolling release 1.14.2 from branch branches/1.14.x@1899108 WARNING:root:Copyright year in NOTICE is not the current year WARNING:root:Copyright year in subversion/libsvn_subr/version.c is not the current year WARNING:root:CHANGES has unmerged revisions: r1886201 r1886461 r1898187 INFO:root:Preparing working copy source INFO:root:Building Windows tarballs ../svnadmin/svnadmin.c:2935: warning: Although being used in a format string position, the msgid is not a valid C format string. Reason: The string ends in the middle of a directive. ../libsvn_fs_fs/verify.c:696: warning: Although being used in a format string position, the msgid is not a valid C format string. Reason: The string ends in the middle of a directive. INFO:root:Building Unix tarballs INFO:root:Moving artifacts and calculating checksums Traceback (most recent call last): File "trunk/tools/dist/release.py", line 1916, in main() File "trunk/tools/dist/release.py", line 1912, in main args.func(args) File "trunk/tools/dist/release.py", line 983, in roll_tarballs download_file(KEYS, filepath, None) File "trunk/tools/dist/release.py", line 289, in download_file response = urlopen(url) File "/usr/lib/python2.7/urllib2.py", line 154, in urlopen return opener.open(url, data, timeout) File "/usr/lib/python2.7/urllib2.py", line 435, in open response = meth(req, response) File "/usr/lib/python2.7/urllib2.py", line 548, in http_response 'http', request, response, code, msg, hdrs) File "/usr/lib/python2.7/urllib2.py", line 473, in error return self._call_chain(*args) File "/usr/lib/python2.7/urllib2.py", line 407, in _call_chain result = func(*args) File "/usr/lib/python2.7/urllib2.py", line 556, in http_error_default raise HTTPError(req.get_full_url(), code, msg, hdrs, fp) urllib2.HTTPError: HTTP Error 404: Not Found Problem 2: Test failure One of the lock_tests consistently fails. I have confirmed I get this failure using the 1.14.1 tarball so I assume it is related to running the tests inside Docker. If this ends up being the only problem, I could probably just safely ignore it and let the others that sign the release verify the tests all work for them. At least one test FAILED, checking /opt/svnrm/subversion-1.14.1/tests.log FAIL: locks-test 14: lock/unlock when 'write-lock' couldn't be obtained Summary of test results: 2518 tests PASSED 163 tests SKIPPED 80 tests XFAILED (17 WORK-IN-PRO
