Re: [Savannah-users] Unable to upload release tarball to Download Area
Asher Gordon wrote: > Bob Proulx wrote: > > I have gained an understanding now. I know what needs to be done to > > solve the problem. At root cause I had broken one of the cron scripts > > in moving it from system to system. It wasn't updating a timestamp > > which caused a monitoring script in the redirector to think the mirror > > was out of sync when it was not. It needs to look not at the absolute > > age but the relative age however. The reference should be the > > upstream source, which is known to the redirector, not the time now. > > And there needs to be a fallback when there are no mirrors. > > That's pretty complicated! I also noticed that many of the directories > in https://download.savannah.nongnu.org/releases/ are empty. Is that > part of the migration? That's just normal. Many projects enable all of the checkboxes which includes a download area but then never do anything with them. Bob
Re: [Savannah-users] Unable to upload release tarball to Download Area
Bob Proulx wrote: > I have gained an understanding now. I know what needs to be done to > solve the problem. At root cause I had broken one of the cron scripts > in moving it from system to system. It wasn't updating a timestamp > which caused a monitoring script in the redirector to think the mirror > was out of sync when it was not. It needs to look not at the absolute > age but the relative age however. The reference should be the > upstream source, which is known to the redirector, not the time now. > And there needs to be a fallback when there are no mirrors. That's pretty complicated! I also noticed that many of the directories in https://download.savannah.nongnu.org/releases/ are empty. Is that part of the migration? > I remember > there being new restrictions on file deletion that didn't exist > before. Therefore sshfs might just not be allowed through the command > filter at this time. That's too bad. Is it possible it will be in the future? Or if not, is there another way to perform more complex tasks in directories than simply uploading files? Specifically, I have in mind creating symlinks. I don't believe that is possible with scp. It would also be nice to be able to delete files in case I accidentally upload the wrong file or something. > As you might imagine it isn't allowed to run arbitrary commands. > Therefore all of the ssh commands are filtered through an sv_membersh > filter before being allowed through. Couldn't sv_membersh allow commands like mv, cp, rm, ln, and whatever sshfs and co. need, but make sure that they only operate on the project directory? Does each Savannah user get his/her own user/group? If not, I think that may be a good idea so you could run more commands, but only under the unprivileged user account. Of course, you should probably call the users something like sv-user-USER where USER is the Savannah username, otherwise a user named "root" would be very problematic. I hope my suggestions help. Asher
Re: [Savannah-users] Download mirror redirection is broken
Savannah Users, > > Unfortunately the mirror redirection for project downloads is broken > > today. > > This is back online now. Unfortunately without the mirror > redirection. This is okay for a while. Most importantly files from > the download areas are available again. And the redirector is online too. Everything should be running normally again now. > > This must be the result of the back-end storage change I made > > yesterday however I promise I thought I had run the regression tests > > after all was done and everything was working at that time. But it is > > definitely failing now. I am debugging the problem now and will get > > it back online as soon as possible. > > Actually FTR it doesn't seem to be related to the back-end change and > the regression suite had actually passed! This simply appears to be > coincidental breakage. Of which I am not quite at the root cause yet > but getting closer. It turned out to both be related and not related. I broke one of the cronjobs when migrating it off the old system to the new. It was a very simple one that did nothing but update a timestamp in the download area. Broken due to new permission requirements. Mirrors mirror everyting including that timestamp file. Everything worked initially because the previous copy of that file had a new enough timestamp in it. Then as time passed the redirector kept looking at that static timestamp being freshly mirrored and decided that the mirrors were too "old" by looking at the time now and deciding they must be out of sync therefore eventually dropping all of the mirrors from the mirror list. Without any mirrors in the list the redirectory failed to produce a valid URL. I fixed this so that even without mirrors it will now produce a valid URL. I added the local mirror fallback so there will always be a valid mirror regardless. I fixed the timestamp user so that it now has permissions and is updating. And with that things seem to be completely back online again. Bob
Re: [Savannah-users] Unable to upload release tarball to Download Area
Asher Gordon wrote: > > I am getting pretty close to having the mirror problem understood. At > > the moment my brains are leaking out my ears from trying to get my > > mind wrapped around it. > > Sounds painful! ;-) Pain is simply weakness leaving the body. Sometimes the soul tags along with it. :-) I have gained an understanding now. I know what needs to be done to solve the problem. At root cause I had broken one of the cron scripts in moving it from system to system. It wasn't updating a timestamp which caused a monitoring script in the redirector to think the mirror was out of sync when it was not. It needs to look not at the absolute age but the relative age however. The reference should be the upstream source, which is known to the redirector, not the time now. And there needs to be a fallback when there are no mirrors. I am a huge fan of continuous improvement. :-) > I am now having problems with sftp and sshfs. See below: > $ mkdir mnt > $ sshfs asd...@dl.sv.nongnu.org:/releases/c2py mnt > Enter passphrase for key '/home/asher/.ssh/id_rsa': > remote host has disconnected > $ sftp asd...@dl.sv.nongnu.org:/releases/c2py > Enter passphrase for key '/home/asher/.ssh/id_rsa': > Connection closed > (scp still works fine) I'll need to investigate. > Are these also related to the same bug? It would be nice to use > sshfs because I could easily manage the directory as if it were > local. This may or may not be related to a previous security vulnerability that Sylvain reported and fixed a couple of years ago which forced a loss of support for some file copy protocols. As such the documentation may not have caught up to the new reality. I remember there being new restrictions on file deletion that didn't exist before. Therefore sshfs might just not be allowed through the command filter at this time. As you might imagine it isn't allowed to run arbitrary commands. Therefore all of the ssh commands are filtered through an sv_membersh filter before being allowed through. > Also, rsync works if I do not use `-a'. However, if I do use `-a', > it gets stuck at "sending incremental file list" (you need `-v' to > see the message). Hopefully this information will help you fix the > bug(s). That one is definitely known due to the sv_membersh filter. I will add that as a note to the rsync method in the documentation. The example shown was without options. scp release.tar.gz y...@dl.sv.nongnu.org:/releases/project/ rsync release.tar.gz y...@dl.sv.nongnu.org:/releases/project/ sftp y...@dl.sv.nongnu.org:/releases/project/ # interactive mode However sshfs is documented as working. I'll give it a try and see what methods it is trying to use for file access. > Thank you very much for your great work on Savannah! Happy to help! Bob
Re: [Savannah-users] Unable to upload release tarball to Download Area
> I am getting pretty close to having the mirror problem understood. At > the moment my brains are leaking out my ears from trying to get my > mind wrapped around it. Sounds painful! ;-) I am now having problems with sftp and sshfs. See below: $ mkdir mnt $ sshfs asd...@dl.sv.nongnu.org:/releases/c2py mnt Enter passphrase for key '/home/asher/.ssh/id_rsa': remote host has disconnected $ sftp asd...@dl.sv.nongnu.org:/releases/c2py Enter passphrase for key '/home/asher/.ssh/id_rsa': Connection closed (scp still works fine) Are these also related to the same bug? It would be nice to use sshfs because I could easily manage the directory as if it were local. Also, rsync works if I do not use `-a'. However, if I do use `-a', it gets stuck at "sending incremental file list" (you need `-v' to see the message). Hopefully this information will help you fix the bug(s). Thank you very much for your great work on Savannah! Asher
Re: [Savannah-users] Download mirror redirection is broken
Savannah Users, > Unfortunately the mirror redirection for project downloads is broken > today. This is back online now. Unfortunately without the mirror redirection. This is okay for a while. Most importantly files from the download areas are available again. Bob P.S. > This must be the result of the back-end storage change I made > yesterday however I promise I thought I had run the regression tests > after all was done and everything was working at that time. But it is > definitely failing now. I am debugging the problem now and will get > it back online as soon as possible. Actually FTR it doesn't seem to be related to the back-end change and the regression suite had actually passed! This simply appears to be coincidental breakage. Of which I am not quite at the root cause yet but getting closer.
Re: [Savannah-users] Unable to upload release tarball to Download Area
Asher Gordon wrote: > Bob Proulx wrote: > > In the meantime I have created that directory for you. Asher you > > should be able to upload to it at this time. Please try it again and > > let us know if it is working for you or not. > > It is working now. Thank you, Jan and Bob, for you help. Awesome! :-) Now I just have three urgent bugs related to this that I need to fix. I am getting pretty close to having the mirror problem understood. At the moment my brains are leaking out my ears from trying to get my mind wrapped around it. Almost there. Soon. Bob
Re: [Savannah-users] Unable to upload release tarball to Download Area
Bob Proulx wrote: > In the meantime I have created that directory for you. Asher you > should be able to upload to it at this time. Please try it again and > let us know if it is working for you or not. It is working now. Thank you, Jan and Bob, for you help. Asher
Re: [Savannah-users] Unable to upload release tarball to Download Area
Hello Asher! And also welcome to Savannah.
Jan Owoc wrote:
> Asher Gordon wrote:
> > I'm very new to Savannah, and I recently added a new project, c2py
> https://savannah.nongnu.org/projects/c2py. I'm trying to upload a
> release tarball to the Download Area, but it is not working.
...
> If you go up a directory, you can see a list of projects that have
> folders, and c2py isn't on the list. It looks like you need to make
> that folder first, or ask a Savannah admin to do it for you.
Thanks Jan for the help! You are correct that the directory was
missing and causing the upload problem. This is due to a storage
back-end change made yesterday. The short answer is that the Savannah
web page UI checkbox is broken at the moment due to it. And another
problem too. I am working on it.
In the meantime I have created that directory for you. Asher you
should be able to upload to it at this time. Please try it again and
let us know if it is working for you or not.
> > scp c2py-0.0.1rc5.tar.gz{,.sig} asd...@dl.sv.nongnu.org:/releases/c2py/
> > Enter passphrase for key '/home/asher/.ssh/id_rsa':
> > scp: /releases/c2py/: No such file or directory
>
> The fact that you are getting the latter message means that everything
> is correctly set up with your SSH keys. Otherwise, the error would be
> something like "authentication failed".
Yes! :-)
> The "Download Area" link on the project page links to
> https://savannah.nongnu.org/files/?group=c2py which redirects to
> https://download.savannah.nongnu.org/releases/c2py/ but that page is
> a 404.
And that is bug problem for me number two to debug this morning. The
mirror redirection is dropping the "/releases" part of the URL from
the redirection and therefore the resulting URL is 404. This is
affecting everyone. I am trying to figure it out now.
In the meantime files that have been uploaded and mirrored out can be
reached by selecting a mirror manually. But of course for new files
they won't have propagated until the next mirror sync is able to copy
them out to any individual mirror site.
https://download.savannah.gnu.org/mirmon/savannah/
Bob
[Savannah-users] Download mirror redirection is broken
Savannah Users, Unfortunately the mirror redirection for project downloads is broken today. It is missing the "/releases" part from the URL at the redirection causing the resulting URL to be a 404 Not Found. The files are there but it is imposible to get to them at the moment. This must be the result of the back-end storage change I made yesterday however I promise I thought I had run the regression tests after all was done and everything was working at that time. But it is definitely failing now. I am debugging the problem now and will get it back online as soon as possible. In the meantime it is still possible to download files by selecting a mirror manually. Visit the mirmon page and select a mirror manually and one should be able to download any of the files. https://download.savannah.gnu.org/mirmon/savannah/ Bob
Re: [Savannah-users] Unable to upload release tarball to Download Area
On Wed, Apr 3, 2019 at 10:35 AM Jan Owoc wrote: > > [...] and try the command: > > scp c2py asd...@dl.sv.nongnu.org:/releases/c2py Sorry. You want to recursively copy the directory and the files in it, so you need a "-r" flag in there: scp -r c2py asd...@dl.sv.nongnu.org:/releases/c2py
Re: [Savannah-users] Unable to upload release tarball to Download Area
Hello Ahser, and welcome to Savannah.
On Wed, Apr 3, 2019 at 6:55 AM Asher Gordon wrote:
>
> I'm very new to Savannah, and I recently added a new project, c2py
> https://savannah.nongnu.org/projects/c2py. I'm trying to upload a release
> tarball to the Download Area, but it is not working. The "Download Area" link
> on the project page links to https://savannah.nongnu.org/files/?group=c2py
> which redirects to https://download.savannah.nongnu.org/releases/c2py/ but
> that page is a 404.
>
If you go up a directory, you can see a list of projects that have
folders, and c2py isn't on the list. It looks like you need to make
that folder first, or ask a Savannah admin to do it for you.
> scp c2py-0.0.1rc5.tar.gz{,.sig} asd...@dl.sv.nongnu.org:/releases/c2py/
> Enter passphrase for key '/home/asher/.ssh/id_rsa':
> scp: /releases/c2py/: No such file or directory
The fact that you are getting the latter message means that everything
is correctly set up with your SSH keys. Otherwise, the error would be
something like "authentication failed".
>From reading the scp documentation, it looks like you cannot copy
files into a directory that doesn't exist. However, you can copy an
entire directory over if it exists locally. Try to make a local folder
named "c2py", put the files that you want copied over into that
folder, and try the command:
scp c2py asd...@dl.sv.nongnu.org:/releases/c2py
Cheers,
Jan
Re: [Savannah-users] Unable to upload release tarball to Download Area
> I have tried uploading using scp as described here: https://savannah.gnu.orI > have uploaded my SSH and GPG public keys. > g/maintenance/DownloadArea/ It seems I accidentally pasted something in the middle of that URL. :P It was meant to read https://savannah.gnu.org/maintenance/DownloadArea/
[Savannah-users] Unable to upload release tarball to Download Area
Hello,
I'm very new to Savannah, and I recently added a new project, c2py
https://savannah.nongnu.org/projects/c2py. I'm trying to upload a release
tarball to the Download Area, but it is not working. The "Download Area" link
on the project page links to https://savannah.nongnu.org/files/?group=c2py
which redirects to https://download.savannah.nongnu.org/releases/c2py/ but that
page is a 404.
I have tried uploading using scp as described here: https://savannah.gnu.orI
have uploaded my SSH and GPG public keys.
g/maintenance/DownloadArea/
scp c2py-0.0.1rc5.tar.gz asd...@dl.sv.nongnu.org:/releases/c2py/
Enter passphrase for key '/home/asher/.ssh/id_rsa':
scp: /releases/c2py/: Is a directory
The "Is a directory" message indicates (somewhat counter-intuitively) that
/releases/c2py/ does not exist on the Savannah server. If I try uploading
multiple files at once (my tarball and its signature), it prints a more
intuitive message (since it expects the destination to be a directory with
multiple sources), but still does not work:
scp c2py-0.0.1rc5.tar.gz{,.sig} asd...@dl.sv.nongnu.org:/releases/c2py/
Enter passphrase for key '/home/asher/.ssh/id_rsa':
scp: /releases/c2py/: No such file or directory
Again, I'm a Savannah newbie, and I'm probably doing something obviously wrong.
Thanks for your help,
Asher
P.S. I have uploaded my SSH and GPG public keys to my account, if it helps.
