Re: Unable to open repository following reboot
Hi Les, Well that reboot was more of an adventure than I had anticipated. And in all the "adventure" we ended up rolling back to the configuration without quotas, but svn is working now. Thanks for all of your time and help. Pat Hi Les, On Wed, Dec 4, 2013 at 7:09 PM, Pat Haley wrote: One thing that didn't stand out in my original Email was the reason for the reboot. We turned quotas on. Would svn react poorly to this? Only on a write that exceeds quota. It is a generic linux box. However doing dmesg before and after generating an svn error doesn't show anything new. Can you run the svnadmin verify on the box with the drives? Maybe it is just a client/network/mount issue and the things that worked (ls, etc.) were running from cache. Can you unmount/mount or reboot a client? svnadmin verify does work on the box with the drives. # svnadmin verify /home/phaley/Papers/2011/ArpitVel/SvnPaper * Verified revision 0. * Verified revision 1. (and so on) I've scheduled a reboot on a client. I'll let you know how that works out. Thanks. -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=- Pat Haley Email: pha...@mit.edu Center for Ocean Engineering Phone: (617) 253-6824 Dept. of Mechanical EngineeringFax:(617) 253-8125 MIT, Room 5-213http://web.mit.edu/phaley/www/ 77 Massachusetts Avenue Cambridge, MA 02139-4301 -- -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=- Pat Haley Email: pha...@mit.edu Center for Ocean Engineering Phone: (617) 253-6824 Dept. of Mechanical EngineeringFax:(617) 253-8125 MIT, Room 5-213http://web.mit.edu/phaley/www/ 77 Massachusetts Avenue Cambridge, MA 02139-4301
Re: Unable to open repository following reboot
Hi Les, On Wed, Dec 4, 2013 at 7:09 PM, Pat Haley wrote: One thing that didn't stand out in my original Email was the reason for the reboot. We turned quotas on. Would svn react poorly to this? Only on a write that exceeds quota. It is a generic linux box. However doing dmesg before and after generating an svn error doesn't show anything new. Can you run the svnadmin verify on the box with the drives? Maybe it is just a client/network/mount issue and the things that worked (ls, etc.) were running from cache. Can you unmount/mount or reboot a client? svnadmin verify does work on the box with the drives. # svnadmin verify /home/phaley/Papers/2011/ArpitVel/SvnPaper * Verified revision 0. * Verified revision 1. (and so on) I've scheduled a reboot on a client. I'll let you know how that works out. Thanks. -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=- Pat Haley Email: pha...@mit.edu Center for Ocean Engineering Phone: (617) 253-6824 Dept. of Mechanical EngineeringFax:(617) 253-8125 MIT, Room 5-213http://web.mit.edu/phaley/www/ 77 Massachusetts Avenue Cambridge, MA 02139-4301
Re: Unable to open repository following reboot
Guten Tag Pat Haley,
am Donnerstag, 5. Dezember 2013 um 02:20 schrieben Sie:
> % file /home/phaley/Papers/2011/ArpitVel/SvnPaper/db/rep-cache.db
> /home/phaley/Papers/2011/ArpitVel/SvnPaper/db/rep-cache.db: SQLite 3.x
> database, user version 1
[...]
> lstat(".svn/log", 0x7fff48dcbec0) = -1 ENOENT (No such file or
> directory)
Can you access the rep-cache.db using a SQLite client? I would copy
the file from the NAS to some other client with some other filesystem
and try to open it there, from my understanding it shouldn't rely on
anything else from the repo and therefore should be usable with any
client which is able to open SQLite database, like DbVisualizer for
convenience.
Afterwards try the same with the file on the NAS using a shell with
the originally mentioned path from your above message and maybe even
on the NAS after the file was copied to another directory within the
same filesystem with the same quotas.
Mit freundlichen Grüßen,
Thorsten Schöning
--
Thorsten Schöning E-Mail:thorsten.schoen...@am-soft.de
AM-SoFT IT-Systeme http://www.AM-SoFT.de/
Telefon...05151- 9468- 55
Fax...05151- 9468- 88
Mobil..0178-8 9468- 04
AM-SoFT GmbH IT-Systeme, Brandenburger Str. 7c, 31789 Hameln
AG Hannover HRB 207 694 - Geschäftsführer: Andreas Muchow
Re: Unable to open repository following reboot
On Wed, Dec 4, 2013 at 7:09 PM, Pat Haley wrote: > >> One thing that didn't stand out in my original Email was the reason > for the reboot. We turned quotas on. Would svn react poorly > to this? Only on a write that exceeds quota. > It is a generic linux box. However doing dmesg before and > after generating an svn error doesn't show anything new. Can you run the svnadmin verify on the box with the drives? Maybe it is just a client/network/mount issue and the things that worked (ls, etc.) were running from cache. Can you unmount/mount or reboot a client? -- Les Mikesell lesmikes...@gmail.com
Re: Unable to open repository following reboot
On 12/4/2013 5:20 PM, Pat Haley wrote:
You can cause strace to write to a file with "-o myoutputfile"
So if you want to post I'd do:
strace -o myoutputfile -s 0 svnadmin verify
/home/phaley/Papers/2011/ArpitVel/SvnPaper
That line is complaining about a stale file handle:
fcntl(3, F_SETLK, {type=F_RDLCK, whence=SEEK_SET, start=1073741824,
len=1}) = -1 ESTALE (Stale NFS file handle)
write(2, "svnadmin: disk I/O error\n", 25) = 25
write(2, "svnadmin: disk I/O error\n", 25) = 25
exit_group(1) = ?
From a couple of lines above it looks like the file it's
complaining about is
/home/phaley/Papers/2011/ArpitVel/SvnPaper/db/rep-cache.db
however, I can use the "file" command successfully on that file
You say that the NAS server was rebooted. Were any of the machines
accessing the NAS rebooted? "Stale file handle" looks like a problem
within the server accessing the data, not the NAS. If you can't reboot,
can you umount/mount the NAS from one of the affected machines and then
try again on that machine?
--
David Chapman dcchap...@acm.org
Chapman Consulting -- San Jose, CA
Software Development Done Right.
www.chapman-consulting-sj.com
Re: Unable to open repository following reboot
Hi Ben,
On 12/4/13 12:52 PM, Pat Haley wrote:
-
(trying local access)
-
mseas(PaperWork)% svn update
svn: Unable to open an ra_local session to URL
svn: Unable to open repository
'file:///home/phaley/Papers/2011/ArpitVel/SvnPaper'
svn: disk I/O error
svn: disk I/O error
---
(trying network access)
---
chris@optiplex:~/Desktop$ svn update 3DDG
Updating '3DDG':
svn: E200030: Unable to connect to a repository at URL
'svn+ssh://mseas/software/Students/3DDG'
svn: E200030: disk I/O error
svn: E200030: disk I/O error
The second error contains an error code for an error from sqlite:
$ which-error.py 200030
00200030 SVN_ERR_SQLITE_ERROR
An update wouldn't be touching a sqlite database on the server since the only
sqlite database on the server side is the rep-cache.db which is only used at
commit time.
Additionally I believe the error message "disk I/O error" only occurs in the
sqlite code.
Are you sure that disk that your working copy is on is ok?
Reasonably so. All the other files are ok. Only svn is
giving us errors.
On 12/4/13 2:31 PM, Pat Haley wrote:
How about 'svnadmin verify /home/phaley/Papers/2011/ArpitVel/SvnPaper'?
No, that also fails
mseas(MeanAvg)% svnadmin verify /home/phaley/Papers/2011/ArpitVel/SvnPaper
svnadmin: disk I/O error
svnadmin: disk I/O error
This still looks like some sort of sqlite error. The rep-cache.db I mentioned
above does get checked as part of svnadmin verify.
Can you run the svnadmin verify under strace:
strace svnadmin verify /home/phaley/Papers/2011/ArpitVel/SvnPaper
You'll get a lot of output but look for the error string in the output and then
look for the preceeding failing system call. It'll probably be a file handle
so you'll need to dig further up looking for where it was opened.
If you want you could post the output someplace. If you want to avoid your
data being exposed use "-s 0" to avoid data being printed (but the error string
won't be included).
You can cause strace to write to a file with "-o myoutputfile"
So if you want to post I'd do:
strace -o myoutputfile -s 0 svnadmin verify
/home/phaley/Papers/2011/ArpitVel/SvnPaper
That line is complaining about a stale file handle:
fcntl(3, F_SETLK, {type=F_RDLCK, whence=SEEK_SET, start=1073741824,
len=1}) = -1 ESTALE (Stale NFS file handle)
write(2, "svnadmin: disk I/O error\n", 25) = 25
write(2, "svnadmin: disk I/O error\n", 25) = 25
exit_group(1) = ?
From a couple of lines above it looks like the file it's
complaining about is
/home/phaley/Papers/2011/ArpitVel/SvnPaper/db/rep-cache.db
however, I can use the "file" command successfully on that file
% file /home/phaley/Papers/2011/ArpitVel/SvnPaper/db/rep-cache.db
/home/phaley/Papers/2011/ArpitVel/SvnPaper/db/rep-cache.db: SQLite 3.x
database, user version 1
I haven't figured out anything more from the full version of that
strace. The post-safe version you suggested is at
http://mseas.mit.edu/download/phaley/Svn/svnverify_post.txt
I also did similar straces on an "svn update" from that
same repository. That is also showing the same
stale file handle error
fcntl(3, F_SETLK, {type=F_RDLCK, whence=SEEK_SET, start=1073741824,
len=1}) = -1 ESTALE (Stale NFS file handle)
write(2, "svn: Unable to open an ra_local "..., 47) = 47
write(2, "svn: Unable to open repository '"..., 89) = 89
write(2, "svn: disk I/O error\n", 20) = 20
write(2, "svn: disk I/O error\n", 20) = 20
lstat(".svn/log", 0x7fff48dcbec0) = -1 ENOENT (No such file or
directory)
unlink("Figures/.svn/lock") = 0
unlink(".svn/lock") = 0
exit_group(1) = ?
The post-safe version of that strace is at
http://mseas.mit.edu/download/phaley/Svn/svnupdate_error_post.txt
Any thoughts on what I should look at next?
Thanks.
-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
Pat Haley Email: pha...@mit.edu
Center for Ocean Engineering Phone: (617) 253-6824
Dept. of Mechanical EngineeringFax:(617) 253-8125
MIT, Room 5-213http://web.mit.edu/phaley/www/
77 Massachusetts Avenue
Cambridge, MA 02139-4301
Re: Unable to open repository following reboot
Hi Les, On Wed, Dec 4, 2013 at 4:31 PM, Pat Haley wrote: How about 'svnadmin verify /home/phaley/Papers/2011/ArpitVel/SvnPaper'? No, that also fails mseas(MeanAvg)% svnadmin verify /home/phaley/Papers/2011/ArpitVel/SvnPaper svnadmin: disk I/O error svnadmin: disk I/O error So, I'm guessing it's a disk error... If it is, it's a mighty peculiar one: (1) We have many other files & directories (including all of our user areas) and nothing else is coming up with errors (2) These different svn repositories are in widely disparate locations under our directory trees, yet all are giving similar errors. One thing that didn't stand out in my original Email was the reason for the reboot. We turned quotas on. Would svn react poorly to this? Also, does the NAS physically holding the disks have any logging facility where you could see physical disk errors reported? I looked in /var/log, but none of the files had any messages at the times I did the above tests. It if is a generic linux box 'dmesg' might show it. If it is some dedicated file server you'll have to find its own diagnostic procedures. It is a generic linux box. However doing dmesg before and after generating an svn error doesn't show anything new. -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=- Pat Haley Email: pha...@mit.edu Center for Ocean Engineering Phone: (617) 253-6824 Dept. of Mechanical EngineeringFax:(617) 253-8125 MIT, Room 5-213http://web.mit.edu/phaley/www/ 77 Massachusetts Avenue Cambridge, MA 02139-4301
Re: Unable to open repository following reboot
On 12/4/13 12:52 PM, Pat Haley wrote: > - > (trying local access) > - > > mseas(PaperWork)% svn update > svn: Unable to open an ra_local session to URL > svn: Unable to open repository > 'file:///home/phaley/Papers/2011/ArpitVel/SvnPaper' > svn: disk I/O error > svn: disk I/O error > > --- > (trying network access) > --- > > chris@optiplex:~/Desktop$ svn update 3DDG > Updating '3DDG': > svn: E200030: Unable to connect to a repository at URL > 'svn+ssh://mseas/software/Students/3DDG' > svn: E200030: disk I/O error > svn: E200030: disk I/O error The second error contains an error code for an error from sqlite: $ which-error.py 200030 00200030 SVN_ERR_SQLITE_ERROR An update wouldn't be touching a sqlite database on the server since the only sqlite database on the server side is the rep-cache.db which is only used at commit time. Additionally I believe the error message "disk I/O error" only occurs in the sqlite code. Are you sure that disk that your working copy is on is ok? On 12/4/13 2:31 PM, Pat Haley wrote: >> How about 'svnadmin verify /home/phaley/Papers/2011/ArpitVel/SvnPaper'? > > No, that also fails > > mseas(MeanAvg)% svnadmin verify /home/phaley/Papers/2011/ArpitVel/SvnPaper > svnadmin: disk I/O error > svnadmin: disk I/O error This still looks like some sort of sqlite error. The rep-cache.db I mentioned above does get checked as part of svnadmin verify. Can you run the svnadmin verify under strace: strace svnadmin verify /home/phaley/Papers/2011/ArpitVel/SvnPaper You'll get a lot of output but look for the error string in the output and then look for the preceeding failing system call. It'll probably be a file handle so you'll need to dig further up looking for where it was opened. If you want you could post the output someplace. If you want to avoid your data being exposed use "-s 0" to avoid data being printed (but the error string won't be included). You can cause strace to write to a file with "-o myoutputfile" So if you want to post I'd do: strace -o myoutputfile -s 0 svnadmin verify /home/phaley/Papers/2011/ArpitVel/SvnPaper
Re: Unable to open repository following reboot
On Wed, Dec 4, 2013 at 4:31 PM, Pat Haley wrote: > >> How about 'svnadmin verify /home/phaley/Papers/2011/ArpitVel/SvnPaper'? > > > No, that also fails > > mseas(MeanAvg)% svnadmin verify /home/phaley/Papers/2011/ArpitVel/SvnPaper > svnadmin: disk I/O error > svnadmin: disk I/O error So, I'm guessing it's a disk error... >> Also, does the NAS physically holding the disks have any logging >> facility where you could see physical disk errors reported? >> > > I looked in /var/log, but none of the files had any > messages at the times I did the above tests. It if is a generic linux box 'dmesg' might show it. If it is some dedicated file server you'll have to find its own diagnostic procedures. -- Les Mikesell lesmikes...@gmail.com
Re: Unable to open repository following reboot
Hi Les, On Wed, Dec 4, 2013 at 4:06 PM, Pat Haley wrote: svn: Unable to open repository 'file:///home/phaley/Papers/2011/ArpitVel/SvnPaper' svn: disk I/O error svn: disk I/O error Sorry that I forgot to mention it in my original Email, but yes I can look in the directory, and I seem to see everything I expected % ls -al /home/phaley/Papers/2011/ArpitVel/SvnPaper How about 'svnadmin verify /home/phaley/Papers/2011/ArpitVel/SvnPaper'? No, that also fails mseas(MeanAvg)% svnadmin verify /home/phaley/Papers/2011/ArpitVel/SvnPaper svnadmin: disk I/O error svnadmin: disk I/O error Also, does the NAS physically holding the disks have any logging facility where you could see physical disk errors reported? I looked in /var/log, but none of the files had any messages at the times I did the above tests. -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=- Pat Haley Email: pha...@mit.edu Center for Ocean Engineering Phone: (617) 253-6824 Dept. of Mechanical EngineeringFax:(617) 253-8125 MIT, Room 5-213http://web.mit.edu/phaley/www/ 77 Massachusetts Avenue Cambridge, MA 02139-4301
Re: Unable to open repository following reboot
On Wed, Dec 4, 2013 at 4:06 PM, Pat Haley wrote: > >>> svn: Unable to open repository >>> 'file:///home/phaley/Papers/2011/ArpitVel/SvnPaper' >>> svn: disk I/O error >>> svn: disk I/O error > Sorry that I forgot to mention it in my original Email, > but yes I can look in the directory, and I seem to > see everything I expected > > % ls -al /home/phaley/Papers/2011/ArpitVel/SvnPaper How about 'svnadmin verify /home/phaley/Papers/2011/ArpitVel/SvnPaper'? Also, does the NAS physically holding the disks have any logging facility where you could see physical disk errors reported? -- Les Mikesell lesmikes...@gmail.com
Re: Unable to open repository following reboot
Hi Henrik,
Last week we rebooted the NAS server which hosts the disk where
(among other things) our svn repositories reside, in order
to enable quotas on that disk. Since that time, we have been
unable to work with these repositories. We see the following
error messages
-
(trying local access)
-
mseas(PaperWork)% svn update
svn: Unable to open an ra_local session to URL
svn: Unable to open repository
'file:///home/phaley/Papers/2011/ArpitVel/SvnPaper'
svn: disk I/O error
svn: disk I/O error
Are you able to do
ls -al /home/phaley/Papers/2011/ArpitVel/SvnPaper
?
Sorry that I forgot to mention it in my original Email,
but yes I can look in the directory, and I seem to
see everything I expected
% ls -al /home/phaley/Papers/2011/ArpitVel/SvnPaper
total 32
drwxrwxr-x 6 phaley phaley 4096 Aug 17 2011 .
drwxrwxr-x 13 phaley phaley 4096 Nov 9 20:31 ..
drwxrwxr-x 2 phaley phaley 4096 Aug 17 2011 conf
drwxrwsr-x 6 phaley phaley 4096 Nov 9 20:36 db
-r--r--r-- 1 phaley phaley2 Aug 17 2011 format
drwxrwxr-x 2 phaley phaley 4096 Aug 17 2011 hooks
drwxrwxr-x 2 phaley phaley 4096 Aug 17 2011 locks
-rw-rw-r-- 1 phaley phaley 229 Aug 17 2011 README.txt
What about:
find /home/phaley/Papers/2011/ArpitVel/SvnPaper -type f -exec cat {}\; >
/dev/null
?
% find /home/phaley/Papers/2011/ArpitVel/SvnPaper -type f
/home/phaley/Papers/2011/ArpitVel/SvnPaper/conf/svnserve.conf
/home/phaley/Papers/2011/ArpitVel/SvnPaper/conf/authz
/home/phaley/Papers/2011/ArpitVel/SvnPaper/conf/passwd
/home/phaley/Papers/2011/ArpitVel/SvnPaper/hooks/pre-unlock.tmpl
/home/phaley/Papers/2011/ArpitVel/SvnPaper/hooks/post-commit.tmpl
/home/phaley/Papers/2011/ArpitVel/SvnPaper/hooks/post-unlock.tmpl
/home/phaley/Papers/2011/ArpitVel/SvnPaper/hooks/start-commit.tmpl
/home/phaley/Papers/2011/ArpitVel/SvnPaper/hooks/post-lock.tmpl
/home/phaley/Papers/2011/ArpitVel/SvnPaper/hooks/pre-lock.tmpl
/home/phaley/Papers/2011/ArpitVel/SvnPaper/hooks/post-revprop-change.tmpl
/home/phaley/Papers/2011/ArpitVel/SvnPaper/hooks/pre-revprop-change.tmpl
/home/phaley/Papers/2011/ArpitVel/SvnPaper/hooks/pre-commit.tmpl
/home/phaley/Papers/2011/ArpitVel/SvnPaper/locks/db-logs.lock
/home/phaley/Papers/2011/ArpitVel/SvnPaper/locks/db.lock
/home/phaley/Papers/2011/ArpitVel/SvnPaper/db/revs/0/65
/home/phaley/Papers/2011/ArpitVel/SvnPaper/db/revs/0/115
/home/phaley/Papers/2011/ArpitVel/SvnPaper/db/revs/0/38
followed by many more similar lines
To me it sounds as if your NAS has some disk problem which might be
hardware related, or, if you are lucky, only a logical error caused by a
clumsy uncontrolled boot. If you have a logical error it might be possible
to fix with some kind of fsck depending upon what kind of NAS you have.
regards Henrik
-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
Pat Haley Email: pha...@mit.edu
Center for Ocean Engineering Phone: (617) 253-6824
Dept. of Mechanical EngineeringFax:(617) 253-8125
MIT, Room 5-213http://web.mit.edu/phaley/www/
77 Massachusetts Avenue
Cambridge, MA 02139-4301
Re: Unable to open repository following reboot
> Last week we rebooted the NAS server which hosts the disk where
> (among other things) our svn repositories reside, in order
> to enable quotas on that disk. Since that time, we have been
> unable to work with these repositories. We see the following
> error messages
>
> -
> (trying local access)
> -
>
> mseas(PaperWork)% svn update
> svn: Unable to open an ra_local session to URL
> svn: Unable to open repository
> 'file:///home/phaley/Papers/2011/ArpitVel/SvnPaper'
> svn: disk I/O error
> svn: disk I/O error
Are you able to do
ls -al /home/phaley/Papers/2011/ArpitVel/SvnPaper
?
What about:
find /home/phaley/Papers/2011/ArpitVel/SvnPaper -type f -exec cat {}\; >
/dev/null
?
To me it sounds as if your NAS has some disk problem which might be
hardware related, or, if you are lucky, only a logical error caused by a
clumsy uncontrolled boot. If you have a logical error it might be possible
to fix with some kind of fsck depending upon what kind of NAS you have.
regards Henrik
Unable to open repository following reboot
Hi, I am not currently a member of this list, so please cc me on any replies to this message. Thanks. We are using svn version 1.6.11 in a linux cluster running CentOS 6.2. We have been using svn for several years. Last week we rebooted the NAS server which hosts the disk where (among other things) our svn repositories reside, in order to enable quotas on that disk. Since that time, we have been unable to work with these repositories. We see the following error messages - (trying local access) - mseas(PaperWork)% svn update svn: Unable to open an ra_local session to URL svn: Unable to open repository 'file:///home/phaley/Papers/2011/ArpitVel/SvnPaper' svn: disk I/O error svn: disk I/O error --- (trying network access) --- chris@optiplex:~/Desktop$ svn update 3DDG Updating '3DDG': svn: E200030: Unable to connect to a repository at URL 'svn+ssh://mseas/software/Students/3DDG' svn: E200030: disk I/O error svn: E200030: disk I/O error Prior to the reboot we were working with these repositories just fine. Any thoughts on how to fix this, or what I should look at to further debug this? Thank you very much. -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=- Pat Haley Email: pha...@mit.edu Center for Ocean Engineering Phone: (617) 253-6824 Dept. of Mechanical EngineeringFax:(617) 253-8125 MIT, Room 5-213http://web.mit.edu/phaley/www/ 77 Massachusetts Avenue Cambridge, MA 02139-4301
