Re: Question about subversion

[Thorsten Schöning]

Guten Tag Marc Davenne,
am Mittwoch, 26. Juni 2013 um 16:36 schrieben Sie:

> Files who should not be   on SVN :
>   
> files   automatically generated

This depends on the files which get automatically generated and their
purpose: It's perfectly valid to version automatically generated class
stubs e.g. for web service clients and servers which get manually
changed afterwards or simply to make sure that one generated version
stays exactly the same and doesn't use new interface names,
automatically generated GUID or whatever those tools out there
implemented.

> files   containing specific information about my
> development   environment (so properties files for example)

Depends again, it's perfectly valid to version project files of
Eclipse, Visual Studio or whatever you use to assure that every member
of you project use the same settings etc.

> executable   files

Depends again?! ;-) There a some projects out there which version
executables, MSI packages and stuff like that and provide direct
download access to their customers using some web application like
ViewVC or WebSVN. If you have an environment were your executables are
not build automatically using some kind of CI, Hudson, Jenkins,
whatever, versioning manually build and once tested applications can
save a lot of time. Besides that, one may be in an situation were one
has only access to third party, pre compiled libraries in which case
there may be no reason to not version those, too.

Mit freundlichen Grüßen,

Thorsten Schöning

-- 
Thorsten Schöning   E-Mail:thorsten.schoen...@am-soft.de
AM-SoFT IT-Systeme  http://www.AM-SoFT.de/

Telefon...05151-  9468- 55
Fax...05151-  9468- 88
Mobil..0178-8 9468- 04

AM-SoFT GmbH IT-Systeme, Brandenburger Str. 7c, 31789 Hameln
AG Hannover HRB 207 694 - Geschäftsführer: Andreas Muchow

RE: Question about subversion

[Bob Archer]

> Hi there. I have a question about subversion.

I think the previous response was quite correct, it mostly all depends.


> I have a theory on what files should not be on SVN and I would like you to 
> tell
> me if you agree. If you dont agree can you tell me why please. If you see more
> files that should not be there, tell me and why.
> Files who should not be on SVN :
> . files automatically generated

Well, Visual Studio generates "designer" files when you edit a web form... we 
do check those in. We also use compass and I check in the compiled .css file. 
Of course, I would like to no longer do that and have the build server generate 
it. 

> . files containing specific information about my development environment (so
> properties files for example) 

For the most part, yes. We check in the files with a .template in the name and 
have an init.bat file that copies them out... this way for any initial checkout 
you at least get reasonable defaults. Our template files also contain the QA 
connection strings so even a dev without the dbs set up will be able to locally 
run the ap. (as was said before, it depends).

. executable files
> PS: I am conscious that we can put anything to subversion but I am looking for
> Best practices Thank you so much

Once again, it depends. Many people feel it is a best practice to be able to do 
a check-out and then build the file without having to do any specific tool 
installs on your PC. Stuff like NuGet (gems npm whatever) help a lot, since 
your build can leverage those tools to get the binaries you need to build test. 
But, sometimes those tools need to be there. For example, we have nant (build 
tool) and mbunit (test tool) checked in so that the build server doesn't need 
to have each of those tools installed, and so also we can rev those tools by 
project rather than having to upgrade all the projects at once.

Also, we have some legacy VB6 code in our project and we use binary 
compatibility, so we need to DLLs checked in. As much as I don't like it. I am 
also considering moving these to a new repo and linking with external so that 
we don't get all the repo bloat... I can just blow away the "binary" repo every 
year or so.

But, yes, for the most part, that artifacts that your project builds aren't 
checked in.. but sometimes dependencies and tools are checked in as binaries. 

So the correct answer, as with almost anything in development is, it 
depends.

BOb


> 
> 
> *
> "Le contenu de ce courriel et ses eventuelles pièces jointes sont 
> confidentiels.
> Ils s'adressent exclusivement à la personne destinataire. Si cet envoi ne vous
> est pas destiné, ou si vous l'avez reçu par erreur, et afin de ne pas violer 
> le
> secret des correspondances, vous ne devez pas le transmettre à d'autres
> personnes ni le reproduire. Merci de le renvoyer à l'émetteur et de le 
> détruire.
> 
> Attention : L'Organisme de l'émetteur du message ne pourra être tenu
> responsable de l'altération du présent courriel. Il appartient au 
> destinataire de
> vérifier que les messages et pièces jointes reçus ne contiennent pas de virus.
> Les opinions contenues dans ce courriel et ses éventuelles pièces jointes sont
> celles de l'émetteur. Elles ne reflètent pas la position de l'Organisme sauf 
> s'il
> en est disposé autrement dans le présent courriel."
> **

RE: Question about subversion

[Andrew Reedick]

> From: Marc Davenne [mailto:marc.dave...@cramif.cnamts.fr] 
> Sent: Wednesday, June 26, 2013 10:37 AM
> To: users@subversion.apache.org
> Subject: Question about subversion
>
> Hi there. I have a question about subversion.
>
>
> I have a theory on what files should not be on SVN and I would like you to 
> tell me if you agree. If you dont agree can you tell me why please. If you 
> see more files that should not be there, tell me and why.
> Files who should not be on SVN :
> * files automatically generated
> * files containing specific information about my development environment (so 
> properties files for example)
> * executable files
> PS: I am conscious that we can put anything to subversion but I am looking 
> for Best practices
> Thank you so much

Wrong question.  The correct question is:  Do I have what I need to reproduce 
the build?

Generally speaking:
* The reasons to avoid checking in generated files are:
- they can be re-created automatically, 
- checking them in results in duplicate data which 
- takes up space unnecessarily
- leads to data synchronization issues, i.e. are the generated files I 
checked in, the same as the files the build generates?  And before you say 
that's a stupid thing to worry about, ask yourself what happens when a checked 
in generated file is no longer generated by the build?  (Someone will have to 
manually delete the no longer needed generated file from the repo.)

* The reason to avoid checking in dev environment files is because they're 
different between everyone's work environment and provide no value to the build 
process.

* The reasons to avoid checking in executable files are:
- they're big
- they can't be merged, and/or
- the executable is generated by the build and thus can be re-built from code 
if necessary.

However, as others have noted, there are times when you want to check in such 
files.


They're guidelines, not hard and fast rules.  Guidelines/Rules exist to support 
your goals.  Your goals are build reproducibility, accurate deployments, and 
other SCM-ish things that allow your organization to deliver a product that 
customers will pay money for so that your company can pay your salary.  Craft 
your guidelines/rules in that context and you'll be fine.

Re: Question about subversion

[Nico Kadel-Garcia]

2013/6/26 Marc Davenne :
> Hi there. I have a question about subversion.
>
> I have a theory on what files should not be on SVN and I would like you to
> tell me if you agree. If you dont agree can you tell me why please. If you
> see more files that should not be there, tell me and why.
>
> Files who should not be on SVN :
>
> files automatically generated
>
> files containing specific information about my development environment (so
> properties files for example)
>
> executable files
>
> PS: I am conscious that we can put anything to subversion but I am looking
> for Best practices
>
> Thank you so much

"Files that should not be on SVN" is like "food that should not be in
a restaurant". It depends tremendously on your purposes. For example,
a "tag" with locked down binaries, or installable packages, can be
very useful for controlling what binaries are deployed in some
environments. They're not efficient to store, however, and if there is
a lot of "churn" of testable or development binaries, the record of
them can get out of hand.

It can be very efficient for a source tree for building software to
avoid ".o" files, ".a" files, and other geneerated compilation
productions that are not part of hte source code itself. But I know
plenty of Subversion and other source control systems where developers
store the ".jar" files from java, or store critical binaries for full
software deployments.

Perhaps you can describe your use case? That affects best practices profoundly.

Re: Question about subversion

[Marc Davenne]

Thank you for your answers... all of you

@Andrew
Thank you, the answers you gave were exactly what I thought about (I was 
a little bit too general when I asked the question)


- concerning answer number 2 (specific files)
Let's say I have a parameter file with paths for my dev environement. 
Typically I would not put it on svn because everybody has different 
ones... but how to I version the param file (without it the application 
does not work)



 Message original 
Sujet: Re: Question about subversion
De : Andrew Reedick 
Pour : Marc Davenne , 
users@subversion.apache.org 

Date : 26/06/2013 18:39




From: Marc Davenne [mailto:marc.dave...@cramif.cnamts.fr]
Sent: Wednesday, June 26, 2013 10:37 AM
To: users@subversion.apache.org
Subject: Question about subversion

Hi there. I have a question about subversion.


I have a theory on what files should not be on SVN and I would like you to tell 
me if you agree. If you dont agree can you tell me why please. If you see more 
files that should not be there, tell me and why.
Files who should not be on SVN :
* files automatically generated
* files containing specific information about my development environment (so 
properties files for example)
* executable files
PS: I am conscious that we can put anything to subversion but I am looking for 
Best practices
Thank you so much


Wrong question.  The correct question is:  Do I have what I need to reproduce 
the build?

Generally speaking:
* The reasons to avoid checking in generated files are:
- they can be re-created automatically,
- checking them in results in duplicate data which
- takes up space unnecessarily
- leads to data synchronization issues, i.e. are the generated files I 
checked in, the same as the files the build generates?  And before you say 
that's a stupid thing to worry about, ask yourself what happens when a checked 
in generated file is no longer generated by the build?  (Someone will have to 
manually delete the no longer needed generated file from the repo.)

* The reason to avoid checking in dev environment files is because they're 
different between everyone's work environment and provide no value to the build 
process.

* The reasons to avoid checking in executable files are:
- they're big
- they can't be merged, and/or
- the executable is generated by the build and thus can be re-built from code 
if necessary.

However, as others have noted, there are times when you want to check in such 
files.


They're guidelines, not hard and fast rules.  Guidelines/Rules exist to support 
your goals.  Your goals are build reproducibility, accurate deployments, and 
other SCM-ish things that allow your organization to deliver a product that 
customers will pay money for so that your company can pay your salary.  Craft 
your guidelines/rules in that context and you'll be fine.





*
"Le contenu de ce courriel et ses eventuelles pièces jointes sont 
confidentiels. Ils s'adressent exclusivement à la personne destinataire. Si cet 
envoi ne vous est pas destiné, ou si vous l'avez reçu par erreur, et afin de ne pas 
violer le secret des correspondances, vous ne devez pas le transmettre à d'autres 
personnes ni le reproduire. Merci de le renvoyer à l'émetteur et de le détruire.

Attention : L'Organisme de l'émetteur du message ne pourra être tenu responsable de 
l'altération du présent courriel. Il appartient au destinataire de vérifier que les 
messages et pièces jointes reçus ne contiennent pas de virus. Les opinions contenues 
dans ce courriel et ses éventuelles pièces jointes sont celles de l'émetteur. Elles 
ne reflètent pas la position de l'Organisme sauf s'il en est disposé autrement dans 
le présent courriel."
**

RE: Question about subversion

[Cooke, Mark]

> -Original Message-
> From: Marc Davenne [mailto:marc.dave...@cramif.cnamts.fr] 
> Sent: 27 June 2013 10:25
> To: Andrew Reedick
> Cc: users@subversion.apache.org
> Subject: Re: Question about subversion
> 
> Thank you for your answers... all of you
> 
> @Andrew
> Thank you, the answers you gave were exactly what I thought 
> about (I was 
> a little bit too general when I asked the question)
> 
> - concerning answer number 2 (specific files)
> Let's say I have a parameter file with paths for my dev environement. 
> Typically I would not put it on svn because everybody has different 
> ones... but how to I version the param file (without it the 
> application does not work)

I think it was BOb who gave an answer for that one...  What I do is to 
svn:ignore the file itself, then check in a default version with a different 
filename (e.g. extra `.tmpl` extension) and provide some sort of 
`initialisation` script that can be used on fresh checkout to copy the template 
file(s) to the right name(s) for use.

For exmaple, I have several .ini files to target different servers 
(development, test, site1, site2 etc) all checked in with the actual .ini file 
name ignored.  I then copy the right one for what I want to the right name.  
The different ini fils are then versioned in the repo.

~ mark c

Re: Question about subversion

[Marc Davenne]

I am not sure I understand it very well. When you say it is ignored, 
does it mean it is on svn ?

And what is this script that would trigger on checkout ?

Can you explain it again ?

 Message original 
Sujet: Re: Question about subversion
De : Cooke, Mark 
Pour : Marc Davenne , Andrew Reedick 


Copie à : "users@subversion.apache.org" 
Date : 27/06/2013 11:40


-Original Message-
From: Marc Davenne [mailto:marc.dave...@cramif.cnamts.fr]
Sent: 27 June 2013 10:25
To: Andrew Reedick
Cc: users@subversion.apache.org
Subject: Re: Question about subversion

Thank you for your answers... all of you

@Andrew
Thank you, the answers you gave were exactly what I thought
about (I was
a little bit too general when I asked the question)

- concerning answer number 2 (specific files)
Let's say I have a parameter file with paths for my dev environement.
Typically I would not put it on svn because everybody has different
ones... but how to I version the param file (without it the
application does not work)


I think it was BOb who gave an answer for that one...  What I do is to 
svn:ignore the file itself, then check in a default version with a different 
filename (e.g. extra `.tmpl` extension) and provide some sort of 
`initialisation` script that can be used on fresh checkout to copy the template 
file(s) to the right name(s) for use.

For exmaple, I have several .ini files to target different servers 
(development, test, site1, site2 etc) all checked in with the actual .ini file 
name ignored.  I then copy the right one for what I want to the right name.  
The different ini fils are then versioned in the repo.

~ mark c





*
"Le contenu de ce courriel et ses eventuelles pièces jointes sont 
confidentiels. Ils s'adressent exclusivement à la personne destinataire. Si cet 
envoi ne vous est pas destiné, ou si vous l'avez reçu par erreur, et afin de ne pas 
violer le secret des correspondances, vous ne devez pas le transmettre à d'autres 
personnes ni le reproduire. Merci de le renvoyer à l'émetteur et de le détruire.

Attention : L'Organisme de l'émetteur du message ne pourra être tenu responsable de 
l'altération du présent courriel. Il appartient au destinataire de vérifier que les 
messages et pièces jointes reçus ne contiennent pas de virus. Les opinions contenues 
dans ce courriel et ses éventuelles pièces jointes sont celles de l'émetteur. Elles 
ne reflètent pas la position de l'Organisme sauf s'il en est disposé autrement dans 
le présent courriel."
**

RE: Question about subversion

[Cooke, Mark]

[Please post inline, it makes it easier to follow...]

>  Message original 
> Sujet: Re: Question about subversion
> De : Cooke, Mark 
> Pour : Marc Davenne , Andrew Reedick 
> 
> Copie à : "users@subversion.apache.org" 
> Date : 27/06/2013 11:40
> 
> >> -Original Message-
> >> From: Marc Davenne [mailto:marc.dave...@cramif.cnamts.fr]
> >> Sent: 27 June 2013 10:25
> >> To: Andrew Reedick
> >> Cc: users@subversion.apache.org
> >> Subject: Re: Question about subversion
> >>
> >> Thank you for your answers... all of you
> >>
> >> @Andrew
> >> Thank you, the answers you gave were exactly what I thought
> >> about (I was
> >> a little bit too general when I asked the question)
> >>
> >> - concerning answer number 2 (specific files)
> >> Let's say I have a parameter file with paths for my dev 
> environement.
> >> Typically I would not put it on svn because everybody has different
> >> ones... but how to I version the param file (without it the
> >> application does not work)
> >
> > I think it was BOb who gave an answer for that one...  What 
> I do is to svn:ignore the file itself, then check in a 
> default version with a different filename (e.g. extra `.tmpl` 
> extension) and provide some sort of `initialisation` script 
> that can be used on fresh checkout to copy the template 
> file(s) to the right name(s) for use.
> >
> > For exmaple, I have several .ini files to target different 
> servers (development, test, site1, site2 etc) all checked in 
> with the actual .ini file name ignored.  I then copy the 
> right one for what I want to the right name.  The different 
> ini fils are then versioned in the repo.
> >
> > ~ mark c

> -Original Message-
> From: Marc Davenne [mailto:marc.dave...@cramif.cnamts.fr] 
> Sent: 27 June 2013 11:15
> To: Cooke, Mark
> Cc: Andrew Reedick; users@subversion.apache.org
> Subject: Re: Question about subversion
> 
> I am not sure I understand it very well. When you say it is ignored, 
> does it mean it is on svn ?

The subversion book explains svn:ignore better than I will:-

http://svnbook.red-bean.com/en/1.7/svn.advanced.props.special.ignore.html

In summary: this is a way to tell svn to ignore a file (or filename pattern) 
when looking for files to check-in.  Hence the file itself is not in the repo, 
but your template version(s) is (are).

> And what is this script that would trigger on checkout ?

That would be (for me, on windows) a simple batch file (that is also checked 
in) that the user knows to run.

> Can you explain it again ?

Does that help?

~ mark c

Re: Question about subversion

[Nico Kadel-Garcia]

On Thu, Jun 27, 2013 at 6:50 AM, Cooke, Mark  wrote:
> [Please post inline, it makes it easier to follow...]
>
>>  Message original ----
>> Sujet: Re: Question about subversion
>> De : Cooke, Mark 
>> Pour : Marc Davenne , Andrew Reedick
>> 
>> Copie à : "users@subversion.apache.org" 
>> Date : 27/06/2013 11:40
>>
>> >> -Original Message-
>> >> From: Marc Davenne [mailto:marc.dave...@cramif.cnamts.fr]
>> >> Sent: 27 June 2013 10:25
>> >> To: Andrew Reedick
>> >> Cc: users@subversion.apache.org
>> >> Subject: Re: Question about subversion
>> >>
>> >> Thank you for your answers... all of you
>> >>
>> >> @Andrew
>> >> Thank you, the answers you gave were exactly what I thought
>> >> about (I was
>> >> a little bit too general when I asked the question)
>> >>
>> >> - concerning answer number 2 (specific files)
>> >> Let's say I have a parameter file with paths for my dev
>> environement.
>> >> Typically I would not put it on svn because everybody has different
>> >> ones... but how to I version the param file (without it the
>> >> application does not work)
>> >
>> > I think it was BOb who gave an answer for that one...  What
>> I do is to svn:ignore the file itself, then check in a
>> default version with a different filename (e.g. extra `.tmpl`
>> extension) and provide some sort of `initialisation` script
>> that can be used on fresh checkout to copy the template
>> file(s) to the right name(s) for use.
>> >
>> > For exmaple, I have several .ini files to target different
>> servers (development, test, site1, site2 etc) all checked in
>> with the actual .ini file name ignored.  I then copy the
>> right one for what I want to the right name.  The different
>> ini fils are then versioned in the repo.
>> >
>> > ~ mark c
>
>> -Original Message-
>> From: Marc Davenne [mailto:marc.dave...@cramif.cnamts.fr]
>> Sent: 27 June 2013 11:15
>> To: Cooke, Mark
>> Cc: Andrew Reedick; users@subversion.apache.org
>> Subject: Re: Question about subversion
>>
>> I am not sure I understand it very well. When you say it is ignored,
>> does it mean it is on svn ?
>
> The subversion book explains svn:ignore better than I will:-
>
> http://svnbook.red-bean.com/en/1.7/svn.advanced.props.special.ignore.html
>
> In summary: this is a way to tell svn to ignore a file (or filename pattern) 
> when looking for files to check-in.  Hence the file itself is not in the 
> repo, but your template version(s) is (are).
>
>> And what is this script that would trigger on checkout ?
>
> That would be (for me, on windows) a simple batch file (that is also checked 
> in) that the user knows to run.

Every software building system has some common tools for this. GNU
make uses "autoconf" and the resulting ./configure file. X11 used to
use "xmkmf" and the "Imakefile". Maven, well, maven has it's own
ideas.

Re: Question about subversion

[Les Mikesell]

On Thu, Jun 27, 2013 at 7:06 AM, Nico Kadel-Garcia  wrote:

>>> http://svnbook.red-bean.com/en/1.7/svn.advanced.props.special.ignore.html
>>
>> In summary: this is a way to tell svn to ignore a file (or filename pattern) 
>> when looking for files to check-in.  Hence the file itself is not in the 
>> repo, but your template version(s) is (are).
>>
>>> And what is this script that would trigger on checkout ?
>>
>> That would be (for me, on windows) a simple batch file (that is also checked 
>> in) that the user knows to run.
>
> Every software building system has some common tools for this. GNU
> make uses "autoconf" and the resulting ./configure file. X11 used to
> use "xmkmf" and the "Imakefile". Maven, well, maven has it's own
> ideas.

Another thing to consider is that you may later wish to add automated
build/CI tools like Jenkins which will do unattended builds for you,
possibly in different environments.   So it is good to plan for that
from the start, providing a script or batch file that does all the
steps needed to configure for any local differences and perform the
build as part of the project.

--
   Les Mikesell
 lesmikes...@gmail.com

Re: Question about Subversion in combination with Samba

[Andy Levy]

On Tue, Mar 12, 2013 at 7:55 AM, Wolfgang Stengel <
wolfgang.sten...@efactory.de> wrote:

>  Hello list,
>
> since a couple of weeks commits from a Windows system in a Samba shared WC
> take much longer than before (using Windows 7, Subversion client 1.7.8).
>
> I tracked this down to Subversion accessing .svn/wc.db-journal and
> .svn/wc.db-wal over and over again (thousands of times). Both files do not
> exist in the WC. It looks like this in the Samba log:
>
> [2013/03/12 12:47:19.260464,  3] smbd/process.c:1662(process_smb)
>   Transaction 719619 of length 150 (0 toread)
> [2013/03/12 12:47:19.260507,  3] smbd/process.c:1467(switch_message)
>   switch message SMBtrans2 (pid 28855) conn 0x7fab8064b800
> [2013/03/12 12:47:19.260542,  3]
> smbd/trans2.c:5117(call_trans2qfilepathinfo)
>   call_trans2qfilepathinfo: TRANSACT2_QPATHINFO: level = 1004
> [2013/03/12 12:47:19.260608,  3] smbd/vfs.c:905(check_reduced_name)
>   check_reduced_name [epson_neon_test/.svn/wc.db-journal]
> [/home/wwwdocs/svn/wolfgang]
> [2013/03/12 12:47:19.260663,  3] smbd/vfs.c:1039(check_reduced_name)
>   check_reduced_name: epson_neon_test/.svn/wc.db-journal reduced to
> /home/wwwdocs/svn/wolfgang/epson_neon_test/.svn/wc.db-journal
> [2013/03/12 12:47:19.260733,  3]
> smbd/trans2.c:5236(call_trans2qfilepathinfo)
>   call_trans2qfilepathinfo: SMB_VFS_STAT of
> epson_neon_test/.svn/wc.db-journal failed (No such file or directory)
> [2013/03/12 12:47:19.260777,  3] smbd/error.c:81(error_packet_set)
>   error packet at smbd/trans2.c(5238) cmd=50 (SMBtrans2)
> NT_STATUS_OBJECT_NAME_NOT_FOUND
> [2013/03/12 12:47:19.261117,  3] smbd/process.c:1662(process_smb)
>   Transaction 719620 of length 142 (0 toread)
> [2013/03/12 12:47:19.261160,  3] smbd/process.c:1467(switch_message)
>   switch message SMBtrans2 (pid 28855) conn 0x7fab8064b800
> [2013/03/12 12:47:19.261195,  3]
> smbd/trans2.c:5117(call_trans2qfilepathinfo)
>   call_trans2qfilepathinfo: TRANSACT2_QPATHINFO: level = 1004
> [2013/03/12 12:47:19.261260,  3] smbd/vfs.c:905(check_reduced_name)
>   check_reduced_name [epson_neon_test/.svn/wc.db-wal]
> [/home/wwwdocs/svn/wolfgang]
> [2013/03/12 12:47:19.261314,  3] smbd/vfs.c:1039(check_reduced_name)
>   check_reduced_name: epson_neon_test/.svn/wc.db-wal reduced to
> /home/wwwdocs/svn/wolfgang/epson_neon_test/.svn/wc.db-wal
> [2013/03/12 12:47:19.261349,  3]
> smbd/trans2.c:5236(call_trans2qfilepathinfo)
>   call_trans2qfilepathinfo: SMB_VFS_STAT of epson_neon_test/.svn/wc.db-wal
> failed (No such file or directory)
> [2013/03/12 12:47:19.261385,  3] smbd/error.c:81(error_packet_set)
>   error packet at smbd/trans2.c(5238) cmd=50 (SMBtrans2)
> NT_STATUS_OBJECT_NAME_NOT_FOUND
>
>
> My question is, is this intended behaviour and just a downside of using a
> network drive, or is this a bug?
>
>
WCs on network drives have never been a recommended or officially supported
configuration, due to variations in permissions handling & locking schemes
between different protocols and implementations. As Subversion has moved
more and more to a database-driven model for the WC, this has become more
true from what I've gathered.


> (Versions are Windows 7, Subversion client 1.7.8
>
>
What Samba version?

Re: Question about Subversion in combination with Samba

[Wolfgang Stengel]

Hi Andy,

Am 12.03.2013 18:44, schrieb Andy Levy:
On Tue, Mar 12, 2013 at 7:55 AM, Wolfgang Stengel 
mailto:wolfgang.sten...@efactory.de>> 
wrote:


Hello list,

since a couple of weeks commits from a Windows system in a Samba
shared WC take much longer than before (using Windows 7,
Subversion client 1.7.8).

I tracked this down to Subversion accessing .svn/wc.db-journal and
.svn/wc.db-wal over and over again (thousands of times). Both
files do not exist in the WC. It looks like this in the Samba log:

[2013/03/12 12:47:19.260464,  3] smbd/process.c:1662(process_smb)
  Transaction 719619 of length 150 (0 toread)
[2013/03/12 12:47:19.260507,  3]
smbd/process.c:1467(switch_message)
  switch message SMBtrans2 (pid 28855) conn 0x7fab8064b800
[2013/03/12 12:47:19.260542,  3]
smbd/trans2.c:5117(call_trans2qfilepathinfo)
  call_trans2qfilepathinfo: TRANSACT2_QPATHINFO: level = 1004
[2013/03/12 12:47:19.260608,  3]
smbd/vfs.c:905(check_reduced_name)
  check_reduced_name [epson_neon_test/.svn/wc.db-journal]
[/home/wwwdocs/svn/wolfgang]
[2013/03/12 12:47:19.260663,  3]
smbd/vfs.c:1039(check_reduced_name)
  check_reduced_name: epson_neon_test/.svn/wc.db-journal
reduced to
/home/wwwdocs/svn/wolfgang/epson_neon_test/.svn/wc.db-journal
[2013/03/12 12:47:19.260733,  3]
smbd/trans2.c:5236(call_trans2qfilepathinfo)
  call_trans2qfilepathinfo: SMB_VFS_STAT of
epson_neon_test/.svn/wc.db-journal failed (No such file or
directory)
[2013/03/12 12:47:19.260777,  3] smbd/error.c:81(error_packet_set)
  error packet at smbd/trans2.c(5238) cmd=50 (SMBtrans2)
NT_STATUS_OBJECT_NAME_NOT_FOUND
[2013/03/12 12:47:19.261117,  3] smbd/process.c:1662(process_smb)
  Transaction 719620 of length 142 (0 toread)
[2013/03/12 12:47:19.261160,  3]
smbd/process.c:1467(switch_message)
  switch message SMBtrans2 (pid 28855) conn 0x7fab8064b800
[2013/03/12 12:47:19.261195,  3]
smbd/trans2.c:5117(call_trans2qfilepathinfo)
  call_trans2qfilepathinfo: TRANSACT2_QPATHINFO: level = 1004
[2013/03/12 12:47:19.261260,  3]
smbd/vfs.c:905(check_reduced_name)
  check_reduced_name [epson_neon_test/.svn/wc.db-wal]
[/home/wwwdocs/svn/wolfgang]
[2013/03/12 12:47:19.261314,  3]
smbd/vfs.c:1039(check_reduced_name)
  check_reduced_name: epson_neon_test/.svn/wc.db-wal reduced
to /home/wwwdocs/svn/wolfgang/epson_neon_test/.svn/wc.db-wal
[2013/03/12 12:47:19.261349,  3]
smbd/trans2.c:5236(call_trans2qfilepathinfo)
  call_trans2qfilepathinfo: SMB_VFS_STAT of
epson_neon_test/.svn/wc.db-wal failed (No such file or directory)
[2013/03/12 12:47:19.261385,  3] smbd/error.c:81(error_packet_set)
  error packet at smbd/trans2.c(5238) cmd=50 (SMBtrans2)
NT_STATUS_OBJECT_NAME_NOT_FOUND


My question is, is this intended behaviour and just a downside of
using a network drive, or is this a bug?


WCs on network drives have never been a recommended or officially 
supported configuration, due to variations in permissions handling & 
locking schemes between different protocols and implementations. As 
Subversion has moved more and more to a database-driven model for the 
WC, this has become more true from what I've gathered.


(Versions are Windows 7, Subversion client 1.7.8


What Samba version?



The version is 3.6.3, however I think all these requests originate from 
the Subversion client itself... is it known why these two files are 
checked repeatedly although they do not exist (at this time at least)?


Wolfgang

Re: Question about Subversion in combination with Samba

[Thorsten Schöning]

Guten Tag Wolfgang Stengel,
am Mittwoch, 13. März 2013 um 08:51 schrieben Sie:

>  The version is 3.6.3, however I think all these requests
> originate from the Subversion client itself... is it known why
> these two files are checked repeatedly although they do not
> exist (at this time at least)?

Those files are internals of SQLite, wc.db is a SQLite database, and
therefore there's nothing SVN can do about it. One either decides to
use SQLite or not, Subversion did. Did the working copy has been
upgraded just some weeks ago to work with Subversion 1.7.x clients?
Then this is expected behavior and there's not much you can do about
it unless tweaking Samba or your client connections "somehow". How
often those files are checked etc. of course depends on the usage of
your clients.

I sometimes commit on a samba share using OpenVPN connection over home
DSL connection with about 60k upload or something around that and it's
slow, but usable. But I'm the only user of the working copy and it's
not really big.

Mit freundlichen Grüßen,

Thorsten Schöning

-- 
Thorsten Schöning   E-Mail:thorsten.schoen...@am-soft.de
AM-SoFT IT-Systeme  http://www.AM-SoFT.de/

Telefon...05151-  9468- 55
Fax...05151-  9468- 88
Mobil..0178-8 9468- 04

AM-SoFT GmbH IT-Systeme, Brandenburger Str. 7c, 31789 Hameln
AG Hannover HRB 207 694 - Geschäftsführer: Andreas Muchow

Re: question about subversion 1.9 unicode normalization status

[Branko Čibej]

On 10.08.2015 18:46, Attila Soki wrote:
> hi,
>
> i saw the entry "reimplement UTF-8 fuzzy conversion using utf8proc (r1511676)"
> in the changelog and hoped this would be the fix for
> http://subversion.tigris.org/issues/show_bug.cgi?id=2464
>
> but after a quick test it seems to be still broken.

In my not even a bit humble opinion, what's broken is Apple's HFS, not
Subversion.


> is there any progress on fixig this bug?

We've not implemented a workaround for the HFS bug in 1.9.

-- Brane

Re: question about subversion 1.9 unicode normalization status

[Dave Huang]

On Aug 11, 2015, at 15:35, Branko Čibej  wrote:
> 
> On 10.08.2015 18:46, Attila Soki wrote:
>> hi,
>> 
>> i saw the entry "reimplement UTF-8 fuzzy conversion using utf8proc 
>> (r1511676)"
>> in the changelog and hoped this would be the fix for
>> http://subversion.tigris.org/issues/show_bug.cgi?id=2464
>> 
>> but after a quick test it seems to be still broken.
> 
> In my not even a bit humble opinion, what's broken is Apple's HFS, not
> Subversion.

Exactly what is broken in Apple's HFS? MacOS uses one of the Unicode 
Normalization Forms. Perhaps it's not the same one that Windows uses, but 
there's nothing wrong with that. While it's unfortunate that SVN didn't handle 
this correctly from the start, it doesn't make it Apple's fault. Unicode 2.0 
talked about normalization/canonicalization in 1996, and TR 15 has been around 
since about the same time--both predating SVN's development by years. Of 
course, most people weren't thinking about Unicode back then, and a filename 
was considered to be some opaque string of bytes, so I don't particularly blame 
SVN either. If anything, Unicode should've just declared one canonical form 
instead of giving options. But while HFS(+) is old and is due for an overhaul, 
its use of Unicode NFD isn't broken.
-- 
Name: Dave Huang |  Mammal, mammal / their names are called /
INet: k...@azeotrope.org |  they raise a paw / the bat, the cat /
FurryMUCK: Dahan |  dolphin and dog / koala bear and hog -- TMBG
Dahan: Hani G Y+C 39 Y++ L+++ W- C++ T++ A+ E+ S++ V++ F- Q+++ P+ B+ PA+ PL++

Re: question about subversion 1.9 unicode normalization status

[Stefan Sperling]

On Tue, Aug 11, 2015 at 05:11:10PM -0500, Dave Huang wrote:
> On Aug 11, 2015, at 15:35, Branko Čibej  wrote:
> > 
> > On 10.08.2015 18:46, Attila Soki wrote:
> >> hi,
> >> 
> >> i saw the entry "reimplement UTF-8 fuzzy conversion using utf8proc 
> >> (r1511676)"
> >> in the changelog and hoped this would be the fix for
> >> http://subversion.tigris.org/issues/show_bug.cgi?id=2464
> >> 
> >> but after a quick test it seems to be still broken.
> > 
> > In my not even a bit humble opinion, what's broken is Apple's HFS, not
> > Subversion.
> 
> Exactly what is broken in Apple's HFS? MacOS uses one of the Unicode 
> Normalization Forms. Perhaps it's not the same one that Windows uses, but 
> there's nothing wrong with that. While it's unfortunate that SVN didn't 
> handle this correctly from the start, it doesn't make it Apple's fault. 
> Unicode 2.0 talked about normalization/canonicalization in 1996, and TR 15 
> has been around since about the same time--both predating SVN's development 
> by years. Of course, most people weren't thinking about Unicode back then, 
> and a filename was considered to be some opaque string of bytes, so I don't 
> particularly blame SVN either. If anything, Unicode should've just declared 
> one canonical form instead of giving options. But while HFS(+) is old and is 
> due for an overhaul, its use of Unicode NFD isn't broken.

Arguing about SVN/Unicode/Apple is all fun and well but won't help with
fixing this problem. Note that Branko was working on this issue for 
quite a while and eventually ran into walls in the FSFS filesystem code.
So I would interpret his statement more as a sign of hard earned
frustration than an actual gripe at whoever came up with HFS(+). He's even
using a mac so it's not like this issue didn't affect him at all.

If anything is needed it's probably kind and encouraging words or even
actual help from fellow sufferers with progress on coding. AFAIK the
design (the most difficult part) is already done (yay!).

Before anyone asks, no, I don't use a mac :)

Re: question about subversion 1.9 unicode normalization status

[Nico Kadel-Garcia]

On Tue, Aug 11, 2015 at 7:12 PM, Stefan Sperling  wrote:
> On Tue, Aug 11, 2015 at 05:11:10PM -0500, Dave Huang wrote:
>> On Aug 11, 2015, at 15:35, Branko Čibej  wrote:
>> >
>> > On 10.08.2015 18:46, Attila Soki wrote:
>> >> hi,
>> >>
>> >> i saw the entry "reimplement UTF-8 fuzzy conversion using utf8proc 
>> >> (r1511676)"
>> >> in the changelog and hoped this would be the fix for
>> >> http://subversion.tigris.org/issues/show_bug.cgi?id=2464
>> >>
>> >> but after a quick test it seems to be still broken.
>> >
>> > In my not even a bit humble opinion, what's broken is Apple's HFS, not
>> > Subversion.
>>
>> Exactly what is broken in Apple's HFS? MacOS uses one of the Unicode 
>> Normalization Forms. Perhaps it's not the same one that Windows uses, but 
>> there's nothing wrong with that. While it's unfortunate that SVN didn't 
>> handle this correctly from the start, it doesn't make it Apple's fault. 
>> Unicode 2.0 talked about normalization/canonicalization in 1996, and TR 15 
>> has been around since about the same time--both predating SVN's development 
>> by years. Of course, most people weren't thinking about Unicode back then, 
>> and a filename was considered to be some opaque string of bytes, so I don't 
>> particularly blame SVN either. If anything, Unicode should've just declared 
>> one canonical form instead of giving options. But while HFS(+) is old and is 
>> due for an overhaul, its use of Unicode NFD isn't broken.

One can also avoid Unicode like the plague of stable programming that
it is. It's awkward enough to stabilize *any* filesystem operations
without trying to "normalize" the translations of anything that is not
internal binary compnents of a file, itself.

Re: question about subversion 1.9 unicode normalization status

[Branko Čibej]

On 12.08.2015 00:11, Dave Huang wrote:
> On Aug 11, 2015, at 15:35, Branko Čibej  wrote:
>> On 10.08.2015 18:46, Attila Soki wrote:
>>> hi,
>>>
>>> i saw the entry "reimplement UTF-8 fuzzy conversion using utf8proc 
>>> (r1511676)"
>>> in the changelog and hoped this would be the fix for
>>> http://subversion.tigris.org/issues/show_bug.cgi?id=2464
>>>
>>> but after a quick test it seems to be still broken.
>> In my not even a bit humble opinion, what's broken is Apple's HFS, not
>> Subversion.
> Exactly what is broken in Apple's HFS? MacOS uses one of the Unicode 
> Normalization Forms. Perhaps it's not the same one that Windows uses, but 
> there's nothing wrong with that.

Yay for misunderstandings. :)

The problem with HFS is that it normalizes paths: regardless of how your
file names are (de)normalised when you create them, they're stored in
HFS in NFD form.

For example, if someone on Linux or Windows creates a file named
"grölsch" and commits it, the Subversion client on the Mac will get a
broken working copy on the next update: you'll see "grölsch" on disk and
"grölsch" in the working copy database, but they'll be different strings.

FWIW, HFS is the only filesystem I'm aware of that does this. Every
other filesystem, including all Windows filesystems, store and return
paths in the exact form they're given. This is true of mounted
filesystems on OSX, too; if you mount a remote ext4 filesystem via NFS,
it will behave differently in this respect than a native HFS volume. The
problem isn't even specific to Subversion; it's encountered by any
software on OSX that has to interact with other filesystems.

This is broken. The filesystem should not be in the business of changing
the (meta)data that it's supposed to store.


>  While it's unfortunate that SVN didn't handle this correctly from the start, 
> it doesn't make it Apple's fault.

See above. It's a fundamental design bug that ignores the common sense
of all other filesystem implementations.

>  Unicode 2.0 talked about normalization/canonicalization in 1996, and TR 15 
> has been around since about the same time--both predating SVN's development 
> by years. Of course, most people weren't thinking about Unicode back then, 
> and a filename was considered to be some opaque string of bytes, so I don't 
> particularly blame SVN either. If anything, Unicode should've just declared 
> one canonical form instead of giving options. But while HFS(+) is old and is 
> due for an overhaul, its use of Unicode NFD isn't broken.

So I'll skip commenting on all this because it's based on a fundamental
misunderstanding of what we're seeing here. Suffice it to say that
normalizing Unicode representations in databases is a very, very bad idea.


The bottom line is: to work around this bug, Subversion needs to make
changes on both the client side, which implies rather fundamental
changes in the working copy structure; and on the server side, to handle
requests made by older clients.

I'm working on this, but slowly because the changes are potentially very
destructive and there are other, far more important things to do.

-- Brane

Re: question about subversion 1.9 unicode normalization status

[Dave Huang]

On Aug 12, 2015, at 4:00, Branko Čibej  wrote:
> 
> Yay for misunderstandings. :)
> 
> The problem with HFS is that it normalizes paths: regardless of how your
> file names are (de)normalised when you create them, they're stored in
> HFS in NFD form.

I don't think I misunderstood anything; I said in my message that filenames are 
stored in NFD, and that I don't think that's broken: "But while HFS(+) is old 
and is due for an overhaul, its use of Unicode NFD isn't broken."

That's great that you're working on the issue though. While I only work in 
English these days and am not affected by this problem, I used to work at a 
place were a lot of l10n stuff was going on, and saw all sorts of languages and 
accented letters :)
-- 
Name: Dave Huang |  Mammal, mammal / their names are called /
INet: k...@azeotrope.org |  they raise a paw / the bat, the cat /
FurryMUCK: Dahan |  dolphin and dog / koala bear and hog -- TMBG
Dahan: Hani G Y+C 39 Y++ L+++ W- C++ T++ A+ E+ S++ V++ F- Q+++ P+ B+ PA+ PL++