Re: SHA-1 collision in repository?

[Myria]

The fulltext whose checksum is 80a10d37de91cadc604ba30e379651b3 I
found out is the first 16384 bytes of the file (see other parts of
this thread).  16384 is SVN__STREAM_CHUNK_SIZE.

On Fri, Mar 2, 2018 at 3:07 PM, Daniel Shahaf  wrote:
> Daniel Shahaf wrote on Fri, Mar 02, 2018 at 22:57:51 +:
>> Myria wrote on Mon, Feb 26, 2018 at 13:41:05 -0800:
>> > In other news, unknown whether related to the current problem, my
>> > attempt to clone the repository to my local computer is failing:
>> >
>> > D:\>svnsync sync file:///d:/svnclone
>> > Transmitting file data
>> > .svnsync:
>> > E16: SHA1 of reps '227170 153 193 57465
>> > bb52be764a04d511ebb06e1889910dcf
>> > e6291ab119036eb783d0136afccdb3b445867364 227184-4vap/_4o' and '-1 0
>> > 193 57465 bb52be764a04d511ebb06e1889910dcf
>> > e6291ab119036eb783d0136afccdb3b445867364 227184-4vap/_4o' matches
>> > (e6291ab119036eb783d0136afccdb3b445867364) but contents differ
>> > svnsync: E160004: Filesystem is corrupt
>> > svnsync: E200014: Checksum mismatch while reading representation:
>> >expected:  bb52be764a04d511ebb06e1889910dcf
>> >  actual:  80a10d37de91cadc604ba30e379651b3
>>
>> When this error happens, could you print the first lines of the two reps
>> identical?  The first line is "PLAIN\n" or "DELTA\n" or "DELTA 42 43 44\n".
>> (I wonder whether we have some stray whitespace that's transparent to parsing
>> but breaks checksums.)
>
> In second thought I'm not sure this makes sense.  A better question is: can we
> obtain the fulltext whose checksum is 80a10d37de91cadc604ba30e379651b3?
>
>> Do you happen to have a copy of the repository lying around that you can run
>> 'grep -a 80a10d37de91cadc604ba30e379651b3 db/revs/{0,1,2,...,227}' on?
>> Admittedly that's a bit of a shot in the dark.
>>
>> Cheers,
>>
>> Daniel

Re: subversion not detecting apr and apr-utils libraries

[Philip Martin]

Guido granda muñoz  writes:

> I'm trying to install subversion 1.9.7 in Ubuntu 16.04 using its source
> code. When I do ./configure ./configure  --with-apr=/usr/lib/apache2
> --with-apr-util=/usr/lib/apache2 , I get the following error:

configure should find apr/aprutil on Ubuntu without you using the --with
parameters.  If that is not happening then you probably need to install
the packages libapr1-dev and libaprutil1-dev.  Look at the Build-depends
used to build the standard Ubuntu package of Subversion:

http://archive.ubuntu.com/ubuntu/pool/main/s/subversion/subversion_1.9.3-2ubuntu1.1.dsc

-- 
Philip

Re: SHA-1 collision in repository?

[Myria]

During rep_write_contents_close, there is a call to get_shared_rep.
get_shared_rep calls svn_fs_fs__get_contents_from_file, which has the
code pasted below.


  /* Build the representation list (delta chain). */
  if (rh->type == svn_fs_fs__rep_plain)
{
  rb->rs_list = apr_array_make(pool, 0, sizeof(rep_state_t *));
  rb->src_state = rs;
}
  else if (rh->type == svn_fs_fs__rep_self_delta)
{
  rb->rs_list = apr_array_make(pool, 1, sizeof(rep_state_t *));
  APR_ARRAY_PUSH(rb->rs_list, rep_state_t *) = rs;
  rb->src_state = NULL;
}
  else
{
  representation_t next_rep = { 0 };

  /* skip "SVNx" diff marker */
  rs->current = 4;

  /* REP's base rep is inside a proper revision.
   * It can be reconstructed in the usual way.  */
  next_rep.revision = rh->base_revision;
  next_rep.item_index = rh->base_item_index;
  next_rep.size = rh->base_length;
  svn_fs_fs__id_txn_reset(_rep.txn_id);

  SVN_ERR(build_rep_list(>rs_list, >base_window,
 >src_state, >len, rb->fs, _rep,
 rb->filehandle_pool));


The bug is occurring because build_rep_list is being called with
first_rep->expanded_size set to zero.  Well, the reason it's zero is
because first_rep is the second to last parameter to build_rep_list,
and the above code initialized expanded_size to zero:

representation_t next_rep = { 0 };

Does the code just need this?  I don't know this call >.<

next_rep.expanded_size = rb->rep.expanded_size;

Melissa

On Wed, Mar 7, 2018 at 9:02 AM, Nathan Hartman  wrote:
> On Mar 5, 2018, at 10:54 PM, Myria  wrote:
>>
>> Final email for the night >.<
>>
>> What's clobbering the expanded_size is this in build_rep_list:
>>
>>  /* The value as stored in the data struct.
>> 0 is either for unknown length or actually zero length. */
>>  *expanded_size = first_rep->expanded_size;
>>
>> first_rep->expanded_size here is zero for the last call to this
>> function before the error.  In every other case before the error, the
>> two values are equal.
>>
>> Then this code executes:
>>
>>  if (*expanded_size == 0)
>>if (rep_header->type == svn_fs_fs__rep_plain || first_rep->size != 4)
>>  *expanded_size = first_rep->size;
>>
>> first_rep->size is 16384, and this is why rb->len becomes 16384,
>> leading to the error.
>>
>> I don't know what all this code is doing, but that's the proximate
>> cause of the failure.
>>
>> Melissa
>
> Has it been possible to determine what is setting expanded_size to 0 before 
> that last call? I wonder if there is specific logic that decides (perhaps 
> incorrectly?) to do that?
>
> Alternatively is it being clobbered by some out-of-bounds access, 
> use-after-free, or another such issue?
>
> Is it possible in your debugger setup to determine the address of that 
> variable and set a breakpoint that triggers when that memory is written? (It 
> may be called a watchpoint?)
>
> Which leads me to another thought: if you can set such a breakpoint / 
> watchpoint and it does not trigger, then this expanded_size might not be the 
> same instance in that final call. Perhaps a shallow copy of an enclosing 
> structure is made which leaves out the known size and sets it to 0 for some 
> reason, and that final call is given that (incomplete) copy.
>
> Caveat: I am not familiar with the codebase but these are my thoughts based 
> on adventures in other code bases.
>

Re: SHA-1 collision in repository?

[Nathan Hartman]

On Mar 5, 2018, at 10:54 PM, Myria  wrote:
> 
> Final email for the night >.<
> 
> What's clobbering the expanded_size is this in build_rep_list:
> 
>  /* The value as stored in the data struct.
> 0 is either for unknown length or actually zero length. */
>  *expanded_size = first_rep->expanded_size;
> 
> first_rep->expanded_size here is zero for the last call to this
> function before the error.  In every other case before the error, the
> two values are equal.
> 
> Then this code executes:
> 
>  if (*expanded_size == 0)
>if (rep_header->type == svn_fs_fs__rep_plain || first_rep->size != 4)
>  *expanded_size = first_rep->size;
> 
> first_rep->size is 16384, and this is why rb->len becomes 16384,
> leading to the error.
> 
> I don't know what all this code is doing, but that's the proximate
> cause of the failure.
> 
> Melissa

Has it been possible to determine what is setting expanded_size to 0 before 
that last call? I wonder if there is specific logic that decides (perhaps 
incorrectly?) to do that?

Alternatively is it being clobbered by some out-of-bounds access, 
use-after-free, or another such issue?

Is it possible in your debugger setup to determine the address of that variable 
and set a breakpoint that triggers when that memory is written? (It may be 
called a watchpoint?)

Which leads me to another thought: if you can set such a breakpoint / 
watchpoint and it does not trigger, then this expanded_size might not be the 
same instance in that final call. Perhaps a shallow copy of an enclosing 
structure is made which leaves out the known size and sets it to 0 for some 
reason, and that final call is given that (incomplete) copy.

Caveat: I am not familiar with the codebase but these are my thoughts based on 
adventures in other code bases.

Re: subversion not detecting apr and apr-utils libraries

[Guido granda muñoz]

Hello,

Thanks for the help. I tried using apr-config and the configuration went
well, however, I got the following error doing the make check:

collect2: error: ld returned 1 exit status
build-outputs.mk:485: recipe for target 'subversion/libsvn_subr/
libsvn_subr-1.la' failed
make: *** [subversion/libsvn_subr/libsvn_subr-1.la] Error 1


What should I Do?
Thanks,

2018-03-07 2:19 GMT-06:00 David Chapman :

> On 3/6/2018 6:54 PM, Guido granda muñoz wrote:
>
>> Hello,
>>
>> I'm trying to install subversion 1.9.7 in Ubuntu 16.04 using its source
>> code. When I do ./configure ./configure --with-apr=/usr/lib/apache2
>> --with-apr-util=/usr/lib/apache2 , I get the following error:
>>
>> configure: Configuring Subversion 1.9.7
>> configure: creating config.nice
>> checking for gcc... gcc
>> checking whether the C compiler works... yes
>> checking for C compiler default output file name... a.out
>> checking for suffix of executables...
>> checking whether we are cross compiling... no
>> checking for suffix of object files... o
>> checking whether we are using the GNU C compiler... yes
>> checking whether gcc accepts -g... yes
>> checking for gcc option to accept ISO C89... none needed
>> checking if gcc accepts -std=c90... yes
>> checking if gcc accepts -w... yes
>> checking if gcc accepts -Werror=unknown-warning-option... no
>> checking for g++... g++
>> checking whether we are using the GNU C++ compiler... yes
>> checking whether g++ accepts -g... yes
>> checking if g++ accepts -std=c++98... yes
>> checking if g++ accepts -w... yes
>> checking if g++ accepts -Werror=unknown-warning-option... no
>> checking how to run the C preprocessor... gcc -E
>> checking for a sed that does not truncate output... /bin/sed
>> checking build system type... x86_64-unknown-linux-gnu
>> checking host system type... x86_64-unknown-linux-gnu
>> checking target system type... x86_64-unknown-linux-gnu
>> checking for grep that handles long lines and -e... /bin/grep
>> checking for egrep... /bin/grep -E
>> checking whether ln -s works... yes
>> checking for a BSD-compatible install... /usr/bin/install -c
>> configure: Apache Portable Runtime (APR) library configuration
>> checking for APR... configure: error: the --with-apr parameter is
>> incorrect. It must specify an install prefix, a build directory, or an
>> apr-config file.
>>
>>
>> I'am pretty sure those provided locations are right because I used the
>> find / -name "apache2" command to obtain them. Please tell me what is wrong.
>>
>> Kind Regards,
>>
>
> Did you look for "apr-config" or just "apache2"?  See
>
> https://unix.stackexchange.com/questions/41910/how-to-make-
> apr-available-for-subversion-install
>
> and of course the reference
>
> https://svn.apache.org/repos/asf/subversion/trunk/INSTALL
>
> You can specify the location of the "apr-config" file directly and it
> should proceed from there.
>
> Disclaimer:  I haven't tried to build Subversion in a long time, and I use
> CentOS, not Ubuntu...
> --
>
> David Chapman  dcchap...@acm.org
> Chapman Consulting -- San Jose, CA
> EDA Software Developer, Expert Witness
> www.chapman-consulting-sj.com
> 2018 Chair, IEEE Consultants' Network of Silicon Valley
>
>


-- 
Guido

Re: subversion not detecting apr and apr-utils libraries

[Paul Hammant]

Guido,

https://issues.apache.org/jira/browse/SERF-188 might have a bearing too
depending on your flavor of Linux.

- Paul

Re: E130003: The XML response contains invalid XML - svn co and log issue on some repos using https/http

[Johan Corveleyn]

On Tue, Mar 6, 2018 at 5:21 PM, NOCERA, ANDY  wrote:
> John,
>
> Good feedback.
>
> Svnadmin 1.9 verify is showing the error also.   I did a dump and load and it 
> resolved that repo.Having reviewed my repos, it looks like around 30%  
> have this issue.  Not sure what we could have done wrong to cause it.   A 
> real simple repo is mine and has only a few commits shows the error.   Is 
> there a correction method quicker than a dump and load?
>

Nice, great that dump+load fixes it. I don't think there is a quicker method.

It might be worth investigating why this happened to begin with. But I
don't really know where to start. One hypothesis is that this
corruption is already lingering there for years (until 1.9 it wouldn't
have been noticed) ... perhaps something outside of SVN manipulated
the rev files years ago? Or perhaps there was a bug once in SVN that
caused this to happen (but the corruption remained benign / unnoticed,
until the stricter validation by 1.9). It's also possible that the
stricter validation by 1.9 contains a bug, and is too strict for some
cases (though in that case I would have expected more reports on this
list).

Maybe you can make a more accurate hypothesis by investigating exactly
what the "Malformed node revision ID string"s looks like. Actually,
danielsh just improved that error message a few days ago, by adding
the actual data to the error message:
http://svn.apache.org/viewvc?view=revision=1825846

So if you can build svn from source you might be able to perform a
build from the latest svn trunk, and run 'svnadmin verify' to get the
more verbose error message (be careful not to use your trunk svn build
on production data without creating a backup of course). Or
alternatively you can try taking a look into the rev files yourself,
to find the "malformed node revision ID".

-- 
Johan

Re: subversion not detecting apr and apr-utils libraries

[David Chapman]

On 3/6/2018 6:54 PM, Guido granda muñoz wrote:

Hello,

I'm trying to install subversion 1.9.7 in Ubuntu 16.04 using its 
source code. When I do ./configure ./configure 
--with-apr=/usr/lib/apache2 --with-apr-util=/usr/lib/apache2 , I get 
the following error:


configure: Configuring Subversion 1.9.7
configure: creating config.nice
checking for gcc... gcc
checking whether the C compiler works... yes
checking for C compiler default output file name... a.out
checking for suffix of executables...
checking whether we are cross compiling... no
checking for suffix of object files... o
checking whether we are using the GNU C compiler... yes
checking whether gcc accepts -g... yes
checking for gcc option to accept ISO C89... none needed
checking if gcc accepts -std=c90... yes
checking if gcc accepts -w... yes
checking if gcc accepts -Werror=unknown-warning-option... no
checking for g++... g++
checking whether we are using the GNU C++ compiler... yes
checking whether g++ accepts -g... yes
checking if g++ accepts -std=c++98... yes
checking if g++ accepts -w... yes
checking if g++ accepts -Werror=unknown-warning-option... no
checking how to run the C preprocessor... gcc -E
checking for a sed that does not truncate output... /bin/sed
checking build system type... x86_64-unknown-linux-gnu
checking host system type... x86_64-unknown-linux-gnu
checking target system type... x86_64-unknown-linux-gnu
checking for grep that handles long lines and -e... /bin/grep
checking for egrep... /bin/grep -E
checking whether ln -s works... yes
checking for a BSD-compatible install... /usr/bin/install -c
configure: Apache Portable Runtime (APR) library configuration
checking for APR... configure: error: the --with-apr parameter is 
incorrect. It must specify an install prefix, a build directory, or an 
apr-config file.



I'am pretty sure those provided locations are right because I used the 
find / -name "apache2" command to obtain them. Please tell me what is 
wrong.


Kind Regards,


Did you look for "apr-config" or just "apache2"?  See

https://unix.stackexchange.com/questions/41910/how-to-make-apr-available-for-subversion-install

and of course the reference

https://svn.apache.org/repos/asf/subversion/trunk/INSTALL

You can specify the location of the "apr-config" file directly and it 
should proceed from there.


Disclaimer:  I haven't tried to build Subversion in a long time, and I 
use CentOS, not Ubuntu...

--

David Chapman  dcchap...@acm.org
Chapman Consulting -- San Jose, CA
EDA Software Developer, Expert Witness
www.chapman-consulting-sj.com
2018 Chair, IEEE Consultants' Network of Silicon Valley