On Fri, Aug 27, 2004 at 02:40:05PM +0100, Jean Delvare wrote:
>>>>OK, so makepp does not store the correct timestamp. How exactly is it distcc's >>>>fault?
It is not at all distcc's fault, its just a problem of using distcc's memory mapped file IO together with makepp using the mtime of a generated object file as signature. So it's just an unhappy interworking effect between two features which on their own are really nice, but put together cause a problem.
Here is what exactly happens when distcc is using memory mapped file IO, and is used in a compilation rule in a makepp build:
- distcc receives the compiled object from a server - when using memory mapped file IO, distcc calls a function mmap, and then considers the file as being written to file, and therefore finishes. - then makepp, noticing that the compilation command completed successfully, gets the mtime of the generated object file, and stores this as signature in a file in a hidden directory. - however, the memory mapped file IO mechanism is not yet finished writing the file actually to disk, this may take between 2-10 seconds!. Accordingly, the mtime of the file is also updated and after the file is completely written, it is 2-10 seconds further in time than the time recorded by makepp.
I tested this in the most simple way:
- I wrote a one line perl script (testFileInfo) which prints the mtime of a file:
use FileInfo; my $finfo = file_info( "$ARGV[ 0 ]" ); print( "file ", $finfo->absolute_filename, " mtime: ", $finfo->file_mtime, "\n" );
- I then used this script in a compilation rule in the following way:
distcc gcc command print the mtime of the generated object file sleep for 5 seconds print the mtime of the generated object file again
The exact commands in the rule are:
$(DIAG) Compiling $(input) to $(output) $(CXX) -c $(absolute-filename $(input)) $(CPPFLAGS) $(CXXFLAGS) -o $(output) /home/cvengelen/tmp/makepp-linux/testFileInfo ${output} sleep 5 /home/cvengelen/tmp/makepp-linux/testFileInfo ${output}
- Then, when compiling files which result in an object file larger than 65 kB, I clearly see that in almost all cases the mtime after the sleep is modified from before the sleep. For example:
debug: Compiling ../../../cih/src/cih_PortGroup.C to ../cih/src/cih_PortGroup.o /swdev/tools/bin/distcc /swdev/tools/cross/bin/powerpc-8xx-linux-g++ -c /home/cvengelen/projects/amu/sw/rel-1.0.1/tih/cih/src/cih_PortGroup.C -I/home/cvengelen/projects/amu/sw/rel-1.0.1 -D_REENTRANT -DPOSIX4 -D_GNU_SOURCE -DLINUX -DUNIX -fsigned-char -fno-builtin -Wall -Wstrict-prototypes -Wmissing-prototypes -W -Winline -Wredundant-decls -DMPC8270 -DMAINCTRL -fno-operator-names -O -o ../cih/src/cih_PortGroup.o /home/cvengelen/tmp/makepp-linux/testFileInfo ../cih/src/cih_PortGroup.o file /home/cvengelen/projects/amu/sw/rel-1.0.1/tih/compiler/gcc-linuxppc+mainctrl/cih/src/cih_PortGroup.o mtime: 1093848988 sleep 5 /home/cvengelen/tmp/makepp-linux/testFileInfo ../cih/src/cih_PortGroup.o file /home/cvengelen/projects/amu/sw/rel-1.0.1/tih/compiler/gcc-linuxppc+mainctrl/cih/src/cih_PortGroup.o mtime: 1093848991
This happens for all files fo which the object file is larger than 65 kB
- I then switch off the memory mapped file IO mechanism of distcc by setting environment variable DISTCC_MMAP to 0, and then the mtime does no longer change:
export DISTCC_MMAP=0
execute makepp command with output:
debug: Compiling ../../../cih/src/cih_PortGroup.C to ../cih/src/cih_PortGroup.o /swdev/tools/bin/distcc /swdev/tools/cross/bin/powerpc-8xx-linux-g++ -c /home/cvengelen/projects/amu/sw/rel-1.0.1/tih/cih/src/cih_PortGroup.C -I/home/cvengelen/projects/amu/sw/rel-1.0.1 -D_REENTRANT -DPOSIX4 -D_GNU_SOURCE -DLINUX -DUNIX -fsigned-char -fno-builtin -Wall -Wstrict-prototypes -Wmissing-prototypes -W -Winline -Wredundant-decls -DMPC8270 -DMAINCTRL -fno-operator-names -O -o ../cih/src/cih_PortGroup.o /home/cvengelen/tmp/makepp-linux/testFileInfo ../cih/src/cih_PortGroup.o file /home/cvengelen/projects/amu/sw/rel-1.0.1/tih/compiler/gcc-linuxppc+mainctrl/cih/src/cih_PortGroup.o mtime: 1093849187 sleep 5 /home/cvengelen/tmp/makepp-linux/testFileInfo ../cih/src/cih_PortGroup.o file /home/cvengelen/projects/amu/sw/rel-1.0.1/tih/compiler/gcc-linuxppc+mainctrl/cih/src/cih_PortGroup.o mtime: 1093849187
The mtime of ALL generated object files no longer is changed after the sleep 5 when using DISTCC_MMAP=0. This is 100% reproducable.
So this proves beyond any doubt that it is the memory mapped file IO used by distcc which causes makepp to store a mtime signature, which is not correct for object files larger than 65 kB, causing recompilation the next time around.
So this is just a fine example of how two fancy mechanisms may lead to unexpected interworking problems, in this case between distcc using memory mapped file IO and makepp using the mtime of a generated file.
Also not that this problem is not related at all to any time differences between distcc servers and hosts: this may have caused other problems, but not this one!
For your information, the related bit of distcc code can be found in distcc source file pump.c, function dcc_r_bulk_plain, which I have attached to this mail. Also notice the check on file size 65536, which explains why I only notice this problem for object files with size larger than 65 kB.
Hope this helps explaining the problem. Please let me know if you need any further information. -- Chris van Engelen AimSys bv Hilversum, The Netherlands Phone: +31 35 689 1935 mailto:[EMAIL PROTECTED]
/* -*- c-file-style: "java"; indent-tabs-mode: nil; fill-column: 78 -*- * * distcc -- A simple distributed compiler system * * Copyright (C) 2003 by Martin Pool <[EMAIL PROTECTED]> * * This program is free software; you can redistribute it and/or * modify it under the terms of the GNU General Public License as * published by the Free Software Foundation; either version 2 of the * License, or (at your option) any later version. * * This program is distributed in the hope that it will be useful, but * WITHOUT ANY WARRANTY; without even the implied warranty of * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU * General Public License for more details. * * You should have received a copy of the GNU General Public License * along with this program; if not, write to the Free Software * Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 * USA */
#include "config.h" #include <stdio.h> #include <stdlib.h> #include <unistd.h> #include <string.h> #include <fcntl.h> #include <errno.h> #include <sys/stat.h> #include <sys/types.h> #include <sys/socket.h> #ifdef HAVE_SYS_SELECT_H # include <sys/select.h> #endif #include <sys/time.h> #ifdef HAVE_SYS_MMAN_H # include <sys/mman.h> #endif #include <netinet/in.h> #include <netinet/tcp.h> #include "distcc.h" #include "trace.h" #include "io.h" #include "util.h" #include "exitcode.h" /** * @file * Transfer of bulk data (source, object code) **/ /** * Receive data from the network to the filesystem. Uses mmap if the data is * large and we can, otherwise reads into a mallocd buffer and writes it out. * * @p out_fd should be open read/write. **/ static int dcc_r_bulk_plain(int out_fd, int in_fd, unsigned len) { int ret; char *buf = NULL; int is_mmapped = 0; if (len == 0) return 0; /* just check */ if (len >= 65536 && dcc_want_mmap()) { #ifdef HAVE_MMAP /* First truncate the file, so that we have space to write to it. This * will fail if we're e.g. writing to stdout. */ buf = mmap(NULL, /* suggested address */ len, PROT_WRITE|PROT_READ, MAP_SHARED, out_fd, 0); /* offset */ if (buf == MAP_FAILED) { rs_trace("mmap output fd%d failed: %s", out_fd, strerror(errno)); } else { is_mmapped = 1; rs_trace("receive %d bytes using mmap", len); /* Now make it big enough to write to. You fall through the floor * if you try to write to nonexistent parts of the file. * * Because this routine is for uncompressed data we know ahead of * time exactly how much space we will need. */ if (ftruncate(out_fd, len) == -1) { rs_log_error("ftruncate fd%d failed: %s", out_fd, strerror(errno)); ret = EXIT_IO_ERROR; goto out; } } #endif } if (!is_mmapped) { if ((buf = malloc(len)) == NULL) { rs_log_error("failed to allocate receipt buffer"); ret = EXIT_OUT_OF_MEMORY; goto out; } } if ((ret = dcc_readx(in_fd, buf, len)) != 0) goto out; if (!is_mmapped) { ret = dcc_writex(out_fd, buf, len); } out: if (is_mmapped) { if (munmap(buf, len) == -1) { rs_log_error("munmap (output) failed: %s", strerror(errno)); ret = ret ? ret : EXIT_IO_ERROR; } } else { free(buf); } return ret; } /** * Common routine for reading a file body used by both fifos and * regular files. * * The output file descriptor is not closed, because this routine is * also used for copying onto stderr. **/ int dcc_r_bulk(int ofd, int ifd, unsigned f_size, enum dcc_compress compression) { if (f_size == 0) return 0; /* don't decompress nothing */ if (compression == DCC_COMPRESS_NONE) { return dcc_r_bulk_plain(ofd, ifd, f_size); } else if (compression == DCC_COMPRESS_LZO1X) { return dcc_r_bulk_lzo1x(ofd, ifd, f_size); } else { rs_fatal("impossible compression %d", compression); } } /** * Copy @p n bytes from @p ifd to @p ofd. * * Does not use sendfile(), so either one may be a socket. * * In the current code at least one of the files will always be a regular * (disk) file, even though it may not be mmapable. That should mean that * writes to it will always complete immediately. That in turn means that on * each pass through the main loop we ought to either completely fill our * buffer, or completely drain it, depending on which one is the disk. * * In future we may put back the ability to feed the compiler from a fifo, in * which case it may be that the writes don't complete. * * We might try selecting on both buffers and handling whichever is ready. * This would require some approximation to a circular buffer though, which * might be more complex. **/ int dcc_pump_readwrite(int ofd, int ifd, size_t n) { static char buf[262144]; /* we're not recursive */ char *p; ssize_t r_in, r_out, wanted; int ret; while (n > 0) { wanted = (n > sizeof buf) ? (sizeof buf) : n; r_in = read(ifd, buf, (size_t) wanted); if (r_in == -1 && errno == EAGAIN) { if ((ret = dcc_select_for_read(ifd, dcc_io_timeout)) != 0) return ret; else continue; } else if (r_in == -1 && errno == EINTR) { continue; } else if (r_in == -1) { rs_log_error("failed to read %ld bytes: %s", (long) wanted, strerror(errno)); return EXIT_IO_ERROR; } else if (r_in == 0) { rs_log_error("unexpected eof on fd%d", ifd); return EXIT_IO_ERROR; } n -= r_in; p = buf; /* We now have r_in bytes waiting to go out, starting at p. Keep * going until they're all written out. */ while (r_in > 0) { r_out = write(ofd, p, (size_t) r_in); if (r_out == -1 && errno == EAGAIN) { if ((ret = dcc_select_for_write(ofd, dcc_io_timeout)) != 0) return ret; else continue; } else if (r_out == -1 && errno == EINTR) { continue; } else if (r_out == -1 || r_out == 0) { rs_log_error("failed to write: %s", strerror(errno)); return EXIT_IO_ERROR; } r_in -= r_out; p += r_out; } } return 0; }
__ distcc mailing list http://distcc.samba.org/ To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/distcc