From: Sean Finney <sean...@debian.org> Date: Mon, 14 Dec 2009 08:27:32 +0100
(Note: this information is largely duplicative of the doxygen documentation comments, found in conffiledb.c and conffiledb.h) The files conffiledb.{c,h} define a set of routines for adding and removing stored versions of "pristine" conffile versions as shipped in their respective packages. Such an implementation allows for a number of features in the future, such as: * 3-way merging of conffile changes. * showing the old->new delta for the "dist" conffiles. * dynamic registering of conffiles (à la ucf). * a general dpkg/conffile cmdline interface to interact with this data. The layout pattern for the conffile database is: <admindir>/conffiles/<pkg>/<version>/<hash> Where: * <admindir> is the standard dpkg administrative dir (i.e. /var/lib/dpkg). * <pkg> is the conffile database directory for a package of the same name. * <version> is the relevant package version. * <hash> is the md5 hash of the installed location of a package's conffile. A hash is used to keep the directory structure flat and balanced. It's assumed that this database is not meant for direct user consumption. However, a "dpkg-conffile" cmdline interface will be provided that will allow the user to perform various forms of inspection with regards to installed conffiles vs the versions in this database. Since there is no loss of functionality with this new code (the previous behavior still exists with respect to the .dpkg-dist/.dpkg-new/.dpkg-old copies of conffiles), it is not considered essential that such a "dpkg-conffile" cmdline interface be provided in the initial implementation. This also gives time for further discussion about how such a tool should function. This commit also contains unit tests covering the basic functionality of the new conffiledb routines. Signed-off-by: Jonathan Nieder <jrnie...@gmail.com> --- Ah, memories. The main piece to revisit might be what sort of identifier to use for <pkg> in these modern days of multiarch. I also think there was some talk about atomic renames across directories versus renaming symlinks to commit to a choice of directory --- perhaps someone can remind me? I notice as I'm sending this that I mistakenly left in some cleanups to src/util.c that were meant to go in a separate patch, if anywhere. Sorry about that. The next version should leave those out. The ..._register_new_conffile API seems a bit odd to me. Would it make sense to return a pathname and let the caller do the write/fsync/rename work? That way it could do a batch of writes followed by a batch of renames if wanted. What happens if ..._register_new_conffile is called and the conffile was already registered? lib/dpkg/test/Makefile.am | 3 +- lib/dpkg/test/t-conffiledb.c | 224 +++++++++++++++++++++++++++++++++++ src/Makefile.am | 1 + src/conffiledb.c | 263 ++++++++++++++++++++++++++++++++++++++++++ src/conffiledb.h | 72 ++++++++++++ src/util.c | 11 +- 6 files changed, 568 insertions(+), 6 deletions(-) create mode 100644 lib/dpkg/test/t-conffiledb.c create mode 100644 src/conffiledb.c create mode 100644 src/conffiledb.h diff --git a/lib/dpkg/test/Makefile.am b/lib/dpkg/test/Makefile.am index 0a81e82..e079ad7 100644 --- a/lib/dpkg/test/Makefile.am +++ b/lib/dpkg/test/Makefile.am @@ -5,7 +5,7 @@ AM_CPPFLAGS = \ -I$(top_builddir) \ -I$(top_srcdir)/lib LDADD = ../libdpkg.a - +t_conffiledb_LDADD = ../../../src/conffiledb.o ../../../src/util.o ../libdpkg.a # The tests are sorted in order of increasing complexity. check_PROGRAMS = \ @@ -18,6 +18,7 @@ check_PROGRAMS = \ t-varbuf \ t-ar \ t-version \ + t-conffiledb \ t-pkginfo \ t-pkg-list \ t-pkg-queue diff --git a/lib/dpkg/test/t-conffiledb.c b/lib/dpkg/test/t-conffiledb.c new file mode 100644 index 0000000..561b159 --- /dev/null +++ b/lib/dpkg/test/t-conffiledb.c @@ -0,0 +1,224 @@ +/** + * @file t-conffiledb.c + * + * Tests for various parts of the conffile database. + * + * Copyright © 2009 Sean Finney <sean...@debian.org> + * + * This is free software; you can redistribute it and/or modify + * it under the terms of the GNU General Public License as published by + * the Free Software Foundation; either version 2 of the License, or + * (at your option) any later version. + * + * This is distributed in the hope that it will be useful, + * but WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the + * GNU General Public License for more details. + * + * You should have received a copy of the GNU General Public License + * along with this program. If not, see <http://www.gnu.org/licenses/>. + */ + +#include <dpkg/test.h> +#include <dpkg/dpkg-db.h> +#include <dpkg/varbuf.h> +#include <dpkg/buffer.h> +#include <../src/conffiledb.h> + +#include <stdlib.h> +#include <sys/types.h> +#include <sys/stat.h> +#include <sys/fcntl.h> +#include <unistd.h> + +/* Forward declaration to avoid needing to include main.h */ +void ensure_pathname_nonexisting(const char *pathname); + +/** + * Our "main program" admindir, which we set to a temporary + * location for our tests. + */ +char *admindir; + +/* Convenience pointer used for inspection of conffiledb contents */ +char *conffiledbdir; + +/** + * Flags for controlling output settings in debug(). + * + * This is needed to link properly with debug() but also convenient + * for debugging. Set to 0 to keep things sane and something like + * (dbg_conff|dbg_conffdetail) to get all the conffiledb debug output + * during test execution. + */ +int f_debug = 0; + +/* some global fixtures used in both tests */ +static struct pkginfo pkg_a, pkg_a2; +static char *pkg_a_cnff_1; +static char *pkg_a_cnff_1_md5; + +/** + * Initialization of various test fixtures, etc. + */ +static void +setup(void) +{ + admindir = m_strdup("/tmp/dpkgtest-admindir-XXXXXX"); + if (mkdtemp(admindir) == NULL) + ohshite("mkdtemp failed in setup"); + + conffiledbdir = m_malloc(strlen(admindir) + strlen("/conffiles") + 1); + strcpy(conffiledbdir, admindir); + strcat(conffiledbdir, "/conffiles"); + if (mkdir(conffiledbdir, 0777)) + ohshite("mkdir failed in setup"); + + /* pkg_a is in the unpack phase of being upgraded */ + pkg_a.name = m_strdup("foopkg"); + pkg_a.installed.version.epoch = 1; + pkg_a.installed.version.version = m_strdup("2.3"); + pkg_a.installed.version.revision = m_strdup("4deb5"); + pkg_a.available.version.epoch = 1; + pkg_a.available.version.version = m_strdup("2.3"); + pkg_a.available.version.revision = m_strdup("5deb6"); + + /* pkg_a2 is in the configure phase of being upgraded */ + pkg_a2.name = m_strdup("foopkg"); + pkg_a2.configversion.epoch = 1; + pkg_a2.configversion.version = m_strdup("2.3"); + pkg_a2.configversion.revision = m_strdup("5deb6"); + pkg_a2.installed.version.epoch = 1; + pkg_a2.installed.version.version = m_strdup("2.3"); + pkg_a2.installed.version.revision = m_strdup("6deb7"); + pkg_a2.available.version.epoch = 1; + pkg_a2.available.version.version = m_strdup("2.3"); + pkg_a2.available.version.revision = m_strdup("6deb7"); + + /* in real use this would be under /etc, but here we set it to + * be under the admindir to allow for testing various things */ + pkg_a_cnff_1 = m_malloc(strlen(admindir) + strlen("/foo.cfg") + 1); + strcpy(pkg_a_cnff_1, admindir); + strcat(pkg_a_cnff_1, "/foo.cfg"); + + pkg_a_cnff_1_md5 = m_malloc(MD5HASHLEN + 1); + buffer_md5(pkg_a_cnff_1, pkg_a_cnff_1_md5, strlen(pkg_a_cnff_1)); +} + +/** + * Cleanup to be done when tests are completed. + * + * Any files creating during tests should be removed here, as well + * as the free()'ing of any allocated memory. This should allow for + * testing the code for memory leaks with valgrind. + */ +static void +teardown(void) +{ + free((char*)pkg_a.name); + free((char*)pkg_a.installed.version.version); + free((char*)pkg_a.installed.version.revision); + free((char*)pkg_a.available.version.version); + free((char*)pkg_a.available.version.revision); + + free((char*)pkg_a2.name); + free((char*)pkg_a2.configversion.version); + free((char*)pkg_a2.configversion.revision); + free((char*)pkg_a2.installed.version.version); + free((char*)pkg_a2.installed.version.revision); + free((char*)pkg_a2.available.version.version); + free((char*)pkg_a2.available.version.revision); + + free(pkg_a_cnff_1); + free(pkg_a_cnff_1_md5); + + ensure_pathname_nonexisting(admindir); + free(admindir); + free(conffiledbdir); +} + +/** + * Ensure that conffiledb_path behaves sanely for all defined uses. + */ +static void +test_conffiledb_paths(void) +{ + char *path, *subdir; + + /* should be the conffiledb root dir */ + path = conffiledb_path(NULL, NULL, NULL); + test_str(path, ==, conffiledbdir); + free(path); + + /* should be root dir + package name */ + path = conffiledb_path(&pkg_a, NULL, NULL); + subdir = path + strlen(admindir); + test_str(subdir, ==, subdir); + free(path); + + /* should be root dir + package name + version */ + path = conffiledb_path(&pkg_a, &pkg_a.installed.version, NULL); + subdir = path + strlen(admindir); + test_str(subdir, ==, "/conffiles/foopkg/1:2.3-4deb5"); + free(path); + + /* should be root dir + package name + version + hash */ + path = conffiledb_path(&pkg_a, &pkg_a.installed.version, pkg_a_cnff_1); + subdir = path + strlen(admindir) + + strlen("/conffiles/foopkg/1:2.3-4deb5/"); + test_str(subdir, ==, pkg_a_cnff_1_md5); + free(path); +} + +/** + * Ensure that get/put type operations function as expected. + */ +static void +test_conffiledb_storage(void) +{ + int pipefds[2]; + struct stat stat_buf; + int fetched_fd; + const char *fake_content = "pretend conffile content\n"; + int content_sz = strlen(fake_content); + char *expected_path = conffiledb_path(&pkg_a, &pkg_a.available.version, + pkg_a_cnff_1); + + /* setup a fake pipe like what we'd get in processarchive */ + if (pipe(pipefds)) + ohshite("pipe failed"); + if (write(pipefds[1], fake_content, content_sz) != content_sz) + ohshite("write failed"); + if (close(pipefds[1])) + ohshite("close failed"); + + /* ensure the file registers, exists, and is the same size */ + conffiledb_register_new_conffile(&pkg_a, pkg_a_cnff_1, pipefds[0], + content_sz); + test_pass(stat(expected_path, &stat_buf) == 0); + test_pass(stat_buf.st_size == content_sz); + + /* now pretend it's later and test fetching the same file */ + fetched_fd = conffiledb_open_conffile(&pkg_a2, &pkg_a2.configversion, + pkg_a_cnff_1); + test_pass(fstat(fetched_fd, &stat_buf) == 0); + test_pass(stat_buf.st_size == content_sz); + + /* now pretend it's removal time */ + conffiledb_remove(&pkg_a2, &pkg_a2.configversion); + test_pass(stat(expected_path, &stat_buf) == -1); + + close(fetched_fd); + free(expected_path); +} + +static void +test(void) +{ + setup(); + + test_conffiledb_paths(); + test_conffiledb_storage(); + + teardown(); +} diff --git a/src/Makefile.am b/src/Makefile.am index 51e2b9f..662ef31 100644 --- a/src/Makefile.am +++ b/src/Makefile.am @@ -28,6 +28,7 @@ bin_PROGRAMS = \ dpkg_SOURCES = \ archives.c archives.h \ cleanup.c \ + conffiledb.c conffiledb.h \ configure.c \ depcon.c \ enquiry.c \ diff --git a/src/conffiledb.c b/src/conffiledb.c new file mode 100644 index 0000000..2dec3b5 --- /dev/null +++ b/src/conffiledb.c @@ -0,0 +1,263 @@ +/** + * @file conffiledb.c + * + * Implementation of conffile database handling routines. + * + * Copyright © 2009 Sean Finney <sean...@debian.org> + * + * This is free software; you can redistribute it and/or modify + * it under the terms of the GNU General Public License as published by + * the Free Software Foundation; either version 2 of the License, or + * (at your option) any later version. + * + * This is distributed in the hope that it will be useful, + * but WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the + * GNU General Public License for more details. + * + * You should have received a copy of the GNU General Public License + * along with this program. If not, see <http://www.gnu.org/licenses/>. + */ + +#include <config.h> +#include <compat.h> + +#include <unistd.h> +#include <sys/types.h> +#include <sys/stat.h> +#include <sys/wait.h> +#include <fcntl.h> +#include <stdlib.h> +#include <errno.h> + +#include <dpkg/dpkg.h> +#include <dpkg/i18n.h> +#include <dpkg/buffer.h> +#include <dpkg/path.h> +#include <dpkg/dpkg-db.h> +#include <dpkg/varbuf.h> + +#include "main.h" +#include "conffiledb.h" + +#define CONFFILEDBDIRROOT "conffiles" /**< @ref conffiledb root location */ + +static char *conffiledb_path_cversion(const struct pkginfo *pkg, + const char *version, const char *path); + +/** + * Return the canonical path for a conffiledb entry or other location. + * + * This is a helper function to resolve the full path to a specific location + * within the conffile database. The primary use is to find the location + * of a conffiledb file, but it can also be used to find the location + * of the @ref conffiledb for a particular package as well as the parent + * directory containing all conffiledb directories for a package. + * + * @param pkg The package information. If NULL, the returned value will + * be the conffiledb root location (admindir + #CONFFILEDBDIRROOT). + * @param version Package version information. This is typically one of + * pkg.installed, pkg.available, or pkg.configversion. If NULL, + * the returned value will be the top level directory for pkg (i.e. + * the directory containing all conffiledb subdirectories for pkg). + * @param path The absolute path to the conffile. If NULL, the returned + * value will be the conffiledb directory which should contain all + * conffiledb files for the combination of pkg and version. + * + * @return An allocated string with the path (caller must free it later). + */ +char * +conffiledb_path(const struct pkginfo *pkg, + const struct versionrevision *version, const char *path) +{ + struct varbuf versionvb = VARBUF_INIT; + char *version_result; + + if (version != NULL) { + varbufversion(&versionvb, version, vdew_nonambig); + varbufaddc(&versionvb, 0); + } + + version_result = conffiledb_path_cversion(pkg, versionvb.buf, path); + + if (version != NULL) + varbuffree(&versionvb); + + return version_result; +} + +/** + * Internal helper function for conffile_db_path. + * + * This function performs most of the work needed by the exposed + * conffiledb_path function. It is seperated from the main function + * to allow for some internal use where a path needs to be generated + * but not from a valid struct versionrevision (i.e. any special "merge" + * or "dynamic" versions). + * + * For more information on the parameters, see conffiledb_path. + * + * @param pkg The package information. + * @param version Package version. + * @param path The absolute path to the conffile. + * + * @return An allocated string with the path (caller must free it later). + */ +char * +conffiledb_path_cversion(const struct pkginfo *pkg, const char *version, + const char *path) +{ + struct varbuf full_path = VARBUF_INIT; + char hash[MD5HASHLEN + 1]; + + varbufprintf(&full_path, "%s/%s", admindir, CONFFILEDBDIRROOT); + + if (pkg != NULL) { + varbufaddc(&full_path, '/'); + varbufaddstr(&full_path, pkg->name); + } + + if (version != NULL) { + varbufaddc(&full_path, '/'); + varbufaddstr(&full_path, version); + } + + /* if a conffile is being requested (not just the db root)... */ + if (path != NULL) { + varbufaddc(&full_path, '/'); + buffer_md5(path, hash, strlen(path)); + varbufaddstr(&full_path, hash); + } + varbufaddc(&full_path, 0); + + debug(dbg_conffdetail, "confffiledb_path(%s, %s) = %s\n", + pkg?(pkg->name):"(null)", path, full_path.buf); + + return full_path.buf; +} + +/** + * Ensure that a particular conffiledb dir exists. + * + * @param pkg The package owning the conffiledb dir. + * @param version Which package version to reference. + */ +static void +conffiledb_ensure_db(const struct pkginfo *pkg, + const struct versionrevision *version) +{ + struct stat s; + int i; + char *dbdirs[2]; + + dbdirs[0] = conffiledb_path(pkg, NULL, NULL); + dbdirs[1] = conffiledb_path(pkg, version, NULL); + + /* ensure each directory exists while reconstructing the path */ + for (i = 0; i < 2; i++) { + if (stat(dbdirs[i], &s) != 0) { + debug(dbg_conffdetail, + "conffiledb_ensure_db: creating %s\n", dbdirs[i]); + if (errno != ENOENT) + ohshite(_("stat() '%s' failed"), dbdirs[i]); + if (mkdir(dbdirs[i], 0700) == -1) + ohshite(_("mkdir() '%s' failed"), dbdirs[i]); + } + free(dbdirs[i]); + } +} + +/** + * Register a new conffiledb file for a package. + * + * Create a conffiledb entry in the conffiledb dir for the newly available + * version of a particular package. + * + * The use of a filedescriptor is due to the fact that the new conffile is + * intercepted very early during the unpack phase, before it exists on disk. + * Therefore this interface is designed to hook cleanly into what's available + * at that point, which is a filedescriptor generated from a struct TarInfo + * (see tarobject in archives.c). + * + * @param pkg The package owning the conffile. + * @param path The installed location of the conffile. + * @param fd A file descriptor holding the contents of the conffile. + * @param sz The size of the conffile. + */ +void +conffiledb_register_new_conffile(const struct pkginfo *pkg, const char *path, + int fd, off_t sz) +{ + int cfgfd; + /* get the path to where the registered file goes */ + char *p = conffiledb_path(pkg, &pkg->available.version, path); + char quoted_path[256]; + struct varbuf tmpfn = VARBUF_INIT; + + varbufprintf(&tmpfn, "%s%s", p, DPKGTEMPEXT); + + conffiledb_ensure_db(pkg, &pkg->available.version); + debug(dbg_conff, "conffile_reg_fd: %s, %s, %s\n", pkg->name, path, p); + + /* make a copy of file to the .dpkg-tmp location and then rename + * it to the proper destination */ + cfgfd = open(tmpfn.buf, O_CREAT|O_TRUNC|O_WRONLY, 0600); + if (cfgfd == -1) + ohshite(_("open '%s' failed"), tmpfn.buf); + fd_fd_copy(fd, cfgfd, sz, _("conffiledb registering '%.255s'"), + path_quote_filename(quoted_path, path, 256)); + + if (fsync(cfgfd) == -1) + ohshite(_("fsync '%s' failed"), p); + if (close(cfgfd) == -1) + ohshite(_("close '%s' failed"), p); + if (rename(tmpfn.buf, p) == -1) + ohshite(_("rename '%s' failed"), p); + + varbuffree(&tmpfn); + free(p); +} + +/** + * Get a readable filedescriptor to the contents of a given conffiledb entry. + * + * @param pkg The package owning the conffile. + * @param version The relevant package version. + * @param path The installed location of the conffile. + * + * @return A read-only file descriptor for the registered conffile. + */ +int +conffiledb_open_conffile(const struct pkginfo *pkg, + const struct versionrevision *version, + const char *path) +{ + int fd; + char *p = conffiledb_path(pkg, version, path); + + fd = open(p, O_RDONLY); + if (fd == -1) + ohshite(_("open '%s' failed"), p); + + free(p); + return fd; +} + +/** + * Remove a conffiledb directory for a package. + * + * @param pkg The package owning the conffile db to be removed. + * @param version The version of the package for which to remove the conffiledb. + * If set to NULL, all information for pkg will be removed. + */ +void +conffiledb_remove(const struct pkginfo *pkg, + const struct versionrevision *version) +{ + char *p = conffiledb_path(pkg, version, NULL); + + debug(dbg_conff, "conffiledb_remove(%s): removing %s\n", pkg->name, p); + ensure_pathname_nonexisting(p); + + free(p); +} diff --git a/src/conffiledb.h b/src/conffiledb.h new file mode 100644 index 0000000..4bb50eb --- /dev/null +++ b/src/conffiledb.h @@ -0,0 +1,72 @@ +/** + * @file conffiledb.h + * + * Functions for tracking referencable versions of packages' conffiles. + * + * This library allows for tracking the pristine versions (and in some specific + * cases a few other versions) of packages' conffiles. This allows for a + * number of desirable features, such as conffile merging and more + * featureful reporting of the differences between conffiles during package + * upgrade, even when the local admin may have modified or removed + * the installed version of the file. + * + * The internal database of conffiles follows the convention: + * + * #CONFFILEDBDIRROOT/\<pkg\>/\<version\>/\<hash\> + * + * where: + * - @b \<pkg\> is the conffile database directory for an individual package. + * - @b \<version\> is the version of the package owning the conffile. + * - @b \<hash\> is the md5 hash of the location of a package's conffile. A + * hash is used to keep a flat and balanced directory structure, and has + * the added benefit of a simpler implementation. + * + * Some further terminology to keep things clear later on: + * + * - @b "conffiledb root" refers to the top-most directory containing + * all conffile databases, and is defined by #CONFFILEDBDIRROOT. + * - @anchor conffiledb @b "conffile database" or @b "conffiledb" refers + * to a directory specific to a package + version tuple. Such a + * directory will contain copies of the conffiles named after the hash + * of their respective installed locations. + * - @b "conffiledb file" or @b "conffiledb entry" refers to a file whose + * contents correspond to a particular conffiledb_base/package/conffile + * triplet. + * + * Copyright © 2009 Sean Finney <sean...@debian.org> + * + * This is free software; you can redistribute it and/or modify + * it under the terms of the GNU General Public License as published by + * the Free Software Foundation; either version 2 of the License, or + * (at your option) any later version. + * + * This is distributed in the hope that it will be useful, + * but WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the + * GNU General Public License for more details. + * + * You should have received a copy of the GNU General Public License + * along with this program. If not, see <http://www.gnu.org/licenses/>. + */ + +#ifndef DPKG_CONFFILEDB_H +#define DPKG_CONFFILEDB_H + +#include <sys/types.h> + +#include <dpkg/dpkg-db.h> + +char *conffiledb_path(const struct pkginfo *pkg, + const struct versionrevision *version, const char *path); + +void conffiledb_register_new_conffile(const struct pkginfo *pkg, + const char *path, int fd, off_t sz); + +int conffiledb_open_conffile(const struct pkginfo *pkg, + const struct versionrevision *version, + const char *path); + +void conffiledb_remove(const struct pkginfo *pkg, + const struct versionrevision *version); + +#endif /* DPKG_CONFFILEDB_H */ diff --git a/src/util.c b/src/util.c index 3dd4fed..58448d3 100644 --- a/src/util.c +++ b/src/util.c @@ -88,7 +88,7 @@ secure_unlink_statted(const char *pathname, const struct stat *stab) * * This function is a generic catch-all delete utility, which * will remove anything that may exist at the given pathname. If - * The pathname is a nonempty directory, it will be recursively + * the pathname is a nonempty directory, it will be recursively * removed with a fork/exec of rm -rf. * * @param pathname the path to be removed. @@ -105,17 +105,18 @@ ensure_pathname_nonexisting(const char *pathname) debug(dbg_eachfile, "ensure_pathname_nonexisting `%s'", pathname); if (!rmdir(pathname)) return; /* Deleted it OK, it was a directory. */ - if (errno == ENOENT || errno == ELOOP) return; + if (errno == ENOENT || errno == ELOOP) + return; if (errno == ENOTDIR) { /* Either it's a file, or one of the path components is. If one * of the path components is this will fail again ... */ if (secure_unlink(pathname) == 0) return; /* OK, it was. */ - if (errno == ENOTDIR) return; + if (errno == ENOTDIR) + return; } - if (errno != ENOTEMPTY && errno != EEXIST) { /* Huh? */ + if (errno != ENOTEMPTY && errno != EEXIST) /* Huh? */ ohshite(_("unable to securely remove '%.255s'"), pathname); - } pid = subproc_fork(); if (pid == 0) { execlp(RM, "rm", "-rf", "--", pathname, NULL); -- 1.7.5.1 -- To UNSUBSCRIBE, email to debian-dpkg-requ...@lists.debian.org with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/20110506075952.GB1040@elie