The branch, master has been updated via 1531fcd vfs_gpfs: Use full_path_tos instead of talloc_asprintf via 1843a2d vfs_gpfs: Remove get_full_smb_filename from is_offline check via 4a29a94 vfs_gpfs: Remove wrong get_full_smb_filename from ntimes function via 01a6aef vfs_gpfs: Adjust debug level when get_winattrs returns EBADF via 9a79a61 smbd: Move dfree_info struct via e30d0c0 smbd: Flush dfree memcache on service reload via 8f12174 smbd: Cache dfree information based on query path via f5d0556 memcache: Add new cache type for dfree information via a55b3d2 selftest: Add test for 'dfree cache' via 7ffcbd5 selftest: Add dfq_cache share with 'dfree cache time' set from 52778af s3: vfs_fake_acls: Correctly implement the chmod/fchmod algorithm on fake acls.
https://git.samba.org/?p=samba.git;a=shortlog;h=master - Log ----------------------------------------------------------------- commit 1531fcde8252efcb2f69dd5462beb83786bb1135 Author: Christof Schmitt <c...@samba.org> Date: Tue May 22 12:52:58 2018 -0700 vfs_gpfs: Use full_path_tos instead of talloc_asprintf full_path_tos avoids the talloc call for most cases; use that instead of talloc_asprintf. Signed-off-by: Christof Schmitt <c...@samba.org> Reviewed-by: Jeremy Allison <j...@samba.org> Autobuild-User(master): Jeremy Allison <j...@samba.org> Autobuild-Date(master): Sat May 26 01:43:09 CEST 2018 on sn-devel-144 commit 1843a2d63259c819eb7a6eefa7f4ab8c46efec37 Author: Christof Schmitt <c...@samba.org> Date: Tue May 22 12:25:42 2018 -0700 vfs_gpfs: Remove get_full_smb_filename from is_offline check No stream information is required here. Signed-off-by: Christof Schmitt <c...@samba.org> Reviewed-by: Jeremy Allison <j...@samba.org> commit 4a29a949ac45fab55b8c41dcbe11a5e07fd62c05 Author: Christof Schmitt <c...@samba.org> Date: Tue May 22 12:22:06 2018 -0700 vfs_gpfs: Remove wrong get_full_smb_filename from ntimes function Updating the timestamps requires the path to the file, but no stream information. Signed-off-by: Christof Schmitt <c...@samba.org> Reviewed-by: Jeremy Allison <j...@samba.org> commit 01a6aef52227e0f029fa107140e670d8af45ad59 Author: Christof Schmitt <c...@samba.org> Date: Wed May 23 15:04:08 2018 -0700 vfs_gpfs: Adjust debug level when get_winattrs returns EBADF This is returned for a get_winattrs call against a non-gpfs file system. This can happen for the .. entry when listing a share on the file system root. Signed-off-by: Christof Schmitt <c...@samba.org> Reviewed-by: Jeremy Allison <j...@samba.org> commit 9a79a61abb08719acde6aec8df9356be36d8dda2 Author: Christof Schmitt <c...@samba.org> Date: Wed May 16 13:25:54 2018 -0700 smbd: Move dfree_info struct As the struct is no longer used as part of connection_struct, move it to dfree.c. This is not backported, as it would change the VFS ABI. Signed-off-by: Christof Schmitt <c...@samba.org> Reviewed-by: Jeremy Allison <j...@samba.org> commit e30d0c0e0d11f65b2d1886be3c0fe9e32eaf3926 Author: Christof Schmitt <c...@samba.org> Date: Fri May 18 20:51:58 2018 -0700 smbd: Flush dfree memcache on service reload BUG: https://bugzilla.samba.org/show_bug.cgi?id=13446 Signed-off-by: Christof Schmitt <c...@samba.org> Reviewed-by: Jeremy Allison <j...@samba.org> commit 8f121747b06ca78cf51801a3931b2ddd1a424c77 Author: Christof Schmitt <c...@samba.org> Date: Wed May 16 13:17:52 2018 -0700 smbd: Cache dfree information based on query path Sub directories in a SMB share can have different free space information (e.g. when a different file system is mounted there). Caching the dfree information per SMB share will return invalid data. Address this by switching to memcache and store the cached data based on the query path. BUG: https://bugzilla.samba.org/show_bug.cgi?id=13446 Signed-off-by: Christof Schmitt <c...@samba.org> Reviewed-by: Jeremy Allison <j...@samba.org> commit f5d05562679f6aa691b98b4a75952f7dda7ed343 Author: Christof Schmitt <c...@samba.org> Date: Wed May 16 13:05:36 2018 -0700 memcache: Add new cache type for dfree information BUG: https://bugzilla.samba.org/show_bug.cgi?id=13446 Signed-off-by: Christof Schmitt <c...@samba.org> Reviewed-by: Jeremy Allison <j...@samba.org> commit a55b3d2fcc2f7737a2702bf908dcf1f80969bf21 Author: Christof Schmitt <c...@samba.org> Date: Wed May 23 11:25:42 2018 -0700 selftest: Add test for 'dfree cache' BUG: https://bugzilla.samba.org/show_bug.cgi?id=13446 Signed-off-by: Christof Schmitt <c...@samba.org> Reviewed-by: Jeremy Allison <j...@samba.org> commit 7ffcbd5ce1222971cb9879f78765d87cdc4102a8 Author: Christof Schmitt <c...@samba.org> Date: Wed May 23 11:07:54 2018 -0700 selftest: Add dfq_cache share with 'dfree cache time' set BUG: https://bugzilla.samba.org/show_bug.cgi?id=13446 Signed-off-by: Christof Schmitt <c...@samba.org> Reviewed-by: Jeremy Allison <j...@samba.org> ----------------------------------------------------------------------- Summary of changes: lib/util/memcache.h | 3 +- selftest/target/Samba3.pm | 6 ++ source3/include/vfs.h | 11 +-- source3/modules/vfs_gpfs.c | 53 +++++++-------- source3/script/tests/test_dfree_quota.sh | 35 ++++++++++ source3/smbd/dfree.c | 112 +++++++++++++++++++++++++------ source3/smbd/proto.h | 1 + source3/smbd/server_reload.c | 1 + 8 files changed, 164 insertions(+), 58 deletions(-) Changeset truncated at 500 lines: diff --git a/lib/util/memcache.h b/lib/util/memcache.h index 670cbd6..4331c2f 100644 --- a/lib/util/memcache.h +++ b/lib/util/memcache.h @@ -45,7 +45,8 @@ enum memcache_number { SINGLETON_CACHE, SMB1_SEARCH_OFFSET_MAP, SHARE_MODE_LOCK_CACHE, /* talloc */ - VIRUSFILTER_SCAN_RESULTS_CACHE_TALLOC /* talloc */ + VIRUSFILTER_SCAN_RESULTS_CACHE_TALLOC, /* talloc */ + DFREE_CACHE, }; /* diff --git a/selftest/target/Samba3.pm b/selftest/target/Samba3.pm index 2b1752f..c6ae7ab 100755 --- a/selftest/target/Samba3.pm +++ b/selftest/target/Samba3.pm @@ -2116,6 +2116,12 @@ sub provision($$$$$$$$$) vfs objects = acl_xattr fake_acls xattr_tdb fake_dfq admin users = $unix_name include = $dfqconffile +[dfq_cache] + path = $shrdir/dfree + vfs objects = acl_xattr fake_acls xattr_tdb fake_dfq + admin users = $unix_name + include = $dfqconffile + dfree cache time = 60 [dfq_owner] path = $shrdir/dfree vfs objects = acl_xattr fake_acls xattr_tdb fake_dfq diff --git a/source3/include/vfs.h b/source3/include/vfs.h index 945e4b6..81fb6c1 100644 --- a/source3/include/vfs.h +++ b/source3/include/vfs.h @@ -253,6 +253,8 @@ All users are now pwrite or async versions. */ /* Version 39 - Remove SMB_VFS_CHMOD_ACL - no longer used. */ /* Version 39 - Remove SMB_VFS_FCHMOD_ACL - no longer used. */ +/* Version 39 - Remove struct dfree_cached_info pointer from + connection struct */ #define SMB_VFS_INTERFACE_VERSION 39 @@ -411,14 +413,6 @@ typedef struct { bool is_wild; } name_compare_entry; -struct dfree_cached_info { - time_t last_dfree_time; - uint64_t dfree_ret; - uint64_t bsize; - uint64_t dfree; - uint64_t dsize; -}; - struct share_params { int service; }; @@ -481,7 +475,6 @@ typedef struct connection_struct { name_compare_entry *veto_list; /* Per-share list of files to veto (never show). */ name_compare_entry *veto_oplock_list; /* Per-share list of files to refuse oplocks on. */ name_compare_entry *aio_write_behind_list; /* Per-share list of files to use aio write behind on. */ - struct dfree_cached_info *dfree_info; struct trans_state *pending_trans; struct rpc_pipe_client *spoolss_pipe; diff --git a/source3/modules/vfs_gpfs.c b/source3/modules/vfs_gpfs.c index a0fd48f..d733df0 100644 --- a/source3/modules/vfs_gpfs.c +++ b/source3/modules/vfs_gpfs.c @@ -243,8 +243,10 @@ static int vfs_gpfs_get_real_filename(struct vfs_handle_struct *handle, char **found_name) { int result; - char *full_path; - char real_pathname[PATH_MAX+1]; + char *full_path = NULL; + char *to_free = NULL; + char real_pathname[PATH_MAX+1], tmpbuf[PATH_MAX]; + size_t full_path_len; int buflen; bool mangled; struct gpfs_config_data *config; @@ -264,8 +266,9 @@ static int vfs_gpfs_get_real_filename(struct vfs_handle_struct *handle, mem_ctx, found_name); } - full_path = talloc_asprintf(talloc_tos(), "%s/%s", path, name); - if (full_path == NULL) { + full_path_len = full_path_tos(path, name, tmpbuf, sizeof(tmpbuf), + &full_path, &to_free); + if (full_path_len == -1) { errno = ENOMEM; return -1; } @@ -275,7 +278,7 @@ static int vfs_gpfs_get_real_filename(struct vfs_handle_struct *handle, result = gpfswrap_get_realfilename_path(full_path, real_pathname, &buflen); - TALLOC_FREE(full_path); + TALLOC_FREE(to_free); if ((result == -1) && (errno == ENOSYS)) { return SMB_VFS_NEXT_GET_REAL_FILENAME( @@ -1612,7 +1615,16 @@ static NTSTATUS vfs_gpfs_get_dos_attributes(struct vfs_handle_struct *handle, if (ret == -1 && errno == EACCES) { ret = get_dos_attr_with_capability(smb_fname, &attrs); } - if (ret == -1) { + + if (ret == -1 && errno == EBADF) { + /* + * Returned for directory listings in gpfs root for + * .. entry which steps out of gpfs. + */ + DBG_DEBUG("Getting winattrs for %s returned EBADF.\n", + smb_fname->base_name); + return map_nt_error_from_unix(errno); + } else if (ret == -1) { DBG_WARNING("Getting winattrs failed for %s: %s\n", smb_fname->base_name, strerror(errno)); return map_nt_error_from_unix(errno); @@ -1879,23 +1891,15 @@ static int vfs_gpfs_ntimes(struct vfs_handle_struct *handle, struct gpfs_winattr attrs; int ret; - char *path = NULL; - NTSTATUS status; struct gpfs_config_data *config; SMB_VFS_HANDLE_GET_DATA(handle, config, struct gpfs_config_data, return -1); - status = get_full_smb_filename(talloc_tos(), smb_fname, &path); - if (!NT_STATUS_IS_OK(status)) { - errno = map_errno_from_nt_status(status); - return -1; - } - /* Try to use gpfs_set_times if it is enabled and available */ if (config->settimes) { - ret = smbd_gpfs_set_times_path(path, ft); + ret = smbd_gpfs_set_times_path(smb_fname->base_name, ft); if (ret == 0 || (ret == -1 && errno != ENOSYS)) { return ret; @@ -1928,7 +1932,7 @@ static int vfs_gpfs_ntimes(struct vfs_handle_struct *handle, attrs.creationTime.tv_sec = ft->create_time.tv_sec; attrs.creationTime.tv_nsec = ft->create_time.tv_nsec; - ret = gpfswrap_set_winattrs_path(discard_const_p(char, path), + ret = gpfswrap_set_winattrs_path(smb_fname->base_name, GPFS_WINATTR_SET_CREATION_TIME, &attrs); if(ret == -1 && errno != ENOSYS){ @@ -2002,8 +2006,6 @@ static bool vfs_gpfs_is_offline(struct vfs_handle_struct *handle, SMB_STRUCT_STAT *sbuf) { struct gpfs_winattr attrs; - char *path = NULL; - NTSTATUS status; struct gpfs_config_data *config; int ret; @@ -2015,24 +2017,17 @@ static bool vfs_gpfs_is_offline(struct vfs_handle_struct *handle, return false; } - status = get_full_smb_filename(talloc_tos(), fname, &path); - if (!NT_STATUS_IS_OK(status)) { - return false; - } - - ret = gpfswrap_get_winattrs_path(path, &attrs); + ret = gpfswrap_get_winattrs_path(fname->base_name, &attrs); if (ret == -1) { - TALLOC_FREE(path); return false; } if ((attrs.winAttrs & GPFS_WINATTR_OFFLINE) != 0) { - DEBUG(10, ("%s is offline\n", path)); - TALLOC_FREE(path); + DBG_DEBUG("%s is offline\n", fname->base_name); return true; } - DEBUG(10, ("%s is online\n", path)); - TALLOC_FREE(path); + + DBG_DEBUG("%s is online\n", fname->base_name); return false; } diff --git a/source3/script/tests/test_dfree_quota.sh b/source3/script/tests/test_dfree_quota.sh index 6e227c4..abd82b4 100755 --- a/source3/script/tests/test_dfree_quota.sh +++ b/source3/script/tests/test_dfree_quota.sh @@ -130,6 +130,35 @@ test_smbclient_dfree() { return $status } +# Issue two queries to different directories in one session to test +# caching effects +test_smbclient_dfree_2() { + name="$1" + share="$2" + dir1="$3" + dir2="$4" + confs="$5" + expected="$6" + subunit_start_test "$name" + setup_conf $confs + output=$($VALGRIND $smbclient //$SERVER/$share \ + -c "cd $dir1; du; cd ..; cd $dir2 ; du" $@ 2>&1) + status=$? + if [ "$status" = "0" ]; then + received=$(echo "$output" | \ + awk '/blocks of size/ {print $1, $5, $6}' | \ + tr '\n' ' ') + if [ "$expected" = "$received" ]; then + subunit_pass_test "$name" + else + echo "$output" | subunit_fail_test "$name" + fi + else + echo "$output" | subunit_fail_test "$name" + fi + return $status +} + test_smbcquotas() { name="$1" conf="$2" @@ -164,6 +193,12 @@ test_smbclient_dfree "Test large disk" dfq "." "conf3 ." "1125899906842624 1024. #basic quota test (SMB1 only) test_smbcquotas "Test user quota" confq1 $USERNAME "40960/4096000/3072000" -U$USERNAME%$PASSWORD --option=clientmaxprotocol=NT1 || failed=`expr $failed + 1` +# Test dfree cache through queries in two different directories +test_smbclient_dfree_2 "Test dfree cache" dfq_cache "." "subdir1" \ + "conf1 . conf2 subdir1" "10 1024. 5 20 1024. 10 " \ + -U$USERNAME%$PASSWORD --option=clientmaxprotocol=SMB3 \ + || failed=`expr $failed + 1` + #quota limit > disk size, remaining quota > disk free test_smbclient_dfree "Test dfree share root df vs quota case 1" dfq "." "confdfq1 ." "80 1024. 40" -U$USERNAME%$PASSWORD --option=clientmaxprotocol=SMB3 || failed=`expr $failed + 1` #quota limit > disk size, remaining quota < disk free diff --git a/source3/smbd/dfree.c b/source3/smbd/dfree.c index a702d08..05f6d69 100644 --- a/source3/smbd/dfree.c +++ b/source3/smbd/dfree.c @@ -21,6 +21,7 @@ #include "smbd/smbd.h" #include "smbd/globals.h" #include "lib/util_file.h" +#include "lib/util/memcache.h" /**************************************************************************** Normalise for DOS usage. @@ -167,48 +168,121 @@ dfree_done: /**************************************************************************** Potentially returned cached dfree info. + + Depending on the file system layout and file system features, the free space + information can be different for different sub directories underneath a SMB + share. Store the cache information in memcache using the query path as the + key to accomodate this. ****************************************************************************/ +struct dfree_cached_info { + time_t last_dfree_time; + uint64_t dfree_ret; + uint64_t bsize; + uint64_t dfree; + uint64_t dsize; +}; + uint64_t get_dfree_info(connection_struct *conn, struct smb_filename *fname, uint64_t *bsize, uint64_t *dfree, uint64_t *dsize) { int dfree_cache_time = lp_dfree_cache_time(SNUM(conn)); - struct dfree_cached_info *dfc = conn->dfree_info; + struct dfree_cached_info *dfc = NULL; + struct dfree_cached_info dfc_new = { 0 }; uint64_t dfree_ret; + char tmpbuf[PATH_MAX]; + char *full_path = NULL; + char *to_free = NULL; + char *key_path = NULL; + size_t len; + DATA_BLOB key, value; + bool found; if (!dfree_cache_time) { return sys_disk_free(conn, fname, bsize, dfree, dsize); } + len = full_path_tos(conn->connectpath, + fname->base_name, + tmpbuf, + sizeof(tmpbuf), + &full_path, + &to_free); + if (len == -1) { + errno = ENOMEM; + return -1; + } + + if (VALID_STAT(fname->st) && S_ISREG(fname->st.st_ex_mode)) { + /* + * In case of a file use the parent directory to reduce number + * of cache entries. + */ + bool ok; + + ok = parent_dirname(talloc_tos(), + full_path, + &key_path, + NULL); + TALLOC_FREE(to_free); /* We're done with full_path */ + + if (!ok) { + errno = ENOMEM; + return -1; + } + + /* + * key_path is always a talloced object. + */ + to_free = key_path; + } else { + /* + * key_path might not be a talloced object; rely on + * to_free set from full_path_tos. + */ + key_path = full_path; + } + + key = data_blob_const(key_path, strlen(key_path)); + found = memcache_lookup(smbd_memcache(), + DFREE_CACHE, + key, + &value); + dfc = found ? (struct dfree_cached_info *)value.data : NULL; + if (dfc && (conn->lastused - dfc->last_dfree_time < dfree_cache_time)) { - /* Return cached info. */ + DBG_DEBUG("Returning dfree cache entry for %s\n", key_path); *bsize = dfc->bsize; *dfree = dfc->dfree; *dsize = dfc->dsize; - return dfc->dfree_ret; + dfree_ret = dfc->dfree_ret; + goto out; } dfree_ret = sys_disk_free(conn, fname, bsize, dfree, dsize); if (dfree_ret == (uint64_t)-1) { /* Don't cache bad data. */ - return dfree_ret; - } - - /* No cached info or time to refresh. */ - if (!dfc) { - dfc = talloc(conn, struct dfree_cached_info); - if (!dfc) { - return dfree_ret; - } - conn->dfree_info = dfc; + goto out; } - dfc->bsize = *bsize; - dfc->dfree = *dfree; - dfc->dsize = *dsize; - dfc->dfree_ret = dfree_ret; - dfc->last_dfree_time = conn->lastused; - + DBG_DEBUG("Creating dfree cache entry for %s\n", key_path); + dfc_new.bsize = *bsize; + dfc_new.dfree = *dfree; + dfc_new.dsize = *dsize; + dfc_new.dfree_ret = dfree_ret; + dfc_new.last_dfree_time = conn->lastused; + memcache_add(smbd_memcache(), + DFREE_CACHE, + key, + data_blob_const(&dfc_new, sizeof(dfc_new))); + +out: + TALLOC_FREE(to_free); return dfree_ret; } + +void flush_dfree_cache(void) +{ + memcache_flush(smbd_memcache(), DFREE_CACHE); +} diff --git a/source3/smbd/proto.h b/source3/smbd/proto.h index 86b6551..7a5ac4c 100644 --- a/source3/smbd/proto.h +++ b/source3/smbd/proto.h @@ -173,6 +173,7 @@ uint64_t sys_disk_free(connection_struct *conn, struct smb_filename *fname, uint64_t *bsize, uint64_t *dfree, uint64_t *dsize); uint64_t get_dfree_info(connection_struct *conn, struct smb_filename *fname, uint64_t *bsize, uint64_t *dfree, uint64_t *dsize); +void flush_dfree_cache(void); /* The following definitions come from smbd/dir.c */ diff --git a/source3/smbd/server_reload.c b/source3/smbd/server_reload.c index c93c077..9b62096 100644 --- a/source3/smbd/server_reload.c +++ b/source3/smbd/server_reload.c @@ -164,6 +164,7 @@ bool reload_services(struct smbd_server_connection *sconn, mangle_reset_cache(); reset_stat_cache(); + flush_dfree_cache(); /* this forces service parameters to be flushed */ set_current_service(NULL,0,True); -- Samba Shared Repository