Re: [Samba] memory consumption with treesize pro and cifs shares

[Linda W]

(some more followup---sorry if I ask too much / too many Q's,
if so, just don't respond! I won't be offended)..

You might look for a file system loop and check for options in treesize pro
to detect such.

Another program to try is "WinDirStat's home is 
http://windirstat.sourceforge.net/.


The reason I mention the loop stuff is that windir stat has options to 
detect

remote mounts and remote symbolic links and to follow them or not.

That can cause it to go in loops but not exhaust memory on th server.

Running it now in normal mode, it runs more aggressively against the 
server and
I see the smbd process at 90% cpu usage  (the fact that the protocol is 
single-server/client

makes it difficult to parallelize cpu usage, so 90% is how much of 1 core
it is using, vs. system wide it would be about 6.3%).

Note -- my instance of 3.6.12 is running with millions of files as well 
(a bit over
9 million at last count) and is running on linux-3.9.8.  I'm not sure, 
but I think
linux's multi-tasking ability is considered more efficient than BSD's, 
though BSD is
has had some record of better security -- though to both those figures 
the relative user
bases need to be considered (fewer users, fewer bugs found, more users, 
more need
for different types of HW and efficient algorithms to handle high loads 
across diverse
platforms, as well as the ability to keep source closed under BSD 
(security through obscurity)).



Of note -- on the server, I see two instances of smbd running -- the 
other is from
another machine where I have a logon instance in windows that I left 
"suspended"
(disconnected from a remote session, so not really suspended -- looks 
like explorer doing
some sort of indexing (which MS refuses to allow their indexer to use to 
update a computer's

local index)).

Anyway, the two of them are running right around 129-137MB Virtual size 
for each

with 24MB resident and 20M Shared...
Server's cpu's are Xeon X5660 @2.8Gh currently running at 1600MHz 
(demand based
scheduler), so they aren't the fastest or the slowest.  Main disk 
subsystem is a RAID50
of reasonable speed.  I would think it unlikely, but perhaps a slow disk 
might cause

a backlog of requests... but I don't see that as likely.

Evidence points to the BSD-samba combination you are using.  :(.
You didn't mention --
* what processor/how many cores the NAS is using/has available?
* What type of disk are in use (Sata/SAS 4000RPM - 15K RPM);
* Is the system using RAID? Type?
* What file system is it using (options?)

Personally, in my limited exploration of home NAS units, I didn't
find any that were "well powered"; not as even as much as a
low-end workstation based server.  With
48G, your's already sounds better than most, but that's only 1 measure.

Also note, my version of treesizepro isn't the latest, it's officially 
'OUTDATED' (says
so next to the version 5.4.4.707);-).  Newer versions may be more 
aggressive

or have different options.



Cy Mike wrote:

More info on this:

The NAS running FreeBSD has 48GB RAM, same as the test NAS we are 
duplicating the error on. Both machines see this error with 3.6.9 
Samba. The initial try at duplicating the error didn't produce it. It 
wasn't until we increased the amount of files in the CIFS share that 
we were able to duplicate it. Number of files is in the millions. 
Drive freespace is large on the test machine and the error still 
occurs. According to LindaW here, the test hasn't been reproduce yet 
using Samba 3.6.16, so we're looking into another test on our box 
using the updated version.


Has anyone else encountered an issue like with using TreeSize Pro? 
Does anyone need more information to help sort this out? We'll be 
running additional tests today looking for a solution and I will post 
back more on this later.�


Thanks,
Mike�


On Tue, Jul 23, 2013 at 6:59 PM, Cy Mike > wrote:


Hi everyone. I'm looking to solve an issue with Samba on a NAS
being accessed with TreeSize Pro. Using that program to scan
through millions of files is eating up memory on swap and
eventually crashing the system. It's scanning mounted CIFS shares
on the NAS running TrueNAS with samba version 3.6.9

We have a test case and have been able to replicate the issue on
another machine.

The "solution" right now is to simply not run TreeSize Pro. Not
the best of plans.

In the meantime, I'm going to continue to check the usual
manuals/google sources to see if I can find anything. I haven't as
yet and am short on time with this. Basically looking to see if
this is an actual bug that might require a patch/upgrade, or
something I can fix with some tuneables.�

Thanks,
Mike




--
"that's not a bald head, that's a solar panel for a dumbass machine" - 
jon stewart 5/9/12


--
To unsubscribe from this list go to the following URL and read the
instructions:  https://lists.samba.org/mailman/options/samba

Re: [Samba] memory consumption with treesize pro and cifs shares

[Cy Mike]

More info on this:

The NAS running FreeBSD has 48GB RAM, same as the test NAS we are
duplicating the error on. Both machines see this error with 3.6.9 Samba.
The initial try at duplicating the error didn't produce it. It wasn't until
we increased the amount of files in the CIFS share that we were able to
duplicate it. Number of files is in the millions. Drive freespace is large
on the test machine and the error still occurs. According to LindaW here,
the test hasn't been reproduce yet using Samba 3.6.16, so we're looking
into another test on our box using the updated version.

Has anyone else encountered an issue like with using TreeSize Pro? Does
anyone need more information to help sort this out? We'll be running
additional tests today looking for a solution and I will post back more on
this later.

Thanks,
Mike


On Tue, Jul 23, 2013 at 6:59 PM, Cy Mike  wrote:

> Hi everyone. I'm looking to solve an issue with Samba on a NAS being
> accessed with TreeSize Pro. Using that program to scan through millions of
> files is eating up memory on swap and eventually crashing the system. It's
> scanning mounted CIFS shares on the NAS running TrueNAS with samba version
> 3.6.9
>
> We have a test case and have been able to replicate the issue on another
> machine.
>
> The "solution" right now is to simply not run TreeSize Pro. Not the best
> of plans.
>
> In the meantime, I'm going to continue to check the usual manuals/google
> sources to see if I can find anything. I haven't as yet and am short on
> time with this. Basically looking to see if this is an actual bug that
> might require a patch/upgrade, or something I can fix with some tuneables.
>
> Thanks,
> Mike
>



-- 
"that's not a bald head, that's a solar panel for a dumbass machine" - jon
stewart 5/9/12
-- 
To unsubscribe from this list go to the following URL and read the
instructions:  https://lists.samba.org/mailman/options/samba

Re: [Samba] memory consumption with treesize pro and cifs shares

[Linda W]

Cy Mike wrote:
> Hi everyone. I'm looking to solve an issue with Samba on a NAS being
> accessed with TreeSize Pro. Using that program to scan through millions of
> files is eating up memory on swap and eventually crashing the system.
---
Which system is crashing?  the NAS or the one running the scan?... I
think you mean the
NAS, but I'm not sure.

I ran 2 copies on different partitions at the same time but didn't
reproduce the problem.
(Never got close).

Note -- at one point I had a problem with TSP finding more disk space on
my disk(S) than
I had...

Under the Tools-Options menu, there is an option which used to default
to 'on'...

"Follow Mount Points and External Symbolic Links to Directories"


That caused it to go out for lunch, a sauna and an afternoon swim...i.e.
it ran into
loops but only TreeSize kept growing bit by bit.

Are you running smb32 or 64?

How big are the disks you are searching through?

Are you using default size INodes or larger?  How much speed to do you get
when using 'dd' to transfer a file?  (ln /dev/zero ~/zero; ln /dev/null
~/null:
dd if=/dev/null of=/dev/zero bs=16-64M seems optimal,

I'm not on the samba team, but am curious -- wanted to see if I could
repro it with
my current samba and server.  I'm running samba 3.6.16 and my server is x64.

You running 32 or 64bit?  maybe that is contributing?








>  It's
> scanning mounted CIFS shares on the NAS running TrueNAS with samba version
> 3.6.9
>
> We have a test case and have been able to replicate the issue on another
> machine.
>   

Awesome...  I only have TreeSize Professional 5.4.4.707 (didn't
bother to upgrade since
this one worked for my needs at the time).

I currently have 3 copies of it running
> The "solution" right now is to simply not run TreeSize Pro. Not the best of
> plans.
>
> In the meantime, I'm going to continue to check the usual manuals/google
> sources to see if I can find anything. I haven't as yet and am short on
> time with this. Basically looking to see if this is an actual bug that
> might require a patch/upgrade, or something I can fix with some tuneables.
>
> Thanks,
> Mike
>   

-- 
To unsubscribe from this list go to the following URL and read the
instructions:  https://lists.samba.org/mailman/options/samba

Re: [Samba] memory consumption with treesize pro and cifs shares

[Gregory Carter]

Add more memory or split the volume into smaller shares.

Gc

Cy Mike  wrote:

>Hi everyone. I'm looking to solve an issue with Samba on a NAS being
>accessed with TreeSize Pro. Using that program to scan through millions of
>files is eating up memory on swap and eventually crashing the system. It's
>scanning mounted CIFS shares on the NAS running TrueNAS with samba version
>3.6.9
>
>We have a test case and have been able to replicate the issue on another
>machine.
>
>The "solution" right now is to simply not run TreeSize Pro. Not the best of
>plans.
>
>In the meantime, I'm going to continue to check the usual manuals/google
>sources to see if I can find anything. I haven't as yet and am short on
>time with this. Basically looking to see if this is an actual bug that
>might require a patch/upgrade, or something I can fix with some tuneables.
>
>Thanks,
>Mike
>-- 
>To unsubscribe from this list go to the following URL and read the
>instructions:  https://lists.samba.org/mailman/options/samba
>
-- 
To unsubscribe from this list go to the following URL and read the
instructions:  https://lists.samba.org/mailman/options/samba

[Samba] memory consumption with treesize pro and cifs shares

[Cy Mike]

Hi everyone. I'm looking to solve an issue with Samba on a NAS being
accessed with TreeSize Pro. Using that program to scan through millions of
files is eating up memory on swap and eventually crashing the system. It's
scanning mounted CIFS shares on the NAS running TrueNAS with samba version
3.6.9

We have a test case and have been able to replicate the issue on another
machine.

The "solution" right now is to simply not run TreeSize Pro. Not the best of
plans.

In the meantime, I'm going to continue to check the usual manuals/google
sources to see if I can find anything. I haven't as yet and am short on
time with this. Basically looking to see if this is an actual bug that
might require a patch/upgrade, or something I can fix with some tuneables.

Thanks,
Mike
-- 
To unsubscribe from this list go to the following URL and read the
instructions:  https://lists.samba.org/mailman/options/samba