On Tue, Sep 23, 2014 at 12:29:22AM +0300, Andriy Gapon wrote:
> On 23/09/2014 00:22, David Wolfskill wrote:
> > ... I rather wish I could get the same information via sysctl.  (Well,
> > something seems to be available via the "opaque" kern.devstat.all
> > sysctl(8) variable, but sysctl(8) doesn't display all of it, and parsing
> > it seems as if that would require knowledge about the internals of the
> > system where the data were acquired.)
> 
> Perhaps sysutils/devstat could be of help?
> ...

On Tue, Sep 23, 2014 at 08:56:54AM +0200, Borja Marcos wrote:
> ... 
> Reading sysctl from a small C program is not hard at all. I did it for 
> devilator (a data recollector for Orca). And
> there's a lot of data available. An advantage is, you avoid launching several 
> processes.
> ...

On Tue, Sep 23, 2014 at 10:45:14AM +0200, Borja Marcos wrote:
> ... 
> Anyway, for disk stats GEOM offers a nice API. You can get delays per GEOM 
> provider, bandwidths, etc.

On Tue, Sep 23, 2014 at 11:38:44AM +0300, Stefan Parvu wrote:
> ... 
> I gave up parsing sysctl via Perl for disks and network devices. It would be
> nice to have devstat properly working via sysctl for disk devices. Similar way
> kern.cp_times does. Currently there is no simple way to extract per disk 
> stats from
> sysctl as a Perl or Sh consumer, unless we build a C module to do that. 


Folks, I appreciate the suggestions, but they address problems other
than the one I am trying to solve.

In particular:
* I require that the tool must only depend on components of base FreeBSD;
  thus, I don't need to perturb the system I want to measure by
  installing otherwise unneeded software on it.

* I am using a shell script (which uses date, sysctl, netstat, and awk)
  so I don't need to recompile my data acquisition tool.

* The tasks I am trying to measure are software builds similar to
  a stable/10 "make universe" -- but about 2 - 3 times the duration
  (and reading and writing a significantly larger amount of data).

  Thus, the number of additional processes caused by my probe firing
  even as often as once/second is lost in the noise.



> ...
> > If iostat(8) could be taught to (optionally) provide a timestamp, that
> > might suffice.
> 
> In fact all performance userland tools should be able to nicely produce 
> timestamp
> CSV records which can be used for capacity planning and feed them to an
> analytic product. Something like:
> 
> 1411445589:4.01:16.03:383.97:1.93:0.29:1.68:0.11:95.99:0.00:0.00:0.00:0.00:123.00:229516.00:570992.00
> 
> or something like iostat:
> 
> 1411445733:ada0:0.00:0.00:0.00:0.00:0.00:0.00:0.00
> 
> where the first field would be always the timestmp, unix time. It is not that 
> complicated
> but it does not exist.


The output of my shell script may be described as:

#   <record>    ::= <fieldlist> newline
#   <fieldlist> ::= <field> | <field> tab <fieldlist>
#   <field>     ::= <tag> : <value>
#   <tag>       ::= [_0-9a-zA-Z][-_.0-9a-zA-Z]+
#   <value>     ::= [^\t\n]*

# Each record will have a field with a tag called "time"; the
# associated value will be the number of seconds since the Epoch,
# but will be coerced as necessary to ensure that it is monotonically
# increasing.

# Note that the colon (":") is a valid character in the value part
# of a field (but not in the tag).  Further, there is no whitespace
# on either side of that delimiting colon: everything to the left of the
# colon (up to, but not including, the previous tab, if any) is part of the
# tag; everything to the right of the colon (up to, but not including,
# the next tab or newline) is part of the value.

A prior version of the script output CSV; for this version, I chose to
use the above format because I had situations where some fields showed
up on some lines, but not on others.  That tends to make the use of CSV
a bit problematic.  (On a machine where I post-process the collected
data, I have some Perl that reads the above format, creates new field
names as necessary to cope with multivariate data (e.g., kern.cp_time or
vm.loadavg), and generates CSV from the result.)

> ...
> My idea of having standard data recorders: sysrec, cpurec, nicrec diskrec 
> which can extract: overall system consumption, per device statistics.
> http://www.systemdatarecorder.org/recording/agents.html 
> ...

That looks interesting and useful for a broad class of similar problems.

However, as far as I can tell, it is not suitable for the problem(s) I
am trying to solve in this case.

Basically, I have something that works "well enough" for things
like CPU counters, memory usage (at the rather coarse granularity
that top(1) provides, vs. "vmstat -m" output), load avergaes, and
NIC counters, and is readily extensible to any univariate (or simple
list of multivariate) (non-opaque) sysctl OIDs.  I'd like to be
able to include information from the I/O subsystem -- in particular,
data that is accessible from "iostat -x".

Peace,
david
-- 
David H. Wolfskill                              da...@catwhisker.org
Taliban: Evil cowards with guns afraid of truth from a 14-year old girl.

See http://www.catwhisker.org/~david/publickey.gpg for my public key.

Attachment: pgpSn41D4lnuF.pgp
Description: PGP signature

Reply via email to