[sqlite] ​ Use Bash to Store Disc Info in SQLite

[Cecil Westerhof]

2016-01-30 3:40 GMT+01:00 Stephen Chrzanowski :

> On Fri, Jan 29, 2016 at 8:59 PM, Cecil Westerhof 
> wrote:
>
> >
> > > However, I've run into a few problems, so I'm offering my findings and
> > > fixes.
> > >
> > > The first, line 3, you've got a reference to a script that doesn't
> exist
> > > AFAIK.  You should maybe put a check to see if the file exists first,
> > > before running it.  I just deleted that entry from my version of the
> > > script.
> > >
> >
> > ?That is my bash library. I will change it so you can use the script
> > without the library.
> > ?
> >
>
> I figured as such. :]
>

?Looking at the other things you wrote, maybe it is useful to install the
library. And if you like me to add functionalities ?

I changed a few things, from my commit:

Made several changes to make it more useful for other users.

- It now works without the Bash library installed.
- It is now possible to overwrite already saved data.
- When data is already saved and not to be overwritten, just give a
message and quit.
- It is now possible to save ?every? second instead of once a day.
- It is possible to show the raw output (-k) from df besides human readable.
- Removed usage of --output-type, because this did not work everywhere.
- There is an option to only output the values and not save them to
the database.

So it is probably a good idea to fetch the latest:

https://github.com/CecilWesterhof/BashLibrary/blob/master/bin/resources.sh
?

?If there are still issues, do not be afraid to ask.? With --run-often you
could run it every second now. (If your systems are up to it.)


In what I do and where I'd be using this most often, I deal with sick
> computers all the time at a software level.  So if something goes rogue, I
> need to know what is changing in as near real-time as necessary sometimes.
> Once I build a front end for this, I'll be able to better monitor the
> health of a system, or at least determine why a drive is getting full
> quicker than expected.  I might even expand this a bit further to look at
> cataloging file sizes, and then have the front end I build show me exactly
> what files have changed in size.
>

?If you have Java on your systems, I could help you with that.



> Do you have any issues with my using this in a commercial environment?
> It'd be just for diagnostic purposes and not resold.
>

?Well I published it with GPL3:
https://github.com/CecilWesterhof/BashLibrary/blob/master/LICENSE
So there is no harm in that.

I would appreciate it when you share where you use it and which
improvements you make to the script.

And a recommendation about I helped you would not hurt either. ;-)


?I have to update the posting, but need to do some other things first.?

-- 
Cecil Westerhof

[sqlite] ​ Use Bash to Store Disc Info in SQLite

[Cecil Westerhof]

Just a short reply, because I should be sleeping now. ;-)

2016-01-30 1:14 GMT+01:00 Stephen Chrzanowski :

> This is going to be useful for me for a couple of reasons.  First, I've got
> a machine at work that mysteriously fills up, and more importantly, I can
> monitor my Minecraft server at home. ;)
>

?I am glad it is useful.?




> However, I've run into a few problems, so I'm offering my findings and
> fixes.
>
> The first, line 3, you've got a reference to a script that doesn't exist
> AFAIK.  You should maybe put a check to see if the file exists first,
> before running it.  I just deleted that entry from my version of the
> script.
>

?That is my bash library. I will change it so you can use the script
without the library.
?



> On line 8, I'm erroring out with "declare: not found".  At first I thought
> it might be something to do with regexp, but even changing it to a static
> file name, it still bombs.  Then I realized that the file needs to be
> executable.  With a chmod, that works.
>

?Strange, when I check it out the bits are correctly set. But I will add a
note about it.?



Third, my version of df doesn't seem to support the --output parameter.
> I've checked online man pages and I can't find an example or a man page
> containing that parameter.  Removing the parameter, it still works.
>

?I will look into that also. What does ?df --version? give and which
version of Linux are you using? (uname -a)

?



> Fourth, I'd maybe suggest that you either run a delete prior to updating
> for todays date, or, change the resolution of the date to the accuracy of a
> second.  The reason being is if you want to test (Like I'm doing) you're
> going to run into constraint violations. With changing to a second
> resolution, you can run it multiple times to get more results to get real
> time info.  I'll also be changing my version so it doesn't show human
> readable since I'll be looking at a few KB worth of changes at a time.
>

?My idea was that you only run it once a day. But I will change that it can
be run on minute base also. I will also implement a parameter to get rid of
human readable.
?



> On my MC server, I have the current partition setup:
> rootfs36G  19G  15G 57% /
> udev  10M0  10M 0% /dev
> tmpfs397M 164K 397M 1% /run
> /dev/disk/by-uuid/75b09020-4711-48d3-a664-7024a0d16db8  36G  19G  15G 57% /
> tmpfs5.0M0 5.0M 0% /run/lock
>
> The fact that I have two  / mentioned, it errors out.  This might be an
> edge case, but, just something I came across.  I'll have to filter out
> either rootfs or dev/disk.  I'll need to see how the work machines are
> setup (They're Redhat, I'm using Debian at home right now) so I'll decide
> what happens then.
>

?I would only use partitions?
? that are mounted on /dev/... But that is the nice thing about this
solution: what is stored is driven by the database.



> And finally, maybe not the scripts fault, but there was an oddball
> directory made by one of the Minecraft mods that pooched the script.
> Literally, the directory was
> "_!'0!bw"k!(}!~@"y!(:!a@"v!'4!d!"y!'%!}w"r!'`!cg!x!#4!;w!u!$%!:!==".
> I deleted the directory, and it went through completely.  I'll just modify
> the script to ignore this particular directory.
>

?You do not need to delete the directory, just do not put it in the
database to be processed. :-D



> All in all, excellent example.
>

?Thank you.
?



> On Fri, Jan 29, 2016 at 12:47 PM, Cecil Westerhof 
> wrote:
>
> > I wrote the following article:
> > ??
> > Use Bash to Store Disc Info in SQLite
> > ??
> >
> >
> https://www.linkedin.com/pulse/use-bash-store-disc-info-sqlite-cecil-westerhof
> >
> > ?No rocket science, but I thought it could be interesting.
> > ?
> >
> > --
> > Cecil Westerhof
> > ___
> > sqlite-users mailing list
> > sqlite-users at mailinglists.sqlite.org
> > http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users
> >
> ___
> sqlite-users mailing list
> sqlite-users at mailinglists.sqlite.org
> http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users
>



-- 
Cecil Westerhof

[sqlite] ​ Use Bash to Store Disc Info in SQLite

[Simon Slavin]

On 30 Jan 2016, at 12:14am, Stephen Chrzanowski  wrote:

> And finally, maybe not the scripts fault, but there was an oddball
> directory made by one of the Minecraft mods that pooched the script.
> Literally, the directory was
> "_!'0!bw"k!(}!~@"y!(:!a@"v!'4!d!"y!'%!}w"r!'`!cg!x!#4!;w!u!$%!:!==".

I don't know anything about the subject of this thread, but the above string is 
the result of storing a Chinese string as Unicode and reading it back out as 
ASCII.  Thought you might find that useful.  Or at least diverting.

Simon.

[sqlite] ​ Use Bash to Store Disc Info in SQLite

[Stephen Chrzanowski]

On Fri, Jan 29, 2016 at 8:59 PM, Cecil Westerhof 
wrote:

>
> > However, I've run into a few problems, so I'm offering my findings and
> > fixes.
> >
> > The first, line 3, you've got a reference to a script that doesn't exist
> > AFAIK.  You should maybe put a check to see if the file exists first,
> > before running it.  I just deleted that entry from my version of the
> > script.
> >
>
> ?That is my bash library. I will change it so you can use the script
> without the library.
> ?
>

I figured as such. :]


>
> Third, my version of df doesn't seem to support the --output parameter.
> > I've checked online man pages and I can't find an example or a man page
> > containing that parameter.  Removing the parameter, it still works.
> >
>
> ?I will look into that also. What does ?df --version? give and which
> version of Linux are you using? (uname -a)
>
>
On my home machine.

mc at core ~$ uname -a
Linux core 3.2.0-4-amd64 #1 SMP Debian 3.2.73-2+deb7u2 x86_64 GNU/Linux
mc at core ~$ df --version
df (GNU coreutils) 8.13
Copyright (C) 2011 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later .
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.

The RH machines are all RH6.  I think all are 6.5, but not sure.  I can't
be assed to VPN in right now to find out.


>
> > Fourth, I'd maybe suggest that you either run a delete prior to updating
> > for todays date, or, change the resolution of the date to the accuracy
> of a
> > second.  The reason being is if you want to test (Like I'm doing) you're
> > going to run into constraint violations. With changing to a second
> > resolution, you can run it multiple times to get more results to get real
> > time info.  I'll also be changing my version so it doesn't show human
> > readable since I'll be looking at a few KB worth of changes at a time.
> >
>
> ?My idea was that you only run it once a day. But I will change that it can
> be run on minute base also. I will also implement a parameter to get rid of
> human readable.
> ?
>
>
In what I do and where I'd be using this most often, I deal with sick
computers all the time at a software level.  So if something goes rogue, I
need to know what is changing in as near real-time as necessary sometimes.
Once I build a front end for this, I'll be able to better monitor the
health of a system, or at least determine why a drive is getting full
quicker than expected.  I might even expand this a bit further to look at
cataloging file sizes, and then have the front end I build show me exactly
what files have changed in size.


>
>
> > On my MC server, I have the current partition setup:
> > rootfs36G  19G  15G 57% /
> > udev  10M0  10M 0% /dev
> > tmpfs397M 164K 397M 1% /run
> > /dev/disk/by-uuid/75b09020-4711-48d3-a664-7024a0d16db8  36G  19G  15G
> 57% /
> > tmpfs5.0M0 5.0M 0% /run/lock
> >
> > The fact that I have two  / mentioned, it errors out.  This might be an
> > edge case, but, just something I came across.  I'll have to filter out
> > either rootfs or dev/disk.  I'll need to see how the work machines are
> > setup (They're Redhat, I'm using Debian at home right now) so I'll decide
> > what happens then.
> >
>
> ?I would only use partitions?
> ? that are mounted on /dev/... But that is the nice thing about this
> solution: what is stored is driven by the database.
>
>
Read my other note.  It isn't / that caused me the grief, but the type of
FS that is mounted.  /run/lock and /run were both tempfs and your
constraint locked me out based on tempfs.  I changed the schema to enforce
the three fields of date, partition, and my new field mountPoint.


>
>
> > And finally, maybe not the scripts fault, but there was an oddball
> > directory made by one of the Minecraft mods that pooched the script.
> > Literally, the directory was
> > "_!'0!bw"k!(}!~@"y!(:!a@"v!'4!d!"y!'%!}w"r!'`!cg!x!#4!;w!u!$%!:!==".
> > I deleted the directory, and it went through completely.  I'll just
> modify
> > the script to ignore this particular directory.
> >
>
> ?You do not need to delete the directory, just do not put it in the
> database to be processed. :-D
>
>
Poh-tay-toe, poh-tah-toe.  To me, before Simon explained, I thought it was
a bad directory descriptor, so, junked it anyways.  But going forward, the
script didn't like the name for some reason, so it ultimately broke.  I'll
need to figure out how to ensure that doesn't happen at least on my MC
server.


>
>
> > All in all, excellent example.
> >
>
> ?Thank you.
> ?
>
> No problem.

Do you have any issues with my using this in a commercial environment?
It'd be just for diagnostic purposes and not resold.

[sqlite] ​ Use Bash to Store Disc Info in SQLite

[Stephen Chrzanowski]

Thanks for the input Simon.  I'm not sure why that'd end up on this
particular machine.  Must be the Author of the mod doing something
different for preferences, as this is the only directory that shows up.

The contents of the directories were just preference files.  I don't think
it'll affect anything in my game, so, not concerned about it, but will most
certainly keep that tidbit of info in the back of my head.

On Fri, Jan 29, 2016 at 7:25 PM, Simon Slavin  wrote:

>
> On 30 Jan 2016, at 12:14am, Stephen Chrzanowski 
> wrote:
>
> > And finally, maybe not the scripts fault, but there was an oddball
> > directory made by one of the Minecraft mods that pooched the script.
> > Literally, the directory was
> > "_!'0!bw"k!(}!~@"y!(:!a@"v!'4!d!"y!'%!}w"r!'`!cg!x!#4!;w!u!$%!:!==".
>
> I don't know anything about the subject of this thread, but the above
> string is the result of storing a Chinese string as Unicode and reading it
> back out as ASCII.  Thought you might find that useful.  Or at least
> diverting.
>
> Simon.
> ___
> sqlite-users mailing list
> sqlite-users at mailinglists.sqlite.org
> http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users
>

[sqlite] ​ Use Bash to Store Disc Info in SQLite

[Stephen Chrzanowski]

The partition portion failed not because of multiple references to / but
multiple references to tempfs.  Maybe set the constraint to not only the
partition type but also the mount point?

On Fri, Jan 29, 2016 at 7:14 PM, Stephen Chrzanowski 
wrote:

> This is going to be useful for me for a couple of reasons.  First, I've
> got a machine at work that mysteriously fills up, and more importantly, I
> can monitor my Minecraft server at home. ;)
>
> However, I've run into a few problems, so I'm offering my findings and
> fixes.
>
> The first, line 3, you've got a reference to a script that doesn't exist
> AFAIK.  You should maybe put a check to see if the file exists first,
> before running it.  I just deleted that entry from my version of the script.
>
> On line 8, I'm erroring out with "declare: not found".  At first I thought
> it might be something to do with regexp, but even changing it to a static
> file name, it still bombs.  Then I realized that the file needs to be
> executable.  With a chmod, that works.
>
> Third, my version of df doesn't seem to support the --output parameter.
> I've checked online man pages and I can't find an example or a man page
> containing that parameter.  Removing the parameter, it still works.
>
> Fourth, I'd maybe suggest that you either run a delete prior to updating
> for todays date, or, change the resolution of the date to the accuracy of a
> second.  The reason being is if you want to test (Like I'm doing) you're
> going to run into constraint violations. With changing to a second
> resolution, you can run it multiple times to get more results to get real
> time info.  I'll also be changing my version so it doesn't show human
> readable since I'll be looking at a few KB worth of changes at a time.
>
> On my MC server, I have the current partition setup:
> rootfs36G  19G  15G 57% /
> udev  10M0  10M 0% /dev
> tmpfs397M 164K 397M 1% /run
> /dev/disk/by-uuid/75b09020-4711-48d3-a664-7024a0d16db8  36G  19G  15G 57% /
> tmpfs5.0M0 5.0M 0% /run/lock
>
> The fact that I have two  / mentioned, it errors out.  This might be an
> edge case, but, just something I came across.  I'll have to filter out
> either rootfs or dev/disk.  I'll need to see how the work machines are
> setup (They're Redhat, I'm using Debian at home right now) so I'll decide
> what happens then.
>
> And finally, maybe not the scripts fault, but there was an oddball
> directory made by one of the Minecraft mods that pooched the script.
> Literally, the directory was 
> "_!'0!bw"k!(}!~@"y!(:!a@"v!'4!d!"y!'%!}w"r!'`!cg!x!#4!;w!u!$%!:!==".
> I deleted the directory, and it went through completely.  I'll just modify
> the script to ignore this particular directory.
>
> All in all, excellent example.
>
> On Fri, Jan 29, 2016 at 12:47 PM, Cecil Westerhof 
> wrote:
>
>> I wrote the following article:
>> ??
>> Use Bash to Store Disc Info in SQLite
>> ??
>>
>> https://www.linkedin.com/pulse/use-bash-store-disc-info-sqlite-cecil-westerhof
>>
>> ?No rocket science, but I thought it could be interesting.
>> ?
>>
>> --
>> Cecil Westerhof
>> ___
>> sqlite-users mailing list
>> sqlite-users at mailinglists.sqlite.org
>> http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users
>>
>
>

[sqlite] ​ Use Bash to Store Disc Info in SQLite

[Stephen Chrzanowski]

This is going to be useful for me for a couple of reasons.  First, I've got
a machine at work that mysteriously fills up, and more importantly, I can
monitor my Minecraft server at home. ;)

However, I've run into a few problems, so I'm offering my findings and
fixes.

The first, line 3, you've got a reference to a script that doesn't exist
AFAIK.  You should maybe put a check to see if the file exists first,
before running it.  I just deleted that entry from my version of the script.

On line 8, I'm erroring out with "declare: not found".  At first I thought
it might be something to do with regexp, but even changing it to a static
file name, it still bombs.  Then I realized that the file needs to be
executable.  With a chmod, that works.

Third, my version of df doesn't seem to support the --output parameter.
I've checked online man pages and I can't find an example or a man page
containing that parameter.  Removing the parameter, it still works.

Fourth, I'd maybe suggest that you either run a delete prior to updating
for todays date, or, change the resolution of the date to the accuracy of a
second.  The reason being is if you want to test (Like I'm doing) you're
going to run into constraint violations. With changing to a second
resolution, you can run it multiple times to get more results to get real
time info.  I'll also be changing my version so it doesn't show human
readable since I'll be looking at a few KB worth of changes at a time.

On my MC server, I have the current partition setup:
rootfs36G  19G  15G 57% /
udev  10M0  10M 0% /dev
tmpfs397M 164K 397M 1% /run
/dev/disk/by-uuid/75b09020-4711-48d3-a664-7024a0d16db8  36G  19G  15G 57% /
tmpfs5.0M0 5.0M 0% /run/lock

The fact that I have two  / mentioned, it errors out.  This might be an
edge case, but, just something I came across.  I'll have to filter out
either rootfs or dev/disk.  I'll need to see how the work machines are
setup (They're Redhat, I'm using Debian at home right now) so I'll decide
what happens then.

And finally, maybe not the scripts fault, but there was an oddball
directory made by one of the Minecraft mods that pooched the script.
Literally, the directory was
"_!'0!bw"k!(}!~@"y!(:!a@"v!'4!d!"y!'%!}w"r!'`!cg!x!#4!;w!u!$%!:!==".
I deleted the directory, and it went through completely.  I'll just modify
the script to ignore this particular directory.

All in all, excellent example.

On Fri, Jan 29, 2016 at 12:47 PM, Cecil Westerhof 
wrote:

> I wrote the following article:
> ??
> Use Bash to Store Disc Info in SQLite
> ??
>
> https://www.linkedin.com/pulse/use-bash-store-disc-info-sqlite-cecil-westerhof
>
> ?No rocket science, but I thought it could be interesting.
> ?
>
> --
> Cecil Westerhof
> ___
> sqlite-users mailing list
> sqlite-users at mailinglists.sqlite.org
> http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users
>

[sqlite] ​ Use Bash to Store Disc Info in SQLite

[Cecil Westerhof]

I wrote the following article:
??
Use Bash to Store Disc Info in SQLite
??
https://www.linkedin.com/pulse/use-bash-store-disc-info-sqlite-cecil-westerhof

?No rocket science, but I thought it could be interesting.
?

-- 
Cecil Westerhof