Hey Bill,
It's been a couple of years for me but I think it may be an easier task
than you realize. Basically, you install robinhood, configure it for a
pretty beefy SQL db, and then you can just run the POSIX scans manually.
So the ingest script will scan your entire filesystem recursively, which
will take forever, but then once all the data is loaded in the SQL db, it's
all the same from there in terms of reporting.
e.g. in a cron job or manually
/usr/sbin/robinhood -f Z1_uncached --once --scan > /dev/null
# cat /etc/robinhood.d/Z1_uncached.conf
General {
fs_path = "/Z1_uncached";
# filesystem type, as displayed by 'mount' (e.g. ext4, xfs, lustre, ...)
fs_type = nfs4;
}
Log {
log_file = "/var/log/robinhood.log";
report_file = "/var/log/robinhood_actions.log";
alert_file = "/var/log/robinhood_alerts.log";
}
ListManager {
MySQL {
server = localhost;
db = robinhood_Z1_unc;
user = robinhood;
password_file = /etc/robinhood.d/.dbpassword;
}
}
The big difference with Lustre is that you can have the robinhood db update
"on the fly" as the filesystem changes, while with POSIX you have to run
the full recursive scan, which I used to do about once a week, and it would
take several days to run. I also didn't have robinhood apply any kind of
policies or rules; I only did scanning and reporting, but you should be
able to test all that out without involving Lustre.
Depending on the performance of you network and fileservers, the file scan
will take a long time to run and depending on the performance of your db
even the queries may take quite a while to run, e.g.
rbh-du -f Z1_uncached --sum --mega subdirectory #might take like minutes or
hours
And obviously file count is the main thing affects performance...
Regards,
Alex
On Wed, Oct 27, 2021 at 2:38 PM Branson, William via robinhood-support <
[email protected]> wrote:
> Esteemed Colleagues,
>
>
>
> I am writing this letter to ask for your assistance in obtaining more
> detailed information about robinhood as it is deployed in a POSIX only
> environment. I understand that most deployments are likely with the Lustre
> file system.
>
> However, we would like to try it with a POSIX only spin. I have been
> looking for documents that cover this type of deployment, but I can’t seem
> to find too much available. One Slide Deck I did see is named
> “robinhood-Per3S.pdf”, dated January 30th, 2017 *(1)*. This document
> explains a bit about why a POSIX only type of deployment is not efficient.
> Let’s say for arguments sake that we wanted to do a POSIX only Proof of
> Concept. Where would one find a comprehensive list of the built in’s for
> triggers, file classes, policies, and rules? I have found examples in other
> documents, but they do not seem to cover any depth. I suppose what we
> really could use is a document that might guide us to a successful POSIX
> deployment with depth that shows us how triggers, policies, and rules
> interact. We understand the limitations of POSIX only, but our HPC
> deployment of a cluster with storage, is not using the Lustre file system.
>
>
>
> I did find a program that I had never used before, DOXYGEN, a document
> generator program, that when run against a variety of un-complied, well
> annotated, source codes, C, C++, etc., will generate many useable forms of
> documentation. Would you recommend using DOXYGEN in order for us to get a
> more in depth understanding of robinhood policy engine, or is there some
> document or documents that may help us better? I have downloaded and been
> studying the documentation that I can find. I have gone from V3, back to
> V2.5 and I have also been looking at the Lustre documentation to see if I
> can find references to POSIX only deployments and configurations. We have
> the V3.1.7 robinhood installed and have been attempting to get it
> configured correctly to test out the triggers and policies and rules and
> such, but we are not being very successful.
>
>
>
> Thank you in advance for your patience, time, and assistance with this
> endeavor.
>
>
>
> Sincerely
>
>
>
> Bill Branson
>
>
>
> William E. Branson, Jr.
>
> Server Engineer II | Research Computing Center
>
> Medical College of Wisconsin
>
> 8701 W. Watertown Plank Road
>
> Milwaukee, WI 53226-3548
>
> 414-955-2475 | [email protected]
>
>
>
>
>
> 1. Leibovici, T. (n.d.). *From research to industry taking back
> control of HPC file systems *
>
> *with RobinHood Policy Engine*. French Atomic Energy Commission.
> Retrieved October
>
> 27, 2021, from
> http://syst.univ-brest.fr/per3s/wp-content/uploads/2017/02/robinhood-Per3S.pdf
> .
>
>
>
>
>
> Bill Branson
>
> Server Engineer II| Research Computing Center
>
> Medical College of Wisconsin
>
> 414-955-2475 | [email protected]
>
>
> _______________________________________________
> robinhood-support mailing list
> [email protected]
> https://lists.sourceforge.net/lists/listinfo/robinhood-support
>
_______________________________________________
robinhood-support mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/robinhood-support
