Hi Alexei, I knopw about jira contributions, but I don't think this should be included into the OM disrtibutions. Looks more like a "tip and tricks" item, you know, things you put in a wiki, than a script to be added to the distribution (as I've said it's quick and dirty). It can be used until OM has it's own clean way of handling these operations (User quota, file expirations, ...).
By the way, here is the little brother of the first script, aimed at dealing with files uploaded to rooms. Regards, Thibault Le 06/04/2012 12:56, Alexei Fedotov a écrit : > Thibault, > > thanks for your contribution. The recommended way is to contribute via > jira attachment, in that case you transfer rights to ASF correctly. > > -- > With best regards / с наилучшими пожеланиями, > Alexei Fedotov / Алексей Федотов, > http://dataved.ru/ > +7 916 562 8095 > > > > On Fri, Apr 6, 2012 at 2:28 PM, Thibault Le Meur > <[email protected]> wrote: >> Hi, >> >> It seems that OM doesn't delete streams form the filesystem. Thus I >> needed a way to track diskspace usage of streams so I developped a "not >> so quick and yet dirty script" to do the following tasks: >> * delete streams marked as deleted in OpenMeetings DB from the filesystem >> * report diskspace used by not deleted streams with associated >> information (owner, room, ...) >> * report diskspace per user, and send an overquota email if needed >> * report streams older than and Expiration duration threshold and send >> an expriation email if needed >> >> It requires: >> * linux system >> * openmeetings with mysql DB (tested on openmeetings_1_9_1_r4707) >> * mysql client, a mysqk user able to do selects to the OM DB >> * usual linux tools like cut, awk, sed and bash 4.x >> >> It comes with no warranty as I had to reverse engineer the DB to >> understand how streams ar recorded and marked as deleted and I'm quite >> sure there are things I missed so It may be considered as a "template". >> I just post it here, in case it can prove to be usefull to someone. >> >> Now I would need help on how uploaded files are recorded to filesystem >> and DB to do the same for these ones. If anyone can help me... ;-) >> >> Regards, >> Thibault
#!/bin/bash - #title :openmeetings_userfiles.sh #description :Delete unused openmeetings user's files and report disk usage or expired user's files #requires :openmeetings with mysql database, mysql client with a user to read the DB, bash 4.x #author :Thibault Le Meur #date :20120405 #version :0.2 #usage :./openmeetings_userfiles.sh #notes : #bash_version :4.1.5(1)-release #============================================================================ #set -x # Setup ## The directory where your OM streams are UPLOADDIR="/opt/red5/webapps/openmeetings/upload/files" ## The MYSQL command line client MYSQL="/usr/bin/mysql" ## The MYSQL User (must have the SELECT right on all openmeetings tables) MYSQLUSR="meetingsinfo" ## The MYSQL Password of user MYSQLUSR MYSQLPWD="VerySecretPassword" ## The MYSQL database for openmeetings MYSQLDB="openmeetings" ## PERUSERQUOTA=20971520 // 20MB PERUSERQUOTA=20971520 ## MAIL command, set to empty string if you don't want user notification MAIL="/usr/bin/mail" ## APPNAME: application name APPNAME="OpenMeetings" ## APPURL: URL of the openmeetings instance APPURL="http://myopenmeetings.mydom.org:5080/openmeetings"; ## ADMINEMAIL: email address of the administrator ADMINEMAIL="[email protected]" ## TIMELIMIT: number of seconds to keep a recording (604800 = 7 days) TIMELIMIT=604800 ## VERBOSE: if set to "Yes", then the scripts outputs information about the process VERBOSE="Yes" localdir=`dirname $0` if [ -r $localdir/local_openmeetings_userfiles.cfg ] then . $localdir/local_openmeetings_userfiles.cfg fi # BEGIN: Helper functions section ## bytesToHuman: returns the Human readable string that corresponds to the number of bytes given in arg1 ## - arg1: <nb of bytes> ## - stdout: string representing the number of bytes in human readable format ## - return: n/A bytesToHuman () { if [ -z $1 -o $1 -eq 0 ] then echo 0 else echo $1 | awk 'BEGIN{hum[1024^3]="Gb";hum[1024^2]="Mb";hum[1024]="Kb";} {for (x=1024^3; x>=1024; x/=1024){ if ($1>=x) { printf "%.2f %s\n",$1/x,hum[x];break };}}'; fi } ## isNotSet: returns 1 if the key "k" of associative array "A" is not set ## - arg1: A[k] ## - stdout: n/a ## - return: 1 if not set isNotSet() { if [[ ! ${!1} && ${!1-_} ]] then return 1 fi } ## logMe: outputs the log message only if VERBOSE mode is enabled logMe() { if [ $VERBOSE = "Yes" ] then echo "$1" fi } # END: Helper functions section # BEGIN: purge of unused streams ## this deletes unsused streams (which are flagged as deleted in OM DB) logMe "BEGIN purging unsused user files flagged as deleted in OM DB" dbquery=$($MYSQL --batch -u $MYSQLUSR -p$MYSQLPWD $MYSQLDB -e "select concat_ws(';',f.fileexploreritem_id,u.login,e.email,f.filehash,f.filename,REPLACE(r.name,' ','_'),DATE_FORMAT(f.updated,'%Y-%m-%d--%T')) as HEADERTODELETE from fileexploreritem as f, rooms as r, users as u, adresses as e WHERE f.owner_id=u.user_id AND f.room_id=r.rooms_id AND u.adresses_id=e.adresses_id and f.deleted='true'" | grep -v HEADERTODELETE) totalSavedBytes=0 for file in $dbquery do fid=`echo $file | cut -d';' -f1` userlogin=`echo $file | cut -d';' -f2` useremail=`echo $file | cut -d';' -f3` filebase1=`echo $file | cut -d';' -f4` origfile=`echo $file | cut -d';' -f5` roomname=`echo $file | cut -d';' -f6` enddate=`echo $file | cut -d';' -f7` if [ -n "$filebase1" -a -d "$UPLOADDIR/$filebase1" ] then savedBytes=`du -b --max-depth=0 $UPLOADDIR/$filebase1 | cut -f1` find $UPLOADDIR -type d -name $filebase1 -exec rm -r {} + savedBytesHr=`bytesToHuman $savedBytes` totalSavedBytes=`expr $totalSavedBytes + $savedBytes` logMe " * Deleting $savedBytesHr from userfile '$origfile' for user '$useremail' [in room $roomname date: $enddate]" fi done # Now let's report used file size (which are flagged as not deleted in OM DB) logMe "BEGIN reporting used files" dbquery=$($MYSQL --batch -u $MYSQLUSR -p$MYSQLPWD $MYSQLDB -e "select concat_ws(';',f.fileexploreritem_id,u.login,e.email,f.filehash,f.filename,REPLACE(r.name,' ','_'),DATE_FORMAT(f.updated,'%Y-%m-%d--%T')) as HEADERTODELETE from fileexploreritem as f, rooms as r, users as u, adresses as e WHERE f.owner_id=u.user_id AND f.room_id=r.rooms_id AND u.adresses_id=e.adresses_id and f.deleted='false'" | grep -v HEADERTODELETE) totalUsedBytes=0 declare -A perUser declare -A perUserExpired nowTimestamp=`date +%s` deleteBefore=`expr $nowTimestamp - $TIMELIMIT` for file in $dbquery do fid=`echo $file | cut -d';' -f1` userlogin=`echo $file | cut -d';' -f2` useremail=`echo $file | cut -d';' -f3` filebase1=`echo $file | cut -d';' -f4` origfile=`echo $file | cut -d';' -f5` roomname=`echo $file | cut -d';' -f6` enddate=`echo $file | cut -d';' -f7 | sed 's/--/ /'` enddatets=`date -d "$enddate" +%s` usedBytes=`du -b --max-depth=0 $UPLOADDIR/$filebase1 | cut -f1` usedBytesHr=`bytesToHuman $usedBytes` totalUsedBytes=`expr $totalUsedBytes + $usedBytes` if [ $usedBytes -gt 0 ] then logMe " * Used $usedBytesHr for file '$origfile' by user '$useremail' [in room $roomname date: $enddate]" ### check for expired streams if [ $enddatets -lt $deleteBefore ] then perUserExpired["${useremail}"]="${perUserExpired["$useremail"]} * Expired file '$origfile' ($usedBytesHr) since $enddate in room '$roomname' \n" fi isNotSet perUser["$useremail"] if [ $? -ne 0 ] then # Not set yet perUser["${useremail}"]=$usedBytes else perUser["${useremail}"]=`expr ${perUser["$useremail"]} + $usedBytes` fi fi done logMe "BEGIN sending email to users for expired files notifications" for user in "${!perUserExpired[@]}" do msg=${perUserExpired[$user]} logMe " * Expired files sending email notification to '$user'" if [ -n "$MAIL" ] then $MAIL -s "Expired files in $APPNAME" $user <<EOM Some of your uploaded files on $APPNAME have expired. Please review the list below: `echo -e "$msg"` 1) Please log in to $APPURL, join a room, and select the Files tab. 2) Right click the file entry, and select "Delete file" option Regards, $ADMINEMAIL EOM fi done logMe "BEGIN reviewing users files diskspace usage" for user in "${!perUser[@]}" do readableUsage=`bytesToHuman ${perUser[$user]}` readableQuota=`bytesToHuman $PERUSERQUOTA` logMe " * Diskspace files usage for '$user' is $readableUsage" if [ ${perUser[$user]} -gt $PERUSERQUOTA ] then logMe " ** Authorized files quota of $readableQuota exceeded, sending email notification" if [ -n "$MAIL" ] then $MAIL -s "Quota of uploaded files exceeded for $APPNAME" $user <<EOM You're using $readableUsage for your uploaded files on $APPNAME, this is more than the $readableQuota authorized quota. 1) Please log in to $APPURL, join a room, and select the Files tab. 2) Right click the file entry, and select "Delete file" option Regards, $ADMINEMAIL EOM fi fi done
