[FOSSology] Move Folder Contents
Good Morning Everyone. My users would like to know if folder contents can be moved into another folder. They recently created another folder in order to organize and prepare for cleanup of the repository. They now want the remaining uploads to be located in that single new folder. Thanks, Ray W. CONFIDENTIALITY NOTICE: This e-mail and any files transmitted with it are intended solely for the use of the individual or entity to whom they are addressed and may contain confidential and privileged information protected by law. If you received this e-mail in error, any review, use, dissemination, distribution, or copying of the e-mail is strictly prohibited. Please notify the sender immediately by return e-mail and delete all copies from your system. ___ fossology mailing list fossology@fossology.org http://fossology.org/mailman/listinfo/fossology
Re: [FOSSology] Clean Out Repository
Thanks for the response Bob. I'm sorry Bob; I don't understand how to use the delete agent. When I list the available default agents as fossy or root ... fossrepo]$ fossjobs -a The available agents are: agent_bucket agent_copyright agent_mimetype agent_nomos agent_pkgagent agent_unpack agent_specagent agent_license PHP Notice: Undefined index: U in /usr/bin/fossjobs on line 167 Is this list complete? Thanks, Ray W. -Original Message- From: Bob Gobeille [mailto:bob.gobei...@hp.com] Sent: Tuesday, June 14, 2011 12:27 PM To: Westphal, Raymond W Cc: fossology@fossology.org Subject: Re: [FOSSology] Clean Out Repository Hi Ray, To find the uploads over a year old: select upload_pk from upload where upload_ts (now() - interval '1 year') order by upload_ts You can then take that list and schedule the delete agent on each one of the upload_pk's with the fossjobs command (man fossjobs). Unfortunately, you can't simply delete the upload from the database because that won't delete the files from the repository, and because cascade isn't turned on for most of the foreign keys. So, at least for now, you have to use the delete agent. Bob Gobeille On Jun 14, 2011, at 6:29 AM, Westphal, Raymond W wrote: Hello Everyone. I'm trying to find a way to automatically cleanup the repository. The users will allow me to delete uploads older than 1 year. I created the following query from watching the database while I selected the Organize/Uploads/Delete Uploaded File menu option. SELECT upload_pk, upload_desc, upload_ts, ufile_name FROM foldercontents,uploadtree,upload WHERE foldercontents.parent_fk = '4' AND upload_ts (select current_date -365) AND foldercontents.foldercontents_mode = 2 AND foldercontents.child_id = upload.upload_pk AND uploadtree.upload_fk = upload.upload_pk AND uploadtree.parent IS NULL; ORDER BY upload_ts; Can I simply delete from the upload table where upload_ts (select current_date -365) ? Thanks, Ray W. CONFIDENTIALITY NOTICE: This e-mail and any files transmitted with it are intended solely for the use of the individual or entity to whom they are addressed and may contain confidential and privileged information protected by law. If you received this e-mail in error, any review, use, dissemination, distribution, or copying of the e-mail is strictly prohibited. Please notify the sender immediately by return e-mail and delete all copies from your system. ___ fossology mailing list fossology@fossology.org http://fossology.org/mailman/listinfo/fossology ___ fossology mailing list fossology@fossology.org http://fossology.org/mailman/listinfo/fossology
[FOSSology] Clean Out Repository
Hello Everyone. I'm trying to find a way to automatically cleanup the repository. The users will allow me to delete uploads older than 1 year. I created the following query from watching the database while I selected the Organize/Uploads/Delete Uploaded File menu option. SELECT upload_pk, upload_desc, upload_ts, ufile_name FROM foldercontents,uploadtree,upload WHERE foldercontents.parent_fk = '4' AND upload_ts (select current_date -365) AND foldercontents.foldercontents_mode = 2 AND foldercontents.child_id = upload.upload_pk AND uploadtree.upload_fk = upload.upload_pk AND uploadtree.parent IS NULL; ORDER BY upload_ts; Can I simply delete from the upload table where upload_ts (select current_date -365) ? Thanks, Ray W. CONFIDENTIALITY NOTICE: This e-mail and any files transmitted with it are intended solely for the use of the individual or entity to whom they are addressed and may contain confidential and privileged information protected by law. If you received this e-mail in error, any review, use, dissemination, distribution, or copying of the e-mail is strictly prohibited. Please notify the sender immediately by return e-mail and delete all copies from your system. ___ fossology mailing list fossology@fossology.org http://fossology.org/mailman/listinfo/fossology
Re: [FOSSology] 1.2.1 to 1.4?
I forgot about that. Yes, I had a similar error. I renamed the copyright directory to copyright.OLD and tried again. And after the install there was an issue with an unusually named copyright directory. So you may want to check for another copyright. (X can be 0-9 or a-z) directory that was created around the time of the first install and remove it. Mary may recall why that weird copyright directory caused an issue. Ray W. From: fossology-boun...@fossology.org [mailto:fossology-boun...@fossology.org] On Behalf Of Fay Michael T Sent: Wednesday, June 01, 2011 12:35 PM To: 'fossology@fossology.org' Subject: Re: [FOSSology] 1.2.1 to 1.4? Okay. I added p7zip-plugins which was needed and ran rpm -Uvh fossology-1.3...rpm and got the following error: # rpm -Uvh fossology-1.3.0-1.el5.i386.rpm Preparing...### [100%] 1:fossology ### [100%] error: unpacking of archive failed on file /usr/lib/fossology/agents/copyright: cpio: rename failed - Is a directory Any ideas? From: Laser, Mary [mailto:mary.la...@hp.com] Sent: Tuesday, May 31, 2011 10:47 AM To: Fay Michael T; 'fossology@fossology.org' Subject: RE: 1.2.1 to 1.4? Hi Michael, We have not tested this. To be safe, I would do as this fossology (rpm) user proposed: I'll backup the database (filesystem level backup plus a dump of the database. Then can I simply perform an rpm -Uvh fossology-1.3.0-1.el5.x86_64.rpm followed by a rpm -Uvh fossology-1.4.0-1.el5.x86_64.rpm Also, Be sure to start the scheduler after upgrade #1. Bring up the UI and browse your repository to verify all is well before proceeding with the 1.4.0 upgrade. Please let us know how it goes. Thanks, Mary -Original Message- From: fossology-boun...@fossology.org [mailto:fossology- boun...@fossology.org] On Behalf Of Fay Michael T Sent: Tuesday, May 31, 2011 8:49 AM To: 'fossology@fossology.org' Subject: [FOSSology] 1.2.1 to 1.4? Has upgrading from 1.2.1 to 1.4 been tested? Or do you really need to go to 3 and then to 4? ___ fossology mailing list fossology@fossology.org http://fossology.org/mailman/listinfo/fossology CONFIDENTIALITY NOTICE: This e-mail and any files transmitted with it are intended solely for the use of the individual or entity to whom they are addressed and may contain confidential and privileged information protected by law. If you received this e-mail in error, any review, use, dissemination, distribution, or copying of the e-mail is strictly prohibited. Please notify the sender immediately by return e-mail and delete all copies from your system. ___ fossology mailing list fossology@fossology.org http://fossology.org/mailman/listinfo/fossology
[FOSSology] Cygwin Analysis causing scheduler restarts
Good Morning Everyone. On Friday my customer uploaded Cygwin for analysis. I think it was a zip file. The job ran from 11:23am until I deleted it at around 8am on Sunday. I had to delete it because it caused Fossology scheduler to keep restarting. And that was causing the load average on the server to slowly climb. Has anyone had any experience with analyzing Cygwin? Any suggestions, please? Thanks, Thanks, Ray W. CONFIDENTIALITY NOTICE: This e-mail and any files transmitted with it are intended solely for the use of the individual or entity to whom they are addressed and may contain confidential and privileged information protected by law. If you received this e-mail in error, any review, use, dissemination, distribution, or copying of the e-mail is strictly prohibited. Please notify the sender immediately by return e-mail and delete all copies from your system. ___ fossology mailing list fossology@fossology.org http://fossology.org/mailman/listinfo/fossology
Re: [FOSSology] Cygwin Analysis causing scheduler restarts
Good Morning Bob/Good Morning Mary. Here is the info. from the log just prior to where the scheduler started its cycle of not responding and then restarting. 2011-05-20 13:51:46 scheduler[8691] : Child[1] 'agent=unpack host=localhost ' state=FREEING(2) @ Fri May 20 13:51:46 2011 2011-05-20 13:51:46 scheduler[8691] : Child[1] 'agent=unpack host=localhost ' state=FREE(1) @ Fri May 20 13:51:46 2011 2011-05-20 13:51:48 scheduler[8691] : Child[2] 'agent=adj2nest host=localhost ' state=SPAWNED(4) @ Fri May 20 13:51:48 2011 2011-05-20 13:51:48 scheduler[8691] : Child[2] 'agent=adj2nest host=localhost ' state=READY(5) @ Fri May 20 13:51:48 2011 2011-05-20 13:57:42 scheduler[8693] : *** Scheduler not responding: killing and restarting *** 2011-05-20 13:57:42 scheduler[8693] : *** Exiting fossology-scheduler PID 8691 QUIT *** 2011-05-20 13:58:46 scheduler[8693] : *** Scheduler restarted successfully by fo_watchdog *** 2011-05-20 13:58:46 scheduler[13944] : Log opened 2011-05-20 13:58:46 scheduler[13944] : Scheduler started. Build version: 1.2.1~3507, exported. I don't know of any resource issues. Disk did not fill up although it grew to 93% with about 2.5GB remaining. I think the ununpack completed. CPU and memory were apparently not an issue. The load average did start to climb slowly after the scheduler kept failing. I think the restarts of the scheduler leave select queries running and generates new queries. So when you watch top you can see the 4 or 5 select queries as the top processes. Bob - I presume your system is running 1.4.0. Can I do an rpm -Uvh to the 1.3.0 Fossology? I noticed the 1.4.0 rpm is for Red Hat 6. We are not running any Red Hat 6 yet. You think the 1.4.0 rpm will run on Red Hat 5? Thanks for the quick response. Ray W. From: Bob Gobeille [mailto:bob.gobei...@hp.com] Sent: Monday, May 23, 2011 9:22 AM To: Westphal, Raymond W Cc: fossology@fossology.org Subject: Re: [FOSSology] Cygwin Analysis causing scheduler restarts Importance: High Hi Ray, I just grabbed the source from the cygwin cvs, tared and uploaded it. On the system I used it took 7 minutes. http://repo.fossology.org/simpleIndex.php?mod=nomoslicenseshow=detailupload=148item=50608165 We have seen the scheduler problem you describe before. I think, as long as you aren't hitting some system resource, stopping the scheduler and restarting it might take care of the problem (btw, v2.0 has a new scheduler). Bob On May 23, 2011, at 6:31 AM, Westphal, Raymond W wrote: On Friday my customer uploaded Cygwin for analysis. I think it was a zip file. The job ran from 11:23am until I deleted it at around 8am on Sunday. I had to delete it because it caused Fossology scheduler to keep restarting. And that was causing the load average on the server to slowly climb. Has anyone had any experience with analyzing Cygwin? CONFIDENTIALITY NOTICE: This e-mail and any files transmitted with it are intended solely for the use of the individual or entity to whom they are addressed and may contain confidential and privileged information protected by law. If you received this e-mail in error, any review, use, dissemination, distribution, or copying of the e-mail is strictly prohibited. Please notify the sender immediately by return e-mail and delete all copies from your system. ___ fossology mailing list fossology@fossology.org http://fossology.org/mailman/listinfo/fossology
Re: [FOSSology] Cygwin Analysis causing scheduler restarts
Hi Mary. I just download the RHEL5 rpm! Hooray! Yes the autovacuum has been configured and appears to be working. Here is an interesting and often repeated section from the postgresql-Sat.log . LOG: could not send data to client: Broken pipe LOG: unexpected EOF on client connection LOG: autovacuum: processing database fossology LOG: could not send data to client: Broken pipe LOG: unexpected EOF on client connection LOG: autovacuum: processing database postgres ERROR: relation license_481 already exists Regarding the upgrade: I'll backup the database (filesystem level backup plus a dump of the database. Then can I simply perform an rpm -Uvh fossology-1.3.0-1.el5.x86_64.rpm followed by a rpm -Uvh fossology-1.4.0-1.el5.x86_64.rpm ? Thanks, Ray W. From: Laser, Mary [mailto:mary.la...@hp.com] Sent: Monday, May 23, 2011 10:19 AM To: Westphal, Raymond W; fossology@fossology.org Subject: RE: [FOSSology] Cygwin Analysis causing scheduler restarts Hi Ray, Thanks for telling us about the missing RHEL5 packages. They're there now :) When I've seen this type of behavior log file messaging, it usually means the unpack was not responding to the scheduler because it was having trouble unpacking a particular file OR the db was having difficulty processing the requests resulting for the unpack. The postgresql log file might have more clues. BTW, did you turn on autovacuum as we discussed last week? Upgrading (rpm -Uvh) is probably the fastest way around this current problem. Mary From: fossology-boun...@fossology.org [mailto:fossology-boun...@fossology.org] On Behalf Of Westphal, Raymond W Sent: Monday, May 23, 2011 8:46 AM To: fossology@fossology.org Subject: Re: [FOSSology] Cygwin Analysis causing scheduler restarts Good Morning Bob/Good Morning Mary. Here is the info. from the log just prior to where the scheduler started its cycle of not responding and then restarting. 2011-05-20 13:51:46 scheduler[8691] : Child[1] 'agent=unpack host=localhost ' state=FREEING(2) @ Fri May 20 13:51:46 2011 2011-05-20 13:51:46 scheduler[8691] : Child[1] 'agent=unpack host=localhost ' state=FREE(1) @ Fri May 20 13:51:46 2011 2011-05-20 13:51:48 scheduler[8691] : Child[2] 'agent=adj2nest host=localhost ' state=SPAWNED(4) @ Fri May 20 13:51:48 2011 2011-05-20 13:51:48 scheduler[8691] : Child[2] 'agent=adj2nest host=localhost ' state=READY(5) @ Fri May 20 13:51:48 2011 2011-05-20 13:57:42 scheduler[8693] : *** Scheduler not responding: killing and restarting *** 2011-05-20 13:57:42 scheduler[8693] : *** Exiting fossology-scheduler PID 8691 QUIT *** 2011-05-20 13:58:46 scheduler[8693] : *** Scheduler restarted successfully by fo_watchdog *** 2011-05-20 13:58:46 scheduler[13944] : Log opened 2011-05-20 13:58:46 scheduler[13944] : Scheduler started. Build version: 1.2.1~3507, exported. I don't know of any resource issues. Disk did not fill up although it grew to 93% with about 2.5GB remaining. I think the ununpack completed. CPU and memory were apparently not an issue. The load average did start to climb slowly after the scheduler kept failing. I think the restarts of the scheduler leave select queries running and generates new queries. So when you watch top you can see the 4 or 5 select queries as the top processes. Bob - I presume your system is running 1.4.0. Can I do an rpm -Uvh to the 1.3.0 Fossology? I noticed the 1.4.0 rpm is for Red Hat 6. We are not running any Red Hat 6 yet. You think the 1.4.0 rpm will run on Red Hat 5? Thanks for the quick response. Ray W. From: Bob Gobeille [mailto:bob.gobei...@hp.com] Sent: Monday, May 23, 2011 9:22 AM To: Westphal, Raymond W Cc: fossology@fossology.org Subject: Re: [FOSSology] Cygwin Analysis causing scheduler restarts Importance: High Hi Ray, I just grabbed the source from the cygwin cvs, tared and uploaded it. On the system I used it took 7 minutes. http://repo.fossology.org/simpleIndex.php?mod=nomoslicenseshow=detailupload=148item=50608165 We have seen the scheduler problem you describe before. I think, as long as you aren't hitting some system resource, stopping the scheduler and restarting it might take care of the problem (btw, v2.0 has a new scheduler). Bob On May 23, 2011, at 6:31 AM, Westphal, Raymond W wrote: On Friday my customer uploaded Cygwin for analysis. I think it was a zip file. The job ran from 11:23am until I deleted it at around 8am on Sunday. I had to delete it because it caused Fossology scheduler to keep restarting. And that was causing the load average on the server to slowly climb. Has anyone had any experience with analyzing Cygwin? CONFIDENTIALITY NOTICE: This e-mail and any files transmitted with it are intended solely for the use of the individual or entity to whom they are addressed and may contain confidential and privileged information protected by law. If you received this e-mail
Re: [FOSSology] Cygwin Analysis causing scheduler restarts
Hello Bob, I learned from the user that the zip file he analyzes is built based upon options he chooses from a Cygwin menu. He believes it contains both source and binary files. So it is very different from the file you analyzed. It is 450MB! I'm going to try the upgrade to Fossology 1.4.0 and see what happens. Thanks, Ray W. From: Bob Gobeille [mailto:bob.gobei...@hp.com] Sent: Monday, May 23, 2011 9:22 AM To: Westphal, Raymond W Cc: fossology@fossology.org Subject: Re: [FOSSology] Cygwin Analysis causing scheduler restarts Importance: High Hi Ray, I just grabbed the source from the cygwin cvs, tared and uploaded it. On the system I used it took 7 minutes. http://repo.fossology.org/simpleIndex.php?mod=nomoslicenseshow=detailupload=148item=50608165 We have seen the scheduler problem you describe before. I think, as long as you aren't hitting some system resource, stopping the scheduler and restarting it might take care of the problem (btw, v2.0 has a new scheduler). Bob On May 23, 2011, at 6:31 AM, Westphal, Raymond W wrote: On Friday my customer uploaded Cygwin for analysis. I think it was a zip file. The job ran from 11:23am until I deleted it at around 8am on Sunday. I had to delete it because it caused Fossology scheduler to keep restarting. And that was causing the load average on the server to slowly climb. Has anyone had any experience with analyzing Cygwin? CONFIDENTIALITY NOTICE: This e-mail and any files transmitted with it are intended solely for the use of the individual or entity to whom they are addressed and may contain confidential and privileged information protected by law. If you received this e-mail in error, any review, use, dissemination, distribution, or copying of the e-mail is strictly prohibited. Please notify the sender immediately by return e-mail and delete all copies from your system. ___ fossology mailing list fossology@fossology.org http://fossology.org/mailman/listinfo/fossology
[FOSSology] swap space
Good Morning. We have a monitor script running on our servers for things like filesystem, load average, swap space usage, etc. Yesterday I received an alert because the swap space usage on our Fossology server had reached 85%. I discovered that the scheduler was using the most virtual memory. I stopped and started the scheduler and the swap usage dropped to 13%. I'll probably increase the swap space. (Our Fossology server is running 1.2.1-3507 and it has 1.5GB of memory, 1GB of swap and 2 AMD 2800Mhz CPUs.) And so I have the following questions: Should I reset the scheduler regularly? If I reset the scheduler will jobs in the queue be lost? Should I check the queue before I reset the scheduler? If I need to check the queue, can you provide the query, please? Thanks, Ray W. CONFIDENTIALITY NOTICE: This e-mail and any files transmitted with it are intended solely for the use of the individual or entity to whom they are addressed and may contain confidential and privileged information protected by law. If you received this e-mail in error, any review, use, dissemination, distribution, or copying of the e-mail is strictly prohibited. Please notify the sender immediately by return e-mail and delete all copies from your system. ___ fossology mailing list fossology@fossology.org http://fossology.org/mailman/listinfo/fossology
[FOSSology] Next Version?
Hello Bob. I noticed In the next version (1.4) ... in the last post. I just recently upgraded 1.2.1. When will 1.4 be available and will I be able to go directly from 1.2.1 to 1.4? Thanks, Ray Westphal. CONFIDENTIALITY NOTICE: This e-mail and any files transmitted with it are intended solely for the use of the individual or entity to whom they are addressed and may contain confidential and privileged information protected by law. If you received this e-mail in error, any review, use, dissemination, distribution, or copying of the e-mail is strictly prohibited. Please notify the sender immediately by return e-mail and delete all copies from your system. ___ fossology mailing list fossology@fossology.org http://fossology.org/mailman/listinfo/fossology
Re: [FOSSology] Undefined Variables
Thanks for your response Bob. Happy New Year! Ray W. From: Bob Gobeille [mailto:bob.gobei...@hp.com] Sent: Wednesday, January 05, 2011 6:14 PM To: Westphal, Raymond W Cc: fossology@fossology.org; Lancaster, Michael J Subject: Re: [FOSSology] Undefined Variables Hi Ray, Sorry for the late response. I've been enjoying the holidays. ;-) The PHP notices are benign. I have been trying to clean them up as I see them. So thanks. I'll take care of these. A quick fix is to configure your php.ini file to not report them. php.ini has a variable to set the reporting level called error_reporting: error_reporting = E_ALL | E_STRICT You could change this to: error_reporting = E_ALL ~E_NOTICE I'll fix the code tomorrow. To understand the differences in the number of licenses, you have to compare which ones were found. The new nomos scanner is much more accurate that the old bsam scanner, but there are differences in the way the licenses are reported. So you really have to look at the two license lists to see if 1.2.1 is missing something or 1.1.0 is reporting false positives. Thanks, Bob Gobeille On Dec 21, 2010, at 10:51 AM, Westphal, Raymond W wrote: Hello All. My users are comparing the 1.1.0 product to the 1.2.1 product by uploading files to each server. They noticed a difference in the analysis of antlr-3.2.jar. To quote: The old said there were 69 licenses id'd. The new said 64 licenses were id'd. During the test we noticed 924 PHP Notice events in the web server logs for Undefined variable. For example: PHP Notice: Undefined variable: Max in /webroot/www/common/common-license.php on line 446, PHP Notice: Undefined variable: Offset in /webroot/www/common/common-license.php on line 441, Is there an easy fix for the undefined variables in the common-license.php file? Would the Undefined variable cause the difference in the number of licenses identified? Thanks, Ray W. CONFIDENTIALITY NOTICE: This e-mail and any files transmitted with it are intended solely for the use of the individual or entity to whom they are addressed and may contain confidential and privileged information protected by law. If you received this e-mail in error, any review, use, dissemination, distribution, or copying of the e-mail is strictly prohibited. Please notify the sender immediately by return e-mail and delete all copies from your system. ATT1..txt ___ fossology mailing list fossology@fossology.org http://fossology.org/mailman/listinfo/fossology
[FOSSology] Securing the httpd server
Good Morning Everyone. ! was successful in importing the data from the old server into a new server and then upgrading to 1.2.0 and then 1.2.1. I also successfully secured the web server. However, the links provided in the email reports contain http and not the https protocol. And when the customer tries to view the license summary (for example) she has to follow several links and login again because her session may have timed out. Is there an easy way to correct the links produced in the emails? Has anyone on the list secured their web server and overcome these issues? Thanks very much. Ray Westphal CONFIDENTIALITY NOTICE: This e-mail and any files transmitted with it are intended solely for the use of the individual or entity to whom they are addressed and may contain confidential and privileged information protected by law. If you received this e-mail in error, any review, use, dissemination, distribution, or copying of the e-mail is strictly prohibited. Please notify the sender immediately by return e-mail and delete all copies from your system. ___ fossology mailing list fossology@fossology.org http://fossology.org/mailman/listinfo/fossology
[FOSSology] Import data from 1.1.0
Hello. We are currently running Fossology version 1.1.0 on RHEL5. Is it possible to build another server running Fossology version 1.2.1 (also on RHEL5) and then rsync the repository and import the data from the old fossology database into the new fossology database? I see there are 6 more tables owned by fossy in the new database. I successfully installed the new version but I don't see any notes in the install documents about moving the data. Thanks, Ray Westphal CONFIDENTIALITY NOTICE: This e-mail and any files transmitted with it are intended solely for the use of the individual or entity to whom they are addressed and may contain confidential and privileged information protected by law. If you received this e-mail in error, any review, use, dissemination, distribution, or copying of the e-mail is strictly prohibited. Please notify the sender immediately by return e-mail and delete all copies from your system. ___ fossology mailing list fossology@fossology.org http://fossology.org/mailman/listinfo/fossology
