Hi Michael, Yeah, big thing is we are trying to ascertain whether or not the file is actually uploaded...certainly the test file I can see uploaded as a job number was assigned and the ununpack started. I'll check out Chrome.
charlie -----Original Message----- From: Michael C. Jaeger [mailto:m...@mcj.de] Sent: Thursday, January 31, 2019 7:10 AM To: Kline, Charles (US) <charles.kl...@lmco.com> Cc: Jaeger, Michael C. <michael.c.jae...@siemens.com>; Stangel, Dan <dan.stan...@hpe.com>; fossol...@fossology.org Subject: Re: EXTERNAL: Re: [FOSSology] Quick Question Hello, looks good. it appears also to me that a part of your waiting time appears to be the upload / transfer time. Not all browsers show the upload progress, Chrome does in the lower left corner. On some OSes you can see the network traffic. I think this is maybe also another effect when dealing with VLP- "very large packages" Maybe FOSSology could improve on this by having an upload status dialogue to inform the user more precisely. I think the last two screenshots show the progress info after the file has been uploaded to the FOSSology server. Kind regards, Michael > On 31. Jan 2019, at 09:53, Kline, Charles <charles.kl...@lmco.com> wrote: > > Hmmm, not sure I’m following you Michael….here’s some screenshots from > a test I did… <image003.png><image011.jpg> CLICK UPLOAD… > <image012.png><image004.png> Then I will see this…but this is after > the uploaded has been completed and the ununpack has started as shown > below… <image014.png><image008.png> <image015.png><image010.png> I > guess I was hoping there was a file or log on the fossology server that one > could tail to see the progress or at least repeatedly do a listing to see the > file growth. > > Charlie > > From: Jaeger, Michael C. [mailto:michael.c.jae...@siemens.com] > Sent: Thursday, January 31, 2019 1:50 AM > To: Kline, Charles (US) <charles.kl...@lmco.com>; Michael C. Jaeger > <m...@mcj.de> > Cc: Stangel, Dan <dan.stan...@hpe.com>; fossol...@fossology.org > Subject: RE: EXTERNAL: Re: [FOSSology] Quick Question > > Hi, > > If you upload a file, there will be a link presented in the “message bar” > right below the menus with the number (id) of the upload. Clicking this > number will show you a table with the progress. > > If you have missed that you could click on “Jobs” in the top menu bar, as for > the options choose “my recent jobs”, it should show what is currently ongoing. > > Kind regards, Michael > > From: Kline, Charles [mailto:charles.kl...@lmco.com] > Sent: Donnerstag, 31. Januar 2019 00:53 > To: Jaeger, Michael C. (CT RDA SSI); Michael C. Jaeger > Cc: Stangel, Dan; fossol...@fossology.org > Subject: RE: EXTERNAL: Re: [FOSSology] Quick Question > > Hi, > Is there a way to monitor (whether it be via the fossology GUI or on the > fossology server) the progress of the upload of a file? Is there a log or > something that can be monitored? In a previous test with a 2G file it took > ~10 min to upload. I don’t know if this is a linear thing where a 4G file > will take 20 minutes, but it would certainly be nice if one could tell how > far along the upload was so you don’t abort the test prematurely because you > felt you had given it enough time. > > Thanks in advance!!!! > > Charlie > > From: Jaeger, Michael C. [mailto:michael.c.jae...@siemens.com] > Sent: Tuesday, January 22, 2019 7:36 AM > To: Kline, Charles (US) <charles.kl...@lmco.com>; Michael C. Jaeger > <m...@mcj.de> > Cc: Stangel, Dan <dan.stan...@hpe.com>; fossol...@fossology.org > Subject: EXTERNAL: RE: EXTERNAL: Re: [FOSSology] Just curious...question > about your installation... > > Hello, > > uploading 15GB in a single upload is not advisable in any case, since it will > lead to speed issues when browsing the source in an aggregated view. > > 15GB of compressed source code will be like … 100GB of uncompressed files? > that will be that will be maybe like >1000 packages into a single upload? > maybe magnitude of 1 mio files in one upload? -> that will lead to endlessly > running queries in the aggregated view unless you are really good at tuning > your postgresql server. > > I would highly recommend to consider splitting your 15GB upload into 30 > packages or so. > > Kind regards, Michael > > > > From: fossology@lists.fossology.org > [mailto:fossology@lists.fossology.org] On Behalf Of Kline, Charles > Sent: Montag, 21. Januar 2019 20:10 > To: Michael C. Jaeger > Cc: Stangel, Dan; fossol...@fossology.org > Subject: Re: EXTERNAL: Re: [FOSSology] Just curious...question about your > installation... > > Hi Michael, > Yeah, this could certainly be an issue...they would like to be able to > process a 15G file and I don’t think that is going to happen…if we > could get 4G that would be nice, but based on the info below that may > not even be possible. I’ll be testing tonight with the following > php.ini properties updated to 4096M but I’m not holding out much hope… > > Maximum allowed size for uploaded files. > upload_max_filesize = 2048M > Maximum size of POST data that PHP will accept. > post_max_size = 2048M > > Excerpt from a 'lscpu': > Architecture: x86_64 > CPU op-mode(s): 32-bit, 64-bit > CPU(s): 4 > > If I do a ‘cat /proc/meminfo’: > MemTotal: 8056932 kB > MemFree: 430432 kB > > And not much running right now…results of ‘top –u fossy’, seems like > we might be up against the limit now… top - 13:29:18 up 8 days, 20:01, 1 > user, load average: 1.00, 1.38, 1.81 > Tasks: 210 total, 2 running, 208 sleeping, 0 stopped, 0 zombie > Cpu(s): 3.3%us, 0.8%sy, 0.0%ni, 94.8%id, 1.1%wa, 0.0%hi, 0.1%si, 0.0%st > Mem: 8056932k total, 7627692k used, 429240k free, 17668k buffers > Swap: 4193276k total, 157492k used, 4035784k free, 5471620k cached > > PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND > 17259 fossy 20 0 99.1m 30m 2820 R 92.3 0.4 10:01.71 nomos > 7075 fossy 20 0 836m 2960 1668 S 0.0 0.0 1:48.17 fo_scheduler > > Charlie > > -----Original Message----- > From: Michael C. Jaeger [mailto:m...@mcj.de] > Sent: Friday, January 18, 2019 5:29 AM > To: Kline, Charles (US) <charles.kl...@lmco.com> > Cc: Stangel, Dan <dan.stan...@hpe.com>; fossol...@fossology.org > Subject: EXTERNAL: Re: [FOSSology] Just curious...question about your > installation... > > Hello, > > I think as for the PHP is it the question how much RAM you have (and > ... do you run a 32-bit OS?) > > restarting apache depends on your linux distro, in some cases > > sudo service apache2 restart > > or a tool from the apache web server software could also help (which > could be installed on your system, I do not know) > > sudo apachectl restart > > in other cases google will help. Reading into apachectl can help you > to do a more careful handling, such as > > apachectl configtest > apachectl graceful > > ... > > Kind regards, Michael > > > On 16. Jan 2019, at 20:07, Kline, Charles <charles.kl...@lmco.com> wrote: > > > > Just curious...what you have in your pkp.ini file for these properties...or > > I guess I should ask can you process files greater than 2G? > > ; Maximum allowed size for uploaded files. > > upload_max_filesize = 2048M > > > > ; Maximum size of POST data that PHP will accept. > > post_max_size = 2048M > > > > Oh, do you know how to restart Apache??? Trying to figure that out. > > > > Charlie > > > > -----Original Message----- > > From: Kline, Charles (US) > > Sent: Wednesday, January 9, 2019 2:31 PM > > To: 'Stangel, Dan' <dan.stan...@hpe.com>; fossol...@fossology.org > > Subject: RE: Question on version 2.5.0 > > > > Thanks so much Dan!!!! > > > > -----Original Message----- > > From: Stangel, Dan [mailto:dan.stan...@hpe.com] > > Sent: Wednesday, January 9, 2019 1:52 PM > > To: Kline, Charles (US) <charles.kl...@lmco.com>; > > fossol...@fossology.org > > Subject: EXTERNAL: RE: Question on version 2.5.0 > > > > Hi Charlie, > > > > Ah, the joys of a large enterprise IT organization. You ask a lot of good > > questions, and I would probably defer to other experts on this list for > > most of them, but I definitely can say that the delete agent has to run > > exclusively, by design, and that it can be rather slow. So your users' > > experiences are not surprising, and I'm not sure there's a good solution. > > Upgrading to a newer version may resolve some of these issues, but not sure > > what the team has done around delagent in the past few releases. > > > > As for maintenance tasks, most of these should be pretty reliable, but to > > your point, I'm not sure what impact they have on running jobs. You would > > definitely want to run the vacuum analyze job somewhat routinely, for the > > health and well-being of Postgres. I think in our configuration it is set > > up as a cron job. > > > > Dan > > > > > > From: Kline, Charles [mailto:charles.kl...@lmco.com] > > Sent: Wednesday, January 09, 2019 11:06 AM > > To: Stangel, Dan <dan.stan...@hpe.com>; fossol...@fossology.org > > Subject: RE: Question on version 2.5.0 > > > > Dan, > > At your convenience if you had any thoughts on the below I would appreciate > > it....having said that I'm aware that I'm kinda taking advantage of you at > > this point so please feel free to decline. I'm sure you have plenty on your > > plate. > > > > Back in Nov (I was out for surgery) the users reported the following: > > To the two server admins who kinda support the fossology server from one of > > the main fossology users: (apparently user was trying to do some > > deletes.assumption is that 'regular' fossology activity was taking place) > > Can one of you folks take a look at the Fossology server? > > It's just kind of sitting there doing nothing. > > > > Admin responded that postgresql and fossology had been restarted. > > > > Subsequent feedback from user. > > I tried deletes again, but it's not processing them. It's not even starting > > the job. > > And subsequently.from another super user. > > > > I had a big job running that didn't finish until 11/14/2018 @ 01:13 AM. > > > > As near as I can tell the delete jobs will wait for current jobs to > > complete their current step, then pause the job(s), then the delete job(s) > > executes, then the other jobs continue on with their next step. > > > > Your deletion jobs have completed last night around 9:54 pm just around the > > time my job completed the ununpack step. > > > > > > > > This kicked off talk (from a server admin who I have never worked with) of > > updating fossology to the latest version and/or moving it to another > > server. This was met with some push back.they finally decided to wait until > > I returned to work.which wasn't until 12/14.I talked to the one super user > > trying to get a sense of where things stood.she insisted that there were > > still issues with trying to do deletes during the week and that they were > > doing them on the weekends. > > > > Meeting was held on 1/7 to discuss. Two main issues. > > 1. The large upload file size, which is being addressed. > > 2. The issue with doing the deletes and things freezing up. > > > > In regards to issue 2, I definitely didn't get a sense that the super user > > was particularly interested in upgrading fossology and in regards to > > getting a new server we now have a mandate that anything new has to be in > > the cloud.and at this point that is not an easy process. > > > > Based on some of my research I ran into the Exclusive and Nokill > > settings.as seen in > > https://github.com/fossology/fossology/wiki/Job-Schedulerand if I look at > > /app/fossology/fossology_2.5/share/fossology I see the following.not sure > > if any of this is relevant to the above. > > Ununpack > > ; Directives: > > ; EXCLUSIVE: the agent cannot run concurrently with any other agent. > > special[] = NOKILL > > > > buckets > > ; Directives: > > ; EXCLUSIVE: the agent cannot run concurrently with any other agent. > > special[]= > > > > copyright > > ; Directives: > > ; EXCLUSIVE: the agent cannot run concurrently with any other agent. > > special[] = > > > > nomos > > ; Directives: > > ; EXCLUSIVE: the agent cannot run concurrently with any other agent. > > special[] = > > > > pkgagent > > ; Directives: > > ; EXCLUSIVE: the agent cannot run concurrently with any other agent. > > special[] = > > > > delagent.conf > > ; Directives: > > ; EXCLUSIVE: the agent cannot run concurrently with any other agent. > > special[] = EXCLUSIVE > > special[] = NOKILL > > > > > > Also, wondering what your experience is with running any of the maintenance > > tasks.any comments would be appreciated. > > > > I am a little hesitant running these since I am not 100% sure what they do > > or impact to the application when they are running.the ones I have > > considered running are "Remove uploads with no pfiles", "Remove orphaned > > temp tables" and "Vacuum Analyze the database". > > > > I am retiring later this year and hope to scrape this off my shoe sooner > > than later. > > > > Thanks in advance for your time and consideration. > > > > Charlie > > > > -----Original Message----- > > From: Kline, Charles (US) > > Sent: Wednesday, January 9, 2019 11:44 AM > > To: 'Stangel, Dan' <dan.stan...@hpe.com>; fossol...@fossology.org > > Subject: RE: Question on version 2.5.0 > > > > Dan, > > Thanks so much for your response...to be honest I didn't think I would get > > one. > > > > Upon further diffing I tracked down /etc/php.ini which looks to what needs > > to be updated. I was being told that only post_max_size needed to be > > updated but totally agree that upload_max_filesize also should be updated. > > Trick now would be to get sudo to enable me to do this or find an admin who > > can copy an updated version I made into /etc. > > > > In regards to restarting Apache...again dumb question, but does > > apache get restarted when the scheduler is restarted or much it be > > restarted separately? Frankly I have never done anything with > > apache. (and the big unfortunately is when I was given fossology to > > support I was told not to spend any time on it learning anything but > > just to respond to issues as they arise...so every time there is an > > issue it's an > > adventure!) > > > > My plan would be to initially increase the size of these 2 properties to > > 4096M, restart the scheduler(fossology), and test. If OK, update the > > properties to a size to accommodate the large file they are trying to > > process and repeat the above. I don't think I'll leave it at that size, > > 16384M, but would probably put it back to 4096M. > > > > Thanks!!!!! > > Charlie > > > > -----Original Message----- > > From: Stangel, Dan [mailto:dan.stan...@hpe.com] > > Sent: Wednesday, January 9, 2019 11:33 AM > > To: Kline, Charles (US) <charles.kl...@lmco.com>; > > fossol...@fossology.org > > Subject: EXTERNAL: RE: Question on version 2.5.0 > > > > Hi Charlie, > > > > On 2018-01-08, Kline, Charles wrote: > >> Yeah, I know...why are we still using 2.5.0...long story. > > > > Don't feel too bad - we're still on 2.6.2 for our production > > instance (again, long story.. Suffice it to say these releases have > > a long > > lifespan) > > > >> Question....users want to process/upload files greater than 2GB which > >> my users are saying is the largest then can process. Trying to > >> figure out the correct place to update this value. > >> > >> In > >> /app/fossology/fossology_2.5/share/fossology/lib/php/common-scheduler. > >> php > >> there is the following... > >> common-scheduler.php: \param $MaxSize - optional max read size, > >> default is 2048 common-scheduler.php:function > >> fo_scheduler_read($SchedObj, $MaxSize=2048) I assume this is in MB > >> > >> But in the fossology online info I am seeing this...assuming these > >> are values for the latest version of fossology...in the link they > >> talk about updating php.ini for > >> apache...(http://archive15.fossology.org/projects/fossology/wiki/Sy > >> sC > >> o > >> nfig/annotate/9) > >> (...) > >> So trying to figure out where I actually need to make an update to > >> change the size of the file that can be uploaded/processed by the users. > > > > You should only need to update the upload_max_filesize and > > post_max_size parameters in your php.ini file. In our case for > > example, we have > > > > ; Maximum allowed size for uploaded files. > > ; http://php.net/upload-max-filesize > > upload_max_filesize = 4096M > > > > ; Maximum size of POST data that PHP will accept. > > ; http://php.net/post-max-size > > post_max_size = 4096M > > > > Once you have updated php.ini, just be sure to restart Apache to pick up > > the changes. Then try to do a > 2G upload from file (ideally from a client > > that isn't too far away, network-wise) to verify that it works. > > > > Dan > > > > > > > > > > -=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#3217): https://lists.fossology.org/g/fossology/message/3217 Mute This Topic: https://lists.fossology.org/mt/29603806/21656 Group Owner: fossology+ow...@lists.fossology.org Unsubscribe: https://lists.fossology.org/g/fossology/unsub [arch...@mail-archive.com] -=-=-=-=-=-=-=-=-=-=-=-