I'm sure iptables can log most everything as well. Someone has probably written an app to format the logs and summarize the data.
----- Original Message ----- From: [email protected] <[email protected]> To: [email protected] <[email protected]> Sent: Wed Apr 28 07:37:52 2010 Subject: Re: [Ntop] how to monitor http and https only Once I had contact with squid and than I realized that squid is a whole chapter in Linux... If is possible I prefer to not install any proxy. I. On 28.04.2010 13:55, Steve Clark wrote: > Or maybe something like squid proxy. I am pretty sure it keeps an > access.log that could provide > the info you are looking for. > > On 04/27/2010 05:08 PM, Gary Gatten wrote: >> Ah, I see.... You just want to see if the users are "surfing" or >> actually working? Not sure if nTop will give you this. The Domain >> report will have some of this info, and rrd may actually store this >> as well. I'm just not sure it will provide exactly what you seek. >> What about "IP -> Summary -> Internet Domain"?, then drill down >> from there? If this report will work for you, maybe run a script >> with several "wget" on the appropriate URL's and save those each >> night? Perhaps you could enable sticky hosts and then run a cron job >> that restarts nTop at midnight (or whenever) each night? >> >> Are you wanting something like "WebSense" - that records every url >> visited, the time of day, the time spent at each site, etc.? You may >> want to check out "OpenDNS". They offer similar service for tracking >> this type of info and it's not "too" expensive for small number of >> users. I'm sure there is Open Source stuff that will do this as well >> - I just don't know of any. >> >> nTop may be able to get what you want - it for sure will capture the >> data, I just don't know of a predefined "report" that will show >> exactly what you want. NTop is good at lots of things, but isn't a >> perfect fit for everything. >> >> Maybe someone else will have other ideas. In the mean time I >> recommend you play with nTop's options a little and see if you can >> get what you need without being too convoluted. >> >> -----Original Message----- >> From: [email protected] >> [mailto:[email protected]] On Behalf Of Istvan Köpe >> Sent: Tuesday, April 27, 2010 3:47 PM >> To: [email protected] >> Subject: Re: [Ntop] how to monitor http and https only >> >> The restrictions are done with iptables. There are only 4 hosts with >> internet access(http and https only), 1 with full access(the manager) >> and 1 test machine with full access. The rest are limited to antivirus >> updates. >> >> I want to save which sites were visited by the users each day. I need 2 >> type of reports: by local IP and by remote hosts. >> >> Istvan >> >> On 27.04.2010 23:35, Gary Gatten wrote: >>> With Sticky hosts, idle hosts are never purged from memory. >>> Therefore, every new host will take more and more until it runs >>> out. Depending on the number of hosts, I can't tell you if 256MB >>> will be enough or not. My guess is not. >>> >>> Maybe Wireshark is all you need? A capture filter will limit your >>> traffic to http (or whatever) and you can tell it to create a new >>> file every hour / 100MB / whatever. Then, some of the summary >>> reports may give the info you need. If you don't capture DNS >>> traffic you may have a hard time reconciling host ip's to urls, so >>> keep that in mind. >>> >>> If you're trying to solve a specific problem or answer a specific >>> question, perhaps post that? >>> >>> G >>> >>> >>> >>> >>> -----Original Message----- >>> From: [email protected] >>> [mailto:[email protected]] On Behalf Of Istvan Köpe >>> Sent: Tuesday, April 27, 2010 3:29 PM >>> To: [email protected] >>> Subject: Re: [Ntop] how to monitor http and https only >>> >>> After all I don't even need graphs, but everywhere I looked, everybody >>> is suggesting ntop, or maybe I'm not asking the right questions. >>> >>> What do you mean by "ntop memory usage continue to grow". The system >>> running ntop is a piece of junk, with 256MB ram. Will it crash >>> within 24h? >>> >>> Istvan >>> >>> On 27.04.2010 23:05, Gary Gatten wrote: >>> >>>> Sounds right. Beware: enabling sticky hosts will cause ntop memory >>>> usage to continue to grow until: ntop is restarted, or ntop crashes >>>> from a malloc error. >>>> >>>> There is probably a way to use "wget" and / or other tools to >>>> "download" reports from ntop and save them somewhere. Then maybe >>>> you could set idle purge for say... 70 minutes, and run this batch >>>> report every hour? >>>> >>>> I think I understand what you're trying to do as I often need the >>>> same thing. You may want to spend a few minutes looking at the >>>> "rrd" settings. There may be some combination of "Data to Dump" >>>> and "RRD Detail" that will do what you wish. I've played with >>>> these settings some, but it's been a long time so can't offer much >>>> guidance. There are several good docs on the web that give details >>>> on what these settings do. If you can get RRD to store the data >>>> you wish, you can then use the "Arbitrary Graph" option to fetch / >>>> display that data. My initial thought is rrd will NOT store >>>> "conversation" level info, but who knows - maybe somewhere in there >>>> you'll get what you need? You could start be enabling all rrd data >>>> sets at the "high" level. >>>> >>>> G >>>> >>>> >>>> -----Original Message----- >>>> From: [email protected] >>>> [mailto:[email protected]] On Behalf Of Istvan Köpe >>>> Sent: Tuesday, April 27, 2010 2:47 PM >>>> To: [email protected] >>>> Subject: Re: [Ntop] how to monitor http and https only >>>> >>>> This means, if I want to see what web pages were opened by one >>>> specific >>>> user(local IP), I need to enable "sticky hosts" or I need to increase >>>> purge hosts to 12 hours, right? >>>> >>>> I'll try with sticky hosts. That seems to be the closest to what I >>>> need. >>>> >>>> Istvan >>>> >>>> On 27.04.2010 18:27, Gary Gatten wrote: >>>> >>>> >>>>> You may be speaking of two different issues: >>>>> 1.) How nTop determines which hosts are local and which are remote >>>>> 2.) Idle host purge timers >>>>> >>>>> First, please make sure you specify "-m all your local network >>>>> ranges" on the command line. Or add via the GUI. This is the >>>>> only way ntop knows local from remote. Anything not defined as >>>>> local is considered remote. >>>>> >>>>> Next, the default idle host purge is 5 minutes. You have two >>>>> options that I know of: >>>>> 1.) Enable "sticky hosts" - which as implies hosts will never >>>>> go away until you restart nTop. Only recommended in unique >>>>> environments. >>>>> 2.) Change the idle purge time in "globals-defines.h" and >>>>> recompile nTop. >>>>> >>>>> Not sure which settings over ride which. If you make a change to >>>>> the startup options, you must restart ntop and most/all recorded >>>>> traffic will be lost. If done by the GUI, some settings are >>>>> dynamic, I can't say for sure which ones. I think the GUI >>>>> settings are saved in the prefsCache.db file. >>>>> >>>>> >>>>> >>>>> -----Original Message---- >>>>> From: [email protected] >>>>> [mailto:[email protected]] On Behalf Of Istvan Köpe >>>>> Sent: Tuesday, April 27, 2010 10:06 AM >>>>> To: [email protected] >>>>> Subject: Re: [Ntop] how to monitor http and https only >>>>> >>>>> Even if I choose All protocols --> Traffic . I choose Hosts: >>>>> All , I >>>>> can't see all the remote hosts. But for a while I could see some >>>>> remote >>>>> hosts which than disappeared. What is the effective time range for >>>>> All >>>>> protocols --> Traffic ? >>>>> Where are the parameters saved if I use the web interface for >>>>> changing >>>>> the configuration(Admin-->Configure-->Startup options)? >>>>> I noticed that if I modify /etc/ntop.conf it overrides the web config >>>>> settings. Is that right? >>>>> If I modify the /etc/ntop.conf, how can I make the settings effective >>>>> without losing the recorded traffic? >>>>> >>>>> On 27.04.2010 17:45, Gary Gatten wrote: >>>>> >>>>> >>>>> >>>>>> There's a startup arg to specify which network ranges are local, >>>>>> it might be -b? Check the man and make sure you have this >>>>>> configured correctly for your environment. >>>>>> >>>>>> ----- Original Message ----- >>>>>> From: >>>>>> [email protected]<[email protected]> >>>>>> To: [email protected]<[email protected]> >>>>>> Sent: Tue Apr 27 09:38:42 2010 >>>>>> Subject: Re: [Ntop] how to monitor http and https only >>>>>> >>>>>> Ok, I got confused. Ntop is set on my Centos router. All the >>>>>> internet >>>>>> traffic goes through it. >>>>>> I go on the web interface All protocols --> Traffic . I >>>>>> choose Hosts: >>>>>> Remote only and I see only some of the remote hosts. I don't >>>>>> understand. >>>>>> Where can I see all the remote hosts which were accessed today? >>>>>> >>>>>> Istvan >>>>>> >>>>>> On 26.04.2010 18:34, Gary Gatten wrote: >>>>>> >>>>>> >>>>>> >>>>>> >>>>>>> You can't disable "everything", but with packet and protocol >>>>>>> filters, and by viewing specific reports - you can get pretty >>>>>>> close to what you need. >>>>>>> >>>>>>> ----- Original Message ----- >>>>>>> From: >>>>>>> [email protected]<[email protected]> >>>>>>> >>>>>>> To: [email protected]<[email protected]> >>>>>>> Sent: Mon Apr 26 09:31:35 2010 >>>>>>> Subject: Re: [Ntop] how to monitor http and https only >>>>>>> >>>>>>> Thanks for the hints. But there is still too much information. >>>>>>> All I want is: >>>>>>> - 192.168.0.xxx, between 08:00-14:00, accessed the following >>>>>>> sites: ... >>>>>>> - www.facebook.com, between 08:00-14:00, was accessed by the >>>>>>> following >>>>>>> local IP-s: ... >>>>>>> >>>>>>> I don't need the: >>>>>>> - Host Traffic Stats >>>>>>> - Packet Statistics >>>>>>> - Protocol Distribution >>>>>>> - TCP/UDP Recently Used Ports >>>>>>> - IP Service Stats: Client Role >>>>>>> - TCP/UDP - Traffic on Other Ports >>>>>>> >>>>>>> How can I do all these? >>>>>>> >>>>>>> Istvan >>>>>>> >>>>>>> On 26.04.2010 17:12, Gary Gatten wrote: >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>>> Good call. One can also restrict the displayed protocols with >>>>>>>> -p, all remaining traffic will be displayed as "other" >>>>>>>> >>>>>>>> ----- Original Message ----- >>>>>>>> From: >>>>>>>> [email protected]<[email protected]> >>>>>>>> >>>>>>>> To: [email protected]<[email protected]>; >>>>>>>> [email protected]<[email protected]> >>>>>>>> Sent: Mon Apr 26 08:44:04 2010 >>>>>>>> Subject: Re: [Ntop] how to monitor http and https only >>>>>>>> >>>>>>>> Have you taken a look at the manpages for ntop? On a unix >>>>>>>> system, the "-B" switch followed by a pcap expression will give >>>>>>>> you want you want. >>>>>>>> >>>>>>>> e.g >>>>>>>> >>>>>>>> ntop -d -w 8080 -B "port 80 or 443" >>>>>>>> >>>>>>>> >>>>>>>> -----Original Message----- >>>>>>>> From: [email protected] >>>>>>>> [mailto:[email protected]] On Behalf Of Istvan >>>>>>>> Köpe >>>>>>>> Sent: Monday, April 26, 2010 9:40 AM >>>>>>>> To: [email protected] >>>>>>>> Subject: [Ntop] how to monitor http and https only >>>>>>>> >>>>>>>> Hello, >>>>>>>> >>>>>>>> I just installed ntop and it gives me much more information I >>>>>>>> need. I >>>>>>>> would like to see only the traffic on ports 80 and 443. >>>>>>>> How can I do that? >>>>>>>> >>>>>>>> Istvan >>>>>>>> _______________________________________________ >>>>>>>> Ntop mailing list >>>>>>>> [email protected] >>>>>>>> http://listgateway.unipi.it/mailman/listinfo/ntop >>>>>>>> _______________________________________________ >>>>>>>> Ntop mailing list >>>>>>>> [email protected] >>>>>>>> http://listgateway.unipi.it/mailman/listinfo/ntop >>>>>>>> _______________________________________________ >>>>>>>> Ntop mailing list >>>>>>>> [email protected] >>>>>>>> http://listgateway.unipi.it/mailman/listinfo/ntop >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>> _______________________________________________ >>>>>>> Ntop mailing list >>>>>>> [email protected] >>>>>>> http://listgateway.unipi.it/mailman/listinfo/ntop >>>>>>> _______________________________________________ >>>>>>> Ntop mailing list >>>>>>> [email protected] >>>>>>> http://listgateway.unipi.it/mailman/listinfo/ntop >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>> _______________________________________________ >>>>>> Ntop mailing list >>>>>> [email protected] >>>>>> http://listgateway.unipi.it/mailman/listinfo/ntop >>>>>> _______________________________________________ >>>>>> Ntop mailing list >>>>>> [email protected] >>>>>> http://listgateway.unipi.it/mailman/listinfo/ntop >>>>>> >>>>>> >>>>>> >>>>>> >>>>> _______________________________________________ >>>>> Ntop mailing list >>>>> [email protected] >>>>> http://listgateway.unipi.it/mailman/listinfo/ntop >>>>> _______________________________________________ >>>>> Ntop mailing list >>>>> [email protected] >>>>> http://listgateway.unipi.it/mailman/listinfo/ntop >>>>> >>>>> >>>>> >>>>> >>>> _______________________________________________ >>>> Ntop mailing list >>>> [email protected] >>>> http://listgateway.unipi.it/mailman/listinfo/ntop >>>> _______________________________________________ >>>> Ntop mailing list >>>> [email protected] >>>> http://listgateway.unipi.it/mailman/listinfo/ntop >>>> >>>> >>>> >>> _______________________________________________ >>> Ntop mailing list >>> [email protected] >>> http://listgateway.unipi.it/mailman/listinfo/ntop >>> _______________________________________________ >>> Ntop mailing list >>> [email protected] >>> http://listgateway.unipi.it/mailman/listinfo/ntop >>> >>> >> >> _______________________________________________ >> Ntop mailing list >> [email protected] >> http://listgateway.unipi.it/mailman/listinfo/ntop >> _______________________________________________ >> Ntop mailing list >> [email protected] >> http://listgateway.unipi.it/mailman/listinfo/ntop >> > > _______________________________________________ Ntop mailing list [email protected] http://listgateway.unipi.it/mailman/listinfo/ntop _______________________________________________ Ntop mailing list [email protected] http://listgateway.unipi.it/mailman/listinfo/ntop
