Skip the Squid then, google around for "iptables logging reports reporting" and see if you can find something useful.
Again, ntop will probably do what you need with some tweaking and add-on scripts, but not "out of the box" from what I can tell. I think the "Internet Domain" reports are your starting place with nTop. G -----Original Message----- From: [email protected] [mailto:[email protected]] On Behalf Of Istvan Köpe Sent: Wednesday, April 28, 2010 11:24 AM To: [email protected] Subject: Re: [Ntop] how to monitor http and https only I googled around and I could not find any article which confirms that https is usable without problems with squid as transparent proxy. Everybody is recommending in case of transparent proxy to not redirect 443 to squid. On 28.04.2010 18:48, Istvan Köpe wrote: > Maybe, but not as transparent proxy > > On 28.04.2010 17:42, Fabio Pardi wrote: >> you need squid 2.5+ i guess >> >> http://www.comfsm.fm/computing/squid/FAQ-1.html#ss1.12 >> >> >> Istvan Köpe wrote: >>> I just learned that squid doesn't handle https. So is not good for what >>> I need. >>> >>> On 28.04.2010 15:51, Gary Gatten wrote: >>>> I'm sure iptables can log most everything as well. Someone has >>>> probably written an app to format the logs and summarize the data. >>>> >>>> ----- Original Message ----- >>>> From: >>>> [email protected]<[email protected]> >>>> To: [email protected]<[email protected]> >>>> Sent: Wed Apr 28 07:37:52 2010 >>>> Subject: Re: [Ntop] how to monitor http and https only >>>> >>>> Once I had contact with squid and than I realized that squid is a >>>> whole >>>> chapter in Linux... If is possible I prefer to not install any proxy. >>>> >>>> I. >>>> >>>> On 28.04.2010 13:55, Steve Clark wrote: >>>> >>>>> Or maybe something like squid proxy. I am pretty sure it keeps an >>>>> access.log that could provide >>>>> the info you are looking for. >>>>> >>>>> On 04/27/2010 05:08 PM, Gary Gatten wrote: >>>>> >>>>>> Ah, I see.... You just want to see if the users are "surfing" or >>>>>> actually working? Not sure if nTop will give you this. The Domain >>>>>> report will have some of this info, and rrd may actually store this >>>>>> as well. I'm just not sure it will provide exactly what you seek. >>>>>> What about "IP -> Summary -> Internet Domain"?, then drill >>>>>> down >>>>>> from there? If this report will work for you, maybe run a script >>>>>> with several "wget" on the appropriate URL's and save those each >>>>>> night? Perhaps you could enable sticky hosts and then run a cron >>>>>> job >>>>>> that restarts nTop at midnight (or whenever) each night? >>>>>> >>>>>> Are you wanting something like "WebSense" - that records every url >>>>>> visited, the time of day, the time spent at each site, etc.? You >>>>>> may >>>>>> want to check out "OpenDNS". They offer similar service for >>>>>> tracking >>>>>> this type of info and it's not "too" expensive for small number of >>>>>> users. I'm sure there is Open Source stuff that will do this as >>>>>> well >>>>>> - I just don't know of any. >>>>>> >>>>>> nTop may be able to get what you want - it for sure will capture the >>>>>> data, I just don't know of a predefined "report" that will show >>>>>> exactly what you want. NTop is good at lots of things, but isn't a >>>>>> perfect fit for everything. >>>>>> >>>>>> Maybe someone else will have other ideas. In the mean time I >>>>>> recommend you play with nTop's options a little and see if you can >>>>>> get what you need without being too convoluted. >>>>>> >>>>>> -----Original Message----- >>>>>> From: [email protected] >>>>>> [mailto:[email protected]] On Behalf Of Istvan Köpe >>>>>> Sent: Tuesday, April 27, 2010 3:47 PM >>>>>> To: [email protected] >>>>>> Subject: Re: [Ntop] how to monitor http and https only >>>>>> >>>>>> The restrictions are done with iptables. There are only 4 hosts with >>>>>> internet access(http and https only), 1 with full access(the >>>>>> manager) >>>>>> and 1 test machine with full access. The rest are limited to >>>>>> antivirus >>>>>> updates. >>>>>> >>>>>> I want to save which sites were visited by the users each day. I >>>>>> need 2 >>>>>> type of reports: by local IP and by remote hosts. >>>>>> >>>>>> Istvan >>>>>> >>>>>> On 27.04.2010 23:35, Gary Gatten wrote: >>>>>> >>>>>>> With Sticky hosts, idle hosts are never purged from memory. >>>>>>> Therefore, every new host will take more and more until it runs >>>>>>> out. Depending on the number of hosts, I can't tell you if 256MB >>>>>>> will be enough or not. My guess is not. >>>>>>> >>>>>>> Maybe Wireshark is all you need? A capture filter will limit your >>>>>>> traffic to http (or whatever) and you can tell it to create a new >>>>>>> file every hour / 100MB / whatever. Then, some of the summary >>>>>>> reports may give the info you need. If you don't capture DNS >>>>>>> traffic you may have a hard time reconciling host ip's to urls, so >>>>>>> keep that in mind. >>>>>>> >>>>>>> If you're trying to solve a specific problem or answer a specific >>>>>>> question, perhaps post that? >>>>>>> >>>>>>> G >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> -----Original Message----- >>>>>>> From: [email protected] >>>>>>> [mailto:[email protected]] On Behalf Of Istvan Köpe >>>>>>> Sent: Tuesday, April 27, 2010 3:29 PM >>>>>>> To: [email protected] >>>>>>> Subject: Re: [Ntop] how to monitor http and https only >>>>>>> >>>>>>> After all I don't even need graphs, but everywhere I looked, >>>>>>> everybody >>>>>>> is suggesting ntop, or maybe I'm not asking the right questions. >>>>>>> >>>>>>> What do you mean by "ntop memory usage continue to grow". The >>>>>>> system >>>>>>> running ntop is a piece of junk, with 256MB ram. Will it crash >>>>>>> within 24h? >>>>>>> >>>>>>> Istvan >>>>>>> >>>>>>> On 27.04.2010 23:05, Gary Gatten wrote: >>>>>>> >>>>>>> >>>>>>>> Sounds right. Beware: enabling sticky hosts will cause ntop memory >>>>>>>> usage to continue to grow until: ntop is restarted, or ntop >>>>>>>> crashes >>>>>>>> from a malloc error. >>>>>>>> >>>>>>>> There is probably a way to use "wget" and / or other tools to >>>>>>>> "download" reports from ntop and save them somewhere. Then maybe >>>>>>>> you could set idle purge for say... 70 minutes, and run this batch >>>>>>>> report every hour? >>>>>>>> >>>>>>>> I think I understand what you're trying to do as I often need the >>>>>>>> same thing. You may want to spend a few minutes looking at the >>>>>>>> "rrd" settings. There may be some combination of "Data to Dump" >>>>>>>> and "RRD Detail" that will do what you wish. I've played with >>>>>>>> these settings some, but it's been a long time so can't offer much >>>>>>>> guidance. There are several good docs on the web that give >>>>>>>> details >>>>>>>> on what these settings do. If you can get RRD to store the data >>>>>>>> you wish, you can then use the "Arbitrary Graph" option to fetch / >>>>>>>> display that data. My initial thought is rrd will NOT store >>>>>>>> "conversation" level info, but who knows - maybe somewhere in >>>>>>>> there >>>>>>>> you'll get what you need? You could start be enabling all rrd >>>>>>>> data >>>>>>>> sets at the "high" level. >>>>>>>> >>>>>>>> G >>>>>>>> >>>>>>>> >>>>>>>> -----Original Message----- >>>>>>>> From: [email protected] >>>>>>>> [mailto:[email protected]] On Behalf Of Istvan >>>>>>>> Köpe >>>>>>>> Sent: Tuesday, April 27, 2010 2:47 PM >>>>>>>> To: [email protected] >>>>>>>> Subject: Re: [Ntop] how to monitor http and https only >>>>>>>> >>>>>>>> This means, if I want to see what web pages were opened by one >>>>>>>> specific >>>>>>>> user(local IP), I need to enable "sticky hosts" or I need to >>>>>>>> increase >>>>>>>> purge hosts to 12 hours, right? >>>>>>>> >>>>>>>> I'll try with sticky hosts. That seems to be the closest to what I >>>>>>>> need. >>>>>>>> >>>>>>>> Istvan >>>>>>>> >>>>>>>> On 27.04.2010 18:27, Gary Gatten wrote: >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>>> You may be speaking of two different issues: >>>>>>>>> 1.) How nTop determines which hosts are local and which are >>>>>>>>> remote >>>>>>>>> 2.) Idle host purge timers >>>>>>>>> >>>>>>>>> First, please make sure you specify "-m all your local network >>>>>>>>> ranges" on the command line. Or add via the GUI. This is the >>>>>>>>> only way ntop knows local from remote. Anything not defined as >>>>>>>>> local is considered remote. >>>>>>>>> >>>>>>>>> Next, the default idle host purge is 5 minutes. You have two >>>>>>>>> options that I know of: >>>>>>>>> 1.) Enable "sticky hosts" - which as implies hosts will >>>>>>>>> never >>>>>>>>> go away until you restart nTop. Only recommended in unique >>>>>>>>> environments. >>>>>>>>> 2.) Change the idle purge time in "globals-defines.h" and >>>>>>>>> recompile nTop. >>>>>>>>> >>>>>>>>> Not sure which settings over ride which. If you make a change to >>>>>>>>> the startup options, you must restart ntop and most/all recorded >>>>>>>>> traffic will be lost. If done by the GUI, some settings are >>>>>>>>> dynamic, I can't say for sure which ones. I think the GUI >>>>>>>>> settings are saved in the prefsCache.db file. >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> -----Original Message---- >>>>>>>>> From: [email protected] >>>>>>>>> [mailto:[email protected]] On Behalf Of Istvan >>>>>>>>> Köpe >>>>>>>>> Sent: Tuesday, April 27, 2010 10:06 AM >>>>>>>>> To: [email protected] >>>>>>>>> Subject: Re: [Ntop] how to monitor http and https only >>>>>>>>> >>>>>>>>> Even if I choose All protocols --> Traffic . I choose >>>>>>>>> Hosts: >>>>>>>>> All , I >>>>>>>>> can't see all the remote hosts. But for a while I could see some >>>>>>>>> remote >>>>>>>>> hosts which than disappeared. What is the effective time range >>>>>>>>> for >>>>>>>>> All >>>>>>>>> protocols --> Traffic ? >>>>>>>>> Where are the parameters saved if I use the web interface for >>>>>>>>> changing >>>>>>>>> the configuration(Admin-->Configure-->Startup options)? >>>>>>>>> I noticed that if I modify /etc/ntop.conf it overrides the web >>>>>>>>> config >>>>>>>>> settings. Is that right? >>>>>>>>> If I modify the /etc/ntop.conf, how can I make the settings >>>>>>>>> effective >>>>>>>>> without losing the recorded traffic? >>>>>>>>> >>>>>>>>> On 27.04.2010 17:45, Gary Gatten wrote: >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>>> There's a startup arg to specify which network ranges are local, >>>>>>>>>> it might be -b? Check the man and make sure you have this >>>>>>>>>> configured correctly for your environment. >>>>>>>>>> >>>>>>>>>> ----- Original Message ----- >>>>>>>>>> From: >>>>>>>>>> [email protected]<[email protected]> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> To: [email protected]<[email protected]> >>>>>>>>>> Sent: Tue Apr 27 09:38:42 2010 >>>>>>>>>> Subject: Re: [Ntop] how to monitor http and https only >>>>>>>>>> >>>>>>>>>> Ok, I got confused. Ntop is set on my Centos router. All the >>>>>>>>>> internet >>>>>>>>>> traffic goes through it. >>>>>>>>>> I go on the web interface All protocols --> Traffic . I >>>>>>>>>> choose Hosts: >>>>>>>>>> Remote only and I see only some of the remote hosts. I don't >>>>>>>>>> understand. >>>>>>>>>> Where can I see all the remote hosts which were accessed today? >>>>>>>>>> >>>>>>>>>> Istvan >>>>>>>>>> >>>>>>>>>> On 26.04.2010 18:34, Gary Gatten wrote: >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>>> You can't disable "everything", but with packet and protocol >>>>>>>>>>> filters, and by viewing specific reports - you can get pretty >>>>>>>>>>> close to what you need. >>>>>>>>>>> >>>>>>>>>>> ----- Original Message ----- >>>>>>>>>>> From: >>>>>>>>>>> [email protected]<[email protected]> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> To: [email protected]<[email protected]> >>>>>>>>>>> Sent: Mon Apr 26 09:31:35 2010 >>>>>>>>>>> Subject: Re: [Ntop] how to monitor http and https only >>>>>>>>>>> >>>>>>>>>>> Thanks for the hints. But there is still too much information. >>>>>>>>>>> All I want is: >>>>>>>>>>> - 192.168.0.xxx, between 08:00-14:00, accessed the following >>>>>>>>>>> sites: ... >>>>>>>>>>> - www.facebook.com, between 08:00-14:00, was accessed by the >>>>>>>>>>> following >>>>>>>>>>> local IP-s: ... >>>>>>>>>>> >>>>>>>>>>> I don't need the: >>>>>>>>>>> - Host Traffic Stats >>>>>>>>>>> - Packet Statistics >>>>>>>>>>> - Protocol Distribution >>>>>>>>>>> - TCP/UDP Recently Used Ports >>>>>>>>>>> - IP Service Stats: Client Role >>>>>>>>>>> - TCP/UDP - Traffic on Other Ports >>>>>>>>>>> >>>>>>>>>>> How can I do all these? >>>>>>>>>>> >>>>>>>>>>> Istvan >>>>>>>>>>> >>>>>>>>>>> On 26.04.2010 17:12, Gary Gatten wrote: >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>>> Good call. One can also restrict the displayed protocols with >>>>>>>>>>>> -p, all remaining traffic will be displayed as "other" >>>>>>>>>>>> >>>>>>>>>>>> ----- Original Message ----- >>>>>>>>>>>> From: >>>>>>>>>>>> [email protected]<[email protected]> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> To: [email protected]<[email protected]>; >>>>>>>>>>>> [email protected]<[email protected]> >>>>>>>>>>>> Sent: Mon Apr 26 08:44:04 2010 >>>>>>>>>>>> Subject: Re: [Ntop] how to monitor http and https only >>>>>>>>>>>> >>>>>>>>>>>> Have you taken a look at the manpages for ntop? On a unix >>>>>>>>>>>> system, the "-B" switch followed by a pcap expression will >>>>>>>>>>>> give >>>>>>>>>>>> you want you want. >>>>>>>>>>>> >>>>>>>>>>>> e.g >>>>>>>>>>>> >>>>>>>>>>>> ntop -d -w 8080 -B "port 80 or 443" >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> -----Original Message----- >>>>>>>>>>>> From: [email protected] >>>>>>>>>>>> [mailto:[email protected]] On Behalf Of Istvan >>>>>>>>>>>> Köpe >>>>>>>>>>>> Sent: Monday, April 26, 2010 9:40 AM >>>>>>>>>>>> To: [email protected] >>>>>>>>>>>> Subject: [Ntop] how to monitor http and https only >>>>>>>>>>>> >>>>>>>>>>>> Hello, >>>>>>>>>>>> >>>>>>>>>>>> I just installed ntop and it gives me much more information I >>>>>>>>>>>> need. I >>>>>>>>>>>> would like to see only the traffic on ports 80 and 443. >>>>>>>>>>>> How can I do that? >>>>>>>>>>>> >>>>>>>>>>>> Istvan >>>>>>>>>>>> _______________________________________________ >>>>>>>>>>>> Ntop mailing list >>>>>>>>>>>> [email protected] >>>>>>>>>>>> http://listgateway.unipi.it/mailman/listinfo/ntop >>>>>>>>>>>> _______________________________________________ >>>>>>>>>>>> Ntop mailing list >>>>>>>>>>>> [email protected] >>>>>>>>>>>> http://listgateway.unipi.it/mailman/listinfo/ntop >>>>>>>>>>>> _______________________________________________ >>>>>>>>>>>> Ntop mailing list >>>>>>>>>>>> [email protected] >>>>>>>>>>>> http://listgateway.unipi.it/mailman/listinfo/ntop >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>> _______________________________________________ >>>>>>>>>>> Ntop mailing list >>>>>>>>>>> [email protected] >>>>>>>>>>> http://listgateway.unipi.it/mailman/listinfo/ntop >>>>>>>>>>> _______________________________________________ >>>>>>>>>>> Ntop mailing list >>>>>>>>>>> [email protected] >>>>>>>>>>> http://listgateway.unipi.it/mailman/listinfo/ntop >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>> _______________________________________________ >>>>>>>>>> Ntop mailing list >>>>>>>>>> [email protected] >>>>>>>>>> http://listgateway.unipi.it/mailman/listinfo/ntop >>>>>>>>>> _______________________________________________ >>>>>>>>>> Ntop mailing list >>>>>>>>>> [email protected] >>>>>>>>>> http://listgateway.unipi.it/mailman/listinfo/ntop >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>> _______________________________________________ >>>>>>>>> Ntop mailing list >>>>>>>>> [email protected] >>>>>>>>> http://listgateway.unipi.it/mailman/listinfo/ntop >>>>>>>>> _______________________________________________ >>>>>>>>> Ntop mailing list >>>>>>>>> [email protected] >>>>>>>>> http://listgateway.unipi.it/mailman/listinfo/ntop >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>> _______________________________________________ >>>>>>>> Ntop mailing list >>>>>>>> [email protected] >>>>>>>> http://listgateway.unipi.it/mailman/listinfo/ntop >>>>>>>> _______________________________________________ >>>>>>>> Ntop mailing list >>>>>>>> [email protected] >>>>>>>> http://listgateway.unipi.it/mailman/listinfo/ntop >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>> _______________________________________________ >>>>>>> Ntop mailing list >>>>>>> [email protected] >>>>>>> http://listgateway.unipi.it/mailman/listinfo/ntop >>>>>>> _______________________________________________ >>>>>>> Ntop mailing list >>>>>>> [email protected] >>>>>>> http://listgateway.unipi.it/mailman/listinfo/ntop >>>>>>> >>>>>>> >>>>>>> >>>>>> _______________________________________________ >>>>>> Ntop mailing list >>>>>> [email protected] >>>>>> http://listgateway.unipi.it/mailman/listinfo/ntop >>>>>> _______________________________________________ >>>>>> Ntop mailing list >>>>>> [email protected] >>>>>> http://listgateway.unipi.it/mailman/listinfo/ntop >>>>>> >>>>>> >>>>> >>>> _______________________________________________ >>>> Ntop mailing list >>>> [email protected] >>>> http://listgateway.unipi.it/mailman/listinfo/ntop >>>> _______________________________________________ >>>> Ntop mailing list >>>> [email protected] >>>> http://listgateway.unipi.it/mailman/listinfo/ntop >>>> >>> _______________________________________________ >>> Ntop mailing list >>> [email protected] >>> http://listgateway.unipi.it/mailman/listinfo/ntop >> _______________________________________________ >> Ntop mailing list >> [email protected] >> http://listgateway.unipi.it/mailman/listinfo/ntop > > _______________________________________________ > Ntop mailing list > [email protected] > http://listgateway.unipi.it/mailman/listinfo/ntop _______________________________________________ Ntop mailing list [email protected] http://listgateway.unipi.it/mailman/listinfo/ntop _______________________________________________ Ntop mailing list [email protected] http://listgateway.unipi.it/mailman/listinfo/ntop
