I figured out how the netflow dumps are formatted. Basically its the raw flow with a few length of flow bytes added to the front of the packet. Its not going to work for what i need to do. I have actually wrote my own udp server to collect the data and dump it every 15 minutes. Then i have a separate job that's parsing out the raw flows. I got netflow v5 complete and started to tackle v9. I have to run this thing on windows so i'm writing the whole thing in C#(blah). If anyone is interested then i'll post the code when i'm done.
Thanks Josh On Tue, Apr 13, 2010 at 12:48 PM, Chris Leonardos <[email protected]>wrote: > Josh, > > After thinking about this, I did have issues with reading the FLOW files > generated by NTOP with flow tools. > > Here's the error: > > flow-cat: ftiheader_read(): Warning, bad magic number > flow-cat: ftiheader_read(1270982473.flow): Failed, ignoring file. > > Does this look familiar? > > Below is information on my build of NTOP. > > I'll be happy to provide more information or work on solving this puzzle, > if there's > someone with a more in depth knowledge of how this works. > > I ended up using a Cisco monitor port and using tcpdump to grab 40Gig of > raw data from a 24 hour period, and > then culling it with Python. > > Kind Regards, > > -Chris > > > > > Welcome to ntop v.3.4-pre2 (32 bit) > [Configured on Mar 4 2010 19:48:49, built on Mar 4 2010 19:49:28] > Copyright 1998-2010 by Luca Deri <[email protected]>. > Get the freshest ntop from http://www.ntop.org/ > > *config.log:* > > build_os='linux-gnu' > build_vendor='suse' > datadir='${datarootdir}' > datarootdir='/usr/local/share' > docdir='${datarootdir}/doc/${PACKAGE_TARNAME}' > dvidir='${docdir}' > exec_prefix='${prefix}' > host='i686-suse-linux-gnu' > host_alias='' > host_cpu='i686' > host_os='linux-gnu' > host_vendor='suse' > htmldir='${docdir}' > includedir='${prefix}/include' > infodir='${datarootdir}/info' > install_sh='$(SHELL) /home/cleonardos/ntop-3.4-pre2/install-sh' > libdir='${exec_prefix}/lib' > libexecdir='${exec_prefix}/libexec' > localedir='${datarootdir}/locale' > localstatedir='${prefix}/var' > lt_ECHO='echo' > mandir='${datarootdir}/man' > mkdir_p='/bin/mkdir -p' > oldincludedir='/usr/include' > pdfdir='${docdir}' > prefix='/usr/local' > program_transform_name='s,x,x,' > psdir='${docdir}' > sbindir='${exec_prefix}/sbin' > sharedstatedir='${prefix}/com' > sysconfdir='${prefix}/etc' > target='i686-suse-linux-gnu' > target_alias='' > target_cpu='i686' > target_os='linux-gnu' > target_vendor='suse' > > ## ----------- ## > ## confdefs.h. ## > ## ----------- ## > > #define PACKAGE_NAME "ntop" > #define PACKAGE_TARNAME "ntop" > #define PACKAGE_VERSION "3.4-pre2" > #define PACKAGE_STRING "ntop 3.4-pre2" > #define PACKAGE_BUGREPORT "" > #define PACKAGE "ntop" > #define VERSION "3.4-pre2" > #define STDC_HEADERS 1 > #define HAVE_SYS_TYPES_H 1 > #define HAVE_SYS_STAT_H 1 > #define HAVE_STDLIB_H 1 > #define HAVE_STRING_H 1 > #define HAVE_MEMORY_H 1 > #define HAVE_STRINGS_H 1 > #define HAVE_INTTYPES_H 1 > #define HAVE_STDINT_H 1 > #define HAVE_UNISTD_H 1 > #define HAVE_DLFCN_H 1 > #define LT_OBJDIR ".libs/" > #define HAVE_PCAP_NEXT_EX 1 > #define HAVE_GDBM_H 1 > #define HAVE_LIBGDBM 1 > #define HAVE_LONG_DOUBLE_WIDER 1 > #define HAVE_LONG_DOUBLE 1 > #define CFG_LITTLE_ENDIAN 1 > #define STDC_HEADERS 1 > #define HAVE_DIRENT_H 1 > #define HAVE_SYS_WAIT_H 1 > #define TIME_WITH_SYS_TIME 1 > #define HAVE_FLOAT_H 1 > #define HAVE_STDDEF_H 1 > #define HAVE_STDLIB_H 1 > #define HAVE_STRING_H 1 > #define HAVE_ERRNO_H 1 > #define HAVE_FCNTL_H 1 > #define HAVE_LIMITS_H 1 > #define HAVE_MATH_H 1 > #define HAVE_SIGNAL_H 1 > #define HAVE_STDARG_H 1 > #define HAVE_UNISTD_H 1 > #define HAVE_STDIO_H 1 > #define HAVE_STRINGS_H 1 > #define HAVE_SYS_IOCTL_H 1 > #define HAVE_SYS_SOCKET_H 1 > #define HAVE_SYS_TIME_H 1 > #define HAVE_SYS_TYPES_H 1 > #define HAVE_SETJMP_H 1 > #define HAVE_SHADOW_H 1 > #define HAVE_SYS_UTSNAME_H 1 > #define HAVE_NETINET_IN_H 1 > #define HAVE_ARPA_INET_H 1 > #define HAVE_ARPA_NAMESER_H 1 > #define HAVE_NET_ETHERNET_H 1 > #define HAVE_LIBZ 1 > #define HAVE_LIBRRD_TH 1 > #define HAVE_NET_IF_H 1 > #define HAVE_NETINET_IF_ETHER_H 1 > #define HAVE_NETINET_IN_SYSTM_H 1 > #define HAVE_NETINET_IP_H 1 > #define HAVE_NETINET_IP_ICMP_H 1 > #define HAVE_NETINET_TCP_H 1 > #define HAVE_NETINET_UDP_H 1 > #define HAVE_SYS_PARAM_H 1 > #define HAVE_SYS_SYSCTL_H 1 > #define HAVE_NET_ROUTE_H 1 > #define HAVE_NET_PPP_DEFS_H 1 > #define HAVE_CRYPT_H 1 > #define HAVE_PWD_H 1 > #define HAVE_SHADOW_H 1 > #define HAVE_DIRENT_H 1 > #define HAVE_DLFCN_H 1 > #define HAVE_GETOPT_H 1 > #define HAVE_INTTYPES_H 1 > #define HAVE_MEMORY_H 1 > #define HAVE_SYS_PARAM_H 1 > #define HAVE_SYS_SELECT_H 1 > #define HAVE_SYS_STAT_H 1 > #define HAVE_SYS_UN_H 1 > #define HAVE_SYS_WAIT_H 1 > #define HAVE_ZLIB_H 1 > #define HAVE_SCHED_H 1 > #define HAVE_PTHREAD_H 1 > #define HAVE_RW_LOCK 1 > #define HAVE_SYS_SYSLOG_H 1 > #define HAVE_SYSLOG_H 1 > #define INET6 1 > #define HAVE_NETINET_IP6_H 1 > #define HAVE_NETINET_ICMP6_H 1 > #define HAVE_SYSCTL 1 > #define HAVE_FINITE 1 > #define HAVE_ISINF 1 > #define TIME_WITH_SYS_TIME 1 > #define HAVE_STRUCT_TM_TM_ZONE 1 > #define HAVE_TM_ZONE 1 > #define HAVE_U_INT64_T 1 > #define HAVE_U_INT32_T 1 > #define HAVE_U_INT16_T 1 > #define HAVE_U_INT8_T 1 > #define HAVE_INT64_T 1 > #define HAVE_INT32_T 1 > #define HAVE_INT16_T 1 > #define HAVE_INT8_T 1 > #define HAVE_LIBC 1 > #define HAVE_LIBCRYPT 1 > #define HAVE_LIBC 1 > #define HAVE_LIBC 1 > #define HAVE_DLADDR 1 > #define HAVE_LIBC 1 > #define HAVE_LIBC 1 > #define HAVE_LIBPTHREAD 1 > #define HAVE_FORK 1 > #define HAVE_VFORK 1 > #define HAVE_WORKING_VFORK 1 > #define HAVE_WORKING_FORK 1 > #define RETSIGTYPE void > #define LSTAT_FOLLOWS_SLASHED_SYMLINK 1 > #define HAVE_STRFTIME 1 > #define HAVE_ALARM 1 > #define HAVE_ENDPWENT 1 > #define HAVE_GETHOSTBYADDR 1 > #define HAVE_GETHOSTBYNAME 1 > #define HAVE_GETHOSTNAME 1 > #define HAVE_GETHOSTBYADDR_R 1 > #define HAVE_GETPASS 1 > #define HAVE_GETTIMEOFDAY 1 > #define HAVE_LOCALTIME_R 1 > #define HAVE_MEMCHR 1 > #define HAVE_MEMSET 1 > #define HAVE_PUTENV 1 > #define HAVE_SELECT 1 > #define HAVE_SOCKET 1 > #define HAVE_SNPRINTF 1 > #define HAVE_SQRTF 1 > #define HAVE_STRCASECMP 1 > #define HAVE_STRNCASECMP 1 > #define HAVE_STRCASESTR 1 > #define HAVE_STRCHR 1 > #define HAVE_STRRCHR 1 > #define HAVE_STRCSPN 1 > #define HAVE_STRDUP 1 > #define HAVE_STRERROR 1 > #define HAVE_STRPBRK 1 > #define HAVE_STRSIGNAL 1 > #define HAVE_STRSPN 1 > #define HAVE_STRSTR 1 > #define HAVE_STRTOUL 1 > #define HAVE_UNAME 1 > #define HAVE_STRTOK_R 1 > #define MAKE_WITH_ZLIB 1 > #define HAVE_PYTHON 1 > #define HAVE_PTHREAD_ATFORK 1 > #define HAVE_BACKTRACE 1 > #define HAVE_GETOPT_LONG 1 > #define HAVE_FACILITYNAMES 1 > #define HAVE_IN6_ADDR 1 > #define RETSIGTYPE void > #define HAVE_GEOIP 1 > #define CFG_DATAFILE_DIR "/usr/local/share/ntop" > #define CFG_CONFIGFILE_DIR "/usr/local/etc/ntop" > #define CFG_RUN_DIR "/usr/local/var/ntop" > #define CFG_PLUGIN_DIR "/usr/local/lib/ntop/plugins" > #define CFG_DBFILE_DIR "/usr/local/var/ntop" > > configure: exit 0 > > ## ---------------------- ## > ## Running config.status. ## > ## ---------------------- ## > > This file was extended by ntop config.status 3.4-pre2, which was > generated by GNU Autoconf 2.63. Invocation command line was > > CONFIG_FILES = > CONFIG_HEADERS = > CONFIG_LINKS = > CONFIG_COMMANDS = > $ ./config.status config.h > > on netmon1 > > config.status:1155: creating config.h > config.status:1377: config.h is unchanged > > > On Tue, Apr 13, 2010 at 9:16 AM, josh summitt <[email protected]> wrote: > >> Yea i have flow tools and silk and a few others but none of them can make >> sense out of the flow dumps that the nTop Netflow plugin generates. I read >> something that said ntop generates netflow v5 dump files. Every tool i've >> used to translate V5 netflow fails on these files. I read something else >> that said these dump files are in a gnu db format or mySQL. >> >> I'm using nTop just as a netflow collector middle man so that i can get >> the netflow data into the analytic software we are using here. Is there a >> better way to collect netflow that will run on windows and support netflow >> V1-9 and IPFIX? I would prefer to have the data in a csv format. >> >> >> Thanks >> Josh >> >> >> >> On Mon, Apr 12, 2010 at 8:33 PM, Gary Gatten <[email protected]> wrote: >> >>> I *think* there are several different "dumps". IIRC there is a dump >>> and/or debug option that basically copies the flow records to a disk file as >>> they're received and look just like netflow flows. >>> >>> ------------------------------ >>> *From*: [email protected] < >>> [email protected]> >>> *To*: [email protected] <[email protected]> >>> *Sent*: Mon Apr 12 20:03:17 2010 >>> *Subject*: Re: [Ntop] Neflow dump format question. >>> >>> Josh, >>> >>> I ran into this problem recently and tried to get the open source >>> flow-tools to compile on Open Suse 11.1 but was unable to get some of the >>> prereq's to compile properly most notably the pypcap python module. >>> >>> Here's a good list of open source tools, including flow-tools. >>> >>> http://www.networkuptime.com/tools/netflow/ >>> >>> -Chris >>> >>> On Mon, Apr 12, 2010 at 3:00 PM, josh summitt <[email protected]> wrote: >>> >>>> I've been searching the forums and internet for the last few days and >>>> have not found the answer so hopefully someone can answer this for me. When >>>> using the Netflow plugin, what format are the netflow data dumps in? I need >>>> to take netflow data and import it into analytic software that we are >>>> using. >>>> What tool do i need to read this data and extract it to another system? >>>> >>>> >>>> Thanks >>>> Josh >>>> >>>> _______________________________________________ >>>> Ntop mailing list >>>> [email protected] >>>> http://listgateway.unipi.it/mailman/listinfo/ntop >>>> >>>> >>> >>> _______________________________________________ >>> Ntop mailing list >>> [email protected] >>> http://listgateway.unipi.it/mailman/listinfo/ntop >>> >>> >> >> _______________________________________________ >> Ntop mailing list >> [email protected] >> http://listgateway.unipi.it/mailman/listinfo/ntop >> >> > > _______________________________________________ > Ntop mailing list > [email protected] > http://listgateway.unipi.it/mailman/listinfo/ntop > >
_______________________________________________ Ntop mailing list [email protected] http://listgateway.unipi.it/mailman/listinfo/ntop
