Reading only part of a really big text file...
If I have a 270 MB log file, can I use CF to split it up into say ten 27 MB files or some such thing? Or maybe into like 100,000 lines per file or some such thing. I'm currently finding it impossible to view the contents of this file due to the size. Purchased a file splitting utility, but it was unable to do the task. Have not been able to load the file in Notepad or Word. I'm hoping there might be a CF way around this. Thanks, Charles Nahm Sonic Networks Inc. __ Structure your ColdFusion code with Fusebox. Get the official book at http://www.fusionauthority.com/bkinfo.cfm FAQ: http://www.thenetprofits.co.uk/coldfusion/faq Archives: http://www.mail-archive.com/cf-talk@houseoffusion.com/ Unsubscribe: http://www.houseoffusion.com/index.cfm?sidebar=lists
Re: Reading only part of a really big text file...
What, may I ask, are you trying to do with it? If you are trying to garner info out of it, may I suggest grep, sed awk, or perl? Depending on your need for searching. If you need to query it as if it were data, do you have access to MSSQL and DTS? You could import the whole darn thing into a table. If you need to analyze the info in it (look for patterns and such), there are a number of free web log analysis tools out there. Jerry Johnson [EMAIL PROTECTED] 05/21/02 05:15PM If I have a 270 MB log file, can I use CF to split it up into say ten 27 MB files or some such thing? Or maybe into like 100,000 lines per file or some such thing. I'm currently finding it impossible to view the contents of this file due to the size. Purchased a file splitting utility, but it was unable to do the task. Have not been able to load the file in Notepad or Word. I'm hoping there might be a CF way around this. Thanks, Charles Nahm Sonic Networks Inc. __ Signup for the Fusion Authority news alert and keep up with the latest news in ColdFusion and related topics. http://www.fusionauthority.com/signup.cfm FAQ: http://www.thenetprofits.co.uk/coldfusion/faq Archives: http://www.mail-archive.com/cf-talk@houseoffusion.com/ Unsubscribe: http://www.houseoffusion.com/index.cfm?sidebar=lists
Re: Reading only part of a really big text file...
If I have a 270 MB log file, can I use CF to split it up into say ten 27 MB files or some such thing? Or maybe into like 100,000 lines per file or some such thing. I'm currently finding it impossible to view the contents of this file due to the size. Purchased a file splitting utility, but it was unable to do the task. Have not been able to load the file in Notepad or Word. I'm hoping there might be a CF way around this. I don't (necessarily) have the definitive answer you might be looking for, but I don't think CF solely can do what you want it to do. To do what you want to do, you would need to use the CFFILE tag. The problems with it (in this case) is that CFFILE will read the entire file into your server's memory before doing any sort of manipulation with the contents. So if you're server doesn't have boatloads of memory, you're out of luck. However, one thought that does come to mind as I'm writing this is to potentially set up a text driver datasource in your CF Administrator and then query the file that way. I'm not entirely sure how to do that (sorry!), but I think I've heard this being done before. Check out a source such as cfcomet.com for more info on this. I'm sure that if it can be done this way, they'll have the answer. Regards, Dave. __ Structure your ColdFusion code with Fusebox. Get the official book at http://www.fusionauthority.com/bkinfo.cfm FAQ: http://www.thenetprofits.co.uk/coldfusion/faq Archives: http://www.mail-archive.com/cf-talk@houseoffusion.com/ Unsubscribe: http://www.houseoffusion.com/index.cfm?sidebar=lists
RE: Reading only part of a really big text file...
-Original Message- From: Jerry Johnson [mailto:[EMAIL PROTECTED]] What, may I ask, are you trying to do with it? I guess analyze for patterns would be the best description. Basically just wanted to do text searches for key phrases. If you are trying to garner info out of it, may I suggest grep, sed awk, or perl? Depending on your need for searching. Are these Unix tools? We are running Windows 2K. If you need to query it as if it were data, do you have access to MSSQL and DTS? You could import the whole darn thing into a table. If you need to analyze the info in it (look for patterns and such), there are a number of free web log analysis tools out there. Thanks, I was not aware of this. Charles Nahm Sonic Networks Inc. __ Your ad could be here. Monies from ads go to support these lists and provide more resources for the community. http://www.fusionauthority.com/ads.cfm FAQ: http://www.thenetprofits.co.uk/coldfusion/faq Archives: http://www.mail-archive.com/cf-talk@houseoffusion.com/ Unsubscribe: http://www.houseoffusion.com/index.cfm?sidebar=lists
RE: Reading only part of a really big text file...
100,000 lines? $ tail -10 filename :) heheh...I couldn't resist. I'm sure there's a wayfind a good perl programmer or a UNIX admin close by. They'll probably be your best source of information. -Original Message- From: Charles Nahm [mailto:[EMAIL PROTECTED]] Sent: Tuesday, May 21, 2002 4:15 PM To: CF-Talk Subject: Reading only part of a really big text file... If I have a 270 MB log file, can I use CF to split it up into say ten 27 MB files or some such thing? Or maybe into like 100,000 lines per file or some such thing. I'm currently finding it impossible to view the contents of this file due to the size. Purchased a file splitting utility, but it was unable to do the task. Have not been able to load the file in Notepad or Word. I'm hoping there might be a CF way around this. Thanks, Charles Nahm Sonic Networks Inc. __ Structure your ColdFusion code with Fusebox. Get the official book at http://www.fusionauthority.com/bkinfo.cfm FAQ: http://www.thenetprofits.co.uk/coldfusion/faq Archives: http://www.mail-archive.com/cf-talk@houseoffusion.com/ Unsubscribe: http://www.houseoffusion.com/index.cfm?sidebar=lists
Re: Reading only part of a really big text file...
My .02... Simply put, cffile previous to MX is not the right tool for parsing large text files. MX might not be great depending on if they used any of Java's io buffering. I haven't put cffile in mx through a torture test yet, and I had to reinstall my machine with the preview release on it. If someone out there feels like doing some comparisons, I for one am really curious :) However parsing text is one of the most basic tasks in any language. You might want to take the oppurtunity to hone your Java skills in preparation for the brave new world we are entering... Check out this UDF for how to use Java's ability to treat a file as a stream allowing you to use CF to parse bite size chunks of the file. It probably isn't that speedy as it sits, but add a buffer and performace would go way up. However if you leave out the part where it concatenates the file into memory and just do your parsing in the loop, it will probably use a lot less memory than cffile alone would require. http://www.cflib.org/udf.cfm?ID=417 On the other hand, using SQL Server would be the easiest way to do this imho. jon - Original Message - From: Charles Nahm [EMAIL PROTECTED] To: CF-Talk [EMAIL PROTECTED] Sent: Tuesday, May 21, 2002 6:05 PM Subject: RE: Reading only part of a really big text file... -Original Message- From: Jerry Johnson [mailto:[EMAIL PROTECTED]] What, may I ask, are you trying to do with it? I guess analyze for patterns would be the best description. Basically just wanted to do text searches for key phrases. If you are trying to garner info out of it, may I suggest grep, sed awk, or perl? Depending on your need for searching. Are these Unix tools? We are running Windows 2K. If you need to query it as if it were data, do you have access to MSSQL and DTS? You could import the whole darn thing into a table. If you need to analyze the info in it (look for patterns and such), there are a number of free web log analysis tools out there. Thanks, I was not aware of this. Charles Nahm Sonic Networks Inc. __ Your ad could be here. Monies from ads go to support these lists and provide more resources for the community. http://www.fusionauthority.com/ads.cfm FAQ: http://www.thenetprofits.co.uk/coldfusion/faq Archives: http://www.mail-archive.com/cf-talk@houseoffusion.com/ Unsubscribe: http://www.houseoffusion.com/index.cfm?sidebar=lists
RE: Reading only part of a really big text file...
(response to original message) Most of the UNIX tools have been ported to NT. Do a search on cygwin. Justin -Original Message- From: Jon Hall [mailto:[EMAIL PROTECTED]] Sent: Tuesday, May 21, 2002 8:12 PM To: CF-Talk Subject: Re: Reading only part of a really big text file... My .02... Simply put, cffile previous to MX is not the right tool for parsing large text files. MX might not be great depending on if they used any of Java's io buffering. I haven't put cffile in mx through a torture test yet, and I had to reinstall my machine with the preview release on it. If someone out there feels like doing some comparisons, I for one am really curious :) However parsing text is one of the most basic tasks in any language. You might want to take the oppurtunity to hone your Java skills in preparation for the brave new world we are entering... Check out this UDF for how to use Java's ability to treat a file as a stream allowing you to use CF to parse bite size chunks of the file. It probably isn't that speedy as it sits, but add a buffer and performace would go way up. However if you leave out the part where it concatenates the file into memory and just do your parsing in the loop, it will probably use a lot less memory than cffile alone would require. http://www.cflib.org/udf.cfm?ID=417 On the other hand, using SQL Server would be the easiest way to do this imho. jon - Original Message - From: Charles Nahm [EMAIL PROTECTED] To: CF-Talk [EMAIL PROTECTED] Sent: Tuesday, May 21, 2002 6:05 PM Subject: RE: Reading only part of a really big text file... -Original Message- From: Jerry Johnson [mailto:[EMAIL PROTECTED]] What, may I ask, are you trying to do with it? I guess analyze for patterns would be the best description. Basically just wanted to do text searches for key phrases. If you are trying to garner info out of it, may I suggest grep, sed awk, or perl? Depending on your need for searching. Are these Unix tools? We are running Windows 2K. If you need to query it as if it were data, do you have access to MSSQL and DTS? You could import the whole darn thing into a table. If you need to analyze the info in it (look for patterns and such), there are a number of free web log analysis tools out there. Thanks, I was not aware of this. Charles Nahm Sonic Networks Inc. __ Get the mailserver that powers this list at http://www.coolfusion.com FAQ: http://www.thenetprofits.co.uk/coldfusion/faq Archives: http://www.mail-archive.com/cf-talk@houseoffusion.com/ Unsubscribe: http://www.houseoffusion.com/index.cfm?sidebar=lists
RE: Reading only part of a really big text file...
I guess analyze for patterns would be the best description. Basically just wanted to do text searches for key phrases. Couldn't you make a verity collection of the log file? You wouldn't be using cffile, but it may get at the info you want. -Craig __ Structure your ColdFusion code with Fusebox. Get the official book at http://www.fusionauthority.com/bkinfo.cfm FAQ: http://www.thenetprofits.co.uk/coldfusion/faq Archives: http://www.mail-archive.com/cf-talk@houseoffusion.com/ Unsubscribe: http://www.houseoffusion.com/index.cfm?sidebar=lists
