RE: Writing Files without CFFILE
Your issue is the number of chunks you are trying to do it in. Try doing it in 2 parts. cffile is probably ok but your cfsavecontent may be what is causing you issues as it may not be clearing down the content. You are still creating 14000 pages inside memory space even though you are not displaying them. Have you considered that cfsavecontent may not be the right thing here? Adam -Original Message- From: Sean Daniels [mailto:[EMAIL PROTECTED] Sent: 09 March 2004 16:16 To: CF-Talk Subject: Writing Files without CFFILE Hey all, I have a CF script that is used to generate about 14,000 static HTML files from dynamic content (don't ask). I am looping over a query using cfsavecontent to save the HTML code for each page and then writing to a file with CFFILE. As this process runs, I sit here and watch the CF process gobble up RAM. When it hits about page 12,000 my RAM is completely spent and the server crashes (hangs). So. I'm thinking CFFILE simply isn't the right tool for this job. Does anyone out there have a suggestion for an alternate way of writing the files. I'm thinking maybe using the underlying Java in CFMX or something. Thanks in advance people. [Todays Threads] [This Message] [Subscription] [Fast Unsubscribe] [User Settings]
Re: Writing Files without CFFILE
On Mar 9, 2004, at 11:50 AM, Adam Reynolds wrote: Your issue is the number of chunks you are trying to do it in. Try doing it in 2 parts. I thought about doing that, but it seems inherently non-scalable. Like, if I break it into two parts, what happens when the database returns 24,000 records instead of 14,000. I know I could do it in 5,000 record chunks I guess, but I thought there might be something obvious that was more elegant. cffile is probably ok but your cfsavecontent may be what is causing you issues as it may not be clearing down the content. You are still creating 14000 pages inside memory space even though you are not displaying them. Have you considered that cfsavecontent may not be the right thing here? That's a good thought. I guess I could try passing the whole HTML string directly into cffile in the output attribute, but that's gonna be hard to read. Is there a way to clear the variable after each iteration of the loop that would release the RAM? [Todays Threads] [This Message] [Subscription] [Fast Unsubscribe] [User Settings]
RE: Writing Files without CFFILE
Are you using the same variable name for the cfsavecontent? -Original Message- From: Sean Daniels [mailto:[EMAIL PROTECTED] Sent: 09 March 2004 17:18 To: CF-Talk Subject: Re: Writing Files without CFFILE On Mar 9, 2004, at 11:50 AM, Adam Reynolds wrote: Your issue is the number of chunks you are trying to do it in. Try doing it in 2 parts. I thought about doing that, but it seems inherently non-scalable. Like, if I break it into two parts, what happens when the database returns 24,000 records instead of 14,000. I know I could do it in 5,000 record chunks I guess, but I thought there might be something obvious that was more elegant. cffile is probably ok but your cfsavecontent may be what is causing you issues as it may not be clearing down the content. You are still creating 14000 pages inside memory space even though you are not displaying them. Have you considered that cfsavecontent may not be the right thing here? That's a good thought. I guess I could try passing the whole HTML string directly into cffile in the output attribute, but that's gonna be hard to read. Is there a way to clear the variable after each iteration of the loop that would release the RAM? _ [Todays Threads] [This Message] [Subscription] [Fast Unsubscribe] [User Settings]
Re: Writing Files without CFFILE
On Mar 9, 2004, at 12:26 PM, Craig Dudley wrote:
Are you using the same variable name for the cfsavecontent?
Yes. On each iteration the savecontent overwrites the previous content
var.
That being said, I believe I solved the problem. I should have
mentioned I wasn't calling CFFILE directly in the loop, rather I had a
method in a CFC called writeStaticFile that accepted two params -
fileID and content. Once I pulled this logic out of the CFC and
directly into the code all is well. I can run the whole batch without
any perceptible increase in RAM usage.
Now, why passing it into the CFC would... is question for another day.
Here's the method, if anyone cares to comment.
cffunction name=writeStaticPage access=public returntype=boolean
output=false
cfargument name=content required=yes type=string/
cfargument name=fileName required=yes type=string/
cffile action=""
file=#expandpath('../static/'arguments.fileName)#
nameconflict=overwrite addnewline=no charset=utf-8
output=#arguments.content#/
cfreturn true/
/cffunction
-Original Message-
From: Sean Daniels [mailto:[EMAIL PROTECTED]
Sent: 09 March 2004 17:18
To: CF-Talk
Subject: Re: Writing Files without CFFILE
On Mar 9, 2004, at 11:50 AM, Adam Reynolds wrote:
Your issue is the number of chunks you are trying to do it in.
Try
doing it
in 2 parts.
I thought about doing that, but it seems inherently
non-scalable. Like,
if I break it into two parts, what happens when the database
returns
24,000 records instead of 14,000. I know I could do it in 5,000
record
chunks I guess, but I thought there might be something obvious
that was
more elegant.
cffile is probably ok but your cfsavecontent may be what is
causing
you
issues as it may not be clearing down the content. You are
still
creating
14000 pages inside memory space even though you are not
displaying
them.
Have you considered that cfsavecontent may not be the right
thing
here?
That's a good thought. I guess I could try passing the whole
HTML
string directly into cffile in the output attribute, but that's
gonna
be hard to read. Is there a way to clear the variable after
each
iteration of the loop that would release the RAM?
_
[Todays Threads]
[This Message]
[Subscription]
[Fast Unsubscribe]
[User Settings]
RE: Writing Files without CFFILE
I can only assume you were getting 14000 instances of the cfc object,
not sure why. You could try creating one instance and storing it in say,
the Application scope, and use that, might help. But since it's working
now anyway, might be a waste of time.
-Original Message-
From: Sean Daniels [mailto:[EMAIL PROTECTED]
Sent: 09 March 2004 17:44
To: CF-Talk
Subject: Re: Writing Files without CFFILE
On Mar 9, 2004, at 12:26 PM, Craig Dudley wrote:
Are you using the same variable name for the cfsavecontent?
Yes. On each iteration the savecontent overwrites the previous
content
var.
That being said, I believe I solved the problem. I should have
mentioned I wasn't calling CFFILE directly in the loop, rather I
had a
method in a CFC called writeStaticFile that accepted two
params -
fileID and content. Once I pulled this logic out of the CFC and
directly into the code all is well. I can run the whole batch
without
any perceptible increase in RAM usage.
Now, why passing it into the CFC would... is question for
another day.
Here's the method, if anyone cares to comment.
cffunction name=writeStaticPage access=public
returntype=boolean
output=false
cfargument name=content required=yes type=string/
cfargument name=fileName required=yes type=string/
cffile action=""
file=#expandpath('../static/'arguments.fileName)#
nameconflict=overwrite addnewline=no charset=utf-8
output=#arguments.content#/
cfreturn true/
/cffunction
-Original Message-
From: Sean Daniels [mailto:[EMAIL PROTECTED]
Sent: 09 March 2004 17:18
To: CF-Talk
Subject: Re: Writing Files without CFFILE
On Mar 9, 2004, at 11:50 AM, Adam Reynolds wrote:
Your issue is the number of chunks you are trying to do it
in.
Try
doing it
in 2 parts.
I thought about doing that, but it seems inherently
non-scalable. Like,
if I break it into two parts, what happens when the database
returns
24,000 records instead of 14,000. I know I could do it in
5,000
record
chunks I guess, but I thought there might be something
obvious
that was
more elegant.
cffile is probably ok but your cfsavecontent may be what
is
causing
you
issues as it may not be clearing down the content. You are
still
creating
14000 pages inside memory space even though you are not
displaying
them.
Have you considered that cfsavecontent may not be the
right
thing
here?
That's a good thought. I guess I could try passing the whole
HTML
string directly into cffile in the output attribute, but
that's
gonna
be hard to read. Is there a way to clear the variable after
each
iteration of the loop that would release the RAM?
_
_
[Todays Threads]
[This Message]
[Subscription]
[Fast Unsubscribe]
[User Settings]
Re: Writing Files without CFFILE
On Mar 9, 2004, at 12:38 PM, Craig Dudley wrote: I can only assume you were getting 14000 instances of the cfc object, not sure why. You could try creating one instance and storing it in say, the Application scope, and use that, might help. But since it's working now anyway, might be a waste of time. I definitely was only instantiating the object once, prior to the looping. I was calling the method however, using this syntax: cfset obj.writeStaticFile(param1,param2)/ Not sure if that would be the problem. I a testing right now using: cfset temp = obj.writeStaticFile(param1,param2)/ And that seems to be gobbling as much RAM as before. In any event, as you said, it's working great when I bypass the CFC so no sense beating it to death. [Todays Threads] [This Message] [Subscription] [Fast Unsubscribe] [User Settings]
Re: Writing Files without CFFILE
And that seems to be gobbling as much RAM as before. In any event, as you said, it's working great when I bypass the CFC so no sense beating it to death. Do you have debugging on by any chance?That can really slow down a CFC-intensive request. Jon [EMAIL PROTECTED] 03/09/04 11:29AM On Mar 9, 2004, at 12:38 PM, Craig Dudley wrote: I can only assume you were getting 14000 instances of the cfc object, not sure why. You could try creating one instance and storing it in say, the Application scope, and use that, might help. But since it's working now anyway, might be a waste of time. I definitely was only instantiating the object once, prior to the looping. I was calling the method however, using this syntax: cfset obj.writeStaticFile(param1,param2)/ Not sure if that would be the problem. I a testing right now using: cfset temp = obj.writeStaticFile(param1,param2)/ And that seems to be gobbling as much RAM as before. In any event, as you said, it's working great when I bypass the CFC so no sense beating it to death. [Todays Threads] [This Message] [Subscription] [Fast Unsubscribe] [User Settings]
