Placeholders in large text file

[Bec Carter]

Good mornin' all!

I've a requirement to put certain values (after computing them) in
various spots in a very large text file. So basically the starting
text file can have placeholders where these computed values will end
up- like a template. Then my code will compute some values based on
user input and i need to fill in the placeholders.
Is there a better way to do this besides a simple string replace?

Cheers.
Bec

Re: Placeholders in large text file

[Noon Silk]

On Mon, Feb 7, 2011 at 10:21 AM, Bec Carter  wrote:
> Good mornin' all!
>
> I've a requirement to put certain values (after computing them) in
> various spots in a very large text file. So basically the starting
> text file can have placeholders where these computed values will end
> up- like a template. Then my code will compute some values based on
> user input and i need to fill in the placeholders.
>
> Is there a better way to do this besides a simple string replace?

Well, no. You'll need to read the file in and find your tokens and
replace them. Depending on how large the file it, you might need to do
this line by line, or chunk by chunk, writing out as you read in, but
inevitably it comes down to looking for a sequence and replacing it
with another.

How large is "very large"? Megs? Gigs?


> Cheers.
> Bec

-- 
Noon Silk

http://dnoondt.wordpress.com/  (Noon Silk) | http://www.mirios.com.au:8081 >

Fancy a quantum lunch?
http://www.mirios.com.au:8081/index.php?title=Quantum_Lunch

"Every morning when I wake up, I experience an exquisite joy — the joy
of being this signature."

Re: Placeholders in large text file

[Bec Carter]

On Mon, Feb 7, 2011 at 10:25 AM, Noon Silk  wrote:
> On Mon, Feb 7, 2011 at 10:21 AM, Bec Carter  wrote:
>> Good mornin' all!
>>
>> I've a requirement to put certain values (after computing them) in
>> various spots in a very large text file. So basically the starting
>> text file can have placeholders where these computed values will end
>> up- like a template. Then my code will compute some values based on
>> user input and i need to fill in the placeholders.
>>
>> Is there a better way to do this besides a simple string replace?
>
> Well, no. You'll need to read the file in and find your tokens and
> replace them. Depending on how large the file it, you might need to do
> this line by line, or chunk by chunk, writing out as you read in, but
> inevitably it comes down to looking for a sequence and replacing it
> with another.
>
> How large is "very large"? Megs? Gigs?
>

Yup reading all into a string right now and replacing. File is around 750 megs

>
>> Cheers.
>> Bec
>
> --
> Noon Silk
>
> http://dnoondt.wordpress.com/  (Noon Silk) | http://www.mirios.com.au:8081 >
>
> Fancy a quantum lunch?
> http://www.mirios.com.au:8081/index.php?title=Quantum_Lunch
>
> "Every morning when I wake up, I experience an exquisite joy — the joy
> of being this signature."
>

Re: Placeholders in large text file

[Noon Silk]

On Mon, Feb 7, 2011 at 10:29 AM, Bec Carter  wrote:
> On Mon, Feb 7, 2011 at 10:25 AM, Noon Silk  wrote:
>> On Mon, Feb 7, 2011 at 10:21 AM, Bec Carter  wrote:
>>> Good mornin' all!
>>>
>>> I've a requirement to put certain values (after computing them) in
>>> various spots in a very large text file. So basically the starting
>>> text file can have placeholders where these computed values will end
>>> up- like a template. Then my code will compute some values based on
>>> user input and i need to fill in the placeholders.
>>>
>>> Is there a better way to do this besides a simple string replace?
>>
>> Well, no. You'll need to read the file in and find your tokens and
>> replace them. Depending on how large the file it, you might need to do
>> this line by line, or chunk by chunk, writing out as you read in, but
>> inevitably it comes down to looking for a sequence and replacing it
>> with another.
>>
>> How large is "very large"? Megs? Gigs?
>>
>
> Yup reading all into a string right now and replacing. File is around 750 megs

Mm, in that case I would definitely think reading in chunk by chunk
would be better.

So, you read in chunks of chars in a char[], and then you must look
for the start of your token. Taking care to note that you could end up
in the middle of your token, something like:

// Pseudocode
char[] data = { "$te" }
char[] nextData = { "st$" }

Where the token is "$test$".

Depending on your data, it might be that reading lines is enough, and
do the replace on that basis. Hopefully this is somewhat clear. I had
a quick search and couldn't find a nice example on doing this, but it
should be easy enough using StreamReader or friends. If it's not clear
I can show an example later on.

-- 
Noon Silk

http://dnoondt.wordpress.com/  (Noon Silk) | http://www.mirios.com.au:8081 >

Fancy a quantum lunch?
http://www.mirios.com.au:8081/index.php?title=Quantum_Lunch

"Every morning when I wake up, I experience an exquisite joy — the joy
of being this signature."

Re: Placeholders in large text file

[Bec Carter]

On Mon, Feb 7, 2011 at 10:35 AM, Noon Silk  wrote:
> On Mon, Feb 7, 2011 at 10:29 AM, Bec Carter  wrote:
>> On Mon, Feb 7, 2011 at 10:25 AM, Noon Silk  wrote:
>>> On Mon, Feb 7, 2011 at 10:21 AM, Bec Carter  wrote:
 Good mornin' all!

 I've a requirement to put certain values (after computing them) in
 various spots in a very large text file. So basically the starting
 text file can have placeholders where these computed values will end
 up- like a template. Then my code will compute some values based on
 user input and i need to fill in the placeholders.

 Is there a better way to do this besides a simple string replace?
>>>
>>> Well, no. You'll need to read the file in and find your tokens and
>>> replace them. Depending on how large the file it, you might need to do
>>> this line by line, or chunk by chunk, writing out as you read in, but
>>> inevitably it comes down to looking for a sequence and replacing it
>>> with another.
>>>
>>> How large is "very large"? Megs? Gigs?
>>>
>>
>> Yup reading all into a string right now and replacing. File is around 750 
>> megs
>
> Mm, in that case I would definitely think reading in chunk by chunk
> would be better.
>
> So, you read in chunks of chars in a char[], and then you must look
> for the start of your token. Taking care to note that you could end up
> in the middle of your token, something like:
>
> // Pseudocode
> char[] data = { "$te" }
> char[] nextData = { "st$" }
>
> Where the token is "$test$".
>
> Depending on your data, it might be that reading lines is enough, and
> do the replace on that basis. Hopefully this is somewhat clear. I had
> a quick search and couldn't find a nice example on doing this, but it
> should be easy enough using StreamReader or friends. If it's not clear
> I can show an example later on.
>

That's fine thanks. Line by line would be ok as the data will never be
broken up and flow onto the next line.

I was kinda hoping there'd be something specifically built for this-
seems like I'm creating mail merge all over again :-)

> --
> Noon Silk
>
> http://dnoondt.wordpress.com/  (Noon Silk) | http://www.mirios.com.au:8081 >
>
> Fancy a quantum lunch?
> http://www.mirios.com.au:8081/index.php?title=Quantum_Lunch
>
> "Every morning when I wake up, I experience an exquisite joy — the joy
> of being this signature."
>

Re: Placeholders in large text file

[Grant Molloy]

Bec,

Read this article and download the demo..
It goes through several different options for "find and replace" in large
text strings..

http://www.codeproject.com/KB/string/fastestcscaseinsstringrep.aspx

Grant

On Mon, Feb 7, 2011 at 9:43 AM, Bec Carter  wrote:

> On Mon, Feb 7, 2011 at 10:35 AM, Noon Silk  wrote:
> > On Mon, Feb 7, 2011 at 10:29 AM, Bec Carter 
> wrote:
> >> On Mon, Feb 7, 2011 at 10:25 AM, Noon Silk 
> wrote:
> >>> On Mon, Feb 7, 2011 at 10:21 AM, Bec Carter 
> wrote:
>  Good mornin' all!
> 
>  I've a requirement to put certain values (after computing them) in
>  various spots in a very large text file. So basically the starting
>  text file can have placeholders where these computed values will end
>  up- like a template. Then my code will compute some values based on
>  user input and i need to fill in the placeholders.
> 
>  Is there a better way to do this besides a simple string replace?
> >>>
> >>> Well, no. You'll need to read the file in and find your tokens and
> >>> replace them. Depending on how large the file it, you might need to do
> >>> this line by line, or chunk by chunk, writing out as you read in, but
> >>> inevitably it comes down to looking for a sequence and replacing it
> >>> with another.
> >>>
> >>> How large is "very large"? Megs? Gigs?
> >>>
> >>
> >> Yup reading all into a string right now and replacing. File is around
> 750 megs
> >
> > Mm, in that case I would definitely think reading in chunk by chunk
> > would be better.
> >
> > So, you read in chunks of chars in a char[], and then you must look
> > for the start of your token. Taking care to note that you could end up
> > in the middle of your token, something like:
> >
> > // Pseudocode
> > char[] data = { "$te" }
> > char[] nextData = { "st$" }
> >
> > Where the token is "$test$".
> >
> > Depending on your data, it might be that reading lines is enough, and
> > do the replace on that basis. Hopefully this is somewhat clear. I had
> > a quick search and couldn't find a nice example on doing this, but it
> > should be easy enough using StreamReader or friends. If it's not clear
> > I can show an example later on.
> >
>
> That's fine thanks. Line by line would be ok as the data will never be
> broken up and flow onto the next line.
>
> I was kinda hoping there'd be something specifically built for this-
> seems like I'm creating mail merge all over again :-)
>
> > --
> > Noon Silk
> >
> > http://dnoondt.wordpress.com/  (Noon Silk) |
> http://www.mirios.com.au:8081 >
> >
> > Fancy a quantum lunch?
> > http://www.mirios.com.au:8081/index.php?title=Quantum_Lunch
> >
> > "Every morning when I wake up, I experience an exquisite joy — the joy
> > of being this signature."
> >
>

Re: Placeholders in large text file

[Michael Minutillo]

If you're in .NET 4.0 land then I'd do something similar to this:

public string ReplaceTokens(string src) { /* ... */ }

File.WriteAllLines(outputFileName,
File.ReadLines(inputFileName).Select(ReplaceTokens));

The ReadLines call (new to .NET 4.0) reads one line at a time and returns it
as you iterate over it so in theory you don't need to have the whole file in
memory. Don't use the ReadAllLines method on a 750MB file which DOES read
the whole thing in before you start.




--
Michael M. Minutillo
Indiscriminate Information Sponge
Blog: http://wolfbyte-net.blogspot.com


On Mon, Feb 7, 2011 at 7:58 AM, Grant Molloy  wrote:

> Bec,
>
> Read this article and download the demo..
> It goes through several different options for "find and replace" in large
> text strings..
>
> http://www.codeproject.com/KB/string/fastestcscaseinsstringrep.aspx
>
> Grant
>
>
> On Mon, Feb 7, 2011 at 9:43 AM, Bec Carter  wrote:
>
>> On Mon, Feb 7, 2011 at 10:35 AM, Noon Silk  wrote:
>> > On Mon, Feb 7, 2011 at 10:29 AM, Bec Carter 
>> wrote:
>> >> On Mon, Feb 7, 2011 at 10:25 AM, Noon Silk 
>> wrote:
>> >>> On Mon, Feb 7, 2011 at 10:21 AM, Bec Carter 
>> wrote:
>>  Good mornin' all!
>> 
>>  I've a requirement to put certain values (after computing them) in
>>  various spots in a very large text file. So basically the starting
>>  text file can have placeholders where these computed values will end
>>  up- like a template. Then my code will compute some values based on
>>  user input and i need to fill in the placeholders.
>> 
>>  Is there a better way to do this besides a simple string replace?
>> >>>
>> >>> Well, no. You'll need to read the file in and find your tokens and
>> >>> replace them. Depending on how large the file it, you might need to do
>> >>> this line by line, or chunk by chunk, writing out as you read in, but
>> >>> inevitably it comes down to looking for a sequence and replacing it
>> >>> with another.
>> >>>
>> >>> How large is "very large"? Megs? Gigs?
>> >>>
>> >>
>> >> Yup reading all into a string right now and replacing. File is around
>> 750 megs
>> >
>> > Mm, in that case I would definitely think reading in chunk by chunk
>> > would be better.
>> >
>> > So, you read in chunks of chars in a char[], and then you must look
>> > for the start of your token. Taking care to note that you could end up
>> > in the middle of your token, something like:
>> >
>> > // Pseudocode
>> > char[] data = { "$te" }
>> > char[] nextData = { "st$" }
>> >
>> > Where the token is "$test$".
>> >
>> > Depending on your data, it might be that reading lines is enough, and
>> > do the replace on that basis. Hopefully this is somewhat clear. I had
>> > a quick search and couldn't find a nice example on doing this, but it
>> > should be easy enough using StreamReader or friends. If it's not clear
>> > I can show an example later on.
>> >
>>
>> That's fine thanks. Line by line would be ok as the data will never be
>> broken up and flow onto the next line.
>>
>> I was kinda hoping there'd be something specifically built for this-
>> seems like I'm creating mail merge all over again :-)
>>
>> > --
>> > Noon Silk
>> >
>> > http://dnoondt.wordpress.com/  (Noon Silk) |
>> http://www.mirios.com.au:8081 >
>> >
>> > Fancy a quantum lunch?
>> > http://www.mirios.com.au:8081/index.php?title=Quantum_Lunch
>> >
>> > "Every morning when I wake up, I experience an exquisite joy — the joy
>> > of being this signature."
>> >
>>
>
>

Re: Placeholders in large text file

[mike smith]

On Mon, Feb 7, 2011 at 12:49 PM, Michael Minutillo <
michael.minuti...@gmail.com> wrote:

> If you're in .NET 4.0 land then I'd do something similar to this:
>
> public string ReplaceTokens(string src) { /* ... */ }
>
> File.WriteAllLines(outputFileName,
> File.ReadLines(inputFileName).Select(ReplaceTokens));
>
> The ReadLines call (new to .NET 4.0) reads one line at a time and returns
> it as you iterate over it so in theory you don't need to have the whole file
> in memory. Don't use the ReadAllLines method on a 750MB file which DOES read
> the whole thing in before you start.
>
>
>

Is that a real problem given physical RAM these days?  If you're going to
write multiple outputs from the one template file of 750 it's going to
rapidly get more efficient to have the template in-ram.   Wait a moment.
 You don't work for Readers Digest, do you?  I have no desire whatever to
make them more efficient.

-- 
Meski

"Going to Starbucks for coffee is like going to prison for sex. Sure, you'll
get it, but it's going to be rough" - Adam Hills

Re: Placeholders in large text file

[Michael Minutillo]

Well, if the template size isn't going change and this is the only app
running on the machine then so be it. Chances are good that neither of those
things is true. I'd still err on the side of having a single line memory at
a time because it's not like the optimization is making it any harder to
read or understand.


Michael M. Minutillo
Indiscriminate Information Sponge
Blog: http://wolfbyte-net.blogspot.com


On Mon, Feb 7, 2011 at 10:30 AM, mike smith  wrote:

> On Mon, Feb 7, 2011 at 12:49 PM, Michael Minutillo <
> michael.minuti...@gmail.com> wrote:
>
>> If you're in .NET 4.0 land then I'd do something similar to this:
>>
>> public string ReplaceTokens(string src) { /* ... */ }
>>
>> File.WriteAllLines(outputFileName,
>> File.ReadLines(inputFileName).Select(ReplaceTokens));
>>
>> The ReadLines call (new to .NET 4.0) reads one line at a time and returns
>> it as you iterate over it so in theory you don't need to have the whole file
>> in memory. Don't use the ReadAllLines method on a 750MB file which DOES read
>> the whole thing in before you start.
>>
>>
>>
>
> Is that a real problem given physical RAM these days?  If you're going to
> write multiple outputs from the one template file of 750 it's going to
> rapidly get more efficient to have the template in-ram.   Wait a moment.
>  You don't work for Readers Digest, do you?  I have no desire whatever to
> make them more efficient.
>
> --
> Meski
>
> "Going to Starbucks for coffee is like going to prison for sex. Sure,
> you'll get it, but it's going to be rough" - Adam Hills
>

Re: Placeholders in large text file

[Bec Carter]

On Mon, Feb 7, 2011 at 12:49 PM, Michael Minutillo
 wrote:
> If you're in .NET 4.0 land then I'd do something similar to this:
> public string ReplaceTokens(string src) { /* ... */ }
> File.WriteAllLines(outputFileName,
> File.ReadLines(inputFileName).Select(ReplaceTokens));
> The ReadLines call (new to .NET 4.0) reads one line at a time and returns it
> as you iterate over it so in theory you don't need to have the whole file in
> memory. Don't use the ReadAllLines method on a 750MB file which DOES read
> the whole thing in before you start.
>
>

Cool I hadn't seen that in .NET 4.0. Cheers

>
> --
> Michael M. Minutillo
> Indiscriminate Information Sponge
> Blog: http://wolfbyte-net.blogspot.com
>
>
> On Mon, Feb 7, 2011 at 7:58 AM, Grant Molloy  wrote:
>>
>> Bec,
>> Read this article and download the demo..
>> It goes through several different options for "find and replace" in large
>> text strings..
>> http://www.codeproject.com/KB/string/fastestcscaseinsstringrep.aspx
>> Grant
>>
>> On Mon, Feb 7, 2011 at 9:43 AM, Bec Carter  wrote:
>>>
>>> On Mon, Feb 7, 2011 at 10:35 AM, Noon Silk  wrote:
>>> > On Mon, Feb 7, 2011 at 10:29 AM, Bec Carter 
>>> > wrote:
>>> >> On Mon, Feb 7, 2011 at 10:25 AM, Noon Silk 
>>> >> wrote:
>>> >>> On Mon, Feb 7, 2011 at 10:21 AM, Bec Carter 
>>> >>> wrote:
>>>  Good mornin' all!
>>> 
>>>  I've a requirement to put certain values (after computing them) in
>>>  various spots in a very large text file. So basically the starting
>>>  text file can have placeholders where these computed values will end
>>>  up- like a template. Then my code will compute some values based on
>>>  user input and i need to fill in the placeholders.
>>> 
>>>  Is there a better way to do this besides a simple string replace?
>>> >>>
>>> >>> Well, no. You'll need to read the file in and find your tokens and
>>> >>> replace them. Depending on how large the file it, you might need to
>>> >>> do
>>> >>> this line by line, or chunk by chunk, writing out as you read in, but
>>> >>> inevitably it comes down to looking for a sequence and replacing it
>>> >>> with another.
>>> >>>
>>> >>> How large is "very large"? Megs? Gigs?
>>> >>>
>>> >>
>>> >> Yup reading all into a string right now and replacing. File is around
>>> >> 750 megs
>>> >
>>> > Mm, in that case I would definitely think reading in chunk by chunk
>>> > would be better.
>>> >
>>> > So, you read in chunks of chars in a char[], and then you must look
>>> > for the start of your token. Taking care to note that you could end up
>>> > in the middle of your token, something like:
>>> >
>>> > // Pseudocode
>>> > char[] data = { "$te" }
>>> > char[] nextData = { "st$" }
>>> >
>>> > Where the token is "$test$".
>>> >
>>> > Depending on your data, it might be that reading lines is enough, and
>>> > do the replace on that basis. Hopefully this is somewhat clear. I had
>>> > a quick search and couldn't find a nice example on doing this, but it
>>> > should be easy enough using StreamReader or friends. If it's not clear
>>> > I can show an example later on.
>>> >
>>>
>>> That's fine thanks. Line by line would be ok as the data will never be
>>> broken up and flow onto the next line.
>>>
>>> I was kinda hoping there'd be something specifically built for this-
>>> seems like I'm creating mail merge all over again :-)
>>>
>>> > --
>>> > Noon Silk
>>> >
>>> > http://dnoondt.wordpress.com/  (Noon Silk) |
>>> > http://www.mirios.com.au:8081 >
>>> >
>>> > Fancy a quantum lunch?
>>> > http://www.mirios.com.au:8081/index.php?title=Quantum_Lunch
>>> >
>>> > "Every morning when I wake up, I experience an exquisite joy — the joy
>>> > of being this signature."
>>> >
>>
>
>

Re: Placeholders in large text file

[Bec Carter]

On Mon, Feb 7, 2011 at 1:33 PM, Michael Minutillo
 wrote:
> Well, if the template size isn't going change and this is the only app
> running on the machine then so be it. Chances are good that neither of those
> things is true. I'd still err on the side of having a single line memory at
> a time because it's not like the optimization is making it any harder to
> read or understand.
>

Yup template file will mostly likely not change and app will always
run locally as an exe. This sort of optimisation you suggested seems
good enough to use as it is fairly simple.

...But I was kinda questioning the design of doing things this way at
all. It seems like what I want is a dynamic "page" (like a webform)
that can run and spit out text just like an asp page does.so the
placeholders would really be <%= %> tags. Is something like this an
option? Can I somehow run an asp.net page locally? Will this cause
performance problems for 750megs of data which is around 70 pages? Am
I going completely crazy? :-)


> Michael M. Minutillo
> Indiscriminate Information Sponge
> Blog: http://wolfbyte-net.blogspot.com
>
>
> On Mon, Feb 7, 2011 at 10:30 AM, mike smith  wrote:
>>
>> On Mon, Feb 7, 2011 at 12:49 PM, Michael Minutillo
>>  wrote:
>>>
>>> If you're in .NET 4.0 land then I'd do something similar to this:
>>> public string ReplaceTokens(string src) { /* ... */ }
>>> File.WriteAllLines(outputFileName,
>>> File.ReadLines(inputFileName).Select(ReplaceTokens));
>>> The ReadLines call (new to .NET 4.0) reads one line at a time and returns
>>> it as you iterate over it so in theory you don't need to have the whole file
>>> in memory. Don't use the ReadAllLines method on a 750MB file which DOES read
>>> the whole thing in before you start.
>>>
>>
>>
>> Is that a real problem given physical RAM these days?  If you're going to
>> write multiple outputs from the one template file of 750 it's going to
>> rapidly get more efficient to have the template in-ram.   Wait a moment.
>>  You don't work for Readers Digest, do you?  I have no desire whatever to
>> make them more efficient.
>> --
>> Meski
>>
>> "Going to Starbucks for coffee is like going to prison for sex. Sure,
>> you'll get it, but it's going to be rough" - Adam Hills
>
>

Re: Placeholders in large text file

[Michael Minutillo]

You could host Razor but I'm not sure how it will handle a template so
large.
http://www.west-wind.com/weblog/posts/864461.aspx

Or possibly you could use NVelocity
http://csharp-source.net/open-source/template-engines/nvelocity
Or StringTemplate
http://www.stringtemplate.org/
Or Spark
http://sparkviewengine.com/

If you're just replacing string tokens with precomputed string values
though, I'd probably just do it one line at a time. Id guess that most of
these template engines are designed to load the template into memory and
turn it into a method that spits out the text given some kind of context
(the way T4 does it [Preprocessed T4 template is another way to go if you
don't mind being all in memory]).


--
Michael M. Minutillo
Indiscriminate Information Sponge
Blog: http://wolfbyte-net.blogspot.com


On Mon, Feb 7, 2011 at 10:42 AM, Bec Carter  wrote:

> On Mon, Feb 7, 2011 at 1:33 PM, Michael Minutillo
>  wrote:
> > Well, if the template size isn't going change and this is the only app
> > running on the machine then so be it. Chances are good that neither of
> those
> > things is true. I'd still err on the side of having a single line memory
> at
> > a time because it's not like the optimization is making it any harder to
> > read or understand.
> >
>
> Yup template file will mostly likely not change and app will always
> run locally as an exe. This sort of optimisation you suggested seems
> good enough to use as it is fairly simple.
>
> ...But I was kinda questioning the design of doing things this way at
> all. It seems like what I want is a dynamic "page" (like a webform)
> that can run and spit out text just like an asp page does.so the
> placeholders would really be <%= %> tags. Is something like this an
> option? Can I somehow run an asp.net page locally? Will this cause
> performance problems for 750megs of data which is around 70 pages? Am
> I going completely crazy? :-)
>
>
> > Michael M. Minutillo
> > Indiscriminate Information Sponge
> > Blog: http://wolfbyte-net.blogspot.com
> >
> >
> > On Mon, Feb 7, 2011 at 10:30 AM, mike smith  wrote:
> >>
> >> On Mon, Feb 7, 2011 at 12:49 PM, Michael Minutillo
> >>  wrote:
> >>>
> >>> If you're in .NET 4.0 land then I'd do something similar to this:
> >>> public string ReplaceTokens(string src) { /* ... */ }
> >>> File.WriteAllLines(outputFileName,
> >>> File.ReadLines(inputFileName).Select(ReplaceTokens));
> >>> The ReadLines call (new to .NET 4.0) reads one line at a time and
> returns
> >>> it as you iterate over it so in theory you don't need to have the whole
> file
> >>> in memory. Don't use the ReadAllLines method on a 750MB file which DOES
> read
> >>> the whole thing in before you start.
> >>>
> >>
> >>
> >> Is that a real problem given physical RAM these days?  If you're going
> to
> >> write multiple outputs from the one template file of 750 it's going to
> >> rapidly get more efficient to have the template in-ram.   Wait a moment.
> >>  You don't work for Readers Digest, do you?  I have no desire whatever
> to
> >> make them more efficient.
> >> --
> >> Meski
> >>
> >> "Going to Starbucks for coffee is like going to prison for sex. Sure,
> >> you'll get it, but it's going to be rough" - Adam Hills
> >
> >
>

RE: Placeholders in large text file

[Bill McCarthy]

If the templates don't change often, but are used somewhat more often, then
it might be worth pre-processing them: that is parse them initially, and
spit them out in parts with a directive file. Then you simply read the
directive file and stitch the parts together.

Eg:
Long text <%=placeholder1 %> more text <%=someotherplaceholder%> even more
text

Would be preprocessed into 3 files and one directive file. The directive
file would just list the parts and placeholder sequence: eg:


   file1.txt
   placeholder1
  file2.txt
someotherplaceholder  
file3.txt


This would save you re-parsing the templates.  Whether or not it is worth
the effort in your case is something you'd have to decide on. I wouldn't
bother unless initial performance and/or you are doing a lot of re-use of
the templates.

As to using a asp.net style parser, it's a possibility, but probably not
worth the overhead or time it would take to investigate another engine.



|-Original Message-
|From: ozdotnet-boun...@ozdotnet.com [mailto:ozdotnet-
|boun...@ozdotnet.com] On Behalf Of Bec Carter
|Sent: Monday, 7 February 2011 1:42 PM
|To: ozDotNet
|Subject: Re: Placeholders in large text file
|
|On Mon, Feb 7, 2011 at 1:33 PM, Michael Minutillo
| wrote:
|> Well, if the template size isn't going change and this is the only app
|> running on the machine then so be it. Chances are good that neither of
|> those things is true. I'd still err on the side of having a single
|> line memory at a time because it's not like the optimization is making
|> it any harder to read or understand.
|>
|
|Yup template file will mostly likely not change and app will always run
locally
|as an exe. This sort of optimisation you suggested seems good enough to use
as
|it is fairly simple.
|
|...But I was kinda questioning the design of doing things this way at all.
It seems
|like what I want is a dynamic "page" (like a webform) that can run and spit
out
|text just like an asp page does.so the placeholders would really be <%=
%>
|tags. Is something like this an option? Can I somehow run an asp.net page
|locally? Will this cause performance problems for 750megs of data which is
|around 70 pages? Am I going completely crazy? :-)
|
|
|> Michael M. Minutillo
|> Indiscriminate Information Sponge
|> Blog: http://wolfbyte-net.blogspot.com
|>
|>
|> On Mon, Feb 7, 2011 at 10:30 AM, mike smith  wrote:
|>>
|>> On Mon, Feb 7, 2011 at 12:49 PM, Michael Minutillo
|>>  wrote:
|>>>
|>>> If you're in .NET 4.0 land then I'd do something similar to this:
|>>> public string ReplaceTokens(string src) { /* ... */ }
|>>> File.WriteAllLines(outputFileName,
|>>> File.ReadLines(inputFileName).Select(ReplaceTokens));
|>>> The ReadLines call (new to .NET 4.0) reads one line at a time and
|>>> returns it as you iterate over it so in theory you don't need to
|>>> have the whole file in memory. Don't use the ReadAllLines method on
|>>> a 750MB file which DOES read the whole thing in before you start.
|>>>
|>>
|>>
|>> Is that a real problem given physical RAM these days?  If you're
|>> going to write multiple outputs from the one template file of 750
|>> it's going to rapidly get more efficient to have the template in-ram.  
Wait a
|moment.
|>>  You don't work for Readers Digest, do you?  I have no desire
|>> whatever to make them more efficient.
|>> --
|>> Meski
|>>
|>> "Going to Starbucks for coffee is like going to prison for sex. Sure,
|>> you'll get it, but it's going to be rough" - Adam Hills
|>
|>

Re: Placeholders in large text file

[Bec Carter]

On Mon, Feb 7, 2011 at 1:59 PM, Bill McCarthy
 wrote:
> If the templates don't change often, but are used somewhat more often, then
> it might be worth pre-processing them: that is parse them initially, and
> spit them out in parts with a directive file. Then you simply read the
> directive file and stitch the parts together.
>
> Eg:
> Long text <%=placeholder1 %> more text <%=someotherplaceholder%> even more
> text
>
> Would be preprocessed into 3 files and one directive file. The directive
> file would just list the parts and placeholder sequence: eg:
>
> 
>   file1.txt
>   placeholder1
>  file2.txt
>    someotherplaceholder
> file3.txt
> 
>
> This would save you re-parsing the templates.  Whether or not it is worth
> the effort in your case is something you'd have to decide on. I wouldn't
> bother unless initial performance and/or you are doing a lot of re-use of
> the templates.
>

Thanks for that Bill this is actually something like what I
considered. I shall re-examine it.

> As to using a asp.net style parser, it's a possibility, but probably not
> worth the overhead or time it would take to investigate another engine.
>
>
>
> |-Original Message-
> |From: ozdotnet-boun...@ozdotnet.com [mailto:ozdotnet-
> |boun...@ozdotnet.com] On Behalf Of Bec Carter
> |Sent: Monday, 7 February 2011 1:42 PM
> |To: ozDotNet
> |Subject: Re: Placeholders in large text file
> |
> |On Mon, Feb 7, 2011 at 1:33 PM, Michael Minutillo
> | wrote:
> |> Well, if the template size isn't going change and this is the only app
> |> running on the machine then so be it. Chances are good that neither of
> |> those things is true. I'd still err on the side of having a single
> |> line memory at a time because it's not like the optimization is making
> |> it any harder to read or understand.
> |>
> |
> |Yup template file will mostly likely not change and app will always run
> locally
> |as an exe. This sort of optimisation you suggested seems good enough to use
> as
> |it is fairly simple.
> |
> |...But I was kinda questioning the design of doing things this way at all.
> It seems
> |like what I want is a dynamic "page" (like a webform) that can run and spit
> out
> |text just like an asp page does.so the placeholders would really be <%=
> %>
> |tags. Is something like this an option? Can I somehow run an asp.net page
> |locally? Will this cause performance problems for 750megs of data which is
> |around 70 pages? Am I going completely crazy? :-)
> |
> |
> |> Michael M. Minutillo
> |> Indiscriminate Information Sponge
> |> Blog: http://wolfbyte-net.blogspot.com
> |>
> |>
> |> On Mon, Feb 7, 2011 at 10:30 AM, mike smith  wrote:
> |>>
> |>> On Mon, Feb 7, 2011 at 12:49 PM, Michael Minutillo
> |>>  wrote:
> |>>>
> |>>> If you're in .NET 4.0 land then I'd do something similar to this:
> |>>> public string ReplaceTokens(string src) { /* ... */ }
> |>>> File.WriteAllLines(outputFileName,
> |>>> File.ReadLines(inputFileName).Select(ReplaceTokens));
> |>>> The ReadLines call (new to .NET 4.0) reads one line at a time and
> |>>> returns it as you iterate over it so in theory you don't need to
> |>>> have the whole file in memory. Don't use the ReadAllLines method on
> |>>> a 750MB file which DOES read the whole thing in before you start.
> |>>>
> |>>
> |>>
> |>> Is that a real problem given physical RAM these days?  If you're
> |>> going to write multiple outputs from the one template file of 750
> |>> it's going to rapidly get more efficient to have the template in-ram.
> Wait a
> |moment.
> |>>  You don't work for Readers Digest, do you?  I have no desire
> |>> whatever to make them more efficient.
> |>> --
> |>> Meski
> |>>
> |>> "Going to Starbucks for coffee is like going to prison for sex. Sure,
> |>> you'll get it, but it's going to be rough" - Adam Hills
> |>
> |>
>
>

Re: Placeholders in large text file

[Bec Carter]

On Mon, Feb 7, 2011 at 1:49 PM, Michael Minutillo
 wrote:
> You could host Razor but I'm not sure how it will handle a template so
> large.
> http://www.west-wind.com/weblog/posts/864461.aspx
> Or possibly you could use NVelocity
> http://csharp-source.net/open-source/template-engines/nvelocity
> Or StringTemplate
> http://www.stringtemplate.org/
> Or Spark
> http://sparkviewengine.com/
> If you're just replacing string tokens with precomputed string values
> though, I'd probably just do it one line at a time. Id guess that most of
> these template engines are designed to load the template into memory and
> turn it into a method that spits out the text given some kind of context
> (the way T4 does it [Preprocessed T4 template is another way to go if you
> don't mind being all in memory]).
>

Thanks Michael I'll hava look into these.

> --
> Michael M. Minutillo
> Indiscriminate Information Sponge
> Blog: http://wolfbyte-net.blogspot.com
>
>
> On Mon, Feb 7, 2011 at 10:42 AM, Bec Carter  wrote:
>>
>> On Mon, Feb 7, 2011 at 1:33 PM, Michael Minutillo
>>  wrote:
>> > Well, if the template size isn't going change and this is the only app
>> > running on the machine then so be it. Chances are good that neither of
>> > those
>> > things is true. I'd still err on the side of having a single line memory
>> > at
>> > a time because it's not like the optimization is making it any harder to
>> > read or understand.
>> >
>>
>> Yup template file will mostly likely not change and app will always
>> run locally as an exe. This sort of optimisation you suggested seems
>> good enough to use as it is fairly simple.
>>
>> ...But I was kinda questioning the design of doing things this way at
>> all. It seems like what I want is a dynamic "page" (like a webform)
>> that can run and spit out text just like an asp page does.so the
>> placeholders would really be <%= %> tags. Is something like this an
>> option? Can I somehow run an asp.net page locally? Will this cause
>> performance problems for 750megs of data which is around 70 pages? Am
>> I going completely crazy? :-)
>>
>>
>> > Michael M. Minutillo
>> > Indiscriminate Information Sponge
>> > Blog: http://wolfbyte-net.blogspot.com
>> >
>> >
>> > On Mon, Feb 7, 2011 at 10:30 AM, mike smith  wrote:
>> >>
>> >> On Mon, Feb 7, 2011 at 12:49 PM, Michael Minutillo
>> >>  wrote:
>> >>>
>> >>> If you're in .NET 4.0 land then I'd do something similar to this:
>> >>> public string ReplaceTokens(string src) { /* ... */ }
>> >>> File.WriteAllLines(outputFileName,
>> >>> File.ReadLines(inputFileName).Select(ReplaceTokens));
>> >>> The ReadLines call (new to .NET 4.0) reads one line at a time and
>> >>> returns
>> >>> it as you iterate over it so in theory you don't need to have the
>> >>> whole file
>> >>> in memory. Don't use the ReadAllLines method on a 750MB file which
>> >>> DOES read
>> >>> the whole thing in before you start.
>> >>>
>> >>
>> >>
>> >> Is that a real problem given physical RAM these days?  If you're going
>> >> to
>> >> write multiple outputs from the one template file of 750 it's going to
>> >> rapidly get more efficient to have the template in-ram.   Wait a
>> >> moment.
>> >>  You don't work for Readers Digest, do you?  I have no desire whatever
>> >> to
>> >> make them more efficient.
>> >> --
>> >> Meski
>> >>
>> >> "Going to Starbucks for coffee is like going to prison for sex. Sure,
>> >> you'll get it, but it's going to be rough" - Adam Hills
>> >
>> >
>
>

Re: Placeholders in large text file

[Noon Silk]

On Mon, Feb 7, 2011 at 1:42 PM, Bec Carter  wrote:
> [...]
>
> Yup template file will mostly likely not change and app will always
> run locally as an exe. This sort of optimisation you suggested seems
> good enough to use as it is fairly simple.
>
> ...But I was kinda questioning the design of doing things this way at
> all. It seems like what I want is a dynamic "page" (like a webform)
> that can run and spit out text just like an asp page does.so the
> placeholders would really be <%= %> tags. Is something like this an
> option? Can I somehow run an asp.net page locally? Will this cause
> performance problems for 750megs of data which is around 70 pages? Am
> I going completely crazy? :-)

I think the reason it "seems" wrong is that you'd typically generate
the surrounding data as well. From the sounds of things, something
else is generating that for you, and then you "add your part". It
could be that there is a better way for you to "add your part", but
we'd need to know how the initial generation is being done.

In this case, I think your proposed solution is appropriate; even
Bills latest comment is "technically acceptable", but would probably,
IMHO, introduce too much confusion if the process were to change a
bit.

-- 
Noon Silk

http://dnoondt.wordpress.com/  (Noon Silk) | http://www.mirios.com.au:8081 >

Fancy a quantum lunch?
http://www.mirios.com.au:8081/index.php?title=Quantum_Lunch

"Every morning when I wake up, I experience an exquisite joy — the joy
of being this signature."

Re: Placeholders in large text file

[mike smith]

On Mon, Feb 7, 2011 at 1:42 PM, Bec Carter  wrote:

> On Mon, Feb 7, 2011 at 1:33 PM, Michael Minutillo
>  wrote:
> > Well, if the template size isn't going change and this is the only app
> > running on the machine then so be it. Chances are good that neither of
> those
> > things is true. I'd still err on the side of having a single line memory
> at
> > a time because it's not like the optimization is making it any harder to
> > read or understand.
> >
>
> Yup template file will mostly likely not change and app will always
> run locally as an exe. This sort of optimisation you suggested seems
> good enough to use as it is fairly simple.
>
> ...But I was kinda questioning the design of doing things this way at
> all. It seems like what I want is a dynamic "page" (like a webform)
> that can run and spit out text just like an asp page does.so the
> placeholders would really be <%= %> tags. Is something like this an
> option? Can I somehow run an asp.net page locally? Will this cause
> performance problems for 750megs of data which is around 70 pages? Am
> I going completely crazy? :-)
>
>
>
If you're repeating the replacement a lot you could keep a structure that
pointed into the template giving the locations.

-- 
Meski

"Going to Starbucks for coffee is like going to prison for sex. Sure, you'll
get it, but it's going to be rough" - Adam Hills

Re: Placeholders in large text file

[Preet Sangha]

N

On 7 February 2011 16:23, mike smith  wrote:

> On Mon, Feb 7, 2011 at 1:42 PM, Bec Carter  wrote:
>
>> On Mon, Feb 7, 2011 at 1:33 PM, Michael Minutillo
>>  wrote:
>> > Well, if the template size isn't going change and this is the only app
>> > running on the machine then so be it. Chances are good that neither of
>> those
>> > things is true. I'd still err on the side of having a single line memory
>> at
>> > a time because it's not like the optimization is making it any harder to
>> > read or understand.
>> >
>>
>> Yup template file will mostly likely not change and app will always
>> run locally as an exe. This sort of optimisation you suggested seems
>> good enough to use as it is fairly simple.
>>
>> ...But I was kinda questioning the design of doing things this way at
>> all. It seems like what I want is a dynamic "page" (like a webform)
>> that can run and spit out text just like an asp page does.so the
>> placeholders would really be <%= %> tags. Is something like this an
>> option? Can I somehow run an asp.net page locally? Will this cause
>> performance problems for 750megs of data which is around 70 pages? Am
>> I going completely crazy? :-)
>>
>>
>>
> If you're repeating the replacement a lot you could keep a structure that
> pointed into the template giving the locations.
>
> --
> Meski
>
> "Going to Starbucks for coffee is like going to prison for sex. Sure,
> you'll get it, but it's going to be rough" - Adam Hills
>



-- 
regards,
Preet, Overlooking the Ocean, Auckland

Re: Placeholders in large text file

[Preet Sangha]

Sorry for hijacking the thread  - but can i recommend the use of Memory
Mapped Files for reading and writing large files? This will make it far far
quicker as changes pages in memory won't need to be stored in the system
page file.



> On 7 February 2011 16:23, mike smith  wrote:
>
>> On Mon, Feb 7, 2011 at 1:42 PM, Bec Carter wrote:
>>
>>> On Mon, Feb 7, 2011 at 1:33 PM, Michael Minutillo
>>>  wrote:
>>> > Well, if the template size isn't going change and this is the only app
>>> > running on the machine then so be it. Chances are good that neither of
>>> those
>>> > things is true. I'd still err on the side of having a single line
>>> memory at
>>> > a time because it's not like the optimization is making it any harder
>>> to
>>> > read or understand.
>>> >
>>>
>>> Yup template file will mostly likely not change and app will always
>>> run locally as an exe. This sort of optimisation you suggested seems
>>> good enough to use as it is fairly simple.
>>>
>>> ...But I was kinda questioning the design of doing things this way at
>>> all. It seems like what I want is a dynamic "page" (like a webform)
>>> that can run and spit out text just like an asp page does.so the
>>> placeholders would really be <%= %> tags. Is something like this an
>>> option? Can I somehow run an asp.net page locally? Will this cause
>>> performance problems for 750megs of data which is around 70 pages? Am
>>> I going completely crazy? :-)
>>>
>>>
>>>
>> If you're repeating the replacement a lot you could keep a structure that
>> pointed into the template giving the locations.
>>
>> --
>> Meski
>>
>> "Going to Starbucks for coffee is like going to prison for sex. Sure,
>> you'll get it, but it's going to be rough" - Adam Hills
>>
>
>
>
> --
> regards,
> Preet, Overlooking the Ocean, Auckland
>



-- 
regards,
Preet, Overlooking the Ocean, Auckland

RE: Placeholders in large text file

[Bill McCarthy]

There's caveats on that. Frist off it means you're using pointers kind-of
approach which is good but it also means you immediately lose all built in
text comparison options other than binary equality. The other big caveat is
you need to endure replacements are the same size otherwise you end up
moving a lot of memory around each time you make a replacement

|-Original Message-
|From: ozdotnet-boun...@ozdotnet.com [mailto:ozdotnet-
|boun...@ozdotnet.com] On Behalf Of Preet Sangha
|Sent: Monday, 7 February 2011 2:28 PM
|To: ozDotNet
|Subject: Re: Placeholders in large text file
|
|Sorry for hijacking the thread  - but can i recommend the use of Memory
|Mapped Files for reading and writing large files? This will make it far far
|quicker as changes pages in memory won't need to be stored in the system
|page file.
|
|
|
|   On 7 February 2011 16:23, mike smith  wrote:
|
|
|   On Mon, Feb 7, 2011 at 1:42 PM, Bec Carter
| wrote:
|
|
|   On Mon, Feb 7, 2011 at 1:33 PM, Michael Minutillo
|
|wrote:
|
|   > Well, if the template size isn't going change and
this is
|the only app
|   > running on the machine then so be it. Chances are
good
|that neither of those
|   > things is true. I'd still err on the side of
having a single
|line memory at
|   > a time because it's not like the optimization is
making it
|any harder to
|   > read or understand.
|   >
|
|
|   Yup template file will mostly likely not change and
app
|will always
|   run locally as an exe. This sort of optimisation you
|suggested seems
|   good enough to use as it is fairly simple.
|
|   ...But I was kinda questioning the design of doing
things
|this way at
|   all. It seems like what I want is a dynamic "page"
(like a
|webform)
|   that can run and spit out text just like an asp page
|does.so the
|   placeholders would really be <%= %> tags. Is
something
|like this an
|   option? Can I somehow run an asp.net page locally?
Will
|this cause
|   performance problems for 750megs of data which is
|around 70 pages? Am
|   I going completely crazy? :-)
|
|
|
|
|
|   If you're repeating the replacement a lot you could keep a
|structure that pointed into the template giving the locations.
|
|   --
|
|   Meski
|
|   "Going to Starbucks for coffee is like going to prison for
sex.
|Sure, you'll get it, but it's going to be rough" - Adam Hills
|
|
|
|
|
|   --
|   regards,
|   Preet, Overlooking the Ocean, Auckland
|
|
|
|
|
|--
|regards,
|Preet, Overlooking the Ocean, Auckland

RE: Placeholders in large text file

[David Kean]

There's no reason you can't treat that memory as a normal text (ie pass the 
view's stream to StreamReader).

-Original Message-
From: ozdotnet-boun...@ozdotnet.com [mailto:ozdotnet-boun...@ozdotnet.com] On 
Behalf Of Bill McCarthy
Sent: Sunday, February 06, 2011 7:32 PM
To: 'ozDotNet'
Subject: RE: Placeholders in large text file

There's caveats on that. Frist off it means you're using pointers kind-of 
approach which is good but it also means you immediately lose all built in text 
comparison options other than binary equality. The other big caveat is you need 
to endure replacements are the same size otherwise you end up moving a lot of 
memory around each time you make a replacement

|-Original Message-
|From: ozdotnet-boun...@ozdotnet.com [mailto:ozdotnet- 
|boun...@ozdotnet.com] On Behalf Of Preet Sangha
|Sent: Monday, 7 February 2011 2:28 PM
|To: ozDotNet
|Subject: Re: Placeholders in large text file
|
|Sorry for hijacking the thread  - but can i recommend the use of Memory 
|Mapped Files for reading and writing large files? This will make it far 
|far quicker as changes pages in memory won't need to be stored in the 
|system page file.
|
|
|
|   On 7 February 2011 16:23, mike smith  wrote:
|
|
|   On Mon, Feb 7, 2011 at 1:42 PM, Bec Carter 
 
|wrote:
|
|
|   On Mon, Feb 7, 2011 at 1:33 PM, Michael Minutillo
|
|wrote:
|
|   > Well, if the template size isn't going change and
this is
|the only app
|   > running on the machine then so be it. Chances are
good
|that neither of those
|   > things is true. I'd still err on the side of
having a single
|line memory at
|   > a time because it's not like the optimization is
making it
|any harder to
|   > read or understand.
|   >
|
|
|   Yup template file will mostly likely not change and
app
|will always
|   run locally as an exe. This sort of optimisation you 
suggested seems
|   good enough to use as it is fairly simple.
|
|   ...But I was kinda questioning the design of doing
things
|this way at
|   all. It seems like what I want is a dynamic "page"
(like a
|webform)
|   that can run and spit out text just like an asp page 
does.so the
|   placeholders would really be <%= %> tags. Is
something
|like this an
|   option? Can I somehow run an asp.net page locally?
Will
|this cause
|   performance problems for 750megs of data which is 
around 70 pages? 
|Am
|   I going completely crazy? :-)
|
|
|
|
|
|   If you're repeating the replacement a lot you could keep a 
structure 
|that pointed into the template giving the locations.
|
|   --
|
|   Meski
|
|   "Going to Starbucks for coffee is like going to prison for
sex.
|Sure, you'll get it, but it's going to be rough" - Adam Hills
|
|
|
|
|
|   --
|   regards,
|   Preet, Overlooking the Ocean, Auckland
|
|
|
|
|
|--
|regards,
|Preet, Overlooking the Ocean, Auckland

RE: Placeholders in large text file

[Ian Thomas]

David
Can you spell that out, please - ie, what in Preet's latest post is not
correct? And why would we use memory-mapped files (in the scenario posed by
Bec Carter)? 


Ian Thomas
Victoria Park, Western Australia



-Original Message-
From: ozdotnet-boun...@ozdotnet.com [mailto:ozdotnet-boun...@ozdotnet.com]
On Behalf Of David Kean
Sent: Tuesday, February 08, 2011 1:29 AM
To: ozDotNet
Subject: RE: Placeholders in large text file

There's no reason you can't treat that memory as a normal text (ie pass the
view's stream to StreamReader).

-Original Message-
From: ozdotnet-boun...@ozdotnet.com [mailto:ozdotnet-boun...@ozdotnet.com]
On Behalf Of Bill McCarthy
Sent: Sunday, February 06, 2011 7:32 PM
To: 'ozDotNet'
Subject: RE: Placeholders in large text file

There's caveats on that. Frist off it means you're using pointers kind-of
approach which is good but it also means you immediately lose all built in
text comparison options other than binary equality. The other big caveat is
you need to endure replacements are the same size otherwise you end up
moving a lot of memory around each time you make a replacement
Bec Carter)? 
|-Original Message-
|From: ozdotnet-boun...@ozdotnet.com [mailto:ozdotnet- 
|boun...@ozdotnet.com] On Behalf Of Preet Sangha
|Sent: Monday, 7 February 2011 2:28 PM
|To: ozDotNet
|Subject: Re: Placeholders in large text file
|
|Sorry for hijacking the thread  - but can i recommend the use of Memory 
|Mapped Files for reading and writing large files? This will make it far 
|far quicker as changes pages in memory won't need to be stored in the 
|system page file.
|
|

RE: Placeholders in large text file

[Bill McCarthy]

I think David is referring to the use of streams with the managed wrappers
of MM's rather than pointers.

|-Original Message-
|From: ozdotnet-boun...@ozdotnet.com [mailto:ozdotnet-
|boun...@ozdotnet.com] On Behalf Of Ian Thomas
|Sent: Tuesday, 8 February 2011 1:54 PM
|To: 'ozDotNet'
|Subject: RE: Placeholders in large text file
|
|David
|Can you spell that out, please - ie, what in Preet's latest post is not
correct? And
|why would we use memory-mapped files (in the scenario posed by Bec Carter)?
|
|
|Ian Thomas
|Victoria Park, Western Australia
|
|
|
|-Original Message-
|From: ozdotnet-boun...@ozdotnet.com [mailto:ozdotnet-
|boun...@ozdotnet.com]
|On Behalf Of David Kean
|Sent: Tuesday, February 08, 2011 1:29 AM
|To: ozDotNet
|Subject: RE: Placeholders in large text file
|
|There's no reason you can't treat that memory as a normal text (ie pass the
|view's stream to StreamReader).
|
|-Original Message-
|From: ozdotnet-boun...@ozdotnet.com [mailto:ozdotnet-
|boun...@ozdotnet.com]
|On Behalf Of Bill McCarthy
|Sent: Sunday, February 06, 2011 7:32 PM
|To: 'ozDotNet'
|Subject: RE: Placeholders in large text file
|
|There's caveats on that. Frist off it means you're using pointers kind-of
|approach which is good but it also means you immediately lose all built in
text
|comparison options other than binary equality. The other big caveat is you
need
|to endure replacements are the same size otherwise you end up moving a lot
of
|memory around each time you make a replacement Bec Carter)?
||-Original Message-
||From: ozdotnet-boun...@ozdotnet.com [mailto:ozdotnet-
||boun...@ozdotnet.com] On Behalf Of Preet Sangha
||Sent: Monday, 7 February 2011 2:28 PM
||To: ozDotNet
||Subject: Re: Placeholders in large text file
||
||Sorry for hijacking the thread  - but can i recommend the use of Memory
||Mapped Files for reading and writing large files? This will make it far
||far quicker as changes pages in memory won't need to be stored in the
||system page file.
||
||

Re: Placeholders in large text file

[Arjang Assadi]

What is the real problem that a solution to it consists of  putting
certain values (after computing them) in very lage file?

Need more info for a meaningfull solutions.

Regards

Arjang


> various spots in a very large text file

On 7 February 2011 10:21, Bec Carter  wrote:
> Good mornin' all!
>
> I've a requirement to put certain values (after computing them) in
> various spots in a very large text file. So basically the starting
> text file can have placeholders where these computed values will end
> up- like a template. Then my code will compute some values based on
> user input and i need to fill in the placeholders.
> Is there a better way to do this besides a simple string replace?
>
> Cheers.
> Bec
>

Re: Placeholders in large text file

[Arjang Assadi]

The first question is why such a huge file is being used, somebody can
learn how to use the slide rule faster but they are never gonna beat a
simple calculator ( these a days), I say before moving to memory
mapped files let's understand the real problem rather than optimise a
possibly wrong solution.

Regards

Arjang

On 7 February 2011 14:27, Preet Sangha  wrote:
> Sorry for hijacking the thread  - but can i recommend the use of Memory
> Mapped Files for reading and writing large files? This will make it far far
> quicker as changes pages in memory won't need to be stored in the system
> page file.
>
>>
>> On 7 February 2011 16:23, mike smith  wrote:
>>>
>>> On Mon, Feb 7, 2011 at 1:42 PM, Bec Carter 
>>> wrote:

 On Mon, Feb 7, 2011 at 1:33 PM, Michael Minutillo
  wrote:
 > Well, if the template size isn't going change and this is the only app
 > running on the machine then so be it. Chances are good that neither of
 > those
 > things is true. I'd still err on the side of having a single line
 > memory at
 > a time because it's not like the optimization is making it any harder
 > to
 > read or understand.
 >

 Yup template file will mostly likely not change and app will always
 run locally as an exe. This sort of optimisation you suggested seems
 good enough to use as it is fairly simple.

 ...But I was kinda questioning the design of doing things this way at
 all. It seems like what I want is a dynamic "page" (like a webform)
 that can run and spit out text just like an asp page does.so the
 placeholders would really be <%= %> tags. Is something like this an
 option? Can I somehow run an asp.net page locally? Will this cause
 performance problems for 750megs of data which is around 70 pages? Am
 I going completely crazy? :-)


>>>
>>> If you're repeating the replacement a lot you could keep a structure that
>>> pointed into the template giving the locations.
>>> --
>>> Meski
>>>
>>> "Going to Starbucks for coffee is like going to prison for sex. Sure,
>>> you'll get it, but it's going to be rough" - Adam Hills
>>
>>
>>
>> --
>> regards,
>> Preet, Overlooking the Ocean, Auckland
>
>
>
> --
> regards,
> Preet, Overlooking the Ocean, Auckland
>

Re: Placeholders in large text file

[Bec Carter]

G'day Arjang!

It's a config file for a legacy system which I have no control over.
As a favour to a friend I created a simple winforms UI allowing them
to enter some values, do some calculations and then insert them into
the file at particular positions.

I ended up going with Michael Minutillo's .NET 4 suggestion- that
worked well. I will be looking into T4 and the other stuff Michael
mentioned also since that stuff sounds pretty cool.

Cheers.
Bec

On Tue, Feb 8, 2011 at 2:03 PM, Arjang Assadi  wrote:
> What is the real problem that a solution to it consists of  putting
> certain values (after computing them) in very lage file?
>
> Need more info for a meaningfull solutions.
>
> Regards
>
> Arjang
>
>
>> various spots in a very large text file
>
> On 7 February 2011 10:21, Bec Carter  wrote:
>> Good mornin' all!
>>
>> I've a requirement to put certain values (after computing them) in
>> various spots in a very large text file. So basically the starting
>> text file can have placeholders where these computed values will end
>> up- like a template. Then my code will compute some values based on
>> user input and i need to fill in the placeholders.
>> Is there a better way to do this besides a simple string replace?
>>
>> Cheers.
>> Bec
>>
>

Re: Placeholders in large text file

[Arjang Assadi]

Hi Bec,

My only advice will be that not to deal directly with the text file or
the memory mapped file.
I would recommend having at least two layers:

1.To deal with the file at the low level directly as
text-file/memory-mapped-file/etc.

2.A logical layer that uses the low level layer. In this layer you
would be dealing with concepts that are abstraction the logic of what
you are trying to do rather than handling low level
string/stream/file/etc. manipulations.

Regards

Arjang


On 9 February 2011 10:03, Bec Carter  wrote:
> G'day Arjang!
>
> It's a config file for a legacy system which I have no control over.
> As a favour to a friend I created a simple winforms UI allowing them
> to enter some values, do some calculations and then insert them into
> the file at particular positions.
>
> I ended up going with Michael Minutillo's .NET 4 suggestion- that
> worked well. I will be looking into T4 and the other stuff Michael
> mentioned also since that stuff sounds pretty cool.
>
> Cheers.
> Bec
>
> On Tue, Feb 8, 2011 at 2:03 PM, Arjang Assadi  wrote:
>> What is the real problem that a solution to it consists of  putting
>> certain values (after computing them) in very lage file?
>>
>> Need more info for a meaningfull solutions.
>>
>> Regards
>>
>> Arjang
>>
>>
>>> various spots in a very large text file
>>
>> On 7 February 2011 10:21, Bec Carter  wrote:
>>> Good mornin' all!
>>>
>>> I've a requirement to put certain values (after computing them) in
>>> various spots in a very large text file. So basically the starting
>>> text file can have placeholders where these computed values will end
>>> up- like a template. Then my code will compute some values based on
>>> user input and i need to fill in the placeholders.
>>> Is there a better way to do this besides a simple string replace?
>>>
>>> Cheers.
>>> Bec
>>>
>>
>