> To summarize, Yes, i can have a lot of those datapoints. And i want to
> interpolate them by Least squares(which i have already figured out for multi
> dimensions) and use only that for further processing in my other numerical
> code. So the problem now is to have a very efficient and scalable wa
David,
I have not yet tried BufferedStream to read the data. I will play around
with that now and will see how it works out. Also, i do have a benchmark
from a standard legacy code which has all the data in binary format. Hence,
the comparison is not straightforward but i do have a clear idea of h
It seems pretty unlikely that you actually will receive a data file (or group
of files) with more than 62 billion data points. You have what seems to be a
"sparse array" -- the fact that you have 50 possible materials and 6 possible
dimensions doesn't mean that you will actually will receive in
So, your main concern is that because the files could be HUGE, your
"load" performance could be bad due to the large number of IO
requests. Is that an accurate description of your problem?
If that is right, then I think I'd first try using a BufferedStream()
to read in the data. You can play wit
David,
The problem i refer to is due to the fact that i do not know the dimension
of the data set i am going to read before hand. The sample input i had
posted was very short and has only 2 dimensions. I could have upto 6
dimensions (dependencies for each parameter). It can very quickly blow up
th
> It is actually not much but the main problem isn't getting the data in
> memory but reading the string and populating them into a double array. And
> that string conversion by parsing is where i am stuck at !
So, how long is it taking you to process this whole thing? It doesn't
"seem" that it w
About the size of each file : they can be around 6KB/material.
So total data size : 6KB * around 50 = 300 KB
It is actually not much but the main problem isn't getting the data in
memory but reading the string and populating them into a double array. And
that string conversion by parsing is where
Hi. Thanks for the prompt reply.
Here's a small piece of the text file i am trying to read.
* Table Input
*
* Tf Rhom Bor Tm.
5 6 0 0
*
*** set # 1
*
* Group No. 1
*
*** DiffC Table
*
.500E+03 .7602200E+03 .8672700E+03 .9218800E+03 .150E+04
.6413994E+03 .7114275E+03 .76946
Vijay,
Are you having trouble correctly reading the data out of the file, or
is it just a performance problem?
Assuming it's the last, a few things. First, consider using a
BufferedStream. That will reduce the number of IO's that are
required. Second, how big is the file that you will be readi
What about sucking the entire contents of the file into memory as a
memory-mapped file and then picking through the format that way? It'll
minimize the amount of file I/O, but aside from that I'm not sure what you
can do to optimize this further.
Ted Neward
Author, Presenter, Consultant
Java, .NET
Can you post an snippet of your text file?
It could help to understand your problem better.
// Ryan
===
This list is hosted by DevelopMentor® http://www.develop.com
View archives and manage your subscription(s) at http://discuss.develop.com
11 matches
Mail list logo
