Yes. I used those for a long time. I think. ;. is faster. On Apr 10, 2014 6:06 PM, "Raul Miller" <[email protected]> wrote:
> I'm going to be building hashes, but currently it looks like it'll be > faster to generate them after finding the unique values rather than > before. > > Hashes have great sales pitches in favor of them, but generating them > takes time. > > Meanwhile, getting lots and lots of real memory is relatively cheap. > According to https://aws.amazon.com/ec2/pricing/ I can use a machine > with 244 gigabytes of ram for $3.50 an hour. I do not think I'll need > that much memory, even for a short time. > > If it helps, though, there's other ways to get a substring: > > 4{.3}. 'abcdefghijk' > defg > (3+i.4){'abcdefghijk' > defg > > Thanks, > > -- > Raul > > > On Thu, Apr 10, 2014 at 6:45 PM, Don Guinn <[email protected]> wrote: > > > > > > Sounds like you can format the data as you want once you get it all. > Given > > the amount of data there is no way to read it > > the entire file > > in an interactive environment. So it has to be indexed or in a database. > > If you really want to use a segmented string you could keep the index to > > the start of each line to > > quickly get > > the line. Also, build a hash for the keys you intend to search on rather > > than trying to search the actual file. > > This index and hash should fit in memory quite easily. > > When you talk about a file that big, mapped or not, it's still going to > > require the amount of virtual storage the size of the file, which will > need > > to be moved from the file or swap into real storage to process. > > Scanning the entire file > > will be very slow unless you have lots and lots of real memory. > > > > I haven't seen anyone mention an easy way to substring from a text file. > It > > took me a long time to find it in the dictionary. > > It really exists. > > And it was not obvious when I finally did find it. So I will post it here > > for anyone who hasn't found it. > > > > > > substr=:4 : 0 NB. x: start and length, y: string > > > > (,:x)];.0 y > > > > ) > > > > 3 4 substr 'abcdefghijk' > > > > defg > > ---------------------------------------------------------------------- > > For information about J forums see http://www.jsoftware.com/forums.htm > ---------------------------------------------------------------------- > For information about J forums see http://www.jsoftware.com/forums.htm > ---------------------------------------------------------------------- For information about J forums see http://www.jsoftware.com/forums.htm
