On 1/14/07, Jim Magnuson <[EMAIL PROTECTED]> wrote:
Hi, I was able to get my Finnish corpus project off the ground this
week with help from this group; thank you very much.
Now I've run into a small problem. After reading in the corpus of
470,000 words and breaking them into syllables, I have created a list
of all possible "nonwords" (words missing from the corpus) that
consist of 2 syllables, each consisting of a consonant and vowel --
CV for short. (I need this list as part of a research project on
language learning, where we will use possible words that don't
already exist in the language).
I was able to generate all possible CV-CV strings and check them
efficiently against entire words in the corpus using hashes. However,
I also want to make sure that the strings do not occur as substrings
of any words in the corpus. I thought grep would be perfect, but the
problem is that there are 66,000 nonwords and 470,000 words; it takes
about 4-5 secs to check each nonword (on my sad old mac laptop) with
the code pasted in below. Can anyone suggest a more efficient method
(I know my code could be much more concise, but I am primarily
interested in speed, of course)?
thank you very much,
jim
while(<SYLFILE>){
chomp;
# on next line, we only care about $ortho, the orthographic string
Then just take what you're interested in:
push @ortho, (split)[1];
}
close SYLFILE;
To get the number of items in an array, just use the array in scalar context:
print STDERR "READ ", scalar(@ortho), " items from ", $sylfile, "\n";
Doing your matches is going to be time-consuming no matter what you
do--your best-case scenario here is 940,000 matches with the initial
substitution--but using a loop should speed things up a bit. The
problem with grep is that it iterates through the entire list no
matter what. Since you don't seem to care how many matches there are
you can bail out of the loop as soon as you match.
while(<NWFILE>){
s/-//g;
my $nw = (split)[0];
my $hit = 0;
foreach (@ortho) {
$hit++ && last if /$nw/;
}
print unless $hit;
}
Depending on your data, it might also make sense to reverse the
operations to read the nonwords into memory and match every syllable
against the list.
If this is going to be a requently-performed operation, you may want
to compute the substrings in each word in advance and embed the list
in your data file. That way you won't have to perform the matches for
every word: you can just check to see if a non-word exists a words
list of substrings.
HTH,
-- jay
--------------------------------------------------
This email and attachment(s): [ ] blogable; [ x ] ask first; [ ]
private and confidential
daggerquill [at] gmail [dot] com
http://www.tuaw.com http://www.downloadsquad.com http://www.engatiki.org
values of β will give rise to dom!
