Hi, I wud like some help in assembling the kit.
Thanks in Advance --- Rob Dixon <[EMAIL PROTECTED]> wrote: > Mohit_jain01 wrote: > > > > > From: Rob Dixon > > > > > > Mohit_jain01 wrote: > > > > > > > > I am facing a problem with text file manipulation with Perl. > > > > > > > > I have a file with over 2 lac lines of data. > > > > I need to find the duplicates(strings) in the file and copy those records into > > > > another > file. > > > > > > > > Is there a function/module in Perl by which I can read the duplicates in a > > > > file at one go > and print them > > > > on to another file. > > > > > > Before we can help you we need to know a little more of your problem. > > > > > > Are you looking for duplicate lines in the file, or duplicate strings defined > > > in some other way? How big is the file you want to read (how many lines > > > or strings do you want to compare)? > > > > > > There are modules which will help you write your program, but exactly > > > how you go about it depends on the details of your problem. > > > > I have a big file containing about 200000 lines. This file basically contains some > > records. A > sample of the file is as given > > below: > > > > dn: cn=1148734,ou=Employees,dc=jci,dc=com > > > > displayname: Herek, Moriah L > > > > jdirlastfourssn: 2888 > > > > dn: cn=1148735,ou=Employees,dc=jci,dc=com > > > > displayname: Pelletier, Michael J > > > > jdirlastfourssn: 8719 > > > > uid: cpellem > > > [snip data] > > > > What I need to do is: > > > > 1. Take the first entry and get the value of the display name and jdirlastfourssn > > attribute. > > > > 2. Check whether there is another record with the same display name attribute > > value. > > (There cud be multiple records) > > > > 3. If so then extract both record and write them into another file. > > > > 4. Delete these duplicate records from the parent file. > > > > 5. Do that for all records. > > > > I'm not clear whether you mean 200K lines or 200K records (which seem to be mostly > 6 lines each except for 'Pelletier, Michael J' which has an additional pair of > lines for a 'uid' value. But, even if it were 200K records at about 100 characters > each this would be 20Mb, which is well within the capacity of all but the smallest > computers these days. This problem is far easier with all the data in memory, so > I'll go that way, and if you find it's not working or is too slow we'll think again. > > OK, so let me rewrite your algorithm a little. > > - Read all records into memory > - While there are records left > - calculate a 'key' from the display name and serial number > - find all records in the data with a matching key > - if there was only one then print it to parent file, else print them to file 2 > - delete them from the list > > From the top: > > > "Read in all of the records" > > It looks like all the information starts with a line beginning with 'dn:'. If this > is wrong we'll have to change it. > > my @records; > > while (<DATA>) { > push @records, '' if /^dn:\s+/; > $records[-1] .= $_; > } > > > Before the loop, how about a subroutine which, given one of the multi-line records, > will return a key value containing the name and serial number. This picks out the > strings and concatenates them with a tab character in between, chosen because it > is unlikely to appear in the data itself. > > sub keyval { > my $rec = shift; > my ($name, $sn); > ($name) = $rec =~ m/^displayname:\s+(.+)/m; > ($sn) = $rec =~ m/^jdirlastfourssn:\s+(\d+)/m; > join "\t", $name, $sn; > } > > > "While there are records left" Here's the loop, including a couple of lines to > remove all non-blank entries from the beginning which will be left by the call > to 'delete' that you see in a moment. > > while (@records) { > until (exists $records[0]) { > shift @records; > } > : > } > > > "Calculate a 'key' from the display name and serial number" > > my $key = keyval($records[0]); > > "Find all records in the data with a matching key" This call to 'map' > returns a list of indices of all the records in the array which have > a matching key value. This is bound to include the index zero as the > first record matches itself. If there are more then the length of the > array will be more than 1. > > my $i = 0; > my @slice = map { > my @i = $i++; > defined $_ and keyval($_) eq $key ? @i : () > } @records; > > > The next two steps together: "If there was only one then print it to > parent file, else print them to file 2. Delete them from the list" > The 'delete' function usefully returns a list of all the records > it deleted, so we can just print the results of deleting the array > slice. > > if (@slice == 1) { > print PARENT delete @[EMAIL PROTECTED]; > } > else { > print FILE2 delete @[EMAIL PROTECTED]; > } > > > And you're done. Clearly you need to open filehandles DATA for read > and PARENT and FILE2 for write, but the program's there otherwise. > > Let us know if you need help assembling the kit. > > HTH, > > Rob > > > > > -- > To unsubscribe, e-mail: [EMAIL PROTECTED] > For additional commands, e-mail: [EMAIL PROTECTED] > __________________________________ Do you Yahoo!? SBC Yahoo! DSL - Now only $29.95 per month! http://sbc.yahoo.com -- To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]