Re: [U2] Regex searching UD files
I'm on UV and from the Unix command line I can do the following: cd /usr/ibm/uv uv port.status | grep j_banker It will list every line with my login. Jerry Banker -Original Message- From: u2-users-boun...@listserver.u2ug.org [mailto:u2-users-boun...@listserver.u2ug.org] On Behalf Of Dan McGrath Sent: Monday, March 29, 2010 4:13 PM To: U2 Users List Subject: Re: [U2] Regex searching UD files Not even close... We run in ECLTYPE P and don't have access to EVAL and the like. ^[Cc][Hh][Qq].*[5-8][0-9][0-9][0-9]x Unless you can explain how to duplicate the above search with ESEARCH? @Stuart, that would be ideal, but unfortunately I can't go dumping out multi-gigabyte files. :( -Original Message- From: u2-users-boun...@listserver.u2ug.org [mailto:u2-users-boun...@listserver.u2ug.org] On Behalf Of Wally Terhune Sent: Tuesday, 30 March 2010 5:22 AM To: U2 Users List Subject: Re: [U2] Regex searching UD files Does UniData ECL ESEARCH command not fill the bill already? Wally Terhune U2 Support Architect Rocket Software 4700 S. Syracuse Street, Suite 400 **Denver, CO 80237 **USA Tel: +1.720.475.8055 Email: wterh...@rs.com Web: www.rocketsoftware.com/u2 -Original Message- From: u2-users-boun...@listserver.u2ug.org [mailto:u2-users-boun...@listserver.u2ug.org] On Behalf Of Boydell, Stuart Sent: Monday, March 29, 2010 12:22 AM To: U2 Users List Subject: Re: [U2] Regex searching UD files I think this should be one for Rocket to implement a RegexSearch in U2. However, if it's size non-impactive, what I would normally do is copy the file to a temporary directory then run grep across that. Stuart Boydell -Original Message- From: u2-users-boun...@listserver.u2ug.org [mailto:u2-users-boun...@listserver.u2ug.org] On Behalf Of Dan McGrath Sent: Monday, 29 March 2010 17:08 To: U2 Users List Subject: [U2] Regex searching UD files I've written a small utility to be able to run egrep on a UD file (not UD directory), however its implementation is not ideal. Essentially, I select the file I'm searching, writing each record one at a time to a temp UNIX file and running egrep on it as follows !egrep -q -f MyTempRegexFile MyTempRecordFile ; echo $? Where MyTempRegexFile is a file containing the desired regex pattern stored earlier and MyTempRecordFile is a file name unique to each user. My problem with it is that I have to do a READ on each record, followed by a WRITE then have egrep read it in as well. That's a lot of seemingly unneeded disk IO if I could only stream the record to egrep without have to do a WRITE after the READ. Any ideas? I feel I'm missing something quite obvious. Regards, Dan ### The information transmitted in this message and attachments (if any) is intended only for the person or entity to which it is addressed. The message may contain confidential and/or privileged material. Any review, retransmission, dissemination or other use of or taking of any action in reliance upon this information by persons or entities other than the intended recipient is prohibited. If you received this in error, please contact the sender and delete the material from any computer. The intended recipient of this e-mail may only use, reproduce, disclose or distribute the information contained in this e-mail and any attached files with the permission of IMB. ### ___ U2-Users mailing list U2-Users@listserver.u2ug.org http://listserver.u2ug.org/mailman/listinfo/u2-users ___ U2-Users mailing list U2-Users@listserver.u2ug.org http://listserver.u2ug.org/mailman/listinfo/u2-users ___ U2-Users mailing list U2-Users@listserver.u2ug.org http://listserver.u2ug.org/mailman/listinfo/u2-users __ This email has been scanned by the MessageLabs Email Security System. For more information please visit http://www.messagelabs.com/email __ ### The information transmitted in this message and attachments (if any) is intended only for the person or entity to which it is addressed. The message may contain confidential and/or privileged material. Any review, retransmission, dissemination or other use of or taking of any action in reliance upon this information by persons or entities other than the intended recipient is prohibited. If you received this in error, please contact the sender and delete the material from any computer. The intended recipient of this e-mail may only use, reproduce, disclose or distribute the information contained
Re: [U2] Regex searching UD files
I'm not sure I understand how regular expressions could be used in ESEARCH, even in Type U. Would you please elaborate? Thanks, Kebbon From: wterh...@rocketsoftware.com To: u2-users@listserver.u2ug.org Date: Mon, 29 Mar 2010 18:22:27 + Subject: Re: [U2] Regex searching UD files Does UniData ECL ESEARCH command not fill the bill already? Wally Terhune U2 Support Architect Rocket Software 4700 S. Syracuse Street, Suite 400 **Denver, CO 80237 **USA Tel: +1.720.475.8055 Email: wterh...@rs.com Web: www.rocketsoftware.com/u2 -Original Message- From: u2-users-boun...@listserver.u2ug.org [mailto:u2-users-boun...@listserver.u2ug.org] On Behalf Of Boydell, Stuart Sent: Monday, March 29, 2010 12:22 AM To: U2 Users List Subject: Re: [U2] Regex searching UD files I think this should be one for Rocket to implement a RegexSearch in U2. However, if it's size non-impactive, what I would normally do is copy the file to a temporary directory then run grep across that. Stuart Boydell -Original Message- From: u2-users-boun...@listserver.u2ug.org [mailto:u2-users-boun...@listserver.u2ug.org] On Behalf Of Dan McGrath Sent: Monday, 29 March 2010 17:08 To: U2 Users List Subject: [U2] Regex searching UD files I've written a small utility to be able to run egrep on a UD file (not UD directory), however its implementation is not ideal. Essentially, I select the file I'm searching, writing each record one at a time to a temp UNIX file and running egrep on it as follows !egrep -q -f MyTempRegexFile MyTempRecordFile ; echo $? Where MyTempRegexFile is a file containing the desired regex pattern stored earlier and MyTempRecordFile is a file name unique to each user. My problem with it is that I have to do a READ on each record, followed by a WRITE then have egrep read it in as well. That's a lot of seemingly unneeded disk IO if I could only stream the record to egrep without have to do a WRITE after the READ. Any ideas? I feel I'm missing something quite obvious. Regards, Dan ### The information transmitted in this message and attachments (if any) is intended only for the person or entity to which it is addressed. The message may contain confidential and/or privileged material. Any review, retransmission, dissemination or other use of or taking of any action in reliance upon this information by persons or entities other than the intended recipient is prohibited. If you received this in error, please contact the sender and delete the material from any computer. The intended recipient of this e-mail may only use, reproduce, disclose or distribute the information contained in this e-mail and any attached files with the permission of IMB. ### ___ U2-Users mailing list U2-Users@listserver.u2ug.org http://listserver.u2ug.org/mailman/listinfo/u2-users ___ U2-Users mailing list U2-Users@listserver.u2ug.org http://listserver.u2ug.org/mailman/listinfo/u2-users ___ U2-Users mailing list U2-Users@listserver.u2ug.org http://listserver.u2ug.org/mailman/listinfo/u2-users ___ U2-Users mailing list U2-Users@listserver.u2ug.org http://listserver.u2ug.org/mailman/listinfo/u2-users
Re: [U2] Regex searching UD files
I found a way to do it with compiled dictionary elements (on UV).
1- I create a dictionnary element called EGREP.ARGS that always returns X:
0001: I
0002: X[1,1]
0003:
0004: EGREP.ARGS
0005: 1L
0006: M
2- I create a dictionnary element called EGREP which is defined like this:
0001: I
0002: SUBR(DICT.EGREP,@RECORD,@ID,@SENTENCE)
0003:
0004: EGREP
0005: 40L
0006: M
DICT.EGREP is defined like this:
SUBROUTINE DICT.EGREP(OUT,RECORD,ID,SENTENCE)
COM /WORKF/WORKF
IF FILEINFO(WORKF,0) = 0 THEN
OPEN , WORK.FILE TO WORKF ELSE STOP 201, WORK.FILE ;* A type 19 file
on Universe (pointer to a directory)
END
STATUS STAT FROM WORKF ELSE STOP
PATH = STAT27
*--- Extract the arguments for egrep from cmd line ---*
ARG.PFX = EGREP.ARGS NE
POS.EGREP = INDEX(SENTENCE,ARG.PFX,1) + LEN(ARG.PFX) + 1
EGREP.ARG = SENTENCE[POS.EGREP,LEN(SENTENCE)-POS.EGREP+1]
EGREP.ARG = EGREP.ARG[1,INDEX(EGREP.ARG,EGREP,1)-1]
EGREP.ARG = CONVERT('','',EGREP.ARG)
TMP.KEY = egrep.tmp ;* Using temporary key not ID to avoid problems (ei:
spaces, parentheses etc)
WRITE RECORD ON WORKF, TMP.KEY ;* Write rec to temp. file
CMD = cd :PATH: ; egrep :EGREP.ARG: :TMP.KEY
EXECUTE SH -c :QUOTE(CMD) CAPTURING OUT
DELETE WORKF, TMP.KEY;* Remove record
RETURN
This is how I do my search:
LIST MYFILE WITH EGREP.ARGS NE -in blahblah EGREP
(This will do the equivalent of egrep -in blahblahblah MYFILE if MYFILE was a
type 19 file)
This method writes only one record at a time and deletes it so while it is slow
(1 write + a sh executed per record), it will work with limited space available
and it uses the Unix egrep command. The arguments to the egrep command are
parsed from the sentence between EGREP.ARGS NE and EGREP.
- Original Message
From: Dan McGrath dmc...@imb.com.au
To: U2 Users List u2-users@listserver.u2ug.org
Sent: Mon, March 29, 2010 5:13:00 PM
Subject: Re: [U2] Regex searching UD files
Not even close... We run in ECLTYPE P and don't have access to EVAL and
the like.
^[Cc][Hh][Qq].*[5-8][0-9][0-9][0-9]x
Unless you can explain how to duplicate the above search with ESEARCH?
@Stuart, that would be ideal, but unfortunately I can't go dumping out
multi-gigabyte files. :(
-Original Message-
From: u2-users-boun...@listserver.u2ug.org
[mailto:u2-users-boun...@listserver.u2ug.org] On Behalf Of Wally Terhune
Sent: Tuesday, 30 March 2010 5:22 AM
To: U2 Users List
Subject: Re: [U2] Regex searching UD files
Does UniData ECL ESEARCH command not fill the bill already?
Wally Terhune
U2 Support Architect
Rocket Software
4700 S. Syracuse Street, Suite 400 **Denver, CO 80237 **USA
Tel: +1.720.475.8055
Email: wterh...@rs.com
Web: www.rocketsoftware.com/u2
-Original Message-
From: u2-users-boun...@listserver.u2ug.org
[mailto:u2-users-boun...@listserver.u2ug.org] On Behalf Of Boydell,
Stuart
Sent: Monday, March 29, 2010 12:22 AM
To: U2 Users List
Subject: Re: [U2] Regex searching UD files
I think this should be one for Rocket to implement a RegexSearch in U2.
However, if it's size non-impactive, what I would normally do is copy
the file to a temporary directory then run grep across that.
Stuart Boydell
-Original Message-
From: u2-users-boun...@listserver.u2ug.org
[mailto:u2-users-boun...@listserver.u2ug.org] On Behalf Of Dan McGrath
Sent: Monday, 29 March 2010 17:08
To: U2 Users List
Subject: [U2] Regex searching UD files
I've written a small utility to be able to run egrep on a UD file (not
UD directory), however its implementation is not ideal.
Essentially, I select the file I'm searching, writing each record one at
a time to a temp UNIX file and running egrep on it as follows
!egrep -q -f MyTempRegexFile MyTempRecordFile ; echo $?
Where MyTempRegexFile is a file containing the desired regex pattern
stored earlier and MyTempRecordFile is a file name unique to each user.
My problem with it is that I have to do a READ on each record, followed
by a WRITE then have egrep read it in as well. That's a lot of seemingly
unneeded disk IO if I could only stream the record to egrep without have
to do a WRITE after the READ.
Any ideas? I feel I'm missing something quite obvious.
Regards,
Dan
###
The information transmitted in this message and attachments (if any) is
intended only
for the person or entity to which it is addressed. The message may
contain confidential
and/or privileged material. Any review, retransmission, dissemination
or other use of
or taking of any action in reliance upon this information by persons or
entities other
than the intended recipient is prohibited. If you received this in
error, please
contact the sender and delete the material from any computer.
The intended recipient of this e-mail may only use, reproduce, disclose
or distribute
the information contained in this e-mail and any attached files with the
permission of IMB
[U2] Regex searching UD files
I've written a small utility to be able to run egrep on a UD file (not UD directory), however its implementation is not ideal. Essentially, I select the file I'm searching, writing each record one at a time to a temp UNIX file and running egrep on it as follows !egrep -q -f MyTempRegexFile MyTempRecordFile ; echo $? Where MyTempRegexFile is a file containing the desired regex pattern stored earlier and MyTempRecordFile is a file name unique to each user. My problem with it is that I have to do a READ on each record, followed by a WRITE then have egrep read it in as well. That's a lot of seemingly unneeded disk IO if I could only stream the record to egrep without have to do a WRITE after the READ. Any ideas? I feel I'm missing something quite obvious. Regards, Dan ### The information transmitted in this message and attachments (if any) is intended only for the person or entity to which it is addressed. The message may contain confidential and/or privileged material. Any review, retransmission, dissemination or other use of or taking of any action in reliance upon this information by persons or entities other than the intended recipient is prohibited. If you received this in error, please contact the sender and delete the material from any computer. The intended recipient of this e-mail may only use, reproduce, disclose or distribute the information contained in this e-mail and any attached files with the permission of IMB. ### ___ U2-Users mailing list U2-Users@listserver.u2ug.org http://listserver.u2ug.org/mailman/listinfo/u2-users
Re: [U2] Regex searching UD files
I think this should be one for Rocket to implement a RegexSearch in U2. However, if it's size non-impactive, what I would normally do is copy the file to a temporary directory then run grep across that. Stuart Boydell -Original Message- From: u2-users-boun...@listserver.u2ug.org [mailto:u2-users-boun...@listserver.u2ug.org] On Behalf Of Dan McGrath Sent: Monday, 29 March 2010 17:08 To: U2 Users List Subject: [U2] Regex searching UD files I've written a small utility to be able to run egrep on a UD file (not UD directory), however its implementation is not ideal. Essentially, I select the file I'm searching, writing each record one at a time to a temp UNIX file and running egrep on it as follows !egrep -q -f MyTempRegexFile MyTempRecordFile ; echo $? Where MyTempRegexFile is a file containing the desired regex pattern stored earlier and MyTempRecordFile is a file name unique to each user. My problem with it is that I have to do a READ on each record, followed by a WRITE then have egrep read it in as well. That's a lot of seemingly unneeded disk IO if I could only stream the record to egrep without have to do a WRITE after the READ. Any ideas? I feel I'm missing something quite obvious. Regards, Dan ### The information transmitted in this message and attachments (if any) is intended only for the person or entity to which it is addressed. The message may contain confidential and/or privileged material. Any review, retransmission, dissemination or other use of or taking of any action in reliance upon this information by persons or entities other than the intended recipient is prohibited. If you received this in error, please contact the sender and delete the material from any computer. The intended recipient of this e-mail may only use, reproduce, disclose or distribute the information contained in this e-mail and any attached files with the permission of IMB. ### ___ U2-Users mailing list U2-Users@listserver.u2ug.org http://listserver.u2ug.org/mailman/listinfo/u2-users ___ U2-Users mailing list U2-Users@listserver.u2ug.org http://listserver.u2ug.org/mailman/listinfo/u2-users
Re: [U2] Regex searching UD files
GCI ! -Original Message- From: u2-users-boun...@listserver.u2ug.org [mailto:u2-users-boun...@listserver.u2ug.org] On Behalf Of Dan McGrath Sent: 29 March 2010 07:08 To: U2 Users List Subject: [U2] Regex searching UD files I've written a small utility to be able to run egrep on a UD file (not UD directory), however its implementation is not ideal. Essentially, I select the file I'm searching, writing each record one at a time to a temp UNIX file and running egrep on it as follows !egrep -q -f MyTempRegexFile MyTempRecordFile ; echo $? Where MyTempRegexFile is a file containing the desired regex pattern stored earlier and MyTempRecordFile is a file name unique to each user. My problem with it is that I have to do a READ on each record, followed by a WRITE then have egrep read it in as well. That's a lot of seemingly unneeded disk IO if I could only stream the record to egrep without have to do a WRITE after the READ. Any ideas? I feel I'm missing something quite obvious. Regards, Dan ### The information transmitted in this message and attachments (if any) is intended only for the person or entity to which it is addressed. The message may contain confidential and/or privileged material. Any review, retransmission, dissemination or other use of or taking of any action in reliance upon this information by persons or entities other than the intended recipient is prohibited. If you received this in error, please contact the sender and delete the material from any computer. The intended recipient of this e-mail may only use, reproduce, disclose or distribute the information contained in this e-mail and any attached files with the permission of IMB. ### ___ U2-Users mailing list U2-Users@listserver.u2ug.org http://listserver.u2ug.org/mailman/listinfo/u2-users ___ U2-Users mailing list U2-Users@listserver.u2ug.org http://listserver.u2ug.org/mailman/listinfo/u2-users
Re: [U2] Regex searching UD files
Dan you can't do it because the records in Universe are not files at all, not even logical files. They are just possible frame-discontinous streams of bytes that *Universe* interprets in run-time as a connected record. Nothing else is going to be able to do that, without re-writing the frame(block)-walker directly in Unix. Alternatively you could write egrep in Universe yourself which would probably be easier! But I wouldn't read and write after each record, I would read and write the entire file at *once*. And then egrep. I sure that would save on thrashing at least. If you're processing at least 10 percent of the file, then you might see that writing the whole file, then switching to Unix for the rest will still be faster than breaking after each write. Will Johnson In a message dated 3/28/2010 11:22:25 P.M. Pacific Daylight Time, stuart.boyd...@spotless.com.au writes: I think this should be one for Rocket to implement a RegexSearch in U2. However, if it's size non-impactive, what I would normally do is copy the file to a temporary directory then run grep across that. Stuart Boydell -Original Message- From: u2-users-boun...@listserver.u2ug.org [mailto:u2-users-boun...@listserver.u2ug.org] On Behalf Of Dan McGrath Sent: Monday, 29 March 2010 17:08 To: U2 Users List Subject: [U2] Regex searching UD files I've written a small utility to be able to run egrep on a UD file (not UD directory), however its implementation is not ideal. Essentially, I select the file I'm searching, writing each record one at a time to a temp UNIX file and running egrep on it as follows !egrep -q -f MyTempRegexFile MyTempRecordFile ; echo $? Where MyTempRegexFile is a file containing the desired regex pattern stored earlier and MyTempRecordFile is a file name unique to each user. My problem with it is that I have to do a READ on each record, followed by a WRITE then have egrep read it in as well. That's a lot of seemingly unneeded disk IO if I could only stream the record to egrep without have to do a WRITE after the READ. Any ideas? I feel I'm missing something quite obvious. Regards, Dan ### The information transmitted in this message and attachments (if any) is intended only for the person or entity to which it is addressed. The message may contain confidential and/or privileged material. Any review, retransmission, dissemination or other use of or taking of any action in reliance upon this information by persons or entities other than the intended recipient is prohibited. If you received this in error, please contact the sender and delete the material from any computer. The intended recipient of this e-mail may only use, reproduce, disclose or distribute the information contained in this e-mail and any attached files with the permission of IMB. ### ___ U2-Users mailing list U2-Users@listserver.u2ug.org http://listserver.u2ug.org/mailman/listinfo/u2-users ___ U2-Users mailing list U2-Users@listserver.u2ug.org http://listserver.u2ug.org/mailman/listinfo/u2-users ___ U2-Users mailing list U2-Users@listserver.u2ug.org http://listserver.u2ug.org/mailman/listinfo/u2-users
Re: [U2] Regex searching UD files
Does UniData ECL ESEARCH command not fill the bill already? Wally Terhune U2 Support Architect Rocket Software 4700 S. Syracuse Street, Suite 400 **Denver, CO 80237 **USA Tel: +1.720.475.8055 Email: wterh...@rs.com Web: www.rocketsoftware.com/u2 -Original Message- From: u2-users-boun...@listserver.u2ug.org [mailto:u2-users-boun...@listserver.u2ug.org] On Behalf Of Boydell, Stuart Sent: Monday, March 29, 2010 12:22 AM To: U2 Users List Subject: Re: [U2] Regex searching UD files I think this should be one for Rocket to implement a RegexSearch in U2. However, if it's size non-impactive, what I would normally do is copy the file to a temporary directory then run grep across that. Stuart Boydell -Original Message- From: u2-users-boun...@listserver.u2ug.org [mailto:u2-users-boun...@listserver.u2ug.org] On Behalf Of Dan McGrath Sent: Monday, 29 March 2010 17:08 To: U2 Users List Subject: [U2] Regex searching UD files I've written a small utility to be able to run egrep on a UD file (not UD directory), however its implementation is not ideal. Essentially, I select the file I'm searching, writing each record one at a time to a temp UNIX file and running egrep on it as follows !egrep -q -f MyTempRegexFile MyTempRecordFile ; echo $? Where MyTempRegexFile is a file containing the desired regex pattern stored earlier and MyTempRecordFile is a file name unique to each user. My problem with it is that I have to do a READ on each record, followed by a WRITE then have egrep read it in as well. That's a lot of seemingly unneeded disk IO if I could only stream the record to egrep without have to do a WRITE after the READ. Any ideas? I feel I'm missing something quite obvious. Regards, Dan ### The information transmitted in this message and attachments (if any) is intended only for the person or entity to which it is addressed. The message may contain confidential and/or privileged material. Any review, retransmission, dissemination or other use of or taking of any action in reliance upon this information by persons or entities other than the intended recipient is prohibited. If you received this in error, please contact the sender and delete the material from any computer. The intended recipient of this e-mail may only use, reproduce, disclose or distribute the information contained in this e-mail and any attached files with the permission of IMB. ### ___ U2-Users mailing list U2-Users@listserver.u2ug.org http://listserver.u2ug.org/mailman/listinfo/u2-users ___ U2-Users mailing list U2-Users@listserver.u2ug.org http://listserver.u2ug.org/mailman/listinfo/u2-users ___ U2-Users mailing list U2-Users@listserver.u2ug.org http://listserver.u2ug.org/mailman/listinfo/u2-users
Re: [U2] Regex searching UD files
Not even close... We run in ECLTYPE P and don't have access to EVAL and the like. ^[Cc][Hh][Qq].*[5-8][0-9][0-9][0-9]x Unless you can explain how to duplicate the above search with ESEARCH? @Stuart, that would be ideal, but unfortunately I can't go dumping out multi-gigabyte files. :( -Original Message- From: u2-users-boun...@listserver.u2ug.org [mailto:u2-users-boun...@listserver.u2ug.org] On Behalf Of Wally Terhune Sent: Tuesday, 30 March 2010 5:22 AM To: U2 Users List Subject: Re: [U2] Regex searching UD files Does UniData ECL ESEARCH command not fill the bill already? Wally Terhune U2 Support Architect Rocket Software 4700 S. Syracuse Street, Suite 400 **Denver, CO 80237 **USA Tel: +1.720.475.8055 Email: wterh...@rs.com Web: www.rocketsoftware.com/u2 -Original Message- From: u2-users-boun...@listserver.u2ug.org [mailto:u2-users-boun...@listserver.u2ug.org] On Behalf Of Boydell, Stuart Sent: Monday, March 29, 2010 12:22 AM To: U2 Users List Subject: Re: [U2] Regex searching UD files I think this should be one for Rocket to implement a RegexSearch in U2. However, if it's size non-impactive, what I would normally do is copy the file to a temporary directory then run grep across that. Stuart Boydell -Original Message- From: u2-users-boun...@listserver.u2ug.org [mailto:u2-users-boun...@listserver.u2ug.org] On Behalf Of Dan McGrath Sent: Monday, 29 March 2010 17:08 To: U2 Users List Subject: [U2] Regex searching UD files I've written a small utility to be able to run egrep on a UD file (not UD directory), however its implementation is not ideal. Essentially, I select the file I'm searching, writing each record one at a time to a temp UNIX file and running egrep on it as follows !egrep -q -f MyTempRegexFile MyTempRecordFile ; echo $? Where MyTempRegexFile is a file containing the desired regex pattern stored earlier and MyTempRecordFile is a file name unique to each user. My problem with it is that I have to do a READ on each record, followed by a WRITE then have egrep read it in as well. That's a lot of seemingly unneeded disk IO if I could only stream the record to egrep without have to do a WRITE after the READ. Any ideas? I feel I'm missing something quite obvious. Regards, Dan ### The information transmitted in this message and attachments (if any) is intended only for the person or entity to which it is addressed. The message may contain confidential and/or privileged material. Any review, retransmission, dissemination or other use of or taking of any action in reliance upon this information by persons or entities other than the intended recipient is prohibited. If you received this in error, please contact the sender and delete the material from any computer. The intended recipient of this e-mail may only use, reproduce, disclose or distribute the information contained in this e-mail and any attached files with the permission of IMB. ### ___ U2-Users mailing list U2-Users@listserver.u2ug.org http://listserver.u2ug.org/mailman/listinfo/u2-users ___ U2-Users mailing list U2-Users@listserver.u2ug.org http://listserver.u2ug.org/mailman/listinfo/u2-users ___ U2-Users mailing list U2-Users@listserver.u2ug.org http://listserver.u2ug.org/mailman/listinfo/u2-users __ This email has been scanned by the MessageLabs Email Security System. For more information please visit http://www.messagelabs.com/email __ ### The information transmitted in this message and attachments (if any) is intended only for the person or entity to which it is addressed. The message may contain confidential and/or privileged material. Any review, retransmission, dissemination or other use of or taking of any action in reliance upon this information by persons or entities other than the intended recipient is prohibited. If you received this in error, please contact the sender and delete the material from any computer. The intended recipient of this e-mail may only use, reproduce, disclose or distribute the information contained in this e-mail and any attached files with the permission of IMB. ### ___ U2-Users mailing list U2-Users@listserver.u2ug.org http://listserver.u2ug.org/mailman/listinfo/u2-users
Re: [U2] Regex searching UD files
Thanks for the reply Will, but I do understand how U2 files work. That's why I'm using UniBasic to read the record then write it out to UNIX. If the files we search weren't so big I would just dump a copy to a dir than search it, but that isn't practical in most cases for us. Maybe I can improve it to write the records in blocks of 'x', search them. That would at least save some of the processing cost of jumping around, but it doesn't solve the Disk I/O bottleneck. -Original Message- From: u2-users-boun...@listserver.u2ug.org [mailto:u2-users-boun...@listserver.u2ug.org] On Behalf Of fft2...@aol.com Sent: Tuesday, 30 March 2010 5:14 AM To: u2-users@listserver.u2ug.org Subject: Re: [U2] Regex searching UD files Dan you can't do it because the records in Universe are not files at all, not even logical files. They are just possible frame-discontinous streams of bytes that *Universe* interprets in run-time as a connected record. Nothing else is going to be able to do that, without re-writing the frame(block)-walker directly in Unix. Alternatively you could write egrep in Universe yourself which would probably be easier! But I wouldn't read and write after each record, I would read and write the entire file at *once*. And then egrep. I sure that would save on thrashing at least. If you're processing at least 10 percent of the file, then you might see that writing the whole file, then switching to Unix for the rest will still be faster than breaking after each write. Will Johnson In a message dated 3/28/2010 11:22:25 P.M. Pacific Daylight Time, stuart.boyd...@spotless.com.au writes: I think this should be one for Rocket to implement a RegexSearch in U2. However, if it's size non-impactive, what I would normally do is copy the file to a temporary directory then run grep across that. Stuart Boydell -Original Message- From: u2-users-boun...@listserver.u2ug.org [mailto:u2-users-boun...@listserver.u2ug.org] On Behalf Of Dan McGrath Sent: Monday, 29 March 2010 17:08 To: U2 Users List Subject: [U2] Regex searching UD files I've written a small utility to be able to run egrep on a UD file (not UD directory), however its implementation is not ideal. Essentially, I select the file I'm searching, writing each record one at a time to a temp UNIX file and running egrep on it as follows !egrep -q -f MyTempRegexFile MyTempRecordFile ; echo $? Where MyTempRegexFile is a file containing the desired regex pattern stored earlier and MyTempRecordFile is a file name unique to each user. My problem with it is that I have to do a READ on each record, followed by a WRITE then have egrep read it in as well. That's a lot of seemingly unneeded disk IO if I could only stream the record to egrep without have to do a WRITE after the READ. Any ideas? I feel I'm missing something quite obvious. Regards, Dan ### The information transmitted in this message and attachments (if any) is intended only for the person or entity to which it is addressed. The message may contain confidential and/or privileged material. Any review, retransmission, dissemination or other use of or taking of any action in reliance upon this information by persons or entities other than the intended recipient is prohibited. If you received this in error, please contact the sender and delete the material from any computer. The intended recipient of this e-mail may only use, reproduce, disclose or distribute the information contained in this e-mail and any attached files with the permission of IMB. ### ___ U2-Users mailing list U2-Users@listserver.u2ug.org http://listserver.u2ug.org/mailman/listinfo/u2-users ___ U2-Users mailing list U2-Users@listserver.u2ug.org http://listserver.u2ug.org/mailman/listinfo/u2-users ___ U2-Users mailing list U2-Users@listserver.u2ug.org http://listserver.u2ug.org/mailman/listinfo/u2-users __ This email has been scanned by the MessageLabs Email Security System. For more information please visit http://www.messagelabs.com/email __ ### The information transmitted in this message and attachments (if any) is intended only for the person or entity to which it is addressed. The message may contain confidential and/or privileged material. Any review, retransmission, dissemination or other use of or taking of any action in reliance upon this information by persons or entities other than
Re: [U2] Regex searching UD files
I don't know what the UD hashed files look like under the hood, and I haven't tried it but with a UV hashed files it might just be possible to pass the whole thing through sed or awk first to convert system marks to CR and parse out the ID and searched for text using grep. Just a thought. Good luck, Stuart Boydell -Original Message- From: u2-users-boun...@listserver.u2ug.org [mailto:u2-users-boun...@listserver.u2ug.org] On Behalf Of Dan McGrath Sent: Tuesday, 30 March 2010 08:50 To: U2 Users List Subject: Re: [U2] Regex searching UD files Thanks for the reply Will, but I do understand how U2 files work. That's why I'm using UniBasic to read the record then write it out to UNIX. If the files we search weren't so big I would just dump a copy to a dir than search it, but that isn't practical in most cases for us. Maybe I can improve it to write the records in blocks of 'x', search them. That would at least save some of the processing cost of jumping around, but it doesn't solve the Disk I/O bottleneck. -Original Message- From: u2-users-boun...@listserver.u2ug.org [mailto:u2-users-boun...@listserver.u2ug.org] On Behalf Of fft2...@aol.com Sent: Tuesday, 30 March 2010 5:14 AM To: u2-users@listserver.u2ug.org Subject: Re: [U2] Regex searching UD files Dan you can't do it because the records in Universe are not files at all, not even logical files. They are just possible frame-discontinous streams of bytes that *Universe* interprets in run-time as a connected record. Nothing else is going to be able to do that, without re-writing the frame(block)-walker directly in Unix. Alternatively you could write egrep in Universe yourself which would probably be easier! But I wouldn't read and write after each record, I would read and write the entire file at *once*. And then egrep. I sure that would save on thrashing at least. If you're processing at least 10 percent of the file, then you might see that writing the whole file, then switching to Unix for the rest will still be faster than breaking after each write. Will Johnson In a message dated 3/28/2010 11:22:25 P.M. Pacific Daylight Time, stuart.boyd...@spotless.com.au writes: I think this should be one for Rocket to implement a RegexSearch in U2. However, if it's size non-impactive, what I would normally do is copy the file to a temporary directory then run grep across that. Stuart Boydell -Original Message- From: u2-users-boun...@listserver.u2ug.org [mailto:u2-users-boun...@listserver.u2ug.org] On Behalf Of Dan McGrath Sent: Monday, 29 March 2010 17:08 To: U2 Users List Subject: [U2] Regex searching UD files I've written a small utility to be able to run egrep on a UD file (not UD directory), however its implementation is not ideal. Essentially, I select the file I'm searching, writing each record one at a time to a temp UNIX file and running egrep on it as follows !egrep -q -f MyTempRegexFile MyTempRecordFile ; echo $? Where MyTempRegexFile is a file containing the desired regex pattern stored earlier and MyTempRecordFile is a file name unique to each user. My problem with it is that I have to do a READ on each record, followed by a WRITE then have egrep read it in as well. That's a lot of seemingly unneeded disk IO if I could only stream the record to egrep without have to do a WRITE after the READ. Any ideas? I feel I'm missing something quite obvious. Regards, Dan ### The information transmitted in this message and attachments (if any) is intended only for the person or entity to which it is addressed. The message may contain confidential and/or privileged material. Any review, retransmission, dissemination or other use of or taking of any action in reliance upon this information by persons or entities other than the intended recipient is prohibited. If you received this in error, please contact the sender and delete the material from any computer. The intended recipient of this e-mail may only use, reproduce, disclose or distribute the information contained in this e-mail and any attached files with the permission of IMB. ### ___ U2-Users mailing list U2-Users@listserver.u2ug.org http://listserver.u2ug.org/mailman/listinfo/u2-users ___ U2-Users mailing list U2-Users@listserver.u2ug.org http://listserver.u2ug.org/mailman/listinfo/u2-users ___ U2-Users mailing list U2-Users@listserver.u2ug.org http://listserver.u2ug.org/mailman/listinfo/u2-users __ This email has been scanned by the MessageLabs Email Security System. For more information please visit
