date:20120703

One of our Java programmers lives and breaths Mac's.  We develop on the Mac
and run on the Mac.  We have for years many years been running on the Mac.
 There are many things that have to be tuned for the Mac on Eclipse.
 Whether it is the the Java libraries that are slightly different or SWT,
there is a lot of work that has to be Mac specific.

Regards,
Doug
www.u2logic.com/tools.html
___
U2-Users mailing list
U2-Users@listserver.u2ug.org
http://listserver.u2ug.org/mailman/listinfo/u2-users

Re: [U2] UniVerse LIST statement question [not-secure]

2012-07-03 Thread Charles Stevenson


What Brian said, except don't replace LIST!
Rather, write your subroutine to be called from an I-descriptors:

 MARKH
0001   FUNCTION MARKH( INARG )
0002   COMMON /MARKH/ START.IDT, END.IDT, PREV.ID, VLIST
0003 *
0004 * Assumes common has been initialized properly before this function
0005 * is called from i-descriptors
0006 *
0007   IF @ID = PREV.ID ELSE
0008  IDTS = @RECORD< 10 >
0009  VLIST = ""
0010  VMC = 0
0011  VMAX = DCOUNT( IDTS, @VM )
0012  FOR V = 1 TO VMAX
0013 IDT = IDTS<1,V>
0014 IF IDT >= START.IDT THEN IF IDT <= END.IDT THEN 
VLIST<-1> = V

0015  NEXT V
0016   END
0017 *
0018   OUTARG = ""
0019   MORE = LEN( VLIST ); * true/false
0020   LOOP WHILE MORE
0021  REMOVE V FROM VLIST SETTING MORE
0022  OUTARG<1,-1> = INARG<1, V >
0023   REPEAT
0024   SETREM 0 ON VLIST
0025 *
0026   RETURN( OUTARG )
0027END

An example using it is below.

It's more flexible and maintainable in the long run than tossing 
RETRIEVE altogether.
(Actually, I do think it can be done with the mv-handling functions but 
an I-descriptor subroutine will be more maintainable.)


Your sub should limits the output to the values you want. If every value 
is out of range, your subr returns null.  It can be called for various 
associated fields, and the 1st time called for a given @ID it figures 
out the value range, and saves to labeled common what vals you want for 
that record.  Subsequent calls to the sub, as long as @ID hasn't 
changed, gets the already calculated value range from common, and 
applies it to the field you pass it.


The only trick is feeding your subroutine the date range you won't know 
until runtime.I don't know a good way to feed an I-descriptor 
command line parameters such as ">= 6/1/2012 &<= 6/30/2012"
You'll need to set them in a control record or, better, in @USER1, 
@USER2, or in your labeled common before the LIST is executed.


Use the I-descriptors that call your subroutine as output criteria and 
not WITH or WHEN selection criteria.  The guts of the subr will do the 
selection limits.


That may not be altogether clear.  Here's an example.
1st the relevant dict entries then an example:


DICT RTAG09:45:22am  03 Jul 2012  Page1

Field. Type & Field Conversion.. Column. Output 
Depth &
Name.. Field. Definition... Code Heading Format 
Assoc..

   Number

LOCD3 12LS
EVENT.CODE D8Event 4L M EVENT
.ASSOC
 Code
EVENT.DT   D   10   D2/  Event 8R M EVENT
.ASSOC
 Date
PRINTERD   13 15LS
MARKH.CODE I  SUBR(  Event 4L M MARKH
'MARKH',  .ASSOC
  EVENT.CODE )
 Code
MARKH.DT   I  SUBR( D2/  Event 8R M MARKH
'MARKH',  .ASSOC
  EVENT.DT )
 Date
MARKH.ASSOCPH MARKH.LINO
  MARKH.CODE
  MARKH.REF
  MARKH.DT
  MARKH.TM
  MARKH.WHO
  MARKH.DTM

7 records listed.
>

>RUN CDS.BP MARKH.INIT 4/1 4/30
>LIST RTAG LOC PRINTER   EVENT.CODE EVENT.DT   MARKH.CODE MARKH.DT


LIST RTAG LOC PRINTER EVENT.CODE EVENT.DT MARKH.CODE MARKH.DT 
09:46:05am  03 Jul 2012  PAGE1

Return.. LOC. PRINTER Event Event... Event Event...
Tag.  Code. Date Code. Date

 5137176 TPAWHSE-IN   ZEBRATPAPRINT 02/08/12
  NEW   11/04/11
10009741 EROCWHSE-OUT PI04/26/12 PI 04/26/12
  NEW   04/26/12 NEW 04/26/12
 5135103 PROCWHSE-DEF REQ   12/30/11
  REQ   12/30/11
  ASGN  12/30/11
  FREE  11/03/11
 5134267 EROCDPO-DIN  ZEBRATPAI 03/01/12
  PRINT 02/08/12
  REQ   12/09/11
  ASGN  12/09/11
  FREE  11/03/11
10010396 DFWWHSE  PI05/22/12
  NEW   05/22/12
 5119929  SI04/10/12 SI 04/10/12
  NEW   09/20/11
10004562 CROCWHSE-OUT ZEBRAEROC   PRINT 04/04/12 PRINT 04/04/12
  PRINT 01/31/12
  PRINT 01/24/12

[U2] Exiting UniData

2012-07-03 Thread Israel, John R.

Going back to my post about runaway processes:
According to our UniData manuals, the official commands to terminate a UniData 
session are BYE, LO and QUIT.

I have since found that LO and OFF (not sure where this came from) simply run 
the QUIT command.

QUIT runs an SB command called SH.OFF (which we obviously do not have the 
source code to review).

BYE still runs the verb BYE, so nothing different there.


We are using Avanté which obviously is built with SB.

My question is this: do ALL SB sites change the value of QUIT to run SH.OFF, or 
was this something that Epicor does?

If anyone is using SB but NOT using an Epicor product, what does your QUIT 
command do?


Thanks

John
___
U2-Users mailing list
U2-Users@listserver.u2ug.org
http://listserver.u2ug.org/mailman/listinfo/u2-users

Re: [U2] Exiting UniData

2012-07-03 Thread Dave Davis

Runs the globally cataloged program _SB_SH.OFF

-Original Message-
From: u2-users-boun...@listserver.u2ug.org 
[mailto:u2-users-boun...@listserver.u2ug.org] On Behalf Of Israel, John R.
Sent: Tuesday, July 03, 2012 10:31 AM
To: 'U2 Users List'
Subject: [U2] Exiting UniData

Going back to my post about runaway processes:
According to our UniData manuals, the official commands to terminate a UniData 
session are BYE, LO and QUIT.

I have since found that LO and OFF (not sure where this came from) simply run 
the QUIT command.

QUIT runs an SB command called SH.OFF (which we obviously do not have the 
source code to review).

BYE still runs the verb BYE, so nothing different there.

We are using Avanté which obviously is built with SB.

My question is this: do ALL SB sites change the value of QUIT to run SH.OFF, or 
was this something that Epicor does?

If anyone is using SB but NOT using an Epicor product, what does your QUIT 
command do?

Thanks

John
___
U2-Users mailing list
U2-Users@listserver.u2ug.org
http://listserver.u2ug.org/mailman/listinfo/u2-users

Dave Davis
Team Lead, R&D

P: 614-875-4910 x108
F: 614-875-4088
E: dda...@harriscomputer.com
[http://www.harriscomputer.com/images/signatures/HarrisSchools.gif]

[http://www.harriscomputer.com/images/signatures/DivisionofHarris.gif]
6110 Enterprise Parkway
Grove City, OH
43123
www.harris-schoolsolutions.com

This message is intended exclusively for the individual or entity to which it 
is addressed. This communication may contain information that is proprietary, 
privileged or confidential or otherwise legally exempt from disclosure. If you 
are not the named addressee, you are not authorized to read, print, retain, 
copy or disseminate this message or any part of it. If you have received this 
message in error, please notify the sender immediately by e-mail and delete all 
copies of the message.

___
U2-Users mailing list
U2-Users@listserver.u2ug.org
http://listserver.u2ug.org/mailman/listinfo/u2-users

Re: [U2] RESIZE - dynamic files


Doug,

When I do the math I come up with a different # (see below):

File name ..   TEST_FILE
Pathname ...   TEST_FILE
File type ..   DYNAMIC
File style and revision    32BIT Revision 12
Hashing Algorithm ..   GENERAL
No. of groups (modulus)    82850 current ( minimum 24, 104 empty,
26225 overflowed, 1441 badly )
Number of records ..   1157122
Large record size ..   2036 bytes
Number of large records    576
Group size .   4096 bytes
Load factors ...   80% (split), 50% (merge) and 80% (actual)
Total size .   449605632 bytes
Total size of record data ..   258687736 bytes
Total size of record IDs ...   19283300 bytes
Unused space ...   171626404 bytes
Total space for records    449597440 bytes

-- 
258,687,736 bytes - Total size of record data
19,283,300 bytes - Total size of record IDs
===
277,971,036 bytes (record + id's)

277,971,036 / 4,084 = 68,063 bytes (minimum modulus)
-- 

68,063 is less than the current modulus of 82,850. Something with this formula 
doesn't seem right because if I use that formula I always calculate a 
minimum modulus of less than the current modulus.

Thanks,

Chris



> Date: Mon, 2 Jul 2012 16:08:16 -0600
> From: dave...@gmail.com
> To: u2-users@listserver.u2ug.org
> Subject: Re: [U2] RESIZE - dynamic files
> 
> Hi Chris:
> 
> You cannot get away with not resizing dynamic files in my experience.  The
> files do not split and merge like we are led to believe.  The separator is
> not used on dynamic files.  Your Universe file is badly sized.  The math
> below will get you reasonably file size.
> 
> Let's do the math:
> 
> 258687736 (Record Size)
> 192283300 (Key Size)
> 
> 450,971,036 (Data and Key Size)
> 
> 4096 (Group Size)
> - 12   (32 Bit Overhead)
> 
> 4084 Usable Space
> 
> 450971036/4084 = Minimum Modulo 110424 (Prime is 110431)
> 
> 
> [ad]
> I hate doing this math all of the time.  I have a reasonably priced resize
> program called XLr8Resizer for $99.00 to do this for me.
> [/ad]
> 
> Regards,
> Doug
> www.u2logic.com/tools.html
> "XLr8Resizer for the rest of us"
> ___
> U2-Users mailing list
> U2-Users@listserver.u2ug.org
> http://listserver.u2ug.org/mailman/listinfo/u2-users
  
___
U2-Users mailing list
U2-Users@listserver.u2ug.org
http://listserver.u2ug.org/mailman/listinfo/u2-users

Re: [U2] UniVerse LIST statement question [not-secure]

2012-07-03 Thread Ed Clark

There are 3 ways I can think of offhand to tell the subroutine what date range 
to use. I like #3 the best:

1: use common variables, which is what the example function MARKH does. The 
downside is that you have to assign the common variables before you run the 
query.

2: You could use @SENTENCE in the function and parse out the date range. That 
could be easy or hard depending on how many people write queries and what odd 
contortions of syntax they use. But both #1 and #2 could be simplified by 
writing a wrapper program. An example command line would look like:
  QLAUNCH "06/01/2012" "06/30/2012" ~ LIST MYFILE INRANGE
QLAUNCH parses arguments before the '~` delimiter and puts them into a common. 
The INRANGE attribute calls the function MARKH which uses the common.

3: use the function directly from the command line with EVAL:
  LIST MYFILE EVAL "SUBR('MARKH','06/01/2012','06/30/2012')"  AS "MYCOL" 
COL.HDG "WHATEVER" FMT "11L"
It's wordier, but you have very fine-grained control over what comes out on the 
report. COL.HDG, and FMT are optional and AS is optional in this case. You 
would us it if you wanted to sort or break on the EVAL column.



On Jul 3, 2012, at 9:55 AM, Charles Stevenson wrote:

> What Brian said, except don't replace LIST!
> Rather, write your subroutine to be called from an I-descriptors:
> 
> MARKH
> 0001   FUNCTION MARKH( INARG )
> 0002   COMMON /MARKH/ START.IDT, END.IDT, PREV.ID, VLIST
> 0003 *
> 0004 * Assumes common has been initialized properly before this function
> 0005 * is called from i-descriptors
> 0006 *
> 0007   IF @ID = PREV.ID ELSE
> 0008  IDTS = @RECORD< 10 >
> 0009  VLIST = ""
> 0010  VMC = 0
> 0011  VMAX = DCOUNT( IDTS, @VM )
> 0012  FOR V = 1 TO VMAX
> 0013 IDT = IDTS<1,V>
> 0014 IF IDT >= START.IDT THEN IF IDT <= END.IDT THEN VLIST<-1> = V
> 0015  NEXT V
> 0016   END
> 0017 *
> 0018   OUTARG = ""
> 0019   MORE = LEN( VLIST ); * true/false
> 0020   LOOP WHILE MORE
> 0021  REMOVE V FROM VLIST SETTING MORE
> 0022  OUTARG<1,-1> = INARG<1, V >
> 0023   REPEAT
> 0024   SETREM 0 ON VLIST
> 0025 *
> 0026   RETURN( OUTARG )
> 0027END
> 
> An example using it is below.
> 
> It's more flexible and maintainable in the long run than tossing RETRIEVE 
> altogether.
> (Actually, I do think it can be done with the mv-handling functions but an 
> I-descriptor subroutine will be more maintainable.)
> 
> Your sub should limits the output to the values you want. If every value is 
> out of range, your subr returns null.  It can be called for various 
> associated fields, and the 1st time called for a given @ID it figures out the 
> value range, and saves to labeled common what vals you want for that record.  
> Subsequent calls to the sub, as long as @ID hasn't changed, gets the already 
> calculated value range from common, and applies it to the field you pass it.
> 
> The only trick is feeding your subroutine the date range you won't know until 
> runtime.I don't know a good way to feed an I-descriptor command line 
> parameters such as ">= 6/1/2012 &<= 6/30/2012"
> You'll need to set them in a control record or, better, in @USER1, @USER2, or 
> in your labeled common before the LIST is executed.
> 
> Use the I-descriptors that call your subroutine as output criteria and not 
> WITH or WHEN selection criteria.  The guts of the subr will do the selection 
> limits.
> 
> That may not be altogether clear.  Here's an example.
> 1st the relevant dict entries then an example:
> 
> 
> DICT RTAG09:45:22am  03 Jul 2012  Page1
> 
> Field. Type & Field Conversion.. Column. Output Depth 
> &
> Name.. Field. Definition... Code Heading Format 
> Assoc..
>   Number
> 
> LOCD3 12LS
> EVENT.CODE D8Event 4L M EVENT
> .ASSOC
> Code
> EVENT.DT   D   10   D2/  Event 8R M EVENT
> .ASSOC
> Date
> PRINTERD   13 15LS
> MARKH.CODE I  SUBR(  Event 4L M MARKH
> 'MARKH',  .ASSOC
>  EVENT.CODE )
> Code
> MARKH.DT   I  SUBR( D2/  Event 8R M MARKH
> 'MARKH',  .ASSOC
>  EVENT.DT )
> Date
> MARKH.ASSOCPH MARKH.LINO
>  MARKH.CODE
>  MARKH.REF
>  MARKH.DT
>  MARKH.TM
>  MARKH.WHO
>  MARKH.DTM
> 
> 7 records listed.
> >
> 
> >RUN CDS.BP MARKH.INIT 4/1 4/30
> >LIST RTAG LOC PRINTER   EVENT.CODE EVENT.DT

Re: [U2] UniVerse LIST statement question [not-secure]

2012-07-03 Thread Charles Stevenson

Ed is quite right.

1: I just did it for simplicity in the example.

2: @SENTENCE, without a wrapper: you have do get ReTrieve to ignore
part of the sentence. I've never come up with a good way to do that
other than via EVAL. Anyone?

3. EVAL: Yes, it is a way to pass run-time parameters!
I tend to forget about it because I - personally - don't like it in
production.

EVAL means dictionaries can't be read-only.
And they tend to permanently leave "temporary" I-descriptors scattered
in the dicts if & when something terminates abnormally.

RetrieVe rudely spits the EVAL compilation report into the output, too.
If you have multiple associated columns, & if your usage gets fancy
(that (future) option is a reason for this approach), then your EVAL may
need ASSOC, ASSOC.WITH, MULTIVALUED keywords, besides AS, FMT, COL.HDG.

Other than that, I love EVAL.

On 7/3/2012 11:08 AM, Ed Clark wrote:

There are 3 ways I can think of offhand to tell the subroutine what date range
to use. I like #3 the best:

1: use common variables, which is what the example function MARKH does. The
downside is that you have to assign the common variables before you run the
query.

2: You could use @SENTENCE in the function and parse out the date range. That
could be easy or hard depending on how many people write queries and what odd
contortions of syntax they use. But both #1 and #2 could be simplified by
writing a wrapper program. An example command line would look like:
QLAUNCH "06/01/2012" "06/30/2012" ~ LIST MYFILE INRANGE
QLAUNCH parses arguments before the '~` delimiter and puts them into a common.
The INRANGE attribute calls the function MARKH which uses the common.

3: use the function directly from the command line with EVAL:
LIST MYFILE EVAL "SUBR('MARKH','06/01/2012','06/30/2012')" AS "MYCOL" COL.HDG
"WHATEVER" FMT "11L"
It's wordier, but you have very fine-grained control over what comes out on the
report. COL.HDG, and FMT are optional and AS is optional in this case. You
would us it if you wanted to sort or break on the EVAL column.

On Jul 3, 2012, at 9:55 AM, Charles Stevenson wrote:

What Brian said, except don't replace LIST!
Rather, write your subroutine to be called from an I-descriptors:

MARKH
0001 FUNCTION MARKH( INARG )
0002 COMMON /MARKH/ START.IDT, END.IDT, PREV.ID, VLIST
0003 *
0004 * Assumes common has been initialized properly before this function
0005 * is called from i-descriptors
0006 *
0007 IF @ID = PREV.ID ELSE
0008 IDTS = @RECORD< 10 >
0009 VLIST = ""
0010 VMC = 0
0011 VMAX = DCOUNT( IDTS, @VM )
0012 FOR V = 1 TO VMAX
0013 IDT = IDTS<1,V>
0014 IF IDT >= START.IDT THEN IF IDT <= END.IDT THEN VLIST<-1> = V
0015 NEXT V
0016 END
0017 *
0018 OUTARG = ""
0019 MORE = LEN( VLIST ); * true/false
0020 LOOP WHILE MORE
0021 REMOVE V FROM VLIST SETTING MORE
0022 OUTARG<1,-1> = INARG<1, V >
0023 REPEAT
0024 SETREM 0 ON VLIST
0025 *
0026 RETURN( OUTARG )
0027END

An example using it is below.

It's more flexible and maintainable in the long run than tossing RETRIEVE
altogether.
(Actually, I do think it can be done with the mv-handling functions but an
I-descriptor subroutine will be more maintainable.)

Your sub should limits the output to the values you want. If every value is out
of range, your subr returns null. It can be called for various associated
fields, and the 1st time called for a given @ID it figures out the value range,
and saves to labeled common what vals you want for that record. Subsequent
calls to the sub, as long as @ID hasn't changed, gets the already calculated
value range from common, and applies it to the field you pass it.

The only trick is feeding your subroutine the date range you won't know until runtime.I don't
know a good way to feed an I-descriptor command line parameters such as ">= 6/1/2012
&<= 6/30/2012"
You'll need to set them in a control record or, better, in @USER1, @USER2, or
in your labeled common before the LIST is executed.

Use the I-descriptors that call your subroutine as output criteria and not WITH
or WHEN selection criteria. The guts of the subr will do the selection limits.

That may not be altogether clear. Here's an example.
1st the relevant dict entries then an example:

DICT RTAG09:45:22am 03 Jul 2012 Page1

Field. Type & Field Conversion.. Column. Output Depth &
Name.. Field. Definition... Code Heading Format Assoc..
Number

LOCD3 12LS
EVENT.CODE D8Event 4L M EVENT
.ASSOC
Code
EVENT.DT D 10 D2/ Event 8R M EVENT
.ASSOC
Date
PRINTERD 13 15LS
MARKH.CODE I

Re: [U2] RESIZE - dynamic files


File name ..   GENACCTRN_POSTED
Pathname ...   GENACCTRN_POSTED
File type ..   DYNAMIC
File style and revision    32BIT Revision 12
Hashing Algorithm ..   GENERAL
No. of groups (modulus)    92776 current ( minimum 31, 89 empty,
28229 overflowed, 2485 badly )
Number of records ..   1290469
Large record size ..   3267 bytes
Number of large records    180
Group size .   4096 bytes
Load factors ...   80% (split), 50% (merge) and 80% (actual)
Total size .   500600832 bytes
Total size of record data ..   287035391 bytes
Total size of record IDs ...   21508449 bytes
Unused space ...   192048800 bytes
Total space for records    500592640 bytes

Using the record above, how would I calculate the following?

1) MINIMUM.MODULUS (Is there a formula to use or should I add 20% to the 
current number)?
2) SPLIT - would 90% seem about right?
3) MERGE - would 20% seem about right? 
4) Large Record Size - does 3276 seem right? 
5) Group Size - should I be using 4096?

I'm just a bit confused as to how to set these, I saw the formula to calculate 
the MINIMUM.MODULUS which is (record + id / 4096 or 2048) but I always get a 
lower number
than my current modulus.. 

I also saw where it said to simply take your current modulus # and add 10-20% 
and set the MINIMUM.MODULUS based on that..

Based on the table above I'm just trying to get an idea of what these should be 
set at.

Thanks,

Chris


> From: cjausti...@hotmail.com
> To: u2-users@listserver.u2ug.org
> Date: Tue, 3 Jul 2012 10:28:17 -0500
> Subject: Re: [U2] RESIZE - dynamic files
> 
> 
> Doug,
> 
> When I do the math I come up with a different # (see below):
> 
> File name ..   TEST_FILE
> Pathname ...   TEST_FILE
> File type ..   DYNAMIC
> File style and revision    32BIT Revision 12
> Hashing Algorithm ..   GENERAL
> No. of groups (modulus)    82850 current ( minimum 24, 104 empty,
> 26225 overflowed, 1441 badly )
> Number of records ..   1157122
> Large record size ..   2036 bytes
> Number of large records    576
> Group size .   4096 bytes
> Load factors ...   80% (split), 50% (merge) and 80% (actual)
> Total size .   449605632 bytes
> Total size of record data ..   258687736 bytes
> Total size of record IDs ...   19283300 bytes
> Unused space ...   171626404 bytes
> Total space for records    449597440 bytes
> 
> -- 
> 258,687,736 bytes - Total size of record data
> 19,283,300 bytes - Total size of record IDs
> ===
> 277,971,036 bytes (record + id's)
> 
> 277,971,036 / 4,084 = 68,063 bytes (minimum modulus)
> -- 
> 
> 68,063 is less than the current modulus of 82,850. Something with this 
> formula doesn't seem right because if I use that formula I always calculate a 
> minimum modulus of less than the current modulus.
> 
> Thanks,
> 
> Chris
> 
> 
> 
> > Date: Mon, 2 Jul 2012 16:08:16 -0600
> > From: dave...@gmail.com
> > To: u2-users@listserver.u2ug.org
> > Subject: Re: [U2] RESIZE - dynamic files
> > 
> > Hi Chris:
> > 
> > You cannot get away with not resizing dynamic files in my experience.  The
> > files do not split and merge like we are led to believe.  The separator is
> > not used on dynamic files.  Your Universe file is badly sized.  The math
> > below will get you reasonably file size.
> > 
> > Let's do the math:
> > 
> > 258687736 (Record Size)
> > 192283300 (Key Size)
> > 
> > 450,971,036 (Data and Key Size)
> > 
> > 4096 (Group Size)
> > - 12   (32 Bit Overhead)
> > 
> > 4084 Usable Space
> > 
> > 450971036/4084 = Minimum Modulo 110424 (Prime is 110431)
> > 
> > 
> > [ad]
> > I hate doing this math all of the time.  I have a reasonably priced resize
> > program called XLr8Resizer for $99.00 to do this for me.
> > [/ad]
> > 
> > Regards,
> > Doug
> > www.u2logic.com/tools.html
> > "XLr8Resizer for the rest of us"
> > ___
> > U2-Users mailing list
> > U2-Users@listserver.u2ug.org
> > http://listserver.u2ug.org/mailman/listinfo/u2-users
> 
> ___
> U2-Users mailing list
> U2-Users@listserver.u2ug.org
> http://listserver.u2ug.org/mailman/listinfo/u2-users
  
___
U2-Users mailing list
U2-Users@listserver.u2ug.org
http://listserver.u2ug.org/mailman/listinfo/u2-users

Re: [U2] RESIZE - dynamic files

Yep, I added an extra 2 in the key value.  Oh, the perils of cut and
paste...
___
U2-Users mailing list
U2-Users@listserver.u2ug.org
http://listserver.u2ug.org/mailman/listinfo/u2-users

Re: [U2] RESIZE - dynamic files


No worries Doug. I'm just wondering if the calculation makes sense (if we use 
the example below):

File name ..   GENACCTRN_POSTED
Pathname ...   GENACCTRN_POSTED
File type ..   DYNAMIC
File style and revision    32BIT Revision 12
Hashing Algorithm ..   GENERAL
No. of groups (modulus)    92776 current ( minimum 31, 89 empty,
28229 overflowed, 2485 badly )
Number of records ..   1290469
Large record size ..   3267 bytes
Number of large records    180
Group size .   4096 bytes
Load factors ...   80% (split), 50% (merge) and 80% (actual)
Total size .   500600832 bytes
Total size of record data ..   287035391 bytes
Total size of record IDs ...   21508449 bytes
Unused space ...   192048800 bytes
Total space for records    500592640 bytes

FORMULA -> (287,035,391+21,508,449) / (4,084) = 75,549  MINIMUM.MODULUS

The question I have is whether 75,549 makes sense for this record. I thought 
the MINIMUM.MODULUS was supposed to be bigger than the number of groups (92,776 
in this case)?

Chris


> Date: Tue, 3 Jul 2012 11:04:53 -0600
> From: dave...@gmail.com
> To: u2-users@listserver.u2ug.org
> Subject: Re: [U2] RESIZE - dynamic files
> 
> Yep, I added an extra 2 in the key value.  Oh, the perils of cut and
> paste...
> ___
> U2-Users mailing list
> U2-Users@listserver.u2ug.org
> http://listserver.u2ug.org/mailman/listinfo/u2-users
  
___
U2-Users mailing list
U2-Users@listserver.u2ug.org
http://listserver.u2ug.org/mailman/listinfo/u2-users

Re: [U2] RESIZE - dynamic files

See comment interspersed...

Using the record above, how would I calculate the following?
>
> 1) MINIMUM.MODULUS (Is there a formula to use or should I add 20% to the
> current number)?
>
Should be less the the current size, if you want the file to merge


> 2) SPLIT - would 90% seem about right?
>
Depends on the history of the file.  Is the data growing over time?  The
way the file looks now the split should be reduced because you have 31% in
overflow.

3) MERGE - would 20% seem about right?
>
Won't be used on a growth file, so the file history is important.

4) Large Record Size - does 3276 seem right?
>
Can be calculated with a lot of effort, but yield little gain.

5) Group Size - should I be using 4096?
>
You have two group sizes on dynamic files 2048 and 4096.  If you lower it
you need to double your modulo, roughly.  If you keep it the same you need
to increase your modulo because 31% of your file is in overflow.

>
>
___
U2-Users mailing list
U2-Users@listserver.u2ug.org
http://listserver.u2ug.org/mailman/listinfo/u2-users

Re: [U2] RESIZE - dynamic files

(record + id / 4096 or 2048)

You need to factor in overhead & the split factor:   (records + ids) * 1.1 * 
1.25  / 4096(for 80%) 

If you use a 20% merge factor and a 80% split factor, the file will start 
merging unless you delete 60 percent of your records.  If you use 90% split 
factor, you will have more overflowed groups.  These numbers refer to the total 
amount of data in the file, not to any individual group.

For records of the size that you have, I do not see any advantage to using a 
larger, 4096, group size. You will end up with twice the number of records per 
group vs 2048 (~ 13 vs ~ 7 ), and a little slower keyed access.

-Rick

-Original Message-
From: u2-users-boun...@listserver.u2ug.org 
[mailto:u2-users-boun...@listserver.u2ug.org] On Behalf Of Chris Austin
Sent: Tuesday, July 03, 2012 9:48 AM
To: u2-users@listserver.u2ug.org
Subject: Re: [U2] RESIZE - dynamic files


File name ..   GENACCTRN_POSTED
Pathname ...   GENACCTRN_POSTED
File type ..   DYNAMIC
File style and revision    32BIT Revision 12
Hashing Algorithm ..   GENERAL
No. of groups (modulus)    92776 current ( minimum 31, 89 empty,
28229 overflowed, 2485 badly )
Number of records ..   1290469
Large record size ..   3267 bytes
Number of large records    180
Group size .   4096 bytes
Load factors ...   80% (split), 50% (merge) and 80% (actual)
Total size .   500600832 bytes
Total size of record data ..   287035391 bytes
Total size of record IDs ...   21508449 bytes
Unused space ...   192048800 bytes
Total space for records    500592640 bytes

Using the record above, how would I calculate the following?

1) MINIMUM.MODULUS (Is there a formula to use or should I add 20% to the 
current number)?
2) SPLIT - would 90% seem about right?
3) MERGE - would 20% seem about right? 
4) Large Record Size - does 3276 seem right? 
5) Group Size - should I be using 4096?

I'm just a bit confused as to how to set these, I saw the formula to calculate 
the MINIMUM.MODULUS which is (record + id / 4096 or 2048) but I always get a 
lower number
than my current modulus.. 

I also saw where it said to simply take your current modulus # and add 10-20% 
and set the MINIMUM.MODULUS based on that..

Based on the table above I'm just trying to get an idea of what these should be 
set at.

Thanks,

Chris


> From: cjausti...@hotmail.com
> To: u2-users@listserver.u2ug.org
> Date: Tue, 3 Jul 2012 10:28:17 -0500
> Subject: Re: [U2] RESIZE - dynamic files
> 
> 
> Doug,
> 
> When I do the math I come up with a different # (see below):
> 
> File name ..   TEST_FILE
> Pathname ...   TEST_FILE
> File type ..   DYNAMIC
> File style and revision    32BIT Revision 12
> Hashing Algorithm ..   GENERAL
> No. of groups (modulus)    82850 current ( minimum 24, 104 empty,
> 26225 overflowed, 1441 badly )
> Number of records ..   1157122
> Large record size ..   2036 bytes
> Number of large records    576
> Group size .   4096 bytes
> Load factors ...   80% (split), 50% (merge) and 80% (actual)
> Total size .   449605632 bytes
> Total size of record data ..   258687736 bytes
> Total size of record IDs ...   19283300 bytes
> Unused space ...   171626404 bytes
> Total space for records    449597440 bytes
> 
> -- 
> 258,687,736 bytes - Total size of record data
> 19,283,300 bytes - Total size of record IDs
> ===
> 277,971,036 bytes (record + id's)
> 
> 277,971,036 / 4,084 = 68,063 bytes (minimum modulus)
> -- 
> 
> 68,063 is less than the current modulus of 82,850. Something with this 
> formula doesn't seem right because if I use that formula I always calculate a 
> minimum modulus of less than the current modulus.
> 
> Thanks,
> 
> Chris
> 
> 
> 
> > Date: Mon, 2 Jul 2012 16:08:16 -0600
> > From: dave...@gmail.com
> > To: u2-users@listserver.u2ug.org
> > Subject: Re: [U2] RESIZE - dynamic files
> > 
> > Hi Chris:
> > 
> > You cannot get away with not resizing dynamic files in my experience.  The
> > files do not split and merge like we are led to believe.  The separator is
> > not used on dynamic files.  Your Universe file is badly sized.  The math
> > below will get you reasonably file size.
> > 
> > Let's do the math:
> > 
> > 258687736 (Record Size)
> > 192283300 (Key Size)
> > 
> > 450,971,036 (Data and Key Size)
> > 
> > 4096 (Group Size)
> > - 12   (32 Bit Overhead)
> > 
> > 4084 Usable Space
> > 
> > 450971036/4084 = Minimum Modulo 110424 (Prime is 110431)
> > 
> > 
> > [ad]
> > I hate doing this math all of the time.  I have a reasonably priced resize
> > program called XLr8Resizer for $99.00 to do this for me.
> > [/ad]
> > 
> > Regards,
> > Doug
> > www.u

Re: [U2] Running XLr8 Tools inside U2 DBTools & new Eclipse release

Hi Dan and others:

I finished testing all of the Rocket tools.  Turns out the only one that
really works as a plug-in platform is BDT for U2logic plug-ins.  The other
products are crippled versions of Eclipse with limited menus by design.

Another big problem is that Rocket's Eclipse products do not share work
spaces, so if you start BDT your work space is in that directory.  If you
start U2Admin the work space is in that directory.  Unfortunately this
situation is very messy for U2logic, a Rocket VAR, to support.  I'm going
to have fun explaining this to the first client that asks why the programs
don't share the server names.

Regards,
Doug
www.u2logic.com
___
U2-Users mailing list
U2-Users@listserver.u2ug.org
http://listserver.u2ug.org/mailman/listinfo/u2-users

Re: [U2] RESIZE - dynamic files


So for this example what would be a good SPLIT level and what would be a good 
MERGE level to use? It was my understanding that I wanted to lower my merge to 
something below 50% and
increase the split to reduce splitting.

Chris


> From: r...@lynden.com
> To: u2-users@listserver.u2ug.org
> Date: Tue, 3 Jul 2012 10:21:16 -0700
> Subject: Re: [U2] RESIZE - dynamic files
> 
> (record + id / 4096 or 2048)
> 
> You need to factor in overhead & the split factor:   (records + ids) * 1.1 * 
> 1.25  / 4096(for 80%) 
> 
> If you use a 20% merge factor and a 80% split factor, the file will start 
> merging unless you delete 60 percent of your records.  If you use 90% split 
> factor, you will have more overflowed groups.  These numbers refer to the 
> total amount of data in the file, not to any individual group.
> 
> For records of the size that you have, I do not see any advantage to using a 
> larger, 4096, group size. You will end up with twice the number of records 
> per group vs 2048 (~ 13 vs ~ 7 ), and a little slower keyed access.
> 
> -Rick
> 
> -Original Message-
> From: u2-users-boun...@listserver.u2ug.org 
> [mailto:u2-users-boun...@listserver.u2ug.org] On Behalf Of Chris Austin
> Sent: Tuesday, July 03, 2012 9:48 AM
> To: u2-users@listserver.u2ug.org
> Subject: Re: [U2] RESIZE - dynamic files
> 
> 
> File name ..   GENACCTRN_POSTED
> Pathname ...   GENACCTRN_POSTED
> File type ..   DYNAMIC
> File style and revision    32BIT Revision 12
> Hashing Algorithm ..   GENERAL
> No. of groups (modulus)    92776 current ( minimum 31, 89 empty,
> 28229 overflowed, 2485 badly )
> Number of records ..   1290469
> Large record size ..   3267 bytes
> Number of large records    180
> Group size .   4096 bytes
> Load factors ...   80% (split), 50% (merge) and 80% (actual)
> Total size .   500600832 bytes
> Total size of record data ..   287035391 bytes
> Total size of record IDs ...   21508449 bytes
> Unused space ...   192048800 bytes
> Total space for records    500592640 bytes
> 
> Using the record above, how would I calculate the following?
> 
> 1) MINIMUM.MODULUS (Is there a formula to use or should I add 20% to the 
> current number)?
> 2) SPLIT - would 90% seem about right?
> 3) MERGE - would 20% seem about right? 
> 4) Large Record Size - does 3276 seem right? 
> 5) Group Size - should I be using 4096?
> 
> I'm just a bit confused as to how to set these, I saw the formula to 
> calculate the MINIMUM.MODULUS which is (record + id / 4096 or 2048) but I 
> always get a lower number
> than my current modulus.. 
> 
> I also saw where it said to simply take your current modulus # and add 10-20% 
> and set the MINIMUM.MODULUS based on that..
> 
> Based on the table above I'm just trying to get an idea of what these should 
> be set at.
> 
> Thanks,
> 
> Chris
> 
> 
> > From: cjausti...@hotmail.com
> > To: u2-users@listserver.u2ug.org
> > Date: Tue, 3 Jul 2012 10:28:17 -0500
> > Subject: Re: [U2] RESIZE - dynamic files
> > 
> > 
> > Doug,
> > 
> > When I do the math I come up with a different # (see below):
> > 
> > File name ..   TEST_FILE
> > Pathname ...   TEST_FILE
> > File type ..   DYNAMIC
> > File style and revision    32BIT Revision 12
> > Hashing Algorithm ..   GENERAL
> > No. of groups (modulus)    82850 current ( minimum 24, 104 empty,
> > 26225 overflowed, 1441 badly )
> > Number of records ..   1157122
> > Large record size ..   2036 bytes
> > Number of large records    576
> > Group size .   4096 bytes
> > Load factors ...   80% (split), 50% (merge) and 80% (actual)
> > Total size .   449605632 bytes
> > Total size of record data ..   258687736 bytes
> > Total size of record IDs ...   19283300 bytes
> > Unused space ...   171626404 bytes
> > Total space for records    449597440 bytes
> > 
> > -- 
> > 258,687,736 bytes - Total size of record data
> > 19,283,300 bytes - Total size of record IDs
> > ===
> > 277,971,036 bytes (record + id's)
> > 
> > 277,971,036 / 4,084 = 68,063 bytes (minimum modulus)
> > -- 
> > 
> > 68,063 is less than the current modulus of 82,850. Something with this 
> > formula doesn't seem right because if I use that formula I always calculate 
> > a 
> > minimum modulus of less than the current modulus.
> > 
> > Thanks,
> > 
> > Chris
> > 
> > 
> > 
> > > Date: Mon, 2 Jul 2012 16:08:16 -0600
> > > From: dave...@gmail.com
> > > To: u2-users@listserver.u2ug.org
> > > Subject: Re: [U2] RESIZE - dynamic files
> > > 
> > > Hi Chris:
> > > 
> > > You cannot get away with not resizing dynamic files in my experience.  The
> > > files do not split and merge like we are led to believe.  The

Re: [U2] RESIZE - dynamic files



Doug,

The data is growing over time with this file. Does that mean I should ignore 
the formula? Or should I still use a lower MINIMUM.MODULO than the
actual modulo #..

Is the idea to reduce overflow by lowering the split? What is this 'overflow' 
referring to?

> > 2) SPLIT - would 90% seem about right?
> >
> Depends on the history of the file.  Is the data growing over time?  The
> way the file looks now the split should be reduced because you have 31% in
> overflow.

So basically don't spend much time worrying about large record size?

> 4) Large Record Size - does 3276 seem right?
> >
> Can be calculated with a lot of effort, but yield little gain.

This seems like a moot point as well, as long as the ratio in regards to the 
MINIMUM.MODULO is set proportionally?

> 5) Group Size - should I be using 4096?
> >
> You have two group sizes on dynamic files 2048 and 4096.  If you lower it
> you need to double your modulo, roughly.  If you keep it the same you need
> to increase your modulo because 31% of your file is in overflow.

Chris


  
___
U2-Users mailing list
U2-Users@listserver.u2ug.org
http://listserver.u2ug.org/mailman/listinfo/u2-users

Re: [U2] RESIZE - dynamic files


Using the formula below, and changing the split to 90% I get the following:

File name ..   GENACCTRN_POSTED
Pathname ...   GENACCTRN_POSTED
File type ..   DYNAMIC
File style and revision    32BIT Revision 12
Hashing Algorithm ..   GENERAL
No. of groups (modulus)    103889 current ( minimum 103889, 114 empty,
22249 overflowed, 1764 badly )
Number of records ..   1290469
Large record size ..   3267 bytes
Number of large records    180
Group size .   4096 bytes
Load factors ...   90% (split), 50% (merge) and 72% (actual)
Total size .   519921664 bytes
Total size of record data ..   287400591 bytes
Total size of record IDs ...   21508497 bytes
Unused space ...   211004384 bytes
Total space for records    519913472 bytes

How does this look in terms of performance? 

My Actual load went down 8% as well as some overflow but it looks like my load 
% is still high at 72% I'm wondering if I should raise the MINIMUM.MODULUS even 
more 
since I still have a decent amount of overflow and not many large records. 

Chris


> From: r...@lynden.com
> To: u2-users@listserver.u2ug.org
> Date: Tue, 3 Jul 2012 10:21:16 -0700
> Subject: Re: [U2] RESIZE - dynamic files
> 
> (record + id / 4096 or 2048)
> 
> You need to factor in overhead & the split factor:   (records + ids) * 1.1 * 
> 1.25  / 4096(for 80%) 
> 
> If you use a 20% merge factor and a 80% split factor, the file will start 
> merging unless you delete 60 percent of your records.  If you use 90% split 
> factor, you will have more overflowed groups.  These numbers refer to the 
> total amount of data in the file, not to any individual group.
> 
> For records of the size that you have, I do not see any advantage to using a 
> larger, 4096, group size. You will end up with twice the number of records 
> per group vs 2048 (~ 13 vs ~ 7 ), and a little slower keyed access.
> 
> -Rick
> 
> -Original Message-
> From: u2-users-boun...@listserver.u2ug.org 
> [mailto:u2-users-boun...@listserver.u2ug.org] On Behalf Of Chris Austin
> Sent: Tuesday, July 03, 2012 9:48 AM
> To: u2-users@listserver.u2ug.org
> Subject: Re: [U2] RESIZE - dynamic files
> 
> 
> File name ..   GENACCTRN_POSTED
> Pathname ...   GENACCTRN_POSTED
> File type ..   DYNAMIC
> File style and revision    32BIT Revision 12
> Hashing Algorithm ..   GENERAL
> No. of groups (modulus)    92776 current ( minimum 31, 89 empty,
> 28229 overflowed, 2485 badly )
> Number of records ..   1290469
> Large record size ..   3267 bytes
> Number of large records    180
> Group size .   4096 bytes
> Load factors ...   80% (split), 50% (merge) and 80% (actual)
> Total size .   500600832 bytes
> Total size of record data ..   287035391 bytes
> Total size of record IDs ...   21508449 bytes
> Unused space ...   192048800 bytes
> Total space for records    500592640 bytes
> 
> Using the record above, how would I calculate the following?
> 
> 1) MINIMUM.MODULUS (Is there a formula to use or should I add 20% to the 
> current number)?
> 2) SPLIT - would 90% seem about right?
> 3) MERGE - would 20% seem about right? 
> 4) Large Record Size - does 3276 seem right? 
> 5) Group Size - should I be using 4096?
> 
> I'm just a bit confused as to how to set these, I saw the formula to 
> calculate the MINIMUM.MODULUS which is (record + id / 4096 or 2048) but I 
> always get a lower number
> than my current modulus.. 
> 
> I also saw where it said to simply take your current modulus # and add 10-20% 
> and set the MINIMUM.MODULUS based on that..
> 
> Based on the table above I'm just trying to get an idea of what these should 
> be set at.
> 
> Thanks,
> 
> Chris
> 
> 
> > From: cjausti...@hotmail.com
> > To: u2-users@listserver.u2ug.org
> > Date: Tue, 3 Jul 2012 10:28:17 -0500
> > Subject: Re: [U2] RESIZE - dynamic files
> > 
> > 
> > Doug,
> > 
> > When I do the math I come up with a different # (see below):
> > 
> > File name ..   TEST_FILE
> > Pathname ...   TEST_FILE
> > File type ..   DYNAMIC
> > File style and revision    32BIT Revision 12
> > Hashing Algorithm ..   GENERAL
> > No. of groups (modulus)    82850 current ( minimum 24, 104 empty,
> > 26225 overflowed, 1441 badly )
> > Number of records ..   1157122
> > Large record size ..   2036 bytes
> > Number of large records    576
> > Group size .   4096 bytes
> > Load factors ...   80% (split), 50% (merge) and 80% (actual)
> > Total size .   449605632 bytes
> > Total size of record data ..   258687736 bytes
> > Total size of rec

Re: [U2] RESIZE - dynamic files

2012-07-03 Thread Jeff Schasny

I would recommend that if you intend to do resizing on a regular basis 
an you want to improve the performance of the file you might consider 
resizing the file to a static file type so that you can have more 
control over the hashing algorithm, separation and modulo.


Chris Austin wrote:

Using the formula below, and changing the split to 90% I get the following:

File name ..   GENACCTRN_POSTED
Pathname ...   GENACCTRN_POSTED
File type ..   DYNAMIC
File style and revision    32BIT Revision 12
Hashing Algorithm ..   GENERAL
No. of groups (modulus)    103889 current ( minimum 103889, 114 empty,
22249 overflowed, 1764 badly )
Number of records ..   1290469
Large record size ..   3267 bytes
Number of large records    180
Group size .   4096 bytes
Load factors ...   90% (split), 50% (merge) and 72% (actual)
Total size .   519921664 bytes
Total size of record data ..   287400591 bytes
Total size of record IDs ...   21508497 bytes
Unused space ...   211004384 bytes
Total space for records    519913472 bytes

How does this look in terms of performance? 

My Actual load went down 8% as well as some overflow but it looks like my load % is still high at 72% I'm wondering if I should raise the MINIMUM.MODULUS even more 
since I still have a decent amount of overflow and not many large records. 


Chris


  

From: r...@lynden.com
To: u2-users@listserver.u2ug.org
Date: Tue, 3 Jul 2012 10:21:16 -0700
Subject: Re: [U2] RESIZE - dynamic files

(record + id / 4096 or 2048)

You need to factor in overhead & the split factor:   (records + ids) * 1.1 * 1.25  / 4096(for 80%) 


If you use a 20% merge factor and a 80% split factor, the file will start 
merging unless you delete 60 percent of your records.  If you use 90% split 
factor, you will have more overflowed groups.  These numbers refer to the total 
amount of data in the file, not to any individual group.

For records of the size that you have, I do not see any advantage to using a 
larger, 4096, group size. You will end up with twice the number of records per 
group vs 2048 (~ 13 vs ~ 7 ), and a little slower keyed access.

-Rick

-Original Message-
From: u2-users-boun...@listserver.u2ug.org 
[mailto:u2-users-boun...@listserver.u2ug.org] On Behalf Of Chris Austin
Sent: Tuesday, July 03, 2012 9:48 AM
To: u2-users@listserver.u2ug.org
Subject: Re: [U2] RESIZE - dynamic files


File name ..   GENACCTRN_POSTED
Pathname ...   GENACCTRN_POSTED
File type ..   DYNAMIC
File style and revision    32BIT Revision 12
Hashing Algorithm ..   GENERAL
No. of groups (modulus)    92776 current ( minimum 31, 89 empty,
28229 overflowed, 2485 badly )
Number of records ..   1290469
Large record size ..   3267 bytes
Number of large records    180
Group size .   4096 bytes
Load factors ...   80% (split), 50% (merge) and 80% (actual)
Total size .   500600832 bytes
Total size of record data ..   287035391 bytes
Total size of record IDs ...   21508449 bytes
Unused space ...   192048800 bytes
Total space for records    500592640 bytes

Using the record above, how would I calculate the following?

1) MINIMUM.MODULUS (Is there a formula to use or should I add 20% to the 
current number)?
2) SPLIT - would 90% seem about right?
3) MERGE - would 20% seem about right? 
4) Large Record Size - does 3276 seem right? 
5) Group Size - should I be using 4096?


I'm just a bit confused as to how to set these, I saw the formula to calculate 
the MINIMUM.MODULUS which is (record + id / 4096 or 2048) but I always get a 
lower number
than my current modulus.. 


I also saw where it said to simply take your current modulus # and add 10-20% 
and set the MINIMUM.MODULUS based on that..

Based on the table above I'm just trying to get an idea of what these should be 
set at.

Thanks,

Chris




From: cjausti...@hotmail.com
To: u2-users@listserver.u2ug.org
Date: Tue, 3 Jul 2012 10:28:17 -0500
Subject: Re: [U2] RESIZE - dynamic files


Doug,

When I do the math I come up with a different # (see below):

File name ..   TEST_FILE
Pathname ...   TEST_FILE
File type ..   DYNAMIC
File style and revision    32BIT Revision 12
Hashing Algorithm ..   GENERAL
No. of groups (modulus)    82850 current ( minimum 24, 104 empty,
26225 overflowed, 1441 badly )
Number of records ..   1157122
Large record size ..   2036 bytes
Number of large records    576
Group size .   4096 bytes
Load factors ...   80% (split), 50% (merge) and 80% (actual)
Total size .   449605632 bytes
Total

Re: [U2] RESIZE - dynamic files

The split load is not affecting anything here, since it is more than the actual 
load.  What your overflow suggests is that you lower the split.load value to 
70$% or below.  You could go ahead and set the merge.load to an arbitrarily low 
number ("1"), and it will probably never do a merge, which would be the same as 
specifying a minimum.modulus equal to "as large as it ever gets".  The 
exception to this is during file creation & clear.file,  when the 
minimum.modulus value determines the initial disk allocation.  Short of going 
to an arbitrarily large minimum.modulus, and a very low split.load, you are 
going to have some overflow (unless you have sequential keys & like sized 
records).

-Rick

-Original Message-
From: u2-users-boun...@listserver.u2ug.org 
[mailto:u2-users-boun...@listserver.u2ug.org] On Behalf Of Chris Austin
Sent: Tuesday, July 03, 2012 12:54 PM
To: u2-users@listserver.u2ug.org
Subject: Re: [U2] RESIZE - dynamic files


Using the formula below, and changing the split to 90% I get the following:

File name ..   GENACCTRN_POSTED
Pathname ...   GENACCTRN_POSTED
File type ..   DYNAMIC
File style and revision    32BIT Revision 12
Hashing Algorithm ..   GENERAL
No. of groups (modulus)    103889 current ( minimum 103889, 114 empty,
22249 overflowed, 1764 badly )
Number of records ..   1290469
Large record size ..   3267 bytes
Number of large records    180
Group size .   4096 bytes
Load factors ...   90% (split), 50% (merge) and 72% (actual)
Total size .   519921664 bytes
Total size of record data ..   287400591 bytes
Total size of record IDs ...   21508497 bytes
Unused space ...   211004384 bytes
Total space for records    519913472 bytes

How does this look in terms of performance? 

My Actual load went down 8% as well as some overflow but it looks like my load 
% is still high at 72% I'm wondering if I should raise the MINIMUM.MODULUS even 
more 
since I still have a decent amount of overflow and not many large records. 

Chris


> From: r...@lynden.com
> To: u2-users@listserver.u2ug.org
> Date: Tue, 3 Jul 2012 10:21:16 -0700
> Subject: Re: [U2] RESIZE - dynamic files
> 
> (record + id / 4096 or 2048)
> 
> You need to factor in overhead & the split factor:   (records + ids) * 1.1 * 
> 1.25  / 4096(for 80%) 
> 
> If you use a 20% merge factor and a 80% split factor, the file will start 
> merging unless you delete 60 percent of your records.  If you use 90% split 
> factor, you will have more overflowed groups.  These numbers refer to the 
> total amount of data in the file, not to any individual group.
> 
> For records of the size that you have, I do not see any advantage to using a 
> larger, 4096, group size. You will end up with twice the number of records 
> per group vs 2048 (~ 13 vs ~ 7 ), and a little slower keyed access.
> 
> -Rick
> 
> -Original Message-
> From: u2-users-boun...@listserver.u2ug.org 
> [mailto:u2-users-boun...@listserver.u2ug.org] On Behalf Of Chris Austin
> Sent: Tuesday, July 03, 2012 9:48 AM
> To: u2-users@listserver.u2ug.org
> Subject: Re: [U2] RESIZE - dynamic files
> 
> 
> File name ..   GENACCTRN_POSTED
> Pathname ...   GENACCTRN_POSTED
> File type ..   DYNAMIC
> File style and revision    32BIT Revision 12
> Hashing Algorithm ..   GENERAL
> No. of groups (modulus)    92776 current ( minimum 31, 89 empty,
> 28229 overflowed, 2485 badly )
> Number of records ..   1290469
> Large record size ..   3267 bytes
> Number of large records    180
> Group size .   4096 bytes
> Load factors ...   80% (split), 50% (merge) and 80% (actual)
> Total size .   500600832 bytes
> Total size of record data ..   287035391 bytes
> Total size of record IDs ...   21508449 bytes
> Unused space ...   192048800 bytes
> Total space for records    500592640 bytes
> 
> Using the record above, how would I calculate the following?
> 
> 1) MINIMUM.MODULUS (Is there a formula to use or should I add 20% to the 
> current number)?
> 2) SPLIT - would 90% seem about right?
> 3) MERGE - would 20% seem about right? 
> 4) Large Record Size - does 3276 seem right? 
> 5) Group Size - should I be using 4096?
> 
> I'm just a bit confused as to how to set these, I saw the formula to 
> calculate the MINIMUM.MODULUS which is (record + id / 4096 or 2048) but I 
> always get a lower number
> than my current modulus.. 
> 
> I also saw where it said to simply take your current modulus # and add 10-20% 
> and set the MINIMUM.MODULUS based on that..
> 
> Based on the table above I'm just trying to get an idea of what these should 
> be set at.
> 
> Thanks,
> 
> Chris
> 
> 
> > From: cjausti...@hotmail.com
> > To: u

Re: [U2] UniVerse LIST statement question [not-secure]

2012-07-03 Thread Ed Clark

There have been more than a few times when I wished that MV query was a little 
more extensible. On a couple of occasions I have created throwaway connectives 
as a means to pass flags to an i-type or a B correlative, but it would be nice 
if there were a keyword that flagged to just ignore everything after it on the 
line.

Hmm, I never knew that EVAL needed to write to the dict on universe. Is that 
try on unidata as well?

On Jul 3, 2012, at 12:47 PM, Charles Stevenson wrote:

> Ed is quite right.
> 
> 1:  I just did it for simplicity in the example.
> 
> 2:  @SENTENCE, without a wrapper:  you have do get ReTrieve to ignore part of 
> the sentence.   I've never come up with a good way to do that other than via 
> EVAL.  Anyone?
> 
> 3.  EVAL: Yes, it is a way to pass run-time parameters!
> I tend to forget about it because I - personally - don't like it in 
> production.
> EVAL means dictionaries can't be read-only.
> And they tend to permanently leave "temporary" I-descriptors scattered in the 
> dicts if & when something terminates abnormally.
> RetrieVe rudely spits the EVAL compilation report into the output, too.
> If you have multiple associated columns, & if your usage gets fancy (that 
> (future) option is a reason for this approach), then your EVAL may need 
> ASSOC, ASSOC.WITH, MULTIVALUED keywords, besides AS, FMT, COL.HDG.
> Other than that, I love EVAL.
> 
> 
> On 7/3/2012 11:08 AM, Ed Clark wrote:
>> There are 3 ways I can think of offhand to tell the subroutine what date 
>> range to use. I like #3 the best:
>> 
>> 1: use common variables, which is what the example function MARKH does. The 
>> downside is that you have to assign the common variables before you run the 
>> query.
>> 
>> 2: You could use @SENTENCE in the function and parse out the date range. 
>> That could be easy or hard depending on how many people write queries and 
>> what odd contortions of syntax they use. But both #1 and #2 could be 
>> simplified by writing a wrapper program. An example command line would look 
>> like:
>>   QLAUNCH "06/01/2012" "06/30/2012" ~ LIST MYFILE INRANGE
>> QLAUNCH parses arguments before the '~` delimiter and puts them into a 
>> common. The INRANGE attribute calls the function MARKH which uses the common.
>> 
>> 3: use the function directly from the command line with EVAL:
>>   LIST MYFILE EVAL "SUBR('MARKH','06/01/2012','06/30/2012')"  AS "MYCOL" 
>> COL.HDG "WHATEVER" FMT "11L"
>> It's wordier, but you have very fine-grained control over what comes out on 
>> the report. COL.HDG, and FMT are optional and AS is optional in this case. 
>> You would us it if you wanted to sort or break on the EVAL column.
>> 
>> 
>> 
>> On Jul 3, 2012, at 9:55 AM, Charles Stevenson wrote:
>> 
>>> What Brian said, except don't replace LIST!
>>> Rather, write your subroutine to be called from an I-descriptors:
>>> 
>>> MARKH
>>> 0001   FUNCTION MARKH( INARG )
>>> 0002   COMMON /MARKH/ START.IDT, END.IDT, PREV.ID, VLIST
>>> 0003 *
>>> 0004 * Assumes common has been initialized properly before this function
>>> 0005 * is called from i-descriptors
>>> 0006 *
>>> 0007   IF @ID = PREV.ID ELSE
>>> 0008  IDTS = @RECORD< 10 >
>>> 0009  VLIST = ""
>>> 0010  VMC = 0
>>> 0011  VMAX = DCOUNT( IDTS, @VM )
>>> 0012  FOR V = 1 TO VMAX
>>> 0013 IDT = IDTS<1,V>
>>> 0014 IF IDT >= START.IDT THEN IF IDT <= END.IDT THEN VLIST<-1> 
>>> = V
>>> 0015  NEXT V
>>> 0016   END
>>> 0017 *
>>> 0018   OUTARG = ""
>>> 0019   MORE = LEN( VLIST ); * true/false
>>> 0020   LOOP WHILE MORE
>>> 0021  REMOVE V FROM VLIST SETTING MORE
>>> 0022  OUTARG<1,-1> = INARG<1, V >
>>> 0023   REPEAT
>>> 0024   SETREM 0 ON VLIST
>>> 0025 *
>>> 0026   RETURN( OUTARG )
>>> 0027END
>>> 
>>> An example using it is below.
>>> 
>>> It's more flexible and maintainable in the long run than tossing RETRIEVE 
>>> altogether.
>>> (Actually, I do think it can be done with the mv-handling functions but an 
>>> I-descriptor subroutine will be more maintainable.)
>>> 
>>> Your sub should limits the output to the values you want. If every value is 
>>> out of range, your subr returns null.  It can be called for various 
>>> associated fields, and the 1st time called for a given @ID it figures out 
>>> the value range, and saves to labeled common what vals you want for that 
>>> record.  Subsequent calls to the sub, as long as @ID hasn't changed, gets 
>>> the already calculated value range from common, and applies it to the field 
>>> you pass it.
>>> 
>>> The only trick is feeding your subroutine the date range you won't know 
>>> until runtime.I don't know a good way to feed an I-descriptor command 
>>> line parameters such as ">= 6/1/2012 &<= 6/30/2012"
>>> You'll need to set them in a control record or, better, in @USER1, @USER2, 
>>> or in your labeled common before the LIST is executed.
>>> 
>>> Use the I-descrip

Re: [U2] RESIZE - dynamic files


I guess what I need to know is what's an acceptable % of overflow for a dynamic 
file? For example, when I change the SPLIT LOAD to 90% (while using the 
calculated min modulus)
I'm still left with ~ 20% of overflow (see below). Is 20% overflow considered 
acceptable on average or should I keep tinkering with it to reach a lower 
overflow %?

Correct me if I'm wrong but it seems the goal here is to REDUCE the overflow % 
while not creating too many modulus (groups). 

Chris


File name ..   GENACCTRN_POSTED
Pathname ...   GENACCTRN_POSTED
File type ..   DYNAMIC
File style and revision    32BIT Revision 12
Hashing Algorithm ..   GENERAL
No. of groups (modulus)    105715 current ( minimum 103889, 114 empty,
21092 overflowed, 1452 badly )
Number of records ..   1290469
Large record size ..   3267 bytes
Number of large records    180
Group size .   4096 bytes
Load factors ...   90% (split), 50% (merge) and 70% (actual)
Total size .   522260480 bytes
Total size of record data ..   287400239 bytes
Total size of record IDs ...   21508521 bytes
Unused space ...   213343528 bytes
Total space for records    522252288 bytes

> From: r...@lynden.com
> To: u2-users@listserver.u2ug.org
> Date: Tue, 3 Jul 2012 13:10:43 -0700
> Subject: Re: [U2] RESIZE - dynamic files
> 
> The split load is not affecting anything here, since it is more than the 
> actual load.  What your overflow suggests is that you lower the split.load 
> value to 70$% or below.  You could go ahead and set the merge.load to an 
> arbitrarily low number ("1"), and it will probably never do a merge, which 
> would be the same as specifying a minimum.modulus equal to "as large as it 
> ever gets".  The exception to this is during file creation & clear.file,  
> when the minimum.modulus value determines the initial disk allocation.  Short 
> of going to an arbitrarily large minimum.modulus, and a very low split.load, 
> you are going to have some overflow (unless you have sequential keys & like 
> sized records).
> 
> -Rick
> 
> -Original Message-
> From: u2-users-boun...@listserver.u2ug.org 
> [mailto:u2-users-boun...@listserver.u2ug.org] On Behalf Of Chris Austin
> Sent: Tuesday, July 03, 2012 12:54 PM
> To: u2-users@listserver.u2ug.org
> Subject: Re: [U2] RESIZE - dynamic files
> 
> 
> Using the formula below, and changing the split to 90% I get the following:
> 
> File name ..   GENACCTRN_POSTED
> Pathname ...   GENACCTRN_POSTED
> File type ..   DYNAMIC
> File style and revision    32BIT Revision 12
> Hashing Algorithm ..   GENERAL
> No. of groups (modulus)    103889 current ( minimum 103889, 114 empty,
> 22249 overflowed, 1764 badly )
> Number of records ..   1290469
> Large record size ..   3267 bytes
> Number of large records    180
> Group size .   4096 bytes
> Load factors ...   90% (split), 50% (merge) and 72% (actual)
> Total size .   519921664 bytes
> Total size of record data ..   287400591 bytes
> Total size of record IDs ...   21508497 bytes
> Unused space ...   211004384 bytes
> Total space for records    519913472 bytes
> 
> How does this look in terms of performance? 
> 
> My Actual load went down 8% as well as some overflow but it looks like my 
> load % is still high at 72% I'm wondering if I should raise the 
> MINIMUM.MODULUS even more 
> since I still have a decent amount of overflow and not many large records. 
> 
> Chris
> 
> 
> > From: r...@lynden.com
> > To: u2-users@listserver.u2ug.org
> > Date: Tue, 3 Jul 2012 10:21:16 -0700
> > Subject: Re: [U2] RESIZE - dynamic files
> > 
> > (record + id / 4096 or 2048)
> > 
> > You need to factor in overhead & the split factor:   (records + ids) * 1.1 
> > * 1.25  / 4096(for 80%) 
> > 
> > If you use a 20% merge factor and a 80% split factor, the file will start 
> > merging unless you delete 60 percent of your records.  If you use 90% split 
> > factor, you will have more overflowed groups.  These numbers refer to the 
> > total amount of data in the file, not to any individual group.
> > 
> > For records of the size that you have, I do not see any advantage to using 
> > a larger, 4096, group size. You will end up with twice the number of 
> > records per group vs 2048 (~ 13 vs ~ 7 ), and a little slower keyed access.
> > 
> > -Rick
> > 
> > -Original Message-
> > From: u2-users-boun...@listserver.u2ug.org 
> > [mailto:u2-users-boun...@listserver.u2ug.org] On Behalf Of Chris Austin
> > Sent: Tuesday, July 03, 2012 9:48 AM
> > To: u2-users@listserver.u2ug.org
> > Subject: Re: [U2] RESIZE - dynamic files
> > 
> > 
> > File name ..   GENACCTRN_POSTED
> > Pathname ...   GE

Re: [U2] RESIZE - dynamic files

The actual load is 70% on your file. The split.load of 90 was set after the 
file was loaded. If you leave it at that value, and add another 100,000 
records, your modulus will not grow, but the number of overflowed groups will. 

Perhaps you need to look at is as "80% not overflowed".  Despite the output, I 
doubt that any of those overflows are that bad. 




-Rick

On Jul 3, 2012, at 1:23 PM, "Chris Austin"  wrote:

> 
> I guess what I need to know is what's an acceptable % of overflow for a 
> dynamic file? For example, when I change the SPLIT LOAD to 90% (while using 
> the calculated min modulus)
> I'm still left with ~ 20% of overflow (see below). Is 20% overflow considered 
> acceptable on average or should I keep tinkering with it to reach a lower 
> overflow %?
> 
> Correct me if I'm wrong but it seems the goal here is to REDUCE the overflow 
> % while not creating too many modulus (groups).
> 
> Chris
> 
> 
> File name ..   GENACCTRN_POSTED
> Pathname ...   GENACCTRN_POSTED
> File type ..   DYNAMIC
> File style and revision    32BIT Revision 12
> Hashing Algorithm ..   GENERAL
> No. of groups (modulus)    105715 current ( minimum 103889, 114 empty,
>21092 overflowed, 1452 badly )
> Number of records ..   1290469
> Large record size ..   3267 bytes
> Number of large records    180
> Group size .   4096 bytes
> Load factors ...   90% (split), 50% (merge) and 70% (actual)
> Total size .   522260480 bytes
> Total size of record data ..   287400239 bytes
> Total size of record IDs ...   21508521 bytes
> Unused space ...   213343528 bytes
> Total space for records    522252288 bytes
> 
>> From: r...@lynden.com
>> To: u2-users@listserver.u2ug.org
>> Date: Tue, 3 Jul 2012 13:10:43 -0700
>> Subject: Re: [U2] RESIZE - dynamic files
>> 
>> The split load is not affecting anything here, since it is more than the 
>> actual load.  What your overflow suggests is that you lower the split.load 
>> value to 70$% or below.  You could go ahead and set the merge.load to an 
>> arbitrarily low number ("1"), and it will probably never do a merge, which 
>> would be the same as specifying a minimum.modulus equal to "as large as it 
>> ever gets".  The exception to this is during file creation & clear.file,  
>> when the minimum.modulus value determines the initial disk allocation.  
>> Short of going to an arbitrarily large minimum.modulus, and a very low 
>> split.load, you are going to have some overflow (unless you have sequential 
>> keys & like sized records).
>> 
>> -Rick
>> 
>> -Original Message-
>> From: u2-users-boun...@listserver.u2ug.org 
>> [mailto:u2-users-boun...@listserver.u2ug.org] On Behalf Of Chris Austin
>> Sent: Tuesday, July 03, 2012 12:54 PM
>> To: u2-users@listserver.u2ug.org
>> Subject: Re: [U2] RESIZE - dynamic files
>> 
>> 
>> Using the formula below, and changing the split to 90% I get the following:
>> 
>> File name ..   GENACCTRN_POSTED
>> Pathname ...   GENACCTRN_POSTED
>> File type ..   DYNAMIC
>> File style and revision    32BIT Revision 12
>> Hashing Algorithm ..   GENERAL
>> No. of groups (modulus)    103889 current ( minimum 103889, 114 empty,
>>22249 overflowed, 1764 badly )
>> Number of records ..   1290469
>> Large record size ..   3267 bytes
>> Number of large records    180
>> Group size .   4096 bytes
>> Load factors ...   90% (split), 50% (merge) and 72% (actual)
>> Total size .   519921664 bytes
>> Total size of record data ..   287400591 bytes
>> Total size of record IDs ...   21508497 bytes
>> Unused space ...   211004384 bytes
>> Total space for records    519913472 bytes
>> 
>> How does this look in terms of performance?
>> 
>> My Actual load went down 8% as well as some overflow but it looks like my 
>> load % is still high at 72% I'm wondering if I should raise the 
>> MINIMUM.MODULUS even more
>> since I still have a decent amount of overflow and not many large records.
>> 
>> Chris
>> 
>> 
>>> From: r...@lynden.com
>>> To: u2-users@listserver.u2ug.org
>>> Date: Tue, 3 Jul 2012 10:21:16 -0700
>>> Subject: Re: [U2] RESIZE - dynamic files
>>> 
>>> (record + id / 4096 or 2048)
>>> 
>>> You need to factor in overhead & the split factor:   (records + ids) * 1.1 
>>> * 1.25  / 4096(for 80%)
>>> 
>>> If you use a 20% merge factor and a 80% split factor, the file will start 
>>> merging unless you delete 60 percent of your records.  If you use 90% split 
>>> factor, you will have more overflowed groups.  These numbers refer to the 
>>> total amount of data in the file, not to any individual group.
>>> 
>>> For records of the size that you have, I do not see any advantage to using 
>>> a larger, 4

Re: [U2] RESIZE - dynamic files

2012-07-03 Thread Dan Fitzgerald


One rule of thumb is to make sure that you have an average of 10 or less items 
in each group. Going by that, you'd want a minimum mod of 130k or more. I've 
also noticed that files approach the "sweet spot" for minimizing overflow 
without having excessive empty groups when the total size is pretty nearly 
twice the data size.
 
The goal can vary according to your situation. I'm personally not all that 
afraid of making the modulus a little too large, as overflow is a pretty bad 
performance hit (overflow means at least two disk reads to retrieve your data, 
"badly" means at least 2 extra disk reads, and I've seen files where that was 
thousands (this file isn't that bad, but 20% of your data is forcing at least 
one extra disk read). Empty groups contribute to overhead on a sequential 
search, so you'd want to consider how often you do a sequential search on a 
file - usually, that's a pretty inefficient way to retrieve data, but, again, 
your mileage may vary. 
 
To me, 20% is too much overflow, and 114 empty groups is trivial; much less 
than 0.2%. I'd be tempted to go to 23 as a minimum Mod, just to see what it 
looks like there. That'll give you an average of 6 records per group, not 
unreasonably shallow, and it's likely to be a while before you have to resize 
again.
 
> From: cjausti...@hotmail.com
> To: u2-users@listserver.u2ug.org
> Date: Tue, 3 Jul 2012 15:23:23 -0500
> Subject: Re: [U2] RESIZE - dynamic files
> 
> 
> I guess what I need to know is what's an acceptable % of overflow for a 
> dynamic file? For example, when I change the SPLIT LOAD to 90% (while using 
> the calculated min modulus)
> I'm still left with ~ 20% of overflow (see below). Is 20% overflow considered 
> acceptable on average or should I keep tinkering with it to reach a lower 
> overflow %?
> 
> Correct me if I'm wrong but it seems the goal here is to REDUCE the overflow 
> % while not creating too many modulus (groups). 
> 
> Chris
> 
> 
> File name ..   GENACCTRN_POSTED
> Pathname ...   GENACCTRN_POSTED
> File type ..   DYNAMIC
> File style and revision    32BIT Revision 12
> Hashing Algorithm ..   GENERAL
> No. of groups (modulus)    105715 current ( minimum 103889, 114 empty,
> 21092 overflowed, 1452 badly )
> Number of records ..   1290469
> Large record size ..   3267 bytes
> Number of large records    180
> Group size .   4096 bytes
> Load factors ...   90% (split), 50% (merge) and 70% (actual)
> Total size .   522260480 bytes
> Total size of record data ..   287400239 bytes
> Total size of record IDs ...   21508521 bytes
> Unused space ...   213343528 bytes
> Total space for records    522252288 bytes
> 
> > From: r...@lynden.com
> > To: u2-users@listserver.u2ug.org
> > Date: Tue, 3 Jul 2012 13:10:43 -0700
> > Subject: Re: [U2] RESIZE - dynamic files
> > 
> > The split load is not affecting anything here, since it is more than the 
> > actual load.  What your overflow suggests is that you lower the split.load 
> > value to 70$% or below.  You could go ahead and set the merge.load to an 
> > arbitrarily low number ("1"), and it will probably never do a merge, which 
> > would be the same as specifying a minimum.modulus equal to "as large as it 
> > ever gets".  The exception to this is during file creation & clear.file,  
> > when the minimum.modulus value determines the initial disk allocation.  
> > Short of going to an arbitrarily large minimum.modulus, and a very low 
> > split.load, you are going to have some overflow (unless you have sequential 
> > keys & like sized records).
> > 
> > -Rick
> > 
> > -Original Message-
> > From: u2-users-boun...@listserver.u2ug.org 
> > [mailto:u2-users-boun...@listserver.u2ug.org] On Behalf Of Chris Austin
> > Sent: Tuesday, July 03, 2012 12:54 PM
> > To: u2-users@listserver.u2ug.org
> > Subject: Re: [U2] RESIZE - dynamic files
> > 
> > 
> > Using the formula below, and changing the split to 90% I get the following:
> > 
> > File name ..   GENACCTRN_POSTED
> > Pathname ...   GENACCTRN_POSTED
> > File type ..   DYNAMIC
> > File style and revision    32BIT Revision 12
> > Hashing Algorithm ..   GENERAL
> > No. of groups (modulus)    103889 current ( minimum 103889, 114 empty,
> > 22249 overflowed, 1764 badly )
> > Number of records ..   1290469
> > Large record size ..   3267 bytes
> > Number of large records    180
> > Group size .   4096 bytes
> > Load factors ...   90% (split), 50% (merge) and 72% (actual)
> > Total size .   519921664 bytes
> > Total size of record data ..   287400591 bytes
> > Total size of record IDs ...   21508497 bytes
> > Unused space ...   211004384 bytes
> > Total space

Re: [U2] Is there any way to tell if I am in an EXECUTE level and CAPTURING is turned on?

2012-07-03 Thread Keith Johnson [DATACOM]

I just noticed this in the help for QM's SYSTEM()

1000 Returns 1 if EXECUTE CAPTURING is in effect, 0 otherwise

Checked if this was an undocumented feature that Martin had reproduced; but no, 
it doesn't work in Universe.

Regards, Keith



___
U2-Users mailing list
U2-Users@listserver.u2ug.org
http://listserver.u2ug.org/mailman/listinfo/u2-users

Re: [U2] RESIZE - dynamic files


Dan,

I changed the MINIMUM.MODULUS to the value of 23 as you suggested and my 
Actual Load has really gone down (as well as overflow). See below for the 
results:

File name ..   GENACCTRN_POSTED
Pathname ...   GENACCTRN_POSTED
File type ..   DYNAMIC
File style and revision    32BIT Revision 12
Hashing Algorithm ..   GENERAL
No. of groups (modulus)    23 current ( minimum 23, 5263 empty,
3957 overflowed, 207 badly )
Number of records ..   1290469
Large record size ..   3267 bytes
Number of large records    180
Group size .   4096 bytes
Load factors ...   90% (split), 50% (merge) and 37% (actual)
Total size .   836235264 bytes
Total size of record data ..   287394719 bytes
Total size of record IDs ...   21508521 bytes
Unused space ...   527323832 bytes
Total space for records    836227072 bytes

My overflow is now @ 2% 
My Load is @ 37% (actual)

granted my empty groups are now up to almost 3% but I hope that won't be a big 
factor. How does this look?

Chris


> From: dangf...@hotmail.com
> To: u2-users@listserver.u2ug.org
> Date: Tue, 3 Jul 2012 16:57:34 -0400
> Subject: Re: [U2] RESIZE - dynamic files
> 
> 
> One rule of thumb is to make sure that you have an average of 10 or less 
> items in each group. Going by that, you'd want a minimum mod of 130k or more. 
> I've also noticed that files approach the "sweet spot" for minimizing 
> overflow without having excessive empty groups when the total size is pretty 
> nearly twice the data size.
>  
> The goal can vary according to your situation. I'm personally not all that 
> afraid of making the modulus a little too large, as overflow is a pretty bad 
> performance hit (overflow means at least two disk reads to retrieve your 
> data, "badly" means at least 2 extra disk reads, and I've seen files where 
> that was thousands (this file isn't that bad, but 20% of your data is forcing 
> at least one extra disk read). Empty groups contribute to overhead on a 
> sequential search, so you'd want to consider how often you do a sequential 
> search on a file - usually, that's a pretty inefficient way to retrieve data, 
> but, again, your mileage may vary. 
>  
> To me, 20% is too much overflow, and 114 empty groups is trivial; much less 
> than 0.2%. I'd be tempted to go to 23 as a minimum Mod, just to see what 
> it looks like there. That'll give you an average of 6 records per group, not 
> unreasonably shallow, and it's likely to be a while before you have to resize 
> again.
>  
> > From: cjausti...@hotmail.com
> > To: u2-users@listserver.u2ug.org
> > Date: Tue, 3 Jul 2012 15:23:23 -0500
> > Subject: Re: [U2] RESIZE - dynamic files
> > 
> > 
> > I guess what I need to know is what's an acceptable % of overflow for a 
> > dynamic file? For example, when I change the SPLIT LOAD to 90% (while using 
> > the calculated min modulus)
> > I'm still left with ~ 20% of overflow (see below). Is 20% overflow 
> > considered acceptable on average or should I keep tinkering with it to 
> > reach a lower overflow %?
> > 
> > Correct me if I'm wrong but it seems the goal here is to REDUCE the 
> > overflow % while not creating too many modulus (groups). 
> > 
> > Chris
> > 
> > 
> > File name ..   GENACCTRN_POSTED
> > Pathname ...   GENACCTRN_POSTED
> > File type ..   DYNAMIC
> > File style and revision    32BIT Revision 12
> > Hashing Algorithm ..   GENERAL
> > No. of groups (modulus)    105715 current ( minimum 103889, 114 empty,
> > 21092 overflowed, 1452 badly )
> > Number of records ..   1290469
> > Large record size ..   3267 bytes
> > Number of large records    180
> > Group size .   4096 bytes
> > Load factors ...   90% (split), 50% (merge) and 70% (actual)
> > Total size .   522260480 bytes
> > Total size of record data ..   287400239 bytes
> > Total size of record IDs ...   21508521 bytes
> > Unused space ...   213343528 bytes
> > Total space for records    522252288 bytes
> > 
> > > From: r...@lynden.com
> > > To: u2-users@listserver.u2ug.org
> > > Date: Tue, 3 Jul 2012 13:10:43 -0700
> > > Subject: Re: [U2] RESIZE - dynamic files
> > > 
> > > The split load is not affecting anything here, since it is more than the 
> > > actual load.  What your overflow suggests is that you lower the 
> > > split.load value to 70$% or below.  You could go ahead and set the 
> > > merge.load to an arbitrarily low number ("1"), and it will probably never 
> > > do a merge, which would be the same as specifying a minimum.modulus equal 
> > > to "as large as it ever gets".  The exception to this is during file 
> > > creation & clear.file,  when the minimum.modulus value determines the 
> > > init

Re: [U2] RESIZE - dynamic files

But the total size of your file is up 60%.  Reading in 60% more records in a 
full select of the file is going to be much slower than a few more overflows.


-Original Message-
From: u2-users-boun...@listserver.u2ug.org 
[mailto:u2-users-boun...@listserver.u2ug.org] On Behalf Of Chris Austin
Sent: Tuesday, July 03, 2012 2:15 PM
To: u2-users@listserver.u2ug.org
Subject: Re: [U2] RESIZE - dynamic files


Dan,

I changed the MINIMUM.MODULUS to the value of 23 as you suggested and my 
Actual Load has really gone down (as well as overflow). See below for the 
results:

File name ..   GENACCTRN_POSTED
Pathname ...   GENACCTRN_POSTED
File type ..   DYNAMIC
File style and revision    32BIT Revision 12
Hashing Algorithm ..   GENERAL
No. of groups (modulus)    23 current ( minimum 23, 5263 empty,
3957 overflowed, 207 badly )
Number of records ..   1290469
Large record size ..   3267 bytes
Number of large records    180
Group size .   4096 bytes
Load factors ...   90% (split), 50% (merge) and 37% (actual)
Total size .   836235264 bytes
Total size of record data ..   287394719 bytes
Total size of record IDs ...   21508521 bytes
Unused space ...   527323832 bytes
Total space for records    836227072 bytes

My overflow is now @ 2%
My Load is @ 37% (actual)

granted my empty groups are now up to almost 3% but I hope that won't be a big 
factor. How does this look?

Chris


> From: dangf...@hotmail.com
> To: u2-users@listserver.u2ug.org
> Date: Tue, 3 Jul 2012 16:57:34 -0400
> Subject: Re: [U2] RESIZE - dynamic files
>
>
> One rule of thumb is to make sure that you have an average of 10 or less 
> items in each group. Going by that, you'd want a minimum mod of 130k or more. 
> I've also noticed that files approach the "sweet spot" for minimizing 
> overflow without having excessive empty groups when the total size is pretty 
> nearly twice the data size.
>
> The goal can vary according to your situation. I'm personally not all that 
> afraid of making the modulus a little too large, as overflow is a pretty bad 
> performance hit (overflow means at least two disk reads to retrieve your 
> data, "badly" means at least 2 extra disk reads, and I've seen files where 
> that was thousands (this file isn't that bad, but 20% of your data is forcing 
> at least one extra disk read). Empty groups contribute to overhead on a 
> sequential search, so you'd want to consider how often you do a sequential 
> search on a file - usually, that's a pretty inefficient way to retrieve data, 
> but, again, your mileage may vary.
>
> To me, 20% is too much overflow, and 114 empty groups is trivial; much less 
> than 0.2%. I'd be tempted to go to 23 as a minimum Mod, just to see what 
> it looks like there. That'll give you an average of 6 records per group, not 
> unreasonably shallow, and it's likely to be a while before you have to resize 
> again.
>
> > From: cjausti...@hotmail.com
> > To: u2-users@listserver.u2ug.org
> > Date: Tue, 3 Jul 2012 15:23:23 -0500
> > Subject: Re: [U2] RESIZE - dynamic files
> >
> >
> > I guess what I need to know is what's an acceptable % of overflow for a 
> > dynamic file? For example, when I change the SPLIT LOAD to 90% (while using 
> > the calculated min modulus)
> > I'm still left with ~ 20% of overflow (see below). Is 20% overflow 
> > considered acceptable on average or should I keep tinkering with it to 
> > reach a lower overflow %?
> >
> > Correct me if I'm wrong but it seems the goal here is to REDUCE the 
> > overflow % while not creating too many modulus (groups).
> >
> > Chris
> >
> >
> > File name ..   GENACCTRN_POSTED
> > Pathname ...   GENACCTRN_POSTED
> > File type ..   DYNAMIC
> > File style and revision    32BIT Revision 12
> > Hashing Algorithm ..   GENERAL
> > No. of groups (modulus)    105715 current ( minimum 103889, 114 empty,
> > 21092 overflowed, 1452 badly )
> > Number of records ..   1290469
> > Large record size ..   3267 bytes
> > Number of large records    180
> > Group size .   4096 bytes
> > Load factors ...   90% (split), 50% (merge) and 70% (actual)
> > Total size .   522260480 bytes
> > Total size of record data ..   287400239 bytes
> > Total size of record IDs ...   21508521 bytes
> > Unused space ...   213343528 bytes
> > Total space for records    522252288 bytes
> >
> > > From: r...@lynden.com
> > > To: u2-users@listserver.u2ug.org
> > > Date: Tue, 3 Jul 2012 13:10:43 -0700
> > > Subject: Re: [U2] RESIZE - dynamic files
> > >
> > > The split load is not affecting anything here, since it is more than the 
> > > actual load.  What your overflow suggests is that you lower the 
> > >

Re: [U2] * GMX Spamverdacht * UniVerse LIST statement question [not-secure]

2012-07-03 Thread Kate Stanton

You can do just about anything by calling a subroutine from an I-type
dictionary item.
On 3 July 2012 05:41, Mecki Foerthmann  wrote:

> Since we don't have outer joins I would build a work file starting with
> selecting all employees, then select the calls and merge them together in a
> loop. How you do it depends on how you want to present the data.
> If you're happy with showing the calls as multi values you can build the
> records that way and if you like one row per call you should probably have
> one row per call.
> Then report on the work file.
> That is the easiest way I can see to include employees who didn't make any
> calls.
> I can also think of a few approaches involving stored parameters and
> dictionaries or a dictionary in the employee file that builds the call info
> in a subroutine as multivalues using an index by employee on the calls file.
> In any case you will most likely need to base your report on the employees
> file not the calls.
>
>
> On 02/07/2012 14:53, Hennessey, Mark F. wrote:
>
>> I need to do a UniVerse LIST statement that would only populate a column
>> if the contents met certain criteria.
>>
>> For example, suppose we have a file with details of telephone usage and
>> that 3 associated mulitvalued fields contain date call was made, duration
>> and if the call was a toll call. Is it possible to limit the output of the
>> "date call made" and associated columns to a date range without that being
>> a select criteria? If I were to do something like:
>>
>> LIST CALLS EMP.NAME EMP.LOCATION WITH DATE.CALL GE "2012-06-01" AND WITH
>> DATE.CALL LE "2012-06-30"  DURATION TOLL WITH @ID EQ '123456'
>>
>> I would get zero record if employee 123456 did not make any calls in
>> June. What I would like to see is the employer name and location returned
>> with the date, duration and toll columns empty. I'm trying to do this in a
>> LIST statement as it will be run by U2 Web Services (and for the time being
>> a subroutine is off the table...)
>>
>> Any advice, or an authoritative "NO, It can not be done" would be greatly
>> appreciated.
>>
>> Mark Hennessey
>> State of Connecticut
>> Department of Social Services
>> Information Technology Services
>> Child Support Systems
>> Voice: 860-424-5261
>> Fax: 860-424-4813
>>
>>
>>
>> CONFIDENTIAL INFORMATION: The information contained in this e-mail may be
>> confidential and protected from general disclosure. If the recipient or
>> reader of this e-mail is not the intended recipient or a person responsible
>> to receive this e-mail for the intended recipient, please do not
>> disseminate, distribute or copy it. If you received this e-mail in error,
>> please notify the sender by replying to this message and delete this e-mail
>> immediately. We will take immediate and appropriate action to see to it
>> that this mistake is corrected.[*LD*]
>> __**_
>> U2-Users mailing list
>> U2-Users@listserver.u2ug.org
>> http://listserver.u2ug.org/**mailman/listinfo/u2-users
>>
>>
> __**_
> U2-Users mailing list
> U2-Users@listserver.u2ug.org
> http://listserver.u2ug.org/**mailman/listinfo/u2-users
>



-- 
Kate Stanton
Walstan Systems Ltd
4 Kelmarna Ave, Herne Bay, Auckland 1011, New Zealand
Phone: + 64 9 360 5310  Mobile: + 64 21 400 486
Email: k...@walstan.com
___
U2-Users mailing list
U2-Users@listserver.u2ug.org
http://listserver.u2ug.org/mailman/listinfo/u2-users

Re: [U2] RESIZE - dynamic files

I should have said "60% more disk records", to be clear.

-Original Message-
From: u2-users-boun...@listserver.u2ug.org 
[mailto:u2-users-boun...@listserver.u2ug.org] On Behalf Of Rick Nuckolls
Sent: Tuesday, July 03, 2012 2:24 PM
To: 'U2 Users List'
Subject: Re: [U2] RESIZE - dynamic files

But the total size of your file is up 60%.  Reading in 60% more records in a 
full select of the file is going to be much slower than a few more overflows.


-Original Message-
From: u2-users-boun...@listserver.u2ug.org 
[mailto:u2-users-boun...@listserver.u2ug.org] On Behalf Of Chris Austin
Sent: Tuesday, July 03, 2012 2:15 PM
To: u2-users@listserver.u2ug.org
Subject: Re: [U2] RESIZE - dynamic files


Dan,

I changed the MINIMUM.MODULUS to the value of 23 as you suggested and my 
Actual Load has really gone down (as well as overflow). See below for the 
results:

File name ..   GENACCTRN_POSTED
Pathname ...   GENACCTRN_POSTED
File type ..   DYNAMIC
File style and revision    32BIT Revision 12
Hashing Algorithm ..   GENERAL
No. of groups (modulus)    23 current ( minimum 23, 5263 empty,
3957 overflowed, 207 badly )
Number of records ..   1290469
Large record size ..   3267 bytes
Number of large records    180
Group size .   4096 bytes
Load factors ...   90% (split), 50% (merge) and 37% (actual)
Total size .   836235264 bytes
Total size of record data ..   287394719 bytes
Total size of record IDs ...   21508521 bytes
Unused space ...   527323832 bytes
Total space for records    836227072 bytes

My overflow is now @ 2%
My Load is @ 37% (actual)

granted my empty groups are now up to almost 3% but I hope that won't be a big 
factor. How does this look?

Chris


> From: dangf...@hotmail.com
> To: u2-users@listserver.u2ug.org
> Date: Tue, 3 Jul 2012 16:57:34 -0400
> Subject: Re: [U2] RESIZE - dynamic files
>
>
> One rule of thumb is to make sure that you have an average of 10 or less 
> items in each group. Going by that, you'd want a minimum mod of 130k or more. 
> I've also noticed that files approach the "sweet spot" for minimizing 
> overflow without having excessive empty groups when the total size is pretty 
> nearly twice the data size.
>
> The goal can vary according to your situation. I'm personally not all that 
> afraid of making the modulus a little too large, as overflow is a pretty bad 
> performance hit (overflow means at least two disk reads to retrieve your 
> data, "badly" means at least 2 extra disk reads, and I've seen files where 
> that was thousands (this file isn't that bad, but 20% of your data is forcing 
> at least one extra disk read). Empty groups contribute to overhead on a 
> sequential search, so you'd want to consider how often you do a sequential 
> search on a file - usually, that's a pretty inefficient way to retrieve data, 
> but, again, your mileage may vary.
>
> To me, 20% is too much overflow, and 114 empty groups is trivial; much less 
> than 0.2%. I'd be tempted to go to 23 as a minimum Mod, just to see what 
> it looks like there. That'll give you an average of 6 records per group, not 
> unreasonably shallow, and it's likely to be a while before you have to resize 
> again.
>
> > From: cjausti...@hotmail.com
> > To: u2-users@listserver.u2ug.org
> > Date: Tue, 3 Jul 2012 15:23:23 -0500
> > Subject: Re: [U2] RESIZE - dynamic files
> >
> >
> > I guess what I need to know is what's an acceptable % of overflow for a 
> > dynamic file? For example, when I change the SPLIT LOAD to 90% (while using 
> > the calculated min modulus)
> > I'm still left with ~ 20% of overflow (see below). Is 20% overflow 
> > considered acceptable on average or should I keep tinkering with it to 
> > reach a lower overflow %?
> >
> > Correct me if I'm wrong but it seems the goal here is to REDUCE the 
> > overflow % while not creating too many modulus (groups).
> >
> > Chris
> >
> >
> > File name ..   GENACCTRN_POSTED
> > Pathname ...   GENACCTRN_POSTED
> > File type ..   DYNAMIC
> > File style and revision    32BIT Revision 12
> > Hashing Algorithm ..   GENERAL
> > No. of groups (modulus)    105715 current ( minimum 103889, 114 empty,
> > 21092 overflowed, 1452 badly )
> > Number of records ..   1290469
> > Large record size ..   3267 bytes
> > Number of large records    180
> > Group size .   4096 bytes
> > Load factors ...   90% (split), 50% (merge) and 70% (actual)
> > Total size .   522260480 bytes
> > Total size of record data ..   287400239 bytes
> > Total size of record IDs ...   21508521 bytes
> > Unused space ...   213343528 bytes
> > Total space for records    522252288 bytes
> >
> > > From: r...@l

Re: [U2] RESIZE - dynamic files


This is why I'm confused.. Is the goal here to reduce 'overflow' or to 
keep the 'Total size' of the disk down? If the goal is to keep the total
 disk size down then it would appear
you would want your actual load % a lot higher than 37%.. and then ignore 
'some' of the overflow..

Chris


> But the total size of your file is up 60%.  Reading in 60% more records in a 
> full select of the file is going to be much slower than a few more overflows.
> 
> 
> -Original Message-
> From: u2-users-boun...@listserver.u2ug.org 
> [mailto:u2-users-boun...@listserver.u2ug.org] On Behalf Of Chris Austin
> Sent: Tuesday, July 03, 2012 2:15 PM
> To: u2-users@listserver.u2ug.org
> Subject: Re: [U2] RESIZE - dynamic files
> 
> 
> Dan,
> 
> I changed the MINIMUM.MODULUS to the value of 23 as you suggested and my 
> Actual Load has really gone down (as well as overflow). See below for the 
> results:
> 
> File name ..   GENACCTRN_POSTED
> Pathname ...   GENACCTRN_POSTED
> File type ..   DYNAMIC
> File style and revision    32BIT Revision 12
> Hashing Algorithm ..   GENERAL
> No. of groups (modulus)    23 current ( minimum 23, 5263 empty,
> 3957 overflowed, 207 badly )
> Number of records ..   1290469
> Large record size ..   3267 bytes
> Number of large records    180
> Group size .   4096 bytes
> Load factors ...   90% (split), 50% (merge) and 37% (actual)
> Total size .   836235264 bytes
> Total size of record data ..   287394719 bytes
> Total size of record IDs ...   21508521 bytes
> Unused space ...   527323832 bytes
> Total space for records    836227072 bytes
> 
> My overflow is now @ 2%
> My Load is @ 37% (actual)
> 
> granted my empty groups are now up to almost 3% but I hope that won't be a 
> big factor. How does this look?
> 
> Chris

  
___
U2-Users mailing list
U2-Users@listserver.u2ug.org
http://listserver.u2ug.org/mailman/listinfo/u2-users

Re: [U2] RESIZE - dynamic files

2012-07-03 Thread Wjhonson

Disks get "bigger" much faster than the rate they get "faster".
So the overflow is the thing to minimize.

-Original Message-
From: Chris Austin 
To: u2-users 
Sent: Tue, Jul 3, 2012 2:38 pm
Subject: Re: [U2] RESIZE - dynamic files

his is why I'm confused.. Is the goal here to reduce 'overflow' or to 
eep the 'Total size' of the disk down? If the goal is to keep the total
disk size down then it would appear
ou would want your actual load % a lot higher than 37%.. and then ignore 'some' 
f the overflow..
Chris

 But the total size of your file is up 60%.  Reading in 60% more records in a 
ull select of the file is going to be much slower than a few more overflows.

 -Original Message-
 From: u2-users-boun...@listserver.u2ug.org 
[mailto:u2-users-boun...@listserver.u2ug.org] 
n Behalf Of Chris Austin
 Sent: Tuesday, July 03, 2012 2:15 PM
 To: u2-users@listserver.u2ug.org
 Subject: Re: [U2] RESIZE - dynamic files

 Dan,

 I changed the MINIMUM.MODULUS to the value of 23 as you suggested and my 
ctual Load has really gone down (as well as overflow). See below for the 
esults:

 File name ..   GENACCTRN_POSTED
 Pathname ...   GENACCTRN_POSTED
 File type ..   DYNAMIC
 File style and revision    32BIT Revision 12
 Hashing Algorithm ..   GENERAL
 No. of groups (modulus)    23 current ( minimum 23, 5263 empty,
 3957 overflowed, 207 badly )
 Number of records ..   1290469
 Large record size ..   3267 bytes
 Number of large records    180
 Group size .   4096 bytes
 Load factors ...   90% (split), 50% (merge) and 37% (actual)
 Total size .   836235264 bytes
 Total size of record data ..   287394719 bytes
 Total size of record IDs ...   21508521 bytes
 Unused space ...   527323832 bytes
 Total space for records    836227072 bytes

 My overflow is now @ 2%
 My Load is @ 37% (actual)

 granted my empty groups are now up to almost 3% but I hope that won't be a big 
actor. How does this look?

 Chris

__
2-Users mailing list
2-us...@listserver.u2ug.org
ttp://listserver.u2ug.org/mailman/listinfo/u2-users

___
U2-Users mailing list
U2-Users@listserver.u2ug.org
http://listserver.u2ug.org/mailman/listinfo/u2-users

Re: [U2] RESIZE - dynamic files

37% is a very low load.  Reading disk records takes much longer than parsing 
the records out of a disk record.  With variable record size and moderately 
poor hashing, overflow is inevitable.  So, do you want 80,000 extra groups, or 
20,000 overflow buffers? I would go with the smaller number.  But for the love 
of Knuth, do not set your split.load to 90% unless you have a perfectly hashed 
file with uniformly sized records.

-Original Message-
From: u2-users-boun...@listserver.u2ug.org 
[mailto:u2-users-boun...@listserver.u2ug.org] On Behalf Of Chris Austin
Sent: Tuesday, July 03, 2012 2:38 PM
To: u2-users@listserver.u2ug.org
Subject: Re: [U2] RESIZE - dynamic files


This is why I'm confused.. Is the goal here to reduce 'overflow' or to 
keep the 'Total size' of the disk down? If the goal is to keep the total
 disk size down then it would appear
you would want your actual load % a lot higher than 37%.. and then ignore 
'some' of the overflow..

Chris


> But the total size of your file is up 60%.  Reading in 60% more records in a 
> full select of the file is going to be much slower than a few more overflows.
> 
> 
> -Original Message-
> From: u2-users-boun...@listserver.u2ug.org 
> [mailto:u2-users-boun...@listserver.u2ug.org] On Behalf Of Chris Austin
> Sent: Tuesday, July 03, 2012 2:15 PM
> To: u2-users@listserver.u2ug.org
> Subject: Re: [U2] RESIZE - dynamic files
> 
> 
> Dan,
> 
> I changed the MINIMUM.MODULUS to the value of 23 as you suggested and my 
> Actual Load has really gone down (as well as overflow). See below for the 
> results:
> 
> File name ..   GENACCTRN_POSTED
> Pathname ...   GENACCTRN_POSTED
> File type ..   DYNAMIC
> File style and revision    32BIT Revision 12
> Hashing Algorithm ..   GENERAL
> No. of groups (modulus)    23 current ( minimum 23, 5263 empty,
> 3957 overflowed, 207 badly )
> Number of records ..   1290469
> Large record size ..   3267 bytes
> Number of large records    180
> Group size .   4096 bytes
> Load factors ...   90% (split), 50% (merge) and 37% (actual)
> Total size .   836235264 bytes
> Total size of record data ..   287394719 bytes
> Total size of record IDs ...   21508521 bytes
> Unused space ...   527323832 bytes
> Total space for records    836227072 bytes
> 
> My overflow is now @ 2%
> My Load is @ 37% (actual)
> 
> granted my empty groups are now up to almost 3% but I hope that won't be a 
> big factor. How does this look?
> 
> Chris

  
___
U2-Users mailing list
U2-Users@listserver.u2ug.org
http://listserver.u2ug.org/mailman/listinfo/u2-users
___
U2-Users mailing list
U2-Users@listserver.u2ug.org
http://listserver.u2ug.org/mailman/listinfo/u2-users

Re: [U2] RESIZE - dynamic files

2012-07-03 Thread Susan Lynch

Chris,

This is why file-sizing is something that requires careful thought.  As
some of the other responders have indicated, sometimes you want to keep
overflow to a minimum (because accessing individual records that are in
overflow takes extra disk reads, which slow down your system, and adding
new records to a group that is already in overflow will inevitably be
slower than adding a new record to a group which is not in overflow), and
sometimes you don't (eg if you have a file that is primarily read in a
sequential fashion where you do a Basic SELECT, and then loop through the
file reading every single record).   Because most of the files that I have
supported in my career have been read and written primarily as
single-record reads, I have always chosen to minimize overflow as my
default criteria, and only sized things for sequential reads when the file
is rarely written, rarely read as anything but a 'read them all in no
particular order' fashion, and that happens rarely in my experience.
However, as other responders have written, 'your mileage may vary'!

Look at how the file is used.  Look at what resources you have.  Then
decide...


Susan M. Lynch
F. W. Davison & Company, Inc.
-Original Message-
From: u2-users-boun...@listserver.u2ug.org
[mailto:u2-users-boun...@listserver.u2ug.org] On Behalf Of Chris Austin
Sent: 07/03/2012 5:38 PM
To: u2-users@listserver.u2ug.org
Subject: Re: [U2] RESIZE - dynamic files


This is why I'm confused.. Is the goal here to reduce 'overflow' or to
keep the 'Total size' of the disk down? If the goal is to keep the total
 disk size down then it would appear
you would want your actual load % a lot higher than 37%.. and then ignore
'some' of the overflow..

Chris


> But the total size of your file is up 60%.  Reading in 60% more records
in a full select of the file is going to be much slower than a few more
overflows.
>
>
> -Original Message-
> From: u2-users-boun...@listserver.u2ug.org
[mailto:u2-users-boun...@listserver.u2ug.org] On Behalf Of Chris Austin
> Sent: Tuesday, July 03, 2012 2:15 PM
> To: u2-users@listserver.u2ug.org
> Subject: Re: [U2] RESIZE - dynamic files
>
>
> Dan,
>
> I changed the MINIMUM.MODULUS to the value of 23 as you suggested
and my Actual Load has really gone down (as well as overflow). See below
for the results:
>
> File name ..   GENACCTRN_POSTED
> Pathname ...   GENACCTRN_POSTED
> File type ..   DYNAMIC
> File style and revision    32BIT Revision 12
> Hashing Algorithm ..   GENERAL
> No. of groups (modulus)    23 current ( minimum 23, 5263
empty,
> 3957 overflowed, 207 badly )
> Number of records ..   1290469
> Large record size ..   3267 bytes
> Number of large records    180
> Group size .   4096 bytes
> Load factors ...   90% (split), 50% (merge) and 37% (actual)
> Total size .   836235264 bytes
> Total size of record data ..   287394719 bytes
> Total size of record IDs ...   21508521 bytes
> Unused space ...   527323832 bytes
> Total space for records    836227072 bytes
>
> My overflow is now @ 2%
> My Load is @ 37% (actual)
>
> granted my empty groups are now up to almost 3% but I hope that won't be
a big factor. How does this look?
>
> Chris


___
U2-Users mailing list
U2-Users@listserver.u2ug.org
http://listserver.u2ug.org/mailman/listinfo/u2-users
___
U2-Users mailing list
U2-Users@listserver.u2ug.org
http://listserver.u2ug.org/mailman/listinfo/u2-users

Re: [U2] RESIZE - dynamic files


I set the split load based on what Dan suggested:

"I'd take the merge down a little, to maybe 30% or even less, and maybe knock 
the split up a bit - say, 90% - to cut down on the splitting."

I thought this would cut down on splitting. Is there a certain formula, or way 
to calculate the split.load? What should my SPLIT.LOAD be around,
and how do you come up with that %?

Chris

> From: r...@lynden.com
> To: u2-users@listserver.u2ug.org
> Date: Tue, 3 Jul 2012 14:45:28 -0700
> Subject: Re: [U2] RESIZE - dynamic files
> 
> 37% is a very low load.  Reading disk records takes much longer than parsing 
> the records out of a disk record.  With variable record size and moderately 
> poor hashing, overflow is inevitable.  So, do you want 80,000 extra groups, 
> or 20,000 overflow buffers? I would go with the smaller number.  But for the 
> love of Knuth, do not set your split.load to 90% unless you have a perfectly 
> hashed file with uniformly sized records.
> 
> -Original Message-
> From: u2-users-boun...@listserver.u2ug.org 
> [mailto:u2-users-boun...@listserver.u2ug.org] On Behalf Of Chris Austin
> Sent: Tuesday, July 03, 2012 2:38 PM
> To: u2-users@listserver.u2ug.org
> Subject: Re: [U2] RESIZE - dynamic files
> 
> 
> This is why I'm confused.. Is the goal here to reduce 'overflow' or to 
> keep the 'Total size' of the disk down? If the goal is to keep the total
>  disk size down then it would appear
> you would want your actual load % a lot higher than 37%.. and then ignore 
> 'some' of the overflow..
> 
> Chris
> 
> 
> > But the total size of your file is up 60%.  Reading in 60% more records in 
> > a full select of the file is going to be much slower than a few more 
> > overflows.
> > 
> > 
> > -Original Message-
> > From: u2-users-boun...@listserver.u2ug.org 
> > [mailto:u2-users-boun...@listserver.u2ug.org] On Behalf Of Chris Austin
> > Sent: Tuesday, July 03, 2012 2:15 PM
> > To: u2-users@listserver.u2ug.org
> > Subject: Re: [U2] RESIZE - dynamic files
> > 
> > 
> > Dan,
> > 
> > I changed the MINIMUM.MODULUS to the value of 23 as you suggested and 
> > my Actual Load has really gone down (as well as overflow). See below for 
> > the results:
> > 
> > File name ..   GENACCTRN_POSTED
> > Pathname ...   GENACCTRN_POSTED
> > File type ..   DYNAMIC
> > File style and revision    32BIT Revision 12
> > Hashing Algorithm ..   GENERAL
> > No. of groups (modulus)    23 current ( minimum 23, 5263 empty,
> > 3957 overflowed, 207 badly )
> > Number of records ..   1290469
> > Large record size ..   3267 bytes
> > Number of large records    180
> > Group size .   4096 bytes
> > Load factors ...   90% (split), 50% (merge) and 37% (actual)
> > Total size .   836235264 bytes
> > Total size of record data ..   287394719 bytes
> > Total size of record IDs ...   21508521 bytes
> > Unused space ...   527323832 bytes
> > Total space for records    836227072 bytes
> > 
> > My overflow is now @ 2%
> > My Load is @ 37% (actual)
> > 
> > granted my empty groups are now up to almost 3% but I hope that won't be a 
> > big factor. How does this look?
> > 
> > Chris
> 
> 
> ___
> U2-Users mailing list
> U2-Users@listserver.u2ug.org
> http://listserver.u2ug.org/mailman/listinfo/u2-users
> ___
> U2-Users mailing list
> U2-Users@listserver.u2ug.org
> http://listserver.u2ug.org/mailman/listinfo/u2-users
  
___
U2-Users mailing list
U2-Users@listserver.u2ug.org
http://listserver.u2ug.org/mailman/listinfo/u2-users

[U2] Account.File.Stats ALL

2012-07-03 Thread Wjhonson


When you execute the Account.File.Stats command in Universe, with the ALL 
directive, the code executes the following command to make it's selection

SSELECT VOC WITH TYPE = "F" AND F2 NOT.MATCHING "'I_'..." AND F2 NOT.MATCHING 
"'../'..." AND F2 NOT.MATCHING "'..\'..." AND F2 NOT.MATCHING "'/'..." AND F2 
NOT.MATCHING "'\'..." AND F2 NOT.MATCHING "1A:'\'..." AND F2 # "." AND F2 # ""


___
U2-Users mailing list
U2-Users@listserver.u2ug.org
http://listserver.u2ug.org/mailman/listinfo/u2-users

Re: [U2] RESIZE - dynamic files

Unless the minimum modulus is configured high enough to artificially lower the 
actual load, the actual load will rise to the designated split.load as the file 
grows. The split.load indicates nothing about the specific load of any given 
group; so if it is set to 90%, then on average, each group will be 90% full, 
and adding a (400byte) record to a group will send it into overflow, but since 
400 bytes is a trivial percentage of your overall file load, many groups will 
be overflowed before the total load factor exceeds 90%.  

Okay, there is a slight distortion with the numbers there, but the idea is that 
all buckets are not loaded equally, so if the average is "almost full" the 
reality is "many overflowed".

-Original Message-
From: u2-users-boun...@listserver.u2ug.org 
[mailto:u2-users-boun...@listserver.u2ug.org] On Behalf Of Chris Austin
Sent: Tuesday, July 03, 2012 2:52 PM
To: u2-users@listserver.u2ug.org
Subject: Re: [U2] RESIZE - dynamic files


I set the split load based on what Dan suggested:

"I'd take the merge down a little, to maybe 30% or even less, and maybe knock 
the split up a bit - say, 90% - to cut down on the splitting."

I thought this would cut down on splitting. Is there a certain formula, or way 
to calculate the split.load? What should my SPLIT.LOAD be around,
and how do you come up with that %?

Chris

> From: r...@lynden.com
> To: u2-users@listserver.u2ug.org
> Date: Tue, 3 Jul 2012 14:45:28 -0700
> Subject: Re: [U2] RESIZE - dynamic files
> 
> 37% is a very low load.  Reading disk records takes much longer than parsing 
> the records out of a disk record.  With variable record size and moderately 
> poor hashing, overflow is inevitable.  So, do you want 80,000 extra groups, 
> or 20,000 overflow buffers? I would go with the smaller number.  But for the 
> love of Knuth, do not set your split.load to 90% unless you have a perfectly 
> hashed file with uniformly sized records.
> 
> -Original Message-
> From: u2-users-boun...@listserver.u2ug.org 
> [mailto:u2-users-boun...@listserver.u2ug.org] On Behalf Of Chris Austin
> Sent: Tuesday, July 03, 2012 2:38 PM
> To: u2-users@listserver.u2ug.org
> Subject: Re: [U2] RESIZE - dynamic files
> 
> 
> This is why I'm confused.. Is the goal here to reduce 'overflow' or to 
> keep the 'Total size' of the disk down? If the goal is to keep the total
>  disk size down then it would appear
> you would want your actual load % a lot higher than 37%.. and then ignore 
> 'some' of the overflow..
> 
> Chris
> 
> 
> > But the total size of your file is up 60%.  Reading in 60% more records in 
> > a full select of the file is going to be much slower than a few more 
> > overflows.
> > 
> > 
> > -Original Message-
> > From: u2-users-boun...@listserver.u2ug.org 
> > [mailto:u2-users-boun...@listserver.u2ug.org] On Behalf Of Chris Austin
> > Sent: Tuesday, July 03, 2012 2:15 PM
> > To: u2-users@listserver.u2ug.org
> > Subject: Re: [U2] RESIZE - dynamic files
> > 
> > 
> > Dan,
> > 
> > I changed the MINIMUM.MODULUS to the value of 23 as you suggested and 
> > my Actual Load has really gone down (as well as overflow). See below for 
> > the results:
> > 
> > File name ..   GENACCTRN_POSTED
> > Pathname ...   GENACCTRN_POSTED
> > File type ..   DYNAMIC
> > File style and revision    32BIT Revision 12
> > Hashing Algorithm ..   GENERAL
> > No. of groups (modulus)    23 current ( minimum 23, 5263 empty,
> > 3957 overflowed, 207 badly )
> > Number of records ..   1290469
> > Large record size ..   3267 bytes
> > Number of large records    180
> > Group size .   4096 bytes
> > Load factors ...   90% (split), 50% (merge) and 37% (actual)
> > Total size .   836235264 bytes
> > Total size of record data ..   287394719 bytes
> > Total size of record IDs ...   21508521 bytes
> > Unused space ...   527323832 bytes
> > Total space for records    836227072 bytes
> > 
> > My overflow is now @ 2%
> > My Load is @ 37% (actual)
> > 
> > granted my empty groups are now up to almost 3% but I hope that won't be a 
> > big factor. How does this look?
> > 
> > Chris
> 
> 
> ___
> U2-Users mailing list
> U2-Users@listserver.u2ug.org
> http://listserver.u2ug.org/mailman/listinfo/u2-users
> ___
> U2-Users mailing list
> U2-Users@listserver.u2ug.org
> http://listserver.u2ug.org/mailman/listinfo/u2-users
  
___
U2-Users mailing list
U2-Users@listserver.u2ug.org
http://listserver.u2ug.org/mailman/listinfo/u2-users
___
U2-Users mailing list
U2-Users@listserver.u2ug.org
http://listserver.u2ug.org/mailman/listinf

Re: [U2] RESIZE - dynamic files

2012-07-03 Thread Charles Stevenson


Chris,
Let's back way up.   I take it your original question is a general one,  
not specific to one poorly performing problematic file.  Is that right?


If so, generally speaking, you just don't get a lot out of fine-tuning 
dynamic files.
Tweaking the default parameters doesn't usually make a whole lot of 
difference.

Several people have said something similar in this thread.

Other than deciding which hashing algorithm,  I generally use the 
defaults and only tweak things once the file proves problematic, which 
usually means slow I/O.


When a problem erupts, look carefully at how that specific file is used, 
as Susan & others have said.   You might get hold of Fitzgerald&Long's 
paper on how dynamic files work.  If you understand the fundamentals, 
you'll understand how to attack your problem file, applying the ideas 
Rick & others have talked about here.


You may go several years without having to resort to that.

Chuck Stevenson


On 7/2/2012 2:22 PM, Chris Austin wrote:

I was wondering if anyone had instructions on RESIZE with a dynamic file? For 
example I have a file called 'TEST_FILE'
with the following:

01 ANALYZE.FILE TEST_FILE
File name ..   TEST_FILE
Pathname ...   TEST_FILE
File type ..   DYNAMIC
File style and revision    32BIT Revision 12
Hashing Algorithm ..   GENERAL
No. of groups (modulus)    83261 current ( minimum 31 )
Large record size ..   3267 bytes
Group size .   4096 bytes
Load factors ...   80% (split), 50% (merge) and 80% (actual)
Total size .   450613248 bytes

How do you calculate what the modulus and separation should be? I can't use 
HASH.HELP on a type 30 file to see the recommended settings
so I was wondering how best you figure out the file RESIZE.

Thanks,

Chris



___
U2-Users mailing list
U2-Users@listserver.u2ug.org
http://listserver.u2ug.org/mailman/listinfo/u2-users

Re: [U2] RESIZE - dynamic files

From the System Description manual:

Important Considerations

Dynamic files are meant to make file management easier for users. The default
parameters are set so that most dynamic files work efficiently. If you decide 
to change
the parameters of a dynamic file, keep the following considerations in mind:

􀂄 Use the SEQ.NUM hashing algorithm only when your record IDs are
numeric, sequential, and consecutive. Nonconsecutive numbers should not
be hashed using the SEQ.NUM hashing algorithm.

􀂄 Use a group size of 2 only if you expect the average record size to be larger
than 1000 bytes. If your record size is larger than 2000 bytes, consider using
a nonhashed file—type 1 or 19.

􀂄 Large record size should generally not be changed. Storing the data of a
large record in the overflow buffer causes that data not to be included in the
split and merge calculations. Also, the extra data length does not slow access
to subsequent records. By choosing a large record size of 0%, all the records
are considered large. In this case, record IDs can be accessed extremely
quickly by commands such as SELECT, but access to the actual data is
much less efficient.

􀂄 A small split load causes less data to be stored in each group buffer, 
resulting
in faster access time and less overflow at the expense of requiring extra
memory. A large split load causes more data to be stored in each group
buffer, resulting in better use of memory at the expense of slower access
time and more overflow. A split load of 100% disables splits.

􀂄 The gap between merge load and split load should be large enough so that
splits and merges do not occur too frequently. The split and merge processes
take a significant amount of processing time. If you make the merge load too
small, memory usage can be very poor. Also, selection time is increased
when record IDs are distributed in more groups than are needed. A merge
load of 0% disables merges.

􀂄 Consider increasing the minimum modulo if you intend to add a lot of initial
data to the file. Much data-entry time can be saved by avoiding the initial
splits that can occur if you enter a lot of initial data. You may want to
readjust this value after

-Original Message-
From: u2-users-boun...@listserver.u2ug.org 
[mailto:u2-users-boun...@listserver.u2ug.org] On Behalf Of Charles Stevenson
Sent: Tuesday, July 03, 2012 3:34 PM
To: U2 Users List
Subject: Re: [U2] RESIZE - dynamic files

Chris,
Let's back way up.   I take it your original question is a general one,  
not specific to one poorly performing problematic file.  Is that right?

If so, generally speaking, you just don't get a lot out of fine-tuning 
dynamic files.
Tweaking the default parameters doesn't usually make a whole lot of 
difference.
Several people have said something similar in this thread.

Other than deciding which hashing algorithm,  I generally use the 
defaults and only tweak things once the file proves problematic, which 
usually means slow I/O.

When a problem erupts, look carefully at how that specific file is used, 
as Susan & others have said.   You might get hold of Fitzgerald&Long's 
paper on how dynamic files work.  If you understand the fundamentals, 
you'll understand how to attack your problem file, applying the ideas 
Rick & others have talked about here.

You may go several years without having to resort to that.

Chuck Stevenson

On 7/2/2012 2:22 PM, Chris Austin wrote:
> I was wondering if anyone had instructions on RESIZE with a dynamic file? For 
> example I have a file called 'TEST_FILE'
> with the following:
>
> 01 ANALYZE.FILE TEST_FILE
> File name ..   TEST_FILE
> Pathname ...   TEST_FILE
> File type ..   DYNAMIC
> File style and revision    32BIT Revision 12
> Hashing Algorithm ..   GENERAL
> No. of groups (modulus)    83261 current ( minimum 31 )
> Large record size ..   3267 bytes
> Group size .   4096 bytes
> Load factors ...   80% (split), 50% (merge) and 80% (actual)
> Total size .   450613248 bytes
>
> How do you calculate what the modulus and separation should be? I can't use 
> HASH.HELP on a type 30 file to see the recommended settings
> so I was wondering how best you figure out the file RESIZE.
>
> Thanks,
>
> Chris
>

___
U2-Users mailing list
U2-Users@listserver.u2ug.org
http://listserver.u2ug.org/mailman/listinfo/u2-users
___
U2-Users mailing list
U2-Users@listserver.u2ug.org
http://listserver.u2ug.org/mailman/listinfo/u2-users

Re: [U2] Running XLr8 Tools inside U2 DBTools & new Eclipse release

2012-07-03 Thread Brian Whitehorn

I don't have BDT or other Rocket tools installed to confirm, but would 
specifying the same "-data workspacepath" in the arguments for each achieve 
what you're after ?
(via: 
http://help.eclipse.org/indigo/index.jsp?topic=%2Forg.eclipse.platform.doc.user%2Ftasks%2Frunning_eclipse.htm)

Regards,
Brian.

-Original Message-
From: u2-users-boun...@listserver.u2ug.org 
[mailto:u2-users-boun...@listserver.u2ug.org] On Behalf Of Doug Averch
Sent: Wednesday, 4 July 2012 3:23 AM
To: U2 Users List
Subject: Re: [U2] Running XLr8 Tools inside U2 DBTools & new Eclipse release

Hi Dan and others:

I finished testing all of the Rocket tools.  Turns out the only one that really 
works as a plug-in platform is BDT for U2logic plug-ins.  The other products 
are crippled versions of Eclipse with limited menus by design.

Another big problem is that Rocket's Eclipse products do not share work spaces, 
so if you start BDT your work space is in that directory.  If you start U2Admin 
the work space is in that directory.  Unfortunately this situation is very 
messy for U2logic, a Rocket VAR, to support.  I'm going to have fun explaining 
this to the first client that asks why the programs don't share the server 
names.

Regards,
Doug
www.u2logic.com
___
U2-Users mailing list
U2-Users@listserver.u2ug.org
http://listserver.u2ug.org/mailman/listinfo/u2-users
--
Message  protected by DealerGuard: e-mail anti-virus, anti-spam and content 
filtering.
http://www.pentanasolutions.com

Click here to report this message as spam:
https://login.mailguard.com.au/report/1F4dy0mK0d/4ZdT5DKh528rfiCd3YgdAP/0.204


This email and any attachments to it are confidential.
You must not use, disclose or act on the email if you are not the intended
recipient.  Liability limited by a scheme approved under Professional
Standards Legislation.
___
U2-Users mailing list
U2-Users@listserver.u2ug.org
http://listserver.u2ug.org/mailman/listinfo/u2-users

Re: [U2] Running XLr8 Tools inside U2 DBTools & new Eclipse release