Re: Duplicate CP Monitor records TCPIP

[Berry van Sleeuwen]

I guess some more testing is required. At least i'm not the only one to 
be supprised. Indeed, a pmr sounds about right at this point.


I guess a test would be pretty easy, restart (logoff/logon) one of the 
TCPIP stacks and see if it produces one record. Next fiddle with NETSTAT 
OBEY and see if the second record appears. Your remark on DIAG 0xDC does 
point in this direction. I also discussed with a collegue and he 
confirmed the NETSTAT OBEY at least once after the last IPL.


Thank you all for the help.

Regards, Berry.

Op 14-06-11 02:49, Alan Altmark schreef:

On Monday, 06/13/2011 at 07:46 EDT, Berry van Sleeuwen
  wrote:


I think so but I'm not quite sure. We use the NETSTAT OBEY sometimes,
usually for stop or start of a link. For instance NETSTAT OBEY START

DEVHW1.

The only way I could think of to get the same record more than once is to
declare it more than once to CP.  (DIAG 0xDC doesn't appear to have a
"this buffer has been previously declared" return code.)

It would be interesting to know if the number of extra LCB (0x05) monitor
records is in any way related to the number of times you START the device
or re-define any of the LINK attributes.

I agree with Bill; you need a PMR to poke at it further.

Alan Altmark

z/VM and Linux on System z Consultant
IBM System Lab Services and Training
ibm.com/systems/services/labservices
office: 607.429.3323
mobile; 607.321.7556
alan_altm...@us.ibm.com
IBM Endicott

Re: Duplicate CP Monitor records TCPIP

[Alan Altmark]

On Monday, 06/13/2011 at 07:46 EDT, Berry van Sleeuwen 
 wrote:

> I think so but I'm not quite sure. We use the NETSTAT OBEY sometimes,
> usually for stop or start of a link. For instance NETSTAT OBEY START 
DEVHW1.

The only way I could think of to get the same record more than once is to 
declare it more than once to CP.  (DIAG 0xDC doesn't appear to have a 
"this buffer has been previously declared" return code.)

It would be interesting to know if the number of extra LCB (0x05) monitor 
records is in any way related to the number of times you START the device 
or re-define any of the LINK attributes.

I agree with Bill; you need a PMR to poke at it further.

Alan Altmark

z/VM and Linux on System z Consultant
IBM System Lab Services and Training 
ibm.com/systems/services/labservices 
office: 607.429.3323
mobile; 607.321.7556
alan_altm...@us.ibm.com
IBM Endicott

Re: Duplicate CP Monitor records TCPIP

[Bill Bitner]

Hi Berry, I'm out of ideas without looking at data. You
may want to open a PMR.

Bill Bitner - z/VM Customer Focus and Care - IBM Endicott - 607-429-3286

Re: Duplicate CP Monitor records TCPIP

[Berry van Sleeuwen]

Alan,

I think so but I'm not quite sure. We use the NETSTAT OBEY sometimes, 
usually for stop or start of a link. For instance NETSTAT OBEY START DEVHW1.


Regards, Berry.

Op 13-06-11 04:01, Alan Altmark schreef:

On Fri, 10 Jun 2011 09:43:52 -0500, Berry van Sleeuwen
  wrote:



The IP stack hasn't been stopped, at least as far as I know. The last time
the stacks were stopped was due to a VM IPL.

If indeed multiple buffers were created, would it make sense that the same
data is reported in all buffers? I would expect, if new buffers were created
for the devices the data in the newest buffers would be incremented with the
new data and the old buffers would stay on their old values.

Berry, has OBEYFILE or NETSTAT OBEY been used against this stack?

Alan Altmark
IBM

Re: Duplicate CP Monitor records TCPIP

[Alan Altmark]

On Fri, 10 Jun 2011 09:43:52 -0500, Berry van Sleeuwen
 wrote:


>The IP stack hasn't been stopped, at least as far as I know. The last ti
me
>the stacks were stopped was due to a VM IPL.
>
>If indeed multiple buffers were created, would it make sense that the sa
me
>data is reported in all buffers? I would expect, if new buffers were cre
ated
>for the devices the data in the newest buffers would be incremented with
 the
>new data and the old buffers would stay on their old values.

Berry, has OBEYFILE or NETSTAT OBEY been used against this stack? 

Alan Altmark
IBM

Re: Duplicate CP Monitor records TCPIP

[Berry van Sleeuwen]

Hi Bill,

The IP stack hasn't been stopped, at least as far as I know. The last tim
e
the stacks were stopped was due to a VM IPL.

If indeed multiple buffers were created, would it make sense that the sam
e
data is reported in all buffers? I would expect, if new buffers were crea
ted
for the devices the data in the newest buffers would be incremented with 
the
new data and the old buffers would stay on their old values.

Regards, Berry.

Re: Duplicate CP Monitor records TCPIP

[Bill Bitner]

If these are the result of multiple links, I would think the link
names would be different. This is a long shot, but is it possible
that this TCP/IP stopped unexpectedly at some point in time and
was restarted without an IPL or Logoff/Logon?
The Stack identifies which buffers the CP monitor will collect
by issuing a diagnose x'DC' during intialization. It's almost like
multiple bufferes got created.

Bill Bitner - z/VM Customer Focus and Care - IBM Endicott - 607-429-3286

Re: Duplicate CP Monitor records TCPIP

[Berry van Sleeuwen]

Multiple links? You mean on the OSA or within the IP stack?

Yes, multiple links on an OSA. In LPAR1 we have one IP stack (EE00-EE02) 
and a VSWITCH (EE04-EE06) on the OSA. In LPAR2 we have two IP stacks on 
the same OSA (EE00--EE06). The customer stack also has a backup on a 
second OSA (EF00-EF02).


Within the IP stack there is one link defined for a device.

Regards, Berry.

Op 09-06-11 22:14, Alan Altmark schreef:

On Thursday, 06/09/2011 at 10:06 EDT, Berry van Sleeuwen
  wrote:


What can cause the CP monitor to have these duplicate records?

A type 0x05 record is, I think, produced for each LCB (Link Control Block)
in the stack.  Does the OSA device have multiple LINKs?

Alan Altmark

z/VM and Linux on System z Consultant
IBM System Lab Services and Training
ibm.com/systems/services/labservices
office: 607.429.3323
mobile; 607.321.7556
alan_altm...@us.ibm.com
IBM Endicott

Re: Duplicate CP Monitor records TCPIP

[Berry van Sleeuwen]

Hi Scott,

Yes they are the same, same VM levels, same basic setup for all VM's and 
IP stacks. In one case, our local IP stack, even uses the same osa 
device. In one LPAR it produces just one record per minute, in the 
second LPAR it produces three records.


And only one virtual CPU. But for these records CPU is not an issue.

Regards, Berry.

Op 09-06-11 21:38, Scott Rohling schreef:

Hmmm..   some questions then:

-  Are the z/VM and TCPIP levels all the same?
-  Are the OSA's attached to TCPIP the same way (DEDICATE in 
directory, via a DTCPARMS, or?)

-  Are the OSA cards the same..  defined the same?
-  Any NICDEF's defined to TCPIP on this LPAR? (though I would think 
it would show as a different OSA )

-  Could a different ports be being used on the same OSA?

You've likely thought of this stuff already - just thinking of stuff I 
would check.   Can't think of any reason offhand.  Well - maybe one: 
 does TCPIP on this LPAR have 2 virtual CPU's defined?   Seems like 
the number of CPU's can influence, but I'm probably thinking of 
accounting records..


Scott Rohling

On Thu, Jun 9, 2011 at 11:56 AM, Berry van Sleeuwen 
mailto:berry.vansleeu...@xs4all.nl>> wrote:


Hi Scott,

Yes, the records are identical. After a SORT UNIQUE the duplicate
records are discarded. The only thing that is not identical is the
TOD field. The second (and third) record are produced a few
miliseconds after the first. When I discard the miliseconds the
records themselves are truly identical. That leads me to believe
that these are seperatly created records.

The record produced by the TCPIP stack contains data from the OSA
device, we are interested in the number of bytes in and out for
the device during the minute interval.

As I understand it there is a buffer for each device. So the OSA
devices each have a record and the CTC devices also have a record.
And indeed that is what we get on other LPARs. Only on this
particular LPAR we have more than one record. And only for the
OSA's, not for the CTC's.

Regards, Berry.

Op 09-06-11 16:51, Scott Rohling schreef:

Just poking around found this description of the monitor record:

DESCRIPTIVE NAME - Monitor Sample Record
Domain 10 - Appldata domain
Record 2  - Application Data Sample Record
 DESCRIPTION - Application data as found in the
application-defined
   buffer at the time of this sample interval. A
   separate record is generated for each buffer
   declared by the virtual machine(s) via the Diagnose
   'DC' START operation.
Seems to imply a separate record is generated for each buffer
declared..  I'm not sure what the application data sample is
in this case.  Are the records truly duplicates?  (the whole
record is absolutely identical, displayable chars or not)  
Perhaps the application data needs to be appended to come up

with the entire field?
Scott Rohling

On Thu, Jun 9, 2011 at 8:04 AM, Berry van Sleeuwen
mailto:berry.vansleeu...@xs4all.nl>
>> wrote:

   Hi listers,

   I am looking at processing the userrecords in the CP
MONITOR for
   the TCPIP
   stacks. These are in CP MONITOR domain 0A record 02. The
data is
   collected
   though a STARMON stage, basically "PIPE STARMON | locate
 | >
   fielid".

   Most LPARs have records like I'd expect them but I have
found that
   in one
   LPAR we have duplicate records for subrecordtype 05 for each of
   the the OSA
   devices.

   The CTC's for VSE systems have one record per minute for
each VSE
   system.
   The OSA links have 2 identical records (apart from a slight
   difference in
   timestamp, a few miliseconds apart). A second IP stack even
writes
   3 records
   for every minute.

   So basically we see:
   00:01 VSE1
   00:01 VSE2
   00:01 OSA1
   00:01 OSA2
   00:01 OSA1
   00:01 OSA2
   00:02 ...

   What can cause the CP monitor to have these duplicate records?

   TIA, Berry.

Re: Duplicate CP Monitor records TCPIP

[Alan Altmark]

On Thursday, 06/09/2011 at 10:06 EDT, Berry van Sleeuwen 
 wrote:

> What can cause the CP monitor to have these duplicate records?

A type 0x05 record is, I think, produced for each LCB (Link Control Block) 
in the stack.  Does the OSA device have multiple LINKs? 

Alan Altmark

z/VM and Linux on System z Consultant
IBM System Lab Services and Training 
ibm.com/systems/services/labservices 
office: 607.429.3323
mobile; 607.321.7556
alan_altm...@us.ibm.com
IBM Endicott

Re: Duplicate CP Monitor records TCPIP

[Scott Rohling]

Hmmm..   some questions then:

-  Are the z/VM and TCPIP levels all the same?
-  Are the OSA's attached to TCPIP the same way (DEDICATE in directory, via
a DTCPARMS, or?)
-  Are the OSA cards the same..  defined the same?
-  Any NICDEF's defined to TCPIP on this LPAR? (though I would think it
would show as a different OSA )
-  Could a different ports be being used on the same OSA?

You've likely thought of this stuff already - just thinking of stuff I would
check.   Can't think of any reason offhand.  Well - maybe one:  does TCPIP
on this LPAR have 2 virtual CPU's defined?   Seems like the number of CPU's
can influence, but I'm probably thinking of accounting records..

Scott Rohling

On Thu, Jun 9, 2011 at 11:56 AM, Berry van Sleeuwen <
berry.vansleeu...@xs4all.nl> wrote:

> Hi Scott,
>
> Yes, the records are identical. After a SORT UNIQUE the duplicate records
> are discarded. The only thing that is not identical is the TOD field. The
> second (and third) record are produced a few miliseconds after the first.
> When I discard the miliseconds the records themselves are truly identical.
> That leads me to believe that these are seperatly created records.
>
> The record produced by the TCPIP stack contains data from the OSA device,
> we are interested in the number of bytes in and out for the device during
> the minute interval.
>
> As I understand it there is a buffer for each device. So the OSA devices
> each have a record and the CTC devices also have a record. And indeed that
> is what we get on other LPARs. Only on this particular LPAR we have more
> than one record. And only for the OSA's, not for the CTC's.
>
> Regards, Berry.
>
> Op 09-06-11 16:51, Scott Rohling schreef:
>
>> Just poking around found this description of the monitor record:
>>
>> DESCRIPTIVE NAME - Monitor Sample Record
>> Domain 10 - Appldata domain
>> Record 2  - Application Data Sample Record
>>  DESCRIPTION - Application data as found in the application-defined
>>buffer at the time of this sample interval. A
>>separate record is generated for each buffer
>>declared by the virtual machine(s) via the Diagnose
>>'DC' START operation.
>> Seems to imply a separate record is generated for each buffer declared..
>>  I'm not sure what the application data sample is in this case.  Are the
>> records truly duplicates?  (the whole record is absolutely identical,
>> displayable chars or not)   Perhaps the application data needs to be
>> appended to come up with the entire field?
>> Scott Rohling
>>
>> On Thu, Jun 9, 2011 at 8:04 AM, Berry van Sleeuwen <
>> berry.vansleeu...@xs4all.nl > wrote:
>>
>>Hi listers,
>>
>>I am looking at processing the userrecords in the CP MONITOR for
>>the TCPIP
>>stacks. These are in CP MONITOR domain 0A record 02. The data is
>>collected
>>though a STARMON stage, basically "PIPE STARMON | locate
>> | >
>>fielid".
>>
>>Most LPARs have records like I'd expect them but I have found that
>>in one
>>LPAR we have duplicate records for subrecordtype 05 for each of
>>the the OSA
>>devices.
>>
>>The CTC's for VSE systems have one record per minute for each VSE
>>system.
>>The OSA links have 2 identical records (apart from a slight
>>difference in
>>timestamp, a few miliseconds apart). A second IP stack even writes
>>3 records
>>for every minute.
>>
>>So basically we see:
>>00:01 VSE1
>>00:01 VSE2
>>00:01 OSA1
>>00:01 OSA2
>>00:01 OSA1
>>00:01 OSA2
>>00:02 ...
>>
>>What can cause the CP monitor to have these duplicate records?
>>
>>TIA, Berry.
>>
>>
>>

Re: Duplicate CP Monitor records TCPIP

[Berry van Sleeuwen]

Hi Scott,

Yes, the records are identical. After a SORT UNIQUE the duplicate 
records are discarded. The only thing that is not identical is the TOD 
field. The second (and third) record are produced a few miliseconds 
after the first. When I discard the miliseconds the records themselves 
are truly identical. That leads me to believe that these are seperatly 
created records.


The record produced by the TCPIP stack contains data from the OSA 
device, we are interested in the number of bytes in and out for the 
device during the minute interval.


As I understand it there is a buffer for each device. So the OSA devices 
each have a record and the CTC devices also have a record. And indeed 
that is what we get on other LPARs. Only on this particular LPAR we have 
more than one record. And only for the OSA's, not for the CTC's.


Regards, Berry.

Op 09-06-11 16:51, Scott Rohling schreef:

Just poking around found this description of the monitor record:

DESCRIPTIVE NAME - Monitor Sample Record
 Domain 10 - Appldata domain
 Record 2  - Application Data Sample Record
  DESCRIPTION - Application data as found in the application-defined
buffer at the time of this sample interval. A
separate record is generated for each buffer
declared by the virtual machine(s) via the Diagnose
'DC' START operation.
Seems to imply a separate record is generated for each buffer declared..  I'm 
not sure what the application data sample is in this case.  Are the records 
truly duplicates?  (the whole record is absolutely identical, displayable chars 
or not)   Perhaps the application data needs to be appended to come up with the 
entire field?
Scott Rohling

On Thu, Jun 9, 2011 at 8:04 AM, Berry van Sleeuwen 
mailto:berry.vansleeu...@xs4all.nl>> wrote:


Hi listers,

I am looking at processing the userrecords in the CP MONITOR for
the TCPIP
stacks. These are in CP MONITOR domain 0A record 02. The data is
collected
though a STARMON stage, basically "PIPE STARMON | locate
 | >
fielid".

Most LPARs have records like I'd expect them but I have found that
in one
LPAR we have duplicate records for subrecordtype 05 for each of
the the OSA
devices.

The CTC's for VSE systems have one record per minute for each VSE
system.
The OSA links have 2 identical records (apart from a slight
difference in
timestamp, a few miliseconds apart). A second IP stack even writes
3 records
for every minute.

So basically we see:
00:01 VSE1
00:01 VSE2
00:01 OSA1
00:01 OSA2
00:01 OSA1
00:01 OSA2
00:02 ...

What can cause the CP monitor to have these duplicate records?

TIA, Berry.

Re: Duplicate CP Monitor records TCPIP

[Scott Rohling]

Just poking around found this description of the monitor record:

DESCRIPTIVE NAME - Monitor Sample Record
Domain 10 - Appldata domain
Record 2  - Application Data Sample Record
 DESCRIPTION - Application data as found in the application-defined
   buffer at the time of this sample interval. A
   separate record is generated for each buffer
   declared by the virtual machine(s) via the Diagnose
   'DC' START operation.

Seems to imply a separate record is generated for each buffer
declared..  I'm not sure what the application data sample is in this
case.  Are the records truly duplicates?  (the whole record is
absolutely identical, displayable chars or not)   Perhaps the
application data needs to be appended to come up with the entire
field?

Scott Rohling


On Thu, Jun 9, 2011 at 8:04 AM, Berry van Sleeuwen <
berry.vansleeu...@xs4all.nl> wrote:

> Hi listers,
>
> I am looking at processing the userrecords in the CP MONITOR for the TCPIP
> stacks. These are in CP MONITOR domain 0A record 02. The data is collected
> though a STARMON stage, basically "PIPE STARMON | locate  | >
> fielid".
>
> Most LPARs have records like I'd expect them but I have found that in one
> LPAR we have duplicate records for subrecordtype 05 for each of the the OSA
> devices.
>
> The CTC's for VSE systems have one record per minute for each VSE system.
> The OSA links have 2 identical records (apart from a slight difference in
> timestamp, a few miliseconds apart). A second IP stack even writes 3
> records
> for every minute.
>
> So basically we see:
> 00:01 VSE1
> 00:01 VSE2
> 00:01 OSA1
> 00:01 OSA2
> 00:01 OSA1
> 00:01 OSA2
> 00:02 ...
>
> What can cause the CP monitor to have these duplicate records?
>
> TIA, Berry.
>