replication failure

[Jeannie Bruno]

Hello.  Has anyone seen these errors when replicating a particular node?

07/05/2017 13:41:22  ANR8340I FILE volume C:\TSMDIR\ORACLE_PRIM7\01706FDD.BFS   
mounted. (SESSION: 47845, PROCESS: 1310)
07/05/2017 13:41:22  ANR0513I Process 1310 opened output volume  
C:\TSMDIR\ORACLE_PRIM7\01706FDD.BFS. (SESSION: 47845, PROCESS: 1310)
07/05/2017 13:42:45  ANR0444W Protocol error on session 47845 for node 
TSMPOK_SERVER1 (Windows) - out-of-sequence verb (type
  BackInsNormEnhanced) received. (SESSION: 47845, PROCESS: 
1310)
07/05/2017 13:42:47  ANR0515I Process 1310 closed volume 
C:\TSMDIR\ORACLE_PRIM7\01706FDD.BFS. (SESSION: 47845, PROCESS: 1310)
07/05/2017 13:42:47  ANR0484W Session 47845 for node TSMPOK_SERVER1 (Windows) 
terminated - protocol violation detected. (SESSION:
  47845, PROCESS: 1310)

I googled the ANR errors and it has:

ANR0444W
Protocol error on session session number for node client node name (client 
platform) - out-of-sequence verb (type verb name) received.
Explanation
The server detects a protocol error on the specified session because a verb has 
been received that does not adhere to the client-server exchange sequence.
System action
The server ends the client session.
User response
If the client generating the error is not an API client, contact your service 
representative. If the client generating the error is an API client, contact 
the owner of the API client. If the client generating the error is a client 
that you have created using WDSF verbs, correct the programming error in your 
client program.
ANR0484W
Session session number for node node name (client platform) terminated - 
protocol violation detected.
Explanation
The server ends the specified session because a communications protocol error 
by the client has been detected.
System action
The server ends the client session.
User response
Examine the client message to determine the problem. Correct the programming 
error in the client program. If the error cannot be isolated and resolved, 
contact your service representative.
If this session is a proxied LAN-Free session for a storage agent, this error 
may be the result of a mount point or volume preemption on the storage agent. 
Review the messages logged by the storage agent to determine if a storage agent 
action caused this to occur.


The node is a linux box, and I'm not sure if it's an API client.  And for the 
other ANR error, it's not a proxied LAN-free session.

I did an audit on the volume in question and it resulted in error:

07/05/2017 16:36:18  ANR2326W Audit volume process terminated for volume  
C:\TSMDIR\ORACLE_PRIM7\01706FDD.BFS - lock conflict detected. (SESSION: 47860, 
PROCESS: 1311)
07/05/2017 16:36:18  ANR0987I Process 1311 for AUDIT VOLUME (REPAIR) running in 
the BACKGROUND processed 789,515 items with a completion
  state of FAILURE at 16:36:18. (SESSION: 47860, PROCESS:   
1311)
07/05/2017 16:36:18  ANR1893E Process 1311 for AUDIT VOLUME (REPAIR) completed 
with a completion state of FAILURE. (SESSION: 47860, PROCESS: 1311)


&&

This node is replicated within a nodegroup which contains 4 other linux nodes 
and they replicate fine.   And the volume in question contains files from the 
other linux nodes.

Any thoughts, help, insight is greatly appreciated.
Thanks.



Jeannie Bruno
Senior Systems Analyst
jbr...@cenhud.com
Central Hudson Gas & Electric
(845) 486-5780
We Care About Your Privacy. This message may contain confidential and/or 
privileged information and is
only for the intended recipient. If the reader of this message is not the 
intended recipient, or an employee or
agent responsible for delivering this message to the intended recipient, please 
notify the sender immediately by
replying to this note and deleting all copies. Thank you.

Re: Best Practices/Best Performance SP/TSM B/A Client Settings ?

[Stefan Folkerts]

Great, and you're very welcome Eric!

On Thu, Jul 6, 2017 at 12:15 PM, Loon, Eric van (ITOPT3) - KLM <
eric-van.l...@klm.com> wrote:

> Hi Stefan!
> I just discovered that raising the sysctl net.core.rmem_max and
> net.core.rmem_max to 124928 fixes the error! Thanks for pointing me in the
> right direction.
> Kind regards,
> Eric van Loon
> Air France/KLM Storage Engineering
>
>
> -Original Message-
> From: ADSM: Dist Stor Manager [mailto:ADSM-L@VM.MARIST.EDU] On Behalf Of
> Stefan Folkerts
> Sent: donderdag 6 juli 2017 9:13
> To: ADSM-L@VM.MARIST.EDU
> Subject: Re: Best Practices/Best Performance SP/TSM B/A Client Settings ?
>
> Hi Eric,
>
> I think some Linux sysctl tuning might be required to raise the Linux OS
> tcp window limit from 224.
> with scaling the system can adjust it if needed, that might work for the
> TSM client as well.
>
> net.ipv4.tcp_window_scaling = 1
>
> Regards,
>   Stefan
>
>
> On Wed, Jul 5, 2017 at 4:30 PM, Loon, Eric van (ITOPT3) - KLM <
> eric-van.l...@klm.com> wrote:
>
> > Hi Del!
> > Because of your mail down below I implemented TCPWINDOWSIZE 512 on the
> > (LINUX) clients on our new TSM server with a directory containerpool,
> > but the client logs are now filled with the following messages:
> >
> > ANS5246W TCPWINDOWSIZE 512 is specified, but exceeds the maximum value
> > allowed by the operating system. TCPWINDOWSIZE 244 will be used instead.
> >
> > What is you recommendation,  from a performance perspective, in this
> > situation?
> > Thanks again for your help!
> > Kind regards,
> > Eric van Loon
> > Air France/KLM Storage Engineering
> >
> > -Original Message-
> > From: ADSM: Dist Stor Manager [mailto:ADSM-L@VM.MARIST.EDU] On Behalf
> > Of Del Hoobler
> > Sent: dinsdag 21 maart 2017 2:11
> > To: ADSM-L@VM.MARIST.EDU
> > Subject: Re: Best Practices/Best Performance SP/TSM B/A Client Settings ?
> >
> > Hi Ben,
> >
> > Here are some items to get you started:
> >
> >
> > Backup-Archive client with limited, high latency network (WAN backups):
> > ===
> > TCPWINDOWSIZE   512
> > RESOURCEUTILIZATION 4
> > COMPRESSION Yes
> > DEDUPLICATION   Yes
> > ENABLEDEDUPCACHEYes
> >
> > Tip:  Do not use the client deduplication caching for applications
> > that use the IBM Spectrum Protect API.  Refer to section 1.2.3.2.1 for
> > additional details.
> >
> >
> > Backup/Archive client or Client API with limited network (Gigabit LAN
> > backups):
> > ===
> > TCPWINDOWSIZE   512
> > RESOURCEUTILIZATION 10
> > COMPRESSION Yes
> > DEDUPLICATION   Yes
> > ENABLEDEDUPCACHENo
> >
> >
> > Backup/Archive client or Client API with high speed network (10
> > Gigabit + LAN backups) ===
> > TCPWINDOWSIZE   512
> > RESOURCEUTILIZATION 10
> > COMPRESSION No
> > DEDUPLICATION   No
> > ENABLEDEDUPCACHENo
> >
> >
> > Tip:  For optimal data reduction, avoid the following client option
> > combination:
> >
> > COMPRESSION Yes
> > DEDUPLICATION   No
> >
> >
> >
> >
> > Del
> >
> > 
> >
> > "ADSM: Dist Stor Manager"  wrote on 03/15/2017
> > 02:39:04 PM:
> >
> > > From: "Alford, Ben" 
> > > To: ADSM-L@VM.MARIST.EDU
> > > Date: 03/15/2017 02:39 PM
> > > Subject: Best Practices/Best Performance SP/TSM B/A Client Settings ?
> > > Sent by: "ADSM: Dist Stor Manager" 
> > >
> > > I've looked at the IBM Blueprint documents but may have missed what
> > > I was looking for - the Best Practices for Best Performance for TSM
> > > B/A client settings.As we move from 6.4 to 7.x or 8.x clients
> > > before the 6.4 EOL, we are looking to test with the current client
> > > settings optimized for things like TCPBUFFSIZE, TCPWINDOWSIZE,
> > > TXNBYTLIMIT, etc., etc.
> > > Thanks!
> > >
> > > Ben Alford
> > > IT Manager, Office of Information Technology
> > > Systems: Shared Services
> > >
> > > The University of Tennessee
> > >
> > 
> > For information, services and offers, please visit our web site:
> > http://www.klm.com. This e-mail and any attachment may contain
> > confidential and privileged material intended for the addressee only.
> > If you are not the addressee, you are notified that no part of the
> > e-mail or any attachment may be disclosed, copied or distributed, and
> > that any other action related to this e-mail or attachment is strictly
> > prohibited, and may be unlawful. If you have received this e-mail by
> > error, please notify the sender immediately by return e-mail, and delete
> this message.
> >
> > Koninklijke Luchtvaart Maatschappij NV (KLM), its subsidiaries and/or
> > its employees shall not be liable for the incorrect or incomplete
> > transmission of this e-mail or any attachments, nor responsible for any
> delay in receipt.
> > Koninklijke Luchtvaart Maatschappij N.V

Re: Best Practices/Best Performance SP/TSM B/A Client Settings ?

[Loon, Eric van (ITOPT3) - KLM]

Hi Stefan!
I just discovered that raising the sysctl net.core.rmem_max and 
net.core.rmem_max to 124928 fixes the error! Thanks for pointing me in the 
right direction.
Kind regards,
Eric van Loon
Air France/KLM Storage Engineering


-Original Message-
From: ADSM: Dist Stor Manager [mailto:ADSM-L@VM.MARIST.EDU] On Behalf Of Stefan 
Folkerts
Sent: donderdag 6 juli 2017 9:13
To: ADSM-L@VM.MARIST.EDU
Subject: Re: Best Practices/Best Performance SP/TSM B/A Client Settings ?

Hi Eric,

I think some Linux sysctl tuning might be required to raise the Linux OS tcp 
window limit from 224.
with scaling the system can adjust it if needed, that might work for the TSM 
client as well.

net.ipv4.tcp_window_scaling = 1

Regards,
  Stefan


On Wed, Jul 5, 2017 at 4:30 PM, Loon, Eric van (ITOPT3) - KLM < 
eric-van.l...@klm.com> wrote:

> Hi Del!
> Because of your mail down below I implemented TCPWINDOWSIZE 512 on the
> (LINUX) clients on our new TSM server with a directory containerpool, 
> but the client logs are now filled with the following messages:
>
> ANS5246W TCPWINDOWSIZE 512 is specified, but exceeds the maximum value 
> allowed by the operating system. TCPWINDOWSIZE 244 will be used instead.
>
> What is you recommendation,  from a performance perspective, in this 
> situation?
> Thanks again for your help!
> Kind regards,
> Eric van Loon
> Air France/KLM Storage Engineering
>
> -Original Message-
> From: ADSM: Dist Stor Manager [mailto:ADSM-L@VM.MARIST.EDU] On Behalf 
> Of Del Hoobler
> Sent: dinsdag 21 maart 2017 2:11
> To: ADSM-L@VM.MARIST.EDU
> Subject: Re: Best Practices/Best Performance SP/TSM B/A Client Settings ?
>
> Hi Ben,
>
> Here are some items to get you started:
>
>
> Backup-Archive client with limited, high latency network (WAN backups):
> ===
> TCPWINDOWSIZE   512
> RESOURCEUTILIZATION 4
> COMPRESSION Yes
> DEDUPLICATION   Yes
> ENABLEDEDUPCACHEYes
>
> Tip:  Do not use the client deduplication caching for applications 
> that use the IBM Spectrum Protect API.  Refer to section 1.2.3.2.1 for 
> additional details.
>
>
> Backup/Archive client or Client API with limited network (Gigabit LAN
> backups):
> ===
> TCPWINDOWSIZE   512
> RESOURCEUTILIZATION 10
> COMPRESSION Yes
> DEDUPLICATION   Yes
> ENABLEDEDUPCACHENo
>
>
> Backup/Archive client or Client API with high speed network (10 
> Gigabit + LAN backups) ===
> TCPWINDOWSIZE   512
> RESOURCEUTILIZATION 10
> COMPRESSION No
> DEDUPLICATION   No
> ENABLEDEDUPCACHENo
>
>
> Tip:  For optimal data reduction, avoid the following client option
> combination:
>
> COMPRESSION Yes
> DEDUPLICATION   No
>
>
>
>
> Del
>
> 
>
> "ADSM: Dist Stor Manager"  wrote on 03/15/2017
> 02:39:04 PM:
>
> > From: "Alford, Ben" 
> > To: ADSM-L@VM.MARIST.EDU
> > Date: 03/15/2017 02:39 PM
> > Subject: Best Practices/Best Performance SP/TSM B/A Client Settings ?
> > Sent by: "ADSM: Dist Stor Manager" 
> >
> > I've looked at the IBM Blueprint documents but may have missed what 
> > I was looking for - the Best Practices for Best Performance for TSM
> > B/A client settings.As we move from 6.4 to 7.x or 8.x clients
> > before the 6.4 EOL, we are looking to test with the current client 
> > settings optimized for things like TCPBUFFSIZE, TCPWINDOWSIZE, 
> > TXNBYTLIMIT, etc., etc.
> > Thanks!
> >
> > Ben Alford
> > IT Manager, Office of Information Technology
> > Systems: Shared Services
> >
> > The University of Tennessee
> >
> 
> For information, services and offers, please visit our web site:
> http://www.klm.com. This e-mail and any attachment may contain 
> confidential and privileged material intended for the addressee only. 
> If you are not the addressee, you are notified that no part of the 
> e-mail or any attachment may be disclosed, copied or distributed, and 
> that any other action related to this e-mail or attachment is strictly 
> prohibited, and may be unlawful. If you have received this e-mail by 
> error, please notify the sender immediately by return e-mail, and delete this 
> message.
>
> Koninklijke Luchtvaart Maatschappij NV (KLM), its subsidiaries and/or 
> its employees shall not be liable for the incorrect or incomplete 
> transmission of this e-mail or any attachments, nor responsible for any delay 
> in receipt.
> Koninklijke Luchtvaart Maatschappij N.V. (also known as KLM Royal 
> Dutch
> Airlines) is registered in Amstelveen, The Netherlands, with 
> registered number 33014286
> 
>

For information, services and offers, please visit our web site: 
http://www.klm.com. This e-mail and any attachment may contain confi

Re: Best Practices/Best Performance SP/TSM B/A Client Settings ?

[Loon, Eric van (ITOPT3) - KLM]

Hi Stefan!
Thanks for your reply! net.ipv4.tcp_window_scaling = 1 seems to be the default 
on Linux, I checked on the host and it is already set to 1...
So indeed there must be another parameter which limits the size to 224 
apparently, but I cannot find any clear instruction on where these limits are 
set...
Kind regards,
Eric van Loon
Air France/KLM Storage Engineering

-Original Message-
From: ADSM: Dist Stor Manager [mailto:ADSM-L@VM.MARIST.EDU] On Behalf Of Stefan 
Folkerts
Sent: donderdag 6 juli 2017 9:13
To: ADSM-L@VM.MARIST.EDU
Subject: Re: Best Practices/Best Performance SP/TSM B/A Client Settings ?

Hi Eric,

I think some Linux sysctl tuning might be required to raise the Linux OS tcp 
window limit from 224.
with scaling the system can adjust it if needed, that might work for the TSM 
client as well.

net.ipv4.tcp_window_scaling = 1

Regards,
  Stefan


On Wed, Jul 5, 2017 at 4:30 PM, Loon, Eric van (ITOPT3) - KLM < 
eric-van.l...@klm.com> wrote:

> Hi Del!
> Because of your mail down below I implemented TCPWINDOWSIZE 512 on the
> (LINUX) clients on our new TSM server with a directory containerpool, 
> but the client logs are now filled with the following messages:
>
> ANS5246W TCPWINDOWSIZE 512 is specified, but exceeds the maximum value 
> allowed by the operating system. TCPWINDOWSIZE 244 will be used instead.
>
> What is you recommendation,  from a performance perspective, in this 
> situation?
> Thanks again for your help!
> Kind regards,
> Eric van Loon
> Air France/KLM Storage Engineering
>
> -Original Message-
> From: ADSM: Dist Stor Manager [mailto:ADSM-L@VM.MARIST.EDU] On Behalf 
> Of Del Hoobler
> Sent: dinsdag 21 maart 2017 2:11
> To: ADSM-L@VM.MARIST.EDU
> Subject: Re: Best Practices/Best Performance SP/TSM B/A Client Settings ?
>
> Hi Ben,
>
> Here are some items to get you started:
>
>
> Backup-Archive client with limited, high latency network (WAN backups):
> ===
> TCPWINDOWSIZE   512
> RESOURCEUTILIZATION 4
> COMPRESSION Yes
> DEDUPLICATION   Yes
> ENABLEDEDUPCACHEYes
>
> Tip:  Do not use the client deduplication caching for applications 
> that use the IBM Spectrum Protect API.  Refer to section 1.2.3.2.1 for 
> additional details.
>
>
> Backup/Archive client or Client API with limited network (Gigabit LAN
> backups):
> ===
> TCPWINDOWSIZE   512
> RESOURCEUTILIZATION 10
> COMPRESSION Yes
> DEDUPLICATION   Yes
> ENABLEDEDUPCACHENo
>
>
> Backup/Archive client or Client API with high speed network (10 
> Gigabit + LAN backups) ===
> TCPWINDOWSIZE   512
> RESOURCEUTILIZATION 10
> COMPRESSION No
> DEDUPLICATION   No
> ENABLEDEDUPCACHENo
>
>
> Tip:  For optimal data reduction, avoid the following client option
> combination:
>
> COMPRESSION Yes
> DEDUPLICATION   No
>
>
>
>
> Del
>
> 
>
> "ADSM: Dist Stor Manager"  wrote on 03/15/2017
> 02:39:04 PM:
>
> > From: "Alford, Ben" 
> > To: ADSM-L@VM.MARIST.EDU
> > Date: 03/15/2017 02:39 PM
> > Subject: Best Practices/Best Performance SP/TSM B/A Client Settings ?
> > Sent by: "ADSM: Dist Stor Manager" 
> >
> > I've looked at the IBM Blueprint documents but may have missed what 
> > I was looking for - the Best Practices for Best Performance for TSM
> > B/A client settings.As we move from 6.4 to 7.x or 8.x clients
> > before the 6.4 EOL, we are looking to test with the current client 
> > settings optimized for things like TCPBUFFSIZE, TCPWINDOWSIZE, 
> > TXNBYTLIMIT, etc., etc.
> > Thanks!
> >
> > Ben Alford
> > IT Manager, Office of Information Technology
> > Systems: Shared Services
> >
> > The University of Tennessee
> >
> 
> For information, services and offers, please visit our web site:
> http://www.klm.com. This e-mail and any attachment may contain 
> confidential and privileged material intended for the addressee only. 
> If you are not the addressee, you are notified that no part of the 
> e-mail or any attachment may be disclosed, copied or distributed, and 
> that any other action related to this e-mail or attachment is strictly 
> prohibited, and may be unlawful. If you have received this e-mail by 
> error, please notify the sender immediately by return e-mail, and delete this 
> message.
>
> Koninklijke Luchtvaart Maatschappij NV (KLM), its subsidiaries and/or 
> its employees shall not be liable for the incorrect or incomplete 
> transmission of this e-mail or any attachments, nor responsible for any delay 
> in receipt.
> Koninklijke Luchtvaart Maatschappij N.V. (also known as KLM Royal 
> Dutch
> Airlines) is registered in Amstelveen, The Netherlands, with 
> registered number 33014286
> 
>
**

Re: Best Practices/Best Performance SP/TSM B/A Client Settings ?

[Stefan Folkerts]

Hi Eric,

I think some Linux sysctl tuning might be required to raise the Linux OS
tcp window limit from 224.
with scaling the system can adjust it if needed, that might work for the
TSM client as well.

net.ipv4.tcp_window_scaling = 1

Regards,
  Stefan


On Wed, Jul 5, 2017 at 4:30 PM, Loon, Eric van (ITOPT3) - KLM <
eric-van.l...@klm.com> wrote:

> Hi Del!
> Because of your mail down below I implemented TCPWINDOWSIZE 512 on the
> (LINUX) clients on our new TSM server with a directory containerpool, but
> the client logs are now filled with the following messages:
>
> ANS5246W TCPWINDOWSIZE 512 is specified, but exceeds the maximum value
> allowed by the operating system. TCPWINDOWSIZE 244 will be used instead.
>
> What is you recommendation,  from a performance perspective, in this
> situation?
> Thanks again for your help!
> Kind regards,
> Eric van Loon
> Air France/KLM Storage Engineering
>
> -Original Message-
> From: ADSM: Dist Stor Manager [mailto:ADSM-L@VM.MARIST.EDU] On Behalf Of
> Del Hoobler
> Sent: dinsdag 21 maart 2017 2:11
> To: ADSM-L@VM.MARIST.EDU
> Subject: Re: Best Practices/Best Performance SP/TSM B/A Client Settings ?
>
> Hi Ben,
>
> Here are some items to get you started:
>
>
> Backup-Archive client with limited, high latency network (WAN backups):
> ===
> TCPWINDOWSIZE   512
> RESOURCEUTILIZATION 4
> COMPRESSION Yes
> DEDUPLICATION   Yes
> ENABLEDEDUPCACHEYes
>
> Tip:  Do not use the client deduplication caching for applications that
> use the IBM Spectrum Protect API.  Refer to section 1.2.3.2.1 for
> additional details.
>
>
> Backup/Archive client or Client API with limited network (Gigabit LAN
> backups):
> ===
> TCPWINDOWSIZE   512
> RESOURCEUTILIZATION 10
> COMPRESSION Yes
> DEDUPLICATION   Yes
> ENABLEDEDUPCACHENo
>
>
> Backup/Archive client or Client API with high speed network (10 Gigabit +
> LAN backups)
> ===
> TCPWINDOWSIZE   512
> RESOURCEUTILIZATION 10
> COMPRESSION No
> DEDUPLICATION   No
> ENABLEDEDUPCACHENo
>
>
> Tip:  For optimal data reduction, avoid the following client option
> combination:
>
> COMPRESSION Yes
> DEDUPLICATION   No
>
>
>
>
> Del
>
> 
>
> "ADSM: Dist Stor Manager"  wrote on 03/15/2017
> 02:39:04 PM:
>
> > From: "Alford, Ben" 
> > To: ADSM-L@VM.MARIST.EDU
> > Date: 03/15/2017 02:39 PM
> > Subject: Best Practices/Best Performance SP/TSM B/A Client Settings ?
> > Sent by: "ADSM: Dist Stor Manager" 
> >
> > I've looked at the IBM Blueprint documents but may have missed what
> > I was looking for - the Best Practices for Best Performance for TSM
> > B/A client settings.As we move from 6.4 to 7.x or 8.x clients
> > before the 6.4 EOL, we are looking to test with the current client
> > settings optimized for things like TCPBUFFSIZE, TCPWINDOWSIZE,
> > TXNBYTLIMIT, etc., etc.
> > Thanks!
> >
> > Ben Alford
> > IT Manager, Office of Information Technology
> > Systems: Shared Services
> >
> > The University of Tennessee
> >
> 
> For information, services and offers, please visit our web site:
> http://www.klm.com. This e-mail and any attachment may contain
> confidential and privileged material intended for the addressee only. If
> you are not the addressee, you are notified that no part of the e-mail or
> any attachment may be disclosed, copied or distributed, and that any other
> action related to this e-mail or attachment is strictly prohibited, and may
> be unlawful. If you have received this e-mail by error, please notify the
> sender immediately by return e-mail, and delete this message.
>
> Koninklijke Luchtvaart Maatschappij NV (KLM), its subsidiaries and/or its
> employees shall not be liable for the incorrect or incomplete transmission
> of this e-mail or any attachments, nor responsible for any delay in receipt.
> Koninklijke Luchtvaart Maatschappij N.V. (also known as KLM Royal Dutch
> Airlines) is registered in Amstelveen, The Netherlands, with registered
> number 33014286
> 
>