Re: [j-nsp] EX-series automation, NETCONF woes

2010-03-05 Thread Ross Vandegrift 
On Fri, Mar 05, 2010 at 02:26:59PM +0300, Alexandre Snarskii wrote:
> On Wed, Jan 28, 2009 at 03:40:10PM -0500, Ross Vandegrift wrote:
> > Can you do that without providing a map that maps the abbreviated
> > namespace back to the fully-qualified namespace?  If so, I'd love to
> > know how.
> 
> Just run into the same problem myself, and workaround is here:
> you can use local-name() function to do namespace-agnostic 
> searches, and you can use partial checks on namespaces to be
> sure that you getting what you want. 
> 
> Example: iterate over physical interfaces:  
> 
> http://xml.juniper.net/junos/')]">
> 
> Select operational state: 
> 
>  select="normalize-space(*[local-name()='oper-status'])"/>
> 
> Select dropped packets for first queue: 
> 
> 
> select="normalize-space(*[local-name()='queue-counters']/*[local-name()='queue'
>  and 
> *[local-name()='queue-number']=0]/*[local-name()='queue-counters-red-packets'])"/>
> 
> 
> Syntax is ugly, but it works... 

I've been meaning to write up my solution to this, but haven't had the
chance.  It turns out that lxml exposes namespace assignments in a
programmatically accessible manner.

Building the map I mention above ends up being very easy and means I
can rid myself of all the absurd XSLT mangling I had to do.  I wish
the exposure of the namespace information had been more clearly
documented - or that I had noticed it earlier!

For Python folks, suppose that ncdoc is a netconf document.  Then
here's the quick version of how to build an appropriate map of
specific namespaces from a given JUNOS response:

namepsaces = set()
nsmap = dict()

for i in ncdoc.getiterator():
namespaces.update(set(i.nsmap.values()))
for ns in namespaces:
shortname = ns.split('/')[-1]
nsmap[shortname]

# Now you can use normal XML tree traversal techniques
e = nsdoc.getroot()
name = e.xpath("./junos-interface:name/text()", namespaces=nsmap)
operstat = e.xpath("./junos-interface:oper-status/text()", namespaces=nsmap)
...etc...

This should works for any JUNOS platform that uses the consistent
namespace abbreviations we're all used to reading.  Hopefully that's
all of them, forever :)

Ross

-- 
Ross Vandegrift
r...@kallisti.us

"If the fight gets hot, the songs get hotter.  If the going gets tough,
the songs get tougher."
--Woody Guthrie


signature.asc
Description: Digital signature
___
juniper-nsp mailing list juniper-nsp@puck.nether.net
https://puck.nether.net/mailman/listinfo/juniper-nsp

Re: [j-nsp] EX-series automation, NETCONF woes

2010-03-05 Thread Alexandre Snarskii 
On Wed, Jan 28, 2009 at 03:40:10PM -0500, Ross Vandegrift wrote:
> On Wed, Jan 28, 2009 at 11:17:11AM -0800, Derick Winkworth wrote:
> > xpath notation can help you find "junos-interface:interfaces" no
> > matter where its located.
> 
> Can you do that without providing a map that maps the abbreviated
> namespace back to the fully-qualified namespace?  If so, I'd love to
> know how.

Just run into the same problem myself, and workaround is here:
you can use local-name() function to do namespace-agnostic 
searches, and you can use partial checks on namespaces to be
sure that you getting what you want. 

Example: iterate over physical interfaces:  

http://xml.juniper.net/junos/')]">

Select operational state: 



Select dropped packets for first queue: 


select="normalize-space(*[local-name()='queue-counters']/*[local-name()='queue' 
and 
*[local-name()='queue-number']=0]/*[local-name()='queue-counters-red-packets'])"/>


Syntax is ugly, but it works... 


> In my understanding, the XPath query ".//junos-interface:interfaces" [1]
> only matches
> "" if I
> can somewhere say that
> "junos-interface = http://xml.juniper.net/junos/9.3R2/junos-interface";.
> 
> That just moves the problem to one of making a namespace map.
> 
> 
> Ross
> 
> [1] - that's the XPath to find the element named "interfaces" from the
> namespace that's been abbreviated "junos-interface" in any subtree.
> 
> -- 
> Ross Vandegrift
> r...@kallisti.us
> 
> "If the fight gets hot, the songs get hotter.  If the going gets tough,
> the songs get tougher."
>   --Woody Guthrie
> ___
> juniper-nsp mailing list juniper-nsp@puck.nether.net
> https://puck.nether.net/mailman/listinfo/juniper-nsp
___
juniper-nsp mailing list juniper-nsp@puck.nether.net
https://puck.nether.net/mailman/listinfo/juniper-nsp

Re: [j-nsp] EX-series automation, NETCONF woes

2009-01-29 Thread Derick Winkworth 
xpath wildcards?

Ross Vandegrift wrote:
> On Wed, Jan 28, 2009 at 11:17:11AM -0800, Derick Winkworth wrote:
>   
>> xpath notation can help you find "junos-interface:interfaces" no
>> matter where its located.
>> 
>
> Can you do that without providing a map that maps the abbreviated
> namespace back to the fully-qualified namespace?  If so, I'd love to
> know how.
>
> In my understanding, the XPath query ".//junos-interface:interfaces" [1]
> only matches
> "" if I
> can somewhere say that
> "junos-interface = http://xml.juniper.net/junos/9.3R2/junos-interface";.
>
> That just moves the problem to one of making a namespace map.
>
>
> Ross
>
> [1] - that's the XPath to find the element named "interfaces" from the
> namespace that's been abbreviated "junos-interface" in any subtree.
>
>   
> 
>
>
> No virus found in this incoming message.
> Checked by AVG - http://www.avg.com 
> Version: 8.0.176 / Virus Database: 270.10.15/1922 - Release Date: 1/28/2009 
> 7:24 PM
>
>   
___
juniper-nsp mailing list juniper-nsp@puck.nether.net
https://puck.nether.net/mailman/listinfo/juniper-nsp

Re: [j-nsp] EX-series automation, NETCONF woes

2009-01-28 Thread Ross Vandegrift 
On Wed, Jan 28, 2009 at 11:17:11AM -0800, Derick Winkworth wrote:
> xpath notation can help you find "junos-interface:interfaces" no
> matter where its located.

Can you do that without providing a map that maps the abbreviated
namespace back to the fully-qualified namespace?  If so, I'd love to
know how.

In my understanding, the XPath query ".//junos-interface:interfaces" [1]
only matches
"" if I
can somewhere say that
"junos-interface = http://xml.juniper.net/junos/9.3R2/junos-interface";.

That just moves the problem to one of making a namespace map.


Ross

[1] - that's the XPath to find the element named "interfaces" from the
namespace that's been abbreviated "junos-interface" in any subtree.

-- 
Ross Vandegrift
r...@kallisti.us

"If the fight gets hot, the songs get hotter.  If the going gets tough,
the songs get tougher."
--Woody Guthrie
___
juniper-nsp mailing list juniper-nsp@puck.nether.net
https://puck.nether.net/mailman/listinfo/juniper-nsp

Re: [j-nsp] EX-series automation, NETCONF woes

2009-01-28 Thread Derick Winkworth 
xpath notation can help you find "junos-interface:interfaces" no matter where 
its located.

xpath notation should be supported by virtually any XML parsing tool.  "should" 
be.





From: Ross Vandegrift 
To: Joe Abley 
Cc: juniper-nsp@puck.nether.net
Sent: Wednesday, January 28, 2009 11:34:13 AM
Subject: Re: [j-nsp] EX-series automation, NETCONF woes

On Wed, Jan 28, 2009 at 11:00:27AM -0500, Joe Abley wrote:
> On 27 Jan 2009, at 16:23, Ross Vandegrift wrote:
> >3) XML is far more complicated than SNMP with marginal benefits to a
> >switching environment.
> 
> This whole message was a great read, and I'm glad you took the time to  
> write it. On this particular point, though, I think you need to  
> compare apples with apples.
> 
> XML is a data representation; SNMP is a protocol. It would make more  
> sense to compare XML with ASN.1, in which case I'd assert ASN.1 is far  
> more complicated -- to the extent that you don't think about  
> constructing or parsing PDUs by hand, and instead use tools to do the  
> job for you.

Well, that's a fair enough point, but not really what I am getting at.
I'm really concerned about the consequences of using XML vs. ASN.1 and
how this affects the naming of things I need to find.

Using ASN.1 with SNMP, I always know how to reach the table of
interfaces.  This is defined in the IF-MIB: .1.3.6.1.2.1.2.2.  All of
devices that implement the IF-MIB use this fixed name.  This makes it
easy for the developer to know where to go for specific information.

In XML with NETCONF, the namespaces mean that names of things aren't
fixed.  For example, on an EX switch with JUNOS 9.2R2, my interfaces
are found under the XML element named:
"<http://xml.juniper.net/junos/9.2R2/junos-interface:interfaces>".
Now let's upgrade to JUNOS 9.3R2.  The same interfaces, in the same
configuration, are now collected under:
"<http://xml.juniper.net/junos/9.3R2/junos-interface:interfaces>".

See how the name in the tag changed based on the version?  XML parsers
enforce proper use of namespaces to maintain document structure.  You
can't search for the "" tag because there is no such tag!
You have to search in a namespace, and that means writing code that
generates version-specific naming for JUNOS.

I'm not saying that ASN.1 is necessarily simpler.  But because the
namespace is global, implementation of ASN.1 in SNMP forces vendors to
produce consistent names.  NETCONF fails to enforce that, permitting
vendors to run wild with their own names and come up with schemes that
are difficult to handle programmatically.

> If you take the same approach with NETCONF and use tools to generate,  
> validate and parse request and response documents, then I'd suggest  
> that the difference in complexity boils down to how good your tools are.

Yup - and that's been my approach.  I have a library that abstracts
out the hacks I had to do to parse the serialized XML before I give
it to the XML toolkits.  It's ugly, but it does work.

-- 
Ross Vandegrift
r...@kallisti.us

"If the fight gets hot, the songs get hotter.  If the going gets tough,
the songs get tougher."
--Woody Guthrie
___
juniper-nsp mailing list juniper-nsp@puck.nether.net
https://puck.nether.net/mailman/listinfo/juniper-nsp
___
juniper-nsp mailing list juniper-nsp@puck.nether.net
https://puck.nether.net/mailman/listinfo/juniper-nsp

Re: [j-nsp] EX-series automation, NETCONF woes

2009-01-28 Thread Ross Vandegrift 
On Wed, Jan 28, 2009 at 11:00:27AM -0500, Joe Abley wrote:
> On 27 Jan 2009, at 16:23, Ross Vandegrift wrote:
> >3) XML is far more complicated than SNMP with marginal benefits to a
> >switching environment.
> 
> This whole message was a great read, and I'm glad you took the time to  
> write it. On this particular point, though, I think you need to  
> compare apples with apples.
> 
> XML is a data representation; SNMP is a protocol. It would make more  
> sense to compare XML with ASN.1, in which case I'd assert ASN.1 is far  
> more complicated -- to the extent that you don't think about  
> constructing or parsing PDUs by hand, and instead use tools to do the  
> job for you.

Well, that's a fair enough point, but not really what I am getting at.
I'm really concerned about the consequences of using XML vs. ASN.1 and
how this affects the naming of things I need to find.

Using ASN.1 with SNMP, I always know how to reach the table of
interfaces.  This is defined in the IF-MIB: .1.3.6.1.2.1.2.2.  All of
devices that implement the IF-MIB use this fixed name.  This makes it
easy for the developer to know where to go for specific information.

In XML with NETCONF, the namespaces mean that names of things aren't
fixed.  For example, on an EX switch with JUNOS 9.2R2, my interfaces
are found under the XML element named:
"".
Now let's upgrade to JUNOS 9.3R2.  The same interfaces, in the same
configuration, are now collected under:
"".

See how the name in the tag changed based on the version?  XML parsers
enforce proper use of namespaces to maintain document structure.  You
can't search for the "" tag because there is no such tag!
You have to search in a namespace, and that means writing code that
generates version-specific naming for JUNOS.

I'm not saying that ASN.1 is necessarily simpler.  But because the
namespace is global, implementation of ASN.1 in SNMP forces vendors to
produce consistent names.  NETCONF fails to enforce that, permitting
vendors to run wild with their own names and come up with schemes that
are difficult to handle programmatically.

> If you take the same approach with NETCONF and use tools to generate,  
> validate and parse request and response documents, then I'd suggest  
> that the difference in complexity boils down to how good your tools are.

Yup - and that's been my approach.  I have a library that abstracts
out the hacks I had to do to parse the serialized XML before I give
it to the XML toolkits.  It's ugly, but it does work.

-- 
Ross Vandegrift
r...@kallisti.us

"If the fight gets hot, the songs get hotter.  If the going gets tough,
the songs get tougher."
--Woody Guthrie
___
juniper-nsp mailing list juniper-nsp@puck.nether.net
https://puck.nether.net/mailman/listinfo/juniper-nsp

Re: [j-nsp] EX-series automation, NETCONF woes

2009-01-28 Thread Joe Abley 


On 27 Jan 2009, at 16:23, Ross Vandegrift wrote:


3) XML is far more complicated than SNMP with marginal benefits to a
switching environment.


This whole message was a great read, and I'm glad you took the time to  
write it. On this particular point, though, I think you need to  
compare apples with apples.


XML is a data representation; SNMP is a protocol. It would make more  
sense to compare XML with ASN.1, in which case I'd assert ASN.1 is far  
more complicated -- to the extent that you don't think about  
constructing or parsing PDUs by hand, and instead use tools to do the  
job for you.


If you take the same approach with NETCONF and use tools to generate,  
validate and parse request and response documents, then I'd suggest  
that the difference in complexity boils down to how good your tools are.



Joe

___
juniper-nsp mailing list juniper-nsp@puck.nether.net
https://puck.nether.net/mailman/listinfo/juniper-nsp

[j-nsp] EX-series automation, NETCONF woes

2009-01-27 Thread Ross Vandegrift 
Hello everyone,

I've spent the past few weeks developing automation software for the
JUNOS EX-series switches.  During this time, I have come to miss IOS
for its SNMP-based configuration.  In case anyone from Juniper is
reading, I'd like to describe why I have found NETCONF to be such a
painful experience.

If anyone has come up with practical solutions to these problems, or
if I have overlooked some solution, I'd be very interested in hearing.
I'd love to hear that I'm barking up the wrong tree!

I'll also note that we're using the EX-4200 as a top-of-cabinet switch
for datacenter deployments.  Config is relatively simple layer 2
switching with LACP and MSTP.  This strongly shapes how we interact
with the devices.


- Four central problems -

1) NETCONF performs very poorly.  Running over SSH means that I'll
always have TCP session and SSH key exchange to wait for.  I can't
start a NETCONF session in less than 3-5 seconds.  SNMP latency is a
fraction of this and it's very easy to make interactive applications
that make realtime changes.

Furthermore, I have to manage connection lifecycles.  My only real
optimization recourse is to try and reuse open connections, which is
tricky to do without leaking connections.


2) NETCONF is inefficient.  Want to enumerate the switch ports and
their assigned VLANs for the server guys?  You'll need
 as well as .  In
my lab setup, this is over 150K of data so I can produce a table like:

Interface   VLAN ID
ge-0/0/05
ge-0/0/16
...

In a production VC of a few members and a hundred or so customers,
this is like half a meg.  What's that going to look like when we have
400 servers on a VC of ten switches?

Almost all of that data is waste - tags and namespaces have no
operational value.  I only wanted a fraction of the info I got back,
but there's no way to make my queries more specific.

I can produce the above info with SNMP by walking IF-MIB::ifName and
either Q-BRIDGE-MIB::dot1qPVid or CISCO-VLAN-MEMBERSHIP-MIB::vmVlan.
Juniper doesn't expose dot1q tags via SNMP - only the internal VLAN
index that JUNOS uses.


3) XML is far more complicated than SNMP with marginal benefits to a
switching environment.  If you're automating MPLS VPN creation on a
router, XML makes sense - you need to load a fairly decent amount of
structured config, and you want to do so atomically.

But in a datacenter switching environment, the vast majority of the
changes are single attribute changes: changing an interface
description or updating an access vlan.  Automating these two things
means 99% of our manual switch work goes away.  Want to update the
description?  Here ya go:


  

  

  

]]>]]>


  

  


  

  
ge-0/0/10
test
  

  

  

]]>]]>

  

]]>]]>

With SNMP, all I need to do is set ifAlias.  One action, it's atomic.
No need to generate documents describing minor operational work in
horrifying detail.  Same deal with changing an access vlan.


4) Juniper seems to change the XML namespaces with every release.
This means that the XML responses from two different versions of JUNOS
require different parsing rules.  The XML patterns that are widely
implemented assume that the namesapce is a granted thing.  Mucking
about with it at runtime, based on responses from a switch, has
required a very ugly stack of hacks.

Now I'm stuck with a very complicated software to do two very basic
things.  If I had to start over - I'd use expect.


- How I would address these -

1) The Q-BRIDGE-MIB on the EX-series is great except for one crucial
oversight: the dot1q VLAN ID (arguably the most important piece of
data) is not exposed anywhere.  This is easy to fix - just add a table
mapping the JUNOS VLAN Index to the dot1q VLAN ID.

2) Permit read-write access to the IF-MIB and Q-BRIDGE-MIB.  Single
attribute changes become trivial.  This would probably be a harder
change, since JUNOS has a candidate config model.


If you've made it this far, thanks for reading,
Ross Vandegrift

-- 
Ross Vandegrift
r...@kallisti.us

"If the fight gets hot, the songs get hotter.  If the going gets tough,
the songs get tougher."
--Woody Guthrie
___
juniper-nsp mailing list juniper-nsp@puck.nether.net
https://puck.nether.net/mailman/listinfo/juniper-nsp