Re: Python3 bindings

[Gabe]

Hi David,

Glad to see some shared interest in this!

I've made a lot of updates to my project since my posting to the mailing
list, including Python 2 (tested with 2.7 at least) support.

The only thing I needed to add to facilitate both Py2 and Py3 functionality
was add a macro in the C code to interface with the PyString vs PyUnicode
as seen on the top here:
https://github.com/xstaticxgpx/netsnmp-py3/blob/master/netsnmp/_api.h

I took a lazier route in terms of interfacing with C from Python. I didn't
implement the PDU structures, and only partially implemented the session
structure. Currently my project lacks all the stuff required for SNMPv3,
for example.

I only focused on implementing the basic  (v1/v2c) polling functions
(GET/GETNEXT/WALK), but basic (v1/v2c) SET should just be a matter of
cloning one of the existing functions, adding a value variable somewhere,
and updating snmp_pdu_create() call.

--

Something you may be interested in looking at is the asynchronous polling
methodology I developed;

https://github.com/xstaticxgpx/netsnmp-py3/blob/master/netsnmp/get_async.c
https://github.com/xstaticxgpx/netsnmp-py3/blob/master/test_async.py

There's some cruft in that test_async.py you can ignore, but it boils down
to this:

Using Python multiprocessing, spawn ZeroMQ IPC receivers, then pass 4096
chunks of devices to multiple get_async() C function instances, all of
which communicate back via the ZeroMQ IPC socket. I was stuffing the
returned data into redis in real time, but there are other ways to deal
with it.

There's a lot going on there and I'm having a hard time typing up a decent
overview. Basically what this allows is to do MASSIVE amounts of SNMP
polling, very efficiently.

For example, we had a use case where we needed to poll (SNMPGET) 8-12 OIDs
from upwards of 14 million devices. This get_async methodology could poll,
from a single Dell R710 (dual xeon, single 1GB nic), all that information
in under 20 minutes. This was all to unique, remote, devices.

If you were just polling basic sysDesc or sysUptime, it could do ~14
million unique polls in 7 minutes if I'm remember correctly. The memory
utilization was totally reasonable during the entire run too, somehow I
managed to avoid any serious memory leaks (at least in my experience)...

I had left the position where I was working on this use case, however I was
looking to extend this to be distributable across more than 1 host, but I
hadn't made any progress on that front.

One of the last things I did however, was *try* to reduce overhead during
PDU creation.
https://github.com/xstaticxgpx/netsnmp-py3/blob/master/netsnmp/interface.c#L148
Here you would pass a OID string, and it would return the void pointer for
that object as a PyLong object, so you can then re-pass that into C...as
seen here:
https://github.com/xstaticxgpx/netsnmp-py3/blob/master/netsnmp/get_async.c#L140
https://github.com/xstaticxgpx/netsnmp-py3/blob/master/netsnmp/get_async.c#L147

In practice, I believe I saw the snmp_clone_pdu() function was doing the
same process as creating a brand new PDU object, so I don't believe that
had any beneficial impact on performance.

--

Depending on what you're trying to accomplish, the above may or may not
make any sense, or seem appealing. I would love to speak with you further
regarding it if you're interested, perhaps on IRC somewhere?

Personally, I found the net-snmp source code disgustingly convoluted and
hard to work with. My opinion is that trying to replicate it 1:1 into
python would not be very beneficial.

Other than that, best of luck in your endeavors!! Hope some of this wall of
text helped, please feel free to dig through my source code on github...

-Gabe

On Thu, Mar 17, 2016 at 1:05 PM, David Hankins  wrote:

> Hi Gabe,
>
> I'm also working on improving the python bindings (tho I'm not targeting
> python 3 specifically, I'm trying to be future proof there);
>
> https://sourceforge.net/u/hcf64/net-snmp/ci/python_api/tree/python/
>
> My approach was also a little different; I made the new python bindings
> (snmp_api.c) as close to a 1:1 map with Net-SNMP API as possible.  My
> strategy there is to make the "pythonic" SNMP library on python's side
> (either a rewrite of netsnmp.py, or a different module).
>
> In retrospect one thing I think I would do differently would be to use
> custom types for the PDU and the two different types of Session objects
> (snmp_open() vs snmp_sess_open()), rather than the Capsule.  The
> implementation I have requires I transform the PDU structure into a
> dictionary to return to the caller.  This creates many python objects and
> causes a GC storm later in heavy use.  A custom type wrapping the Net-SNMP
> structs would construct objects on demand, so I don't think it would have
> that GC penalty, and it would be more reverse compatible (to 2.6 and
> earlier).
>
> I also think I should consider a new strategy for freeing and taking the
> GIL, to handle threaded 

Re: Unresolved hostname

[Niels Baggesen]

On Thu, Mar 17, 2016 at 01:03:07PM +0200, Ruth Glushkin wrote:
> but when I run agent(s), I got the following error:
> 
> cannot resolve source hostname
> 
> 
> If I remove the port from config file and run agent, it works.
> 
> How could I set different ports for agents?

You use the agentaddress parameter to set the IP address and port
number.

/Niels

-- 
Niels Baggesen - @home - Århus - Denmark - n...@users.sourceforge.net
The purpose of computing is insight, not numbers   ---   R W Hamming

--
Transform Data into Opportunity.
Accelerate data analysis in your applications with
Intel Data Analytics Acceleration Library.
Click to learn more.
http://pubads.g.doubleclick.net/gampad/clk?id=278785231=/4140
___
Net-snmp-coders mailing list
Net-snmp-coders@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/net-snmp-coders

Re: Python3 bindings

[David Hankins]

Hi Gabe,

I'm also working on improving the python bindings (tho I'm not targeting
python 3 specifically, I'm trying to be future proof there);

https://sourceforge.net/u/hcf64/net-snmp/ci/python_api/tree/python/

My approach was also a little different; I made the new python bindings
(snmp_api.c) as close to a 1:1 map with Net-SNMP API as possible.  My
strategy there is to make the "pythonic" SNMP library on python's side
(either a rewrite of netsnmp.py, or a different module).

In retrospect one thing I think I would do differently would be to use
custom types for the PDU and the two different types of Session objects
(snmp_open() vs snmp_sess_open()), rather than the Capsule.  The
implementation I have requires I transform the PDU structure into a
dictionary to return to the caller.  This creates many python objects and
causes a GC storm later in heavy use.  A custom type wrapping the Net-SNMP
structs would construct objects on demand, so I don't think it would have
that GC penalty, and it would be more reverse compatible (to 2.6 and
earlier).

I also think I should consider a new strategy for freeing and taking the
GIL, to handle threaded python applications.

On Thu, Mar 3, 2016 at 9:48 AM, Wes Hardaker  wrote:

> Gabe  writes:
>
> > I've been working on a Python3 C Extension for the net-snmp API. Still
> very much a work in progress, but I feel like
> > it's at a point now that I can share:
> >
> > https://github.com/xstaticxgpx/netsnmp-py3
> >
> > Mostly just wanted to gauge interest on this, however any feedback
> > would be much appreciated.
>
> Glad to hear someone is plugging away at it.  Did you eventually want to
> contribute it back to the Net-SNMP project, or keep it as a separate piece
> of work?
> --
> Wes Hardaker
> Please mail all replies to net-snmp-coders@lists.sourceforge.net
>
>
> --
> Transform Data into Opportunity.
> Accelerate data analysis in your applications with
> Intel Data Analytics Acceleration Library.
> Click to learn more.
> http://pubads.g.doubleclick.net/gampad/clk?id=278785231=/4140
> ___
> Net-snmp-coders mailing list
> Net-snmp-coders@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/net-snmp-coders
>



-- 
Network Tools & Automation
Twitter, Inc.
--
Transform Data into Opportunity.
Accelerate data analysis in your applications with
Intel Data Analytics Acceleration Library.
Click to learn more.
http://pubads.g.doubleclick.net/gampad/clk?id=278785231=/4140___
Net-snmp-coders mailing list
Net-snmp-coders@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/net-snmp-coders

Re: perl 5.23.2+ and net-snmp don't get along

[Magnus Fromreide]

On Wed, Mar 16, 2016 at 01:33:14PM -0400, Robert Story wrote:
> On Wed, 16 Mar 2016 12:28:14 -0400 Bill wrote:
> BF> > > > I think this needs some discussion. While the change does
> BF> > > > not affect binary compatibility, it does remove a typedef
> BF> > > > that has been around since 1998. It's possible and even
> BF> > > > likely that people are using this typedef in their own
> BF> > > > modules, and this will break backwards compatibility for
> BF> > > > them.  
> BF> > >
> BF> > > Restoring that typedef is easy - all that is needed is
> BF> > > something like the patch below. But I'm not sure whether we
> BF> > > really should restore that typedef. The name of the U64
> BF> > > type is so short that there might be other projects than
> BF> > > Perl that define a conflicting type with the same name.  
> BF> >
> BF> > Yes, but then, counter64 and oid ain't that good as names
> BF> > goes either... (And yes, the perl folks are just as bad as us
> BF> > when it comes to inventing good
> BF> > names)
> BF> >
> BF> > I can surely see how both points of view have merit.
> BF> 
> BF> Just a data point: one of the first things we did to be able to
> BF> use the net-snmp client libraries was to change the net-snmp
> BF> U64 typedef, since our system has its own idea of what a U64
> BF> is.  (I think this is Bart's point.)
> 
> I'm fine with changing the typedef to a new typedef (NETSNMP_U64?)
> and updating all the code to use it.
> 
> The question is whether or not compatibility is on by default. I'd
> argue that in existing branches it should be, possibly with a flag
> that can be defined to remove it. i.e.
> 
> /** default to compatibility */
> #ifndef NETSNMP_U64_NOCOMPAT
> #define U64 NETSNMP_U64
> #endif
> 
> Then in future branches updating to something like
> 
> /** do not define U64 unless requested */
> #ifdef NETSNMP_U64_COMPAT
> #define U64 NETSNMP_U64
> #endif
> 
> Thoughts?

Since c99 uint64_t exists. Why do we need U64 in the first place?

/MF

--
Transform Data into Opportunity.
Accelerate data analysis in your applications with
Intel Data Analytics Acceleration Library.
Click to learn more.
http://pubads.g.doubleclick.net/gampad/clk?id=278785231=/4140
___
Net-snmp-coders mailing list
Net-snmp-coders@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/net-snmp-coders