Hi David,
Glad to see some shared interest in this!
I've made a lot of updates to my project since my posting to the mailing
list, including Python 2 (tested with 2.7 at least) support.
The only thing I needed to add to facilitate both Py2 and Py3 functionality
was add a macro in the C code to interface with the PyString vs PyUnicode
as seen on the top here:
https://github.com/xstaticxgpx/netsnmp-py3/blob/master/netsnmp/_api.h
I took a lazier route in terms of interfacing with C from Python. I didn't
implement the PDU structures, and only partially implemented the session
structure. Currently my project lacks all the stuff required for SNMPv3,
for example.
I only focused on implementing the basic (v1/v2c) polling functions
(GET/GETNEXT/WALK), but basic (v1/v2c) SET should just be a matter of
cloning one of the existing functions, adding a value variable somewhere,
and updating snmp_pdu_create(<type>) call.
--
Something you may be interested in looking at is the asynchronous polling
methodology I developed;
https://github.com/xstaticxgpx/netsnmp-py3/blob/master/netsnmp/get_async.c
https://github.com/xstaticxgpx/netsnmp-py3/blob/master/test_async.py
There's some cruft in that test_async.py you can ignore, but it boils down
to this:
Using Python multiprocessing, spawn ZeroMQ IPC receivers, then pass 4096
chunks of devices to multiple get_async() C function instances, all of
which communicate back via the ZeroMQ IPC socket. I was stuffing the
returned data into redis in real time, but there are other ways to deal
with it.
There's a lot going on there and I'm having a hard time typing up a decent
overview. Basically what this allows is to do MASSIVE amounts of SNMP
polling, very efficiently.
For example, we had a use case where we needed to poll (SNMPGET) 8-12 OIDs
from upwards of 14 million devices. This get_async methodology could poll,
from a single Dell R710 (dual xeon, single 1GB nic), all that information
in under 20 minutes. This was all to unique, remote, devices.
If you were just polling basic sysDesc or sysUptime, it could do ~14
million unique polls in 7 minutes if I'm remember correctly. The memory
utilization was totally reasonable during the entire run too, somehow I
managed to avoid any serious memory leaks (at least in my experience)...
I had left the position where I was working on this use case, however I was
looking to extend this to be distributable across more than 1 host, but I
hadn't made any progress on that front.
One of the last things I did however, was *try* to reduce overhead during
PDU creation.
https://github.com/xstaticxgpx/netsnmp-py3/blob/master/netsnmp/interface.c#L148
Here you would pass a OID string, and it would return the void pointer for
that object as a PyLong object, so you can then re-pass that into C...as
seen here:
https://github.com/xstaticxgpx/netsnmp-py3/blob/master/netsnmp/get_async.c#L140
https://github.com/xstaticxgpx/netsnmp-py3/blob/master/netsnmp/get_async.c#L147
In practice, I believe I saw the snmp_clone_pdu() function was doing the
same process as creating a brand new PDU object, so I don't believe that
had any beneficial impact on performance.
--
Depending on what you're trying to accomplish, the above may or may not
make any sense, or seem appealing. I would love to speak with you further
regarding it if you're interested, perhaps on IRC somewhere?
Personally, I found the net-snmp source code disgustingly convoluted and
hard to work with. My opinion is that trying to replicate it 1:1 into
python would not be very beneficial.
Other than that, best of luck in your endeavors!! Hope some of this wall of
text helped, please feel free to dig through my source code on github...
-Gabe
On Thu, Mar 17, 2016 at 1:05 PM, David Hankins <dhank...@twitter.com> wrote:
> Hi Gabe,
>
> I'm also working on improving the python bindings (tho I'm not targeting
> python 3 specifically, I'm trying to be future proof there);
>
> https://sourceforge.net/u/hcf64/net-snmp/ci/python_api/tree/python/
>
> My approach was also a little different; I made the new python bindings
> (snmp_api.c) as close to a 1:1 map with Net-SNMP API as possible. My
> strategy there is to make the "pythonic" SNMP library on python's side
> (either a rewrite of netsnmp.py, or a different module).
>
> In retrospect one thing I think I would do differently would be to use
> custom types for the PDU and the two different types of Session objects
> (snmp_open() vs snmp_sess_open()), rather than the Capsule. The
> implementation I have requires I transform the PDU structure into a
> dictionary to return to the caller. This creates many python objects and
> causes a GC storm later in heavy use. A custom type wrapping the Net-SNMP
> structs would construct objects on demand, so I don't think it would have
> that GC penalty, and it would be more reverse compatible (to 2.6 and
> earlier).
>
> I also think I should consider a new strategy for freeing and taking the
> GIL, to handle threaded python applications.
>
> On Thu, Mar 3, 2016 at 9:48 AM, Wes Hardaker <w...@net-snmp-pro.com> wrote:
>
>> Gabe <r...@un1x.su> writes:
>>
>> > I've been working on a Python3 C Extension for the net-snmp API. Still
>> very much a work in progress, but I feel like
>> > it's at a point now that I can share:
>> >
>> > https://github.com/xstaticxgpx/netsnmp-py3
>> >
>> > Mostly just wanted to gauge interest on this, however any feedback
>> > would be much appreciated.
>>
>> Glad to hear someone is plugging away at it. Did you eventually want to
>> contribute it back to the Net-SNMP project, or keep it as a separate
>> piece of work?
>> --
>> Wes Hardaker
>> Please mail all replies to net-snmp-coders@lists.sourceforge.net
>>
>>
>> ------------------------------------------------------------------------------
>> Transform Data into Opportunity.
>> Accelerate data analysis in your applications with
>> Intel Data Analytics Acceleration Library.
>> Click to learn more.
>> http://pubads.g.doubleclick.net/gampad/clk?id=278785231&iu=/4140
>> _______________________________________________
>> Net-snmp-coders mailing list
>> Net-snmp-coders@lists.sourceforge.net
>> https://lists.sourceforge.net/lists/listinfo/net-snmp-coders
>>
>
>
>
> --
> Network Tools & Automation
> Twitter, Inc.
>
------------------------------------------------------------------------------
Transform Data into Opportunity.
Accelerate data analysis in your applications with
Intel Data Analytics Acceleration Library.
Click to learn more.
http://pubads.g.doubleclick.net/gampad/clk?id=278785231&iu=/4140
_______________________________________________
Net-snmp-coders mailing list
Net-snmp-coders@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/net-snmp-coders
