Re: PSPAT subsystem Implementation in FreeBSD - GSoC 2018

[Sumit Lakra]

Hello,

I tried the sysctl and it worked in that I was able to intercept the
packets with DIR  == DIR_OUT | PROTO_LAYER2, but I am beginning to face
some other increasingly difficult and unanticipated problems in trying to
attach the PSPAT code to work with the present networking system. As you
mentioned you are a bit busy now, I was hoping maybe Alexander will be able
to help me a little here. It will be good to hear a different viewpoint as
well. Also, there are issues I am facing which I believe even you may not
be aware of, hence I am also sending this mail to the mailing lists in hope
of getting additional opinions from other experts of dummynet/ipfw and the
FreeBSD network stack.

PSPAT WIP branch - https://github.com/theGodlessL
akra/freebsd-pspat/tree/pspat-temp

Firstly, as per our previous ideas we had the plan to intercept the packets
from dummynet... pass it through PSPAT... and finally dispatch them out
from the dispatcher queue via the arbiter or a dedicated dispatcher thread
using functions like ip_output() or ether_output_frame() similar to
dummynet_send(). I had already spent a good deal of time trying to get
these working but it failed every time and resulted in kernel panics. My
first thoughts were that the packets are not complete enough for these
functions. (net.link.ether.ipfw worked but it also resulted in an error
when sending the packet to ether_output_frame). So, in order to test it, I
wrote a simple commit to test whether these packets can really be sent to
these functions without making them go through PSPAT at all. Turns out,
they failed.

The first one can be seen here
..
sending DIR_OUT packets to ip_output() directly from dummynt_io() with
nothing to do with PSPAT failed.
The second one can be seen here
..
a similar failure with DIR_OUT | PROTO_LAYER2 packets. These both attempts
resulted in kernel panics.

The conclusion was that neither of these are a good match for PSPAT input
and output interception. (Also, in case of the DIR_OUT | PROTO_LAYER2
packets, they were successfully intercepted and put on the PSPAT client
mailboxes but when the arbiter scanned them, it somehow returned NULL. This
did not happen with DIR_OUT packets which successfully reached the PSPAT
exit point)

So, next I tried to check if we can let dummynet tag the packets and then
call the dummynet_send() functions to dispatch them directly. The first try
with no PSPAT looked like this
..
and it worked without any errors. Although I am unable to make out anything
special being done by the code here, but somehow, letting dummynet tag the
packets and just reading those tags in dummynet_send() before calling
ip_output() or ether_output_frame() seems to work better than trying to
call these functions directly.

Anyway, I figured then it would be a good idea to let dummynet tag these
packets before redirecting them to PSPAT and then calling dummynet_send()
itself at the PSPAT exit point pspat_txqs_flush(), and so I did as can be
seen here
..,
but it didn't work out again. The packets were successfully intercepted and
reached pspat_txqs_flush() but when dummynet_send() is called on them they
result in kernel panics.. I can't figure out how and why ?

So after all these attempts and many more like them, when I was unable to
get it working, I decided to intercept the packets in the originally
planned way that we thought of at the start of GSoC, i.e. to intercept them
where if_output() is called. I also thought that it would be better to call
the exact same function during which we intercept it, while dispatching, as
we don't do anything other than scheduling in PSPAT so the packet remains
in the same state and calling a lower layer function on the packet may not
end well. So, I wrote a bunch of printf statements to see which is the most
commonly used if_output() function call. For testing, I wanted to intercept
at only one position instead of all the dozen places where this is called
in the code. The chosen point was ip_output.c line 662, which is what is
almost always used..I guess. I wrote the code to intercept packets here as
can be seen here
.
On testing, I found that the interception was successful and the packets
were stored in the PSPAT client queues, but the arbiter always returned
NULL while scanning these queues for packets. This issue was similar to
intercepting DIR_OUT | PROTO_LAYER2 packets.

Lastly, leaving the search for the perfect point for packet interception, I
decided to try and implement the Scheduler Algorithm use in PSPAT. I am yet
to use the patch and see how it works. I was mo

Re: PSPAT subsystem Implementation in FreeBSD - GSoC 2018

[Sumit Lakra]

Hello,

I tried some other simpler tests today. First I tried intercepting the
packets from ip_output.c again and added some printf statements to track
the path of package (code
).
As before, they were successfully intercepted and placed in the PSPAT
client queues but the arbiter was unable to find them most of the time (not
always), when scanning the queues. As per my previous assumption this was
probably due to client threads early return without any error indications
which assumed that the packet was dispatched. So, to test it I did this
..
I couldn't make the client threads pause as they apparently had some non
sleepable locks held, so I made them go through a really long loop before
returning hoping this would allow PSPAT enough time to pick them up and
dispatch.. and bingo.. it worked. The packets no longer disappeared from
the PSPAT client queues and reached the pspat_txqs_flush().

This could also be the same reason how the packets with PROTO_LAYER2 tags
disappeared, although as I mentioned in the previous mail, they were really
not good for interception anyway.

Next, I uncommented the actual if_output() call in the pspat_txqs_flush()
to dispatch the packets that were reaching this point, but somehow the
function call failed again(code
).
In order to check if the function was called with correct parameters, I
used some printf statements to check them (code
)..
they were intact. But the function call was failing when called by the
arbiter thread to dispatch packets. The exact same function called with the
exact same arguments and yet it fails when called from a thread other then
the client thread... Why does this happen ??.. I can't figure out !!

This makes my second assumption from the previous mail to be possibly
correct too, and this is probably why calling dummynet_send() from
pspat_txqs_flush() didn't work either.. Put simply, there is some thread
specific stuff going on with the client threads and they don't like any
other threads trying to step in their shoes and dispatch their packets, and
this is not restricted to dummynet/ipfw but maybe true for the entire
network stack and many other functions

Like I had said, I have already completed the PSPAT part and tested it to
be working well but trying to make it work with the existing networking
subsystem is turning out to be increasingly complex. I have no idea how to
get around this problem, but will keep trying to come up with something.
Any help/ideas will be greatly appreciated.

Thanks and Regards,
Sumit

On Sat, Jul 28, 2018 at 1:52 AM, Sumit Lakra 
wrote:

> Hello,
>
> I tried the sysctl and it worked in that I was able to intercept the
> packets with DIR  == DIR_OUT | PROTO_LAYER2, but I am beginning to face
> some other increasingly difficult and unanticipated problems in trying to
> attach the PSPAT code to work with the present networking system. As you
> mentioned you are a bit busy now, I was hoping maybe Alexander will be able
> to help me a little here. It will be good to hear a different viewpoint as
> well. Also, there are issues I am facing which I believe even you may not
> be aware of, hence I am also sending this mail to the mailing lists in hope
> of getting additional opinions from other experts of dummynet/ipfw and the
> FreeBSD network stack.
>
> PSPAT WIP branch - https://github.com/theGodlessL
> akra/freebsd-pspat/tree/pspat-temp
>
> Firstly, as per our previous ideas we had the plan to intercept the
> packets from dummynet... pass it through PSPAT... and finally dispatch them
> out from the dispatcher queue via the arbiter or a dedicated dispatcher
> thread using functions like ip_output() or ether_output_frame() similar to
> dummynet_send(). I had already spent a good deal of time trying to get
> these working but it failed every time and resulted in kernel panics. My
> first thoughts were that the packets are not complete enough for these
> functions. (net.link.ether.ipfw worked but it also resulted in an error
> when sending the packet to ether_output_frame). So, in order to test it, I
> wrote a simple commit to test whether these packets can really be sent to
> these functions without making them go through PSPAT at all. Turns out,
> they failed.
>
> The first one can be seen here
> ..
> sending DIR_OUT packets to ip_output() directly from dummynt_io() with
> nothing to do with PSPAT failed.
> The second one can be seen here
> ..
> a similar failure with DIR_OUT | PROTO_LAYER2 packets. These both attempts
> resulted i