Hi Daniel - Thanks for the feedback. See my comments below . . .
On Thu, Nov 7, 2013 at 8:40 AM, Daniel Wagner <w...@monom.org> wrote: > Hi Glenn, > > [Sorry for the late response. I am kind of pretty busy] > > > On 11/05/2013 08:27 PM, Glenn Schmottlach wrote: >>>> >>>> As I understand it, Connman sessions are tracked, in part, by marking >>>> packets associated with the session's UID, GID, or SELinux context >>>> information. This is translated into iptable rules to which "mark" the >>>> connections as described in session-overview.txt. >>>> >>>> Per session iptables rules: >>>> >>>> iptables -t mangle -A OUTPUT -m owner [--uid-owner|--gid-owner] $OWNER \ >>>> -j MARK --set-mark $MARK >>>> >>>> iptables -t filter -A INPUT -m mark --mark $MARK \ >>>> -m nfacct --nfacct-name session-input-$MARK >>>> iptables -t filter -A OUTPUT -m mark --mark $MARK \ >>>> -m nfacct --nfacct-name session-output-$MARK >>> >>> >>> >>> BTW, the nfacct is going away. We are going to use NFQUEUE in future. >>> Though we still need the MARK unless we can convince the netdev >>> guys that the lookup for a policy routing table could be something else >>> e.g. cgroup id. For the time being I keep the assumption we need >>> the marker. >>> >> >> I'm curious how you anticipate using NFQUEUE in the future unless >> you're considering re-directing the network traffic to a user-space >> application (like Connman). Are you trying to do a transparent proxy >> or intercept the packets in some way? If so, have you considered >> TPROXY or even using NFLOG to get at the these packets? My only >> concern/confusion here is the intent for using NFQUEUE since it seems >> you want to funnel matching packets into user-space. Can you elaborate >> on your plans? Wouldn't this have a negative impact on the performance >> of the connections managed by Connman if you are moving in that >> direction? > > > The Session API is also designed to allow per application statistics > and routing. The current implementation is trying to use NFACCT > for the statistics. That doesn't really scale though. We need one > rule per direction and protocol type per application. Furthermore, > we need to poll for the updates. > > Android supports the same feature via xt_qtaguid. There was a good > presentation during the LPC on this topic: > > http://www.youtube.com/watch?v=Fi_iyaF7Gw0 > > Eric Dumazet and Thomas Graf were attending this session and both > said NFQUEUE is what we should use for this kind of feature we > want to support. They also pointed out that the operation is > dirt cheap. Let's see, at least we have them on tape and can > bother them to fix it if it isn't :D (just jocking). > I was not aware of this approach. I'll have to investigate this further and get back to you on whether is fits my use-case as well. Thanks for the link . . . I'll watch the video. > I and a colleague of mine already working on this. We need > to extend the connection tracking information with skuid > and skgid. We create a 'statistic daemon' to collect the data. > The main reason not add it to ConnMan directly is that > libnetfiler_queue (and friends) have a blocking API which > is completely unfit for ConnMan. Besides this I don't think > it is a great idea to have ConnMan to look at all packets. > I hope to have soon some real numbers on performance > and power impact. It sounds like an interesting approach. I would be very interested in how it turns out. Is this statistics daemon something you expect that you can share with the Connman community? > >>>> These rules instruct iptables to collect basic usage metrics >>>> (bytes/packets sent/received) on behalf of Connman. >>>> >>>> I have a requirement to do additional filtering and statistics >>>> collection using my own set of rules and would like to filter on these >>>> same MARK'ed packets while adding additional "marks" (or fields) to >>>> the packets so that I can separate them into additional categories >>>> needed by my application. This is where I need some additional >>>> information from Connman and a slight modification to how packets are >>>> MARK'ed and compared. >>>> >>>> As I understand it, each netfilter packet has an associated 32-bit >>>> mark value which is being set to $MARK in the above rules. When >>>> setting a MARK value you can provide a mask which will only allow >>>> specific bits to be set. Looking at the session.c code (line 39) I see >>>> that new sessions are initialized starting at 256 and are incremented >>>> upwards from there. So my assumption is that Connman assumes it "owns" >>>> the upper 24-bits of the netfiler mark value. Likewise, it would seem >>>> from the iptables rules that it does not expect other rules in the >>>> iptable to use these same bits for marking packets. >>> >>> >>> >>> ConnMan starts counting at 256 because that is the first 'unmanaged' >>> ID for a policy routing table. To simplify I used the same value >>> for the routing table as for the marker. Obviously the current >>> implementation doesn't really play nicely by assuming it has complete >>> control. I have no problem to change that. >>> >>> >>>> What I am proposing is the following: >>>> >>>> 1) Modify Connman to provide a mechanism to retrieve the iptable >>>> "MARK" value associated with a session as well as the bitmask used to >>>> filter/extract those bits from the underlying U32 netfilter value. >>>> Perhaps both the session MARK value and bitmask could be considered >>>> read-only "settings" of the session object and would be updated when >>>> the iptable rule is written. >>> >>> >>> >>> I don't really follow here. Could you please elaborate it a bit? >> >> >> Let me try to explain what I'm trying to accomplish. On my target >> platform there exist applications that run as separate processes. My >> intention is to run each "app" as a separate Linux user. > > > Similar to what we want to do (and Android does) > > >> These apps >> might still be further divided into groups with similar features (e.g. >> streaming apps, music apps, etc...). My thought was to assign each app >> to a Linux group, or as you indicate, perhaps a specific cgroup if I >> utilize that Linux feature. I need to enforce both a per-application >> network data quota (bytes sent + received < data limit) as well as an >> overall application "group" data limit. > > > I think we share the same use case here. Apart of the grouping of several > application I want to support the same use case That's good news. I suspect our worlds and needs are quite similar. > > >> My thought is that I could create a Connman policy (or session >> configuration) for *each* application. Connman seems to already >> provide the mechanism to provide a prioritized Bearer list and the >> necessary indications to tell an application whether the underlying >> Bearer/service is connected (or not) and to request a connection >> through the Session API. > > > Yes, the Session API is not only there for notification. > > >> It seems that when a Session is created new >> rules are added to the iptable to "mark" these packets from a >> particular application based on the UID, GID, or SELinux context. > > > Yes. > > >> I want to set up additional filters on these same marked packets >> except I also want to add a field to mark a GID association in the >> same U32 netfilter value. I need to know the netfilter mask I can use >> to mask off the fields Connman uses for each packet. My thought was to >> do this outside of Connman since it's not clear this is something >> everyone needs/desires in their architectures. > > > Maybe. Could you list your rules you want to use? Here's a smattering of rules I've been prototyping. As you can see there is nothing complex here but I suspect it's enough to give you the gist of what I'm trying to do. Right now these rules add a quota to a given UID. The GID rules would be similar but come *before* the UID quota rules since these supersede those (e.g. if it passes the GID related data limits, then apply the per UID limits). Assuming I know Connman's UID marker, I can match on the same packets for that user. CHAIN_PREFIX="chain_$UID" CHAIN_QUOTA="${CHAIN_PREFIX}_quota" ACCT="ACCT_${UID}_${MARK}" CHAIN_ACCT="${CHAIN_PREFIX}_acct" QUOTA=65536 # Create the new custom chains iptables -N $CHAIN_QUOTA iptables -N $CHAIN_ACCT # This marks all outgoing packets with a special marker per user # which will be restored when the response comes pack on the input. # We use it to track incoming packet for a specific UID. iptables -t mangle -A OUTPUT -m owner --uid-owner $UID \ -j MARK --set-mark $MARK # Hook in our custom chains iptables -t filter -A INPUT -m mark --mark $MARK -j $CHAIN_QUOTA iptables -t filter -A OUTPUT -m owner --uid-owner $UID -j $CHAIN_QUOTA iptables -t filter -A $CHAIN_QUOTA -m quota --quota $QUOTA -g $CHAIN_ACCT # Send a TCP reset for all TCP traffic iptables -t filter -A $CHAIN_QUOTA -p tcp -j REJECT --reject-with tcp-reset # Drop UDP and anything else on the floor iptables -t filter -A $CHAIN_QUOTA -j DROP iptables -t filter -A $CHAIN_ACCT -m nfacct --nfacct-name $ACCT -j RETURN > > >> Also, I want to use >> netfilter to collect metrics for a combined send/receive byte count >> since currently Connman tracks sent/received bytes separately. So in >> my solution, I add additional rules for each app (using the Connman >> "mark" value) to collect a send/receive application statistic. I also >> use my own netfilter "mark" value to track the application's Linux >> "group" so I can do group statistics and impose a quota at that level >> as well (which could override the quota at the application level). > > > This sound pretty reasonable. I don't see why something like this > should be supported upstream by ConnMan. Note we completely lack > code for supporting the quota per application so far. Yes, I'd much prefer up-streaming my changes to Connman if at all possible. Forking is always an option but I'd rather implement a n approach that can be supported from the main-line. > > >> I >> can do all of this directly with iptables (I've prototyped it by hand) >> but need Connman to provide me with the per-session/application "mark" >> value and the filter I can used to mask off that value for my own >> iptable rules. Retrieving this information as a read-only Session >> property seemed the most convenient way. I'm open to other suggestions >> however. > > > I would suggest to we join forces here and add the needed code > directly to ConnMan or the 'statistic daemon'. I think I'd be open to that. I'll have to learn more about the "statistics daemon" and it's roles/responsibilities to see how it might fit into the approach I'm considering. Can you share any of these details? > >>>> 2) Provide a configuration option (either at compile-time or via >>>> configuration file item), that specifies the offset/mask for the >>>> session's MARK value. For instance, currently the inferred offset (by >>>> examining the code) is 8 bits with 0xFFFFFFFF as the default mask. >>>> This doesn't allow the Connman markings to co-exist with any other >>>> external markings. For my application I would like to reserve the >>>> upper 16 bits for the Connman session value and the lower 16 bits for >>>> my own use. The Connman code that generates the iptables rule might be >>>> modified to look like this: >>>> >>>> iptables -t mangle -A OUTPUT -m owner [--uid-owner|--gid-owner] $OWNER \ >>>> -j MARK --set-mark $MARK/0x0000FFFF >>>> >>>> iptables -t filter -A INPUT -m mark --mark $MARK/0xFFFF0000 \ >>>> -m nfacct --nfacct-name session-input-$MARK >>>> iptables -t filter -A OUTPUT -m mark --mark $MARK/0xFFFF0000 \ >>>> -m nfacct --nfacct-name session-output-$MARK >>> >>> >>> >>> That sounds reasonable to me. I don't think we need 2^24 ids, we >>> should be happy with 2^16 applications :) >>> >> >> I thought so too . . . I can't imagine (at least for the near-term) >> having more the 2^16 apps installed. In the process of testing how >> Sessions work in Connman I believe I've stumbled upon either a bug in >> the Sessions implementation or an intended (purposeful) behavior. > > > The code is pretty new, so it is quite likely you find bugs. Yes, I think it is a bug (or an unexpected use case that may not have been considered). > >> I >> rather hope it's a bug since otherwise it will force me to put a >> "shim" in front of the Connman Session API. What I have observed is >> that for *every* session that is created, a new set of iptable rules >> is created . . . even if the same user/group/SELinux context is used >> to create the session. > > > Yeah, the assumption is that each application uses it's own unique > identifier. That code doesn't check if more then one Session uses > the same identifier (GID). So this is a bug. Glad you agree. > >> For instance, if I have two applications (both >> the same user - with a session policy keyed to the UID) and they >> create a Session instance then in the iptable's "mangle" table >> (OUTPUT) looks something like this: >> >> Chain connman-OUTPUT (1 references) >> pkts bytes target prot opt in out source >> destination >> 20 1134 MARK all -- * * 0.0.0.0/0 >> 0.0.0.0/0 owner UID match 1000 MARK set 0x100 >> 20 1134 MARK all -- * * 0.0.0.0/0 >> 0.0.0.0/0 owner UID match 1000 MARK set 0x101 >> >> So it would appear the *last* session that is created will effectively >> mark (and clobber) the first marking for the packet (0x101 mark >> clobbers the earlier 0x100 mark since it's last in the chain). I hope >> this is *not* what was intended. I expected that iptable rules for >> Session policies for the same UID/GID/SELinux context would "share" >> the same session (and thus statistics). So if two applications running >> as the same UID/GID/SELinux context requested a Connman Session, the >> first requestor would create it while the second application would get >> a "reference" to that session. This session would remain valid until >> the last application (with the same UID/GID/SELinux context) destroyed >> the session. In a sense the Sessions become reference counted per >> session policy. As it stands today it would appear Connman creates far >> more iptable rules than necessary to support multiple applications >> running as the same user (or in the same group or SELinux context). Am >> I correct in assuming this is a design error in the current >> implementation of Sessions? > > > You are right. We need to fix this. Patches? :D I can certainly work on this now that I understand what was intended. Just didn't want to invest the time if Connman was operating under a different premise or approach. > >>>> As you can see, setting the Connman MARK is no longer destructive to a >>>> value that might already exist in the lower 16-bits of the 32-bit >>>> word. Likewise, when reading and comparing the value the lower 16-bits >>>> of the 32-bits value are effectively masked out so it won't impact the >>>> Connman session match test. >>>> >>>> Part of my problem is that I need to create additional iptable rules >>>> that place data-caps (e.g. a quota) on certain sessions (apps) which >>>> are then optionally partitioned into additional groups. I can use the >>>> power/flexibility of iptables to implement this logic but I need to >>>> cooperatively use the same netfilter 32-bit value to store additional >>>> information about the connection and their associated packets. This >>>> change would help Connman behave as a better netfilter citizen while >>>> allowing me to piggy-back on top of the existing marking done by >>>> Connman. >>> >>> >>> >>> Well, the quota thing is something we want also to support natively >>> with sessions. That is also why count the packets (currently with >>> nfacct but we don't use that information yet). >>> >> >> I'm sure you realize that you don't explicitly need nfacct to collect >> statistics. Granted, nfacct gives you nicely named buckets you can >> access with the nfacct utility and likely over netlink as well. But >> each iptable's rule has a byte/packet count associated with it. > > > Dumping the iptables all the time is kind of excessive. Agreed . . . I'd rather not poll for these stats. But they are available and I could keep the frequency of polling to a minimum since the quota enforcement is done within a netfilter rule rather than trying to track and apply it from a user-space program. > >> So if >> a packet merely passes through a rule (a simple "do nothing" rule) it >> will increment that rule's counters. Plus, iptables gives give you a >> way to query/reset the stats for a specific rule if I'm not mistaken. > > > Yes, that would probably also work but this is a polling which I > try to avoid. Agreed. Polling is generally bad . . . but there are times when you might have no other choice. > >>>> I'm willing to make the necessary code changes to implement this >>>> solution. I'm just seeking feedback of whether a better solution >>>> exists (that I'm unaware of) or perhaps recommendations on my >>>> approach. Of course any patches I make would be available to the >>>> community and hopefully could eventually be integrated into the main >>>> baseline. >>> >>> >>> >>> I am happy to see patches! >>> >>> cheers, >>> daniel >> >> >> I'm really interested in the direction you're headed with Connman >> Sessions. I'd like to leverage this work and perhaps contribute the >> project's generic parts for the solutions I devise. I realize not >> everyone has similar needs but it would appear (at least so far) >> Connman's goals are fairly closely aligned with mine. > > > That is great to hear. We really should strive to make it useful for > more than only my use cases. Therefore I am really happy if you > join the effort to push forward with the Session API and make it > a great feature! > > cheers, > daniel _______________________________________________ connman mailing list connman@connman.net https://lists.connman.net/mailman/listinfo/connman
