Re: Improving qos_flows mark feature - obtaining mark later
Sorry for the delay, stand by for further emails on this subject as well as the comments below... On Mon, 2011-10-17 at 16:30 +1300, Amos Jeffries wrote: On 17/10/11 11:23, Andrew Beverley wrote: Hi, I've been using the qos_flows feature for preserving a netfilter mark, but have run into some problems. Currently, the netfilter mark for the connection is obtained in forward.cc, during the stages of opening a connection to the remote server. The problem with this is that the connection mark may change once the reply is received. So, I would like to move Ip::Qos::getNfmarkFromServer() from FwdState::dispatch to a function that is called during the server reply stage. However, I can't work out where it should go. I've looked at moving it to client_side_reply.cc, but I can't find a way to retrieve the remote server connection information. I've also looked at comm_read() in comm.cc, but I can't find the client connection information and it looks like the wrong place anyway. Where is the best place to move it to, where it has access to both the client and server side connection, but where it is late enough to read the mark value if it changes during the server reply? Hope this makes sense! FwdState::dispatch is called at the start of Server request sending. TO start the construction and write of request headers to the server. For persistent and pinned connections it is essentially in the pause position between end of reading one reply and start of sending the next request. I think you mean that it may change on reading the first bytes of reply, yes? Yes. In which case the best position is in HttpStateData::processReplyHeader(). What I actually meant was that it could change during the sending of data, as well as the first bytes. I'll explain further in a follow up email. With matching positions in ftp.cc and tunnel.cc reply starting handlers. The server connection is produced by a method dataConnection() of the state object. The client connection is not exposed. Although it could be. It is in the parent ServerStateData::FwdState::clientConn. Amos Andy
Re: Improving qos_flows mark feature - obtaining mark later
On Mon, 2011-10-17 at 10:34 -0600, Alex Rousskov wrote: On 10/16/2011 04:23 PM, Andrew Beverley wrote: I've been using the qos_flows feature for preserving a netfilter mark, but have run into some problems. Currently, the netfilter mark for the connection is obtained in forward.cc, during the stages of opening a connection to the remote server. The problem with this is that the connection mark may change once the reply is received. So, I would like to move Ip::Qos::getNfmarkFromServer() from FwdState::dispatch to a function that is called during the server reply stage. However, I can't work out where it should go. Could you define server reply stage more precisely, please? If you want that function to be called once per [HTTP] response, then the definition should be that of a single response state change and not a continuous reply stage. If you want that function to be called once per connection, then the definition should be based on connection-specific state changes and not on the response state change. Etc. What I meant was during the receiving of data from the server, as opposed to (currently) during the sending of the request to the server. Having thought about this further, I think what I was trying to achieve was getting the mark every time a packet was received. I've looked at moving it to client_side_reply.cc, but I can't find a way to retrieve the remote server connection information. I've also looked at comm_read() in comm.cc, but I can't find the client connection information and it looks like the wrong place anyway. client_side_reply.cc is dealing with receiving responses from Store. Such responses may have no connection associated with them. If you must have access to the server connection, then you should look on the server side (forward.cc, Server.cc, http.cc, ftp.cc, etc. with tunnel.cc being a special case). See Amos' email for more details. I tried some test code in processReplyHeader() as per Amos' suggestion, but as that function is only called once per reply it doesn't fulfil the requirement of retrieving the mark every time new data arrives. BTW, code common to HTTP and FTP handling should go into Server.cc to the extent possible. I have tried putting the relevant code into ServerStateData::addVirginReplyBody() which seems to work for different protocols as well as being called for each packet received. Is there any problems with this approach? If not, I will refine and submit a patch. Where is the best place to move it to, where it has access to both the client and server side connection, but where it is late enough to read the mark value if it changes during the server reply? Please note that during the server reply may take an hour. Do you want to do something once per reply, every time there are new bytes read from the origin server, or every time a new piece of response body is given to the client side, for example? Ideally it would be for every packet of data received, as each packet of data will have its own mark value. Doing it every time ServerStateData::addVirginReplyBody() is called seems to be pretty close. To give some more detail as to what I was actually trying to achieve: I was trying to mark the *connection* with one value whilst a packet was leaving the internet-facing interface, but a different value as the reply was received. This was because I wanted a different mark value for the packets going to the client from Squid, as to the upstream connection going to the server. However, this just doesn't work very well for reasons that are probably obvious, and I've found with all the code above that the mark value was almost random (presumably because by the time the mark value has been read, the next packet is on its way out). So... I still plan to progress the work above, assuming that you are happy, but my plan for my particular application is to add a mask option for the qos_flows mark miss option. I'll send a patch separately once I've finished it. Thanks, Andy
Re: Improving qos_flows mark feature - obtaining mark later
On Mon, 2011-10-17 at 10:03 +0200, Kinkie wrote: On Mon, Oct 17, 2011 at 5:30 AM, Amos Jeffries squ...@treenet.co.nz wrote: On 17/10/11 11:23, Andrew Beverley wrote: Hi, I've been using the qos_flows feature for preserving a netfilter mark, but have run into some problems. Currently, the netfilter mark for the connection is obtained in forward.cc, during the stages of opening a connection to the remote server. The problem with this is that the connection mark may change once the reply is received. So, I would like to move Ip::Qos::getNfmarkFromServer() from FwdState::dispatch to a function that is called during the server reply stage. However, I can't work out where it should go. I've looked at moving it to client_side_reply.cc, but I can't find a way to retrieve the remote server connection information. I've also looked at comm_read() in comm.cc, but I can't find the client connection information and it looks like the wrong place anyway. Where is the best place to move it to, where it has access to both the client and server side connection, but where it is late enough to read the mark value if it changes during the server reply? Hope this makes sense! FwdState::dispatch is called at the start of Server request sending. TO start the construction and write of request headers to the server. For persistent and pinned connections it is essentially in the pause position between end of reading one reply and start of sending the next request. I think you mean that it may change on reading the first bytes of reply, yes? In which case the best position is in HttpStateData::processReplyHeader(). With matching positions in ftp.cc and tunnel.cc reply starting handlers. The server connection is produced by a method dataConnection() of the state object. The client connection is not exposed. Although it could be. It is in the parent ServerStateData::FwdState::clientConn. Instead of moving it , why not doing both control points? like http_access / http_reply_access. It shouldn't hurt performance, and give greater flexibility.. I don't think that there is much point to having it in both places, as the value is only used when sending data to the client. Something like this will happen: the mark will be retrieved during the request to the server and saved (as now); the mark will then be retrieved again during the server reply; and then the mark will be used to mark packets leaving Squid to the client. I think it makes sense just to retrieve the mark during the server reply stage and use that. Andy
Re: Improving qos_flows mark feature - obtaining mark later
On 10/31/2011 03:03 PM, Andrew Beverley wrote: Having thought about this further, I think what I was trying to achieve was getting the mark every time a packet was received. I have tried putting the relevant code into ServerStateData::addVirginReplyBody() which seems to work for different protocols as well as being called for each packet received. Is there any problems with this approach? If not, I will refine and submit a patch. Yes, if you do not care about HTTP response header or FTP control messages: ServerStateData::addVirginReplyBody() is not called when the HTTP header or FTP control messages are received. If you want to catch those cases (it sounds like you do based on your every packet description above), then you need HttpStateData::readReply() and equivalent read handlers in other protocols. I do not think there is a single Server method that is always called for every Squid read. You should probably add it and call it from the corresponding HTTP and FTP-specific code. Please keep in mind that Squid does not work on a TCP packet level. It uses the TCP socket interface and the read(2) may happen when there are multiple packets in the buffer already. HTH, Alex.
Re: Improving qos_flows mark feature - obtaining mark later
On Mon, 2011-10-31 at 15:42 -0600, Alex Rousskov wrote: On 10/31/2011 03:03 PM, Andrew Beverley wrote: Having thought about this further, I think what I was trying to achieve was getting the mark every time a packet was received. I have tried putting the relevant code into ServerStateData::addVirginReplyBody() which seems to work for different protocols as well as being called for each packet received. Is there any problems with this approach? If not, I will refine and submit a patch. Yes, if you do not care about HTTP response header or FTP control messages: ServerStateData::addVirginReplyBody() is not called when the HTTP header or FTP control messages are received. If you want to catch those cases (it sounds like you do based on your every packet description above), then you need HttpStateData::readReply() and equivalent read handlers in other protocols. Ah, okay. Thanks for the explanation. I do not think there is a single Server method that is always called for every Squid read. You should probably add it and call it from the corresponding HTTP and FTP-specific code. I assume that I should add it to all the protocols? I may need some advice as to the correct functions within each one. Which one do you think for FTP? dataRead()? processReplyBody()? Please keep in mind that Squid does not work on a TCP packet level. It uses the TCP socket interface and the read(2) may happen when there are multiple packets in the buffer already. Got you. I was wondering that, as it seemed that the data was being buffered somewhere, hence the reason I kept testing my code further and further back, but to no avail. Thanks, Andy
Re: Improving qos_flows mark feature - obtaining mark later
On Mon, 31 Oct 2011 22:19:12 +, Andrew Beverley wrote: On Mon, 2011-10-31 at 15:42 -0600, Alex Rousskov wrote: On 10/31/2011 03:03 PM, Andrew Beverley wrote: Having thought about this further, I think what I was trying to achieve was getting the mark every time a packet was received. I have tried putting the relevant code into ServerStateData::addVirginReplyBody() which seems to work for different protocols as well as being called for each packet received. Is there any problems with this approach? If not, I will refine and submit a patch. Yes, if you do not care about HTTP response header or FTP control messages: ServerStateData::addVirginReplyBody() is not called when the HTTP header or FTP control messages are received. If you want to catch those cases (it sounds like you do based on your every packet description above), then you need HttpStateData::readReply() and equivalent read handlers in other protocols. Ah, okay. Thanks for the explanation. I do not think there is a single Server method that is always called for every Squid read. You should probably add it and call it from the corresponding HTTP and FTP-specific code. I assume that I should add it to all the protocols? I may need some advice as to the correct functions within each one. Which one do you think for FTP? dataRead()? processReplyBody()? All the read handlers. With type IOCB or taking CommIoCb* handlers needs a loot at. If you really want it every packet you can't get any closer than commHandleRead() in comm.cc which is what schedules those handlers. The ccb-conn available in there is the connection descriptor being read. Please keep in mind that Squid does not work on a TCP packet level. It uses the TCP socket interface and the read(2) may happen when there are multiple packets in the buffer already. Got you. I was wondering that, as it seemed that the data was being buffered somewhere, hence the reason I kept testing my code further and further back, but to no avail. Yes, its buffered in TCP, its buffered in squid, its processed in async chunk in squid (adaptation means a few more buffers to transit) and its buffered on write, then its buffered in TCP again. Each of these steps has a 'random' asynchronous delay as well as possibly aggregating (parser, chunking) and/or splitting (de-chunking, adaptation) the data size. Overall, I'm a little mystified as to why you need per-packet marks at all. Some real-time QoS or such? That is all simply outside of Squids scope of operation. Squid working with the HTTP per-request metric focus does make a lot more sense normally to grab the marks at or just after significant MiME boundaries. Before request-line, before headers, before body, and after body. Possibly at chunk boundaries. Amos
Re: Improving qos_flows mark feature - obtaining mark later
On Tue, 2011-11-01 at 12:12 +1300, Amos Jeffries wrote: On Mon, 31 Oct 2011 22:19:12 +, Andrew Beverley wrote: On Mon, 2011-10-31 at 15:42 -0600, Alex Rousskov wrote: On 10/31/2011 03:03 PM, Andrew Beverley wrote: Having thought about this further, I think what I was trying to achieve was getting the mark every time a packet was received. I have tried putting the relevant code into ServerStateData::addVirginReplyBody() which seems to work for different protocols as well as being called for each packet received. Is there any problems with this approach? If not, I will refine and submit a patch. Yes, if you do not care about HTTP response header or FTP control messages: ServerStateData::addVirginReplyBody() is not called when the HTTP header or FTP control messages are received. If you want to catch those cases (it sounds like you do based on your every packet description above), then you need HttpStateData::readReply() and equivalent read handlers in other protocols. Ah, okay. Thanks for the explanation. I do not think there is a single Server method that is always called for every Squid read. You should probably add it and call it from the corresponding HTTP and FTP-specific code. I assume that I should add it to all the protocols? I may need some advice as to the correct functions within each one. Which one do you think for FTP? dataRead()? processReplyBody()? All the read handlers. With type IOCB or taking CommIoCb* handlers needs a loot at. Okay. If you really want it every packet you can't get any closer than commHandleRead() in comm.cc which is what schedules those handlers. The ccb-conn available in there is the connection descriptor being read. It's been established that trying to achieve this is pointless, so I'll either stick with the above, or alternatively reduce it further as below. Please keep in mind that Squid does not work on a TCP packet level. It uses the TCP socket interface and the read(2) may happen when there are multiple packets in the buffer already. Got you. I was wondering that, as it seemed that the data was being buffered somewhere, hence the reason I kept testing my code further and further back, but to no avail. Yes, its buffered in TCP, its buffered in squid, its processed in async chunk in squid (adaptation means a few more buffers to transit) and its buffered on write, then its buffered in TCP again. Each of these steps has a 'random' asynchronous delay as well as possibly aggregating (parser, chunking) and/or splitting (de-chunking, adaptation) the data size. Overall, I'm a little mystified as to why you need per-packet marks at all. Some real-time QoS or such? That is all simply outside of Squids scope of operation. I'm doing both routing and traffic shaping with the mark value. Different bits in the mark value affect each of them. I only want the routing bits present in the connection to the server, otherwise they mess up the routing to the client connection. Routing is done before the packets have hit iptables, so it's too late to change the mark value there. It soon became apparent to me that trying to achieve the above by constantly changing the connection mark value was stupid, and that a better way was to mask the bits instead on the outbound packet, hence the patch I have just submitted. However, having looked into it, I thought that obtaining the mark value could be better anyway, so I thought I would work on that as well. Squid working with the HTTP per-request metric focus does make a lot more sense normally to grab the marks at or just after significant MiME boundaries. Before request-line, before headers, before body, and after body. Possibly at chunk boundaries. Okay, I'll take your recommendation on that. I don't suppose there is any particular single function that I can target, but instead that it needs to be done within each of those? Thanks, Andy
Re: Improving qos_flows mark feature - obtaining mark later
On Mon, Oct 17, 2011 at 5:30 AM, Amos Jeffries squ...@treenet.co.nz wrote: On 17/10/11 11:23, Andrew Beverley wrote: Hi, I've been using the qos_flows feature for preserving a netfilter mark, but have run into some problems. Currently, the netfilter mark for the connection is obtained in forward.cc, during the stages of opening a connection to the remote server. The problem with this is that the connection mark may change once the reply is received. So, I would like to move Ip::Qos::getNfmarkFromServer() from FwdState::dispatch to a function that is called during the server reply stage. However, I can't work out where it should go. I've looked at moving it to client_side_reply.cc, but I can't find a way to retrieve the remote server connection information. I've also looked at comm_read() in comm.cc, but I can't find the client connection information and it looks like the wrong place anyway. Where is the best place to move it to, where it has access to both the client and server side connection, but where it is late enough to read the mark value if it changes during the server reply? Hope this makes sense! FwdState::dispatch is called at the start of Server request sending. TO start the construction and write of request headers to the server. For persistent and pinned connections it is essentially in the pause position between end of reading one reply and start of sending the next request. I think you mean that it may change on reading the first bytes of reply, yes? In which case the best position is in HttpStateData::processReplyHeader(). With matching positions in ftp.cc and tunnel.cc reply starting handlers. The server connection is produced by a method dataConnection() of the state object. The client connection is not exposed. Although it could be. It is in the parent ServerStateData::FwdState::clientConn. Instead of moving it , why not doing both control points? like http_access / http_reply_access. It shouldn't hurt performance, and give greater flexibility.. -- /kinkie
Re: Improving qos_flows mark feature - obtaining mark later
On 10/16/2011 04:23 PM, Andrew Beverley wrote: I've been using the qos_flows feature for preserving a netfilter mark, but have run into some problems. Currently, the netfilter mark for the connection is obtained in forward.cc, during the stages of opening a connection to the remote server. The problem with this is that the connection mark may change once the reply is received. So, I would like to move Ip::Qos::getNfmarkFromServer() from FwdState::dispatch to a function that is called during the server reply stage. However, I can't work out where it should go. Could you define server reply stage more precisely, please? If you want that function to be called once per [HTTP] response, then the definition should be that of a single response state change and not a continuous reply stage. If you want that function to be called once per connection, then the definition should be based on connection-specific state changes and not on the response state change. Etc. I've looked at moving it to client_side_reply.cc, but I can't find a way to retrieve the remote server connection information. I've also looked at comm_read() in comm.cc, but I can't find the client connection information and it looks like the wrong place anyway. client_side_reply.cc is dealing with receiving responses from Store. Such responses may have no connection associated with them. If you must have access to the server connection, then you should look on the server side (forward.cc, Server.cc, http.cc, ftp.cc, etc. with tunnel.cc being a special case). See Amos' email for more details. BTW, code common to HTTP and FTP handling should go into Server.cc to the extent possible. Where is the best place to move it to, where it has access to both the client and server side connection, but where it is late enough to read the mark value if it changes during the server reply? Please note that during the server reply may take an hour. Do you want to do something once per reply, every time there are new bytes read from the origin server, or every time a new piece of response body is given to the client side, for example? Thank you, Alex.
Improving qos_flows mark feature - obtaining mark later
Hi, I've been using the qos_flows feature for preserving a netfilter mark, but have run into some problems. Currently, the netfilter mark for the connection is obtained in forward.cc, during the stages of opening a connection to the remote server. The problem with this is that the connection mark may change once the reply is received. So, I would like to move Ip::Qos::getNfmarkFromServer() from FwdState::dispatch to a function that is called during the server reply stage. However, I can't work out where it should go. I've looked at moving it to client_side_reply.cc, but I can't find a way to retrieve the remote server connection information. I've also looked at comm_read() in comm.cc, but I can't find the client connection information and it looks like the wrong place anyway. Where is the best place to move it to, where it has access to both the client and server side connection, but where it is late enough to read the mark value if it changes during the server reply? Hope this makes sense! Thanks, Andy
Re: Improving qos_flows mark feature - obtaining mark later
On 17/10/11 11:23, Andrew Beverley wrote: Hi, I've been using the qos_flows feature for preserving a netfilter mark, but have run into some problems. Currently, the netfilter mark for the connection is obtained in forward.cc, during the stages of opening a connection to the remote server. The problem with this is that the connection mark may change once the reply is received. So, I would like to move Ip::Qos::getNfmarkFromServer() from FwdState::dispatch to a function that is called during the server reply stage. However, I can't work out where it should go. I've looked at moving it to client_side_reply.cc, but I can't find a way to retrieve the remote server connection information. I've also looked at comm_read() in comm.cc, but I can't find the client connection information and it looks like the wrong place anyway. Where is the best place to move it to, where it has access to both the client and server side connection, but where it is late enough to read the mark value if it changes during the server reply? Hope this makes sense! FwdState::dispatch is called at the start of Server request sending. TO start the construction and write of request headers to the server. For persistent and pinned connections it is essentially in the pause position between end of reading one reply and start of sending the next request. I think you mean that it may change on reading the first bytes of reply, yes? In which case the best position is in HttpStateData::processReplyHeader(). With matching positions in ftp.cc and tunnel.cc reply starting handlers. The server connection is produced by a method dataConnection() of the state object. The client connection is not exposed. Although it could be. It is in the parent ServerStateData::FwdState::clientConn. Amos
