Otto, THANK YOU. The Gzip Plugin makes a whole bunch more sense and I must say that I like the way that the code is laid out.
After reading you code I think I can say that TSVConnWrite Tells the downstream VConn how much data it will ultimately need to expect. Presumably so that it can send the content-length header or alternatively send the data chunked encoding. Does that about sum it up? On 1/25/13 12:37 PM, "Otto van der Schaaf" <[email protected]> wrote: >re: "and I think it's also a little more experimental " > >I meant to say: and I think it has a more realistic function > >2013/1/25 Otto van der Schaaf <[email protected]>: >> I hope that works out, might be helpful and prevent other from having >> to go down the same road :-) I would be happy to help out with that. >> >> Regarding INKVIONBytesGet: >> >> "Gets the number of bytes to be performed by the IO operation >> described by viop. This is the nbytes parameter that's passed to >> INKVConnRead or INKVConnWrite." >> >> So, no, I don't think that modifying anything. If I remember >> correctly, that call returns the total number of bytes that the will >> be written to the passed in vio. So in the context of the picture >> above, you will write as much bytes to the output as you get from the >> input I guess. You can specify INT64_MAX if you don't know yet how >> much bytes you will write, in which case trafficserver will either >> send a connection:close header or send a transfer-encoding chunked. >> >> You might want to have a look at the gzip plugin in /experimental: I >> made an effort to make to code a little more understandable then the >> null-transform, and I think it's also a little more experimental :-) >> >> >> Regards, >> >> Otto >> >> >> 2013/1/25 Owens, Steve <[email protected]>: >>> Otto, >>> >>> I have started mapping code examples from the null transform example to >>> the diagram to try to connect the dots see attached. >>> >>> Steve Owens >>> >>> >>> >>> On 1/25/13 11:44 AM, "Owens, Steve" <[email protected]> wrote: >>> >>>>Otto, yes I have seen them and read them several times. The second >>>>link >>>>explains in rather abstract terms what VIO's do and how they relate to >>>>buffers. >>>> >>>>But I personally could use a little more amplification on what the >>>>specific methods are that manipulate these structures and what effect >>>>that >>>>calls to these methods have on the structures depicted in the images. >>>> >>>>The null transform plugin re-uses variables and takes enough shortcuts >>>>that what it is doing is seems obscured and I am trying to pierce that >>>>obscurity. I will continue to try to pierce the veil but after having >>>>done so I would love to write up additional info to make it clearer to >>>>dunderheads such as myself what these methods actually do in order to >>>>reduce the slope of the learning curve for this technology. >>>> >>>>Thank you for re-directing me to these I am sure that given enough time >>>>and patience I will crack the code. >>>> >>>> >>>> >>>>On 1/25/13 11:25 AM, "Otto van der Schaaf" <[email protected]> wrote: >>>> >>>>>Hi Steve, >>>>> >>>>>Did you get a chance already to have a look at >>>>>http://trafficserver.apache.org/docs/v2/sdk/images/vconnection.jpg ? >>>>>That image is from >>>>>http://trafficserver.apache.org/docs/v2/sdk/HTTPTransformationPlugins. >>>>>htm >>>>>l >>>>>#VIOs >>>>>Both helped my understanding of the transformation api's >>>>> >>>>>Regards, >>>>> >>>>>Otto >>>>> >>>>> >>>>>2013/1/25 Owens, Steve <[email protected]>: >>>>>> One step closer >>>>>> >>>>>> VIO * >>>>>> >>>>>> PluginVC::do_io_write(Continuation * c, int64_t nbytes, >>>>>>IOBufferReader >>>>>>* >>>>>> abuffer, bool owner) >>>>>> >>>>>> { >>>>>> >>>>>> >>>>>> ink_assert(!closed); >>>>>> >>>>>> ink_assert(magic == PLUGIN_VC_MAGIC_ALIVE); >>>>>> >>>>>> >>>>>> if (abuffer) { >>>>>> >>>>>> ink_assert(!owner); >>>>>> >>>>>> write_state.vio.buffer.reader_for(abuffer); >>>>>> >>>>>> } else { >>>>>> >>>>>> write_state.vio.buffer.clear(); >>>>>> >>>>>> } >>>>>> >>>>>> >>>>>> // Note: we set vio.op last because process_write_side looks at >>>>>>it to >>>>>> >>>>>> // tell if the VConnection is active. >>>>>> >>>>>> write_state.vio.mutex = c->mutex; >>>>>> >>>>>> write_state.vio._cont = c; >>>>>> >>>>>> write_state.vio.nbytes = nbytes; >>>>>> >>>>>> write_state.vio.ndone = 0; >>>>>> >>>>>> write_state.vio.vc_server = (VConnection *) this; >>>>>> >>>>>> write_state.vio.op = VIO::WRITE; >>>>>> >>>>>> >>>>>> Debug("pvc", "[%u] %s: do_io_write for %"PRId64" bytes", PVC_ID, >>>>>>PVC_TYPE, >>>>>> nbytes); >>>>>> >>>>>> >>>>>> // Since reentrant callbacks are not allowed on from do_io >>>>>> >>>>>> // functions schedule ourselves get on a different stack >>>>>> >>>>>> need_write_process = true; >>>>>> >>>>>> setup_event_cb(0, &sm_lock_retry_event); >>>>>> >>>>>> >>>>>> return &write_state.vio; >>>>>> >>>>>> } >>>>>> >>>>>> >>>>>> It would appear that the value passed in to the third argument of >>>>>> TSVConnWrite ends up being assigned to write_state.vio.nbytes >>>>>> >>>>>> the docs on write_state.vio.nbytes state: >>>>>> >>>>>> /** >>>>>> >>>>>> Number of bytes to be done for this operation. >>>>>> >>>>>> >>>>>> The total number of bytes this operation must complete. >>>>>> >>>>>> >>>>>> */ >>>>>> >>>>>> int64_t nbytes; >>>>>> >>>>>> >>>>>> So far clear as mud. Apologies in advance for my mental density. >>>>>> >>>>>> >>>>>> >>>>>> From: "Owens, Steve" <[email protected]> >>>>>> Reply-To: "[email protected]" >>>>>><[email protected]> >>>>>> Date: Fri, 25 Jan 2013 11:09:08 -0800 >>>>>> To: "[email protected]" >>>>>><[email protected]> >>>>>> Subject: Re: Api Questions >>>>>> >>>>>> An addendum to the previous message I have resorted to reading the >>>>>>traffic >>>>>> server code in order to try to discern what it does: >>>>>> >>>>>> I am examining >>>>>> >>>>>> TSVIO >>>>>> >>>>>> TSVConnWrite(TSVConn connp, TSCont contp, TSIOBufferReader readerp, >>>>>>int64_t >>>>>> nbytes) >>>>>> >>>>>> { >>>>>> >>>>>> sdk_assert(sdk_sanity_check_iocore_structure(connp) == >>>>>>TS_SUCCESS); >>>>>> >>>>>> sdk_assert(sdk_sanity_check_iocore_structure(contp) == >>>>>>TS_SUCCESS); >>>>>> >>>>>> sdk_assert(sdk_sanity_check_iocore_structure(readerp) == >>>>>>TS_SUCCESS); >>>>>> >>>>>> sdk_assert(nbytes >= 0); >>>>>> >>>>>> >>>>>> FORCE_PLUGIN_MUTEX(contp); >>>>>> >>>>>> VConnection *vc = (VConnection *) connp; >>>>>> >>>>>> >>>>>> return reinterpret_cast<TSVIO>(vc->do_io_write((INKContInternal *) >>>>>>contp, >>>>>> nbytes, (IOBufferReader *) readerp)); >>>>>> >>>>>> } >>>>>> >>>>>> >>>>>> It would appear that the answer to what this method does lies in >>>>>>what >>>>>> happens in the function "vc->do_io_write" yet when I search the >>>>>> trafficserver 3.2.0 codebase for any file that actually contains an >>>>>> implementation of "vc->do_io_write" I find that "do_io_write" is >>>>>>declared in >>>>>> several .h files but not defined in any .c or .cpp files. >>>>>> >>>>>> >>>>>> Does anyone know where in the code base the implementation of >>>>>>"do_io_write" >>>>>> resides? >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> From: "Owens, Steve" <[email protected]> >>>>>> Reply-To: "[email protected]" >>>>>><[email protected]> >>>>>> Date: Fri, 25 Jan 2013 10:48:11 -0800 >>>>>> To: "[email protected]" >>>>>><[email protected]> >>>>>> Subject: Api Questions >>>>>> >>>>>> I am trying to understand the relationship between the following >>>>>>functions >>>>>> as they are used within a transformation plugin and so far no matter >>>>>>how >>>>>> hard I dig the mystery remains. I am hoping that people who have a >>>>>>better >>>>>> understanding of ATS than I do would be willing to help clear up any >>>>>> misunderstandings I have. >>>>>> >>>>>> Here are the functions I find confusing: >>>>>> >>>>>> TSVIONBytesSet >>>>>> >>>>>> (TSIOBufferCopy and or TSIOBufferWrite) >>>>>> >>>>>> And finally >>>>>> >>>>>> TSVConnWrite >>>>>> >>>>>> >>>>>> Each of the above functions takes an int_64 argument at the end to >>>>>>indicate >>>>>> the number of bytes but I don't fully understand the purpose of this >>>>>> parameter with respect to each of the functions. >>>>>> >>>>>> >>>>>> Suppose I have a transformation which is intended to write a prefix >>>>>>spew the >>>>>> response and then write a suffix to the down stream I would imagine >>>>>>it >>>>>>looks >>>>>> something like this: >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> void handleTransformOperation(TSCont contp) { >>>>>> >>>>>> TSIOBuffer buf_test; >>>>>> >>>>>> int_64 written = 0; >>>>>> >>>>>> TransormationData* pData = (TransformationData*) >>>>>>TSContDataGet(contp); >>>>>> >>>>>> TSVIO input_vio = TSTransformOutputVConnGet(contp); >>>>>> >>>>>> >>>>>> if(data->state == STATE_BEGIN) { >>>>>> >>>>>> // Initialize the output_buffer and output_reader >>>>>> >>>>>> data->output_buffer = TSIOBufferCreate(); >>>>>> >>>>>> data->output_reader = >>>>>>TSIOBufferReaderAlloc(data->output_buffer); >>>>>> >>>>>> >>>>>> >>>>>> // Write out a prefix >>>>>> >>>>>> written += TSIOBufferWrite(data->output_buffer, "{Hello I am a >>>>>>prefix}", >>>>>> strlen(prefix) * sizeof(char)); >>>>>> >>>>>> data->state == STATE_MIDDLE; >>>>>> >>>>>> >>>>>> // Figure out if the upstream wants to stop receiving WRITEREADY >>>>>>and >>>>>> WRITE_COMPLETE events >>>>>> >>>>>> buf_test = TSVIOBufferGet(input_vio); >>>>>> >>>>>> if (!buf_test) { >>>>>> >>>>>> int64_t doneget = TSVIONDoneGet(input_vio); >>>>>> >>>>>> TSVIONBytesSet(data->output_vio, /*What goes here doneget or >>>>>>written >>>>>> + doneget or just written? Bonus question: why? /* ???); >>>>>> >>>>>> data->state = STATE_END; >>>>>> >>>>>> } else >>>>>> >>>>>> data->state = STATE_MIDDLE; >>>>>> >>>>>> } >>>>>> >>>>>> >>>>>> // Notice that the above if falls through to the next state so it is >>>>>> possible that the following code may or may not >>>>>> >>>>>> // get executed based on the buf_test result above. >>>>>> >>>>>> if(data->state == STATE_MIDDLE) { >>>>>> >>>>>> // Figure out if the upstream wants to stop receiving WRITEREADY >>>>>>and >>>>>> WRITE_COMPLETE events >>>>>> >>>>>> buf_test = TSVIOBufferGet(input_vio); >>>>>> >>>>>> if (!buf_test) { >>>>>> >>>>>> int64_t doneget = TSVIONDoneGet(input_vio); >>>>>> >>>>>> TSVIONBytesSet(data->output_vio, /*What goes here doneget or >>>>>>written >>>>>> + doneget or just written? Bonus question: why? /* ???); >>>>>> >>>>>> data->state = STATE_END; >>>>>> >>>>>> } else { >>>>>> >>>>>> // Figure out how much data is left to read from the upstream >>>>>> >>>>>> int64_t upstreamBytesRemaining = TSVIONTodoGet(input_vio); >>>>>> >>>>>> int64_t upstreamBytesAvail = >>>>>> TSIOBufferReaderAvail(TSVIOReaderGet(input_vio)); >>>>>> >>>>>> if(upstreamBytesRemaining > 0) { >>>>>> >>>>>> int64_t bytesToWrite = upstreamBytesRemaining; >>>>>> >>>>>> if(bytesToWrite > upstreamBytesAvail) >>>>>> >>>>>> bytesToWrite = upstreamBytesAvail; >>>>>> >>>>>> >>>>>> >>>>>> /* Copy the data from the read buffer to the output >>>>>>buffer. >>>>>>*/ >>>>>> >>>>>> TSIOBufferCopy(TSVIOBufferGet(data->output_vio), >>>>>> TSVIOReaderGet(input_vio), bytesToWrite, 0); >>>>>> >>>>>> >>>>>> /* Tell the read buffer that we have read the data and >>>>>>are >>>>>>no >>>>>> >>>>>> * longer interested in it. >>>>>> >>>>>> */ >>>>>> >>>>>> TSIOBufferReaderConsume(TSVIOReaderGet(input_vio), >>>>>>bytesToWrite); >>>>>> >>>>>> >>>>>> /* Modify the input VIO to reflect how much data we've >>>>>> >>>>>> * completed. >>>>>> >>>>>> */ >>>>>> >>>>>> int64_t doneget = TSVIONDoneGet(input_vio); >>>>>> >>>>>> TSVIONDoneSet(input_vio, /* This is where things get really muddy >>>>>>what >>>>>> goes here? doneget or doneget + bytesToWrite or doneget + >>>>>>bytesToWrite >>>>>>+ >>>>>> written */ ??); >>>>>> >>>>>> /* What is the impact of calling this later in the >>>>>>handler >>>>>>if it >>>>>> has already been called here? >>>>>> >>>>>> } >>>>>> >>>>>> >>>>>> /* Now we check the input VIO to see if there is data left to >>>>>> >>>>>> * read. >>>>>> >>>>>> */ >>>>>> >>>>>> upstreamBytesRemaining = TSVIONTodoGet(input_vio); >>>>>> >>>>>> if (upstreamBytesRemaining > 0) { >>>>>> >>>>>> if (upstreamBytesAvail > 0) { >>>>>> >>>>>> /* If there is data left to read, then we reenable the output >>>>>> >>>>>> * connection by reenabling the output VIO. This will wake up >>>>>> >>>>>> * the output connection and allow it to consume data from the >>>>>> >>>>>> * output buffer. >>>>>> >>>>>> */ >>>>>> >>>>>> TSVIOReenable(data->output_vio); >>>>>> >>>>>> >>>>>> /* Call back the input VIO continuation to let it know that >>>>>>we >>>>>> >>>>>> * are ready for more data. >>>>>> >>>>>> */ >>>>>> >>>>>> TSContCall(TSVIOContGet(input_vio), >>>>>> >>>>>> TS_EVENT_VCONN_WRITE_READY, input_vio); >>>>>> >>>>>> } >>>>>> >>>>>> } else { >>>>>> >>>>>> /* There is no data left to read so set state to STATE_END and >>>>>>*/ >>>>>> >>>>>> data->state = STATE_END; >>>>>> >>>>>> } >>>>>> >>>>>> } >>>>>> >>>>>> >>>>>> // The above can fall through to this if block >>>>>> >>>>>> if(data->state == STATE_END) { >>>>>> >>>>>> /* Spew a suffix */ >>>>>> >>>>>> const char* suffix = "{i am a suffix}"; >>>>>> >>>>>> int64_t localWritten = TSIOBufferWrite(data->output_buffer, >>>>>>suffix, >>>>>> (strlen(suffix) - 1) * sizeof(char)); >>>>>> >>>>>> >>>>>> >>>>>> int64_t doneget = TSVIONDoneGet(input_vio); >>>>>> >>>>>> TSVIONBytesSet(data->output_vio, /* What goes here? Why? >>>>>>*/??); >>>>>> >>>>>> TSVIOReenable(data->output_vio); >>>>>> >>>>>> >>>>>> /* Call back the input VIO continuation to let it know that we >>>>>> >>>>>> * have completed the write operation. >>>>>> >>>>>> */ >>>>>> >>>>>> TSContCall(TSVIOContGet(input_vio), >>>>>>TS_EVENT_VCONN_WRITE_COMPLETE, >>>>>> input_vio); >>>>>> >>>>>> } >>>>>> >>>>>> >>>>>> /* I am presuming we only want to call this once per invocation >>>>>>of >>>>>> handlerOperation But I am not sure */ >>>>>> >>>>>> data->output_vio = TSVConnWrite(output_conn, contp, >>>>>>data->output_reader, >>>>>> /* What to put here hmmm and why? */ ?? ); >>>>>> >>>>>> } >>>>>> >>>>>> >>>>>> >>>>>> I really want to understand how these methods relate and what they >>>>>>do. >>>>>>I >>>>>> would be willing to help update/contribute to the ATS docs but >>>>>>first I >>>>>>would >>>>>> need to understand what they do and so far the examples that are >>>>>>available >>>>>> have not made it clear to me what is going on as yet so any help >>>>>>would >>>>>>be >>>>>> appreciated. >>>>>> >>>>>> >>>>>> >>>>>> >>>> >>>
