Re: [U-Boot] [PATCH] TCP and wget implementation v4

[Joe Hershberger]

On Fri, Jan 5, 2018 at 4:10 PM, Duncan Hare  wrote:
>> >
>> > A note on this TCP implementation. In TCP the transmitting TCP
>> > guarantees delivery of a stream, and the receiving TCP guarantees
>> > ordered of delivery of the stream. In this implementation The
>> > kernel memory buffer and the TCP sequence number is used to order
>> > the stream. for the application, and the application is the kernel
>> > itself. wget is not considered the application, and does receive
>> > packets "out of order."
>>
>> It seems like it would be possible to just store off a packet that is
>> ahead of its neighbor and not call any upper handler until the needed
>> packet arrives. Then all upper layers wouldn't need to know about the
>> reordering.
>
> There is, for u-boot a large number of buffers used on RX, in order
> to have a long work queue  of kernel data.
>
> CONFIG_SYS_RX_ETH_BUFFER 50
>
> The TCP transmit window is approximately 25 such packets, so overruns
> are eliminated.
>
> There are about 3300 kernel data packets in this test kernel, so missing
> a few, 3 to 5, has little impact on elapsed time, and they remain in the
> input buffer pool ahead of the HTTP header,
>
> The only critical order for this implementation is the TCP header.
> Without processing the TCP header we do not know the stream
> address of the first byte of kernel data. It is easy when we know this
> to map TCP stream address to kernel data offset.
>
>> >
>> > Advice on the reset buffer approach are invited. It requires an
>> > interface between the wget application to reset the buffer index.
>>
>> Between wget and what? The TCP implementation? It seems like something
>> that should be abstracted from wget.
>
> wget receives packets without intermediate ordering.
> Ordering data is a function of wget placing the kernel data at the
> correct offset, based of TCP stream location, at the correct memory
> address.
>
> wget is the agent which correctly orders data. There is no
> socket analogue, the abstraction, and a socket's linked list buffer
> reordering, which is the generally recognized reordering point for TCP
> data.
>
> Correct ordering is a primary task of this wget implementation,
> becuse the destination is a kernel image, and this choice very greatly
> simplifies the TCP stack, while reducing code complexity, and limits
> code size.
>
> Reordering would require a new piece of code similar to the fragment
> assemble piece of code in net.h, and that's an amazing, but complex,
> piece of code. My objective was to produce something as
> simple as possible.
>
> The TCP stack delivers packets in the order they come off the wire.
> Wget puts them in the correct order based on TCP sequence number to
> build the kernel image, and the TCP sequence number/ack protocol ensures
> complete delivery of a stream.

OK, it sounds like you've got a solution and a preference, so if it
makes more sense for this embedded implementation to maintain
simplicity, then so be it.

-Joe
___
U-Boot mailing list
U-Boot@lists.denx.de
https://lists.denx.de/listinfo/u-boot

Re: [U-Boot] [PATCH] TCP and wget implementation v4

[Duncan Hare]

> >
> > A note on this TCP implementation. In TCP the transmitting TCP
> > guarantees delivery of a stream, and the receiving TCP guarantees
> > ordered of delivery of the stream. In this implementation The
> > kernel memory buffer and the TCP sequence number is used to order
> > the stream. for the application, and the application is the kernel
> > itself. wget is not considered the application, and does receive
> > packets "out of order."  
> 
> It seems like it would be possible to just store off a packet that is
> ahead of its neighbor and not call any upper handler until the needed
> packet arrives. Then all upper layers wouldn't need to know about the
> reordering.

There is, for u-boot a large number of buffers used on RX, in order
to have a long work queue  of kernel data. 

CONFIG_SYS_RX_ETH_BUFFER 50 

The TCP transmit window is approximately 25 such packets, so overruns
are eliminated.

There are about 3300 kernel data packets in this test kernel, so missing
a few, 3 to 5, has little impact on elapsed time, and they remain in the
input buffer pool ahead of the HTTP header,

The only critical order for this implementation is the TCP header.
Without processing the TCP header we do not know the stream
address of the first byte of kernel data. It is easy when we know this
to map TCP stream address to kernel data offset.

> >
> > Advice on the reset buffer approach are invited. It requires an
> > interface between the wget application to reset the buffer index.  
> 
> Between wget and what? The TCP implementation? It seems like something
> that should be abstracted from wget.

wget receives packets without intermediate ordering.
Ordering data is a function of wget placing the kernel data at the
correct offset, based of TCP stream location, at the correct memory
address. 

wget is the agent which correctly orders data. There is no
socket analogue, the abstraction, and a socket's linked list buffer
reordering, which is the generally recognized reordering point for TCP
data.

Correct ordering is a primary task of this wget implementation,
becuse the destination is a kernel image, and this choice very greatly
simplifies the TCP stack, while reducing code complexity, and limits
code size.

Reordering would require a new piece of code similar to the fragment
assemble piece of code in net.h, and that's an amazing, but complex,
piece of code. My objective was to produce something as
simple as possible.

The TCP stack delivers packets in the order they come off the wire.
Wget puts them in the correct order based on TCP sequence number to
build the kernel image, and the TCP sequence number/ack protocol ensures
complete delivery of a stream.

  

___
U-Boot mailing list
U-Boot@lists.denx.de
https://lists.denx.de/listinfo/u-boot

Re: [U-Boot] [PATCH] TCP and wget implementation v4

[Joe Hershberger]

On Wed, Jan 3, 2018 at 5:01 PM, Duncan Hare  wrote:
>> >>>selects the LIB_RAND feature since it is required.
>> >
>> > Thanks: will be in u-boot/cmd/Kconfig
>> >
>> >>Are we lookin at a series of patches, or a concurrent set?
>> >
>> > At this time a series of three, but I'd take advice on the preferred
>> > procedure.
>>
>> Remember that the goal is to be atomic.
>>
>> You should be able to build and use U-Boot after each patch.
>>
>> Also, any changes to existing code that is not changing behavior but
>> simply making way for new functionality should be done separately.
>> Thanks
>> -Joe
>
> A note on this TCP implementation. In TCP the transmitting TCP
> guarantees delivery of a stream, and the receiving TCP guarantees
> ordered of delivery of the stream. In this implementation The
> kernel memory buffer and the TCP sequence number is used to order the stream.
> for the application, and the application is the kernel itself. wget is
> not considered the application, and does receive packets "out of order."

It seems like it would be possible to just store off a packet that is
ahead of its neighbor and not call any upper handler until the needed
packet arrives. Then all upper layers wouldn't need to know about the
reordering.

>
> This places a constraint on the incoming data stream. All
> (disordered) packets received before the HTTP header are
> ignored, which means the sending TCP will re-transmit them. This forced
> re-transmission could be avoided with a change to reprocess the
> incoming packet stream back to the first packet received, directly
> following processing the TCP header.
>
> This behavior was detected and failed downloads fixed in tests
> downloading Linux kernels from the cloud.
>
> Advice on the reset buffer approach are invited. It requires an
> interface between the wget application to reset the buffer index.

Between wget and what? The TCP implementation? It seems like something
that should be abstracted from wget.

>
> Joe Thanks, Good advice, based on that the order of 3 patches is:
>
> (1) Prepares the interfaces, no new behavior, CONFIG-TCP is introduced.
> net/Kconfig
> net/net.c
> net/ping.c
> include/net.h
>
> (2) Introduces TCP
> net/tcp.c
> net/tcp.h
> net/Makefile
>
> (3) Introduces wget, CONFIG-WGET introduced
> cmd/Kconfig
> cmd/net.c
> net/wget.c
> net/wget.h
>
>
> ___
> U-Boot mailing list
> U-Boot@lists.denx.de
> https://lists.denx.de/listinfo/u-boot
___
U-Boot mailing list
U-Boot@lists.denx.de
https://lists.denx.de/listinfo/u-boot

Re: [U-Boot] [PATCH] TCP and wget implementation v4

[Duncan Hare]

> >>>selects the LIB_RAND feature since it is required.  
> >
> > Thanks: will be in u-boot/cmd/Kconfig
> >  
> >>Are we lookin at a series of patches, or a concurrent set?  
> >
> > At this time a series of three, but I'd take advice on the preferred
> > procedure.  
> 
> Remember that the goal is to be atomic.
> 
> You should be able to build and use U-Boot after each patch.
> 
> Also, any changes to existing code that is not changing behavior but
> simply making way for new functionality should be done separately.
> Thanks
> -Joe

A note on this TCP implementation. In TCP the transmitting TCP
guarantees delivery of a stream, and the receiving TCP guarantees
ordered of delivery of the stream. In this implementation The
kernel memory buffer and the TCP sequence number is used to order the stream.
for the application, and the application is the kernel itself. wget is
not considered the application, and does receive packets "out of order."

This places a constraint on the incoming data stream. All
(disordered) packets received before the HTTP header are 
ignored, which means the sending TCP will re-transmit them. This forced
re-transmission could be avoided with a change to reprocess the
incoming packet stream back to the first packet received, directly
following processing the TCP header.

This behavior was detected and failed downloads fixed in tests
downloading Linux kernels from the cloud.

Advice on the reset buffer approach are invited. It requires an
interface between the wget application to reset the buffer index.


Joe Thanks, Good advice, based on that the order of 3 patches is:
 
(1) Prepares the interfaces, no new behavior, CONFIG-TCP is introduced.
net/Kconfig
net/net.c
net/ping.c
include/net.h

(2) Introduces TCP
net/tcp.c
net/tcp.h
net/Makefile

(3) Introduces wget, CONFIG-WGET introduced
cmd/Kconfig
cmd/net.c
net/wget.c
net/wget.h


___
U-Boot mailing list
U-Boot@lists.denx.de
https://lists.denx.de/listinfo/u-boot