Re: [lwip-users] lwIP with FreeRTOS memory problem

2016-11-28 Thread Noam Weissman
Hi,

I am not sure with my reply as this area is not so documented.

What I can see in your lwipopts that you use tons of memory for the LwIP heap 
but
other defines are way to small

For example:
#define MEMP_NUM_PBUF 16<<< ?
#define MEMP_NUM_UDP_PCB 4
#define MEMP_NUM_TCP_PCB 32
#define MEMP_NUM_TCP_PCB_LISTEN 8

I am using a micro (STM32F4 and F2) with limited memory and never had to use 
more than 20K RAM.

LwIP own opts.h file has macro’s in it and there is a reason for that. Some 
values have relations with other values.

I have attached my own lwipopts.h for your reference. Use it and hopefully it 
will help you.
Do understand that I have set up RAW API and Socket API as well. I use mostly 
RAW API but as
I am using PolarSSL I need the Socket API as well.


Good luck,
Noam.



From: lwip-users [mailto:lwip-users-bounces+noam=silrd@nongnu.org] On 
Behalf Of pekez
Sent: Monday, November 28, 2016 12:18 PM
To: Mailing list for lwIP users
Subject: [lwip-users] lwIP with FreeRTOS memory problem


Hello people,

I've been trying to figure out how lwIP uses memory and it's really important 
to know exact (or maximum) amount of memory that lwIP uses. I am using lwIP 
v1.4.1 with FreeRTOS v8.2.3 on ZYNQ custom board. Right now, I am working with 
application examples provided by Xilinx 
(XAPP1026).
 On ZYNQ board, I am running iperf server, 
and on PC iperf client.

When both MEMP_MEM_MALLOC and MEM_LIBC_MALLOC are 0 then everything works 
completely expected. However, when I set MEMP_MEM_MALLOC to 1 (according to RAM 
usage article from lwIP wiki), so that 
every piece of dynamically allocated memory comes from heap of MEM_SIZE size, 
application always crashes, no matter how big or small MEM_SIZE is (I have a 
lot of RAM, so I tried to put MEM_SIZE to even more then 20 MB!). Application 
crashes because two of FreeRTOS asserts fail, I get these two prints: "Assert 
failed in file queue.c, line 1224" and "Assert failed in file port.c, line 
424". According to call trace (which I provided in attachments), it crashes 
when XEmacPs_IntrHandler (I am using Xilinx layer 2 drivers) tries to allocate 
memory for pbuf.

Is this problem even related to lwIP or maybe to FreeRTOS or Xilinx drivers, 
what do you think? I am also providing the entire lwipopts.h, so that you can 
see whether some other options need to be changed in order to make application 
work with MEMP_MEM_MALLOC set to 1.

Thanks in advance.
/**
  **
  * @filelwipopts.h
  * @author  MCD Application Team
  * @version V1.1.0
  * @date07-October-2011
  * @brief   lwIP Options Configuration.
  *  This file is based on Utilities\lwip_v1.3.2\src\include\lwip\opt.h 
  *  and contains the lwIP configuration for the STM32F2x7 
demonstration.
  **
  * @attention
  *
  * THE PRESENT FIRMWARE WHICH IS FOR GUIDANCE ONLY AIMS AT PROVIDING CUSTOMERS
  * WITH CODING INFORMATION REGARDING THEIR PRODUCTS IN ORDER FOR THEM TO SAVE
  * TIME. AS A RESULT, STMICROELECTRONICS SHALL NOT BE HELD LIABLE FOR ANY
  * DIRECT, INDIRECT OR CONSEQUENTIAL DAMAGES WITH RESPECT TO ANY CLAIMS ARISING
  * FROM THE CONTENT OF SUCH FIRMWARE AND/OR THE USE MADE BY CUSTOMERS OF THE
  * CODING INFORMATION CONTAINED HEREIN IN CONNECTION WITH THEIR PRODUCTS.
  *
  *  COPYRIGHT 2011 STMicroelectronics
  **
  */

#ifndef __LWIPOPTS_H__
#define __LWIPOPTS_H__

#include "stdlib.h"

#include "Main.h"



/**
 * SYS_LIGHTWEIGHT_PROT==1: if you want inter-task protection for certain
 * critical regions during buffer allocation, deallocation and memory
 * allocation and deallocation.
 */
#define SYS_LIGHTWEIGHT_PROT1

#define ETHARP_TRUST_IP_MAC 0
#define IP_REASSEMBLY   0
#define IP_FRAG 0
#define ARP_QUEUEING0
#define TCP_LISTEN_BACKLOG  1
#define LWIP_DNS1
#define LWIP_NETIF_STATUS_CALLBACK 1

#define IP_SOF_BROADCAST1
#define IP_SOF_BROADCAST_RECV   1


#define LWIP_IGMP   1

#define SO_REUSE1
#define LWIP_NETIF_HOSTNAME 1

/**
 * NO_SYS==1: Provides VERY minimal functionality. Otherwise,
 * use lwIP facilities.
 */
#define NO_SYS  0

/* -- Memory options -- */
/* MEM_ALIGNMENT: should be set to the alignment of the CPU for which
   lwIP is compiled. 4 byte alignment -> define MEM_ALIGNMENT to 4, 2
   byte alignment -> define MEM_ALIGNMENT to 2. */
#define MEM_ALIGNMENT   4

/* MEM_SIZE: the size of the heap memory. If the application will send
   a lot of data that needs to be copied, this should be set high. */
#define MEM_SIZE(14 * 

Re: [lwip-users] lwIP with FreeRTOS memory problem

2016-11-28 Thread FreeRTOS Info


On 28/11/2016 12:17, Nenad Pekez wrote:

Richard, first of all, thanks for your quick response.

What I forgot to mention is that application doesn't crash at the very
beginning, it actually works for some time how it should work and than
suddenly crashes. I suppose that problem you mention would manifest
itself right from the beginning, when XEmacPs_IntrHandler is called for
the first time? Moreover, I thought that lwIP doesn't use standard C
malloc() and free() functions (if you are talking about those), but have
custom heap-based function mem_malloc(), UNLESS we set option
MEM_LIBC_MALLOC to 1 , which I haven't done.


Mistake on my part - I thought you were using malloc and free.

You said the assert gets hit.  The one on line queue.c:1224 guards 
against the use of a mutex in an interrupt and the one in port.c against 
a non-interrupt safe function being called from an interrupt - so if a 
mutex is being used in an interrupt, the deferred handler bit of my last 
email would still hold true.


Regards,
Richard.

+ http://www.FreeRTOS.org
The de facto standard, downloaded every 4.2 minutes during 2015.

+ http://www.FreeRTOS.org/plus
IoT, Trace, Certification, TCP/IP, FAT FS, Training, and more...


___
lwip-users mailing list
lwip-users@nongnu.org
https://lists.nongnu.org/mailman/listinfo/lwip-users


Re: [lwip-users] lwIP with FreeRTOS memory problem

2016-11-28 Thread Nenad Pekez

Richard, first of all, thanks for your quick response.

What I forgot to mention is that application doesn't crash at the very 
beginning, it actually works for some time how it should work and than 
suddenly crashes. I suppose that problem you mention would manifest 
itself right from the beginning, when XEmacPs_IntrHandler is called for 
the first time? Moreover, I thought that lwIP doesn't use standard C 
malloc() and free() functions (if you are talking about those), but have 
custom heap-based function mem_malloc(), UNLESS we set option 
MEM_LIBC_MALLOC to 1 , which I haven't done.


Regards,
Nenad

On 11/28/2016 4:16 PM, FreeRTOS Info wrote:


On 28/11/2016 02:17, pekez wrote:

Hello people,

I've been trying to figure out how lwIP uses memory and it's really
important to know exact (or maximum) amount of memory that lwIP uses. I
am using lwIP v1.4.1 with FreeRTOS v8.2.3 on ZYNQ custom board. Right
now, I am working with application examples provided by Xilinx (XAPP1026
). 


On ZYNQ board, I am running iperf 
server, and on PC iperf client.

When both MEMP_MEM_MALLOC and MEM_LIBC_MALLOC are 0 then everything
works completely expected. However, when I set MEMP_MEM_MALLOC to 1
(according to RAM usage article from lwIP wiki
), so that every piece of
dynamically allocated memory comes from heap of MEM_SIZE size,
application always crashes, no matter how big or small MEM_SIZE is (I
have a lot of RAM, so I tried to put MEM_SIZE to even more then 20 MB!).
Application crashes because two of FreeRTOS asserts fail, I get these
two prints: "Assert failed in file queue.c, line 1224" and "Assert
failed in file port.c, line 424". According to call trace (which I
provided in attachments), it crashes when XEmacPs_IntrHandler (I am
using Xilinx layer 2 drivers) tries to allocate memory for pbuf.

Is this problem even related to lwIP or maybe to FreeRTOS or Xilinx
drivers, what do you think? I am also providing the entire lwipopts.h,
so that you can see whether some other options need to be changed in
order to make application work with MEMP_MEM_MALLOC set to 1.


It sounds like you are trying to call malloc from an interrupt handler 
- which you can't do [in FreeRTOS] as the malloc is thread safe but 
not interrupt safe. One solution, albeit with a time penalty, is to 
use a deferred interrupt handler, where the IRQ does nothing more then 
unblock a very high priority task and clear the interrupt. Then the 
interrupt will return directly to the unblocked task to perform the 
actual processing necessary (the processing that was deferred from the 
interrupt). You can then optimise it so only interrupts that want to 
allocate memory get deferred to a task.


Example code snippets of how to do this can be found on the following 
page: 
http://www.freertos.org/RTOS_Task_Notification_As_Counting_Semaphore.html


Alternatively provide an additional heap for use just by the ISR so 
you don't get re-entrancy problems, or update the heap_n.c file 
(http://www.freertos.org/a00111.html) so it uses interrupt safe 
critical sections rather than task only critical sections (although 
this last option could make interrupts less responsive).


Regards,
Richard.

+ http://www.FreeRTOS.org
The de facto standard, downloaded every 4.2 minutes during 2015.

+ http://www.FreeRTOS.org/plus
IoT, Trace, Certification, TCP/IP, FAT FS, Training, and more...

___
lwip-users mailing list
lwip-users@nongnu.org
https://lists.nongnu.org/mailman/listinfo/lwip-users




---
This email has been checked for viruses by Avast antivirus software.
https://www.avast.com/antivirus


___
lwip-users mailing list
lwip-users@nongnu.org
https://lists.nongnu.org/mailman/listinfo/lwip-users


Re: [lwip-users] lwIP with FreeRTOS memory problem

2016-11-28 Thread FreeRTOS Info


On 28/11/2016 02:17, pekez wrote:

Hello people,

I've been trying to figure out how lwIP uses memory and it's really
important to know exact (or maximum) amount of memory that lwIP uses. I
am using lwIP v1.4.1 with FreeRTOS v8.2.3 on ZYNQ custom board. Right
now, I am working with application examples provided by Xilinx (XAPP1026
).
On ZYNQ board, I am running iperf 
server, and on PC iperf client.

When both MEMP_MEM_MALLOC and MEM_LIBC_MALLOC are 0 then everything
works completely expected. However, when I set MEMP_MEM_MALLOC to 1
(according to RAM usage article from lwIP wiki
), so that every piece of
dynamically allocated memory comes from heap of MEM_SIZE size,
application always crashes, no matter how big or small MEM_SIZE is (I
have a lot of RAM, so I tried to put MEM_SIZE to even more then 20 MB!).
Application crashes because two of FreeRTOS asserts fail, I get these
two prints: "Assert failed in file queue.c, line 1224" and "Assert
failed in file port.c, line 424". According to call trace (which I
provided in attachments), it crashes when XEmacPs_IntrHandler (I am
using Xilinx layer 2 drivers) tries to allocate memory for pbuf.

Is this problem even related to lwIP or maybe to FreeRTOS or Xilinx
drivers, what do you think? I am also providing the entire lwipopts.h,
so that you can see whether some other options need to be changed in
order to make application work with MEMP_MEM_MALLOC set to 1.


It sounds like you are trying to call malloc from an interrupt handler - 
which you can't do [in FreeRTOS] as the malloc is thread safe but not 
interrupt safe. One solution, albeit with a time penalty, is to use a 
deferred interrupt handler, where the IRQ does nothing more then unblock 
a very high priority task and clear the interrupt. Then the interrupt 
will return directly to the unblocked task to perform the actual 
processing necessary (the processing that was deferred from the 
interrupt). You can then optimise it so only interrupts that want to 
allocate memory get deferred to a task.


Example code snippets of how to do this can be found on the following 
page: 
http://www.freertos.org/RTOS_Task_Notification_As_Counting_Semaphore.html


Alternatively provide an additional heap for use just by the ISR so you 
don't get re-entrancy problems, or update the heap_n.c file 
(http://www.freertos.org/a00111.html) so it uses interrupt safe critical 
sections rather than task only critical sections (although this last 
option could make interrupts less responsive).


Regards,
Richard.

+ http://www.FreeRTOS.org
The de facto standard, downloaded every 4.2 minutes during 2015.

+ http://www.FreeRTOS.org/plus
IoT, Trace, Certification, TCP/IP, FAT FS, Training, and more...

___
lwip-users mailing list
lwip-users@nongnu.org
https://lists.nongnu.org/mailman/listinfo/lwip-users


Re: [lwip-users] TCP fragmentation over PPP - fragments are lost in LwIP if data is queued for sending

2016-11-28 Thread Simon Goldschmidt
Marco Jakobs wrote:
> We're not using nonblocking connections as this never did work properly … in 
> my today's
> debug session I've found why. In api_lib.c, line 610 this code just prevents 
> using
> NETCONN_WRITE for nonblocking connections:
> 
>  dontblock = netconn_is_nonblocking(conn) || (apiflags & NETCONN_DONTBLOCK);
>  if (dontblock && !bytes_written) {
>    /* This implies netconn_write() cannot be used for non-blocking send, since
>   it has no way to return the number of bytes written. */
>    return ERR_VAL;
>  }

This code is correct. You're just using the wrong function. For nonblocking, 
you need to
use 'netconn_write_partly()' instead of 'netconn_write()'. This ensures you get 
the number
of actually written bytes - as you saw, ERR_OK can be returned without all 
bytes written.

Returning ERR_WOULDBLOCK is not a solution as some bytes are already written 
(in that case at least).

The above code is simply missing a check for send-timeout. Using send-timeout 
without passing
a 'bytes_written' pointer (or using 'netconn_write()' instead of 
'netconn_write_partly()'),
does not work (for the reason described above). So the bug you found is that 
this check is missing.

> [..]
> Giving back an ERR_OK where data is definitely lost is clearly a bug,

Yes. I'll add the check that nonblocking write needs a 'bytes_left' pointer, 
too. Thanks for
tracking that down.

> so as a solution we've changed the SNDTIMEO handling in do_writemore in this 
> way:

Congrats, with this change, you effectively turned send-timeout into 
nonblocking ;-)


Simon

___
lwip-users mailing list
lwip-users@nongnu.org
https://lists.nongnu.org/mailman/listinfo/lwip-users

Re: [lwip-users] TCP fragmentation over PPP - fragments are lost in LwIP if data is queued for sending

2016-11-28 Thread Marco Jakobs
Hi Simon, Hi Sylvain,
 
we've found it!
 
We're not using nonblocking connections as this never did work properly … in my 
today's debug session I've found why. In api_lib.c, line 610 this code just 
prevents using NETCONN_WRITE for nonblocking connections:
 
  dontblock = netconn_is_nonblocking(conn) || (apiflags & NETCONN_DONTBLOCK);
  if (dontblock && !bytes_written) {
    /* This implies netconn_write() cannot be used for non-blocking send, since
   it has no way to return the number of bytes written. */
    return ERR_VAL;
  }
 
 
To avoid a too long blocking we're using send timeouts. And here was the issue.
 
As Neil already correctly has found, a part of the data was not written into 
the TCP send buffer. This happened in do_writemore in api_msg.c in the send 
timeout handling:
 
    if (conn->write_offset == 0) {
  /* nothing has been written */
  err = ERR_WOULDBLOCK;
  conn->current_msg->msg.w.len = 0;
    } else {
  /* partial write */
  err = ERR_OK;
  conn->current_msg->msg.w.len = conn->write_offset;
  conn->write_offset = 0;
    }
 
If NETCONN_WRITE wants to write data and the remaining space in the TCP send 
buffer is less than the data to be written, do_writemore will write as many 
data as it can. As the outgoing flow is quite slow, it'll then run into the 
send timeout to avoid a too long blocking. We'll get the "partial write" 
condition above and it'll return an ERR_OK which is passed as the result of the 
NETCONN_WRITE, so my application routing assumes the data is all written. In 
fact, it's only partly written and the data which did not match into the 
TCP_SNDBUF is lost here (which is the loss we have faced).
 
Giving back an ERR_OK where data is definitely lost is clearly a bug, so as a 
solution we've changed the SNDTIMEO handling in do_writemore in this way:
 
#if LWIP_SO_SNDTIMEO
    if (conn->send_timeout != 0)
    {
    dataptr = (u8_t*)conn->current_msg->msg.w.dataptr + conn->write_offset;
    diff = conn->current_msg->msg.w.len - conn->write_offset;
    if (diff > 0xUL)  /* max_u16_t */
   {
  len = 0x;
   }
   else
   {
  len = (u16_t)diff;
   }
    available = tcp_sndbuf(conn->pcb.tcp);
    if (available < len)
   {
  err = ERR_WOULDBLOCK;
  goto err_mem;
   }
    }
  if ((conn->send_timeout != 0) &&
  ((s32_t)(sys_now() - conn->current_msg->msg.w.time_started) >= 
conn->send_timeout)) {
    write_finished = 1;
    if (conn->write_offset == 0) {
  /* nothing has been written */
  err = ERR_WOULDBLOCK;
  conn->current_msg->msg.w.len = 0;
    } else {
  /* partial write */
  err = ERR_OK;
  conn->current_msg->msg.w.len = conn->write_offset;
  conn->write_offset = 0;
    }
  } else
#endif /* LWIP_SO_SNDTIMEO */
 
If the timeout is enabled and a sent_timeout is set for the connection, we 
check if the data to be sent will completely fit into the TCP_SNDBUF. If this 
is not the case, as we have no way to return the amount of unwritten bytes to 
the NETCONN_WRITE output, we'll not write anything but return an ERR_WOULDBLOCK 
back. That's a proper return for the my upper  layers to know that the data was 
not sent. It'll kept in my output queue and my IP-Server task will periodically 
retry to send the data out.
 
With that modification we don't see any data loss anymore.
 
This condition should however be checked from your side as it allows a 
potential data loss without the application getting aware of it.
 
Thank you for your time and ideas in that issue!
 
Marco___
lwip-users mailing list
lwip-users@nongnu.org
https://lists.nongnu.org/mailman/listinfo/lwip-users

[lwip-users] lwIP with FreeRTOS memory problem

2016-11-28 Thread pekez

Hello people,

I've been trying to figure out how lwIP uses memory and it's really 
important to know exact (or maximum) amount of memory that lwIP uses. I 
am using lwIP v1.4.1 with FreeRTOS v8.2.3 on ZYNQ custom board. Right 
now, I am working with application examples provided by Xilinx (XAPP1026 
). 
On ZYNQ board, I am running iperf  
server, and on PC iperf client.


When both MEMP_MEM_MALLOC and MEM_LIBC_MALLOC are 0 then everything 
works completely expected. However, when I set MEMP_MEM_MALLOC to 1 
(according to RAM usage article from lwIP wiki 
), so that every piece of 
dynamically allocated memory comes from heap of MEM_SIZE size, 
application always crashes, no matter how big or small MEM_SIZE is (I 
have a lot of RAM, so I tried to put MEM_SIZE to even more then 20 MB!). 
Application crashes because two of FreeRTOS asserts fail, I get these 
two prints: "Assert failed in file queue.c, line 1224" and "Assert 
failed in file port.c, line 424". According to call trace (which I 
provided in attachments), it crashes when XEmacPs_IntrHandler (I am 
using Xilinx layer 2 drivers) tries to allocate memory for pbuf.


Is this problem even related to lwIP or maybe to FreeRTOS or Xilinx 
drivers, what do you think? I am also providing the entire lwipopts.h, 
so that you can see whether some other options need to be changed in 
order to make application work with MEMP_MEM_MALLOC set to 1.


Thanks in advance.

#ifndef __LWIPOPTS_H_
#define __LWIPOPTS_H_

#ifndef PROCESSOR_LITTLE_ENDIAN
#define PROCESSOR_LITTLE_ENDIAN
#endif

#define SYS_LIGHTWEIGHT_PROT 1


#define OS_IS_FREERTOS
#define DEFAULT_THREAD_PRIO 2
#define TCPIP_THREAD_PRIO (2 + 1)
#define TCPIP_THREAD_STACKSIZE 1024
#define DEFAULT_TCP_RECVMBOX_SIZE   200
#define DEFAULT_ACCEPTMBOX_SIZE 5
#define TCPIP_MBOX_SIZE 200
#define DEFAULT_UDP_RECVMBOX_SIZE   100
#define DEFAULT_RAW_RECVMBOX_SIZE   30
#define LWIP_COMPAT_MUTEX 0
#define LWIP_ALLOW_MEM_FREE_FROM_OTHER_CONTEXT 1

#define MEM_ALIGNMENT 64
#define MEM_SIZE 786432
#define MEMP_NUM_PBUF 16
#define MEMP_NUM_UDP_PCB 4
#define MEMP_NUM_TCP_PCB 32
#define MEMP_NUM_TCP_PCB_LISTEN 8
#define MEMP_NUM_TCP_SEG 256
#define MEMP_NUM_SYS_TIMEOUT 8
#define MEMP_NUM_NETBUF 8
#define MEMP_NUM_NETCONN 16
#define MEMP_NUM_TCPIP_MSG_API 16
#define MEMP_NUM_TCPIP_MSG_INPKT 64
#define MEMP_MEM_MALLOC 1

#define MEMP_NUM_NETBUF 8
#define MEMP_NUM_NETCONN16
#define LWIP_PROVIDE_ERRNO  1
#define MEMP_NUM_SYS_TIMEOUT 8
#define PBUF_POOL_SIZE 256
#define PBUF_POOL_BUFSIZE 1700
#define PBUF_LINK_HLEN 16

#define ARP_TABLE_SIZE 10
#define ARP_QUEUEING 1

#define ICMP_TTL 255
#define LWIP_BROADCAST_PING 1
#define LWIP_MULTICAST_PING 1

#define IP_OPTIONS 0
#define IP_FORWARD 0
#define IP_REASSEMBLY 1
#define IP_FRAG 1
#define IP_REASS_MAX_PBUFS 128
#define IP_FRAG_MAX_MTU 1500
#define IP_DEFAULT_TTL 255
#define LWIP_CHKSUM_ALGORITHM 3

#define LWIP_UDP 0
#define UDP_TTL 255

#define LWIP_TCP 1
#define TCP_MSS 1460
#define TCP_SND_BUF 8192
#define TCP_WND 2048
#define TCP_TTL 255
#define TCP_MAXRTX 12
#define TCP_SYNMAXRTX 4
#define TCP_QUEUE_OOSEQ 1
#define TCP_SND_QUEUELEN   ((16 * TCP_SND_BUF)/TCP_MSS)
#define LWIP_SO_RECVBUF 0
#define LWIP_SO_RECVTIMEO 0
#define RECV_BUFSIZE_DEFAULT 8192
#define CHECKSUM_GEN_TCP0
#define CHECKSUM_GEN_UDP0
#define CHECKSUM_GEN_IP 0
#define CHECKSUM_CHECK_TCP  0
#define CHECKSUM_CHECK_UDP  0
#define CHECKSUM_CHECK_IP   0
#define LWIP_FULL_CSUM_OFFLOAD_RX  1
#define LWIP_FULL_CSUM_OFFLOAD_TX  1

#define NO_SYS_NO_TIMERS 1
#define MEMP_SEPARATE_POOLS 1
#define MEMP_NUM_FRAG_PBUF 256
#define IP_OPTIONS_ALLOWED 0
#define TCP_OVERSIZE TCP_MSS

#define LWIP_DHCP 0
#define DHCP_DOES_ARP_CHECK 0

#define CONFIG_LINKSPEED_AUTODETECT 1

#define LWIP_STATS 1
#define LWIP_STATS_DISPLAY 1

#endif
___
lwip-users mailing list
lwip-users@nongnu.org
https://lists.nongnu.org/mailman/listinfo/lwip-users