Re: CBOR encoding problems.

[Aditya Xavier]

Never mind, I think I got it working.

The problem it seems is that the Array has to be a const or declared inside the 
method.

Even though the compiler doesn’t throw a warning / error, thats what it needs.

PS, doing a os_mbuf_clear_chain  and then doing os_mbuf_get is that an 
efficient way to do this ?

Thanks,
Aditya Xavier.

> On 03-Apr-2018, at 1:19 PM, Aditya Xavier  wrote:
> 
> Hi Andrzej,
> 
> I have implemented the changes you suggested, however cbor_read_mbuf_attrs 
> causes a crash.
> 
> Side note, I don’t think, the problem has ever been regarding cbor decoding. 
> Rather its about cbor encoding, so am not sure how much cbor_read_mbuf_attrs 
> would help.
> 
> Also, no mater what method I use, am unable to get the cbor_int aray printed.
> 
> Thanks,
> Aditya Xavier.
> 
> 
> 
>> On 03-Apr-2018, at 11:25 AM, Aditya Xavier  wrote:
>> 
>> Thanks Andrzej,
>> 
>> Yeah I have made the changes to the app, to get it working ( Didn’t test the 
>> BLE message Truncation & MSYS issues yet).
>> 
>> Regarding your suggestion about using cbor_read_mbuf_attrs(), yes it does 
>> makes sense.
>> 
>> However, is there any way to use a mbuf as a scratchpad ? i.e. write 
>> something to it, erase it / overwrite on it, without re-initializing it ?
>> 
>> Because, even if am doing a os_mbuf_free / os_mbuf_free_chain, am still able 
>> to access the previous values.
>> 
>> Thanks,
>> Aditya Xavier.
>> 
>> 
>>> On 02-Apr-2018, at 11:34 PM, Andrzej Kaczmarek 
>>>  wrote:
>>> 
>>> Hi Aditya,
>>> 
>>> On Sun, Apr 1, 2018 at 3:51 PM, Aditya Xavier  wrote:
 Hi Andrzej,
 
 PFB my the test code, this is a torn down version of the code we were 
 trying to do.
 
 However it should be good enough for you to understand the objective and 
 the issues at hand.
 
 I did not personally try the code myself, I would be able to do so and let 
 you know the same tomorrow.
>>> 
>>> I checked the code and there are some issues wrt to mbuf handling
>>> which can cause problems.
>>> 
>>> Firstly, I don't see where you actually initialize cbor_codex_mbuf
>>> with proper mbuf as you neither received it from somewhere or call
>>> os_msys_get()/os_mbuf_get() to get it from a pool.
>>> Calling net_buf_simple_init() does not work here since it only
>>> initializes existing mbuf and you probably should not use it at all
>>> unless you're using this code for mesh. In case of mesh you can get
>>> valid mbuf by using NET_BUF_SIMPLE(). I assume this looks a bit
>>> different in your app as this sample app will most likely crash during
>>> startup.
>>> 
>>> Secondly, this probably causes data truncation:
>>> cbor_decode_action((char *)cbor_codex_mbuf->om_data, 
>>> cbor_codex_mbuf->om_len);
>>> Basically this way you pass contents of first mbuf in chain to
>>> decoding function, not complete chain. Internally this calls
>>> cbor_read_flat_attrs() but when working with mbufs you should use
>>> cbor_read_mbuf_attrs() instead and pass mbuf instead of its data.
>>> 
 Thanks,
 Aditya Xavier.
>>> 
>>> Best,
>>> Andrzej
>>> 
 
 
> On 31-Mar-2018, at 3:06 PM, Andrzej Kaczmarek 
>  wrote:
> 
> Hi Aditya,
> 
> On Sat, Mar 31, 2018 at 5:51 AM, Aditya Xavier  
> wrote:
>> Increasing MSYS_1_BLOCK_COUNT to 30 gets it to work, but if I increase 
>> it to 40 it doesn’t.
>> Block size being 110.
>> 
>> Any reasoning behind this pattern?
> 
> This is strange. Can you post some piece of code to show how you handle 
> this?
> 
>> Also what would is behaviour of os_mbuf_free? Do I need to re init it 
>> again?
> 
> Not sure what you mean by "reinit". Calling os_mbuf_free() releases
> mbuf back to pool and it should no longer be used. You should call
> os_mbuf_get() or os_msys_get() to get new mbuf from appropriate pool.
> Also, if your CBOR stream exceeds size of block then you will have
> chain of mbufs which should be freed using os_mbuf_free_chain()
> instead. Basically, if you're not sure whether your mbuf is just a
> single mbuf or chain of mbufs, you can always use os_mbuf_free_chain()
> instead of os_mbuf_free().
> 
>> Sent from my iPhone
>> 
>>> On 30-Mar-2018, at 1:58 AM, Christopher Collins  
>>> wrote:
>>> 
 On Thu, Mar 29, 2018 at 10:27:36PM +0530, Aditya Xavier wrote:
 Thanks for the tip. Would try that too..
 
 However, I have tried few variations to check the BLE side ( whether 
 it was responsible for truncating )
 
 1. I am able to send the same message using encoding only.. i.e. if I 
 trigger the same using a button to send over ble.
 i.e. BLE Connection, Button -> Encoding -> BLE Output, works.
 
 2. The method which 

Re: CBOR encoding problems.

[Andrzej Kaczmarek]

Hi Aditya,

On Sat, Mar 31, 2018 at 5:51 AM, Aditya Xavier  wrote:
> Increasing MSYS_1_BLOCK_COUNT to 30 gets it to work, but if I increase it to 
> 40 it doesn’t.
> Block size being 110.
>
> Any reasoning behind this pattern?

This is strange. Can you post some piece of code to show how you handle this?

> Also what would is behaviour of os_mbuf_free? Do I need to re init it again?

Not sure what you mean by "reinit". Calling os_mbuf_free() releases
mbuf back to pool and it should no longer be used. You should call
os_mbuf_get() or os_msys_get() to get new mbuf from appropriate pool.
Also, if your CBOR stream exceeds size of block then you will have
chain of mbufs which should be freed using os_mbuf_free_chain()
instead. Basically, if you're not sure whether your mbuf is just a
single mbuf or chain of mbufs, you can always use os_mbuf_free_chain()
instead of os_mbuf_free().

> Sent from my iPhone
>
>> On 30-Mar-2018, at 1:58 AM, Christopher Collins  wrote:
>>
>>> On Thu, Mar 29, 2018 at 10:27:36PM +0530, Aditya Xavier wrote:
>>> Thanks for the tip. Would try that too..
>>>
>>> However, I have tried few variations to check the BLE side ( whether it was 
>>> responsible for truncating )
>>>
>>> 1. I am able to send the same message using encoding only.. i.e. if I 
>>> trigger the same using a button to send over ble.
>>>i.e. BLE Connection, Button -> Encoding -> BLE Output, works.
>>>
>>> 2. The method which receives the char * data, itself receives a truncated 
>>> value.
>>>BLE Connection, BLE Incoming -> Decoding -> Encoding -> BLE Outgoing, 
>>> does not work. Truncation.
>>>
>>> 3. I am able to send the same message using encoding only.. i.e if I 
>>> trigger only the encoding method after receiving a message over BLE.
>>>i.e. BLE Connection, BLE Incoming -> Encoding -> BLE Outgoing, works.
>>>
>>> This kinda makes me feel that it should be a memory issue when its BLE + 
>>> Decoding + Encoding. With or without using mbuf / mbuf_pool
>>>
>>> Let me know if you require to see the code, I can write another small test 
>>> app for you.
>>
>> I think I understand the sequences you described.  I'm afraid I don't
>> have a good answer.
>>
>> Are you checking the return code of the encoding function?  If the
>> system is running out of mbufs during encoding, the function should
>> return `CborErrorOutOfMemory`, and the mbuf chain will be partially
>> filled.
>>
>> If you are running in gdb, another way to check for mbuf exhaustion is:
>> ```
>> p os_msys_init_1_mempool
>> ```
>>
>> and look at the `mp_min_free` value.  If it is 0, that means the pool
>> has been exhausted at some point, and it is likely that some allocations
>> have failed.
>>
>> You can also just try increasing the number of mbufs in the system:
>> ```
>> newt target amend  syscfg=MSYS_1_BLOCK_COUNT=16
>> ```
>>
>> (assuming you are currently using the default value of 12; adjust
>> accordingly if not)
>>
>> Chris

Re: CBOR encoding problems.

[Aditya Xavier]

Increasing MSYS_1_BLOCK_COUNT to 30 gets it to work, but if I increase it to 40 
it doesn’t.
Block size being 110.

Any reasoning behind this pattern?

Also what would is behaviour of os_mbuf_free? Do I need to re init it again?

Sent from my iPhone

> On 30-Mar-2018, at 1:58 AM, Christopher Collins  wrote:
> 
>> On Thu, Mar 29, 2018 at 10:27:36PM +0530, Aditya Xavier wrote:
>> Thanks for the tip. Would try that too..
>> 
>> However, I have tried few variations to check the BLE side ( whether it was 
>> responsible for truncating )
>> 
>> 1. I am able to send the same message using encoding only.. i.e. if I 
>> trigger the same using a button to send over ble.
>>i.e. BLE Connection, Button -> Encoding -> BLE Output, works.
>> 
>> 2. The method which receives the char * data, itself receives a truncated 
>> value.
>>BLE Connection, BLE Incoming -> Decoding -> Encoding -> BLE Outgoing, 
>> does not work. Truncation.
>> 
>> 3. I am able to send the same message using encoding only.. i.e if I trigger 
>> only the encoding method after receiving a message over BLE.
>>i.e. BLE Connection, BLE Incoming -> Encoding -> BLE Outgoing, works.
>> 
>> This kinda makes me feel that it should be a memory issue when its BLE + 
>> Decoding + Encoding. With or without using mbuf / mbuf_pool
>> 
>> Let me know if you require to see the code, I can write another small test 
>> app for you.
> 
> I think I understand the sequences you described.  I'm afraid I don't
> have a good answer.
> 
> Are you checking the return code of the encoding function?  If the
> system is running out of mbufs during encoding, the function should
> return `CborErrorOutOfMemory`, and the mbuf chain will be partially
> filled.
> 
> If you are running in gdb, another way to check for mbuf exhaustion is:
> ```
> p os_msys_init_1_mempool
> ```
> 
> and look at the `mp_min_free` value.  If it is 0, that means the pool
> has been exhausted at some point, and it is likely that some allocations
> have failed.
> 
> You can also just try increasing the number of mbufs in the system:
> ```
> newt target amend  syscfg=MSYS_1_BLOCK_COUNT=16
> ```
> 
> (assuming you are currently using the default value of 12; adjust
> accordingly if not)
> 
> Chris

Re: CBOR encoding problems.

[Christopher Collins]

On Thu, Mar 29, 2018 at 10:27:36PM +0530, Aditya Xavier wrote:
> Thanks for the tip. Would try that too..
> 
> However, I have tried few variations to check the BLE side ( whether it was 
> responsible for truncating )
> 
> 1. I am able to send the same message using encoding only.. i.e. if I trigger 
> the same using a button to send over ble.
>   i.e. BLE Connection, Button -> Encoding -> BLE Output, works.
> 
> 2. The method which receives the char * data, itself receives a truncated 
> value.
>   BLE Connection, BLE Incoming -> Decoding -> Encoding -> BLE Outgoing, 
> does not work. Truncation.
> 
> 3. I am able to send the same message using encoding only.. i.e if I trigger 
> only the encoding method after receiving a message over BLE.
>   i.e. BLE Connection, BLE Incoming -> Encoding -> BLE Outgoing, works.
> 
> This kinda makes me feel that it should be a memory issue when its BLE + 
> Decoding + Encoding. With or without using mbuf / mbuf_pool
> 
> Let me know if you require to see the code, I can write another small test 
> app for you.

I think I understand the sequences you described.  I'm afraid I don't
have a good answer.

Are you checking the return code of the encoding function?  If the
system is running out of mbufs during encoding, the function should
return `CborErrorOutOfMemory`, and the mbuf chain will be partially
filled.

If you are running in gdb, another way to check for mbuf exhaustion is:
```
p os_msys_init_1_mempool
```

and look at the `mp_min_free` value.  If it is 0, that means the pool
has been exhausted at some point, and it is likely that some allocations
have failed.

You can also just try increasing the number of mbufs in the system:
```
newt target amend  syscfg=MSYS_1_BLOCK_COUNT=16
```

(assuming you are currently using the default value of 12; adjust
accordingly if not)

Chris

Re: CBOR encoding problems.

[Christopher Collins]

Hi Aditya,

On Thu, Mar 29, 2018 at 08:52:08PM +0530, Aditya Xavier wrote:
[...]
> And it doesn’t work when am trying to trigger it from BLE. 
> 
> Assuming it was a memory issue, I used a mem_pool, mbuf, allocating
> and reserving space. However, problem remains.
> 
> It usually encodes and sends the following structure:
> 
> {“field1”:1, “field2”:2, “field3”:[{“field4”:4,
> 
> Which naturally fails because of incomplete cbor structure.

My guess is that the Bluetooth ATT MTU is too low to accommodate the
full packet.  The ATT layer is specified with the somewhat surprising
behavior of silently truncating overly-long attribute values [1].

How are you sending the CBOR data?  If you are using a simple "write
characteristic" procedure (`ble_gattc_write()`), you can use the "write
long characteristics" procedure instead (`ble_gattc_write_long()`),
which will fragment the long attribute value for you.  If you are
sending the CBOR data in a notification, on the other hand, then I'm
afraid the BLE stack doesn't offer any means of fragmentation; you'll
need to implement an application-layer fragmentation scheme.

Chris

[1] Perhaps we should add a syscfg setting which causes an over-long
write to indicate an error to the application instead of silently
truncating the data.  The stack wouldn't be standards-conforming in this
mode, but it seems more useful.