On Tue, May 30, 2023 at 12:10 PM 'Dieter Weidenbrück' via
emscripten-discuss <emscripten-discuss@googlegroups.com> wrote:
> All,
> with lots of debug lines I am able to isolate one error in my code,
> however, I have not been able yet to build a reduced test case for it.
> I tested this with a single worker to avoid races.
>
> The pointer in question got allocated in the worker first. Later I needed
> to enlarge the size. Using code like the following snippet I detected that
> the content of the reallocated block was different from the original one
> (considering the old size only, of course):
> Assume p is the block that was allocated initially with size oldSize:
> #define smallChunkSize 256
> uint8_T test1[smallChunkSize],test2[smallChunkSize];
> uint8Ptr_T p = (uint8Ptr_T)myP;
> uint8Ptr_T newP = 0L;
> int i,n;
>
> n = min(oldSize,smallChunkSize); memcpy(test1,p,n);
> newP = realloc(p,newSize);
> assert(newP != 0L);
>
> memcpy(test2,newP,n);
>
> for (i=0;i<n;i++){
> assert(test1[i] == test2[i]);
> }
>
>
Is newSize always greater than n ?
Are there other workers (or the main thread) that might be accessing `myP`
while this code is running?
> The newP contained garbage.
> I can't say when this happens, but I can reproduce it with my code. A wild
> guess is that it happens during a heap resize, but I don't have tools to
> verify this.
> Just as a reminder, if I set the number of workers to 0 and run this code
> in a single thread nothing happens.
>
> As a workaround I tried to allocate a new block, copy everything from the
> old block to the new one, and dispose of the old block. That resolves this
> particular problem, but then other problems arise.
> So I will keep searching.
>
> Cheers,
> Dieter
>
>
> jj schrieb am Sonntag, 28. Mai 2023 um 14:51:57 UTC+2:
>
>> Oops, that looks like an oversight with emmalloc + multithreading +
>> memvalidate assertion checks implementation. Posted a fix PR to that at
>> https://github.com/emscripten-core/emscripten/pull/19465
>>
>> On Sun, May 28, 2023 at 11:32 AM 'Dieter Weidenbrück' via
>> emscripten-discuss <emscripte...@googlegroups.com> wrote:
>>
>>> One more detail:
>>> - use hello_wasm_worker.c from the test lib without changes
>>> - use
>>> emcc ./hello_wasm_worker.c ^
>>> -g3 ^
>>> --source-map-base ./ ^
>>> -gsource-map ^
>>> -s ALLOW_MEMORY_GROWTH=1 ^
>>> -s ENVIRONMENT=web,worker ^
>>> -s SUPPORT_ERRNO=0 ^
>>> -sWASM_WORKERS ^
>>> -s ASSERTIONS=2 ^
>>> -sMALLOC=emmalloc-memvalidate ^
>>> -o hello_wasm_worker.html
>>>
>>> Result:
>>> [image: hello_error.png]
>>>
>>> Tested on Dell XPS17, 12th Gen Intel(R) Core(TM) i9-12900HK 2.50 GHz,
>>> 64 GB RAM, Windows 11 64bit
>>> Browsers: current Chrome and Firefox
>>>
>>> Hope this helps,
>>> Dieter
>>> Dieter Weidenbrück schrieb am Samstag, 27. Mai 2023 um 18:03:37 UTC+2:
>>>
>>>> JJ,
>>>> thanks for your recommendations. Here are some first results (I hope it
>>>> is ok to post these screenshots here, if not I can post links):
>>>>
>>>> SAFE_HEAP switched off.
>>>>
>>>> Using *emmalloc*:
>>>> First try:
>>>> [image: emmalloc.png]
>>>>
>>>> Second try:
>>>> [image: emmalloc2.png]
>>>> Third try (try == same binary, just restarted and performed the same
>>>> steps)
>>>> [image: emmalloc3.png]
>>>> So the behavior is not really predictable nor reproducable.
>>>>
>>>> Using *emmalloc-debug*:
>>>> First try:
>>>> [image: emmalloc-debug.png]
>>>>
>>>> Second try:
>>>> [image: emmalloc-debug2.png]
>>>>
>>>> In both cases Chrome was left in a state using 98+% of the CPU without
>>>> returning. I had to kill Chrome
>>>>
>>>> Using *emmalloc-memvalidate*:
>>>> [image: emmalloc-memvalidate.png]
>>>> Right after startup without any interaction from my side, the app
>>>> aborted with this message.
>>>>
>>>> Setting the number of wasm workers used to 0 (i.e. run as a single
>>>> thread app), everything worked fine with emmalloc and emmalloc-debug. For
>>>> emmalloc-memvalidate I had to #if 0 all calls to wasm workers and remove
>>>> -sWASM_WORKERS, then the app would run without problems.
>>>>
>>>> *A comment about debugging in general:*
>>>> source code debugging in a browser is a tedious task. Every line
>>>> requires several clicks/keys to advance, and I have not found a way to
>>>> inspect the values of variables.
>>>> If I have to run through a loop with a couple of hundred or thousand
>>>> iterations this doesn't help a lot to raise my enthusiasm.
>>>> So what I do is, I have set up a project in Visual Studio with a
>>>> slightly different main function as a cmdline-app. From there I can call
>>>> specific functions and debug them with all bells and whistles.
>>>> So I have some good confidence that the code works before entering the
>>>> wasm/browser world.
>>>>
>>>> Cheers,
>>>> Dieter
>>>>
>>>> jj schrieb am Samstag, 27. Mai 2023 um 01:25:24 UTC+2:
>>>>
>>>>> This is a good bug report about debuggability shortcomings of Wasm on
>>>>> two accounts:
>>>>> 1. when we had the old JS-based SAFE_HEAP feature, the feature said
>>>>> something along the lines of "segmentation fault: attempted to write i32
>>>>> value x to memory address y", so one could deduce whether the memory
>>>>> address was zero, out of wasm memory bounds, or compare it against the
>>>>> memory map, e.g. if it is out of the dynamic malloc region memory bounds
>>>>> even if it might have been a dynamically allocated ptr. And one could
>>>>> verify the alignment as well, to use that as a heuristic to figure out if
>>>>> the ptr was stomped on and was garbled.
>>>>>
>>>>> The wasm-based SAFE_HEAP feature does not provide any of above, but
>>>>> only prints that "Aborted(segmentation fault)".
>>>>>
>>>>> 2. Wasm browser debuggers should be able to show the line of code that
>>>>> does the segfault operation. Not sure if the browser debuggers are yet
>>>>> powerful enough to do that. Also currently we have no integration with
>>>>> those - maybe a SAFE_HEAP should generate a wasm trap instead of going out
>>>>> to JS to throw a JS exception? (although catching that exception should
>>>>> allow one to still manually inspect the callstack to find the offending
>>>>> line)
>>>>>
>>>>> Given that there exists an error print into dlmalloc, might be
>>>>> interesting to see how the error message adjust if you build against
>>>>> emmalloc, by using the linker flag -sMALLOC=emmalloc. There also exist
>>>>> variants -sMALLOC=emmalloc-debug, -sMALLOC=emmalloc-memvalidate, and
>>>>> -sMALLOC=emmalloc-memvalidate-verbose with increasing levels of dynamic
>>>>> memory allocator internal consistency checks (and increasing levels of
>>>>> slowness and log spamming).
>>>>>
>>>>> If nothing stands out, getting to a reduced repro test case would be
>>>>> helpful.
>>>>>
>>>>> On Fri, May 26, 2023 at 11:47 PM 'Sam Clegg' via emscripten-discuss <
>>>>> emscripte...@googlegroups.com> wrote:
>>>>>
>>>>>> Can I ask why you chose not to use pthreads to start with? I'd like
>>>>>> to understand better why folks would choose wasm workers over pthreads.
>>>>>>
>>>>>> On Fri, May 26, 2023 at 3:25 AM 'Dieter Weidenbrück' via
>>>>>> emscripten-discuss <emscripte...@googlegroups.com> wrote:
>>>>>>
>>>>>>> Hi Sam,
>>>>>>> IIRC, when I started with Emscripten a while ago the program would
>>>>>>> abort in case of a memory error. As my app is comparable to a desktop
>>>>>>> app,
>>>>>>> this was not acceptable, so I set ABORTING_MALLOC to 0. I understand
>>>>>>> that
>>>>>>> this flag has a different meaning today. Here is how all my allocation
>>>>>>> calls work:
>>>>>>>
>>>>>>> Error_T allocMemPtr(MemPtr_T *p,uint32_T size,boolean_T clear) {
>>>>>>> _MemPtr_T mp;
>>>>>>>
>>>>>>> if (clear)
>>>>>>> mp = (_MemPtr_T)calloc(1,size + sizeof(_Mem_T));
>>>>>>> else
>>>>>>> mp = (_MemPtr_T)malloc(size + sizeof(_Mem_T));
>>>>>>> if (mp) {
>>>>>>> mp->size = size;
>>>>>>> *p = (MemPtr_T)((char_T*)mp + sizeof(_Mem_T));
>>>>>>> return kErr_NoErr;
>>>>>>> }
>>>>>>> return kErr_MemErr;
>>>>>>> }
>>>>>>> Error_T setMemPtrSize(MemPtr_T *p,uint32_T size){
>>>>>>> _MemPtr_T m = _MP(*p);
>>>>>>> MemPtr_T newPtr;
>>>>>>>
>>>>>>> newPtr = realloc(m,size + sizeof(_Mem_T));
>>>>>>> if (newPtr) {
>>>>>>> m = (_MemPtr_T)newPtr;
>>>>>>> m->size = size;
>>>>>>> *p = (MemPtr_T)((char_T*)m + sizeof(_Mem_T));
>>>>>>> return kErr_NoErr;
>>>>>>> }
>>>>>>> return kErr_MemErr;
>>>>>>> }
>>>>>>>
>>>>>>> So I should catch all errors. However, errors (i.e. return value ==
>>>>>>> 0) are not reported by malloc or calloc during the problems I am
>>>>>>> experiencing. I added debug lines, but not a single failure was
>>>>>>> recorded.
>>>>>>> Removing ABORTING_MALLOC did not result in any change of error
>>>>>>> outcome.
>>>>>>>
>>>>>>> I see two different behaviors now:
>>>>>>> - setting up workers and checking that they run by
>>>>>>> static void startUpWorker(void) {
>>>>>>> #ifdef __EMSCRIPTEN__
>>>>>>> int32_T w = emscripten_wasm_worker_self_id();
>>>>>>> if (! emscripten_current_thread_is_wasm_worker()){
>>>>>>> EM_ASM_({
>>>>>>> console.log("Error: No worker: " + $0);
>>>>>>> },w);
>>>>>>> }
>>>>>>> #endif //__EMSCRIPTEN__
>>>>>>> }
>>>>>>> - then I do my stuff and receive about 10 of the "Uncaught
>>>>>>> RuntimeError: memory access out of bounds" errors.
>>>>>>> - no failures of malloc/calloc recognized
>>>>>>>
>>>>>>> The second behavior is
>>>>>>> - in main() I call this routine:
>>>>>>> static void memtest(void) {
>>>>>>> #define NUM_CHUNKS 15
>>>>>>> const int CHUNK_SIZE = 100 * 1024 * 1024;
>>>>>>> int i;
>>>>>>> void* p[NUM_CHUNKS];
>>>>>>> Error_T err = kErr_NoErr;
>>>>>>>
>>>>>>> for (int i = 0; i < NUM_CHUNKS; i++) {
>>>>>>> err = allocMemPtr(&p[i],CHUNK_SIZE,FALSE); //see function above
>>>>>>> if (err != kErr_NoErr || p[i] == NULLPTR) {
>>>>>>> printf("Error chunk %d\n",i);
>>>>>>> break;
>>>>>>> }
>>>>>>> }
>>>>>>> for (int i = 0; i < NUM_CHUNKS; i++) {
>>>>>>> if (p[i] == NULLPTR)
>>>>>>> break;
>>>>>>> disposeMemPtr(p[i]);
>>>>>>> }
>>>>>>> }
>>>>>>> - then I start up the workers as described above
>>>>>>> - then I do my stuff
>>>>>>> - sometimes this results in error free behavior, but not always. If
>>>>>>> an error occurs, I only get one "Uncaught RuntimeError" message.
>>>>>>>
>>>>>>> I am pretty confident that I handle memory allocation correctly,
>>>>>>> because my background is in development of desktop apps in C for 30+
>>>>>>> years,
>>>>>>> and there you better not have any leaks and keep the app running
>>>>>>> whenever
>>>>>>> possible. So I must be doing something wrong when dealing with multiple
>>>>>>> threads.
>>>>>>> I will try out pthreads next, because I have no idea anymore what
>>>>>>> the cause could be here.
>>>>>>>
>>>>>>> Cheers,
>>>>>>> Dieter
>>>>>>> s...@google.com schrieb am Donnerstag, 25. Mai 2023 um 23:20:33
>>>>>>> UTC+2:
>>>>>>>
>>>>>>>> Is there some reason you added `-sABORTING_MALLOC=0`.. that looks a
>>>>>>>> little suspicious, since it means the program can continue after malloc
>>>>>>>> fails.. which mean that any callsite that doesn't check the return
>>>>>>>> value of
>>>>>>>> malloc can lead to segfaults. If you remove that setting does the
>>>>>>>> behaviour change?
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> On Thu, May 25, 2023 at 1:27 PM 'Dieter Weidenbrück' via
>>>>>>>> emscripten-discuss <emscripte...@googlegroups.com> wrote:
>>>>>>>>
>>>>>>>>> Hi Sam,
>>>>>>>>>
>>>>>>>>> I can run the code in a single thread without problems, and I have
>>>>>>>>> done that for a while. So I assume that the code is stable.
>>>>>>>>>
>>>>>>>>> Here is the command line I use in a .bat file:
>>>>>>>>> emcc ./src/main.c ^
>>>>>>>>> ...
>>>>>>>>> ./src/w_com.c ^
>>>>>>>>> -I ./include/ ^
>>>>>>>>> -g3 ^
>>>>>>>>> --source-map-base ./ ^
>>>>>>>>> -gsource-map ^
>>>>>>>>> -s ALLOW_MEMORY_GROWTH=1 ^
>>>>>>>>> -s ENVIRONMENT=web,worker ^
>>>>>>>>> --shell-file ./index_template.html ^
>>>>>>>>> -s SUPPORT_ERRNO=0 ^
>>>>>>>>> -s MODULARIZE=1 ^
>>>>>>>>> -s ABORTING_MALLOC=0 ^
>>>>>>>>> -sWASM_WORKERS ^
>>>>>>>>> -s "EXPORT_NAME='wasmMod'" ^
>>>>>>>>> -s EXPORTED_FUNCTIONS="['_malloc','_free','_main']" ^
>>>>>>>>> -s EXPORTED_RUNTIME_METHODS=
>>>>>>>>> "['cwrap','UTF16ToString','UTF8ToString','stringToUTF8','allocateUTF8']"
>>>>>>>>> ^
>>>>>>>>> -o index.html
>>>>>>>>>
>>>>>>>>> I will start familiarizing myself with pthreads to test whether
>>>>>>>>> that would work better.
>>>>>>>>>
>>>>>>>>> BTW, as an old C programmer I am fascinated by emscripten and its
>>>>>>>>> possibilities. Excellent job!
>>>>>>>>>
>>>>>>>>> Cheers,
>>>>>>>>> Dieter
>>>>>>>>>
>>>>>>>>> s...@google.com schrieb am Donnerstag, 25. Mai 2023 um 20:29:58
>>>>>>>>> UTC+2:
>>>>>>>>>
>>>>>>>>>> This looks like some kind of memory corruption, most likely due
>>>>>>>>>> to the use of muiltithreading/wasm_workers Are you able to build a
>>>>>>>>>> single threaded version of your program, or one that uses normal
>>>>>>>>>> pthreads
>>>>>>>>>> rather than wasm workers?
>>>>>>>>>>
>>>>>>>>>> Also, can you share the full link command you are using?
>>>>>>>>>>
>>>>>>>>>> cheers,
>>>>>>>>>> sam
>>>>>>>>>>
>>>>>>>>>> On Thu, May 25, 2023 at 9:20 AM 'Dieter Weidenbrück' via
>>>>>>>>>> emscripten-discuss <emscripte...@googlegroups.com> wrote:
>>>>>>>>>>
>>>>>>>>>>> This is a memory snapshot when using SAFE_HEAP. So here I am
>>>>>>>>>>> quite below the browser limits, still the segfault occurs in
>>>>>>>>>>> different
>>>>>>>>>>> places.
>>>>>>>>>>> Ignore the first console line, it results from Norton Utilities
>>>>>>>>>>> I think.
>>>>>>>>>>>
>>>>>>>>>>> [image: error2.png]
>>>>>>>>>>>
>>>>>>>>>>> Dieter Weidenbrück schrieb am Donnerstag, 25. Mai 2023 um
>>>>>>>>>>> 18:06:27 UTC+2:
>>>>>>>>>>>
>>>>>>>>>>>> Hi Sam,
>>>>>>>>>>>> I noticed already that I am bumping against browser limits,
>>>>>>>>>>>> especially with sanitizer switched on, so I reduced the
>>>>>>>>>>>> pre-allocation
>>>>>>>>>>>> calls.
>>>>>>>>>>>> It turns out that asan uses so much memory that I can't use it
>>>>>>>>>>>> to analyze this case.
>>>>>>>>>>>>
>>>>>>>>>>>> I use
>>>>>>>>>>>> -s ALLOW_MEMORY_GROWTH=1
>>>>>>>>>>>> but don't specify any MAXIMUM_MEMORY.
>>>>>>>>>>>>
>>>>>>>>>>>> No pthreads version so far. I might try this next.
>>>>>>>>>>>>
>>>>>>>>>>>> Cheers,
>>>>>>>>>>>> Dieter
>>>>>>>>>>>>
>>>>>>>>>>>> s...@google.com schrieb am Donnerstag, 25. Mai 2023 um
>>>>>>>>>>>> 17:55:41 UTC+2:
>>>>>>>>>>>>
>>>>>>>>>>>>> Firstly, if you are allocating 1.8Gb you are likely pushing up
>>>>>>>>>>>>> against browser limits. Are you specifying a MAXIMUM_MEMORY of
>>>>>>>>>>>>> larger than
>>>>>>>>>>>>> 2GB?
>>>>>>>>>>>>>
>>>>>>>>>>>>> Secondly, it looks like you are using wasm workers, which are
>>>>>>>>>>>>> still relatively new. Do you have a version of your code that
>>>>>>>>>>>>> uses
>>>>>>>>>>>>> pthreads instead? It might tell is if the issue is related to
>>>>>>>>>>>>> wasm workers.
>>>>>>>>>>>>>
>>>>>>>>>>>>> cheers,
>>>>>>>>>>>>> sam
>>>>>>>>>>>>>
>>>>>>>>>>>>> On Thu, May 25, 2023 at 8:06 AM 'Dieter Weidenbrück' via
>>>>>>>>>>>>> emscripten-discuss <emscripte...@googlegroups.com> wrote:
>>>>>>>>>>>>>
>>>>>>>>>>>>>> The joy was premature, even with pre-allocated heap size
>>>>>>>>>>>>>> segfaults occur. :(
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Dieter Weidenbrück schrieb am Donnerstag, 25. Mai 2023 um
>>>>>>>>>>>>>> 16:28:37 UTC+2:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> All,
>>>>>>>>>>>>>>> I am experiencing segmentation faults when using wasm
>>>>>>>>>>>>>>> workers.
>>>>>>>>>>>>>>> Overview:
>>>>>>>>>>>>>>> I am working on a project with considerable 3D data sets.
>>>>>>>>>>>>>>> The code has been stable for a while when running in the main
>>>>>>>>>>>>>>> thread alone.
>>>>>>>>>>>>>>> Then I started using js workers (no shared memory), and again
>>>>>>>>>>>>>>> all was well.
>>>>>>>>>>>>>>> Now I've switched to SharedArrayBuffers and wasm workers,
>>>>>>>>>>>>>>> and I keep running into random problems.
>>>>>>>>>>>>>>> I have prepared the code such that I can run with 0 workers
>>>>>>>>>>>>>>> up to hardware.concurrency workers. All is well with 0 workers,
>>>>>>>>>>>>>>> but as soon
>>>>>>>>>>>>>>> as I use one or more workers, I keep getting segfaults because
>>>>>>>>>>>>>>> of invalid
>>>>>>>>>>>>>>> pointers, access out of bounds and similar.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> What happens in main thread and what in the wasm workers:
>>>>>>>>>>>>>>> I allocate all objects in the main thread when importing the
>>>>>>>>>>>>>>> 3D file. Then i fire off a function for each object that will
>>>>>>>>>>>>>>> do some
>>>>>>>>>>>>>>> serious calculations of the data, including allocating and
>>>>>>>>>>>>>>> disposing of
>>>>>>>>>>>>>>> memory. The workers allocate approx. 300 to 400 MB in addition
>>>>>>>>>>>>>>> to the main
>>>>>>>>>>>>>>> thread. All this happens in the same sharedArrayBuffer, of
>>>>>>>>>>>>>>> course.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Here is what I've tried so far:
>>>>>>>>>>>>>>> - compiling with SAFE_HEAP=1
>>>>>>>>>>>>>>> not a lot of helpful information,
>>>>>>>>>>>>>>> - compiling with -fsanitize=address
>>>>>>>>>>>>>>> everything works without problems here!
>>>>>>>>>>>>>>> - compiling with ASSERTIONS=2
>>>>>>>>>>>>>>> gave me this information:
>>>>>>>>>>>>>>> [image: error.png]
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> To me it looks like another resize call is executed while
>>>>>>>>>>>>>>> other workers keep working on the buffer, and then something
>>>>>>>>>>>>>>> gets into
>>>>>>>>>>>>>>> conflict.
>>>>>>>>>>>>>>> To test this, I allocated 1.8 GB right after startup in the
>>>>>>>>>>>>>>> main thread and disposed the mem blocks again just to trigger
>>>>>>>>>>>>>>> heap resize.
>>>>>>>>>>>>>>> After that everything works like a charm.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Is there anything I am doing wrong?
>>>>>>>>>>>>>>> Sorry for not providing a sample, but there is a lot of code
>>>>>>>>>>>>>>> involved, and it is not easy to simulate this behavior. Happy
>>>>>>>>>>>>>>> to answer
>>>>>>>>>>>>>>> questions.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> All comments are appreciated.
>>>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>>>> Dieter
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>> --
>>>>>>>>>>>>>> You received this message because you are subscribed to the
>>>>>>>>>>>>>> Google Groups "emscripten-discuss" group.
>>>>>>>>>>>>>> To unsubscribe from this group and stop receiving emails from
>>>>>>>>>>>>>> it, send an email to emscripten-disc...@googlegroups.com.
>>>>>>>>>>>>>> To view this discussion on the web visit
>>>>>>>>>>>>>> https://groups.google.com/d/msgid/emscripten-discuss/80d56314-59d8-4332-bb2e-ebe00fe52ea3n%40googlegroups.com
>>>>>>>>>>>>>> <https://groups.google.com/d/msgid/emscripten-discuss/80d56314-59d8-4332-bb2e-ebe00fe52ea3n%40googlegroups.com?utm_medium=email&utm_source=footer>
>>>>>>>>>>>>>> .
>>>>>>>>>>>>>>
>>>>>>>>>>>>> --
>>>>>>>>>>> You received this message because you are subscribed to the
>>>>>>>>>>> Google Groups "emscripten-discuss" group.
>>>>>>>>>>> To unsubscribe from this group and stop receiving emails from
>>>>>>>>>>> it, send an email to emscripten-disc...@googlegroups.com.
>>>>>>>>>>>
>>>>>>>>>> To view this discussion on the web visit
>>>>>>>>>>> https://groups.google.com/d/msgid/emscripten-discuss/cfc03512-f69f-44b0-8c14-1f1a8e4ffe9fn%40googlegroups.com
>>>>>>>>>>> <https://groups.google.com/d/msgid/emscripten-discuss/cfc03512-f69f-44b0-8c14-1f1a8e4ffe9fn%40googlegroups.com?utm_medium=email&utm_source=footer>
>>>>>>>>>>> .
>>>>>>>>>>>
>>>>>>>>>> --
>>>>>>>>> You received this message because you are subscribed to the Google
>>>>>>>>> Groups "emscripten-discuss" group.
>>>>>>>>> To unsubscribe from this group and stop receiving emails from it,
>>>>>>>>> send an email to emscripten-disc...@googlegroups.com.
>>>>>>>>>
>>>>>>>> To view this discussion on the web visit
>>>>>>>>> https://groups.google.com/d/msgid/emscripten-discuss/e568e189-4259-460f-9601-e7996927cdb7n%40googlegroups.com
>>>>>>>>> <https://groups.google.com/d/msgid/emscripten-discuss/e568e189-4259-460f-9601-e7996927cdb7n%40googlegroups.com?utm_medium=email&utm_source=footer>
>>>>>>>>> .
>>>>>>>>>
>>>>>>>> --
>>>>>>> You received this message because you are subscribed to the Google
>>>>>>> Groups "emscripten-discuss" group.
>>>>>>> To unsubscribe from this group and stop receiving emails from it,
>>>>>>> send an email to emscripten-disc...@googlegroups.com.
>>>>>>> To view this discussion on the web visit
>>>>>>> https://groups.google.com/d/msgid/emscripten-discuss/b20d2de8-2532-4441-b8fc-3ef8f049f7f0n%40googlegroups.com
>>>>>>> <https://groups.google.com/d/msgid/emscripten-discuss/b20d2de8-2532-4441-b8fc-3ef8f049f7f0n%40googlegroups.com?utm_medium=email&utm_source=footer>
>>>>>>> .
>>>>>>>
>>>>>> --
>>>>>> You received this message because you are subscribed to the Google
>>>>>> Groups "emscripten-discuss" group.
>>>>>> To unsubscribe from this group and stop receiving emails from it,
>>>>>> send an email to emscripten-disc...@googlegroups.com.
>>>>>>
>>>>> To view this discussion on the web visit
>>>>>> https://groups.google.com/d/msgid/emscripten-discuss/CAL_va28k7RyF2n-x6B8M9pbgri2bCDCQA7N%2BG7x-6GVP%2Bpqumg%40mail.gmail.com
>>>>>> <https://groups.google.com/d/msgid/emscripten-discuss/CAL_va28k7RyF2n-x6B8M9pbgri2bCDCQA7N%2BG7x-6GVP%2Bpqumg%40mail.gmail.com?utm_medium=email&utm_source=footer>
>>>>>> .
>>>>>>
>>>>> --
>>> You received this message because you are subscribed to the Google
>>> Groups "emscripten-discuss" group.
>>> To unsubscribe from this group and stop receiving emails from it, send
>>> an email to emscripten-disc...@googlegroups.com.
>>>
>> To view this discussion on the web visit
>>> https://groups.google.com/d/msgid/emscripten-discuss/7de28d6a-0cc9-4953-b9eb-bb9a2844ac9dn%40googlegroups.com
>>> <https://groups.google.com/d/msgid/emscripten-discuss/7de28d6a-0cc9-4953-b9eb-bb9a2844ac9dn%40googlegroups.com?utm_medium=email&utm_source=footer>
>>> .
>>>
>> --
> You received this message because you are subscribed to the Google Groups
> "emscripten-discuss" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to emscripten-discuss+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/emscripten-discuss/c438a02c-af55-4c44-9999-064c49a27d78n%40googlegroups.com
> <https://groups.google.com/d/msgid/emscripten-discuss/c438a02c-af55-4c44-9999-064c49a27d78n%40googlegroups.com?utm_medium=email&utm_source=footer>
> .
>
--
You received this message because you are subscribed to the Google Groups
"emscripten-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to emscripten-discuss+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/emscripten-discuss/CAL_va2_Uza3Ak5t2eVik4a5Tvbf4qoYQWMGk7rrvcunF%2BKZcBA%40mail.gmail.com.
