Re: Thread behavior in 7.8.3
Hi Michael, Are you already using usb-1.3.0.0? If not, could you upgrade and test again? That release fixed the deadlock that Ben and Carter where talking about. Good luck, Bas ___ Glasgow-haskell-users mailing list Glasgow-haskell-users@haskell.org http://www.haskell.org/mailman/listinfo/glasgow-haskell-users
Re: Thread behavior in 7.8.3
Bas, I have not upgraded, mainly because my problems manifest without enabling USB. However, I think I can upgrade in a few days and move forward. Are you using ghc 7.8.10 these days or something older? Mike On Jan 21, 2015, at 12:52 PM, Bas van Dijk v.dijk@gmail.com wrote: Hi Michael, Are you already using usb-1.3.0.0? If not, could you upgrade and test again? That release fixed the deadlock that Ben and Carter where talking about. Good luck, Bas ___ Glasgow-haskell-users mailing list Glasgow-haskell-users@haskell.org http://www.haskell.org/mailman/listinfo/glasgow-haskell-users
Re: Thread behavior in 7.8.3
Bas, I checked my cabal file and I was already using 1.3.0.0. Mike On Jan 21, 2015, at 12:52 PM, Bas van Dijk v.dijk@gmail.com wrote: Hi Michael, Are you already using usb-1.3.0.0? If not, could you upgrade and test again? That release fixed the deadlock that Ben and Carter where talking about. Good luck, Bas ___ Glasgow-haskell-users mailing list Glasgow-haskell-users@haskell.org http://www.haskell.org/mailman/listinfo/glasgow-haskell-users
Re: Thread behavior in 7.8.3
visible. From last week, I know that an non-gui version of the app is getting stuck, but I do not know if it eventually runs like this case. Is there some convention that when I look at an event log you can tell which threads are OS threads vs threads from fork? Perhaps someone that knows the scheduler might have some advice. It seems odd that a scheduler could behave this way. The scheduler should have some built in notion of fairness. On Jan 12, 2015, at 11:02 PM, Michael Jones m...@proclivis.com mailto:m...@proclivis.com wrote: Sorry I am reviving an old problem, but it has resurfaced, such that one system behaves different than another. Using -C0.001 solved problems on a Mac + VM + Ubuntu 14. It worked on a single core 32 bit Atom NUC. But on a dual core Atom MinnowBoardMax, something bad is going on. In summary, the same code that runs on two machines does not run on a third machine. So this indicates I have not made any breaking changes to the code or cabal files. Compiling with GHC 7.8.3. This bad system has Ubuntu 14 installed, with an updated Linux 3.18.1 kernel. It is a dual core 64 bit I86 Atom processor. The application hangs at startup. If I remove the -C0.00N option and instead use -V0, the application runs. It has bad timing properties, but it does at least run. Note that a hang hangs an IO thread talking USB, and the GUI thread. When testing with the -C0.00N option, it did run 2 times out of 20 tries, so fail means fail most but not all of the time. When it did run, it continued to run properly. This perhaps indicates some kind of internal race condition. In the fail to run case, it does some printing up to the point where it tries to create a wxHaskell frame. In another non-UI version of the program it also fails to run. Logging to a file gives a similar indication. It is clear that the program starts up, then fails during the run in some form of lockup, well after the initial startup code. If I run with the strace command, it always runs with -C0.00N. All the above was done with profiling enabled, so I removed that and instead enabled eventlog to look for clues. In this case it lies between good and bad, in that IO to my USB is working, but the GUI comes up blank and never paints. Running this case without -v0 (event log) the gui partially paints and stops, but USB continues. Questions: 1) Does ghc 7.8.4 have any improvements that might pertain to these kinds of scheduling/thread problems? 2) Is there anything about the nature of a thread using USB, I2C, or wxHaskell IO that leads to problems that a pure calculation app would not have? 3) Any ideas how to track down the problem when changing conditions (compiler or runtime options) affects behavior? 4) Are there other options besides -V and -C for the runtime that might apply? 5) What does -V0 do that makes a problem program run? Mike On Oct 29, 2014, at 6:02 PM, Michael Jones m...@proclivis.com mailto:m...@proclivis.com wrote: John, Adding -C0.005 makes it much better. Using -C0.001 makes it behave more like -N4. Thanks. This saves my project, as I need to deploy on a single core Atom and was stuck. Mike On Oct 29, 2014, at 5:12 PM, John Lato jwl...@gmail.com mailto:jwl...@gmail.com wrote: By any chance do the delays get shorter if you run your program with `+RTS -C0.005` ? If so, I suspect you're having a problem very similar to one that we had with ghc-7.8 (7.6 too, but it's worse on ghc-7.8 for some reason), involving possible misbehavior of the thread scheduler. On Wed, Oct 29, 2014 at 2:18 PM, Michael Jones m...@proclivis.com mailto:m...@proclivis.com wrote: I have a general question about thread behavior in 7.8.3 vs 7.6.X I moved from 7.6 to 7.8 and my application behaves very differently. I have three threads, an application thread that plots data with wxhaskell or sends it over a network (depends on settings), a thread doing usb bulk writes, and a thread doing usb bulk reads. Data is moved around with TChan, and TVar is used for coordination. When the application was compiled with 7.6, my stream of usb traffic was smooth. With 7.8, there are lots of delays where nothing seems to be running. These delays are up to 40ms, whereas with 7.6 delays were a 1ms or so. When I add -N2 or -N4, the 7.8 program runs fine. But on 7.6 it runs fine without with -N2/4. The program is compiled -O2 with profiling. The -N2/4 version uses more memory, but in both cases with 7.8 and with 7.6 there is no space leak. I tired to compile and use -ls so I could take a look with threadscope, but the application hangs and writes no data to the file. The CPU fans run wild like it is in an infinite loop. It at least pops an unpainted wxhaskell window, so it got partially running. One of my libraries uses option -fsimpl-tick-factor=200 to get around the compiler. What do I need to know about changes to threading and event
Re: Thread behavior in 7.8.3
in an immediate GUI and i2c IO. The behavior suggests to me that some initial threads are starving the ability for other threads to start, and perhaps on a dual core machine it is more of a problem than single or quad core machines. For certain, due to some printing, I know that the main thread is starting, and that a print just before the first fork is not printing. Code between them is evaluating wxhaskell functions, but the main frame is not yet asked to become visible. From last week, I know that an non-gui version of the app is getting stuck, but I do not know if it eventually runs like this case. Is there some convention that when I look at an event log you can tell which threads are OS threads vs threads from fork? Perhaps someone that knows the scheduler might have some advice. It seems odd that a scheduler could behave this way. The scheduler should have some built in notion of fairness. On Jan 12, 2015, at 11:02 PM, Michael Jones m...@proclivis.com mailto:m...@proclivis.com wrote: Sorry I am reviving an old problem, but it has resurfaced, such that one system behaves different than another. Using -C0.001 solved problems on a Mac + VM + Ubuntu 14. It worked on a single core 32 bit Atom NUC. But on a dual core Atom MinnowBoardMax, something bad is going on. In summary, the same code that runs on two machines does not run on a third machine. So this indicates I have not made any breaking changes to the code or cabal files. Compiling with GHC 7.8.3. This bad system has Ubuntu 14 installed, with an updated Linux 3.18.1 kernel. It is a dual core 64 bit I86 Atom processor. The application hangs at startup. If I remove the -C0.00N option and instead use -V0, the application runs. It has bad timing properties, but it does at least run. Note that a hang hangs an IO thread talking USB, and the GUI thread. When testing with the -C0.00N option, it did run 2 times out of 20 tries, so fail means fail most but not all of the time. When it did run, it continued to run properly. This perhaps indicates some kind of internal race condition. In the fail to run case, it does some printing up to the point where it tries to create a wxHaskell frame. In another non-UI version of the program it also fails to run. Logging to a file gives a similar indication. It is clear that the program starts up, then fails during the run in some form of lockup, well after the initial startup code. If I run with the strace command, it always runs with -C0.00N. All the above was done with profiling enabled, so I removed that and instead enabled eventlog to look for clues. In this case it lies between good and bad, in that IO to my USB is working, but the GUI comes up blank and never paints. Running this case without -v0 (event log) the gui partially paints and stops, but USB continues. Questions: 1) Does ghc 7.8.4 have any improvements that might pertain to these kinds of scheduling/thread problems? 2) Is there anything about the nature of a thread using USB, I2C, or wxHaskell IO that leads to problems that a pure calculation app would not have? 3) Any ideas how to track down the problem when changing conditions (compiler or runtime options) affects behavior? 4) Are there other options besides -V and -C for the runtime that might apply? 5) What does -V0 do that makes a problem program run? Mike On Oct 29, 2014, at 6:02 PM, Michael Jones m...@proclivis.com mailto:m...@proclivis.com wrote: John, Adding -C0.005 makes it much better. Using -C0.001 makes it behave more like -N4. Thanks. This saves my project, as I need to deploy on a single core Atom and was stuck. Mike On Oct 29, 2014, at 5:12 PM, John Lato jwl...@gmail.com mailto:jwl...@gmail.com wrote: By any chance do the delays get shorter if you run your program with `+RTS -C0.005` ? If so, I suspect you're having a problem very similar to one that we had with ghc-7.8 (7.6 too, but it's worse on ghc-7.8 for some reason), involving possible misbehavior of the thread scheduler. On Wed, Oct 29, 2014 at 2:18 PM, Michael Jones m...@proclivis.com mailto:m...@proclivis.com wrote: I have a general question about thread behavior in 7.8.3 vs 7.6.X I moved from 7.6 to 7.8 and my application behaves very differently. I have three threads, an application thread that plots data with wxhaskell or sends it over a network (depends on settings), a thread doing usb bulk writes, and a thread doing usb bulk reads. Data is moved around with TChan, and TVar is used for coordination. When the application was compiled with 7.6, my stream of usb traffic was smooth. With 7.8, there are lots of delays where nothing seems to be running. These delays are up to 40ms, whereas with 7.6 delays were a 1ms or so. When I add -N2 or -N4, the 7.8 program runs fine. But on 7.6 it runs fine without with -N2/4. The program is compiled -O2
Re: Thread behavior in 7.8.3
-N4. Thanks. This saves my project, as I need to deploy on a single core Atom and was stuck. Mike On Oct 29, 2014, at 5:12 PM, John Lato jwl...@gmail.com mailto:jwl...@gmail.com wrote: By any chance do the delays get shorter if you run your program with `+RTS -C0.005` ? If so, I suspect you're having a problem very similar to one that we had with ghc-7.8 (7.6 too, but it's worse on ghc-7.8 for some reason), involving possible misbehavior of the thread scheduler. On Wed, Oct 29, 2014 at 2:18 PM, Michael Jones m...@proclivis.com mailto:m...@proclivis.com wrote: I have a general question about thread behavior in 7.8.3 vs 7.6.X I moved from 7.6 to 7.8 and my application behaves very differently. I have three threads, an application thread that plots data with wxhaskell or sends it over a network (depends on settings), a thread doing usb bulk writes, and a thread doing usb bulk reads. Data is moved around with TChan, and TVar is used for coordination. When the application was compiled with 7.6, my stream of usb traffic was smooth. With 7.8, there are lots of delays where nothing seems to be running. These delays are up to 40ms, whereas with 7.6 delays were a 1ms or so. When I add -N2 or -N4, the 7.8 program runs fine. But on 7.6 it runs fine without with -N2/4. The program is compiled -O2 with profiling. The -N2/4 version uses more memory, but in both cases with 7.8 and with 7.6 there is no space leak. I tired to compile and use -ls so I could take a look with threadscope, but the application hangs and writes no data to the file. The CPU fans run wild like it is in an infinite loop. It at least pops an unpainted wxhaskell window, so it got partially running. One of my libraries uses option -fsimpl-tick-factor=200 to get around the compiler. What do I need to know about changes to threading and event logging between 7.6 and 7.8? Is there some general documentation somewhere that might help? I am on Ubuntu 14.04 LTS. I downloaded the 7.8 tool chain tar ball and installed myself, after removing 7.6 with apt-get. Any hints appreciated. Mike ___ Glasgow-haskell-users mailing list Glasgow-haskell-users@haskell.org mailto:Glasgow-haskell-users@haskell.org http://www.haskell.org/mailman/listinfo/glasgow-haskell-users ___ Glasgow-haskell-users mailing list Glasgow-haskell-users@haskell.org mailto:Glasgow-haskell-users@haskell.org http://www.haskell.org/mailman/listinfo/glasgow-haskell-users ___ Glasgow-haskell-users mailing list Glasgow-haskell-users@haskell.org http://www.haskell.org/mailman/listinfo/glasgow-haskell-users ___ Glasgow-haskell-users mailing list Glasgow-haskell-users@haskell.org http://www.haskell.org/mailman/listinfo/glasgow-haskell-users ___ Glasgow-haskell-users mailing list Glasgow-haskell-users@haskell.org http://www.haskell.org/mailman/listinfo/glasgow-haskell-users
Re: Thread behavior in 7.8.3
the nature of a thread using USB, I2C, or wxHaskell IO that leads to problems that a pure calculation app would not have? 3) Any ideas how to track down the problem when changing conditions (compiler or runtime options) affects behavior? 4) Are there other options besides -V and -C for the runtime that might apply? 5) What does -V0 do that makes a problem program run? Mike On Oct 29, 2014, at 6:02 PM, Michael Jones m...@proclivis.com mailto:m...@proclivis.com wrote: John, Adding -C0.005 makes it much better. Using -C0.001 makes it behave more like -N4. Thanks. This saves my project, as I need to deploy on a single core Atom and was stuck. Mike On Oct 29, 2014, at 5:12 PM, John Lato jwl...@gmail.com mailto:jwl...@gmail.com wrote: By any chance do the delays get shorter if you run your program with `+RTS -C0.005` ? If so, I suspect you're having a problem very similar to one that we had with ghc-7.8 (7.6 too, but it's worse on ghc-7.8 for some reason), involving possible misbehavior of the thread scheduler. On Wed, Oct 29, 2014 at 2:18 PM, Michael Jones m...@proclivis.com mailto:m...@proclivis.com wrote: I have a general question about thread behavior in 7.8.3 vs 7.6.X I moved from 7.6 to 7.8 and my application behaves very differently. I have three threads, an application thread that plots data with wxhaskell or sends it over a network (depends on settings), a thread doing usb bulk writes, and a thread doing usb bulk reads. Data is moved around with TChan, and TVar is used for coordination. When the application was compiled with 7.6, my stream of usb traffic was smooth. With 7.8, there are lots of delays where nothing seems to be running. These delays are up to 40ms, whereas with 7.6 delays were a 1ms or so. When I add -N2 or -N4, the 7.8 program runs fine. But on 7.6 it runs fine without with -N2/4. The program is compiled -O2 with profiling. The -N2/4 version uses more memory, but in both cases with 7.8 and with 7.6 there is no space leak. I tired to compile and use -ls so I could take a look with threadscope, but the application hangs and writes no data to the file. The CPU fans run wild like it is in an infinite loop. It at least pops an unpainted wxhaskell window, so it got partially running. One of my libraries uses option -fsimpl-tick-factor=200 to get around the compiler. What do I need to know about changes to threading and event logging between 7.6 and 7.8? Is there some general documentation somewhere that might help? I am on Ubuntu 14.04 LTS. I downloaded the 7.8 tool chain tar ball and installed myself, after removing 7.6 with apt-get. Any hints appreciated. Mike ___ Glasgow-haskell-users mailing list Glasgow-haskell-users@haskell.org mailto:Glasgow-haskell-users@haskell.org http://www.haskell.org/mailman/listinfo/glasgow-haskell-users ___ Glasgow-haskell-users mailing list Glasgow-haskell-users@haskell.org mailto:Glasgow-haskell-users@haskell.org http://www.haskell.org/mailman/listinfo/glasgow-haskell-users ___ Glasgow-haskell-users mailing list Glasgow-haskell-users@haskell.org http://www.haskell.org/mailman/listinfo/glasgow-haskell-users ___ Glasgow-haskell-users mailing list Glasgow-haskell-users@haskell.org http://www.haskell.org/mailman/listinfo/glasgow-haskell-users ___ Glasgow-haskell-users mailing list Glasgow-haskell-users@haskell.org http://www.haskell.org/mailman/listinfo/glasgow-haskell-users
Re: Thread behavior in 7.8.3
without -v0 (event log) the gui partially paints and stops, but USB continues. Questions: 1) Does ghc 7.8.4 have any improvements that might pertain to these kinds of scheduling/thread problems? 2) Is there anything about the nature of a thread using USB, I2C, or wxHaskell IO that leads to problems that a pure calculation app would not have? 3) Any ideas how to track down the problem when changing conditions (compiler or runtime options) affects behavior? 4) Are there other options besides -V and -C for the runtime that might apply? 5) What does -V0 do that makes a problem program run? Mike On Oct 29, 2014, at 6:02 PM, Michael Jones m...@proclivis.com mailto:m...@proclivis.com wrote: John, Adding -C0.005 makes it much better. Using -C0.001 makes it behave more like -N4. Thanks. This saves my project, as I need to deploy on a single core Atom and was stuck. Mike On Oct 29, 2014, at 5:12 PM, John Lato jwl...@gmail.com mailto:jwl...@gmail.com wrote: By any chance do the delays get shorter if you run your program with `+RTS -C0.005` ? If so, I suspect you're having a problem very similar to one that we had with ghc-7.8 (7.6 too, but it's worse on ghc-7.8 for some reason), involving possible misbehavior of the thread scheduler. On Wed, Oct 29, 2014 at 2:18 PM, Michael Jones m...@proclivis.com mailto:m...@proclivis.com wrote: I have a general question about thread behavior in 7.8.3 vs 7.6.X I moved from 7.6 to 7.8 and my application behaves very differently. I have three threads, an application thread that plots data with wxhaskell or sends it over a network (depends on settings), a thread doing usb bulk writes, and a thread doing usb bulk reads. Data is moved around with TChan, and TVar is used for coordination. When the application was compiled with 7.6, my stream of usb traffic was smooth. With 7.8, there are lots of delays where nothing seems to be running. These delays are up to 40ms, whereas with 7.6 delays were a 1ms or so. When I add -N2 or -N4, the 7.8 program runs fine. But on 7.6 it runs fine without with -N2/4. The program is compiled -O2 with profiling. The -N2/4 version uses more memory, but in both cases with 7.8 and with 7.6 there is no space leak. I tired to compile and use -ls so I could take a look with threadscope, but the application hangs and writes no data to the file. The CPU fans run wild like it is in an infinite loop. It at least pops an unpainted wxhaskell window, so it got partially running. One of my libraries uses option -fsimpl-tick-factor=200 to get around the compiler. What do I need to know about changes to threading and event logging between 7.6 and 7.8? Is there some general documentation somewhere that might help? I am on Ubuntu 14.04 LTS. I downloaded the 7.8 tool chain tar ball and installed myself, after removing 7.6 with apt-get. Any hints appreciated. Mike ___ Glasgow-haskell-users mailing list Glasgow-haskell-users@haskell.org mailto:Glasgow-haskell-users@haskell.org http://www.haskell.org/mailman/listinfo/glasgow-haskell-users ___ Glasgow-haskell-users mailing list Glasgow-haskell-users@haskell.org mailto:Glasgow-haskell-users@haskell.org http://www.haskell.org/mailman/listinfo/glasgow-haskell-users ___ Glasgow-haskell-users mailing list Glasgow-haskell-users@haskell.org http://www.haskell.org/mailman/listinfo/glasgow-haskell-users ___ Glasgow-haskell-users mailing list Glasgow-haskell-users@haskell.org http://www.haskell.org/mailman/listinfo/glasgow-haskell-users ___ Glasgow-haskell-users mailing list Glasgow-haskell-users@haskell.org http://www.haskell.org/mailman/listinfo/glasgow-haskell-users
Re: Thread behavior in 7.8.3
threads are OS threads vs threads from fork? Perhaps someone that knows the scheduler might have some advice. It seems odd that a scheduler could behave this way. The scheduler should have some built in notion of fairness. On Jan 12, 2015, at 11:02 PM, Michael Jones m...@proclivis.com mailto:m...@proclivis.com wrote: Sorry I am reviving an old problem, but it has resurfaced, such that one system behaves different than another. Using -C0.001 solved problems on a Mac + VM + Ubuntu 14. It worked on a single core 32 bit Atom NUC. But on a dual core Atom MinnowBoardMax, something bad is going on. In summary, the same code that runs on two machines does not run on a third machine. So this indicates I have not made any breaking changes to the code or cabal files. Compiling with GHC 7.8.3. This bad system has Ubuntu 14 installed, with an updated Linux 3.18.1 kernel. It is a dual core 64 bit I86 Atom processor. The application hangs at startup. If I remove the -C0.00N option and instead use -V0, the application runs. It has bad timing properties, but it does at least run. Note that a hang hangs an IO thread talking USB, and the GUI thread. When testing with the -C0.00N option, it did run 2 times out of 20 tries, so fail means fail most but not all of the time. When it did run, it continued to run properly. This perhaps indicates some kind of internal race condition. In the fail to run case, it does some printing up to the point where it tries to create a wxHaskell frame. In another non-UI version of the program it also fails to run. Logging to a file gives a similar indication. It is clear that the program starts up, then fails during the run in some form of lockup, well after the initial startup code. If I run with the strace command, it always runs with -C0.00N. All the above was done with profiling enabled, so I removed that and instead enabled eventlog to look for clues. In this case it lies between good and bad, in that IO to my USB is working, but the GUI comes up blank and never paints. Running this case without -v0 (event log) the gui partially paints and stops, but USB continues. Questions: 1) Does ghc 7.8.4 have any improvements that might pertain to these kinds of scheduling/thread problems? 2) Is there anything about the nature of a thread using USB, I2C, or wxHaskell IO that leads to problems that a pure calculation app would not have? 3) Any ideas how to track down the problem when changing conditions (compiler or runtime options) affects behavior? 4) Are there other options besides -V and -C for the runtime that might apply? 5) What does -V0 do that makes a problem program run? Mike On Oct 29, 2014, at 6:02 PM, Michael Jones m...@proclivis.com mailto:m...@proclivis.com wrote: John, Adding -C0.005 makes it much better. Using -C0.001 makes it behave more like -N4. Thanks. This saves my project, as I need to deploy on a single core Atom and was stuck. Mike On Oct 29, 2014, at 5:12 PM, John Lato jwl...@gmail.com mailto:jwl...@gmail.com wrote: By any chance do the delays get shorter if you run your program with `+RTS -C0.005` ? If so, I suspect you're having a problem very similar to one that we had with ghc-7.8 (7.6 too, but it's worse on ghc-7.8 for some reason), involving possible misbehavior of the thread scheduler. On Wed, Oct 29, 2014 at 2:18 PM, Michael Jones m...@proclivis.com mailto:m...@proclivis.com wrote: I have a general question about thread behavior in 7.8.3 vs 7.6.X I moved from 7.6 to 7.8 and my application behaves very differently. I have three threads, an application thread that plots data with wxhaskell or sends it over a network (depends on settings), a thread doing usb bulk writes, and a thread doing usb bulk reads. Data is moved around with TChan, and TVar is used for coordination. When the application was compiled with 7.6, my stream of usb traffic was smooth. With 7.8, there are lots of delays where nothing seems to be running. These delays are up to 40ms, whereas with 7.6 delays were a 1ms or so. When I add -N2 or -N4, the 7.8 program runs fine. But on 7.6 it runs fine without with -N2/4. The program is compiled -O2 with profiling. The -N2/4 version uses more memory, but in both cases with 7.8 and with 7.6 there is no space leak. I tired to compile and use -ls so I could take a look with threadscope, but the application hangs and writes no data to the file. The CPU fans run wild like it is in an infinite loop. It at least pops an unpainted wxhaskell window, so it got partially running. One of my libraries uses option -fsimpl-tick-factor=200 to get around the compiler. What do I need to know about changes to threading and event logging between 7.6 and 7.8? Is there some general documentation somewhere that might help? I am on Ubuntu 14.04 LTS. I
Re: Thread behavior in 7.8.3
them is evaluating wxhaskell functions, but the main frame is not yet asked to become visible. From last week, I know that an non-gui version of the app is getting stuck, but I do not know if it eventually runs like this case. Is there some convention that when I look at an event log you can tell which threads are OS threads vs threads from fork? Perhaps someone that knows the scheduler might have some advice. It seems odd that a scheduler could behave this way. The scheduler should have some built in notion of fairness. On Jan 12, 2015, at 11:02 PM, Michael Jones m...@proclivis.com mailto:m...@proclivis.com wrote: Sorry I am reviving an old problem, but it has resurfaced, such that one system behaves different than another. Using -C0.001 solved problems on a Mac + VM + Ubuntu 14. It worked on a single core 32 bit Atom NUC. But on a dual core Atom MinnowBoardMax, something bad is going on. In summary, the same code that runs on two machines does not run on a third machine. So this indicates I have not made any breaking changes to the code or cabal files. Compiling with GHC 7.8.3. This bad system has Ubuntu 14 installed, with an updated Linux 3.18.1 kernel. It is a dual core 64 bit I86 Atom processor. The application hangs at startup. If I remove the -C0.00N option and instead use -V0, the application runs. It has bad timing properties, but it does at least run. Note that a hang hangs an IO thread talking USB, and the GUI thread. When testing with the -C0.00N option, it did run 2 times out of 20 tries, so fail means fail most but not all of the time. When it did run, it continued to run properly. This perhaps indicates some kind of internal race condition. In the fail to run case, it does some printing up to the point where it tries to create a wxHaskell frame. In another non-UI version of the program it also fails to run. Logging to a file gives a similar indication. It is clear that the program starts up, then fails during the run in some form of lockup, well after the initial startup code. If I run with the strace command, it always runs with -C0.00N. All the above was done with profiling enabled, so I removed that and instead enabled eventlog to look for clues. In this case it lies between good and bad, in that IO to my USB is working, but the GUI comes up blank and never paints. Running this case without -v0 (event log) the gui partially paints and stops, but USB continues. Questions: 1) Does ghc 7.8.4 have any improvements that might pertain to these kinds of scheduling/thread problems? 2) Is there anything about the nature of a thread using USB, I2C, or wxHaskell IO that leads to problems that a pure calculation app would not have? 3) Any ideas how to track down the problem when changing conditions (compiler or runtime options) affects behavior? 4) Are there other options besides -V and -C for the runtime that might apply? 5) What does -V0 do that makes a problem program run? Mike On Oct 29, 2014, at 6:02 PM, Michael Jones m...@proclivis.com mailto:m...@proclivis.com wrote: John, Adding -C0.005 makes it much better. Using -C0.001 makes it behave more like -N4. Thanks. This saves my project, as I need to deploy on a single core Atom and was stuck. Mike On Oct 29, 2014, at 5:12 PM, John Lato jwl...@gmail.com mailto:jwl...@gmail.com wrote: By any chance do the delays get shorter if you run your program with `+RTS -C0.005` ? If so, I suspect you're having a problem very similar to one that we had with ghc-7.8 (7.6 too, but it's worse on ghc-7.8 for some reason), involving possible misbehavior of the thread scheduler. On Wed, Oct 29, 2014 at 2:18 PM, Michael Jones m...@proclivis.com mailto:m...@proclivis.com wrote: I have a general question about thread behavior in 7.8.3 vs 7.6.X I moved from 7.6 to 7.8 and my application behaves very differently. I have three threads, an application thread that plots data with wxhaskell or sends it over a network (depends on settings), a thread doing usb bulk writes, and a thread doing usb bulk reads. Data is moved around with TChan, and TVar is used for coordination. When the application was compiled with 7.6, my stream of usb traffic was smooth. With 7.8, there are lots of delays where nothing seems to be running. These delays are up to 40ms, whereas with 7.6 delays were a 1ms or so. When I add -N2 or -N4, the 7.8 program runs fine. But on 7.6 it runs fine without with -N2/4. The program is compiled -O2 with profiling. The -N2/4 version uses more memory, but in both cases with 7.8 and with 7.6 there is no space leak. I tired to compile and use -ls so I could take a look with threadscope, but the application hangs and writes no data to the file. The CPU fans run wild
Re: Thread behavior in 7.8.3
. Mike On Oct 29, 2014, at 5:12 PM, John Lato jwl...@gmail.com mailto:jwl...@gmail.com wrote: By any chance do the delays get shorter if you run your program with `+RTS -C0.005` ? If so, I suspect you're having a problem very similar to one that we had with ghc-7.8 (7.6 too, but it's worse on ghc-7.8 for some reason), involving possible misbehavior of the thread scheduler. On Wed, Oct 29, 2014 at 2:18 PM, Michael Jones m...@proclivis.com mailto:m...@proclivis.com wrote: I have a general question about thread behavior in 7.8.3 vs 7.6.X I moved from 7.6 to 7.8 and my application behaves very differently. I have three threads, an application thread that plots data with wxhaskell or sends it over a network (depends on settings), a thread doing usb bulk writes, and a thread doing usb bulk reads. Data is moved around with TChan, and TVar is used for coordination. When the application was compiled with 7.6, my stream of usb traffic was smooth. With 7.8, there are lots of delays where nothing seems to be running. These delays are up to 40ms, whereas with 7.6 delays were a 1ms or so. When I add -N2 or -N4, the 7.8 program runs fine. But on 7.6 it runs fine without with -N2/4. The program is compiled -O2 with profiling. The -N2/4 version uses more memory, but in both cases with 7.8 and with 7.6 there is no space leak. I tired to compile and use -ls so I could take a look with threadscope, but the application hangs and writes no data to the file. The CPU fans run wild like it is in an infinite loop. It at least pops an unpainted wxhaskell window, so it got partially running. One of my libraries uses option -fsimpl-tick-factor=200 to get around the compiler. What do I need to know about changes to threading and event logging between 7.6 and 7.8? Is there some general documentation somewhere that might help? I am on Ubuntu 14.04 LTS. I downloaded the 7.8 tool chain tar ball and installed myself, after removing 7.6 with apt-get. Any hints appreciated. Mike ___ Glasgow-haskell-users mailing list Glasgow-haskell-users@haskell.org mailto:Glasgow-haskell-users@haskell.org http://www.haskell.org/mailman/listinfo/glasgow-haskell-users ___ Glasgow-haskell-users mailing list Glasgow-haskell-users@haskell.org mailto:Glasgow-haskell-users@haskell.org http://www.haskell.org/mailman/listinfo/glasgow-haskell-users ___ Glasgow-haskell-users mailing list Glasgow-haskell-users@haskell.org http://www.haskell.org/mailman/listinfo/glasgow-haskell-users ___ Glasgow-haskell-users mailing list Glasgow-haskell-users@haskell.org http://www.haskell.org/mailman/listinfo/glasgow-haskell-users
Re: Thread behavior in 7.8.3
Donn, True, but in that case I was using a driver for the Aardvark, and my current two test cases use: A) DC1613A from LTC B) /dev/i2c driver with FFI wrapper I wrote Case A uses the haskell usb package and libusb. I suppose SIGVALRM could be in used in the libusb driver. I know for sure it is not used by my I2C stuff, unless it is behind the /dev/i2c user mode calls. But interesting. Obviously the scheduler is using timers from the OS. Is it really an advantage not to use OS threads all around? Is there anyway to enable such behavior to see if things are better? Mike On Jan 17, 2015, at 11:00 PM, Donn Cave d...@avvanta.com wrote: Quoth Michael Jones m...@proclivis.com, ... 5) What does -V0 do that makes a problem program run? Well, there's that SIGVTALRM barrage, you may remember we went over that mid-August. I expect there are other effects. Donn ___ Glasgow-haskell-users mailing list Glasgow-haskell-users@haskell.org http://www.haskell.org/mailman/listinfo/glasgow-haskell-users
Re: Thread behavior in 7.8.3
I have narrowed down the problem a bit. It turns out that many times if I run the program and wait long enough, it will start. Given an event log, it may take from 1000-1 entries sometimes. When I look at a good start vs. slow start, I see that in both cases things startup and there is some thread activity for thread 2 and 3, then the application starts creating other threads, which is when the wxhaskell GUI pops up and IO out my /dev/i2c begins. In the slow case, it just gets stuck on thread 2/3 activity for a very long time. If I switch from -C0.001 to -C0.010, the startup is more reliable, in that most starts result in an immediate GUI and i2c IO. The behavior suggests to me that some initial threads are starving the ability for other threads to start, and perhaps on a dual core machine it is more of a problem than single or quad core machines. For certain, due to some printing, I know that the main thread is starting, and that a print just before the first fork is not printing. Code between them is evaluating wxhaskell functions, but the main frame is not yet asked to become visible. From last week, I know that an non-gui version of the app is getting stuck, but I do not know if it eventually runs like this case. Is there some convention that when I look at an event log you can tell which threads are OS threads vs threads from fork? Perhaps someone that knows the scheduler might have some advice. It seems odd that a scheduler could behave this way. The scheduler should have some built in notion of fairness. On Jan 12, 2015, at 11:02 PM, Michael Jones m...@proclivis.com wrote: Sorry I am reviving an old problem, but it has resurfaced, such that one system behaves different than another. Using -C0.001 solved problems on a Mac + VM + Ubuntu 14. It worked on a single core 32 bit Atom NUC. But on a dual core Atom MinnowBoardMax, something bad is going on. In summary, the same code that runs on two machines does not run on a third machine. So this indicates I have not made any breaking changes to the code or cabal files. Compiling with GHC 7.8.3. This bad system has Ubuntu 14 installed, with an updated Linux 3.18.1 kernel. It is a dual core 64 bit I86 Atom processor. The application hangs at startup. If I remove the -C0.00N option and instead use -V0, the application runs. It has bad timing properties, but it does at least run. Note that a hang hangs an IO thread talking USB, and the GUI thread. When testing with the -C0.00N option, it did run 2 times out of 20 tries, so fail means fail most but not all of the time. When it did run, it continued to run properly. This perhaps indicates some kind of internal race condition. In the fail to run case, it does some printing up to the point where it tries to create a wxHaskell frame. In another non-UI version of the program it also fails to run. Logging to a file gives a similar indication. It is clear that the program starts up, then fails during the run in some form of lockup, well after the initial startup code. If I run with the strace command, it always runs with -C0.00N. All the above was done with profiling enabled, so I removed that and instead enabled eventlog to look for clues. In this case it lies between good and bad, in that IO to my USB is working, but the GUI comes up blank and never paints. Running this case without -v0 (event log) the gui partially paints and stops, but USB continues. Questions: 1) Does ghc 7.8.4 have any improvements that might pertain to these kinds of scheduling/thread problems? 2) Is there anything about the nature of a thread using USB, I2C, or wxHaskell IO that leads to problems that a pure calculation app would not have? 3) Any ideas how to track down the problem when changing conditions (compiler or runtime options) affects behavior? 4) Are there other options besides -V and -C for the runtime that might apply? 5) What does -V0 do that makes a problem program run? Mike On Oct 29, 2014, at 6:02 PM, Michael Jones m...@proclivis.com wrote: John, Adding -C0.005 makes it much better. Using -C0.001 makes it behave more like -N4. Thanks. This saves my project, as I need to deploy on a single core Atom and was stuck. Mike On Oct 29, 2014, at 5:12 PM, John Lato jwl...@gmail.com wrote: By any chance do the delays get shorter if you run your program with `+RTS -C0.005` ? If so, I suspect you're having a problem very similar to one that we had with ghc-7.8 (7.6 too, but it's worse on ghc-7.8 for some reason), involving possible misbehavior of the thread scheduler. On Wed, Oct 29, 2014 at 2:18 PM, Michael Jones m...@proclivis.com wrote: I have a general question about thread behavior in 7.8.3 vs 7.6.X I moved from 7.6 to 7.8 and my application behaves very differently. I have three threads, an application thread that plots data with wxhaskell or sends it over a network (depends
Re: Thread behavior in 7.8.3
Quoth Michael Jones m...@proclivis.com, ... 5) What does -V0 do that makes a problem program run? Well, there's that SIGVTALRM barrage, you may remember we went over that mid-August. I expect there are other effects. Donn ___ Glasgow-haskell-users mailing list Glasgow-haskell-users@haskell.org http://www.haskell.org/mailman/listinfo/glasgow-haskell-users
Re: Thread behavior in 7.8.3
Michael Jones m...@proclivis.com writes: Sorry I am reviving an old problem, but it has resurfaced, such that one system behaves different than another. [snip] 1) Does ghc 7.8.4 have any improvements that might pertain to these kinds of scheduling/thread problems? 2) Is there anything about the nature of a thread using USB, I2C, or wxHaskell IO that leads to problems that a pure calculation app would not have? Do you know about [1]? This is a regression due to an interface change that arose from the new event manager. `usb` 1.3.0.0` has a workaround, GHC 7.10 will have a fixed event manager [2]. Given that it sounds like your program works some of the time this may not be relevant but I thought it would be negligent not to mention it. Cheers, - Ben [1] https://github.com/basvandijk/usb/issues/7 [2] https://phabricator.haskell.org/D347 pgpcJkuBM6EEF.pgp Description: PGP signature ___ Glasgow-haskell-users mailing list Glasgow-haskell-users@haskell.org http://www.haskell.org/mailman/listinfo/glasgow-haskell-users
Re: Thread behavior in 7.8.3
Ben, Interesting. In this case, I can duplicate the problem when not using USB (USB to i2c dongle) by using /dev/i2c_n, and when I do use USB, in some cases the USB is working (can see i2c on scope), but the GUI is hung. So I believe this is not causing the problem. Thanks, Mike On Jan 13, 2015, at 1:02 PM, Ben Gamari bgamari.f...@gmail.com wrote: Michael Jones m...@proclivis.com writes: Sorry I am reviving an old problem, but it has resurfaced, such that one system behaves different than another. [snip] 1) Does ghc 7.8.4 have any improvements that might pertain to these kinds of scheduling/thread problems? 2) Is there anything about the nature of a thread using USB, I2C, or wxHaskell IO that leads to problems that a pure calculation app would not have? Do you know about [1]? This is a regression due to an interface change that arose from the new event manager. `usb` 1.3.0.0` has a workaround, GHC 7.10 will have a fixed event manager [2]. Given that it sounds like your program works some of the time this may not be relevant but I thought it would be negligent not to mention it. Cheers, - Ben [1] https://github.com/basvandijk/usb/issues/7 [2] https://phabricator.haskell.org/D347 ___ Glasgow-haskell-users mailing list Glasgow-haskell-users@haskell.org http://www.haskell.org/mailman/listinfo/glasgow-haskell-users
Re: Thread behavior in 7.8.3
Sorry I am reviving an old problem, but it has resurfaced, such that one system behaves different than another. Using -C0.001 solved problems on a Mac + VM + Ubuntu 14. It worked on a single core 32 bit Atom NUC. But on a dual core Atom MinnowBoardMax, something bad is going on. In summary, the same code that runs on two machines does not run on a third machine. So this indicates I have not made any breaking changes to the code or cabal files. Compiling with GHC 7.8.3. This bad system has Ubuntu 14 installed, with an updated Linux 3.18.1 kernel. It is a dual core 64 bit I86 Atom processor. The application hangs at startup. If I remove the -C0.00N option and instead use -V0, the application runs. It has bad timing properties, but it does at least run. Note that a hang hangs an IO thread talking USB, and the GUI thread. When testing with the -C0.00N option, it did run 2 times out of 20 tries, so fail means fail most but not all of the time. When it did run, it continued to run properly. This perhaps indicates some kind of internal race condition. In the fail to run case, it does some printing up to the point where it tries to create a wxHaskell frame. In another non-UI version of the program it also fails to run. Logging to a file gives a similar indication. It is clear that the program starts up, then fails during the run in some form of lockup, well after the initial startup code. If I run with the strace command, it always runs with -C0.00N. All the above was done with profiling enabled, so I removed that and instead enabled eventlog to look for clues. In this case it lies between good and bad, in that IO to my USB is working, but the GUI comes up blank and never paints. Running this case without -v0 (event log) the gui partially paints and stops, but USB continues. Questions: 1) Does ghc 7.8.4 have any improvements that might pertain to these kinds of scheduling/thread problems? 2) Is there anything about the nature of a thread using USB, I2C, or wxHaskell IO that leads to problems that a pure calculation app would not have? 3) Any ideas how to track down the problem when changing conditions (compiler or runtime options) affects behavior? 4) Are there other options besides -V and -C for the runtime that might apply? 5) What does -V0 do that makes a problem program run? Mike On Oct 29, 2014, at 6:02 PM, Michael Jones m...@proclivis.com wrote: John, Adding -C0.005 makes it much better. Using -C0.001 makes it behave more like -N4. Thanks. This saves my project, as I need to deploy on a single core Atom and was stuck. Mike On Oct 29, 2014, at 5:12 PM, John Lato jwl...@gmail.com wrote: By any chance do the delays get shorter if you run your program with `+RTS -C0.005` ? If so, I suspect you're having a problem very similar to one that we had with ghc-7.8 (7.6 too, but it's worse on ghc-7.8 for some reason), involving possible misbehavior of the thread scheduler. On Wed, Oct 29, 2014 at 2:18 PM, Michael Jones m...@proclivis.com wrote: I have a general question about thread behavior in 7.8.3 vs 7.6.X I moved from 7.6 to 7.8 and my application behaves very differently. I have three threads, an application thread that plots data with wxhaskell or sends it over a network (depends on settings), a thread doing usb bulk writes, and a thread doing usb bulk reads. Data is moved around with TChan, and TVar is used for coordination. When the application was compiled with 7.6, my stream of usb traffic was smooth. With 7.8, there are lots of delays where nothing seems to be running. These delays are up to 40ms, whereas with 7.6 delays were a 1ms or so. When I add -N2 or -N4, the 7.8 program runs fine. But on 7.6 it runs fine without with -N2/4. The program is compiled -O2 with profiling. The -N2/4 version uses more memory, but in both cases with 7.8 and with 7.6 there is no space leak. I tired to compile and use -ls so I could take a look with threadscope, but the application hangs and writes no data to the file. The CPU fans run wild like it is in an infinite loop. It at least pops an unpainted wxhaskell window, so it got partially running. One of my libraries uses option -fsimpl-tick-factor=200 to get around the compiler. What do I need to know about changes to threading and event logging between 7.6 and 7.8? Is there some general documentation somewhere that might help? I am on Ubuntu 14.04 LTS. I downloaded the 7.8 tool chain tar ball and installed myself, after removing 7.6 with apt-get. Any hints appreciated. Mike ___ Glasgow-haskell-users mailing list Glasgow-haskell-users@haskell.org http://www.haskell.org/mailman/listinfo/glasgow-haskell-users ___ Glasgow-haskell-users mailing list Glasgow-haskell-users@haskell.org http://www.haskell.org/mailman/listinfo/glasgow-haskell-users
RE: Thread behavior in 7.8.3
I wonder if the knowledge embodied in this thread might usefully be summarised in the user manual? Or on the GHC section of the Haskell wiki https://www.haskell.org/haskellwiki/GHC? Simon | -Original Message- | From: Glasgow-haskell-users [mailto:glasgow-haskell-users- | boun...@haskell.org] On Behalf Of Edward Z. Yang | Sent: 30 October 2014 00:41 | To: John Lato | Cc: GHC Users List | Subject: Re: Thread behavior in 7.8.3 | | Yes, that's right. | | I brought it up because you mentioned that there might still be | occasional delays, and those might be caused by a thread not being | preemptible for a while. | | Edward | | Excerpts from John Lato's message of 2014-10-29 17:31:45 -0700: | My understanding is that -fno-omit-yields is subtly different. I | think that's for the case when a function loops without performing | any | heap allocations, and thus would never yield even after the context | switch timeout. In my case the looping function does perform heap | allocations and does eventually yield, just not until after the | timeout. | | Is that understanding correct? | | (technically, doesn't it change to yielding after stack checks or | something like that?) | | On Thu, Oct 30, 2014 at 8:24 AM, Edward Z. Yang ezy...@mit.edu | wrote: | |I don't think this is directly related to the problem, but if you |have a thread that isn't yielding, you can force it to yield by |using -fno-omit-yields on your code. It won't help if the |non-yielding code is in a library, and it won't help if the | problem |was that you just weren't setting timeouts finely enough (which |sounds like what was happening). FYI. | |Edward | |Excerpts from John Lato's message of 2014-10-29 17:19:46 -0700: | I guess I should explain what that flag does... | | The GHC RTS maintains capabilities, the number of capabilities | is |specified | by the `+RTS -N` option. Each capability is a virtual machine | that executes Haskell code, and maintains its own runqueue of | threads to |process. | | A capability will perform a context switch at the next heap | block | allocation (every 4k of allocation) after the timer expires. | The | timer defaults to 20ms, and can be set by the -C flag. | Capabilities perform context switches in other circumstances as | well, such as when a thread yields or blocks. | | My guess is that either the context switching logic changed in | ghc-7.8, |or | possibly your code used to trigger a switch via some other | mechanism |(stack | overflow or something maybe?), but is optimized differently now | so |instead | it needs to wait for the timer to expire. | | The problem we had was that a time-sensitive thread was getting | scheduled on the same capability as a long-running non-yielding | thread, so the time-sensitive thread had to wait for a context | switch timeout (even |though | there were free cores available!). I expect even with -N4 | you'll | still |see | occasional delays (perhaps 5% of calls). | | We've solved our problem with judicious use of `forkOn`, but | that | won't help at N1. | | We did see this behavior in 7.6, but it's definitely worse in | 7.8. | | Incidentally, has there been any interest in a work-stealing | scheduler? | There was a discussion from about 2 years ago, in which Simon | Marlow |noted | it might be tricky, but it would definitely help in situations | like this. | | John L. | | On Thu, Oct 30, 2014 at 8:02 AM, Michael Jones | m...@proclivis.com |wrote: | | John, | | Adding -C0.005 makes it much better. Using -C0.001 makes it | behave more like -N4. | | Thanks. This saves my project, as I need to deploy on a single | core |Atom | and was stuck. | | Mike | | On Oct 29, 2014, at 5:12 PM, John Lato jwl...@gmail.com | wrote: | | By any chance do the delays get shorter if you run your | program | with |`+RTS | -C0.005` ? If so, I suspect you're having a problem very | similar to |one | that we had with ghc-7.8 (7.6 too, but it's worse on ghc-7.8 | for | some reason), involving possible misbehavior of the thread | scheduler. | | On Wed, Oct 29, 2014 at 2:18 PM, Michael Jones | m...@proclivis.com |wrote: | | I have a general question about thread behavior in 7.8.3 vs | 7.6.X | | I moved from 7.6 to 7.8 and my application behaves very | differently. I have three threads, an application thread that | plots data with |wxhaskell or | sends it over a network (depends on settings), a thread doing | usb bulk writes, and a thread doing usb bulk reads. Data is | moved around with |TChan
Re: Thread behavior in 7.8.3
on ghc-7.8 for some reason), involving possible misbehavior of the thread scheduler. On Wed, Oct 29, 2014 at 2:18 PM, Michael Jones m...@proclivis.com wrote: I have a general question about thread behavior in 7.8.3 vs 7.6.X I moved from 7.6 to 7.8 and my application behaves very differently. I have three threads, an application thread that plots data with wxhaskell or sends it over a network (depends on settings), a thread doing usb bulk writes, and a thread doing usb bulk reads. Data is moved around with TChan, and TVar is used for coordination. When the application was compiled with 7.6, my stream of usb traffic was smooth. With 7.8, there are lots of delays where nothing seems to be running. These delays are up to 40ms, whereas with 7.6 delays were a 1ms or so. When I add -N2 or -N4, the 7.8 program runs fine. But on 7.6 it runs fine without with -N2/4. The program is compiled -O2 with profiling. The -N2/4 version uses more memory, but in both cases with 7.8 and with 7.6 there is no space leak. I tired to compile and use -ls so I could take a look with threadscope, but the application hangs and writes no data to the file. The CPU fans run wild like it is in an infinite loop. It at least pops an unpainted wxhaskell window, so it got partially running. One of my libraries uses option -fsimpl-tick-factor=200 to get around the compiler. What do I need to know about changes to threading and event logging between 7.6 and 7.8? Is there some general documentation somewhere that might help? I am on Ubuntu 14.04 LTS. I downloaded the 7.8 tool chain tar ball and installed myself, after removing 7.6 with apt-get. Any hints appreciated. Mike ___ Glasgow-haskell-users mailing list Glasgow-haskell-users@haskell.org http://www.haskell.org/mailman/listinfo/glasgow-haskell-users ___ Glasgow-haskell-users mailing list Glasgow-haskell-users@haskell.org http://www.haskell.org/mailman/listinfo/glasgow-haskell-users
Re: Thread behavior in 7.8.3
did see this behavior in 7.6, but it's definitely worse in 7.8. Incidentally, has there been any interest in a work-stealing scheduler? There was a discussion from about 2 years ago, in which Simon Marlow noted it might be tricky, but it would definitely help in situations like this. John L. On Thu, Oct 30, 2014 at 8:02 AM, Michael Jones m...@proclivis.com wrote: John, Adding -C0.005 makes it much better. Using -C0.001 makes it behave more like -N4. Thanks. This saves my project, as I need to deploy on a single core Atom and was stuck. Mike On Oct 29, 2014, at 5:12 PM, John Lato jwl...@gmail.com wrote: By any chance do the delays get shorter if you run your program with `+RTS -C0.005` ? If so, I suspect you're having a problem very similar to one that we had with ghc-7.8 (7.6 too, but it's worse on ghc-7.8 for some reason), involving possible misbehavior of the thread scheduler. On Wed, Oct 29, 2014 at 2:18 PM, Michael Jones m...@proclivis.com wrote: I have a general question about thread behavior in 7.8.3 vs 7.6.X I moved from 7.6 to 7.8 and my application behaves very differently. I have three threads, an application thread that plots data with wxhaskell or sends it over a network (depends on settings), a thread doing usb bulk writes, and a thread doing usb bulk reads. Data is moved around with TChan, and TVar is used for coordination. When the application was compiled with 7.6, my stream of usb traffic was smooth. With 7.8, there are lots of delays where nothing seems to be running. These delays are up to 40ms, whereas with 7.6 delays were a 1ms or so. When I add -N2 or -N4, the 7.8 program runs fine. But on 7.6 it runs fine without with -N2/4. The program is compiled -O2 with profiling. The -N2/4 version uses more memory, but in both cases with 7.8 and with 7.6 there is no space leak. I tired to compile and use -ls so I could take a look with threadscope, but the application hangs and writes no data to the file. The CPU fans run wild like it is in an infinite loop. It at least pops an unpainted wxhaskell window, so it got partially running. One of my libraries uses option -fsimpl-tick-factor=200 to get around the compiler. What do I need to know about changes to threading and event logging between 7.6 and 7.8? Is there some general documentation somewhere that might help? I am on Ubuntu 14.04 LTS. I downloaded the 7.8 tool chain tar ball and installed myself, after removing 7.6 with apt-get. Any hints appreciated. Mike ___ Glasgow-haskell-users mailing list Glasgow-haskell-users@haskell.org http://www.haskell.org/mailman/listinfo/glasgow-haskell-users ___ Glasgow-haskell-users mailing list Glasgow-haskell-users@haskell.org http://www.haskell.org/mailman/listinfo/glasgow-haskell-users
Thread behavior in 7.8.3
I have a general question about thread behavior in 7.8.3 vs 7.6.X I moved from 7.6 to 7.8 and my application behaves very differently. I have three threads, an application thread that plots data with wxhaskell or sends it over a network (depends on settings), a thread doing usb bulk writes, and a thread doing usb bulk reads. Data is moved around with TChan, and TVar is used for coordination. When the application was compiled with 7.6, my stream of usb traffic was smooth. With 7.8, there are lots of delays where nothing seems to be running. These delays are up to 40ms, whereas with 7.6 delays were a 1ms or so. When I add -N2 or -N4, the 7.8 program runs fine. But on 7.6 it runs fine without with -N2/4. The program is compiled -O2 with profiling. The -N2/4 version uses more memory, but in both cases with 7.8 and with 7.6 there is no space leak. I tired to compile and use -ls so I could take a look with threadscope, but the application hangs and writes no data to the file. The CPU fans run wild like it is in an infinite loop. It at least pops an unpainted wxhaskell window, so it got partially running. One of my libraries uses option -fsimpl-tick-factor=200 to get around the compiler. What do I need to know about changes to threading and event logging between 7.6 and 7.8? Is there some general documentation somewhere that might help? I am on Ubuntu 14.04 LTS. I downloaded the 7.8 tool chain tar ball and installed myself, after removing 7.6 with apt-get. Any hints appreciated. Mike ___ Glasgow-haskell-users mailing list Glasgow-haskell-users@haskell.org http://www.haskell.org/mailman/listinfo/glasgow-haskell-users
Re: Thread behavior in 7.8.3
Michael Jones m...@proclivis.com writes: I have a general question about thread behavior in 7.8.3 vs 7.6.X I moved from 7.6 to 7.8 and my application behaves very differently. I have three threads, an application thread that plots data with wxhaskell or sends it over a network (depends on settings), a thread doing usb bulk writes, and a thread doing usb bulk reads. Data is moved around with TChan, and TVar is used for coordination. Are you using Bas van Dijk's `usb` library by any chance? If so, you should be aware of this [1] issue. When the application was compiled with 7.6, my stream of usb traffic was smooth. With 7.8, there are lots of delays where nothing seems to be running. These delays are up to 40ms, whereas with 7.6 delays were a 1ms or so. When I add -N2 or -N4, the 7.8 program runs fine. But on 7.6 it runs fine without with -N2/4. The program is compiled -O2 with profiling. The -N2/4 version uses more memory, but in both cases with 7.8 and with 7.6 there is no space leak. Have you looked at the RTS's output when run with `+RTS -sstderr`? Is productivity any different between the two tests? I tired to compile and use -ls so I could take a look with threadscope, but the application hangs and writes no data to the file. The CPU fans run wild like it is in an infinite loop. Oh dear, this doesn't sound good at all. Have you tried getting a backtrace out of gdb? Usually this isn't terribly useful but in this case since the event log is involved it might be getting stuck in the RTS which should give a useful backtrace. If not, perhaps strace will give some clues as to what is happening (you'll probably want to hide SIGVTALM to improve signal/noise)? Cheers, - Ben [1] https://github.com/basvandijk/usb/issues/7 pgpGwIoqAmMbI.pgp Description: PGP signature ___ Glasgow-haskell-users mailing list Glasgow-haskell-users@haskell.org http://www.haskell.org/mailman/listinfo/glasgow-haskell-users
Re: Thread behavior in 7.8.3
Ben, I am using Bas van Dijk’s usb, and I am past the -threading issue by using the latest commit. I don’t have any easy way of making comparisons between 7.6 and 7.8 productivity, but from oscilloscope activity, I can’t see any difference. The only difference I see is the thread scheduling on 7.8 for -N1 vs -N2/4. If —sstderr gives some notion of productivity, I’ll have to do an experiment between -N1 and -N2/4. Unchartered territory for me. I’ll setup and experiment tonight. I am not familiar with strace. I’ll fix that soon. Mike On Oct 29, 2014, at 10:24 AM, Ben Gamari bgamari.f...@gmail.com wrote: Michael Jones m...@proclivis.com writes: I have a general question about thread behavior in 7.8.3 vs 7.6.X I moved from 7.6 to 7.8 and my application behaves very differently. I have three threads, an application thread that plots data with wxhaskell or sends it over a network (depends on settings), a thread doing usb bulk writes, and a thread doing usb bulk reads. Data is moved around with TChan, and TVar is used for coordination. Are you using Bas van Dijk's `usb` library by any chance? If so, you should be aware of this [1] issue. When the application was compiled with 7.6, my stream of usb traffic was smooth. With 7.8, there are lots of delays where nothing seems to be running. These delays are up to 40ms, whereas with 7.6 delays were a 1ms or so. When I add -N2 or -N4, the 7.8 program runs fine. But on 7.6 it runs fine without with -N2/4. The program is compiled -O2 with profiling. The -N2/4 version uses more memory, but in both cases with 7.8 and with 7.6 there is no space leak. Have you looked at the RTS's output when run with `+RTS -sstderr`? Is productivity any different between the two tests? I tired to compile and use -ls so I could take a look with threadscope, but the application hangs and writes no data to the file. The CPU fans run wild like it is in an infinite loop. Oh dear, this doesn't sound good at all. Have you tried getting a backtrace out of gdb? Usually this isn't terribly useful but in this case since the event log is involved it might be getting stuck in the RTS which should give a useful backtrace. If not, perhaps strace will give some clues as to what is happening (you'll probably want to hide SIGVTALM to improve signal/noise)? Cheers, - Ben [1] https://github.com/basvandijk/usb/issues/7 ___ Glasgow-haskell-users mailing list Glasgow-haskell-users@haskell.org http://www.haskell.org/mailman/listinfo/glasgow-haskell-users
Re: Thread behavior in 7.8.3
Michael Jones m...@proclivis.com writes: Ben, I am using Bas van Dijk’s usb, and I am past the -threading issue by using the latest commit. Excellent; I hadn't noticed the proclivis in your email address I don’t have any easy way of making comparisons between 7.6 and 7.8 productivity, but from oscilloscope activity, I can’t see any difference. The only difference I see is the thread scheduling on 7.8 for -N1 vs -N2/4. If —sstderr gives some notion of productivity, I’ll have to do an experiment between -N1 and -N2/4. Unchartered territory for me. I’ll setup and experiment tonight. Indeed it does; productivity in this context refers to the fraction of runtime spent in evaluation (as opposed to in the garbage collector, for instance). I am not familiar with strace. I’ll fix that soon. It's often an invaluable tool; that being said it remains to seen whether it yields anything useful in this particular case. Cheers, - Ben pgp2rrHbAUnFW.pgp Description: PGP signature ___ Glasgow-haskell-users mailing list Glasgow-haskell-users@haskell.org http://www.haskell.org/mailman/listinfo/glasgow-haskell-users
Re: Thread behavior in 7.8.3
By any chance do the delays get shorter if you run your program with `+RTS -C0.005` ? If so, I suspect you're having a problem very similar to one that we had with ghc-7.8 (7.6 too, but it's worse on ghc-7.8 for some reason), involving possible misbehavior of the thread scheduler. On Wed, Oct 29, 2014 at 2:18 PM, Michael Jones m...@proclivis.com wrote: I have a general question about thread behavior in 7.8.3 vs 7.6.X I moved from 7.6 to 7.8 and my application behaves very differently. I have three threads, an application thread that plots data with wxhaskell or sends it over a network (depends on settings), a thread doing usb bulk writes, and a thread doing usb bulk reads. Data is moved around with TChan, and TVar is used for coordination. When the application was compiled with 7.6, my stream of usb traffic was smooth. With 7.8, there are lots of delays where nothing seems to be running. These delays are up to 40ms, whereas with 7.6 delays were a 1ms or so. When I add -N2 or -N4, the 7.8 program runs fine. But on 7.6 it runs fine without with -N2/4. The program is compiled -O2 with profiling. The -N2/4 version uses more memory, but in both cases with 7.8 and with 7.6 there is no space leak. I tired to compile and use -ls so I could take a look with threadscope, but the application hangs and writes no data to the file. The CPU fans run wild like it is in an infinite loop. It at least pops an unpainted wxhaskell window, so it got partially running. One of my libraries uses option -fsimpl-tick-factor=200 to get around the compiler. What do I need to know about changes to threading and event logging between 7.6 and 7.8? Is there some general documentation somewhere that might help? I am on Ubuntu 14.04 LTS. I downloaded the 7.8 tool chain tar ball and installed myself, after removing 7.6 with apt-get. Any hints appreciated. Mike ___ Glasgow-haskell-users mailing list Glasgow-haskell-users@haskell.org http://www.haskell.org/mailman/listinfo/glasgow-haskell-users ___ Glasgow-haskell-users mailing list Glasgow-haskell-users@haskell.org http://www.haskell.org/mailman/listinfo/glasgow-haskell-users
Re: Thread behavior in 7.8.3
John, Adding -C0.005 makes it much better. Using -C0.001 makes it behave more like -N4. Thanks. This saves my project, as I need to deploy on a single core Atom and was stuck. Mike On Oct 29, 2014, at 5:12 PM, John Lato jwl...@gmail.com wrote: By any chance do the delays get shorter if you run your program with `+RTS -C0.005` ? If so, I suspect you're having a problem very similar to one that we had with ghc-7.8 (7.6 too, but it's worse on ghc-7.8 for some reason), involving possible misbehavior of the thread scheduler. On Wed, Oct 29, 2014 at 2:18 PM, Michael Jones m...@proclivis.com wrote: I have a general question about thread behavior in 7.8.3 vs 7.6.X I moved from 7.6 to 7.8 and my application behaves very differently. I have three threads, an application thread that plots data with wxhaskell or sends it over a network (depends on settings), a thread doing usb bulk writes, and a thread doing usb bulk reads. Data is moved around with TChan, and TVar is used for coordination. When the application was compiled with 7.6, my stream of usb traffic was smooth. With 7.8, there are lots of delays where nothing seems to be running. These delays are up to 40ms, whereas with 7.6 delays were a 1ms or so. When I add -N2 or -N4, the 7.8 program runs fine. But on 7.6 it runs fine without with -N2/4. The program is compiled -O2 with profiling. The -N2/4 version uses more memory, but in both cases with 7.8 and with 7.6 there is no space leak. I tired to compile and use -ls so I could take a look with threadscope, but the application hangs and writes no data to the file. The CPU fans run wild like it is in an infinite loop. It at least pops an unpainted wxhaskell window, so it got partially running. One of my libraries uses option -fsimpl-tick-factor=200 to get around the compiler. What do I need to know about changes to threading and event logging between 7.6 and 7.8? Is there some general documentation somewhere that might help? I am on Ubuntu 14.04 LTS. I downloaded the 7.8 tool chain tar ball and installed myself, after removing 7.6 with apt-get. Any hints appreciated. Mike ___ Glasgow-haskell-users mailing list Glasgow-haskell-users@haskell.org http://www.haskell.org/mailman/listinfo/glasgow-haskell-users ___ Glasgow-haskell-users mailing list Glasgow-haskell-users@haskell.org http://www.haskell.org/mailman/listinfo/glasgow-haskell-users
Re: Thread behavior in 7.8.3
I guess I should explain what that flag does... The GHC RTS maintains capabilities, the number of capabilities is specified by the `+RTS -N` option. Each capability is a virtual machine that executes Haskell code, and maintains its own runqueue of threads to process. A capability will perform a context switch at the next heap block allocation (every 4k of allocation) after the timer expires. The timer defaults to 20ms, and can be set by the -C flag. Capabilities perform context switches in other circumstances as well, such as when a thread yields or blocks. My guess is that either the context switching logic changed in ghc-7.8, or possibly your code used to trigger a switch via some other mechanism (stack overflow or something maybe?), but is optimized differently now so instead it needs to wait for the timer to expire. The problem we had was that a time-sensitive thread was getting scheduled on the same capability as a long-running non-yielding thread, so the time-sensitive thread had to wait for a context switch timeout (even though there were free cores available!). I expect even with -N4 you'll still see occasional delays (perhaps 5% of calls). We've solved our problem with judicious use of `forkOn`, but that won't help at N1. We did see this behavior in 7.6, but it's definitely worse in 7.8. Incidentally, has there been any interest in a work-stealing scheduler? There was a discussion from about 2 years ago, in which Simon Marlow noted it might be tricky, but it would definitely help in situations like this. John L. On Thu, Oct 30, 2014 at 8:02 AM, Michael Jones m...@proclivis.com wrote: John, Adding -C0.005 makes it much better. Using -C0.001 makes it behave more like -N4. Thanks. This saves my project, as I need to deploy on a single core Atom and was stuck. Mike On Oct 29, 2014, at 5:12 PM, John Lato jwl...@gmail.com wrote: By any chance do the delays get shorter if you run your program with `+RTS -C0.005` ? If so, I suspect you're having a problem very similar to one that we had with ghc-7.8 (7.6 too, but it's worse on ghc-7.8 for some reason), involving possible misbehavior of the thread scheduler. On Wed, Oct 29, 2014 at 2:18 PM, Michael Jones m...@proclivis.com wrote: I have a general question about thread behavior in 7.8.3 vs 7.6.X I moved from 7.6 to 7.8 and my application behaves very differently. I have three threads, an application thread that plots data with wxhaskell or sends it over a network (depends on settings), a thread doing usb bulk writes, and a thread doing usb bulk reads. Data is moved around with TChan, and TVar is used for coordination. When the application was compiled with 7.6, my stream of usb traffic was smooth. With 7.8, there are lots of delays where nothing seems to be running. These delays are up to 40ms, whereas with 7.6 delays were a 1ms or so. When I add -N2 or -N4, the 7.8 program runs fine. But on 7.6 it runs fine without with -N2/4. The program is compiled -O2 with profiling. The -N2/4 version uses more memory, but in both cases with 7.8 and with 7.6 there is no space leak. I tired to compile and use -ls so I could take a look with threadscope, but the application hangs and writes no data to the file. The CPU fans run wild like it is in an infinite loop. It at least pops an unpainted wxhaskell window, so it got partially running. One of my libraries uses option -fsimpl-tick-factor=200 to get around the compiler. What do I need to know about changes to threading and event logging between 7.6 and 7.8? Is there some general documentation somewhere that might help? I am on Ubuntu 14.04 LTS. I downloaded the 7.8 tool chain tar ball and installed myself, after removing 7.6 with apt-get. Any hints appreciated. Mike ___ Glasgow-haskell-users mailing list Glasgow-haskell-users@haskell.org http://www.haskell.org/mailman/listinfo/glasgow-haskell-users ___ Glasgow-haskell-users mailing list Glasgow-haskell-users@haskell.org http://www.haskell.org/mailman/listinfo/glasgow-haskell-users
Re: Thread behavior in 7.8.3
I don't think this is directly related to the problem, but if you have a thread that isn't yielding, you can force it to yield by using -fno-omit-yields on your code. It won't help if the non-yielding code is in a library, and it won't help if the problem was that you just weren't setting timeouts finely enough (which sounds like what was happening). FYI. Edward Excerpts from John Lato's message of 2014-10-29 17:19:46 -0700: I guess I should explain what that flag does... The GHC RTS maintains capabilities, the number of capabilities is specified by the `+RTS -N` option. Each capability is a virtual machine that executes Haskell code, and maintains its own runqueue of threads to process. A capability will perform a context switch at the next heap block allocation (every 4k of allocation) after the timer expires. The timer defaults to 20ms, and can be set by the -C flag. Capabilities perform context switches in other circumstances as well, such as when a thread yields or blocks. My guess is that either the context switching logic changed in ghc-7.8, or possibly your code used to trigger a switch via some other mechanism (stack overflow or something maybe?), but is optimized differently now so instead it needs to wait for the timer to expire. The problem we had was that a time-sensitive thread was getting scheduled on the same capability as a long-running non-yielding thread, so the time-sensitive thread had to wait for a context switch timeout (even though there were free cores available!). I expect even with -N4 you'll still see occasional delays (perhaps 5% of calls). We've solved our problem with judicious use of `forkOn`, but that won't help at N1. We did see this behavior in 7.6, but it's definitely worse in 7.8. Incidentally, has there been any interest in a work-stealing scheduler? There was a discussion from about 2 years ago, in which Simon Marlow noted it might be tricky, but it would definitely help in situations like this. John L. On Thu, Oct 30, 2014 at 8:02 AM, Michael Jones m...@proclivis.com wrote: John, Adding -C0.005 makes it much better. Using -C0.001 makes it behave more like -N4. Thanks. This saves my project, as I need to deploy on a single core Atom and was stuck. Mike On Oct 29, 2014, at 5:12 PM, John Lato jwl...@gmail.com wrote: By any chance do the delays get shorter if you run your program with `+RTS -C0.005` ? If so, I suspect you're having a problem very similar to one that we had with ghc-7.8 (7.6 too, but it's worse on ghc-7.8 for some reason), involving possible misbehavior of the thread scheduler. On Wed, Oct 29, 2014 at 2:18 PM, Michael Jones m...@proclivis.com wrote: I have a general question about thread behavior in 7.8.3 vs 7.6.X I moved from 7.6 to 7.8 and my application behaves very differently. I have three threads, an application thread that plots data with wxhaskell or sends it over a network (depends on settings), a thread doing usb bulk writes, and a thread doing usb bulk reads. Data is moved around with TChan, and TVar is used for coordination. When the application was compiled with 7.6, my stream of usb traffic was smooth. With 7.8, there are lots of delays where nothing seems to be running. These delays are up to 40ms, whereas with 7.6 delays were a 1ms or so. When I add -N2 or -N4, the 7.8 program runs fine. But on 7.6 it runs fine without with -N2/4. The program is compiled -O2 with profiling. The -N2/4 version uses more memory, but in both cases with 7.8 and with 7.6 there is no space leak. I tired to compile and use -ls so I could take a look with threadscope, but the application hangs and writes no data to the file. The CPU fans run wild like it is in an infinite loop. It at least pops an unpainted wxhaskell window, so it got partially running. One of my libraries uses option -fsimpl-tick-factor=200 to get around the compiler. What do I need to know about changes to threading and event logging between 7.6 and 7.8? Is there some general documentation somewhere that might help? I am on Ubuntu 14.04 LTS. I downloaded the 7.8 tool chain tar ball and installed myself, after removing 7.6 with apt-get. Any hints appreciated. Mike ___ Glasgow-haskell-users mailing list Glasgow-haskell-users@haskell.org http://www.haskell.org/mailman/listinfo/glasgow-haskell-users ___ Glasgow-haskell-users mailing list Glasgow-haskell-users@haskell.org http://www.haskell.org/mailman/listinfo/glasgow-haskell-users
Re: Thread behavior in 7.8.3
Yes, that's right. I brought it up because you mentioned that there might still be occasional delays, and those might be caused by a thread not being preemptible for a while. Edward Excerpts from John Lato's message of 2014-10-29 17:31:45 -0700: My understanding is that -fno-omit-yields is subtly different. I think that's for the case when a function loops without performing any heap allocations, and thus would never yield even after the context switch timeout. In my case the looping function does perform heap allocations and does eventually yield, just not until after the timeout. Is that understanding correct? (technically, doesn't it change to yielding after stack checks or something like that?) On Thu, Oct 30, 2014 at 8:24 AM, Edward Z. Yang ezy...@mit.edu wrote: I don't think this is directly related to the problem, but if you have a thread that isn't yielding, you can force it to yield by using -fno-omit-yields on your code. It won't help if the non-yielding code is in a library, and it won't help if the problem was that you just weren't setting timeouts finely enough (which sounds like what was happening). FYI. Edward Excerpts from John Lato's message of 2014-10-29 17:19:46 -0700: I guess I should explain what that flag does... The GHC RTS maintains capabilities, the number of capabilities is specified by the `+RTS -N` option. Each capability is a virtual machine that executes Haskell code, and maintains its own runqueue of threads to process. A capability will perform a context switch at the next heap block allocation (every 4k of allocation) after the timer expires. The timer defaults to 20ms, and can be set by the -C flag. Capabilities perform context switches in other circumstances as well, such as when a thread yields or blocks. My guess is that either the context switching logic changed in ghc-7.8, or possibly your code used to trigger a switch via some other mechanism (stack overflow or something maybe?), but is optimized differently now so instead it needs to wait for the timer to expire. The problem we had was that a time-sensitive thread was getting scheduled on the same capability as a long-running non-yielding thread, so the time-sensitive thread had to wait for a context switch timeout (even though there were free cores available!). I expect even with -N4 you'll still see occasional delays (perhaps 5% of calls). We've solved our problem with judicious use of `forkOn`, but that won't help at N1. We did see this behavior in 7.6, but it's definitely worse in 7.8. Incidentally, has there been any interest in a work-stealing scheduler? There was a discussion from about 2 years ago, in which Simon Marlow noted it might be tricky, but it would definitely help in situations like this. John L. On Thu, Oct 30, 2014 at 8:02 AM, Michael Jones m...@proclivis.com wrote: John, Adding -C0.005 makes it much better. Using -C0.001 makes it behave more like -N4. Thanks. This saves my project, as I need to deploy on a single core Atom and was stuck. Mike On Oct 29, 2014, at 5:12 PM, John Lato jwl...@gmail.com wrote: By any chance do the delays get shorter if you run your program with `+RTS -C0.005` ? If so, I suspect you're having a problem very similar to one that we had with ghc-7.8 (7.6 too, but it's worse on ghc-7.8 for some reason), involving possible misbehavior of the thread scheduler. On Wed, Oct 29, 2014 at 2:18 PM, Michael Jones m...@proclivis.com wrote: I have a general question about thread behavior in 7.8.3 vs 7.6.X I moved from 7.6 to 7.8 and my application behaves very differently. I have three threads, an application thread that plots data with wxhaskell or sends it over a network (depends on settings), a thread doing usb bulk writes, and a thread doing usb bulk reads. Data is moved around with TChan, and TVar is used for coordination. When the application was compiled with 7.6, my stream of usb traffic was smooth. With 7.8, there are lots of delays where nothing seems to be running. These delays are up to 40ms, whereas with 7.6 delays were a 1ms or so. When I add -N2 or -N4, the 7.8 program runs fine. But on 7.6 it runs fine without with -N2/4. The program is compiled -O2 with profiling. The -N2/4 version uses more memory, but in both cases with 7.8 and with 7.6 there is no space leak. I tired to compile and use -ls so I could take a look with threadscope, but the application hangs and writes no data to the file. The CPU fans run wild like it is in an infinite loop. It at least pops an unpainted wxhaskell window, so it got partially running. One of my libraries uses option -fsimpl-tick-factor=200 to get around
