Re: [Development] Is QtConcurrent's code generator still in use?
On terça-feira, 20 de novembro de 2012 13.15.47, Sorvig Morten wrote: > > Let me give you an example of a thread pool used differently: the DNS > > resolver code in QtNetwork. It uses 5 threads, regardless of how many > > CPU cores you have. It involves a blocking call. And as such, it must not > > use the global thread pool because it would prevent other tasks from > > running. > > Exactly, there must be a way of separating different types of tasks. > > The interesting thing about Qt is that we "own" a complete API, from > threading to file system to network stack. This means that the concurrency > engine could be informed by the network stack that a worker thread is about > to block on I/O. Indeed, we can do that. But right now, telling QtConcurrent that the thread is going to sleep / block means it will schedule a new thread to continue processing tasks. We don't want that for the DNS resolver: we want to limit the number of queries in parallel for a reason. And even if that were not the case, we wouldn't want to share the global thread pool: even if the user is busy creating thumbnails for an image collection, we still want to send DNS requests to find flickr.com. -- Thiago Macieira - thiago.macieira (AT) intel.com Software Architect - Intel Open Source Technology Center signature.asc Description: This is a digitally signed message part. ___ Development mailing list Development@qt-project.org http://lists.qt-project.org/mailman/listinfo/development
Re: [Development] Is QtConcurrent's code generator still in use?
On Nov 19, 2012, at 4:42 PM, Thiago Macieira wrote: > On segunda-feira, 19 de novembro de 2012 14.03.17, Konstantin Tokarev wrote: >>> - The core of a concurrency engine should be a work-stealing data >>> structure/scheduler. Qt Concurrent has simple work-stealing >>> functionality, but is to tied to the global thread pool. >> Since number of available CPU cores is usually constant over process >> lifetime (let's forget HPC clusters :), there's nothing wrong with global >> thread pool. > > By that, you immediately restrict QtConcurrent's use to CPU-bound tasks. One > can't use it anymore for tasks that involve blocking I/O, for example. > > Let me give you an example of a thread pool used differently: the DNS > resolver > code in QtNetwork. It uses 5 threads, regardless of how many CPU cores you > have. It involves a blocking call. And as such, it must not use the global > thread pool because it would prevent other tasks from running. Exactly, there must be a way of separating different types of tasks. The interesting thing about Qt is that we "own" a complete API, from threading to file system to network stack. This means that the concurrency engine could be informed by the network stack that a worker thread is about to block on I/O. Morten ___ Development mailing list Development@qt-project.org http://lists.qt-project.org/mailman/listinfo/development
Re: [Development] Is QtConcurrent's code generator still in use?
On segunda-feira, 19 de novembro de 2012 14.03.17, Konstantin Tokarev wrote: > > - The core of a concurrency engine should be a work-stealing data > > structure/scheduler. Qt Concurrent has simple work-stealing > > functionality, but is to tied to the global thread pool. > Since number of available CPU cores is usually constant over process > lifetime (let's forget HPC clusters :), there's nothing wrong with global > thread pool. By that, you immediately restrict QtConcurrent's use to CPU-bound tasks. One can't use it anymore for tasks that involve blocking I/O, for example. Let me give you an example of a thread pool used differently: the DNS resolver code in QtNetwork. It uses 5 threads, regardless of how many CPU cores you have. It involves a blocking call. And as such, it must not use the global thread pool because it would prevent other tasks from running. -- Thiago Macieira - thiago.macieira (AT) intel.com Software Architect - Intel Open Source Technology Center signature.asc Description: This is a digitally signed message part. ___ Development mailing list Development@qt-project.org http://lists.qt-project.org/mailman/listinfo/development
Re: [Development] Is QtConcurrent's code generator still in use?
19.11.2012, 10:32, "Sorvig Morten" : > On Nov 14, 2012, at 4:24 PM, Christian Kandeler > wrote: > >> On 11/14/2012 12:17 PM, Sorvig Morten wrote: >>> QtConcurrent is done. The implementation is not good enough to be used as >>> a base for further development. >> Can you be a bit more specific? What are the general problems and why >> can't they be easily solved? > > Off the top of my head: > > - It uses a code generator which falls out of sync (there were no variadic > templates at the time). There are no variadic templates now either, unless you are using C++11. > - Unclear separation between inline template code and non-template library > code. It's hard to say where the binary compatible interface is. Make it all template/inline and don't bother. > - To much template code, causing bloat. There's space/speed trade-off here. > - The core of a concurrency engine should be a work-stealing data > structure/scheduler. Qt Concurrent has simple work-stealing functionality, > but is to tied to the global thread pool. Since number of available CPU cores is usually constant over process lifetime (let's forget HPC clusters :), there's nothing wrong with global thread pool. -- Regards, Konstantin ___ Development mailing list Development@qt-project.org http://lists.qt-project.org/mailman/listinfo/development
Re: [Development] Is QtConcurrent's code generator still in use?
On Nov 14, 2012, at 4:24 PM, Christian Kandeler wrote: > On 11/14/2012 12:17 PM, Sorvig Morten wrote: >> QtConcurrent is done. The implementation is not good enough to be used as a >> base for further development. > > Can you be a bit more specific? What are the general problems and why > can't they be easily solved? Off the top of my head: - It uses a code generator which falls out of sync (there were no variadic templates at the time). - Unclear separation between inline template code and non-template library code. It's hard to say where the binary compatible interface is. - To much template code, causing bloat. - The core of a concurrency engine should be a work-stealing data structure/scheduler. Qt Concurrent has simple work-stealing functionality, but is to tied to the global thread pool. The API works fine for what it does, but if we want to improve it it looks more like a rewrite rather than incremental patching to me. Morten ___ Development mailing list Development@qt-project.org http://lists.qt-project.org/mailman/listinfo/development
Re: [Development] Is QtConcurrent's code generator still in use?
On 2012-11-14, Christian Kandeler wrote: > On 11/14/2012 12:17 PM, Sorvig Morten wrote: >> QtConcurrent is done. The implementation is not good enough to be used as a >> base for further development. > > Can you be a bit more specific? What are the general problems and why > can't they be easily solved? I think it is quite limited and hard to use. I recently tried and after giving up on that, I switched a project to the ThreadWeaver library by KDE which not only made my code simpler, it also made it much easier to understand and having threading cote separated out. The ThreadWeaver library is a quite simple, but yet powerful thing built on qtcore. /Sune ___ Development mailing list Development@qt-project.org http://lists.qt-project.org/mailman/listinfo/development
Re: [Development] Is QtConcurrent's code generator still in use?
On 11/14/2012 12:17 PM, Sorvig Morten wrote: > QtConcurrent is done. The implementation is not good enough to be used as a > base for further development. Can you be a bit more specific? What are the general problems and why can't they be easily solved? Christian ___ Development mailing list Development@qt-project.org http://lists.qt-project.org/mailman/listinfo/development
Re: [Development] Is QtConcurrent's code generator still in use?
On Nov 11, 2012, at 6:07 AM, Sze Howe Koh wrote: > > Thanks for the explanation. I've opened tickets for this: > https://bugreports.qt-project.org/browse/QTBUG-27940 > https://bugreports.qt-project.org/browse/QTBUG-27941 > > I asked because I'm working on patches for Qt Concurrent now; I don't > know all the updates that the tool needs, but I'll patch it with my > changes. > I think what QtConcurrent really needs is a plan. What do we want it to be? There are several options, and I'd like to offer the "null" option: QtConcurrent is done. The implementation is not good enough to be used as a base for further development. We should limit ourselves to fixing bugs and making small improvements where we can. This is in line with moving QtConcurrent out of Qt Core. It also has the advantage of not taking much developer time. - Morten ___ Development mailing list Development@qt-project.org http://lists.qt-project.org/mailman/listinfo/development
Re: [Development] Is QtConcurrent's code generator still in use?
On 10 November 2012 18:34, Olivier Goffart wrote: > On Saturday 10 November 2012 10:09:42 Sze Howe Koh wrote: >> I found a code generator in qttools/src/qtconcurrent/generaterun, >> which is supposedly the tool for generating >> qtbase/src/concurrent/qtconcurrentrun.h >> qtbase/src/concurrent/qtconcurrentstoredfunctioncall.h >> >> However, the code generator seems to have fallen out of sync with its >> targets. It produces lines like "#include >> ", which is reminiscent of the old Qt >> 4.8 structure. >> >> Should it be updated, or is it no longer used? > > Hi, > > It seems like none of the people that have maintained QtConcurrent recently or > applied or reviewed patch to those files, including me, knew about this tool. > (My fault since there is a even comment in the files telling not to edit which > I must have missed) > > There are a few commit even in the 4.x series that are not included in that > generator. > > > Anyway... First, the location: It should probably not have been in the qttools > repository, but rather in the qtrepotools. > And yes, it probably would make send to update the tool to take in account all > the commits since at least the time the repository went public around Qt 4.5. Thanks for the explanation. I've opened tickets for this: https://bugreports.qt-project.org/browse/QTBUG-27940 https://bugreports.qt-project.org/browse/QTBUG-27941 I asked because I'm working on patches for Qt Concurrent now; I don't know all the updates that the tool needs, but I'll patch it with my changes. Regards, Sze-Howe ___ Development mailing list Development@qt-project.org http://lists.qt-project.org/mailman/listinfo/development
Re: [Development] Is QtConcurrent's code generator still in use?
On Saturday 10 November 2012 10:09:42 Sze Howe Koh wrote: > I found a code generator in qttools/src/qtconcurrent/generaterun, > which is supposedly the tool for generating > qtbase/src/concurrent/qtconcurrentrun.h > qtbase/src/concurrent/qtconcurrentstoredfunctioncall.h > > However, the code generator seems to have fallen out of sync with its > targets. It produces lines like "#include > ", which is reminiscent of the old Qt > 4.8 structure. > > Should it be updated, or is it no longer used? Hi, It seems like none of the people that have maintained QtConcurrent recently or applied or reviewed patch to those files, including me, knew about this tool. (My fault since there is a even comment in the files telling not to edit which I must have missed) There are a few commit even in the 4.x series that are not included in that generator. Anyway... First, the location: It should probably not have been in the qttools repository, but rather in the qtrepotools. And yes, it probably would make send to update the tool to take in account all the commits since at least the time the repository went public around Qt 4.5. -- Olivier Woboq - Qt services and support - http://woboq.com ___ Development mailing list Development@qt-project.org http://lists.qt-project.org/mailman/listinfo/development
[Development] Is QtConcurrent's code generator still in use?
I found a code generator in qttools/src/qtconcurrent/generaterun, which is supposedly the tool for generating qtbase/src/concurrent/qtconcurrentrun.h qtbase/src/concurrent/qtconcurrentstoredfunctioncall.h However, the code generator seems to have fallen out of sync with its targets. It produces lines like "#include ", which is reminiscent of the old Qt 4.8 structure. Should it be updated, or is it no longer used? Regards, Sze-Howe ___ Development mailing list Development@qt-project.org http://lists.qt-project.org/mailman/listinfo/development
