Re: [Interest] QProcess unbuffered

[Björn Schäpers]

Am 07.03.2023 um 23:39 schrieb Thiago Macieira:

On Tuesday, 7 March 2023 14:23:00 PST Björn Schäpers wrote:

  From the code I reviewed back then (and remember it now) it unpacks the
handles directly to a HANDLE, which is pointer size (although it only uses
the lower 32 bit). But Scott will tell us what the field width has to be
for a 64 bit process.

Yup.

BTW, Scott, when using QProcess::setCreateProcessArgumentsModifier(), you get
access to QProcess' standard handles so you can place them in your fd table.
Björn's code probably predates that function (Qt 5.7).


It's actually code which doesn't use Qt at all. ;) I'm sadly far away from 
getting Qt into my main work projects.

___
Interest mailing list
Interest@qt-project.org
https://lists.qt-project.org/listinfo/interest

Re: [Interest] QProcess unbuffered

[Björn Schäpers]

Am 08.03.2023 um 03:18 schrieb Scott Bloom:

I really want to thank Björn

With his starting point, I was able to tweak his changes, and seem to have it 
working  I have more testing to do. But it was actually pretty straight 
forward.

* I use VS (2022) so the packing was different
__attribute__((packed)) became  __pragma( pack( push, 1 ) ) struct 
HackedHandlePasser
* The handles are 64 bit in the existing Startup Info, so I kept them as such, 
which changed the definition of HANDLE32
  using HANDLE32 = std::int32_t; was removed, and the standard HANDLE 
defined in windows.h was used
* The static assert was updated to reflect this
* I copy all 3 handles from the existing startupinfo
std::memcpy([0], , 4);
std::memcpy([1], , 4);
std::memcpy([2], , 4);

became

 std::memcpy( [ 0 ], >startupInfo->hStdInput, 
sizeof( HANDLE ) );
 std::memcpy( [ 1 ], >startupInfo->hStdOutput, 
sizeof( HANDLE ) );
 std::memcpy( [ 2 ], >startupInfo->hStdError, 
sizeof( HANDLE ) );

Then I wrapped this into a function that returns a 
QProcess::QCreateProcessArgumentModifier.

And finally in my process, I set the modifier using process-> 
setCreateProcessArgumentsModifier( func );

You can find the code
https://github.com/towel42-com/SABUtils/blob/trunk/ForceUnbufferedProcessModifier.h
and
https://github.com/towel42-com/SABUtils/blob/trunk/ForceUnbufferedProcessModifier.cpp

and its usage at
https://github.com/towel42-com/MediaManager/blob/trunk/Models/DirModel.cpp

Thanks again.
Scott



Happy for you.
I assume you don't use write() to the process, right? Because I think if you 
want to do that, you'd have to set FOPEN for the hStdInput handle.

___
Interest mailing list
Interest@qt-project.org
https://lists.qt-project.org/listinfo/interest

Re: [Interest] QProcess unbuffered

[Björn Schäpers]

That's right. It depends on the runtime, but I have not seen a process which 
uses not a Microsoft provided runtime (of course I didn't check all the programs 
I used).


And that's all I've found. It serves my purposes to the fullest.

From the code I reviewed back then (and remember it now) it unpacks the handles 
directly to a HANDLE, which is pointer size (although it only uses the lower 32 
bit). But Scott will tell us what the field width has to be for a 64 bit process.


Am 07.03.2023 um 23:06 schrieb Thiago Macieira:

[quoting out of order]

On Tuesday, 7 March 2023 12:57:59 PST Björn Schäpers wrote:

startInf.cbReserved2 = sizeof(HackedHandlePasser);
startInf.lpReserved2 = reinterpret_cast();

These reserved fields are how the runtimes "pass file descriptors" to child
processes. This mimics the Unix fork() behaviour that the parent's open file
descriptors are available in the child, without implementing the full fork()
behaviour. The fields are undocumented (AFAIK) but since both msvcrt and ucrt
depend on the specific behaviours, they can't change.

That includes these bit fields for flags:


#ifndef FOPEN
#define FOPEN 0x01
#endif
#ifndef FDEV
#define FDEV 0x40
#endif

handles.FlagsPerHandle[0] = 0;

std::memset([1], FOPEN | FDEV, 2);

According to the UCRT source code, the FDEV flag controls whether the standard
streams will be buffered or not. The UCRT implementation of isatty() is a check
to see if this flag is set.

The UCRT source code isn't complete (ucrtbase isn't included AFAICS), so I
couldn't find all details when trying to give you a more specific answer.


* I have the HANDLE32 because I start a 32 bit application from within a 64
bit, as far as I understood the structure needs the right HANDLE size, thus
for a 64 bit application it should be std:int64_t, but since I don't need
that I never tested it.

The parent process wouldn't know whether the child is 32-bit or not, so I
imagine the size must be fixed at 32 bits in order to be inheritable. If you
have MSVC, you have access to the UCRT source code too and you can search for
those two reserved fields to see if they hardcode to 32-bit or pointer sizes.


So when using it with QProcess I think one should copy the handles from the
STARTUPINFOW structure into this struct, then I think the normal QIODevice
interface should keep working. If one should clear the STARTF_USESTDHANDLES
flag I also don't know. I stopped my experiments when I achieved success
for my "simple" use case.

QProcess uses CreateProcessW directly, not the _spawnv* CRT family of
functions, which is why file descriptors aren't usually inherited via QProcess.
I don't know either whether the STARTF_USESTDHANDLES flag must be cleared
because they're handled at separate times.

If this flag is passed, then the three handles in STARTUPINFO[1] are passed to
the child as its default handles, by the Win32 call itself. That means those
handles are installed in the child before the runtime gets a chance to
initialise and read the file descriptor table passed in those reserved fields.
Whether the runtime overwrites the standard file descriptors or not, I couldn't
find in the UCRT sources.

And for the same reason, I can't tell either how the FDEV flag is set for the
standard file descriptors from outside the file descriptor table.

Moreover, please note that this is *runtime* -driven behaviour, not OS-
mandated. So it's entirely possible that this executable that Scott wants to
run operates differently because it uses a weird runtime. (Strictly speaking,
it's the same on Unix systems, but all libcs have the same behaviour)

[1] 
https://learn.microsoft.com/en-us/windows/win32/api/processthreadsapi/ns-processthreadsapi-startupinfoa

___
Interest mailing list
Interest@qt-project.org
https://lists.qt-project.org/listinfo/interest


___
Interest mailing list
Interest@qt-project.org
https://lists.qt-project.org/listinfo/interest

Re: [Interest] QProcess unbuffered

[Björn Schäpers]

So here is what I have:

struct __attribute__((packed)) HackedHandlePasser {
 using HANDLE32 = std::int32_t;

 DWORD    NumberOfHandles = 3; // 4 Byte
 BYTE FlagsPerHandle[3];   // 3 * 1 Byte
 HANDLE32 Handles[3];  // 3 * 4 Byte
  };

  static_assert(sizeof(HackedHandlePasser) == 19);
  HackedHandlePasser handles;

  #ifndef FOPEN
#define FOPEN 0x01
  #endif
  #ifndef FDEV
#define FDEV 0x40
  #endif

  handles.FlagsPerHandle[0] = 0;
  std::memset([1], FOPEN | FDEV, 2);

  const HANDLE invalid = INVALID_HANDLE_VALUE;
  std::memcpy([0], , 4);
  std::memcpy([1], , 4);
  std::memcpy([2], , 4);

  startInf.cbReserved2 = sizeof(HackedHandlePasser);
  startInf.lpReserved2 = reinterpret_cast();

Some explanations:
* I have the HANDLE32 because I start a 32 bit application from within a 64 bit, 
as far as I understood the structure needs the right HANDLE size, thus for a 64 
bit application it should be std:int64_t, but since I don't need that I never 
tested it.

* In cbReserved2 there is the size of the data in lpReserved2.
* The data in lpReserved2 starts with 2 bytes which indicate how many handles 
are passed to the spawned process (they may be INVALID_HANDLE_VALUE). Followed 
by one byte per handle with the opened flags - and here comes the trick: If it 
has the flag FDEV the ms runtime printf assumes it writes to a console and thus 
flushes. And then followed by the handles.
* My use case needs only stdout (and I also open stderr, in my case to the same 
handle), these are the handles number 1 and 2. Number 0 would be stdin.


So when using it with QProcess I think one should copy the handles from the 
STARTUPINFOW structure into this struct, then I think the normal QIODevice 
interface should keep working. If one should clear the STARTF_USESTDHANDLES flag 
I also don't know. I stopped my experiments when I achieved success for my 
"simple" use case.


Let me know what works for you.

And last but not least some sources:
https://stackoverflow.com/questions/40965496/windows-how-to-stop-buffering-of-redirected-stdout-using-createprocess 
which apparently now also has a C/C++ answer, when I needed it there was only 
the python code.
http://www.catch22.net/tuts/undocumented-createprocess does use the fields for 
some other stuff, but explains a bit of background.
https://github.com/cansou/msvcrt/blob/master/src/isatty.c One if the involved 
files to handle the input.


Kind Regards,
Björn.
___
Interest mailing list
Interest@qt-project.org
https://lists.qt-project.org/listinfo/interest

Re: [Interest] QProcess unbuffered

[Björn Schäpers]

Am 06.03.2023 um 02:11 schrieb Scott Bloom:


I have an external executable (mkvalidator from 
https://www.matroska.org/downloads/mkvalidator.html


It never flushes the output, and it uses linefeed without carriage returns to 
overwrite existing text on the output.



The problem is, when I run it via QProcess (on windows, Qt 5.15.10), I get 
zero output until the process ends.  As if it was buffered, and the output was 
finally flushed


When running from the command line, I do get output continuously during the 
processing.


I setup a timer to waitForRead, didn’t do anything. I cant seem to force the 
QProcess to read all stdout or stderror info.


Any thoughts?


TIA

Scott


___
Interest mailing list
Interest@qt-project.org
https://lists.qt-project.org/listinfo/interest

Hi,

I had the exact same problem (but without QProcess). You have to use a trick to 
disable the buffering, I don't know if its possible with QProcess, you will have 
to use QProcess::CreateProcessArgumentModifier or most likely CreateProcess 
directly.


If there is interest I can copy some of my data, but currently I can't access 
it.

Kind regards,
Björn.___
Interest mailing list
Interest@qt-project.org
https://lists.qt-project.org/listinfo/interest

Re: [Interest] spurious breakpoints in Qt Creator

[Björn Schäpers]

Am 16.05.2022 um 09:09 schrieb Elvis Stansvik:
Den sön 15 maj 2022 21:45Matej Košík > skrev:


Hi,

While using Qt Creator, I noticed that it generates "spurious breakpoints".

How to reproduce:
- I start Qt Creator
- I generate a new project
  (Qt Widgets Application with QMainWindow as a base class)
- from the menu I choose
  - Debug
    - Start Debugging
      - Start debugging of startup project

then the debugged project first stops in "gtk3/main.cpp" and then stops
in other places.

When I let the program continue, it then stops again in the file
"ico/main.cpp".

When I let the program continue, it then stops again in the file
"pdf/main.cpp".

I guess that I am doing something wrong, but I am unable to figure out
what. Any help is welcome. I would like the program to stop only in
breakpoints defined by me.

I am using:
- Linux (version 5.10.0-14-amd64)
- GNOME (version 3.38.5)
- Qt Creator (4.14.1)


Not related to your problem, but would highly recommend using latest version 
of Qt Creator.


Elvis

- Qt (version 6.3 compiled from source)
___
Interest mailing list
Interest@qt-project.org 
https://lists.qt-project.org/listinfo/interest


___
Interest mailing list
Interest@qt-project.org
https://lists.qt-project.org/listinfo/interest

And maybe post on the creator list.
___
Interest mailing list
Interest@qt-project.org
https://lists.qt-project.org/listinfo/interest

Re: [Interest] Windows: "loadlibrary failed with error 87" on a configuration with a fake video card?

[Björn Schäpers]

If you can use the Process Monitor from SysInternals you can also see which 
library should be loaded.


But I don't know it that has an command line interface.

Am 18.03.2022 um 05:19 schrieb Alexander Dyagilev:

Yes. But what I meant is that it has to be triggered by Qt somehow.

As a first attempt I asked the user to set QT_OPENGL env var to "software". 
This did not help. So it's probably not related to OpenGL.


Anyway, thanks to Thiago Macieira, I asked the user to send me the dump file. 
Let's see. :)



On 3/18/2022 6:52 AM, Henry Skoglund wrote:

On 2022-03-18 04:41, Thiago Macieira wrote:

On Thursday, 17 March 2022 19:50:45 PDT Alexander Dyagilev wrote:

This error message is shown by Windows OS when it's failed to load a
library the process sayed it to load.

I.e. Qt calls LoadLibrary (directly, or by loading some other modules
which calls it to load another modules they depends on).

Short googling tells that it's an graphics card driver related issue
(e.g. https://github.com/spyder-ide/spyder/issues/10084).

But, maybe there is some known work around for this...

If you can generate a backtrace from the application at the moment that dialog
is shown, it can help pinpoint what was being loaded and where from. There are
too many LoadLibrary in the source code to guess which one has incorrect
parameters.

Hi, note: it's not certain that the LoadLibrary error is from Qt, for example 
a similar error:
"Load Library failed with error 126: The specified module could not be 
found." was found to be from within the bowels of Windows itself (non-Qt apps 
also got this error).
More here about error 126: 
https://forum.qt.io/topic/132880/qt-creator-error-126-running-on-windows-10-via-rdp


___
Interest mailing list
Interest@qt-project.org
https://lists.qt-project.org/listinfo/interest

___
Interest mailing list
Interest@qt-project.org
https://lists.qt-project.org/listinfo/interest


___
Interest mailing list
Interest@qt-project.org
https://lists.qt-project.org/listinfo/interest

Re: [Interest] Proxied view not updating live when source model does

[Björn Schäpers]

You emit dataChanged with the given role (which is Qt::EditRole), but not for 
Qt::DisplayRole. Is it that?


Regards,
Björn.

Am 20.09.2021 um 19:15 schrieb Murphy, Sean:

I'm having an issue where my proxy model and its associated view aren't 
updating live when changes
are made in the source model.

My proxy model simply is attempting to invert the rows of the source model. So 
it inherits from a
QSortFilterProxyModel and implements mapFromSource() and mapToSource() as 
follows:

 QModelIndex invertRowsModel::mapFromSource(const QModelIndex ) 
const
 {
 if(sourceIndex.isValid() && rowCount() > 0)
 {
 return this->index(rowCount() - sourceIndex.row() - 1, 
sourceIndex.column());
 }

 return QModelIndex();
 }

 QModelIndex invertRowsModel::mapToSource(const QModelIndex ) 
const
 {
 if(proxyIndex.isValid() && sourceModel() && rowCount() > 0)
 {
 return sourceModel()->index(rowCount() - proxyIndex.row() - 1, 
proxyIndex.column());
 }

 return QModelIndex();
 }

I load up my source model with some data, connect the models up to their 
respective views,
connect the source model to the proxy and launch the application. The original 
views are
correct - the proxied rows are flipped when compared to the source model. But 
then if the
user makes a change to some of the data in the source model's view, the 
corresponding index
in proxy model's view is NOT updated automatically. HOWEVER, if the user then 
clicks within
the proxy view, or clicks outside of my application, the proxy model will then 
repaint and the
updated information is correctly shown.

I've attached a minimal example that shows the behavior. Things to note:
1. When the source model emits its dataChanged(), this seems to trigger a 
dataChanged()
   emission in the proxy model as well. The weird (at least to me) thing is 
that both of those
   signals have the SAME model index row & column numbers. I would have 
expected that
   when the source model emits data changed, that the proxy model would 
translate the
   source model's indices into the proper proxy model's indices via the 
mapFromSource()
   function and then emit dataChanged() with those indices. I've got debug 
statements in
   my example to illustrate that no dataChanged() translation is happening - if 
the user
   updates data in row 0 in the source model's view, the proxy model also emits 
that data
   changed in its row 0, but that isn't the correct row for the proxy.
2. We discovered that if we connect the proxy model's dataChanged() signal to a 
custom
   slot within the proxy model, and have that slot emit layoutChanged() then we 
can get
   the live update behavior when the source model updates. This is at line 25 of
   invertrowsmodel.cpp. It's commented out initially, but if you uncomment it 
and re-run
   the app then proxy view does update instantly. Other similar signals would 
work as well -
   we just need something triggers the proxy view to refresh the data in the 
proxied view,
   but we don't think we should really have to do this...
3. I have a second view attached to the source model and both source model 
views update
   each other live - changes in one source model view are instantly updated in 
the other
   source model view, but not in the proxy model view without further user 
interaction
   (or by emitting that layoutChanged() signal)

Any thoughts as to why the proxy model isn't updating correctly?
Sean

___
Interest mailing list
Interest@qt-project.org
https://lists.qt-project.org/listinfo/interest



___
Interest mailing list
Interest@qt-project.org
https://lists.qt-project.org/listinfo/interest

Re: [Interest] Difficulty running Timer from QThread

[Björn Schäpers]

Am 20.07.2021 um 06:19 schrieb Thiago Macieira:

On Monday, 19 July 2021 11:51:30 PDT Björn Schäpers wrote:

Yes, and the Timer was created inside the VLCWorker::initVLC, and said
worker was moved, but before initVLC, but initVLC was called in the moved
from thread.

But the QTimer had no parent. From the OP:

rtspStartTimer=new QTimer;
rtspStartTimer->setSingleShot(true);
connect(rtspStartTimer, ::timeout,
 this, ::rtspStarting);

That means this QTimer wasn't moved and therefore was firing on the thread
that created the object. Whichever that was.

Last response. Yes that's why I advised to parent it, and THEN move to the other 
thread. The Timer would have been moved with its parent.


___
Interest mailing list
Interest@qt-project.org
https://lists.qt-project.org/listinfo/interest

Re: [Interest] Difficulty running Timer from QThread

[Björn Schäpers]

Am 19.07.2021 um 17:57 schrieb Thiago Macieira:

On Sunday, 18 July 2021 23:08:13 PDT Björn Schäpers wrote:

you should parent your QTimer, and moveToThread afterwards.

That shouldn't be "AND". That should be an exclusive OR: you either parent
your QObject or you moveToThread, not both. You cannot move a QObject that has
a parent to another thread. You can only move the entire hierarchy.

Yes, and the Timer was created inside the VLCWorker::initVLC, and said worker 
was moved, but before initVLC, but initVLC was called in the moved from thread.


___
Interest mailing list
Interest@qt-project.org
https://lists.qt-project.org/listinfo/interest

Re: [Interest] Difficulty running Timer from QThread

[Björn Schäpers]

Hi,

you should parent your QTimer, and moveToThread afterwards. Then the Timer will 
also be moved to the thread, otherwise it remains whereever you created the worker.


Regards,
Björn.

Am 19.07.2021 um 05:29 schrieb Israel Brewster:
Yes, this is a FAQ, but I can’t seem to figure out what I am doing wrong. I am 
using the worker pattern of QThread, (as opposed to subclassing), running Qt 
5.15.2. In my controller’s constructor, I have the following code to 
initialize a worker:


worker=newVLCWorker();
worker->moveToThread();
worker->initVLC(videoWin);
vlcWorkerThread.start();

The initVLC function, among other things, creates a timer and connects it to a 
slot in the worker:


rtspStartTimer=newQTimer;
rtspStartTimer->setSingleShot(true);
connect(rtspStartTimer,::timeout,
this,::rtspStarting);

Later, at some point during program execution, I try to start this timer from 
a function in the worker (which itself was triggered by a signal from the main 
thread):


qDebug()<<"*Tryingtostarttimerwhichlivesinthread"<<(long)rtspStartTimer->thread()<<"fromthread"<<(long)this->thread();
rtspStartTimer->start(250);
qDebug()<<"*Timerattemptcomplete";

Unfortunately, this doesn’t work, giving me the following output:

Debug: 2021-07-18T19:26:12.009 - *Trying to start timer which lives in 
thread 140235980165688  from thread  140235980165688 
(../DoorBellCamC/vlccontroller.cpp:121, void VLCWorker::rtspStarting())
Warning: 2021-07-18T19:26:12.009 - QObject::startTimer: Timers cannot be 
started from another thread (:0, )
Debug: 2021-07-18T19:26:12.009 - *Timer attempt complete 
(../DoorBellCamC/vlccontroller.cpp:123, void VLCWorker::rtspStarting())


So even though the output of the debug confirms that the timer lives in the 
same thread I am trying to call start from, I still get the error about 
“timers cannot be started from another thread”. What am I doing wrong?

---
Israel Brewster
Software Engineer
Alaska Volcano Observatory
Geophysical Institute - UAF
2156 Koyukuk Drive
Fairbanks AK 99775-7320
Work: 907-474-5172
cell:  907-328-9145


___
Interest mailing list
Interest@qt-project.org
https://lists.qt-project.org/listinfo/interest



___
Interest mailing list
Interest@qt-project.org
https://lists.qt-project.org/listinfo/interest

Re: [Interest] Windows installer for Qt programs

[Björn Schäpers]

Am 21.04.2020 um 09:19 schrieb "Alexander Carôt":

Hello all,

I consider shipping my Qt built software with a conventional installer on 
Windows.

Can anyone give me a recommendation of a solid freeware for this purpose ?

Thanks in advance,
best

Alex

We use Innosetup at Work for that. I've not used any other tools, but when I 
started I made the migration from Innosetup 4 to 5 and did some online 
comparisons beforehand. I came to the conclusion Innosetup was still the best 
choice.


Regards,
Björn.

___
Interest mailing list
Interest@qt-project.org
https://lists.qt-project.org/listinfo/interest

Re: [Interest] Example for: QTreeView + QAbstractItemModel + external data

[Björn Schäpers]

Am 26.04.2016 um 19:05 schrieb anton:

Hi,

I have subclassed QAbstractItemModel,
and I want to load additionally data in the model
as soon as somebody clicks on the [+]
in the treeview to load the children of this
item form an external data ( web- app which gives me json data back).

Now:
  I reimplemented hasChildren() in the model
but i am not sure about the moment to load the data.

I suppose that the rowCount() method of the model is called
when somebody expands an item in the view?
If not what is the moment when my model should ask for additional Data?

Does there exist a small example which demonstrates how to
handle this in an optimal way.
Not sure about beginInsertRows() or layoutChanged() ..
I am experimenting but i get also some crashes so an
example project would be nice.

Thanks

Anton
I think you are looking for QAbstractItemModel::canFetchMore() and fetchMore(). 
See also the last paragraph in http://doc.qt.io/qt-5/qabstractitemmodel.html


To create models that populate incrementally, you can reimplement fetchMore() 
and canFetchMore(). If the reimplementation of fetchMore() adds rows to the 
model, beginInsertRows() and endInsertRows() must be called.

___
Interest mailing list
Interest@qt-project.org
http://lists.qt-project.org/mailman/listinfo/interest