The high-level GIO API and threading

[Ayo]

Hello devs,

I've been browsing the GIO and glib-networking source code for a
while, and I'd like to get some clarification on the usage of the
GIOStream API with respect to threading. It was my understanding that
the API allows running a g_input_stream_read() in one thread while
doing a g_output_stream_write() in another - much like regular socket
programming. Building upon this idea, g_input_stream_read_async() and
g_output_stream_write_async() are implemented by pushing a job to a
GThreadPool and doing the blocking _read() and _write() there. Thus
allowing one to read and write from the same GIOStream in an event
based manner, by calling the following functions in a single go:

GSocketConnection *n; // obtained this from a GSocketClient or GSocketListener
GInputStream *in = g_io_stream_get_input_stream(G_IO_STREAM(n));
GOutputStream *out = g_io_stream_get_output_stream(G_IO_STREAM(n));
g_input_stream_read_async(in, ..);
g_output_stream_write_async(out, ..);

...at least, I thought the API supported that. Browsing through the
code of the gnutls implementation in glib-networking, I see a single
internal state being used for both the read and write functions, with
apparently no synchronisation whatsoever. Am I misinterpreting the
code, or was I assuming things wrongly?

Ayo.
___
gtk-devel-list mailing list
gtk-devel-list@gnome.org
http://mail.gnome.org/mailman/listinfo/gtk-devel-list

[GIO] Using GBufferedOutputStream

[Ayo]

Hello,

I am a bit at a loss as to how GBufferedOutputStream is supposed to be
used. My aim is to have a simple high-level non-blocking
send_message() function that buffers a new message and flushes the
data on the background. A naive method of doing so could be the
following (partially-pseudo) code (assuming I understood the docs
right):


GOutputStream *out;

void on_connect() {
  out = 
g_buffered_output_stream_new(g_io_stream_get_output_stream(G_IO_STREAM(connection)));
  g_buffered_output_stream_set_auto_grow(G_BUFFERED_OUTPUT_STREAM(out), TRUE);
}

void handle_flush(..) {
  // close connection on error
}

void send_message(char *msg) {
  // this write should neither block nor fail, since we should be
writing to a buffer and auto-grow is true
  g_output_stream_write(out, msg, ..);
  g_output_stream_flush_async(out, .., handle_flush, ..);
}


The above method assumes that send_message() is not called before the
message of a previous call has been flushed. Otherwise the later flush
would, I assume, fail because there is a pending operation on the
stream. Now suppose I would want to handle this situation and allow
send_message() to be called at any time:


void handle_flush(..) {
  if(error)
report_error_and_close_connection()
  else if(buffer_is_not_empty())
g_output_stream_flush_async(out, .., handle_flush, ..);
}

void send_message(char *msg) {
  // this write should neither block nor fail, since we should be
writing to a buffer and auto-grow is true
  g_output_stream_write(out, msg, ..);
  if(!g_output_stream_has_pending(out))
g_output_stream_flush_async(out, .., handle_flush, ..);
}


However, there are two issues with the above solution:
- GBufferedOutputStream does not provide any method to check whether
the buffer is empty, or whether there are any outstanding bytes that
need to be flushed. This is required in handle_flush() to determine
whether a new flush is required.
- Will GBufferedOutputStream correctly handle the situation where new
data is added to the buffer while it is performing a flush operation
on the background? I've skimmed a bit through the code, but get the
impression it is not supposed to be used like this.

I also just encountered a problem with g_output_stream_write(): in
some situations, it reports to have written less bytes than I asked it
to. I had assumed this to be impossible when writing to an
auto-growing buffer. I have not investigated this problem any furher
yet, though, and have no idea whether it is related to the above
problem.

Anyway, I have the impression that I'm trying to use
GBufferedOutputStream in a way it is not supposed to be used, yet I
don't think I'm trying to do anything obscure. I could have
send_message() queue messages in a separate queue and write() them in
the background from handle_flush(), but that is rather inefficient
(many of the messages are pretty small and could be combined in a
single write()) and would probably require more code than simply
implementing my own buffer.

I keep wondering if there is no better solution to do this, and how
GBufferedOutputStream *is* supposed to be used...


Ayo.
___
gtk-app-devel-list mailing list
gtk-app-devel-list@gnome.org
http://mail.gnome.org/mailman/listinfo/gtk-app-devel-list

Re: [GIO] Using GBufferedOutputStream

[Ayo]

(Replying to myself)

On 7 April 2011 12:24, Ayo a...@blicky.net wrote:
 [snip]
 - Will GBufferedOutputStream correctly handle the situation where new
 data is added to the buffer while it is performing a flush operation
 on the background? I've skimmed a bit through the code, but get the
 impression it is not supposed to be used like this.

Learned what happens after some experimentation: the
g_output_stream_write() will fail with Stream has outstanding
operation, thus not even allowing new writes to the buffer while it
is flushing. This confirms that I'm using it in a way it is not
supposed to be used.

 I also just encountered a problem with g_output_stream_write(): in
 some situations, it reports to have written less bytes than I asked it
 to. I had assumed this to be impossible when writing to an
 auto-growing buffer. I have not investigated this problem any furher
 yet, though, and have no idea whether it is related to the above
 problem.

Also solved this mistery. This was caused by a bug in my code and
_write() was called with an already unref()'ed GOutputStream.

 [snip]

 I keep wondering if there is no better solution to do this, and how
 GBufferedOutputStream *is* supposed to be used...

While I'm still wondering how (or more importantly: when)
GBufferedOutputStream is supposed to be used, it apparently isn't what
I was looking for. Implementing my own buffer seems like the best
option...

On a related note: why are all these high-level I/O modules using
threads for async operations, rather than using the main loop for
polling the underlying filehandles (where possible). That would be
both more efficient and allows better cancellation. I'm inclined to
fall back to plain old socket programming with all these levels of
indirection, added complexity and inefficiencies. :-(

Ayo
___
gtk-app-devel-list mailing list
gtk-app-devel-list@gnome.org
http://mail.gnome.org/mailman/listinfo/gtk-app-devel-list