Re: Line-oriented blocking input from sockets?
On Fri, Jul 11, 2003 at 03:04:18PM -0700, Damon Hastings wrote: This is probably a stupid question, but how do you do line-oriented blocking socket reads in Pth? There's a pth_read() which I assume blocks until a specified number of bytes (or eof) are received -- but I'm looking for something like pth_gets() to block until a newline is received. I don't think you could even write a simple webserver in Pth without such a function, though you could of course implement it yourself via a non-blocking pth_read in a loop. I'll implement it myself if need be, though I would rather trust someone else's code, as I've never used Pth before and there are lots of tricky error conditions in socket programming. The documentation at http://www.gnu.org/software/pth/pth-manual.html includes a simple output-only server as a code example -- is there an input/output example somewhere? You may want to look at the mmfd.c/h library which is included into mmftpd and can be found at http://mmondor.gobot.ca/software.html What it does is custom buffering with the supplied functions, in mmftpd case it uses pth_read()/pth_write(), that is pretty efficient... It is released under a BSD-style license. The library comes with an mdoc manual page. Matt __ GNU Portable Threads (Pth)http://www.gnu.org/software/pth/ Development Site http://www.ossp.org/pkg/lib/pth/ Distribution Files ftp://ftp.gnu.org/gnu/pth/ Distribution Snapshots ftp://ftp.ossp.org/pkg/lib/pth/ User Support Mailing List[EMAIL PROTECTED] Automated List Manager (Majordomo) [EMAIL PROTECTED]
pth_malloc()/pth_realloc()/pth_free()
Hi, Has there ever been a plan to eventually include memory allocation primitives as part of the PTh library to minimize the frequency of calls to malloc()/free()? The reason why I am wondering is that kernels will often cause a process to sleep for an undetermined amount of time if memory resources become sparce or if processes need to be swapped out when malloc() uses sbrk() or mmap() to enlarge the process heap. An alternate implementation could call the libc malloc()/free() internally but provide pth_malloc()/pth_free() (and possibly pth_realloc()) using a managed heap, which I beleive could probably enhance performance when many threads need to allocate memory. If such an interface was provided, it also would allow the caller to specify custom malloc()/free() replacements which should be used internally to manage the internal heap, it thus would be possible to even provide a custom pair which uses mmap() with MAP_ANON and mlock(), if it was necessary that the memory never be written to swap, etc. Because those functions would be called rarely, and to allocate relatively large blocks at once, it would work Another area where it would help is on systems which provide poor unefficient implementations in their libc (which aren't too rare). The pth library itself could use pth_malloc() where it currently uses malloc(), etc. Just ideas I thought I would share... Also, if I wrote such a module for pth, following the general programming style, and posted the diff (against the current CVS repository sources) would it have a chance to eventually be included, or would it be considered as useless bloat by the community (or the author)? The implementation would not enlarge the library much. Thanks, Matt __ GNU Portable Threads (Pth)http://www.gnu.org/software/pth/ Development Site http://www.ossp.org/pkg/lib/pth/ Distribution Files ftp://ftp.gnu.org/gnu/pth/ Distribution Snapshots ftp://ftp.ossp.org/pkg/lib/pth/ User Support Mailing List[EMAIL PROTECTED] Automated List Manager (Majordomo) [EMAIL PROTECTED]
Re: PTH is broke or I don't understand threads.
On Tue, Apr 01, 2003 at 12:15:44PM -0800, Steve Alstrin wrote: I compiled the test program, included with distro, and it appears that no thread will start until a join is done on it. When I compile the test program test_pthread with native threads the threads start after the pthread_create call, whereas when compiled with pth threads, each thread only starts when the join for that thread is done, and will not allow the other threads to run until the previous join has completed. Is this a bug, or does all the other thread packages behave incorrectly? Any help would be greatly appreciated, i.e. my job depends on it. I am not 100% sure if I understand the question, but I have the impression that your problem may be related to the non-preemptive nature of PTh... Your applications should either call pth_yield() or pth_*() IO functions (which can internally be called automatically by standard syscalls if PTh was compiled with hard-syscall option) for it to delegate control back to the CPU and therefore allow other threads to execute. (The same applies calling pthread_*() functions rather than the Pth specific pth_*() ones) Hmm also make sure that you are using pth(read)_join() according to POSIX defined semantics (threads should only be joined when the current thread wants to wait until thread termination, and that thread should not have been started using the asynchroneous flag). Otherwise, if the threads were started using the asynchroneous DETACH flag, they are expected to free their resources and exit naturally themselves at pth(read)_exit(), and should not be joined. This behavior is expected for all POSIX pthread_*() functions, no matter the implementation. Hope this helps, Matt __ GNU Portable Threads (Pth)http://www.gnu.org/software/pth/ Development Site http://www.ossp.org/pkg/lib/pth/ Distribution Files ftp://ftp.gnu.org/gnu/pth/ Distribution Snapshots ftp://ftp.ossp.org/pkg/lib/pth/ User Support Mailing List[EMAIL PROTECTED] Automated List Manager (Majordomo) [EMAIL PROTECTED]
Message size and data fields
Hi again,
In the following bit of code, I set the message size field of a message.
If they are internally moved among ports by only relinking the message
node, is this size relevant? I'm sure it would be if we had message
allocation and/or pool management backend for them, but we allocate them
ourselves in the user code... So is this field used for anything? Or is it
only there to possibly permit the library user to perhaps detect the type
of message it was sent by it's size? (Which it could do using other
methods as well... since the thread/task receiving a message is supposed
to know the expected types aleready, as a general rule, which optionally
could comport a user size field)...
Like in this code, the receiver of the message already knows it's size and
won't need to consult the field; Is it safe to not define the size then
(for future implementations as well)? Or should I still do it because Pth
may require it eventually? If it is the case perhaps a function or macro
could be provided to initialize messages so that internal structures do
not have to be modified by the user and API remain consistent among
revisions eg:
void pth_message_init(pth_message_t *, pth_msgport_t, unsigned int, void *);
It would not matter if it was called multiple times on the same message,
it would not clear the message but only reset the size, replyport and
data fields to requested values (which of course can be NULL for
replyport and data)
Or, an API similar to POSIX provides in some circumstances
int pth_message_setattr(pth_message_t *, int, void *);
int pth_message_getattr(void *, pth_message_t *, int);
where int could be MESG_RPORT, MESG_SIZE, MESG_DATA
Alternatively, if the structure will not change among revisions, it could
simply be specified in the man page and publically be exported in it
static bool
start_transfer_thread(clientenv *clenv)
{
/* XXX Which fits best for all future Pth implementations?
* clenv-tmsg.msg.m_size = sizeof(transfermsg);
* clenv-tmsg.msg.m_replyport = clenv-rport;
* clenv-tmsg.msg.m_data = NULL;
*/
/* ||
* pth_message_setattr((clenv-tmsg.msg), PTH_MSGATTR_RPORT,
* clenv-rport);
*/
/* ||
* pth_message_init((clenv-tmsg.msg), clenv-rport,
* sizeof(transfermsg), NULL);
*/
if ((clenv-tthread = pth_spawn(tthreadattr, (void *)transferthread,
clenv))) {
if (transfer_request(REQ_NONE, TRUE, clenv)) return (TRUE);
pth_join(clenv-tthread, NULL);
}
return (FALSE);
}
Hmm also, is there any useage of the data * field in the message
structure or is it left to the user to optionally allow opaque messages?
My message was defined similarly to on AmigaOS, where the pth_message_t
structure only provides the message linking node:
typedef struct transfermsg {
pth_message_t msg;/* Internal message node */
int request; /* Request to send to device */
bool result; /* Result status after device request */
int64_t rbytes, wbytes; /* Number of bytes transfered */
int64_t dlfiles, ulfiles; /* Number of files transfered */
int port; /* PORT or PASV port for next transfer*/
int file; /* File descriptor to read/write from/to*/
int rrate, wrate; /* Transfer dl/ul rate */
char *ipaddr; /* If PORT, address to connect to */
char *path; /* Path to be passed to ls() if list=TRUE*/
bool passive; /* Either next xfer is PORT or PASV mode*/
bool ongoing; /* TRUE when a transfer is in progress*/
char list;/* 0=File xfer, 1=LIST, 2=NLST, 3=empty*/
} transfermsg;
The way I see it, it appears possible with the size and data fields to
process opaque messages. If this is a useful feature, the message
initialization function seems a good idea... It may be useful to add
some notes in the man page as well to clarify this point so that users
don't need to temper internal Pth structures which theoretically could
eventually change in a future implementation. If it will not, the
pth_message_t structure contents could be shown in the man page under
the message ports section, clarifying that it is always safe to
user-set these fields among various Pth revisions...
If the goal is to allow opaque messages, perhaps that a type field
could be introduced (since message size is usually useful but
not appropriate to evaluate a message type).
Thanks again,
Matt
__
GNU Portable Threads (Pth)http://www.gnu.org/software/pth/
User Support Mailing List[EMAIL PROTECTED]
Automated List Manager (Majordomo) [EMAIL PROTECTED]
Re: pth_uctx_* (was: Re: pth_msgport_create() and pth_mctx_*())
On Sun, Nov 03, 2002 at 11:02:53AM +0100, Ralf S. Engelschall wrote: On Thu, Oct 24, 2002, Matthew Mondor wrote: Yes, that is also why I suggest to perhaps provide an SVR4/SUS like interface, which could consist of C functions... and then exported and documented. The API is already well known so it could be a plus... [...] I've now finished this TODO list point, too. Details you can find under http://cvs.ossp.org/chngview?cn=2718. For trying it out fetch the snapshot ftp://ftp.ossp.org/pkg/lib/pth/pth-SNAP-20021103.tar.gz or higher. An example you can find in test_uctx.c. Documentation is in pth.pod. Thank you very much, I will try to test it soon. Matt __ GNU Portable Threads (Pth)http://www.gnu.org/software/pth/ User Support Mailing List[EMAIL PROTECTED] Automated List Manager (Majordomo) [EMAIL PROTECTED]
Re: netbsd and pth threads
On Tue, 22 Oct 2002 16:11:24 -0500 Jeff Rhine [EMAIL PROTECTED] wrote: Trying to compile ntop with netbsd 1.5.x and pth for provide threads support ? You tried using the binary package netbsd provides, or pkgsrc? Note that also pth is provided, which compiles by default with POSIX thread wrapper API. pkgsrc/devel/pth/ pkgsrc/net/ntop/ Matt __ GNU Portable Threads (Pth)http://www.gnu.org/software/pth/ User Support Mailing List[EMAIL PROTECTED] Automated List Manager (Majordomo) [EMAIL PROTECTED]
Re: pth_msgport_create() and pth_mctx_*()
On Wed, 23 Oct 2002 15:58:40 +0200
Ralf S. Engelschall [EMAIL PROTECTED] wrote:
Good idea. The name for pth_msgport_create() is actually only required
of one wants to search the message port via pth_msgport_find(). So I've
changed this now for GNU Pth 1.5.0 to allow a NULL name, too. Thanks for
the hint.
Indeed, and effectively in my case I did not need to locate the port, on
AmigaOS tasks really needed the functionality, although it's of course
optional with user-space threads where memory is already shared among
them. Thanks, I'm happy to see that it's still being maintained actively,
it's a great piece of work.
Yes, the same idea is also on my TODO list. Unfortunately it is perhaps
not as easy as it looks, because some of the pth_mctx_*() stuff is just
macro based and hence exporting it means moving it physically to pth.h
-- perhaps with other side-effects. I'll look into this, perhaps we
could wrap the pth_mctx_*() functionality with extra C function for
easier exporting.
Yes, that is also why I suggest to perhaps provide an SVR4/SUS like
interface, which could consist of C functions... and then exported and
documented. The API is already well known so it could be a plus...
BTW, yesterday night I wrote a quick small test, attempting to still write
a scheduler around current Pth, after I had the idea about using
pth_yield() and specifying the thread to switch to :) unfortunately it
does not seem to work yet. Of course it's only a small hack, but I beleive
it was enough to try it (attached, there may be some extra stuff which was copy-pasted
from an older project to code it faster).
- microxisop.h
#ifndef MICROXISOP_H
#define MICROXISOP_H
#include sys/types.h
#include stdlib.h
#include pth.h
#include mmtypes.h
#include mmlist.h
#define STATE_READY 1
#define STATE_WAIT 2
#define STATE_SUSPENDED 3
#define STATE_DEAD 4
#define xisop_malloc(s) malloc((s))
#define xisop_free(b) free((b))
typedef int xisop_sig_t;
typedef u_int64_t xisop_sigmask_t;
typedef int xisop_lock_t;
struct xisop_context {
/* A simple hack to run Xisop on unix */
pth_t id;
};
struct xisop_task {
node nod;
node usernod;
void *(*start)(void *);
struct xisop_context context;
char state, priority, credits;
xisop_sigmask_t sigalloc, sigwait, sigrecv;
list private_msgports;
};
struct xisop_root {
xisop_lock_t sched_lock;
list *task_pool;
list queue_new, queue_ready, queue_wait, queue_suspended, queue_dead;
list msgports;
list libraries;
list devices;
list handlers;
list volumes;
struct xisop_task *root_task, *current_task;
struct xisop_context context;
u_int64_t ticks;
};
struct xisop_msgport {
node nod;
node usernod;
xisop_lock_t lock;
u_int64_t hash;
struct xisop_task *sigtask;
xisop_sig_t signum;
int maxqueue;
list queue;
};
struct xisop_message {
node nod;
struct xisop_msgport *replyport;
};
struct xisop_timerhook {
node nod;
void (*hook)(void);
};
extern struct xisop_root *Root;
voidxisop_init(void);
/* Task */
struct xisop_task * xisop_gettask(void);
struct xisop_task * xisop_newtask(void *(*)(void *), char);
struct xisop_task * xisop_freetask(struct xisop_task *);
voidxisop_runtask(struct xisop_task *);
voidxisop_deadtask(void);
/* Time */
voidxisop_timer_on(void);
voidxisop_timer_off(void);
voidxisop_timer_setfrequency(u_int32_t);
voidxisop_timer_attach_hook(void (*)(void));
voidxisop_timer_unlink_hook(void (*)(void));
/* Schedule */
voidxisop_forbid(void);
voidxisop_permit(void);
voidxisop_disable(void);
voidxisop_enable(void);
voidxisop_yield(void);
voidxisop_lock_init(xisop_lock_t *);
voidxisop_simplelock(xisop_lock_t *, bool);
/* IPC */
xisop_sig_t xisop_sigalloc(void);
voidxisop_sigfree(xisop_sig_t);
voidxisop_signal(struct xisop_task *, int);
xisop_sigmask_t xisop_wait(xisop_sigmask_t);
struct xisop_msgport * xisop_openport(u_int64_t, size_t);
struct xisop_msgport * xisop_closeport(struct xisop_msgport *);
struct xisop_msgport * xisop_findport(u_int64_t);
boolxisop_sendmsg(struct xisop_msgport *,
struct xisop_message *);
struct xisop_message * xisop_getmsg(struct xisop_msgport *);
boolxisop_replymsg(struct xisop_message *);
voidxisop_waitport(struct xisop_msgport *);
struct xisop_message * xisop_openmsg(struct xisop_msgport *, size_t);
struct xisop_message * xisop_closemsg(struct xisop_message *);
#endif
- microxisop.c
/* This only
pth_msgport_create() and pth_mctx_*()
I really enjoy Pth, and am using it's functions quite extensively in my mmftpd and mmmail servers (http://mmondor.gobot.ca/software.html or cvs -z9 -d:pserver:[EMAIL PROTECTED]/cvsroot co mmsoftware). I especially like it's event handling, which even POSIX threads don't provide. Here are notes of interest on things that I feel could be modified, I possibly could perform the changes myself as long as the community likes the modifications. Of course suggestions are first requested. First, I noticed that pth_msgport_create(), although inspired from the AmigaOS API, does not support NULL for port identifyer, which would be very useful for thread-specific private message ports (mmftpd uses those and unfortunately currently has to generate unique strings to create ports). AmigaOS had this functionality... Another suggestion would be exporting the low-level pth_mctx_*() interface so that applications that only need portable context switching/manipulation features could possibly use it. Those could then be documented as well in the man page... For instance a real-time kernel scheduler implementation I have been working on a while back, could easily be tested on unix in a portable manner. It's unfortunate that the SVR4 API is not part of all unices, and that sigaltstack fiddling is architecture + platform dependant. But Pth has all this for us already, it's just not exported publically. Possibly that an SVR4-inspired public API could be made part of future Pth versions. Matt __ GNU Portable Threads (Pth)http://www.gnu.org/software/pth/ User Support Mailing List[EMAIL PROTECTED] Automated List Manager (Majordomo) [EMAIL PROTECTED]
