Hi Brandon -
Replies to things below, inline.
On 12/02/2013 07:42 PM, Brandon Ross wrote:
> Hmm, what if getaddrinfo was used directly as an extern proc in the Net
> module code instead of through a wrapper in sys.c? Though that would be
> an issue on systems without a getaddrinfo. We'd be able to handle that
> if it was possible to have conditional compilation of Chapel code
> depending on the OS type. Is there a pragma for that?
Hmm. Given the current situation - that sys.c only provides a
single implementation DNS resolution - getaddrinfo - I do believe that
you could improve the portability goal by making it an extern
proc in the Net module. I still think that forcing the C compiler
to inline a version in sys.h would have the same effect. But either
way, the Net module would not be a standard module, and so that
Chapel code would only be compiled if the program said 'use Net;'.
Of course, the other way would be to add an optional runtime layer,
like CHPL_REGEXP or CHPL_GMP. That might make sense if the networking
library has other dependencies that might not always be available,
such as libevent or libcurl. If you agree with me that it would make
sense to build atop libcurl and libevent, this is probably the sensible
way forward. I think the #define - type mess for system support
is best left to C.
Chapel doesn't have anything like #ifdef, but the generic programming
facility is very flexible and could support string comparison on
config params that indicate the current platform. I believe that
should work now, but havn't tried it. These are defined in ChapelBase.chpl
line 1-123. You should be able to say something like (I'm intentionally
ignoring details):
proc mygetaddrinfo() {
extern proc getaddrinfo();
if CHPL_TARGET_PLATFORM == "cray" {
// do something else
} else {
getaddrinfo()
}
}
Since CHPL_TARGET_PLATFORM (and the other similar variables) are config param
(ie compile-time) constants, the compiler is obligated to remove the branch
of the if statement that does not apply. Now, of course CHPL_TARGET_PLATFORM
has a variety of versions for Crays, so perhaps your strstr would be
useful at compile time, like
if CHPL_TARGET_PLATFORM.contains("cray")
but that's a bit of a separate issue, which you should feel free to
put off for the moment.
> Go's net package provides a pretty intuitive high-level way to do
> connect and listen without all the low-level structures and call
> sequences. Connecting looks like net.Dial("tcp", "example.com:8080") and
> listening looks like net.Listen("tcp", "127.0.0.1:8080"). You pass
> strings, and all the low-level DNS lookup, socket binding, and
> translation to system-specific constants and whatnot is hidden. Dial
> returns a Conn, which you can read and write to directly and use
> wherever a Reader or Writer is expected, while Listen returns a Listener
> with an Accept() which returns Conns. Access to the low level stuff is
> still provided, but these functions are generally all you need.
>
> I imagine Chapel's net package would have similar facilities, and maybe
> let you pass URL strings for convenience. It would be nice to connect to
> a web server with net_connect("tcp://example.com:80") or even
> net_connect("http://example.com";) and have it return a channel (or pair
> of). Similarly, net_listen will let you specify a bind address in the
> URL and return an object which has an accept() method that returns channels.
>
> It would be nice to offer some expressive flexibility and sensible
> defaults with regard to accepted URL formats for these functions. E.g.,
> net_listen("tcp://0.0.0.0:80"), net_listen("tcp://:80"),
> net_listen("0.0.0.0:80"), and net_listen(":80") should be equivalent.
> Using a different socket type is as easy as changing the URL, e.g.
> net_listen("udp://:80"), net_listen("unix:/path/to/socket"),
> net_listen("raw://:80"). net_connect and net_listen would of course be
> overloaded to let you pass the arguments separately, e.g.,
> net_connect("tcp", "example.com:80") or net_connect("tcp",
> "example.com", 80) as well. Ideally it should be as easy as possible to
> hack together a working network application and incrementally optimize
> it with lower-level library functions (which will of course be provided
> as well) as necessary.
These all sound pretty reasonable to me, with some stylistic points
(Chapel code usually uses camelCase instead of the underscores, but
last time I tried to push for an official Chapel modules style guide,
there was lots of noise but no consensus).
I have in the past considered including libcurl. I don't really think
it would be reasonable to write our own. libcurl could easily handle
most client-based situations. If you want a web-server though, we
might need something else. Turns out though that libevent (which I
had earlier described a desire to incorporate) has basic http
support (including running a server). My preference would be to
incorporate both libevent and libcurl, and use libevent for
anything that it can do and libcurl for everything else. (It turns
out that you can convince libcurl to run atop libevent, see
http://stackoverflow.com/questions/6927710/using-libcurl-with-libevent -
doing so, combined with libevent support I described earlier, would
give us processor-efficient implementations of a variety of protocols,
so that e.g. a web crawler would be easy to write and efficient.
> I just finished a simple URL parser for this purpose, which is the
> reason behind the string-oriented patch I submitted. Even if we opt not
> to use URLs for describing network connections, it's always nice to have
> a good URL parser in a networking package. (At least it's something I'm
> always looking for.)
Right, we'll discuss your review request separately. Improvements to the
Chapel modules are always appreciated. But if you agree
with me that incorporating libcurl and libevent is the most reasonable
way forward, we might want to rely on their existing URL parsing...
Cheers,
-michael
> -- Brandon
>
> On 12/02/2013 05:36 PM, Michael Ferguson wrote:
>> Hi -
>>
>> Yes, we commented out getaddrinfo because we were getting warnings
>> in testing and didn't want to spend much time chasing them
>> down since nobody was using that functionality. So I'd recommend
>> you un-comment it and sorting out the portability problem will
>> become worth doing (whether you or somebody else does it). As Brad's
>> comment says - it was triggering a warning in some Cray environments...
>> (not a fatal error). It's not clear to me that using a different
>> function would solve the warning... it's really a build issue.
>> The warning we are talking about is something like
>> warning: Using 'getaddrinfo' in statically linked applications
>> requires at
>> runtime the shared libraries from the glibc version used for linking
>> We could:
>> - decide to ignore the warning
>> - try to disable the warning
>> - try to limit the warning to cases when getaddrinfo is actually used
>> (with some form of conditional compilation, inlining, or maybe
>> putting it in a "net support file" that is only included when
>> requested)
>> - investigate if there's a way to build differently to avoid
>> the warning. For example, maybe we don't really need to statically
>> link?
>> Or maybe there is some other magic linker flag to solve the problem?
>>
>> I'm not familiar with Go's net package. Could you briefly describe how
>> Go-inspired network functions in Chapel might look to a user?
>>
>> Also, it's important to understand the existing I/O design
>> which separates a 'file' (e.g. something storing data on disk) from
>> a 'channel' (buffered, unidirectional I/O). You can't read or write
>> a 'file' directly in Chapel - you must open a 'channel' first. The
>> reason is that we wanted to avoid the race condition inherent in
>> the pre-threads UNIX design which has a file position stored in a
>> file. So we store the file position in a 'channel', which cannot
>> seek, and addresses a defined portion of a file. Different channels
>> can be openned to the same file and don't have to have immediate
>> visibility into each others I/Os (vs how writes to a FILE* must
>> be immediately available for reading). Anyway, I'm probably
>> belaboring this point, but accept and connect would return channels,
>> not files, in Chapel.
>>
>> An IRC channel might be a good idea, but there isn't one now.
>> It might make it shorter to respond to questions, when we don't
>> have to set the record for posterity...
>>
>> (to do atoi, cast a string containing a number to an integer).
>>
>> -michael
------------------------------------------------------------------------------
Rapidly troubleshoot problems before they affect your business. Most IT
organizations don't have a clear picture of how application performance
affects their revenue. With AppDynamics, you get 100% visibility into your
Java,.NET, & PHP application. Start your 15-day FREE TRIAL of AppDynamics Pro!
http://pubads.g.doubleclick.net/gampad/clk?id=84349351&iu=/4140/ostg.clktrk
_______________________________________________
Chapel-developers mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/chapel-developers
