On Wed, 2012-08-22 at 23:35 +0000, Hefty, Sean wrote:
> I'm haven't identified the specific problem with fork support, but I did see
> this in libmlx4:
>
> mlx4_alloc_context()
> {
> ...
> context->uar = mmap(NULL, to_mdev(ibdev)->page_size, PROT_WRITE,
> MAP_SHARED, cmd_fd, 0);
> if (context->uar == MAP_FAILED)
> goto err_free;
>
> if (resp.bf_reg_size) {
> context->bf_page = mmap(NULL, to_mdev(ibdev)->page_size,
> PROT_WRITE, MAP_SHARED, cmd_fd,
> to_mdev(ibdev)->page_size);
> ...
> }
>
> I don't know for certain that these mmap() calls cause an issue, but the
> preload library socket() function calls rsocket(), which loads and
> initializes libibverbs. This calls mlx4_alloc_context() before fork() has
> been called.
>
> I added the following hack to the preload socket() call, which skips calling
> rsocket().
>
> if ((domain == PF_INET || domain == PF_INET6) &&
> (type == SOCK_STREAM) && (!protocol || protocol == IPPROTO_TCP) &&
> fork_support) {
> printf("skipping rsocket call\n");
> goto realsock;
> }
>
> recursive = 1;
> ret = rsocket(domain, type, protocol);
> recursive = 0;
> if (ret >= 0) {
> if (fork_support) {
> rclose(ret);
> realsock:
> ret = real.socket(domain, type, protocol);
>
> With the above hack, I no longer see hangs when running with fork() and the
> netperf performance is split. This should delay libibverbs initializing
> until after fork() has been called.
Yes. This fixes the hangs i am seeing with netperf too.
I saw this code in preload library and was wondering why rsocket() is called
and closed immediately if fork_support is enabled. I guess you are doing this
so that you can fallback to real socket at the initial socket() call instead
of waiting all the way until fork_active/fork_passive. This should be OK
specifically if a user is using preload library and enabling fork_support.
This doesn't look like a hack to me. However it looks like a bug if rsocket()
followed by rclose() doesn't cleanup all the resources correctly.
Thanks
Sridhar
> I'm trying to decide whether to keep this work-around or find a better
> solution.
>
> - Sean
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majord...@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
