[perl #129994] [NATIVECALL][BUG] C functions required to be called once in one thread are hung up

[jn...@jnthn.net via RT]

On Tue Nov 01 09:01:04 2016, cookbook_...@yahoo.co.jp wrote:
> I'm creating a Perl 6 bindings for
> MeCab ( http://taku910.github.io/mecab/ )
> 
> See the following codes and documentations(translations).
> 
> My repository is:
> https://github.com/titsuki/p6-MeCab/tree/for-RT
> 
> 
> The following subtest in the t/02-lattice.t in the above repository is
> hung up.
> 
> subtest {
>     my MeCab::Model $model .= new;
>     my @texts = (("私","僕") xx 3).flat;
> 
>     my @actual = (@texts.hyper(:batch(1))\
>                   .map(
>                          {
>                              my MeCab::Tagger $tagger = $model.create-
> tagger;
>                              my MeCab::Lattice $lattice =
> $model.create-lattice;
>                              $lattice.sentence($_);
>                              $lattice.tostr if
> $tagger.parse($lattice);
>                              $tagger.DESTROY;
>                              $lattice.DESTROY;
>                          }
>                      ).list);
> 
>     my Str $r1 = ("私\t名詞,代名詞,一般,*,*,*,私,ワタシ,ワタシ\nEOS\n");
>     my Str $r2 = ("僕\t名詞,代名詞,一般,*,*,*,僕,ボク,ボク\nEOS\n");
>     my @expected = (($r1, $r2) xx 3).flat;
>     is @actual, @expected;
> }, "MeCab::Tagger should work in the multithread environment";
> ---
> 
> 
> In the MeCab documentation
> page(Japanese) https://taku910.github.io/mecab/libmecab.html
> "C++ サンプルコード" (en: C++ sample code) section says that:
> In the multithread environment (MeCab::Tagger, MeCab::Model,
> MeCab::Lattice)
> 
> * Call MeCab::createModel() and create the Model object.
> 
> * Call model->createTagger and create the Tagger object. The Tagger
> objects share the same model, even though you create the multiple
> models under the one model per thread constraint.
> 
> * Call model->createLattice or MeCab::createLattice() and crate the
> Lattice object. The Lattice objects include all local variables for
> the morpheme analyze. Must keep the constraint that you can create
> only one object per thread.
> 
> * model->swap(antoher_model) function replaces the Tagger object
> models created from the invocant model by another_model. This
> operation is thread-safe.
> 
> 
> 
> 
> I think I properly call the MeCab functions from Perl 6(e.g.
> $model.create-tagger, $model.create-lattice) according to the above
> instructions.
> So I think something is wrong in the NativeCall.

I think the hang was due to long-running native functions called on one thread 
blocking GC (and thus progress) in all other threads. That is addressed in 
https://github.com/MoarVM/MoarVM/commit/5c659178a07d86fe73b7c506c4ea7b10ad38b0e8
 and 
https://github.com/MoarVM/MoarVM/commit/f769569b78f099d59b7a17966d9096a0837c4d42
 and tested in 
https://github.com/rakudo/rakudo/commit/5ba75f445dc74560f9dabceeb4b4dc13f78c786f.

If you still have problems, please re-open this issue. Also, if possible, see 
if you can find a smaller reproduction (for example, try just running the code 
in a number of start blocks, or threads, instead of using `.hyper`).

Thanks,

/jnthn

[perl #129994] [NATIVECALL][BUG] C functions required to be called once in one thread are hung up

[jn...@jnthn.net via RT]

On Tue Nov 01 09:01:04 2016, cookbook_...@yahoo.co.jp wrote:
> I'm creating a Perl 6 bindings for
> MeCab ( http://taku910.github.io/mecab/ )
> 
> See the following codes and documentations(translations).
> 
> My repository is:
> https://github.com/titsuki/p6-MeCab/tree/for-RT
> 
> 
> The following subtest in the t/02-lattice.t in the above repository is
> hung up.
> 
> subtest {
>     my MeCab::Model $model .= new;
>     my @texts = (("私","僕") xx 3).flat;
> 
>     my @actual = (@texts.hyper(:batch(1))\
>                   .map(
>                          {
>                              my MeCab::Tagger $tagger = $model.create-
> tagger;
>                              my MeCab::Lattice $lattice =
> $model.create-lattice;
>                              $lattice.sentence($_);
>                              $lattice.tostr if
> $tagger.parse($lattice);
>                              $tagger.DESTROY;
>                              $lattice.DESTROY;
>                          }
>                      ).list);
> 
>     my Str $r1 = ("私\t名詞,代名詞,一般,*,*,*,私,ワタシ,ワタシ\nEOS\n");
>     my Str $r2 = ("僕\t名詞,代名詞,一般,*,*,*,僕,ボク,ボク\nEOS\n");
>     my @expected = (($r1, $r2) xx 3).flat;
>     is @actual, @expected;
> }, "MeCab::Tagger should work in the multithread environment";
> ---
> 
> 
> In the MeCab documentation
> page(Japanese) https://taku910.github.io/mecab/libmecab.html
> "C++ サンプルコード" (en: C++ sample code) section says that:
> In the multithread environment (MeCab::Tagger, MeCab::Model,
> MeCab::Lattice)
> 
> * Call MeCab::createModel() and create the Model object.
> 
> * Call model->createTagger and create the Tagger object. The Tagger
> objects share the same model, even though you create the multiple
> models under the one model per thread constraint.
> 
> * Call model->createLattice or MeCab::createLattice() and crate the
> Lattice object. The Lattice objects include all local variables for
> the morpheme analyze. Must keep the constraint that you can create
> only one object per thread.
> 
> * model->swap(antoher_model) function replaces the Tagger object
> models created from the invocant model by another_model. This
> operation is thread-safe.
> 
> 
> 
> 
> I think I properly call the MeCab functions from Perl 6(e.g.
> $model.create-tagger, $model.create-lattice) according to the above
> instructions.
> So I think something is wrong in the NativeCall.

I think the hang was due to long-running native functions called on one thread 
blocking GC (and thus progress) in all other threads. That is addressed in 
https://github.com/MoarVM/MoarVM/commit/5c659178a07d86fe73b7c506c4ea7b10ad38b0e8
 and 
https://github.com/MoarVM/MoarVM/commit/f769569b78f099d59b7a17966d9096a0837c4d42
 and tested in 
https://github.com/rakudo/rakudo/commit/5ba75f445dc74560f9dabceeb4b4dc13f78c786f.

If you still have problems, please re-open this issue. Also, if possible, see 
if you can find a smaller reproduction (for example, try just running the code 
in a number of start blocks, or threads, instead of using `.hyper`).

Thanks,

/jnthn

Re: [perl #129994] [NATIVECALL][BUG] C functions required to be called once in one thread are hung up

[Brandon Allbery via RT]

On Tue, Nov 1, 2016 at 12:01 PM, Itsuki Toyota  wrote:

> I think I properly call the MeCab functions from Perl 6(e.g.
> $model.create-tagger, $model.create-lattice) according to the above
> instructions.
> So I think something is wrong in the NativeCall.
>

Not sure NativeCall even knows about TLS as yet. Probably needs a new
attribute for functions that use TLS, and code to ensure those always run
on the same thread. (This also needs to group functions so they will all
use the same thread... which, given the usage for this particular library,
may need to be dynamic where the init call picks an available thread and
"pins" it for subsequent calls. Which will be tricky, because it needs to
be attached to the object returned by init. Urgh.)

-- 
brandon s allbery kf8nh   sine nomine associates
allber...@gmail.com  ballb...@sinenomine.net
unix, openafs, kerberos, infrastructure, xmonadhttp://sinenomine.net

Re: [perl #129994] [NATIVECALL][BUG] C functions required to be called once in one thread are hung up

[Brandon Allbery]

On Tue, Nov 1, 2016 at 12:01 PM, Itsuki Toyota  wrote:

> I think I properly call the MeCab functions from Perl 6(e.g.
> $model.create-tagger, $model.create-lattice) according to the above
> instructions.
> So I think something is wrong in the NativeCall.
>

Not sure NativeCall even knows about TLS as yet. Probably needs a new
attribute for functions that use TLS, and code to ensure those always run
on the same thread. (This also needs to group functions so they will all
use the same thread... which, given the usage for this particular library,
may need to be dynamic where the init call picks an available thread and
"pins" it for subsequent calls. Which will be tricky, because it needs to
be attached to the object returned by init. Urgh.)

-- 
brandon s allbery kf8nh   sine nomine associates
allber...@gmail.com  ballb...@sinenomine.net
unix, openafs, kerberos, infrastructure, xmonadhttp://sinenomine.net

[perl #129994] [NATIVECALL][BUG] C functions required to be called once in one thread are hung up

[via RT]

# New Ticket Created by  Itsuki Toyota 
# Please include the string:  [perl #129994]
# in the subject line of all future correspondence about this issue. 
# https://rt.perl.org/Ticket/Display.html?id=129994 >


I'm creating a Perl 6 bindings for MeCab ( http://taku910.github.io/mecab/ )

See the following codes and documentations(translations).

My repository is:
https://github.com/titsuki/p6-MeCab/tree/for-RT


The following subtest in the t/02-lattice.t in the above repository is hung up.

subtest {
    my MeCab::Model $model .= new;
    my @texts = (("私","僕") xx 3).flat;

    my @actual = (@texts.hyper(:batch(1))\
                  .map(
                         {
                             my MeCab::Tagger $tagger = $model.create-tagger;
                             my MeCab::Lattice $lattice = $model.create-lattice;
                             $lattice.sentence($_);
                             $lattice.tostr if $tagger.parse($lattice);
                             $tagger.DESTROY;
                             $lattice.DESTROY;
                         }
                     ).list);

    my Str $r1 = ("私\t名詞,代名詞,一般,*,*,*,私,ワタシ,ワタシ\nEOS\n");
    my Str $r2 = ("僕\t名詞,代名詞,一般,*,*,*,僕,ボク,ボク\nEOS\n");
    my @expected = (($r1, $r2) xx 3).flat;
    is @actual, @expected;
}, "MeCab::Tagger should work in the multithread environment";
---


In the MeCab documentation page(Japanese) 
https://taku910.github.io/mecab/libmecab.html
"C++ サンプルコード" (en: C++ sample code) section says that:
In the multithread environment (MeCab::Tagger, MeCab::Model, MeCab::Lattice)

* Call MeCab::createModel() and create the Model object.

* Call model->createTagger and create the Tagger object. The Tagger objects 
share the same model, even though you create the multiple models under the one 
model per thread constraint.

* Call model->createLattice or MeCab::createLattice() and crate the Lattice 
object. The Lattice objects include all local variables for the morpheme 
analyze. Must keep the constraint that you can create only one object per 
thread.

* model->swap(antoher_model) function replaces the Tagger object models created 
from the invocant model by another_model. This operation is thread-safe.




I think I properly call the MeCab functions from Perl 6(e.g. 
$model.create-tagger, $model.create-lattice) according to the above 
instructions.
So I think something is wrong in the NativeCall.