Re: [Apertium-stuff] GSoC 19: Unsupervised weighting of automata - Implementing the supervised method of weighing autoamata

[Amr Mohamed Hosny Anwar]

Hi all,

I would like to share the status of implementing the supervised method of 
weighting of automata script.

First, I used this page for understanding the XEROX regex syntax "Syntax of 
Regular Expressions (Finite-State Calculus)": 
ftp://ftp.cis.upenn.edu/pub/cis639/public_html/docs/fssyntax.html

Then, I wrote a bash script (URL: 
https://github.com/AMR-KELEG/lttoolbox/tree/supervised-weighing-of-automata-experiment/scripts)
 whose inputs are:

- A tagged corpus (such as: 
https://github.com/apertium/apertium-eng/blob/master/texts/eng.tagged)
- A compiled FST in apertium's format

Currently, the script isn't working as lt-comp fails to compile FSTs with 
initial epsilon transitions or multiple FSTs in the same .att file.
I haven't inspect the source of the error yet.

The thrown exception is:

terminate called after throwing an instance of 'std::invalid_argument'
  what():  stoi
  Aborted (core dumped)

I tried adding  [?*] at the start and the end of the regex but the script 
became much much slower.
I left it for about two hours and it was still doing computations!
I will have to estimate the complexity of the compose function to predict 
whether the script's running time is reasonable or not.

In order to keep all the paths that aren't part of the tagged corpus, I added a 
regex in the form [?*]::100.
So, Basically all the paths that are part of the input FST will have a very 
large weight (We should represent INF in a better way).
However, this generates the problem of having two analyses for any of the 
weighted regexps.
I will experiment with subtracting FSTs so that weighted analyses don't receive 
trivial unweighted analysis.
I have tested  the script using the apertium-eng FST and I am attaching the 
results of using the unweighted/weighted HFST FST.

Should I investigate the causes of the errors related to lttoolbox and may be 
implement methods that will help in compiling hfst-generated att files?
I believe we will need to figure out a way to weight FSTs regardless of the way 
these weights were generated.

Regards,
Amr



From: Hèctor Alòs i Font 
Sent: Wednesday, May 22, 2019 5:55 AM
To: [apertium-stuff]
Subject: Re: [Apertium-stuff] GSoC 19: Unsupervised weighting of automata - 
Implementing the supervised method of weighing autoamata

Hi Amr,

The files should be there. It was a mistake. I added them. Each line has a 
word. If something is not clear, the "raw" files have the original texts 
without morphological analysis.

About other manually tagged corpora, I cannot help. This is the only one we did 
in the projects I've been working. Unfortunately, we didn't find the time for 
creating a similar one for Sardinian two years ago.

Best,
Hèctor

Missatge de Amr Mohamed Hosny Anwar 
mailto:amr.ke...@eng.asu.edu.eg>> del dia dt., 21 de 
maig 2019 a les 19:05:

Hi Hector,

Yes, these files are for sure what I need.
However, it seems like these files (*.tagged.txt) aren't part of the upstream 
repository: https://github.com/apertium/apertium-oci/tree/master/texts

I am currently experimenting with the English and Italian tagged 
corpora/morphological analysers.
The more languages we have, the better we can compare between weighting 
methodologies.
I don't have a strong background in linguistics so I thought it'd be better if 
you can recommend me corpora from different diverse languages.

Thanks,
Amr


From: Hèctor Alòs i Font mailto:hectora...@gmail.com>>
Sent: Tuesday, May 21, 2019 12:42:02 PM
To: [apertium-stuff]
Subject: Re: [Apertium-stuff] GSoC 19: Unsupervised weighting of automata - 
Implementing the supervised method of weighing autoamata

Hi Amr,

I'm not sure it may help you, but in apertium-oci/texts there are several texts 
in Occitan manually disambiguated. Aprox. 14,000 words. They are:
atom_gascon.tagged.txt
continent.tagged.txt
glacier.tagged.txt
cors_aran.tagged.txt
hlama_coming.tagged.txt
uranus_prov.tagged.txt

Best,
Hector

Missatge de Amr Mohamed Hosny Anwar 
mailto:amr.ke...@eng.asu.edu.eg>> del dia dg., 19 de 
maig 2019 a les 2:59:

Dear maintainers, contributors,

Hope this email finds you well.

This mail can be considered as a status report for detailing next week's plan 
in addition to seeking feedback/ suggestions regarding the project.
After a fruitful discussion with my mentors Nick, Flammie and Francis, we have 
agreed on implementing the supervised way of weighing automata as follows:

The command will look like: lt-weight transducer.bin corpus.tagged

transducer.bin: A FST compiled using lttoolbox.
corpus.tagged: A tagged corpus that will be used to estimate the weights.

The weighting will be done by composing the main "unweighted" FST with a set of 
simple FSTs that are generated for each token.
A simplified example: If the main FST had an edge a:b::0 and the estimated 
weight for this edge is W, then The main FST will be composed with a simple FST 
of an edge b:b::w generating a new 

Re: [Apertium-stuff] GSoC 19: Unsupervised weighting of automata - Implementing the supervised method of weighing autoamata

[Hèctor Alòs i Font]

Hi Amr,

The files should be there. It was a mistake. I added them. Each line has a
word. If something is not clear, the "raw" files have the original texts
without morphological analysis.

About other manually tagged corpora, I cannot help. This is the only one we
did in the projects I've been working. Unfortunately, we didn't find the
time for creating a similar one for Sardinian two years ago.

Best,
Hèctor

Missatge de Amr Mohamed Hosny Anwar  del dia dt.,
21 de maig 2019 a les 19:05:

> Hi Hector,
>
> Yes, these files are for sure what I need.
> However, it seems like these files (*.tagged.txt) aren't part of the
> upstream repository:
> https://github.com/apertium/apertium-oci/tree/master/texts
>
> I am currently experimenting with the English and Italian tagged
> corpora/morphological analysers.
> The more languages we have, the better we can compare between weighting
> methodologies.
> I don't have a strong background in linguistics so I thought it'd be
> better if you can recommend me corpora from different diverse languages.
>
> Thanks,
> Amr
>
> --
> *From:* Hèctor Alòs i Font 
> *Sent:* Tuesday, May 21, 2019 12:42:02 PM
> *To:* [apertium-stuff]
> *Subject:* Re: [Apertium-stuff] GSoC 19: Unsupervised weighting of
> automata - Implementing the supervised method of weighing autoamata
>
> Hi Amr,
>
> I'm not sure it may help you, but in apertium-oci/texts there are several
> texts in Occitan manually disambiguated. Aprox. 14,000 words. They are:
> atom_gascon.tagged.txt
> continent.tagged.txt
> glacier.tagged.txt
> cors_aran.tagged.txt
> hlama_coming.tagged.txt
> uranus_prov.tagged.txt
>
> Best,
> Hector
>
> Missatge de Amr Mohamed Hosny Anwar  del dia
> dg., 19 de maig 2019 a les 2:59:
>
> Dear maintainers, contributors,
>
> Hope this email finds you well.
>
> This mail can be considered as a status report for detailing next week's
> plan in addition to seeking feedback/ suggestions regarding the project.
> After a fruitful discussion with my mentors Nick, Flammie and Francis, we
> have agreed on implementing the supervised way of weighing automata as
> follows:
>
> The command will look like: lt-weight transducer.bin corpus.tagged
>
> transducer.bin: A FST compiled using lttoolbox.
> corpus.tagged: A tagged corpus that will be used to estimate the weights.
>
> The weighting will be done by composing the main "unweighted" FST with a
> set of simple FSTs that are generated for each token.
> A simplified example: If the main FST had an edge a:b::0 and the estimated
> weight for this edge is W, then The main FST will be composed with a simple
> FST of an edge b:b::w generating a new FST with an edge a:b::W.
>
> To achieve this, I will create a new shell script that makes use of hfst's
> compose (Instead of implementing/adding a compose function to the
> lttoolbox). We will approve and use this approach if the prototype has
> proven to be functioning as expected.
>
> The shell script will work as follows:
> 1) lt-print will be used to convert the FST to at format.
> 2) The weights will be estimated from the tagged corpus by counting the
> unigram lexical forms (A clever set of shell commands can do the job but I
> am not an expert in shell scripting so it will take me some time - I am
> open to suggestions/ sources/ examples for doing so).
> 3) For each weighted string, hfst-str2fst (or the corresponding regex
> version) will be used to generate simple FSTS.
> 4) The FSTs will be composed using hfst-compose.
> 5) The final FST will be converted to at format.
> 6) lt-comp will be be used to regenerate a weighted FST that is compatable
> with all the tools that rely on apertium.
>
> In this version, We will just use unigram counts for the lexical forms to
> estimate the weights.
> Additionally, The weight will be assigned to the final state and won't be
> distributed among the edges (We will most probably want to change this
> later).
>
> On the other hand, I will try to improve the list of publications/ideas
> that will be used to weigh automata in an unsupervised way.
> I would be grateful if you can share with me resources/ ideas regarding
> this part.
>
> Finally, Do you have recommendations for tagged corpora that can be used
> throughout the project for benchmarking?
> I am using this English Tagged corpus from the apertium-eng repository (
> https://github.com/apertium/apertium-eng/blob/master/texts/eng.tagged)
> It would be better if we can do benchmarking on corpora and FSTs of
> different sizes and complexity.
>
> Thanks and looking forward to hearing from you.
> Your suggestions, feedback, feature requests are more than welcome.
>
> Best Regards,
> Amr
>
> --
> *From:* Amr Mohamed Hosny Anwar
> *Sent:* Sunday, May 19, 2019 12:50:52 AM
> *To:* apertium-stuff
> *Cc:* nlhow...@gmail.com
> *Subject:* GSoC 19: Unsupervised weighting of automata - Implementing the
> supervised method of weighing autoamata
>
>
> Dear maintainers, contributors,

Re: [Apertium-stuff] GSoC 19: Unsupervised weighting of automata - Implementing the supervised method of weighing autoamata

[Amr Mohamed Hosny Anwar]

Hi Hector,

Yes, these files are for sure what I need.
However, it seems like these files (*.tagged.txt) aren't part of the upstream 
repository: https://github.com/apertium/apertium-oci/tree/master/texts

I am currently experimenting with the English and Italian tagged 
corpora/morphological analysers.
The more languages we have, the better we can compare between weighting 
methodologies.
I don't have a strong background in linguistics so I thought it'd be better if 
you can recommend me corpora from different diverse languages.

Thanks,
Amr


From: H?ctor Al?s i Font 
Sent: Tuesday, May 21, 2019 12:42:02 PM
To: [apertium-stuff]
Subject: Re: [Apertium-stuff] GSoC 19: Unsupervised weighting of automata - 
Implementing the supervised method of weighing autoamata

Hi Amr,

I'm not sure it may help you, but in apertium-oci/texts there are several texts 
in Occitan manually disambiguated. Aprox. 14,000 words. They are:
atom_gascon.tagged.txt
continent.tagged.txt
glacier.tagged.txt
cors_aran.tagged.txt
hlama_coming.tagged.txt
uranus_prov.tagged.txt

Best,
Hector

Missatge de Amr Mohamed Hosny Anwar 
mailto:amr.ke...@eng.asu.edu.eg>> del dia dg., 19 de 
maig 2019 a les 2:59:

Dear maintainers, contributors,

Hope this email finds you well.

This mail can be considered as a status report for detailing next week's plan 
in addition to seeking feedback/ suggestions regarding the project.
After a fruitful discussion with my mentors Nick, Flammie and Francis, we have 
agreed on implementing the supervised way of weighing automata as follows:

The command will look like: lt-weight transducer.bin corpus.tagged

transducer.bin: A FST compiled using lttoolbox.
corpus.tagged: A tagged corpus that will be used to estimate the weights.

The weighting will be done by composing the main "unweighted" FST with a set of 
simple FSTs that are generated for each token.
A simplified example: If the main FST had an edge a:b::0 and the estimated 
weight for this edge is W, then The main FST will be composed with a simple FST 
of an edge b:b::w generating a new FST with an edge a:b::W.

To achieve this, I will create a new shell script that makes use of hfst's 
compose (Instead of implementing/adding a compose function to the lttoolbox). 
We will approve and use this approach if the prototype has proven to be 
functioning as expected.

The shell script will work as follows:
1) lt-print will be used to convert the FST to at format.
2) The weights will be estimated from the tagged corpus by counting the unigram 
lexical forms (A clever set of shell commands can do the job but I am not an 
expert in shell scripting so it will take me some time - I am open to 
suggestions/ sources/ examples for doing so).
3) For each weighted string, hfst-str2fst (or the corresponding regex version) 
will be used to generate simple FSTS.
4) The FSTs will be composed using hfst-compose.
5) The final FST will be converted to at format.
6) lt-comp will be be used to regenerate a weighted FST that is compatable with 
all the tools that rely on apertium.

In this version, We will just use unigram counts for the lexical forms to 
estimate the weights.
Additionally, The weight will be assigned to the final state and won't be 
distributed among the edges (We will most probably want to change this later).

On the other hand, I will try to improve the list of publications/ideas that 
will be used to weigh automata in an unsupervised way.
I would be grateful if you can share with me resources/ ideas regarding this 
part.

Finally, Do you have recommendations for tagged corpora that can be used 
throughout the project for benchmarking?
I am using this English Tagged corpus from the apertium-eng repository 
(https://github.com/apertium/apertium-eng/blob/master/texts/eng.tagged)
It would be better if we can do benchmarking on corpora and FSTs of different 
sizes and complexity.

Thanks and looking forward to hearing from you.
Your suggestions, feedback, feature requests are more than welcome.

Best Regards,
Amr


From: Amr Mohamed Hosny Anwar
Sent: Sunday, May 19, 2019 12:50:52 AM
To: apertium-stuff
Cc: nlhow...@gmail.com
Subject: GSoC 19: Unsupervised weighting of automata - Implementing the 
supervised method of weighing autoamata


Dear maintainers, contributors,

Hope this email finds you well.

This mail can be considered as a status report for detailing next week's plan 
in addition to seeking feedback/ suggestions regarding the project.



Best Regards,
Amr Keleg

___
Apertium-stuff mailing list
Apertium-stuff@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/apertium-stuff
___
Apertium-stuff mailing list
Apertium-stuff@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/apertium-stuff

Re: [Apertium-stuff] GSoC 19: Unsupervised weighting of automata - Implementing the supervised method of weighing autoamata

[Hèctor Alòs i Font]

Hi Amr,

I'm not sure it may help you, but in apertium-oci/texts there are several
texts in Occitan manually disambiguated. Aprox. 14,000 words. They are:
atom_gascon.tagged.txt
continent.tagged.txt
glacier.tagged.txt
cors_aran.tagged.txt
hlama_coming.tagged.txt
uranus_prov.tagged.txt

Best,
Hector

Missatge de Amr Mohamed Hosny Anwar  del dia dg.,
19 de maig 2019 a les 2:59:

> Dear maintainers, contributors,
>
> Hope this email finds you well.
>
> This mail can be considered as a status report for detailing next week's
> plan in addition to seeking feedback/ suggestions regarding the project.
> After a fruitful discussion with my mentors Nick, Flammie and Francis, we
> have agreed on implementing the supervised way of weighing automata as
> follows:
>
> The command will look like: lt-weight transducer.bin corpus.tagged
>
> transducer.bin: A FST compiled using lttoolbox.
> corpus.tagged: A tagged corpus that will be used to estimate the weights.
>
> The weighting will be done by composing the main "unweighted" FST with a
> set of simple FSTs that are generated for each token.
> A simplified example: If the main FST had an edge a:b::0 and the estimated
> weight for this edge is W, then The main FST will be composed with a simple
> FST of an edge b:b::w generating a new FST with an edge a:b::W.
>
> To achieve this, I will create a new shell script that makes use of hfst's
> compose (Instead of implementing/adding a compose function to the
> lttoolbox). We will approve and use this approach if the prototype has
> proven to be functioning as expected.
>
> The shell script will work as follows:
> 1) lt-print will be used to convert the FST to at format.
> 2) The weights will be estimated from the tagged corpus by counting the
> unigram lexical forms (A clever set of shell commands can do the job but I
> am not an expert in shell scripting so it will take me some time - I am
> open to suggestions/ sources/ examples for doing so).
> 3) For each weighted string, hfst-str2fst (or the corresponding regex
> version) will be used to generate simple FSTS.
> 4) The FSTs will be composed using hfst-compose.
> 5) The final FST will be converted to at format.
> 6) lt-comp will be be used to regenerate a weighted FST that is compatable
> with all the tools that rely on apertium.
>
> In this version, We will just use unigram counts for the lexical forms to
> estimate the weights.
> Additionally, The weight will be assigned to the final state and won't be
> distributed among the edges (We will most probably want to change this
> later).
>
> On the other hand, I will try to improve the list of publications/ideas
> that will be used to weigh automata in an unsupervised way.
> I would be grateful if you can share with me resources/ ideas regarding
> this part.
>
> Finally, Do you have recommendations for tagged corpora that can be used
> throughout the project for benchmarking?
> I am using this English Tagged corpus from the apertium-eng repository (
> https://github.com/apertium/apertium-eng/blob/master/texts/eng.tagged)
> It would be better if we can do benchmarking on corpora and FSTs of
> different sizes and complexity.
>
> Thanks and looking forward to hearing from you.
> Your suggestions, feedback, feature requests are more than welcome.
>
> Best Regards,
> Amr
>
> --
> *From:* Amr Mohamed Hosny Anwar
> *Sent:* Sunday, May 19, 2019 12:50:52 AM
> *To:* apertium-stuff
> *Cc:* nlhow...@gmail.com
> *Subject:* GSoC 19: Unsupervised weighting of automata - Implementing the
> supervised method of weighing autoamata
>
>
> Dear maintainers, contributors,
>
>
> Hope this email finds you well.
>
> This mail can be considered as a status report for detailing next week's
> plan in addition to seeking feedback/ suggestions regarding the project.
>
>
>
> Best Regards,
> Amr Keleg
> ___
> Apertium-stuff mailing list
> Apertium-stuff@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/apertium-stuff
>
___
Apertium-stuff mailing list
Apertium-stuff@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/apertium-stuff

Re: [Apertium-stuff] GSoC 19: Unsupervised weighting of automata - Implementing the supervised method of weighing autoamata

[Amr Mohamed Hosny Anwar]

Dear maintainers, contributors,

Hope this email finds you well.

This mail can be considered as a status report for detailing next week's plan 
in addition to seeking feedback/ suggestions regarding the project.
After a fruitful discussion with my mentors Nick, Flammie and Francis, we have 
agreed on implementing the supervised way of weighing automata as follows:

The command will look like: lt-weight transducer.bin corpus.tagged

transducer.bin: A FST compiled using lttoolbox.
corpus.tagged: A tagged corpus that will be used to estimate the weights.

The weighting will be done by composing the main "unweighted" FST with a set of 
simple FSTs that are generated for each token.
A simplified example: If the main FST had an edge a:b::0 and the estimated 
weight for this edge is W, then The main FST will be composed with a simple FST 
of an edge b:b::w generating a new FST with an edge a:b::W.

To achieve this, I will create a new shell script that makes use of hfst's 
compose (Instead of implementing/adding a compose function to the lttoolbox). 
We will approve and use this approach if the prototype has proven to be 
functioning as expected.

The shell script will work as follows:
1) lt-print will be used to convert the FST to at format.
2) The weights will be estimated from the tagged corpus by counting the unigram 
lexical forms (A clever set of shell commands can do the job but I am not an 
expert in shell scripting so it will take me some time - I am open to 
suggestions/ sources/ examples for doing so).
3) For each weighted string, hfst-str2fst (or the corresponding regex version) 
will be used to generate simple FSTS.
4) The FSTs will be composed using hfst-compose.
5) The final FST will be converted to at format.
6) lt-comp will be be used to regenerate a weighted FST that is compatable with 
all the tools that rely on apertium.

In this version, We will just use unigram counts for the lexical forms to 
estimate the weights.
Additionally, The weight will be assigned to the final state and won't be 
distributed among the edges (We will most probably want to change this later).

On the other hand, I will try to improve the list of publications/ideas that 
will be used to weigh automata in an unsupervised way.
I would be grateful if you can share with me resources/ ideas regarding this 
part.

Finally, Do you have recommendations for tagged corpora that can be used 
throughout the project for benchmarking?
I am using this English Tagged corpus from the apertium-eng repository 
(https://github.com/apertium/apertium-eng/blob/master/texts/eng.tagged)
It would be better if we can do benchmarking on corpora and FSTs of different 
sizes and complexity.

Thanks and looking forward to hearing from you.
Your suggestions, feedback, feature requests are more than welcome.

Best Regards,
Amr


From: Amr Mohamed Hosny Anwar
Sent: Sunday, May 19, 2019 12:50:52 AM
To: apertium-stuff
Cc: nlhow...@gmail.com
Subject: GSoC 19: Unsupervised weighting of automata - Implementing the 
supervised method of weighing autoamata


Dear maintainers, contributors,

Hope this email finds you well.

This mail can be considered as a status report for detailing next week's plan 
in addition to seeking feedback/ suggestions regarding the project.



Best Regards,
Amr Keleg
___
Apertium-stuff mailing list
Apertium-stuff@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/apertium-stuff