Hey Apertiumers,
This mail is regarding an ongoing project to eliminate dictionary trimming.
The project idea can be found here
<https://wiki.apertium.org/wiki/Ideas_for_Google_Summer_of_Code/Eliminate_trimming>.
The project description was to work around everything in Why we trim
<https://wiki.apertium.org/wiki/Why_we_trim>.
In the current state of the project, it was proposed that using secondary
tags to propagate surface information in the pipe and weighting the
monodix with the bidix would preserve most of the benefits of trimming
while getting rid of the disadvantages. However, it doesn't seem like we
all agree on the importance of aforementioned benefits and disadvantages. I
am a potential language developer with Apertium, but I haven't made a pair
yet, which makes me anything but an authority on what's actually
beneficial. When I wrote the proposal, I weighed the pros and cons based on
my knowledge of linguistics, MT systems, and Apertium.
However, Francis has made some really good points about the benefits of
trimming that I wasn't aware of, which has now led to a dilemma. I will do
my best to list these advantages and disadvantages here, and you guys can
decide between: *making trimming the norm and having the option of
eliminating it, or making eliminating trimming the norm and having the
option of activating it, or to have partial trimming, as discussed later.*
*Disadvantages:*
1. The monodix has some erroneous analyses - wrong surface forms, wrong
analyses, or even MWEs that aren't really MWEs and can be translated word
by word. These are currently removed since bidixes are more carefully
maintained. If trimming is eliminated, and none of the analyses of a word
are in the bidix, then one of the analyses will be chosen, and there is a
chance that it is erroneous. If it's an MWE that doesn't exist in the
bidix, it won't be translated word by word even though that was ok.
2. If your monodix is used by lots of other pair developers, you don't want
*your* pair to get messed up because someone somewhere decided "take
precautions" should be an MWE, and suddenly where your old output had "ta
forholdsregler" you now get "*take precautions".
- Unhammer
3. Having trimming gives the ability to control the monodix using the bidix
in your language pair. This ability isn't lost, because we're still
weighting the monodix, but if the bidix has none of the analyses for a
word, earlier it was discarded and now it will be retained.
4. Weighting the monodix will take more compile time than just trimming it.
*Advantages:*
1. Since the monodix will be weighted with the bidix (if decided), then a
lot of erroneous analyses will be ignored if even one of them exists in the
bidix and that will be chosen.
2. If none of the analyses of a surface form are in the bidix, then the
most likely one will be chosen and used for context disambiguation and
transfer rules, thereby giving a more comprehensible and more post-editable
output. Even enables us to do something like:
*Basque to English "Andonik izarak izeki zuen" ('Andoni hung up the
sheets') → 'Andoni *izeki-ed the sheets"?*
*Doing this won't be possible without eliminating dictionary trimming.*
3. Philosophically, it would make sense to not discard the knowledge of a
word's analysis if we can use it, even though we cannot translate it.
4. It helps with debugging as now we can distinguish errors from word not
being in the bidix and word not being in the monodix.
5. Earlier without trimming we get @source_lemma, which harms the
translated output, but now we can get @source_surfaceform, which is the
same as what we get with trimming (*source_surfaceform), but with the
benefits of disambiguation and transfer.
If there are any advantages or disadvantages I missed, please add them here
so people can make a more informed decision. *Another possible solution
that was discussed was to continue trimming multiwords since they present a
unique disadvantage, but to eliminate trimming for words without spaces. *Any
other possible solution can be discussed as well.
Hopefully after an informed discussion we can come to an acceptable
conclusion.
Thanks and Regards,
Tanmai Khanna
--
*Khanna, Tanmai*
_______________________________________________
Apertium-stuff mailing list
Apertium-stuff@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/apertium-stuff
