Re: [Moses-support] string of Words + states in feature functions
Hoang, Hieu and Matthias,Thank you all so much, your explanations really helped me. Hieu, could you please add your explanations about compare function to adding-feature-function web page? I'm confused about how compare function works. which hypothesis are going to be compared with each other? those hypos that covers the same source spans (for example all hypos that cover [x y] spans)? or those ones that covers the source spans with the same length( [x x+d] spans)? Cheers On Wed, Dec 10, 2014 at 2:02 PM, Matthias Huck mh...@inf.ed.ac.uk wrote: Hi Amir, The input is passed to the feature functions via InitializeForInput(InputType const source). This method is called before search and collecting of translation options (cf. moses/FF/FeatureFunction.h). You can set a member variable to have access to the input in your scoring method. Alternatively, if you implement EvaluateWithSourceContext(), the input is passed directly to the method as a parameter (const InputType input) and you can use that. Finally, there's another option in the EvaluateWhenApplied() methods. You can get the input from the Hypothesis object: const InputType input = hypo.GetManager().GetSource(); The input is an InputType object. Moses knows different input types, see InputTypeEnum in moses/TypeDef.h . So what you get might differ depending on what was passed to the decoder. If you're happy with implementing your feature for sentence input only, then you can cast the input to a Sentence object. The Sentence object gives you convenient access methods, in particular GetSize() and GetWord(size_t pos). You can thus obtain the sequence of words in the input. Words can contain several factors in Moses. The factor with index 0 is typically the surface form. Access it using the [] operator. I guess you will never really want to work directly with the string representation of the factor, but at this point you would be able to get it and for instance print it to your debug output. Hope this was helpful as another answer to your first question. Cheers, Matthias On Wed, 2014-12-10 at 11:41 +0330, amir haghighi wrote: Hi everyone I'm implementing a feature function in moses-chart. I need the source words string and also their indexes in the source sentence. I've written a function that gets the source words but I don't know how extract word string from a word. could anyone guide me how to do that? as I know, each word is implemented as an array of factors, which of them is its string? I have also some questions about the states in the stateful features, what kind of variables should be stored in each state? only those ones that should be used in the compare function? or any variable from the previous hypothesis that we use in our feature? Thanks in advance! Cheers Amir ___ Moses-support mailing list Moses-support@mit.edu http://mailman.mit.edu/mailman/listinfo/moses-support -- The University of Edinburgh is a charitable body, registered in Scotland, with registration number SC005336. ___ Moses-support mailing list Moses-support@mit.edu http://mailman.mit.edu/mailman/listinfo/moses-support
[Moses-support] string of Words + states in feature functions
Hi everyone I'm implementing a feature function in moses-chart. I need the source words string and also their indexes in the source sentence. I've written a function that gets the source words but I don't know how extract word string from a word. could anyone guide me how to do that? as I know, each word is implemented as an array of factors, which of them is its string? I have also some questions about the states in the stateful features, what kind of variables should be stored in each state? only those ones that should be used in the compare function? or any variable from the previous hypothesis that we use in our feature? Thanks in advance! Cheers Amir ___ Moses-support mailing list Moses-support@mit.edu http://mailman.mit.edu/mailman/listinfo/moses-support
Re: [Moses-support] string of Words + states in feature functions
Hi Amir, I'm implementing a feature function in moses-chart. I need the source words string and also their indexes in the source sentence. I've written a function that gets the source words but I don't know how extract word string from a word. could anyone guide me how to do that? as I know, each word is implemented as an array of factors, which of them is its string? You can utilize some of the following functions to get the source information: //target phrase and range const TargetPhrase currTargetPhrase = cur_hypo.GetCurrTargetPhrase(); const WordsRange sourceWordRage = cur_hypo.GetCurrSourceWordsRange(); //source sentence Manager manager = cur_hypo.GetManager(); const Sentence source_sent = static_castconst Sentence(manager.GetSource()); //alignment const AlignmentInfo alignments = targetPhrase.GetAlignTerm(); I have also some questions about the states in the stateful features, what kind of variables should be stored in each state? only those ones that should be used in the compare function? or any variable from the previous hypothesis that we use in our feature? Normally, for stateful functions, for instance, previous target words will be stored. -- Cheers, Vu On Wed, Dec 10, 2014 at 4:11 PM, amir haghighi amir.haghighi...@gmail.com wrote: Hi everyone I'm implementing a feature function in moses-chart. I need the source words string and also their indexes in the source sentence. I've written a function that gets the source words but I don't know how extract word string from a word. could anyone guide me how to do that? as I know, each word is implemented as an array of factors, which of them is its string? I have also some questions about the states in the stateful features, what kind of variables should be stored in each state? only those ones that should be used in the compare function? or any variable from the previous hypothesis that we use in our feature? Thanks in advance! Cheers Amir ___ Moses-support mailing list Moses-support@mit.edu http://mailman.mit.edu/mailman/listinfo/moses-support ___ Moses-support mailing list Moses-support@mit.edu http://mailman.mit.edu/mailman/listinfo/moses-support
Re: [Moses-support] string of Words + states in feature functions
More: word_str = source_sent.GetWord(pos).GetString(m_factorType) -- Cheers, Vu On Wed, Dec 10, 2014 at 5:26 PM, HOANG Cong Duy Vu duyvu...@gmail.com wrote: Hi Amir, I'm implementing a feature function in moses-chart. I need the source words string and also their indexes in the source sentence. I've written a function that gets the source words but I don't know how extract word string from a word. could anyone guide me how to do that? as I know, each word is implemented as an array of factors, which of them is its string? You can utilize some of the following functions to get the source information: //target phrase and range const TargetPhrase currTargetPhrase = cur_hypo.GetCurrTargetPhrase(); const WordsRange sourceWordRage = cur_hypo.GetCurrSourceWordsRange(); //source sentence Manager manager = cur_hypo.GetManager(); const Sentence source_sent = static_castconst Sentence(manager.GetSource()); //alignment const AlignmentInfo alignments = targetPhrase.GetAlignTerm(); I have also some questions about the states in the stateful features, what kind of variables should be stored in each state? only those ones that should be used in the compare function? or any variable from the previous hypothesis that we use in our feature? Normally, for stateful functions, for instance, previous target words will be stored. -- Cheers, Vu On Wed, Dec 10, 2014 at 4:11 PM, amir haghighi amir.haghighi...@gmail.com wrote: Hi everyone I'm implementing a feature function in moses-chart. I need the source words string and also their indexes in the source sentence. I've written a function that gets the source words but I don't know how extract word string from a word. could anyone guide me how to do that? as I know, each word is implemented as an array of factors, which of them is its string? I have also some questions about the states in the stateful features, what kind of variables should be stored in each state? only those ones that should be used in the compare function? or any variable from the previous hypothesis that we use in our feature? Thanks in advance! Cheers Amir ___ Moses-support mailing list Moses-support@mit.edu http://mailman.mit.edu/mailman/listinfo/moses-support ___ Moses-support mailing list Moses-support@mit.edu http://mailman.mit.edu/mailman/listinfo/moses-support
Re: [Moses-support] string of Words + states in feature functions
On 10/12/14 08:11, amir haghighi wrote: Hi everyone I'm implementing a feature function in moses-chart. I need the source words string and also their indexes in the source sentence. I've written a function that gets the source words but I don't know how extract word string from a word. could anyone guide me how to do that? as I know, each word is implemented as an array of factors, which of them is its string? I have also some questions about the states in the stateful features, what kind of variables should be stored in each state? only those ones that should be used in the compare function? or any variable from the previous hypothesis that we use in our feature? You can store anything in the state information that will help you to calculate function FFState::Compare(other) This is the only function you need to implement. The example SkeletonStatefulFF class compares the length of the target side of the translation rule from 2 hypotheses, therefore, it the state only need to store the length of the target string, not the entire string. It 2 states are the same Compare() should return 0. If 2 states are different, then it should return -1 or +1. you MUST make sure if, ie. if a b and b c then a c and if a b then b a Thanks in advance! Cheers Amir ___ Moses-support mailing list Moses-support@mit.edu http://mailman.mit.edu/mailman/listinfo/moses-support ___ Moses-support mailing list Moses-support@mit.edu http://mailman.mit.edu/mailman/listinfo/moses-support
Re: [Moses-support] string of Words + states in feature functions
Hi Amir, The input is passed to the feature functions via InitializeForInput(InputType const source). This method is called before search and collecting of translation options (cf. moses/FF/FeatureFunction.h). You can set a member variable to have access to the input in your scoring method. Alternatively, if you implement EvaluateWithSourceContext(), the input is passed directly to the method as a parameter (const InputType input) and you can use that. Finally, there's another option in the EvaluateWhenApplied() methods. You can get the input from the Hypothesis object: const InputType input = hypo.GetManager().GetSource(); The input is an InputType object. Moses knows different input types, see InputTypeEnum in moses/TypeDef.h . So what you get might differ depending on what was passed to the decoder. If you're happy with implementing your feature for sentence input only, then you can cast the input to a Sentence object. The Sentence object gives you convenient access methods, in particular GetSize() and GetWord(size_t pos). You can thus obtain the sequence of words in the input. Words can contain several factors in Moses. The factor with index 0 is typically the surface form. Access it using the [] operator. I guess you will never really want to work directly with the string representation of the factor, but at this point you would be able to get it and for instance print it to your debug output. Hope this was helpful as another answer to your first question. Cheers, Matthias On Wed, 2014-12-10 at 11:41 +0330, amir haghighi wrote: Hi everyone I'm implementing a feature function in moses-chart. I need the source words string and also their indexes in the source sentence. I've written a function that gets the source words but I don't know how extract word string from a word. could anyone guide me how to do that? as I know, each word is implemented as an array of factors, which of them is its string? I have also some questions about the states in the stateful features, what kind of variables should be stored in each state? only those ones that should be used in the compare function? or any variable from the previous hypothesis that we use in our feature? Thanks in advance! Cheers Amir ___ Moses-support mailing list Moses-support@mit.edu http://mailman.mit.edu/mailman/listinfo/moses-support -- The University of Edinburgh is a charitable body, registered in Scotland, with registration number SC005336. ___ Moses-support mailing list Moses-support@mit.edu http://mailman.mit.edu/mailman/listinfo/moses-support
