Yep, you've hit the nail right on the head. This is why I said my main concern would be the inordinate amounts of training data you would need to get something useful up and running.
When translating sentences from one language to another there can be a lot of variance but there can also be a lot of consistency at some level and so it possible to identify a limited number of patterns. The domain you are trying to train seems to me to be so much more open to variance that I would expect you would need much larger training sets and/or much more intelligent learning algorithms to be able to extract useful generalisations. Of course, I could be wrong. The only way to tell would be to suck it and see. We would need to set up some kind of empirical pipeline to train and test the system with varying amounts and types of data to see how it performs. I'm not sure how we would test such a system. I guess a quick approximation of performance of your translation model would be to see how highly the output sentences score on a well trained language model. This would give you an idea of how fluent the utterances generated are but would give you no idea of how appropriate a user would rate the responses. I guess you could use one of the bag of metrics to measure the distance of output sentences from responses in a test corpus. Again, I'm not sure how good a predictor of user judgements this would be. I suppose you could measure the average time a user is willing to chat with your bot to get an idea of how well it's performing. But if the output is particularly bad then some users may keep chatting with the bot just for the comical value. Have you got a system running yet? Could you show us some sample output? James ________________________________ From: Andrew [rave...@hotmail.com] Sent: 09 December 2013 21:46 To: Read, James C; moses-support@mit.edu Subject: RE: [Moses-support] using Moses in Monolingual dialogue setting Thanx for the insights. I've already done approach 2, and the result didn't seem bad to me, so I became curious if it would've made significant difference had I chosen the first approach. I was worried that approach 2 might've resulted in over-training, but judging from your comments, I guess it's only a matter of having broader entries. (or could it have been over-trained?) > I suppose my main concern would be the inordinate amounts of training data > you would need to get something useful up and running. This leads me to my next question. I trained my system with about 650k pairs of stimulus-response collected from Twitter. Each pair is part of a conversation which consists of 3~10 utterances. For example, suppose we have a conversation that has 4 utterances labeled A,B,C,D where A is the "root"of the conversation, and B is the response to A, C is the response to B, and D is the response to C. Following my second approach, A and B, B and C, C and D are pairs, so source file will contain A,B,C and target file will contain B,C,D, making 3 pairs from 1 conversation. In this way, I have 650k pairs from about 80k conversations. I've seen that when you use Moses for actual translation task, say German to English, the amount of training data seems pretty low, somewhere around 50K. So my 650k is already much bigger than this. However, in the paper that I mentioned http://aritter.github.io/mt_chat.pdf the author used about 1.3M pairs, which is twice bigger than mine, and I've seen research in similar setting http://www.aclweb.org/anthology/P13-1095 which used 4M pairs.(!) So, given the unpredictable nature of monolingual conversation setting, what would you think is the appropriate, or minimum amount of training data? And how much would the quality of the response-generation task depend on the amount of training data? I know this is out-of-nowhere question which may be hard to answer, but even a rough guess would great me assist me. Thank you very much in advance. > From: jcr...@essex.ac.uk > To: kgim...@cs.cmu.edu > Date: Mon, 9 Dec 2013 17:33:00 +0000 > CC: moses-support@mit.edu > Subject: Re: [Moses-support] using Moses in Monolingual dialogue setting > > I guess if you were to change the subject and ask a question from a list of > well formed common questions if the probability of the response is below some > sensible threshold then you could make a system which fools a user some of > the time. > > James > > ________________________________________ > From: moses-support-boun...@mit.edu [moses-support-boun...@mit.edu] on behalf > of Read, James C [jcr...@essex.ac.uk] > Sent: 09 December 2013 17:14 > To: Kevin Gimpel > Cc: moses-support@mit.edu > Subject: Re: [Moses-support] using Moses in Monolingual dialogue setting > > I'm guessing he wants to make a conversational agent that produces a most > likely response based on the stimulus. > > In any case, the distinction between 1 and 2 is probably redundant if GIZA++ > is being used to train in both directions. The two phrase tables could be > merged I guess. I guess the advantage of 2 over 1 is that you don't need to > worry about the merging logic at the cost of more training time. > > I'm not sure I understand the question of A1~B3. Unless I'm reading his > question wrong I don't see how this could happen. > > I suppose my main concern would be the inordinate amounts of training data > you would need to get something useful up and running. > > James > > ________________________________ > From: kgim...@gmail.com [kgim...@gmail.com] on behalf of Kevin Gimpel > [kgim...@cs.cmu.edu] > Sent: 09 December 2013 15:17 > To: Read, James C > Cc: Andrew; moses-support@mit.edu > Subject: Re: [Moses-support] using Moses in Monolingual dialogue setting > > Hi Andrew, it's an interesting idea.. I would guess that it would depend on > what the data look like. If the A's and B's are of fundamentally different > type (e.g., they are utterances in an automatic dialogue system, where A's > are always questions and B's are always responses), then approach 2 seems a > bit odd as it will conflate A's and B's utterances. However, if the A's and > B's are just part of a conversation, e.g., in IM chats, then they are of the > same "type" and approach 2 would make sense. In fact, I think approach 2 > would make more sense than approach 1 in that case. It also of course depends > on how you want to use the resulting translation system. > Kevin > > > > On Mon, Dec 9, 2013 at 5:18 AM, Read, James C > <jcr...@essex.ac.uk<mailto:jcr...@essex.ac.uk>> wrote: > Are you trying to figure out the probability of a response given a stimulus? > > Given that GIZA++ aligns words and makes heavy use of co-occurrence > statistics I doubt this is likely to produce very fruitful results. How big > is your data set? > > Give it a whirl and see what happens. I would be interested to hear what > comes of it? > > James > > ________________________________ > From: moses-support-boun...@mit.edu<mailto:moses-support-boun...@mit.edu> > [moses-support-boun...@mit.edu<mailto:moses-support-boun...@mit.edu>] on > behalf of Andrew [rave...@hotmail.com<mailto:rave...@hotmail.com>] > Sent: 08 December 2013 20:10 > To: moses-support@mit.edu<mailto:moses-support@mit.edu> > Subject: [Moses-support] using Moses in Monolingual dialogue setting > > > Hi, > > > I'm using Moses in monolingual dialogue setting as in > http://aritter.github.io/mt_chat.pdf, > where source and target are both in English and target is a response to > source. > I'd like to propose a little thought experiment in this setting, and hear > what you think would happen. > > > Suppose we have a conversation with six utterances, A1,B1,A2,B2,A3,B3 where A > and B indicate speakers, > and the number indicates n-th statement by the speaker. They are all in one > conversation of continuous topic. > > Now suppose we train it using Moses in two different ways as following: > 1) Source file contains A1, A2, A3 and target contains B1, B2, B3 so that > A1-B1 is a pair and so on. > 2) Source contains A1,B1,A2,B2,A3 and target contains B1,A2,B2,A3,B3, taking > advantage of the fact that response is a stimulus to the next response. > > Then, How will the results be different and why? > Since GIZA++ gets alignment in both directions, will 2) result in any of > A1~B3 being the translation of any other? > > > This may be a strange question, but I would really like to get your insight. > > > _______________________________________________ > Moses-support mailing list > Moses-support@mit.edu<mailto:Moses-support@mit.edu> > http://mailman.mit.edu/mailman/listinfo/moses-support > > > _______________________________________________ > Moses-support mailing list > Moses-support@mit.edu > http://mailman.mit.edu/mailman/listinfo/moses-support > > _______________________________________________ > Moses-support mailing list > Moses-support@mit.edu > http://mailman.mit.edu/mailman/listinfo/moses-support _______________________________________________ Moses-support mailing list Moses-support@mit.edu http://mailman.mit.edu/mailman/listinfo/moses-support
