i/o

Like every aspect of human languages (or maybe better, human intelligence), modeling morphology has turned out to be much trickier than at first thought. Thus the extensive list of background work…

The current list of modeling and learning possibilities:

1. EM.
   • pro: Simple; flows from nice, generative model.
   • con: Greatly biased toward classes with fewer inflections (especially atomic), since the affix vector distribution is less sparse. Different hypotheses within a (class, stem) group do contribute to each other’s likelihood, but this doesn’t translate across classes.
   • potential extensions:
     1. Putting a prior on classes may help, and very simple to try. Could go Bayesian, but I am skeptical of how much more that will help.
     2. Need to experiment with initialization.
     3. Higher-order model will help, but trickier in the setting where we analyze only a few word classes and assume all other words are atomic, which makes prior context less informative. One solution for that might be hierarchical class splitting of atomics, cf. Petrov.
     4. Try to model some of the more global effects that we want. To balance the varying affix sparseness, we might want to focus on maximizing the (class, stem) likelihood, and after that, we have differing expectations of a specific affix vector based on the size of the space for that class. There’s also a global expectation that the ratios between different affix values will remain roughly constant across a class. Not sure how to model these effects with a pure stochastic model; a really scary option is to try to adapt M-estimation to unsupervised learning and use the EM model as q₀.
2. Contrastive Estimation
   • pro: Allows arbitrary features, in principle may avoid the sparse affix space problem.
   • con: The only obvious neighborhood function, using inflections of the (class, stem) hypotheses, will generate many valid words. Uncertain how this will work, then, especially having the tagging problem at the same time.
   • possible extensions:
     1. Initialization and regularization will be very important.
     2. Multiple neighborhoods, maximized jointly.
     3. Alternate objective functions?
     4. Adding support and/or conflict sets.

This entry was posted on Monday, October 29th, 2007 at 10:42 am and is filed under morfo. You can follow any responses to this entry through the RSS 2.0 feed. You can leave a response, or trackback from your own site.