But it has a few incredibly annoying weaknesses. The worst of these is an abysmal indentation model. (The second worst is terrible support for large files and projects, but I haven’t solved that yet.)

Happily, I ran across this nice post to indent R files using Emacs in batch mode from Textmate. A very simple and effective idea (Emacs afficionados are of course scoffing, why not use Emacs? Why not? Because I still haven’t found the button to zap-phrase-long-counter-intuitive-command-names-out-of-the-1970s-and-into-a-modern-ui).

I did a reworking in Ruby that can handle any file type for which an Emacs mode is installed, which is specified as a command-line argument to the script. So just put the Ruby script somewhere (it probably should go in a Textmate support directory, but I just put it in a personal script dir), and then create a ‘Tidy’ command in Textmate for each desired language, specifying an appropriate file ending to tell Emacs the language. For example:

/path/to/tidy.rb cpp   # C++

/path/to/tidy.rb java   # Java

/path/to/tidy.rb ml      # OCaml

Voilà!

Again, the script (you’ll need to rename and make executable): tidy.rb

(Note: The p2 patch adds header installation to the earlier version.)

The terrible thing about being leftward-leaning in this country is to watch the quadrennial self-destruction of the Democratic Party, nominating yet another New England intellectual who can dish off stacks of well reasoned policy documents, but can’t connect with anyone.

The one living exception is Bill, but we’ve been reminded all too well in the past few weeks of his character and the divisive environment he helps create and thrives upon. And for those nostalgic for the Clinton Nineties, are they really worth reliving? Or just better than the past seven years?

We’ve been told for at least four years that Hillary was coming, and that she was inevitable, which I resented right off the bat. Add to that the fact that, while she certainly possesses her husband’s brains and probably better policy, she possesses none of his charisma and common touch, but has definitely acquired a good measure of his sleaze and self-righteousness (ok, maybe she already had some of the latter) through years of osmosis and simple association. At times they both seem indignant and even incredulous that anyone would dare challenge their birthright to return to the throne.

Now, we as the Good Democratic Herd are supposed to make the usual calculations and triangulations and support The Party’s Anointed. Vote HHH, not RFK. Mondale and Dukakis, Gore and Kerry and … Billary. And while some years there have been other, more inspiring candidates, who may or may not have done better in November, the Fear always whispered, “No, too X, too Y. We need neutral, we need safe.”

Well, we may not win this one, either. I’m not saying Obama is flawless or the Chosen One, but this is the first candidate in my life that I’m genuinely excited about, not just tacking away from this or that. And so I’m elated that this month, finally, millions of others are feeling the permission to dream:

From a morphology perspective, however, we might say that the text is over-segmented into a string of morphs, so that our task is to group morphs into chunks of words. The nice aspect of this view is that it falls nicely under what I would call sequential models of morphological analysis for non-isolating languages, where we have a lattice of possible segmentations (from some hypothesis generator) and we aim to find the best path. The isolating case just has a more constrained graph topology, with N+1 states for the N tokens and possible arcs limited by the maximum token length of words in the languages.

It’s also interesting that we run into the same type of problem here as with the analysis of non-isolating languages. Specifically, if the arc weights are something like probabilities, we will be biased toward longer words and thus fewer arcs, unless the costs of longer arcs are suitably balanced. With token classes and some labeled training data, we certainly might build highly accurate models. For a more unsupervised approach, however, seeking some intrinsic measure for an optimal segmentation, we run into the usual issue that there is no universally correct trade-off between coarse and fine analyses. We are always left with parameters, hidden or overt, that guide our models one way or another toward our preferences, which are, to a large degree, quite arbitrary.

Except for Kile. Which still sucks since it doesn’t have WYSIWYG or at least real-time preview, but at least they figured out that dynamic command completion is really frigging useful. So that’s what I use.

But there’s one big problem: I can’t copy from a Mac application to Kile. Or any KDE application for that matter. As the following post discusses

http://lists.macosforge.org/pipermail/macports-users/2007-July/004451.html

you can paste to other X11 applications using the mouse middle button (option-click) (see http://developer.apple.com/qa/qa2001/qa1232.html), but it doesn’t work for KDE apps. Unfortunately the post hasn’t been answered, but I finally figured out a not-so-odious work-around: Just run xclipboard.

Yes, old, crusty, xclipboard in all of its Athena widget glory scoops up your native Mac copying activity and magically makes it available to KDE applications with the usual Control-v.

So that’s why today’s UNIX GUI distros still dump hundreds of old and often redundant X applications. Sometimes you actually need them!

I’ve banned myself from working on morphology learning right now, because it’s clearly risky, with a lot of work needed, and that looming January 10th ACL deadline says I need to get cracking on the morphemic MT paper until it’s nearing a finished state. But I feel anxious to spend some time at least thinking about the morphology, so I’ll grant an exception…

(On the other hand, the MT paper looks more hopeful for EMNLP than ACL; the morphology work is more of an ACL flavor, but I’m losing confidence that it will be ready in time.)

As I’ve already noted, the decision to do morphology is fraught with a different kind of peril compared to the quixotic grammar work. While the morphology is much more likely to achieve results in the nearer term, it is also much more likely to achieve similar results for others, and in fact there have been a gazillion papers on this in the past 8-10 years. I need to do more prior-work reading, but it seems probable that even the weakly supervised space is a little crowded already. And the state-of-the-art unsupervised systems are crafted by hardy Finns who live and breathe agglutinative morphology every day, and if my (cheating) system can’t even match that, well, why bother?

Contingent on doing more background reading, I think the following areas are important to focus on:

• Model analysis
  • It looks like (Creutz and Lagus, 2007) have done a good job analyzing the different types of information that models need to capture, but there may be some room left, especially if it leads to a successful model in the end.
  • One major issue for generative modeling: Differing entropy of affix space distributions for different classes. Hack solution is to scale affix probabilities where there are more allowed values, or perhaps some Bayesian technique. Another possibility is to model the letters directly, and their likelihood to generate a separate morpheme, so that, e.g. we assign a likelihood to the “ly” suffix being part of the stem given after the characters “sal” vs. “slow.”
• Global effects
  • Closely related is the issue of how to model the more global aspects of a morphological system, e.g. the frequency ratios, expectation that a stem of one class will not be seen with morphemes of another class.
  • It is possible that the (successful) use of contrastive estimation would be the clincher for the work, perhaps with the addition of support sets (though this needs better theoretical justification).
• Joint tagging and analysis
  • It seems that much of the focus in morphology learning is on providing the correct segmentation, without identifying the type of the word and the functional roles of the segmented morphemes.
  • Having now finished the (Creutz and Lagus, 2007) paper, it’s clear that (a) state-of-the-art unsupervised morphology is still a large number of heuristics generally tuned for a few languages (here, Finnish); and (b) the focus is very much on segmentation of the surface characters and less on identifying the word type(s), affix functions, etc. So it’s good that this is my focus.
• Morphology and applications
  • A good deal of the motivation for my approach is for the practical use of morphology in applications such as MT, parsing, and (need to add) IR.
  • Aside from showing performance improvements with morphology, the thing to emphasize is the importance of adapting the morphological analysis to the problem, both in the structure (how to segment words) and the amount (which types of words and affixes to segment). Good to do: Apply two different segmentation structures–one traditional linguistic, one not–to a problem and compare results.

(Boobear woke up before I finished this yesterday. This is enough for now.)

Put more cynically, should I cling to that last idealistic drop of PhD motivation in my body, the drive to do something novel and exciting, the last, tenuous hope for a home run that will make the last 9 1/2 innings of drudgery seem worthwhile? Or just accept that those dreams are done and that now all I want is the paper reward, that piece of parchment suitable for framing and the little acronym that says: resistiré.

The Dreamer declaims the following:

1. These ideas are exciting! They are novel, with nice linguistic foundations (albeit unorthodox), and could be a strong development in unsupervised and low-resource grammar learning, and in MT.
2. The high bar for the quals are a bit self-imposed.
   1. The two morphology chapters plus the syntax smoothing (probably feasible for the spring) are sufficient for the quals, so I can still meet that deadline.
   2. With the smoothing completed, the grammar transformations are mostly done.
   3. Then I spend most of the final year on unsupervised learning, with the MT results limited to the most straightforward applications of it.
3. If not syntax, what then? What novel work would you do in morphology to fill out a thesis? Especially since everyone and their cousin has taken a pass at it!

To which The Pragmatist retorts:

1. They are exciting, but extremely speculative and risky. If you’d developed them in year two or even three, that would have been a great time to try something big. But we’re starting year five now, and it’s time to finish, not to finesse.
2. Yes, but then you push more work to do after the quals, and do you really want to be here past May 2009?
   1. A bit hopeful, assuming mountains of SpeechLinks work doesn’t come crashing down, also no chance for a COLING paper, because it’s pretty clear that the current papers will occupy me fully through January 10th.
   2. Yes, but again no small piece of work. 6 months is a safe estimate, so that takes us through the NAACL deadline, without starting on the unsupervised learning, which is harder!
   3. I’d call it 18 months after the quals. Want to stay through December?
3. Ah, you have me there a bit, but I can come up with something. Just watch me….

The Pragmatist’s Plan B

The mantra of Plan B is this: My hammer is morphology, so what can I nail? Put more concretely, we find languages with interesting morphology, model them, and evaluate the performance of the segmentation and its use in applications.

A little brainstorming:

1. Models
   1. Basic concatenative morphology
   2. Functional and surface affixes
   3. Agglutinative morphology
   4. Template morphology
   5. Affix ordering (e.g. Chingtang)
   6. Phonological modeling
2. Learning
   1. EM and variants
   2. Log-linear: contrastive, etc.
   3. Full semi-supervised, i.e. combine small supervised model with unsupervised.
   4. Using Y&W approaches to discover patterns, in addition to specification.
3. MT
   1. Segmentation and training techniques (IN PROGRESS).
   2. Factored morphological models.
   3. Translation with isolating language, mapping morphemes to particles (Vietnamese, Chinese).
4. Other applications
   1. Dependency parsing? Train McDonald on segmented data.
   2. Joint tagging and segmentation.

Now let’s translate this into chapters for the quals:

1. Models of Morphological Systems
2. Learning Morphology
   1. EM and variants
   2. Contrastive estimation
   3. Partial specification (Y&W proposals)
3. Translating Morphemes
   1. Persian and Spanish, maybe Czech (but what to offer over G&M).
4. Proposed: Factored Translation
   1. Basic improvements
   2. Isolating languages: identifying particles, etc.
5. Proposed: Other Applications
   1. Joint tagging
   2. Dependency parsing

I’m not sure I like this division of the work, especially separating the models from the learning. Let’s try another:

1. Learning Morphology by Specification
   1. Basic concatenative model, with stem change, orders 1-3.
   2. EM and variants.
   3. Contrastive estimation (note: this requires generation!)
2. Beyond Concatenative Morphology
   1. Agglutinative, templatic, permutative.
   2. Functional and surface affix modeling.
   3. Phonological modeling.
3. Translating Morphemes
   1. Persian work, with patterns and above models.
   2. Some other exciting language(s).
4. Proposed: Learning from Partial Specifications
   1. Use hypotheses proposals from Y&W models and others, extend specification.
   2. Also semi-supervised training, with small annotated data set.
5. Proposed: Factored Translation
   1. Basic improvements on morpheme translation.
   2. Isolating languages: Identifying particles, etc.
6. Proposed: Other Applications
   1. Dependency parsing of morphemes (McDonald plus certain attachment constraints).
   2. Joint tagging and segmentation (anything really to be done here?).

Ok, I think that’s better. Call it a plan.

The current list of modeling and learning possibilities:

1. EM.
   • pro: Simple; flows from nice, generative model.
   • con: Greatly biased toward classes with fewer inflections (especially atomic), since the affix vector distribution is less sparse. Different hypotheses within a (class, stem) group do contribute to each other’s likelihood, but this doesn’t translate across classes.
   • potential extensions:
     1. Putting a prior on classes may help, and very simple to try. Could go Bayesian, but I am skeptical of how much more that will help.
     2. Need to experiment with initialization.
     3. Higher-order model will help, but trickier in the setting where we analyze only a few word classes and assume all other words are atomic, which makes prior context less informative. One solution for that might be hierarchical class splitting of atomics, cf. Petrov.
     4. Try to model some of the more global effects that we want. To balance the varying affix sparseness, we might want to focus on maximizing the (class, stem) likelihood, and after that, we have differing expectations of a specific affix vector based on the size of the space for that class. There’s also a global expectation that the ratios between different affix values will remain roughly constant across a class. Not sure how to model these effects with a pure stochastic model; a really scary option is to try to adapt M-estimation to unsupervised learning and use the EM model as q₀.
2. Contrastive Estimation
   • pro: Allows arbitrary features, in principle may avoid the sparse affix space problem.
   • con: The only obvious neighborhood function, using inflections of the (class, stem) hypotheses, will generate many valid words. Uncertain how this will work, then, especially having the tagging problem at the same time.
   • possible extensions:
     1. Initialization and regularization will be very important.
     2. Multiple neighborhoods, maximized jointly.
     3. Alternate objective functions?
     4. Adding support and/or conflict sets.

First, we define some concepts:

• stem
  The underlying base of a non-compound word, possibly corresponding to the word’s lemma form or root, but we view the choice of stem as a modeling decision. We use the word “stem” and not “root,” to include more shallow types of analysis that may seek only to identify the affixes (and what’s left behind), and not to recover the fundamental word root.
• stem class
  The type of the stem, often related to the familiar notion of part-of-speech, which determines which affixes may join with it. For more limited morphological analysis, we may constrain the stem and final word classes to be equal.
• surface affix
  What is normally referred to as an affix, realized as zero or more characters of text.
• functional affix
  An abstract affix that expresses a single underlying function of a surface affix, e.g. Number, Person, Tense, Aspect, etc., which may determine the final word class. Note that multiple functional affixes often map to a single surface affix.
• affix vector
  A value in the space (either surface or functional) of all possible affix combinations allowed in a language.
• affix position
  A set of one or more affixes that are mutually dependent and thus modeled jointly, occupying a single index in an affix vector. The concept of a position is also useful for specifying the mapping between functional and surface affixes. For example, in Spanish we may have multiple function positions for person, tense, etc. mapping to a single, “verb suffix” surface position.

Word Formation

Next, we describe a simplified model of word formation that, as suggested above, ignores stem-to-word class changes from affixes, which is sufficient for many applications, such as smoothing to address data sparsity. This framework allows us to model, for example, “working” as verbal stem “work” plus progressive suffix “ing,” but not “worker” as “work” plus nominalizing suffix “er.” Voilà:

1. Select a word class t, given the utterance context.
2. Select a word stem s in t, given the utterance context.
3. Select a vector of functional affixes f, given (t, s) and the utterance context.
4. Given (t, s, f), select a stem transformation (possibly identity) to produce the final stem s’.
5. Given (t, s’, f), map f to a surface affix vector a.
6. Generate the final word from w from (t, s’, a). For most familiar languages, this process is deterministic, but there are languages that allow some affix permutations with each other and even the stem.

Now we extend this process with a few modifications to handle affixes that change the class of the stem:

1. Select a word class t, given the utterance context.
2. Select a stem class c such that t is derivable, that is, c admits one or more affix vectors that will produce a final word class t when adjoined to a stem of class c.
3. Select a word stem s in c, given the utterance context.
4. Select a vector of functional affixes f, given (t, c, s) and the utterance context, such that f maps the stem class c to the final word class t.
5. Given (t, c, s, f), select a stem transformation (possibly identity) to produce the final stem s’.
6. Given (t, c, s’, f), map f to a surface affix vector a.
7. Generate the final word from w from (t, c, s’, a).

Desiderata

Finally, the promised desiderata for a morphology learner:

1. If an analysis for word w with the stem (c, s) is correct, we expect to observe a number of other words that also have an analysis with (c, s), but with different, valid affix vectors.
2. Conversely, the hypothesis that a word is atomic (unanalyzed) is an implicit claim that the word will not appear as the stem in other words with affixes.
3. Homographs make things more difficult, as one word token may appear with different classes, some atomic and some not, e.g. “get caught up in the ins and outs of something” or Persian “(در در(ها” (”dr dr(hA)” = “at the door(s)”, an atomic preposition plus inflecting noun). Thus context must be part of the model and learning process, and a word’s marginal hypothesis distribution should never be too skewed, especially between atomic and analyzed classes.
4. It would be nice to be able to integrate a discovery process, such as the method of Yarowski and Wicentowski, with (possibly partial) knowledge of the morphological system.
5. Another aspect of Y&W that would be nice: Modeling the expectation of regularity over the relative frequency between different related word pairs, e.g. we expect count(walk)/count(walking) to be roughly equal to count(singe)/count(singeing).