[This post is a followup to my previous post, Generative vs. discriminative; Bayesian vs. frequentist.] I had a brief chat with Andrew Gelman about the topic of generative vs. discriminative models. It came up when I was asking him why he didn't like the frequentist semicolon notation for variables that are not random. He said […]

A while back Bob blogged about The Latin1 Transcoding Trick for Java Servlets, etc. Suppose you have an API that insists on converting an as-yet-unseen stream of bytes to characters for you (e.g. servlets), but lets you set the character encoding if you want. Because Latin1 (officially, ISO-8859-1) maps bytes one-to-one to Unicode code points, […]

[Theres now a followup post, All Bayesian models are generative (in theory).] I was helping Boyi Xie get ready for his Ph.D. qualifying exams in computer science at Columbia and at one point I wrote the following diagram on the board to lay out the generative/discriminative and Bayesian/frequentist distinctions in what gets modeled. To keep […]

I had the hardest time trying to understand variational inference. All of the presentations I've seen (MacKay, Bishop, Wikipedia, Gelman's draft for the third edition of Bayesian Data Analysis) are deeply tied up with the details of a particular model being fit. I wanted to see the algorithm and get the big picture before being […]

Mitzi and I pitched O'Reilly books a revision of the Text Processing in Java book that she's been finishing off. The response from their editor was that they'd love to have an NLP book based on Java, but what we provided looked like everything-but-the-NLP you'd need for such a book. Insightful, these editors. That's exactly […]

Bayesian inference for LDA is intractable. And I mean really really deeply intractable in a way that nobody has figured or is ever likely to figure out how to solve. Before sending me a "but, but, but, …" reply, you might want to bone up on the technical definition of Bayesian inference, which is a […]

Johan Bos and his crew at University of Groningen have a new suite of games aimed at linguistic data data collection. You can find them at: http://www.wordrobe.org/ Wordrobe is currently hosting four games. Twins is aimed at part-of-speech tagging, Senses is for word sense annotation, Pointers for coref data, and Names for proper name classification. […]

Bernoulli Model Consider the following very simple model of drawing the components of a binary random N-vector y i.i.d. from a Bernoulli distribution with chance of success theta. data { int N; // number of items int y[N]; // binary outcome for item i } parameters { real theta; // Prob(y[n]=1) = theta } model […]

For those of you who like mystery novels, Mitzi's just written one. The added bonus for readers of this blog is that there's natural language processing involved in the detective work (I don't want to give too much away, so I can't tell you how). Poetic Justice is in the cozy mystery sub-genre, where the […]

I'm not a big fan of kappa statistics, to say the least. I point out several problems with kappa statistics right after the initial studies in this talk on annotation modeling. I just got back from another talk on annotation where I was ranting again about the uselessness of kappa. In particular, this blog post […]