Hidden topic Markov Models

Amit Gruber*, Michal Rosen-Zvi, Yair Weiss

*Corresponding author for this work

Research output: Contribution to journalConference articlepeer-review

115 Scopus citations


Algorithms such as Latent Dirichlet Allocation (LDA) have achieved significant progress in modeling word document relationships. These algorithms assume each word in the document was generated by a hidden topic and explicitly model the word distribution of each topic as well as the prior distribution over topics in the document. Given these parameters, the topics of all words in the same document are assumed to be independent. In this paper, we propose modeling the topics of words in the document as a Markov chain. Specifically, we assume that all words in the same sentence have the same topic, and successive sentences are more likely to have the same topics. Since the topics are hidden, this leads to using the well-known tools of Hidden Markov Models for learning and inference. We show that incorporating this dependency allows us to learn better topics and to disambiguate words that can belong to different topics. Quantitatively, we show that we obtain better perplexity in modeling documents with only a modest increase in learning and inference complexity.

Original languageEnglish
Pages (from-to)163-170
Number of pages8
JournalProceedings of Machine Learning Research
StatePublished - 2007
Event11th International Conference on Artificial Intelligence and Statistics, AISTATS 2007 - San Juan, Puerto Rico
Duration: 21 Mar 200724 Mar 2007

Cite this