Logistic Markov decision processes

User modeling in advertising and recommendation has typically focused on myopic predictors of user responses. In this work, we consider the long-term decision problem associated with user interaction. We propose a concise specification of long-term interaction dynamics by combining factored dynamic Bayesian networks with logistic predictors of user responses, allowing state-of-the-art prediction models to be seamlessly extended. We show how to solve such models at scale by providing a constraint generation approach for approximate linear programming that overcomes the variable coupling and nonlinearity induced by the logistic regression predictor. The efficacy of the approach is demonstrated on advertising domains with up to 2⁵⁴ states and 2³⁹ actions.