Most present applications of time-dependent density functional theory use adiabatic functionals, i.e., the effective potential at time r is determined solely by the density at the same time. This paper discusses a method that aims to go beyond this approximation, by incorporating "memory" effects: the derived exchange-correlation potential will depend not only on present densities but also on the past. In order to ensure the potentials are causal, we formulate the action on the Keldysh contour for electrons in electromagnetic fields, from which we derive suitable Kohn-Sham equations. The exchange-correlation action is now a functional of the electron density and velocity field. A specific action functional is constructed which is Galilean invariant and yields a causal exchange-correlation vector potential for the Kohn-Sham equations incorporating memory effects. We show explicitly that the net exchange-correlation Lorentz force is zero. The potential is consistent with known dynamical properties of the homogeneous electron gas (in the linear response limit).