The stability and dynamics of geological faults are often controlled by the frictional strength of the fault gouge, a granular layer that accumulates between the fault blocks due to wear. One of the main factors controlling the shear strength (as measured by an apparent friction coefficient) of such granular layers is the systems resistance to dilation, a by-process accompanying granular shear. This paper develops theoretical models connecting shear resistance and dilation during the different types of motions that occur in shear of uniform grains and compares theoretical predictions to results from two-dimensional discrete element simulations. We find that minimization of dilation-associated work (and thus minimization of apparent friction) is the factor that determines whether granular deformation will be localized or distributed and whether grains will predominantly slide or roll. Simulations of more realistic systems, with heterogeneous grain sizes, suggest that the apparent friction and dilation rate follow a physical picture consisting of two of the deformation mechanisms identified in shear of uniform grain systems: (1) shear localization onto a few layers of rolling grains and (2) brief compaction events in which dilated regions outside the shear zone compact. The first mechanism provides a nearly linear relationship between apparent friction and dilation rate. The second mechanism produces significant deviations from the linear relationship, with strongly negative dilation rates reflecting compaction events. Because observed compaction events occur outside of shear zones, dilation rate is not always simply related to apparent friction.