In the past decade, great strides have been made to quantify the dynamics of single-cell growth and division in microbes. In order to make sense of the evolutionary history of these organisms, we must understand how features of single-cell growth and division influence evolutionary dynamics. This requires us to connect processes on the single-cell scale to population dynamics. Here, we consider a model of microbial growth in finite populations which explicitly incorporates the single-cell dynamics. We study the behavior of a mutant population in such a model and ask: can the evolutionary dynamics be coarse-grained so that the forces of natural selection and genetic drift can be expressed in terms of the long-term fitness? We show that it is in fact not possible, as there is no way to define a single fitness parameter (or reproductive rate) that defines the fate of an organism even in a constant environment. This is due to fluctuations in the population averaged division rate. As a result, various details of the single-cell dynamics affect the fate of a new mutant independently from how they affect the long-term growth rate of the mutant population. In particular, we show that in the case of neutral mutations, variability in generation times increases the rate of genetic drift, and in the case of beneficial mutations, variability decreases its fixation probability. Furthermore, we explain the source of the persistent division rate fluctuations and provide analytic solutions for the fixation probability as a multi-species generalization of the Euler-Lotka equation.