A new self-consistent projection-operator method for treating non-Markovian time evolution in the case of interactions between atoms and field (reservoir) is developed. It is shown that the application of this new method makes possible a complete and gauge-invariant solution of the long-standing problem of the spontaneously decaying unstable atomic state. All shortcomings of the Weisskopf-Wigner gauge-dependent method (Markov approximation and the neglect of virtual-transition, retardation and electron-spin effects) are removed. After deriving a gauge-invariant equation of motion for the decaying state, a new physical interaction picture (in which the unobservable interaction of the free electron with the vacuum radiation field is eliminated) is introduced. This new interaction picture leads automatically to a natural solution of the renormalization problem and thus finite Lamb-shift results without applying the conventional mass renormalization concept are explicitly calculated. Furthermore, in this picture explicit finite analytic results for the whole non-Markovian Lyman-α spontaneous decay are derived. Finally, a quasi-Lorentzian expression for the radiative line shape containing the correct (complete) finite non-relativistic frequency shift, stemming from the Lamb shifts of both levels 1S and 2P, is calculated.