This work addresses the question of the intimate coupling of plastic and damaging processes during the deformation of semi-crystalline polymers at small strains. The evolution of the spherulitic structure in the pre-yield strain range under tensile testing is investigated by atomic force microscopy for three semi-crystalline polymers, namely polycaprolactone, poly(1-butene) and polyamide 6. These materials have different spherulite size, crystallinity index and lamella thickness, and different glass transition temperature of the amorphous phase. Strain-induced damage is clearly evidenced through the gradual loss of elastic properties upon cyclic tensile tests, since the early stage of stretching. In parallel, volume strain appears to be about nil up to the yield point for the three polymers. AFM reveals that fragmentation of the crystalline lamellae occurs well before the yield strain at room temperature, starting about the core region of the spherulites and extending towards the periphery, for all polymers. This is claimed as evidence that lamella fragmentation is a basic mechanism of damage without significant cavitation at low strain. An approach of damage modeling is carried out via preliminary assessment of the viscoelastic contribution from low strain dynamic mechanical analysis using a generalized Maxwell model. It is shown that computing the viscoelastic contribution in the strain range up to yielding, in the assumption of linearity, fairly account for the loading-unloading hysteresis of the tensile cycles. A phenomenological plasticity/damage coupling law is established from the elastic modulus drop with increasing plastic strain, both assessed from the "relaxed" tensile cycles. The same kind of law is shown to apply for the three polymers. A physical meaning to the phenomenological law is proposed via a simple model of fiber rupture in single-fiber-reinforced composite.