Effects of magnetization on Young’s modulus of natural polymer, carrageenan and agar, gels and polyurethane elastomers have been investigated by dynamic viscoelastic measurements. The carrageenan magnetic gel exhibited a large nonlinear viscoelastic response suggesting the magnetic particles connect each other with a weak force. The storage modulus above a volume fraction of 0.16 significantly increased with the volume fraction, which is not satisfying the Krieger-Dougherty equation. This strongly suggests that the magnetic particles come in contact with each other and make a structure with high modulus. The carrageenan gel demonstrated the giant and negative magnetorheological effect due to magnetization. At maximum, the modulus reduction exceeded 10⁷ Pa, which was much larger than the theoretical value by 10⁴ times. Both of the nonlinear viscoelasticity and the magnetostriction with an order of 10^-3 may bring the giant and negative magnetorheological effect. Similar phenomenon, the Young’s modulus decreased by ~10⁵ Pa due to magnetization, was observed also in the soft magnetic materials with matrices of agar and polyurethane. The gel and elastomer demonstrated an enhanced viscoelasticity, high Young’s modulus, and magnetostriction as well as carrageenan magnetic gels. Both the viscoelastic feature characterized by the Payne effect and  the macroscopic deformation induced by the magnetization may bring the giant and negative magnetorheological effect.