Shape memory polymers (SMPs) have enormous potential in various applications. Some of them are biocompatible and biodegradable, and they can be used in biomedical engineering and tissue engineering. Light-sensitive, heat-responsive and electro-sensitive shape memory polymers have potential to be used in self-deployable structures and fields of automatic control. Liquid crystal elastomers with shape memory effect (SME) are expected to play an important role in artificial intelligence systems as man-made muscle. Shape memory hydrogels that are responsive to specific molecules, such as glucose, antigens or PH, can be used as biosensors as well as drug delivery systems. All of these are attractive enough to pursue a more extensive research of shape memory polymers. As is well known, performances of materials are determined by their architectures. Shape memory polymers are no exception. Shape memory effects of polymers must be ascribed to their structures.  From the viewpoint of structure, this review has summarized and commented on the latest advances in shape memory polymers. In short, thermoset shape memory polymers have chemical cross-linking and crystalline structures. Inversely, thermoplastic shape memory polymers have physical networks or entangled points. Shape memory effects of liquid crystal elastomers come from the phase transition of liquid crystal states. Swelling and deswelling functions of amphiphilic groups in molecule are responsible for shape memory effects of polymeric hydrogel. Cross-linked polyolefins, shape memory polyurethane and intelligent gels have come into our life, but, the development and improvement of some new polymers such as biomedical shape memory polymers and liquid crystal elastomers with practical properties is still a formidable challenge. If the achievements of the past can be extrapolated into the future, however, it is highly likely that new polymers with a wide array of desirable properties can be made.