Abstract: A novel prefabricated structural system, i.e. the frame structure with energy dissipating cladding panels, was proposed with an emphasis on the collaborative vibration reduction of the cladding panels and the main structure. To investigate the seismic performance of such structures, a multi-scale numerical simulation method was proposed based on a experiment. The collaborative mechanism of the vibration reduction was identified through an RC frame structure. The effects of the stiffness ratio and yield force ratio on such a mechanism were revealed based on 28 study cases. The results indicate that the multi-scale numerical simulation method can reflect the damage evolution mode and mechanical characteristics of such structures, and that the U-shaped dampers in the energy dissipating cladding panels yielded and started to dissipate energy before the main structure did. Therefore, the seismic response of the main structure and the damage to the beams and columns were effectively controlled without changing the damage evolution mode of the structure. With the increase of some critical design parameters (i.e., the stiffness ratio and yield force ratio), the damping effect on the inter-story drift ratio and the energy dissipated by the dampers was gradually increased. When such parameters reached certain values, the damping effect and the energy dissipated by the dampers were stabilized. The research results provided a useful reference for the development of high-performance structural systems with an emphasis on the collaboration of cladding panels and the main structure.