Abstract: Gear pair meshing and coupling vibration systems are multi-degree-of-freedom parametric vibration systems under the influence of internal and/or external linear and nonlinear factors. This paper investigates the dynamics of a spur gear meshing and coupling system with time-varying meshing stiffness, transmission error, and square nonlinearity. First, the equations of motion of a 4-DOF system are established and transformed into regular model equations. The multiple scale method is then used to analyze the parametric instability under main and subharmonic resonant conditions. The non-parametric resonant response and parametric resonant response are numerically simulated. The results reveal that the parametric resonance occurs as the gear meshing frequency approaches the combination frequency. The response can be boundless. The non-parametric resonant response is almost periodic, and contains various combined frequency components.