Abstract: The stability of artificial boundary condition is the premise to obtain reliable simulation results in infinite wave simulation. The numerical schemes combining spectral element method and transmitting boundary with high order accuracy show better numerical accuracy and stability, but there still exits instability problem. The mechanism of instability and stability condition of transmitting boundary is still not known, and the corresponding theoretical analysis is extremely lacking. Aiming at the stability of transmitting boundary in spectral element method, according to the periodic extension property of the spectral element nodes that are not evenly spaced, the vector forms of spectral element method and transmitting boundary numerical scheme are presented. Based on the vector form, the reflection coefficient of transmitting boundary is deduced. Then the stability condition of transmitting boundary is obtained by guaranteeing the reflection coefficients of the real mode and all spurious modes in spectral element method are less than or equal to one. The stability condition shows the relationship between dimensionless parameter of transmitting boundary and that of spectral element method, that means the ratio of the artificial wave velocity to the physical wave velocity is limited to a certain range. Meanwhile, it is revealed that the mechanism of instability is the multi-reflection amplification of the spurious modes caused by the artificial boundary in the finite calculation area. Finally, the stability condition of transmitting boundary is verified by numerical experiments.