Abstract: Combined with Newmark-β integration scheme, an improved algorithm for iterative process, based on the finite element method and nonlinear contact theory, is presented and applied to analyze the dynamic response of a vehicle-track coupling system. For vehicle and track systems, vibration equations are established respectively. In the process of coupling and decoupling iteration, a relaxation factor function and a convergence criterion function are constructed, and the compatibility conditions at the wheel/rail interface are simplified. The wheel/rail interaction force is used to complete iterations in a rapid succession between two systems to realize the efficient solution for the coupling relationship. This algorithm improves the convergence precision of iteration and the stability of the iterative process. At the same time it reduces the difficulty of program design. The vibration responses of a coupling system caused by two kinds of line defects, the stepped irregularity and differential settlement of subgrade, are calculated using the proposed algorithm. The results show that the algorithm has advantages in iterative speed and iterative stability, and it can be widely used in vehicle-track coupled vibration analysis.