Abstract: Vibration displacement is one of the most significant indicators in structural health monitoring and condition assessment of bridges. The traditional monitoring means are relatively expensive and have limited measuring points for displacement measurement of bridges. This study presents a low-cost, non-contact monocular vision system with the KCF-based algorithm. A calculation method for the geometric scale factor was established by considering different depths of the field to cope with the loss of depth information in images when a monocular camera is used to track multiple targets. Shaking table tests were conducted when a double-column pier with buckling-restrained braces (BRBs) was subjected to seismic waves with different amplitudes and frequencies. The feasibility, accuracy, effectiveness and robustness of a monocular vision system with the KCF-based algorithm were validated by referring to the displacement recorded by the laser displacement sensor. The results show that the monocular vision system with the KCF-based algorithm can effectively and accurately identify the vibration displacement of the double-column pier with BRBs. Compared with the peak displacement measured by laser displacement sensors under different earthquake amplitudes and frequencies, the errors identified by the KCF algorithm are within 6%. The root mean square errors of the displacement time history measured by the KCF algorithm and laser displacement sensor are slight. The spectral characteristics of identifying displacement based on the KCF algorithm are consistent with those recorded by laser displacement sensors.