Abstract: Numerous achievements have been made in the research of conventional fracture detection under a single background, but the study on preventive deformation monitoring of cracks under complex backgrounds in long-term conditions is still at its early stages. Considering that the mural walls of ancient buildings have the characteristics of micro-deformation and non-interference, a non-contact and preventive deformation monitoring system has been established. This system is based on computer vision technology, including traditional image segmentation processing technology, as well as a neural network model for semantic segmentation of crack images. In order to reduce the interference of color-painted murals, environmental lighting, and noise that are unique to cracks in mural walls and effectively separate the cracks from the complex background, traditional threshold segmentation algorithm systems and intelligent semantic segmentation systems are studied. The former system utilizes feature matching of scale-invariant feature transform and the solution of homography matrix to solve the problem of image perspective differences. Meanwhile, by comparing different filtering algorithms, a monitoring algorithm is selected, which combines bilateral filtering and threshold segmentation that is more suitable for mural crack segmentation. The latter system uses multi-layer convolution, sampling and splicing operations to remove redundant features, reconstruct the high-level semantic feature map of cracks, and select multi-step optimization strategy to improve the original U-Net model network architecture, which improves the average accuracy of model to 0.9899. The crack of typical mural walls with stone masonry material of Tibetan ancient buildings has been monitored for 12 days, and relevant characteristic parameters such as crack contours and crack skeleton lines are extracted as key indicators to quantitatively describe the changing information during the crack growth process. The results show that the coefficient of variation (COV) of the two-dimensional feature indicators (such as crack area, density, and width) obtained by the traditional threshold segmentation algorithm system ranges from 4.50% to 6.52%. However, the improved U-Net model reduces the COV of the crack area feature parameter, which has the largest data fluctuation in the traditional threshold segmentation system, from 6.52% to 3.53%, and thus improves the monitoring system's robustness to mural color, lighting, and shadow interference. Both algorithms process 12 images of the same crack image from different perspectives, and the output data is uniformly consistent, with a COV not exceeding 7%. These results demonstrate the technical feasibility of the monitoring system for real-time, non-destructive monitoring of long-term deformation of mural wall cracks.