Building information modeling (BIM) provides a wealth of space and attribute information for construction, and the development of 3D information acquisition technologies such as 3D laser scanning makes it more convenient and efficient to obtain 3D information of construction scenes. By integrating the point clouds of actual construction scenes and BIM, the automation of construction quality and progress management is expected to be improved. Existing studies generally compare the global information of point clouds with BIM, which needs further inspection and judgement by engineers based on different rules, and also makes it difficult to quantify the results. Meanwhile, the comparison results cannot be automatically matched with component properties in BIM and then serve intelligent analysis. The point cloud semantic segmentation using deep learning is introduced to realize the object-oriented comparison of components for the frame nodes, which increases the automation and digitization level of point-cloud-vs-BIM comparison. Synthetic experiments show that the adopted PointNet++ achieves high accuracy of semantic segmentation in the task, and has good robustness to data acquisition errors, which can provide a solid foundation for object-oriented construction deviation comparison. The object-oriented deviation comparison of point clouds with BIM realized by this method will have a promising prospect in the application to construction deviation visualization, measurement data digitization, and construction progress management.