Abstract: In order to overcome the problem that the general elastoplastic damage model cannot reflect the influence and nonlinear failure characteristic of rock mass, rock strength, and stress state, it established an elastoplastic damage model based on Hoek-Brown criterion, considering the coupling effect of two failure mechanisms, namely stiffness degradation due to damage and plastic flow. The modified effective stress principle is introduced to consider the effect of pore water pressure. The damage variable definition and evolution equation are developed. The fully implicit return mapping algorithm in the principal stress space is derived, using three steps:elastic trial, plastic correction and damage correction. The calculation progress is implemented in Umat of ABAQUS. Uniaxial and triaxial compression tests and tunnel examples are adopted to verify and analyze the model algorithm. The results show that the model can well describe the mechanical properties of rock mass materials, and the effect is satisfactory in the damage simulation of practical rock mass engineering, and the calculated results are of guiding significance to engineering.