Abstract: Stability of the system (rock specimen in uniaxial compression) composed of inclined shear band and elastic rock outside the band was analyzed in terms of energy principle. Axial deformation of the specimen is decomposed into two parts. One is due to the compression of elastic rock; the other is induced by the shear slip along shear band. The latter is related to the relative shear deformation between the top and base of shear band through a simple geometrical relation. Total potential energy is composed of elastic and dissipated potential energies in shear band as well as work done by external force. Potential energies in the band depend on the volume of shear band. The thickness of the band is determined by gradient-dependent plasticity. The first-order derivative equal to zero of total potential energy with respect to the relative shear deformation of shear band leads to the equilibrium condition of elastic rock. The second-order derivative less than zero of total potential energy with respect to the relative shear deformation of shear band results in the unstable criterion of the system. The present unstable criterion can reflect the influences of the constitutive parameters of rock in elastic and strain-softening stages, the volumes of elastic rock and shear band as well as the structural form on the stability of the system. The present analytical unstable criterion is strict and accurate, which is a generalization from existing analytical unstable criterion by Wang et al.