Abstract: A theoretical analysis of localized strain and strain rate in shear band after localization occurs beyond the peak shear stress is carried out based on the strain gradient plasticity theory. The influences of the shear descending modulus, the internal length parameter of rock material, the differential stress and shear stress rate on shear localization are investigated. A theoretical(or constitutive)relation between flow shear stress and relative sliding displacement of shear band is deduced. A constitutive relation between flow shear stress rate and the relative sliding velocity of shear band is also obtained. The influences of the shear descending modulus and the internal length parameter of rock material on instability of shear band are analyzed. The direct shear test machine is simplified as a system composed of a rock specimen(i.e. shear band)and a steel block. The energy criterion is applied to the instability analysis of the system and a criterion of instability is put forward. If the shear descending modulus of rock and the equivalent length of the steel block is low or the internal length parameter of rock material and the shear stiffness of the steel block is large, then shear band can slide easily along its tangential direction, i.e., the system is prone to instability.