Abstract: The non-linear dynamics and control of friction-induced vibration in a brake system are investigated. Adopting the LuGre friction model, the dynamical equations of a 2-DOF disc brake system are derived by considering the friction force between the disc and the pad. The critical speed of Hopf bifurcation of the equilibrium is obtained through stability analysis. The design method based on differential geometry and linear quadratic optimal control for SISO nonlinear system controller is used to reduce the chatter phenomena during deceleration-type braking by postponing the critical speed of Hopf bifurcation. At last, the influences of the controller and the system parameters are analyzed. The simulation result shows that the method based on differential geometry and linear quadratic optimal is effective to control the self-excited vibration.