Abstract: Based on the principle of minimum potential energy, the helical buckling of a drillstring is studied, with its top end free circumferentially and lower end hinged. To simplify the problem from contact nonlinear to geometric nonlinear, cylindrical coordinate system is used in the derivations. Using Rayleigh-Ritz method, the non-dimensional nonlinear integral equations are developed for determining the helical buckling loads corresponding to a certain lateral load. Practical examples are analyzed with the help of symbolic computational software Maple 10. It is found that the numerical results agree with existing experimental data. It implies that the helical buckling of a drillstring is a nonlinear problem and the established nonlinear analytical method may be used for analysis of drillstrings in inclined wellbores.