Abstract: The buckling failure of pile can be studied by the catastrophe theory. In this paper, the calculating width of pile was analyzed firstly, and then the cusp catastrophe theory was applied to determining the critical load for pile failure. For partially buried pile with one end embedded in stronger bearing stratum, the paper established catastrophe model with top-free and bottom-fixed, and top-hinged and bottom-hinged for entirely buried pile with one end embedded in relative bearing stratum. A new method to determine the critical load was proposed, and the calculated results were compared with those from lab and field model tests, which verified the feasibility of the method. The critical loads of Y style vibro-pile, square pile and circular pile with identical cross sectional area were compared in conditions of partially and entirely buried in the ground. It is found that the bucking load of Y style vibro-pile is much greater than that of square pile and circular pile. Under the conditions specified in this paper, the buckling critical loads of square and circular pile are only 65% and 62% that of Y style vibro-pile. The reason is that: due to the special-shaped section, the Y style vibro-pile enhances the skin friction, thus to increase the buckling critical load effectively.