A compressed stiff film/compliant substrate system undergoes a morphology transition from wrinkling to period-doubling. The perturbation method is used to obtain the approximate analytical solution incorporating both the quadratic and cubic nonlinearities of the substrate, which have a significant effect on the post-buckling behavior of the system. Based on the perturbation method, the post-buckling equilibrium path of the system is presented with the multi-modal analysis, and two bifurcation points appear on the stable equilibrium path. The wrinkling instability occurs at the first bifurcation point, where the uncoupled path bifurcates from the fundamental unbuckled state. Under further compression, the period-doubling instability occurs at the second bifurcation point due to the coupling of different modes, which is referred to as the mode coupling. The two-mode analysis shows that the coupled equilibrium path is hyperbola-like and there exists a stable branch which bifurcates from the primary uncoupled path. When more modes are included, the model is more accurate to predict the critical strain of the period-doubling bifurcation and the evolution of the amplitudes.