In this paper, a modified incremental harmonic balance (IHB) method combined with Tikhonov regularization has been proposed to achieve the semi-analytical solution for the periodic nonlinear system. To the best of our knowledge, the convergence of the traditional IHB method is bound up with the iterative initial values of harmonic coefficients, especially near the bifurcation point. Thus, the Tikhonov regularization is introduced into the linear incremental equation to tackle the ill-posed situation in the iteration. To this end, the convergence performance of the traditional IHB method has been improved significantly. Moreover, the proof of convergence of the proposed method also has been given in this paper. Finally, a van der Pol-Duffing oscillator with external excitation and a cubic nonlinear airfoil system with the external store are adopted as numerical examples to illustrate the efficiency and the performance of the present method. The numerical examples show that the results achieved by the proposed method are in excellent agreement with those from the Runge–Kutta method, and the accuracy is not significantly reduced compared with the traditional IHB method. Especially, the results from the proposed method also can converge to the exact solution from the initial values that the traditional IHB method cannot obtain the converged results.