In order to identify more accurately and efficiently the unknown parameters of a ship motions model, a novel Nonlinear Least Squares Support Vector Machine (NLSSVM) algorithm, whose penalty factor and Radial Basis Function (RBF) kernel parameters are optimised by the Beetle Antennae Search algorithm (BAS), is proposed and investigated. Aiming at validating the accuracy and applicability of the proposed method, the method is employed to identify the linear and nonlinear parameters of the first-order nonlinear Nomoto model with training samples from numerical simulation and experimental data. Subsequently, the identified parameters are applied in predicting the ship motion. The predicted results illustrate that the new NLSSVM-BAS algorithm can be applied in identifying ship motion’s model, and the effectiveness is verified. Compared among traditional identification approaches with the proposed method, the results display that the accuracy is improved. Moreover, the robust and stability of the NLSSVM-BAS are verified by adding noise in the training sample data.