Rotary axes are the key components for five-axis computerized numerical control machines, while their motions are dramatically influenced by thermal issues. To precisely model the thermal error of rotary axis, a convolutional neural network (CNN) model is developed. To form data sets for the CNN, a laser interferometer is used to measure the angular positioning error at different temperatures and a thermal imager is taken to obtain thermal images of the rotary axis. The measured thermal error is fitted to a sine curve so that training parameters are reduced. And the thermal pixel values of the initial thermal image are subtracted from all the thermal images to consider the incremental thermal effect, so the influence of the initial temperature is negligible. Finally, a deep CNN model with multiple output classifications is designed to complete the data training, verifying and testing. The experimental results show that the prediction accuracy for the parameters is higher than 90%, and the percentage reduction in error is higher than 80%.