Due to harsh operating environments in underground coal seams, the key components (e.g., gear pairs and bearings) in the power transmission systems of coal cutters suffer from extreme wear and functional damages. To guarantee the safe and reliable operation of the coal cutters, it is important to monitor the condition of their transmission systems and detect possible faults in a timely manner. A challenging task here is to diagnose multiple concurrent faults. A literature review indicates that the current interests lie on the decoupling of multiple co-existing faults and that the very limited work has been done to deal with the dependence/correlation between the fault signals. To address this issue, this work extends our previous work on gear crack detection using the bounded component analysis (BCA) and proposes an improved BCA-based approach for decoupling hybrid faults with high dependence/correlation in coal cutter transmission systems. The proposed approach incorporates the Vold–Kalman order tracking and spectral kurtosis into an improved BCA framework (OTBCA-SK). Owing to the uniform sampling of order tracking, the influence of background noise and rotational speed variation on vibration signals can be effectively reduced. Since BCA is capable of handling vibration sources that are statistically dependent, OTBCA-SK can decouple both independent and dependent source signals. As a result, the vibration sources excited by hybrid faults, although maybe dependent/correlated, can be fully decoupled into single-fault vibration source signals. Three specially designed case studies were used to evaluate the effectiveness of the proposed OTBCA-SK approach in decoupling hybrid gear faults. The analysis results demonstrate better performance of hybrid fault decoupling using OTBCA-SK than that of three representative techniques, i.e., Erdogan's BCA (E-BCA), joint approximate diagonalization of eigen matrices (JADE) and fast independent component analysis (FastICA). These case studies also suggest that the proposed OTBCA-SK approach can retain the physical meaning of the original vibration and is hence suitable for hybrid fault diagnosis in practical applications.