The present computational investigation aims to investigate the effect of varied buoyancy ratios on mixed convection and entropy formation in a lid-driven trapezoidal enclosure under magnetic field with two rotating cylinders. The effects of SWCNT–water, Cu–water, and Al2O3–water nanofluids individually, as well as effects of three different types of SWCNT–Cu–Al2O3–water hybrid nanofluids are examined. The governing Navier–Stokes, thermal energy, and mass conservation equations are solved using the Galerkin weighted residual finite element method to obtain results as average Nusselt number, Sherwood number, temperature, and Bejan number as output parameters inside the enclosure for different parameter values. Then, an innovative artificial neural network model for effective prediction is created using the simulation data. The optimum values of each of these input parameters are obtained by finite element method (FEM) and artificial neural network (ANN), and a comparative study between FEM and ANN is done to get best results for the output parameters. The performance of the created ANN model for novel scenarios is evaluated using Cu–Al2O3–water hybrid nanofluid. The proposed innovative ANN model predicts the findings with less time and sufficient accuracy for each type of studied governing fluids. The model’s accuracy for predicting convective heat and mass transfer, along with average dimensionless temperature and Bejan number, was 96.81% and 98.74%, respectively, when tested on training and validation data. On test data, the accuracy was 97.03% for convective heat and mass transfer and 99.17% for average dimensionless temperature and Bejan number.