A geothermal heat exchanger (GHE) uses the geothermal energy for heating or cooling of residential places during winter or summer. Two different designs of GHEs, the straight-pipe and coiled-pipe designs, are evaluated in this study, and the effect of nanofluid as the working fluid is investigated. For this purpose, a mathematical model is developed, validated, and used to predict the temperature gain, heat gain, exergy gain, and pressure loss of the working fluid for different concentration of additive ceramic nanoparticles (Al2O3 and MgO) into the working fluid. It is shown that the coiled-pipe design has better performance compared to the straight-pipe design for GHEs. It also shown that how the temperature, heat gain and exergy gain change with increasing the additive nanoparticles into the base-fluid (water), while the pressure loss does not change significantly. The temperature gain increases about 60% when the volume fraction of nanoparticles in the base-fluid reaches 2%. This also helps to improve the natural circulation of working fluid and the GHE may not need a circulating pump to run at low flow rates. It is also shown that the additive MgO nanoparticles is more effective than Al2O3 nanoparticles to improve the GHE performance.