In ground air heat exchanger (GAHE) system, the heat transfer between air and underground soil largely depends on soil thermal properties and therefore, any improvement in soil thermal properties will shorten the pipe length required and the land area needed for its installation. The objective of the present study is to investigate the effect of different backfilling materials (low cost and locally available) on the thermal performance of GAHE system using a small-scale laboratory experimental setup. Seven different backfilling materials have been considered for the study and it was observed that after 6 h of continuous operation, the drop in air temperature was 6.2 °C at outlet section of pipe (2.4 m away from inlet) for the native soil. However, for sand–bentonite with graphite as a backfilling material (BFM), the drop in air temperature of 6.2 °C was obtained at a pipe length of 1.15 m only. Therefore, the use of sand–bentonite with graphite as a BFM reduces the pipe length of GAHE system by more than 50%. The study establishes the fact that the length of pipe and land area requirement for GAHE system can be substantially reduced by using thermally enhanced backfilling materials at the close vicinity of GAHE pipes.