The design, energy output, and cost effectiveness of solar projects using concentrators critically depend on the resource in direct normal irradiance (DNI). Many modeled DNI datasets now exist, and a recent preliminary study has shown some areas of serious disagreement in Europe. So far, no rigorous performance assessment has been undertaken for other parts of the world. The present contribution focuses on North Africa and bordering regions, which have great solar potential for power plants based on thermal or photovoltaic concentration systems. The mean monthly and annual performance of eight different modeled datasets providing DNI is analyzed here, with respect to measured radiation data at 14 sites, which are used as “ground-truth”. Relatively good results are generally obtained for sites in southern Europe. Serious problems, however, are found at various sites in North Africa or the Middle East. Most of these problems appear linked to inadequate aerosol optical depth data used by the models, and to the dust storms from the Sahara that regularly, and strongly, modify the aerosol regime. A method that can potentially correct these problems, or allow for model benchmarking based on a reference aerosol database, is proposed. The bankability of current datasets is questioned.