A large share of cross-country differences in productivity is explained by differences in agricultural productivity. Using a combination of sub-national agricultural statistics and geospatial datasets on crop-specific potential yields, we study the main drivers of this variation from a macroeconomic perspective. We find that differences in geographically-induced crop-specific comparative advantages can explain a substantial share of the variation in yields across the world. Data reveal substantial gaps between potential and observed yields in most countries. When decomposing these within country gaps, we find that crop selection gaps are on average larger than those induced by input usage alone. The results highlight the importance of understanding the interaction of geography and crop selection drivers in assessing aggregate agricultural productivity differences.