The Mantel test is widely used in biology, including landscape ecology and genetics, to detect the presence of spatial structures in data or control for spatial correlation in the relationship between two data sets, e.g. community composition and environment. The paper demonstrates that this is an incorrect use of that test.

The null hypothesis of the Mantel test differs from that of a correlation analysis; the statistics computed in the two types of analyses differ. We examine the basic assumptions of the Mantel test in spatial analysis and show that they are not verified in most studies. Finally, we show the consequences, in terms of power, of the mismatch between this assumption and the Mantel testing procedure.

The Mantel test is a test of the absence of relationship (H₀) between the values in two dissimilarity matrices, not the independence between two random variables or data tables. A demonstration is provided that the Mantel R² differs from the R² of correlation, regression and canonical analysis; these two statistics cannot be reduced to one another. Using simulated data, we show that in spatial analysis, the assumptions of linearity and homoscedasticity of the Mantel test (H₁: small values of D₁ correspond to small values of D₂ and large values of D₁ to large values of D₂) do not hold in most cases, except when spatial correlation extends over the whole study area; this is a novel contribution to the Mantel debate. Finally, using extensive simulations of spatially correlated data involving different representations of geographic relationships, we show that the power of the Mantel test is always lower than that of Moran’s eigenvector map (MEM) analysis, and that the Mantel R² is always smaller than in MEM analysis, and uninterpretable (also a novel contribution).

Our main conclusion is that Mantel tests should be restricted to questions that, in the domain of application, only concern dissimilarity matrices, and are not derived from questions that can be formulated as the analysis of raw data tables, meaning the vectors and matrices from which one can compute dissimilarity or distance matrices.