We study the splitting of Fermat Jacobians of prime degree $\ell$ over an algebraic closure of a finite field of characteristic $p$ not equal to $\ell$. We prove that their decomposition is determined by the residue degree of $p$ in the cyclotomic field of the $\ell$-th roots of unity. We provide a numerical criterion that allows to compute the absolutely simple subvarieties and their multiplicity in the Fermat Jacobian.