In this paper we study genus $2$ curves whose Jacobians admit a polarized $(4,4)$-isogeny to a product of elliptic curves. We consider base fields of characteristic different from $2$ and $3$, which we do not assume to be algebraically closed. We obtain a full classification of all principally polarized abelian surfaces that can arise from gluing two elliptic curves along their $4$-torsion, and we derive the relation their absolute invariants satisfy. As an intermediate step, we give a general description of Richelot isogenies between Jacobians of genus $2$ curves, where previously only Richelot isogenies with kernels that are pointwise defined over the base field were considered. Our main tool is a Galois theoretic characterization of genus $2$ curves admitting multiple Richelot isogenies.