Given a group automorphism $\phi:\Gamma\longrightarrow \Gamma$, one has an action of $\Gamma$ on itself by $\phi$-twisted conjugacy, namely, $g.x=gx\phi(g^{-1})$. The orbits of this action are called $\phi$-twisted conjugacy classes. One says that $\Gamma$ has the $R_\infty$-property if there are infinitely many $\phi$-twisted conjugacy classes for every automorphism $\phi$ of $\Gamma$. In this paper we show that $\operatorname{SL}(n,\mathbb{Z})$ and its congruence subgroups have the $R_\infty$-property. Further we show that any (countable) abelian extension of $\Gamma$ has the $R_\infty$-property where $\Gamma$ is a torsion free non-elementary hyperbolic group, or $\operatorname{SL}(n,\mathbb{Z}), \operatorname{Sp}(2n,\mathbb{Z})$ or a principal congruence subgroup of $\operatorname{SL}(n,\mathbb{Z})$ or the fundamental group of a complete Riemannian manifold of constant negative curvature.