We say that an abelian variety over a $p$-adic field $K$ has anisotropic reduction (AR) if the special fiber of its N\'eron minimal model does not contain a nontrivial split torus. This includes all abelian varieties with potentially good reduction and, in particular, those with complex or quaternionic multiplication. We give a bound for the size of the $K$-rational torsion subgroup of a $g$-dimensional AR variety depending only on $g$ and the numerical invariants of $K$ (the absolute ramification index and the cardinality of the residue field). Applying these bounds to abelian varieties over a number field with everywhere locally anisotropic reduction, we get bounds which, as a function of $g$, are close to optimal. In particular, we determine the possible cardinalities of the torsion subgroup of an AR abelian surface over the rational numbers, up to a set of 11 values which are not known to occur. The largest such value is 72.