Let $\mathcal{H}_n$ be the real linear space of $n\times n$ complex Hermitian matrices. The unitary (similarity) orbit $\mathcal{U} (C)$ of $C \in \mathcal{H}_n$ is the collection of all matrices unitarily similar to $C$. We characterize those $C \in \mathcal{H}_n$ such that every matrix in the convex hull of $\mathcal{U}(C)$ can be written as the average of two matrices in $\mathcal{U}(C)$. The result is used to study spectral properties of submatrices of matrices in $\mathcal{U}(C)$, the convexity of images of $\mathcal{U} (C)$ under linear transformations, and some related questions concerning the joint $C$-numerical range of Hermitian matrices. Analogous results on real symmetric matrices are also discussed.