Nonconvexity of the Generalized Numerical Range Associated with the Principal Character

http://dx.doi.org/10.4153/CMB-2000-053-3
Canad. Math. Bull. 43(2000), 448-458

  Published:2000-12-01
 Printed: Dec 2000

Suppose $m$ and $n$ are integers such that $1 \le m \le n$. For a subgroup $H$ of the symmetric group $S_m$ of degree $m$, consider the {\it generalized matrix function} on $m\times m$ matrices $B = (b_{ij})$ defined by $d^H(B) = \sum_{\sigma \in H} \prod_{j=1}^m b_{j\sigma(j)}$ and the {\it generalized numerical range} of an $n\times n$ complex matrix $A$ associated with $d^H$ defined by $$ \wmp(A) = \{d^H (X^*AX): X \text{ is } n \times m \text{ such that } X^*X = I_m\}. $$ It is known that $\wmp(A)$ is convex if $m = 1$ or if $m = n = 2$. We show that there exist normal matrices $A$ for which $\wmp(A)$ is not convex if $3 \le m \le n$. Moreover, for $m = 2 < n$, we prove that a normal matrix $A $ with eigenvalues lying on a straight line has convex $\wmp(A)$ if and only if $\nu A$ is Hermitian for some nonzero $\nu \in \IC$. These results extend those of Hu, Hurley and Tam, who studied the special case when $2 \le m \le 3 \le n$ and $H = S_m$.