We study the geometry of non-reductive $4$-dimensional homogeneous spaces. In particular, after describing their Levi-Civita connection and curvature properties, we classify homogeneous Ricci solitons on these spaces, proving the existence of shrinking, expanding and steady examples. For all the non-trivial examples we find, the Ricci operator is diagonalizable.

{Correspondence} between torsion-free connections with {nilpotent skew-symmetric curvature operator} and IP Riemann extensions is shown. Some consequences are derived in the study of four-dimensional IP metrics and locally homogeneous affine surfaces.