We study the geometry of non-reductive $4$-dimensional homogeneous spaces. In particular, after describing their Levi-Civita connection and curvature properties, we classify homogeneous Ricci solitons on these spaces, proving the existence of shrinking, expanding and steady examples. For all the non-trivial examples we find, the Ricci operator is diagonalizable.