Let $X$ be a locally finite, connected graph without vertices of degree $1$. Non-backtracking random walk moves at each step with equal probability to one of the ``forward'' neighbours of the actual state, \emph{i.e.,} it does not go back along the preceding edge to the preceding state. This is not a Markov chain, but can be turned into a Markov chain whose state space is the set of oriented edges of $X$. Thus we obtain for infinite $X$ that the $n$-step non-backtracking transition probabilities tend to zero, and we can also compute their limit when $X$ is finite. This provides a short proof of old results concerning cogrowth of groups, and makes the extension of that result to arbitrary regular graphs rigorous. Even when $X$ is non-regular, but \emph{small cycles are dense in} $X$, we show that the graph $X$ is non-amenable if and only if the non-backtracking $n$-step transition probabilities decay exponentially fast. This is a partial generalization of the cogrowth criterion for regular graphs which comprises the original cogrowth criterion for finitely generated groups of Grigorchuk and Cohen.