Exactly this. Housing as "investment" requires constant increase in prices, the faster the better. Housing as housing does not.
The deeper problem - perhaps the fundamental problem in this area - is that we have organized housing so that provision of a basic service is completely intertwined with a speculative asset market. The long run goal should be to disentangle them.