Dan Lepard says on this thread on his forum "Wheat prices were high during 1790 - 1810". Due to the rise in malt taxes in this period it also became more expensive to brew. In addition, industrial common brewers were fast displacing the production of beer in private homes and on the premises in pubs, as you could buy it cheaper from the common brewer than you could make it yourself.
So we have a double whammy: there were far fewer people producing ale-barm and the cost of producing it had increased too. With higher prices and decreased availability of ale-barm it looks like bakers were squeezed all round, not just by wheat prices.
I assume that a typical bakery around 1800 would be a very small operation with maybe a master and apprentice and one shop. Possibly they had even counted among those who had recently given up brewing themselves and no longer had their own supply of free yeast.
I am fascinated by the relationships between brewers and bakers. I can't help but think there must be something deeper behind such a significant change in practice.
Beer historian Ron Pattinson says here:
This period saw a key change in British brewing practices. Though brewers had already been aware of the better yield of pale malt compared to brown, it was only with the introduction of the hydrometer that this could be quantified. It became apparent that, despite the lower price of brown malt, it was still cheaper to brew from pale malt. Brown malt produced only about two thirds as much fermentable material as pale.
But I can't for the life of me think what possible adverse effect moving to mostly pale malt would have on the yeast.