Regarding long-term play, sitting out conflicts will slow leveling considerably. At higher levels, the hardest thing is spending enough Fate.

Tests are relatively scarce in Torchbearer, especially in a larger group. That means you’re only likely to get a few tests per session where you can spend Fate.

Conflicts are one of the few places where you’re always looking for extra successes and so can justify spending that Fate.