I think handling time grows faster than group size. Noncombat rolls are fine, and can offer meaningful choices if you roll multiple sets that would affect quality or speed differently. That does mean you need both time pressure and consequences for poor quality to make that work, but when you do it’s really nice.

Initiative order and too many NPCs was always my biggest annoyance in combat. Everybody kind of rolls at once and then you figure out WTF just happened to everyone. My group called it the “headshot circus”, after the ending of The Departed, although now the gunfight in Wind River also comes to mind.