It all depends on what you build, the lumber you tend to work with, and how much it bothers you to rip down lumber just to mill it. Everybody will have different answers to the above criteria, but in the realm of jointers, bigger is rarely worse unless you simply don’t have the space or electrical requirements for a larger machine. I’ve used 6” machines, used to own an 8” delta, currently own a 12” Oliver, and have used machines up to 16” wide doing all types of personal and professional solid wood woodworking and there’s never a been a time when I’ve thought “man, if only this jointer were smaller...”
Longer beds and a stouter build quality alone is usually reason enough for me to gravitate towards a bigger machine, even if I’m just edge jointing. It seems that the OP works in a particularly distinctive way that avoids lumber wider than 4” (?) and that’s a unique situation, but the first time you want to mill a 8/4 board that’s 10” wide and 6’ long, I don’t think you would regret having a bigger jointer.
Different strokes for different folks.