Hey no worries! I find RP's work better between a fewer amount of people!
If anything, one of us can play more players if need be, but I don't think it is. Besides, groups of four often rely on every member contributing in a more divided manner. I think just the few of us can handle a good story, if nothing else! It's not quantity, it's quality. And if our quality is amazing with a few people as opposed to good with four, I'd rather opt for amazing. After all, you can still recruit once we've started a bit. Write a backstory with a new player before they "happen upon us" so to speak. :)