You can start from one cell and go from there, but the reason you use a certain amount of yeast is cell density. For one cell, you'd be using a few ml to grow it into lots of cells, moving up the volume slowly, etc. The yeast in one packet is good for 5 gallons (of moderate gravity) because the cell density is appropriate. Starting with not enough yeast cells, the yeast will have to work harder & longer to reach optimum density. This does a couple things: less sugar is turned into alcohol (more is turned into yeast), more byproducts of yeast growth are produced (possible off flavors), the possibility of cell mutation is high (due to more generations of yeast, predictable fermentation behavior is lost), the possibility of stuck fermentation is higher (cells which bud a lot are less able to ferment fully, mutant daughters might ferment less too), and with a sufficiently low pitch rate the chances of infection are higher (you'd have to seriously underpitch for this to be a concern). You would also have to feed the yeast more in order to give them the required compounds for extra cell division.
For moderate gravity (up to about 1.090-1.100), about 1 g/gal is a good amount of yeast to pitch assuming you properly rehydrate for maximum cell count. Higher gravity and you should add a little more yeast (~1.2 g/gal for about SG 1.115 for example). Adding too much yeast is also a problem: certain flavor compounds are produced during the "lag phase" which you can essentially skip by pitching tons of yeast. This is not a huge problem, and honestly you'd have to overpitch by a factor of about 10 to run into it. Even then, you might not notice unless you're using a very particular strain known for yeast-derived flavors (some Belgian ale strains, for example).