There are some people who are sensitive to the taste of sorbate who may pick up a flavor change even at relatively low levels, but for most folks the taste does not become detectable until over about 150-200 mg/L. Some claim that with long aging it may become more noticeable, but I cannot say whether this is true or not.
What is true is that backsweetening without doing something to stop the yeast is asking for bottle bombs, and in most cases the sorbate is a good idea.