View Full Version : Blind tasting reveals judging inconsistancies!

01-29-2009, 05:10 PM
Well, I was reading this today and it reminds me once again of the value of true blind tastings. This from a study done over the course of four years by a retired College Professor from Northern California.

By Jerry Hirsch (http://articles.latimes.com/writers/jerry-hirsch)
January 29, 2009 (http://articles.latimes.com/2009/jan/29/business)

Judges at the California State Fair wine competition scored poorly at giving the same wine an identical rating when they tasted it multiple times in a blind tasting.

That was the conclusion of a four-year study of judging decisions at the California State Fair Wine Competition by retired Humboldt State professor Robert Hodgson.

“Consumers should have a healthy skepticism about the medals awarded to wines from the various competitions,” he said.

Hodgson’s findings have prompted state fair officials to consider making changes in the way they operate future wine competitions.

In a study published today by the Journal of Wine Economics, Hodgson wrote that only 10% of the judges were able to consistently give the same rating, or something very close, to the identical wine sampled multiple times in a large blind tasting.

At the opposite end, another 10% of the judges gave the same wine far different ratings, ranging from worthy of a gold medal to deserving of no medal at all on successive tastings. The remaining 80% of judges also varied in their ratings, but by a narrower range.

State fair officials, who cooperated with the study, said Hodgson’s findings provided valuable information that can be used to improve wine competitions. The next state wine judging is set for June 10-12.

“We want to do whatever we can to smell and taste wines to get the best results,” said G.M. “Pooch” Pucilowski, a professional wine educator and chief judge for the state fair.

The competition plans to reduce the number of wines sampled per day – which at some competitions can approach 150 or more – to as few as 75 to help judges avoid the sensory fatigue that can cloud their ratings.

The fair is also using the data “to weed out” judges who year after year are unable to rate different samples of an identical wine consistently, Pucilowski said.

Finding ways to evaluate the skills and consistency of judges is an important issue for wine competitions, which often draw from the same small pool of industry members and aficionados for their rating panels. It’s not unusual for judges to work as many as six different competitions annually.

Wine shoppers such as Glendora urologist Dr. Charles Metzger rarely looks for wines that are medal winners when deciding what wine to purchase.

“I mostly look to what other people recommend, and then I look at the 100-point ratings from publications like the Wine Spectator,” Metzger said.

“I found that some of the Los Angeles County Fair winners that I tried were just not that good,” he said.

But Metzger’s wife, Barbara, has had good luck purchasing wines that have won awards at competitions, he said.

“The medal winners tend to be less expensive than the $40, $60, $80 wines you see in Wine Spectator,” Metzger said.

Hodgson said he doesn’t put any more trust in the 100-point-scale ratings of wines from magazines and newsletters than he does in medal winners.

“Consumers need to gain more self-confidence in their own opinions and tastes rather than listen to what other people think wine should be like,” Hodgson said.

Last year, 649 wineries entered 2,917 California wines into the annual state fair contest, the oldest wine competition in America. More than half, or 1,587 wines, won awards.

Hodgson, an oceanography professor who also taught statistics at the university, owns the small Fieldbrook Winery just north of Eureka, in Humboldt County. He said he designed the study because he didn’t understand why “we would have wines that we sent off and would get gold medals in some competitions and in others would get poop. It seemed like a gold medal was just a matter of luck.”

Hodgson said he learned the judging business is characterized by inconsistent decision making by judges and wide variability among competitions.

“Wine judges in the setting of a competition must make about a hundred decisions a day. It is in this environment where I think their ability is taxed beyond a reasonable level,” Hodgson said.

He also discovered that a “super judge” who is consistent in his or her ratings one year does not maintain that superiority the next year.
“This emphasizes the chance argument in placing awards,” Hodgson said. The findings of the study create a conundrum for Hodgson and his small winery.

“I use gold medals to sell my wine,” he said, “and now I have written this paper saying the wine competition system that awards those medals isn’t perfect.”

This is a pretty strong statement about the "trained" palates of a majority of wine judges who tout themselves as experts. It also brings into question the validity of the scores, the quality of the wines that win, and the overall value of a system that has only a 10% consistency in judging over several years. I really feel this supports my opinion that judging should start with a calibrated panel using blind tasting (triangular) to screen judges prior to the actual competition. Unfortunately with the number of people available for judging, it is problematic since a majority of applicants will more than likely be eliminated from judging.

To me this is where the issue of actual judge training, sensory analysis and standardization of judging criteria come into play. Anyhow, on the more optimistic side, I think that in the mead world we'll see a better overall percentage of judges who can really discriminate between different meads in triangular tastings. We come at mead a lot differently than wine judges approach wine. That is, we don't have the "presumption" of expertise in my opinion. Most of us are pretty clear about not being experts, not wanting to be held as an expert, and most importantly are very aware of what we do not know. Of course there are folks who think they know it all, but, based on the people I've met and gotten to know over the years, that percentage of the mead crowd is pretty low.

I think this also presents an excellent opportunity for the mead world to establish sensory standards in a way that will allow for more consistency from competition to competition and from judge to judge. Over time style standards and a judging standard can be engineered to minimize guesswork and increase concordance with established standards. The check-box approach is a good start based on the scores I've seen where this type of judging sheet is used at BJCP sanctioned competitions. I think this can evolve into a way of tallying a score for each sensory category (nose, aroma, flavor, character, body, mouthfeel, etc) that will give a spot-on grade to the mead being judged.

Now all we have to do is make it happen!

01-29-2009, 05:24 PM
I'm not familiar with how judging takes place for wine...but I have been involved in food competitions when the judging was by three different groups...the first judges narrowed the choices from say 25 down to 10...a second and different set of judges then tasted and narrowed the 10 down to 5....and yet a third set of judges chose the winners from the remaining 5.

Would a similar system (along with better trained judges) work for meads?

01-29-2009, 06:13 PM
I think judging will eventually move toward that type of model based on the type and number of meads to be judged. Honestly I think there needs to be an evolution "revolution" in beer, mead and wine judging. I've been seated at a number of different BoS panels in beer, wine and mead competitions and I'm always amazed at how often entries with great complexity, structure and subtlety are overlooked in favor of an entry hits you over the head. In my opinion this is because many judges are taking the style guidelines so literally that they think everything has to be represented equally.

Many disjointed, tight and still developing entries get the nod because the flavors are obvious and it doesn't take a lot to distinguish them from one another. I don't think a lot of judges can really tell you how a well integrated wine, mead or beer develops across the palate; and honestly I still struggle with that as well. To me it takes a lot of practice so I'm out there every week hitting different wines mostly because there is a limited selection of mead available commercially through the wine, beer and spirit stores in my area. You become more in touch with your palate the more you practice; just as you become more aware of your shortcomings.

To me a palate is just like a muscle that you have to exercise regularly to keep it in it's prime. It's also about treating your palate right, I can't tell you how many people I see smoking, drinking coffee, having spicy foods, using minted toothpastes, wearing cologne, scented skin lotion, etc. prior to judging. They all must be better than me because I wouldn't ever be able to judge effectively with any of that stuff dulling my palate. But hey...that's just me!

Cheers, Oskaar

01-29-2009, 06:32 PM
Even though I'm new to Mead & Wine making I am interested in sending in entries mainly for feedback to see how what I produce compares to others.

I plan to enter several competitions this year for my beer, and hope to do the same next year with a couple of Meads and Wines.

Ultimately what my wife and I like will determine the direction I go with brewing as well as Mead and Wine making, but having feedback from my peers can also help me to monitor how I am progressing.

01-29-2009, 06:46 PM
That was a very interesting read. I had read of a similar type of study (name of researcher, time, and location escape me). White wine was dyed red and given to tasting experts to evaluate, and they evaluated it just like they would a red wine, and had no idea it was actually white. Also, the same wine was bottled in two different bottles, one fancy and one plain, and the fancier bottle consistently was given a better taste rating.

There were other parameters, but those are just the ones that stuck out the most to me. Goes to show you what appearance can do in judging.

01-29-2009, 07:29 PM
Not surprising, but at the same time it is, if that makes any sense at all.

My better half can eat the same food just a few days apart, and will like it more/less than the first time. The palate is a delicate instrument that needs care and attention, and is affected by the mood of the individual, plus whatever they have eaten/drunk recently. As Pete says, practice makes perfect.

What us surprising is that all of the wine officionados that I have talked to are so full of themselves, and eager to stroke their own egos, that it strikes me as odd that they do not "exercise" their palate to make sure they are never caught out by a blind test.


01-30-2009, 04:56 PM
Well, I thing this concept of exercising the palate is a good idea, who's time has come!

Seeing as I'm at work right now, I'll have to wait, but as soon as I'm home I plan on exercising mine... Repeatedly!