My Rating Philosophy

Few subjects concerning Writing.Com arouse as much discussion on a regular basis as the rating and reviewing system. Even The StoryMistress wrote her own take on the system: "Comment-In-A-Box"  [E]. There are many strong opinions out there on this topic, and mine are among them, so I thought that I would take a moment to share my rating philosophy. The reason the StoryMistress wrote her outline, as I understand it, is because she felt people were rating too highly. My average given rating is above 4.0, so I feel it’s necessary for me to explore the issue and speak out on the method to my madness.

The StoryMistress asks that you consider her method next time you decide to rate an item on Writing.Com. Here is the rating breakdown:


         Perfect

         Above Average

         Average

         Below Average

         Needs Serious Work

What could be so contentious about this? My eye immediately zeroes in on two reasons: “Perfect” and “average” are two extremely subjective words. Seeing as how four out of five of the ratings are defined using those words, is it any wonder that people’s thoughts on ratings differ so much? Views on what exactly constitutes “serious work” will no doubt vary as well.

First, let me talk about “perfect.” The StoryMistress says that in order to garner a 5.0 rating from her, she must find an item to be “absolutely perfect,” which some raters have taken completely to heart. I have heard stories of people who will simply not give a 5.0 because nothing in the Universe is perfect; that is, before they even read something, they will prejudge it as unworthy of a 5.0. Perhaps some of my reviewers have given me a 4.5 instead of a 5.0 for this reason. I can be sure about at least one person, who openly stated she couldn’t give me a 5.0 because perfection doesn’t exist. If we weren’t meant to give 5.0-star ratings, then they wouldn’t be included as an option in the rating scale. What’s worse is that these people almost certainly will accept a 5.0 rating without objection if one is given to them, in effect stating that they are absolutely perfect. If they were consistent, they would respond to the reviewer by requesting, “Please lower my rating, since nothing deserves 5.0 stars.” That sounds ridiculous, right? Well, it doesn’t sound any more ridiculous than the idea that nothing can earn 5.0 stars.

Don’t misunderstand me; I think that requiring a piece to meet high standards in order to achieve a 5.0 rating is a good thing. A work doesn’t need to have something “wrong” with it in order to merit less than a 5.0. Some writings are good, but others make us say, “Wow.” Years ago, in a comment for one of my items, the reviewer stated that he’d never given a 5.0 before, but in this case he felt it was justified. However, even this item that earned a 5.0 rating from someone who’d never given one before continues to be improved to this very day. A 5.0 rating should not be taken as an indication that an item is as good as it could possibly get, for that will rarely if ever be true.

Because the Writing.Com rating system uses five stars and bases itself on the term “average,” it’s only natural to draw a parallel between it and the letter grades from the standard educational grading system we all know and love from our years of schooling. Those letter grades would logically seem to correspond to the number ratings in this manner:


         A

         B

         C

         D

         F

Factoring in half-star ratings and plusses and minuses, it would look something more like below:


         A+ or A

         A- or B+

         B

         B- or C+

         C

         C- or D+

         D

         D-

         F

Focus on 5.0 right now. This means an item that would receive a grade of A or A+ from me will achieve a 5.0 rating. On a 100 point scale, a minus letter grade often has a last digit 0–2, a straight letter grade 3–6, and a plus letter grade 7–9. For example, an A- would be 90–92, an A would be 93–96, and an A+ would be 97–99—and since there is no higher letter grade, 100 is an A+. The raters about whom I previously spoke claim that they would only give a 5.0 to an item that scores 100, but they maintain that is impossible anyway. I, on the other hand, would bestow a 5.0 upon any item that I feel scored 93 or better. Do I read an item and then sit around deciding on a specific number? No, this is all theoretical. The practical message that I want you to take away is that an item does not have to be “absolutely perfect” to be worthy of 5.0 stars.

The second highly subjective term to examine is “average.” “Comment-In-A-Box” places the average rating at 3.0, which I would equate with a C grade. Believe me, a C is not average—at least in a statistical sense, which is how Writing.Com means it, constantly bombarding us with messages to dispense more ratings closer to 3.0. Perhaps a C is the statistically average grade if you go to an inner city public high school in Minneapolis like I did, where—at least when I was there—the dropout rate was above 50%. In general, however, if you check out the distribution of grades in high school, you will more than likely see the mean centered near B. In college, it’s probably more like a B+. The quality of the average students’ work may not be outstanding, but teachers and professors feel better than ambivalent about it, so a mediocre, so-so, “average” grade of C doesn’t seem appropriate. A C grade is far enough below normal quality, in fact, that a 2.0 grade point average (C) was the bare minimum required to graduate from my college. So, in terms of Writing.Com, I would say that the statistically average grade would probably fall around a B, which would mean a rating of 4.0. As a Writing.Com Moderator, I can see the average rating any member has given to others, and I will tell you that even most of those on the site who promote the idea that 3.0 is average have an average given rating closer to 4.0.

Another reason average ratings tend toward the high side is that most of us, as much as we want to help out our fellow authors, prefer to enjoy ourselves at the same time, so we’re going to click on and read items we think we’ll like. I know my average is certainly skewed by the fact that a large percentage of my reviews are attributed to a single author’s epic, whose chapters have received a well-deserved average rating far above 4.5 from me. Just as we tend to seek out items that interest us, most of us avoid items that probably won’t, and items we don’t want to finish, either because they’re riddled with errors or fail to hold our attention even slightly, may be left without a rating instead of given a low rating.

Much of the animosity toward low ratings, I sense, is directed at the low ratings given without a review. Though such rate-and-runs are called “hate rates” by some, they are not necessarily given out of malice. As a veteran of this site who is secure in his writing abilities, I’m not fazed by these, but they can be frustrating or downright devastating for less experienced writers. Not only is the author not offered any indication of what’s wrong, let alone how he or she can remedy it, but he or she has no way to contact the reviewer in the future. If people have at least a Basic Membership, they can always review anonymously, and that will supply more guidance to the author in addition to providing a means for communication. Anonymous reviews often get a bad rap, but if they’re as constructive as a non-anonymous review would have been, there’s no shame in “hiding.” I also believe that the time and words put into a review allow for a deeper consideration of the rating. I know the rating I’ve ultimately chosen for some items in the past was not the original, “knee-jerk” rating I had in mind; it either rose or fell over the course of composing the review.

For anyone who questions my views on ratings, I should point out that the review is, or at least should be, more important to the author than the rating, and I have reviewed and will continue to review every single item that I rate. Honestly, I seriously dislike summing up my evaluation of an item with a single number—probably because I’m a scientific guy, and rating is an art, not an exact science. Things get even murkier when we take the average of these single numbers, all of which come from people with different interpretations of “perfect” and “average,” in order to arrive at that item’s average rating.

The average rating that Writing.Com displays isn’t always even the best summary of those individual ratings. Let’s say that you have an item with four 5.0-star ratings and one 1.0-star rating, so, in ascending order, your ratings are 1.0, 5.0, 5.0, 5.0, 5.0. What does the “average” person consider the quality of this item to be? Well, I don’t know about you, but I would say 5.0 stars. Of course, no reviewer will know all the ratings unless you personally tell them, and not even you know all the ratings if you don’t have at least an Upgraded Membership.

Writing.Com takes the mean, the most commonly used average, which is found by adding all the ratings together and then dividing by the number of ratings. (I’m sure that most of you know about averages, but I want to make sure I don’t lose anybody.) In this item’s case, that would be (1+5+5+5+5)/5 = 21/5 = 4.20, which would display to the public as 4.0 stars. The mean is not always the most effective average, however, since it doesn’t manage outliers (extreme data values) very well. In this case, the 1.0-star rating is an outlier. When the average household income is calculated for a city, state, province, or country, the mean average is not used because it would be greatly influenced by those who are obscenely wealthy, and the average income would be overstated and not reflect the actual financial situation of most people. Instead, the median household income is calculated, which takes the middle value in the ordered data set. In the case of 1.0, 5.0, 5.0, 5.0, 5.0, there are five ratings, so the third rating is the median. (If there were an even number of ratings—four, for example—the median would be found by taking the mean of the second and third ratings.) The third rating in ascending or descending order is 5.0, so the median average is 5.0 stars. There is yet another average, the mode, which simply takes the most frequent value. Since there are four 5.0’s and only one 1.0, 5.0 is the most numerous rating, so the mode average is 5.0 stars.

As the number of ratings increases, the influence of outliers is greatly reduced, so as an item receives more ratings, the average rating will more accurately depict what the “average” person thinks of the quality. However, if you take a look at the Sitewide Stats and divide the Ratings Currently On Writing.Com by the Items Currently Stored On Writing.Com, you’ll notice that the (mean) average item on this site has about four ratings, and when you consider the items that have tens or hundreds of ratings, those need to be balanced out by many items that have even fewer ratings than average.

There’s nothing that says the statistically average item on this site merits a mediocre rating of 3.0, but even if I we assume for the sake of argument that it does, expecting actual rating averages to match that mark means assuming that we choose items to read completely at random and rate every single one, which simply isn’t how most members approach reading and rating, nor should we be expected to. It takes time for new members to even get a sense of what’s “average” on the site. One shouldn’t necessarily fret over an average given rating that leans to the high side; as long as one rates each individual item with as much honesty as possible, it follows that the average of all those individual ratings will be honest as well.