Review Scoring?

[katiesquilts]

Hi everyone,

I was just wondering how the editors who score official reviews actually assign scores. Is there a grading template, or are they allowed to assign random numbers based on their overall opinion of the review?

I'm not complaining or anything, but I think it's very odd that I can get a comment full of compliments for a review, but the score is in the 70~79 range. If there's a template, I'd rather know where the points are being docked from.

[rssllue]

Good question.

[jamesabr]

I just reviewed my first paid book, so I was wondering this too. I'll see if I can let you know once I get my score back.

[katiesquilts]

Bumping this topic. Perhaps someone from the editing team can share a bit of the process? For example, how they choose reviews to edit (or if they're assigned), how the grades are assigned, whether they can see the username of the reviewer, etc.

[gali]

Bumping this topic. Perhaps someone from the editing team can share a bit of the process? For example, how they choose reviews to edit (or if they're assigned), how the grades are assigned, whether they can see the username of the reviewer, etc.

Reviews are assigned randomly and the editors can't see the username of the reviewer. The grades are assigned based on the quality of the review (grammar and spelling issues, typos, format, etc). You can read about the requirements of the reviews on your RT page. Good luck!

[katiesquilts]

Reviews are assigned randomly and the editors can't see the username of the reviewer. The grades are assigned based on the quality of the review (grammar and spelling issues, typos, format, etc). You can read about the requirements of the reviews on your RT page. Good luck!

Thanks, gali! I've been reviewing books for a while, but I only recently became curious about the scoring process. It's great to know that everything is anonymous!

[bookowlie]

Until a reviewer gets to the highest levels, a review must be edited and scored by two different editors. The overall score you receive for a review is not an average; it is the lower of the two editors' scores. I am not sure if both editors' comments are shown for each review. I think the comments are randomly picked for display. That might explain why you might see comments that are very complimentary with a lower score. You can always contact Scott if you have questions about the score and/or comments you received Hope this helps.

-- 26 Dec 2016, 22:07 --

Gali is correct - the identity of the reviewer is not revealed. Also, there should be a plot summary, opinions, and clear justification for the rating.

[MrsCatInTheHat]

Until a reviewer gets to the highest levels, a review must be edited and scored by two different editors. The overall score you receive for a review is not an average; it is the lower of the two editors' scores. I am not sure if both editors' comments are shown for each review. I think the comments are randomly picked for display. That might explain why you might see comments that are very complimentary with a lower score. You can always contact Scott if you have questions about the score and/or comments you received Hope this helps.

I can see comments from both editors, if there were two. Sometimes there is only one editor, though, and in one case, there were three sets of comments.

[bookowlie]

Until a reviewer gets to the highest levels, a review must be edited and scored by two different editors. The overall score you receive for a review is not an average; it is the lower of the two editors' scores. I am not sure if both editors' comments are shown for each review. I think the comments are randomly picked for display. That might explain why you might see comments that are very complimentary with a lower score. You can always contact Scott if you have questions about the score and/or comments you received Hope this helps.

I can see comments from both editors, if there were two. Sometimes there is only one editor, though, and in one case, there were three sets of comments.

I think the general and detailed (3 sets of comments) questions regarding scoring guidelines are best answered by Scott. The editors receive random reviews to edit and score, but are not involved in the rest of the process.

[katiesquilts]

Until a reviewer gets to the highest levels, a review must be edited and scored by two different editors. The overall score you receive for a review is not an average; it is the lower of the two editors' scores. I am not sure if both editors' comments are shown for each review. I think the comments are randomly picked for display. That might explain why you might see comments that are very complimentary with a lower score.

It's not an average?? That doesn't make very much sense to me. If you're not going to average the scores, what's the point of having multiple people grade the review?
Hmm... I wonder what the purpose of only using the lowest score is?

[bookowlie]

Until a reviewer gets to the highest levels, a review must be edited and scored by two different editors. The overall score you receive for a review is not an average; it is the lower of the two editors' scores. I am not sure if both editors' comments are shown for each review. I think the comments are randomly picked for display. That might explain why you might see comments that are very complimentary with a lower score.

It's not an average?? That doesn't make very much sense to me. If you're not going to average the scores, what's the point of having multiple people grade the review?
Hmm... I wonder what the purpose of only using the lowest score is?

I assume you didn't mean me when you wrote, "If you're not going to average the scores." I was just passing along what was explained to me at the time. It would be better to discuss these issues with Scott directly.

[MrsCatInTheHat]

Until a reviewer gets to the highest levels, a review must be edited and scored by two different editors. The overall score you receive for a review is not an average; it is the lower of the two editors' scores. I am not sure if both editors' comments are shown for each review. I think the comments are randomly picked for display. That might explain why you might see comments that are very complimentary with a lower score. You can always contact Scott if you have questions about the score and/or comments you received Hope this helps.

I can see comments from both editors, if there were two. Sometimes there is only one editor, though, and in one case, there were three sets of comments.

I think the general and detailed (3 sets of comments) questions regarding scoring guidelines are best answered by Scott. The editors receive random reviews to edit and score, but are not involved in the rest of the process.

I didn't have a question or need clarification It was more to let people know who were wondering how much they see that more than one editor's comments can be seen.

[bookowlie]

CatintheHat - Hope I didn't offend you...it was not my intention. My comments were directed more toward the various members who continually have questions regarding scoring. I probably shouldn't have quoted you since it looked like I was referring only to you. Sorry about that.

[MrsCatInTheHat]

CatintheHat - Hope I didn't offend you...it was not my intention. My comments were directed more toward the various members who continually have questions regarding scoring. I probably shouldn't have quoted you since it looked like I was referring only to you. Sorry about that.

You offend me? No, I was just clarifying. You've been very helpful to me in the past.

[bookowlie]

No worries. I just wanted to mention it since sometimes online comments (instead of in person) can be misconstrued.

It's just a touchy subject sometimes when different members bring up issues about editors' scores and comments. The editors are trained to look at many aspects of a review, including whether it's interesting, insightful, etc. So...slight differences in scores between editors can be compared to the way different teachers in school might not grade an essay the same way. One teacher might overlook or not notice a grammatical error, be stricter or more generous in their overall assessment, etc. I had an Art History teacher in college that gave an A on term papers only if someone brought up something exceptional in their discussion of a painting or artist. Honestly, how many things can you think of to say about a painting that hasn't already been said?