Ranked Choice Scholarship Scoring

JoAnnaDarda

Our organization utilizes a ranked scoring system for our Lilly Endowment Community Scholarship Program, and we are struggling to translate that system effectively into the evaluation tools in SLM.

For this process, our evaluators are evaluating a pool of 10 finalists (one representing each eligible high school in the county). There are 10 questions/categories where evaluators must rank each applicant on a scale from 1 to 10, with only one student at each rank. We send evaluators a spreadsheet where they can work out their scores more easily. Then, we ask the evaluators to enter their scores for each student in the evaluation forms in SLM.

This year, when pulling evaluation scores from SLM, we noticed a majority of evaluators had at least one error when translating their ranked scores to the SLM evaluation form. Some errors were minor, but some evaluators faced some significant scoring errors. This required a lot of communication between Foundation staff and evaluators to correct.

The SLM evaluation form is linked to the individual applicant records, but our method for scoring this scholarship requires a comparison of all applicants for each question. I'm looking for ways to make this process easier for our evaluators, and more accurate for next year. Does your organization utilize ranked scoring? Do you face these challenges as well, or do you have an effective system in place? I would love to hear all your thoughts and ideas!!