## Any way to recover differences in subjective judging?

For the discussion of math. Duh.

Moderators: gmalivuk, Moderators General, Prelates

hnooch
Posts: 128
Joined: Mon Nov 26, 2007 6:55 pm UTC

### Any way to recover differences in subjective judging?

Suppose we have m competitors and n judges. Each judge uses her own private evaluation method to assign a value to each competitor. The judge then releases (to the public) her ranking of the competitors, from best to worst — the actual values are not released, so we don't know how much better the first place winner is from the runner-up.

A sample ranking might look like:

Code: Select all

`        Judge  Xavier Yeats ZaneCompetitorAlan              1     1     2Beth              3     2     1Carter            4     3     3Dima              2     4     4`

Now we have each judge's ranking of the competitors, but we don't know their weighting system. Given the similarity or differences across different judges' rankings, is there anything we can say about how similar their systems are or how different in value the competitors are? In the example above, Alan looks like a better competitor than Dima. Can we make this precise somehow?

It seems like too much information is lost. If the judges have similar ranking systems, then if their rankings seem to agree, the competitors were likely spread out, and if the rankings are random-looking then the competitors were probably bunched together. But if the judges have different enough ranking systems, then the results will probably look random. Is there any way to quantify this?

To make the setting more concrete, an assumption that sounds reasonable to me is to represent competitors by vectors in a large Euclidean space (one dimension for each criterion the judges care about) and represent judges by (semi)norms on this space; a judge evaluates a competitor by taking its norm.

Anyone have any thoughts? Is there any other information about the judges that could reveal their internal correlations with each other? (e.g. their rankings of another set of competitors)

rhetorical
Posts: 103
Joined: Wed Apr 21, 2010 12:44 am UTC

### Re: Any way to recover differences in subjective judging?

By different ranking systems, do you mean they look for different things in the competitors performance? or that, for example, a 2 is much closer to a 1 that 3 is to 4 for one judge, and the opposite for another?

Velifer
Posts: 1132
Joined: Wed Dec 26, 2007 4:05 pm UTC
Location: 40ºN, 83ºW

### Re: Any way to recover differences in subjective judging?

Start digging into the math behind nonparametric statistical tests like Mann-Whitney-Wilcoxon ranksum, and also check out Cohen's Kappa for inter-rater reliability (although in this case, inter-rater differences will hold more information).
Time flies like an arrow, fruit flies have nothing to lose but their chains -Marx

hnooch
Posts: 128
Joined: Mon Nov 26, 2007 6:55 pm UTC

### Re: Any way to recover differences in subjective judging?

Thanks for the references! Cohen's kappa sounds similar to what I want; I'll keep looking.

### Who is online

Users browsing this forum: No registered users and 10 guests