Any way to recover differences in subjective judging?

For the discussion of math. Duh.

Moderators: gmalivuk, Moderators General, Prelates

hnooch
Posts: 128
Joined: Mon Nov 26, 2007 6:55 pm UTC

Any way to recover differences in subjective judging?

Postby hnooch » Tue Jun 15, 2010 3:37 am UTC

Suppose we have m competitors and n judges. Each judge uses her own private evaluation method to assign a value to each competitor. The judge then releases (to the public) her ranking of the competitors, from best to worst — the actual values are not released, so we don't know how much better the first place winner is from the runner-up.

A sample ranking might look like:

Code: Select all

        Judge  Xavier Yeats Zane
Competitor
Alan              1     1     2
Beth              3     2     1
Carter            4     3     3
Dima              2     4     4

Now we have each judge's ranking of the competitors, but we don't know their weighting system. Given the similarity or differences across different judges' rankings, is there anything we can say about how similar their systems are or how different in value the competitors are? In the example above, Alan looks like a better competitor than Dima. Can we make this precise somehow?

It seems like too much information is lost. If the judges have similar ranking systems, then if their rankings seem to agree, the competitors were likely spread out, and if the rankings are random-looking then the competitors were probably bunched together. But if the judges have different enough ranking systems, then the results will probably look random. Is there any way to quantify this?

To make the setting more concrete, an assumption that sounds reasonable to me is to represent competitors by vectors in a large Euclidean space (one dimension for each criterion the judges care about) and represent judges by (semi)norms on this space; a judge evaluates a competitor by taking its norm.

Anyone have any thoughts? Is there any other information about the judges that could reveal their internal correlations with each other? (e.g. their rankings of another set of competitors)

rhetorical
Posts: 103
Joined: Wed Apr 21, 2010 12:44 am UTC

Re: Any way to recover differences in subjective judging?

Postby rhetorical » Tue Jun 15, 2010 1:01 pm UTC

By different ranking systems, do you mean they look for different things in the competitors performance? or that, for example, a 2 is much closer to a 1 that 3 is to 4 for one judge, and the opposite for another?

User avatar
Velifer
Posts: 1132
Joined: Wed Dec 26, 2007 4:05 pm UTC
Location: 40ºN, 83ºW

Re: Any way to recover differences in subjective judging?

Postby Velifer » Tue Jun 15, 2010 1:04 pm UTC

Start digging into the math behind nonparametric statistical tests like Mann-Whitney-Wilcoxon ranksum, and also check out Cohen's Kappa for inter-rater reliability (although in this case, inter-rater differences will hold more information).
Time flies like an arrow, fruit flies have nothing to lose but their chains -Marx

hnooch
Posts: 128
Joined: Mon Nov 26, 2007 6:55 pm UTC

Re: Any way to recover differences in subjective judging?

Postby hnooch » Tue Jun 15, 2010 5:29 pm UTC

Thanks for the references! Cohen's kappa sounds similar to what I want; I'll keep looking.


Return to “Mathematics”

Who is online

Users browsing this forum: No registered users and 10 guests