Online rating system

[AmbivalentDuck]

I've been thinking of implementing some sort of online "rating" program for some time (and maybe tying a PHP bulletin board like TMD into it).  The idea would be creating a rating service so people could find suitable opponents on MWS and let "kitchen table" games count.  I'm guessing that other people would also be interested in quantifying their play skill without driving to or sitting through tournaments to do it.

My thought so far on something relatively difficult to scam would require both players to request a matchslip.  Then require both to submit a copy of the slip. (And some way to decide who's right if they disagree...or just punish both?)

I'd then use network theory based analysis to figure out who's played who and give rankings within networks.  I'm guessing that we'd have *very* few subnetworks if I trim inactive players every month or two.

The hardest part is what to do *automatically* when people disagree.  The second hardest part is how to handle rankings.  I think that I'd just mark it as *worse* than a loss for both to encourage people to work out their differences.  Some sort of citizenship/agreement ranking might help players to weed out any bad apples.  On the second point, I'm thinking I should research and choose one of the "established" rating systems.  Any recommendations on which?  A point system?  A ladder system?

Any thoughts?  This would theoretically be pretty easy to write in PHP and not take too much bandwidth to host.  And obviously I'd hand source code to anyone who wanted it.  I'm thinking the results would be stored in a MySQL or PostGRESQL database.

[Anusien]

Try http://www.o-gaming or http://www.magic-league.com.  They already do exactly what you want them to do.  Along with tournaments.

[AmbivalentDuck]

Those essentially implement tournaments/leagues.  That forces your hand on who you play and when.

The "big idea" behind using a network theory approach is that your could use the elegant analytical techniques that exist for analyzing networks and get a much, much better idea of where someone stands in actual playskill.

The "randomer" sampling also would enable players to play who they want, when they want, while still getting an accurate rating.

The questions are (I think) what network model to choose and how to "rank" isolated networks.  Since many network models could ignore most simple types of abuse, it's actually a secondary (though important) consideration.

[kirdape3]

I'm pretty sure that the actual ranking system that paper Magic uses goes by tournament placing only.  It leads to some inflated ratings in the short term, but the rating system generally is an accurate measure of playskill over enough matches (for most people, >100).

[Godder]

The Magic rating system is a bastardised version of the Chess Elo system, but goes round by round, whereas Chess is done tournament by tournament, using the same rating across a three month period with ratings updated every three months. It's much more statistically sound, whereas the Magic version completely misses the point, thus having little statistical relevance.

[Anusien]

I'm not certain about O-Gaming, but Magic-League allows you to play single matches.  They just have a smaller k-rating.

It seems like the purpose of this is to both establish a ranking system and facilitate matchmaking.  Both those systems do exactly that.

[cssamerican]

The best system I have seen on the internet for handling reporting was quite simple. Loser reports the loss. If someone doesn't report a lose the winner simply reports a complaint against them for not reporting. Once someone gets a reputation for not reporting people will quit playing them. The system I used you just look up the persons name on the reporting site, there you can see their rating and how many complaints they have for non-reporting.

A player can only post a complaint for a particular player non-reporting once in a designated time period, say three months. And the complaints aren't wriiten complants, it is just a counter for failure to report. Every six months the counter loses the first three months of complaint counts. This way people can repair their reputation yet it is not instantaneous.

[AmbivalentDuck]

I'm not certain about O-Gaming, but Magic-League allows you to play single matches.  They just have a smaller k-rating.

It seems like the purpose of this is to both establish a ranking system and facilitate matchmaking.  Both those systems do exactly that.

The notion of a k-rating or any sort of statistic that require 100+ trials (per player) to be accurate is precisely what I'm trying to avoid.  My hope would be to abuse something like eigenvector centrality to apply a traditional rating scale only to players with a certain level of importance, and then assign relative (and less statistically sound) ratings to less important players based on their performance against more important players.  Importance here would actually be a measure of how "Magically" promiscuous a person is.

The idea would be that everyone wouldn't have to start at some median value and play tons of games to "work their way up:" their relative skill would be apparent (with appropriate error margins) as long as at least some players were playing many games and promiscuously choosing their partners.

[Godder]

Another bonus of the chess system is that unrated players don't start at a set rating. Rather, their rating is calculated from their results in their first active rating period.

[AmbivalentDuck]

The chess rating system is really attactive, but it has some pretty stiff assumptions built in:

-Your "win percentage" stays nearly constant over time.
-Game/match/tourney wins are direct indicators of skill (you have to pick one of the three; match and game wins probably don't compare well).
-You play enough matches that assumptions about win percentages being normally distributed work out.

That system works just fine as long as:

-Different decks don't wildly swing your win percentage game to game
-You only play single games/matches/tourneys
-You play a lot of them

So, it's probably fairly reasonable to use a Elo-like system for pro players.

That said, it's probably not reasonable when applied to pretty much everybody else as people have been known to choose sub-optimal decks inconsistently and play a very few games with them.

[Godder]

It's not so much that your individual win percentage will remain the same, but that your win percentage against a particular rating will remain the same. If it doesn't bear out, your rating goes up or down accordingly.

Also, club chessplayers are not nearly as active as active FNM Magic players in terms of rated games (which I'd liken to matches in Magic) - I play about 50 rated chess games per year, for example, and that's quite high for my club, and yet, my rating is considered accurate. An active FNM player would play around 200 rated matches per year, so I can't see activity being an issue for most players. For relatively inactive players (say, a tournament every year or so), their rating won't be very accurate no matter how you do it.

An unrated chessplayer will be given a provisional rating which is calculated from his score vs rated players, and then it's recalculated with new games/tournaments until their rated games reaches whatever threshold is in place (here in NZ it's 24).

As a side note, the DCI claims that its rating system is an Elo system, because it uses the Elo calculation, even though it misses the point entirely.

[MarkPharaoh]

An unrated chessplayer will be given a provisional rating which is calculated from his score vs rated players, and then it's recalculated with new games/tournaments until their rated games reaches whatever threshold is in place (here in NZ it's 24).

It's 20 here in the States.  Also, fifty games for a club player a year is indeed pretty high.