Matt Kopala

Software Development, Technology, Travel

The oDesk Rating System Is Broken

| Comments

I have been using oDesk a lot lately to outsource software development work for clients. It has been working great for me so far, and I now have 6+ people working for me, and I plan to hire more.

oDesk employs a ratings system for feedback for both Clients & Contractors. Although this seems like a good idea on the surface, it has some severe flaws, in my opinion.

The Problem

I believe many of the issues listed below artificially drive ratings higher than they probably should be.

It is important to note that Contractors and Clients do not see feedback until both parties have entered their ratings & feedback. While this is a good idea and prevents more blatant abuse, it doesn’t solve the problems.

Clients want Contractors w/ high ratings

A client will naturally want to hire a contractor with a higher rating, because there’s a higher chance that if other clients liked their work, they will too.

But even more importantly on oDesk, a client will avoid a contractor will a low rating, because they don’t want to hire someone that someone else didn’t like.

When clients search for contractors on oDesk, they see the overall rating (1 – 5 stars) of contractors prominently in the search results, under the contractor’s profile photo. When performing a search, you can even filter by rating. I have read blogs that recommend only showing contractors with a rating of 4.5 & higher.

Contractors want Clients with high ratings

A contractor wants to work for someone that is going to treat them well, and be enjoyable to work with. If applying for fixed-price jobs, I suspect (I only post hourly jobs) that a high rating is MUCH more important than for hourly jobs. For fixed-price jobs, there is no guarantee that you will get paid at all as a contractor.

The Flaws

So, what’s wrong with the rating system?

Ratings are not a bell curve

Look at the screenshot above for feedback. On a scale of 1 to 5, you’d hope that 3 is average. That is statistically impossible will the skew in ratings evident by the totals. Out of 57,000 ratings, 75% of those are 4.5 – 5.0 stars. Four stars represents Excellent and five starts represents Exceptional, in terms of oDesk.

Either a lot of people have pretty low standards, every contractor on oDesk is awesome (definitely not), or something is broken.

Contractor rates Client higher than deserved

Contractors are motivated to give high ratings to any client that they want to hire rehire them for another job. Because other contractors see the ratings from previous contractors on a client profile, a client wants to keep a high rating to remain attractive. If a contractor gives them a low rating, they will either request it be changed, or probably won’t hire that contractor again.

Client rates Contractor higher than deserved

Clients are motivated to give high ratings to contractors for a couple of reasons.

  • If they are going to rehire them, they want the the contractor to remain happy, and most contractors on oDesk are not happy with anything less than a 5 star rating (due to the flaws described in this blog post).
  • Low ratings for contractors show up in the client history. Future potential contractors see this, and will be less likely to apply or take an offer, because they are afraid they might not receive a 5-star rating as well.

Ratings can be changed

This is both good and bad.

If there’s a misunderstanding, and emotions were involved, you might want to come back and change your rating later. It’s not uncommon for contractors to request that you change ratings of less than 5 stars.

On the other hand, contractors & clients can adjust their ratings because of the very problems I described above. I doubt that ratings & feedback ever get changed to make them worse.

It’s all about the Average

Your rating is average of all your ratings. Therefore, if you’re looking to have/keep a high rating, it’s in your best interest to collect as many high ratings as possible. The easiest way to do this is:

  1. posting lots of small jobs (more work) if you’re a client
  2. working on lots of smaller jobs, if you’re a contractor

This isn’t itself a flaw. The problem is that for both clients and contractors with few ratings, it’s dangerous to give or receive a low rating (again, see above). If you’ve got a lot of ratings, you can probably afford to be a bit more objective & risk the consequences.

If you know what you’re doing as a client, finding a fresh contractor with no ratings that is skilled & competent is actually fantastic. They’ll be more motivated to please you (their average depends highly on it), and they probably won’t be getting as many offers from clients that require a rating.

Refunds to remove ratings

The only way to remove a rating from your profile is for the contractor to fully refund the client.

This isn’t so bad if you’re a client, and it’s a way to remove bad feedback & ratings on both sides.

This could easily be abused though, via the “I’ll screw your rating or give me a refund” tactic:

  1. Send a polite message to the contractor thanking them for a job well done
  2. End the contract
  3. Give a low rating and very harsh feedback
  4. Wait for the contractor to complain, and then send them the link above

I do not recommend this, and it is against the oDesk feedback regulations. I actually stumbled upon this idea when thinking of ways the system could be abused. This would be harder to prove or detect than the violations listed listed on the regulations page. Obviously, a client would not re-hire the same contractor if doing this.

Fully fabricated ratings

This is only a theory, but I wouldn’t be surprised if there existed groups contractors & clients created just to create profiles with high test scores & high ratings. Experts could be hired to pass tests, and many contracts could be made at the cost of the 10% oDesk fee in order to generate profiles that can then be used to secure real contracts with unsuspecting clients.

I have no idea if this takes place, but it’s something to think about.

The Solution

I haven’t really thought about a good solution to this overall. I think there is always going to be flaws in a rating system where money is involved. All I can do is share what has worked for me.

Here’s what I do as a client:

  • Post jobs without specifying any rating preference
  • Include a link to a short (10 – 15 minute) skills test (see below)
  • Decline any applicants that ignore or don’t complete the test
    • unless I like their profile a lot – then I will send a response telling them to complete the test
  • Review the test results, and Decline those that don’t pass
  • Respond to successful applicants and have them add me on Skype
  • Have a conversation, and conduct a simple interview
  • Hire the contractor for 5 hours (set a time limit, and have them send over what they’ve got at the end of the 5 hours

I look at ratings, but I place much less importance on them. I usually don’t spend much time looking at contractor profiles, cover letters, or oDesk test scores either. I don’t care how someone did on a job for someone else – I want to know how they’ll do on my job.

Even though I consider myself to have high standards, and would like to rate contractors objectively, I also succumb to giving higher ratings that I might otherwise. If everyone else is playing the game, it’s risky for you not to.

Testing Applicants

I’ve thrown together several short tests that I use to screen applicants. I include a link in the job posting to the test that matches the technology & skills required for the job. I’ll create a new test if the job involves technology that isn’t covered by one of my existing tests.

Although it requires a bit more work up-front on my part, I think it saves potential time wasted on spam or unqualified applicants, and helps me find contractors that are interested enough in my project to spend 10 – 15 minutes on what should be rather trivial to them, and shows me that they can follow instructions.

Conclusion

No matter what, I believe you should always be polite, respectful, and considerate when dealing with people on oDesk, whether they are clients or contractors. And remember, we’re only human.

Links

oDesk Discussions

Comments