State of the Art in Digital Agency Ranking
Our industry is a very subjective one. A client likes your design today and hates it tomorrow. You enter a Behance* profile and find very nice looking portfolios with sometimes, too much curation and end up with an under representative sample of work that only shows you the very best productions of a particular professional.
The last group in the graph is essentially award-winning work, the middle 95% is most likely representative of the usual performance of the professional, or agency, while the low 2.5% can be thought of as “unpublishable blunders”. The problem in the web design industry is that, right now, there is absolutely zero visibility into the blunders and very low visibility of the typical 95%. Agencies and designers only show their very best work and this makes it harder for us to pick the most consistent players. Both as employers and as end customers of digital agencies.
Agencies and sometimes designers are ranked primarily based in the number of awards they have obtained. This rank, while a subjective measure is also a very unfair one because it makes it harder for the unestablished designers to move up and get better clients. However, rankings such as these use only award counts to rank agencies. In conclusion the current of affairs as awards can be subjective, can over-celebrate brand association and can reward friendships and particular design styles.
Enter AgencyRank
A better way is to rank professionals and agencies based on the success of their work and the importance of each production. In this case, the judgment should be performed by clients not judges, so that we capture the very important 95% piece of productions.
The way to calculate this is very simple, we first need a catalog of sites in a given industry, say energy for example. Then we produce a public list with the sites and key public figures provided by clients: page rank variation (or logarithm variation to account for an exponential raise in page rank), bounce rate change, average time spent on page, and so on. You can click the following image to see a sample calculation**.
In this particular competition, the judges assign more weight to bounce rate reduction because, probably, page rank is not naturally distributed in this particular industry. Or maybe this is a web design and not a web marketing competition. Making these equations public, giving clients the ability to self-report the numbers and come up with a more reasonable set of criteria can produce better rankings and help the whole industry rank creative professionals in a more reasonable manner.
A similar metric can then be self-reported in oDesk and the like to give us principals are more accurate view of the capabilities of each professional or group of creative types. Giving clients the wheel in this scenario also helps in capturing more of that 95% of the performance that still remains in the dark.
* Behance is a showcase site for creative professionals. A good tool for sharing and discovering the best work of creatives but not a particularly enlightening one when it comes to hard-figure analysis.
** I don’t account for page rank’s exponential nature. Feel free to make adjustments to the formula and consider including logarithms in your page rank calculations to adequately account for big jumps in page rank.