JAMS International ADR Blog

Get the latest updates from our blog:

Evaluating arbitrators: you be the judge

by Matthew Rushton on 10 Nov 2016

evaluating_arbitratorsIn 2007/8, I set up a now long-since defunct website (disputesloop.com) with the lofty aim of introducing some transparency into the appointment process of ADR neutrals. At its core, the site offered a battery of neutrals' CVs supplemented with free-form written feedback from users.

On one level, it aimed to substitute the kind of third-hand market gossip that makes some ADR careers and breaks others, with something substantial, first-person and, importantly, in the public domain. I was keen for this feedback repository to develop into a body of market knowledge to inform the uninitiated and to develop a framework for objective discussion of the competencies of mediators and arbitrators.

The hope was that users would become more discriminating in their choice of ADR professionals, more concerned to match skills and competences to the requirements of the case, and more articulate about what they wanted from ADR professionals. I also hoped that the site would guide the market's invisible hand to push out those who overtraded and under-delivered, while rewarding the capable and conscientious.  

So far, so naïve. 150 or so noble souls signed up (and paid!) to offer their services via the site, and to subject themselves to the market's slings and arrows. The only constraint on clients was an insistence that comments be attributed. And over time, and with some prompting, feedback did appear on the site.

You'd be right to infer, however, from the site's non-existence that the venture was not a roaring success. The trouble was that the feedback wasn't terribly valuable and the market wasn't terribly interested. With hindsight, the reasons are entirely predictable. The first problem with the site was lack of confidentiality leading to lack of candour. Being mostly British, the comments were mostly polite. Anodyne, one might say.

But so what? If an arbitrator was "well prepared", "responsive", "polite" and the award was "well-reasoned" and "promptly delivered", is that not a good enough reason to consider him or her for the next appointment? Well, no, actually.

Pinpointing the arbitrators not to work with

I went back to my market (private practice lawyers) to investigate their desultory interest in the site. The message was clear: the positive attributes were somewhat helpful, but the information they really valued what not "who’s any good", but "who’s seriously dreadful". Knowing who to avoid was, and remains, the aim of the game. A good arbitrator does not guarantee a good result, but a careless, disorganised, disreputable arbitrator does guarantee the destruction of a hard-won client relationship.

If you are imagining that my site's ultimate demise came as a result of defamation claims from angry arbitrators, you'd be wrong. Nobody wants to criticise an arbitrator in a public forum for fear of reprisals in any future hearing. That, too, is unhealthy and sustains many a creaking practice.

However, feedback is not the ne plus ultra of transparency that some imagine it to be. It's biased (consciously and unconsciously), flawed, and ultimately misleading. Mediators, particularly, can get a raw deal. In caucus sessions, parties don't and cannot know what's going on in the other room. Your mediator might be working hard with one party to achieve a shift in attitude; it might take hours, and it might ultimately fail. The other party, feeling abandoned, and with no settlement forthcoming might feel justified in giving poisonous feedback. And sometimes they do; it's first-hand, objective, but also entirely unfair.

Lack of fairness, however, is not a reason to refuse to engage with feedback; rather, it is something that must be factored in. If one dismisses both the unmerited abuse and unjustified praise, somewhere in the middle are valuable insights. The mere fact that such feedback has been put in writing gives it a status and credibility above word-of-mouth recommendations, which exert disproportionate influence.

Almost a decade ago, I had an illuminating experience of the shortcomings of word-of-mouth feedback. I shadowed one UK mediator whom I thought I knew well from feedback accumulated verbally over the years. I thought I had a good sense of his style and how he might approach the case and handle the parties. But to my great surprise the caricature assembled from multiple sources in no way resembled the reality. I was confounded on every front: where I'd anticipated an austere, and forbidding manner, he was levity, modesty and bonhomie. Where I'd anticipated the heavy slugging of evaluation, he tiptoed around the issues with finesse and grace. Where I'd expected an imposing ego, I found a man who, being shown the door, stood obligingly in the corridor for hours at a time.

Two further points arise from this encounter: first (tangentially), I'd been told by one law firm involved just days before that the mediator concerned was one that they would never use again. His name was inked in their black book under the heading "never more". Patent nonsense. Secondly, whenever I've related this experience to people in the market they've refused to believe me: he was putting on an act for your benefit, they say. Such is the incontrovertible power of rumour.

You be the judge

Flawed or not, market feedback is here to stay. The old adage that you can't improve what you can't measure applies to ADR. And while we're still in the early stages of understanding what and how to measure performance, it remains in my view critical that the market is given a voice and provoked into dialogue. JAMS International this week is launching a revamped tool ("you be the judge") for users to give their feedback – anonymously if necessary before, during and after ADR processes. Any and all feedback is welcome. The challenge for providers, in light of the above, is interpreting and acting on feedback. I look forward to hearing from you.


Topics: ADR, Arbitration

Matthew Rushton

Written by Matthew Rushton