In deciding which papers to recommend for acceptance, I looked at the following things: recommendation score, confidence score, text of the review, text of the author response, discussion between reviewers, who the reviewer is and what I think they understood about the paper, and the paper itself — I read or at least skimmed the paper in all of the borderline cases.

Things I did not look at: any of the other numerical scores. But having reviewers to fill out these scores might still help the reviewing process, by forcing reviewers to think about the criteria that the program chairs feel are important.

I don’t know how the program chairs made the final decisions, but area chairs were given a lot of leeway in making recommendations. One piece of advice that we got, which I agree with, is to favor papers that elicited strong opinions (as long as at least one opinion was positive) over papers that all reviewers were tepid about. The logic is that if 20% of the ACL audience is excited about a paper and 40% hate it, that’s better than a paper that 100% of the audience is apathetic about.