Abstract: LLM-as-a-judge is a large language model (LLM) that evaluates the output of another LLM. While LLMs excel at qualitative evaluation, they often struggle to predict human preferences and numeric scores. We propose quantitative LLM judges, which align LLM judges in a given domain using post-hoc calibration. Specifically, we train regression models to improve the score of the original judge using its rationale and score. We present four quantitative judges for different types of absolute and relative feedback, which shows the generality and versatility of our framework. Our framework does not require fine-tuning and can be used with limited human feedback, which is expected in practice. We validate our claims on four datasets. Our experiments show that quantitative judges consistently outperform base judges on optimized metrics and are competitive with specialized LLM judges, a larger off-the-shelf model, and fine-tuning by TRACT.