At calibration time, the Checkerboard calibration tool computes both the linear and nonlinear components (when the tool is in nonlinear mode) of the overall transformation of the vertices from raw calibrated space to the calibration image's selected space. The Checkerboard calibration computes the RMS error of the computed transformation.
The tool also generates error strings that help you evaluate the size and possible cause of the computed error. In general, the most likely cause of excessive error is specifying linear mode calibration when the calibration image exhibits significant nonlinear distortion. Since the Checkerboard calibration tool automatically determines the correspondence between points in the two spaces (unlike the N-Point tool), high RMS error is unlikely to indicate a correspondence error. Instead, a high RMS error may indicate a failure to locate tile vertices correctly, or it may indicate the presence of distortion other than simple perspective and radial distortion.