An example demonstrating repeatability can be seen in figure 1, which shows a spring-loaded position sensor in a typical gaging stand being calibrated with a precision gage block of 0.5000 inch dimension. The sensor delivers an output of 0 to 10 Volts DC full scale for 0 to 1 inch of probe movement. The tip of the sensor is moved inward to allow the gage block to be inserted between the tip and a flat base, and then released to contact the block. In 5 trials, system outputs are: 5.0012, 5.0016, 5.0013, 5.0010, and 5.0015 Volts DC. The average value is slightly over 5.0013 Volts and the maximum variance is ±0.0003 Volts, which is equal to ±30 ppm of FSO, or 0.003% of FSO.
Resolution is a measure of the smallest change in the input to a sensor-based measuring system that will produce a measurable change in the electrical output from that system. While this may seem like a fairly simple concept, it is impacted by factors external to the sensor itself, the most significant of which is the signal-to-noise ratio of the system's analog output. Electrical noise present on the system's output can reduce the effective resolution of the system by masking any small changes in the sensor's output. For example, if the sensor's resolution specification is 0.25 mV, but the system output noise and ripple is 2 mVp-p, clearly sensor output changes smaller than 2 mV will not be discernable within that noise. Thus, the actual system resolution is only about 12% of what the sensor resolution specification offers.