Abstract: A new method based on Radial Basis Function(RBF) neural network was proposed to correct the system error of a optical encoder. The modeling method of RBF was introduced in detail and the theoretical basis for adjusting the parameters of the model was given. A new model for error correction was set up by taking the test values of the high precision instrument as outputs and the angle values of sample points as inputs. The testing results show that the precision of the encoder by this method has increased by 3 times as compared with that of traditional method and the precision of measuring system is improved greatly by using the RBF model as error compensation.