In recent years, there has been a demand for turbomachinery with increasing speed, pressure, and compactness owing to the increasing demand for high output and efficiency. Consequently, various shaft vibration problems have become apparent. The rotor dynamic (RD) forces are one of the causes of these axial vibrations, which can lead to instability of the rotor system. To ensure the reliability of turbomachinery, the effect of RD fluid force should be analyzed at the design stage. In this study, high-precision tracking control of a complex orbit was analyzed using an experimental approach, and a highly efficient and low-dispersion estimation of the RD fluid force was achieved. The target was a parallel annular seal, and an experimental system was developed to control the rotor position using a piezo-electric actuator. Using the adaptive feedforward cancellation (AFC), we suppressed the periodic disturbance caused by rotation, whirling, and disturbances, such as noise that existed uniformly at all frequencies. Thus, a high-precision tracking control with an orbit tracking error of approximately 1 μm was achieved. The RD fluid force was estimated from the output signal of the AFC during high-precision tracking control, and the RD coefficient was calculated using spectral analysis. Consequently, a highly accurate RD fluid force estimation with reduced dispersion was achieved. In addition, a multifrequency orbit control and RD fluid force estimation method were developed to improve the efficiency of the RD force estimation, which agreed well with the results of a single-frequency orbit.