In this paper, an optimum insertion/deletion point selection algorithm for fractional sample rate conversion (SRC) is proposed. The direct insertion/deletion technique achieves low complexity and low power consumption as compared to the other fractional SRC methods. Using a multiple set insertion/deletion technique is efficient for reduction of distortion caused by the insertion/deletion step. When the conversion factor is (N±α)=N, the number of possible patterns of insertion/deletion points and the number of combinations for multiple set inserters/deleters grow as α increases. The proposed algorithm minimizes the distortion due to SRC by selecting the patterns and the combinations.