Recently, the agricultural population is decreasing, and the demand for autonomous agricultural robots has increased. For a robot to successfully harvest fruits in diverse environments, various obstacles must be pushed away from its path. During the pushing motion, however, gravity acts on the fruits, causing them to move back and forth and return to their original state after being pushed away. This results in fruit-to-fruit contact, and fruit damage may occur. Therefore, a method to avoid obstacles while reducing the risk of fruit damage is required. However, the direct consideration of fruit damage is difficult because it requires many parameters that are difficult to observe. Herein, we propose a motion planning method by considering the potential energy of fruits as an evaluation index, which could be calculated based on observable parameters; this consideration could also suppress the reciprocating motion of fruits, thus preventing fruit damage. The proposed method was evaluated through simulations, and its effectiveness was verified.