To prevent the occurrence of the kidnapped robot problem, it is vital to evaluate the likelihood of each particle considering an environmental change. Moving objects are one of the leading causes of environmental change, and each object has its own movability. For example, a chair has high movability because it is designed to move and often interacts with humans. However, walls or shelves have low movability because they are designed not to move, and they interact less often with humans. Therefore, in this study, we define classes of objects and their movability. We propose a localization approach that focuses on the association between sensor information obtained from objects whose movability is low and prior map by considering classes and movability, to prevent the occurrence of the kidnapped robot problem.