Recently damage detection methods based on measured vibration data for structural health monitoring (SHM) have been intensively studied. In order to decrease the number of required sensors, however, most of their methods focus only on single dimensional systems, in spite that there are some cases that torsional vibration greatly affect for structural damage. Although some studies consider multiple dimensional systems using frame structures, usually they need lots of sensors and calculation is time-consuming. Therefore, the balance between the cost and the particularity is very important for SHM system. In this paper, a method to localize the damaged area of multi-story buildings considering torsional components is proposed to detect the damage simply and particularly. This method focuses on shift in the center of rigidity caused by induced damage. The damaged quadrant of a certain story is identified comparing story eccentric distances of before and after damage-inducing seismic events. An adaptive extended Kalman filter (AEKF) is utilized to identify unknown structural parameters. Using a model which has four columns in each floor, several cases are considered in the verification study to disclose the capability of our proposed method.