We propose a new multilevel POLSK (Polarization-Shift-Keying) coherent optical system based on QAM (quadrature-amplitude-modulation) mapping. In the designing high-spectral efficiency systems, problems concern the utilization of the single-channel bandwidth, which is determined mainly by the receiver bandwidth. One way to increase the single-channel transmission rate is multilevel transmission, however due to the laser phase noise, it is difficult for conventional modulation schemes such as PSK and QAM to realize coherent optical multilevel transmission systems. The proposed system, which is based on the exploitation of the property of POLSK, can cancel the laser phase noise. Therefore, this system makes multilevel transmission possible in coherent optical communication. We theoretically analyze the receiver sensitivity of the proposed system compared with conventional M-PSK and M-QAM systems and evaluate the effect of the fluctuation of the state of polarization (SOP) on the receiver sensitivity of the proposed system. It is found that the proposed system is more efficient in utilizing the optical single-channel bandwidth and has good receiver sensitivity when increasing transmission rate, compared with conventional M-PSK and M-QAM heterodyne system with existence of phase noise.