Fences extensively cover the image in a variety of shapes, making fence removal from the image is a challenging task. Most fence removal in a single image is performed in two stages: fence area estimation and inpainting as shown in the Fig 1. Conventional fence detection has utilized information such as edge information, luminance information, and fence regularity to deal with various shapes. However, those methods have resulted in missing fences and false detections of structures similar to fences. In this paper, we propose a network that incorporates a module that includes a fast Fourier transform. This allows us to extract global features of fences that extensively cover the image in various shapes. Furthermore, we create a dataset of rectangular fences, which is not available in previous datasets. Experimental results on existing and proposed datasets show that our method achieves better results, both quantitatively and qualitatively, by suppressing missing fences and false positives of other structures.