ILU (or Incomplete LU) decomposition is one of the most popular preconditioners for large and sparse linear systems of equations. However, it is difficult to implement the ILU preconditioner on distributed memory parallel computers, because the process consists of forward and backward substitution. The block divided method is one of the algorithms that can paralletize the ILU preconditioner for the linear system obtained by applying the finite difference method to discretize the elliptic boundary value problem of the PDE (or partial differential equation). However, on a distributed memory parallel computer, since the communication overhead is significantly large, the ILU preconditioner does not perform well. We propose an algorithm that decreases the communication overhead on the block divided method and determines the appropriate band-size. Based on our approach, the BiCGStab(g) method with the ILU preconditioner is implemented on the distributed memory parallel computer, Fujitsu AP3000. We also analyze the performance of parallelism in the operation of the ILU preconditioner through numerical results.