Lately, there has been a need to improve the efficiency of material movements within factories and multi-agents are required to perform these tasks. In this study, graphical representation and mixed-integer programming have been adopted for simultaneous optimization of task allocation and path planning for each agent to achieve the following three goals. First, this study realizes time and capacity constrained multi-agent pickup and delivery (TCMAPD) that simultaneously considers time constraints, capacity constraints, and collision avoidance. Previous studies have not considered these constraints simultaneously. Thus, we can solve the problems associated with using multi-agents in actual factories. Second, we achieved TCMAPD that optimizes the collision avoidance between multi-agents. In conventional research, only a single collision avoidance method can be used. However, an appropriate route was selected from a variety of avoidance methods in this study. Hence, we could achieve a more efficient task allocation and path planning with collision avoidance. Third, the proposed method simultaneously optimizes task allocation and path planning for each agent. Previous studies have separately considered the approach of optimizing task allocation and path planning or used the cost of path planning after task allocation to again perform task allocation and path planning. To simultaneously optimize them in a single plan, we have developed a solution-derivable formulation using mixed-integer programming to derive a globally optimal solution. This enables efficient planning with a reduced total time traveled by the agents.