Particle size strongly influences the shear strength of granular materials. However, previous studies of the particle size effect have focused mainly on the macroscopic behavior of granular materials, neglecting the associated micro-mechanism. In this study, the effect of particle size on the shear strength of uncrushable granular materials in biaxial testing is investigated using the discrete element method (DEM). First, a comprehensive calibration against experimental results is conducted to obtain the DEM parameters for two types of quartz sand. Then, a series of biaxial tests are simulated on sands with parallel particle size distributions to investigate the effect of particle size on macro- and microscopic behaviors. Finally, by adopting the rolling resistance method and the clump method, irregular-shaped particles are simulated to investigate how the particle size effect will be influenced by the particle shape. Simulation results demonstrate that (1) the peak shear strength increases with particle size, whereas the residual shear strength is independent of particle size; (2) the thickness of the shear band increases with the particle size, but its ratio decreases with particle size; (3) the particle size effect can be explained by the increase of friction utilization ratio with particle size; and (4) the particle size effect is more significant in granular materials that consist of particles with higher angularity.