Because of spatial variability and complex compositions, the mechanical test results of natural soil–rock mixtures (SRMs) are often discrete and lack reproducibility, which has greatly restricted the practical application of the experimental findings. The objective of this study was to examine the general mechanical behavior of SRMs under the influences of some hidden factors (e.g., structural parameters, parent rock type and weathering degree). To that end, the abstraction idea was adopted to prepare purified SRM samples. Large-scale triaxial tests were performed on these purified materials. On this basis, the influences of three structural parameters on the mechanical behavior of SRMs were studied. Moreover, the relationship between the shear strength and parent rock type and that between the shear strength and the spatial distribution of rock blocks were quantified. Some additional intrinsic behavior was distinguished from individual experimental phenomena through the comparative analysis of the test data in this study and those reported in the literature. The results show that the hidden factors had significant influences on the mechanical behavior of SRMs. A greater saturated uniaxial compressive strength of rock blocks generally led to a larger shear strength of SRMs. According to the significance of their influences on the shear strength parameters of SRMs, the structural parameters are ordered as: the gradation of rock blocks, the initial dry density of sample and the spatial distribution of rock blocks. The deformation and failure feature of SRMs were considerably affected by the spatial distribution of rock blocks and shear rate. And the shear strength parameters of SRMs were mainly influenced by the content of grains between 40 and 60 mm. The findings of this study would provide useful guidance for engineering practice.