Gully slope is one of the most active areas of soil erosion in small watershed of the Chinese Loess hilly–gully region. Although its soil erosion has been effectively controlled with the implementation of the “Grain-for-Green Program” in this region, the soil water storage and distribution have been also impacted. In particular, unreasonable revegetation model has aggravated the water shortage, which may in turn threaten the health of ecosystems. However, yet little is known about the effect mechanism of vegetation on soil water in the gully slope. In this study, we examined the relationship between two revegetation modes, including afforestation (i.e. black locust forest) and natural revegetation (i.e. the grassland), and soil water in a depth of 0–120 cm of the gully slope, during the rainy season. The results showed that the effect of the vegetation to the soil water was smaller than that of the precipitation. Furthermore, the response of soil water to the environmental factors was higher in afforestation vegetation due to its lower soil water content, resulting in higher space dependence for soil water, compared to the natural revegetation. The lower soil water content of the black locust forest was mainly caused by its higher recession rate, not its supply. The soil water was deficient for a long time, caused by afforestation, with a shallower formation depth of the dried soil layer and stronger desiccation degree. However, this deficient could not be effectively relieved until in wet year. In comparison with the ridge slope, the effect of vegetation to soil water in gully slope was stronger, with greater water consumption in afforestation vegetation and the higher water storage in natural revegetation. From the aspect of water resources conservation on the water scale, the natural revegetation was the optimal revegetation mode in the gully slope of the loess hilly region.