The effect law of deformation and failure of a jointed rock mass is essential for underground engineering safety and stability evaluation. In order to study the evolution mechanism and precursory characteristics of instability and failure of jointed rock masses, uniaxial compression and acoustic emission (AE) tests are conducted on sandstones with different joint dip angles. To simulate the mechanical behavior of the rock, a jointed rock mass damage constitutive model with AE characteristic parameters is created based on damage mechanics theory and taking into account the effect of rock mass structure and load coupling. To quantify the mechanism of rock instability, a cusp catastrophe model with AE characteristic parameters is created based on catastrophe theory. The results indicate that when the joint dip angle increases from 0° to 90°, the failure mechanism of sandstone shifts from tensile to shear, with 45° being the critical failure mode. Sandstone's compressive strength reduces initially and subsequently increases, resulting in a U-shaped distribution. The developed damage constitutive model's theoretical curve closely matches the test curve, indicating that the model can reasonably describe the damage evolution of sandstone. The cusp catastrophe model has a high forecast accuracy, and when combined with the damage constitutive model, the prediction accuracy can be increased further. The research results can provide theoretical guidance for the safety and stability evaluation of underground engineering.