Fatigue failure of gear transmission is one of the key factors that restrict the performance and service life of wind turbines. One of the major concerns in gear transmission under random loading conditions is the disregard of dynamic fatigue reliability in conventional design methods. Various issues, such as overweight structure or insufficient fatigue reliability, require continuous improvements in the reliability-based design optimization (RBDO) methodology. In this work, a novel gear transmission optimization model based on dynamic fatigue reliability sensitivity is developed to predict the optimal structural parameters of a wind turbine gear transmission. In the model, the dynamic fatigue reliability of the gear transmission is evaluated based on stress–strength interference theory. Design variables are determined based on the reliability sensitivity and correlation coefficient of the initial design parameters. The optimal structural parameters with the minimum volume are identified using the genetic algorithm in consideration of the dynamic fatigue reliability constraints. Comparison of the initial and optimized structures shows that the volume decreases by 3.58% while ensuring fatigue reliability. This work provides new insights into the RBDO of transmission systems from the perspective of reliability sensitivity.