This paper addresses low detection accuracy in rail surface defect detection. The problem comes from many defect types, large scale changes, and small dense targets. Hence, an improved model based on RT-DETR is proposed namely PSR-DETR. The PR_BasicBlock module first simplifies the model structure. It reduces parameters and computation cost. Meanwhile, it maintains satisfactory detection performance. Consequently, the network becomes more lightweight. After that, the RetC3 module adds a new attention mechanism. It enhances feature integration. It also strengthens the model’s capability to represent and distinguish targets of different scales. Finally, the SSFF module adds extra feature fusion paths. It helps the model emphasize critical regions. As a result, the detection performance is further improved. Experimental results show clear improvements, where the model does not greatly increase parameters or computation. The mAP@0.5 achieves 68.0%. The mAP@0.5:0.95 attains 44.7%, which are improvements of 6.3% and 2.7% over the original model. These findings show that the proposed method is effective and practical for enhancing detection performance.