The detection of wheel-to-tread defects in high-speed trains focuses on the identification and positioning of defects in the image, and pays more attention to the real-time model reasoning link in the actual industrial application, which makes it difficult for the relevant methods to accurately detect wheel-to-tread defects and ensure real-time performance. The implementation of defect detection relies on special inspection datasets containing expensive manual annotations. This paper proposes a multi-scale feature fusion detection system based on Yolov4, and focuses on practical industrial applications, such as high-speed train wheelset tread defect detection. To achieve strong classification capability, the system continues to use the Cspdarknet53 and SPP of each stage of the feature map of the Yolov4 backbone network. Then, the proposed multi-scale feature fusion network uses L2 normalization to standardize the features before fusion, so that the defect feature vectors of the tread of the wheelset are in the same order of magnitude. Subsequently, the features of the four stages are fused to output three fused features of different resolutions, which can contain more semantic information and location details. Based on these multi-scale fusion features, the path aggregation network is used to perform secondary fusion of feature maps with different resolutions. The improved loss function can speed up the convergence of the network. Finally, the wheelset tread defect detection dataset is established WT-DET used to train and evaluate the method. The experimental results show that the improved Yolov4 target detection algorithm proposed in this paper can achieve a detection accuracy of 86.25%, which is better than the current mainstream detection algorithm. Single GPU detection speed can reach 37.05 fps, real-time detection performance. Therefore, it has certain practical application value.