In view of the problems of slow detection speed, low accuracy and missing detection and false detection of current forestry pest detection methods, a real-time detection system of forestry pests based on attention model and lightweight YOLOv4 was designed. The specific work includes: constructing the data set, using geometric transformation, random color dithering and mosaic data enhancement techniques to preprocess the data set; replacing the YOLOv4 backbone network with a lightweight MobileNetV3 network model, and adding CBAM attention model to the improved PANet network to build an improved lightweight YOLOv4 network model; introducing Focal Loss to optimize the loss function of YOLOv4 network model; the pre-processed data set is input into the improved network model, and the detection results containing the pest species and location information are output. The improvement points of the network designed by this system have contributed to the performance improvement of the system; the detection speed is faster than the original YOLOv4 model, the average detection accuracy is higher, and the problem of missed detection and false detection is effectively solved; the comparative experiment proves that it is better than other mainstream network models, and meets the accuracy and speed requirements of real-time pest detection.