The detection system, established in the background of ordinary surveillance video, mainly for offline two-dimensional video for abnormal behavior detection, which does not contain audio information, depth information, etc., but also no camera internal reference, height, angle and other information, and no additional calibration reference point and other operations. First, the improved YOLOv4 network is used to detect the target pedestrian in the video, and the position and number of pedestrians are obtained to realize the calculation of the number of people. Then, the illegal crowd is clustered based on L2 norm to realize social distance monitoring. Then, combined with the time attention mechanism, the EfficientNet network and ConvLSTM network are used to determine whether there is abnormal behavior in the video and realize the detection of violent behavior. Finally, the social security distance and abnormal behavior detection system of surveillance video combined with deep learning is designed and implemented.