Award-winning Work Information & Description
MetaThroat-Real-time Sign Language Translation System Based on Edge Computing AI TPU, YOLOv4-SPP and Multi-sensor Fusion
Description
Aiming at the problems of low recognition accuracy, slow recognition speed, dependence on the network, heavy equipment, high failure rate, poor practicability, and inability to popularize the current sign language translation device, this work uses edge-based artificial intelligence tensor processor TPU, big data processing, computer vision deep learning neural network, multi-sensor data fusion and voice control technology to develop a portable sign language real-time translation system with high accuracy and fast recognition speed, which can be used for high accuracy recognition of own sign language different from the standard sign language library and does not rely on the network. And for the deaf can not perceive the outside sudden alarm sound and easy to fall into the dangerous pain point, this product realizes the vibration and multi-color light flashing way to remind, with high practicality. The system development is based on the TPU( BM1864) platform of computing energy edge computing artificial intelligence processor. It is equipped with USB camera, speaker, display screen, heart rate/skin electric sensor module, sound detection module, color-changing LED lamp and micro motor reminder. It includes data acquisition, annotation, model training, model conversion, OpenCV video capture and flow resolution, gesture recognition YOLOv4-SPP neural network algorithm, UDP push recognition category, it is composed of screen display, multi-sensor data fusion emotion detection, voice broadcast UI, voice control lighting and vibration prompt. Its innovation is mainly reflected in: (1) the use of TPU BM1684 deep learning proprietary tensor processor power to achieve local computing, fast and independent of the network. (2) Support custom proprietary gesture sample library, to solve the real-world sign language types, large differences, low recognition accuracy, can not be commercialized core problems, become the industry's first. (3) Target detection based on deep convolutional neural network YOLOv4-SPP supports 1600.00g accuracies of INT and FP, which can optimize recognition speed and accuracy. (4) Through the video capture and decoding based on OpenCV, the image analysis reasoning and post-processing intelligent scheme reduces the false detection rate and greatly improves the recognition accuracy. (5) Through the proprietary training model library and the common standard library, the reasoning can be switched to realize the semantic intelligent push of fuzzy expression. (6) And by building a UDP user datagram protocol platform, the category of the recognition result is uploaded and broadcast through the display and voice to form a friendly UI scheme. (7) At the same time, the sound detection module, multi-color LED lights and micro-motors remind the deaf in time through colorful lights or vibrations when someone outside the deaf's field of vision greets them or warns them loudly. (8 Integrated multi-biosensor module to detect the emotion of sign language users in real time, and broadcast the speech of sign language translation with the intonation and speed that best matches the emotion. (9) This system does not rely on the network, the algorithm is carried out in the machine, which can be applied to the non-network environment, expands the user environment, and greatly facilitates the sign language users. (10) This scheme is a self-carrying device, without the assistance of others, easy to use. The system has independent intellectual property rights, and the software and hardware system has entered the pilot stage. The measured results of the system show that the recognition accuracy is as high as 96%, the recognition speed is 20 frames FP32 and 34 frames INT8 per second, which can almost achieve lossless translation of personalized sign language and is worthy of great promotion. The novelty search results of the project show that there are no similar functional products. At present, a soft book, two related patents in the application. We will further improve core technologies such as recognition algorithms and emotional interaction, and miniaturize products to enhance the ease of use of products, and bring products to the market as soon as possible. At the same time, it will cooperate with diversified product suppliers, such as providing technical interfaces for voice-to-gesture and smart wearable products, and creating a technology secondary development interface platform and a big data platform. It also works with government departments to provide disability protection services. Build a complete ecosystem by complemented with the model provided by the integration of technical solutions.