Award-winning Work Information & Description
Visual Analysis Platform of “Smart Logistics” Based on Hadoop
Description
The "smart logistics" visual analysis platform based on big data designed by this team takes the logistics data information of Hunan Province as an example, through the investigation of the information of Hunan Railway Bureau, Hunan Road Transport Administration Bureau and National Bureau of Statistics and other websites, on the basis of studying the characteristics of logistics information, through the study of big data framework and distributed service framework technology, and the corresponding expansion and algorithm mechanism design, targeted to solve the problems related to logistics information data processing. This platform uses a big data platform based on Hadoop distributed cluster and intelligent analysis algorithm to clean, standardize, analyze and process the data information set. Modular processing enhances the reuse of the system. Working in parallel improves the running speed of the system. Distributed computing platform can process massive logistics data more quickly. Design a logistics data algorithm model based on K-means, to judge, classify and count the data information such as freight volume, turnover volume and business volume; to design a real-time data streaming computing model, under this framework, the business logic can be realized freely, the analysis speed can be accelerated, and the parallel computing scalability is high; the powerful HiveQL technology is used as the data warehouse processing means to reduce the difficulty of data call and speed up the operation of the system; finally, the use of HTML5 and CSS3 technology to process the data page dynamic visual display, interactive, good visual effect, the user's sensory experience is good, greatly improving the work efficiency. The product has the following innovative features: ● Design and implement real-time data processing solutions in the platform for the real-time nature of logistics big data. Considering the impact of massive real-time data and complex services on the processing system, the platform real-time data processing needs to meet the requirements of low latency, high performance, distributed, scalable and fault-tolerant; ● Innovative design of micro-service-oriented big data service management system mainly provides visualization, log service and unified data access functions for the platform, and the big data platform with big data service management system can provide logistics information standardization, high flexibility and high transparency; After the big data service management system is added to the platform, the system administrator does not need to care about the underlying architecture and application implementation, but can view, monitor, control and manage the platform at any time through the Web page; ● In view of the characteristics of various data structure types of logistics big data, integrate various data persistence methods in the big data platform for logistics information, and establish a unified data access interface; Logistics information has a variety of data structure types, including structured, semi-structured, unstructured data; ● Make the platform efficient, stable and independent through distributed coordination services, load balancing and backup mechanisms. The main framework technologies used in the big data platform for logistics information are distributed, with the advantages of load balancing, dynamic expansion, parallel computing, distributed storage, fault restart, disaster recovery, etc; ● Design a log management service that combines custom parameters of the log framework and log stream processing to maintain a unified log mechanism of the platform; ● The platform provides a complete set of security authentication mechanism to guarantee the security of the system and information.