Univerza v Ljubljani , Fakulteta za elektrotehniko
Research projects (co)funded by the Slovenian Research Agency .
Project	Slovenska verzija
Member of  University of Ljubljana	UL Faculty of Electrical Engineering
Code	L2-3168
Project	Development of a self-learning system for optimizing the driving rules of autonomous transport vehicles and their temporally and spatially coordinated activities
Period	1.10.2021 - 30.9.2024
Range on year	1,84 FTE
Head	Gregor Klančar
Research activity	Engineering sciences and technologies
Research Organisation	link on SICRIS
Abstract	Modern manufacturing and warehousing systems are facing the challenges of the market, which dictates increasingly personalized products, low manufacturing costs, unconditional quality, and the achievement of ever more demanding logistic objectives. Product variability is increasing and batch sizes are decreasing. Production and storage systems need to be increasingly flexible and able to be reconfigured quickly, placing new requirements for internal material transport systems (intralogistics). The solution for most current intralogistics implementations is provided by automatically guided vehicles (AGVs), usually guided on the basis of a magnetic tape attached to the floor. The tape defines the transport routes in advance, which facilitates the management of the AGV fleet, but decreases the flexibility of the system. The state-of-the-art systems employ autonomous mobile robots (AMR), which better meet the growing demands for flexibility and reconfigurability of intralogistics systems. They contain all the necessary sensors for autonomous movement in free space, without additional floor markings in the form of painted lines or magnetic tape. Therefore, if the layout of the production system is changed, new work systems are integrated, or new movement constraints are introduced (e.g. obstacles), it is no longer necessary to redefine the paths by changing the layout of the magnetic tapes. Moreover, AMRs can detect changes and independently find alternative paths, as well as share this information with other robots, allowing them to immediately adjust their actions. However, due to the greater flexibility of AMR systems, the problem of fleet management becomes much more challenging. While the problem of AGV fleet management is restricted to relatively simple heuristics operating with relatively deterministic transport times, the problem of managing a fleet of AMRs encompasses high-level functionalities, such as monitoring, scheduling, task assignment, and time slot allocation, through mid-level functionalities, such as path planning, navigation, and motion control, to low-level functionalities, such as robot localization, sensing, and control. Inputs to the Fleet Management System (FMS) are generated based on high-level planning systems such as Enterprise Resource Planning, Warehouse Management systems, and Material Flow Control systems. In this context, functionalities of all levels are tightly coupled, since, for example, obstacle detection at the sensory level has an impact on the control of the movement in the presence of the obstacle, and furthermore, on the travel times and the allocation of time windows for the entire fleet, which must be coordinated with the higher-level planning systems. The problem of fleet management requires new approaches due to its interdisciplinary nature intersecting systems management and robotics. The question is how to determine the movement rules in free space, plan routes, and assign and execute transportation tasks based on a known intralogistic problem, i.e., known layout, pick-up locations, drop-down locations, and dynamics of transportation tasks. Today, movement rules are usually determined by humans, e.g. by the system planner, based on their experience. Route planning is then handled by systems that use simple rules or heuristics and do not solve problems that arise in multi-robot systems, such as joint route planning and conflict resolution. Also, the allocation of transportation tasks to robots is often very simplistic and does not take into account the route planning system, resulting in suboptimal operation. In addition, current systems are not able to react quickly to changes in the intralogistic problem, because the system's operating rules are fixed for a particular problem and do not adapt to changes. The project addresses an intelligent fleet management system for autonomous mobile robots and brings together groups from three project partners: 1. The Faculty of Electrical Engineering (FE), the University of Ljubljana (UL), which deals with mobile robotics, route planning, robot control and prediction methods, 2. The Faculty of Mechanical Engineering (FS), UL, which deals with distributed control and artificial intelligence applications in production systems, and 3. The company Epilog d.o.o., a leading Slovenian company developing autonomous mobile robots and software for warehouse intralogistics, logistic process management, and fleet management of autonomous mobile robots. Recently, the company has also developed a high-performance autonomous mobile robot platform. The main goal of the project is to develop algorithms for efficient and flexible multi-robot transportation. Important innovations compared to existing approaches in the industry will be better flexibility through automatic creation and adaptation of map (layout) configuration, which will enable more efficient transport by AMRs (e.g., shorter transport times, less congestion, and fewer conflicts when planning AMR routes). Another important innovation will be the self-learning module for assigning tasks to AMRs, which will adapt the rules to the current situation (current map, current transport order statistics, properties of the route planning algorithm used, etc.), thus enabling better performance over time through more efficient planning and reduced complexity (given the combinatorial complexity where task assignment and route planning are solved simultaneously). Priority is also given to coordinated route planning for a group of AMRs by defining occupancy windows for sections on the map, taking into account transportation priorities, with minimal necessary coordination, without conflicts and collisions, facilitating local management with fewer corrections during transportation. These algorithms will be tested, analyzed, and validated at multiple levels of fleet management. It will be shown that new approaches to abstract the intralogistic problem, multi-robot route planning, and task assignment can achieve more efficient and robust solutions than existing ones. The results of the research and applications using the aforementioned prototype robots will be put to good use by Epilog and its partners in improving existing and future solutions.
Researchers	link on SICRIS
Work packages of the project and their realization	The duration of the applied research project is three years or 36 months. The project is divided into six work packages (WPs), which are in turn divided into several tasks (T). The WP1 package defines activities related to project management. The other packages (WP2 to WP5) specify the objectives, tasks, deliverables and milestones. The WP6 package additionally specifies the activities related to the dissemination of the deliverables. Work packages are:     WP1: Project Management     WP2: Method for determining the configuration of the AMR logistics system                 for a given intralogistic problem     WP3: Algorithms for multi-robot route planning     WP4: Self-learning algorithms for assigning transport tasks     WP5: Integration and demonstration     WP6: Dissemination The following is a detailed description of the objectives, the tasks and results for each project package. ______________________________________________________________ 1. WORK PACKAGE (WP1): Project Management 1.1 Objectives WP1:     Under the work package, work organization and research activities are carried out, work progress is monitored, implementation problems are identified, and appropriate actions are formulated. In evaluating the adequacy of implementation, we will introduce measures such as standardized performance (activities performed according to the schedule), time compliance with the plan, evaluation of the effort required to use the solution in the following work packages and the target application, solution efficiency compared to those existing in the company, quality of the reports and possibility of use for dissemination purposes (WP6). 1.2 Tasks WP1:     T1.1. Coordination of project activities (FE, FS, Epilog): Includes the establishment of the project council and its activity at the beginning of the project, the supervision of finances, the organization of working meetings and the resolution of possible conflicts. [m1-m36]     T1.2. Coordination of research activities (FE, FS): Includes coordination of activities between work packages, ongoing risk assessment and monitoring, and coordination of reporting of research results. The task also governs intellectual property created and harmonized application and research results. [m1-m36]     T1.3. Quality assurance (FE, FS, Epilog): Establishing measures to assess the quality of project implementation and the adequacy of all deliverables (D). Internal monitoring of the preparation of results every 6 months. [m1-m36] 1.3 Results WP1:     D1.1.1. - D1.1.6. Periodic report on activities performed, resources used, achievements, and risk assessment. [m6, m12, m18, m24, m30, m36] ______________________________________________________________ 2. WORK PACKAGE (WP2): Method for determining the configuration of the AMR logistics system for a given intralogistic problem 2.1 Objectives WP2:     The objective of the work package is to develop a method for determining the configuration of the logistics system. The input information is a known layout of the space (e.g. placement of static objects in the workshop or warehouse, given movement constraints, given pick-up and drop-down locations) and known dynamics of transport tasks (e.g. distribution of arrivals of transport tasks). The intermediate result of the developed method will be a map of the space containing the rules of movement. These will be given in the form of the price of movements according to the direction of movement, allowing, for example, one-way traffic in narrow corridors, and in the form of conflict resolution rules, for example, prioritization at intersections. On this basis, the final result of the method will be formed, a graph of directed routes, which will be the basis for further multi-robot route planning (WP3). The approach will thus be an extension of previous work, where the description of the space was based on the potential field. Simulation methods will be used for development and validation. The results of the method will be compared with classical approaches (using the A* algorithm) and with the previously developed methods based on the potential field. The basic measures for comparison will be the amount of tasks performed per unit time, the total length of trips performed, and the amount of close encounters and conflicts in the movement of robots. 2.2 Tasks WP2:     T2.1. Intralogistic problem abstraction (FS, Epilog, FE): Includes the definition of the intralogistic problem set and its associated data structures: directional map and movement rules. Modifications to routing algorithms required to use the newly defined data structures will be analyzed. The company will help to create examples of typical intralogistic problems, which will be used to further develop and validate the methods. [m1-m3]     T2.2. Development of a method for determining the configuration of the intralogistic system (FS): It includes the creation of a simulation environment. This is initially based on the representation of a map with a discrete grid of cells. Algorithms for determining the costs of directed motion in such a discretized space and methods for determining the encounter rules will be developed, which will be based on the simulation of a multi-robot system. An algorithm for determining a directed graph of a route based on a map of directed prices will be developed. [m4-m12]     T2.3. Validation by simulation (FS, Epilog): The algorithms will first be validated in a simulation developed in T2.2, where the space is represented by discrete cells. Validation will then also take place in an extended simulation environment that takes into account the physics of motion and simulates the robot sensors and actuators. ROS / Gazebo will be used as the simulation tool. The results of the method will be compared and evaluated as described in the objectives of the work package. In addition, the results of the method will also be evaluated by experts from the company. [m10-m15]     T2.4. Development of interfaces for integration (FS, Epilog, FE): Interfaces will be prepared that accept as input a map of the space in .pgm format, which is supported by default by ROS (Robot Operating System). The format of the output directed graph will be determined and developed. The interfaces will be tested based on the developed simulations. [m12-m18] 2.3 Results WP2:     D2.1. Abstraction of the intralogistic problem (report). [m3]     D2.2. Method for determining the configuration of the intralogistics system (software and report). [m12]     D2.3. Validation of the method by simulation (report). [m12]     D2.4. Integration interfaces (software and documentation). [m12] ______________________________________________________________ 3. WORK PACKAGE (WP3): Algorithms for multi-robot route planning 3.1 Objectives WP3:     The goal of the work package is to develop algorithms for controlling the movement of the fleet of autonomous mobile robots. The input information for this problem is task-robot pairs that define, among other things, the start and end positions of the load and indicate which robot is assigned the task. Fleet management algorithms then solve the problem of route planning, assigning time windows for trips, and implementing routes in a multi-robot system. First, the problem will be formulated mathematically, and based on this, algorithms will then be developed to automatically determine the movement constraints and movement control of the fleet. The validation of the algorithms using a simulator, developed in the project preparation phase and completed during the project, will allow to compare the effectiveness of the different approaches (e.g.: computational complexity, optimality of results, evaluation of improvement over a selfish planner without AMR coordination, analysis of achieved coordination, robustness and adaptability to current changes, such as response to new tasks during implementation, changing tasks or goals, changes in the environment or priorities in existing implementations). 3.2 Tasks WP3:     T3.1. Definition of multi-robot route management problem (FE, Epilog): Includes a mathematical formulation of the fleet motion management problem. Emphasis will be placed on a holistic view of the multi-robot problem, since robot motion is generally not independent. Benchmarks for measuring efficiency will be developed, on the basis of which it will be possible to compare solutions. [m1-m3]     T3.2. Development of multi-robot route management (FE) Includes the development of new algorithms that comprehensively solve the fleet movement management problem. Algorithms are developed that automatically determine the rules or movement constraints considering loads and spatial arrangements. Multi-robot motion algorithms will then be developed on this basis. The possibility of distributed implementation of these algorithms or a suitable trade-off between centralized and local problem solving and a comparison with centralized implementations will be performed. [m4-m18]     T3.3. Validation of algorithms with simulation (FE): Includes testing of algorithms using simulation. Scenarios are prepared for testing that include different room layouts and different numbers of robots. Scenarios are also prepared where the environment is dynamic, e.g., a robot fails or a new one is added during scenario execution. [m12-m24]     T3.4. Development of integration interfaces (FE, Epilog, FS): It includes the preparation of the developed algorithms for integration with the task assignment system (WP4) and with the real demonstrator (WP5). ROS will be used for the integration. [m19-m27] 3.3 Results WP3:     D3.1. Definition of the problem of fleet movement management (report). [m6]     D3.2. Fleet motion control algorithms (software). [m18]     D3.3. Validation of algorithms by simulation (report). [m24]     D3.4. Integration interfaces (software and documentation). [m27] ______________________________________________________________ 4. WORK PACKAGE (WP4): Self-learning algorithms for assigning transport tasks 4.1 Objectives WP4:     The goal of the work package is to develop an algorithm for transportation task assignment based on reinforcement learning methods. This will make the assignment more efficient over time, adaptable to a specific intralogistic problem, and more robust or resilient to changes in the system. The input information is transportation tasks obtained from information systems (WMS, ERP). The output of the task, on the other hand, will be task-robot pairs. The problem will be defined as a reinforcement learning problem that tests the use of multiple and different input features, such as the map of the space (with and without directional paths, with and without known current location of all robots,...), graph of directed paths, current locations and loads of robots, etc. The two methods of centralized allocation with a single agent and distributed allocation where the agents represent individual robots will be tested. The two approaches will be compared with different classical heuristics (e.g., unoccupied-first, closest-first,...) in different scenarios (stable system operation, system operation in the presence of a disruption, e.g., failure of one or more robots). Fast simulation methods on a simplified discrete simulator will be used for learning and more detailed simulations will be used in the validation. 4.2 Tasks WP4:     T4.1. Definition of the task assignment problem (FS, Epilog): It includes formulating the task assignment problem in a form suitable for reinforcement learning, and producing a set of meaningful features for decision making. It also involves a review and evaluation of the suitability of existing reinforcement learning algorithms. [m7-m9]     T4.2. Development of self-learning algorithms (FS): Includes setting up a simple discrete simulation environment to stimulate learning, and testing and improving existing algorithms. The Python software library RLlib will be used, which allows easy development and comparison of different algorithms. For the learning problem, different combinations of features and two configurations will be used, one with a central agent that has all the information available and the other based on distributed learning (e.g. based on an auction as presented in the preliminary research review). In improving algorithms, the efficiency of learning will be particularly emphasized, i.e., how much data or samples are needed to formulate a working policy. [m10-m18]     T4.3. Validation of algorithms with simulation (FS): It includes validation and comparison of algorithms and analysis of the impact of input features on efficiency. This will be measured by the number of successfully completed transportation tasks per time. Efficiency will be analyzed both in stable operation of the system and in the presence of disruptions, thus assessing the robustness of the solution (loss of efficiency due to disruptions). [m15-m24]     T4.4. Development of integration interfaces (FS, Epilog, FE): Includes the preparation of the developed algorithms for integration with a multi-robot path planning system (WP3) and a real demonstrator (WP5). ROS will be used for the integration. [m19-m27] 4.3 Results WP4:     D4.1. Definition of the task assignment problem (report). [m8]     D4.2. Development of self-learning algorithms (software, report). [m18]     D4.3. Validation of algorithms by simulation (report). [m24]     D4.4. Integration interfaces (software and documentation). [m27] ______________________________________________________________ 5. WORK PACKAGE (WP5): Integration and demonstration 5.1 Objectives WP5:     The goals of the work package are to integrate the results of the work packages WP2, WP3 and WP4 and to build a real demonstration system. For the integration, ROS (Robot Operating System) will be used to facilitate the integration and to enable the transition from simulated environments to real environments. Software interfaces for integration and demonstration will be defined, implemented, and tested. A demonstration system will be prepared, based on the company's three existing autonomous mobile robots and a test area, prepared for this purpose. Four demonstration scenarios will be developed: one for each work package (WP2, WP3, WP4) and a final one to demonstrate integrated operation. 5.2 Tasks WP5:     T5.1. Specification of software interfaces for integration (Epilog, FE, FS): Software interfaces will be specified to enable integration of the results of work packages WP3 and WP4. The specification will be given in terms of an input and output data structure for each module and then implemented in terms of messages, services and actions as enabled by the ROS middleware. [m12-m18]     T5.2. Development of software interfaces for integration (Epilog, FE, FS): This task implements the interfaces specified in T5.1. and then integrates the interfaces developed in T2.4, T3.4. and T4.4. into an integrated system. [m19-m27]     T5.3. Development of the demonstrator (Epilog): It includes the preparation of hardware and software for the implementation of a physical demonstrator comprising three robots and a test area. [m12-m30]     T5.4. Testing (Epilog, FE, FS): It includes tests on a real polygon with the aim of validating the results in the most realistic environment possible. During testing, all the developed solutions will be tested in detail in the listed scenarios. The efficiency of the system is measured and compared with the classical centralized fleet management. [m31-m36] 5.3 Results WP5:     D5.1. Specification of integration interfaces (report). [m18]     D5.2. Integrated fleet management system (software). [m27]     D5.3. Demonstration system (report). [m30]     D5.4. Testing (report). [m36] ______________________________________________________________ 6. WORK PACKAGE (WP6): Dissemination 6.1 Objectives WP6:     The aim of the work package is to present the research to the general domestic and foreign public, both professionally (companies) and scientifically (articles and conferences). An important aspect is also the transfer of knowledge to the employees in the company Epilog and through the pedagogical process and the supervision of theses at the faculties. Dissemination will also be through applications, starting with the installation of a demonstration application and later on the possible commercialization of solutions. 6.2 Tasks WP6:     T6.1. Dissemination activities (FE, FS, Epilog): The research results of the project will be disseminated through 2 domestic (AIG, ERK) and 2 foreign scientific conferences (e.g. IROS, ICRA, ICIEA, IFAC, CASE), in the form of scientific articles (at least 2 SCI articles, e.g. in CIRP Annals, Robotics and Computer-Integrated Manufacturing, Robotics and Autonomous Systems, Automation, ...), at least 2 domestic journals (World of Mechatronics, IRT 3000), and 2 mentorships of theses. [m1-m36]     T6.2. Application activities (Epilog, FS, FE): The results of the project will be disseminated by Epilog d.o.o. (demonstrations to collaborators, partners, at trade shows) and potentially commercialized in real industrial applications. For this purpose, there will be a continuous evaluation of the applicability of the developed solutions for industrial applications (feasibility assessment, applicability in industry, preparation of suitable interfaces for compatibility with existing equipment) as part of this task. [m1-m36] 6.3 Results WP6:     D6.1. Review and assessment of industrial applicability of solutions (report). [m36]
Citations for bibliographic records	link on SICRIS