Optimal and Simulation-Based Approximate Dynamic Programming Approaches for the Control of Re-Entrant Line Manufacturing Models.

摘要

This dissertation considers the application of simulation-based Approximate Dynamic Programming Approaches (ADP) for near-optimal control of Re-entrant Line Manufacturing (RLM) models. This study departs from the analysis of the optimal control problem under a discounted cost (DC) criterion in two simple RLM models with both job sequencing and job releasing control operations. Results on optimality conditions, structural properties of the optimal control policy, and sufficient conditions for optimality are provided. For the same models, four different simulation-based ADP approaches, namely, Q-Learning, Q-Learning with State Aggregation, SARSA(Lambda), and an Actor-Critic Architecture were utilized for control optimization. The ADP approaches studied include methods based on lookup tables and methods based on parametric approximations of the optimal cost function using temporal difference learning. Numerous simulation experiments were conducted to evaluate and compare the performance of the ADP methods employed against that of optimal solutions. Results indicate that the Actor-Critic approach consistently obtained a performance close to the optimal solutions while providing the best features for scalability in the state and action spaces which is essential for implementations of ADP in realistic RLM models. Upon these results, an extension of the Actor-Critic for larger RLM models is proposed under both a DC and average cost (AC) criterion. The formulation of the proposed approach is based on the representation of the RLM system as a model with an arbitrary number of single exponential servers and binary controls which can be seen as an abstraction of the simple RLM models previously studied. The proposed model is then amenable for the application of the uniformization procedure, which in turns allows for the derivation of optimality equations and conditions. These provide structural properties that also facilitate the definition and implementation of the control or actor in the proposed ADP algorithm. As an example, the so-called Intel Mini-Fab model was utilized in numerous simulation experiments on the optimization of job sequencing operations under an AC criterion. These experiments compared the performance of policies obtained with the proposed ADP against that of well known dispatching rules. Results from these experiments demonstrated the applicability of the proposed approach under different operational conditions, including different preventive maintenance schedules, random and deterministic processing times in the machines, and different load factors in the system. The results also demonstrate that, in general, the policies obtained with the proposed ADP approach provided good performance when compared to the dispatching rules considered. Moreover, the results show that under given operational conditions ADP-generated policies can even outperform the dispatching rules considered in the experiments. Finally, this dissertation also provides experimental results from the application of a simulation-based ADP approach for the optimization of preventive maintenance (PM) schedules in RLM models. The proposed approach utilizes an Actor-Critic architecture and the so-called post-decision state variable approach to define the actor in the ADP architecture. As an illustrative example, simulation experiments were conducted with the Intel Mini-Fab model. Results from these experiments demonstrated that ADP-generated PM policies were able to significatively reduce both the average work-in-process and the average cycle-time when compared to selected fixed PM policies.