RUcore Resource Object
RUcore Resource Object
PDF-1(12.38 MB)
TitlePerception-based generalization in model-based reinforcement learning
NameLeffler, Bethany R. (author), Littman, Michael (chair), Stone, Matthew (internal member), Pavlovic, Vladimir (internal member), Roy, Nicholas (outside member), Rutgers University, Graduate School - New Brunswick,
Degree Date2009-01
Date Created2009
SubjectComputer Science,
Reinforcement learning--Mathematical models,
Machine learning
DescriptionIn recent years, the advances in robotics have allowed for robots to venture into places too dangerous for humans. Unfortunately, the terrain in which these robots are being deployed may not be known by humans in advance, making it difficult to create motion programs robust enough to handle all scenarios that the robot may encounter. For this reason, research is being done to add learning capabilities to improve the robot's ability to adapt to its environment. Reinforcement learning is well suited for these robot domains because often the desired outcome is known, but the best way to achieve this outcome is unknown.
In a real world domain, a reinforcement-learning agent has to learn a great deal from experience. Therefore, it must be sample-size efficient. To do so, it must balance the amount of exploration that is needed to properly model the environment with the need to use the information that it has already obtained to complete its original task. In robot domains, the exploration process is especially costly in both time and energy. Therefore, it is important to make the best possible use of the robot's limited opportunities for exploration without degrading the robot's performance.
This dissertation discusses a specialization of the standard Markov Decision Process (MDP) framework that allows for easier transfer of experience between similar states and introduces an algorithm that uses this new framework to perform more efficient exploration in robot-navigation problems. It then develops methods for an agent to determine how to accurately group similar states. One proposed technique clusters states by their observed outcomes. To make it possible to extrapolate observed outcomes to as-yet unvisited states, a second approach uses perceptual information such as the output of an image-processing system to group perceptually similar states with the hope that they will also be related in terms of outcomes. However, there are many different percepts from which a robot could obtain state groupings. To address this issue, a third algorithm is presented that determines how to group states when the agent has multiple, possibly conflicting, inputs from which to choose. Robot experiments of all algorithms proposed are included to demonstrate the improvements that can be obtained by using the approaches presented.
In a real world domain, a reinforcement-learning agent has to learn a great deal from experience. Therefore, it must be sample-size efficient. To do so, it must balance the amount of exploration that is needed to properly model the environment with the need to use the information that it has already obtained to complete its original task. In robot domains, the exploration process is especially costly in both time and energy. Therefore, it is important to make the best possible use of the robot's limited opportunities for exploration without degrading the robot's performance.
This dissertation discusses a specialization of the standard Markov Decision Process (MDP) framework that allows for easier transfer of experience between similar states and introduces an algorithm that uses this new framework to perform more efficient exploration in robot-navigation problems. It then develops methods for an agent to determine how to accurately group similar states. One proposed technique clusters states by their observed outcomes. To make it possible to extrapolate observed outcomes to as-yet unvisited states, a second approach uses perceptual information such as the output of an image-processing system to group perceptually similar states with the hope that they will also be related in terms of outcomes. However, there are many different percepts from which a robot could obtain state groupings. To address this issue, a third algorithm is presented that determines how to group states when the agent has multiple, possibly conflicting, inputs from which to choose. Robot experiments of all algorithms proposed are included to demonstrate the improvements that can be obtained by using the approaches presented.
NotePh.D.
NoteIncludes bibliographical references (p. 100-104)
Noteby Bethany R. Leffler
Genretheses
Persistent URLhttp://hdl.rutgers.edu/1782.2/rucore10001600001.ETD.000051041
Languageeng
CollectionGraduate School - New Brunswick Electronic Theses and Dissertations
Organization Name
RightsThe author owns the copyright to this work.