I am not sure if the following is a legitimate question for this board.

I am looking for examples of Partially observed Markov decision processes (preferably infinite horizon, Discrete time, Discrete state) where the actions impact the transitions of the underlying Markov chain. I know for sure that Machine replacement is one such example. However, I do not find other useful examples. Quite often people study POMDPs for active sensing/ target tracking via sensor networks where different actions only impact the observations not the actions.

I would appreciate it if somebody could point me to references of POMDP where the actions impact the transition probabilities of the underlying markov chain.

asked 21 May '14, 18:40

anon123's gravatar image

accept rate: 0%

edited 22 May '14, 04:08

fbahr's gravatar image

fbahr ♦

It is interesting. I always thought there must be an example of POMDP where the actions affect the transition probability. But, I cannot find one, either.

If there is an MDP with action-dependent transition and unobservable state, it might be the situation where you can not get any useful information from observation, I guess.

(21 May '14, 22:29) ksphil

Mykel Kochenderfer, an aero/astro prof at Stanford, has done work in which he uses POMDPs for aircraft collision avoidance. In this case the control actions involve changing where an aircraft goes, so they would impact the state transition probabilities. ACAS-X, a collision-avoidance system that looks likely to be deployed operationally, is based on this work (although I'm not sure if in that case it is an MDP or a POMDP formulation).

Here is a paper that discusses aircraft collision avoidance as a POMDP. There are a few other papers that might be relevant on his website.


answered 22 May '14, 15:46

mbloem's gravatar image

accept rate: 50%

thanks for the references

(23 May '14, 16:27) anon123

Partially observed linear quadratic Gaussian model, which are used quite often in control theory.


answered 24 Feb '16, 00:40

adityam's gravatar image

accept rate: 0%

Your answer
toggle preview

Follow this question

By Email:

Once you sign in you will be able to subscribe for any updates here



Answers and Comments

Markdown Basics

  • *italic* or _italic_
  • **bold** or __bold__
  • link:[text](http://url.com/ "Title")
  • image?![alt text](/path/img.jpg "Title")
  • numbered list: 1. Foo 2. Bar
  • to add a line break simply add two spaces to where you would like the new line to be.
  • basic HTML tags are also supported



Asked: 21 May '14, 18:40

Seen: 5,251 times

Last updated: 24 Feb '16, 00:40

OR-Exchange! Your site for questions, answers, and announcements about operations research.