An Overview and Recent Developments in the Analysis of Multistate Processes

However, in many application contexts, more complicated event processes may arise. For instance, a subject of interest may experience a succession of intermediate events, each representing a disease/health status. Multistate models (MSM) extend the traditional survival model to encompass more states, providing a useful probabilistic framework for modelling complex longitudinal event history data ( Cook and Lawless,, 2018 ) . Well-known models such as competing risks and illness-death models are special cases of MSM. MSM can be conveniently specified via a collection of time-to-event models, one for each permitted transition between two states. They have recently been incorporated into joint models, typically by linking the longitudinal and transition-specific regression models via shared random effects, allowing inference of association patterns between markers and different transitions.

We study the expected accumulated reward for a discrete-time Markov reward model with absorbing states. The rewards are impulse rewards, where a reward $ρ_{ij}$ is accumulated when transitioning from state $i$ to state $j$. We derive an explicit, single-letter expression for the expected accumulated reward as a function of the number of time steps $n$ and include in our analysis the limit in which $n \to \infty$.

The creation and destruction of agents in cooperative multi-agent reinforcement learning (MARL) is a critically under-explored area of research. Current MARL algorithms often assume that the number of agents within a group remains fixed throughout an experiment. However, in many practical problems, an agent may terminate before their teammates. This early termination issue presents a challenge: the terminated agent must learn from the group's success or failure which occurs beyond its own existence. We refer to propagating value from rewards earned by remaining teammates to terminated agents as the Posthumous Credit Assignment problem. Current MARL methods handle this problem by placing these agents in an absorbing state until the entire group of agents reaches a termination condition. Although absorbing states enable existing algorithms and APIs to handle terminated agents without modification, practical training efficiency and resource use problems exist. In this work, we first demon …

We study phase transitions in a long-range one-dimensional cellular automaton with two symmetric absorbing states. It includes and extends several other models, like the Ising and Domany-Kinzel ones. It is characterized by competing ferromagnetic linear and antiferromagnetic nonlinear couplings. Despite its simplicity, this model exhibits an extremely rich phase diagram. We present numerical results and mean-field approximations.

We study the long-time behaviour of stochastic models with an absorbing state, conditioned on survival. For a large class of processes, in which saturation prevents unlimited growth, statistical properties of the surviving sample attain time-independent limiting values. We may then define a quasi-stationary probability distribution as one in which the ratios p n ( t )/ p m ( t ) (for any pair of non-absorbing states n and m ) are time-independent. This is not a true stationary distribution, since the overall normalization decays as probability flows irreversibly to the absorbing state. We construct quasi-stationary solutions for the contact process on a complete graph, the Malthus–Verhulst process, Schlögl's second model and the voter model on a complete graph. We also construct the master equation and quasi-stationary state in a two-site approximation for the contact process and for a pair of competing Malthus–Verhulst processes.