Overview of the project

Reinforcement learning (RL), both classical algorithms and their deep variants, critically rely on the fact that the underlying system is assumed stationary, i.e., how the environment responds to the agent’s actions does not vary with time. Unfortunately, many real-world problems fail to exhibit such stationary property. Therefore, to bring out the full potential of RL in complex application domains, existing methods must be extended to cope with non-stationarity in a principled way.

RELEvaNT investigates new models and methods for efficient deep RL in non-stationary environments and the potential applications on several “human-centered” domains. In particular, RELEvaNT investigates:

Model-based RL in which the learned model captures a low-dimensional factorized representation of the world. We investigate the extent to which such low-dimensional representations enable the agent to robustly cope with changes in the dynamics of the world.
Meta-learning approaches to model-based RL, in order to render the process of learning the low-dimensional models mentioned above more data-efficient. In particular, we build on existing frameworks for model-agnostic meta-learning to construct pre-trained “prototypical” representations that can then be adjusted, at interaction time, using a small number of samples, thus enabling the agent to adjust to changes in the environment.

The outcomes of the project will be evaluated in several real-world non-stationary domains, exploiting the application of RL in robot control and human-robot interaction.

Research team

Research in RELEvaNT is held by a dynamic team that includes both senior and junior researchers. Several MSc students have also been advised in the context of the project.

Senior researchers

Francisco S. Melo (Principal Researcher, Full Professor, IST)
José Alberto Sardinha (Associate Professor, PUC-Rio)
Manuel Lopes (Associate Professor, IST)

Junior researchers

Diogo Carvalho (PhD student at IST)
Pedro Pinto Santos (PhD student at IST)
Jacopo Silvestrin (PhD student at IST)
Bernardo Esteves (PhD student at IST)

Former members

António Soares
Fábio Vital (currently a PhD student at IST)
Francisco Martins (former MSc student)
Gonçalo Querido (former MSc student)
José Rocha (currently a PhD student at IST)
Margarida Serralheiro (currently a PhD student at IST)
Miguel Vasco (currently a post-doc at KTH)
Rustam Zayanov (former MSc student)

Publications

Edited volumes

U. Endriss, F. Melo, K. Bach, A. Bugarin-Diz, J. Alonso-Moral, S. Barro, and F. Heintz, eds. ECAI 2024 - 27th European Conference on Artificial Intelligence. IOS Press, 2024 (link).

F. Melo, F. Fang. (2022) Autonomous Agents and Multiagent Systems: Best and Visionary Papers, AAMAS 2022, Lecture Notes in Computer Science 13441, Springer (link).

International journals

D. Carvalho, P. Santos, F. Melo. Multi-Bellman operator for convergence of Q-learning with linear function approximation. Trans. Machine Learning Research, vol. 2025, 2025. (pdf)

D. Carvalho, P. Santos, F. Melo. Reinforcement learning in convergently non-stationary environments: Feudal hierarchies and learned representations. Artificial Intelligence, vol. 347, p. 104382, 2025. (pdf)

H. Fonseca, C. de Melo, K. Terada, J. Gratch, A. Paiva, F. Santos. Evolution of indirect reciprocity under emotion expression. Nature Scientific Reports, vol. 15, p. 9151, 2025. (pdf)

P. Santos, F. Melo, A. Sardinha, D. Carvalho. Understanding the impact of data distribution on Q-learning with function approximation. Machine Learning, vol. 113, pp. 6141-6163, 2024. (pdf)

P. Santos, D. Carvalho, M. Vasco, A. Sardinha, P. Santos, A. Paiva, F. Melo. Centralized training with hybrid execution in multi-agent reinforcement learning via predictive observation imputation. Artificial Intelligence, vol. 348, p. 104404, 2025. (pdf)

M. Vasco, H. Yin, F. Melo, A. Paiva. Leveraging hierarchy in multimodal generative models for effective cross-modality inference. Neural Networks, vol. 146, pp. 238-255, 2022. (pdf)

International conferences

D. Carvalho, F. Melo, P. Santos. Theoretical remarks on feudal hierarchies and reinforcement learning. In Proc. 26th Eur. Conf. Artificial Intelligence, pp. 351-356, 2023. (pdf)

B. Esteves, M. Vasco, F. Melo. NeuralSolver: Learning algorithms for consistent and efficient extrapolation across general tasks. In Advances in Neural Information Proc. Systems 37, 2024. (pdf)

P. Poklukar, M. Vasco, H. Yin, F. Melo, A. Paiva, D. Kragic. Geometric multi-modal contrastive representation learning. In Proc. 39th Int. Conf. Machine Learning, pp. 17782-17800, 2022. (pdf)

G. Querido, A. Sardinha, F. Melo. Learning to perceive in deep model-free reinforcement learning. In Proc. 22nd Int. Conf. Autonomous Agents and Multiagent Systems, pp. 2595-2597, 2023. (pdf)

J. Ribeiro, C. Martinho, A. Sardinha, F. Melo. Making friends in the dark: Ad hoc teamwork under partial observability. In Proc. 26th Eur. Conf. Artificial Intelligence, pp. 1954-1961, 2023. (pdf)

P. Santos, D. Carvalho, M. Vasco, A. Sardinha, P. Santos, A. Paiva, F. Melo. Centralized training with hybrid execution in multi-agent reinforcement learning. In Proc. 23rd Int. Conf. Autonomous Agents and Multiagent Systems, pp. 2453-2455, 2024. (pdf)

M. Vasco, H. Yin, F. Melo, A. Paiva. “How to sense the world”: Leveraging hierarchy in multimodal perception for robust reinforcement learning agents. In Proc. 21st Int. Conf. Autonomous Agents and Multiagent Systems, pp. 1301-1309, 2022. (pdf)

F. Vital, M. Vasco, A. Sardinha, F. Melo. Perceive, represent, generate: Translating multimodal information to robotic motion trajectories”. In Proc. 2022 IEEE/RSJ Int. Conf. Intelligent Robots and Systems, pp. 5855-5860, 2022. (pdf)

G. Varela, A. Sardinha, F. Melo. Distributed value decomposition networks with networked agents. In Proc. 24th Int. Conf. Autonomous Agents and Multiagent Systems, pp. 2774-2776, 2025. (pdf)

G. Varela, A. Sardinha, F. Melo. Networked agents in the dark: Team value learning under partial observability. In Proc. 24th Int. Conf. Autonomous Agents and Multiagent Systems, pp. 2087-2095, 2025. (pdf)

F. Vital, A. Sardinha, F. Melo. Implicit repair with reinforcement learning in emergent communication. In Proc. 24th Int. Conf. Autonomous Agents and Multiagent Systems, pp. 2115-2124, 2025. (pdf)

R. Zayanov, F. Melo, M. Lopes. Interactively teaching an inverse reinforcement learner with limited feedback. In Proc. 16th Int. Conf. Agents and Artificial Intelligence, pp. 15-24, 2024. (pdf)

National conferences

B. Esteves, M. Vasco, F. Melo. Pre-training with augmentations for efficient transfer in model-based reinforcement learning. In Proc. 22nd EPIA Conf. Artificial Intelligence, pp. 133–145, 2023. (pdf)

A. Neves, A. Sardinha. Learning to cooperate with completely unknown teammates. In Proc. 21st EPIA Conf. Artificial Intelligence, pp. 739-750, 2022. (pdf)

J. Silvestrin, J., Melo, F., and Lopes, M. A comparative study of continual backpropagation. In Proc. 23rd EPIA Conf. Artificial Intelligence, pp. 324–334, 2024. (pdf)

PhD theses

J. Ribeiro. Ad Hoc Teamwork in Realistic Environments. PhD thesis, Instituto Superior Técnico, Univ. Lisboa, 2025. (pdf)

M. Vasco. Multimodal Representation Learning for Agent Perception and Action. PhD thesis, Instituto Superior Técnico, Univ. Lisboa, 2023. (pdf)

MSc theses

S. Carvalho. Using autonomous vehicles to improve traffic conditions. MSc thesis, Instituto Superior Técnico, Univ. Lisboa, 2022. (pdf)

B. Esteves. Efficient transfer in model-based reinforcement learning. MSc thesis, Instituto Superior Técnico, Univ. Lisboa, 2022. (pdf)

A. Fernandes. Leveraging deep unsupervised models towards learning robust multimodal representations. MSc thesis, Instituto Superior Técnico, Univ. Lisboa, 2023. (pdf)

G. Jardim. Using feudal hierarchies for non-stationary reinforcement learning. MSc thesis, Instituto Superior Técnico, Univ. Lisboa, 2024. (pdf)

E. Paiva. Using deep model-based RL in ad hoc teamwork with partial observability. MSc thesis, Instituto Superior Técnico, Univ. Lisboa, 2023. (pdf)

G. Querido. Learning to perceive in deep model-free reinforcement learning. MSc thesis, Instituto Superior Técnico, Univ. Lisboa, 2022. (pdf)

G. Salvador. Discounting strategies for planning in hazardous environments. MSc thesis, Instituto Superior Técnico, Univ. Lisboa, 2024. (pdf)

R. Zayanov. Interactively teaching an inverse reinforcement learner with limited feedback. MSc thesis, Instituto Superior Técnico, Univ. Lisboa, 2022. (pdf)

Datasets and software

M. Vasco, H. Yin, Francisco S. Melo, A. Paiva. (2022) Multimodal Handwritten Digits (MHD) dataset