Transfer is a ubiquitous concept in machine learning. The most common form is transfer learning from big pretrained models (e.g. by finetuning or zero-shot predictions), but it is also present in multi-task learning, meta-learning, and continual learning. Still, we have very little understanding of when transfer is successful (when we can learn new tasks very fast) and when it isn't. Given a pretrained model, how could we know whether it will adapt well to a new task or not?

This topic is about gaining understanding and potentially automating this process. It would have significant impact in all kinds of applications.

Imagine having a large library of pretrained models (e.g. for computer vision or LLMs). Then, given a new task/dataset, you would have to automatically define the process for finetuning that model for the new dataset. I.e. you would need to decide:

* Which model and pretrained weights to use

* How to finetune it to the new task. E.g. if you have a very small target dataset, you will want to do few-shot learning. If it is larger, maybe you can simply finetune the model. If it is a very similar problem, maybe continual learning is the best thing to do. 

* Maybe you can't predefine the neural architecture, maybe you need to do a bot of (efficient) Neural Architecture Search to finetine the architecture to the target problem as well as finetuning the weights.

There are many unanswered questions. Is it worth pretraining different models for different situations, or do we need only one big pretrained model (big pretraining) for all future tasks? If so, how do you optimally train that big pretrained model?

This is a general topic that we can finetune further after initial discussions. You'll need a researcher's attitude. You'll have to (help) finetune the research question after reading the relevant literature and potentially doing experiments. You'll also need to able to answer questions by yourself via reading and experimentation.