Project: [Closed] (PwC) Question bank generator for Applied GenAI
Description
(PwC) Question bank generator for Applied GenAI
PwC has developed several GenAI applications using models that have been trained on a large corpus of text and can retrieve relevant parts of that corpus when prompted by a user's questions (known as RAG-LLMs). Though many businesses are interested in this functionality, it's proven difficult to quantify a model's performance, besides running expensive trials with end users. This may hurt our ability to market such applications. In this project, we propose to revert the underlying architecture of an existing RAG-LLM to create, essentially, a question generator. While building the generator, the student will analyse the suitability of the model's output, both in terms of comprehensiveness (i.e. do the questions test/cover the entirety of the input text) and specificity (i.e. would the questions make for a difficult but fair test?). Additional avenues of analysis could be the 'reversal curse' and model bias. If successful, the resulting question banks can serve as test case input for other GenAI models so their accuracy is more easily judged by a human end users, before sending it out into the wild.
Please contact Bart Engelen ( l.j.t.engelen@tue.nl ) for questions on this project.
• Adolphs, L., Huebscher, M. C., Buck, C., Girgin, S., Bachem, O., Ciaramita, M., & Hofmann, T. (2022). Decoding A Neural Retriever's Latent Space for Query Suggestion. arXiv preprint arXiv:2210.12084.
• Morris, J. X., Zhao, W., Chiu, J. T., Shmatikov, V., & Rush, A. M. (2023). Language model inversion. arXiv preprint arXiv:2311.13647.
Details
- Supervisor
-
Bart Engelen
- External location
- PWC
- Link
- Thesis