Recent advances in Large Language Models (LLMs) and AI agents have enabled systems that can perform software engineering tasks such as requirements analysis, code generation, testing, debugging, and code reviews. While these systems show promising results on isolated tasks, their effectiveness in end-to-end software development workflows remains poorly understood.

Organizations are interested in understanding how far autonomous software engineering can be taken in practice, and which software development activities can be safely delegated to AI agents while maintaining quality, security, and compliance standards. It remains unclear which level of autonomy can realistically be achieved and how autonomous software engineering agents should be evaluated.

Investigate the feasibility, reliability, and limitations of autonomous software engineering agents in realistic software development environments.

The student will design and implement an experimental framework in which AI agents perform software engineering tasks with varying levels of autonomy. The framework will be used to evaluate the performance of agents across different development scenarios and investigate factors that contribute to success and failure.

· Autonomous feature implementation

· Automated bug fixing

· Agent-driven software testing

· Multi-agent collaboration during software development

· Human-AI collaboration models

· Self-improving software engineering agents

· Which software engineering activities can be reliably performed by autonomous AI agents?

· How does agent autonomy impact software quality, productivity, and maintainability?

· What are the most common failure modes of autonomous software engineering agents?

· How can autonomous software engineering systems be evaluated beyond traditional coding benchmarks?

· How do single-agent and multi-agent approaches compare for complex software engineering tasks?

· What level of human oversight is required to ensure trustworthy operation?

Possible scientific contributions include:

· A novel evaluation framework for autonomous software engineering systems

· New metrics for measuring agent autonomy and reliability

· Empirical comparison of single-agent and multi-agent software engineering approaches

· Identification and categorization of failure modes in autonomous development workflows

· Guidelines for designing trustworthy software engineering agents in enterprise environments

· Literature review on AI agents and autonomous software engineering

· Experimental framework and benchmark scenarios

· Prototype implementation of one or more autonomous software engineering agents

· Empirical evaluation using quantitative and qualitative metrics

· Analysis of limitations, risks, and future research directions

· Final thesis report and presentation

Generative AI systems are increasingly used in software development and business processes, creating new cost-management challenges. Unlike traditional cloud costs, token usage is often variable, difficult to attribute, and influenced by prompts, context size, model selection, output length, and retry behavior.

Organizations need a way to monitor and optimize AI usage without slowing down experimentation, adoption, or developer productivity. This thesis investigates how an AI FinOps way of working can create cost transparency and responsible usage while preserving business value.

Develop a practical operating model and measurement framework for AI token monitoring, cost attribution, and process optimization in enterprise environments.

The student will analyze AI usage patterns, identify token waste drivers, and design governance mechanisms such as showback dashboards, model tiering, guardrails, and feedback loops that support responsible AI adoption.

· Token monitoring and cost attribution

· Prompt and context optimization

· Model tiering and usage policies

· Cost-per-outcome metrics

· Governance models for AI adoption

· How can token usage be monitored and attributed in enterprise AI environments?

· Which usage patterns create the largest token waste?

· How can organizations reduce AI costs without limiting productivity or adoption?

· Which governance mechanisms support responsible and effective AI usage?

· A framework for AI FinOps in token-based and agentic AI workloads

· A taxonomy of token waste drivers in enterprise AI usage

· A measurement model linking token cost to productivity and business value

· Design guidelines for governance mechanisms that improve transparency without blocking useful AI adoption

· Literature review on AI FinOps and GenAI cost management

· AI FinOps operating model and measurement framework

· Analysis of token waste drivers and optimization opportunities

· Empirical evaluation using selected AI workflows or proof-of-concept scenarios

· Final thesis report and presentation

Organizations are scaling GenAI and ML, but often lack governance, compliance, and auditability in their MLOps pipelines. The EU AI Act increases the need for demonstrable control mechanisms, while standard pipelines typically do not include automated quality and risk checks.

Design and implement a Responsible-AI Gate that automatically evaluates data quality, bias, fairness, model behavior, explainability, documentation, audit trails, and deployment approval criteria in a CI/CD pipeline.

· Responsible-AI metrics

· MLOps governance

· Bias, fairness, and safety checks

· Auditability and deployment approval

· Which Responsible-AI metrics are minimally required based on NIST and the EU AI Act?

· How can Responsible-AI checks be integrated into DevOps without reducing delivery speed?

· What are the trade-offs between reliability, cost, and latency?

· Responsible-AI Gate module

· Python evaluation scripts for bias, fairness, and safety

· Governance dashboard

· Model card template and architecture document

· Final thesis report with impact analysis

GenAI systems can show inconsistent quality due to hallucinations, weak retrieval, variable latency, and changing costs. Organizations need predictable KPIs and a reliable evaluation framework for RAG, LLM agents, and Copilot scenarios.

Develop an evaluation harness that automatically tests GenAI systems on groundedness, correctness, hallucination detection, retrieval quality, latency, token costs, robustness, and regression behavior after model updates.

· Groundedness and hallucination metrics

· RAG evaluation

· Regression testing for LLMs

· Latency and token-cost analysis

· How can reliable groundedness and hallucination metrics be defined?

· How can RAG systems be evaluated systematically?

· Which techniques are suitable for LLM regression testing?

· Where are the main opportunities for cost optimization?

· Python evaluation framework

· Benchmark dataset and judge prompts

· Test report generator for quality, risk, and cost

· CI integration for regression tests

· Final thesis report with recommendations

Software teams use AI tools such as GitHub Copilot and Claude Code, but often lack an integrated workflow that connects code repositories, ticketing systems, bug flows, and pull request generation.

Develop a prototype of an LLM-driven development workflow manager that reacts to GitHub events, integrates with Jira or Azure DevOps, links bugs to code locations, and generates initial pull request proposals using an LLM.

· LLM-based code patch generation

· GitHub event automation

· Integration with Jira or Azure DevOps

· Safety checks for automated pull requests

· How reliable are LLMs in generating small, controlled code patches for bug fixes?

· Which GitHub event triggers provide the most value for automation?

· How can unwanted changes in automated pull requests be prevented?

· What are the relevant limitations regarding security, tokens, permissions, and rate limits?

· Prototype LLM Development Cycle Manager

· Observability dashboard

· Cost governance playbook

· Final thesis report and demonstration environment

AI-Enabled Data Verification and Reporting for Industrial Data Migration (with ASML / Eviden)

Industrial data migrations transform source data so that it fits the target platform, the target data model, and the related data-quality requirements. Verification therefore has to reason over three information sources at once: the original source data, the transformed data, and the documented transformation requirements. In the ASML context this is slow work, because quality must be checked across every processing step and deviations must be explained rather than only counted. Transformation is also not a single jump. Source data is first translated into an abstract, concept-level model, as in the transformation-engine set-up developed by Wim, before it is written to the target. Each phase can lose or distort data, so verification and reporting should follow the same phasing instead of comparing source and target as one black box. For complex PLM and engineering data this matters: completeness spans parts, revisions, datasets, named references and BOM lines, and cannot be judged on record counts alone. This assignment continues the earlier student research supervised by Yuri Visser and widens the question: across the full migration chain, from requirements capture and export through transformation, validation and reporting, where can AI add value and what are the risks? A central sub-question is how transformation requirements must be written and structured so that an AI can genuinely work with them.

The precise thesis scope is agreed with the company supervisor and the academic supervisor; the student is not expected to cover all objectives.

• Survey where AI can be applied across the whole migration chain — requirements, export, transformation, validation and reporting — and establish the state of practice in ETL testing, automated reconciliation and data-quality monitoring.

• Define the semantics and structure that transformation requirements, mappings and validation rules need in order to serve as machine-readable input for AI, including whether Croissant or a comparable metadata format is suitable.

• Define a phased verification model that treats source data, the concept-level model and the target separately, rather than as one transformation step.

• Investigate how AI can support validation-rule generation, anomaly detection, discrepancy classification, root-cause explanation and management-ready reporting.

• Propose reusable concepts and architecture patterns, using the existing transformation-engine set-up as reference implementation, so the approach carries across clients, data domains and migration waves.

• Explore risks and boundary conditions: hallucination, traceability, auditability, confidentiality, and the minimum human-in-the-loop requirements.

Phase 1: Previous work and solution landscape

Start from the earlier thesis work in Yuri Visser's track and from the internal ASML migration-quality material, then review external practice: row counts, checksums, schema comparison, mapping validation and audit trails on the traditional side; anomaly detection, discrepancy prioritisation and automated test generation on the AI side. Identify commercial and open-source tools that already claim AI-supported migration validation.

Phase 2: Phase model and requirement semantics

Split the migration into its real phases and describe what must be verified at each boundary. The reference case is the existing transformation-engine set-up, which translates source data into abstract tables before loading into the target. Define, per phase, how the inputs are represented and compared: source data as it stands before transformation; the concept-level model of abstract tables, where most requirement logic is applied; and the target data together with the rules, mappings and constraints that define the intended transformation, expressed in a form an AI can consume.

Phase 3: Prototype an AI-supported verification workflow

Build a proof of concept for one or more verification tasks, keeping deterministic checks strictly separate from AI-generated insight: deterministic checks supply the evidence, AI supplies interpretation, prioritisation and explanation. Candidate functions are generating validation checks from transformation requirements, comparing records across phases and explaining the differences, classifying discrepancies as expected deviation, transformation error, missing data or data-quality issue, and summarising migration quality for business and technical stakeholders.

Phase 4: Reporting and explainability

Define a reporting format that turns technical verification results into actionable migration-quality insight. The internal completeness reporting suggests it should carry an overall quality summary with completeness percentages, completeness by object type (parts, revisions, datasets, named references, BOM lines), verification status per processing step (export, transfer, transform, load), the top discrepancy categories, AI-generated explanations with confidence indicators, and traceability back to source records, transformation rules and validation evidence.

Phase 5: Risks and governance

Assess the risks of applying AI here, hallucination first among them, and propose safeguards: audit trails, a hard separation between AI suggestions and approved evidence, rule-based cross-checks, prompt and version control, data-access constraints, and mandatory review of high-impact findings. Define when AI output may be accepted automatically and when human review is compulsory.

• AI-assisted validation rule generation: derive validation rules automatically from transformation requirements, mapping specifications and example datasets.

• AI-supported discrepancy explanation and classification: compare data across migration phases and classify differences as expected, explainable, suspicious or erroneous.

• Metadata-driven reusable validation framework: determine how source, concept-level and target data plus transformation requirements should be structured for reuse across clients and projects, including an assessment of Croissant or a comparable schema-documentation approach.

• AI-for-transformation with independent validation: use AI to propose or perform transformations, verified independently by a separate AI agent or a deterministic validation layer.

Supervisors: Eric Lakerveld and Yuri Visser. Students with an interest in applied AI, data migration, PLM and engineering data, data governance and enterprise reporting are invited to apply. The assignment offers a practical, high-value industrial problem: making data-migration verification faster, more reliable, more transparent and reusable.