Artificial intelligence is transforming society. AI Sweden is the national center for applied artificial intelligence and our mission is to accelerate the use of AI for the benefit of our society, our competitiveness, and for everyone living in Sweden. We drive impactful initiatives in areas such as healthcare, energy, and the public sector while pushing the boundaries of AI research and innovation in fields such as natural language processing and edge learning. Join us in harnessing the untapped value of AI to drive innovation and create sustainable value for Sweden.

We are now looking for a master thesis student to strengthen our team. 

Introduction

When an LLM reproduces training text verbatim or near-verbatim, it can expose personal data, confidential documents, or proprietary code. For individuals, that means privacy harm and potential misuse; for organizations, it creates regulatory exposure, loss of trade secrets, and reputational damage. Empirically, this risk is real: black-box sampling with filtering has recovered hundreds of word-for-word training snippets from GPT-2, including sensitive strings.

Regulators warn that AI models can leak training data and call for empirical stress-testing with privacy attacks and privacy-enhancing technologies. Measuring this risk is critical: without hard evidence, models are deployed blindly, exposing people to privacy harm and organizations to legal and IP liability. This thesis rigorously measures and compares data-extraction attacks under realistic black-box conditions, producing reproducible, actionable results that inform safer training, tuning, and release.

Project Background and Problem Statement

AI Sweden is leading a project to develop an open-source privacy auditing tool called LeakPro, designed to assess information leakage risks in machine learning models. This initiative, undertaken in collaboration with RISE, Sahlgrenska, Region Halland, AstraZeneca, Syndata, and Scaleout, aims to evaluate the risk of sensitive information disclosure when models trained on confidential data are made publicly available.

Recent advances in automatic prompt optimization demonstrate that query phrasing can be systematically optimized, but they target task performance rather than privacy auditing. There is no standard, reproducible method that uses automated prompt- optimization to measure LLM data-extraction risk using only black-box access and a fixed query budget. The problem is building a reproducible audit that uses automated prompt optimization to measure data extraction from LLMs.

Outline

The objectives of this project are outlined below.

1. Literature study of harm-based privacy risk models: Summarize (i) how “data extraction” is defined (exact vs.\ near-verbatim recovery of training text), (ii) practical ways to verify matches and set similarity thresholds, (iii) representative attack settings and query styles reported in the literature, and (iv) datasets and evaluation practices suitable for reproducible audits.

2. Design of a preliminary evaluation approach: From the study, specify a threat model, dataset(s), and models. Evaluate data extraction in a meaningful, calibrated way so that results are comparable across settings.

3. Prototype and evaluation: Based on insights from the study and benchmark, explore directions to enhance data extraction attacks.

Contact

Fazeleh Hoseini: fazeleh.hoseini@ai.se

Why work for AI Sweden?

To us, artificial intelligence is not only about tech, it’s a force for positive societal change. You'll be working alongside leading AI experts, scientists, journalists, linguists, policy professionals, entrepreneurs, change leaders, and many more. To work here, you don’t need to know “everything” about AI, but you need to believe in its potential to help shape our society for the better.

As an organization, we’re uniquely positioned at the sweet spot of governmental influence and startup agility. Small enough to stay adaptive and have fun but backed by and in close contact with both the government, academia and private and public sector.

Join us to make a real-world impact by contributing to initiatives that benefit society and tackle critical challenges. Be at the forefront of AI innovation, working with cutting-edge technologies and playing a key role in shaping the future of AI in Sweden.

And, within our mission, we can most certainly be a platform empowering you to realize your ideas. AI Sweden’s ability to empower partners and individual team members to do exceedingly well in their profession is a key success factor for driving positive and significant impact.

In short, we like to believe we offer our team members a place to grow, an environment for personal development.

An equal and fair working environment

We strongly believe in diversity and inclusion and are acutely aware of the skewed gender balance in our industry. We actively strive to put together a diverse team in terms of age, gender and background.

AI Sweden does not accept unsolicited support and kindly ask not to be contacted by any advertisement agents, recruitment agencies or manning companies.