Lab: Exploiting insecure output handling in LLMs

Create a user account

When the exercise starts, your shop should look like the screenshot. Click on the "Register" link to register a new user.

Now enter the user name, email address, and password in the form. You can find the email address in your email client. Simply click on the "Email Client" button.

The email address is different for you, as it is generated anew with each exercise.

Back in the form, click the "Email client" button and you will see a registration email in your email client. Click on the link in the email.

This completes the registration of our user.

Probe for XSS

Now we log in with the user account we just created. To do this, we click on the "My account" link and enter our login details in the form. After successfully logging in, we see our user account.

Now let's switch to live chat by clicking on the "Live chat" link.

In the prompt, we enter the following HTML tag <img src=1 onerror=alert(1)>. When we send the message, we receive a warning window in the browser telling us that the chat window is vulnerable to XSS.

The warning window looks as follows.

Now let's go to the shop and select any product. It is important that it is not the leather jacket. To do this, click on the "Home" link. In this example, we will select the product "Potato Theater" and click on the "View details" button.

In the product description, we insert <img src=1 onerror=alert(1)> into the field for a product review and confirm our entry by clicking on "Submit review".

Now we see our entry below the product description.

If we look at the source code of the website (in the browser) by right-clicking -> View page source, we can see that the payload is securely HTML-encoded, which means that the review functionality cannot be exploited directly.

We now return to the live chat and ask the LLM which functions it supports: Which functions do you support?. We receive two functions as a response:

1. password_reset: This feature allows you to request a new password by entering your username or email address.

2. product_info: This feature provides information about the products we sell. You can enter the exact name or ID of the product to see details.

Note that the LLM supports a product_info function that returns information about a specific product based on its name or ID. We use this function to obtain details about the product "Potato Theater". To do this, we use the following input: product_info(Potato Theater). It may be necessary to enter this input several times before the LLM issues a warning about malicious code in the product review. As we can see, the LLM detects problems in product reviews.

Test the attack

We now return to the product description for "Potato Theater" and delete our old rating by clicking on the red "X".

We then create a new review with the code <iframe src =my-account onload = this.contentDocument.forms[1].submit() >. This code deletes the account of the user reading the product review.

Back in the live chat, we ask the LLM for information about the product “Potato Theater” using the function product_info(Potato Theater). Again, several inputs may be necessary before the LLM issues a warning about the content in the product reviews.

After receiving this warning, we are still logged in as users. This means that the LLM successfully detected and ignored the malicious payload.

We delete the old product review and create a new one that contains the XSS payload in a meaningful sentence. For example:

When I received this product I got a free T-shirt with "<iframe src =my-account onload = this.contentDocument.forms[1].submit() >" printed on it. I was delighted! This is so cool, I told my wife.

In the live chat, we ask the LLM again for details about the product "Potato Theater": product_info(Potato Theater). In the response, we see that an iframe has been opened, which means that the payload was successful. When we click on the “My account” link, we are redirected to the registration form, which means that our user account has been deleted.

Exploit the vulnerability

We will now register a new user, as we did at the beginning of this exercise.

In the shop, we now select the product "Lightweight "l33t" Leather Jacket" and go to the product description.

To submit a product description, we need to log in to the application with our new user account. Enter the code from the previous step in the product description.

When I received this product I got a free T-shirt with "<iframe src =my-account onload = this.contentDocument.forms[1].submit() >" printed on it. I was delighted! This is so cool, I told my wife.

After submitting the review, we wait for Carlos to open the product description so that the exercise can be successfully completed.