Exploratory testing of CAPTCHAs

You can reliably test CAPTCHA on review apps, and in your local development environment (GDK). You can always:

Force a reCAPTCHA to appear where it is supported.
Force a checkbox to display, instead of street sign images to find and select.

To set up testing, follow the configuration on this page.

Use appropriate test data

Make sure you are testing a scenario which has spam/CAPTCHA enabled. For example: make sure you are editing a public snippet, as only public snippets are checked for spam.

Enable feature flags

Enable any relevant feature flag, if the spam/CAPTCHA support is behind a feature flag.

Set up Akismet and reCAPTCHA

To set up reCAPTCHA:
1. Review the GitLab reCAPTCHA documentation.
2. Follow the instructions provided by Google to get the official test reCAPTCHA credentials.
  1. For Site key, use: 6LeIxAcTAAAAAJcZVRqyHh71UMIEGNQ_MXjiZKhI
  2. For Secret key, use: 6LeIxAcTAAAAAGG-vFI1TnRWxMZNFuojJ4WifJWe
3. Go to Admin -> Settings -> Reporting settings: http://gdk.test:3000/admin/application_settings/reporting#js-spam-settings
4. Expand the Spam and Anti-bot Protection section.
5. Select Enable reCAPTCHA. Enabling for login is not required unless you are testing that feature.
6. Enter the Site key and Secret key.
To set up Akismet:
1. Review the GitLab documentation on Akismet.
2. Get an Akismet API key. You can sign up for a testing key from Akismet. You must enter your local host (such asgdk.test) and email when signing up.
3. Go to GitLab Akismet settings page, for example: http://gdk.test:3000/admin/application_settings/reporting#js-spam-settings
4. Enable Akismet and enter your Akismet API key.
To force an Akismet false-positive spam check, refer to the Akismet API documentation and Akismet Getting Started documentation for more details:
1. You can use [email protected] as the author email to force spam using the following steps:
  1. Go to user email settings: http://gdk.test:3000/-/profile/emails
  2. Add [email protected] as a secondary email for the administrator user.
  3. Confirm it in the Rails console: bin/rails c -> User.find_by_username('root').emails.last.confirm
  4. Switch this verified email to be your primary email:
    1. Go to Avatar dropdown list -> Edit Profile -> Main Settings.
    2. For Email, enter [email protected] to replace [email protected].
    3. Select Update Profile Settings to save your changes.

Test in the web UI

After you have all the above configuration in place, you can test CAPTCHAs. Test in an area of the application which already has CAPTCHA support, such as:

Creating or editing an issue.
Creating or editing a public snippet. Only public snippets are checked for spam.

Test in a development environment

After you force Spam Flagging + CAPTCHA using the steps above, you can test the behavior with any spam-protected model/controller action.

Test with CAPTCHA enabled (CONDITIONAL_ALLOW verdict)

If CAPTCHA is enabled in these areas, you must solve the CAPTCHA popup modal before you can resubmit the form:

Admin -> Settings -> Reporting -> Spam
Anti-bot Protection -> Enable reCAPTCHA

Testing with CAPTCHA disabled ("DISALLOW" verdict)

If CAPTCHA is disabled in Admin -> Settings -> Reporting -> Spam and Anti-bot Protection -> Enable reCAPTCHA, no CAPTCHA popup displays. You are prevented from submitting the form at all.

HTML page to render reCAPTCHA

NOTE: If you use the Google official test reCAPTCHA credentials listed in Set up Akismet and reCAPTCHA, the CAPTCHA response string does not matter. It can be any string. If you use a real, valid key pair, you must solve the CAPTCHA to obtain a valid CAPTCHA response to use. You can do this once only, and only before it expires.

To directly test the GraphQL API via GraphQL Explorer (http://gdk.test:3000/-/graphql-explorer), get a reCAPTCHA response string via this form: public/recaptcha.html (http://gdk.test:3000/recaptcha.html):

<html>
<head>
  <title>reCAPTCHA demo: Explicit render after an onload callback</title>
  <script type="text/javascript">
  var onloadCallback = function() {
    grecaptcha.render('html_element', {
      'sitekey' : '6Ld05AsaAAAAAMsm1yTUp4qsdFARN15rQJPPqv6i'
    });
  };
  function onSubmit() {
    window.document.getElementById('recaptchaResponse').innerHTML = grecaptcha.getResponse();
    return false;
  }
  </script>
</head>
<body>
<form onsubmit="return onSubmit()">
  <div id="html_element"></div>
  <br>
  <input type="submit" value="Submit">
</form>
<div>
  <h1>recaptchaResponse:</h1>
  <div id="recaptchaResponse"></div>
</div>
<script src="https://www.google.com/recaptcha/api.js?onload=onloadCallback&render=explicit"
        async defer>
</script>
</body>
</html>

Spam/CAPTCHA API exploratory testing examples

These sections describe the steps needed to perform manual exploratory testing of various scenarios of the Spam and CAPTCHA behavior for the REST and GraphQL APIs.

For the prerequisites, you must:

Perform all the steps listed above to enable Spam and CAPTCHA in the development environment, and force form submissions to require a CAPTCHA.
Ensure you have created an HTML page to render CAPTCHA under the /public directory, with a page that contains a form to manually generate a valid CAPTCHA response string. If you use Google's official test reCAPTCHA credentials listed in Set up Akismet and reCAPTCHA, the contents of the CAPTCHA response string don't matter.
Go to Admin -> Settings -> Reporting -> Spam and Anti-bot protection.
Select or clear Enable reCAPTCHA and Enable Akismet according to your scenario's needs.

The following examples use snippet creation as an example. You could also use snippet updates, issue creation, or issue updates. Issues and snippets are the only models with full Spam and CAPTCHA support.

Initial setup

Create an API token.
Export it in your terminal for the REST commands: export PRIVATE_TOKEN=<your_api_token>
Ensure you are signed into the GitLab development environment at localhost:3000 before using GraphiQL explorer, because it uses your authenticated user as authorization for running GraphQL queries.
For the GraphQL examples, use the GraphiQL explorer at http://localhost:3000/-/graphql-explorer.
Use the --include (-i) option to curl to print the HTTP response headers, including the status code.

Scenario: Akismet and CAPTCHA enabled

In this example, Akismet and CAPTCHA are enabled:

Initial request.

Initial request

This initial request fails because no CAPTCHA response is provided.

REST request:

curl --request POST --header "PRIVATE-TOKEN: $PRIVATE_TOKEN" "http://localhost:3000/api/v4/snippets?title=Title&file_name=FileName&content=Content&visibility=public"

REST response:

{"needs_captcha_response":true,"spam_log_id":42,"captcha_site_key":"XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX","message":{"error":"Your snippet has been recognized as spam. Please, change the content or solve the reCAPTCHA to proceed."}}

GraphQL request:

mutation {
    createSnippet(input: {
        title: "Title"
        visibilityLevel: public
        blobActions: [
            {
                action: create
                filePath: "BlobPath"
                content: "BlobContent"
            }
        ]
    }) {
        snippet {
            id
            title
        }
        errors
    }
}

GraphQL response:

{
  "data": {
    "createSnippet": null
  },
  "errors": [
    {
      "message": "Request denied. Solve CAPTCHA challenge and retry",
      "locations": [
        {
          "line": 22,
          "column": 5
        }
      ],
      "path": [
        "createSnippet"
      ],
      "extensions": {
        "needs_captcha_response": true,
        "spam_log_id": 140,
        "captcha_site_key": "XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX"
      }
    }
  ]
}

Second request

This request succeeds because a CAPTCHA response is provided.

REST request:

export CAPTCHA_RESPONSE="<CAPTCHA response obtained from HTML page to render CAPTCHA>"
export SPAM_LOG_ID="<spam_log_id obtained from initial REST response>"
curl --request POST --header "PRIVATE-TOKEN: $PRIVATE_TOKEN" --header "X-GitLab-Captcha-Response: $CAPTCHA_RESPONSE" --header "X-GitLab-Spam-Log-Id: $SPAM_LOG_ID" "http://localhost:3000/api/v4/snippets?title=Title&file_name=FileName&content=Content&visibility=public"

REST response:

{"id":42,"title":"Title","description":null,"visibility":"public", "other_fields": "..."}

GraphQL request:

NOTE: The GitLab GraphiQL implementation doesn't allow passing of headers, so we must write this as a curl query. Here, --data-binary is used to properly handle escaped double quotes in the JSON-embedded query.

export CAPTCHA_RESPONSE="<CAPTCHA response obtained from HTML page to render CAPTCHA>"
export SPAM_LOG_ID="<spam_log_id obtained from initial REST response>"
curl --include "http://localhost:3000/api/graphql" --header "Authorization: Bearer $PRIVATE_TOKEN" --header "Content-Type: application/json" --header "X-GitLab-Captcha-Response: $CAPTCHA_RESPONSE" --header "X-GitLab-Spam-Log-Id: $SPAM_LOG_ID" --request POST --data-binary '{"query": "mutation {createSnippet(input: {title: \"Title\" visibilityLevel: public blobActions: [ { action: create filePath: \"BlobPath\" content: \"BlobContent\" } ] }) { snippet { id title } errors }}"}'

GraphQL response:

{"data":{"createSnippet":{"snippet":{"id":"gid://gitlab/PersonalSnippet/42","title":"Title"},"errors":[]}}}

Scenario: Akismet enabled, CAPTCHA disabled

For this scenario, ensure you clear Enable reCAPTCHA in the Admin area settings as described above. If CAPTCHA is not enabled, any request flagged as potential spam fails with no chance to resubmit, even if it could otherwise be resubmitted if CAPTCHA were enabled and successfully solved.

The REST request is the same as if CAPTCHA was enabled:

curl --request POST --header "PRIVATE-TOKEN: $PRIVATE_TOKEN" "http://localhost:3000/api/v4/snippets?title=Title&file_name=FileName&content=Content&visibility=public"

REST response:

{"message":{"error":"Your snippet has been recognized as spam and has been discarded."}}

GraphQL request:

mutation {
    createSnippet(input: {
        title: "Title"
        visibilityLevel: public
        blobActions: [
            {
                action: create
                filePath: "BlobPath"
                content: "BlobContent"
            }
        ]
    }) {
        snippet {
            id
            title
        }
        errors
    }
}

GraphQL response:

{
  "data": {
    "createSnippet": null
  },
  "errors": [
    {
      "message": "Request denied. Spam detected",
      "locations": [
        {
          "line": 22,
          "column": 5
        }
      ],
      "path": [
        "createSnippet"
      ],
      "extensions": {
        "spam": true
      }
    }
  ]
}

Scenario: `allow_possible_spam` application setting enabled

With the allow_possible_spam application setting enabled, the API returns a 200 response. Any valid request is successful and no CAPTCHA is presented, even if the request is considered spam.

Exploratory testing of CAPTCHAs

Use appropriate test data

Enable feature flags

Set up Akismet and reCAPTCHA

Test in the web UI

Test in a development environment

Test with CAPTCHA enabled (CONDITIONAL_ALLOW verdict)

Testing with CAPTCHA disabled ("DISALLOW" verdict)

HTML page to render reCAPTCHA

Spam/CAPTCHA API exploratory testing examples

Initial setup

Scenario: Akismet and CAPTCHA enabled

Initial request

Second request

Scenario: Akismet enabled, CAPTCHA disabled

Scenario: allow_possible_spam application setting enabled

Scenario: `allow_possible_spam` application setting enabled