Category: .NET

Hi everyone,

For a project I’m currently working on I needed a dynamic robots.txt. Because our test environment is public facing we want to keep it from being indexed by Google etc. It took a bit of Googling to find a solution that worked, but in the end it was actually pretty simple.

Here’s the action in one of the API Controllers:

    public class UtilitiesController : CustomBaseApiController
    {
        [Route("Robots.txt")]
        [HttpGet]
        public HttpResponseMessage GetRobotsFile()
        {
            var resp = new HttpResponseMessage(HttpStatusCode.OK);
            var stringBuilder = new StringBuilder();

            if (Helpers.IsProduction())
            {
                // Allow bots in production
                stringBuilder.AppendLine("user-agent: *");
                stringBuilder.AppendLine("disallow: ");
            }
            else
            {
                // Don't allow bots in non-production environments
                stringBuilder.AppendLine("user-agent: *");
                stringBuilder.AppendLine("disallow: *");
            }

            resp.Content = new StringContent(stringBuilder.ToString());

            return resp;
        }
    }

Also need to add the following to your web.config so that the robots.txt file can processed by the routing handler. Without this IIS will attempt to serve it as a static file and will return a 404 when it’s not found:

In production you’ll end up with the following:

user-agent: *
disallow:

And any other environments:

user-agent: *
disallow: *

Thanks to these answers on stackoverflow for the info:
https://stackoverflow.com/a/52270877/522859
https://stackoverflow.com/a/17037935/522859