I created a robots.txt Orchard CMS Module that leverages InfoSet in Orchard 1.8 for Content Parts. This alleviates the need for custom tables via SchemaBuilder.CreateTable, which can reduce the number of table joins, reduce the possibility of Select N+1 queries, increase Orchard performance, and eliminate extra noise in your Orchard code base and SQL Server database. I mentioned a couple of other modules that use Infoset, too, Smart App Banners and Live Chat Modules for Orchard CMS.
I created a robots.txt module quite some time ago, but this new version is for the upcoming Orchard 1.8 that uses Infoset for saving Content Part data. Simply enable the module and enter the robots.txt instructions and the content will be provided to all robots and web crawlers that request robots.txt. In addition to using Infoset, all information is cached to provide the best Orchard performance possible.
I mentioned a bit more detail about Infoset and Content Parts in my post on converting the Olark Live Chat Module to use Infoset.
I am really excited for Orchard CMS 1.8 and its support for ASP.NET MVC 5, ASP.NET Web API 2, and Razor 3 as well as Infoset for storing Content Part data. Check out other Orchard Tutorials as well as ASP.NET MVC 5 Tutorials.