Blog items tagged with "robots-txt"
When reading blogs that discuss topics related to SEO, most of the time you'll read about on-site strategies such as keyword optimization, image optimization, cross linking, etc.
Most of these on-site strategies can be easily executed through the administration interface of your Exponent CMS website.
Some of the mission critical items for SEO that you can't execute through the admin interface of your Exponent CMS must be done on the web server through a FTP or cPanel/WHM type interface.
This post is the first in a series of submissions I'll write over the next few days discussing advanced server-side SEO tips for your Exponent CMS website include editing your robots.txt and .htaccess files, as well as pushing webmaster tools verification files and XML sitemaps to your web server.
In this edition, Part I, I'll discuss how modifying the Robots.txt file that resides on your web server can eliminate issues with duplicate content.
The Robots.txt file on your web server defines crawling parameters for robots that crawl websites all over the Internet. For SEO, the Robots.txt file is a way to allow/disallow search engine robots (such as Googlebot) that index your web pages from crawling specific directories on your website.
If you're working with a new install of Exponent CMS, the Robots.txt file will be included in your installation with a basic set of disallow parameters that will look like this:
Because Exponent CMS is a dynamic platform, there are several modules within the system such as blog/news/portfolio or ecommerce that can pose some SEO headaches revolving around duplicate content.
Some of the most frequent issues that I encounter from Exponent CMS websites in terms of duplicate title tags, content, etc., come from the tagging functionality (blogs in specific) and when login/view cart links are hard coded for users on ecommerce websites.
If a robot can crawl a link to a blog tag or a link to login, this often times causes that URL to be indexed – more often than not with a duplicate Title or Description tag as other dynamic pages on your site. The example below shows an ecommerce Exponent CMS website that has had its login module link indexed. As you can see, the meta description is exactly the same as the homepage meta description:
You can easily prevent this type of SEO issue by disallowing the /loginmodule (or for newer versions of Exponent the /login) or any other troublesome directories within your Robots.txt file.
In the screenshot below, you can see where this Robots.txt file has added more parameters than the initial file that comes with your Exponent CMS install.
Simply pull down this file from public_html through your FTP or cPanel/WHM interface, make these modifications and push that file back up to your web server to ensure you don't experience these easily avoidable SEO headaches.
In Part II of these server-side SEO tips for Exponent CMS sites, I'll discuss how to modify the .htaccess in order to redirect URLs and set a preferred domain URL structure.
About the Author
Chris J. Everett is an SEO expert and the founder of Captivate Search Marketing, a digital marketing firm based in Atlanta, Georgia.
Chris has worked with Exponent CMS in a variety of roles since 2007 and continues to work with the software's developers to further enhance its SEO friendliness.