Robot Text

 # Robots.txt Introduction and Guide | Google Search Central

# Introduction to robots.txt

[Video](https://www.youtube.com/watch?v=IXNEVt9rZG8)

A robots.txt file tells search engine crawlers which URLs the crawler can access on your site.

This is used mainly to avoid overloading your site with requests; **it is not a

mechanism for keeping a web page out of Google** . To keep a web page out of Google,

[block indexing with `noindex`](https://developers.google.com/search/docs/crawling-indexing/block-indexing)

or password-protect the page.

> [!IMPORTANT]

>

> **If you use a CMS, such as Wix or Blogger**, you might not need to (or

> be able to) edit your robots.txt file directly. Instead, your CMS might expose a search

> settings page or some other mechanism to tell search engines whether or not to crawl your

> page.

>

>

> If you want to hide or unhide one of your pages from search engines, search for instructions

> about modifying your page visibility in search engines on your CMS (for example,

> search for "wix hide page from search engines").

## What is a robots.txt file used for?

A robots.txt file is used primarily to manage crawler traffic to your site, and

*usually* to keep a file off Google, depending on the file type:

| robots.txt effect on different file types ||

|---|---|

| Web page | You can use a robots.txt file for web pages (HTML, PDF, or other [non-media formats that Google can read](https://developers.google.com/search/docs/crawling-indexing/indexable-file-types)), to manage crawling traffic if you think your server will be overwhelmed by requests from Google's crawler, or to avoid crawling unimportant or similar pages on your site. > [!WARNING] > **Warning**: Don't use a robots.txt file as a means to hide your web pages > (including PDFs and other text-based formats supported by Google) from Google Search > results. > If other pages point to your page with descriptive text, Google could still index > the URL without visiting the page. If you want to block your page from search > results, use another method such as password protection or > [`noindex`](https://developers.google.com/search/docs/crawling-indexing/block-indexing). **If your web page is blocked with a robots.txt file** , its URL can still appear in search results, but the search result [won't have a description](https://support.google.com/webmasters/answer/7489871). Image files, video files, PDFs, and other non-HTML files embedded in the blocked page will be excluded from crawling, too, unless they're referenced by other pages that are allowed for crawling. If you see this search result for your page and want to fix it, remove the robots.txt entry blocking the page. If you want to hide the page completely from Search, use [another method](https://developers.google.com/search/docs/crawling-indexing/remove-information#i-control-the-web-page). |

| Media file | Use a robots.txt file to manage crawl traffic, and also to prevent image, video, and audio files from appearing in Google Search results. This won't prevent other pages or users from linking to your image, video, or audio file. - [Read more about preventing images from appearing on Google.](https://developers.google.com/search/docs/crawling-indexing/prevent-images-on-your-page) - [Read more about how to remove or restrict your video files from appearing on Google.](https://developers.google.com/search/docs/appearance/video#remove) |

| Resource file | You can use a robots.txt file to block resource files such as unimportant image, script, or style files, **if you think that pages loaded without these resources won't be significantly affected by the loss**. However, if the absence of these resources make the page harder for Google's crawler to understand the page, don't block them, or else Google won't do a good job of analyzing pages that depend on those resources. |

## Understand the limitations of a robots.txt file

Before you create or edit a robots.txt file, you should know the limits of this URL blocking

method. Depending on your goals and situation, you might want to consider other mechanisms to

ensure your URLs are not findable on the web.

- **robots.txt rules may not be supported by all search engines.**   

  The instructions in robots.txt files cannot enforce crawler behavior to your site; it's up to the crawler to obey them. While Googlebot and other respectable web crawlers obey the instructions in a robots.txt file, other crawlers might not. Therefore, if you want to keep information secure from web crawlers, it's better to use other blocking methods, such as [password-protecting private files on your server](https://developers.google.com/search/docs/crawling-indexing/control-what-you-share).

- **Different crawlers interpret syntax differently.**   

  Although respectable web crawlers follow the rules in a robots.txt file, each crawler might interpret the rules differently. You should know the [proper syntax](https://developers.google.com/search/docs/crawling-indexing/robots/robots_txt#syntax) for addressing different web crawlers as some might not understand certain instructions.

- **A page that's disallowed in robots.txt can

  still be indexed if linked to from other sites.**   

  While Google won't crawl or index the content blocked by a robots.txt file, we might still find and index a disallowed URL if it is linked from other places on the web. As a result, the URL address and, potentially, other publicly available information such as anchor text in links to the page can still appear in Google Search results. To properly prevent your URL from appearing in Google Search results, [password-protect the files on your server](https://developers.google.com/search/docs/crawling-indexing/control-what-you-share), [use the `noindex` `meta` tag or response header](https://developers.google.com/search/docs/crawling-indexing/block-indexing), or remove the page entirely.

> [!CAUTION]

> **Caution** : Combining multiple crawling and indexing rules might cause some rules to counteract other rules. Learn how to [combine crawling with indexing and serving rules](https://developers.google.com/search/docs/crawling-indexing/robots-meta-tag#combining).

## Create or update a robots.txt file

If you decided that you need one, learn how to

[create a robots.txt file](https://developers.google.com/search/docs/crawling-indexing/robots/create-robots-txt). Or if

you already have one, learn how to

[update it](https://developers.google.com/search/docs/crawling-indexing/robots/submit-updated-robots-txt).

Want to learn more? Check out the following resources:

- [How to write and submit a robots.txt file](https://developers.google.com/crawling/docs/robots-txt/create-robots-txt)

- [Update your robots.txt file](https://developers.google.com/crawling/docs/robots-txt/submit-updated-robots-txt)

- [How Google interprets the robots.txt specification](https://developers.google.com/crawling/docs/robots-txt/robots-txt-spec)

Google adsense ad

 https://ads.google.com/aw/unifiedassetreport/vacassetdetails?ocid=1536123827&entityId=682991003246&adGroupIdForEntity=151392614090&isPMax=false&adId=682991003246&euid=827295086&__u=3706758014&uscid=1536123827&__c=5710364123&authuser=0&__e=3597558392&subid=pk-en-gdn-awa-pr-a-pmx1%21o3-s3000g1500%7Cib%3A7966379274%7Cib%3A6653296821%7Cib%3A9279461727%7C~CjwKCAjwpqHTBhAcEiwAj2Afuj1X0ftKzPeSUNap9Sp8QgHM4UG_ZnrKg_U4LxbTsN5vSSpBTZe-rBoClP8QAvD_BwE~~~21753609787~-ahpm-0000000179-0000000001

Leaderboard

  <script async src="https://pagead2.googlesyndication.com/pagead/js/adsbygoogle.js?client=ca-pub-1234567890123456" crossorigin="anonymous"></script>

<!-- Homepage Leaderboard -->
<ins class="adsbygoogle"
style="display:inline-block;width:728px;height:90px"
data-ad-client="ca-pub-1234567890123456"
data-ad-slot="1234567890"></ins>
<script>
(adsbygoogle = window.adsbygoogle || []).push({});
</script>

      </div>

Auto ads div

  <script async src="https://pagead2.googlesyndication.com/pagead/js/adsbygoogle.js?client=ca-pub-1234567890123456" crossorigin="anonymous"></script>

<!-- Homepage Leaderboard -->

<ins class="adsbygoogle"

style="display:inline-block;width:728px;height:90px"

data-ad-client="ca-pub-1234567890123456"

data-ad-slot="1234567890"></ins>

<script>

(adsbygoogle = window.adsbygoogle || []).push({});

</script>

      </div>

Auto ads Code

 Skip to main content

Help Center

Community

Announcements

Code implementation guide

Where to place ad unit code in your HTML

Next: Connect to AdSense and show ads on your WordPress site

This article describes where to add ad unit code in your HTML to serve AdSense ads.

If you are not familiar with working with HTML, you should first see the instructions on how to paste ad code in your HTML pages, depending on the tools that you use. Before proceeding, you should also know the differences between AdSense code and ad unit code.

Tip: For help with HTML, try searching for online resources using terms like "HTML basics"

On this page

Where to add ad unit code

How to position ads on your site with HTML

Where to add ad unit code

After you get and copy your ad unit code, you need to paste it between the <body> and </body> tags of your page. If you paste the ad code outside the <body> tags it will prevent your ads from appearing correctly.

Note: This is sample code only which shows where to add ad unit code. Don't use this code exactly in your own site's code.

Example of an HTML page before ad unit code has been added

<html>

   <head>

      This is the head of your page.

      <title>Example HTML page</title>

   </head>

   <body>

      This is the body of your page.

   </body>

</html>

Example of an HTML page after ad unit code has been added in the <body> tag

<html>

   <head>

      This is the head of your page.

      <title>Example HTML page</title>

   </head>

   <body>

      This is the body of your page.

      <script async src="https://pagead2.googlesyndication.com/pagead/js/adsbygoogle.js?client=ca-pub-1234567890123456" crossorigin="anonymous"></script>

<!-- Homepage Leaderboard -->

<ins class="adsbygoogle"

style="display:inline-block;width:728px;height:90px"

data-ad-client="ca-pub-1234567890123456"

data-ad-slot="1234567890"></ins>

<script>

(adsbygoogle = window.adsbygoogle || []).push({});

</script>

   </body>

   </html>

How to position ads on your site with HTML

Just like the other elements of your site (images, paragraphs, etc.), you can change the position of your Google ads by using HTML tags (<div>, <table>, etc.). For example, if you surround your ad code with the HTML tags <div align="center"> and </div>, your ads will appear in the center of your page.

Example of an HTML <div> element used to position an ad in the <body> tag

<html>

   <head>

      This is the head of your page.

      <title>Example HTML page</title>

   </head>

   <body>

      This is the body of your page.

      <div align="center">

      <script async src="https://pagead2.googlesyndication.com/pagead/js/adsbygoogle.js?client=ca-pub-1234567890123456" crossorigin="anonymous"></script>

<!-- Homepage Leaderboard -->

<ins class="adsbygoogle"

style="display:inline-block;width:728px;height:90px"

data-ad-client="ca-pub-1234567890123456"

data-ad-slot="1234567890"></ins>

<script>

(adsbygoogle = window.adsbygoogle || []).push({});

</script>

      </div>

   </body>

   </html>

Give feedback about this article

Next: Connect to AdSense and show ads on your WordPress site

Was this helpful?

YesNo

Need more help?

Try these next steps:

Post to the help community

Get answers from community members

Contact us

Tell us more and we’ll help you get there

Code implementation guide

About the AdSense code

Get and copy the AdSense code

How to paste the ad code in your page

Where to place AdSense code in your HTML

Where to place ad unit code in your HTML

WordPress sites

Common issues with ad code implementation

Use Chrome DevTools to troubleshoot issues with your ads not showing

Integrate the AdSense ad code with a Content Security Policy (CSP)

Unlock your revenue with personalized tips today!

Visit the 

Optimization tips for AdSense page to see personalized suggestions to maximize your revenue.

Unlock now

Language

Help guide

Auto add code

 <head>

      This is the head of your page.

      <title>Example HTML page</title>

   </head>

   <body>

      This is the body of your page.

   </body>

Sitemap Images

 # Image Sitemaps | Google Search Central

# Image sitemaps

Image sitemaps are a way of telling Google about other images on your site, especially those that

we might not otherwise find (such as images your site reaches with JavaScript code). You can

create a separate image sitemap or add image sitemap tags to your existing sitemap; either

approach is equally fine for Google.

Image sitemaps are based on generic sitemaps so the

[general sitemap best practices](https://developers.google.com/search/docs/crawling-indexing/sitemaps/build-sitemap#general-guidelines)

also apply to image sitemaps. We also recommend that you follow the

[general best practices for publishing images](https://developers.google.com/search/docs/appearance/google-images).

## Example image sitemap

The following example shows a regular sitemap with image sitemap extension, with two

`<url>` elements:

- `https://example.com/sample1.html`, which contains two images

- `https://example.com/sample2.html`, which contains one image

```

<?xml version="1.0" encoding="UTF-8"?>

<urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9"

    xmlns:image="http://www.google.com/schemas/sitemap-image/1.1">

  <url>

    <loc>https://example.com/sample1.html</loc>

    <image:image>

      <image:loc>https://example.com/image.jpg</image:loc>

    </image:image>

    <image:image>

      <image:loc>https://example.com/photo.jpg</image:loc>

    </image:image>

  </url>

  <url>

    <loc>https://example.com/sample2.html</loc>

    <image:image>

      <image:loc>https://example.com/picture.jpg</image:loc>

    </image:image>

  </url>

</urlset>

```

## Image sitemap reference

The `image` tags are defined in the Image Sitemaps namespace:

[`http://www.google.com/schemas/sitemap-image/1.1`](http://www.google.com/schemas/sitemap-image/1.1)

To make sure Google can use your image sitemap, you must use the following required tags:

| Required tags ||

|---|---|

| `<image:image>` | Encloses all information about a single image. Each `<url>` tag can contain up to 1,000 `<image:image>` tags. |

| `<image:loc>` | The URL of the image. In some cases, the image URL may not be on the same domain as your main site. This is fine, as long as you verify both domains in Search Console. If, for example, you use a content delivery network such as Google Sites to host your images, make sure that the hosting site is verified in Search Console. In addition, make sure that your [robots.txt](https://developers.google.com/search/docs/crawling-indexing/robots/intro) file doesn't disallow the crawling of any content you want indexed. |

### Deprecated tags and attributes

We removed the following tags and attributes from our documentation:

`<image:caption>`, `<image:geo_location>`,

`<image:title>`, `<image:license>`. See the

[deprecation announcement](https://developers.google.com/search/blog/2022/05/spring-cleaning-sitemap-extensions)

for more information.

## Troubleshooting sitemaps

If you're having trouble with your sitemap, you can investigate the errors with Google Search Console.

See Search Console's

[sitemaps troubleshooting guide](https://support.google.com/webmasters/answer/7451001#errors)

for help.

## Additional resources

Want to learn more? Check out the following resources:

- [Submit your sitemap to Google](https://developers.google.com/search/docs/crawling-indexing/sitemaps/build-sitemap#addsitemap)

- [Learn how to combine sitemap extensions](https://developers.google.com/search/docs/crawling-indexing/sitemaps/combine-sitemap-extensions)