While OpenAI allows any publisher to block GPTBot, the crawler that powers its ChatGPT/GPT-4 models, Google then also offered webmasters a quick way to do the same to block Google Bard.
Google-Extended: the new Google user agent dedicated to Bard
On September 28, 2023, Google revealed the introduction of Google extended.
This new user agent from Google gives website administrators better control over how their content is used by the generative artificial intelligence of the American company Mountainview.
It allows publishers to decide whether their sites can help improve the generative APIs of the Google Bard chatbot and Vertex AI, Google’s machine learning platform.
The initiative aims to address growing concerns about the use of copyrighted content to train AI models. It aligns with a similar approach taken by OpenAI for its GPTBot robot.
Site editors can now decide to do so block or not block the new Google-Extended user agent via the robots.txt file.
What are the challenges for publishers?
The main benefit for publishers is regaining control of their data. However, some questions remain. Although Google and OpenAI now offer the ability to block their AI learning bots, a lot of content has already been used to train their AI models.
Additionally, with the growing importance of AI in online search, publishers who choose to block AI bots may face visibility issues in the future: their sites may not be listed as sources in responses made by chatbots, which could result in a loss of visibility. in the medium/long term.
How to block Google Bard crawler?
To prevent the Google Bard crawler from accessing your content and using it for learning, simply insert this line of code into the robots.txt text file located at the root of your site:
User-agent: Google-Extended
Disallow: /We remind you that if you also want to block the OpenAi crawler, you should add this line to your robots.txt file:
User-agent: GPTBot
Disallow: /This line of code must be added in addition to the existing ones.
In short
The new Google-Extended user agent is a good initiative to provide greater transparency and control to publishers, however, it is a delicate balance between protecting publishers’ rights and the potential benefits of artificial intelligence in optimizing their online visibility. Before blocking AI chatbot crawlers like Google Bard and ChatGPT, publishers should also think about the long-term implications of their decisions.
Before leaving…
If this article is active blocking Google Bard in robots.txt you liked it, don’t hesitate to share it on social networks and sign up for our digital newsletter to receive our next articles.
You can also follow our best articles via our RSS feed: https://www.leptidigital.fr/tag/newsletter-digitale/feed/ (just insert it into your favorite RSS feed reader (e.g. Feedly)).
We are also active on LinkedIn, Twitter, Facebook and YouTube. We see them?
For any questions related to this article, do not hesitate to use the “comments” section to let us know your comment, we will reply as soon as possible (with pleasure).
You would like to receive Our best articles ?
(You can cancel at any time.)
