I helped write some of the earliest parts of this code from 1999-2002. Lots of fun times:
— Jeff Dean (@JeffDean) July 2, 2019
What should you do with robots.txt files in MS Word format?
One site had:
User-agent *
Disallow: /
Instead of:
User-agent: *
Disallow: /
(We made our parser less strict) https://t.co/7FnX8lFKqu
from Twitter https://twitter.com/JeffDean
July 01, 2019 at 09:27PM
via IFTTT
No comments:
Post a Comment