THIS IS A READ-ONLY ARCHIVE FROM THE SORABJI.COM MESSAGE BOARDS (1995-2016). |
---|
By Sorabji on Thursday, January 22, 1998 - 10:47 pm: |
i was looking forward to this being the week of the pope meeting castro. instead it is a footnote. not that the press is any barometer of what really matters, but it's been tough getting any information at all about the event. anyway, this story would have made an interesting afterthought to a vastly meaningful week. days earlier, i accidentally re-discovered The Road to Nowhere, sort of an interesting little place which doesn't look like it gets updated much. i actually spent a few minutes last week exploring robots.txt files of the big, fat, huge websites. robots.txt files are the files which search engines look for so they can know what parts of a website not to index. to look at it another way, these are the files which tell you where the good stuff is. for instance, the CNN robots.txt file leads one to the such jewels as the CNN.com webmaster logs, (this link chokes MSIE 3.0) freely available to the world except for one internal frame. this page shows you how the CNN servers are doing, in case you care. Usually, though, robots files are pretty innocuous. Take abcnews.com. very lame. From the the Yahoo robots file we learn that Roverbot is a bad seed. I never even heard of it. What I've always wondered, though, is why my robots.txt file gets almost all its hits from dial-up slip and ppp customers? you'd think only bots and machines and search crawlers of loving grace would care about these files, but perhaps there are others like me out here on dialup. You can also, if you have aboslutely nothing else to do, find interesting pages by typing in "index2.html" or "test.html" or "index.test.html" or something like that after your favorite domain name. Web types are so predictable. |
By R.C. on Friday, January 23, 1998 - 12:13 am: |
|
By R.C. on Friday, January 23, 1998 - 12:58 am: |
|
By Christop.com on Tuesday, April 7, 1998 - 07:36 pm: |
my favourite so far is benicetobears. this gave me a chuckle. |