Welcome Guest ( Log In | Register ) | Resend Validation Email |
tsukikomi | Posted: Dec 16 2014, 01:13 PM |
Falbury~<3 Group: TRAP CLUB Posts: 708 Member No.: 43,994 Joined: Jun 23rd 2014 Location: Southcenter Parkway | Whats is there purpose here exactly? Is it to collect info on us to sell to 3rd parties? |
kyonpalm | Posted: Dec 16 2014, 02:49 PM |
Professional Amateur Group: ADMINISTRATOR Posts: 10,568 Member No.: 30,882 Joined: Oct 16th 2008 Location: Laniakea | Google's bot trawls the site to keep cached versions, I believe. I assume the other bots do the same, but I don't use any other search engine. |
Proud Contributor of the Music Section Revival Project |
Perry | Posted: Dec 16 2014, 06:10 PM |
Like an eagle! Group: SITE OWNER Posts: 8,014 Member No.: 1 Joined: Sep 15th 2002 Location: San Leandro, California | I think GoogleBot and Baidu are the only ones doing the caching. The other bots are simply crawling for links and calculating relevance on their own search engine index. |
Proud Contributor of the Music Section Revival Project |
Falbere | Posted: Dec 16 2014, 06:13 PM | ||
Back from the dead, baby Group: IDP Admin Posts: 1,327 Member No.: 43,254 Joined: Mar 31st 2014 Location: Singapore |
Interesting. But there is only like 1 bot that is not baidu or google. | ||
tsukikomi | Posted: Dec 16 2014, 08:41 PM | ||
Falbury~<3 Group: TRAP CLUB Posts: 708 Member No.: 43,994 Joined: Jun 23rd 2014 Location: Southcenter Parkway |
Why do we need them? | ||
THE_HONDA_CG2 | Posted: Dec 16 2014, 09:04 PM |
Patient Zero Group: Advanced Members Posts: 4,279 Member No.: 37,947 Joined: Oct 1st 2011 Location: Update Profile | THEY'RE SPYING ON US AND WHAT WE DO. Duh. |
kyonpalm | Posted: Dec 16 2014, 09:09 PM | ||
Professional Amateur Group: ADMINISTRATOR Posts: 10,568 Member No.: 30,882 Joined: Oct 16th 2008 Location: Laniakea |
Gear up. | ||
Proud Contributor of the Music Section Revival Project |
Nomake Wan | Posted: Dec 17 2014, 10:16 PM | ||
ShiMACHaze Group: Advanced Members Posts: 19,542 Member No.: 5,394 Joined: Feb 5th 2005 Location: Drydock |
This page was not cached due to the site's robots.txt. | ||
Proud Contributor of IDW Forums and the Music Section Revival Project |
tsukikomi | Posted: Dec 17 2014, 10:30 PM | ||
Falbury~<3 Group: TRAP CLUB Posts: 708 Member No.: 43,994 Joined: Jun 23rd 2014 Location: Southcenter Parkway |
Thats good to hear | ||
Falbere | Posted: Dec 17 2014, 11:40 PM | ||
Back from the dead, baby Group: IDP Admin Posts: 1,327 Member No.: 43,254 Joined: Mar 31st 2014 Location: Singapore |
But robots.txt can't really stop bots right? Like if the maker of the bot decide to not follow robots.txt. | ||
Spaz | Posted: Dec 18 2014, 09:42 AM |
Just a guy towing a car across the country to chase a dream. Group: FORUM MODERATOR Posts: 9,272 Member No.: 30,193 Joined: Jul 25th 2008 Location: Plymouth, MN | ITT: All of the tinfoils. |
tsukikomi | Posted: Dec 18 2014, 10:37 AM | ||
Falbury~<3 Group: TRAP CLUB Posts: 708 Member No.: 43,994 Joined: Jun 23rd 2014 Location: Southcenter Parkway |
Its official, botnets are gaining free will. | ||
Nomake Wan | Posted: Dec 19 2014, 09:42 PM | ||||
ShiMACHaze Group: Advanced Members Posts: 19,542 Member No.: 5,394 Joined: Feb 5th 2005 Location: Drydock |
Do you have access to the robots.txt file? I was just answering the question with the response--that is, what happens to search results when crawlers are refused access. I have no idea about whether this page is cached or not, and neither do you. If I had to guess, though? This is a public part of the forum so it's probably cached. | ||||
Proud Contributor of IDW Forums and the Music Section Revival Project |
Falbere | Posted: Dec 19 2014, 11:30 PM | ||
Back from the dead, baby Group: IDP Admin Posts: 1,327 Member No.: 43,254 Joined: Mar 31st 2014 Location: Singapore |
robots.txt So this page can totally be cached This post has been edited by Falbere on Dec 19 2014, 11:31 PM | ||
Spaz | Posted: Dec 20 2014, 06:52 AM | ||
Just a guy towing a car across the country to chase a dream. Group: FORUM MODERATOR Posts: 9,272 Member No.: 30,193 Joined: Jul 25th 2008 Location: Plymouth, MN |
Better delete your posts in here or they'll come after you first for disapproving. | ||
Falbere | Posted: Dec 20 2014, 07:37 AM | ||
Back from the dead, baby Group: IDP Admin Posts: 1,327 Member No.: 43,254 Joined: Mar 31st 2014 Location: Singapore |
"OH SHI-!" *gets censored* | ||
tsukikomi | Posted: Dec 20 2014, 09:51 AM | ||
Falbury~<3 Group: TRAP CLUB Posts: 708 Member No.: 43,994 Joined: Jun 23rd 2014 Location: Southcenter Parkway |
Too late they already contacted potential 3rd party buyers for your information/data. | ||
Shirogane | Posted: Dec 20 2014, 12:07 PM | ||
SCREEEEEEECHING INTENSIFIES Group: Advanced Members Posts: 5,595 Member No.: 17,722 Joined: May 10th 2006 Location: Washington |
Am I doing it right? | ||
Spaz | Posted: Dec 21 2014, 11:37 AM |
Just a guy towing a car across the country to chase a dream. Group: FORUM MODERATOR Posts: 9,272 Member No.: 30,193 Joined: Jul 25th 2008 Location: Plymouth, MN | The long and short of it is that if the unintentional distribution of the basic, harmless info you give to a web forum bothers you, you have more impactful issues in your life than said info distribution and really need to see somebody about them. This post has been edited by Spaz on Dec 21 2014, 11:38 AM |