| Printable Version of Topic
Click here to view this topic in its original format |
| Initial D World - Discussion Board / Forums > Feedback and Support > Always been meaning ask |
| Posted by: tsukikomi Dec 16 2014, 01:13 PM |
| Whats is there purpose here exactly? Is it to collect info on us to sell to 3rd parties? |
| Posted by: kyonpalm Dec 16 2014, 02:49 PM |
| Google's bot trawls the site to keep cached versions, I believe. I assume the other bots do the same, but I don't use any other search engine. |
| Posted by: Perry Dec 16 2014, 06:10 PM |
| I think GoogleBot and Baidu are the only ones doing the caching. The other bots are simply crawling for links and calculating relevance on their own search engine index. |
| Posted by: Falbere Dec 16 2014, 06:13 PM | ||
Interesting. But there is only like 1 bot that is not baidu or google. |
| Posted by: tsukikomi Dec 16 2014, 08:41 PM | ||
Why do we need them? |
| Posted by: THE_HONDA_CG2 Dec 16 2014, 09:04 PM |
| THEY'RE SPYING ON US AND WHAT WE DO. Duh. |
| Posted by: kyonpalm Dec 16 2014, 09:09 PM | ||
 Gear up. |
| Posted by: Nomake Wan Dec 17 2014, 10:16 PM | ||
This page was not cached due to the site's robots.txt. |
| Posted by: tsukikomi Dec 17 2014, 10:30 PM | ||
Thats good to hear  |
| Posted by: Falbere Dec 17 2014, 11:40 PM | ||
But robots.txt can't really stop bots right? Like if the maker of the bot decide to not follow robots.txt. |
| Posted by: Spaz Dec 18 2014, 09:42 AM |
| ITT: All of the tinfoils. |
| Posted by: tsukikomi Dec 18 2014, 10:37 AM | ||
Its official, botnets are gaining free will. |
| Posted by: Nomake Wan Dec 19 2014, 09:42 PM | ||||
Do you have access to the robots.txt file? I was just answering the question with the response--that is, what happens to search results when crawlers are refused access. I have no idea about whether this page is cached or not, and neither do you. If I had to guess, though? This is a public part of the forum so it's probably cached. |
| Posted by: Falbere Dec 19 2014, 11:30 PM | ||
https://idforums.net/robots.txt So this page can totally be cached |
| Posted by: Spaz Dec 20 2014, 06:52 AM | ||
Better delete your posts in here or they'll come after you first for disapproving. |
| Posted by: Falbere Dec 20 2014, 07:37 AM | ||
"OH SHI-!" *gets censored* |
| Posted by: tsukikomi Dec 20 2014, 09:51 AM | ||
Too late they already contacted potential 3rd party buyers for your information/data. |
| Posted by: Shirogane Dec 20 2014, 12:07 PM | ||
 Am I doing it right? |
| Posted by: Spaz Dec 21 2014, 11:37 AM |
| The long and short of it is that if the unintentional distribution of the basic, harmless info you give to a web forum bothers you, you have more impactful issues in your life than said info distribution and really need to see somebody about them. |