Initial D World - Discussion Board / Forums
   
Welcome Guest ( Log In | Register )Resend Validation Email

DJ Panel ( Server Stats )   Song History   Initial D World Chat Room (Discord)   Broadband Stream
RADIO BROADCAST » streaming at 96kbps with 5 unique listeners, playing (Battle Stage 2) Mr. M - Welcome People

       

 

Views: 9,025  ·  Replies: 18 
> Always been meaning ask, About the search engine bots
tsukikomi
  Posted: Dec 16 2014, 01:13 PM


Falbury~<3
Group Icon

Group: TRAP CLUB
Posts: 708
Member No.: 43,994
Joined: Jun 23rd 2014
Location: Southcenter Parkway





Whats is there purpose here exactly? Is it to collect info on us to sell to 3rd parties?
kyonpalm
Posted: Dec 16 2014, 02:49 PM


Professional Amateur
Group Icon

Group: ADMINISTRATOR
Posts: 10,568
Member No.: 30,882
Joined: Oct 16th 2008
Location: Laniakea





Google's bot trawls the site to keep cached versions, I believe. I assume the other bots do the same, but I don't use any other search engine.
Proud Contributor of the Music Section Revival Project
Perry
Posted: Dec 16 2014, 06:10 PM


Like an eagle!
Group Icon

Group: SITE OWNER
Posts: 8,014
Member No.: 1
Joined: Sep 15th 2002
Location: San Leandro, California





I think GoogleBot and Baidu are the only ones doing the caching. The other bots are simply crawling for links and calculating relevance on their own search engine index.
Proud Contributor of the Music Section Revival Project
Falbere
Posted: Dec 16 2014, 06:13 PM


Back from the dead, baby
Group Icon

Group: IDP Admin
Posts: 1,327
Member No.: 43,254
Joined: Mar 31st 2014
Location: Singapore





QUOTE (Perry @ 3 minutes, 24 seconds ago)
I think GoogleBot and Baidu are the only ones doing the caching. The other bots are simply crawling for links and calculating relevance on their own search engine index.

Interesting. But there is only like 1 bot that is not baidu or google.
Proud Contributor of Initial D World Forums
tsukikomi
  Posted: Dec 16 2014, 08:41 PM


Falbury~<3
Group Icon

Group: TRAP CLUB
Posts: 708
Member No.: 43,994
Joined: Jun 23rd 2014
Location: Southcenter Parkway





QUOTE (Perry @ 2 hours, 30 minutes ago)
I think GoogleBot and Baidu are the only ones doing the caching. The other bots are simply crawling for links and calculating relevance on their own search engine index.

Why do we need them?
THE_HONDA_CG2
Posted: Dec 16 2014, 09:04 PM


Patient Zero
**********

Group: Advanced Members
Posts: 4,279
Member No.: 37,947
Joined: Oct 1st 2011
Location: Update Profile





THEY'RE SPYING ON US AND WHAT WE DO. Duh. derp.gif
kyonpalm
Posted: Dec 16 2014, 09:09 PM


Professional Amateur
Group Icon

Group: ADMINISTRATOR
Posts: 10,568
Member No.: 30,882
Joined: Oct 16th 2008
Location: Laniakea





QUOTE (THE_HONDA_CG2 @ 4 minutes, 22 seconds ago)
THEY'RE SPYING ON US AND WHAT WE DO.

user posted image

Gear up.
Proud Contributor of the Music Section Revival Project
Nomake Wan
Posted: Dec 17 2014, 10:16 PM


ShiMACHaze
**********

Group: Advanced Members
Posts: 19,542
Member No.: 5,394
Joined: Feb 5th 2005
Location: Drydock





QUOTE (THE_HONDA_CG2 @ Yesterday, 10:04 PM)
THEY'RE SPYING ON US AND WHAT WE DO. Duh. derp.gif

This page was not cached due to the site's robots.txt.
Proud Contributor of IDW Forums and the Music Section Revival Project
tsukikomi
  Posted: Dec 17 2014, 10:30 PM


Falbury~<3
Group Icon

Group: TRAP CLUB
Posts: 708
Member No.: 43,994
Joined: Jun 23rd 2014
Location: Southcenter Parkway





QUOTE (Nomake Wan @ 13 minutes, 54 seconds ago)
This page was not cached due to the site's robots.txt.

Thats good to hear unsure.gif
Falbere
Posted: Dec 17 2014, 11:40 PM


Back from the dead, baby
Group Icon

Group: IDP Admin
Posts: 1,327
Member No.: 43,254
Joined: Mar 31st 2014
Location: Singapore





QUOTE (Nomake Wan @ 1 hour, 23 minutes ago)
This page was not cached due to the site's robots.txt.

But robots.txt can't really stop bots right? Like if the maker of the bot decide to not follow robots.txt.
Proud Contributor of Initial D World Forums
Spaz
Posted: Dec 18 2014, 09:42 AM


Just a guy towing a car across the country to chase a dream.
Group Icon

Group: FORUM MODERATOR
Posts: 9,272
Member No.: 30,193
Joined: Jul 25th 2008
Location: Plymouth, MN





ITT: All of the tinfoils.
Proud Contributor of Initial D World Forums
tsukikomi
  Posted: Dec 18 2014, 10:37 AM


Falbury~<3
Group Icon

Group: TRAP CLUB
Posts: 708
Member No.: 43,994
Joined: Jun 23rd 2014
Location: Southcenter Parkway





QUOTE (Falbere @ Yesterday, 11:40 PM)
But robots.txt can't really stop bots right? Like if the maker of the bot decide to not follow robots.txt.

Its official, botnets are gaining free will.
Nomake Wan
Posted: Dec 19 2014, 09:42 PM


ShiMACHaze
**********

Group: Advanced Members
Posts: 19,542
Member No.: 5,394
Joined: Feb 5th 2005
Location: Drydock





QUOTE (tsukikomi @ Dec 17 2014, 11:30 PM)
Thats good to hear unsure.gif

QUOTE (Falbere @ Yesterday, 12:40 AM)
But robots.txt can't really stop bots right? Like if the maker of the bot decide to not follow robots.txt.

Do you have access to the robots.txt file? I was just answering the question with the response--that is, what happens to search results when crawlers are refused access.

I have no idea about whether this page is cached or not, and neither do you. If I had to guess, though? This is a public part of the forum so it's probably cached.
Proud Contributor of IDW Forums and the Music Section Revival Project
Falbere
Posted: Dec 19 2014, 11:30 PM


Back from the dead, baby
Group Icon

Group: IDP Admin
Posts: 1,327
Member No.: 43,254
Joined: Mar 31st 2014
Location: Singapore





QUOTE (Nomake Wan @ 1 hour, 47 minutes ago)
Do you have access to the robots.txt file? I was just answering the question with the response--that is, what happens to search results when crawlers are refused access.

I have no idea about whether this page is cached or not, and neither do you. If I had to guess, though? This is a public part of the forum so it's probably cached.

robots.txt
So this page can totally be cached

This post has been edited by Falbere on Dec 19 2014, 11:31 PM
Proud Contributor of Initial D World Forums
Spaz
Posted: Dec 20 2014, 06:52 AM


Just a guy towing a car across the country to chase a dream.
Group Icon

Group: FORUM MODERATOR
Posts: 9,272
Member No.: 30,193
Joined: Jul 25th 2008
Location: Plymouth, MN





QUOTE (Falbere @ 7 hours, 22 minutes ago)
robots.txt
So this page can totally be cached

Better delete your posts in here or they'll come after you first for disapproving.
Proud Contributor of Initial D World Forums
Falbere
Posted: Dec 20 2014, 07:37 AM


Back from the dead, baby
Group Icon

Group: IDP Admin
Posts: 1,327
Member No.: 43,254
Joined: Mar 31st 2014
Location: Singapore





QUOTE (Spaz @ 44 minutes, 26 seconds ago)
Better delete your posts in here or they'll come after you first for disapproving.

"OH SHI-!"
*gets censored*
Proud Contributor of Initial D World Forums
tsukikomi
  Posted: Dec 20 2014, 09:51 AM


Falbury~<3
Group Icon

Group: TRAP CLUB
Posts: 708
Member No.: 43,994
Joined: Jun 23rd 2014
Location: Southcenter Parkway





QUOTE (Spaz @ 2 hours, 58 minutes ago)
Better delete your posts in here or they'll come after you first for disapproving.

Too late they already contacted potential 3rd party buyers for your information/data.
Shirogane
Posted: Dec 20 2014, 12:07 PM


SCREEEEEEECHING INTENSIFIES
**********

Group: Advanced Members
Posts: 5,595
Member No.: 17,722
Joined: May 10th 2006
Location: Washington





QUOTE (kyonpalm @ Dec 16 2014, 10:09 PM)
http://i.minus.com/ibz21ClXUxDFHd.jpg

Gear up.

user posted image
Am I doing it right?
Spaz
Posted: Dec 21 2014, 11:37 AM


Just a guy towing a car across the country to chase a dream.
Group Icon

Group: FORUM MODERATOR
Posts: 9,272
Member No.: 30,193
Joined: Jul 25th 2008
Location: Plymouth, MN





The long and short of it is that if the unintentional distribution of the basic, harmless info you give to a web forum bothers you, you have more impactful issues in your life than said info distribution and really need to see somebody about them.

This post has been edited by Spaz on Dec 21 2014, 11:38 AM
Proud Contributor of Initial D World Forums