Initial D World - Discussion Board / Forums
   
Welcome Guest ( Log In | Register )Resend Validation Email

DJ Panel ( Server Stats )   Song History   Initial D World Chat Room (IRC)   Broadband Stream
RADIO BROADCAST » streaming at 96kbps with 13 unique listeners, playing Yuzo Koshiro - There is No Way Out

       

 

Views: 1,595  ·  Replies: 18 
> Always been meaning ask, About the search engine bots
tsukikomi
  Posted: Dec 16 2014, 01:13 PM


Falbury~<3
Group Icon

Group: TRAP CLUB
Posts: 708
Member No.: 43,994
Joined: Jun 23rd 2014
Location: Southcenter Parkway





Whats is there purpose here exactly? Is it to collect info on us to sell to 3rd parties?
kyonpalm
Posted: Dec 16 2014, 02:49 PM


Professional Amateur
Group Icon

Group: ADMINISTRATOR
Posts: 10,396
Member No.: 30,882
Joined: Oct 16th 2008
Location: Laniakea





Google's bot trawls the site to keep cached versions, I believe. I assume the other bots do the same, but I don't use any other search engine.
Proud Contributor of the Music Section Revival Project
Perry
Posted: Dec 16 2014, 06:10 PM


Like an eagle!
Group Icon

Group: SITE OWNER
Posts: 7,947
Member No.: 1
Joined: Sep 15th 2002
Location: San Leandro, California





I think GoogleBot and Baidu are the only ones doing the caching. The other bots are simply crawling for links and calculating relevance on their own search engine index.
Proud Contributor of the Music Section Revival Project
Falbere
Posted: Dec 16 2014, 06:13 PM


osu★noob
Group Icon

Group: TRAP CLUB
Posts: 1,271
Member No.: 43,254
Joined: Mar 31st 2014
Location: Singaporu!





QUOTE (Perry @ 3 minutes, 24 seconds ago)
I think GoogleBot and Baidu are the only ones doing the caching. The other bots are simply crawling for links and calculating relevance on their own search engine index.

Interesting. But there is only like 1 bot that is not baidu or google.
tsukikomi
  Posted: Dec 16 2014, 08:41 PM


Falbury~<3
Group Icon

Group: TRAP CLUB
Posts: 708
Member No.: 43,994
Joined: Jun 23rd 2014
Location: Southcenter Parkway





QUOTE (Perry @ 2 hours, 30 minutes ago)
I think GoogleBot and Baidu are the only ones doing the caching. The other bots are simply crawling for links and calculating relevance on their own search engine index.

Why do we need them?
THE_HONDA_CG2
Posted: Dec 16 2014, 09:04 PM


Patient Zero
**********

Group: Advanced Members
Posts: 4,272
Member No.: 37,947
Joined: Oct 1st 2011
Location: Update Profile





THEY'RE SPYING ON US AND WHAT WE DO. Duh. derp.gif
kyonpalm
Posted: Dec 16 2014, 09:09 PM


Professional Amateur
Group Icon

Group: ADMINISTRATOR
Posts: 10,396
Member No.: 30,882
Joined: Oct 16th 2008
Location: Laniakea





QUOTE (THE_HONDA_CG2 @ 4 minutes, 22 seconds ago)
THEY'RE SPYING ON US AND WHAT WE DO.

user posted image

Gear up.
Proud Contributor of the Music Section Revival Project
Nomake Wan
Posted: Dec 17 2014, 10:16 PM


ShiMACHaze
**********

Group: Advanced Members
Posts: 19,016
Member No.: 5,394
Joined: Feb 5th 2005
Location: Drydock





QUOTE (THE_HONDA_CG2 @ Yesterday, 10:04 PM)
THEY'RE SPYING ON US AND WHAT WE DO. Duh. derp.gif

This page was not cached due to the site's robots.txt.
Proud Contributor of the Music Section Revival Project
tsukikomi
  Posted: Dec 17 2014, 10:30 PM


Falbury~<3
Group Icon

Group: TRAP CLUB
Posts: 708
Member No.: 43,994
Joined: Jun 23rd 2014
Location: Southcenter Parkway





QUOTE (Nomake Wan @ 13 minutes, 54 seconds ago)
This page was not cached due to the site's robots.txt.

Thats good to hear unsure.gif
Falbere
Posted: Dec 17 2014, 11:40 PM


osu★noob
Group Icon

Group: TRAP CLUB
Posts: 1,271
Member No.: 43,254
Joined: Mar 31st 2014
Location: Singaporu!





QUOTE (Nomake Wan @ 1 hour, 23 minutes ago)
This page was not cached due to the site's robots.txt.

But robots.txt can't really stop bots right? Like if the maker of the bot decide to not follow robots.txt.
Spaz
Posted: Dec 18 2014, 09:42 AM


I just wanna go fast
Group Icon

Group: FORUM MODERATOR
Posts: 9,141
Member No.: 30,193
Joined: Jul 25th 2008
Location: Plymouth, MN





ITT: All of the tinfoils.
tsukikomi
  Posted: Dec 18 2014, 10:37 AM


Falbury~<3
Group Icon

Group: TRAP CLUB
Posts: 708
Member No.: 43,994
Joined: Jun 23rd 2014
Location: Southcenter Parkway





QUOTE (Falbere @ Yesterday, 11:40 PM)
But robots.txt can't really stop bots right? Like if the maker of the bot decide to not follow robots.txt.

Its official, botnets are gaining free will.
Nomake Wan
Posted: Dec 19 2014, 09:42 PM


ShiMACHaze
**********

Group: Advanced Members
Posts: 19,016
Member No.: 5,394
Joined: Feb 5th 2005
Location: Drydock





QUOTE (tsukikomi @ Dec 17 2014, 11:30 PM)
Thats good to hear unsure.gif

QUOTE (Falbere @ Yesterday, 12:40 AM)
But robots.txt can't really stop bots right? Like if the maker of the bot decide to not follow robots.txt.

Do you have access to the robots.txt file? I was just answering the question with the response--that is, what happens to search results when crawlers are refused access.

I have no idea about whether this page is cached or not, and neither do you. If I had to guess, though? This is a public part of the forum so it's probably cached.
Proud Contributor of the Music Section Revival Project
Falbere
Posted: Dec 19 2014, 11:30 PM


osu★noob
Group Icon

Group: TRAP CLUB
Posts: 1,271
Member No.: 43,254
Joined: Mar 31st 2014
Location: Singaporu!





QUOTE (Nomake Wan @ 1 hour, 47 minutes ago)
Do you have access to the robots.txt file? I was just answering the question with the response--that is, what happens to search results when crawlers are refused access.

I have no idea about whether this page is cached or not, and neither do you. If I had to guess, though? This is a public part of the forum so it's probably cached.

robots.txt
So this page can totally be cached

This post has been edited by Falbere on Dec 19 2014, 11:31 PM
Spaz
Posted: Dec 20 2014, 06:52 AM


I just wanna go fast
Group Icon

Group: FORUM MODERATOR
Posts: 9,141
Member No.: 30,193
Joined: Jul 25th 2008
Location: Plymouth, MN





QUOTE (Falbere @ 7 hours, 22 minutes ago)
robots.txt
So this page can totally be cached

Better delete your posts in here or they'll come after you first for disapproving.
Falbere
Posted: Dec 20 2014, 07:37 AM


osu★noob
Group Icon

Group: TRAP CLUB
Posts: 1,271
Member No.: 43,254
Joined: Mar 31st 2014
Location: Singaporu!





QUOTE (Spaz @ 44 minutes, 26 seconds ago)
Better delete your posts in here or they'll come after you first for disapproving.

"OH SHI-!"
*gets censored*
tsukikomi
  Posted: Dec 20 2014, 09:51 AM


Falbury~<3
Group Icon

Group: TRAP CLUB
Posts: 708
Member No.: 43,994
Joined: Jun 23rd 2014
Location: Southcenter Parkway





QUOTE (Spaz @ 2 hours, 58 minutes ago)
Better delete your posts in here or they'll come after you first for disapproving.

Too late they already contacted potential 3rd party buyers for your information/data.
Shirogane
Posted: Dec 20 2014, 12:07 PM


SCREEEEEEECHING INTENSIFIES
**********

Group: Advanced Members
Posts: 5,589
Member No.: 17,722
Joined: May 10th 2006
Location: Washington





QUOTE (kyonpalm @ Dec 16 2014, 10:09 PM)
http://i.minus.com/ibz21ClXUxDFHd.jpg

Gear up.

user posted image
Am I doing it right?
Spaz
Posted: Dec 21 2014, 11:37 AM


I just wanna go fast
Group Icon

Group: FORUM MODERATOR
Posts: 9,141
Member No.: 30,193
Joined: Jul 25th 2008
Location: Plymouth, MN





The long and short of it is that if the unintentional distribution of the basic, harmless info you give to a web forum bothers you, you have more impactful issues in your life than said info distribution and really need to see somebody about them.

This post has been edited by Spaz on Dec 21 2014, 11:38 AM