服务器禁用爬虫
2022-09-26 17:05 By
"Powerless"
2697
0
3
【Nginx禁止爬虫访问的方法】
if ($http_user_agent ~* "Scrapy|Baiduspider|Curl|HttpClient|Bytespider|FeedDemon|JikeSpider|Indy Library|Alexa Toolbar|AskTbFXTV|AhrefsBot|CrawlDaddy|CoolpadWebkit|Java|Feedly|UniversalFeedParser
|ApacheBench|Microsoft URL Control|Swiftbot|ZmEu|oBot|jaunty|Python-urllib|lightDeckReports Bot|YYSpider|DigExt|YisouSpider|HttpClient|MJ12bot|heritrix|EasouSp
ider|Ezooms|^$"){
return 403;
}
如需跳转其他页面，只需要吧return 403 换成对于的地址即可，配置如下:
if ($http_user_agent ~* "Scrapy|Baiduspider|Curl|HttpClient|Bytespider|FeedDemon|JikeSpider|Indy Library|Alexa Toolbar|AskTbFXTV|AhrefsBot|CrawlDaddy|CoolpadWebkit|Java|Feedly|UniversalFeedParser
|ApacheBench|Microsoft URL Control|Swiftbot|ZmEu|oBot|jaunty|Python-urllib|lightDeckReports Bot|YYSpider|DigExt|YisouSpider|HttpClient|MJ12bot|heritrix|EasouSp
ider|Ezooms|^$") {
return 301 https://yoursite.com;
}
如需禁止特定来源用户，配置如下:
if ($http_referer ~ "baidu\.com|google\.net|bing\.com") {
return 403;
}
如需仅允许GET,HEAD和POST请求，配置如下:
#fbrbidden not GET|HEAD|POST method access
if ($request_method !~ ^(GET|HEAD|POST)$) {
return 403;
}
【Apache禁用爬虫的配置】
mod_rewrite模块确定开启的前提下，在.htaccess文件或者相应的.conf文件,添加以下内容:
RewriteEngine On
RewriteCond %{HTTP_USER_AGENT} (^$|FeedDemon|Indy Library|Alexa Toolbar|AskTbFXTV|AhrefsBot|CrawlDaddy|CoolpadWebkit|Java|Feedly|UniversalFeedParser|ApacheBench|Microsoft URL Control|Swiftbot|ZmEu|oBot|jaunty|Python-urllib|lightDeckReports Bot|YYSpider|DigExt|HttpClient|MJ12bot|heritrix|EasouSpider|Ezooms) [NC]
RewriteRule . - [R=403,L]
View in WeChat
Others Discussion
-
初识七层、五层、四层网络协议 Posted on 2021-04-09 16:52
-
投票通过，PHP 8 确认引入 Union Types 2.0 Posted on 2019-11-18 22:22
-
Linux工具 - NM目标文件格式分析 Posted on 2019-04-24 10:29
-
PHP扩展安装 Posted on 2019-06-24 11:28
-
Redis各种数据类型的使用场景举例分析【三】 Posted on 2018-11-22 17:00
-
ACID原则 Posted on 2020-12-17 16:36
-
PHP7不兼容性 Posted on 2018-03-07 15:59
-
MySQL分组 Posted on 2019-11-18 14:00
评 论