Beware Evil Google Bots
How Google bots act as an attack vector
Just when you thought cookies were bad, now we have evil Bots !! .[Spoiler (click to open)].Google is Good & Evil !
Each search engine has its unique set of algorithms, but they all work in a similar way: they visit a given website, look at the content and links they find (this is called crawling), and then grade and list the resources. After one of these bots finds your website, it will visit and index it. For a good ranking, you need to make sure that search engine bots can crawl your website without issues. Google specifically recommends that you avoid blocking search bots in order to achieve successful indexing. Attackers are aware of these permissions
and have developed an interesting technique to exploit them: abusing Google bots.
When F5 Labs inspected the CroniX cryptominer. When F5 Labs researchers analyzed some malicious requests they had logged, they discovered that the requests originated from Google bots, they discovered that the requests were indeed sent from Google. Google bots were abused in order to make them behave maliciously.
Herein lie your concerns:
• How do I allow the good bots into my life?
• Can invisible reCAPTCHA defend against headless bots such as phantom.js or selenium?
• At some point in the future (if not already), Google will maintain a blacklist of IPs once they’ve been flagged.
• How often will this be cleaned up?
• How do I whitelist my tools for scanning my application? (Rapid7, Qualys, Whitehat, Qanon, etc)
• By embedding this javascript into my application, I get protection, but what PII data do I surrender?
• How do I troubleshoot false positives?
Incapsula showed that more than half of internet traffic is bots - some good (like Google’s search crawlers and monitoring tools) and some nefarious (like scrapers and malware).
1. VPN (Hide my ass) and Proxies (preferably Residentials).
2. Random user agents.
3. Random Screen sizes.
4. Unique browser fingerprint on every request
5. Recaptcha API (2captcha integrated).
6. many more…
How did the Google bots turn evil?
When Google bots encounter this URL, they’ll visit it in order to index it. The request that includes the payload will be made by a Google bot. This scenario is only possible in GET requests where the payload can be sent through the URL.
Abusing Googlebot Services to Deliver Crypto-Mining Malware both our friendly and less friendly bots tend to report as ‘standard’ agents, i.e., 1024×768 running on Windows and IE.
If you can make sense of this cryptic programming, be my guest. Sounds malicious to me..
https://www.invicti.com/blog/web-security/using-google-bots-attack-vector/
Sven Morgenroth - Senior Security Engineer
Google Adwords Bot - SCRAPESCREWWITHBOTS : https://scrapewithbots.com/product/google-adwords-bot/
Youtube Playlist Downloader Bot : https://scrapewithbots.com/product/youtube-playlist-downloader/
Mining Your Data - 6 Ways SEO Pros Are Using ChatGPT Right Now :
https://www.searchenginejournal.com/ways-seos-are-using-chatgpt-right-now/475896/
So, Vaaht da fock?
.
Internet Bots are Clouding Insight from Web Metrics
Google Adwords-Bots and users beware, Google’s reCAPTCHA
Bots are insidious on the web so filter them out.
dr. π (pi)
.
.
Just when you thought cookies were bad, now we have evil Bots !! .[Spoiler (click to open)].Google is Good & Evil !
Each search engine has its unique set of algorithms, but they all work in a similar way: they visit a given website, look at the content and links they find (this is called crawling), and then grade and list the resources. After one of these bots finds your website, it will visit and index it. For a good ranking, you need to make sure that search engine bots can crawl your website without issues. Google specifically recommends that you avoid blocking search bots in order to achieve successful indexing. Attackers are aware of these permissions
and have developed an interesting technique to exploit them: abusing Google bots.
When F5 Labs inspected the CroniX cryptominer. When F5 Labs researchers analyzed some malicious requests they had logged, they discovered that the requests originated from Google bots, they discovered that the requests were indeed sent from Google. Google bots were abused in order to make them behave maliciously.
Herein lie your concerns:
• How do I allow the good bots into my life?
• Can invisible reCAPTCHA defend against headless bots such as phantom.js or selenium?
• At some point in the future (if not already), Google will maintain a blacklist of IPs once they’ve been flagged.
• How often will this be cleaned up?
• How do I whitelist my tools for scanning my application? (Rapid7, Qualys, Whitehat, Qanon, etc)
• By embedding this javascript into my application, I get protection, but what PII data do I surrender?
• How do I troubleshoot false positives?
Incapsula showed that more than half of internet traffic is bots - some good (like Google’s search crawlers and monitoring tools) and some nefarious (like scrapers and malware).
1. VPN (Hide my ass) and Proxies (preferably Residentials).
2. Random user agents.
3. Random Screen sizes.
4. Unique browser fingerprint on every request
5. Recaptcha API (2captcha integrated).
6. many more…
How did the Google bots turn evil?
When Google bots encounter this URL, they’ll visit it in order to index it. The request that includes the payload will be made by a Google bot. This scenario is only possible in GET requests where the payload can be sent through the URL.
Abusing Googlebot Services to Deliver Crypto-Mining Malware both our friendly and less friendly bots tend to report as ‘standard’ agents, i.e., 1024×768 running on Windows and IE.
If you can make sense of this cryptic programming, be my guest. Sounds malicious to me..
https://www.invicti.com/blog/web-security/using-google-bots-attack-vector/
Sven Morgenroth - Senior Security Engineer
Google Adwords Bot - SCRAPESCREWWITHBOTS : https://scrapewithbots.com/product/google-adwords-bot/
Youtube Playlist Downloader Bot : https://scrapewithbots.com/product/youtube-playlist-downloader/
Mining Your Data - 6 Ways SEO Pros Are Using ChatGPT Right Now :
https://www.searchenginejournal.com/ways-seos-are-using-chatgpt-right-now/475896/
So, Vaaht da fock?
.
Internet Bots are Clouding Insight from Web Metrics
Google Adwords-Bots and users beware, Google’s reCAPTCHA
Bots are insidious on the web so filter them out.
dr. π (pi)
.
.