HTTP Robots.txt Content Scanner
05/30/2018
Description
Detect robots.txt files and analyze its content
Author(s)
- et <et@metasploit.com>
Development
Module Options
To display the available options, load the module within the Metasploit console and run the commands 'show options' or 'show advanced':
msf > use auxiliary/scanner/http/robots_txt
msf auxiliary(robots_txt) > show actions
...actions...
msf auxiliary(robots_txt) > set ACTION < action-name >
msf auxiliary(robots_txt) > show options
...show and set options...
msf auxiliary(robots_txt) > run 
Penetration testing software for offensive security teams.
Key Features
- Collect and share all the information you need to conduct a successful and efficient penetration test
- Simulate complex attacks against your systems and users
- Test your defenses to make sure they’re ready
- Automate Every Step of Your Penetration Test
Time is precious, so I don’t want to do something manually that I can automate. Leveraging the Metasploit Framework when automating any task keeps us from having to re-create the wheel as we can use the existing libraries and focus our efforts where it matters.
– Jim O’Gorman | President, Offensive Security