Basic Pentest Recon

Basic Recon on Domain/Company name to support network pentest

Overview

Below is a list of the current techniques I use to do a basic recon search for an organization prior to a network pentest. There are a number of APIs in use (All are free minus DeHashed) that you will need to register for prior to running (see API Keys). This API registration process shouldn't take more than an hour or two.

I would recommend keeping the API key file subfinder-keys.yaml in a password manager and pasting it in for each run. The below command will remove the file after subfinder runs

Recon Checks:

1. Subdomain Enumeration w/ HTTP Screenshot Scan

2. Cloud Resource Enumeration

3. Breach Credential Lookup

4. American Registry for Internet Numbers (ARIN) Data

All of the below block of code are made to copy/paste straight into a shell. If any input is needed from you, it will prompt for it or drop you into a vim editor.

Install Tools

sudo apt install subfinder miller xxd
sudo subfinder -up
subfinder -ls
go install -v github.com/projectdiscovery/httpx/cmd/httpx@latest
mv "$HOME/go/bin/httpx" "$HOME/go/bin/httpx-ng"  # Competing package names on kali
# pipx install git+https://github.com/hmaverickadams/DeHashed-API-Tool  # Changes not merged
pipx install git+https://github.com/d-woosley/DeHashed-API-Tool

Setup

command -v vared >/dev/null && vared -p 'Domain: ' -c DOMAIN || read -p "Domain: " DOMAIN
vim subfinder-keys.yaml  # Add API Keys

API Keys

Subfinder will need a number of API to get good results from subdomain searches. The ones listed below are all available via free trails that should be sufficient for most pentest use cases. You will need to create an account and get an API key for each service. There are more free APIs out there, this is just the list of APIs that I could easily identify and sign up for without paying.

Site Name	Subfinder Key Tag	URL
DNSDumpster	dnsdumpster: []	https://dnsdumpster.com/
Fullhunt	fullhunt: []	https://fullhunt.io
Hunter	hunter: []	https://hunter.io
LeakIX	leakix: []	https://leakix.net
SecurityTrails	securitytrails: []	https://securitytrails.com
SSLMate (Certspotter)	certspotter: []	https://sslmate.com
Threat Book	threatbook: []	https://i.threatbook.io/
URLScan	urlscan: []	https://urlscan.io
VirusTotal	virustotal: []	https://www.virustotal.com
WhoIsXMLAPI	whoisxmlapi: []	https://main.whoisxmlapi.com
ZoomEye	zoomeye: [] zoomeyeapi: []	https://www.zoomeye.ai
Censys	censys: []	https://accounts.censys.io
Chaos (ProjectDiscovery)	chaos: []	https://chaos.projectdiscovery.io/
GitHub (*Use burner account)	github: []	https://github.com/
Shodan	shodan: []	https://account.shodan.io/

More information about this process can be found at: https://dhiyaneshgeek.github.io/bug/bounty/2020/02/06/recon-with-me/

subfiner-keys.yaml template

censys: []
certspotter: []
chaos: []
dnsdumpster: []
fullhunt: []
github: []
hunter: []
leakix: []
securitytrails: []
shodan: []
threatbook: []
urlscan: []
virustotal: []
whoisxmlapi: []
zoomeye: []
zoomeyeapi: []

Run

Subdomain Enumeration

# Get Subdomains
subfinder -d $DOMAIN -all -pc subfinder-keys.yaml -stats -o "Subdomain-candidates-$DOMAIN.txt"
rm subfinder-keys.yaml

# Create Subdomain Wordlist
wget -q https://raw.githubusercontent.com/danielmiessler/SecLists/master/Discovery/DNS/sortedcombined-knock-dnsrecon-fierce-reconng.txt -O common-subdomain.txt
cat common-subdomain.txt "Subdomain-candidates-$DOMAIN.txt" | sort -u > possible-subdomains.txt

# DNS Subdomain Confirmation
gobuster dns --domain $DOMAIN -w possible-subdomains.txt -o DNS_Subdomain_Enumeration_Results.txt --no-color --resolver 1.1.1.1

# Convert results to CSV/TXT
cat DNS_Subdomain_Enumeration_Results.txt | awk -F " " 'NR==1 {print "\"Subdomain\",\"IP(s)\""};{print "\""$1"\",\""$2"\""}' > DNS_Subdomain_Enumeration_Results.csv
cat DNS_Subdomain_Enumeration_Results.txt | cut -d ' ' -f 1 >> Subdomains.txt

# Check for HTTP (Maybe not needed?)
httpx-ng -status-code -title -web-server -t 15 -no-fallback -follow-redirects -ports 80,8080,443,8443,4443,8888 -probe-all-ips -random-agent -o "HTTP-Server-$DOMAIN.txt" -l Subdomains.txt 

# Screenshot Scan
grep -P '(,|\[)\x1b\[32m?200\x1b\[0m?\]' "HTTP-Server-$DOMAIN.txt" | cut -d ' ' -f 1 | sort -u > gowitness-http.hosts
gowitness scan file -f gowitness-http.hosts --write-db
gowitness report generate --zip-name Subdomain-Screenshots.zip

Cloud Enumeration

echo "$DOMAIN" | cut -d '.' -f 1 > org-base-words.txt
echo "$DOMAIN" >> org-base-words.txt
vim org-base-words.txt  # Add in any extra org base words or product names
curl -s https://raw.githubusercontent.com/initstring/cloud_enum/refs/heads/master/enum_tools/fuzz.txt -o cloud_enum-mutations.txt
while read MUTATION; do while read BASE; do echo "$BASE.$MUTATION\n$BASE-$MUTATION\n$BASE$MUTATION\n$MUTATION.$BASE\n$MUTATION-$BASE\n$MUTATION$BASE"; done; done < org-base-words.txt < cloud_enum-mutations.txt > cloud-wordlist_$DOMAIN.txt
nuclei -t ~/.local/nuclei-templates/cloud/enum -esc -var wordlist="cloud-wordlist_$DOMAIN.txt" -allow-local-file-access -o Cloud-Enumeration_Results.txt -je Cloud-Enumeration_Results.json

Breached Credentials Lookup

dat --key -o Publicly_Leaked_Credential.csv --summary -d "$DOMAIN"

# Create Redacted CSV of breach creds
mlr --icsv --ocsv filter '$password != ""' \
  then cut -f email,password,database_name \
  then put '$password = substr($password,1,3) . gsub(substr($password,4,length($password)-3),".","*")' \
  then reorder -f email,password,database_name \
  Publicly_Leaked_Credential.csv > Redacted-Publicly_Leaked_Credential.csv

American Registry for Internet Numbers (ARIN) Data

# Find ARIN org via contact domain
ARIN_CONTACTS=$(curl --retry 5 --connect-timeout 3 -s "https://whois.arin.net/rest/pocs;domain=$DOMAIN.json" | jq -r '.pocs.pocRef | (if type == "array" then .[] else . end)."$"' 2> /dev/null)
ARIN_ORGS=$(while IFS= read -r CONTACT_URL; do curl --retry 5 --connect-timeout 3 -s $CONTACT_URL/orgs.json | jq -r '.orgs.orgPocLinkRef | (if type == "array" then .[] else . end)."$"' 2> /dev/null; done <<< $ARIN_CONTACTS | sort -u)

# Grab org data
ARIN_ASN_URLS=$(while IFS= read -r ORG_URL; do curl --retry 5 --connect-timeout 3 -s $ORG_URL/asns.json | jq -r '.asns.asnRef | (if type == "array" then .[] else . end)."$"' 2> /dev/null; done <<< $ARIN_ORGS | sort -u)
ARIN_POC_URLS=$(while IFS= read -r ORG_URL; do curl --retry 5 --connect-timeout 3 -s $ORG_URL/pocs.json | jq -r '.pocs.pocLinkRef | (if type == "array" then .[] else . end)."$"' 2> /dev/null; done <<< $ARIN_ORGS | sort -u)
ARIN_NET_URLS=$(while IFS= read -r ORG_URL; do curl --retry 5 --connect-timeout 3 -s $ORG_URL/nets.json | jq -r '.nets.netRef | (if type == "array" then .[] else . end)."$"' 2> /dev/null; done <<< $ARIN_ORGS | sort -u)

# Extract Org Data
ASNS=$(while IFS= read -r ASN_URL; do curl --retry 5 --connect-timeout 3 -s $ASN_URL.json | jq -r '.asn.handle["$"]' 2> /dev/null; done <<< $ARIN_ASN_URLS | sort -u)
NETBLOCKS=$(while IFS= read -r NET_URL; do curl --retry 5 --connect-timeout 3 -s $NET_URL.json | jq -r '.net.netBlocks.netBlock | "\(.startAddress["$"])/\(.cidrLength["$"])"' 2> /dev/null; done <<< $ARIN_NET_URLS | sort -u)
POC_INFO=$(while IFS= read -r POC_URL; do curl --retry 5 --connect-timeout 3 -s https://whois.arin.net/rest/poc/KWI70-ARIN.json | jq -r '.poc | (.streetAddress.line | if type=="array" then . else [.] end | map(.["$"]) | join(", ")) as $street | (.emails.email | if type=="array" then . else [.] end | map(.["$"]) | join(", ")) as $emails | (.phones.phone | if type=="array" then . else [.] end | map(.number["$"] + " (" + .type.description["$"] + ")") | join(", ")) as $phones | .firstName["$"] as $fn | .lastName["$"] as $ln | .city["$"] as $city | .["iso3166-2"]["$"] as $st | .postalCode["$"] as $zip | [$fn + " " + $ln, $street + "," + $city + ", " + $st + " " + $zip, $emails + " - " + $phones] | @csv' 2> /dev/null; done <<< $ARIN_POC_URLS  | sort -u)

# Save CSV
{ echo "Name,Address,Contact Info"; printf '%s\n' "$POC_INFO"; } > ARIN_Point_of_Contact_Information.csv

# Display Info
echo;echo;echo
echo "External Network Block(s): $(printf '%s' "$NETBLOCKS" | awk '{printf "%s%s", sep, $0; sep=", "}')"
echo "ASN(s): $(printf '%s' "$ASNS" | awk '{printf "%s%s", sep, $0; sep=", "}')"
echo
echo "Point of Contact Information"
mlr --c2t cat ARIN_Point_of_Contact_Information.csv

Package/Export Results

zip recon-artifacts.zip Redacted-Publicly_Leaked_Credential.csv Cloud-Enumeration_Results.txt Subdomain-Screenshots.zip DNS_Subdomain_Enumeration_Results.csv Subdomain-candidates-$DOMAIN.txt ARIN_Point_of_Contact_Information.csv
echo
echo
echo "Download the results at: $(realpath recon-artifacts.zip)"