OSINT and Reconnaissance for Web Application Penetration Testing

Reconnaissance is the phase that separates efficient penetration tests from scattered ones. Before touching a target application directly, experienced testers spend significant time building a detailed picture from publicly available information — and that picture shapes every subsequent testing decision.

This guide covers the passive and semi-passive reconnaissance techniques used in professional web application penetration tests, the tools that make them tractable at scale, and the indicators that signal a promising attack surface.

Passive vs. Active Reconnaissance

The distinction matters for rules of engagement:

• Passive reconnaissance — gathering information from sources that do not require direct contact with the target. Uses public data: search engines, certificate logs, WHOIS records, DNS databases, social media. Leaves no trace on target systems.
• Semi-passive reconnaissance — light interaction that mimics normal user traffic. Visiting the application, inspecting client-side assets, analyzing HTTP responses. Would appear in web server logs as a normal request.
• Active reconnaissance — direct probing: port scanning, service fingerprinting, directory brute-forcing. Clearly visible in server logs and WAF alerting. Requires explicit scope authorization.

Most engagements permit all three categories within the defined scope. Black-box engagements that restrict active recon to simulate a real attacker still permit comprehensive passive and semi-passive techniques — which often yield more information than aggressive scanning.

Scope Mapping: What Are You Actually Testing?

Before any tool runs, build an asset inventory. Engagements often define scope as "the application at app.company.com" — but the real attack surface is wider.

Domain and Subdomain Enumeration

Start by mapping all hostnames associated with the target organization. Certificate Transparency (CT) logs are the most reliable public source:

# Certificate Transparency log search
# crt.sh — free, comprehensive CT log aggregation
curl -s "https://crt.sh/?q=%.example.com&output=json" | \
  jq -r '.[].name_value' | sort -u

# subfinder — multi-source passive subdomain enumeration
subfinder -d example.com -all -silent

# amass — OWASP project, multi-source enumeration
amass enum -passive -d example.com

Common subdomain patterns that expand attack surface:

• api., api-v2., api-staging. — API backends that may lack frontend-level security controls
• admin., internal., ops. — administrative interfaces; often less hardened than public-facing apps
• staging., dev., qa., uat. — pre-production environments frequently using production data or relaxed security configs
• mail., smtp., mx. — mail infrastructure; email headers may reveal internal hostnames or cloud provider details
• vpn., remote., citrix., portal. — remote access infrastructure
• cdn., assets., static., media. — CDN origins that might expose storage bucket misconfigurations

DNS Intelligence

DNS records reveal infrastructure choices and historical IP associations:

# A records — current IP addresses
dig A example.com +short

# MX records — email provider (reveals cloud/SaaS posture)
dig MX example.com +short

# TXT records — SPF, DKIM, DMARC, domain verification tokens
# SPF record reveals mail relay infrastructure
dig TXT example.com +short

# NS records — DNS provider and potential zone transfer target
dig NS example.com +short

# Attempt zone transfer (usually blocked, sometimes not)
dig axfr @ns1.example.com example.com

# Historical DNS — resolves previous IP addresses
# Useful when current IP is behind CDN/WAF
# Tools: SecurityTrails, PassiveDNS, Shodan

Historical DNS is particularly useful when the current IP resolves to a CDN like Cloudflare. The origin IP — what requests hit when bypassing the CDN — is often visible in historical DNS records. Direct-to-origin requests bypass WAF controls and rate limiting.

Google Dorks: Search Engine Reconnaissance

Search engine operators expose indexed content that shouldn't be public. Effective Google dorking against a target:

Finding Exposed Files and Directories

site:example.com filetype:pdf
site:example.com filetype:xlsx OR filetype:csv
site:example.com filetype:env OR filetype:log OR filetype:bak
site:example.com intitle:"index of"
site:example.com inurl:"/wp-content/uploads/"
site:example.com inurl:"/phpinfo.php"

Discovering Administrative Interfaces

site:example.com inurl:admin OR inurl:administrator OR inurl:wp-admin
site:example.com inurl:login OR inurl:signin OR inurl:auth
site:example.com intitle:"dashboard" OR intitle:"control panel"
site:example.com inurl:"/api/" filetype:json

Finding Error Pages and Stack Traces

site:example.com intext:"Warning: mysql_"
site:example.com intext:"syntax error" OR intext:"stack trace"
site:example.com intext:"Connection refused" OR intext:"SQLSTATE"
site:example.com intext:"java.lang.NullPointerException"

Credential and Secret Discovery

site:example.com intext:"password" filetype:txt
site:example.com intext:"api_key" OR intext:"apikey"
site:example.com intext:"BEGIN RSA PRIVATE KEY"

# GitHub search for org-related secrets
org:company-name "example.com" AND "password"
org:company-name ".env" extension:env

Certificate Transparency Analysis

Every public TLS certificate is logged in certificate transparency logs — a public audit trail of SSL/TLS certificate issuance. CT log search is the most reliable source of subdomain enumeration because it captures what was actually deployed, not just what's currently in DNS.

# crt.sh search for all wildcard and specific certs
curl -s "https://crt.sh/?q=example.com&output=json" | \
  jq -r '.[].name_value' | \
  sed 's/\*\.//g' | sort -u | grep -v "^$"

# Filter for interesting subdomain patterns
curl -s "https://crt.sh/?q=example.com&output=json" | \
  jq -r '.[].name_value' | \
  grep -E "(dev|staging|api|admin|internal|test)"

CT logs reveal subdomains that have been active at any point in the domain's history — including decommissioned services that may still have dangling DNS records (a subdomain takeover opportunity) or infrastructure that was quietly removed from production but still exists.

Shodan: Internet-Wide Device and Service Scanning

Shodan indexes public-facing internet infrastructure. For web application targets:

# Search by hostname
hostname:example.com

# Search by organization
org:"Company Name"

# Search by network range (if you have IP ranges)
net:192.0.2.0/24

# Find specific technologies
hostname:example.com http.title:"Jenkins"
hostname:example.com product:"Apache Tomcat"
hostname:example.com http.component:"WordPress"

# Find exposed databases
hostname:example.com port:3306 # MySQL
hostname:example.com port:5432 # PostgreSQL
hostname:example.com port:27017 # MongoDB
hostname:example.com port:6379 # Redis

Shodan findings to prioritize:

• Development services on production IPs — Jupyter notebooks, development web servers, database management UIs (phpMyAdmin, Adminer)
• Exposed admin interfaces — Jenkins, Grafana, Kibana, Prometheus, Kubernetes dashboards without authentication
• Outdated software versions — Shodan reports banners; old version strings map directly to known CVEs
• Non-standard ports — services running on unusual ports are often less hardened and less monitored
• Default credentials indicators — specific product fingerprints where default credential lists are publicly known

Web Archive and Historical Content

The Wayback Machine and other web archives capture historical versions of web applications, sometimes revealing:

• Endpoints that have been removed but not disabled on the server
• API documentation or developer notes that were briefly public
• Previous versions of authentication flows that indicate where legacy code might still exist
• JavaScript source files from before minification/obfuscation was introduced

# waybackurls — fetch all archived URLs for a domain
echo "example.com" | waybackurls | sort -u > archived_urls.txt

# Filter for potentially interesting archived paths
cat archived_urls.txt | grep -E "(api|admin|config|backup|\.bak|\.sql|\.env)"

# gau — get all URLs from multiple archive sources
gau example.com | sort -u > all_historical_urls.txt

Technology Fingerprinting

Knowing the target's technology stack before active testing significantly focuses subsequent work. Technology fingerprinting combines several passive and semi-passive signals:

HTTP Response Headers

Server response headers often reveal framework, server, and infrastructure details without any active probing:

curl -I https://example.com

# Look for:
# Server: Apache/2.4.41 (Ubuntu)
# X-Powered-By: PHP/8.1.12
# X-Generator: Drupal 9
# X-AspNet-Version: 4.0.30319
# X-AspNetMvc-Version: 5.2

Client-Side Asset Analysis

JavaScript files, HTML source, and asset paths are goldmines for technology fingerprinting:

# Download and analyze JavaScript bundles
curl -s https://example.com | grep -E "src=\"[^\"]+\.js\""

# Extract all JavaScript endpoints from a page
curl -s https://example.com | grep -oP '"(https?://[^"]+|/[^"]+)"' | \
  sort -u | grep -E "\.js"

Modern JavaScript bundles often contain API endpoint paths, internal service names, feature flags, and sometimes developer comments that reveal internal architecture. Bundle analysis with tools like source-map-explorer or simply searching for internal strings is consistently productive.

Wappalyzer and BuiltWith

Browser extensions and API services that aggregate technology fingerprinting signals across hundreds of patterns. Use whatweb for command-line equivalent:

whatweb -v https://example.com
whatweb -a 3 https://example.com  # aggressive fingerprinting

WHOIS and IP Intelligence

# WHOIS for domain registration details
whois example.com

# IP range lookup for ASN
whois -h whois.radb.net "!gAS12345"

# Reverse IP lookup — find all domains on same IP
# Useful for shared hosting environments
host 192.0.2.1

# BGP prefix lookup — maps company to IP ranges
# Useful for broad external attack surface mapping

WHOIS often reveals registrar, registration dates, and sometimes registrant contact information. More practically: it reveals whether the target uses a privacy proxy service, and the nameservers used (which may themselves be a target for DNS misconfiguration testing).

GitHub and Code Repository Intelligence

Public code repositories are consistently high-value OSINT targets. Developers commit secrets, internal hostnames, and API documentation to public repositories surprisingly often:

# GitHub search operators
org:company-name "api.example.com"
org:company-name "AKIA" # AWS access key prefix
org:company-name "database" extension:env
org:company-name "password" extension:yml language:yaml
user:developer-name "example.com"

# Tools for automated GitHub secret search
# truffleHog — scans git history for high-entropy strings
trufflehog github --org=company-name

# gitleaks — SAST tool for secret detection in repos
gitleaks detect --source=cloned-repo

Beyond secrets, GitHub repositories reveal:

• Internal service names and hostnames referenced in configuration examples
• API endpoint structures from client libraries or SDK documentation
• Infrastructure-as-code patterns revealing cloud provider, regions, and resource naming
• CI/CD configuration files that expose deployment workflows and environments
• Dependency lists that map to specific library versions and known CVEs

Social Media and Professional Network Intelligence

LinkedIn and professional networks provide organizational intelligence that shapes social engineering assessments and password policy testing:

• Technology stack indicators — job postings listing required technologies confirm the stack and infrastructure patterns
• Organizational structure — engineering team organization, reporting lines, key technical contacts
• Employee email format — determines username structure for credential stuffing or phishing assessments
• Recent hiring patterns — active hiring in specific technology areas indicates new initiatives and potentially new attack surface

# theHarvester — multi-source email and hostname gathering
theHarvester -d example.com -b google,bing,linkedin,twitter

# hunter.io API — professional email format discovery
curl "https://api.hunter.io/v2/domain-search?domain=example.com&api_key=YOUR_KEY"

Cloud Storage and Asset Exposure

Public cloud storage is a recurring source of sensitive data exposure. Common patterns to check:

# AWS S3 bucket enumeration
# Common naming patterns based on domain
for bucket in example company-name company-assets company-backups company-logs; do
  curl -s -o /dev/null -w "%{http_code} $bucket\n" \
    "https://${bucket}.s3.amazonaws.com/"
done

# GCP Cloud Storage
for bucket in example company-name; do
  curl -s -o /dev/null -w "%{http_code} $bucket\n" \
    "https://storage.googleapis.com/${bucket}/"
done

# Azure Blob Storage
for container in example company; do
  curl -s -o /dev/null -w "%{http_code} $container\n" \
    "https://${container}.blob.core.windows.net/"
done

Shodan and Censys also index publicly accessible cloud storage buckets. Search for the organization name combined with storage service patterns.

Recon Output: Building the Target Profile

Effective recon synthesis produces a structured target profile before active testing begins:

Automating Reconnaissance

Manual recon is thorough but slow. Production-quality security testing programs combine automated tooling for broad coverage with manual analysis of results:

#!/bin/bash
# Basic recon automation script
TARGET=$1

echo "[*] Subdomain enumeration..."
subfinder -d $TARGET -silent | anew subdomains.txt
amass enum -passive -d $TARGET | anew subdomains.txt

echo "[*] DNS resolution..."
cat subdomains.txt | dnsx -silent -a -resp | anew resolved.txt

echo "[*] HTTP service discovery..."
cat resolved.txt | httpx -silent -title -tech-detect -status-code | \
  tee http_services.txt

echo "[*] Wayback URLs..."
cat subdomains.txt | waybackurls | sort -u | anew historical_urls.txt

echo "[*] JavaScript discovery..."
cat http_services.txt | awk '{print $1}' | \
  hakrawler -js -plain | sort -u | anew js_files.txt

echo "[*] Recon complete. Review:"
echo "  Subdomains: $(wc -l < subdomains.txt)"
echo "  Live services: $(wc -l < http_services.txt)"
echo "  Historical URLs: $(wc -l < historical_urls.txt)"

Legal and Ethical Boundaries

Passive reconnaissance uses only public data and is universally permissible during authorized engagements. The legal considerations apply at the active boundary:

• Active scanning (port scanning, directory enumeration) requires explicit authorization for the IP ranges and domains being tested
• Out-of-scope subdomains discovered during passive recon should be reported to the client but not actively tested without scope expansion
• Credentials discovered during OSINT should be reported immediately — do not use them to access systems, even if technically possible within scope
• Third-party services (CDN providers, cloud platforms) have their own terms of service; don't test shared infrastructure that doesn't belong to the target