Skip to main content

Introducing the Columbus Project

· 3 min read
Daniel Gorbe
Daniel Gorbe
SysAdmin

An open source append only database of known subdomains. The goals are to discover, store and serve subdomains as fast as possible in a hassle free way.

NOTE: This project is in early stage! It takes some time to be up-to-date.

The current speed of finding subdomains of tesla.com is ~0.311s with 604 unique entry.

$ time curl "https://columbus.elmasy.com/lookup/tesla.com"
...
real	0m0.311s
user	0m0.021s
sys	0m0.010s

Query​

Website​

A website is built into the binary to present the capabilities of Columbus. URL: https://columbus.elmasy.com/

CLI​

Using our CLI tool:

$ columbus lookup elmasy.com
dev.elmasy.com
elmasy.com
one.scanner.elmasy.com
status.elmasy.com
autoconfig.elmasy.com
columbus.elmasy.com
three.scanner.elmasy.com
four.scanner.elmasy.com
autodiscover.elmasy.com
scan.elmasy.com
two.scanner.elmasy.com
gw.elmasy.com
www.elmasy.com
db.elmasy.com
api.elmasy.com
blog.elmasy.com

HTTP API​

By default, the API returns a JSON array:

$ curl "https://columbus.elmasy.com/lookup/elmasy.com"
["dev","","one.scanner","status","autoconfig","columbus","three.scanner","four.scanner","autodiscover","scan","two.scanner","gw","www","db","api","blog"]

Set Accept header to text/plain to get a newline delimed string:

$ curl -H "Accept: text/plain" "https://columbus.elmasy.com/lookup/elmasy.com"
dev

one.scanner
status
autoconfig
columbus
three.scanner
four.scanner
autodiscover
scan
two.scanner
gw
www
db
api
blog

NOTE: The empty string (or line) represent the domain itself.

Sources​

Certificate Transparency​

Multiple Scanners are always running and inserting every found domain.

Logs:

  • Argon
  • Mammoth
  • Nessie
  • Nimbus
  • Oak
  • Sabre
  • Trustasia
  • Xenon
  • Yeti

The full list of logs can be found here.

DNS Servers​

Using our DNS servers is the easiest way to contribute to the project. Every valid record will be inserted into the database.

List of servers (TCP + UDP):

  • 194.163.175.29:53
  • 178.18.241.104:53
  • 149.102.156.147:53
  • 149.102.156.148:53

Contributing is this simple:

amass enum -tr 194.163.175.29,178.18.241.104,149.102.156.147,149.102.156.148 -d example.com
subfinder -r 194.163.175.29,178.18.241.104,149.102.156.147,149.102.156.148 -d example.com

Or just:

dig @149.102.156.148 example.com

This external shell script collects commonly used, "popular" domains and run external tools on it using our DNS servers to ensure, that the most important domains are always up to date.

Integration​

One of the main aspect was the ease of integration into other systems with the API or embedding the result into a website as the CORS setting allows it.