Table filled in from the top 10m most popular web-sites.

• List Quality
  • This list is prioritized on page rank vs traffic
    • May include both inactive or redirected sites
    • Does not reflect actual traffic or views
  • Other lists that might be interesting:

https://s3-us-west-1.amazonaws.com/umbrella-static/index.html https://www.commoncrawl.org/ https://tranco-list.eu/

• Not that concerned if this list is exact
  • Should provide good sampling of top sites
  • Order really isn't important in this case

• This domain is decades old and not in the list!
  • Don't get much traffic, but 10M+??
• Good performance test of a database table with 10M rows
  • Cold load times take several seconds when paged out
  • Well indexed but not performing as well as exepected
    • Much larger tables have perfomed much better
    • Appears like it is doing a full table-scan
  • This database is running in a SQL Server under docker.
  • Will load the same dataset into postgres for comparison.
• Started crawling the home page for the first 1M domains.
  • Interested in stats on use of html5, proper html, etc..
  • Started with the first 100K sites and expanding to first 1M.
• Observations
  • Surprising number of domains without a proper html lang attr
  • Surprising number of domains not using a proper HTML 5 doctype
  • The domains are naked without fully qualified with hostname.
  • Many domains don't have a dns entry for the naked domain.
    • domain.com vs www.domain.com which should redirect.
  • Suprising number of SSL errors on the naked domain
    • Clients can't connect without dropping ssl verification
    • With standard security checks clients will never get a redirect
    • domains that redirect behind an invalid ssl cert
      • this is super easy to fix/park to handle redirects.
      • lost revenue with that much linking to get on this list