Links are rotting, and we have pages which were created in 1993, so the links are pretty old. There are no links which are that old, because pages are overhauled, but there are old links. I started to add the date about 15 years ago, and there are numerous links without a date.
At the beginning it was hard to find links, and so there were numerous linklists. Those lists either had a single topic, or they were much bigger and had a sort of tree structure where every leaf was a topic, and contained links for this topic. Some were made by persons, some by non-profit organizations, some by universities. They are mostly gone, some are unchanged for a decade. The most famous was the Open Directory Project (ODP), better known as DMOZ. It was discontinued in 2017. The reason was the same why our links are broken.
We found that there are many broken links on showcaves.com, depending on the tool we use more than 10,000. There are 404 errors where the page was removed, or the domain was given up and there are pages which were part of a web framework, and when they changed the framework, the url changed.
We invested several weeks to remove the broken links, which is quite difficult. Its necessary to click on the link, wait what happens.
- if the page still exists anything is okay
- if the page exists but they installed https, add the s
- if the page is redirected to a new address but has the same content, replace the old url by the new one
- if it is now a portal, or simply no redirect configured, I often try a search with the heading of the old link. Sometimes there is the content still on the site.
- if the article is behind a paywall, delete the link
- if the article is gone, delete the link
But in the end its an outdated concept. We decided to still list the official webpage and pages by institutions like wikipedia, Atlas Obscura, mindat.org and others.