So, I accidentally took Game Jolt down today and I thought it'd be interesting to share why and how it happened, but first I have to come clean - it doesn't actually have anything to do with homeland security

It does, however, have everything to do with website security.

TLDR; no we didn't get hacked and nothing was stolen. the only damage is my trust in humanity and devkind.

Website Security (in a nushell)

This is a super simplified description of HTTPS and certificates. If you're already familiar with this you can move on eyy

Many websites (Game Jolt included) are served in a secure manner by encrypting the traffic between your computer and the server and establishing trust. By trust I mean that when you open your browser and go to gamejolt.com you can "trust" that you are indeed connecting to our server and not someone else's server that's just pretending to be us.

This trust is established by the server sending your browser a certificate. The certificate is like an online ID card. It tells the browser who the certificate belongs to and also who created it. What makes certificates so secure is the fact that they cannot be forged. If a certificate says Bob created it for gamejolt.com - you can trust that Bob indeed created it for gamejolt.com.

But who the f$#* is Bob??

Well therein lies the magic of trust! If I don't trust Bob, it means I don't trust whatever certificates they created. So for example, if I visit gamejolt.com and the server shows me a certificate that Bob created - the browser will tell me "yo, I don't trust that this server really is gamejolt.com because I don't know who this Bob is, he can't vouch for the server's authenticity".

To solve this, operating systems like Windows already ship with a list of trustworthy certificate creators, so any certificate that was made by any of these trustworthy people can be trusted! Those trusted entities are called Certificate Authorities (or CA for short).

If you click on the lock icon next to the url in your browser's address bar you can actually see the certificates for that website. For example, here's ours:

So what happened?

For some reason, our CA gave us the wrong certificate to use, resulting in everyone seeing an error message like this:

Now why the fnaf would that happen??

Our certificates are short lived, usually we need to renew them once every 2-3 months or they wont work anymore. To automate these renewals we use a tool our CA provides.

This tool runs daily and checks if the certificates are about to expire. If they are, it renews them automatically. It's like magic!

Except when it isn't...

See, our CA is trying to avoid overusing their services so they can offer it to more people. Fair enough! For that reason they enforce a very strict limit on how many certificates we can create each month.

However, while testing the tool and developing I have to be able to try a bunch of different things to make sure everything works, yeah? That means I need to generate potentially a lot more certificates than I would normally generate.

For this purpose, our CA lets us generate "fake" certificates. We can generate as many of these as we need which is perfect for testing. What do I mean by "fake" certificates tho? They are fake in the sense that they are not trusted by your computer. This means if you visit a website that shows you one of those fake certificates your browser will complain that the website is not secure and cannot be trusted. Those certificates are meant to be used in development only.

Turns out, due to a bug, feature, continuous lapse in judgement or higher power - our CA gave us a "fake" cert.

Again, why??

We recently switched our host provider. It's been a huge project that took months of preparations during which we had to make sure literally EVERYTHING works on the new hardware.

One of those things was of course the tool we use to renew our certificates. (see where this is going?)

Since I needed to do some testing and didn't want to hit our monthly limit of certificates, I made the tool generate those fake certificates I mentioned earlier. After making sure it all works (and it did), I switched it back to generating the real certificates.

Around 2 months after the host provider migration was finally done and dusted it was that time again - renewing certificates time! The tool ran as per usual, but then instead of getting a new real certificate like it was calibrated to do, it discovered the old fake certificate i was using while testing the tool and decided to renew that instead.

Seems like the tool always prioritizes renewing an existing certificate over getting the certificate that you specified you wanted. Even if they are configured differently! Talk about counter intuitive!

The end result is fake certificates being used across our site and a very grumpy me.

Thanks for reading!

I appreciate you for taking the time to relate to my pain