Regarding Our Recent Outage, And What We’re Doing About It

Hey all, Tyler here.

As many of you may already know, DoRoyal’s entire web hosting infrastructure has been experiencing a Distributed Denial of Service attack since at least March of 2024. Unfortunately, things just kept getting worse for us, with our entire network going down for approximately two whole days, requiring that we do a full server reset to get everything back up and running again.

Many of you are also constantly asking us the same question. What happened, and will I be compensated?

To start this article off, I’d like to take this time to formally apologize for the outage. We’ve always been known for our overall reliability, with us only ever experiencing a small handful of outages since our doors first opened in April of 2012. Sadly, our reputation as of late has taken a pretty big nosedive, primarily fuelled by the network issues, and our lack of or otherwise slow technical support. (sometimes going days, or even weeks without a ticket response)

I’m sorry. We as a hosting provider failed you. But that doesn’t mean we’re giving up.

1. What Happened?

While we’ve already discussed our DDoS attack in great detail, I can thankfully (and unfortunately) confirm that the “big one” was entirely unrelated to our DDoS attack. In fact, it was a literal “act of God”.

At least, that’s the legal term for what happened. You see, our datacenter is located in Kansas City, and Kansas City experienced one of its largest power outages ever. As in, the entire city went dark for days. This included our datacenter.

DoRoyal was operating on backup and emergency power for days.

Yes, we have backup generators. Yes, they were active. Yes, one of them failed.

The UPS (Uninterrupted Power Supply) failure ended up taking all of our servers offline, including the entire DoRoyal Hosting network, the UlarHost network, and every VPS hosted at ServerTent. Thankfully, UlarHost and ServerTent were both restored somewhat quickly. DoRoyal, on the other hand? Yeah, things sorta got worse.

You see, we now had power. The server was starting up. We were hopeful! At least until we found out that the server wasn’t booting into its operating system. This is kind of a critical thing, as without the OS, we sort of just… well, crash. And that’s what happened. DoRoyal crashed.

Okay, but why? If we have all of these backup systems in place, how was it even possible for DoRoyal to crash? How could a simple power outage take out the entire DoRoyal network?

Well, that’s actually remarkably simple. The power outage fried our SSD. (SSD = storage device)

So we had a physical box, with all of the processing power and RAM you could ever ask for, but no way to actually store anything on it. All because Mother Nature decided it didn’t want us to exist.

This sucked.

2. What About The Accounts? User Data?

Any self-respecting website hosting provider will have multiple off-site backups in place for these exact situations. Well, perhaps not exact, but my point stands.

We performed an emergency restore of every account that we host. This was done on a new SSD. That’s the good news. The bad news?

Our backup server is freaking SLOW! So even though we had everything remotely backed up, it took us well over 24 additional hours to restore 2TB of user account data. But thankfully, slow doesn’t mean unreliable, and eventually, everything was restored.

DoRoyal is back up and operating normally.
Oh, and as a bonus? Even the DDoS attacks stopped.

3. Okay, So What About Compensation?

Yeah, so do you remember how I mentioned how this outage was caused by “an act of God”? That means that the outage was entirely out of our control. So, no compensation shall be given.

…that’s what most other companies would say, at least. But we’re DoRoyal. We’re not here to make as much money as possible, and we also fully understand how severe the outage was. I personally host my own websites here, so this outage affected me as a business, as well as somebody that just wants to have reliable web hosting for my small blogs and personal projects.

So yes, every single one of our active clients will be compensated for this.

Once we’ve fully finalized everything on our end, we will be adding $30 of credit to every account. So, need a new domain? Renewal? Got a monthly hosting bill you want covered? Yeah, we’ve got it.

Once again, every single client will be given $30 in credit.

4. Neat! But uh, What’s Next?

We hate when things go down, but we also know that not everything is within our power to control. Having said that, we can at least do what we can to ensure better overall reliability. That’s why, moving forward, we will be deploying the following backup and mitigation techniques.

Dual SSD – all of our current and future servers will have an additional Solid State Drive installed, which will act as a mirror. Any data stored on the primary drive, will also be added to the secondary drive. So if one fails, the other will be able to take its place.
ISO Backups – we will be utilizing additional backup technologies, which basically has us create a full clone of our entire server, and saving it to our offsite backup device. If our server should ever fail again in the future, we will be able to restore it at any time.
Multiple Remote Mirrors – while this feature isn’t yet available, we will be deploying several additional servers hosted in multiple locations worldwide. Each server will serve as a mirror of our primary Royal 2, so if Royal 2 ever goes down, things will automatically fallback to one of our other servers, ensuring near 100% uptime at all times.

5. What About Customer Support? You’ve Been Radio Silent!

We’ve actually been replying back to tickets every single day for multiple months, and are doing so on a first come, first serve basis. The older your ticket, the more likely it is we’ll reply back to you soon.

We brought the Jester AI in to try and help with most common issues and questions, which was our first step to ensuring support activity. There’s even a whole dedicated department just for our little AI, which is kinda neat.

Ultimately though, and no matter how justified our reasoning may be, we’ve failed you. DoRoyal has failed to provide you with a reliable hosting environment, as well as fast and consistent access to support.

I’m Tyler, the owner of DoRoyal, and I truly cannot express just how terribly sorry I am.

6. The Future of DoRoyal Hosting

Now that the storm is (quite literally) behind us, I can safely and positively say that DoRoyal is here to stay. Never once did we risk “going out of business”, nor did we ever even consider that as a possibility. In fact, despite our constant outages, reliability issues, and our slow support, I’ve never lost confidence in our network.

DoRoyal will thrive, that is my personal promise. Not only that, but we’ve got a ton of really neat partnerships, too!

Oh, and the reviews we’ve been getting? You now, the really negative ones? While it sucks that we’re no longer holding a high rating on TrustPilot, it’s also totally understandable and justified. That said, I’m not overly concerned about these negative reviews impacting our reputation, as quite frankly, I do not expect for the current situation to repeat itself.

So while it sucks that we’re getting negative reviews, it’s only a temporary thing. We’ve been around for over 12 years, and I doubt that a few bad reviews will hurt us over the next 12.

Final Words

Phew, that took a while to write out, and based on the estimates I’m seeing, should take a few minutes to read, too. (sorry about that)

All in all, DoRoyal is here to stay. It’s business as usual.