News

June 6th outage resolved!

Update, 01:00 June 7th:

The sales server and updates mechanism outage is now resolved! We have replaced the certificate on our sales server and everything should now be working properly.

If you tried to make a purchase yesterday and could not, please try again today. Similarly, the updates mechanism should now be working again, so you should now be able to retrieve updates within Army Builder and Hero Lab.

Thanks for your patience, everyone!

———————-

Update, 00:30 June 7th:

The mail server switch that we attempted didn’t complete, so we’ve contacted tech support (a third tech support team for a different service – we are going for some sort of record here!) to have them figure out what went wrong. They are trying to resolve the problem now. We are contemplating what we did wrong in our previous lives to deserve this many problems hitting all in one day.

———————-

Hi everyone,

Our online purchase servers have been unavailable for most of today (Friday June 6th, 2014), so we wanted to give you a status update and share some details on what’s causing these problems.

What’s affected by this problem?

Our online sales server, and the Hero Lab and Army Builder updates mechanisms, are currently unavailable. However, our licensing mechanism is unaffected – Army Builder and Hero Lab licenses can still be activated and transferred as normal. This means you can’t purchase Hero Lab or Army Builder, but if you already own one of our products and are activating your license on a new computer, you should still be able to do so.

What happened? (short version)

Our online purchasing site is currently unavailable, due to some (frankly bizarre) circumstances outside our control. While investigating the problem, we encountered other unrelated issues which complicated things further. We are currently waiting on technical support from multiple organizations to help us resolve these problems.

We are hoping to have these issues resolved later tonight. If we can make that happen, our purchase server will be back to normal before the weekend starts. If not, we’ll make sure to keep everyone updated.

Thanks for your patience while we deal with this problem – we’re doing our best to get things back up and running as soon as possible. We’re sorry for any inconvenience caused by this outage.

What happened? (long version)

This morning, we woke up to find a few users reporting errors checking updates in Army Builder and Hero Lab. What originally appeared to be a problem affecting a few people quickly spread. Over the next hour we realized that this was a serious problem, affecting everyone who tried to access our online store or updates mechanism.

So what happened? First, some (very simplified) background: When you make a purchase online, you want the site you’re buying from to be protected by an SSL certificate. If a web site has an SSL certificate, when you’re using that site, nobody can listen in on information you send to it (for example, your credit card number). SSL certificates are very important.

For reasons still unknown, on Friday morning, the SSL certificate that protects our online purchase site was revoked. We don’t know why, or who did it. Even more concerning, the company that issued the certificate doesn’t know why or who did it. Not only that, the certificate has vanished from our account at said company. When we called their technical support line, the person who answered was very confused because we were showing them a certificate they appeared to have issued, and yet had no record of.

The problem was escalated, then escalated again. At this point it was around midday and we were looking at waiting 12-24 hours before we could find out what happened to our certificate. We wanted to get everything available to our customers again as soon as possible, so we decided to go with the simplest solution available to us – abandon the old certificate and purchase a new one (from a different company, hopefully one that won’t randomly lose our certificates).

When we tried to purchase the new certificate, we ran into a second problem. Before you can buy an SSL certificate, the people selling it want to check that you have access to the server you say you’ll use it on. To do this, they send an email to that server. Simple, right? Unfortunately, at some time over the last couple of weeks, it appears that our ISP has (without warning us) blocked access to that server via email. (This is a different server from the one that handles most of our email, so we hadn’t yet noticed the problem – email volume to it is very low.)

After spending some time talking with support at our ISP, they couldn’t find anything wrong. The problem has been escalated to another department, who will hopefully be getting back to us tonight. At the same time, we’re trying to transfer email service to another mail server which doesn’t have these problems, and are now waiting for those changes to propagate through the internet.

Once either of these things happen, we’ll be able to complete our purchase of the new SSL certificate. We then have two more steps we have to take before everything is back up and running:

1) Complete the purchase of the new certificate and wait for it to be delivered to us. This can take up to 24 hours in some circumstances, but we’ve been told that because we’ve dealt with this company before, they trust us and can issue the certificate very quickly once we have all the details in place.

2) Install the certificate on our server. Hopefully this should be simple, but then nothing else about today has been simple, so we are reserving judgement and keeping our fingers crossed

So, when will this be sorted out? Assuming we can get the email issue resolved in the next couple of hours, and no other major roadblocks arise, we should hopefully have all this fixed by sometime tonight. That means our online purchase server will be back up for this weekend. If we can’t make that happen, we’ll keep everyone updated with our status as we get more information.

Thanks for your support

Thanks for your patience while we deal with this problem – we’re doing our best to get things back up and running as soon as possible. We’re sorry for any inconvenience caused by this outage.

When we have more information, we’ll post it on this site, as well as our usual sources like facebook and twitter.

– the Lone Wolf team

Twitter

  • We've sent out initial invites for the HLO Beta and will expand to include more soon! It's not too late to sign up! ,
  • Check out all these awesome Paizo books in the current Humble Bundle! ,
  • Check out our newsletter for up to date info on Realm Works, Hero Lab, and to sign up for the Hero Lab Online Beta! ,