TUXEDO Computers website and online shop haven't been available - background information - TUXEDO Computers

As you have already noticed, our website and the corresponding store were unfortunately unavailable for some time! We had to perform unpostponable maintenance work on the servers, which proved to be more difficult than it seemed at first. But only our website and online store were affected, all other services were perfectly functional!

Some technical background information

Our store system runs on a cluster consisting of three so-called nodes, i.e. three synchronously connected servers. These were supposed to be updated to Ubuntu 22.04 last week. In the process, unforeseeable problems arose regarding the dependencies on Ceph, the underlying cluster software. So far so good, since one node at a time should be updated. Although this takes time, it only has a minor impact on ongoing operations.

But as if the effort in time wasn't enough, the NVMe SSD of one of the remaining nodes went off while the first node was still disconnected. The cluster construct had collapsed, resulting in days of downtime - and a lot of work to recover. And even worse, the latest backups of the online store - which are taken hourly - were also no longer available, so we had to revert to a backup from late December now.

In fact, we have a service provider for hosting that specializes in the subject and operates 24/7 management, as well as offering special backup strategies. Obviously, however, all this did not save us from the fact that we now had to suffer a week-long downtime of the website and the store. In addition, the service provider did not clearly communicate how long the offline time would last, so we were late in working on an alternative.

Our other services, which we host and maintain 100 percent by ourselves, were not affected by the downtime, so we had to make a few decisions now.

The good news: We are back!

The recovery work on our store is now complete and we are back online since May 6th. You can now get information about TUXEDOs and place orders again! If it is not possible to log in to your customer account, we ask you for security reasons to use the "forgotten password" function and to set a new password. Thank you for your understanding!

However, there is one thing...

Due to the combination of the server and data crash, some orders have not been registered in the webshop. This means that for customers who have registered since the beginning of the year, no customer account has been created and no corresponding order has been saved in the store. Customers who had registered earlier also probably did not have any orders recorded between January and the end of April. However, since we make an import of the orders for further processing and we receive a copy of the order by e-mail, we can still track some data and enter it manually.

What we will do differently in the future

We have had to learn some hard lessons! First and foremost, we had to realize that we work best when we can take charge and do things ourselves. Therefore, we decided to stop using our current service provider and to buy additional servers to host the website and store system by ourselves. Furthermore, we will distribute our backups more widely than was previously the case, in order to be able to fall back on more copies in case of emergency. Also, as we have been planning for a long time, we will increase our staff.

Thank you very much for your patience!

Despite all the adversities of the last few days, we are happy to be able to offer you an online store again and we would like to thank you very much for your patience as well as the mental support we have received from you, especially via the social media channels!

Your TUXEDO Computers Team