Dark Developments Where Knowledge Meets Power

16May/110

Recent forum downtime

I apologise to all our users for the delay in us getting information out to you about our forums downtime. Originally we assumed it was that the internet node had dropped (not an uncommon event for our server I regret to say). However after getting in contact with our host, it turns out, that it was a whole load worse.

Our host informed us that our server was down because the last command issued by the root user was
rm -rf /
As we have no doubt any user here knows... that command should never be run.

As we run a security aware site (this one, amongst others) we where rather concerned that someone cracked the 128 character root password, that was randomly set and then rather sharply forgotten. Our host says the command was issued from root, and they are correct that only root would be able to issue that command, however, we also know that our server was on Fedora Core 14.

Here is me, reissuing this command on the newly setup box, to demonstrate what would happen if we where to run that as root.

[root@netw0rksecurity ~]# rm -rf /
rm: it is dangerous to operate recursively on `/'
rm: use --no-preserve-root to override this failsafe

as is shown here, in order for this command to have been run, the last logged command should have been "rm -rf / --no-preserve-root" - rather suspicious that it wasn't, and our host had no backups of the logs to share with us about the incident.

Unfortunately we have lost all our data. Yes all our tutorials on backup and recovery and yet we cannot backup our own site. We are sorry and this has been rectified.

So what are we doing to make sure this doesn't happen again?
Well to start, we are ensuring that all logs are available... when the logs are generated they are now emailed to our sys-admin - that is, around the same time they are written to the OS, this means if someone starts messing about, we should hear about it.

Second, we've altered our security policies. We realised that a 128 character password, could indeed be cracked. So we've removed passwords. There are various other changes we've made internally, but nothing should affect our users.

We do not believe that any user data was compromised, all passwords are stored encrypted, so don't worry we believe your data was safe. However, we also no longer have it, so when we put the forums back online, you will have to re-register, again sorry about that.

I hope you enjoy reading this (rant) update, as much as I did writing it. The forums will be back online as soon as we can finish sorting it's server, and an update will be posted to let you all know when that happens.

Kind Regards,
Ollie.
DarkDevelopments Administration Team

Print Friendly
Comments (0) Trackbacks (0)

No comments yet.


Leave a comment

No trackbacks yet.