UPDATE: Restart has been completed.
We have a grid restart planned for 6pm Grid Time on Tuesday November 15th, 2016.
This is just a routine restart to keep things fresh and downtime is expected to be less than 2 hours.
Happy Friday Digizens!
Some of you might not be able to login at the moment. Fear not, its just the datacenter adding/configuring some equipment to help us fight any future DDOS attempts to take our grid down. Please bear with us while they make they final adjustments and then we should be back business as usual!
Recently, we have been tweaking and experimenting with the maximum number of users in a single region.
You may have noticed our “Avatars Online” count has seemed quite high lately.
By default, Opensim regions are limited to only 40 avatars in a region. This figure can be increased in the config files for the region and/or from the console.
Most viewers have the limit set to only 100.
This can be somewhat confusing as if the grid is still set to the default of only 40 avies in a region, and you use the estate tools in the viewer to try to change this, you can set the value in the viewer to any number you like, but the maximum number of avies in a region will be overridden by the opensim config values.
After some careful tweaking, and some experimenting, we have been able to get 325 “Real” avies in a region. These were not NPC’s.
Making matters worse, the region was a large var (10×10).
These avatars were in a “Ruthed” default state so they did not have mesh clothing, attachments, etc., but the fact we were able to get this many avatars in a region without it crashing is a very cool thing.
Granted, at 325 avatars we experienced some lag, lots of lag, but we were pleased the region did not crash, even after we dropped thousands of physical balls in the region.
The region became usable after we cut the number of avatars down to 175 and cut down on the number of physical balls in the region.
We are using the pcampbot utility and logging in batches of 25 avatars from several different machines.
These avatars are randomly jumping, walking, flying, running, chatting, etc… it is quite busy as you can imagine.
As a test, On the evening of 5/12/2016 we logged in 175 of these bots in a normal sized region and left it run.. so far it has been running for about 19 hours straight with no crashes, negligible chat lag, and other avatars are able to walk, build, fly, etc. with minimal lag.
If you want to see for yourself, visit login.digiworldz.com:8002:rnd1
Be warned, these bots are quite rude!
This test will end later this evening, but we’ll keep testing off and on over the coming weeks as we want to find a combination which will allow a region to have 200-250 avatars on a single region and the region still be usable. That’s our goal, as while having that many avatars in a region is cool, it’s useless unless you can still do things normally there.
I have created a DigiWorldz Users Skype group.. please feel free to join if you like:
The DigiWorldz Grid is currently offline due to a DDOS style attack on our login server.
We have temporarily suspended logins and have taken the grid offline to mitigate traffic in and out of our servers.
As soon as we get the attack under control we will resume normal operations.
Sorry for the inconvenience.
Our preliminary investigations have shown that the Great Canadian Grid was attacked first. When the Great Canadian Grid was taken offline to end the attack, the attackers then moved to the DigiWorldz Grid.
It appears only the core servers at both the Great Canadian Grid and the DigiWorldz grid have been targeted.
DigiWorldz and The Great Canadian grid combined have 48 servers at our datacenter, but only our core machines were targeted.
Seems maybe a competitor has a grudge? Maybe a user?
The fact is, we don’t yet know as we have an enormous amount of data to sift through to find our clues.
Rest assured once we are able to pinpoint where this attack came from, we will prosecute to the fullest extent allowed by law.
A DDOS attack happens when someone directs many machines to “poke” a specific machine at the same time or very near the same time without stopping.
This has the effect of “overloading” the services found on the server and essentially “Bogs” the server down trying to answer these many requests thus making the intended services unavailable for real users wanting to use our services as they were intended.
Think of it as “Thousands” of visitors showing up on your doorstep, ringing the doorbell asking if “Joe” lives here.. you answer no, the visitor leaves, then you close the door. As soon as the door is closed, the bell rings again, this time someone different asks if “Joe” lives here. Before you say no, you notice that many others are lining up behind this person. You tell this person, no, “Joe” does not live here, they leave, but the next person behind him asks the same question. You give the same answer, and so on, and so on.
As you answer each visitor, you notice now that 100’s of other visitors are still lining up.. then thousands, then 10’s of thousands, etc..
Most of these users are here to ask the same question… “Does Joe Live Here?”, but.. in the midst of all of these users who grow by the second, there are real users in here asking if they can login to your grid, but as the number of visitors increase, the time it takes to get to a real user, provide permission for them to use the grid and go to the next visitor keeps increasing as the number of “Bad” users taking up your time increases.
Eventually you become overwhelmed and you slam the door closed.
That is essentially what we’ve done. We can only wait until the person or persons directing this attack get tired and go away.
Why don’t we block them?
Basically we have, we’ve slammed the door… meaning we’ve turned off internet access to our core servers in hopes the bad users will go away and find something useful to do with their time and resources.
Until they all go away and stop ringing the doorbell, it is pointless to keep trying to answer them.. all it is doing is wasting precious, costly bandwidth while at the same time denying our real users the usage of our grid.
The nature of DDOS attacks is such that if we block the bad users, we also block the good users.
DDOS attacks take advantage of the fact that in order for a machine to function, it must accept public requests on a specific port, but a machine can only handle so many requests before it essentially becomes overwhelmed.
The idea here is that they keep sending these requests in numbers great enough to overwhelm the server thus essentially denying good users the ability to contact the server.
DDOS attacks are very basic in nature, they aren’t “Hacks”, they are essentially many computers/servers trying to make a valid request to a single server in an attempt to overload them and these sorts of attacks are quite common.
Often times these types of attacks are launched from 100’s or 1000’s of machines which have been compromised in some way and are under the control of a single individual. These “Networks” of compromised machines are often called “Botnets”.
Sometimes they are caused by a program on only a few computers which launch the attack.
The logs will show which mac addresses and ip addresses these attacks came from and we will analyze them closely to try to pinpoint the source of these attacks, but in the meantime, I’m afraid we have no choice but to make our services unavailable until we can put measures in place to mitigate these attacks.
Both the DigiWorldz Grid and The Great Canadian Grid will be offline for at least 24 hours while we sort all of this out.