As many readers may know, I often provide help and perform work on other grids as well as DigiWorldz.

Several months ago, I was contacted by a person who was looking for servers to start a grid.
I quoted 2 prices, one price if they managed the servers, and another if DigiWorldz were to handle the management.

My rates were fair and reasonable, but this person decided to go to another provider to get servers because they were a bit less expensive than mine, and they then would manage these servers themselves.
The grid was born.

The owner of this grid contacted me recently as they were having some issues on their grid and wondered if I might give them a clue on what could be causing them.
As always, I offered free advice on what and where to check in hopes of leading them into a direction of correcting the issues on their own.

After several rounds of asking questions and me providing my advice on what to check and where, the issues still persisted and I was asked what I would charge to correct them on their behalf.
I quoted a price, of which they agreed to pay, they gave me the login information for their servers and I logged in to take a look around to see if I could find the problem.
I won’t detail the issues, as that might lead one to suspect a specific grid and I’m not looking to cause them any problems.

After about 20 minutes, I corrected the issues.
The problem had been one of the recommended checks I gave them previously which, after they checked them, they responded were fine.
No big deal, an oversight on their part was now fixed.

Now, I can say that this grid has become somewhat popular and has attracted many users.
These users, like users on other grids expect the basics… ie, being able to login, stay logged in, have a great experience while there, and do not expect to have basic troubles like assets and inventory disappearing, etc.

After correcting the issues for this grid, I gave some advice on some changes they should make to their platform based on what I had seen while in their server.
This grid was running on a single database, assets, regions, inventory, etc, all in one database.
This grid has no slave databases at all, and when I asked about how long ago their last backups were made and checked, they replied that it had been several weeks.
To make matters worse, this grid was storing it’s last backups on the same server which their core database and assets lived on.
If something were to happen to their main server, the entire grid and all of their backups would be lost.

So, here is a grid, running on a single database with no slave databases at all, storing their only backups on the same server as their core databases and assets were on.
This is simply tempting fate in my opinion.

Running a single database requires them to shut down the entire grid so that they can make a backup of their database.
This obviously then, introduces downtime.

After I gave my warnings about their current platform and offered ways to correct them, I was told that they were having other issues too.
After more investigating I found their main hard drive on their core machine was showing I/O errors.
UH OH!! DANGER, DANGER!!

I immediately made them aware of my findings and told them they should obtain storage space from Amazon S3 and move their backups to this area for safe keeping as I/O errors on their main drive were very scary indeed.
I also indicated that they should make a current set of backups and to prepare for the server’s drive to crash.
I advised them to try to obtain 2 more servers, one to replace the current core machine where the drive was throwing errors, and a second machine to act as a slave machine.
I explained having this slave was a good idea as they could then setup a slave database on this machine which would allow them to take backups from this database which would not require taking the grid offline.
I also explained they should additionally use this machine as an area to store their backups as well.
This would give this grid 3 separate storage areas in which to store backups, and would provide a way for them to make a backup of their database without requiring the grid to be taken offline to do so.

I also indicated they should try to get servers which could also be networked together locally, so they would have an internal network in which to move files around, etc.

I was also told this grid planned on a “Region Density” on each server of 50 regions.
They were currently using 4 core machines, which will run opensim fine, but in my opinion is far too weak to run 50 regions on each server.

This goes back to what I’ve talked about in the past.
A new grid starts up using low cost, underpowered machines, running on questionable networks, advertising regions at a very low cost with lots of prims to attract new users.
The new users buy regions as it seems like a great deal, but as more and more users flock to the grid, the regions they purchase are “Built up” and the regions then are consuming far more resources than the inexperienced grid owner has planned for.
Soon, the grid owner finds the servers cannot handle the load of 50 “Loaded” regions on a server.
In order to cope with lag they are forced to buy more and more servers in which to move some of the regions to to alleviate the load and relieve the lag.
Profits drop.
Soon, they may only get 25 regions on a server which is only costing them $80-90/mo.
If the grid is selling at only $5-$6/mo for each region then the gross income on the machine with 25 regions is between $125 and $150 per month.
Let’s say the server is costing them $90/mo., this leaves just $60/mo profit on that server.

Let’s say the grid currently has a core machine like this grid has, and 5 region servers.
6 servers times $90/mo for each server = $540/mo in total server costs per month.
if each server is generating $60 profit each month, 6 servers times $60 = $360 profit each month.

Now, this grid at the very least should have an additional server to run as a slave database and extra storage for backups, etc.. so now we have to subtract $90 from the gross profit to cover the cost of the additional slave/storage server. This leaves the grid with just $270 gross profit each month in this particular operation.

Additionally, if additional cloud storage is purchased as a third location in which to use to store backups, the cost of this space would be subtracted as well.
Usually, this storage space is cheap, a grid this size would probably not even be charged $10/mo for this space, but we’ll use $10 because it’s a nice round number.
$270 – $10 = $260/mo gross profit for a grid of 150 – 200 regions.

Just in a normal month of operation the users will ask for region renames, oars to be uploaded, need help with the usual opensim related issues like slow inventory, can’t connect, etc..
Lets say the grid owner spends just 7 hours per week answering support tickets.. that’s approx 28 hours per month in support.

Now, the grid owner should also be doing backups, upgrades, marketing, etc as well.
Let’s say the grid owners spends only another 8 hours per month on these activities.

This is a total of 36 hours spent on the grid.
If the owner divides $260/mo gross profit from above by the amount of hours spent on grid activities, they are earning $7.22/hr., but there are far more expenses the grid will realize as well.
Maybe they have software licenses to pay, they’ll have domain name fees, they’ll have web hosting fees, etc.

In the case of this particular grid, they have partners, so the grid would have to divide the profit by how ever many partners they have.

Additionally in the case of this grid, they need additional work performed by someone else which they are incapable of doing themselves.
There will be an additional fee for that which will have to be factored in.

In the end, this grid is in a position that is very scary.
Their platform is inadequate, their expectations are based on inexperienced assumptions, the servers will quickly become overloaded, they are selling their services far to cheap, they cannot do all that is required themselves, causing them to bring in outside help to correct the issues which they cannot afford, they have a server which is failing, very old backups on which to fall back on should they need them, their backups are in danger due to being stored on the same server, they are only storing their backups in a single location, they have no hardware failure plans, etc.

After talking about all of their issues and needs, I was asked if I could provide what was needed and if so, what the cost of providing it would be.

We chatted about the costs, and in the end they decided to purchase 2 servers from me, but they would manage them.
They indicated they would purchase the additional servers the following week.
They would then pay me to setup their platform as described.

This was on Friday, December 23rd.
I explained that it would take a day or so to get the new core and slave servers online, and that I would be leaving for the holidays, but would return on the evening of christmas around 6pm my time.

They paid me for the 2 servers, I ordered them and a few hours later on the 23rd they were placed online.
I prepared the servers to become their new core and slave machines and indicated when I returned on christmas day, I would login to their server, make a backup of their database and assets and starting Monday morning I would install their backups on the new core and would setup database replication on their new slave server.
In the meantime, I advised them to move their only backups to the newly purchased amazon storage space for safe keeping.

I left, celebrated christmas with my family and returned on Christmas day as indicated.
Shortly before the predefined time I was to begin work on their server, I poked them in skype to make sure we were still set to take their grid offline and perform a backup.

I was told their had been a change of plans and they had decided to buy their servers somewhere else as having another grid with access to their servers was unacceptable.
I indicated that our operation was professional and that we would be happy to enter into a legal “Non Disclosure Agreement” to protect them. They indicated they were still not comfortable with this idea and they would not be using our servers and would like to have their money refunded.

I refunded the money they sent for the servers and have directed our datacenter to take them offline.

In the end, this grid has decided to take my recommendations and try to implement them on their own.
I wish them well, but I’m not confident, based on what I’ve seen and heard while talking to them that they will take care of all of the issues.

I hope they do, both for their sake and the sake of their users.

I setup my schedule to help a grid in need on Christmas day, jumped through hoops to get them the hardware they needed quickly, warned and advised them on what “Might” happen and pointed out they should quickly move what backups they have to a different location, and I returned from my holiday celebrations prepared to do the work to help them, and now they’ve decided they did not “Trust” in a competing grid to help them, even after I suggested entering into a non disclosure with them for their protection.
They’ve taken the free information I’ve provided, with the plans of implementing the indicated changes themselves, and now want a refund for the servers they purchased.

I am thankful not all business transactions are like this.
I’ve always told everyone, my advise and information is free, but if you want me to develop something for you, or fix something for you I will charge a fee, but info is free.
Most of the time, people appreciate the advise and suggestions they are given and often I’m rewarded with not only a thank you, but many times they will offer me the opportunity to do more work for them since I’ve already helped them for free.
I’m quite proud of my reputation within the community, and I’m quite pleased to answer any questions one might have about anything I might know. Sometimes it’s grid setup info, sometimes it’s about how to cure a problem, sometimes it’s about how we do things on DigiWorldz. In each case, I’m quite happy to share.

So here I sit, knowing the grid’s users are in a very precarious position and they don’t even know it.
Even if the grid is able to implement the recommendations on their own, their choice of hardware, the provider they’ve chosen, the price point they’ve chosen, and their plan to have 50 regions on each server will cause nothing but issues for them in the future.

I feel a business obligation to protect what I’ve learned about the grid, not to share it with anyone, which could cause them issues, but I also feel an obligation to the users. They should know what kind of position they are in.

I’m afraid due to the legal implications, I cannot “Point out” this particular grid and I also do not wish to cause them any further issues, so don’t ask… I will not tell you…  they are nice people who are trying hard to provide a platform for users to enjoy, but their lack of experience, and their poor choices are, in my opinion, going to be the downfall of them.

Maybe one day I’ll write up a blog detailing my ideas for starting a new grid, but for now, I’ll close with saying this…

Users should always ask questions of their provider.
Nothing stinks like the smell of losing everything you’ve built due to choosing the wrong provider.
Chasing the best deal is fine, just be sure you know where you stand in regards to the grid’s backup policies and business practices.

If you are contemplating starting a new grid, find someone who has been there and done that. Ask lots of questions and learn all that you can. Learn from what those who have ventured before you have learned and be sure to choose the right hardware and the right network.
Plan for the loads you might see on each region once they’ve been “Built up”, not what you are seeing on empty, or lightly loaded regions. Be sure to have a regular, workable backup plan and perform those backups regularly and store your backups in several different locations, on different hardware and preferably in different datacenters. Be sure to know how to fix and do everything yourself, or be prepared to pay someone to help you if you can’t do it yourself.
Make sure to charge not only what is needed sustain what you have, but to grow and expand your features, do marketing, etc.

Do not think the most important thing is price. The lowest price, as anyone should know by now is not always the best price. Do your research, it will pay off many times in the future.