Yes, another disgruntled customer. We opened a ticket on 6/17/2009 and have gotten a couple responses and then nothing for weeks. We keep updating the ticket for a response....and nothing.

We have suffered thru the recent outages, past outages, hardware failures, human errors, network screwups and are now experiencing a CDP Backup failure.

I have waited for weeks thinking they would eventually get around to us, but nothing. Total disregard for us, our company needs, or the crap we have put up with. And to top it all off, PEER1 has solicited our business offering us a slightly better configuration than we currently have with ThePlanet and all for a little more than $600/mo less than we are currently paying...and we have told them that!! Even faced with the eminent loss of a customer, we are ignored.

Anyone considering ThePlanet, read these forums before making your decision.

In regards to my backup situation, we purchased a 500gb CDP Server to handle backing up 6 servers. All 6 backup daily and 3 backup hourly in addition. Currently there might be 3 servers doing a backup...I can't get in to know for sure.

ThePlanet response:
This server looks to be running low on memory which causes hang/extremely slow responses. In general, how many backups are running on CdP Server at a moment? I'm not able to login to CdP Server to check the amount of hosts and backup schedules there are currently. I would recommend a memory upgrade first to resolve low memory issue and then tuning CdP Server.

I guess the moral of this story can also include: Don't purchase a CDP Backup server from the planet if you intend to backup any more than....X servers. I have no idea what x is. 1? 2? We were pitched this backup solution as a great alternative to EVault and being able to handle all our servers. It is definitely cheaper than the stored space that would be required thru evault...but none of that is worth anything if it doesn't actually work.

Anyone at the planet interested in my tickets? (5949237PLNT, 5794343PLNT)

I am extremly frustrated right now and will leave it at that.

Thanks for flagging these tickets for us. I've escalated the two you've included (and attached this thread) to the managers in charge of our storage and data protection offerings along with a product engineer to investigate the capabilities of this dedicated CDP server.

I understand your frustration, and I want to better understand how we can help you. I see the pricing request that has been going through sales and finance approval over the past few weeks, and I don't understand why it hasn't been updated with any customer-facing communication (while it has been updated 8 times internally since the last response you've seen to it).

On the CDP server side, I'm not sure if it's going to be an issue that we'll need to enlist R1Soft's help in or if it's simply a one-off server hardware/configuration issue that is limiting the performance of the server.

We encourage all prospective customers to visit our publicly-accessible community forums to learn more about our company. While we hate that issues like this come up every now and then, it's important to have an avenue to escalate and follow up on our performance, and we're certainly accountable here. My direct email address is KHazard AT theplanet.com, and I'll be keeping an eye on your tickets until they get resolved. If you have any questions or additional concerns, please shoot me an email, post an update here in this thread or send me a PM.

I just got off the phone with tech support and was suggested that we need to schedule a time when we can take down our servers for 4 hours to run hardware checks. That won't work unless we do it 1 server at a time and failover as we go. I would prefer to not have to disrupt business and, unfortunately, we are a 24/7 company in which there is never a real good time to just take servers down. He indicated it "could" be a hardware issue on our servers or it "could" be a software issue with R1Soft. My vote is software with R1Soft. Having scoured the web for R1Soft reviews, they have some serious issues. I am not the only person being left with a bad taste over this software. Furthermore, the backup issues/failures have occured on 3 separate servers. I would say it is more likely that the software is failing rather than 3 servers all having hardware failures at once....and yet no other issues manifesting themselves.

I did suggest that we look into getting back on EVault since it was always flawless...but, understandably, he was not the person to persue the discussion with.

And, though EVault was perfect, it is still not very cost effective. SQL Server grows, file servers grow. We are currently utilizing 72% of our 500gb which would be close to $400 a month on EVault. The CDP Server is roughtly 200 a month...but inreliable. The proverbial Rock and Hard Place comes to mind.

You guys have to have plenty of customers that need offsite backups and I would find it hard to believe they don't need huge storage capacity. Are they using EVault or is there a secret order or society of backup people that get a break on storage or have other solutions not publicly offered. I don't see any reason we won't eventually exceed 1TB of required backup space. That's $1000 a month just for backup. I can go buy a 1TB drive at bestbuy for pratically nothing compared to that. There is something seriously wrong here. There has to be a better way.

And from that I gathered during this phone conversation, the CDP Server we have is really only capable of backing up a couple servers, due to it's limited hardware capabilities. Hmmm, and when we purchased it you knew we had 6 servers. It's not like things aren't transparent. Either you knew you were giving us a sub par solution, or your hard ware guys are provisioning sub par hardware and your sales guy just don't know any better OR no one really knows what's going on and we are all flying blind in the dark. I certainly knew nothing more about the CDP servers than you told me and marketed, and no one every mentioned a memory upgrade would be necessary or that the thing would crumble before we would really know a memory upgrade would be necessary.

I appreciate the fact that you guys are now looking into our issues. However, and that's a big HOWEVER, I do not appreciate the fact that it requires blasting the forums, blasting the ticket queue, and making calls to tech support to get it done.

And the Billing Modification ticket, clearly you get it...nothing more I can really do but sit back and wait to see what this accomplishes.

We can replace the chassis and memory without the need for dealing with diagnostics. We can run diagnostics on those components later down the road.

If it's suspected to be an issue specifically with the hard drives, then the need for diagnostics will take longer since we're trying retain the data on the drives. If you're willing to go through an OS reload, RAID rebuild, and re-seed your backups, then we can skip diagnostics on the drives and just swap them out. We can also replace the RAID controller without diagnostics, but I recommend that it gets lumped in with the hard drives.

I've seen the CDP solution run well with more than 6 machines using incremental backups set for every 30 minutes before with similar specs to your server. How much is being backed up with each increment? If it's a lot, you may want to use more frequent increments which should even out the load at any given time. Keep in mind that the CDP solution was designed to do this. If you're running only Daily backups, then the benefits of the solution are somewhat lessened.

While I have your attention. A couple of us got this email from the guy we finally got on the phone to discuss our issues:

******************************
Hello,

Thanks for being a valued customer with The Planet. I do apologize for any inconvenience you’ve received in regard to your server. It was great speaking to you and if you ever need anything additional or any assistance please do not hesitate to contact me directly. Hopefully the senior tech support representative I got you on the phone with will be able to get everything fixed for you. If you are looking to setup some servers or need more information on any of the items we offer please email me back when you’re ready to set something up. If you are going to place any order on our website, I would appreciate it if you selected me, Cedric C, as your Sales Representative. There is a drop down box on the checkout page where you can select this.... Cedric Ihegword. If you have any additional questions please don’t hesitate to email me back. Thanks!
******************************

Now, this is not our sales rep. This was just a random recipient of our phone call. He knows nothing about our account, who we are, what we do, or what troubles we are having. Why exactly would be direct any sales in his direction, and furthermor, how is his soliciting sales from us something he should be doing?

I get the feeling I am working with car salesmen. I have a mental image of 10 sales guys standing outside like vultures, just waiting for the next unsuspecting victim to meander by. Is it normal for sales reps at ThePlanet to solicit sales from customers of other reps? Or is everyone working on salary and it doesn't matter?

This just seems odd. Doesn't it defeat the purpose of assigning a rep to our account?

I know I am jumping around here, but you guys are really doing a lot of sketchy stuff.

I've been watching the updates to the ticket, and I think there are two real schools of thought to resolving the technical issues: 1) Do not modify your backup timeline and add RAM to accommodate for multiple large concurrent backups or 2) Modify your backup timeline to better utilize the continuous nature of CDP ...

By scheduling hourly and daily backups with CDP, you're taking the iterative changes that are designed to back up continuously (let's say every 5 minutes) and stockpiling them all to run hourly or daily, so when it comes time for your daily backups to run, you're running one backup for a server when, with CDP, you could have broken that up into 288 5-minute backups on the box getting daily backups. Now if you've got three servers scheduled to do that at the same time, you're processing equivalent of 864 "continuous" backups (assuming that different files would be changing in each of the 5-minute backups. In actuality, that equivalent number would be lower, but it is being used to illustrate the problem you're running into).

The same thing is happening in the hourly backups: you want to back up three specific servers more regularly, so one could assume that those backups are the most dynamic and most important to you. Instead of doing one backup every five minutes, you're waiting and performing 12 of those 5-minute iterations at one time per hour per server (likely all at the same time).

When you moved over from Evault, it doesn't seem like your backup strategy took into account the abilities (and limitations) of the new platform. R1Soft allows you to customize and distribute the backup load however you'd like, so before simply adding RAM, I'd recommend staggering and breaking down the backup load to see if the same problems persist. The key to effectively utilizing the CDP server is to distribute the workload as evenly as possible to keep the incremental backups smaller (and more regular).

Here's an example of what I mean by staggering the backups:

Servers Currently on Hourly Backups -- Move to every 20 minutes
Server 1
*:05, *:25, *:45
Server 2 (Currently on Hourly Backups)
*:10, *:30, *:50
Server 3 (Currently on Hourly Backups)
*:15, *:35, *:55

Servers Currently on Daily Backups -- Move to every hour
Server 4
*:00
Server 5
*:20
Server 6
*:40

These backups should all be incremental (not full backups). If they are taking more than 5 minutes each, the schedule would need to be adjusted and tweaked.

We got an email from this very same guy today trying to steal another reps sale.

Can you forward the email to tdurden%at%theplanet.com?

Patrick,

I've updated the ticket. I'll be out of the office here in a while, but I'll checkup on the ticket tonight.

I think the scheduling changes will benefit you the most. If the issue persists, then we'll want to consider other options. Depending on the situation after the scheduling change, we'll want to consider doing diagnostics on the drives or memory upgrade(if there's no indication of drive failures).

I usually prefer to start with the no/low-cost options and work up from there.

After discussing this with both ThePlanet and R1Soft, R1Soft suggests I keep my backups rotation about the same but with a little tweak. I do agree that I should stagger the timing a little...but I am not sure that will work since my hourly backups sometimes take longer than an hour and my daily backups difinitely take longer than an hour and therefor my hourly backups kick in before the daily is complete. Anyway, R1Soft suggests I continue doing:

23 Hourly
+
6 Daily
+
x Weekly - I will probably do 2.

For databases, this seems like a good strategy. I don't really need 14 straight days of backups, as I currently have. I doubt I would ever even need to go back 2 weeks, but better safe than sorry. The 24 hours of backups is the most critical requirement. That's where we will be recovering from in a loss.

This reduces the necessary disksafes, and as R1Soft advises, would "boost" performance. Here is R1Softs actual response.

*******************************************************
Hello,

Keep in mind that each snapshot is a "full backup and recovery point".
The more snapshots in the disk safe, the larger and more loaded the disk safe becomes.

For 14 days of backups, the smallest, fastest, and simplest would be 14 daily snapshots.
(set to a max of 14 recovery points)

In your example of having 336 or even 672 snapshots in the disk safe at any one time,
it sounds like you are needing hourly backups as well, at least for the first 24 hrs.
( would you really ever restore back to the 335th or 671st snapshot in a disaster? )

So , what about:
One scheduled task of 23 hourly from 01:00 to 23:00 ( set to keep a maximum of 23 )
One scheduled task set to 14 daily , at 00:00 ( set to keep 14 maximum )

This would limit the maximum amount of recovery points in the disk safe to 37,
and still provide you with up to the hour snapshots for the last 24 hours,
and a daily backup for the previous 13 days.

Or choose to keep only 6 days of daily backups (Mon-Sat), and create another scheduled task that runs on a weekly basis.
One scheduled task set to 4 weekly on Sunday - ( set to keep 4 points maximum )
This would only be a total of 33 snapshots, 23 + 6 + 4, and allow you to go back up to 4 weeks.

This type of plan would create smaller disk safes, which defragment and verify faster,
( which can take as much or more time to process than the actual snapshots themselves.)
and improve restore speeds due to the reduced size in which the restore task must sift thru data.

You could also schedule in a weekend stand-alone defragment task, to occur on Sundays, for example,
that would allow you to eliminate all of the defragment tasks within every scheduled backup, ( improving speed )
and limit this activity to the one day a week that you select.

Regards,
-Kent Porter
-R1Soft Support

And we were down again today for an hour. Customers livid as usual and one even let me have it for having been down sooo frequently in such a short period of time. I had to eat crow as usual. Had to appologize, even though I have no reason why and for what reason..just that I am sorry and that we are just as upset as you are. All I could say was that the outages were just as unacceptable to us as it is to him and that we are looking for a better solution. This is not the type of hosting we had in mind when we joined. These past few months have been hell...and in this economy, we really don't need to be giving "our" customers any reasons to leave us...just like you guys shouldn't be as well.

It is only the backup server not your actual web servers that went down for an hour - do I understand this correctly? Does it cause any impact to the web servers when the backup server is down?

No, not just the backup server. Another outage in the datacenter. All of my servers were inaccessible.

Response from tech:

Dear Customer,

I apologize for the inconvenience. There appears to be a partial outage in the data center where your server is located. Our Network team is aware of the issue and is working to get this resolved as soon as possible.

Thank you for choosing the Planet!