Retrospective on Backup

Overdue responses...

Happy New Year, everyone. I hope that y'all enjoyed yourselves over the holidays and that 2009 is already looking good from your perspectives.

Here at Retrospect HQ, we're running flat out to get a good preview/beta ready for Macworld. There's a lot that isn't finished or hooked up, but you'll soon be able to download a build of the new Retrospect for Mac and at least run a backup (or several at the same time) and get a feel for where we're going with it.

I wanted to take a few minutes to respond to some of the great questions and comments you all have asked here. It's unfortunate that the team doesn't have time available to respond to each comment, but we do read and talk about them. Here are a few of those questions with our overdue responses...

From Russ:

The hardest thing for newbies to grok seems to be the overall Retrospect paradigm, especially those coming from other backup programs. Those who are accustomed to the baseline (or "full") backup, followed by daily incrementals, have a hard time understanding Retrospect's restore model that uses snapshots, and keep trying to fit Retrospect's paradigm into a "restore full, then restore all successive incrementals in order" model.

You're quite right here, Russ. 90% of the file-based (as opposed to image-based) backup software out there, and especially in the Windows world, must be told what type of a backup to perform, such as a full, incremental, or differential. These types of backups choose files based on archive bits, which is a braindead of choosing files. There's no such thing as a type of backup in Retrospect...or in Time Machine for that matter, which helps us somewhat.

Retrospect always performs Smart Incremental backups (yep, that's a new term, though more of a marketing one). Point Retrospect at a destination, and it automatically copies just the unique files needed to complete that destination. Give Retrospect an empty destination, and it will add one copy of every unique file it finds. Give Retrospect a destination with some files already present, and Retrospect will only add a copy of each unique file that isn't already there.

I don't think we've ever done a good job explaining this up front in the documentation, but we will be doing that this time around, and we'll be discussing briefly what some key terminology means up front. And tool tips in the user interface will help, too (though they won't be in the first preview/beta).

From Rick (and several others with similar comments):

I sure hope PPC support is in the initial release, otherwise it's a wait (and not too long or else look for other solutions) situation for me. I don't know whether I'm like other small installations, but while I have several intel based macs on my network, I use an older PPC machine as my central server, which is where Retrospect will need to run.

I estimate that very close to 50% of existing Retrospect installations are on PowerPC-based Macs. After all, Retrospect 6.1 is not Intel-native, and many small businesses tend to give newer Macs to their users, not to the backup/restore task.

While support for PowerPC-based Macs will not make it into next week's beta preview, it will definitely be in the shipping version. The engine (called the Retrospect server) will run under 10.4 and 10.5; the UI (called the Retrospect console) will be 10.5 only, because we're making use of some key interface elements that appeared in 10.5.

From Ronald:

...one of the most important features of Retrospect for me is the ftp backup. Yet it's treated like a stepchild. I hope this gets properly updated in Retrospect X, incl. removal of certain long-standing limitations.

It's funny with all the online/cloud backup services taking off these days, it's easy to see how far ahead of its time Retrospect was. FTP backup was added with the 4.0 release, and that was more than 10 years ago.

Unfortunately, the new engine does not yet have FTP backup capability. We will update that capability with SFTP (WebDAV would be nice, too) and make it available in a point release in the near future.

From Jay:

my biggest problem with the current version of Retrospect is the performance when doing the initial file scan, not performance writing to the backup device.

I have one Intel Xserve with about 300GB of data on it which currently takes approx 3 hours to do the scan, followed by 1-2 hours to write the backup...

How is performance on scanning looking?

Scanning is faster and uses fewer system resources. I'll see if I can post some performance data here before Macworld Expo.

Here are a series of questions from Maurice:

Will I be able to make report rules that tell me if a system that should be getting backed up hasn't had one in x number of days? Will I be able to make a report rule that can actually check on backup scripts to let me know there is some misconfiguration that's preventing it from running?

Yes and yes.

Where you display dates, I'd suggest following Eudora's idea of naming the last seven dates rather than displaying date itself.

This won't be in the beta, unfortunately, but we hope to get it in the final release.

A minor quirk: I don't think there's enough room to display double digit hours, for example, 12:30 PM.

On the adding client screen, the default display should display the full name of the system—no ellipsis—and its DNS name. They don't always match. The IP address should be shown in full without any ellipsis. Don't we all have big-screen monitors?

Columns are resizable and reorder-able, and many views allow you to turn columns on and off to meet specific needs. This may be one of the most useful features of the new UI: It's all customizable. Check it out:

I can't see scrollbars on the right side of the Sources pane, but from the looks of it, the default should be display way more sources at once.

Not only will there be scroll bars where you expect, but the entire detail view (in the lower half of the window) can be collapsed/hidden.

How will I easily be able to assign, say, 50 computers to a "source group"? It seems as if I'd have to manually enter a tag for each system. Also, how can see every system in a particular group?

Tagging, which replaces Source Groups, won't be completely implemented for the beta, nor will multiple source selection. But they'll be in the release, and you'll be able to assign tags to multiple sources.

From David H:

I think the backup to disk options are a big issue, and all your screenshots here are eye-candy, which do nothing to reassure me that fundamental issues with Retrospect will be actually resolved. Until such time as I don't have to deal with Retrospect flat-out INSISTING that a backup set begin with "1-" , I'm done with this program. That and scripts that are written (!!) to use (for example) "media 1" rolling over to use "media 2" and then failing when "media 2" is not found...

Retrospect 6.1 for the Mac doesn't really have true disk-based backups; it either uses the Removable Disk Backup Set, which forces erasing and renaming of each disk, or it uses the File Backup Set, which places everything in a single file structure.

The new release has true Disk Media Sets, which allow you to keep disks named whatever you want and back up to folders contained therein. You can add members (essentially folders on local or network-attached volumes) to a Disk Media Set, and as long as one with free space is available, the backups will run. The new Retrospect also has a "Skip to blank media" option for folks who use tape and other removable media.

From Marcel:

Will Retrospect finally run as a server process?

As I mentioned above, yes it will, but it doesn't hurt to mention again, since this is the single largest change from 6.1 (and anything we have for Windows, for that matter).

The new Retrospect has three components:

1) the Retrospect server (engine), which runs as a daemon;
2) the Retrospect console (user interface), a Cocoa application; and
3) the Retrospect System Preference pane, which can start/stop the server.

While it's certainly valid for all three components to be installed on the same Mac (that's how I'm using it right now), this architecture also allows the Retrospect server to be installed on one computer, while the console is installed on an entirely different one. This allows a consultant who runs the backups for several clients to manage their Retrospect servers in a single window. Very powerful.

Now, this architecture also changes how things work, even on a single Mac. For example, we can't use the standard Open/Save dialog boxes, since the Retrospect server we're controlling could be across a network, and that's where you need to browse through files and folders. So take note: It's going to feel different.

From Fred:

...can I make a plea that the configuration files are text based rather than binary files? Several times these have got corrupted and I have had to start over from scratch...

The settings for the Retrospect console are contained in a standard plist file. The Retrospect server, however, still uses a binary config file. While this is something we'd like to maneuver away from and hope to do so in the future, you will find that the new config file is far less likely to become corrupted.

Two from Erik:

Retrospect 6.1 doesn't seem to be capable of handling more than 12 tapes in a single library. It will choke as soon as it tries to scan the library after I insert the 13th tape.

You will find much improved support for tape libraries, including much better scanning, automated drive cleaning, and support for multiple drives in a library (the latter with an Advanced Tape add-on).

While Retrospect 6.1 is performing a backup or restore operation, it will prevent any other actions from being done within the Retrospect software. For instance, while I am overseeing a lengthy restore, it would be nice to be able to prepare the next restore operation, or indeed be able to restore from one tape library while backing up to another.

This, you will be able to do with ease. This new version of Retrospect has complete support for handling multiple streams of data.

And finally, two from Matthew:

Email notifications. I want to be able to set triggers for email notifications to be sent to my admin email address. Triggering events may include successful backups completed, errors, timeouts for accessing remote items and timeouts for waiting for new media to be mounted (say, a 3rd member to a backup set).

Email notifications aren't going to be hooked up in the beta, though you'll see the options in Preferences:

Look for more email options in future releases.

Setting auto-timeouts. If I run a backup server, for instance, I want to be able to set a time for the server to give up on a client if it loses communication. Like, say, wait 20 minutes, if the execution remains stalled, abandon.

This is a fabulous request, and I know it has been asked by others as well. The only reason we haven't added that as an option for standard backup scripts is because Proactive Backup scripts (called Backup Server in 6.1) has a "Retry failure after n minutes" feature. So if a connection drops, Retrospect waits that amount of time and tries to hit that machine again until the backup is successful.

Still, I'd like to see an option for this. We'll see what we can do.

This post ended up being a little longer than I expected, but I hope that everyone finds some useful info here.

Happy New Year, all.

-Eric

New Retrospect X screenshots!

After being out of the office at Apple's Sales Engineer Fall Camp up in Tahoe City on Wednesday and Thursday, I didn't have time to get the new screenshots up yesterday, so here they are (click any screenshot to see a full-size version)...

Dashboard/Reports
This is the main dashboard view showing a single Retrospect server in the left-nav. Retrospect is showing the four reports that I saved from various other views. Reports are like smart folders; you can save a set of rules for each one. Clicking the Edit button on the right side of each report's title bar takes you to the corresponding view where you can make changes to the criteria you've set.

You will notice that several of the reports have green check marks in their leftmost column. These are placeholders for a variety of status icons that will be added in a future build. Status icons will allow a user to tell at-a-glance if an activity is running, has generated an error or warning, etc.

Adding Clients
Neither Retrospect 6.1 for Mac nor 7.6 for Windows do a very good job today of dealing with long lists of network clients. They only provide alphabetical sorting, and they don't really provide any information other than client name and computing platform. As you can see in this screenshot, Retrospect X will not only show a variety of information for each client computer, but the view can be sorted by any column. Here, we're sorting by computing platform as an example.

New to the Mac with Retrospect X is the ability to define multiple network interfaces, so you could have, say, one set up for either side of a firewall, or you could keep backup traffic from crossing subnets.

You'll also likely notice that apostrophes aren't yet displaying properly. That's because we're still making significant changes to the UI and engine, but we'll soon begin the work of fine-tuning and refining.

Tagging Sources
The Sources view shows all local volumes, client computers and their volumes, as well as network-attached storage devices or volumes accessed through some file sharing protocol (instead of the Retrospect Client software). Also visible in this view are favorite folders defined by the administrator, which replace Retrospect's "subvolume" moniker. Again, this is a fairly rough view, but there are several key things to point out...

At the very top of the Sources list are several scope buttons: Servers, Desktops & Laptops, Local, Client, and Shares. Clicking these buttons will cause the Sources list to be filtered according to the type of operating system installed on a source's host computer, or according to how the Retrospect server is connecting to that source.

Below the scope buttons are sortable columns, with only a portion of the available columns shown. By control/right-clicking on any column heading, users can turn the available headings on or off to customize the view as needed.

Below the Sources list, in the Options pane, you can see that we've assigned two tags to Robin's MacBook: Building2 and Critical Systems. Tags replace the Source Groups container in previous editions of Retrospect. Source Groups were essentially folders containing aliases/shortcuts to whichever sources/volumes the user assigned. They provided a level of abstraction and an ability to logically group sources for backups. For example, one might have a Notebooks Source Group that contained aliases to all the laptop computers they were backing up with Retrospect, and then they would just point their backup script at the Notebooks Source Group. With Retrospect X, tags provide a more flexible and easily recognizable method for doing the same thing.

Scheduling a Script
This screenshot shows the interface for scheduling a script, the Daily Backup script in this case. Just as with previous versions of Retrospect, scripts can contain multiple schedules and target multiple destination media sets, so you only need a single script for each group of sources to which you want the same schedule to apply

To add a schedule, one simply clicks the + button in the Schedule pane and selects the various options in the Details pane below. Here, our backup will be targeting Test Set A as its destination media set, and no special media action will be performed. Media actions tell Retrospect what to prior to starting the actual backup. "No media action" (like "normal" today) means that Retrospect will start writing to empty media if no backup has yet been performed to the chosen media set, or, if data is already contained in the destination media set, then Retrospect will append to the existing media. Retrospect can also be told to skip to new media, recycle/erase existing media, or to archive the existing media set and start a new one.

Retrospect X gets a calendar that makes it easier to select a script's start date, and it also gains a stop time option, where the administrator can force Retrospect X to stop backing up at a certain time.

I hope that everyone finds these new screenshots and accompanying text helpful as far as seeing where we are with Retrospect X. We certainly appreciate your feedback here on the blog, and we invite you to share your thoughts in the comments section.

The real test will be getting a beta in your hands, which we're working towards as hard as we can. Thanks for reading—I'll post another update soon.

-Eric

A brief Retrospect X update

It has been a little while since my last post. We've all been heads-down working on Retrospect X, and tomorrow, we'll be showing a build (still pre-beta) at Apple's SE Fall Camp, where their sales engineers gather from around the country.

Right now, we're working on the last major elements of the UI, specifically for scheduling scripts (yeah, we kept the name the same as in 6.1) and building rules (formerly selectors). To say that the new UI for these tasks will be much improved over today's Mac and Windows versions of Retrospect is an understatement. I should be able to post some new screenshots by Friday.

We've also started to gather early performance data, and the numbers that Retrospect X is putting up are encouraging. With the architecture changes, improved throughput to devices was pretty much a given, but it's nice to see the proof in the pudding, so to speak.

One area where we continue to be challenged is with support for PowerPC Macs. Just to be clear, the Retrospect Client software will still support PowerPCs and even Macs running Mac OS 9, but the new UI and engine are Intel-only at this point in time. We think we have PPC support pretty well figured out, but it may be that we roll out PPC support shortly after shipping an Intel-only initial release. To do so would not be our first choice, but we'll have to see how it goes.

Thanks, all. I'll be back Friday with some cool, new screenshots.

-Eric

RetroSpeak (or, "What the heck is a Subvolume?")

I've been up to my eyeballs in Retrospect terminology this week, as we're making decisions on what specific terms will be used in the UI and user documentation. Retrospect uses...uncommon terminology for some things, in both the Mac and Windows versions. In fact, Retrospect uses so many unique terms, or different meanings for common terms, we jokingly refer to the use of such as "RetroSpeak".

We keep several goals in mind as we go through the terminology process:

1. Choose explanatory/descriptive terminology whenever possible.
2. If a new term is not significantly better than the current term, we probably shouldn't make the change.
3. Avoid changing terms that will confuse/hinder existing Retrospect customers.
4. Use generic terms that will work across Windows, Mac, etc.
5. Call out Retrospect's technology, but don't overdo it.

These are good guidelines, and they definitely help us with most decisions, but some aren't as clear as others, and external input is just the thing we need right now.

Script/Job/Task
Today, in both the Mac and Windows versions of Retrospect, a saved backup/duplicate/restore/whatever operation is called a "script". A script includes one or more sources, destinations, options, and usually at least one schedule. While Retrospect is designed for use primarily at small and medium businesses, it is often used by individuals who aren't IT pros, and the term script can be intimidating (i.e., some people relate that to a scripting language like AppleScript or VB). Most other backup software calls these saved operations "jobs", but I personally have some sort of allergy to that term, and just try to come up with an icon for it.

The term that seems to make the most sense is "task", but I don't think it passes Rule #2 above. Your thoughts?

Proactive Backup/Backup Server/Policy Backup
One of Retrospect's cool technologies that has been around for a long time is its ability to go down a list of sources and back them up to available media based on priority, with the least-recently backed up sources having the highest priority. Version 6.1 on the Mac calls these "Backup Server" scripts, while they're called "Proactive Backup" scripts in version 7.6 on Windows.

We've put some time and effort into branding/marketing Proactive Backup, and it's certainly more descriptive than Backup Server, but "Policy Backup" or "Policy-based Backup" might be even more descriptive. Still, we're leaning towards using Proactive Backup. Your opinion?

Thanks for the feedback, and please keep it coming!

Happy Halloween, all.

-Eric

Testing a fix for Mac clients turning themselves off

Just a quick note to let folks know that we've been testing a fix for our #1 bug, where Mac Retrospect Clients turn themselves off for no apparent reason, and we hope to make new client and app builds available in the next few days.

-Eric

Thank you, commenters!

To those of you who have commented on the blog: Thank you. Thank you VERY much. Although we haven't had much time to reply to comments, we read (and discuss) every one. We thrive on your feedback and the critical influence it has in Retrospect's development.

Please keep the comments coming, and thanks again for sharing your opinions and ideas.

-Eric

Scheduling backups in Retrospect

Retrospect uses the term "script" to refer to a saved set of rules that apply to a backup (or a duplicate, transfer, restore, etc.) operation. A script tells Retrospect what, where, how, and when to execute the operation. Right. That's all very straightforward.

Where Retrospect is different from pretty much all other backup software, though, is in the area of scheduling. Unlike in most other products, a single Retrospect script can have multiple schedules that each specify the type of action to take and the action's destination. For example, a single nightly backup script might contain all of the following "schedulers" (a term that makes me, and quite possibly some of you, cringe):

• Every other Fri at 10:00 PM, Recycle Backup, Tape Set A
  
• Every other Mon-Thu at 10:00 PM, Normal Backup, Tape Set A
  
• Every other Fri at 10:00 PM (offset +1 week), Recycle Backup, Tape Set B
  
• Every other Mon-Thu at 10:00 PM (offset +1 week), Normal Backup, Tape Set B
  
• Every 12 weeks on Fri at 10:00 PM, New Media Backup, Tape Set A
  
• Every 12 weeks on Fri at 10:00 PM (offset +6 weeks), New Media Backup, Tape Set B

Most other backup software products would require six separate backup jobs to perform the same actions. While one could argue that Retrospect's method can be confusing to set up, it's certainly easier to make changes to sources, destinations, selectors/filters, etc, when you're dealing with just a single script. On the other hand, making schedule changes—especially with the current interface—can be more difficult.

One of our goals with Retrospect X is to make scheduling easier and more self-explanatory. To that end, we'd love to hear your comments on scheduling Retrospect scripts. Do you find the concept of multiple schedules per script easy to manage, or would you create six separate scripts, each with a single scheduler (D'oh!), to accomplish the tasks in the example above? We'd love to hear your thoughts.

-Eric

We're working with Iconfactory!

If you saw the early screenshots of Retrospect X that I posted a couple of months ago, you likely noticed our placeholder icons. As a UI element, icons play a huge role in Mac apps. Finder icons in Leopard scale up to 512x512 pixels, and even toolbar icons can be gorgeous.

When we had our first UI feedback meeting at Apple, we showed a Retrospect X interface where all the icon placeholders were Indiana Jones icons from Anthony Piraino and the Iconfactory. It was pretty funny, actually, even though it made no sense whatsoever.

We definitely know how important it is to have good-looking icons in Retrospect X that best represent the concepts and information they need to convey to the user, so I am pleased to say that we are going to be working with the Iconfactory's design team to come up with our final artwork. They do great work, and we're thrilled to have their help.

Expect to see some of their work here in the coming weeks.

-Eric

A great day at Apple

Yesterday, JG Heithcock (director of engineering and primary software architect for Retrospect), Brian Dunagan (lead Retrospect UI engineer), and I (lead Retrospect windbag...er...product manager) had our third UI review meeting at Apple. While the exact details of the meeting are confidential, I can say that Apple's response to the latest iteration of Retrospect X's UI was highly favorable. As always, we walked away with some gems of advice, but the tweaks were only minor this time.

I think you're going to like what we've done, and I can hardly wait to show you. We'll get some new screenshots posted soon.

-Eric

A milestone achieved

We've been so busy that I almost completely forgot to let everyone know about an important milestone that we reached a little over a week ago. It was only when the photo shown below arrived in my inbox this morning that I remembered this important achievement.

One of the goals we set at the beginning of the summer was to have Retrospect X far enough along that we would be able to show it at Apple Expo in Paris. Well, there in the photo is our very own Michael Schneider, manning the Retrospect X station in Iomega's booth at Apple Expo. Unfortunately, the technical requirements of running the demo were a little high, and it ultimately became easier to show slides instead.

One really cool thing is that the new, native Mac UI was connecting to and controlling the new Retrospect engine running behind the scenes in a Windows XP VM in VMware Fusion.

We're thrilled to have hit this milestone, and we're well on our way to reaching the next. More on that soon.

-Eric