Readers of this blog will know that backup, or more pertainately, restore has been a regular theme.
I use my laptop everyday for work and when as is the case every few years the hard drive or motherboard fails then it usually costs me several days labour and stress to get up and working again.
Over the years I have tried many backup solutions but the difficulty is always in the restore.
The perfect solution is virtually obvious
In a world where CPU, storage and bandwidth are free I would simply log into a cloudy service which would change its presentation to match my display and input devices. I would never have to worry about backing up data or configuration because my service provider would do this for me. If my endpoint device, laptop or phone, was lost I would simply use another and take up from where I left off.
We don’t live in a perfect world
We are making good progress towards that future but at present it is still convenient to have local CPU and storage. This is because bandwidth varies between quite good and non-existent and remote storage costs are still quite high. In the real world I use my laptop on the train and it goes into tunnels. So we need to consider how to ensure business continuity when using physical computers.
Introducing the “service” paradigm
In the 1980s when I started using personal computers I did not think about “services” I had a computer and it ran software.
Computers were expensive specialized devices. You bought this very expensive bit of kit and looked after it carefully. It had a tiny (10 MB – that is MB not GB) hard disk and you had a variety of tools to care for it. We academics had the internet but no-one else had heard of it. No-one regarded computing as a “service”.
In 2016 things are very different. Computer hardware is cheap and ubiquitous and the concepts of service management are well established.
In 1980 when a computer crashed we would ask “how can we get it working again?”, in 2016 we ask “how can we restore services?”
What application services does my laptop provide?
- Web browsing
- Standard personal productivity services
- Word processing
- Note taking
- Custom software
- UML modelling tool
- Image cataloging
- Software development platform
- Data storage for offline access
What sort of incidents must we protect against?
The most common incidents are hard drive failure, equipment damage and total loss due to theft.
A new threat is losing access to data due to a cryptolocker virus.
How do we restore application services after an incident?
The 1980s approach to this problem is to take a “full backup” or “system image” of the computer each week (or night) and then to do a restore.
Unfortunately this does not work for home users,
It is not possible to reliably back-up and restore a physical computer except in a very well-managed commercial environment that has stock of replacement computers and parts.
A complete backup of a physical computer requires a “system image”. This is a block-by-block copy of each of the hard disks. Fortunately it is easy and relatively fast to create a system image. Unfortunately the restore is very fragile. Any discrepancies between the original source system and the targeted system for restore will result in failure. Discrepancies could include slightly different hard drives or network cards. With new secure boot capabilities there are even more things to go wrong.
Microsoft are so aware of this issue that they removed the System Image capability from Windows 8.1 and although it has returned in Windows 10 it is deeply hidden.
The 21 Century solution is to use different processes to protect user data, service availability and the physical computer. We will see how near-real-time backup and online (cloud) service provision greatly reduce the time that services are degraded.
Protecting User Data
Use a Microsoft Account
This entire article assumes that your Windows 10 user identity is a Microsoft Account not a local user. This is necessary to make all the cloudy stuff work nicely.
Protecting Current Versions
The primary protection for user data is to store it in a well-managed secure location away from the device. This offsite backup is now provided by cloud services such as Microsoft OneDrive, Google GDrive and DropBox. I use Windows 10 and want to stick to inbuilt tools so my rule is that,
- all user data must be stored in OneDrive
Windows 10 protects against accidental deletion of data – fat finger errors, by giving you recycle bin both on your PC and in OneDrive. Items can be recovered from OneDrive for 30 days.
The OneDrive sync client ensures that all files are protected whenever the laptop is connected to the Internet.
Protecting Previous Versions
It is also desirable to protect previous versions of data. For example you may accidentally save “Lecture notes 11” as “Lecture notes 1” thus obliterating the original contents of “Lecture notes 1”.
In the 1980s we would have gone to our weekly system backup and found the file “Lecture notes 1” and restored it.
In the 21st Century Windows 10 provides “File History”. This will store every previous version of a document on an external device. It works transparently with both desktops and laptops.
After you enable File History all previous versions are available through Windows’s explorer. Simply right click on a file or folder to view its properties and previous versions.
File History is an important part of my backup strategy and I don’t want to lose data. It can also grow quite large over time. To address these issues I have purchased a Buffalo Linkstation 520 NAS with two 3 TB disks with RAID 1 mirroring. It only cost £ 180 and it simply plugs into my home router and can be the backup target for all the computers in the house. (If you would like to see my future posts about this device search for “Buffalo”).
- Only use a RAID 1 device as a backup target
It is not possible to protect File History from Cryptolocker attacks so you must also make periodic data backups and put them in safe location on the NAS.
Configuring File History to mitigate Cryptolocker attacks
You need to ensure that if one computer or user is compromised you don’t lose all the data. Use different NAS shares with different usernames and passwords for each computer you want to backup.
- Create a share (different user) for each computer’s File History, eg. OfficeComputerFileHistory, TimsBedroomComputerFileHistory…
Previous Versions and cloud services
The OneDrive service provides users with access to previous versions of Word, Excel and PowerPoint documents. Other providers such as DropBox provide access to previous versions of all document types. Cloud services require the regular payment of a fee and you may wish to change provider at some time.
- Cloud services are a supplement to, not a replacement for Windows 10 File History stored on a NAS.
You should still make off-line archival copies sometimes
It is not easy to protect against Cryptolocker type attacks so it necessary to make offline copies. This can be done using your NAS like this,
- Create a Public share (a NAS will usually have one by default) for moving data around on the NAS.
- Create a share called “Offline”
- Turn off all sharing including SMB, FTP, etc
- Copy your most valuable data such as photos to your Public share and then use your NAS’s web user interface to move this data to your Offline folder.
You may also wish to periodically take a system image or simple copy of your data and put it in your Offline folder.
System Image – Cheap, dirty and quick
Windows 10 still offers us the opportunity to create a System Image. Although we can’t rely on this 1980’s technology to restore a computer it can provide very quick data backups. The reason for this is that it performs a block level backup. Therefore although the first backup may take a couple of hours subsequent backups will be very quick. The resulting system image contains a VHD file that can be mounted as a drive (say S:) on any Windows 10 computer to provide file access. Making a system image is a good way to ensure you have an up to date copy of all your drivers.
- Make a System Image backup to your NAS. Update it occasionally.
If you are considering actually using a System Image for restoring a computer in the future then you should not use Windows Backup but Veeam Endpoint Backup. This excellent programme is free and very reliable.
To mitigate the risk of Cryptolocker attacks the backups should be manually copied to your Offline folder.
Protecting your software development environment
As an application developer I have a large number of applications and utilities that I install to make my life easier. Although in future I expect Chocolaty (1,2) to make this easier at present rebuilding a development environment is a real pain. In my opinion all development environments should be virtual machines and backed up using simple file copies to the NAS. I use VirtualBox rather than Hyper-V because it is available for all platforms.
- A suitable computer for development using virtual machines is i5, 8 GB RAM, 256 GB SSD. (I use a Microsoft Surface Pro 4 but more sensibly priced devices are available).*
For backing up the virtual machine Windows 10 System Image would be an excellent choice because it does a block level backup however I prefer to use GoodSync to back-up the VirtualBox machines directory because it has can be easily configured to run on a schedule and send an email if the job fails.
Amazingly I found that to back up a Windows 10 virtual machine a second time only took 1 min 40s. This is because I had created a snapshot and VirtualBox is incorporating my changes into the small snapshot file rather the the whole disk.
- Backup by copying the entire Virtual Machines directory to the NAS
To mitigate the risk of Cryptolocker attacks create a share on the NAS that is only accessible by FTP and then use GoodSync’s FTP capability to backup to it.
To prevent confusing the consumer sync applications
- Do not put your virtual machines in the OneDrive directory.
- Exclude the virtual machine directory from File History.
* Hyper-V and VirtualBox performance
Hyper-V and VirtualBox on laptops are Type 2 Hypervisors they benefit from the host operating system providing access to the underlying hardware such as sound and graphics cards. Because of this they cannot use all the host’s resources and you need a powerful computer to run them. It would be nice to boot your laptop to a type 1 hypervisors or “bare-metal” hypervisors that would then load your virtual machine of choice. Unfortunately writing and maintaining such an operating system is so hard, essentially that is what Windows or Linux vendors do, that no-one makes bare-metal hypervisors for consumer kit.
How do we restore application services?
Let us consider how I restore services in the scenario where I am using OneDrive and Office 365.
- Data storage for offline access
- Login to any other Windows 10 laptop using my Microsoft Account and sync the data that I need.
- Web browsing
- I don’t need my laptop to browse the web. I can use any computer or my phone. If necessary bookmarks can be synced using a suitable service.
- I don’t need my laptop to manage email. Although I like using the Desktop Outlook client I can access my email and calendars at Outlook.com.
- Standard personal productivity services (Microsoft Office)
- I don’t need my laptop to access or create Microsoft Office documents. I can either use the online versions or because I have an Office 365 license I can download and install the desktop applications on any other Windows 10 computer.
- Custom software (UML Modelling Tool, Image Cataloging, Software development)
- Please see below
We can restore our virtual machines by installing VirtualBox and copying them back from the NAS but this is not applicable to all tools we might want.
At best a Hyper-V or VirtualBox PC will give you 50% of the performance of the host*. There is some software such as Photoshop that it does not make sense to run in a virtual machine and other software such as Notepad++ or SnagIt that you just want to hand. Until these installs can be scripted using Chocolaty or Powershell they will need to be done by hand. The pain can be mitigated by keeping distributions and license keys in your OneDrive directory so that they are immediately available.
Errrm – What about fixing the computer then?
So far none of my “restore” strategies have actually required me to fix the broken device. This is a consequence of my emphasis on restoring access to data and services in a era of data and device ubiquity. The actual hardware you use is almost irrelevant. If your old hardware has seen its best days and you want something with bells and whistles or a touch screen and pen like my Surface 4 Pro you may wish simply to buy another device. Using the methods outlined about you can get it up and running in a couple of hours.
However given that you paid good money for your old kit it now time to fix it.
Preparations before it goes wrong
- Turn on System Restore and create your first restore point
- This creates restore points on your device which are triggered when major update are performed. This is great for reverting after a bad driver update.
- Create a Windows 10 Recovery Drive
- This contains copies of your key operating system files and drivers. I recommend that on devices with more than 32GB of storage you do not delete your recovery partition. (update 2016-02-12 Windows 10 is different from earlier versions. You should never manually delete the recovery partition on a working system)
- Get a copy of the Windows 10 distribution and store it on your NAS
- This is sometimes necessary if you are to restore corrupt files
- (Microsoft Surface users can download special recovery media with all the drivers in it.)
How to fix a corrupt Windows 10 Operating System
This post details a quick, brute force solution you may wish to try before doing the reset that Microsoft recommend below.
Fix the image using SFC /SCANNOW and DISM
Here is the official and sensible Microsoft advice from Recovery options in Windows 10
If you’re having problems with your PC, the following table can help you decide which recovery option to use.
|Problem||See this section|
Your PC isn’t working well and you recently installed an app, driver, or update.
Restore from a system restore point
Your PC isn’t working well and it’s been a while since you installed an app, driver, or update.
Reset your PC
Your PC won’t start and you’ve created a recovery drive.
Use a recovery drive to restore or reset your PC
Your PC won’t start and you haven’t created a recovery drive.
Use installation media to restore or reset your PC
Your PC won’t start, you haven’t created a recovery drive, and resetting your PC didn’t work.
Use installation media to reinstall Windows 10
- Restore from a system restore point
- Reset your PC
- Use a recovery drive to restore or reset your PC
- Use installation media to restore or reset your PC
- Use installation media to reinstall Windows 10
Configuring a new or fixed computer
If you have replaced a major system component such a disk or motherboard the simplest and most reliable method is simply to reinstall Windows using one of the options above. If you have just changed the disk to a larger one and you have a recent System Image you might want to try using it but don’t expect it to work.
Having reinstalled windows (or purchased a new machine)
- Login with your Microsoft Account
- Turn on System Restore and make your first Restore Point
- Run Windows Update repeatedly until you are up to date.
- Create another restore point
- Install Microsoft Office 365 making careful note of which version you choose.
- Run Windows Update repeatedly until you are up to date
- Connect with OneDrive and sync your data back to the device
- Configure File History – it should pick up your previous archive on your NAS
- Create a restore point
- Install all your custom software
- Create a System Image (optionally use Veeam Endpoint Backup for this)
- Configure your GoodSync jobs if necessary.
Pingback: Better than Apple Time Machine but for a PC | Dr James Bayley
Pingback: Windows 8.1 is Android Kitkat and other thoughts | Dr James Bayley
Pingback: The best Acronis backup scheme | Dr James Bayley
Pingback: Why I removed Acronis True Image 2014 | Dr James Bayley
Pingback: Backing up a virtual machine to the cloud | Dr James Bayley
Pingback: How to add a static IP to a Virtualbox on your laptop | Dr James Bayley
Pingback: How to restore a Surface image | Dr James Bayley
Great blog and article.
You are not using ShadowProtect anymore?
Dr James Bayley said:
Thank you Ondiz, I have stopped using ShadowProtect. It is a really great product and I might well use it again in a commercial environment. It is a powerful back-up image management tool and as I say in this post, image management is not really needed for personal computers used in cloudy way. It is also quite expensive ~ $100. If you do want a traditional backup tool I strongly recommend Veeam Endpoint Backup, this is free and rock-solid. This does not offer you the image management tools that Storage Protect has but as I said you should never need them.
For private use I’ll evaluate Veeam.
For production Environment I’m testing Shadow Protect, Acronis and BAckupAssist…
Pingback: Adding a MicroSD card to a Surface Pro 4 | Dr James Bayley
Pingback: Playing Ark Survival Evolved on a Microsoft Surface 4 Pro | Dr James Bayley
Dr James Bayley said:
I removed Acronis.(https://blog.jamesbayley.com/2014/03/10/why-i-removed-acronis-true-image-2014/)
If your production environment is virtualised then you may wish to consider the Veeam products which are very good.
Dr James Bayley said:
I have done a major update to this article detailing the correct NAS configuration to protect against Cryptolocker.
Pingback: New year – annual backups | Dr James Bayley
some dude said:
“definitive” is not spelt correctly.
Dr James Bayley said:
Thank you dude, I have corrected the title.
I have used “previous versions” for years as my first-line backup so I could restore files quickly from the local drive. Unfortunately, this feature is unavailable in Microsoft’s new storage tool: Storage Spaces. I have a tiered & mirrored storage space in my PC (4 drives) that is both fast and fault tolerant. The negative is that version control now needs to be done elsewhere. I’m still evaluating options to do reliable version control.
Dr James Bayley said:
Thank you for this very interesting update. I have been thinking about the benefits of software RAID for fault tolerance on home computers this week. I was wondering if the benefits outweighed the complexity.
I am wondering if this might not be a Storage Spaces issue. Rather in Windows 10 there is no ability to make volume snapshots to create Previous Versions, this feature has been replaced with File History – thereby moving backup to an external drive. Do you think I am right?
Win10 has “previous versions” just like in older versions of Windows. I use it on my other volumes but can’t enable it on my storage spaces volume.
Historically, I have used hardware RAID as it’s much faster. However, storage spaces has the ability to set up tiered storage which was not possible with with Microsoft’s software RAID. I now a have tiered and mirrored storage space with 2xHDD and 2xSSD which gives me a 6TB “drive” in windows that is almost as fast as an SSD and has fault tolerance.
Dr James Bayley said:
Wow. That is really cool.
Yes, it is cool but it’s not for the average home user unfortunately. In Win10, MS gives you a GUI to set up simple storage spaces. That is handy but the true power for me is the tiered storage. That can only be set up with Powershell commands which is daunting for anyone not familiar with it. However, the benefit so far has been awesome, I have ran benchmarks on my SSDs (NVMe and SATA), HDD, mirrored storage space, and tiered/mirrored storage space. The results are impressive.
Pingback: [Fixed] Error Using Veeam Agent for Windows Free with a NAS | Dr James Bayley