Backing Up a Synology NAS to a WD My Cloud NAS

The built-in Synology Backup & Replication feature has not been very reliable for me, prompting me to explore other options.  Since the underlying backup process was rsync based, I kept with the theme and rolled my own script; really convenient since I did something similar in ESXi years ago with the ‘poor-mans VMotion’.

TL;DR

The script below does have a few requirements:

• You need rsyncd running on the remote host
• The poor man’s script will need:
  • The volume where your data resides (probably /volume1)
  • The rsync user you setup on the remote host
  • The remote host be it via IP or hostname
  • The remote path to store your data
  • Your backup jobs at the bottom in the form of:
    StartBackup ‘sharename’
• Please note there’s a Dry-run enabled by default and it will need to be removed when you’re ready to execute for real. (so delete -n in the rsync command line)

Setup rsync on WD MyCloud

The steps below require a combination of access to the WD MyCloud web portal and ssh access to the box.  For setting up ssh I’ll refer you to the Internets (aka search for it) and WD:

• How to enable SSH (Secure Shell) on a My Cloud (single bay) device
• How to enable SSH (Secure Shell) on a My Cloud EX4 device

Two big things I want to point out if you don’t already know this.

1. You could seriously break something so be careful.  I recommend starting with a clean slate on your WD or at least backing up whatever data you have on there some place else in the event you break something.
   .
2. Linux/Unix is a case SeNsItIve world.  Everything from file, to user names to directory names and commands are all case sensitive.  So where possible, error on the side of caution and keep everything in lowercase to be sure.

Create a New Share via the WD MyCloud Portal

Login to your WD MyCloud and create a new share (e.g.: synologynas) specifically for this purpose.

Create an rsync.secrets file in /etc

Keep it simple like rsyncd.secrets in /etc

synologynasuser:super sekret 1337 passw0rd!

Create an rsyncd.conf file in /etc

pid file = /var/run/rsyncd.pid
lock file = /var/run/rsync.lock
log file = /var/log/rsync.log

# This is the rsync module name you're going to access
[SynologyNAS Share for rsync]
comment = "My rsync share for Synology"
# This is the local path on the WD MyCloud
path = /shares/synologynas
# Or if you prefer
#path = /DataVolumes/shares
use chroot = true
uid = root
gid = share
read only = no
list = yes
# add the user you specified via the WD MyCloud portal
auth users = synologynas
# put the path to your secrets file here
secrets file = /etc/rsyncd.secrets
# OPTIONAL But Recommended: Put the IP of your Synology NAS here & uncomment (meaning remove the # in front of the line below)
#hosts allow = Synology.NAS.IP.ADDRESS

Enable rync in /etc/default/rsync

Edit /etc/default/rsync and change RSYNC_ENABLE=false to RSYNC_ENABLE=true.

Start rsync

You could reboot your WD MyCloud but since you already have a shell open just start or restart the service

# Check to see if the service is running

/etc/init.d/rsync status

# If not running, start it

/etc/init.d/rsync start

# If it is running, reload it

/etc/init.d/rsync reload

# or force-reload it but you shouldn't have to

/etc/init.d/rsync force-reload

# or just restart it

/etc/init.d/rsync restart

At this point, rsync is setup and ready to receive connections.

Bonus: Create a New User via the WD MyCloud Portal

This is more of an optional step.  I spend a lot of time in DSM so I decided to mount the remote CIFS folder I setup on the WD MyCloud so I could access it in DSM.  To do that, login to your WD MyCloud and create a new user specifically for this purpose and grant that user full rights to the share you setup above.  To keep things simple, you could use the same username and password you setup above in the secrets file.

The Script

I tested this script in DSM 5.2 through DSM 6 Beta 2 and an early build of DSM 6 when it went GA.

Keep in mind:

• I’m no rsync or [ba|k]sh, guru but it works perfectly for my needs.
• Minimal error checking in this version
• You’re free to edit to suit your needs. (obviously)
• The rsync options are what Synology uses by default for its rsync-based backup tasks; feel free to alter as they’re not necessarily ideal.

#!/bin/sh

Volume=/volumeN
RsyncPasswordFile=/path/to/passwordFile
RemoteUser=synologynas
RemoteHost=127.0.0.1
RemotePath=Backups

LogDir=/var/log
LogFile=$LogDir/MyCustomBackup.log

echo "vvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvv" >> $LogFile
echo "Started Main Script" >> $LogFile

StartBackup() {
 if [ -z $1 ]; then
      echo "ERROR 1 IS EMPTY [$1]" >> $LogFile
      return
 fi

 local SourceDir=$1
 echo "SourceDir is [$SourceDir]" >> $LogFile

 local DestinationDir=$SourceDir
 echo "DestinationDir is [$DestinationDir]" >> $LogFile

 if [ ! -d $Volume/$SourceDir ]; then
      echo "ERROR: SourceDir [$SourceDir] was NOT found" >> $LogFile
      return
 fi

 local JobLog=$LogDir/MyCustomBackup-$(echo $SourceDir | sed -e 's/\//_/g').log
 echo "Rsync log is [$JobLog]" >> $LogFile

 echo ">>>>>> Starting Backup Job for [$SourceDir] to [$DestinationDir]">> $LogFile

 # Remember - dry-run is enabled by default
 /usr/syno/bin/rsync --timeout=600 -rlt -p --chmod=ugo=rwx -H -W --password-file=$RsyncPasswordFile $Volume/$SourceDir $RemoteUser@$RemoteHost::$RemotePath/$DestinationDir --exclude=/*/#recycle/ -Phriivv --stats --log-file=$JobLog -n

 echo "Finished Backup Job for [$SourceDir] to [$DestinationDir] <<<<<<" >> $LogFile
}

StartBackup 'photos'
StartBackup 'music'
StartBackup 'video'

echo "Finished Main Script" >> $LogFile
echo "^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^" >> $LogFile

I have customized this script further because I’m a crazy man, so if you’re interested in seeing that version, let me know and I’ll put up a link here.

Good Providence to you!

How Did I Get Here?

I have an aging, but still fast & reliable, Synology DS411+ii that we, and several members of our family, rely on.  I won’t list all the features, just know that it does a lot and is capable of much more.  As with most NAS, a built-in backup feature exists so you might be wondering why I’m not using that.

Synology does have a ‘Backup and Restore’ application package that can be used to backup data to a variety of sources:

• Amazon S3
• Microsoft Azure
• CHT hicloud
• SFR NAS BACKUP
• Another Synology
• An rsync-compatible server
• Local storage
• External storage
• Network share

And that is where I began.  I created a new share on the WD My Cloud, added a user and gave them rights to said share, setup rsyncd on the WD My CLoud then added the WD My Cloud as ‘an rsync-compatible server backup destination’ on the Synology as part of the backup task.

However, although my initial test backup completed successfully, after creating 6 or 7 other backup tasks (basically one for each share on the Synology) the results were hit or miss:

• One or two data backup task seemed to always complete successfully
• Most data backup tasks seemed to fail most of the time but succeeded on the second or third attempt
• Some data backup tasks fail more often than they succeed
• One or two data backup tasks always fail; never completed successfully once

I thought maybe I screwed something up so I deleted all the data backup tasks that were failing and re-created them one at a time, testing between each.  I initially had better success the second time around so I pressed on for a few kept failing.  Thinking maybe there was something wrong with the existing backed up data on the WD My Cloud I deleted everything and recated.  Still, a couple tasks wouldn’t work, and after a few days things that were successful were now failing.  Bonkers.

Since the WD My Cloud isn’t exactly a workhorse, I looked there thinking it was struggling, and aside from the indexing service bogging the system, which I nerfed:

for i in wdphotodbmergerd wdmcserverd
do
     for j in $( ls /etc/rc2.d/*$i ); do
          $j stop
     done
     update-rc.d $i disable
done

There were no logs that suggested rsyncd or the system were the source of the failures.

I grabbed my shovel and went back to the Synology to dig into some logs and the backup process as a whole.  Unfortunately very few logs are created, the logs contain minimal information and Synology has obfuscated the backup process by using custom scripts and binaries making it impossible (for me anyway) to alter the rsync command line to create verbose log files.

It’s been well over a month now, the Synology backup process runs nightly, the failure rate is still no better and I’m getting worried: Back at the end of November I ran into a series of unfortunate events that resulted in nearly complete data loss:

• A drive died in the NAS – no big deal because RAID5
• The next day I lost power ONLY in the room where the NAS was located and when it came back up the volume couldn’t be mounted (bad superblock) – UPS was not connected (shame on me) because I was in the middle of a move
• A few days after that, the external drive where a subset of the data on the NAS was backed up to died – no idea what happened there!

Thanks be to God, I was able to get everything restored with zero data loss!  I was able to fix the superblock error on the NAS, mount the volume read-only on the NAS and restore the data.  Since I had about 5TB of data on the NAS and only 1-something TB on my Server, I needed a place to back it up to.  Best Buy had a 4TB WD My Cloud NAS on sale for around $170 which was about the same price as a 4TB drive – score!

Once I backed everything up to my Server and the WD My Cloud I recreated the volume, restored the data and re-evaluated my backup strategy which is what prompted me to look at the ‘Backup and Restore’ process.  Not wanting to wait for another emergency, I took matters into my own hands and rolled my own rsync shell script above to backup the NAS when the built-in process was failing.

My custom backup script has been been running nightly for the past month and, as far as I can tell, it works every time.  I’ve spot checked the data as well as the logs to verify the process completes successfully, which it does, giving me peace of mind.

Synology is on the cusp of releasing DSM 6 and I hope they’ve improved the backup process so I can simply use that.  I’m considering getting a DS1515+ (or the 16 series equivalent) so maybe I’ll be able to tap into the Synology-to-Synology backups and block-level replication.

Good Providence (again) to you!