Downloading iCloud Photo Library to your Synology for Backup

Thanks to Tom Insam for the inspiration

What are we attempting?

This takes all the photos out of your iCloud Photo Library and syncs them to your Synology. You end up with exactly the same photos on your synology volume as you do in iCloud. It’s important to note that this is a sync — if you delete a photo from iCloud, it’ll also delete from your Synology. This is done by watching the ‘deleted’ folder in iCloud, so if a photo goes missing without appearing in your deleted folder it will remain.

Updated August 11, 2023: I’ve modified this guide to use a virtual environment instead of installing as root. It should run much more reliably and be less likely to break with DSM updates now. You also no longer need sudo.



Why would I do this?

Having your entire photo library in iCloud is a mild risk. It means your photos aren’t stored in a recognizable, hierarchical folder format and your access to the library depends on Apple continuing to offer a reliable service. Any cloud storage that syncs comes with a risk that all your photos could disappear suddenly — this prevents that from happening. If you’re a digital pack rat with every photo you’ve ever taken, this is for you.



Other notes from me:

  • For even more absurd redundancy, you can use Synology’s Glacier backup to copy your entire photo library up to AWS Glacier for cheap, offsite replication.

  • These instructions assume you’re using a Mac, but Windows users should be able to accomplish this with WSL (Windows Systems for Linux) terminal commands.

  • You probably shouldn’t empty the deleted photos folder more frequently than you run this script or it will leave photos on your Synology that you’ve removed from iCloud



Ingredients

  1. iCloud Photo Library

  2. Synology with plenty of extra storage space

    1. Recommended: Not a slow CPU

    2. DSM 7.0 or later

    3. Your Synology’s local IP or domain (something like <synologyname>.local)

    4. Your Synology username and password

  3. Familiarity with ssh, the terminal in general, and other computer inner workings



Instructions

0) Before you Begin

  1. Allow SSH through your Synology firewall (only if your firewall is on)

    1. Open the Control Panel

    2. Go to Security > Firewall > Edit Rules

    3. Make sure ‘Encrypted Terminal Service’ appears on one of the entries with an action of ‘allow’

  2. Turn on SSH

    1. Open the Control Panel

    2. Go to Terminal & SNMP

    3. Check ‘Enable SSH Service’

    4. Click ‘Apply’

  3. SSH to Synology from your local machine

    1. Open Terminal on your computer

    2. Enter the command `ssh <synologyusername>@<synology domain or IP>’ as in ‘ssh sam@harddrive.local'

    3. Enter your synology password when prompted

    4. You’ll see a slightly different command prompt

1) Install and set up Python

  1. It’s probably already done for you!

    1. Check that it is by typing python -V into your terminal and pressing enter

    2. The result needs to be at least 3.6 (Mine was 3.8.8 which I believe is included with DSM 7.1)

  2. Create a virtual environment in which to do all of the further work. This effectively sections off the python configuration being used for this project from anything else on your Synology, making sure different work you (or DSM) does won’t impact the iCloud Photos work. It’s recommended

    1. Change to your user’s home directory by running cd ~

      1. Create a temp directory for use later by running mkdir $HOME/tmp

    2. Create the virtual environment using python3 -m venv icloudpd

    3. Navigate into that folder with cd icloudpd

    4. Activate the virtual environment by source bin/activate

      1. Note: If you experience any disconnections after this point, re-activate your virtual environment using cd ~/icloudpd;source bin/activate

  3. In order to install more Python packages, we need to install an up-to-date version of pip, the Python package manager

    1. I followed this great guide to enable pip

    2. Run the following commands in your terminal

      1. python -m ensurepip

      2. python3 -m pip install --upgrade pip

  4. The order of this step is very important — we need to install WHEEL first

    1. Run the following commands in your terminal

      1. pip install wheel

      2. pip install --upgrade wheel

2) Install the iCloud Photo Downloader

  1. icloudpd is available on GitHub

  2. Run pip install icloudpd

3) Set up iCloud authentication

  1. Run icloud —username <youremailaddress>

  2. Enter your real iCloud password when prompted — app-specific passwords do NOT work because it is the API

  3. Two factor users will get prompted for your code. For some reason this defaults to the SMS fallback (if you’re still not using MFA on iCloud, you really should be)

4) Get your Synology ready

  1. Return to the DSM web user interface but do not disconnect your terminal yet

  2. Create a dedicated folder for your photos

    1. Copy the path to your folder out to a note somewhere and save it for the next step

    2. Remember that you need to add /volume1 (or a different number if you have more than one volume) to the beginning of your path

5) Run some tests

  1. Return to your terminal. If your computer went to sleep at any point, you may have to ssh to your Synology again. Remember to reactivate your virtualenv

  2. Tell your subsequent work to use that temp directory we created earlier. If you don’t do this, everything will break. export TMPDIR=$HOME/tmp

  3. Run icloudpd -d <path_to_your_folder> -u <iCloud_username> --recent 10 --only-print-filenames --threads-num 2

    If it fails — if you have spaces in your path, you’ll need to enclose in single quotes

  4. See if it succeeds (it should print 10 filenames)

  5. Try downloading a few dozen photos

    1. Run icloudpd -d <path_to_your_folder> -u <iCloud_username> --recent 50 --set-exif-datetime --auto-delete --threads-num 2

  6. Check your folder using the Synology file browser to see if the photos are there and whether they open correctly

6) Set up scheduling

  1. In the Synology web UI, open Synology > Control Panel > Task Scheduler

  2. Create an upgrade script to keep the iCloud Photos Downloader up to date

    1. Click Create > Scheduled Task > User-defined Script

    2. Fill out each tab as follows:

      1. General: Give it a name, run as your user

      2. Schedule: your preference, I chose once a month

      3. Task Settings

        1. Notification: Your email here, only if abnormal termination (if you want)

        2. Command: cd icloudpd; source bin/activate; python -m pip install --upgrade icloudpd

          Note: This is actually three commands in one — it changes to the icloudpd folder that we installed the python package in, activates the virtual environment, then does the upgrade

    3. Highlight the resulting entry and click ‘run’

    4. Click ‘Action’ > ‘View Result’ and ensure that the Current Status is Normal (0) or Requirement already satisfied…

  3. Set up the script itself

    1. Click Create > Scheduled Task > User-defined script again

    2. Fill out each tab as follows:

      1. General: Give it a name, run as your user

      2. Schedule: Your preference, I chose daily at 4am

        1. Note that this may occasionally take a long time to run, frequencies <1 day aren’t recommended

        2. Because of the way this script tracks (and removes) deleted photos, it must be run at a frequency less than 30 days — deleted photos are pruned from iCloud after 30 days.

      3. Task Settings

        1. Notification: Same as above

        2. Command: export TMPDIR=$HOME/tmp; cd icloudpd; source bin/activate; icloudpd -d /volume1/'Media Storage'/Pictures/'iCloud Photo Library' -u samgross144@gmail.com --set-exif-datetime --no-progress-bar --auto-delete --threads-num 2 --log-level error --recent 10 --only-print-filenames --dry-run

7) (Optional) Save logs which is helpful

  1. Open Synology > Control Panel > Task Scheduler > Settings

    1. Check ‘save output results’ and give it a folder that you are ok with filling up

    2. Click ok

8) Final Test

  1. Highlight the upgrade task by clicking on it and then click Run

    1. Select Action > View Result to see terminal output

    2. Make sure that the current status says ‘Normal (0)’

  2. Highlight the download task and then click Run

    1. This will download your entire iCloud Photo Library don’t do this when you need your bandwidth. My Synology maxed out at about 10Mbps download, a fraction of my available speed

    2. Select Action > View Result to see terminal output

      1. You should see ‘current status: running’ with no data as the output

9) Clean Up

  1. Quit your terminal app

  2. Turn off SSH on the Synology (remember to hit ‘apply’)

  3. Change log level if desired in the task settings by editing your scheduled task and replacing the last word (info) with error

  4. Limit the comparisons to recent photos only if you’d like by adding —-recent 1000 to the end of your configuration (you can change the number if desired)

The Car-Based Truck

A few weeks ago, Steve Troughton-Smith and Guilherme Rambo found a software update that contained the code for Apple’s upcoming iMac Pro. Confirming some earlier rumors, it appears that the iMac will have both an Intel processor, as previously announced, and an A10 — the Apple-designed processor powering the iPhone 7. Inevitably, someone will soon bring up the famous quote from Steve Jobs about how traditional desktop computing is a truck, and iOS devices are the cars that everyone will use from now on. In this case, it seems even more appropriate than most. 

In 1979, AMC introduced the Eagle. It was ungainly, slow, and ugly — yet was a popular car for most of its eight-year production run. Essentially, it was a compact car with four wheel drive and a 3” lift — ordinarily not enough to cover up the underlying flaws inevitable in a car built cheaply by a dying company. But the Eagle itself started a revolution in transportation — it was the first car-based truck.

I can guarantee nobody ever used their Eagle like this

I can guarantee nobody ever used their Eagle like this

Until the Eagle’s introduction, the only way to get a family car that could also handle heavy snows, dirt tracks, and withstand the rigors of even moderate abuse was a heavy, slow, noisy, and crude truck-based ‘wagon.’ Suddenly, it was possible to take your family car to the grocery store, and make it home even when the weather had turned completely sour. This was a revelation at a time when trucks (and their family-wagon derivatives) had barely advanced in the previous thirty years — one of the headlining features of the 1973 Chevrolet Suburban was a fourth door. By taking advantage of the advances that had happened in the car sector, the Eagle provided creature comforts with an acceptable compromise to its off-road abilities. 

A few years later, Jeep (then a sister brand to AMC) introduced the XJ Cherokee which — rather than being based on a previous car body — was purpose-built as a car-truck hybrid. It was built for over 30 years, selling more than 2.1 million units, and created an entire segment of the automotive market that hadn’t existed. Today, a plurality of vehicles sold are crossovers, the name that was coined to represent these blends of heavy-duty and approachable transportation. 

The iMac Pro, operating under this somewhat-tortured analogy, is the AMC Eagle. It’s the first Mac to be car-first, run by the A-series processor which then boots the legacy Intel hardware. For now, it will (like the Eagle) only provide a few benefits to security. Eventually, this will become the way all mainstream computers are built, providing the power of a truck with the ease of use and simplicity of maintenance of a car.  

This iMac, along the Touch Bar-equipped MacBook Pro, may not be a lasting design. Like the Eagle, they might turn out to be a mere hint of the future; with time, they’ll fade into history as the awkward predecessors of a new kind of computer. Like the automotive industry, the line between light-duty, everyday tools and powerful special-purpose machines will begin to blur. Today, you can buy an eighty-thousand-dollar Ford SuperDuty pickup truck with more features than the most luxurious cars from the’90s. Similarly, desktop computing will begin to take on some of the characteristics of mobile devices until it is difficult to tell where one ends and the other begins.

Some Personal News

I'm thrilled to share that I've accepted a position with Oscar Health here in New York, working on their internal product team. Here's to helping build the future of health insurance (sorry, Mom & Dad!) 

 

Hi Oscar!