Migrating a large Google Photos library with a Cloud Server
Google Takeout hands over every byte. Turning those bytes back into a photo library is the part nobody warns you about. Leaving a photo service takes more than a download: It is a small infrastructure project. Mine cost a rented server, several days time and a number of wrong turns.If you are planning the same move, this guide is the wrong turns taken out.
Contents
Step 1 - Prepare and secure the server
Step 2 - Get the export onto the Volume
Step 3 - Unpack the archives
Step 4 - Verify the extracted data
Step 5 - Run the desktop application on a virtual display
Step 6 - Connect from your own computer
Step 7 - Import in batches and verify
Step 8 - Clean up
Introduction
Moving a photo library out of Google Photos is easy to start and hard to finish. Google Takeout will export everything, but what arrives is a set of large ZIP archives in which each image is separated from its own metadata. Capture date, album membership, geolocation and description live in a companion JSON file next to the picture, not inside it. If those companion files are lost or ignored, the photos arrive at the destination dated on the day of the import, and the chronology of the library is gone.
For a small library you can do all of this on your own computer. For a large one you cannot. A 300 GB export means you need roughly twice that in free disk space to unpack it, days of sustained upload, and a machine that stays powered on and busy for the whole time. A laptop on a domestic internet connection is not suitable equipment for that job.
This tutorial uses a Cloud Server as a temporary workbench instead. The export is transferred machine to machine, unpacked and verified on a Volume, and uploaded from the data center. Because many photo services offer no command line import and accept uploads only through their desktop application, the tutorial also shows how to run a graphical application on a server that has no screen, and how to watch it from your own computer through an encrypted connection.
The example destination is Ente, an open source, end to end encrypted photo service whose command line tool can export but not import. The same approach works for any destination whose only import path is a desktop application.
At the end you will have the full library in the destination service, a verified count of what arrived, and a cold copy of the original archives that you can keep or discard.
Prerequisites
A Hetzner Cloud Server - it’s my personal go-to, but other providers will also work well. For a library of a few hundred gigabytes, plan for at least 4 vCPU and 8 GB RAM. The upload is limited by client side encryption, which is processor bound, and the desktop application needs memory to index tens of thousands of files.
A Volume with at least twice the size of your compressed export, because you will hold the archives and the extracted files at the same time.
Optionally a Storage Box as cheap cold storage for the original archives.
An SSH key on the server.
An account with the destination photo service, and enough quota there for the whole library.
A VNC viewer on your own computer. On macOS, Screen Sharing is part of the system. On Linux, Remmina or TigerVNC will do.
Will and endurance to fix all the little problems that may occur - you will not stay on the happy path only.
Example naming
Username: holu
Server IP address: <10.0.0.1>
Volume mount point: /mnt/volume1
Storage Box user and host: <u123456>@<u123456>.your-storagebox.de
Step 1 - Prepare and secure the server
Do all of this before any data reaches the machine. Later in the process an import will be running inside a long lived terminal session, and at that point a reboot costs you the entire indexing run, while installing packages replaces libraries that the running application has already loaded. Updating and hardening first is not just good practice here, it is the cheapest moment to do it.
Bring the system up to date, and reboot if the update asks for it:
sudo apt update && sudo apt full-upgrade -y ls /var/run/reboot-required 2>/dev/null && sudo reboot
Filter incoming traffic on two levels. A Cloud Firewall works in the network before packets reach the server, which means it also protects any service you might expose by accident. Configure it in the Cloud Console with a single inbound rule for TCP port 22 and apply it to the server.
On the server itself, ufw acts as a second line:
sudo apt install -y ufw sudo ufw default deny incoming sudo ufw default allow outgoing sudo ufw allow 22/tcp sudo ufw enable sudo ufw status verbose
Three things to keep in mind:
Do not add a rule for the VNC port 5900 on either level. Step 6 forwards it through the SSH connection, which is precisely why it never has to be reachable from the internet.
The server has a public IPv6 address as well as an IPv4 one, so make sure your rules cover both.
Outbound traffic is unrestricted by default on both levels. If you decide to define outbound rules, the upload needs TCP 443 and name resolution on port 53, otherwise it will stall without an obvious error.
Now attach the Volume in the Cloud Console and note the mount point it gives you. The rest of this tutorial assumes /mnt/volume1.
The graphical application should not run as root. Electron based applications refuse to start as root unless you disable their sandbox, and there is no reason to give an import job administrative rights. Create a normal user and give it ownership of the working directory:
sudo adduser holu sudo mkdir -p /mnt/volume1/takeout /mnt/volume1/extracted sudo chown -R holu:holu /mnt/volume1/takeout /mnt/volume1/extracted
Add swap as a safety net. It will not make the import fast, but it prevents the kernel from killing the application if memory use spikes while it indexes the library:
sudo fallocate -l 8G /swapfile sudo chmod 600 /swapfile sudo mkswap /swapfile sudo swapon /swapfile echo ‘/swapfile none swap sw 0 0’ | sudo tee -a /etc/fstab
Check the result:
free -h df -h /mnt/volume1
Step 2 - Get the export onto the Volume
In Google Takeout, select Google Photos only, and make two choices that matter.
Set the delivery method to Google Drive rather than a download link. Takeout links expire after a few days, and pulling several 50 GB archives over a home connection without dependable resume is unreliable. Delivering to Drive keeps the data on the provider side until you are ready to move it. Be aware that the export occupies storage in your own Google account while it sits there, so you may need free space equal to the size of your library.
Set the archive size to the largest option offered, usually 50 GB. Eight large files are much easier to handle and verify than two hundred small ones.
When the export is ready, transfer it machine to machine. rclone can talk to Drive directly, resumes interrupted transfers and compares checksums, which is what you want at this size. Configure a remote for your Drive account and copy the archives to the Storage Box first, so that you keep an untouched copy:
rclone config rclone copy gdrive:Takeout <u123456>@<u123456>.your-storagebox.de:/home/takeout \ --sftp-port 23 --progress --transfers 4
Then bring them onto the Volume, where you have fast local disk for unpacking:
rsync -av --progress -e ‘ssh -p 23’ \ <u123456>@<u123456>.your-storagebox.de:/home/takeout/ /mnt/volume1/takeout/
If you do not use a Storage Box, copy from Drive to the Volume directly with rclone copy. The intermediate step exists only to give you a second copy that no later command can touch.
Step 3 - Unpack the archives
Unpacking several hundred gigabytes takes hours, and a dropped SSH connection would kill the job halfway through. Run it inside a terminal session that survives disconnection. screen provides this:
sudo apt update && sudo apt install -y screen unzip screen -S unpack
Inside the session:
cd /mnt/volume1/takeout for f in *.zip; do unzip -n “$f” -d /mnt/volume1/extracted; done
The -n flag tells unzip never to overwrite an existing file, which matters because the archives overlap slightly at their boundaries.
Press Ctrl+a followed by d to detach and leave the job running. Reconnect later with:
screen -r unpack
When it finishes, the export will be at /mnt/volume1/extracted/Takeout/Google Photos.
Step 4 - Verify the extracted data
This is the step people skip, and it is the one that protects the library. A missing archive or a truncated file produces no error at the destination. You simply end up with fewer photos and no way to notice.
Move into the export and count what you have:
cd “/mnt/volume1/extracted/Takeout/Google Photos” du -sh .
Count media files against companion files. Note that Takeout names them either image.jpg.json or image.jpg.supplemental-metadata.json depending on when the export was created:
find . -type f -name ‘*.json’ | wc -l find . -type f ! -name ‘*.json’ | wc -l
The number of companion files should be slightly below the number of media files. A large gap means the metadata was lost somewhere, and importing in that state would date every photo to today. A count of only a few dozen JSON files means they are missing entirely.
Get an inventory of file types, which is also a quick check that your raw formats survived:
find . -type f ! -name ‘*.json’ | sed ‘s/.*\.//’ | tr ‘A-Z’ ‘a-z’ \ | sort | uniq -c | sort -rn
Look for truncated files. This must return zero:
find . -type f -size 0 | wc -l
Finally, check the archive numbering. Google numbers the parts of a series consecutively, so a series ending in -007 must contain seven parts. Compare what you unpacked against what was delivered:
ls -1 /mnt/volume1/takeout/*.zip | wc -l ls -lh /mnt/volume1/takeout/
Two properties of the export are worth knowing before you import. A photo that belongs to an album is delivered twice, once in a year folder and once in the album folder, so the total size of the export is larger than the actual library. And Google ships its own edited versions of images as separate files, which the destination will count as distinct photos. Depending on your language setting the suffix is -edited or its localized equivalent:
find . -type f -iname ‘*-edited*’ | wc -l
If you do not want those, move them aside now. Removing them afterwards is considerably more work.
Step 5 - Run the desktop application on a virtual display
The server has no screen, so create one. Xvfb provides a display that exists only in memory, openbox gives it a minimal window manager, and x11vnc makes it viewable over the network.
sudo apt install -y xvfb x11vnc openbox
Install the destination application. A distribution package is preferable to an AppImage here, because the package manager resolves the many libraries an Electron application needs:
cd /home/holu wget https://github.com/ente/photos-desktop/releases/download/v1.7.26/ente-1.7.26-amd64.deb sudo apt install -y ./ente-1.7.26-amd64.deb
Set a VNC password. Some viewers, including the one built into macOS, refuse to connect to a server that accepts unauthenticated sessions. The protocol truncates passwords after eight characters, so keep it short:
x11vnc -storepasswd
Start everything inside a screen session so that the upload survives a dropped connection:
screen -S import Xvfb :99 -screen 0 1920x1080x24 & export DISPLAY=:99 openbox & x11vnc -display :99 -localhost -rfbauth ~/.vnc/passwd -forever -shared & ente &
Confirm that the VNC server is listening on the loopback address only. If it shows 0.0.0.0, it is reachable from the internet, and you should stop it and restart it with -localhost:
ss -tlnp | grep 5900
Detach with Ctrl+a then d.
Step 6 - Connect from your own computer
This is the reason step 1 left port 5900 closed on both firewall levels. Instead of opening it, forward it through the SSH connection, which encrypts the session and keeps the port unreachable from outside. Run this on your own computer, not on the server:
ssh -L 5900:localhost:5900 holu@<10.0.0.1>
Leave that terminal open, then point your VNC viewer at localhost:5900. On macOS, press Cmd+K in the Finder and enter vnc://localhost:5900.
You should see a grey desktop with the application window on it. If the screen is empty, the window may be positioned outside the visible area. Right click the desktop to open the Openbox menu, which includes a window list.
Log in to the service. This is where most people get stuck for a while, so it deserves its own explanation.
The virtual display has no keyboard attached and defaults to a US layout. Your own keyboard sends key positions, not characters, so what appears on screen is whatever a US layout produces at that position. The at sign is the immediate problem, because you need it for your email address. On a German keyboard it is printed on the Q key, but the virtual display expects it at Shift+2, where a German keyboard prints the double quote.
Do not fix this by switching the layout on the server. Running setxkbmap de looks like the obvious solution and creates a worse problem: a German layout expects AltGr for the at sign, and Apple keyboards have no AltGr key at all. You end up with a layout you cannot fully reach.
Two approaches work reliably instead.
The shared clipboard is the simpler one. x11vnc synchronizes it in both directions, so you can copy text on your own computer and paste it into the application window with Ctrl+V. Note that this is Linux inside the session, so the modifier is Ctrl and not the Cmd key you would use on macOS. This is also the sensible way to transfer a password or a confirmation code from your mail client.
If the clipboard does not come through, send the keystrokes from the server side. xdotool writes directly into the focused input field and is completely independent of any keyboard layout:
sudo apt install -y xdotool
Click into the input field in the VNC window first so that the cursor is blinking there, then run on the server:
DISPLAY=:99 xdotool type --delay 80 ‘address@example.com’
Use single quotes, otherwise the shell will interpret special characters before xdotool ever sees them. If nothing appears, the window has lost focus, which you can restore:
DISPLAY=:99 xdotool search --name -i ente windowactivate
For a single character rather than a whole string:
DISPLAY=:99 xdotool key at
The same method works for the password and the confirmation code. Be aware that anything typed this way ends up in the shell history of the server. Putting a space in front of the command keeps it out of the history in most shells.
Step 7 - Import in batches and verify
Do not select the whole export at once, if you are one of the cautious ones. If something fails after two days you will have no way to tell where it stopped. Start with the smallest folder as a test:
cd “/mnt/volume1/extracted/Takeout/Google Photos” du -sh “Photos from 20”* | sort -h
Import that one folder. When the application asks whether to create one album for everything or one album per folder, choose one per folder, otherwise the album structure is lost.
Then verify. Count the files on the server and compare against the album in the application:
find “Photos from 2016” -type f ! -name ‘*.json’ | wc -l
Open a few images and check that the capture date shows the correct year and not today. If it does, the companion files were read correctly and the rest of the import can be trusted.
If you want to assemble larger batches without copying data, use bind mounts. They present existing directories under a new path, which the application treats as ordinary folders:
SRC=“/mnt/volume1/extracted/Takeout/Google Photos” sudo mkdir -p /mnt/volume1/batch for y in 2017 2018 2019; do sudo mkdir -p “/mnt/volume1/batch/Photos from $y” sudo mount --bind “$SRC/Photos from $y” “/mnt/volume1/batch/Photos from $y” done
Import /mnt/volume1/batch, then release the mounts before assembling the next batch:
for d in /mnt/volume1/batch/*/; do sudo umount “$d”; done sudo rmdir /mnt/volume1/batch/*/ mount | grep batch
Always confirm with the last command that nothing is still bound. A bind mount left in place points at your original data, and a delete command in the batch directory would remove the real photos.
While a batch is running, disconnect the VNC viewer. x11vnc polls the screen continuously, and a progress bar that redraws several times per second can occupy a processor core that the encryption needs. The upload continues without a viewer attached, because Xvfb keeps the display alive.
To check that data is actually leaving the machine rather than merely being indexed:
a=$(cat /sys/class/net/eth0/statistics/tx_bytes); sleep 60 b=$(cat /sys/class/net/eth0/statistics/tx_bytes) echo “$(( (b-a)/1048576 )) MB/min”
Around 100 MB per minute corresponds to roughly 140 GB per day. Expect the process to take days rather than hours for a large library. Video files are the slowest part, because the client generates preview images for them.
I honestly just selected the whole Google Photos folder, because I was pretty fed up by all of this at this point and it worked out fine.
Step 8 - Clean up
Verify the total first. Compare the number of media files in the export against the number of items in the destination, and take samples from your oldest years, where the original files often carry no embedded date and everything depends on the companion file. If the album folders were imported as well as the year folders, look for a duplicate detection feature in the destination and use it, because the overlap described in step 4 will otherwise consume quota twice.
Only then take the workbench apart, in this order:
Release any remaining bind mounts.
Detach and delete the Volume. A Volume is billed by its provisioned size, not by how full it is, so deleting files inside it saves nothing.
Scale the server back down or delete it. A server that is only powered off continues to be billed, because billing follows its existence rather than its uptime.
Keep a cold copy. If the Storage Box is the only remaining copy of the original archives, keep it. In an end to end encrypted service, a lost password cannot be recovered by the provider, and a library that exists in exactly one place is a fragile arrangement.
Some final suggestions
If you followed along, you most likely have successfully moved a large photo library out of Google Photos without involving your own hardware, and you have verified at every stage that nothing was silently lost. The parts worth keeping from this procedure are not specific to photos: transfer machine to machine rather than through your own connection, unpack and count before you trust an export, work in batches small enough to verify, and treat a graphical application as something you can run headless when it is the only interface a service offers.
Three Hetzner features made this kind of temporary project inexpensive:
Billing is hourly with the monthly price as a ceiling, so a larger machine for a few days costs a fraction of its monthly figure.
CPU and RAM can be raised and lowered again on an existing server, provided you do not grow its disk, which keeps the change reversible.
A Volume can be attached when the space is needed and deleted when it is not, so the storage exists only for as long as the migration does.
Whichever provider you decide on, check those three things before you start. Being able to size a machine up for the migration and back down afterwards is what turns this from a purchase into a rental.