Digital Decluttering: Photo Collection Consolidation is Back on Track (Premium)

The kids—who are both adults now—were home for Thanksgiving and so I finally asked them about the digital photo collection as part of a conversation about the work my wife and I have done over the years to downsize and that I have done, especially this past year, to organize our digital archives.

As a bit of background, we have a relative who is a hoarder, and each time she’s moved in recent years, the family has discovered an astonishing amount of stuff piled up in every nook and cranny of her home. And while you can’t ever plan this perfectly, our goal is to not add the responsibility of dealing with all that clutter when we pass away. And that’s as true of the physical stuff as it is the digital stuff.

Anyway, I told the kids that I had two big sets of archives, a documents archive that includes everything I’ve ever written over about 30 years and a photos archive that includes all my—and, more importantly, our—photos from an entire lifetime, dating back almost 60 years. I explained that they would likely be uninterested in my documents archive for the most part, though there are some personal things in there, and that there may be colleagues out in the world who wouldn’t mind going through it. But the photos archive will be a bit trickier because there are thousands and thousands of photos (and home videos) of them in there. And that it was likely that they would want to save at least some of it.

There was no need for an immediate decision, I added, since I was literally working to see whether it was even worth taking the time to consolidate this archive, which consists of three separate collections (each with lots of overlap but also each with some unique photos) into a single photos archive. If you’ve been following along with this past year’s digital decluttering efforts, you are likely familiar with some of this work: I very successfully organized and replicated my documents archive but had mixed results with the photos for a variety of reasons, including its vast size and the lack of good meta-data in many cases. The kids were only peripherally aware of this work because I had shared some newly re-digitized home videos with them when I did that work a few months ago. But I never mentioned the photos.

Our son Mark initially joked that I should make the photos archive consolidation my “retirement job,” and he then suggested that perhaps making a “best of” or “favorites” collection and sharing that would be enough. But after thinking about it for a bit, he said that this archive needed to be saved, and that he, at least, would like to have it when we were gone. Kelly, our daughter, was nodding her head to all this. She, too, was interested in the “favorites” collection and agreed that the entire thing should be saved if possible.

I was a bit surprised by that, but I told them that their mother and I had at one point curated a photo collection, literally called Favorites, 15 or 20 years ago, and most likely with an eye towards having these images appear in a digital photo frame of some sort. We hadn’t kept up with it, for any number of reasons, but the flood of smartphone-based photos that arrived soon after that would have made that work difficult anyway. Today, our digital photo frame is a Google smart display that auto-curates photos of family members, friends, and pets in Google Photos, with no work required.

And so the promise was made: My wife and I would work on the Favorites collection. And I would continue my work trying to consolidate the three separate photo collections into a single archive, with the understanding that my success to date had been limited and that this was something I may never finish: At the very least, they could have all the photos, however they were organized. And I would investigate the best ways to share them with the kids.

And then they were off, as is the norm these days, with Mark driving back to Rochester, New York and his job and Kelly flying back to Charlotte, North Carolina and her final semester of college as an undergrad. These moments are always curiously poignant, not just because we love our kids but because we like them as people and enjoy spending time with them. But this time was also a bit different: We had just moved into a new place the weekend before and after suffering some earlier decluttering setbacks, I had promised myself that I would get on top of this work as soon as we had finished moving: There is still so much work to do, and as noted above, my goal is not to burden my wife or kids with it should anything happen to me. And so I walked back into my new office, surveyed the boxes and bins that needed decluttering, some out, some in the closet, and sighed. It was time to get to work.

Some of that work will likely come up in future articles in this series. But here, I would like to finally discuss my plans for the photos archive, and the work I’m doing to get it consolidated. This work is based on months of organizational experiments and diversions that came about in the wake of my decluttering implosion two months ago, when I took on too much work at once and failed at most of it. Since then, I scaled back my plans and made a bit of progress with some possible digital photo organization and consolidation techniques. But I was also dealt a bad hand by Google when it killed the paid additional storage subscription for Workspace Basic customers that I had been relying on. This forced me to move some of my online content, including my Google Photos-based photo collection, from Workspace to my personal Gmail account, work that I completed quickly.

So here’s where I’m at.

I have three collections of photos across Google Photos and OneDrive. My Google Photos-based photos collection is about 570 GB when downloaded through Google Takeout. And in OneDrive, I have separate collections in its automatically created “Camera roll” folder and in a “Photo collection” folder that I created and organized using date-based folders (with top-level year folders and date-based event folders inside each). The Camera roll folder is about 250 GB when synced locally. And the Photo collection folder is about 175 GB. That’s roughly 850 GB of photos, with lots of redundancy/overlap, but also lots of unique photos in each. (And this is aside from my ongoing work finishing up paper-based photo and document scans and whatever home movies we have. One thing at a time.)

I’ve written a lot about the nothing I’ve accomplished with this task to date, but the good news is that I’ve learned a lot and have arrived, after constant experimentation, at a method for automating the consolidation as much as possible, on a year-by-year basis, before having to move into more manual work. And I think that I will be able to make this all work, though the amount of time and effort it will require is still daunting.

But the key thing I learned, oddly, is that I was going about this the wrong way from the get-go. And so my new push to complete this work is coming at it from a different angle. And as with any daunting task, like getting out from under credit card debt, I am starting with some small wins with the hope that they inspire me to keep going when the work gets more complicated.

This is important because that 850 GB of photos is comprised, currently, of over 350,000 files. Most of them, of course, are photos, but some are videos that need to be organized too, while many are other things—superfluous “live photo” videos, screenshots, and other images and files—that do not. These files are inconsistently named and inconsistently tagged with “Date taken” meta-data, and there are different versions of some files with different resolutions and quality levels.

One of the big things I learned early on in this process is that I needed to work on these files locally on a PC, and not across a network or the Internet. And that means that I need all three of these collections, all 850 GB worth, downloaded or synced to some computer. More specifically, I need a PC with 2 TB of storage, ideally, since I need some storage wiggle room during the consolidation process. And that neatly limits my choices: I have many laptops here and a few desktop PCs, but few with over 1 TB of storage.

Our having just moved helps: I again have all my PCs at my disposal, whereas many were unavailable, packed in boxes, during the pre-move period that started in September. So I spent some time looking at the available storage allotments on my PCs, and was hoping to find a portable choice so that I could work on this stuff around the house when desired rather than being stuck at a desk. And I lucked out: I have an HP Envy 16 with a 2 TB SSD, so I reset that, upgraded it to 23H2, removed a lot of cruft, installed a few key tools, and started downloading and syncing my photo collections to it. In fact, this work started as soon as my son pulled away in his car to drive home Sunday.

OK, technically, it started a bit earlier: In the few days I had between our return from Mexico and my trip to Seattle, I had decided that I wanted to consolidate the two OneDrive-based photo collections first, and that the first step in doing that would be to organize the Camera roll folder, which did have a few year-based folders but was mostly just a raw dump of photos in its root, into something more organized. The goal was simple enough: Move all the files in Camera roll into the correct year-based subfolders (2012, 2013, and so own) so that I could then work year-by-year to compare the files in each with the photos in my Photo collection folder.

And that process was surprisingly easy thanks to all the experimentation I’d done in the previous two months. I started with Directory Opus because its two-pane view makes it easy to sort files by “Date taken” meta-data in one pane and then just drag them over to the appropriate Year older in the other pane. When “Date taken” failed me—and it did because so many files didn’t use this meta-data—I used the app’s file name matching feature because many of the file names had the date in their names. And with a bit of work, I was able to move every single loose photo from the root of the Camera roll folder into the “correct” year folder.

I was pretty excited about this first step, but I had to fly to Seattle the next day.  While I was away, I actually spent some time, on the plane and in a few random free periods, experimenting further with a few of the year folders (like 2004 and 2005) for which I had made offline copies to see how difficult it would be to further refine the structure of each to include day/event-based folders. So for example, instead of having a dump of all the photos from 2004 in a single folder, I would have subfolders like 2004-01-01, 2004-02-04, and so on. That, I felt, would make it much easier to make future comparisons and consolidations.

Bulk Rename Utility (BRU) is the right tool for this job, as it can re-sort a folder of files into a folder of subfolders based on virtually any scheme you can configure. Its success rate is entirely dependent on the meta-data of the contained files: For older years, like 1987, where I had scanned in paper-based photos and applied the correct “Dake taken” meta-data, it was a breeze. For newer folders (say, 2012 and more recent), I would still need to do some manual sorting work because many of the files (screenshots, downloads, other files) have terrible or non-existent meta-data.

And that was the epiphany, if you will, that led me to what I’m doing now: It would be time-consuming but easy to consolidate the earliest photos (again, probably anything older than 2012) but harder to consolidate the more recent photos. And so I should start with the older photos first, consolidating many years’ worth of memories in this first phase. When that was done, I could move on to the newer years, which would be a slower, more tedious process that would also require a de-duplication app like AllDup and a lot of manual work. Small wins first, then the hard stuff.

So I came home, we moved, and the kids came and went over the Thanksgiving long weekend, as noted, and the whole time I had this idea in the back of my head. Before they left, I ordered a newer version of my Google Photos collection via Google Takeout. And this past Sunday, I configured my OneDrive-based Camera roll and Photo collection folders to be always available on that PC, syncing them both locally. I downloaded and unzipped the 12 50+ GB Takeout ZIP files pretty quickly using 7Zip, which I measured to be at least five times as fast as File Explorer’s compressed folders feature. And then I connected the laptop to Ethernet on Sunday night so it could sync those two OneDrive folders as quickly as possible. That finished up while I was at the gym on Monday morning, sometime between 10 and 10:30 am most likely.

With that done, I decided to consolidate the older years of photos in the two OneDrive folders first. (In contrast to much of my work over the past month, which usually involved experimenting with more recent years in the Google Photos collection). But that, too, was an epiphany of a sort: Three photo collections is one too many, and consolidating the two OneDrive collections into a single collection seemed like the easier job, with the understanding that it would still be pretty tedious. I wanted to keep the organized Photo collection folder, so I would whittle down the Camera roll folder, which as noted above was now organized into year-based folders like 2011, 2012, and so on.

By starting with the oldest years first, I was sure I could blow through the first several year-based folders very quickly because they contain relatively few files, and those files are mostly pretty clean from a meta-data perspective. And that meant I could probably start with File Explorer, as lame as it is: Once I started manipulating folders full of thousands of files, I would move to a more sophisticated tool like Directory Opus.

This part of the process was manual. I opened a File Explorer window to the Camera roll folder and snapped that to the left side of the screen. Then I opened a second File Explorer window to the Photo collection folder and navigated into its Old subfolder, which contains year-based folders from 1965 through 1999. This was an ideal subset of photos to work on: It covers many years but is by far the smallest part of the collections by overall size and file count.

Putting these two folders side-by-side, I got to work. This was a semi-time-consuming but rote process of comparing what was on the left (in Camera roll) with what was on the right (in Photo collection\old), year-by-year. If a file on the left was present on the right, that file on the left was deleted. In the rare instances in which a file on the left folder was not present on the right, I would move it to the right folder. This happened between 12 and 15 times for the first 15 years I got through Monday afternoon, covering the years 1965 through 2008, or about half of the Camera roll folder’s folders. And that kind of thing of course validates my efforts. I might have lost some pictures otherwise.

Most times, of course, the contents in Camera roll and Photo collection were identical. That’s because the OneDrive Camera roll folder didn’t formally exist until sometime in the 2010s or so, and so anything older than that in there now was somehow downloaded to a phone at some point and backed up by OneDrive.

Here’s the general process I used: I would bring up a year-based folder, like 1986, in Camera roll on the left and compare it with the photos in Photo collection on the right, which again has year folders and event-based subfolders.

If the photos on the left were already on the right, I deleted the versions on the left (from Camera roll). This was simple, repetitive work at first, thanks to how well these older photos were organized and correctly tagged.

I started this work around 2:00 pm Monday and by the time I finished the first 15 (!) year-based folders, it was about three hours later. By this point, only the 2009 through 2023 folders were left in Camera roll, plus 500+ loose and recently backed-up photos in the folder itself. This was a nice bit of progress on one level, but it didn’t impact the size of Camera roll much because most of the photos are in its more recent year-based folders. More specifically, it was now 248 GB in size, down just 2 GB or so from the start.

And that’s just fine: I had guessed that there would be a division between the old and the new and that the newer, larger folders would need more automated consolidation methods. The question at this point was how far into those folders could I go—to what year, in other words—before I had to make that switch? (Related to this, I figured I would use that dividing line/year as a hard stop for incorporating the Google Photos collection too. That is, once I hit a wall on file volume, I’d start working on the files in Google Photos until I hit that same year. Baby steps.)

Looking through the next few folders revealed the answer. The 2009 folder has only 107 files, fewer than many of the folders I’d just consolidated. 2010 has 300 files and 2011 has 436 files, but 2012 has 1900 files, so the volume was finally growing. 2013, the year I finally switched to smartphones full time for photography thanks to the Lumia 1020, has about 3,800 files. And it goes up from there: 2017, for example, has about 8600 files, but from there it gets even uglier: The folders for the past three years (including 2023) have 11,000; 14,100; and 7500 files respectively. Ugh.

I decided to continue this work through 2012, if possible, the year before I switched to smartphones for photos. It seemed right, somehow. And so Monday night I plowed through 2008, 2009, 2010, 2011, and 2012, deleting most of the files in those folders in Camera roll and moving the others over to Photo collection. These newer folders required an additional step because they contained several hundred photos each: I used BRU to resort the photos in each into date-based folders like 2012-05-26, 2012-06-01, and so on.

And then I could compare the date folders on the left with those in Photo collection on the right more easily. Here again, the work proceeded pretty quickly, and I finished up with the Camera roll folder (through 2011 inclusive) on Monday. The old photos in OneDrive were now consolidated into a single location, the Photo collection folder.

By Tuesday, I was ready to start consolidating the local copy of my Google Photos collection, year by year, into that Photo collection folder in OneDrive. One collection to rule them all!

First, I was reminded that the Takeout download comes with a lot of cruft, including an incredible 197,000 superfluous JSON files. So I used Directory Opus to find and delete all of that first.

And then I used side-by-side File Explorer windows to compare the first year-based folders in Google Photos (Photos from 1966, Photos from 1967, and so on) on the left with the year-based folders in OneDrive’s Photo collection folder (1966, 1967, and so on) on the right.

This consolidation work was manual—where I would compare, move as needed, and then delete from the left otherwise—but it also went quickly, at least through the 1980s, when the Google Photo folders started having several hundred files (or more) in each. But I was ready for this, and my BRU skills made short work of organizing each year in Google Photos into date-based subfolders (again, 1990-05-26 and so on) so that I could more easily compare left and right.

And, almost always, just delete from the left (Google Photos).

By this morning, I had completed consolidating all three collections through 1997. And now I am positive that I will finish the first phase of this work, through 2011 inclusive, by this weekend. And at that point, it will be decision time.

But I will likely upload the updated and consolidated folders (roughly 1966 through 2011) to various services and the NAS, and then move on to the more grueling work awaiting me in 2012 through 2023. But thanks to the experiments I did over the past two months, I feel confident that I’ll get through that too: In testing my consolidation skills on newer years in various ways over time, I arrived at a workable system that includes both automated and manual organization And while it will be time-consuming, it’s doable.

And that alone is an amazing accomplishment: This task that I never thought I’d complete is suddenly on track for the first time, and I can see how to get through the rest of it. I can’t make any promises on the schedule, but my goal is to finish this as quickly as possible so I can move on to my other digital and physical decluttering tasks. It never ends, I guess. But I can see the light at the end of the tunnel.

Finally.