Trying to manage a massive Photos Library

Gregor Samsa · 09-09-2020 11:37am #1

I've a MBP with a 1TB SSD.

I've a pretty massive Photos Library with 84,606 photos taking up over 776GB of space, which is unsustainable as the library is growing - two decades of photography as a hobby. I've two 24mp cameras, so the file sizes from those two would be a fair bit bigger than the phone pictures, but my main immediate problem is videos - as well as the two cameras and the phone, I've a GoPro, and the file sizes of the high res videos are (obviously) massive.

I've got a 1TB external SSD, and my ideal plan would be that I keep all my still photos on the MPB internal drive (so I always have access to them), and all the videos on the external drive (don't need access as much).

Ideally, I'd like everything stored in iCloud Photos too, to access on my phone, iPad and Apple TVs (I know, yeah, I want the moon on a stick!)

I don't want to only rely only on iCloud for the photos. I have Optimized Storage turned on on my Phone and iPad, but I 100% want to keep a copy of my photos local . Many are irreplaceable, and I'm not taking chances with them (I've a Time Machine backup too, and occasionally do another off-line backup). So using Optimised Storage on my MPB - while the obvious solution - isn't an option.

Photos lets you use multiple Libraries, but you have to manually switch when opening the application to load one or another. So I could have one on my internal disk for stills, and another on my external disk for videos, and switch between the two as necessary.

Does anyone know how this would work with iCloud? Will my iCloud library somehow merge the two local libraries? Or will I have to turn iCloud off to stop content from one library syncing to the cloud and then back down to the other library - basically giving me two duplicate libraries with everything in it.

Or does anyone have any other solutions? I was thinking maybe of using Google Photos for the videos, so I could keep the two completely separate and still have cloud access on my other devices. GP doesn't run on ATV, but there is a 3rd party app that gives you access. Amazon Photos would be an option too (I have Prime), but there seems to be no ATV app at all. GoPro have an unlimited cloud storage plan, but I'd rather not pay for another service, and anyway, I want to store all videos from all cameras, not just the GoPro.

I know the easy answer is "you have too many photos, delete them", and I am in the middle of a cull, but just park that for the moment and pretend that every one's a keeper

tl;dr: What's the best way to split photos and video local storage on the Apple ecosystem, and still have cloud access?

Cabaal · 16-09-2020 8:53am

I used to use Google Drive with a storage plan but I found i had issues with file dates on them (it was resetting them) so I switched to Dropbox.

I got about 1.2TB of photos and another 700GB of misc files stored on a NAS locally, the files are also mirrored (via dropbox) to a second NAS locally (in case one fails) and they are also of course backed up to Dropbox which I can then access via my phone or other devices.

I'm running off a Mac but opted against icloud as I wanted NAS intergration and my NAS's didn't offer an icloud option.

Gregor Samsa · 11-11-2020 11:55am

I'm just going to give an update on what I did to solve my problem, in case it's of help to anyone else.

I got an external 4TB drive. I set up a new account on my Mac called "Photos". In "Photos", I'm logged into the same iCloud account as I am in my regular Mac account.

On my regular Mac account, I turned off local storage of my photos, so they're all just in the cloud with local previews. This obviously frees up a massive amount of space on my internal hard disk (776GB), solving the first problem - the lack of storage space. Next step is to solve the problem of having both cloud storage AND local backup on the same iCloud account, but using an external disk.

Then, in the "Photos" account, I set up a new Photo library on my external disk. I set this to be the system library, turned on iCloud photos, and turned on local storage. It turns out that the "system library" setting is contained within the account. So you can have a different "system library" for each account on your Mac, but run them off the one iCloud account. All the photos and videos then downloaded to the external drive from iCloud.

So I now have two Photos libraries running on my Mac: One, on the login I use for day-to-day work, is cloud based, so not taking up much local storage on the Mac.The other, on the "Photos" login, has all the same photos stored locally, but still syncing to the same iCloud account.

The great thing is that when I'm logged into my regular account (which is most of the time), the second "Photos" account is asleep in the background - but on the Mac, "asleep" means that it still fetches iCloud updates.

So, If I take a photo on my phone, and it syncs to iCloud, it appears in both Photos libraries I have set up on the Mac (the cloud one and the local one) simultaneously, even though I'm only logged into the account that is running the cloud based one. This is perfect, because it means I don't have to do any kind of manual intervention ot updating or syncing to have a local copy of all my photos, no matter what device they're taken on or imported from.

So while my Mac is sitting on the desk connected to the external hard disk, it's keeping a local copy of all my photos. When I grab the Mac and bring it off somewhere without the HD, I still have access to all the photos in the cloud, and I can still import new photos. As soon as I reconnect the HD, any new photos get pulled down to the second library in the background, and stored on the external HD. Plus, I have Time Machine running backing everything up to a second external networked drive. So my 80-odd thousand photos are in the cloud (and accessible on all my devices), stored locally on my Mac's external drive, and backed up to a network drive. And I'm not tethered to the external drive when I don't want to be.

The only problem was that the initial download of all the photos from iCloud is sloooooow - even with a 500mb fibre connection, and plugged into the router via ethernet. It took almost a week. Plus, iPhoto is very bad at telling you what activity is going on: sometimes it's impossible to know whether it's actually downloading the photos or whether it's stuck. But it got there in the end. I did try just physically moving the old local photo library from the internal HD to the external HD, and then linking it to the Photos app in the "Photos" account, but it didn't like that, and kept giving errors, despite running a repair a few times.

5uspect · 15-11-2020 2:23pm

Personally l’ve always separated my phone photos from my camera photos. All the phone stuff goes into the Photos app and iCloud while I keep 15 odd years of my photography (almost all RAW) in a reasonably well organised file system and duplicate it via rsysc onto an external disk and also a small NAS. I keep JPEGs of some of them on my Macs.

Currently my photography is on on a 4TB internal disk in my PC which is backed up to Back Blaze. We moved into our own home recently and I’m interested in how best to maintain my NAS or replace it.

Gregor Samsa · 15-11-2020 2:47pm

Yeah, I used to shoot RAW, and kept my camera photos (from two 24mp Sony Alphas) separate in Aperture or Lightroom (I switched back and forth a few times before Aperture disappeared), and the casual phone stuff in Photos (or iPhoto then). That’s certainly the right way to do it.

But a few years ago I found that I had no time for post processing, and the RAW file sizes were just too big to be worth it for me. So I switched to shooting in JPEG. Then I found that the main place people wanted to see the photos was on the TV, and so having them all in iCloud so that they could be seen via the Photos app on Apple TV was just so convenient.

I know Photos isn’t what I should be using, but I guess that even thought I have a “serious” photo collection, I just don’t have the time to take photography seriously any more, and am looking for convenience.

I had a MacBook (and all my cameras) stolen in Berlin with a decades worth of digital photos in it ten years ago. Fortunately, I had backed them up for the very first time just before I left, so I I only lost the photos I took that week. So backing up my photos is of paramount importance to me. As well as the whole library in iCloud, the MBP itself and Time Machine, I’ve select photos on about 3 other hard disks, and some also on Google Drive, Flickr and OneDrive.

5uspect · 15-11-2020 3:33pm

I hear you about about having time. When the kids arrived I found myself mostly using my phone for photos. My Canon gear was used less and less. Any photos from my gear would be of the kids and they many end up in iCloud Photo Library.

I’m hoping to make more time for personal photography and get back to processing photos like I used to. That means getting my on-site backup under control.

Fann Linn · 15-11-2020 3:38pm

On a related issue I've a fair few photos myself and when trying to organise same I've come across a few copies of the same photos which is in effect almost doubling my stock.
I don't know how this has happened but can anyone recommend a way of removing these rather than individually deleting them.
Thanks.

Gregor Samsa · 15-11-2020 4:04pm

Fann Linn wrote: »

On a related issue I've a fair few photos myself and when trying to organise same I've come across a few copies of the same photos which is in effect almost doubling my stock.
I don't know how this has happened but can anyone recommend a way of removing these rather than individually deleting them.
Thanks.

I found that myself over the past few weeks when I've been working on getting all this set up. I had duplicates and in some cases triplicates of random photos - sometimes the duplicates would be rotated 90 degrees to the original. They usually occurred in batches (a group of 10 or so photos with duplicates, all beside each other in the library). No idea how or when they got there, but I saved probably a gigabyte in removing them.

I just manually removed them, but there's software like this: https://www.cisdem.com/duplicate-finder-mac.html
that claims that it can scan your library for duplicates (that one doesn't just do the Photos Library, but can handle any file). Haven't used such things myself, so I don't know how safe, accurate or convenient it is.

nofools · 15-11-2020 4:08pm

I have had both a macbook and hard drive fail on me with irretrievably lost files.

I would zip and burn some master copies to DVD and lock them away somewhere for safe keeping and then carry on with the rest of your plan.

Cloud storage is imperfect for serious backups.

Gregor Samsa · 15-11-2020 4:31pm

nofools wrote: »

I have had both a macbook and hard drive fail on me with irretrievably lost files.

I would zip and burn some master copies to DVD and lock them away somewhere for safe keeping and then carry on with the rest of your plan.

Cloud storage is imperfect for serious backups.

DVDs - particularly writeable ones, deteriorate with time, so while they can be part of the solution, they can’t be relied upon, no more than anything else can. I’ve DVD-Rs from previous computers that are unusable now. Plus as we’re seeing, access to the players is getting rarer. Industry standard for personal backup in the 90s used to be to Iomega Zip and Jaz disks. If you don’t remember them, then there’s a lesson in how something ubiquitous can totally disappear in a short time in tech. I’ve loads of them lying around still, and the data is probably still intact in them. But try finding a Zip or Jaz drive to read them on now.

The best bet is a mixture of all kinds media, as diverse as possible. That’s why I’ve local, NAS and cloud for everything on my Mac, and then more hard disks, and a few other cloud providers for the very important stuff. I’ve about 10 copies of the vital stuff in 10 different places across multiple media and providers. The chances of them all failing at the same time are slim enough to discount (unless the magnetosphere disappears, but then we’ve got bigger problems).

DVDs certainly could be part of that mixture, I’ve just opted to diversify in different ways.

nofools · 15-11-2020 6:37pm

Gregor Samsa wrote: »

DVDs - particularly writeable ones, deteriorate with time, so while they can be part of the solution, they can’t be relied upon, no more than anything else can. I’ve DVD-Rs from previous computers that are unusable now. Plus as we’re seeing, access to the players is getting rarer. Industry standard for personal backup in the 90s used to be to Iomega Zip and Jaz disks. If you don’t remember them, then there’s a lesson in how something ubiquitous can totally disappear in a short time in tech. I’ve loads of them lying around still, and the data is probably still intact in them. But try finding a Zip or Jaz drive to read them on now.

The best bet is a mixture of all kinds media, as diverse as possible. That’s why I’ve local, NAS and cloud for everything on my Mac, and then more hard disks, and a few other cloud providers for the very important stuff. I’ve about 10 copies of the vital stuff in 10 different places across multiple media and providers. The chances of them all failing at the same time are slim enough to discount (unless the magnetosphere disappears, but then we’ve got bigger problems).

DVDs certainly could be part of that mixture, I’ve just opted to diversify in different ways.

It is getting harder for sure. I am reluctant to put important things to the cloud without another backup.

Cloud storage is a big money spinner

roboshatner · 06-12-2020 7:46pm

Awesome Duplicate Photo Finder is out of this world
to use.

It works on windows it torn through my folders of images and I had a lot of duplicates.

It ended up after days and hours of used that it saved me gigs of data from deleting Duplicates or photos taken 4 times more then they need to be taken.

jakdublin · 10-12-2020 1:12pm

roboshatner wrote: »

Awesome Duplicate Photo Finder is out of this world
to use.

It works on windows it torn through my folders of images and I had a lot of duplicates.

It ended up after days and hours of used that it saved me gigs of data from deleting Duplicates or photos taken 4 times more then they need to be taken.

There isn't a Mac version. This is a Mac forum.

faceman · 20-12-2020 12:57pm

I’m heading down a similar road to you OP. Looking to remove my massive photo library and stick it on an SSD that the Mac treats as it’s photo library. I keep getting round to doing it and never do! I’m due to upgrade my iMac next year and they don’t ship with large storage drives anymore so I must do it before then.

The challenge I have is that migrating photo library’s to new macs in the past hasn’t always gone smooth. Photos and/or thumbnails have gone missing or been duplicated for many of my albums. A cleanse is necessary

Gregor Samsa · 20-12-2020 1:38pm

faceman wrote: »

The challenge I have is that migrating photo library’s to new macs in the past hasn’t always gone smooth. Photos and/or thumbnails have gone missing or been duplicated for many of my albums. A cleanse is necessary

Yeah, it wasn’t a smooth process for me. I’ve ended up with a fair few duplicates (and triplicates and quadruplates). It’s weird, they seem to appear in clusters of 6 or 7 consecutive photos spread randomly throughout the library, with some of the copies rotated 90 degrees. I’ve been doing a fair bit of cleaning up. I’m also finding that having the library on an external disk is not as reliable as on the internal one. If I disconnect the disk, I often find I have to reboot the machine to get the library on the external disk to reload when I connect it up again.

I just got a new camera, and I’d love to be shooting in RAW, but I just can’t be dealing with the 24MB file sizes from three cameras. I know, Photos isn’t meant for this, and I should be using Lightroom (or Apple shouldn’t have abandoned Aperture), but when it works, Photos is so convenient across devices.

Next year I might look into going back to Lightroom for importing, processing and local “master” storage of my SLT/Mirrorless camera photos in RAW, and see if I can somehow set up an automatic export of JPEGs to Photos, mixing in with the casual iPhone stuff and the GoPro stuff, for easy access via iCloud across devices.

Life was much simpler when I just had one 35mm film SLR!

Gregor Samsa · 10-07-2023 10:08am

https://www.boards.ie/discussion/comment/120810988#Comment_120810988

Yeah, I spoke about the duplicates in some earlier posts, but my main issue was that I genuinely had over 80 thousand photos.

That was from 2020. In the meantime, I got back into photography and a 50mp camera that I shoot RAW (so approximately 50mb files), so I needed to sort the whole thing out for once and for all.

I got a Mac Studio and went back to using Lightroom for importing and processing my non-phone photos. I got a fast external 1tb NVMe drive in a fast Thunderbolt enclosure, and I have my Lightroom files on this. Any 100% keepers, I then export as Jepg to Photos, which also has my casual iPhone photos, so I have access to them across all of my devices. My Photos library is running on the NVMe drive too. It's rock solid. I had tried a few different standard USC-C SSDs, but they all suffered from random disconnects, and at one stage corruption of my Lightroom catalogue. The NVMe has never disconnected or given any issues in the 10 months I've been running it.

Separately then everything gets backed up by TimeMachine to a local HDD, but I'm getting a Synology NAS this week.

So I'm perfectly happy with the setup now.

Trying to manage a massive Photos Library

Comments