• Please review our updated Terms and Rules here

File Dupe Checker

NeXT

NeXT

Veteran Member
Joined
Oct 22, 2008
Messages
8,149
Location
Kamloops, BC, Canada
My goal this winter is to migrate my old server and it's multiple overflow drives to a new NAS but after five years of downtime I got multiple volumes with hundreds of duplicate or outdated files totaling a few terabytes. I can drop them all into the NAS and overwrite older files as part of my first-pass at screening and organizing into new folder structuring but I'll still end up with duplicate files hiding in renamed folders or the files themselves were renamed so it's the age-old challenge of a program to flag duplicates for manual review. I know there's also a bunch of duplicate AND corrupt photographs floating around.
You search something like that into Google and I get a few results, all of them reek either of dodgy trial versions or auto-generated top 10 lists. I'm pretty certain I'm not the only person with large amounts of small files so I'm wondering what other members are using that supports at least Windows 7 and compares files beyond name, date and location.
 
Last edited:
I would be interested in what comes of this. I have a similar problem with my NAS having many duplicate copies of files and folders. I know years ago when I was working at a site they had a similar problem. A huge backup system but with multiple (and I mean multiple) copies of (maybe) the same file or slightly changed. I never did find a way to clean that up, but it needed cleaning. Having all that data and pretty much useless if you can find the latest copy or what is really old.
 
I can if bad comes to worse do it manually, but I'll probably be at it for months! x_x
During a server failure in 2019 I had to merge two datasets together and my Canon RAW file catalog got clobbered. I was able to recover all the images, but in the process several hundred images with identical names split off into a neighboring directory. They still kind of load within my software but they are clearly corrupt. I've been weeding them out ever since.
 
Sorry for the hasty reply the first time. Since that post I've had cause to look at this closer myself and I see CCleaner isn't what I thought it was.

I've looked a little further and discovered that there is a program that will take care of much of the problem:

https://www.jdupes.com/

Downloads are here:

https://github.com/jbruchon/jdupes/releases

I've tried jdupes myself on a drive that had been saving data when a power failure occurred. The root directory et al had been trashed but (all? of) the files were still there. I was able to use a brute force recovery method to get thousands of files onto a different drive, but the file names were lost. jdupes successfully matched a lot of them up to a recent backup set and I was able to find the few recent files that weren't backed up and rename/restore them by hand.
 
That is the Linux program that jdupes is forked from. The original request was for a program that ran on Windows 7 which fdupes can't do (natively) but jdupes has been made into a superset of fdupes that runs on the current big 3 OSes: Windows, Mac OS, and Linux.

I think your fdupes experience would directly translate to jdupes. @NeXT is looking for positive experiences/recommendations, so how well do you find fdupes to work?
 
You must log in or register to reply here.
Back
Top