Rams Poster
Ever opened your Downloads folder and wondered why it’s exploding with copies of the same file?
A quick duplicate‑file finder can clean that up in minutes, and you can build one from scratch in a single sitting.
What you’ll need
- pathlib – modern, object‑oriented way to work with file paths (instead of the old os.path).
- hashlib – creates a digital fingerprint (hash) of a file’s contents; identical files produce identical hashes.
- collections.defaultdict – a dictionary that automatically creates a list for each new key, perfect for grouping files by their hash.
- os – for walking through directories (os.walk).
- (optional) tqdm – a tiny progress bar so you can see the scan move.
How it works, step by step
1. Walk the folder – os.walk yields every sub‑folder and file, letting you examine the whole tree.
2. Hash each file – open the file in binary mode, read it chunk‑by‑chunk (e.g., 64 KB blocks) and feed those bytes into hashlib.sha256(). This avoids loading huge files into memory.
3. Group by hash – store the resulting hash as a key in a defaultdict(list). Every time you see the same hash, you append the new path to that list.
4. Report duplicates – after the scan, any hash with more than one path means those files are byte‑for‑byte identical. Print them out, or later add a prompt to delete the extras.
Why hashing is reliable
A SHA‑256 hash is 256 bits long; the chance of two different files sharing the same hash (a collision) is astronomically low—practically zero for everyday use. So comparing hashes is far faster than comparing every file byte‑by‑byte.
Mini‑example (core logic)
import os
import hashlib
from pathlib import Path
from collections import defaultdict
def hash_file(path: Path, chunk_size: int = 65536) -> str:
"""Return SHA‑256 hash of file contents."""
hasher = hashlib.sha256()
with path.open('rb') as f:
for chunk in iter(lambda: f.read(chunk_size), b''):
hasher.update(chunk)
return hasher.hexdigest()
def find_duplicates(root_folder: str) -> dict:
"""Scan root_folder and return dict {hash: [paths...] } for duplicates."""
hash_map = defaultdict(list)
for dirpath, _, filenames in os.walk(root_folder):
for name in filenames:
file_path = Path(dirpath) / name
try:
file_hash = hash_file(file_path)
hash_map[file_hash].append(file_path)
except (OSError, PermissionError):
# Skip files we can't read
continue
# Keep only entries with more than one file (real duplicates)
return {h: p for h, p in hash_map.items() if len(p) > 1}
# ---- demo ----
if __name__ == "__main__":
folder = input("Folder to scan: ").strip()
duplicates = find_duplicates(folder)
if not duplicates:
print("✅ No duplicates found!")
else:
print("🔎 Duplicate groups:")
for i, (h, paths) in enumerate(duplicates.items(), 1):
print(f"\nGroup {i} (hash {h[:8]}…):")
for p in paths:
print(f" {p}")
What you can add next
- Command‑line arguments with argparse (e.g., --delete to auto‑remove all but the first copy).
- Progress bar via tqdm to keep the UI friendly for big folders.
- Export the report to a CSV or JSON file for later review.
Give it a try: point the script at a cluttered directory, run it, and watch the list of duplicates appear. You’ll see how a few standard libraries can turn a messy folder into a tidy workspace in minutes.
Happy cleaning! 🚀
8 ·