All lessons

Module 1: Ingest anything · Lesson 1.3 · about 35 min

Landing files from folders and shared drives

Land files from shared folders safely: skip half-written files, fingerprint each one with SHA-256, keep a manifest, and never move a source file.

Key points, meanings and formulas on one page: choose "Save as PDF" in the print box.

Module 1 · about 35 minutes · read first: 1.2. You’ll be able to land files from a shared folder so that reruns add nothing, re-sends are spotted, corrections are kept as new versions, and half-written files are never read.

Land files like a careful librarian: never take the original off the shelf, photocopy it, stamp the copy with a fingerprint, and write it in the register, so a second copy of the same pages is recognised at once.

1. Why this matters

In many medium-sized companies the most common data source is still a folder: a network share, a SharePoint library synced to a server, a vendor’s export directory. Folder loading looks trivial, and it is where most silent errors live: the file read twice, the file read while it was still being written, the file replaced with a corrected version under the same name.

2. The landing pattern

  1. Discover: list the candidate files and skip anything not finished.
  2. Fingerprint: compute a SHA-256 hash of the file’s bytes. A hash is a 64-character code worked out from every byte; change one byte and the code changes completely. Two files with the same hash have the same content.
  3. Check the manifest: the manifest is your register of every file landed. Is this hash already in it? If so, skip and log duplicate.
  4. Copy into the raw zone at a path that says the source and the business date: raw/qsr/business_date=2026-09-14/pos_Q204__a3f91c2e.csv (the first 8 characters of the hash keep names unique).
  5. Record a manifest row: source, original path, size, modified time, hash, when you received it, raw path.

You copy, you never move. The source folder belongs to someone else. Moving files breaks their process and destroys your ability to check the source again.

import hashlib, shutil, sqlite3, time
from pathlib import Path

def sha256(path: Path, chunk=1 << 20) -> str:
    h = hashlib.sha256()
    with path.open("rb") as f:
        for block in iter(lambda: f.read(chunk), b""):   # 1 MB at a time: works for huge files
            h.update(block)
    return h.hexdigest()

def is_stable(path: Path, wait=2.0) -> bool:
    """A file still being written keeps growing. Skip partials and temp names."""
    if path.suffix in {".partial", ".tmp", ".crdownload"} or path.name.startswith("~$"):
        return False
    s1 = path.stat().st_size; time.sleep(wait); s2 = path.stat().st_size
    return s1 == s2 and s1 > 0

def land(src_dir: Path, raw_dir: Path, db: sqlite3.Connection, source: str):
    db.execute("""create table if not exists manifest(
        sha256 text primary key, source text, src_path text, size int, mtime real,
        received_at text, raw_path text)""")
    for p in sorted(src_dir.rglob("*.csv")):
        if not is_stable(p):
            continue
        digest = sha256(p)
        if db.execute("select 1 from manifest where sha256=?", (digest,)).fetchone():
            continue                                       # exact same bytes already landed
        business_date = p.parent.name                      # in this source, the folder is the date
        dest = raw_dir / source / f"business_date={business_date}" / f"{p.stem}__{digest[:8]}{p.suffix}"
        dest.parent.mkdir(parents=True, exist_ok=True)
        tmp = dest.with_suffix(dest.suffix + ".tmp")
        shutil.copy2(p, tmp); tmp.replace(dest)            # rename is atomic: no half files
        db.execute("insert into manifest values (?,?,?,?,?,datetime('now'),?)",
                   (digest, source, str(p), p.stat().st_size, p.stat().st_mtime, str(dest)))
        db.commit()

Watch one run over a folder with four files in it:

Discover. The job lists four files. pos_Q207.csv.partial is still being uploaded, so it is held back until it is renamed.
New content. pos_Q204.csv hashes to a3f91c2e…, which is not in the manifest. It is copied to raw and a row is added.
Exact re-send. pos_Q205.csv hashes to 7be04d19…, already in the manifest from yesterday. Same bytes, so it is skipped and logged as a duplicate.
Correction. pos_Q206.csv has yesterday's name but a new hash: someone fixed the file. It lands as a second version; staging will use the latest.
Rerun. Run the job again a minute later. Every hash is already known, so nothing is added. Running it twice has the same effect as once.
Maths and principle

Principle: a step is idempotent if doing it twice has the same effect as doing it once. Landing keyed by content hash is idempotent.

Equation: f(f(x))=f(x). With a manifest M (a set of hashes) and a file F with hash h(F):

land(M,F)=M∪{h(F)},M∪{h}∪{h}=M∪{h}

because a set ignores repeats.

Analogy: pressing a lift button twice; the lift still comes once. Where it breaks: the same business data can arrive with different bytes (a re-export with a new “printed at” line). Then the hash differs, and you need a second, business-level key (source, store, business date) with a rule such as “latest version wins”.

Try it: a folder has 18 files today; 2 are byte-identical re-sends of yesterday’s. How many manifest rows does today add if the job runs twice? (16. The re-sends and the second run add nothing.)

3. Same name, new content

Vendors often fix a file and drop it again under the same name. Decide the rule out loud and write it in the README:

Situation Hash Name Rule
Exact re-send same any skip
Correction new same land as a new version; staging takes the latest by received time
New day new new land

Never trust the file’s modified time alone: copy tools, sync clients and unzipping all rewrite it. Never trust dates in file names alone either: people rename files. Record name, modified time and received time, and choose one business date rule per source.

Hash (SHA-256)
A fingerprint of a file's bytes. Same bytes, same fingerprint; one byte different, a completely different one.
Manifest
Your register of every file landed: where it came from, its hash, when it arrived and where the copy lives.
Idempotent
Safe to repeat: running it again changes nothing.
Atomic rename
A rename that happens all at once, so nobody can see a half-written file.
Physical meaning
Six months from now, finance will ask why March's numbers changed. The manifest answers in one query: which version of which file fed which load, and when it arrived. Because the job is idempotent, you can also rerun it after any failure without fear of double-counting.

“If the file name is in the manifest, I already have it.” Names lie: a corrected file can reuse yesterday’s name, and two different stores can send files with the same name. Key the manifest on the hash of the content, and keep the name only as information.

4. Service accounts, mapped drives and polling

A scheduled job usually runs as a service account (a login for a program, not a person). The Z: drive you mapped in your own session does not exist for it. Use full network paths (\\fileserver\exports\qsr) and give the service account read-only access. Read-only is a feature: it makes deleting the source impossible.

You can poll (list the folder every few minutes) or watch (ask the operating system for change events, for example with Python’s watchdog). Polling is simpler and works on network shares, where events are unreliable. It is what Dagster sensors do in Module 3.

5. Try it: land the restaurant shared drive

  1. In retail-ingest, create src/retail_ingest/sources/folder.py with sha256, is_stable and land (use wait=0.2 locally).
  2. Run it over data/landing/shared_drive/qsr. Count the manifest rows; it should equal the number of files.
  3. Run it again: zero new rows. Write tests/test_folder_idempotent.py that proves it, using pytest’s tmp_path.
  4. Simulate a correction: add a line to one source file and rerun. Check a second version lands with a new hash.
  5. Simulate an upload in progress: create pos_Q999.csv.partial and check it is skipped.
  6. Move the manifest to DuckDB (duckdb.connect("data/manifest.duckdb")) and query files per day, total bytes, and any day with fewer than 18 files.

Commit: feat(ingest): idempotent folder landing with sha256 manifest.

What to check: both tests pass, and your “fewer than 18 files” query returns nothing for the restaurant source (each of the 18 restaurants sends one file a day).

6. Practice

  1. A folder holds 18 files. You run the job three times. How many manifest rows?
  2. Why does is_stable check the size twice?
  3. The sync client rewrote every file’s modified time overnight. Does your job re-land them?
Answers
    1. The second and third runs add nothing.
  1. A file still being written grows between the two checks. Same size twice (and above zero) suggests the writer has finished.
  2. No. The hashes have not changed, so the manifest already has them.

New words: hash, SHA-256, manifest, idempotent, atomic rename, service account, polling.

  • Copy, never move: the source folder belongs to someone else.
  • Fingerprint every file with SHA-256 and keep a manifest keyed on that hash.
  • Skip half-written files (temp names, growing sizes) and write your own copies to a temp name, then rename.
  • A correction under the same name has a new hash: land it as a new version and let staging take the latest.

Check yourself

1Why copy files to raw instead of moving them out of the shared folder?
2A file has the same name as yesterday's but a different hash. What is the safest default?
3Why write to a temporary name and then rename?

Sources

Your ticks stay in this browser. Save them as a file on the course home.

Updated 2026-11-22. All company data in the examples is synthetic. Tool versions and prices change: check the official docs before you rely on them.