The Monthly Export Nobody Documented: Building a Repeatable Data Workflow

The Task That Exists Only in Someone's Head

On the third working day of every month, someone on your team exports a report, opens it in a spreadsheet, deletes a couple of rows, fixes a date column, converts it to SQL, and loads it into a reporting table. It takes them forty minutes. They have done it eighteen times. They are very good at it.

Then they take two weeks off, and someone else has to do it. That person spends three hours, gets the date column wrong, and produces a report that finance quietly distrusts for a quarter.

This pattern is extremely common and it is almost never treated as a real problem, because from the outside the task looks trivial. The work is not the difficulty. The difficulty is that the task contains perhaps fifteen small decisions, and fourteen of them have never been written down.

Why These Procedures Resist Documentation

It is worth understanding the resistance, because "just write it down" has been suggested before and did not work.

Expertise Becomes Invisible

After the fifth repetition, the person doing the task stops perceiving their own decisions as decisions. "Delete the summary row" does not feel like a step; it feels like something obviously true about the file. When they write documentation, they document the steps they still consciously perform and omit the ones that have become automatic. Those omitted steps are precisely the ones that break for a newcomer.

The Cost Is Deferred and Diffuse

Writing the procedure costs an hour this month. Not writing it costs three hours in some future month, to a different person, probably charged to a different project. The incentive structure points the wrong way, and it points the wrong way every single month.

Documentation Rots Silently

Someone writes a procedure. Two months later the source system changes a column name. The procedure is now wrong, but nothing announces this. The next person follows it, gets an error, concludes the documentation is unreliable, and reverts to asking the original person. After that the document is dead, and everyone has learned that documentation does not work here.

All three problems are addressable, but only if you write the procedure differently than people usually do.

Documenting Decisions, Not Keystrokes

The standard failure is a document that describes clicks. Click here, then here, choose this option. These break the moment a UI changes, and they teach the reader nothing, which means the reader cannot recover when something differs from the description.

A durable procedure documents four things per step, and the second and third are the ones usually missing.

• The intent. What state should exist after this step. "The data region should contain only the header and the actual records."
• The judgement. Which decision the operator is making, and on what basis. "Rows at the bottom that repeat the column totals are summary rows and must go. A row with a blank customer name but real amounts is a genuine record with missing data and must stay."
• The verification. How the operator confirms the step worked before proceeding. "The last row should be a normal record. If it says TOTAL, you have not finished."
• The failure signature. What it looks like downstream if this step was done wrong. "If a summary row survives, revenue totals will be roughly double."

That fourth item is what makes the document self-correcting. Someone debugging a wrong figure can search the procedure for the symptom and find the step that causes it, which is a far better experience than reading a checklist top to bottom.

A Worked Example

Here is the monthly export from the opening, written as a real procedure. Note how much of it is judgement rather than mechanics.

Step 1: Obtain the Source Extract

Intent: A file covering exactly the previous calendar month, with no overlap into adjacent months.

Judgement: The source system's date filter is inclusive on both ends and uses the report timezone, not UTC. Requesting the 1st to the 31st will include a handful of records that belong to the following month for teams in eastern timezones. Filter on the 1st to the last day, then verify.

Verification: Minimum and maximum of the date column fall inside the intended month.

Failure signature: Month-over-month figures show a small unexplained shift, and two consecutive months both contain the same handful of records.

Step 2: Inspect and Clean the Extract

Intent: A rectangular region containing one header row and one row per record.

Judgement: This export includes three metadata lines above the header and a totals row at the bottom. It also leaves the region column blank after the first occurrence in each group, which must be filled down. Do not delete rows with blank optional fields; only delete rows that are entirely empty or that are summary lines.

Verification: First row is the header. Last row is a normal record. Row count matches the record count shown in the source system's UI.

Failure signature: Totals inflated; or a region appearing with far fewer records than expected because blanks were never filled.

Step 3: Resolve Column Types

Intent: Every column unambiguously typed before any SQL is generated.

Judgement: The account code column looks numeric but is a fixed-width code with leading zeros, so it is text. The date column arrives as an Excel serial number and must be formatted as a date in the spreadsheet before export. Amounts use a comma decimal separator in the European regional export and must be normalised.

Verification: Account codes all have the same length. Dates render as dates, not five-digit numbers. Amount magnitudes are plausible against last month.

Failure signature: Codes failing to join against the account dimension; dates landing in 1899; amounts off by exactly 100.

Step 4: Generate the Load Script

Intent: A batched, dialect-correct INSERT script targeting the staging table.

Judgement: Target is PostgreSQL, so identifiers use double quotes rather than backticks. Table name is the staging table, never the reporting table directly. Batch insert must be enabled, since a row-by-row script for 40,000 rows takes tens of minutes rather than seconds.

Verification: Read the first and last generated batch. Column order matches the staging table. Quoting style is correct for the dialect.

Failure signature: Syntax error on execution, or a very slow load, or values silently landing in the wrong columns.

Step 5: Load, Validate, Promote

Intent: Data in the reporting table, verified before anyone sees it.

Judgement: Always truncate staging first so it reflects exactly one extract. Promotion uses upsert on the natural key so that a rerun converges rather than duplicating.

Verification: Run the saved validation query set: row count, null counts per column, min and max on dates and amounts, distinct count on the natural key, and a sum compared against the source system's own total.

Failure signature: Any of the above out of range. Stop and investigate rather than proceeding.

Tool Choices That Make Procedures Durable

The tooling you specify affects how long the document stays accurate. Some choices age much better than others.

The Develop Box Online Converters score well on all five, which is largely incidental to their design but genuinely useful here: a procedure that says "open this URL, paste, set these three options" does not break when someone's laptop is reimaged, and it does not require a licence request when the usual operator is away.

Keeping It Alive

A procedure that is never re-read is indistinguishable from no procedure. Three habits keep it accurate.

• Rotate the operator. Have a different person run it at least once a quarter, following only the document, with the usual operator available but silent. Every question the newcomer has to ask is a gap, and it should be filled immediately rather than answered verbally.
• Record the validation output each run. Not just "passed", but the actual counts and ranges. Over six months this becomes a baseline, and an out-of-family value becomes visible before anyone builds a report on it.
• Log deviations in the document itself. When something differs from the description, the fix goes into the procedure in the same session, not into a message thread. A document that visibly absorbs corrections earns trust; one that does not gets abandoned.

Frequently Asked Questions

Should we not just automate this instead of documenting it?

Eventually yes, and documenting first is how you get there. You cannot automate decisions nobody has articulated. Writing the procedure surfaces every judgement call, and at that point you can see which ones are mechanical enough to encode and which genuinely need a human. Automating before that produces a script that silently does the wrong thing.

How detailed is too detailed?

Describing which button to click is usually too detailed, because it breaks on any UI change and conveys no understanding. Describing the intended end state and the judgement involved is the right level. A competent colleague should be able to reach the same result even if the interface has moved.

Who should write it, the expert or the newcomer?

The newcomer, with the expert watching. The expert has lost visibility into their own automatic decisions, so they will omit exactly the steps that matter. A newcomer writing while being corrected captures the tacit knowledge, because every correction marks a step the expert would not have thought to mention.

Conclusion: The Value Is in the Judgement

Recurring data tasks are not hard, which is why they stay undocumented. What makes them fragile is the accumulated set of small decisions the regular operator no longer notices making.

Write those decisions down along with how to verify each one and what going wrong looks like. Rotate who runs it so the document gets tested. That converts a person-shaped dependency into a process, and it is usually one hour of work to eliminate a recurring three-hour failure.