A folder of incoming files is not yet a reliable data pipeline. Files can arrive late, carry unexpected encodings, reuse old names, or contain records that look valid until a downstream system tries to use them. Batch file processing works best when it treats intake, transformation, and acceptance as separate stages.
Imagine a fictional team receiving inventory exports from several suppliers. The exports describe similar concepts, but their column names and conventions differ. The team needs one normalized dataset and a clear account of anything it could not accept. This guide uses that example to show how a manifest and explicit validation rules make file processing explainable, repeatable, and easier to repair.
Decide what a completed batch means
Start with the receiving team's needs. Must every expected supplier file be present, or can a partial dataset be published? Are all records mandatory, or can invalid rows be set aside? Does a late correction replace an earlier file or create a new version of the dataset?
Write these decisions into the batch contract. In our example, each supplier contributes a versioned export, and publication waits until every required supplier either passes validation or has an approved exception. A successful worker exit is not enough to declare the dataset complete.
Define a cutoff and a policy for late arrivals. Otherwise the meaning of the batch can change while consumers are reading it. Keep the accepted input set fixed for each published version, even when the overall collection process continues receiving new material.
Create an intake manifest before transformation
Record each expected file with a stable identifier, supplier identifier, source location, checksum, and expected schema version. Add observed properties when the file arrives, including its actual type and size. The manifest connects the business expectation to the bytes being processed.
Keep original files in a controlled, immutable location for the run. Do not normalize them in place. A later correction should be a new source version so reviewers can explain why a published record changed and which input supplied the revised value.
Avoid treating a filename as trustworthy metadata. A file called current-inventory.csv could be old, incomplete, or not a CSV at all. Use the manifest to express what the pipeline expects and use validation to determine whether the received content actually meets that expectation.
Separate parsing from business validation
Parsing asks whether the file can be read according to a format. Business validation asks whether the extracted values are acceptable. A CSV row can parse successfully while carrying an impossible quantity, an unknown supplier code, or a date in the wrong business period.
Python's CSV module documentation describes reader and writer support, including dictionary-based access through column names. It also documents opening file objects with newline handling appropriate to the module. Those mechanics help read a CSV correctly; they do not replace a schema or business rules.
For each supplier, define accepted columns, required fields, encoding expectations, and the interpretation of empty values. Decide whether an empty quantity means unknown, zero, or invalid. Do not silently choose a meaning because it makes the transformation easier to complete.
Normalize with reversible, documented rules
A transformation should be specific enough that another person can reproduce it. Record mappings from source columns to destination fields, units, date interpretation, whitespace treatment, and accepted identifiers. Keep those mappings versioned alongside the code that applies them.
In the inventory example, two suppliers might report the same item in individual units and cartons. The conversion requires an explicit pack size from an approved reference, not an assumption based on the item name. Preserve the original quantity and unit when that evidence is needed to explain the normalized value.
Avoid destructive cleanup that hides a problem. Removing every nonnumeric character from a quantity might turn an annotated or invalid value into a plausible but wrong number. Route ambiguous values to review or rejection with a reason that tells the supplier what needs correction.
Quarantine failures at the right level
Some problems affect one row, while others invalidate the whole file. A single unknown product code may be a row-level exception. A missing header, unreadable encoding, or unrecognized schema version may mean the entire file cannot be interpreted safely.
Give each exception a stable code, source reference, and location within the input where appropriate. Include enough context for repair without copying sensitive content into broad-access logs. The person responsible for fixing the file should be able to locate the problem without guessing which run produced the message.
Do not let a quarantine folder become an invisible dead end. Record ownership and the conditions for reprocessing. A corrected file should enter as a new version, while the rejected version remains traceable in the history. The original failed run should not quietly change its accepted inputs.
Protect the processing boundary
Treat incoming files as untrusted until they pass the required checks. Limit accepted types and sizes, control where temporary files can be written, and avoid executing embedded content or shell instructions taken from a filename. Apply resource limits to decoding and extraction work.
Archives need their own policy. Decide whether they are accepted, how deeply nested content may be, and how extraction destinations are validated. A supplier export workflow usually benefits from a narrow set of supported formats rather than accepting every container a sender might attach.
Keep access scoped to the job's inputs and outputs. A worker that normalizes supplier inventory does not need broad access to unrelated employee documents. These boundaries also make later self-hosted deployment planning clearer because the data paths and operational responsibilities are already explicit.
Publish only after reconciliation
Produce a per-file and per-record summary before publishing. Count accepted, rejected, and unprocessed items so the totals reconcile to the input manifest. When a business total is meaningful, compare it before and after transformation using a rule that accounts for approved filtering or unit conversion.
Write output to a staging location, then validate the published representation itself. A correct in-memory dataset does not prove that an exported file preserved the intended encoding, field order, or numeric precision. Test the format that the downstream team will actually consume.
Publish a versioned dataset with its completion manifest. Include the transform version, accepted source versions, and any approved exceptions. A consumer should be able to identify the exact dataset it used without relying on whichever file happens to have the newest modification time.
Make reprocessing a deliberate operation
A rerun should declare whether it is retrying an interrupted operation or producing a new interpretation of the inputs. The first aims to finish the original contract. The second may use a changed schema, corrected source, or revised transformation and should create a new result version.
Use item-level state to avoid repeating already accepted work unnecessarily. If one supplier's file failed to download, a retry need not transform all the other suppliers again. However, shared reference data changes may legitimately require a broader rebuild; record that dependency rather than assuming every output is independent.
The batch files guide collects the main design decisions, and the batch manifest reference shows a small example structure. Keep the operational playbook close to those contracts so repair instructions evolve with the pipeline rather than becoming a separate, outdated document.
Conclusion: make every file accountable
A dependable file pipeline can explain what it expected, what it received, what it accepted, and what remains unresolved. Preserve original inputs, separate parsing from business validation, and publish only after reconciling the result to the manifest. The resulting dataset is useful not simply because it exists, but because another team can understand and trust the process that produced it.
