Recipe

Write a program that processes a data file

Use when: a program must turn records in a file into a summary or a cleaned copy

Goal: a short program of one function per stage that reports bad records instead of crashing

Taught in Data-processing programs

1. 1

   Open the file as plain text and read the first lines: the separator, the header, the encoding, the kinds of mess.

   Example

   sensor,time,celsius, commas, UTF-8; one empty reading, one n/a, one name with spaces.

   Why: every later step depends on what the file contains, not what it should contain.

2. 2

   Write read_rows(path) that returns the records as they are, all strings.

   Example

   list(csv.DictReader(f)) on a file opened with encoding="utf-8", newline="".

   Why: keeping reading separate means a different source only changes this one function.

3. 3

   Write check_row(row, line) that returns a message for a bad record or None, checking text before converting anything.

   Example

   "line 6: celsius 'n/a' is not a number", found with is_number, never with float.

   Why: one bad record should be reported and skipped, not stop the program.

4. 4

   Convert the good records, then group and compute in a pure function.

   Example

   summarize(readings) gives count, low, high and mean per sensor.

   Why: a pure function can be checked with assert on a few hand-made records.

5. 5

   Create the output folder, write each format the readers need, and print how many records were read, kept and skipped.

   Example

   out/summary.csv for people, out/report.json for programs, and a one-line summary on screen.

   Why: a report makes skipped records visible, so someone can fix the source.