Sources & schema — name your columns

The question for this lesson: before Clinker can transform your data, it has to read it and know what each column is: text, a number, a date. Where does that come from?

What you’ll be able to do

• Declare a schema on a source: name and type each column.
• Choose the right type for a column, and know which types exist.
• Decide what happens to columns that aren’t in your schema.

Clinker does not guess types

✋ Predict first

A CSV has a column `amount` with values like 4000 and 1200. If you don't tell Clinker that `amount` is a number, how should it treat the value 4000?

Decide on your answer before you open the reveal — committing a guess is what makes it stick.

Reveal the answer

As text: the characters “4000”, not the number 4000. Clinker never infers types from the file; a value is whatever your schema says it is, and an undeclared value stays text. That’s why you declare a schema: so amount can be a number you can compare and do math on.

This is the most important thing to know about sources: you declare the columns and their types; Clinker does not infer them. A CSV header only names columns; it never sets types.

Declaring a schema

A source’s schema: is a list of { name, type } entries:

- type: source
  name: customers
  config:
    type: csv
    path: ./data/customers.csv
    options: { has_header: true }
    schema:
      - { name: customer_id, type: string }
      - { name: status,      type: string }
      - { name: amount,      type: int }
      - { name: signed_up,   type: date }

The types you can use in an inline schema:

Type	For
string	text
int	whole numbers
float	decimal numbers
bool	true / false
date, date_time	a date, or a date + time
array, map	a list, or a set of key/values

🌱 New here? — picking a type

Match the type to how you’ll use the column, not just what it looks like. A zip code looks like a number but you never do math on it, so make it string and the leading zero (“01970”) survives. A price you’ll compare or total should be int or float.

Two footguns

The inline schema spells the date-and-time type date_time (with an underscore), and it has no decimal type, so use float. (A separate .schema.yaml format for fixed-width files uses different spellings; you won’t need it here.)

Only the listed types are real

“Declared, not inferred” cuts both ways: the type you write has to be one Clinker knows. A made-up type like dollars is not silently accepted or guessed. It’s rejected before any data is read. If you change amount’s type to dollars and validate:

schema:
  - { name: customer_id, type: string }
  - { name: amount, type: dollars }    # not a real type

clinker run pipeline.yaml --dry-run stops with exit 1 and tells you exactly which names are valid:

pipeline error in pipeline.yaml: config error: YAML parse error:
unknown variant `dollars`, expected one of null, bool, int, float,
string, date, date_time, array, map, numeric, any, nullable

That list (null, bool, int, float, string, date, date_time, array, map, numeric, any, nullable) is the full set of names the inline schema accepts.

// quick check

A column holds US zip codes like 01970. What type should it be?

• string, because you never do math on it and you must keep the leading zero
• int, because it looks like a number
• float, to be safe

Columns that don’t fit: on_unmapped

What about columns in the file that you didn’t list in the schema? The on_unmapped setting decides:

• auto_widen (default): keep them, as text.
• reject: fail if the file has a column you didn’t declare.
• drop: silently ignore them.

config:
  type: csv
  path: ./data/customers.csv
  options: { has_header: true }
  on_unmapped: reject     # be strict: an unexpected column is an error
  schema:
    - { name: status, type: string }

Try it

Predict, then run: this source declares amount as int. Trace what the output type of amount is, versus customer_id (declared string).

✋ Predict first

amount is declared int and customer_id is declared string. After the source reads a row with customer_id=C001 and amount=4000, what type is each value?

Reveal the expected result

=== Resolved Outputs ===

'out' → ./output/customers.csv
    [if_exists=overwrite, write_meta=false]

Dry run: config valid, 1 inputs, 1 outputs, 0 transforms

pipeline.yaml // editable

reset ⌖ explain ✓ dry-run ▶ run

pipeline: name: customer_schema nodes: - type: source name: customers config: name: customers type: csv path: ./data/customers.csv options: { has_header: true } schema: - { name: customer_id, type: string } - { name: amount, type: int } - type: output name: out input: customers config: name: out type: csv path: ./output/customers.csv

A --dry-run validates the config and the declared schema without reading any data, so the summary line reports the shape of the pipeline (here, one source, one output, and no transforms) and exits cleanly. It does not print a column-by-column type table; the place the types live is the schema: you wrote.

✓ Checkpoint

// quick check

Where do a column's types come from?

• The schema you declare on the source, where Clinker never infers them
• Clinker guesses from the first few rows of the file
• The output node decides them

// quick check

Looking back at the notional machine from lesson 00 (source → nodes → output, with a DLQ off to the side): where does the source sit, and what does it hand to the rest of the pipeline?

• At the front, reading the file and producing records that flow into the downstream nodes
• At the end, writing the final output file
• Off to the side, only catching rejected rows

Common mistakes

• Leaving a number as text: if you’ll compare or total a column, declare it int/float.
• Using decimal or datetime: the inline schema uses float and date_time.

You should be able to:

• Declare a schema with the right type for each column
• Explain why a zip code should be string, not int
• Say what auto_widen, reject, and drop do to undeclared columns

Verify in the checkout:

clinker run customer_etl.yaml --dry-run
clinker run customer_etl.yaml --explain