Data file input

This step reads records from a file produced by the Data file output step.

The file format is detected automatically from the filename extension — the same extensions that Data file output writes are read back here.

Supported file types

Extension	Format
.ndjson, .json	Newline-delimited JSON: a JSON schema line followed by one JSON array of values per row. The field layout (names, types, lengths) is rebuilt from the schema line.
.csv	Comma-separated values (RFC 4180): a JSON schema line followed by comma-separated records. Records may span several physical lines when a quoted field contains a line break. Dates are read as ISO-8601 in UTC and the file is treated as UTF-8.
.ndjson.gzip, .csv.gzip	The ndjson / csv formats above, gzip-compressed. .gz works as well.
any other extension	Cube — the gzip-compressed binary format written by Data file output.

Note

Files are streamed one row at a time, so the step can read large files with a constant, small memory footprint. Use Preview in the dialog to check the field layout and the first rows.

File options

Option	Description
Transform name	Name of the transform. This name has to be unique in a single pipeline.
Filename	The filename and location of the input file. The extension decides the file format (see above).
Limit size	Maximum number of rows to read. 0 reads the whole file.
Add filename to result	Adds the read filename to the internal result filename set for further processing.