Task
Sales exports came from different systems in different encodings and with different separators, with repeats and empty cells. They had to land in one place without letting broken files through, and the data had to be easy to look at without Excel.
Solution
Built a web application on Django and pandas. On save the file is copied to the raw layer, read with a fallback over encodings and run through the quality checks. If a check fails, every layer of that upload is removed and the user sees the list of errors. Otherwise the data goes to ingest, table and destilled, where the preview, the charts and the export read it.
What was done
- Built CSV uploads in the admin with a choice of separator
- Detect the encoding automatically: UTF-8, UTF-8 with BOM or Windows-1251
- Check the file before saving: repeated rows and gaps reject the upload
- Lay the data out in layers: source, UTF-8 CSV, typed Parquet and deduplicated Parquet
- Built a preview of the first rows of a dataset
- Built a chart builder where columns are dragged onto the axes
- Show column statistics and export the dataset back to CSV
Result
A broken export never reaches the data: the check stops it and shows what to fix. A dataset can be previewed, charted and exported back without Excel. The code is open on GitHub.
Technologies
PythonDjangopandaspyarrowParquetPlotlyBootstrap
