Online / 5 & 6 February 2022

visit

Transforming and ingesting complex JSON data with Python

Transform and insert complex JSONs into a relational database - without Pandas


Python is a very popular language to write data pipelines, or just process data in general. In this talk, I will show you practical solutions to common data processing challenges when working with complex data files in Python. I will cover some lessons about transforming and ingesting large and deeply nested JSON files into a relational database like PostgreSQL. Even better: using only Python dictionaries and a database connector library (Psycopg2).

During my work, I often need to download and ingest data from different sources. Routinely, I use Python to download the data, transform it, then insert it into a database. But sometimes I come across challenges that make me scratch my head. I’d like to explain how to efficiently organize, transform, and eventually ingest complex JSON files into a PostgreSQL (or any other) relational database. I’m going to use real-world examples and make it easy for everyone to replicate the same solutions in other projects.

In this talk, I will show you practical solutions to common data processing challenges when working with complex data files in Python. I will cover some lessons about transforming and ingesting large and deeply nested JSON files into a relational database like PostgreSQL. Even better: using only Python dictionaries and a database connector library (Psycopg2).

Making sense of complex JSON files can be difficult, especially if you are dealing with loosely structured data or data that you don’t know much about. In this session, you will see a blueprint for downloading, transforming, and (batch) ingesting data from an API with Python. You will see how to tackle some of the challenges of working with deeply nested data structures, a difficult database schema with lots of foreign keys, all the while trying not to rely on large external dependencies like Pandas.

Speakers

Photo of Attila Tóth Attila Tóth

Links