{ "cells": [ { "cell_type": "markdown", "metadata": { "id": "bPVVve29bu6Z" }, "source": [ "# Building a Data Pipeline with dlt\n", "\n", "In this notebook, we will build a complete data pipeline from scratch using **dlt**.\n", "\n", "Our goal is simple:\n", "\n", "→ Fetch real data from an API \n", "→ Turn it into clean relational tables \n", "→ Load it into a database \n", "→ Explore and analyze it \n", "\n", "We will use the **Open Library API** as our data source and **DuckDB** as our database.\n", "\n", "Along the way, you will learn:\n", "\n", "- What a dlt source is \n", "- What a dlt pipeline does \n", "- How data moves through Extract → Normalize → Load \n", "- How to inspect and explore the final dataset \n", "\n", "By the end, you will understand not just how to run a pipeline, but what happens at each stage.\n" ] }, { "cell_type": "markdown", "metadata": { "id": "u9eCv60qV5PS" }, "source": [ "## 📦 Step 0: Install Dependencies\n" ] }, { "cell_type": "code", "execution_count": 1, "metadata": { "id": "Arp4d7KZNRTS" }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "zsh:1: no matches found: dlt[duckdb]\n" ] } ], "source": [ "# install dependencies first\n", "!pip -q install dlt[duckdb]" ] }, { "cell_type": "markdown", "metadata": { "id": "x7VGYS5hWNKQ" }, "source": [ "

In this notebook we will use:

\n", "\n", "\n", "\n", "

\n", " DuckDB is great for beginners because it requires no setup and no credentials.\n", "

" ] }, { "cell_type": "markdown", "metadata": { "id": "aQTSvnvnHWBd" }, "source": [ "## 📚 Step 1: Import Libraries" ] }, { "cell_type": "markdown", "metadata": { "id": "YFQGLTECWkpn" }, "source": [ "\n", "

In this cell we import the libraries we will use throughout the notebook:

\n", "\n", "\n" ] }, { "cell_type": "code", "execution_count": 2, "metadata": { "id": "Lm8AbbHBImjI" }, "outputs": [], "source": [ "import dlt\n", "import dlt\n", "from itertools import islice\n", "from dlt.sources.rest_api import rest_api_source" ] }, { "cell_type": "markdown", "metadata": { "id": "UFoBTwDVhzRL" }, "source": [ "## 🔗 Step 2: Define the API Source (Open Library)" ] }, { "cell_type": "markdown", "metadata": { "id": "VdKrEM-VXEY2" }, "source": [ "

\n", " In dlt, a source is the part of your pipeline that knows how to fetch data from somewhere.\n", " In this notebook, our source fetches data from the Open Library Search API.\n", "

\n", "\n", "

\n", " We define the source using rest_api_source, which lets us describe an API in a simple\n", " Python dictionary instead of writing lots of request code.\n", "

\n", "\n", "

\n", " 📖 Open Library Search API docs:
\n", " \n", " https://openlibrary.org/dev/docs/api/search\n", " \n", "

" ] }, { "cell_type": "code", "execution_count": 3, "metadata": { "id": "hOxkEKy4Kaj4" }, "outputs": [], "source": [ "def openlibrary_source(query: str = \"harry potter\"):\n", "\n", " return rest_api_source({\n", " \"client\": {\n", " \"base_url\": \"https://openlibrary.org\",\n", " },\n", " \"resource_defaults\": {\n", " \"primary_key\": \"key\",\n", " \"write_disposition\": \"replace\",\n", " },\n", " \"resources\": [\n", " {\n", " \"name\": \"books\",\n", " \"endpoint\": {\n", " \"path\": \"search.json\",\n", " \"params\": {\n", " \"q\": query,\n", " \"limit\": 100,\n", " },\n", " \"data_selector\": \"docs\",\n", " \"paginator\": {\n", " \"type\": \"offset\",\n", " \"limit\": 100,\n", " \"offset_param\": \"offset\",\n", " \"limit_param\": \"limit\",\n", " \"total_path\": \"numFound\",\n", " },\n", " },\n", " },\n", " ],\n", " })\n" ] }, { "cell_type": "markdown", "metadata": { "id": "ntKAVaEGYFgw" }, "source": [ "## 🔧 Step 3: Create the dlt Pipeline" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "id": "bxpFEetGh3lS" }, "outputs": [], "source": [ "pipeline = dlt.pipeline(\n", " pipeline_name=\"ol_demo\",\n", " destination=\"duckdb\",\n", " dataset_name=\"ol_data\",\n", " progress=\"log\" # logs the pipeline run (Optiona)\n", ")" ] }, { "cell_type": "markdown", "metadata": { "id": "y7CJ9A2HXsFb" }, "source": [ "## 🔍 Understanding the Pipeline\n", "\n", "At this point we have defined two key building blocks:\n", "\n", "- **The source** describes where the data comes from and how to fetch it from the API. \n", "- **The pipeline** describes where the data should go (DuckDB) and keeps track of tables, schemas, and run history. \n", "\n", "---\n", "\n", "Instead of running everything at once, we will now run the pipeline in three separate phases so you can clearly see what happens at each stage:\n", "\n", "1. **Extract**: download raw data from the API \n", "2. **Normalize**: turn nested JSON into relational tables \n", "3. **Load**: write those tables into DuckDB \n", "\n", "\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "![ETL Diagram](./images/etl_diagram.png)" ] }, { "cell_type": "markdown", "metadata": { "id": "pAYgUUJIw-c4" }, "source": [ "Once these steps make sense, we will run the full workflow again using one command:\n", "\n", "```python\n", "pipeline.run(source)\n" ] }, { "cell_type": "markdown", "metadata": { "id": "JsfcBA7McJMo" }, "source": [ "## ⬇️ Step 4: Extract\n", "\n", "Now we run the first stage of the pipeline: **Extract**.\n", "\n", "Extract means:\n", "\n", "- dlt sends requests to the Open Library API\n", "- the raw JSON responses are downloaded\n", "- the results are stored in dlt’s local working folder\n", "\n", "At this stage, the data is **not** in DuckDB yet. We are just confirming that we successfully pulled data from the API." ] }, { "cell_type": "code", "execution_count": 5, "metadata": { "id": "yifCIPxSKJZ4" }, "outputs": [], "source": [ "extract_info = pipeline.extract(openlibrary_source())" ] }, { "cell_type": "markdown", "metadata": { "id": "NLRRVLnLcNgl" }, "source": [ "---\n", "\n", "### What we will print\n", "\n", "After extraction, we will print a small summary showing:\n", "\n", "- which **resources** were extracted\n", "- which **tables** will be created later\n", "- how many rows were extracted per resource\n", "\n", "This helps confirm that the pipeline is working before we move on to normalization." ] }, { "cell_type": "code", "execution_count": 6, "metadata": { "colab": { "base_uri": "https://localhost:8080/" }, "id": "wtDasHRNNNN0", "outputId": "51c71eeb-5435-40a1-8728-ea48c59bfd58" }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Resources: ['books']\n", "Tables: ['books']\n", "Load ID: 1770907406.962898\n", "\n", "Resource: books\n", "rows extracted: 3756\n", "\n" ] } ], "source": [ "load_id = extract_info.loads_ids[-1]\n", "m = extract_info.metrics[load_id][0]\n", "\n", "print(\"Resources:\", list(m[\"resource_metrics\"].keys()))\n", "print(\"Tables:\", list(m[\"table_metrics\"].keys()))\n", "print(\"Load ID:\", load_id)\n", "print()\n", "\n", "for resource, rm in m[\"resource_metrics\"].items():\n", " print(f\"Resource: {resource}\")\n", " print(f\"rows extracted: {rm.items_count}\")\n", " print()" ] }, { "cell_type": "markdown", "metadata": { "id": "f6MwYtznc3UX" }, "source": [ "### What you should see after Extract\n", "\n", "In our case, Extract shows only **one resource and one table**:\n", "\n", "- **Resources:** `['books']` \n", "- **Tables:** `['books']`\n", "\n", "That is expected.\n", "\n", "The `search` endpoint returns a list of book results, so dlt stores those rows in a single table called `books`. The interesting part comes next, because many fields inside each row are lists or nested objects. Those will turn into additional tables during **Normalize**.\n", "\n", "Example output:\n", "\n", "- **25 rows extracted** means we pulled 25 search results (books) \n", "\n", "---" ] }, { "cell_type": "markdown", "metadata": { "id": "lQVLZMcyXWkm" }, "source": [ "## 🔄 Step 5: Normalize\n", "\n", "Now we run **Normalize**. This is where dlt transforms raw JSON into a clean relational structure.\n", "\n", "During normalization, dlt does three key things:\n", "\n", "### 1. Adds Tracking Columns to the Main Table\n", "\n", "dlt adds special columns to every table:\n", "- `_dlt_id`: A unique identifier for each row\n", "- `_dlt_load_id`: Links each row to the load job that created it\n", "\n", "### 2. Flattens Nested Data into Child Tables\n", "\n", "APIs often return nested JSON. For example, a book can have multiple authors (a list), multiple editions, and multiple identifiers.\n", "\n", "dlt flattens these nested structures into separate **child tables** with names like:\n", "- `books__author_name`\n", "- `books__author_key`\n", "- `books__language`\n", "\n", "Each child table has a `_dlt_parent_id` column that references `_dlt_id` in the parent table. This is how dlt maintains relationships.\n", "\n", "### 3. Creates Metadata Tables\n", "\n", "dlt also creates internal tables to track pipeline state:\n", "- `_dlt_loads`: Tracks load history (when data was loaded, status)\n", "- `_dlt_pipeline_state`: Stores pipeline state for incremental loading\n", "- `_dlt_version`: Tracks schema versions\n", "\n", "In the next cell, we will print a summary showing which tables were created.\n" ] }, { "cell_type": "code", "execution_count": 7, "metadata": { "id": "LCmiiG3tXXwh" }, "outputs": [], "source": [ "normalize_info = pipeline.normalize()" ] }, { "cell_type": "code", "execution_count": 8, "metadata": { "colab": { "base_uri": "https://localhost:8080/" }, "id": "-kNiY112Xvuk", "outputId": "502bff6b-edb2-4bd8-a9e9-1f1b88f20c48" }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Load ID: 1770907406.962898\n", "\n", "Tables created/updated:\n", " - books: 3756 rows\n", " - books__author_key: 4600 rows\n", " - books__author_name: 4600 rows\n", " - books__ia: 3422 rows\n", " - books__ia_collection: 2724 rows\n", " - books__language: 3748 rows\n", " - books__id_standard_ebooks: 12 rows\n", " - books__id_librivox: 60 rows\n", " - books__id_project_gutenberg: 54 rows\n" ] } ], "source": [ "load_id = normalize_info.loads_ids[-1]\n", "m = normalize_info.metrics[load_id][0]\n", "\n", "print(\"Load ID:\", load_id)\n", "print()\n", "\n", "print(\"Tables created/updated:\")\n", "for table_name, tm in m[\"table_metrics\"].items():\n", " # skip dlt internal tables to keep it beginner-friendly\n", " if table_name.startswith(\"_dlt\"):\n", " continue\n", " print(f\" - {table_name}: {tm.items_count} rows\")\n" ] }, { "cell_type": "markdown", "metadata": { "id": "ctHuJ0yEdNaq" }, "source": [ "### What happened during Normalize?\n", "\n", "After running `pipeline.normalize()`, we now see multiple tables instead of just one.\n", "\n", "Tables created/updated:\n", "\n", "- `books`\n", "- `books__author_key`\n", "- `books__author_name`\n", "- `books__editions__docs`\n", "- `books__editions__docs__language`\n", "- `books__ia`\n", "\n", "---\n", "\n", "### What does this mean?\n", "\n", "We started with **N book search results** in the `books` table.\n", "\n", "During normalization:\n", "\n", "- Each book may have **more than N authors**, so those were split into:\n", " - `books__author_name`\n", " - `books__author_key`\n", "\n", "- Each book may contain **edition information**, which became:\n", " - `books__editions__docs`\n", "\n", "- Some editions contain **language information**, which became:\n", " - `books__editions__docs__language`\n", "\n", "- The `ia` field (Internet Archive IDs) is a list, so it became:\n", " - `books__ia`\n", "\n", "This is the key moment in the pipeline.\n", "\n", "The data has been transformed from nested JSON into a **relational structure** with multiple linked tables. This makes it much easier to query and analyze.\n", "\n", "---\n", "\n", "### Schema Visualization\n", "\n", "dlt can render the schema as a visual diagram. Run the next cell to see the parent-child table relationships:" ] }, { "cell_type": "code", "execution_count": 13, "metadata": {}, "outputs": [ { "data": { "text/html": [ "\n", "\n", "\n", "\n", "\n", "
\n", "\n" ], "text/plain": [ "" ] }, "execution_count": 13, "metadata": {}, "output_type": "execute_result" } ], "source": [ "# Display schema \n", "pipeline.default_schema" ] }, { "cell_type": "markdown", "metadata": { "id": "lJ5QzSnYdidK" }, "source": [ "## 📤 Step 6: Load\n", "\n", "Now we run the final stage of the pipeline: **Load**.\n", "\n", "Load means:\n", "\n", "- dlt creates tables in DuckDB (if they do not already exist)\n", "- the normalized rows are inserted into those tables\n", "- the pipeline records the load in its internal tracking tables\n", "\n" ] }, { "cell_type": "code", "execution_count": 10, "metadata": { "id": "d9Xb67c5XfL5" }, "outputs": [], "source": [ "load_info = pipeline.load()" ] }, { "cell_type": "markdown", "metadata": { "id": "ehkz8lESGGdm" }, "source": [ "\n", "After this step, the data is fully stored in the database and ready to query.\n", "\n", "At this point:\n", "\n", "- The `books` table contains our books\n", "- The related tables (such as `books__author_name` and `books__editions__docs`) contain the exploded nested data\n", "- Everything is now queryable using `pipeline.dataset()` or SQL\n", "\n", "This is the moment where the data officially moves from “pipeline processing” into a database you can explore." ] }, { "cell_type": "markdown", "metadata": { "id": "jBznxM00eCOF" }, "source": [ "## 🚀 Step 7: Run the Full Pipeline\n", "\n", "Now that we have walked through each step individually, we can run the entire workflow using a single command:\n", "\n" ] }, { "cell_type": "code", "execution_count": 11, "metadata": { "id": "YQLigkh-f7Ey" }, "outputs": [], "source": [ "load_info = pipeline.run(openlibrary_source())" ] }, { "cell_type": "markdown", "metadata": { "id": "SbLkA8W7eNPb" }, "source": [ "

What does pipeline.run() do?

\n", "\n", "

\n", " pipeline.run() simply combines the three steps we already executed manually:\n", "

\n", "\n", "
    \n", "
  1. Extract – fetch data from the Open Library API
    2. \n", "
  2. Normalize – convert nested JSON into relational tables
    3. \n", "
  3. Load – write those tables into DuckDB
    4. \n", "
\n", "\n", "

In other words, this:

\n", "\n", "
pipeline.run(source)
\n", "\n", "

is equivalent to:

\n", "\n", "
pipeline.extract(source)\n",
    "pipeline.normalize()\n",
    "pipeline.load()
\n", "\n", "

\n", " There is no hidden magic. It just runs the full ELT process in order.\n", "

\n" ] }, { "cell_type": "markdown", "metadata": { "id": "7ViMq6gIfJj_" }, "source": [ "## 🔎 Step 8: Inspect the Loaded Data\n", "\n", "Now that the data is loaded into DuckDB, we can inspect it using `pipeline.dataset()`.\n", "\n", "This gives us a convenient Python interface for exploring the tables that dlt created, without writing SQL.\n", "\n", "---\n", "\n", "### List available tables\n", "\n", "First, let’s see what tables exist in the dataset:" ] }, { "cell_type": "code", "execution_count": 12, "metadata": { "id": "bmnrK1aVZXPO" }, "outputs": [], "source": [ "ds = pipeline.dataset()" ] }, { "cell_type": "code", "execution_count": 13, "metadata": { "colab": { "base_uri": "https://localhost:8080/" }, "id": "SV6J6AtBf0xq", "outputId": "19ad26bf-f34a-4f8e-c30c-5acd3342c3c5" }, "outputs": [ { "data": { "text/plain": [ "['books',\n", " 'books__author_key',\n", " 'books__author_name',\n", " 'books__ia',\n", " 'books__ia_collection',\n", " 'books__language',\n", " 'books__id_standard_ebooks',\n", " 'books__id_librivox',\n", " 'books__id_project_gutenberg',\n", " '_dlt_version',\n", " '_dlt_loads',\n", " '_dlt_pipeline_state']" ] }, "execution_count": 13, "metadata": {}, "output_type": "execute_result" } ], "source": [ "ds.tables" ] }, { "cell_type": "code", "execution_count": 17, "metadata": { "colab": { "base_uri": "https://localhost:8080/", "height": 315 }, "id": "WLa4yN7lf1TF", "outputId": "d2da841b-a8bf-461f-a011-eb1db644656f" }, "outputs": [ { "data": { "application/vnd.google.colaboratory.intrinsic+json": { "summary": "{\n \"name\": \"df\",\n \"rows\": 3756,\n \"fields\": [\n {\n \"column\": \"cover_edition_key\",\n \"properties\": {\n \"dtype\": \"category\",\n \"num_unique_values\": 1192,\n \"samples\": [\n \"OL24951484M\",\n \"OL9131663M\",\n \"OL47198575M\"\n ],\n \"semantic_type\": \"\",\n \"description\": \"\"\n }\n },\n {\n \"column\": \"cover_i\",\n \"properties\": {\n \"dtype\": \"Int64\",\n \"num_unique_values\": 1288,\n \"samples\": [\n 842156,\n 10365881,\n 3341732\n ],\n \"semantic_type\": \"\",\n \"description\": \"\"\n }\n },\n {\n \"column\": \"ebook_access\",\n \"properties\": {\n \"dtype\": \"category\",\n \"num_unique_values\": 5,\n \"samples\": [\n \"printdisabled\",\n \"unclassified\",\n \"no_ebook\"\n ],\n \"semantic_type\": \"\",\n \"description\": \"\"\n }\n },\n {\n \"column\": \"edition_count\",\n \"properties\": {\n \"dtype\": \"number\",\n \"std\": 108,\n \"min\": 0,\n \"max\": 3546,\n \"num_unique_values\": 62,\n \"samples\": [\n 44,\n 92,\n 396\n ],\n \"semantic_type\": \"\",\n \"description\": \"\"\n }\n },\n {\n \"column\": \"first_publish_year\",\n \"properties\": {\n \"dtype\": \"Int64\",\n \"num_unique_values\": 127,\n \"samples\": [\n 2008,\n 1622,\n 1962\n ],\n \"semantic_type\": \"\",\n \"description\": \"\"\n }\n },\n {\n \"column\": \"has_fulltext\",\n \"properties\": {\n \"dtype\": \"boolean\",\n \"num_unique_values\": 2,\n \"samples\": [\n false,\n true\n ],\n \"semantic_type\": \"\",\n \"description\": \"\"\n }\n },\n {\n \"column\": \"key\",\n \"properties\": {\n \"dtype\": \"string\",\n \"num_unique_values\": 3756,\n \"samples\": [\n \"/works/OL34662215W\",\n \"/works/OL39702699W\"\n ],\n \"semantic_type\": \"\",\n \"description\": \"\"\n }\n },\n {\n \"column\": \"lending_edition_s\",\n \"properties\": {\n \"dtype\": \"category\",\n \"num_unique_values\": 281,\n \"samples\": [\n \"OL45637056M\",\n \"OL26064272M\"\n ],\n \"semantic_type\": \"\",\n \"description\": \"\"\n }\n },\n {\n \"column\": \"lending_identifier_s\",\n \"properties\": {\n \"dtype\": \"category\",\n \"num_unique_values\": 281,\n \"samples\": [\n \"alicesadventures0000unse_v7d2\",\n \"harrypottermagic0000unse_n5w6\"\n ],\n \"semantic_type\": \"\",\n \"description\": \"\"\n }\n },\n {\n \"column\": \"public_scan_b\",\n \"properties\": {\n \"dtype\": \"boolean\",\n \"num_unique_values\": 2,\n \"samples\": [\n true,\n false\n ],\n \"semantic_type\": \"\",\n \"description\": \"\"\n }\n },\n {\n \"column\": \"title\",\n \"properties\": {\n \"dtype\": \"string\",\n \"num_unique_values\": 2984,\n \"samples\": [\n \"1000 Facts and Trivia about Marvel Cinematic Universe, Game of Thrones, Disney, Star Wars, Harry Potter 1\",\n \"The Unofficial Harry Potter Insults Handbook\"\n ],\n \"semantic_type\": \"\",\n \"description\": \"\"\n }\n },\n {\n \"column\": \"_dlt_load_id\",\n \"properties\": {\n \"dtype\": \"category\",\n \"num_unique_values\": 1,\n \"samples\": [\n \"1770819876.9353185\"\n ],\n \"semantic_type\": \"\",\n \"description\": \"\"\n }\n },\n {\n \"column\": \"_dlt_id\",\n \"properties\": {\n \"dtype\": \"string\",\n \"num_unique_values\": 3756,\n \"samples\": [\n \"ZN3UfCkWBXFxSw\"\n ],\n \"semantic_type\": \"\",\n \"description\": \"\"\n }\n },\n {\n \"column\": \"subtitle\",\n \"properties\": {\n \"dtype\": \"category\",\n \"num_unique_values\": 59,\n \"samples\": [\n \"Hogwarts Through the Years\"\n ],\n \"semantic_type\": \"\",\n \"description\": \"\"\n }\n }\n ]\n}", "type": "dataframe", "variable_name": "df" }, "text/html": [ "\n", "
\n", "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
cover_edition_keycover_iebook_accessedition_countfirst_publish_yearhas_fulltextkeylending_edition_slending_identifier_spublic_scan_btitle_dlt_load_id_dlt_idsubtitle
0OL61027601M15155833borrowable3961997True/works/OL82563WOL38565767Mharrypotterylapi0000rowl_q5r6FalseHarry Potter and the Philosopher's Stone1770819876.9353185lGJrV2BS8Z9qJQNone
1OL26378158M15158660printdisabled1442007True/works/OL82586WNoneNoneFalseHarry Potter and the Deathly Hallows1770819876.9353185F9W0WQlLwgvsFwNone
2OL26234270M10580435borrowable2781999True/works/OL82536WOL48101764Mbdrc-W8LS66814FalseHarry Potter and the Prisoner of Azkaban1770819876.9353185kSdfO1XbBVAjmQNone
\n", "
\n", "
\n", "\n", "
\n", " \n", "\n", " \n", "\n", " \n", "
\n", "\n", "\n", "
\n", "
\n" ], "text/plain": [ " cover_edition_key cover_i ebook_access edition_count \\\n", "0 OL61027601M 15155833 borrowable 396 \n", "1 OL26378158M 15158660 printdisabled 144 \n", "2 OL26234270M 10580435 borrowable 278 \n", "\n", " first_publish_year has_fulltext key lending_edition_s \\\n", "0 1997 True /works/OL82563W OL38565767M \n", "1 2007 True /works/OL82586W None \n", "2 1999 True /works/OL82536W OL48101764M \n", "\n", " lending_identifier_s public_scan_b \\\n", "0 harrypotterylapi0000rowl_q5r6 False \n", "1 None False \n", "2 bdrc-W8LS66814 False \n", "\n", " title _dlt_load_id \\\n", "0 Harry Potter and the Philosopher's Stone 1770819876.9353185 \n", "1 Harry Potter and the Deathly Hallows 1770819876.9353185 \n", "2 Harry Potter and the Prisoner of Azkaban 1770819876.9353185 \n", "\n", " _dlt_id subtitle \n", "0 lGJrV2BS8Z9qJQ None \n", "1 F9W0WQlLwgvsFw None \n", "2 kSdfO1XbBVAjmQ None " ] }, "execution_count": 17, "metadata": {}, "output_type": "execute_result" } ], "source": [ "df = ds.books.df() # main table\n", "df.head(3)" ] }, { "cell_type": "markdown", "metadata": { "id": "OWFqaH2wgCWR" }, "source": [ "## 💡 Conclusion\n", "\n", "### What dlt handled for us\n", "\n", "✔ API requests \n", "✔ JSON normalization \n", "✔ Table creation \n", "✔ Database loading \n", "✔ Simple dataset inspection \n", "\n", "---\n", "\n", "### But there are still friction points\n", "\n", "• Getting the REST API config exactly right \n", "• Remembering paginator syntax \n", "• Remembering how to inspect tables \n", "• Debugging schema or pagination issues \n", "• Writing Python or SQL to get insights \n", "\n", "It works... but it still takes effort.\n", "\n", "---\n", "\n", "## 🚀 Next Up: LLM-Powered Workflows\n", "\n", "dlt now integrates LLMs directly into the workflow to make:\n", "\n", "• Pipeline runs easier \n", "• Debugging faster \n", "• Schema inspection simpler \n", "• Data analysis more natural \n", "\n", "Instead of writing glue code, you can use natural language.\n", "\n", "In the workshop, we will see what that looks like.\n" ] }, { "cell_type": "code", "execution_count": 17, "metadata": { "id": "BweSVO3igErN" }, "outputs": [], "source": [] } ], "metadata": { "colab": { "provenance": [], "toc_visible": true }, "kernelspec": { "display_name": "Python 3 (ipykernel)", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.13.5" } }, "nbformat": 4, "nbformat_minor": 4 }