← 3c.2 Initial PAL demos


PAL DEMO D7: Deep Dive: Building Your First Pipeline (26.0720)

See the working text for this QS in

2 Setting up your Project and Folder
  2.1 Introduction	
  2.2 Create a Foundry Learning Project
  2.3 Create a Course-Specific Training Folder
3 Raw data (Data Preprocessing and Cleaning)
  3.1 Introduction
  3.2 Download the Data Sources
  xx3.3 [Optional] Deploy Datasets via Marketplace
  3.4 Create a New Pipeline
3b clean data
  3.5 Basic Transforms: Clean the Products Dataset
  3.6 Advanced Transforms: Clean the Customers Dataset
  3.6b fix the json file (BINGO BINGO) 
  3.7 Work with Different File Types: Preprocess and Clean the Transactions Dataset	
4 Joined data (Data Integration, Joining, and Aggregations)
  4.1 Introduction	
  4.2 Simple Joins: Join Transactions and Products (BINGO)	
  4.3 Chaining Joins: Join Transactions x Products with Customers	
4b business ready dataset	
  4.4 Add a Dataset Output	
  4.5 Identify a Bad Join: Drill into Transactions x Products	
  4.6 Create an Operationally Valuable Dataset


7-03
xxxxxx




NOTES



Yes — another docs/UI mismatch.

The transform the docs call:

Extract rows from a JSON file

may now appear under a different name, usually something like:

  • Parse JSON
  • JSON to rows
  • Extract records from JSON
  • Read JSON files
  • Files → JSON
  • Create dataset from JSON

Most likely causes

1. You are in the wrong tool/mode

That step is probably meant for Pipeline Builder, not Workshop, Ontology Manager, or Functions.

You should be in something like:

Pipeline Builder / Data pipeline / Add transform

Then search the transform picker for:

json

not for the full phrase.

2. Your input is not a raw file dataset

The “extract rows from JSON file” transform usually appears only if the input is a file-based dataset containing JSON files.

If your input is already a structured dataset/table, Foundry may hide that transform because there is no JSON file to extract from.

3. The transform was renamed

The current equivalent is probably:

Parse JSON

or:

Read JSON

or:

Extract records from JSON

What I would try

  1. Open the pipeline / transform builder.
  2. Click Add transform.

  3. Search only:

    json
  4. Pick the transform that converts JSON/file content into rows.
  5. If nothing appears, confirm that your input is a raw JSON file dataset, not a structured table.

Short answer

The docs are likely outdated. Look for Parse JSON or search for json in the transform picker. If no JSON transform appears, you are probably not using a raw JSON file dataset as the input.



Python transforms run in a Code Repository, not directly in Pipeline Builder. Here are the steps:

Step 1 — Create a Python Code Repository

  1. From the Foundry home screen, click New and search for Code Repository.
  2. Select Python as the language.
  3. Give it a name (e.g. transactions_json_parser) and choose a location in your project folder.

Step 2 — Install the required library

Once inside the repository, open the Libraries panel on the left side. Search for and install:

  • transforms-media — required for MediaSetInput to read from your Media Set [1]

Commit the change after installing.

Step 3 — Add the transform code

Open the default Python file (usually transforms/my_transform.py) and replace its contents with the script from before, updated with your actual output path:

from transforms.api import transform, Output
from transforms.mediasets import MediaSetInput
import json
import pandas as pd

@transform(
    media_input=MediaSetInput("ri.mio.main.media-set.e6b065a6-5fc7-455d-8051-57c964cb03cd"),
    output=Output("/ziptieai-5313a6/Pal_7/pal_7_first_pipeline/transactions_parsed"),
)
def parse_json_transactions(media_input, output):
    rows = []
    for media_item in media_input.list_media_items():
        content = media_input.get_media_item(media_item.media_item_rid).read().decode("utf-8")
        for line in content.strip().splitlines():
            if line.strip():
                rows.append(json.loads(line))
    df = pd.DataFrame(rows, columns=[
        "transaction_id",
        "transaction_date",
        "units",
        "customer_id",
        "variation_id",
        "product_id"
    ])
    output.write_dataframe(df)

Update the Output(...) path to wherever you want the clean tabular dataset to land in your project.

Step 4 — Build and run

Click the Build button (top right of the Code Repository). Foundry will run the transform and write the output dataset to the path you specified.

Step 5 — Connect to Pipeline Builder

Once the output dataset exists, you can add it as an input node in ri.eddie.main.pipeline.2b185cec-e69d-480e-91f1-d3ce7d86a0fd and continue building your pipeline from there.

Note: The transforms-media library is required because your source is a Media Set. [1] Without it, the MediaSetInput import will fail at build time.



Yes — it worked.

Your JSON file was processed into a structured dataset:

transactions_parsed

You can see it has:

50 rows
6 columns

Columns:

transaction_id
transaction_date
units
customer_id
variation_id
product_id

So the flow succeeded:

transactions_raw.json
        ↓
Python transform: fix json
        ↓
transactions_parsed dataset/table

A few notes:

  • units was converted correctly to double.
  • transaction_date is still string, which is okay for now because the source date format is messy JavaScript-style text.
  • The output is a Foundry dataset, not another JSON file.
  • The left panel confirms it was Updated via fix json, so your transform produced this dataset.

So yes: the file was processed, and the result is the dataset currently open in your screenshot.


this all looks correct (amazing .. thanks for your expert help)… but do i need to verify ? or is this already verification (looks like in the speedrun demo)

Yes — this preview is already a basic verification.

It shows the join is working:

  • Left table: clean transactions
  • Right table: clean products

  • Join condition:

    clean transactions.product_id = clean products.product_id

  • Output preview has product fields added:

    products_price
products_product_name
products_category

So visually, this matches the speedrun expectation.

But I would still do two quick checks

1. Check for unmatched products

Because this is a left join, all transactions stay even if no product match exists. If there are unmatched rows, the product columns will be null.

In the preview, look at:

products_price
products_product_name
products_category

If many are null, the join is not fully matching.

2. Check row count

Your input had 50 transaction rows, but this preview shows:

Previewing 172 rows

That could be okay if the products table has multiple rows per product_id, but it may also mean the join is multiplying rows.

This is important.

If each transaction should match exactly one product, then the joined result should still have about:

50 rows

not 172.

Possible issue

You are joining only on:

product_id

But your data also has:

variation_id

and the products table appears to have:

products_variation_id
products_product_id

If product variations are unique, the correct join may need both:

transaction.product_id = products.product_id
transaction.variation_id = products.variation_id

That would prevent row multiplication.

Recommendation

Before moving on, add a second match condition:

variation_id is equal to variation_id

or if prefixed on the right:

variation_id is equal to products_variation_id

Then preview again.

Expected result:

50 rows

If it becomes 50 rows, that is the correct join.

yes, 50 rows :)

Great — that verifies the join is correct.

Before adding the second condition, the join produced 172 rows, which meant rows were being multiplied. After adding the variation/product match, it produces:

12 That matches the transaction input row count.

So your join is now logically correct:

123 And the output keeps:

12 with product details added.

So yes — this is verified.


01
drones


~26.0720