Pipelines · pipeline/

Pipelines that load, model and check your data, on a schedule in code.

Ingestion, models, checks and schedules in Python, in the pipeline/ folder of your repository. The platform builds the graph, runs it on managed compute and shows the lineage of every table.

Pipeline jobs with their schedules, assets and runs. Start one by hand, or let the schedule in code run it.

cloud.segmentstream.com
The console's jobs page: pipeline jobs with their schedules, asset counts and the status of the last run

Ingestion

Ingestion assets load any API by day through a connection, with retries and backfills. Re-running a day replaces it exactly.

Models

Models are Python dataframe expressions compiled to warehouse SQL, with declared schemas, tiers and partitions.

Checks

Checks run with the pipeline and record their result next to the table, where lineage and agents can see it.

Schedules

Schedules are declared in code beside the jobs, with a timezone and the partitions each run covers. Pause or resume them from the CLI.

Lineage and run history

See what feeds what, which run produced a table and what failed upstream, in the console and the CLI.

Managed runs

Runs on managed compute. There are no clusters, workers or orchestrator to operate.

In code

pipeline/definitions.py
import segmentstream as srevenue = s.job("revenue_pipeline", assets=[revenue_summary], include_upstream=True)defs = s.definitions(    assets=[raw_orders, normalized_orders, revenue_summary],    checks=[positive_order_amounts],    jobs=[revenue],    schedules=[        s.schedule(            "revenue_morning",            job=revenue,            cron="0 6 * * *",            timezone="Europe/London",            partitions=s.LatestPartitions(3),        ),    ],)
Pipeline SDK reference →

Use it, build on it, or have it built

Use it

Run, backfill and inspect jobs from the console or the CLI.

Build it with your agent

Your agent writes ingestion assets, models and checks, one per file, and validates the graph before it deploys.

Have our engineers build it

Our engineers build the first pipelines with you, from the tools you already run.

Replaces Fivetran, dbt Cloud, Airflow and Hand-written alert queries.

Works with

Solutions built with it

Read more

Enterprise pilot

Start a pilot.

Want pipelines set up for you? Our forward-deployed engineers build it with you in your workspace, as code you own.

  • One solution, on your data, in your own workspace
  • First version live in two weeks
  • Parity proven before anything is switched off
  • Fixed scope and price, credited to your contract if you continue

Not ready for a pilot? Send the form anyway with your question. An engineer will reply, not a sales rep.

How the pilot works

Goes to our founder’s inbox. No sales queue, no newsletter.

Your company’s software, built by agents, owned by you.

$ segmentstream workspace create acme --region eu --subdomain acme

▸ Created workspace acme · database, warehouse and a private git repository

$ segmentstream workspace WORKSPACE-ID clone acme && cd acme

▸ your agent writes the five folders and runs the checks

$ git add -A && git commit -m "First version"

$ segmentstream deploy

▸ app, ontology API and MCP tools live for your invited members

Start a pilot

Our engineers build the first version on your data in two weeks, at a fixed price.

Or join the waitlist for self-serve:

We’ll email you product updates, tips and onboarding help. Unsubscribe anytime.

Self-serve opens to the waitlist first.