DagsterDocs
GitHub iconSlack iconTwitter iconStack Overflow icon

Execution#

Dagster provides several different ways to execute pipelines.

Relevant APIs#

NameDescription
execute_pipelineA method to execute a pipeline synchronously, typically for running scripts or testing.

Overview#

There are several different ways to execute pipelines. This page explains different ways to do one-off execution of pipelines: Dagit, Dagster CLI, or Python APIs.

You can also launch pipelines in other ways:

  • Schedules can be used to launch runs on a fixed interval.
  • Sensors allow you to launch runs based on external state changes.

Executing a Pipeline#

from dagster import pipeline, solid


@solid
def return_one():
    return 1


@solid
def add_two(i: int):
    return i + 2


@solid
def multi_three(i: int):
    return i * 3


@pipeline
def my_pipeline():
    multi_three(add_two(return_one()))

Dagit#

Dagster comes with a web-based interface for viewing and interacting with pipelines and other Dagster objects.

To view your pipeline in Dagit, you can use the dagit command:

dagit -f my_pipeline.py

Then navigate to http://localhost:3000 to start using Dagit:

pipeline-def

Click on the "Playground" tab, then press the "Launch Execution" button to execute the pipeline. You will then see Dagit launches a pipeline run:

pipeline-run

Dagit Playground also offers a configuration editor to let you interactively build up the configuration. See details in Dagit.

Dagster CLI#

The dagster CLI includes both dagster pipeline execute for direct execution and dagster pipeline launch for launching runs asynchronously using the run launcher on your instance.

To execute your pipeline directly, you can simply run:

dagster pipeline execute -f my_pipeline.py

Python APIs#

Dagster includes Python APIs for execution that are useful when writing tests or scripts.

execute_pipeline executes a pipeline and returns a PipelineExecutionResult.
from dagster import execute_pipeline

if __name__ == "__main__":
    result = execute_pipeline(my_pipeline)

You can find the full API documentation in Execution API and learn more about the testing use cases in Testing.

Executing Pipeline Subset#

Dagster supports ways to run a subset of a pipeline, called Solid Selection.

Solid Selection Syntax#

To specify solid selection, Dagster supports a simple query syntax.

It works as follows:

  • A query includes a list of clauses.
  • A clause can be a solid name, in which case that solid is selected.
  • A clause can be a solid name preceded by *, in which case that solid and all of its ancestors (upstream dependencies) are selected.
  • A clause can be a solid name followed by *, in which case that solid and all of its descendents (downstream dependencies) are selected.
  • A clause can be a solid name followed by any number of +s, in which case that solid and descendents up to that many hops away are selected.
  • A clause can be a solid name preceded by any number of +s, in which case that solid and ancestors up to that many hops away are selected.

Clause examples

  • some_solid: select "some_solid" itself
  • *some_solid: select "some_solid" and all ancestors (upstream dependencies).
  • some_solid*: select "some_solid" and all descendants (downstream dependencies).
  • *some_solid*: select "some_solid" and all of its ancestors and descendants.
  • +some_solid: select "some_solid" and its direct parents.
  • some_solid+++: select "some_solid" and its children, its children's children, and its children's children's children.

Specifying Solid Selection#

You can use this selection syntax in the solid_selection argument to the execute_pipeline:

execute_pipeline(my_pipeline, solid_selection=["*add_two"])

Similarly, you can specify the same solid selection in Dagit Playground:

solid-selection