9/3/2023 0 Comments Airflow dag decoratorWithin each mapped task group instance two tasks will run using that instances' value for my_num as an input. 6 mapped task group instances are created, one for each input. This DAG dynamically maps over the task group group1 with different inputs for the my_num parameter. # creating 6 mapped task group instances of the task group group1 (2.5 feature) # will print out a list of results from map index 2 and 3 of the add_42 task # only pull Xcom from specific mapped task group instances (2.5 feature) # reference a task in a task group with task_group_id.task_id # a downstream task to print out resulting pull_xcom ( ** context ) : # creating a task group using the decorator with the dynamic input ( group_id = "group1" ) decorators import dag, task_group, task Task_group_example = task_group_example ( )įrom airflow. Load ( transform_values ( extract_data ( ) ) ) datetime ( 2021, 1, 1, tz = "UTC" ), catchup = False )ĭef task_group_example ( ) ( task_id = "extract", retries = 2 )ĭata_string = ' """ Import ( schedule = None, start_date = pendulum. decorators import dag, task, task_group To use the decorator, add before a Python function which calls the functions of tasks that should go in the task group. Using task group decorators doesn't change the functionality of task groups, but they can make your code formatting more consistent if you're already using them in your DAGs. The task group decorator functions like other Airflow decorators and allows you to define your task group with the TaskFlow API. The task group decorator is available in Airflow 2.1 and later. Use the task group decorator Īnother way of defining task groups in your DAGs is by using the task group decorator. It is important that you use this format when calling specific tasks with XCOM passing or branching operator decisions. This ensures the task_id is unique across the DAG. When your task is within a task group, your callable task_id is the task_id prefixed with the group_id. The task group dependencies are shown in the following animation: The task immediately to the right of the first blue circle ( t1) gets the group's upstream dependencies and the task immediately to the left ( t2) of the last blue circle gets the group's downstream dependencies. When you click and expand group1, blue circles identify the task group dependencies. In the Airflow UI, blue highlighting is used to identify tasks and task groups. """Example DAG demonstrating the usage of the sensor decorator.""" from _future_ import annotations # import pendulum from corators import dag, task from import PokeReturnValue # ( schedule = None, start_date = pendulum. See the License for the # specific language governing permissions and limitations # under the License. You may obtain a copy of the License at # Unless required by applicable law or agreed to in writing, # software distributed under the License is distributed on an # "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY # KIND, either express or implied. The ASF licenses this file # to you under the Apache License, Version 2.0 (the # "License") you may not use this file except in compliance # with the License. See the NOTICE file # distributed with this work for additional information # regarding copyright ownership. # Licensed to the Apache Software Foundation (ASF) under one # or more contributor license agreements.
0 Comments
Leave a Reply. |
AuthorWrite something about yourself. No need to be fancy, just an overview. ArchivesCategories |