| Dependency | Reason |
|---|---|
| Dagrun Running | Task instance's dagrun was not in the 'running' state but in the state 'success'. |
| Task Instance State | Task is in the 'success' state which is not a valid state for execution. The task must be cleared in order to be run. |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 | def terminate_emr_clusters():
# Initialize Boto3 client for EMR
aws_region = Variable.get("aws_region")
emr_client = boto3.client('emr', region_name=aws_region)
logging.info("Retrieving EMR clusters...")
# List clusters matching the specified criteria
all_clusters_response = emr_client.list_clusters(ClusterStates=['WAITING'])
# Extract cluster IDs for clusters with names ending with '-viooh-Custom-EMR-cluster'
cluster_ids = [cluster['Id'] for cluster in all_clusters_response['Clusters'] if cluster['Name'].endswith('-viooh-Custom-EMR-cluster')]
if not cluster_ids:
logging.info("No clusters found for Custom EMR.")
return
logging.info(f"Found {len(cluster_ids)} clusters to terminate.")
# Terminate each UAT cluster
for cluster_id in cluster_ids:
logging.info(f"Terminating cluster with ID: {cluster_id}")
emr_client.terminate_job_flows(JobFlowIds=[cluster_id])
logging.info(f"Cluster with ID {cluster_id} terminated successfully at {datetime.utcnow()}.")
|
| Attribute | Value |
|---|---|
| dag_id | Terminate_Custom_EMR_Clusters |
| duration | 0.486057 |
| end_date | 2024-05-14 18:30:09.679785+00:00 |
| execution_date | 2024-05-13T18:30:00+00:00 |
| executor_config | {} |
| generate_command | <function TaskInstance.generate_command at 0x7ffa4fe777b8> |
| hostname | airflow-worker-0.airflow-worker.data-eng.svc.cluster.local |
| is_premature | False |
| job_id | 541578 |
| key | ('Terminate_Custom_EMR_Clusters', 'terminate_emr_clusters_task', <Pendulum [2024-05-13T18:30:00+00:00]>, 2) |
| log | <Logger airflow.task (INFO)> |
| log_filepath | /opt/airflow/logs/Terminate_Custom_EMR_Clusters/terminate_emr_clusters_task/2024-05-13T18:30:00+00:00.log |
| log_url | https://airflow.devel.viooh.net.cn/admin/airflow/log?execution_date=2024-05-13T18%3A30%3A00%2B00%3A00&task_id=terminate_emr_clusters_task&dag_id=Terminate_Custom_EMR_Clusters |
| logger | <Logger airflow.task (INFO)> |
| mark_success_url | https://airflow.devel.viooh.net.cn/success?task_id=terminate_emr_clusters_task&dag_id=Terminate_Custom_EMR_Clusters&execution_date=2024-05-13T18%3A30%3A00%2B00%3A00&upstream=false&downstream=false |
| max_tries | 1 |
| metadata | MetaData(bind=None) |
| next_try_number | 2 |
| operator | PythonOperator |
| pid | 37799 |
| pool | default_pool |
| pool_slots | 1 |
| prev_attempted_tries | 1 |
| previous_execution_date_success | 2024-05-12 18:30:00+00:00 |
| previous_start_date_success | 2024-05-13 18:30:04.765117+00:00 |
| previous_ti | <TaskInstance: Terminate_Custom_EMR_Clusters.terminate_emr_clusters_task 2024-05-12 18:30:00+00:00 [success]> |
| previous_ti_success | <TaskInstance: Terminate_Custom_EMR_Clusters.terminate_emr_clusters_task 2024-05-12 18:30:00+00:00 [success]> |
| priority_weight | 1 |
| queue | default |
| queued_dttm | 2024-05-14 18:30:06.934397+00:00 |
| raw | False |
| run_as_user | None |
| start_date | 2024-05-14 18:30:09.193728+00:00 |
| state | success |
| task | <Task(PythonOperator): terminate_emr_clusters_task> |
| task_id | terminate_emr_clusters_task |
| test_mode | False |
| try_number | 2 |
| unixname | airflow |
| Attribute | Value |
|---|---|
| dag | <DAG: Terminate_Custom_EMR_Clusters> |
| dag_id | Terminate_Custom_EMR_Clusters |
| depends_on_past | False |
| deps | {<TIDep(Previous Dagrun State)>, <TIDep(Not Previously Skipped)>, <TIDep(Trigger Rule)>, <TIDep(Not In Retry Period)>} |
| do_xcom_push | True |
| downstream_list | [] |
| downstream_task_ids | set() |
| None | |
| email_on_failure | True |
| email_on_retry | True |
| end_date | None |
| execution_timeout | None |
| executor_config | {} |
| extra_links | [] |
| global_operator_extra_link_dict | {} |
| inlets | [] |
| lineage_data | None |
| log | <Logger airflow.task.operators (INFO)> |
| logger | <Logger airflow.task.operators (INFO)> |
| max_retry_delay | None |
| on_failure_callback | None |
| on_retry_callback | None |
| on_success_callback | None |
| op_args | [] |
| op_kwargs | {} |
| operator_extra_link_dict | {} |
| operator_extra_links | () |
| outlets | [] |
| owner | data.engineers@viooh.com |
| params | {} |
| pool | default_pool |
| pool_slots | 1 |
| priority_weight | 1 |
| priority_weight_total | 1 |
| provide_context | False |
| queue | default |
| resources | None |
| retries | 1 |
| retry_delay | 0:05:00 |
| retry_exponential_backoff | False |
| run_as_user | None |
| schedule_interval | 30 18 * * * |
| shallow_copy_attrs | ('python_callable', 'op_kwargs') |
| sla | None |
| start_date | 2024-05-01T00:00:00+00:00 |
| subdag | None |
| task_concurrency | None |
| task_id | terminate_emr_clusters_task |
| task_type | PythonOperator |
| template_ext | [] |
| template_fields | ('templates_dict', 'op_args', 'op_kwargs') |
| templates_dict | None |
| trigger_rule | all_success |
| ui_color | #ffefeb |
| ui_fgcolor | #000 |
| upstream_list | [] |
| upstream_task_ids | set() |
| wait_for_downstream | False |
| weight_rule | downstream |