dev-serum
  • README.md
  • sitemap
  • cheat sheet
    • Front-End
    • IDE
      • JetBrain Products
      • VScode (feat. Cursor IDE)
  • devops-cloud
    • [Monitoring] Datadog
    • [FinOps] AWS
    • Docker
      • Docker env-variable mapping
      • Make snapshot
      • compose
      • Networking
      • [Docker] context (multi profile)
    • [Kubernates]
      • [Kubernates][Linux] check stdout log and restart
      • [Kubernates] with MSK
      • [Kebernbates] Commands
      • [Kubernates] EKS
    • Infrastructure as Code
      • cloud key,value manage
      • Terraform, Terragrunt
        • [Terraform] basic
        • Terraform with OnePassword
      • [Terraform] CDKTF
        • [Terraform] CDKTF Best Practice
        • [Terraform] CDKTF module apply
    • AWS Command
    • AWS
      • AWS ElsticCache VS MemoryDB
      • AWS Route 53
      • AWS Machine Learning
      • AWS::CloudWatch
      • AWS RAM (Resource Access Manager)
      • S3 vs CloudFront
      • AWS VPC & Subnet
        • AWS VPC & Subnet Strategy
        • AWS NAT gateway
      • AWS: Check List
      • AWS Fargate
      • AWS: Good Things To Know
      • AWS ElasticCache
      • AWS Load Balancer
      • AWS EFS
        • mount EFS on AWS ECS
      • [devserum] AWS setup guide for ECR event-based deploy
      • TECH-NEWS
        • AWS Fargate support spot instance
      • _Glossary
      • About Task volume
      • AWS Price
      • AWS EC2
        • Launch EC2 instance for ECS cluster
        • EC2 log
        • assign EC2 to ECS
      • Learn AWS
      • Cron jobs
      • AWS Codebuild
      • AWS API Gateway
      • AWS ECS
        • AWS ECS Quick Start
        • ECS service deployment and autoscaling
        • AWS ECS Task
        • Container Instance Draining
        • [editing][Serum] Create 'EC2 Linux + Networking Cluster' + Spot instance type
        • [editing] AWS ECS Guide
      • AWS S3
        • create S3 bucket for logging
      • AWS CodePipeline
        • CodePipeline setup based ECS push event
        • AWS Codebuild
      • AWS RDS
        • Aurora
      • AWS IAM
        • AWS IAM Custom Policy
      • AWS System Manager (parameter store)
      • AWS ECR
      • AWS Lambda (serverless)
      • [guide] network setup
      • [editing][Serum] AWS CodePipeline
        • create ECR
        • create CodeDeploy
        • create new pipeline
      • AWS Elastic IP
  • Computer Science
    • Database
    • Network
    • Design Pattern
    • Computer Science ETC
      • SIMD
      • Narcissistic number
      • Prime number
      • Fold (high-order function)
      • DFS (Deterministic finite automaton)
    • Object-Oriented Programming
  • Convention
    • API Design (RESTful) & Naming Convention
      • API Gateway & OAuth & API key
      • Authentication & Authorization
        • OAuth
    • File Naming Convention
  • DevOps & MLOps
    • Linux
      • Linux Tips
        • [Linux] tmux
        • [Linux] server terminal resolution
        • [Linux] SSH access control
          • SSH
        • [Linux] user & user group manage
      • CentOS
      • Linux commands
      • macOS command
      • Page 1
    • MLOps
      • ONNX on AWS Inferentia
      • Apache Airflow
      • [MLOps] Multi-GPU server setup
    • DevOps
      • Apache Flink
      • [cheatsheets] nginx
        • [cheetsheets] nginx & SSL certbot
      • CI / CD
        • [argocd]
      • Log ingeration
      • JMeter
      • Github
        • Github action
          • codeball
      • [Yaml] how to manage
    • Test
      • jmeter (stress test)
    • Jira
      • Jira Workflow Setup
      • Jira Board Configuraiton
      • Tips
      • JQL
      • Jira Ticket Automation
        • github action cheatsheets
        • reuse github action (github action template)
    • Git
      • git rollback process
      • private npm package
      • git token manage
    • [DevEnv]
      • [DevEnd] Ubuntu on Mac
      • [DevEnv] NodeJS
      • [DevEnv] Python setup
        • Pycharm
        • pipenv
        • [DevEnv] python setup on Windows
      • [DevEnv] Mac Setup QuickStart
      • [DevEnv] IDE - jetbrain
      • Java on Mac
      • asdf
      • Page 2
      • git
      • vimdiff
      • intellij + ammonite
      • iTerm2 + (fish-shell + advance) for macOS
    • DevOps: Good Things To Know
      • API Gateway & Reverse Proxy
      • Site Reliability Engineering
      • DevOps Topologies
    • Virtualization
    • Logging & Monitoring
      • Kibana
      • Fluent Bit & FluentD
    • Tech Standards
      • SIP: Session Initiation Protocol
  • Spring Framework
    • [Spring] N+1 solution
    • [Spring] Transaction
    • Gradle
    • [Spring] how autowired works?
    • [Spring] Test
  • Machine Learning
  • System Design
    • System Design
      • telemetry
      • Scalability
      • Stream Processing
      • System Design Pattern
      • Architecture
      • Cache
        • Chrome Cache
        • Redis as Cache
      • API Gateway
    • Serverless
      • Serverless framework deploy
      • AWS CloudFront
      • AWS:: Lambda@edge & CloudFront Function
        • deploy lambda@edge on existing cloudfront
      • lambda@edge (on the fly)
    • Image Server
  • Database
    • Data engineering
      • Metabase
    • RDBMS
      • PostgreSQL
        • Performance Tips
        • Postgres CDC with Airbyte on AWS RDS
    • Database Common Knowledge
    • In-memory
      • Redis
      • Redis
    • NoSQL
      • AWS Dynamo DB
      • Elasticsearch
  • Dive into
    • TS & JS
      • Express
      • Jest
      • NestJS
      • NodeJS
        • PM2
        • npm & yarn
        • node-http-proxy
        • Rollup JS
        • ExpressJS
      • JavaScript
        • for...in, for...of
        • RegExp
        • for iterator optimize
        • forEach vs map
      • TypeScript
        • Tips
        • Type & Interface
        • tslint on eslint
    • Golang
      • Go: Bootstrap
      • Go - Cheatsheet
      • Syntax
    • Python
      • Tips
      • pylint
      • packaging
      • Code Quality
      • Django
      • What's new
      • DRF
        • personal tutoring
      • Future of Language
      • TDD
      • Performance
    • Scala
    • C++
    • ETC
    • Kotlin & Spring Toy
  • Mathematics
    • Coursera Math
      • Systems of Equations
      • Systems of Sentences
    • Calculus
    • Math Cheatsheet
    • README.md Math
    • ETC
      • Why is a negative times a negative positive?
      • Interesting articles
    • Mathematical expression
    • Mathematics README.md
    • Pre-algebra
      • Factors and multiples
        • Least common multiple
    • Algebra I
      • Quadratic functions & equations
    • Algebra 2
    • Geometry
      • Circle
  • English
    • README.md - English
  • Data Structures & Algorithms
    • Math with Algorithm
    • Coding Challenges
      • Codewars-SQL
        • 7kyu
          • SQL Basics: Simple JOIN
        • 8kyu
          • Easy SQL: Convert to Hexadecimal
          • Easy SQL: Square Root and Log
          • Easy SQL: LowerCase
          • SQL Basics: Simple SUM
          • Countries Capitals for Trivia Night (SQL for Beginners #6)
          • SQL Basics: Simple DISTINCT
          • SQL Basics: Simple WHERE and ORDER BY
          • Easy SQL - Ordering
          • Adults only (SQL for Beginners #1)
      • Codewars
        • 4kyu
          • Strings Mix
        • 5kyu
          • Simple Pig Latin
        • 6kyu
          • Street Fighter 2 - Character Selection
          • String average
          • Casino chips
          • Design a Simple Automaton (Finite State Machine)
          • Numericals of a String
          • Strip Url Params
          • Row of the odd triangle
          • Calculate String Rotation
          • Prize Draw
          • Multiplication Tables
          • Primorial Of a Number
          • English beggars
          • Highest Rank Number in an Array
          • Tank Truck
          • IPv4 to int32
          • Sum The Tree
          • Prefill an Array
          • The Enigma Machine - Part 1: The Plugboard
          • If you can read this...
          • [draft]1/n- Cycle
          • Berlin Clock
          • Simple Fun #265: The Janitor And His Mop
          • Function Composition
          • Coordinates Validator
          • Meeting
          • Two Joggers
          • The Deaf Rats of Hamelin
          • Roman Numerals Decoder
        • 7kyu
          • Even numbers in an array
          • The highest profit wins!
          • Remove the minimum
          • Small enough? - Beginner
          • Compare Strings by Sum of Chars
          • Convert a linked list to a string
          • Halving Sum
          • My Languages
          • Folding your way to the moon
          • Gauß needs help! (Sums of a lot of numbers).
          • Sum of angles
          • Sum of Triangular Numbers
          • Flatten and sort an array
          • Triangular Treasure
          • Sorted? yes? no? how?
          • Most digits
          • Are the numbers in order?
          • Sort array by string length
          • Make a function that does arithmetic!
          • Find the stray number
          • two fighters, one winner.
          • Don't give me five!
          • Sum of the first nth term of Series
          • Largest 5 digit number in a series
          • Palindrome chain length
          • Sum of a sequence
          • Form The Largest
          • Credit Card Mask
          • List Filtering
          • Beginner Series #3 Sum of Numbers
          • Find the divisors!
          • Is this a triangle?
          • Find the next perfect square!
          • Square Every Digit
          • Every possible sum of two digits
          • A Gift Well Spent
          • Exclamation marks series #3: Remove all exclamation marks from sentence except at the end
          • Simple Fun #215: Properly Closed Bracket Word
          • Bit Plane Return
          • Playing with Sets : Sup/Sub
          • All Star Code Challenge #22
          • Simple Fun #213: Is Function?
          • Consecutive letters
          • Maximum Length Difference
          • L2: Triple X
          • Simple Fun #136: Missing Values
          • Merge two arrays
        • 8Kyu
          • Grasshopper - Array Mean
          • Lario and Muigi Pipe Problem
          • Add Length
          • Duck Duck Goose
          • Reverse List Order
          • Welcome!
          • Heads and Legs
          • Collatz Conjecture (3n+1)
          • Pirates!! Are the Cannons ready!??
          • The falling speed of petals
          • Formatting decimal places #0
          • Merge two sorted arrays into one
          • Find the Difference in Age between Oldest and Youngest Family Members
          • Convert to Binary
          • Thinkful - Number Drills: Pixelart planning
          • Holiday VIII - Duty Free
          • Closest elevator
          • Surface Area and Volume of a Box
          • Bin to Decimal
          • Is it even?
          • Check the exam
          • Hex to Decimal
          • Watermelon
          • Can we divide it?
          • Parse float
          • Twice as old
          • Get Nth Even Number
          • No zeros for heros
          • Volume of a Cuboid
          • Parse nice int from char problem
          • Will there be enough space?
          • Reversed Words
          • Short Long Short
          • Vowel remover
          • Beginner - Reduce but Grow
          • To square(root) or not to square(root)
          • Get the mean of an array
          • Count Odd Numbers below n
          • Convert a String to a Number!
          • You only need one - Beginner
          • Invert values
          • Keep Hydrated!
          • Convert a Number to a String!
          • Return Negative
          • Count of positives / sum of negatives
          • Square(n) Sum
          • Basic Mathematical Operations
          • Geometry Basics: Distance between points in 2D
          • Find the smallest integer in the array
          • Simple validation of a username with regex
          • Exclamation marks series #11: Replace all vowel to exclamation mark in the sentence
          • Removing Elements
          • Stringy Strings
          • Correct the mistakes of the character recognition software
          • Double Char
          • Rock Paper Scissors!
          • Count by X
          • Coefficients of the Quadratic Equation
          • Dollars and Cents
          • ES6 string addition
          • Character Frequency
          • SpeedCode #2 - Array Madness
    • Famous Algorithm Problems
      • Deterministic finite automaton
      • Fibonacci
      • Wheat and chessboard problem
    • Algorithm Technic & Optimization
      • JavaScript Algorithm Technic & Optimization
        • array compare optimize
      • Python Algorithm Technic
  • OpenStack
    • Manage IP addresses
  • Standards
    • ISO
    • IEEE
  • Good things to know
    • Web Technology
      • Untitled
  • BOOK-REVIEW
    • Tech
      • [KR-230607] 가상 면접 사례로 배우는 대규모 시스템 설계 기초 (원제 : System Design Interview)
      • [KR-2001061] '빅데이터를 지탱하는 기술' 리뷰
      • [KR-2001060] '성당과 시장' 리뷰
    • Candidates to read
    • Human Mind & Behavior
      • Unlimited Memory
      • Everybody Lies
    • Economics & Stock
      • One Up On Wall Street
    • Productivity
      • Get Organized: Do More in Less Time
  • quick-reference
    • API
      • API URL naming convention
    • TECH
      • Free Certificate (HTTPS)
      • GDPR + PostgreSQL & Sequelize Encryption
      • API Gateway
    • security
    • ETC
      • IDE / JetBrains / PyCharm
      • dev-env
        • web-storm
      • Github
    • Front-end
      • next.js
  • -------------------------------
  • ⇣ ⇣Legacy⇣ ⇣
  • -------------------------------
  • Cheat-sheet
    • JavaScript
      • eslint
    • ETC
      • [cheat-sheet] Git
  • -------------------------------
  • ⇣ ⇣Korean Contents ⇣ ⇣
  • -------------------------------
  • [kr] Tech-knowledge
    • Article
      • [KR-1912150] 모던 소프트웨어 개발, 배포 라이프사이클
  • [kr] dive-into
    • Software Design Principle
      • [KR-2001110] DRY - don't repeat yourself
  • [kr] Book-review
    • Candidates to read
  • ETC
    • Personal notes
      • [review] AWS Innovate Online Conference –AI and Machine Learning Edition
Powered by GitBook
On this page
  • AirbnbEng - Airflow: a workflow management platform
  • Airflow vs. Luigi vs. Argo vs. MLFlow vs. KubeFlow
  • Airflow, MLflow or Kubeflow for MLOps?
  • Pros & Cons
  • Screenshots
  • Graph view
  • Dataset view

Was this helpful?

  1. DevOps & MLOps
  2. MLOps

Apache Airflow

PreviousONNX on AWS InferentiaNext[MLOps] Multi-GPU server setup

Last updated 1 year ago

Was this helpful?

AirbnbEng - Airflow: a workflow management platform

  • https://github.com/apache/airflow

Airflow vs. Luigi vs. Argo vs. MLFlow vs. KubeFlow

For a quick overview, we’ve compared the libraries when it comes to:

  • Maturity: based on the age of the project and the number of fixes and commits;

  • Popularity: based on adoption and GitHub stars;

  • Simplicity: based on ease of onboarding and adoption;

  • Breadth: based on how specialized vs. how adaptable each project is;

  • Language: based on the primary way you interact with the tool.

Airflow, MLflow or Kubeflow for MLOps?

Pros & Cons

https://youtu.be/YQ056EKzCyw?t=394

Screenshots

Graph view

https://medium.com/airbnb-engineering/airflow-a-workflow-management-platform-46318b977fd8

Dataset view

https://airflow.apache.org/docs/apache-airflow/stable/ui.html#datasets-view

LogoUI / Screenshots — Airflow Documentation