Github<\/a> Actions are already well-known for its functionalities in the software development domain, while in recent years, also discovered as offering compelling benefits in streamlining data workflows:<\/p>\n GitHub Action is a feature of GitHub that allows users to automate workflows directly within their repositories. These workflows are defined using YAML files and can be triggered by various events such as code pushes, pull requests, issue creation, or scheduled intervals. With its extensive library of pre-built actions and the ability to write custom scripts, GitHub Actions is a versatile tool for automating tasks.<\/p>\n We will demonstrate the implementation GitHub actions through 4 levels of difficulty, starting with the \u201cminimal viable product\u201d and progressively introducing additional components and customization in each level.<\/p>\n Start by creating a GitHub repository where you want to store your workflow and the Python script. In your repository, create a The It consists of three sections: First, The first workflow demonstrated the minimal viable version of the GitHub Action, but it did not take full advantage of the GitHub Actions. At the second level, we will add a bit more customization and functionalities \u2013 automatically set up the environment with Python version 3.11, install required packages and execute the script whenever changes are pushed to main branch.<\/p>\n If you are interested in the basics of setting up a development environment for your data science projects, my previous blog post \u201c7 Tips to Future-Proof Machine Learning Projects<\/a><\/strong>\u201d provides a bit more explanation.<\/p>\n At the third level, we add more dynamics and complexity to make it more suitable for real-world applications. We introduce scheduled jobs as they bring even more benefits to a data science project, enabling periodic fetching of more recent data and reducing the need to manually run the script whenever data refresh is required. Additionally, we utilize dynamic argument parsing to execute the script based on different date range parameters according to the schedule.<\/p>\n When the command As shown above, the last step \u201cCommit changes\u201d commits the changes, configures the git user email and name, stages the changes, commits them, and pushes to the remote GitHub repository. This is a necessary step when running GitHub Actions that result in changes to the working directory (e.g., output file \u201cdata.jsonl\u201d is created). Otherwise, the output is only saved in the The final level focuses on improving the security and performance of the GitHub workflow by addressing non-functional requirements.<\/p>\n To improve workflow efficiency and reduce errors, we add a check before committing changes, ensuring that commits and pushes only occur when there are actual changes since the last commit. This is achieved through the command Lastly, we introduce the environment secret, which enhances security and avoids exposing sensitive information (e.g., API token, personal access token) in the codebase. Additionally, environment secrets offer the benefit of separating the development environment. This means you can have different secrets for different stages of your development and deployment pipeline. For example, the testing environment (e.g., in the dev branch) can only access the test token, whereas the production environment (e.g. in the main branch) will be able to access the token linked to the production instance.<\/p>\n To set up environment secrets in GitHub:<\/p>\n After setting up the GitHub environment secrets, we will need to add the secret to the workflow environment, for example below we added We then update the Python script This guide explores the implementation of GitHub Actions for building dynamic data pipelines, progressing through four different levels of workflow implementations:<\/p>\n Each level builds upon the previous one, demonstrating how GitHub Actions can be effectively utilized in the data domain to streamline data solutions and speed up the development lifecycle.<\/p>\n The post 4 Levels of GitHub Actions: A Guide to Data Workflow Automation<\/a> appeared first on Towards Data Science<\/a>.<\/p>\n<\/div>\nGitHub Action Benefits<\/h2>\n
![\"\"](\"https:\/\/i0.wp.com\/contributor.insightmediagroup.io\/wp-content\/uploads\/2025\/04\/Screenshot-2025-04-01-at-10.13.26%25E2%2580%25AFPM-1024x220.png?resize=1024%2C220&ssl=1\")
<\/figure>\n\n
GitHub Action Building Blocks<\/h2>\n
![\"\"](\"https:\/\/i0.wp.com\/contributor.insightmediagroup.io\/wp-content\/uploads\/2025\/04\/Screenshot-2025-04-01-at-10.13.46%25E2%2580%25AFPM-1024x220.png?resize=1024%2C220&ssl=1\")
<\/figure>\n\n
.github\/workflow<\/code> directory in a GitHub repository.<\/li>\n
\n4 Levels of Github Actions<\/h2>\n
![\"\"](\"https:\/\/i0.wp.com\/contributor.insightmediagroup.io\/wp-content\/uploads\/2025\/04\/Screenshot-2025-04-01-at-10.19.34%25E2%2580%25AFPM-1024x476.png?resize=1024%2C476&ssl=1\")
<\/figure>\n1. \u201cSimple Workflow\u201d with Python Script Execution<\/h3>\n
.github\/workflows<\/code> directory (please note that this directory must be placed within the workflows<\/code> folder for the action to be executed successfully). Inside this directory, create a YAML file (e.g., simple-workflow.yaml<\/code>) that defines your workflow.<\/p>\nhello_world.py<\/code> based on a manual trigger.<\/p>\nname: simple-workflow\n\non: \n workflow_dispatch:\n \njobs:\n run-hello-world:\n runs-on: ubuntu-latest\n steps:\n - name: Checkout repo content\n uses: actions\/checkout@v4\n - name: run hello world\n run: python code\/hello_world.py<\/code><\/pre>\nname: simple-workflow<\/code> defines the workflow name. Second, on: workflow_dispatch<\/code> specifies the condition for running the workflow, which is manually triggering each action. Last, the workflow jobs jobs: run-hello-world<\/code> break down into the following steps:<\/p>\n\n
runs-on: ubuntu-latest<\/code>: Specify the runner (i.e., a virtual machine) to run the workflow \u2014 ubuntu-latest<\/code> is a standard GitHub hosted runner containing an environment of tools, packages, and settings available for GitHub Actions to use.<\/li>\nuses: actions\/checkout@v4<\/code>: Apply a pre-built GitHub Action checkout@v4<\/code> to pull the repository content into the runner\u2019s environment. This ensures that the workflow has access to all necessary files and scripts stored in the repository.<\/li>\nrun: python code\/hello_world.py<\/code>: Execute the Python script located in the code<\/code> sub-directory by running shell commands directly in your YAML workflow file.<\/li>\n<\/ul>\n2. \u201cPush Workflow\u201d with Environment Setup<\/h3>\n
name: push-workflow\n\non: \n push:\n branches:\n - main\n\njobs:\n run-hello-world:\n runs-on: ubuntu-latest\n steps:\n - name: Checkout repo content\n uses: actions\/checkout@v4\n - name: Set up Python\n uses: actions\/setup-python@v5\n with:\n python-version: '3.11' \n - name: Install dependencies\n run: |\n python -m pip install --upgrade pip\n pip install -r requirements.txt\n - name: Run hello world\n run: python code\/hello_world.py<\/code><\/pre>\n\n
on: push<\/code>: Instead of being activated by manual workflow dispatch, this allows the action to run whenever there is a push from the local repository to the remote repository. This condition is commonly used in a software development setting for integration and deployment processes, which is also adopted in the Mlops<\/a> workflow, ensuring that code changes are consistently tested and validated before being merged into a different branch. Additionally, it facilitates continuous deployment by automatically deploying updates to production or staging environments as soon as changes are pushed. Here we add an optional condition branches: -main<\/code> to only trigger this action when it is pushed to the main branch.<\/li>\nuses: actions\/setup-python@v5<\/code>: We added the \u201cSet up Python\u201d step using GitHub\u2019s built-in action setup-python@v5<\/code>. Using the setup-python<\/code> action is the recommended way of using Python with GitHub Actions because it ensures consistent behavior across different runners and versions of Python.<\/li>\npip install -r requirements.txt<\/code>: Streamlined the installation of required packages for the environment, which are saved in the requirements.txt<\/code> file, thus speed up the further building of data pipeline and data science solution.<\/li>\n<\/ul>\n3. \u201cScheduled Workflow\u201d with Argument Parsing<\/h3>\n
name: scheduled-workflow\n\non: \n workflow_dispatch:\n schedule:\n - cron: \"0 12 1 * *\" # run 1st day of every month\n\njobs:\n run-data-pipeline:\n runs-on: ubuntu-latest\n steps:\n - name: Checkout repo content\n uses: actions\/checkout@v4\n - name: Set up Python\n uses: actions\/setup-python@v5\n with:\n python-version: '3.11' # Specify your Python version here\n - name: Install dependencies\n run: |\n python -m pip install --upgrade pip\n python -m http.client\n pip install -r requirements.txt\n - name: Run data pipeline\n run: |\n PREV_MONTH_START=$(date -d \"`date +%Y%m01` -1 month\" +%Y-%m-%d)\n PREV_MONTH_END=$(date -d \"`date +%Y%m01` -1 day\" +%Y-%m-%d)\n python code\/fetch_data.py --start $PREV_MONTH_START --end $PREV_MONTH_END\n - name: Commit changes\n run: |\n git config user.name '<github-actions>'\n git config user.email '<github-actions@github.com>'\n git add .\n git commit -m \"update data\"\n git push<\/code><\/pre>\n\n
on: schedule: - cron: \"0 12 1 * *\"<\/code>: Specify a time based trigger using the cron expression \u201c0 12 1 * *\u201d \u2013 run at 12:00 pm on the 1st day of every month. You can use crontab.guru<\/a> to help create and validate cron expressions, which follow the format: \u201cminute\/hour\/ day of month\/month\/day of week\u201d.<\/li>\npython code\/fetch_data.py --start $PREV_MONTH_START --end $PREV_MONTH_END<\/code>: \u201cRun data pipeline\u201d step runs a series of shell commands. It defines two variables PREV_MONTH_START<\/code> and PREV_MONTH_END<\/code> to get the first day and the last day of the previous month. These two variables are passed to the python script \u201cfetch_data.py\u201d to dynamically fetch data for the previous month relative to whenever the action is run. To allow the Python script to accept custom variables via command-line arguments, we use argparse<\/code> library to build the script. This deserves a separate topic, but here is quick preview of how the python script would look like using the argparse<\/code> library to handle command-line arguments \u2018\u2013start\u2019 and \u2018\u2013end\u2019 parameters.<\/li>\n<\/ul>\n## fetch_data.py\n\nimport argparse\nimport os\nimport urllib\n\ndef main(args=None):\n\t parser = argparse.ArgumentParser()\n\t parser.add_argument('--start', type=str)\n\t parser.add_argument('--end', type=str)\n\t args = parser.parse_args(args=args)\n\t print(\"Start Date is: \", args.start)\n\t print(\"End Date is: \", args.end)\n\t \n\t date_range = pd.date_range(start=args.start, end=args.end)\n\t content_lst = []\n\t\n\t for date in date_range:\n\t date = date.strftime('%Y-%m-%d')\n\t\n\t\t params = urllib.parse.urlencode({\n\t 'api_token': '<NEWS_API_TOKEN>',\n\t 'published_on': date,\n\t 'search': search_term,\n\t })\n\t\t url = '\/v1\/news\/all?{}'.format(params)\n\t\t \n\t\t content_json = parse_news_json(url, date)\n\t\t content_lst.append(content_json)\n\n\t with open('data.jsonl', 'w') as f:\n\t for item in content_lst:\n\t json.dump(item, f)\n\t f.write('n')\n\t \n return content_lst<\/code><\/pre>\npython code\/fetch_data.py --start $PREV_MONTH_START --end $PREV_MONTH_END<\/code> executes, it creates a date range between $PREV_MONTH_START <\/code>and $PREV_MONTH_END<\/code>. For each day in the date range, it generates a URL, fetches the daily news through the API, parses the JSON response, and collects all the content into a JSON list. We then output this JSON list to the file \u201cdata.jsonl\u201d. <\/p>\n- name: Commit changes\n run: |\n git config user.name '<github-actions>'\n git config user.email '<github-actions@github.com>'\n git add .\n git commit -m \"update data\"\n git push<\/code><\/pre>\n\/temp<\/code> folder within the runner environment, and appears as if no changes have been made after the action is completed.<\/p>\n4. \u201cSecure Workflow\u201d with Secrets and Environment Variables Management<\/h3>\n
name: secure-workflow\n\non: \n workflow_dispatch:\n schedule:\n - cron: \"34 23 1 * *\" # run 1st day of every month\n\njobs:\n run-data-pipeline:\n runs-on: ubuntu-latest\n steps:\n - name: Checkout repo content\n uses: actions\/checkout@v4\n - name: Set up Python\n uses: actions\/setup-python@v5\n with:\n python-version: '3.11' # Specify your Python version here\n - name: Install dependencies\n run: |\n python -m pip install --upgrade pip\n python -m http.client\n pip install -r requirements.txt\n - name: Run data pipeline\n env:\n NEWS_API_TOKEN: ${{ secrets.NEWS_API_TOKEN }} \n run: |\n PREV_MONTH_START=$(date -d \"`date +%Y%m01` -1 month\" +%Y-%m-%d)\n PREV_MONTH_END=$(date -d \"`date +%Y%m01` -1 day\" +%Y-%m-%d)\n python code\/fetch_data.py --start $PREV_MONTH_START --end $PREV_MONTH_END\n - name: Check changes\n id: git-check\n run: |\n git config user.name 'github-actions'\n git config user.email 'github-actions@github.com'\n git add .\n git diff --staged --quiet || echo \"changes=true\" >> $GITHUB_ENV\n - name: Commit and push if changes\n if: env.changes == 'true'\n run: |\n git commit -m \"update data\"\n git push\n <\/code><\/pre>\ngit diff --staged --quiet || echo \"changes=true\" >> $GITHUB_ENV<\/code>.<\/p>\n\n
git diff --staged<\/code> checks the difference between the staging area and the last commit.<\/li>\n--quiet<\/code> suppresses the output \u2014 it returns 0 when there are no changes between the staged environment and working directory; whereas it returns exit code 1 (general error) when there are changes between the staged environment and working directory<\/li>\necho \"changes=true\" >> $GITHUB_ENV<\/code> through the OR operator ||<\/code> which tells the shell to run the rest of the line if the first command failed. Therefore, if changes exist, \u201cchanges=true\u201d is passed to the environment variable $GITHUB_ENV<\/code> and accessed at the next step to trigger git commit and push conditioned on env.changes == 'true'<\/code>.<\/li>\n<\/ul>\n\n
${{ secrets.NEWS_API_TOKEN }}<\/code> to the step \u201cRun data pipeline\u201d. <\/p>\n- name: Run data pipeline\n env:\n NEWS_API_TOKEN: ${{ secrets.NEWS_API_TOKEN }} \n run: |\n PREV_MONTH_START=$(date -d \"`date +%Y%m01` -1 month\" +%Y-%m-%d)\n PREV_MONTH_END=$(date -d \"`date +%Y%m01` -1 day\" +%Y-%m-%d)\n python code\/fetch_data.py --start $PREV_MONTH_START --end $PREV_MONTH_END<\/code><\/pre>\nfetch_data.py<\/code> to access the environment secret using os.environ.get()<\/code>.<\/p>\nimport os api_token = os.environ.get('NEWS_API_TOKEN')<\/code><\/pre>\n
\nTake-Home Message<\/h2>\n
\n