You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Copy file name to clipboardExpand all lines: README.md
+51-32Lines changed: 51 additions & 32 deletions
Original file line number
Diff line number
Diff line change
@@ -1,31 +1,52 @@
1
-
# Github Actions for Scientific Data Workflows
1
+
# Welcome to GitHub Actions for Scientific Data Workflows
2
2
3
-
Tutorial presented at [US-RSE'24 Conference](https://us-rse.org/usrse24/program/tutorials/)
3
+
4
+
Tutorial presented at [US-RSE'24 Conference](https://us-rse.org/usrse24/program/tutorials/#github-actions-for-scientific-data-workflows)
4
5
5
6
Authors: Valentina Staneva, Quinn Brencher, Scott Henderson
6
7
7
8
## Abstract
8
9
9
-
In this tutorial we will introduce GitHub Actions to scientists as a tool for lightweight automation of scientific data workflows. We will demonstrate that GitHub Actions are not just a tool for software testing, but can be used in various ways to improve the reproducibility and impact of scientific analysis. Through a sequence of examples, we will demonstrate some of Github Actions' applications to scientific workflows, such as scheduled deployment of algorithms to sensor streams, updating visualizations based on new data, processing large datasets, model versioning and performance benchmarking. GitHub Actions can particularly empower Python scientific programmers who are not willing to build fully-fledged applications or set up complex computational infrastructure, but would like to increase the impact of their work. The goal is that participants will leave with their own ideas of how to integrate Github Actions in their own work.
10
-
11
-
## Description:
12
-
13
-
GitHub Actions are quite popular within the software engineering community, but a scientific Python programmer may not have seen their use beyond a continuous integration framework for unit testing. We would like to increase their visibility through a scientific workflow lens. We will use examples that are relevant to the community: wrangling a messy realtime hydrophone data stream to display noise sounds from the Puget Sound (not far from the conference venue!) or processing hundreds of satellite radar images over glacial lakes in High-Mountain Asia to study flood hazards. We assume no knowledge on Github Actions and will start slowly with a “Hello World” step, but build quickly to create complex and exciting workflows. We will also showcase their value for scientific collaborations across institutions as a means to share reproducible workflows and computing infrastructure.
14
-
15
-
## Prerequisites:
16
-
GitHub account, familiarity with git, GitHub, and Python (conda, scipy, matplotlib), some maturity in manipulating scientific data and exposure to the challenges associated with it, ability to read code (our examples may use libraries not familiar to the audience, but the focus will be on the steps these libraries accomplish rather than the details)
17
-
18
-
## Installation Instructions:
19
-
Participants can make edits from the GitHub interface, but if they are willing to make updates locally, they need to have a functioning git ([set up instructions](https://swcarpentry.github.io/git-novice/#installing-git))
10
+
In this tutorial we will introduce GitHub Actions to scientists as a tool for lightweight automation of scientific data workflows. We will
11
+
demonstrate that GitHub Actions are not just a tool for software testing, but can be used in various ways to improve the reproducibility
12
+
and impact of scientific analysis. Through a sequence of examples, we will demonstrate some of GitHub Actions' applications to scientific
13
+
workflows, such as scheduled deployment of algorithms to sensor streams, updating visualizations based on new data, processing large
14
+
datasets, model versioning and performance benchmarking. GitHub Actions can particularly empower Python scientific programmers who are not
15
+
willing to build fully-fledged applications or set up complex computational infrastructure, but would like to increase the impact of their
16
+
work. The goal is that participants will leave with their own ideas of how to integrate Github Actions in their own work.
17
+
18
+
## Description
19
+
20
+
GitHub Actions are quite popular within the software engineering community, but a scientific Python programmer may not have seen their use
21
+
beyond a continuous integration framework for unit testing. We would like to increase their visibility through a scientific workflow lens.
22
+
We will use examples that are relevant to the community: wrangling a messy realtime hydrophone data stream to display noise sounds from the
23
+
Puget Sound (not far from the conference venue!) or processing hundreds of satellite radar images over glacial lakes in High-Mountain Asia
24
+
to study flood hazards. We assume no knowledge on GitHub Actions and will start slowly with a “Hello World” step, but build quickly to
25
+
create complex and exciting workflows. We will also showcase their value for scientific collaborations across institutions as a means to
26
+
share reproducible workflows and computing infrastructure.
27
+
28
+
## Prerequisites
29
+
GitHub account, familiarity with git (commits, versioning), GitHub (push, pull requests), and Python (conda, scipy, matplotlib), some maturity in manipulating scientific data and
30
+
exposure to the challenges associated with it, ability to read code (our examples may use libraries not familiar to the audience, but the
31
+
focus will be on the steps these libraries accomplish rather than the details)
32
+
33
+
## Installation Instructions
34
+
Participants can make edits from the GitHub interface, but if they are willing to make updates locally, they need to have a functioning git
35
+
([set up instructions](https://swcarpentry.github.io/git-novice/#installing-git))
20
36
21
37
## Outline
22
38
23
-
Outline:
39
+
### Short Version
40
+
```{tableofcontents}
41
+
```
24
42
25
-
* Overview of GitHub Actions and Workflows and their popular uses in Python software development (examples of testing, listing, packaging)(30 min)
43
+
### Long Version (with approximate schedule)
44
+
* Overview of GitHub Actions and Workflows and their popular uses in Python software development (examples of testing, listing,
45
+
packaging)(15 min)
26
46
* We will explain the main components of GitHub Actions and associated terminology
27
47
* We will summarize their typical uses in software development
28
-
* We will point to popular GitHub Actions used in Python software development and packaging (the focus of this tutorial will not be on them but rather on scientific pipelines)
48
+
* We will point to popular GitHub Actions used in Python software development and packaging (the focus of this tutorial will not be
49
+
on them but rather on scientific pipelines)
29
50
30
51
* Setting up your first workflow: a scientific Python environment (20 min)
31
52
* participants will update a workflow `.yml` file to create an environment with their favorite Python libraries
@@ -35,36 +56,34 @@ Outline:
35
56
* we will deploy a typical scientific workflow: reading data, converting to a new format, and making a visualization
36
57
* participants will update the deployment schedule to trigger a new workflow and will monitor the progress in the GitHub interface
37
58
38
-
* Break (10 min)
39
-
40
-
* Exporting results (30 min)
59
+
* Exporting results (25 min)
41
60
* participants will learn about various ways to store the results:
42
61
* caching
43
-
* creating GitHub artifacts
44
62
* committing to GitHub
45
-
* storing to own storage
46
-
* they will modify the code to make their own plot which will be automatically updated
63
+
* creating GitHub artifacts
64
+
* storing to personal storage
65
+
* they will modify the code to make a new plot which will be automatically updated
47
66
* they will use either matplotlib or an interactive library such as plotly
48
67
49
-
* Update results on a webpage (30 min)
50
-
* we will overview different ways to display scientific results on a webpage
51
-
* we will demonstrate the workflow to deploy the webpage
52
-
* participants will rerender the webpage based on the updates in GitHub
68
+
* Break (15 min)
53
69
54
-
* Large-scale data processing (30 min)
55
-
* we will demonstrate a use-case of processing large data sets with Github Actions
70
+
* Large-scale data processing (45 min)
71
+
* we will demonstrate a use-case of processing large data sets with GitHub Actions
56
72
* participants will fiddle with problem size to understand the power and limits of the computational infrastructure
57
73
* we will discuss connections to cluster/cloud computing
58
74
59
-
* Break (10 min)
75
+
* Update results on a webpage (30 min)
76
+
* we will overview different ways to display scientific results on a webpage
77
+
* we will demonstrate the workflow to deploy the webpage
78
+
* participants will rerender the webpage based on the updates in GitHub
60
79
61
-
* Model Versioning and Comparison (30 min)
80
+
* Model Versioning and Benchmarking (10 min)
62
81
* we will introduce how to leverage GitHub’s version control to version different models and performance
63
82
* participants can contribute a new model and check its performance
64
83
* we will discuss how this can be used as a community network to share methods and results
65
84
66
-
* Recap and Discussion (or buffer time) (20 min)
67
-
* we will have a discussion on potential uses of Github Actions within the work of the participants
85
+
* Recap and Discussion (or buffer time) (5 min)
86
+
* we will have a discussion on potential uses of GitHub Actions within the work of the participants
0 commit comments