Programming in Parallel
While not strictly necessary when using the HPC, programming in parallel is one of the primary benefits to using the HPC. Even if you don't plan on programming in parallel, it is still helpful to understand how it works so you can spot opportunities to reduce the runtime of your program.
What is parallel processing?
Parallel processing is when a program runs multiple tasks simultaneously. It has the potential to save a serious amount of time when programming. Unfortunately, programming in parallel isn't as simple as just telling the compiler "Hey, I want to run this in parallel!". We need to change how our code is written if we want the benefits of parallelization.
How it works
Programs can be broken down into distinct "tasks" that need to be done to achieve the final goal. A task can be any basic step that the program takes. In simple programs, this can be as few as one or even zero (if the file does nothing). In these simple programs there is no opportunity for the program to run in parallel because there aren't enough tasks.
Things get more interesting once you increase the number of tasks in program. For example, can a program with 5 tasks be ran in parallel? Well, the answer depends on if the tasks are independent or not. If each task requires information from the previous task, then the program cannot be run in parallel. This is because in order for a task to run, every previous task needs to be completed. This is an example of a sequential program (as seen in the diagram).
A program can be run in parallel if it contains independent tasks, meaning that a task doesn't require all previous tasks to be complete in order to run. Let's go back to our program with 5 distinct tasks. If tasks 2, 3, and 4 only require task 1, then they can all be run simultaneously. This example can be seen in the diagram. If the program runs tasks 2, 3, and 4 simultaneously then it is an example of a parallel program, even if they need to wait for task 1.
Since the time it takes to complete each task is the same, the difference in processing time between the sequential and parallel programs can be found by comparing the heights of the two programs in the diagram. If each task takes an hour, then the sequential program will complete in 5 hours while the parallel program will complete in 3 hours. Similarly, the width of the diagram shows how many nodes we need to run the task. The sequential program will use 1 node while the parallel program will need 3 nodes.
How to implement
When looking for places to implement parallelization, it is important to spot independent tasks. If Task B needs information returned by Task A, then Task B cannot run until Task A is finished. Conversely, if Task A and Task B don't need to exchange or share any information, they can be run concurrently. So, the key to implementing parallelization is to find tasks that are independent and running them simultaneously. If you are curious about the specific syntax on how to do this, each programming language is different and may need specific libraries to run on multiple threads. Information about your specific language can be easily found online.