Parallelization & Scaling

Parallel Computing

Limiting factors

• Data dependencies ⇒ Can force you to do some task sequentially.
• Start-up
• Bottlenecks: access to shared / slow resources.
• Communication
• Resources required for management.

Q1: How much faster can a problem be solved using $N$ cores instead of 1.

Strong Scaling

• Constant Problem size, varying number of workers (cores)
• However, A compute Problem in reality
• Serial Compute time $T_s=S+P$
• Parallel Compute time $T_P=S+\frac{P}{N}$
• Andahl’s Law for Speed-up (under Strong Scaling)

Q2: How much more work can be done in parallel instead of in serial.

Roofline Model

Q3: How fast can a task be executed?

Roofline Graph