CMPT231 Lecture 1

# CMPT231
## Data Structures and Algorithms
### Dr. Sean Ho

>>>
Speaker notes go here.

---

## Psalm 127:1-2 (NIV)
Unless the Lord **builds** the house, <br/>
the builders **labor** in vain. <br/>
Unless the Lord **watches** over the city, <br/>
the guards **stand watch** in vain.

In vain you **rise early** and **stay up** late, <br/>
**toiling** for food to eat -- <br/>
for he **grants sleep** to those he loves.

>>>
+ "work smarter, not harder"
  + smarter **algos**, not just better **hardware**
  + what **God** builds, not what we build
+ success, security, sleep:
  + all from God's **grace**
  + all **glory** to God

---

>>>
+ Website: **static** content (schedule, HW assignments, etc.)
+ myCourses: HW **submissions**
+ 9 **HW**, roughly 1/wk
  + due **Thu 10pm** at myCourses
  + **coding** in later HWs
+ 1 **midterm** 25Oct, 1 **final**

---
## What you need to succeed in 231
+ **Explorer's** heart (self-motivated)
+ Discrete **math** (e.g., MATH150)
  + Logic, **proofs**
+ Comfortable **coding** environment
  + Python, C++, Java, etc.
  + but not until **later** in semester

---

## Outline for today
+ **Algorithmic** analysis
  + **Insertion** sort
+ Discrete **math** review
  + **Logic** and proofs
  + Monotonicity, limits, iterated functions
  + **Fibonacci** sequence and golden ratio
+ **Asymptotic** notation: Θ, O, Ω, o, ω
  + **Proving** asymptotic bounds

---
## What is an algorithm?
+ Precise **process** for solving a problem:
  + Input → Compute → Output
+ Various languages for **expression**:
  + English, pseudocode, UML diagrams, etc.
+ Programming languages for **implementation**:
  + Python, C, Java, etc.
+ Focus: not **toolkits** but **problem solving**

---
## Algorithmic complexity
+ How many machine **instr** to **execute**
  + As function of input **size**
  + Ignoring **constant** factors
+ Depends on machine **architecture**
  + CPUs generally **sequential**
  + GPUs are massively **parallel**
+ **Running time** is more complex than this
  + Cache / **memory** very important

---
## Basic machine model
+ Our simple CPU **instruction set**:
  + **Arith**: +, -, \*, /, <, >, ≠
  + **Data**: `load` (read), `store` (write), copy
  + **Control**: `if`/`else`, `for`/`while`, functions
  + **Types**: char, int, float
  + Data **Structures**: pointers, fixed-length arrays
    + but **not** Python list / STL vec!
+ Assume each takes **constant** time

---
## Problem definition: Sorting
+ Input: **array** of key-value pairs
  + wlog, assume **keys** are *1* ... *n*
  + **values** (payload) can be any data
+ Output: array **sorted** by key
  + **in-place**: modify original array
  + **out-of-place**: return a copy
+ In standard **libraries**:
  + Python: `sort()`, `sorted()`
  + C++/Java: `sort()`
  + **How** do they do it?

---

## Outline for today
+ Algorithmic analysis
  + **Insertion sort**
+ Discrete math review
  + Logic and proofs
  + Monotonicity, limits, iterated functions
  + Fibonacci sequence and golden ratio
+ Asymptotic notation: Θ, O, Ω, o, ω
  + Proving asymptotic bounds

---

## Insertion sort: hand of cards

```
insertion_sort(A, n):
  for j = 2 to n:
    key = A[j]
    i = j - 1
    while i > 0 and A[i] > key:
      A[i+1] = A[i]
      i = i - 1
    A[i+1] = key
```

</div><div>

| In: |  *E*  | **B** |   D   |   F   |   A   |   C   |
|-----|-------|-------|-------|-------|-------|-------|
| j=3 |   B   |  *E*  | **D** |   F   |   A   |   C   |
| j=4 |   B   |   D   |  *E*  | **F** |   A   |   C   |
| j=5 |  *B*  |  *D*  |  *E*  |  *F*  | **A** |   C   |
| j=6 |   A   |   B   |  *D*  |  *E*  |  *F*  | **C** |
| Out:|   A   |   B   |   C   |   D   |   E   |   F   |

</div></div>

>>>
Not the **fastest** solution, but pretty **easy** to code

---
## Proof of correctness
+ Loop **invariant**:
+ Property that is true **before**, **during**, and **after** loop
+ For insertion sort: at each iteration of loop,
  + part of array `A[1 .. j-1]` is in **sorted** order

```
insertion_sort(A, n):
  for j = 2 to n:
    key = A[j]
    i = j - 1
    while i > 0 and A[i] > key:
      A[i+1] = A[i]
      i = i - 1
    A[i+1] = key
```

---
## Complexity analysis
Let \`t_j\` be the number of times the loop condition is checked
in the inner "while" loop:

```
insertion_sort(A, n):
  for j = 2 to n: # c0 * n
    key = A[j] # c1 * (n-1)
    i = j - 1 # c2 * (n-1)
    while i > 0 and A[i] > key: # c3 * sum (t_j)
      A[i+1] = A[i] # c4 * sum (t_j - 1)
      i = i - 1 # c5 * sum (t_j - 1)
    A[i+1] = key # c6 * (n-1)
```

Summation notation: \`sum_2^n t_j = t_2 + t_3 + ... + t_n\`

---
## Best vs. worst case
+ **Best** case is if input is **pre-sorted**:
  + Still need to **scan** to verify sorted,
  + But inner **while** loop only has 1 iteration: \`t_j=1\`
  + Total complexity: T(n) = a n + b, for some a, b
  + **Linear** in n
+ **Worst** case: input is in **reverse** order!
  + Inner "while" loop always max iterations: \`t_j=j\`
  + Calculate **total** complexity T(n):
  + Pick a line in inner loop, e.g., **line 5**: `A[i+1] = A[i]`
  + Complexity of other lines **similar**

---
## Worst case complexity
+ **Total** complexity for line 5, worst-case:
  \` c_4 sum_2^n(t_j-1) = c_4 sum_2^n(j-1) \`
  \` = c_4 (n-1)n/2 = (c_4/2)n^2 - (c_4/2)n \`
  + **Quadratic** in n
+ **Average** case: input is random, \`t_j=j/2\` on average
  + Still quadratic (only changes by a **constant** factor)

---
## Theta (Θ) notation
+ Insertion sort, line 5: \` (c_4 / 2) n^2 - (c_4/2)n \`
+ **Constants** \`c_1, c_2, ...\` may vary for different computers
  + As n gets **big**, constants become **irrelevant**
  + Even the n term is dominated by the \`n^2\` term
+ Complexity of insertion sort is on **order** of \`n^2\`
  + Notation: \`T(n) = Theta(n^2)\` ("theta")
+ \`Theta(1)\` means an algorithm runs in **constant time**
  + i.e., does not depend on **size** of input
+ We'll define Θ more **precisely** later today

---

## Outline for today
+ Algorithmic analysis
  + Insertion sort
+ **Discrete math review**
  + **Logic and proofs**
  + Monotonicity, limits, iterated functions
  + Fibonacci sequence and golden ratio
+ Asymptotic notation: Θ, O, Ω, o, ω
  + Proving asymptotic bounds

---
## Logic notation
+ \`not A\` (or !A): "**not** A"
  + e.g., let A = "*it is Tue*": then \`not A\` = "*it is not Tue*"
+ A ⇒ B: "**implies**", "if A, then B"
  + e.g., let B = "*meatloaf*":
  + then A ⇒ B = "*if Tue, then meatloaf*"
+ A ⇔ B: if and only if ("**iff**"):
  + **equivalence**: (A ⇒ B) and (B ⇒ A)
+ **John 14:15**: "*If you love me, keep my commands*"
+ **v21**: "*Whoever keeps my commands loves me*"
+ **v24**: "*He who does not love me will not obey my teaching*"

---
## Logic notation: ∀ and ∃
+ ∀: "**for all**",
  + e.g., "∀ day: meal(day) = meatloaf"
  + "*For all days, the meal on that day is meatloaf*"
+ ∃: "there **exists**" (not necessarily unique)
  + e.g., "∃ day: meal(day) = meatloaf"
  + "*There exists a day on which the meal is meatloaf*"

---
## Logic: contrapos and converse
+ **Contrapositive** of "A ⇒ B" is \`not B => not A\`
  + **Equivalent** to original statement
  + "*If Tue, then meatloaf*" ⇔ <br/> "*if not meatloaf, then not Tue*"
+ **Converse** of "A ⇒ B" is "\`not A => not B\`"
  + **Not** equivalent to original statement!
  + "*if not Tue, then not meatloaf*"

---

## Outline for today
+ Algorithmic analysis
  + Insertion sort
+ Discrete math review
  + Logic and proofs
  + **Monotonicity, limits, iterated functions**
  + Fibonacci sequence and golden ratio
+ Asymptotic notation: Θ, O, Ω, o, ω
  + Proving asymptotic bounds

---
## Monotonicity
+ f(x) is **monotone increasing** iff: *x < y ⇒ f(x) ≤ f(y)*
  + Also called "non-decreasing"
  + Can be **flat**
+ f(x) is **strictly increasing** iff: *x < y ⇒ f(x) < f(y)*
  + note inequality is **strict**
+ "*a mod n*" is the **remainder** of a when divided by n
  + e.g., 17 mod 5 = 2 (in Python: `17 % 5`)

---
## Limits
+ Formal **definitions** of limits involve ∀ and ∃
+ \` lim_(x->a) f(x) = b \`: "**limit** of *f(x)* as *x* goes to *a*"
  + ∀ *ε* > 0, ∃ *δ* > 0:
    *|x-a| < δ* ⇒ *|f(x)-b| < ε*
  + When *x* is "close" to *a*, then *f(x)* is "close" to *b*
+ \` lim_(n->oo) f(n) = b \`: "**limit** of *f(n)* as *n* goes to infinity":
  + ∀ *ε* > 0, ∃ *n0*:
    *n > n0* ⇒ *|f(n)-b| < ε*
  + When *n* is "big", *f(n)* is "close" to *b*

---
## Iterated functions (recursion)
+ \` f^((i))(x) \`: function *f*, **applied** *i* times to *x*:
  f(f(f(... f(x) ...)))
  + **Not** the same as \` f^i(x) = (f(x))^i \`
+ e.g., \` log^((2))(1000) \` = log(log(1000)) = log(3) ≈ *0.477*
  + But \` log^2(1000) = (log(1000))^2 = 3^2 \` = *9*
+ By convention, \` f^((0))(x) = x \` (apply *f* **zero** times)

---
## Iterated log: log\*(n)
+ \` log^**(n) = min(i>=0: log^((i))(n)<=1) \`
+ \# **times** *log* needs to be applied to *n* until the result is *≤1*
+ e.g., let lg = \` log_2 \`:
  + then \` text(lg)^**(16) = 3 \`, because
  + lg( lg( lg(16) ) ) = lg( lg(4) ) = lg(2) = 1

---

## Outline for today
+ Algorithmic analysis
  + Insertion sort
+ Discrete math review
  + Logic and proofs
  + Monotonicity, limits, iterated functions
  + **Fibonacci sequence and golden ratio**
+ Asymptotic notation: Θ, O, Ω, o, ω
  + Proving asymptotic bounds

---

## Fibonacci sequence
+ The n-th **Fibonacci number** is \`F\_n=F\_(n-1) + F\_(n-2)\`
  + Start with \`F_0=0, F_1=1:\`
  + \`F_n\` = *0, 1, 1, 2, 3, 5, 8, 13, 21,* ...
  + (Lucas numbers start with \`F_0=2\`)
+ Shows up all over **nature**
  + Num of **spirals** on sunflowers, pinecones, etc.
  + [Vi Hart video: Doodling in Math ![YouTube](img/vi-fib.jpg)](https://www.youtube.com/watch?v=ahXIMUkSXX0)

---

## The Golden ratio φ
+ φ is the **solution** to the equation \`x^2 = x+1\`
  + \` phi = (1 +- sqrt 5)/2 \`
  + Actually, two solutions: φ and its **conjugate**, \`bar phi\`
  + φ ≈ *1.61803*, and \`bar phi\` ≈ *-0.61803*
+ Also shows up all over **nature**
  + Dimensions of **Nautilus** seashells, spiral **galaxies**, etc.
  + **Aspect ratio** in architecture, e.g., Parthenon

---

## Fibonacci + golden ratio
+ Can **prove** *(#3.2-7)* that \`F\_n = (phi^n - (bar phi)^n)/sqrt 5\`
+ Second term is **fractional**: \` |(bar phi)^n|/sqrt 5 < 1/2\`
+ Thus: \`F\_n = |\_\_ phi^n/sqrt 5 + 1/2 \_\_| = text(round)( phi^n / sqrt 5) \`
+ i.e., Fibonacci grows **exponentially**!

---

## Outline for today
+ Algorithmic analysis
  + Insertion sort
+ Discrete math review
  + Logic and proofs
  + Monotonicity, limits, iterated functions
  + Fibonacci sequence and golden ratio
+ **Asymptotic notation: Θ, O, Ω, o, ω**
  + Proving asymptotic bounds

---

## Asymptotic growth: Θ, O, Ω
+ Behaviour "**in the limit**" (big n)
+ Define Θ as **class** of functions: *f(n)* ∈ *Θ( g(n) )* iff
  + ∃ *c1, c2, n0*: ∀ *n* > n0,
  0 ≤ *c1 g(n)* ≤ *f(n)* ≤ *c2 g(n)*
  + *f* is "**sandwiched**" between two multiples of *g*:
    + *c1 g(n)* and *c2 g(n)*
+ "**Big O**": specify only **upper** bound: f(n) ∈ *O( g(n) )* iff
  + ∃ *c2, n0*: ∀ *n* > n0, 0 ≤ *f(n)* ≤ *c2 g(n)*
  + e.g., \`Theta(n^2) sub O(n^2) sub O(n^3) \`
+ "**Big Omega**": Ω( g(n) ) specifies only the **lower** bound
  + Think of other examples?

---
## Proving asymptotic growth
+ **(p.52 #3.1-2)** ∀ a, b > 0, prove: \`(n+a)^b in Theta(n^b) \`
+ From definition: we need to **find** \`n_0, c_1, c_2\` such that
  \` forall n > n_0: c_1 n^b <= (n+a)^b <= c_2 n^b \`
+ i.e., find constants so we can **sandwich** \`(n+a)^b\` in between
  two multiples of \`n^b\`

---
## Prove: (n+a)^b ∈ Θ(n^b)
+ **Observe** that *n+a ≥ n/2*, as long as n > *2|a|*
  + Also, *n+a ≤ 2n*, as long as n > *|a|*
  + Hence, *n+a* is **sandwiched** by *n/2* and *2n* (if n > 2|a|):
    + *n/2* ≤ *n+a* ≤ *2n*
+ **Raise** to the *b* power (\`x^b\` is **monotone** if x > 1, b > 0)
  + Thus, \`(n/2)^b <= (n+a)^b <= (2n)^b\` (for n > 2|a|)
+ So we select \`n_0 = 2|a|, c_1 = 2^(-b), c_2 = 2^b\`
  + This **proves** the Theta bound.

---
## Asymptotic shorthand
+ Θ(g) is a **class** of functions
  + But for convenience, some **short-hand** notation:
+ When *Θ* (et al) are on the **right** side of =:
  + It means "there **exists**" \`f in Theta(g)\`
  + e.g., \` 2n^2 + 3n = Theta(n^2) \`
+ When *Θ* (et al) are on the **left** side of =:
  + It means "**for all**" \`f in Theta(g)\`
  + e.g., \` 4n^2 + Theta(n log(n)) = Theta(n^2) \`
  + True for **any** function in \`Theta(n log(n))\`

---
## Asymptotic domination: o, ω
+ "**Little o**": like a strict **less than** inequality: *f* ∈ *o(g)* iff
  + ∀ *c* > 0 ∃ *n0*: ∀ *n* > n0, 0 ≤ *f(n)* < *c g(n)*
  + i.e., the **limit** of *f(n)/g(n)* → 0 as n → ∞
+ "**Little omega**": like a strict **greater than**: *f* ∈ *ω(g)* iff
  + ∀ *c* > 0 ∃ *n0*: ∀ *n* > n0, 0 ≤ *c g(n)* < *f(n)*
  + i.e., the **limit** of *f(n)/g(n)* → ∞ as n → ∞

---
## Examples of o and ω
+ **Little o**: \`n^1.999 in o(n^2)\`, and \`n^2/log(n) in o(n^2) \`
  + but \` n^2/10000 notin o(n^2) \`
+ **Little omega**: \` n^2.0001 in omega(n^2) and n^2 log(n) = omega(n^2) \`

---
## Useful math identities
+ All *logs* are the **same** up to a constant factor:
  + \` log\_a(n) = (1/log\_b(a)) log\_b(n) \`
  + So we often use *lg* = \` log_2 \` for convenience
+ \` Theta(1) sub o(log(n)) sub o(n) sub o(n^(p)) sub o(p^n) \`
  + For any constant p > 1
+ In fact, ∀ a>1, b: \`lim_(n->oo) n^b / a^n = 0 \`
  + Hence, \` n^b in o(a^n) \`
  + *"Exponentials dominate polynomials"*

---
## Stirling's approximation
+ **Factorial**: n! = n(n-1)(n-2)...(2)(1)
  + Number of **permutations** of n distinct objects
+ **Stirling's approx**:
  \` n! = sqrt(2pi n)(n/e)^n (1+Theta(1/n)) \`
+ Hence, \` log(n!) in Theta(n log(n)) \`

---

## Outline for today
+ Algorithmic analysis
  + Insertion sort
+ Discrete math review
  + Logic and proofs
  + Monotonicity, limits, iterated functions
  + Fibonacci sequence and golden ratio
+ Asymptotic notation: Θ, O, Ω, o, ω
  + **Proving asymptotic bounds**

---
## Example asymptotic proof
+ *(p.62 #3-3)*: **Prove**: \`(log n)! in omega(n^3)\`
+ **Approach**: take *log* of both sides (log is monotone)
+ **Left side**: use Stirling:
  \` n! = sqrt(2pi n)(n/e)^n (1+Theta(1/n)) \`
  + So log( n! ) ∈ Θ( *n log(n)* )
  + Now **substitute** log(n) for n, using monotonicity of log:
    + So log( (log n)! ) ∈ Θ( *(log n) log(log n)* )

---
## Prove: (log n)! = ω(n^3)
+ ... so: log((log n)!) ∈ Θ( *(log n) log(log n)* )
+ **Right side**: \`log(n^3) = 3log n\`
  + This is **close** to the left side, with *3* instead of *log( log n )*
  + But we only need an **ω** bound, and *log( log n )* ∈ ω(*3*)
+ **Combining**: \`log( (log n)! ) in Theta( (log n) log( log n ) )\`
  + \` = omega( (log n) 3 ) = omega( log(n^3) ) \`
+ So by **monotonicity**, \`(log n)! in omega(n^3)\`

---

>>>
2016-09-13: finished here with 10min to spare, so started lec2 merge sort