Deep Computer Vision (CNN)
Lesson 2Deep learning's role in computer vision
From hand-crafted features to representations learned from data
Deep learning's role in computer vision
Over the last decade, deep learning turned computer vision from hand-tuned features and brittle rules into models that learn straight from pixels. That change is not a small tweak; it is a different way to think about perception.
| Classical ML + Handcrafted Features | Deep Neural Networks (Dense / MLPs) |
|---|---|
| Manual feature engineering required | Learns features directly from pixels |
| Raw pixels were weak inputs for traditional ML algorithms | Pixels can be fed directly into neural networks |
| Required image processing expertise | Reduced dependence on handcrafted features |
| Features: , , , , | Representation learning emerges during training |
| Separate feature extraction + classifier pipeline | End-to-end learning becomes possible |
| Traditional ML: SVMs, Random Forests | Deep fully connected neural networks |
| Limited performance on complex visual tasks | Better learning capacity with large datasets |
Math: y = ML(f(x)) where f(x) is handcrafted features | Math: y = NN(x) where the network learns features automatically |
Code intuition: features = hog(image) | Code intuition: prediction = model(image_tensor) |
Two pipelines, same goal
Classical ML — features hand-engineered
Deep learning — features learned from pixels
Classical vision needs a separate feature step you design by hand; deep learning folds feature extraction into the trained network.
Neural networks are now the default engine for serious vision work, from datacenter training down to phones and wearables.
What the dense network gives up
Look again at the right-hand column: those are fully connected networks. A dense layer takes a flat vector, so the image has to be flattened first — a 28×28 grid becomes a 784-long list before the first weight is ever applied.
That step is lossless in values and catastrophic in structure. Pixel (5, 5) and pixel (5, 6) are neighbours in the image and 1 apart in the list; pixel (5, 5) and (6, 5) are also neighbours and 28 apart. The network is handed no reason to treat either pair as related. Shuffle every pixel with a fixed permutation and a dense network trains to exactly the same accuracy — proof that it never used the geometry at all.
Press Flatten below and watch the grid unravel into a row — the values all survive, the geometry does not:
Animation · Flattening Destroys Underlying Spatial Information
2D image (8 × 8) — a tiny cityscape
So learning features from pixels was the right move, but doing it with dense layers throws away the one thing images have that vectors do not: locality. Lesson 5 is about putting it back.
Real-world computer vision: cloud to edge
Vision models run across very different compute environments, from datacenter GPUs to low-power edge devices.
| Environment | Typical constraints | Common vision tasks |
|---|---|---|
| Mobile / edge devices | Low latency, low power, small memory | Face unlock, portrait enhancement, AR depth |
| Robots / drones / wearables | Real-time inference, constrained hardware | Navigation, tracking, obstacle perception |
| Cloud / datacenter | Large-scale training and serving | Foundation models, large-scale inference |
Edge models are often:
- quantized, lower-precision weights and activations
- pruned, redundant connections removed
- distilled, a small student trained to mimic a large teacher
- hardware-optimized, kernels tuned for specific accelerators
Vision in healthcare
Medical imaging already contains high-signal visual patterns, making it a strong domain for deep learning.
| Domain | Typical vision tasks |
|---|---|
| Radiology (CT / MRI / X-ray) | Detection, localization, triaging |
| Pathology | Tissue and cell-level analysis |
| Retina imaging | Disease screening (e.g. diabetic retinopathy) |
Models help surface subtle and repeatable patterns across large datasets, usually as decision-support systems alongside clinicians, not a replacement for clinical judgment.
Autonomous driving
Self-driving systems combine:
- perception, what's in the scene
- prediction, what will it do next
- planning, what should we do
- control, how do we execute it
One important research direction explored a much simpler framing:
| Input | Output |
|---|---|
| Camera frames / video | Steering and driving commands |
This introduced the idea of:
pixels → actions (end-to-end learning)
The challenge:
- safety-critical edge cases
- rare events ("long tail")
- interpretability
- real-world robustness
Modern autonomy stacks therefore combine learned vision systems, rules, mapping, planning, and systems engineering, rather than betting everything on a single end-to-end network.
Facial recognition and understanding
Computer vision evolved from simple face detection to rich facial understanding.
| Earlier systems | Modern deep learning systems |
|---|---|
| Detect whether a face exists | Detect, align, and embed faces |
| Limited geometric features | Identity, landmarks, gaze, pose, expressions |
| Simple classifiers | Deep metric-learning pipelines |
Modern systems must additionally handle:
- occlusion (masks, hands, hair)
- lighting variation
- pose changes
- privacy and bias concerns
Try it: zero-shot vision with CLIP
The "one flexible toolkit" claim above is easy to wave away in prose. Here it is concretely. CLIP is a vision-language model trained on image-caption pairs scraped from the web. It learned a shared embedding space: an image and a sentence describing that image end up close together; an unrelated sentence ends up far away.
Once you have that, classification stops being a fixed list of categories. You feed the model an image and whatever candidate captions you want, "a photo of a cat", "an indoor scene", "the year 1985", and it scores how well each one matches. No retraining, no fine-tuning, no labeled dataset. The same model handles "is this a cat?" and "is this happy?" because it lives in language-space.
Pick a preset image (or upload your own), tweak the candidate labels, and watch the scores shift:
The model running here is clip-vit-base-patch32, the same architecture used in production by Stable Diffusion, DALL·E text encoders, and countless retrieval systems. ~150 MB; cached in your browser after the first run.
Check your understanding
1 / 6What is the key shift deep learning brought to computer vision?
Practice it yourself
From edge-shrinking tricks to CLIP's zero-shot magic — implement the ideas, real Python, hidden tests, no setup.