Deep Computer Vision (CNN)
Lesson 2Deep learning's role in computer vision
From hand-crafted features to representations learned from data
Deep learning's role in computer vision
Over the last decade, deep learning turned computer vision from hand-tuned features and brittle rules into models that learn straight from pixels. That change is not a small tweak; it is a different way to think about perception.
| Classical ML + Handcrafted Features | Deep Neural Networks (Dense / MLPs) |
|---|---|
| Manual feature engineering required | Learns features directly from pixels |
| Raw pixels were weak inputs for traditional ML algorithms | Pixels can be fed directly into neural networks |
| Required image processing expertise | Reduced dependence on handcrafted features |
| Features: , , , , | Representation learning emerges during training |
| Separate feature extraction + classifier pipeline | End-to-end learning becomes possible |
| Traditional ML: SVMs, Random Forests | Deep fully connected neural networks |
| Limited performance on complex visual tasks | Better learning capacity with large datasets |
Math: y = ML(f(x)) where f(x) is handcrafted features | Math: y = NN(x) where the network learns features automatically |
Code intuition: features = hog(image) | Code intuition: prediction = model(image_tensor) |
Two pipelines, same goal
Classical ML — features hand-engineered
Deep learning — features learned from pixels
Classical vision needs a separate feature step you design by hand; deep learning folds feature extraction into the trained network.
Neural networks are now the default engine for serious vision work, from datacenter training down to phones and wearables.
Real-world computer vision: cloud to edge
Vision models run across very different compute environments, from datacenter GPUs to low-power edge devices.
| Environment | Typical constraints | Common vision tasks |
|---|---|---|
| Mobile / edge devices | Low latency, low power, small memory | Face unlock, portrait enhancement, AR depth |
| Robots / drones / wearables | Real-time inference, constrained hardware | Navigation, tracking, obstacle perception |
| Cloud / datacenter | Large-scale training and serving | Foundation models, large-scale inference |
Edge models are often:
- quantized, lower-precision weights and activations
- pruned, redundant connections removed
- distilled, a small student trained to mimic a large teacher
- hardware-optimized, kernels tuned for specific accelerators
Vision in healthcare
Medical imaging already contains high-signal visual patterns, making it a strong domain for deep learning.
| Domain | Typical vision tasks |
|---|---|
| Radiology (CT / MRI / X-ray) | Detection, localization, triaging |
| Pathology | Tissue and cell-level analysis |
| Retina imaging | Disease screening (e.g. diabetic retinopathy) |
Models help surface subtle and repeatable patterns across large datasets, usually as decision-support systems alongside clinicians, not a replacement for clinical judgment.
Autonomous driving
Self-driving systems combine:
- perception, what's in the scene
- prediction, what will it do next
- planning, what should we do
- control, how do we execute it
One important research direction explored a much simpler framing:
| Input | Output |
|---|---|
| Camera frames / video | Steering and driving commands |
This introduced the idea of:
pixels → actions (end-to-end learning)
The challenge:
- safety-critical edge cases
- rare events ("long tail")
- interpretability
- real-world robustness
Modern autonomy stacks therefore combine learned vision systems, rules, mapping, planning, and systems engineering, rather than betting everything on a single end-to-end network.
Facial recognition and understanding
Computer vision evolved from simple face detection to rich facial understanding.
| Earlier systems | Modern deep learning systems |
|---|---|
| Detect whether a face exists | Detect, align, and embed faces |
| Limited geometric features | Identity, landmarks, gaze, pose, expressions |
| Simple classifiers | Deep metric-learning pipelines |
Modern systems must additionally handle:
- occlusion (masks, hands, hair)
- lighting variation
- pose changes
- privacy and bias concerns
Try it: zero-shot vision with CLIP
The "one flexible toolkit" claim above is easy to wave away in prose. Here it is concretely. CLIP is a vision-language model trained on image-caption pairs scraped from the web. It learned a shared embedding space: an image and a sentence describing that image end up close together; an unrelated sentence ends up far away.
Once you have that, classification stops being a fixed list of categories. You feed the model an image and whatever candidate captions you want, "a photo of a cat", "an indoor scene", "the year 1985", and it scores how well each one matches. No retraining, no fine-tuning, no labeled dataset. The same model handles "is this a cat?" and "is this happy?" because it lives in language-space.
Pick a preset image (or upload your own), tweak the candidate labels, and watch the scores shift:
The model running here is clip-vit-base-patch32, the same architecture used in production by Stable Diffusion, DALL·E text encoders, and countless retrieval systems. ~150 MB; cached in your browser after the first run.
Check your understanding
1 / 6What is the key shift deep learning brought to computer vision?
Practice it yourself
From edge-shrinking tricks to CLIP's zero-shot magic — implement the ideas, real Python, hidden tests, no setup.