Deep Computer Vision (CNN)
Lesson 4Feature detection & hierarchical structure
Edges → textures → shapes → objects, layer by layer
Feature detection & hierarchical structure
In this chapter, we will talk about essentially feature extraction and feature detection. Why this feature detection is harder, and what are the main reasons that understanding images in raw pixel space is not enough and we need to understand the spatial relationship and the hierarchical features hidden inside an image.
Visual classification
At the heart of any image classification problem lies the task of feature detection.
For instance, if you are trying to classify an image as a human face, you might define the necessary features as eyes, a nose, and a mouth. Once these components are detected in the image, you gain confidence that the object is indeed a face. Similarly, if you are trying to detect a house, you might define features such as doors, windows, and rooftops. The more of these features the system detects, the more confident it becomes in its classification.
This process may appear intuitive, but building a system that performs it reliably across diverse inputs is highly nontrivial.
| You want to detect… | Typical features you might check for |
|---|---|
| Human face | Eyes, nose, mouth |
| House | Doors, windows, rooftops |
Why feature detection is harder than it looks
The difficulty arises from two core challenges:
1. Feature hierarchy and recursion
2. High variability in input space
Feature hierarchy and recursion
To detect a face, you must detect eyes. (face → depends on eyes → depends on the eye ball → a dark black or brown dot → so on… )
But to detect eyes, you now have a subproblem that is structurally identical to the original problem.
Each feature itself requires its own sub-feature detection process.
This introduces a hierarchical structure where higher-level features are composed of lower-level ones.
Idea: nested detection
Same pattern repeats: each level needs its own detector.
High variability in input space
A single object class, such as a face, can appear vastly different depending on:
A single class (e.g. a face) can vary because of… |
|---|
| Lighting conditions |
| Pose and orientation |
| Occlusion (e.g. sunglasses, hats, shadows) |
| Camera sensor differences |
| Scale and resolution |
In raw pixel space, even small transformations can lead to large changes in numerical values.
This makes it hard for classical machine learning models to generalize, as they are sensitive to exact pixel patterns.
The limitations of classical machine learning
In traditional machine learning, feature engineering is a manual process.
Domain experts are responsible for defining what to look for in the data. For image tasks, this means designing filters or extracting geometric and statistical descriptors like SIFT, HOG, or edge detectors.
While effective in constrained settings, this approach fails under real-world variation. It lacks adaptability and requires constant tweaking for new tasks or domains.
Most importantly, these hand-engineered pipelines are brittle. If the features are poorly chosen or do not generalize, the performance collapses. The system’s success depends entirely on human insight and intuition.
Deep learning as the paradigm shift
Deep neural networks were a major shift from classical ML because models could now learn features directly from data instead of relying on handcrafted image processing pipelines.
But fully connected deep networks still struggled with images:
- they ignored spatial locality
- parameter count exploded for large images
- they did not naturally exploit image structure
Convolutional Neural Networks (CNNs) solved this by introducing architectures specifically designed for visual data.
Interactive · DNN vs CNN
Adjust the sliders to feed an image into a single fully-connected layer on the left, or a single convolutional layer on the right. Watch the parameter counts diverge.
Dense / FC layer
DNNH × W × C × hidden
224 × 224 × 3 × 1000
150.53M
= 150,528,000 parameters
Convolutional layer
CNNK × K × C × filters
3 × 3 × 3 × 32
864
= 864 parameters
Side-by-side
FC needs this many times more parameters than Conv
174.2K×
And FC is for one image. You'd repeat this overhead for every layer.
| Deep Neural Networks (Dense / MLPs) | Convolutional Neural Networks (CNNs) |
|---|---|
| Learns features directly from pixels | Learns spatial hierarchies from pixels |
| Fully connected architecture | Convolution-based architecture |
| Treats image as a flattened vector | Preserves 2D spatial structure |
✗ Huge parameter count for images | Weight sharing drastically reduces parameters |
✗ Weak inductive bias for images | Built specifically for visual patterns |
✗ Struggles to capture locality and translation patterns | Learns local receptive fields naturally |
| Representation learning emerges during training | Hierarchical feature learning emerges layer-by-layer |
✗ Lower scalability for high-resolution images | Efficient and scalable for vision workloads |
| Dense matrix multiplications dominate computation | Convolution operations dominate computation |
| End-to-end learning becomes possible | End-to-end visual feature extraction becomes practical |
| Limited robustness to visual variation | Better practical invariance to translation, scale, texture, brightness |
| Mostly generic deep learning architecture | Backbone of modern computer vision |
Hierarchical feature learning in CNNs
One of the biggest breakthroughs of CNNs was automatic hierarchical representation learning.
| CNN layer depth | What CNNs often learn |
|---|---|
| Lower layers | Edges, gradients, textures |
| Middle layers | Shapes, corners, object parts |
| Higher layers | Object-level abstractions |
Depth → abstraction
This hierarchy emerges automatically during training. The network itself learns:
- which visual patterns matter
- where they matter
- how to combine them for prediction
…without manually engineered image features like:
- edge detectors
- handcrafted histograms
Why this works: distributed & compositional representations
The power of this approach lies in two properties of the representations CNNs learn:
Distributed
No single neuron defines a concept. Instead, many neurons jointly encode patterns, robustness emerges from redundancy.
Compositional
High-level features are composed of lower-level ones, learned through successive transformations, small detectors stack into big concepts.
The feature extractor is no longer manually programmed by humans. It is learned directly from data through optimization.
Check your understanding
1 / 8In a deep CNN, what kind of features do the EARLY (low) layers typically learn?
Practice it yourself
From a hand-coded edge filter to the parameter math that makes CNNs win — implement the hierarchy story. Real Python, hidden tests, no setup.