Knowledge Bytes logoKnowledge BytesLearn Something New, One Byte at a Time.
Home/Learn AI/Module 2

Module 2 · Deep Learning Foundations

Activation Functions: ReLU, Sigmoid, and Tanh

Compare common neural-network activation functions and understand why nonlinearity is necessary for deep learning.

Written and reviewed by Mahendra Akula Knowledge Bytes

Core idea

Activation functions give networks nonlinear power

Without nonlinear activation functions, stacked layers collapse into a linear transformation and cannot model many complex relationships. ReLU is common in hidden layers, sigmoid maps values into a zero-to-one range, and tanh maps values between negative one and one.

Visual process

How it works

1Weighted sum z→
2Choose activation→
3Transform z→
4Pass representation forward→
5Backpropagate gradient

Practical context

Applied example

A binary classifier may use ReLU in hidden layers to learn nonlinear features and sigmoid at the output to produce a probability-like score. The score still needs calibration, threshold selection, and evaluation.
InputDefine the numerical, image, or labeled information supplied to the system.
TransformationTrace the learned calculation rather than treating the model as magic.
OutputInterpret scores and predictions in the application context.
ValidationTest unseen data, failure modes, drift, latency, cost, and human controls.

AWS Certified AI Practitioner

Exam signals

  • ReLU outputs max(0,z).
  • Sigmoid is common for binary outputs.
  • Softmax supports multi-class output distributions.
  • Activation choice affects gradient flow and model behavior.

Key takeaways

  • Activation functions add nonlinearity
  • ReLU is efficient for many hidden layers
  • Sigmoid is useful for binary probabilities
  • Function choice affects training behavior

Check your understanding

  1. Can you explain this concept in two sentences without using jargon?
  2. Can you identify the input, process, output, and validation step in the example?
  3. Can you name one suitable use case and one case where another approach is better?
  4. Which risk or limitation should a responsible implementation address?

Research references

Public sources and further reading

This lesson is original educational writing informed by the public references below. Use the sources to explore definitions, technical details, and current AWS exam objectives.