Improvements

Improved Implementation of Softmax

Numerical Stability Issues

Floating-Point Precision

• The previous implementation of neural networks with softmax layers works, but has potential numerical issues
• Problem stems from floating-point precision limitations in computers

Illustrative Example:

1. Option 1: Calculate x = 2/10,000 directly
2. Option 2: Calculate x = (1 + 1/10,000) - (1 - 1/10,000)
   • Mathematically equivalent to 2/10,000
   • Computationally produces round-off errors

Note

When computers store decimal numbers with finite precision (floating-point), different calculation approaches can lead to different levels of numerical round-off errors.

Improving Logistic Regression First

Conceptual Bridge

Original Implementation:

1. Compute activation: a = g(z) = 1/(1+e^(-z))
2. Compute loss: -y*log(a) - (1-y)*log(1-a)

Original Implementation

model = tf.keras.Sequential([
 # Previous layers...
 tf.keras.layers.Dense(1, activation='sigmoid')
])

model.compile(
 loss='BinaryCrossentropy',
 optimizer='adam',
 metrics=['accuracy']
)

Improved Implementation:

• Instead of computing a separately, provide the complete loss expression
• Let TensorFlow optimize the computation internally

Improved Implementation

model = tf.keras.Sequential([
 # Previous layers...
 tf.keras.layers.Dense(1, activation='linear')
])

model.compile(
 loss=tf.keras.losses.BinaryCrossentropy(from_logits=True),
 optimizer='adam',
 metrics=['accuracy']
)

Tip

The from_logits=True parameter tells TensorFlow that the model outputs raw scores (logits) rather than probabilities, giving it flexibility to rearrange calculations for better numerical stability.

Applying to Softmax

Key Improvement

Original Softmax Implementation:

1. Compute activations: aⱼ = e^(zⱼ) / Σ e^(zᵢ)
2. Compute loss: -log(aⱼ) where j is the correct class

Original Softmax Implementation

model = tf.keras.Sequential([
 # Previous layers...
 tf.keras.layers.Dense(10, activation='softmax')
])

model.compile(
 loss='SparseCategoricalCrossentropy',
 optimizer='adam',
 metrics=['accuracy']
)

Improved Softmax Implementation:

• Output layer uses linear activation (outputs z values directly)
• Loss function handles both softmax activation and cross-entropy calculation

Improved Softmax Implementation

model = tf.keras.Sequential([
 # Previous layers...
 tf.keras.layers.Dense(10, activation='linear')
])

model.compile(
 loss=tf.keras.losses.SparseCategoricalCrossentropy(from_logits=True),
 optimizer='adam',
 metrics=['accuracy']
)

Why This Matters

Numerical Stability

• Avoids computing extremely small values (e^(-large_number))
• Avoids computing extremely large values (e^(large_number))
• TensorFlow can rearrange terms for more accurate computation

Trade-offs

• More numerically accurate
• Less intuitive/readable code
• Conceptually equivalent to original implementation

Important Note About Model Outputs

Caution

With this implementation, the neural network’s final layer no longer outputs probabilities (a₁ through a₁₀). Instead, it outputs the raw logits (z₁ through z₁₀). If you need actual probabilities for predictions, you’ll need to apply the softmax function to these outputs.

Looking Ahead

• You now know how to implement multiclass classification with softmax output layers
• You understand how to do it in a numerically stable way
• Next topic: Multi-label classification (a different type of classification problem)

The improved implementation of softmax neural networks focuses on numerical stability by separating the activation and loss functions in the code, even though conceptually they remain connected. This approach allows TensorFlow to optimize the calculations internally, reducing rounding errors that can occur with very large or very small numbers.