← Concepts


This page describes how CNN and TF are quite similar. Based on demo xxx. (make this simplified version of 2.2b CNN<>TF comparison (WIP))

In future I will also add NN description. But basically CNN and TF have 2 sets of NNs.

  • Initial NN
    • CNN: Convolution (can be compared to NN style matrix math).
    • TF: QKV computation (context).
  • Final NN
    • CNN: Dense layers.
    • TF: FFN and final Wunembed.

    Notes: 1) TF FFN is performed at every layer. This is probably necessary because language is much more complicated to compute from crude token input compared to the input pixels of a CNN. 2) Each TF token has its own FFN (NN). But the final classification of the input is performed in the dims of the final token only (because this is how the TF is trained).

drones


1a CNN convolution

For one new pixel in one new filter:

  • 1) For each kernel position Compute (Kernel scalar) * (all old-filter pixel values at that position)
  • 2) Sum all those products
  • 3) Add ONE bias
  • 4) Apply ReLU

Example:

  • (6) 14x14 pix’s, 8 dims (feature maps)
  • (7) CONV2 3x3, 16 filters (feature maps to be create from the 8 existing)
  • (7b) Convolution2.
    • This shows a new pixel (red) in a new feature map.
    • The white kernel (3x3) pix’s are = 0 (paddding).
    • The kernel red pix value is used to multiply the same pix location value for all 8 maps.
    • Same is done for each yellow pix (multiply the same pix location in all other maps).
    • 1) Add up all the pix sums (4 in this case).
    • 3) Add bias.
    • 4) Apply (9) RELU2. Repeat the above
    • For each pixel (14x14) for the first filter.
      • You now have a 14x14x1 set. Repeat the above
    • For remaining 15 filters.
      • Each filter has different Kernel values.

drones


1b TF attention heads


2a CNN max pooling

  • Continue at P2 in the above CNN diagram.
  • In each filter for each set of 4 pix’s choose the highest pix value.
  • All other pixs are discarded.
  • You now have 7x7 with 16 feature maps.
    • Basically the pix’s now represent higher level features than before.


2b TF feature extraction (FFN)


3a CNN dense layers

  • The 7x7x16 (784) resulting pix’s from convolution/pooling are now fed into NN.
  • Result is 10 “logits” that represent the probability of what digit was input.

drones


3b TF FFN / Wunembed


4 How CNN/TF are similar

CNN and TF are solving the same kind of problem:

  • Providing a computational “surface” that matches the input.
  • Pixels are related to neighbors; for TF relation is context (QKV).

  • etc…..

  • I still need to embellish this with more detail.
  • Will do that after I add TF content.
  • See #609 for source chats.

CNN

  • Fixed local neighbors
  • Neighbor importance = learned kernel weights
  • Values = pixel/filter activations
  • Spatial size often reduced by pooling
  • Final dense classifier

TF

  • Dynamic semantic neighbors
  • Neighbor importance = QK scores + softmax
  • Values = V vectors
  • Sequence length usually preserved
  • Meaning/story builds across many layers
  • Final W_unembed maps hidden state → vocabulary logits

Your key insight is right:

  • CNN kernel = fixed local mixing rule
  • TF attention = dynamic meaning-based mixing rule

And yes, the reason there is no big dense classifier tail like CNN is basically:

  • The transformer block already keeps refining each token vector.
  • The final unembedding is just the last projection into vocab space.


26.0624 (v1 26.0624)