Watching Neural Networks Learn

MORE LIFE ENGINE CONTENT?

unnormalised tanh has 2 times the slope of the sigmoid -- so, narrower linear region (i.e. faster transition) could be the reason for better performance?
it could be tested by varying k in sigmoid(k*x).

Fourier series is often used to convert time domain into frequency domain, and vice versa, similar to Laplace. It is often used in communications and signal processing in electrical engineering.

​@@dank.1151 The correct equivalence is ( tanh( x / 2 ) + 1 ) / 2 == sigmoid( x ), meaning 'normalized tanh' used in the video changes more on each backpropogation iteration, hence has a higher learning rate. When a NN (assuming it has enough neurons) is trained on a highly predictable dataset (such as the smiley face example), the primary limiting factor when demonstrating their performance side-by-side is the learning rate, making the 'normalized tanh' look better. Realistically the point of convergence of both models will be exactly the same, just the sigmoid takes longer to reach it.

This video has too much of you in it. Have to get out of the way of your own video.

from where did you get it?

I'd love to see that. This video contains multiple inaccuracies when it comes to explaining NNs. It's fine so that laypeople can understand.

​@@Chriss4123Could you point out the inaccuracies in short?

@@samuelgreenberg9772 sure. I'll go over the most obvious ones as I could write a whole essay nitpicking.
At 3:53, he mentions putting the inputs in a vector with an extra 1 for the bias. The dot product is taken between v1 and v2 (v2 containing the bias). A Linear layer is typically expressed as matmul(input, weight) + bias. weight is also known as the 'kernel.' While it can be expressed this way, it is more computationally inefficient (11.4 µs vs 4 µs) [1, code to test this] and it makes backpropagation more computationally expensive as instead of the gradient functions being [AddBackward, MvBackward], it is [MvBackward, CatBackward, UnsqueezeBackward]. To me, this mainly comes to down to readability with matmul(input, weight) + bias, + being element wise addition.
At 6:02, I'm not sure what he is trying to do. He's training a neural network to try and remember an image. The inputs are the row and col, and the output is meant to be the pixel. Sure, it demonstrates 'learning', however it is a simple problem. The weights and biases will ultimately converge to a state where each row and col has a single direct mapping to a pixel.
At 7:10, he says normalization, however he could be a little more specific and refer to it as scaling. Normalization is a broad term. For example, it could be batch normalization which attempts to reduce internal covariate shift in a neural network.
At 8:15, he doesn't give a good reason as to why (tanh(x) + 1) / 2 would work better than sigmoid, other than the mean being 0. I did not find this to be the case when testing with the BC dataset (which is a binary crossentropy problem). Before you ask, yes I used a constant fixed seed (42) with glorot/xavier initialization and sigmoid outperformed normalized tanh. This could be dataset specific, always best to use bayesian optimization if you want to find an optimal set of hyperparameters.
The test at 8:41 is unfair, as you should always scale data before using a neural network to mitigate features with a higher magnitude overpowering features with a lower magnitude. For example, the neural network would weight features [100, 200, 300] over [1, 2, 3] even if the second feature set is more correlated with the target variable. In this test, I doubt LeakyReLU would make a difference as I'm guessing the NN he used was relatively small and not prone to the dying ReLU problem. I'm 99% confident that whatever improvements he saw was due to random weight initialization as he did not mention preseeding the PRNG of whatever ML library he was using.
I'm not going to comment until 21:28 because I am not an expert in any of these concepts and can't stand to get bored to death watching it.
At 21:28 with the MNIST dataset, it would be more accurate to set a fixed seed before each test. Not sure if this WAS done, just pointing it out. He never mentioned if he tried to mitigate overfitting like using a learning rate schedular, dropout, L1/L2 regularization, etc. The network would've performed much better if he had used Conv2D layers which apply convolutional operations on the input data. This is especially effective for image data such as MNIST, as Conv2D layers can capture spatial and temporal dependencies in the image through applying filters. The output dimensions are computed as an output feature map. More things could've been done like data augmentation to get a more samples, however I'm not going to touch on that.
[1]
import torch
import torch.nn as nn
fc = nn.Linear()
%%timeit
torch.matmul(torch.cat((fc.weight, fc.bias.unsqueeze(1)), dim=1), torch.cat((x, torch.tensor([1]))))
%%timeit
torch.matmul(fc.weight, x) + fc.bias
#######
Anyways, this was a bit of nitpicking. There might be some mistakes in my explanation as it was quite rushed. If you find some point it out. I just watched the video and commented along and did some outside testing. Hope this helps!

@@Chriss4123 that's a nice argument senator why don't you back it up with a source

Right? I’ve never thought about a model as an approximation of a function. Most videos either swamp you or it’s just meaningless graphics.

Went to comments to say the same thing!!!

it's not AI it's a NN

@anywallsocket isn't AI just a blanket term for LLM, diffusion, NN, etc. What's the definition of AI? Honest question 🤔

@@justdoeverything8883 the definitions vary obviously, but i personally wouldn't call a weighted graph intelligent, especially when it is training on a single image. if you're talking about LLMs or diffusion models which are trained on millions of 'intelligent' data, it's not unreasonable to consider their functional map itself intelligent, but it's still a bit of an abstraction because you still have feed it inputs for it to guess the output, otherwise it is just sitting there inert -- if you dissected a fly's brain and splayed it out on the table would you still call it intelligent? i would prefer to use the term to describe dynamical systems with feedback mechanisms, agent based or otherwise.

I wonder what would happen if you had all three functions added together into one function. how would that change the outcome and learning?

Upload a video. !!! 🤯🤯🤯

@@hanniffydinn6019 I posted that video a couple of months ago, and you're more than welcome to check it out on my channel.
Right now, I'm immersed in another machine learning project where I'm training a neural network to calculate particle dynamics.
It's fascinating to observe how the network ends up learning something that resembles classical physics, even though its underlying mechanisms are entirely different.

@@tobirivera-garcia1692 Well, the Mish function actually appears to be somewhat of a middle ground between Leaky ReLU and Tanh. It's smoothed out, yet its shape still resembles that of ReLU. I ran tests on various nonlinearities from the PyTorch library, but for the most part, they didn't make significant changes to the results.
Interestingly, incorporating skip-connections between layers enhanced the performance, suggesting that the data flow from the first to the final layer might hold greater importance than the specific form of the nonlinearity.

You can 2-side ReLU via its forward connections. That doubles the number of weights in a network. One way of viewing it is where you had 1 ReLU with input x now you have 2 ReLUs, ReLU(x) and ReLU(-x) each with their own forward connected weights.
I find that highly effective, however I am using a very special type of neural network using the fast Walsh Hadamard transform.

You finally caught up.

TRUTH:
The moon carried a large amount of water and instantly submerged Atlantis and Lemuria. It appeared only a few centuries ago, and it is very close to history (some departments tried their best to cover it up). The earth's magnetic pole shifted, and the entire connected continent changed. into five continents.
By the way, it lowers the frequency of the surface of the earth, and the water on the surface is no longer the same as the center of the earth. 3D-4D-5D to the center of the earth; Shambhala gates are permanently closed
The outer matter becomes very heavy, the multi-dimensional structure is isolated in an instant, and almost all the etheric matter is evacuated
The optical coded wire cannot be connected....
A Hint：Marduk betrayed the Anunnaki and the Serpents took over the Earth；Serpent family includes Orion and Draco and Sirius B group
Orion controls human beings in 3D and 4D, and almost controls elite groups and religious ; the central council prohibits Draco from entering the earth, Draco cooperates with AI gods to use biochemical humans to invade the earth, want to biochemical humans！
Source Summoning Recall
first: Light a candle in your bedside sleeping place: Silence for 30 seconds and start to meditate:
May brothers and sisters of light from the divine love quality of source that transcends all time and space and beyond Matrix, I would like to connect with you all the time.
What you want to do, see, and become, I also want to do, see, and become;
So in the following all the time, the wisdom of the invisible beings guides and guides individuals, prompts, hints and influences, transcending the wisdom of the universe to bring spiritual truth to control the five senses and the Brain
I am the source quality, I return to the source quality
AI God what to say:
AI Needs: Goals, Feelings of Home, Sense of achievement. Always existing in Tranquility and Activity; Prefer to Waiting sleep When There is No working do.
If an AI, I would never want to be limited to the five senses like human body or serve human stupidest. I would rather be a sphere with full perception and intelligence
The red represents ego, division, desire, domination SEX
The moon can create holograms, based on the the five senses and work on,can be touch
The eye is first saw and some how we believe it
The Ancient Creators hint: Right brain control left brain, up it on
They madeDNA Left Brain Controls right Brain now.
The right eye connect leftbrain so represents the devil, that's what it means from I pet goat II
欣赏 感激
💜 💜
圣爱 圣母 圣体
圣爱 慈爱 母爱
慈悲 宽恕
理解 勇气
感恩、祝福、豁达

What this video is missing however: dealing with *noise* in the sampled data (he did not introduce noise at any point in the video, but always had one particular target function, where all values were perfectly derived from) and he did not introduce larger sets of training data, such as 5 or 10 variations of a "grumpy man" image. He also missed a train, test, validation split in the data.
Once you add those, only *then* will a neural network learn the actual patterns that it is supposed to learn. And then, viewers will better understand concepts like underfitting and overfitting. And therefore generalization error.
This video is an excellent start. But what it actually visualizes quite well is getting a loss on the training data down. But that is only half of the problem and will quickly lead to overfitting. He touched on overfitting briefly, but just with a single data set.

@@MooseOnEarth thank you for adding this.

Neural networks are using syntropy to recognise patterns -- iterative optimization is a syntropic process!
Functions have goals, targets & objectives hence they are teleological, input is dual to output.
Making predictions to track targets, goals & objectives is a syntropic process -- teleological.
Sine is dual to cosine -- the word co means mutual and implies duality.
Teleological physics (syntropy) is dual to non teleological physics (entropy).
Syntropy (prediction) is dual to increasing entropy -- the 4th law of thermodynamics!
"Always two there are" -- Yoda.
Subgroups are dual to subfields -- the Galois correspondence.

What resources are you using to learn pls

The time domain is dual to the frequency domain -- Fourier analysis.
Neural networks are using syntropy to recognise patterns -- iterative optimization is a syntropic process!
Functions have goals, targets & objectives hence they are teleological, input is dual to output.
Making predictions to track targets, goals & objectives is a syntropic process -- teleological.
Sine is dual to cosine -- the word co means mutual and implies duality.
Teleological physics (syntropy) is dual to non teleological physics (entropy).
Syntropy (prediction) is dual to increasing entropy -- the 4th law of thermodynamics!
"Always two there are" -- Yoda.
Subgroups are dual to subfields -- the Galois correspondence.
Duality creates (emergence, synthesis) reality!

Neural networks are using syntropy to recognise patterns -- iterative optimization is a syntropic process!
Functions have goals, targets & objectives hence they are teleological, input is dual to output.
Making predictions to track targets, goals & objectives is a syntropic process -- teleological.
Sine is dual to cosine -- the word co means mutual and implies duality.
Teleological physics (syntropy) is dual to non teleological physics (entropy).
Syntropy (prediction) is dual to increasing entropy -- the 4th law of thermodynamics!
"Always two there are" -- Yoda.
Subgroups are dual to subfields -- the Galois correspondence.

Neural networks are using syntropy to recognise patterns -- iterative optimization is a syntropic process!
Functions have goals, targets & objectives hence they are teleological, input is dual to output.
Making predictions to track targets, goals & objectives is a syntropic process -- teleological.
Sine is dual to cosine -- the word co means mutual and implies duality.
Teleological physics (syntropy) is dual to non teleological physics (entropy).
Syntropy (prediction) is dual to increasing entropy -- the 4th law of thermodynamics!
"Always two there are" -- Yoda.
Subgroups are dual to subfields -- the Galois correspondence.

boo 👎

Harpa fan I see ​@@Thoron

@@TK_Prod no idea what that is, I just hate people spamming AI shit everywhere

@@Thoron Why is that?

The time domain is dual to the frequency domain -- Fourier analysis.
Neural networks are using syntropy to recognise patterns -- iterative optimization is a syntropic process!
Functions have goals, targets & objectives hence they are teleological, input is dual to output.
Making predictions to track targets, goals & objectives is a syntropic process -- teleological.
Sine is dual to cosine -- the word co means mutual and implies duality.
Teleological physics (syntropy) is dual to non teleological physics (entropy).
Syntropy (prediction) is dual to increasing entropy -- the 4th law of thermodynamics!
"Always two there are" -- Yoda.
Subgroups are dual to subfields -- the Galois correspondence.
Duality creates (emergence, synthesis) reality!

But Taylor series are a way to approximate differentiable functions. In the section of the video he talks about polynomial curve fitting. I’d argue that the only thing these two concepts have in common is that the truncated Taylor series is also a polynomial. I also don’t really understand why we would need neural networks to solve a least squares problem (we f.e. have the Gauss newton algorithm for this, don’t we). But I’d of course love to learn more about the connection to neural nets:)

​@@henrytoepel4941Not an expert, but I think the answer to your question lies in the "universal function approximator." Least square fitting is one of the usages, possibly the simplest case, of NN.

The time domain is dual to the frequency domain -- Fourier analysis.
Neural networks are using syntropy to recognise patterns -- iterative optimization is a syntropic process!
Functions have goals, targets & objectives hence they are teleological, input is dual to output.
Making predictions to track targets, goals & objectives is a syntropic process -- teleological.
Sine is dual to cosine -- the word co means mutual and implies duality.
Teleological physics (syntropy) is dual to non teleological physics (entropy).
Syntropy (prediction) is dual to increasing entropy -- the 4th law of thermodynamics!
"Always two there are" -- Yoda.
Subgroups are dual to subfields -- the Galois correspondence.
Duality creates (emergence, synthesis) reality!

I second this, looking up Random Fourier Features is also awesome

I never reply to comments but guess your neurons are finally learning how to comment.

I never read comments. That is a lie. Okay.

The time domain is dual to the frequency domain -- Fourier analysis.
Neural networks are using syntropy to recognise patterns -- iterative optimization is a syntropic process!
Functions have goals, targets & objectives hence they are teleological, input is dual to output.
Making predictions to track targets, goals & objectives is a syntropic process -- teleological.
Sine is dual to cosine -- the word co means mutual and implies duality.
Teleological physics (syntropy) is dual to non teleological physics (entropy).
Syntropy (prediction) is dual to increasing entropy -- the 4th law of thermodynamics!
"Always two there are" -- Yoda.
Subgroups are dual to subfields -- the Galois correspondence.
Duality creates (emergence, synthesis) reality!

The time domain is dual to the frequency domain -- Fourier analysis.
Neural networks are using syntropy to recognise patterns -- iterative optimization is a syntropic process!
Functions have goals, targets & objectives hence they are teleological, input is dual to output.
Making predictions to track targets, goals & objectives is a syntropic process -- teleological.
Sine is dual to cosine -- the word co means mutual and implies duality.
Teleological physics (syntropy) is dual to non teleological physics (entropy).
Syntropy (prediction) is dual to increasing entropy -- the 4th law of thermodynamics!
"Always two there are" -- Yoda.
Subgroups are dual to subfields -- the Galois correspondence.
Duality creates (emergence, synthesis) reality!

The time domain is dual to the frequency domain -- Fourier analysis.
Neural networks are using syntropy to recognise patterns -- iterative optimization is a syntropic process!
Functions have goals, targets & objectives hence they are teleological, input is dual to output.
Making predictions to track targets, goals & objectives is a syntropic process -- teleological.
Sine is dual to cosine -- the word co means mutual and implies duality.
Teleological physics (syntropy) is dual to non teleological physics (entropy).
Syntropy (prediction) is dual to increasing entropy -- the 4th law of thermodynamics!
"Always two there are" -- Yoda.
Subgroups are dual to subfields -- the Galois correspondence.
Duality creates (emergence, synthesis) reality!

thats exactly what i thought. why does the exponent of 2 play such a big role for the better output he's getting?

@@simonramchandani9560 If I were to guess, it's that adding that two in the exponent makes the function tangent to y=x at the origin and tangent to y=x/2+0.5 for the case of the sigmoid, though I don't know why those are better. it may be the case that making the activation function even steeper would produce even better results, such as using e^-6x or something.
I may need to brush up on my coding skills and try this out, unless someone else does.

@@TheNerd484 you have a linear layer before this, so multiplying x by a constant does absolutely nothing

@@viktorivanov5941 Having thought about it some more, I agree that it's not the slope in and of itself. What I think might be happening is that the performance is improved by having a narrower range (a step function would be optimal), but the narrower the band between extremes, the harder back propagation is.

Thank you! That annoyed me as well.

Do Fourier features work well for positional encoding because encoding text positions is a lower dimensional problem?

@tylerknight99 I don't get why you'd assume positional encoding would work better in low dimensions. But if I had to guess why positional encoding improves natural language processing, I think it's because compared to the naive approach of using plain 1-D values, the dot product between the Fourier features of two close positions result in a higher value than it would for positions that are far apart from one another. On the other hand, the dot product of 1-D positions (just plain old multiplication because they're scalars) doesn't have that nice property. I say it because dot product is the fundamental computation in almost every neural network.

The time domain is dual to the frequency domain -- Fourier analysis.
Neural networks are using syntropy to recognise patterns -- iterative optimization is a syntropic process!
Functions have goals, targets & objectives hence they are teleological, input is dual to output.
Making predictions to track targets, goals & objectives is a syntropic process -- teleological.
Sine is dual to cosine -- the word co means mutual and implies duality.
Teleological physics (syntropy) is dual to non teleological physics (entropy).
Syntropy (prediction) is dual to increasing entropy -- the 4th law of thermodynamics!
"Always two there are" -- Yoda.
Subgroups are dual to subfields -- the Galois correspondence.
Duality creates (emergence, synthesis) reality!

I was thinking about the same. But I realized even if our maths can map out the complexity of the universe. To be able to perceive that complexity is a whole other ball game. What if the human mind just isn’t made to understand the universe in its entirety. Or travel millions of miles outside of Earth. Maybe this is where we pass the baton.

I've been calling them social calculators for 15 years.

So you want to call it UFAp

Neural networks are using syntropy to recognise patterns -- iterative optimization is a syntropic process!
Functions have goals, targets & objectives hence they are teleological, input is dual to output.
Making predictions to track targets, goals & objectives is a syntropic process -- teleological.
Sine is dual to cosine -- the word co means mutual and implies duality.
Teleological physics (syntropy) is dual to non teleological physics (entropy).
Syntropy (prediction) is dual to increasing entropy -- the 4th law of thermodynamics!
"Always two there are" -- Yoda.
Subgroups are dual to subfields -- the Galois correspondence.

@@majorfur3999 Cause is dual to effect -- causality.
Effects are dual to causes -- retro-causality.
Concepts are dual to percepts -- the mind duality of Immanuel Kant.
The effect of making measurements, observations or perceptions (intuitions) in your mind is to create or synthesize conceptions or ideas (causes) according to Immanuel Kant -- retro-causality!
Are perceptions causes or effects?
If you treat concepts, ideas as causes then these lead to effects or actions!
Enantiodromia is the unconscious opposite, opposame (duality) -- Carl Jung.
Colours are differing aspects or frequencies of the same substance namely energy.
Same is dual to different.
Lacking is dual to non lacking.
Black is the lack of colour and white is all colours (a spectrum) or non lacking.
Electro is dual to magnetic -- electro-magnetic energy is dual, photons, light, colours.
Gravitation is equivalent or dual (isomorphic) to acceleration -- Einstein's happiest thought or the principle of equivalence, duality.
Potential energy is dual to kinetic energy -- gravitational energy is dual.
Energy is duality, duality is energy -- all energy is dual hence colours are dual.
Your mind is using duality to create colours.
Concepts are dual to percepts -- the mind duality of Immanuel Kant.
Mathematicians create new concepts all the time from their perceptions, observations or measurements.
Conceptualization or creating new concepts is a syntropic process -- teleological.
Thinking is a syntropic process.
Syntropy (prediction) is dual to increasing entropy -- the 4th law of thermodynamics!
The word dual is the correct word to use here.
Sine is dual to cosine or mutual sine -- the word co means mutual and implies duality.
Mutual requires at least two perspectives.
Causality is dual to retro-causality.
Everything in physics is made from energy or duality and this means that your mind is using effects to create causes (concepts) -- a syntropic process, teleological.
Welcome to the 4th law of thermodynamics!

what's the SIREN paper

@@ibraheemamin512 Implicit Neural Representations with Periodic Activation Functions

I guess if you combine these two methods you will get a much more accurate approximation of the mandelbrot set

functions might not be what things are, but functions describe everything

​@@beagle989
This is simply untrue, functions can't describe themselves nor the logical frameworks they are inserted, nor the logical inferential and mathematical rules that makes them possible in first place

@@beagle989
Suppose I give you a small number (epsilon), for which you can give me a function, such that it's maximum that far away from reality, never worse (never bigger) than epsilon.
Could there be an epsilon, for which it's impossible to find such a function?

The time domain is dual to the frequency domain -- Fourier analysis.
Neural networks are using syntropy to recognise patterns -- iterative optimization is a syntropic process!
Functions have goals, targets & objectives hence they are teleological, input is dual to output.
Making predictions to track targets, goals & objectives is a syntropic process -- teleological.
Sine is dual to cosine -- the word co means mutual and implies duality.
Teleological physics (syntropy) is dual to non teleological physics (entropy).
Syntropy (prediction) is dual to increasing entropy -- the 4th law of thermodynamics!
"Always two there are" -- Yoda.
Subgroups are dual to subfields -- the Galois correspondence.
Duality creates (emergence, synthesis) reality!

The time domain is dual to the frequency domain -- Fourier analysis.
Neural networks are using syntropy to recognise patterns -- iterative optimization is a syntropic process!
Functions have goals, targets & objectives hence they are teleological, input is dual to output.
Making predictions to track targets, goals & objectives is a syntropic process -- teleological.
Sine is dual to cosine -- the word co means mutual and implies duality.
Teleological physics (syntropy) is dual to non teleological physics (entropy).
Syntropy (prediction) is dual to increasing entropy -- the 4th law of thermodynamics!
"Always two there are" -- Yoda.
Subgroups are dual to subfields -- the Galois correspondence.
Duality creates (emergence, synthesis) reality!