Neural networks were designed to mimic the architecture of the brain and serve as the underpinning of modern [[artificial intelligence]]. In reality, neural networks and the brain operate in ways that are significantly different. Neural networks are built from the [[perceptron]] with an input layer, some number of hidden layers (typically), and an output layer. If all neurons in adjacent layers are connected, it is called a **fully-connected neural network** and can be referred to as **dense**. The simplest implementation of a neural net is the [[feed forward neural net]]. Architecture hyperparameters include the number of hidden layers, number of nodes per layer, and activation function. Training hyperparameters include learning rate, momentum, optimization method, and regularization (among others). ## training neural networks The process of updating the parameter values (weights and biases) of a neural network is called training. [[Back propagation]] is used to update the weights for each layer. - Monitor overfitting as epoch grows - Hyperparameter tuning: learning rate, etc. - Architecture: number of layers, number of neurons, activation function, etc. - Optimization methods: RMSProp, Adam - Regularization: dropout and batch normalization, or add L1/L2 regularization on the loss. - Dropout If the learning rate is too high, it may skip right past the mimima and shoot off in another direction, or roll back and forth without settling. A learning rate that is too low will not only take a long time to find the minima, but may in fact get stuck in some local depression of the error surface. Dropout is used to randomly exclude some nodes in each step to find a more generalizable solution. Batch normalization is used to normalize inputs for each batch in [[stochastic gradient descent]]. ## history of the neural network Scientists first began to understand the architecture of the brain and its role in cognition in the mid 1800s. Santiago Ramon y Cajal is known as the father of neuroscience for his work in this era, which led to a branch of science known as "connectionism" which sought to explain mental phenomena using artificial neural networks, led by scientists like Alexander Bain.  *Santiago Ramon y Cajal's depiction of neurons in the cerebellum.*