Neural networks

A micrograph of 256 × 256 pixels is 65,536 numbers. A dense layer that maps it to 32 features needs 2,097,184 parameters; a convolutional layer with 32 filters of 3 × 3 pixels needs 320, and it finds an edge wherever in the image the edge sits. Much of deep learning on scientific data is choosing such a structure, so that the network has few enough parameters to learn from the data you have. The problems people bring are of that kind: segmenting cells in microscope images, separating signal from background in a particle detector, learning the energy of a molecule from quantum-chemistry calculations, picking earthquake arrivals in seismograms, replacing a slow engineering simulation by a fast surrogate.

In Python the standard is PyTorch. A network is a subclass of torch.nn.Module built from layers such as nn.Linear and nn.Conv2d. Calling loss.backward() computes the gradient of the loss with respect to every parameter, and an optimizer from torch.optim, usually Adam, takes the step. You write the training loop yourself, which takes more lines than a scikit-learn fit and shows you what actually happens. In Julia, Flux.jl stacks layers with Chain, Dense, and Conv, takes gradients with Zygote, and trains with Flux.train! or a loop of your own.

Start with a small dense network fitted to a function of one variable, with the training loop written out. Then validation: hold back part of the data and stop training when the loss on it starts to rise. Convolutional networks for images come next, networks that build in a physical law last. On a table of a few hundred measured rows, try the random forest of machine learning first. Even a small network with two hidden layers of 64 units on three inputs has 4,481 parameters.

What belongs here

Learning with neural networks: layers, losses, and training by gradient descent, convolutional and recurrent networks, regularization and validation on small scientific data sets, and networks that respect physics, with PyTorch in Python and Flux.jl in Julia. Classical methods such as trees, forests, and linear models belong to machine-learning.