Skip to content

Linear

Linear applies an affine transformation y = x @ W + b to its input. It is the fundamental building block of fully connected networks. The weight matrix and bias are both initialised from a uniform distribution over [-sqrt(1/in_features), +sqrt(1/in_features)]. Both are registered as learnable parameters and updated by any Optimizer.

import simplegrad as sg
import simplegrad.nn as nn

fc = nn.Linear(in_features=784, out_features=256)
x = sg.normal((32, 784))
out = fc(x)  # shape: (32, 256)
sg.sum(out).backward()

Linear

Bases: Module

Fully-connected linear layer: output = x @ W + b.

Weights are initialized with Kaiming uniform (range [-1/sqrt(in), 1/sqrt(in)]).

Parameters:

  • in_features (int | None, default: None ) –

    Number of input features.

  • out_features (int | None, default: None ) –

    Number of output features.

  • weight (Tensor | None, default: None ) –

    Optional pre-built weight tensor of shape (in_features, out_features).

  • bias (Tensor | None, default: None ) –

    Optional pre-built bias tensor of shape (out_features,).

  • use_bias (bool, default: True ) –

    Add a bias term. Defaults to True.

  • dtype (str, default: None ) –

    Data type string. Defaults to "float32".

  • weight_label (str, default: 'W' ) –

    Label for the weight tensor (used in graph visualization).

  • bias_label (str, default: 'b' ) –

    Label for the bias tensor.

Attributes

Attribute Type Description
.weight Tensor Weight matrix of shape (in_features, out_features). Learnable.
.bias Tensor Bias vector of shape (out_features,). Only present when use_bias=True.
.in_features int Number of input features.
.out_features int Number of output features.
.use_bias bool Whether a bias term is included.
.dtype str Data type of the weight and bias tensors.

Methods

Method Description
.forward() Compute x @ W + b.

Inherits all methods from Module: .parameters(), .submodules(), .to_device(), .summary(), .set_train_mode(), .set_eval_mode().