metrics¶

Easy evaluation metrics for PyTorch.

class torchutils.metrics.Accuracy(keep_hist=False, hist_size=0, hist_freq=1)¶

Calculate and track accuracy of predictions.

Parameters

keep_hist (bool) – Set as True to save accuracy history. (default:False)
hist_size (int) – Size of accuracy history buffer. (default:0 means infinite)
hist_freq (int) – Frequency of storing the history. (default:1 means store every iteration)

Example:

import torch
import torch.nn as nn
import torch.nn.functional as F
import torch.optim as optim
import torchvision
import torchvision.transforms as transforms
import torchutils as tu

# define your network
model = MyNet()
criterion = nn.CrossEntropyLoss()
optimizer = optim.Adam(model.parameters())
trainset = torchvision.datasets.MNIST(root='./data/', train=True,
                                    download=True,
                                    transform=transforms.ToTensor())
trainloader = torch.utils.data.DataLoader(trainset, batch_size=60,
                                        shuffle=True, num_workers=2,
                                        drop_last=True)
n_epochs = 1
model.train()
for epoch in range(n_epochs):
    print('Epoch: %d/%d' % (epoch + 1, n_epochs))
    acc_tracker = tu.Accuracy()
    for batch_idx, (data, target) in enumerate(trainloader):
        optimizer.zero_grad()
        outputs = model(data)
        loss = criterion(outputs, target)
        loss.backward()
        optimizer.step()
        _, predicted = torch.max(F.softmax(outputs, dim=1), 1)
        acc_tracker.update(target, predicted)
        if batch_idx % 100 == 0:
            print(acc_tracker)

Out:

Epoch: 1/1
Accuracy - Val: 10.0000 Avg: 10.0000
Accuracy - Val: 91.6667 Avg: 70.9406
Accuracy - Val: 86.6667 Avg: 79.5937
Accuracy - Val: 93.3333 Avg: 83.1063
Accuracy - Val: 90.0000 Avg: 85.4032
Accuracy - Val: 88.3333 Avg: 86.9627
Accuracy - Val: 95.0000 Avg: 88.1364
Accuracy - Val: 95.0000 Avg: 89.1702
Accuracy - Val: 93.3333 Avg: 89.9459
Accuracy - Val: 95.0000 Avg: 90.5161

property accuracy¶

Current running accuracy (percentage).

Type: float

property history¶

Accuracy values for past iterations.

Type: dict {“metric” -> list of accuracy values, “iteration” -> list of iteration numbers}

reset()¶

Reset accuracy tracker.

Returns: Returns nothing.
Return type: None

update(targets, predictions)¶

Update accuracy tracker.

Parameters

targets (torch.Tensor) – Targets. Must be (N, *).
predictions (torch.Tensor) – Model predictions. Must be (N, *).

Returns

Accuracy of current batch of predictions (percentage).

Return type

float

class torchutils.metrics.HammingLoss(keep_hist=False, hist_size=0, hist_freq=1)¶

Calculate and track hamming loss of predictions.

Hamming loss is an evaluation metric for multilabel classification problem. It is the fraction of labels that are incorrectly predicted.

Parameters

keep_hist (bool) – Set as True to save accuracy history. (default:False)
hist_size (int) – Size of accuracy history buffer. (default:0 means infinite)
hist_freq (int) – Frequency of storing the history. (default:1 means store every iteration)

Example:

import torch
import torch.nn as nn
import torch.optim as optim
import torchutils as tu

# define your network and trainloader
model = MyNet()
optimizer = optim.Adam(model.parameters())
criterion = nn.MultiLabelSoftMarginLoss()

n_epochs = 1
model.train()
for epoch in range(n_epochs):
    print('Epoch: %d/%d' % (epoch + 1, n_epochs))
    ham_tracker = tu.HammingLoss()
    for batch_idx, (data, target) in enumerate(trainloader):
        optimizer.zero_grad()
        output = model(data)
        loss = criterion(output, target)
        loss.backward()
        optimizer.step()
        predicted = torch.sigmoid(output) > 0.5
        ham_tracker.update(target, predicted)
        if batch_idx % 100 == 0:
            print(ham_tracker)

Out:

Epoch: 1/1
Hamming Loss - Val: 0.6667 Avg: 0.6667
Hamming Loss - Val: 1.0000 Avg: 0.8333
Hamming Loss - Val: 1.0000 Avg: 0.8889
Hamming Loss - Val: 0.0000 Avg: 0.6667
Hamming Loss - Val: 0.6667 Avg: 0.6667
Hamming Loss - Val: 0.6667 Avg: 0.6667
Hamming Loss - Val: 1.0000 Avg: 0.7143
Hamming Loss - Val: 0.6667 Avg: 0.7083
Hamming Loss - Val: 0.0000 Avg: 0.6296
Hamming Loss - Val: 0.0000 Avg: 0.5667

property history¶

Hamming loss values for past iterations.

Type: dict {“metric” -> list of hamming loss values, “iteration” -> list of iteration numbers}

property loss¶

Current running hamming loss.

Type: float

reset()¶

Reset hamming loss tracker.

Returns: Returns nothing.
Return type: None

update(targets, predictions)¶

Update hamming loss tracker.

Parameters

targets (bool torch.Tensor) – Targets. Must be (N, Classes).
predictions (bool torch.Tensor) – Model predictions. Must be (N, Classes).

Returns

Hamming loss of current batch of predictions.

Return type

float

class torchutils.metrics.RunningLoss(keep_hist=False, hist_size=0, hist_freq=1)¶

Track and maintain running average of loss.

Parameters

keep_hist (bool) – Set as True to save accuracy history. (default:False)
hist_size (int) – Size of accuracy history buffer. (default:0 means infinite)
hist_freq (int) – Frequency of storing the history. (default:1 means store every iteration)

Example:

import torch
import torch.nn as nn
import torch.optim as optim
import torchvision
import torchvision.transforms as transforms
import torchutils as tu

# define your network
model = MyNet()
criterion = nn.CrossEntropyLoss()
optimizer = optim.Adam(model.parameters())
trainset = torchvision.datasets.MNIST(root='./data/', train=True,
                                    download=True,
                                    transform=transforms.ToTensor())
trainloader = torch.utils.data.DataLoader(trainset, batch_size=60,
                                        shuffle=True, num_workers=2,
                                        drop_last=True)
n_epochs = 1
model.train()
for epoch in range(n_epochs):
    print('Epoch: %d/%d' % (epoch + 1, n_epochs))
    loss_tracker = tu.RunningLoss()
    for batch_idx, (data, target) in enumerate(trainloader):
        optimizer.zero_grad()
        outputs = model(data)
        loss = criterion(outputs, target)
        loss_tracker.update(loss.item())
        loss.backward()
        optimizer.step()
        if batch_idx % 100 == 0:
            print(loss_tracker)

Out:

Epoch: 1/1
Loss - Val: 2.2921 Avg: 2.2921
Loss - Val: 0.5084 Avg: 0.9639
Loss - Val: 0.6027 Avg: 0.6588
Loss - Val: 0.1817 Avg: 0.5255
Loss - Val: 0.1005 Avg: 0.4493
Loss - Val: 0.2982 Avg: 0.3984
Loss - Val: 0.3103 Avg: 0.3615
Loss - Val: 0.0940 Avg: 0.3296
Loss - Val: 0.0957 Avg: 0.3071
Loss - Val: 0.0229 Avg: 0.2875

property history¶

Loss values for past iterations.

Type: dict {“metric” -> list of loss values, “iteration” -> list of iteration numbers}

property loss¶

Current running (average) loss.

Type: float

reset()¶

Reset running loss tracker.

Returns: Returns nothing.
Return type: None

update(val)¶

Update running loss tracker.

Parameters: val (float) – Loss value.
Returns: Running (average) loss after latest update.
Return type: float