Wyaaaattwho’s Blog

入门强化学习 Easy Reinforcement Learning

Fri, 22 Nov 2024 00:00:00 GMT

🔥

强化学习（Reinforcement Learning, RL）是一种模仿人类学习方式的机器学习方法，它通过“试错”的交互过程，从环境中获得反馈信号（奖励或惩罚），逐步优化决策策略。无论是围棋中击败人类顶尖棋手的 AlphaGo，还是帮助机器人实现灵活运动的智能控制系统，强化学习都展示出了极大的潜力和应用价值。

本博客旨在为强化学习初学者提供一条清晰、易懂的入门路径，持续更新中。

1️⃣

Imitation: Supervised Learning of Behaviors

2️⃣

Reintroduction to Reinforcement Learning

3️⃣

Policy Gradients

4️⃣

Actor-Critic Algorithms

5️⃣

Value Function Methods

6️⃣

Deep RL with Q-Functions

7️⃣

Advanced Policy Gradient

ControlNet Breakdown

Mon, 17 Mar 2025 00:00:00 GMT

Modules UNet ControlNet DDIMSampler Models DDPM LDM ControledLDM

Modules

UNet

1️⃣ UNetModel 代码结构解析（in ldm/modules/openaimodel.py）

UNetModel 是一个标准的 U-Net 结构，包含：

编码器（input_blocks）

瓶颈层（middle_block）

解码器（output_blocks）

最终输出层（self.out）

📌 主要流程

1. input_blocks（编码器）：

通过多个 ResBlock（残差块）和 Downsample 逐层提取特征

attention_resolutions 控制是否加入注意力机制

特征 h 被存入 hs，用于后续跳跃连接

2. middle_block（瓶颈层）：

通过 ResBlock + AttentionBlock 进一步处理特征

3. output_blocks（解码器）：

通过 Upsample 层恢复分辨率

核心：与 hs 进行跳跃连接（skip connection）

如果有注意力机制，加入 AttentionBlock

4. self.out（最终输出层）

归一化 + SiLU 激活

通过 conv_nd 生成最终输出

2️⃣ ControlledUnetModel 代码解析

ControlledUnetModel 继承 UNetModel，并 增加了 control 变量，用于外部引导 U-Net 处理特定任务。

📌 主要改进点

1. 编码器部分（input_blocks）

加速优化：使用 torch.no_grad()，避免计算梯度，提高推理速度

hs.append(h)：存储编码器输出，用于跳跃连接

2. 中间层（middle_block）

control 的第一次作用：如果提供了 control，则 在 middle_block 之后 进行相加 h += control.pop()

这里 control.pop() 代表 取出控制变量的最后一层

3. 解码器部分（output_blocks）

加入 control 的方式

如果 only_mid_control=True，则 control 仅在 middle_block 作用，后续解码过程正常否则，在解码过程中不断加入 control.pop()

h = torch.cat([h, hs.pop() + control.pop()], dim=1)

h 是当前特征图
hs.pop() 是来自编码器的跳跃连接
control.pop() 是控制信号（用于影响解码过程）

ControlNet

ControlNet 解析

ControlNet 是一种改进的 U-Net 结构，它通过引入额外的“控制”信息（如 hint 输入）来引导神经网络的生成过程，使其能够更加精准地遵循某些约束或先验信息。在扩散模型（Diffusion Models）或图像生成任务中，ControlNet 可以用于特定任务，如边缘检测、深度信息控制、姿态引导等。

🔹 ControlNet 相较于标准 U-Net 的主要改进

1️⃣ 额外的 hint 信息输入

传统的 U-Net 主要依赖输入 x 进行图像生成，而 ControlNet 额外引入 hint 作为辅助信息，提供某种先验引导。

hint 通过 input_hint_block 进行处理，这个模块是一个深度卷积网络（CNN），它将 hint 逐步下采样并映射到 model_channels 维度，使其与主 U-Net 结构兼容。

2️⃣ zero_convs 额外控制分支

zero_convs 是一系列 零初始化卷积层，用于让 ControlNet 直接在各个层级学习额外的偏差信息。

这意味着 ControlNet 不是直接干预主 U-Net 的计算，而是以 残差（Residual） 方式调整输出，这样可以在保持预训练 U-Net 结构的同时，让 ControlNet 提供新的信息。

3️⃣ 编码器 input_blocks 中引入 guided_hint

在前向传播过程中，ControlNet 逐层融合 hint 信息：

1. hint 经过 input_hint_block 处理后，变成 guided_hint

2. 在编码阶段的 第一层 input_blocks，guided_hint 直接加到 h 上，使得 ControlNet 受 hint 影响

3. 后续层不会重复加 guided_hint

🔹 代码解析

1️⃣ __init__ 构造函数

💡 新增的 hint 处理模块

这个 input_hint_block 负责 处理 hint 数据，它是一个深度卷积网络：

从 hint_channels 开始，逐步增加通道数，并进行三次 下采样 (stride=2)，最终与 U-Net 主网络的 model_channels 维度匹配。

这样可以确保 hint 信息可以直接用于调整 U-Net 的隐藏层特征。

2️⃣ forward() 计算流程

这个 forward() 接受 4 个主要输入：

1. x - 原始图像

2. hint - 额外的控制信号

3. timesteps - 扩散模型的时间步

4. context - 额外的上下文信息（如文本）

💡 计算时间嵌入

时间步 timesteps 先经过 timestep_embedding() 变换，再经过 self.time_embed 提取时间特征。

💡 处理 hint 额外信息

hint 经过 input_hint_block 处理，生成 guided_hint，它将在 ControlNet 结构中 融合到 U-Net 的编码过程中。

💡 编码时融合 guided_hint

逐层遍历 input_blocks 进行编码：

在 第一层（guided_hint 仍然存在）

计算 h = module(h, emb, context)（标准 U-Net 计算）
直接 h += guided_hint，融合额外控制信息
然后 guided_hint = None，确保后续层不会继续加 hint

其他层正常执行 h = module(h, emb, context)。

3️⃣ middle_block 处理

在 瓶颈层 middle_block 继续处理，并存入 outs 作为最终控制信号。

🔹 ControlNet 主要作用

与 UNet 主要区别：

ControlNet 增加了 hint 作为额外控制信号，用于引导扩散过程。
额外的 zero_convs 让 ControlNet 以残差方式调整特征，不会破坏 U-Net 原始权重。
只在编码器 input_blocks 第一层添加 guided_hint，避免后续层过度干预。

🔹 ControlNet VS ControlledUnetModel

模型	核心机制	是否使用 hint	影响位置	作用
UNetModel	经典 U-Net	❌	仅依赖输入 x	ㅤ
ControlledUnetModel	允许 control 影响解码	✅ (可选)	仅影响 output_blocks	受控扩散模型，风格迁移
ControlNet	hint 直接影响编码器	✅ (必须)	仅影响 input_blocks 第一层	条件生成（草图，深度图）

DDIMSampler

DDIMSampler 主要用于 扩散模型（Diffusion Model） 的高效采样，采用 DDIM (Denoising Diffusion Implicit Models) 进行降噪，相比于传统 DDPM (Denoising Diffusion Probabilistic Model) 具有更快的采样速度，同时可以调整 eta 来控制生成样本的多样性。

🔹 DDIMSampler 主要结构

1️⃣ 初始化

绑定 扩散模型 model，并读取 总时间步长 ddpm_num_timesteps。

schedule="linear" 代表时间步（timestep）的调度方式。

2️⃣ 预计算采样公式

该函数预计算 DDIM 采样所需的时间步、方差、累计 α (alphas_cumprod) 等参数。

ddim_eta=0.0 对应 确定性采样（即 DDIM 采样）。

ddim_eta>0.0 时会引入 随机性（向 DDPM 采样靠近）。

关键参数

计算 累计噪声方差 sqrt_one_minus_alphas_cumprod，用于后续 从噪声恢复图像。

3️⃣ 主要采样流程

🟢 sample() 入口

S: 采样步数（通常 S << self.ddpm_num_timesteps，即少量步数生成高质量图像）。

conditioning: 控制信号（如文本、深度图等）。

x_T: 初始噪声（如果 None，默认 torch.randn(shape)）。

unconditional_guidance_scale: 文本控制权重（如 CLIP 指导）。

🟢 ddim_sampling() 采样核心

x_T=None 时，使用随机噪声 torch.randn(shape, device=device) 作为初始图像。

时间步 timesteps 递减（由 T 逐步减少到 0）。

self.p_sample_ddim() 进行逐步去噪。

4️⃣ 单步降噪 p_sample_ddim()

计算预测噪声 ：

计算去噪后的 x_0：

计算下一步 x_prev：

其中：

dir_xt = (1. - a_prev - sigma_t**2).sqrt() * e_t 计算噪声方向。

noise = sigma_t * noise_like(...) 加权随机噪声（当 eta=0 时无噪声）。

eta 影响采样方程

在 DDIM 采样公式 中：

其中：

表示噪声尺度，由 eta 控制：

当 eta=0.0 时，，完全确定性采样。

当 eta>0.0 时，，加入随机噪声，增加多样性。

🔹 DDIMSampler 的作用

1. 更快的采样

S << T，可使用 10~50 步采样，而 DDPM 需要 1000+ 步。

2. 可调控的生成

eta=0.0：完全确定性采样（一致的结果）。

eta>0.0：增加随机性（提升多样性）。

3. 支持文本引导（Guidance）

unconditional_guidance_scale 控制文本影响程度。

🔹 结论

✅ DDIM 采样器是 扩散模型中高效的采样方法，可以大幅提升推理速度，并允许调整多样性，使得 扩散模型生成更加灵活高效。

Models

ControlNet里面模型的继承思路是DDPM→ LDM→其他。

DDPM

DDPM (Denoising Diffusion Probabilistic Models) 解析

DDPM（去噪扩散概率模型），它是一种基于扩散过程的生成模型。DDPM 通过向数据添加噪声进行训练，然后通过学习逆过程去噪，从随机噪声生成清晰的图像。

代码核心结构

1️⃣ 初始化 (__init__)

作用：初始化 DDPM 关键参数，并定义模型结构。

unet_config：U-Net 结构的配置文件，扩散模型通常基于 U-Net 进行训练。

timesteps=1000：扩散过程的时间步数（T），表示扩散链的长度。

beta_schedule="linear"：用于控制扩散过程的噪声增加方式，常见的有线性 (linear) 或余弦 (cosine) 。

loss_type="l2"：损失函数选择，默认为 L2损失（均方误差 MSE）。

parameterization="eps"：指示模型的预测目标：

"eps"：预测加噪的噪声 ε（最常用）。

"x0"：直接预测无噪声图像 x₀ 。

"v"：预测 x₀ 和 ε 的加权组合。

2️⃣ 噪声调度 (register_schedule)

作用：定义噪声的扩散调度，即 β 和 α 的计算方式。

关键变量：

betas：表示每个时间步添加的噪声量（βₜ）。

alphas = 1 - betas：表示保持原始数据的部分。

alphas_cumprod = np.cumprod(alphas, axis=0)：累乘所有 αₜ，用于计算 q(x_t | x_0) 的均值。

这些变量决定了扩散过程的噪声模式，使模型能够从噪声数据中恢复出原始数据。

3️⃣ 前向传播 (forward)

作用：计算训练过程中的损失。

t = torch.randint(0, self.num_timesteps, (x.shape[0],))

这行代码随机选择 t，表示在扩散链中某个时间步对图像 x 进行加噪。

self.p_losses(x, t, *args, **kwargs)：

计算训练损失，核心方法如下：

其中：

x_noisy = self.q_sample(x_start, t, noise)：生成加噪后的数据 x_t。

model_out = self.model(x_noisy, t)：让 U-Net 预测噪声（或 x0）。

计算损失

如果 parameterization == "eps"，那么 model_out 代表的是噪声 ε，损失函数就是：

计算 L2 Loss：

4️⃣ 前向扩散过程 (q_sample)

作用：在训练过程中，把真实数据 x_0 逐步加噪，生成 x_t。

x_t = sqrt(alpha_cumprod) * x_0 + sqrt(1 - alpha_cumprod) * noise

这个公式来自于 前向扩散公式：

sqrt(alphas_cumprod) 控制 x_0 贡献的部分

sqrt(1 - alphas_cumprod) 控制噪声的比例。

5️⃣ 反向去噪 (p_sample)

作用：在推理阶段，通过去噪逐步恢复数据。

model_mean：当前 x_t 估计出的均值 μ_t。

model_log_variance：估计出的方差 Σ_t。

noise：采样的高斯噪声。

计算：

其中：

μ_t = model_mean

σ_t = exp(0.5 * model_log_variance)

ε = noise

6️⃣ 图像生成 (p_sample_loop)

作用：从纯噪声开始，通过多步去噪生成样本。

输入：随机噪声 img = torch.randn(shape, device=device)。

去噪：逐步从 x_T 逆扩散到 x_0。

返回：最终去噪后的 img。

总结

📌 DDPM 的核心流程

1. 训练阶段

从 x_0 开始，逐步加噪 x_t = sqrt(alphas_cumprod) * x_0 + sqrt(1 - alphas_cumprod) * noise。

让 U-Net 学习，即预测 x_0 或 ε。

2. 推理阶段

从 x_T（纯噪声）开始，逆扩散回 x_0。

逐步去噪 x_t = model_mean + sigma * noise，最终生成清晰图像。

LDM

LatentDiffusion 继承自 DDPM（Denoising Diffusion Probabilistic Models），是 稳定扩散模型（Stable Diffusion） 及类似框架的核心。它结合了扩散模型与 VAE（变分自编码器），在潜空间（latent space）中进行扩散，从而降低计算成本并提高生成质量。

📌 1. 核心思想

与标准 DDPM 直接在像素空间进行扩散不同，LatentDiffusion：

1. 首先使用 VAE（Autoencoder）将图像编码到潜变量 z，然后在潜变量上进行扩散。

2. 扩散过程 仍然遵循 DDPM 逻辑，但在 更低维的潜空间 中进行，减少计算复杂度。

3. 反扩散后，VAE 将 z 解码回图像，生成最终结果。

这个和Dreamer是一个思路

📌 2. 代码结构

① 初始化 (__init__)

主要参数：

first_stage_config：定义 VAE 编码器/解码器 的参数。

cond_stage_config：定义 条件模型（文本/图像等） 的参数。

num_timesteps_cond：定义 条件调度 的时间步数。

scale_factor：用于缩放潜变量 z，控制图像的动态范围。

初始化过程中：

1. 加载 DDPM 的扩散调度。

2. 创建 VAE (first_stage_model) 和条件模型 (cond_stage_model)。

3. 设置 conditioning_key，确定 U-Net 采用 拼接 (concat) 还是交叉注意力 (cross-attention) 作为条件方式。

② 训练前预处理

作用：

在 第一个训练 batch 之前，如果 scale_by_std=True，则：

计算 z 的标准差，并 动态缩放 scale_factor 以标准化分布。

③ VAE 编码 & 解码

作用：

encode_first_stage(x)：将 x 通过 VAE 编码 得到 z。

decode_first_stage(z)：将 z 解码回图像，最终生成。

④ 训练输入 (get_input)

作用：

1. 获取 x 并编码为 z。

2. 获取条件信息 c：

若 conditioning_key = "crossattn"，则使用文本编码 c_crossattn。

若 conditioning_key = "concat"，则使用额外的图像 c_concat 作为拼接条件。

⑤ 计算损失 (p_losses)

扩散过程：

1. 从 x_start 生成 x_noisy，模拟 t 时刻的噪声图像。

2. U-Net 预测 ε 或 x0，从 x_noisy 还原干净图像。

3. 计算损失：

如果 parameterization = "eps"，则 预测噪声 ε，损失函数为 MSE(ε_pred, ε)。

如果 parameterization = "x0"，则 直接预测 x0，损失为 MSE(x0_pred, x0)。

⑥ 采样 (p_sample_loop & sample)

作用：

通过 p_sample_loop() 从纯噪声 x_T 逆扩散回 x_0，最终生成图像。

⑦ 采样过程 (p_sample)

核心方程：

其中：

μ_t 由 model_mean 计算。

σ_t 由 model_log_variance 计算。

ε 为高斯噪声。

⑧ 额外特性

💡 1. 无条件采样 (get_unconditional_conditioning)

用于 classifier-free guidance（无分类标签引导）。

生成 空白条件 以提高采样质量。

💡 2. 进阶采样 (progressive_denoising)

逐步去噪并 记录中间状态，可用于可视化整个扩散过程。

📌 3. 总结

💡 LatentDiffusion（LDM）结合了 扩散模型（DDPM）+ VAE（变分自编码器）+ 条件输入（文本/图像）：

通过 VAE 降维：在潜变量 z 上运行扩散，提高效率。

U-Net 预测 ε 或 x0：指导采样过程。

支持文本 & 图像条件：

c_crossattn（文本条件）
c_concat（额外图像条件）

最终生成高质量图像。

✅ LDM 是 Stable Diffusion 的核心框架！ 🚀

比 DDPM 更高效（在潜变量上运行）。

支持文本-图像扩散（如 Stable Diffusion）。

采样方式丰富（DDIM、progressive_denoising）。

ControledLDM

ControlLDM 是一个基于 Latent Diffusion Model (LDM) 的 受控扩散模型，它在标准 LDM 基础上加入了 ControlNet 控制机制，用于在扩散过程中施加额外的引导（control hints），以便更好地控制生成图像的结构和内容。

🔹 ControlLDM 的主要改进

1. 加入 ControlNet

额外增加 control_model，用于处理控制信息（例如边缘检测、深度图等）。

通过 control_key 指定输入数据中的控制信息字段。

only_mid_control 控制是否仅在中间层施加控制。

2. 自定义输入处理

在 get_input 方法中，额外提取 control hints（如边缘检测、深度信息）。

这些控制数据被转换为 通道优先格式（b c h w），并送入 ControlNet。

3. 修改 apply_model 方法

apply_model 负责模型前向传播。

ControlNet 生成 控制向量 并施加到 diffusion_model 之中。

计算方式：

这样，ControlNet 影响 LDM 的采样过程，使得生成结果更符合控制信息。

4. 日志记录 log_images

记录重建图像、控制条件、去噪过程等信息，以便可视化 LDM 的行为。

计算 unconditional guidance，用于 CFG 采样（Classifier-Free Guidance）。

5. 优化器 configure_optimizers

只优化 control_model，保持 LDM 其他部分不变（sd_locked 控制是否锁定 Stable Diffusion 组件）。

使用 AdamW 进行优化。

6. 低显存模式 low_vram_shift

在推理时释放部分显存，将不必要的组件移到 CPU，提高大模型的可用性。

🔹 代码解析

组件	作用
control_model	额外的 ControlNet，提供额外的控制信号
get_input	解析输入，并提取 control 作为额外的条件
apply_model	结合 ControlNet 的输出，影响扩散模型
log_images	记录生成过程中的各个阶段，便于调试
configure_optimizers	只优化 ControlNet，不影响 LDM 主体
low_vram_shift	在推理时减少显存占用，提高效率

数据结构与算法

Fri, 27 Dec 2024 00:00:00 GMT

绪论

线性表

栈和队列

图

排序

Convex optimization

Fri, 22 Nov 2024 00:00:00 GMT

Convex optimization problems

5️⃣

Duality

简明机器学习 Easy Machine Learning

Fri, 02 Aug 2024 00:00:00 GMT

Easy Machine Learning guide！

This is a simplified machine learning guidebook written entirely by myself. The initial idea was to find a path to learn machine learning independently, as our school doesn’t offer any useful courses on the subject. Later, I realized I could write down what I’d learned as a blog for my friends and others to read, which also serves as a way to review the material myself.

The blog is primarily based on lectures from Stanford’s CS229 and Machine Learning by Zhi-Hua Zhou. I removed some outdated or irrelevant content and added insights from blogs I’ve read. I typed this all in Notion myself, so there might be some typo and so on, please let me know if you get one.

这里是简明机器学习系列博客。鉴于现在国内大多数本科课程大多无法应对大家就业和实际使用的需求，直接看《机器学习》或者《统计学习方法》又难免比较难以入门，特参考CS229和周，李两位老师的书写出了这个博客，希望你能喜欢。对于《机器学习》一书中部分现在看来过于冷门的算法，我几乎全部砍掉了，感觉实在用处不大。

Mathematic Preparations Supervised Learning Deep Learning Regulations and Generalization Unsupervised Learning Reinforcement Learning

Mathematic Preparations

✊

Mathematic preparations

Supervised Learning

🫥

Linear regression

📔

Classification and logistic regression

🧞

Generalized linear models

🌲

Decision trees and Ensemble learning

🧬

Generative learning algorithm

🍩

Kernel method

🎰

Support vector machines

Deep Learning

🧠

Deep learning

Regulations and Generalization

🎟️

Bias-Variance Tradeoff and so more

📉

The double descent phenomenon and so more

®️

Regularization and model selection

Unsupervised Learning

🧩

Clustering algorithms

Reinforcement Learning

🦾

Reinforcement learning

🧀

LQR, DDP and LQG

📐

Policy Gradient (REINFORCE)

🧑‍💻

Practice code

ISP技术入门 AI+ISP

Fri, 02 Aug 2024 00:00:00 GMT

Dr. Michael S. Brown, Professor, EECS, York University, Canada

Michael S. Brown, Professor, School of Computing (SoC), National University of Singapore (NUS), York University, Canada, York, Toronto

http://www.cse.yorku.ca/~mbrown/

Color tutorials Color and color spaces Spectral power distribution (SPD)Tristimulus color theory Radiometry vs. photometry/colorimetry Luminance-chromaticity space (CIE xyY)Color constancy and color temperature Color constancy Color temperature Color model versus color space In-camera rending pipeline Camera Image signal processor (ISP)Camera sensor ISO gain and raw-image processing ISO signal amplification (gain)Black light subtraction Defective pixel mask Flat-field correction RGB demoasicing CFA/Bayer pattern demosaicing Noise reduction (NR)White-balance & color space transform(CIE XYZ)Color mapping/colorimetric stage AWB Color space transform Color manipulation (Photo-finishing)Image rescaling (or up-scaling)Mapping to output color space JPEG/HEIC compression Save to file Circular buffer for ZSL(Zero Shutter Lag) and multi-frame processing

Color tutorials

Color and color spaces

The most common color space is CIE XYZ, but before we get into that, we first need a more basic view of how we perceive the colors.

Spectral power distribution (SPD)

Every color (visible spectrum) has a special combined wavelength, this can be described by a spectral power distribution (SPD) shown below. The SPD plot shows the relative amount of each wavelength reflected over the visible spectrum.

Due to the accumulation effect of the cones, two different SPDs can be perceived as the same color (such SPDs are called “metamers”).

Tristimulus color theory

People found that all color could be expressed as a linear combination of three primary color. Grassman’s Law states that a source color can be matched by a linear combination of three independent “primaries”.

Radiometry vs. photometry/colorimetry

Radiometry

Quantitative measurements of radiant energy.

Often shown as spectral power distributions (SPD).

Measures light coming from a source (radiance) or light falling on a surface (irradiance).

Photometry/ colorimetry (a psychophysical)

Quantitative measurement of perceived radiant energy based on human’s sensitivity tolight.

Perceived in terms of “brightness” (photometry) and color (colorimetry).

Now we face the problem to bridging up the gap between the radiometry and photometry/colorimetry. How can we do that? Well in the flicker experiments, we are finally able to measure how we perceive different color of different SPD byb their brightness.

We invert the result, and we shall get what is called CIE (1924) Photopic luminosity function

The Luminosity Function (written as y(λ) or V(λ)) shows the eye’s sensitivity to radiant energy into luminous energy (or perceived radiant energy) based on human experiments (flicker fusion test).

And finally, we can define a quantity to measure the perceived brightness of different light. That is

And now let’s figure how to go from radiometry to colorimetry. Based on tristimulus color theory, colorimetry attempts to quantify all visible colors in terms of a standard set of primaries. So by hiring “standard observers”, we can easily carry out experiments and thus set up a measurement for our purpose.

Noticeable that there are negative values in, we can apply an linear transformation to avoid this.

In 1931, the CIE met and approved defining a new canonical basis, termed XYZ that would be derived from Wright-Guild’s CIE RGB data.

Properties desired in this conversion:

Positive values only
Pure white light (flat SPD) to lie at X=1/3, Y=1/3, Z=1/3
Y would be the luminosity function (V(λ))

Quite a bit of freedom in selecting the XYZ basis

In the end, the adopted transform was:

To transfer from SPD to CIE 1931 XYZ we have to carry out three integrates respectively

CIE 1931 XYZ is one of the most common color space we gonna mention later.

Luminance-chromaticity space (CIE xyY)

Sometimes it is useful to discuss color in terms of luminance (perceived brightness) and chromaticity (we can think of as the hue-saturation combined). CIE xyY space is used for this purpose.

A few things you need to know about this image:

Color constancy and color temperature

Color constancy

Our visual system has an amazing ability to compensate for environmental illumination such that objects are perceived as the same color.

Color constancy (chromatic adaptation) is the ability of the human visual system to adapt to scene illumination. This ability is not perfect, but it works fairly well. Image sensors do not have this ability! We will discuss this in part 2 . . this is related to the camera’s white-balance module.

We can use The Von Kries transform to discribe this kind of adaptation

And it’s noticeable that people always adapts printed media rather than emissive media.

All we have discussed about indicates that

Color is intimately connected to scene illumination.

Even for emissive displays, we have to consider (or make assumptions) about the illumination in the viewing environment of the display.

Keep this in mind because it will play a role when we define color spaces used to encode our images.

Color temperature

To better describe color of different illuminant, we introduce color temperature. In the photography and display communities, an illumination’s “color” is described using a correlated color temperature (CCT). This is an excellent example of where metamers are used.

As mentioned, illuminants are often described by their “color temperature.” This mapping is based on theoretical blackbody radiators that produce SPDs for a given temperature expressed in Kelvin (K). We map light sources (both real and synthetic) to their closest color temperature.

We can plot visible SPDs in CIE xy chromaticity

How to check the CCT of a given illuminant is simple, just do as the following steps:

Find the light sources SPD mapping to CIE XYZ using the CIE 1931 mapping functions.

Project the CIE xyY value to the Planckian locus line.

An example of an OLED light is here

Where the projection falls is the Correlated Color Temperature (CCT) of this light source. So, in this example, the OLED light source is roughly 4500K. While we often say "color temperature", we should say "correlated color temperature.” The concept is not always related to the physical temperature of the light source, but its correlation with the black body radiator's color temperature.

After we defined the CCT, we can easily define the White point. A white point is a color defined in CIE xyY that we want to be considered “white” (or achromatic/neutral). This is essentially an illuminant’s SPD in terms of CIE XYZ/CIE xyY. Think of it as CIE Yxy value of a white piece of paper under some illumination.

🚨

Summary

Color constancy is our ability to adapt to illumination in the scene.

Correlated Color Temperature (CCT) — or just color temperature — is a system used to describe scene illumination.

Note: we must factor in the scene illumination when capturing and displaying color images.

Color model versus color space

A color model is a mathematical system for describing a color as a tuple of numbers (RGB, HSV, HSL, more. . . )

A color space is a specific range of colors within a color model. The range of color (gamut) can be expressed in CIE XYZ. Color spaces typically also define the viewing environment and, therefore, the “white point” of the space

We must understood that no specific color model can fully cover up the CIE XYZ, but just part of it, and we must realize that different definition of white point define a different color model, which implies the existence of a vast range different color models.

In 1996, Microsoft and HP defined a set of “standard” RGB primaries. R=CIE xyY (0.64, 0.33, 0.2126) G=CIE xyY (0.30, 0.60, 0.7153) B=CIE xyY (0.15, 0.06, 0.0721). This was considered an RGB space achievable by most devices at the time. The white point was set to the D65 illuminant. This is an important to note. It means sRGB has built in the assumed viewing condition (6500K daylight).

A matrix decides the transformation from CIE XYZ to the sRGB, which is

Before introducing someting new, let’s grab a look at Stevens' power law

That tells us, if we want our color seemed more natural, we need it to be in-linear, that’s why we have our gamma curve

The actual formula is a bit complicated, but effectively this is gamma () where I’ is the output intensity and I is the linear sRGB ranged 0-1, with a small linear transfer for linearized sRGB values close to 0 (not shown in this plot). This is known as “perceptual encoding” and is intended to allocate more bits based on our nonlinear response to radiant power.

A huge difference between linear and in-linear sRGB

Generally speaking, the relation is like a clan of color spaces

At last we introduce a little more common color spaces.

CIE Lab

CIE Lab transforms CIE to a new space where color (and brightness) differences are more uniform.

And in CIE Lab it’s safe to define Color error metric – CIE 2000 Delta E (ΔE). Delta E is a color metric based on Lab space. Since Lab is more uniformly perceptual, distances (e.g., Euclidean distance) in L*ab have more meaning than in CIE XYZ. Delta E values have an interpretation as follows.

Others to be aware of

Adobe RGB

Medium gamut color space
Used for photo-editing

Display P3

Medium gamut color space
Used by Apple devices to accommodate better display technology
Similar to Adobe RGB

ProPhoto (ROMM)

Developed by Kodak
Intended to encode a wide range of colors and dynamic range

In-camera rending pipeline

Camera

The image directly captured from the camera’s sensor needs to be processed. We can call this process “rendering,” as the goal is to render a digital image suitable for viewing.

Image signal processor (ISP)

An ISP is dedicated hardware that renders the sensor image to produce the final output.

Companies such as Qualcomm, HiSilicon, Intel (and more) sell ISP chips (often as part of a System on a Chip – SoC). Companies can customize the ISP. Many ISPs now have neural processing units (NPUs).

Camera sensor

Almost all consumer camera sensors are based on complementary metaloxide-semiconductor (CMOS) technology. We generally describe sensors in terms of number of pixels and size. The larger the sensor, the better the noise performance as more light can fall on each pixel. Smart phones have small sensors!

Camera uses a certain pattern of sensors to catch photons.

The color filter array (CFA) on the camera filters the light into three sensor-specific RGB primaries.

Remember: physical world is measured by radiometric spectral power distributions. Your camera sensor RGB filter is sensitive to different regions of the incoming SPD. Raw-RGB represents the physical world's SPD "projected" onto the sensor's spectral filters.

Note that Sensors are linear to irradiance, if you double the amount of light hitting a sensor's pixel, the digital value output of that pixel will double.

Digital value I is a linear function of irradiate i and exposure t.

🚨

IMPORTANT: raw-RGB sensor images are not in a standard color space.

So if you try to display a raw-RGB pic it would be weird(like what you see before in your “pixeliser” )

ISO gain and raw-image processing

ISO signal amplification (gain)

Imaging sensor signal is amplified and digitized. Amplification to assist A/D conversion. You need to get the voltage to the range required for the desired digital output. This gain is used to accommodate camera ISO settings. Gain to signal applied on the sensor. Note that gaining the signal also gains image noise.

Black light subtraction

And let’s talk about the pixel intensity. We often talk about a pixel's intensity, however, a pixel's numerical value has no unit.

The digital value of a pixel is based on several factors.

Exposure (which is a function of both shutter speed and exposure)

Gain (ISO setting on the camera)

Camera hardware that digitizes the signal.

That’s why we use relative digital values in the image and not the absolute digital values.

Sensor values for pixels with “no light” should be zero. However, this is not the case due to sensor noise, the black level often changes as the sensor heats up. This can be corrected by capturing a set of pixels that do not see light. Place a dark shield around the sensor. Subtract the level from the “black” pixels.

Defective pixel mask

In CMOS (complementary metal-oxide-semiconductor) image sensors, defective pixels are a common issue. Here’s a breakdown of the process you mentioned for handling defective pixels:

1. Dead Pixel Masks: Dead pixel masks are used to identify and manage defective pixels in CMOS sensors. These masks are pre-calibrated at the factory to ensure that the sensor performs optimally when deployed.

2. Dark Current Calibration: Dark current refers to the small amount of current that flows through a pixel even when it is not exposed to light. To create a dead pixel mask, a “dark current” calibration is performed. This involves taking an image of the sensor with no light exposure, effectively capturing the dark current levels.

3. Recording Values: During the dark current calibration, the sensor’s output is recorded. Pixels that report values significantly different from their neighbors are identified as defective or bad.

4. Creating the Mask: The recorded values are used to create a mask that identifies which pixels are defective. This mask is essentially a map that indicates which pixels should be considered faulty.

5. Interpolating Bad Pixels: After creating the mask, any data from defective pixels are interpolated. This means that the values from these bad pixels are replaced with values calculated based on neighboring pixels. This interpolation helps to ensure that the image sensor’s output remains accurate and usable despite the presence of defective pixels.

Flat-field correction

RGB demoasicing

You’ve already know some part of this

CFA/Bayer pattern demosaicing

Color filter array (CFA) pattern placed over pixel sensors. We want an RGB value at each pixel, so we need to perform interpolation.

Most simply, we just simply interpolation with average values, yet it’s more intuitive that if we use a weight mask.

There are newer CFAs such as Quad/Tetra (2x2) and Nona (3x3), they aer common on smart phones.

Noise reduction (NR)

Most cameras apply additional NR after A/D conversion

Analog-to-Digital Converte（ADC）

Two main sources of image noise:

The quantum nature of light (photon noise/shot noise); unrelated to the imaging sensor. Follows a Poisson distribution.

Electronic sources associated with the imaging sensor circuitry (dark current and dark noise). Often follows a Normal distribution.

Gain factor g amplifies noise.

Blur the image based on the ISO setting (higher ISO = more blur). Blurring will reduce noise, but also remove detail. Add image detail back for regions that have a high signal. We can even boost some parts of the signal to enhance detail (i.e. "sharpening")

White-balance & color space transform(CIE XYZ)

Color mapping/colorimetric stage

This step in the IPS converts the sensor raw-RGB values to a device-independent color space. It’s a two step procedure, first apply a white-balance correction to the raw-RGB values, then map the white-balanced raw-RGB values to CIE XYZ.

How do white balance (computational color constancy) work? We have a special matrix

Users can manually set the white balance, otherwise auto white balance (AWB) is performed. In computer vision, we often refer to AWB as "illumination estimation". Since the hard part is trying to determine what the illumination in the scene is. AWB is not easy and this remains an open research problem.

WB manual

AWB

Given an arbitrary raw-RGB image, determine what is the camera's response to the illumination. The idea is that something that is white* is a natural reflector of the scene's illuminations SPD. So, if we can identify what is "white" in the raw-RGB image, we are observing the sensor's RGB response to the illumination.

First we introduce "Gray world" algorithm.

Also we have "White patch" algorithm.

🚨

you can actually think of two different pool NN in PyTorch

Gray world and white patch are very basic algorithms. These both tend to fail when the image is dominated by large regions of a single color (e.g. a sky image).

Color space transform

Color manipulation (Photo-finishing)

This is the stage where a camera applies its "secret sauce" to make the images look good.

This procedure is called by many names:

Color manipulation
Photo-finishing
Color rendering or selective color rendering
Yuv processing engine

DSLR will often allow the user to select various photo-finishing styles.

Smartphones often compute this per-image.

Photo-finishing may also tied to geographical regions!

Color manipulation can be implemented using a 3D look up table (LUT) and a 1D LUT tone-curve. The 3D LUT table acts like a 3D function: 𝑓 (𝑅, 𝐺, 𝐵) → 𝑅′, 𝐺′, 𝐵′ The 1D LUT table is applied per channel: g (𝑅) → 𝑅′, 𝑔 (𝐺) → 𝐺′, 𝑔 (𝐵) → 𝐵′ The 3D and 1D LUT can change based on picture style

There’s also Local tone mapping (LTM) compared to the Global tone mapping (GTM)

🚨

On many cameras, esp smartphones, a local tone map is applied as part of the photo-finishing. This helps bring out highlights in the image.

Selective color manipulation, "Select" colors can be manipulated, especially skin tone, sometimes called preferred color correction (PCC)

Image rescaling (or up-scaling)

Re-scaling image and sRGB conversion, often, the entire image is processed by the ISP and then rescaled for the view-finder or to fit the requested output resolution.

Mapping to output color space

JPEG/HEIC compression

The amount of quantization applied on the DCT coefficients amounts to a “quality” factor.

More quantization = better compression (smaller file size)

More quantization = lower quality

🚨

Note that both sRGB and JPEG are being slowly replaced

Save to file

Circular buffer for ZSL(Zero Shutter Lag) and multi-frame processing

Many ISPs store the last few RAW images in memory (i.e., a circular buffer).

These images can be used for many purposes (image stabilization, temporal noise reduction, etc.).

Another purpose is to ensure “Zero Shutter Lag” (ZSL).

There is often a delay between when a person presses the “capture” and the camera capturing the image. This is called “shutter lag”.
So, a previous image in the buffer can saved instead.

🚨

Again, it is important to stress that the exact steps mentioned in these notes only serve as a guide to what takes place in a camera