Blog

• June 11, 2026 MoE autoresearch

  My first attempt at autoresearch and scaling laws.

• June 5, 2026 How much autoregressive training does Diffusion Language Model need?

  If GPT-7 will be a diffusion LM, should they pretrain from scratch or from an AR checkpoint? I swept the whole range for a 1M dense transformer and a 70M MoE and consistently found the optimum near 30%.

• May 20, 2026 A New Chapter?

  Irresistible temptation to understand intelligence