The promise of DeBERTa is that it does away with absolute positional embeddings added at the start of the transformer encoder, and replaces it instead with an attention mechanism that takes into account the relative distance between tokens when doing attention, in every head of every layer. Since absolute positional embeddings are the only part of a transformer whose dimensionality is related to the length of the input, would taking it out mean DeBERTa can process an infinite amount of tokens at once? Let’s analyse.

The famous roberta-base HuggingFace checkpoint is a serialised version of a RobertaForMaskedLM model, consisting of a roberta field and an lm_head field. Yet, despite this, you can still call .from_pretrained("roberta-base") on RobertaForTokenClassification and get an object that has a roberta field with exactly the checkpoint’s roberta weights, but a head with a different architecture and randomly initialised weights. Even more strikingly, you can call .from_pretrained("roberta-base") on RobertaModel, which is what normally sits inside that roberta field and consists of the fields embeddings and encoder, and somehow it can still match all the relevant checkpoint weights to the correct fields. Ever wondered how that’s done? Here’s how.

I recently watched a video online of someone complaining about the existence of millionaires and billionaires. I commented with the realistic and optimistic message that anyone has the ability to become a multi-millionaire, even if just through supplying a silly backend commodity like toilet rolls. Then, all hell broke loose.

Knowing how to build mobile apps is likely going to be a skill that will stay relevant for the foreseeable future. I’ve had some ideas for mobile apps in the past, and wanted to make sure that if I was going to learn to develop them, it had better be in a framework that allows having one code base and yet runs on both iOS and Android. We’re in luck: Google’s Flutter framework, built on top of the Dart language, does exactly that. Here are my notes learning it.

At the time of writing, I’ve publicly released a handful of Python packages (five, to be precise: Fiject, BPE-knockout, TkTkT, LaMoTO, and MoDeST) covering various parts of the NLP research workflow, with more on the way. It took me a bit to learn how to set up Python code as a package (also inaccurately called “library” or more accurately called a “module”), and as I later discovered, it’s not so trivial to have one custom package be installed automatically upon installing another custom package, especially when you are the author of both and are already using a working version. Let’s dive straight in!

I was recently reading a paper on how to train an adapter between a given model and any tokeniser, and noticed that they measured their causal language modelling performance in bits-per-character (BPC) rather than the more standard perplexity (PPL) with no citation to show for it. Let’s dive into that!

Let me present to you an allegory on misguided claims, and how we should deal with them.

After a harrowing round of peer review earlier this year (story for another day), I’m pleased to say that I am officially a published author. This article gives a short overview of my first paper, BPE-knockout: Pruning Pre-existing BPE Tokenisers with Backwards-compatible Morphological Semi-supervision. For a more detailed and slightly more technical overview, watch the 13-minute video explainer here.

When I was starting out in hobby photography in my early teens, I felt that I had to cling to all forms of automatic assistance to coordinate my shots for me: I would merely point the camera, and would let auto-focus and auto-exposure (the infamous P mode, or even worse, AUTO mode, with assistance of the on-camera flash) handle the rest. After all, computers are much better at multi-tasking than we mortals are, and that dreaded M mode seemed to have way too many settings with opposite effects. Better safe than sorry… right?

On the 21st of July, my wonderful home country of Belgium (autonymously België or Belgique) celebrates its National Day (or literally “the national holiday” as we call it). As with other nations, this is hypothetically a day to show off everyone’s patriotism. A day to show gratitude for living in one of the most peaceful, fair and prosperous places that the world has known throughout its history. A day to show our unity as a people and to stress why it makes sense for our country to exist as a sovereign country. Well, not here.