Free Standard Shipping in the US on orders over $99

Layout Optimization Best Practices: The Corpus (Part 1)

Aug 19, 2024181 VIEWS

In this series we are designing our own custom keymaps, logical layouts, you name it. We’ve laid the groundwork by looking into how good/bad QWERTY is, the power of layers (SpaceFN), and also the huge potential of alternative layouts and custom keymaps. Today, we take the first step in designing your ultimate keymap by exploring our options for compiling a corpus. What's a corpus? Essentially, it's just a fancy term for a big chunk of text. In this context it means a usually large collection of textual data used directly or indirectly as an input for our layout optimization algorithms. Often literally a single text file. Why does it matter to you? Because a well-crafted, personalized corpus is crucial for keymap wizards. If you're aiming to design your own custom logical layout, the corpus plays a key role in determining the language statistics that reflect your typing habits, thus the outcome of the optimization. These statistics, which we extract through analysis of the corpus, are fundamental to achieving the best optimization results, so let's explore how to compile a tailor-fit corpus with this in mind.

TLDR; A good personal corpus is the best way to ensure you end up with a great layout during optimization. Sources for your personal corpus can include your diary, emails, posts and comments, professional writings. Surprisingly, you don't need a large amount of text for layout optimization, a few megabytes are more than enough.

Custom corpora for logical layout design There are different approaches to obtaining or compiling corpora. This can quickly become a deep rabbit hole in itself, some people seem downright obsessive, so let's focus on how to put together a corpus effectively, without spending weeks on the process. Basically, the goal here is to gather as many texts as you can/need into a single file (or database). The list of the most common sources may include:

general language statistics
downloading existing corpora
grabbing online articles, ebooks, etc.
keylogging
aggregating your own texts

I'm an advocate for personalized corpora based on texts you've personally typed out. One reason is that the overwhelming majority of people type in multiple languages, while available corpora typically focus on a single language. In addition, "average English" (or any other language) doesn't really exist on the personal level. Nonetheless, let's walk through the most common options: General language statistics Layout optimization algorithms simulate typing in various ways, but in general, they need the text of our corpus only for generating language statistics: letter, bigram and trigram frequencies, to calculate a score based on finger movement, hurdles, rolls, and all the various metrics. So basically you could use the language stats directly, skipping the corpus step.

(Letter frequencies of various languages differ a lot! Source: Wikipedia) However, it's highly unlikely that general letter frequencies found online would accurately represent your typing habits or provide enough data beyond basic statistics. It’s interesting and often useful to contemplate on the difference of letter frequencies between languages, but these numbers won’t really help you when compiling your corpus, so I recommend not wasting time on these, at least for now. Existing corpora Similarly, downloading massive corpora like the full Wikipedia or Project Gutenberg is not only overkill but also involves someone else's text -- often from hundreds of years ago and on topics you'll never think about, let alone type about. In conclusion, if I were in your shoes, I'd disregard these easily accessible but too general and thus irrelevant corpora. Grabbing articles, e-books, etc. A more targeted selection of online content, easy to crawl and grab automatically or even to harvest manually, might be a better idea. If you're an aspiring writer, ebooks in the genre and style you're pursuing could be useful, as they better represent the words and character frequencies you'll encounter in your work. The same goes for journalists and bloggers: compiling a corpus based on articles and blog posts similar to your main topics and writing style may be quicker and easier than creating a fully personalized corpus. However, the result will only be as good as the effort you put into it. And if you happen to have a portfolio of published articles or books already, use those instead. Keyloggers vs aggregating When compiling a personal corpus tailored to your own unique typing habits, you are left with two main approaches: using a simple and trustworthy keylogger or meticulously creating a text file from your own written content.

Yikes, keyloggers are dangerous and scary. Yep. But trustworthy, basic keyloggers, open-source options or ideally ones you code yourself, are an easy way to harvest keystrokes over time. Ensure that the keylogger does not transmit any data externally. Ideally, write your own basic keylogger or opt for one that counts keystrokes and bi/trigrams rather than saves the entire text to log files.

Pros: Registers every keypress (including hotkeys, modifiers, filling out forms, and yep, passwords too.)
Cons: Registers typing errors. Gaming input, typing practice (monkeytype), language learning (Duolingo) may completely mess up the statistics. Also security of course. And it may take some time to gather enough data this way.

Aggregating text manually or semi-manually is a cleaner, safer and quicker approach. However, this method may result in losing some useful data, such as modifiers, hotkeys, form data, passwords, navigation, etc. Pros and cons of aggregating text, i.e. copypasting your texts into a single file:

Pros: More controlled input, safer, almost instant result if the appropriate content is available.
Cons: Labor intensive. No backspace, del, modifiers, function keys, hotkeys, etc.

Summary After trying pretty much all of the options above, I've found that manually compiling a corpus from my own typewritten text works best for me. However, I'm aware that some people prefer using yearlong keylogged data. Both approaches can work as long as you’re aware of their advantages and disadvantages, and keep your corpus clean and sanitized. Either way, if I were in your shoes, I'd quickly start compiling my own personal corpus to aim for more optimal keymaps. Next time, in Part 2, we will look into how to do it exactly, and how long of a corpus you need for the best results. (Cover image by Michael D Beckwith.)

(Edited)

100% upvoted

Comment

181

dovenyi

Sort by: Newest

Let’s get the conversation started!

Be the first to comment.

PRODUCTS YOU MAY LIKE

Drop + MiTo GMK Laser Custom Keycap Set

$49

Drop + Matt3o MT3 Susuwatari Custom Keycap Set

$99

Drop + biip MT3 Extended 2048 Custom Keycap Set

$52

Drop Holy Panda X Mechanical Switches

$35

Drop Skylight Series Keycap Set

$45

Drop + Oblotzky SA Oblivion V2 Custom Keycap Set

$20

Drop + RedSuns GMK Red Samurai Keycap Set

$90

Drop GMK White-on-Black Custom Keycap Set

$110

Drop + The Lord of the Rings™ MT3 Elvish Keycap Set

$99

Drop + MiTo SA Godspeed Custom Keycap Set

$39

Trending Posts in Mechanical Keyboards

Peckinpaw

Help with purchasing a keyboard

Hi, I currently have a Razer Huntsman V2, full size. I like the programmable macros and global shortcuts. I also like the feel of the switches (purple). Is there something similar available in the Drop shop? It seems like options that include ten-key are pretty limited. Thanks!

Sep 11, 2024

Mech Keys

Questions

fuseman30

College themed keyboard

My wife wanted a keyboard that is themed after her college colors. It was exhausting/fun searching around for options. This hobby has expanded so much that the options are almost endless. I settled on the Daring Run DR-70f, Gateron EF Curry Linear Switches and DROP DCX Sapphire. She was very happy with the end result and most importantly, I got to build another keyboard. I love the layout of the Daring run and it was crazy affordable! Think I might take on the Flip 6 next.

Sep 10, 2024

Mech Keys

Projects

SimSimCo

Put your dream keyboard on your virtual desktop!

Check it out here: Desktop Keyboard by Simulated Simulation Co. (itch.io)

Sep 8, 2024

Mech Keys

Projects

legendeffects

Failing Switches

Sep 5, 2024

Mech Keys

Photos

Zombiekiller5072

For all you Keyboard Club members, Is the membership worth it and how often are new keycaps come out?

Hey Keyboard Club members and everyone in general. I got a quick question for you all, is the Keyboard Club membership a worthwhile purchase? I would like to buy a couple keycap sets and maybe a keyboard or two, but i don't know how often I would use the discounts the Keyboard club offers. And second, how often does Drop roll out new keycap sets for purchase? Thanks guys! Looking forward to hearing what you all have to say.

Sep 5, 2024

Mech Keys

Questions

dovenyi

Layout optimization best practices: sources of your personal corpus (part 2)

Welcome back to our series on designing custom keymaps! After looking into how good/bad QWERTY is, the power of layers, and the potential of custom keymaps, last time we took the first real step by examining your options for compiling a corpus. As a recap: The corpus is simply a big chunk of text. We use this collection of textual data, often a single text file, to characterize your typing habits (calculating various language statistics), and feed it directly or indirectly as an input into layout optimization algorithms – to find the optimal keymap for you! Today we’ll expand on this idea by exploring your options if, like me, you prefer a personalized corpus rather than grabbing some general (and mostly irrelevant) data available online. Image 1: Letter frequencies – the most basic use of corpora – in this very post We've seen that a well-crafted, personalized corpus plays a key role in determining the outcome of the optimization process. We've also explored...

Sep 4, 2024

Mech Keys

Guides

Drop Updates