We’ve known for some time that Wikipedia readers and communities wanted a dark mode. The fact the feature has been absent has been a constant source of complaint on our social media accounts. It has been the most requested feature from our editors since at least 2019 and our analytics show that 20% of our readership has specified on the operating system level that they would prefer dark mode. While research is mixed, dark mode is often praised as reducing eye strain and helping people with medical conditions.

The problem has never been why, it’s always been how and as my team embarked on a project to improve accessibility for readers [1], we finally set out to answer that question.

The first challenge we had to overcome was making such a feature accessible to anonymous users, without compromising our site performance. Historically, because we cache page HTML using Varnish, we have to serve the same HTML page to all users. After consideration of various options, we addressed this during the Wikipedia appearance rollout by using a small inline script that read from a cookie and altered the classes on the HTML tag.

More difficult challenges remained…

Turning off the lights

Typically, dark mode experiences are achieved by switching CSS stylesheets to use CSS variables. The challenge for Wikipedia is that we have 20+ years of code, much of it without active maintainers, and some with large chunks of code written in libraries we deprecated over a decade ago such as jQueryUI. For over a decade, thanks to the long-term vision of several individuals, we have slowly been reviewing and standardising colours across our vast codebases. Much of this work has been thankless and difficult to quantify until recent years.

Then, in August 2021 the Wikimedia Foundation decided to adopt Vue.js as its future JavaScript framework. With this decision, the Codex design system followed for Wikimedia code, and we have been empowered to accelerate and benefit from these standardization efforts. The new Wikipedia appearance we rolled out in 2023 was a big catalyst and an essential stepping stone for providing a dark mode.

While this provided us with a path forward for making our user interfaces available in a dark theme, we had one remaining hurdle to overcome — user-generated content.

User-generated content

Wikipedia is one of the most unique websites. I think MySpace during its peak years is the only comparable site in the history of the Internet that has allowed the expression of user-generated content at the level that Wikipedia has. While most of Wikipedia is text, a lot of Wikipedia’s content is built with HTML markup, often via pages called templates which have very few limitations. The lack of limitations has allowed editors to build complex things like route diagrams and taxonomy tables which make use of colour for a variety of reasons, whether they be conveying information or expressing a brand. Much of this relies on the assumption that the text has always been black.

Wikipedia can be viewed outside our website, in apps such as the official Wikimedia mobile apps and WikiMed so it is important to consider where the content is viewed. The Wikimedia Foundation’s mission is: “to empower and engage people around the world to collect and develop educational content under a free license or in the public domain, and to disseminate it effectively and globally.”

Using colours in our content causes friction to this mission, as it means it is more challenging to embed our content in foreign places where the background is not white and the text is not black. This is therefore something that should be fixed.

Dark mode shortcuts

There is a well-known shortcut in developer communities for getting dark mode working on any site that various popular browser extensions have exploited. It works as follows: A small amount of CSS applies a filter across the page to flip colours to darker equivalents and an additional filter undoes that transformation for images to avoid them being color-shifted incorrectly.

We have been aware of this shortcut for some time, and it has been popular with many of our users with accounts for some time through Volker Eckl’s dark mode gadget. As with any shortcut, it pays to be suspicious. While this approach is popular with developers and can be shipped relatively quickly, it is not well understood and has problems:

1. This technique can be computationally expensive. A substantial amount of people who tried out browser extensions utilizing this method report high CPU usage and in some cases slow scrolling or crashing browsers which correlated with older devices. While we failed to replicate this in synthetic testing, it was deeply concerning.
2. Native emojis were unreadable in a way that was not correctable by editors.
3. Content can be altered in a way that misinforms or confuses the reader.

Misinformation was a deal-breaker for us and should be for any website using user-generated content where colour can be used.

When colour is important

Sometimes colour plays an important part in an article. The shortcut approach had a worrisome trait best illustrated by two examples.

Consider this article on International Orange and notice, how the colour swatches are subtly different in both articles.

In the first example, the colours are correct. In the second image, the colours have been distorted by the “dark mode shortcut” and are now misinformation. However, to the average reader, nothing is wrong and there is a risk such an issue could remain undetected similar to many of Wikipedia’s hoax articles.

While in this particular situation, the stakes are low, it takes little imagination to see where this sort of colour misinformation can be problematic. For example, consider the implications of the recent United Kingdom election where colour distortion in a legend for a diagram could cause misunderstandings about which party is in the lead; or where someone is trying to determine what coloured gas is filling their room.

An accessible opportunity

As we began to build dark mode and evaluate potential fixes for the colour issues outlined above, we used the Web Content Accessibility Guidelines on colour contrast to determine which pages had colours and colour combinations that might make reading more difficult. Through this process, we discovered that regrettably Wikipedia already had many accessibility issues in its content in its previous light-only state, which were caused by editors selecting specific colours for templates or other parts of the content without checking contrast guidelines. The Wrexham A.F.C article for example has inaccessible red backgrounds on all its table headers, simply because the football team plays in red (this is now fixed!).

While colour can have semantic use in conveying knowledge — for example if describing the colour International orange, a colour helps. However most of the time it is a stylistic choice. Some colours also have undesired connotations that can lead to misunderstandings of information — for example, red can be associated with warning and danger. In general, colour in Wikipedia should be used wisely. As part of this project, we wanted to make it easier for communities to find and fix these issues with user-generated content.

Because Wikipedia can be edited by everyone, accessibility is all of our responsibility. Adding the dark mode feature provided an opportunity to improve our content. What better way to bring greater awareness to accessibility and existing issues, than using a widely desired feature as the vehicle?

While there are many editors well-versed in accessibility, there are many that are understandably not.

Improving our content

We decided that dark mode was an important driver for better accessibility, and that despite the challenges ahead relating to improving articles, it was important not to take a shortcut and instead make our content better. After all, Wikipedia is all about improving the encyclopedia and if there is one thing our community excels at it is fixing things that do not speak the truth. While this felt uncomfortable it felt like the right strategic choice.

By not taking the shortcut of distorting colours, we needed to update many articles working under the assumption that the text was black. For example, it was quite common for tables to have green and red rows, which would now have white text.

To support this, we created various tools for flagging issues to empower editors to make our content more accessible:

1. We created a tool to generate reports of color contrast issues which we update regularly on a separate web page.
2. We created a tool that flags pages where colour usage is ambiguous. At the time of writing, there are over 40 million pages where this is the case.
3. We created a recommendations page to guide editors on best practices in this new world where dark mode exists.
4. The desktop version will allow readers to flag issues with articles directly to editors.

Since our tooling and testing flagged many issues, to reduce the strain this would create for our editing community, we applied several generic rules to autocorrect many of the problems with content that we found (for example making text black whenever we could detect a background).

Given the challenges with dark mode because of these issues, and the time it would take to revisit these colour choices through an accessibility lens, we were more conservative with the initial dark mode release and have qualified it as an opt-in feature with a “beta” status and only released it to our websites where editors were actively engaged with fixing these issue. We’ll continue to monitor this for some time and plan to expand availability to other languages when we feel more confident that the work has been done to make our content more accessible than the current reading experience. If you edit Wikipedia and can help, we encourage you to do so.

As a reader, we hope you enjoy reading Wikipedia in dark mode. We’re sorry this took so long. Hopefully, it was worth the wait!

With thanks

This has been a journey with many challenging conversations! Special thanks go to Bernard, Edward, George, Jan, Jennifer, Justin, Kim, Mo, Olga, Nat, Szymon, and Steph in the Web team and to Anne, Barbara, Chris, Eric, Derek, Lauralyn, Roan and Volker in the Design Systems team who have incredibly patient and responsive throughout this project with our rapidly-changing needs. Thanks also to Volker and Leon whose early work many years ago provided the impetus for much of this work. Our community ambassadors Isabel, Martin, Mehman, Mohammed, and Phuong have played an essential role by keeping us connected to our editing community.

In addition, thank you as always to our community that helped us along the way and the key contributions from other colleagues and friends that helped us scale across our codebases and content. The Wikimedia Hackathon 2024 in Estonia played an important role! Rather than miss out on someone, I’ve created this page to celebrate all those many people — please help us expand it!

Thanks to Olga for helping me with this blog post!

¹ This project also included recently introduced controls for other aspects of the appearance like font size and colour — if you’re interested, you can read more about these changes in this Diff post by our designer Justin Scherer.

This blog post is available on my website and Medium.com.

This week my team reached an important milestone in our biggest project — the deployment of a new look for English Wikipedia. This was the most significant change to the desktop site in over a decade.

However, far more has changed than just appearance, and in this blog post, I hope to shed some light on what we’ve achieved and the people that did it¹.

Since the first patchset back in January 2020, the redesign of Wikipedia’s desktop site has led to various improvements to our codebases and tooling.

I’m sure many would smirk at the idea a redesign could take three whole years, rather than three months, but much of this time has been consulting with various partners and refactoring the code that has evolved over twenty years. We’ve followed an agile approach to development, incrementally building out a grand vision set by our designers. Our first iteration, to our Basque language Wikipedia, was in fact deployed as far back as July 2020. We’ve built responsibly, intentionally, and strategically, thinking holistically about the entire behemoth that is our codebase. When I look back at our release timeline at all the things we did as a means to an end of this project, it’s incredible to see what’s been achieved over three years.

We became better at disagreeing

I’ve noticed huge improvements in the way Wikimedia Foundation works through internal impasses. As an engineer, I am so grateful that this project had technical program managers supporting us throughout our daily rituals and helping us overcome any obstacle that was thrown our way. Whether it was throwing people in a room together or just asking the right questions, or negotiating conflicting strongly held opinions, we found solutions to make big impactful changes whether it was using Vue.js or growing trust with each other. Among others, thank you to Jazmin, Lani, Suman, Nat, and Daniel for all your help here.

We worked effectively with our editing community

We’ve strengthened our relationship with our editing communities.

Although we deployed to English Wikipedia in January 2023, the new look and feel has been the default for many of our other projects, in far less complete form for a much longer time.

We owe a tremendous amount to our pilot communities and their feedback as we’ve iteratively built out the design. I am astounded by the quality and respectful feedback many volunteers were able to give us, and by the lengths our team has gone to describe our motivations and goals. For example, the limited width improvement for reading has been a controversial feature ever since we introduced it, and when we realized there was no accessible documentation for why something which had been of obvious importance to us, we wrote documentation ourselves. My work colleagues have organized and executed various office hours, some of which I’ve attended, which often although only scheduled for thirty minutes have massively overrun due to the enthusiasm and engaging conversation we’ve had. Kudos to Alex, Szymon, Olga, Phuong, Martin, Letizia, Sherry, and all the volunteers for their feedback that helped us tweak the new design.

We improved our frontend development practices

Our codebase was very much a backend-first piece of software, and working with frontend code historically has been a little lackluster, with a lack of support for modern JavaScript, a lack of industry-standard build tools, no frontend framework, and lockin to a myriad of less-desirable libraries such as JQuery.UI, moment and homegrown libraries.

One big challenge, we’ve faced historically has been deciding on which of the many JavaScript frameworks to use in our stack. This had a thorny decision to make, but with good management, and a process for discussion and discourse, we agreed that we should use Vue.js going forward for frontend development. We also found ways to support the writing of ES6 code without tooling, and our designers have been working with teams across the organization to build a design system toolkit.

Since our initial pilot experiment of Vue.js, which is used in the new search experience, our toolkit has grown, incorporating views from a variety of teams and projects. Thanks go to Eric, Anne, Volker, Roan, Nick, Jan, Stephen, Alex, Barbara, Nat, and many others for trailblazing this tremendous effort.

We made improvements to backward compatibility

As I wrote previously in 2018, when we update the design of our site, old versions tend to stay around for a very long time. This traditionally hasn’t come for free, and historically often small things, such as markup changes, and removing CSS, require changes in multiple versions that can result in dozens of patches.

We’ve had a huge over-reliance on markup, where changing the HTML structure or an ID broke key integrations, such as unstyled menu items. To allow us to make the big changes we had planned, and reduce the burden on developers, we created a shared data and presentation layer using templates. This resulted in a 20% decrease in code maintained by my team.

To counter this we added APIs and tightened contracts where code integrates. For example, we now have stable methods for modifying the page subtitle and adding items to menus with icons in a consistent manner.

We’ve broken features many times during the project and I appreciate the patience and forgiveness of developers as we’ve tried to prevent that from happening again. Thank you, Leon, Ed, Roan, Ammarpad, Clare, Bernard, Mo, and Bartosz for all your help and patience here.

We improved language support

Language support in the new version is far more prominent than it historically has been and appears beside the article title. Making this feature more prominent invited considerable feedback from our community and required working closely with our language team, and distracting them from their own projects. Thanks to Santhosh, Pau, and the language team for working closely with us!

We simplified APIs and removed code paths

As we approached what looked like straightforward changes, such as adding items to menus, we met an unnecessarily high level of complexity. In the case of menus, there were over 5 different APIs for code to add an item to a menu. We simplified a lot of that for our own sanity. Thanks to Ammarpad, Clare, and Mo for your efforts here.

We retained a good level of accessibility

The old desktop look and feel actually enjoyed very good accessibility so we wanted to keep this standard going into the new redesign. We were lucky to benefit from not one but three engineers who specialized in this field and kept us accountable throughout the project and were fortunate enough to receive lots of feedback from users and experts in the field. Kudos in particular to Bernard, Jan, and Volker who’ve all been wonderful, and to the managers that have enabled them. Bernard in particular organized some very useful feedback from the American Foundation for the Blind.

More information: How the new Wikipedia design focused on accessibility

We built with performance in mind

We’ve improved our performance. We’ve removed unused CSS and limited CSS to pages that need it and we’ve standardized thumbnail traffic. We’ve monitored big changes such as the switch to Vue.js and a new API for search. Thanks in particular to Volker, Nick, and Timo.

We restructured our Wikitext markup language

For the new table of contents experience, we needed to restructure Wikitext. In the old version, the table of contents was part of the parsed article HTML content rather than metadata that could be rendered elsewhere. This change in particular was a tricky change to make while maintaining the older style version. This hopefully serves as a blueprint for a major restructuring of article HTML in the future. Thanks to the content transform team — in particular Subbu, C Scott, and James.

We improved third-party support

Our software is run by various websites that are not Wikipedia and many of these look different from Wikipedia, using plugins known as skins. When we began the project various skins for our third parties were broken, unmaintained, or more complex than necessary due to limitations in the existing architecture and shared similar challenges to the ones that were blocking our goal. Auditing these helped us predict challenges we would face. There are now 101 skins that our 3rd parties can use with much better support and we have better systems in place for backward compatibility. You can explore these beautiful creations at the Skins lab and even make your own. Thanks to all the skin developers for their patience, engagement, and insight: Mainframe, Alistair to name a few.

We started to monitor visual regressions

About a year into the project we were noticing lots of visual regressions sneaking into our releases, which slowed down our overall velocity.

To counter this we created a visual regression framework and incorporated it into our deployment process. This tool is now starting to be used outside the team. Kudos to Suman and Nick for most of the work here.

More information: Visual Testing: Building A More Robust Wikipedia Interface By Spotting the Differences³.

We started to log JavaScript errors

At the start of the project, we had no JavaScript error logging. We now do. Thanks to Jason L and Daniel for your support here.

We improved the mobile experience

While we didn’t quite get as far as we would have liked with this. The new desktop skin is more responsive than the old one. Mostly this has been an issue of prioritization — we have a perfectly capable mobile site right now, but the choice is important, and we’re keen to at least have a viable skin.

Meanwhile, our mobile traffic grows, so we’ve kept one eye open at all times. We’ve brought the technology stack for the mobile site closer to the desktop site and vice-versa. Features previously absent on the mobile site, are now available to use on mobile and some of the best ideas on mobile have been merged with the desktop versions. There are new projects over the next year that look at improving our diff pages.

Thanks in particular to Bernard, Jan, Bartosz, and Jason S here.

… but we’ve still got work to do

Importantly our work is not done. There are several imperfections that we had to deprioritize or delay. In particular:

• The new desktop skin is not as responsive as we’d like it to be. This wasn’t prioritized as we have a mobile site and much of the challenge here lies in working with our content creators to make article content responsive (in particular large tables).
• The new skin is not as performant as it could be. We’re currently shipping more than 2kb of CSS than we need to, for example, due to the way we roll out features, and overall the site’s critical CSS is a little larger than before. The flip side of this is added functionality that should make the site more useful, and a design that degrades on older browsers
• The font size is smaller than we’d like it to be because changing it at this point would create significant work for our editors.
• There are still remaining issues with accessibility on the site. We’ve received consultancy and believe we have identified those issues and continue to work towards fixing them.
• Several blog posts about our work are still being written²

I look forward to improving these imperfections and more. Most importantly because of the last three years, I feel we have a much better foundation to work from now. The next improvement to the desktop experience will surely not take another decade.

Footnotes

¹Apologies if I have missed out any of the village. If I have, please let me know and I’ll happily and apologetically credit you in this post.

² I will be updating this post with links to these so please check back in a week or so!

³ This was updated from the original blog post on 19th January to add a link to Nick’s blog post and one 24th January to add a link to Bernard’s

This blog post is available on my personal blog as well as on Medium.com.

Thanks Alex for the last minute technical writing support!

This has also been posted on my personal blog.

On a day in November, with the support of others¹, a somewhat absurd bug involving semicolons that took a week to fix was finally squashed. It was part of a group of changes that had led to a 47% bump in moderation tool usage on mobile. What follows is my account of this unremarkable incident as a reminder that a bug is never elementary; I will reflect on the the art of making progress in legacy code without disruption; and I will boldly hypothesize how the removal of a semicolon will improve the Wikipedia encyclopedia.

Disaster strikes

This was not the usual kind of semicolon drama that developers are familiar with. I didn’t crash a site with a PHP syntax (at least not this time). I didn’t add a semicolon in a string boundary by accident. This was a semicolon in one of our many user interfaces for editors hidden away behind the scenes. It was a semicolon that was plaguing the mobile interface, making it a little more confusing.

Thursday was when the drama began to unfold. Like a virus, semicolons were showing up in all sorts of places. If you are an editor you may have noticed some semicolons in your editing interfaces that were not there a week ago. For instance, notice these two semicolons before the date:

The bug reports were coming in — seven in total. All of them described the same underlying issue but in a different flavor. Semicolons had disappeared in some places — in some places pushing words together that shouldn’t be together! In some cases they’d appeared where they were not before. I was pointed to a long detailed conversation on English Wikipedia where editors reported semicolons were appearing in unexpected places. One reported:

“I’m a big fan of semicolons, used correctly; this one, I have to say, just looks weird.”

Another pondered:

“There could be a reason for this, perhaps?”

A third added:

“I don’t remember there being an announcement of this happening. Is this a glitch or an intended feature?”

A semicolon is clearly not a feature. It’s not something you write blog posts about. “Wikipedia finally removes semicolon from mobile” is not exactly the kind of “feature” you might see at the top of Hacker News or Reddit. However, here I am writing about it.

So, how did we get here?

How we got here

The team has been slowly winding down the Advanced Mobile Contributions project. This project has added various moderation features to the Wikipedia mobile site². It also had strong, inflexible requirements from our department which were:

1 ) Don’t interfere with the desktop experience.

2) Don’t interfere with the standard mobile experience of ordinary users.

3) Make legacy interfaces available on mobile to users that opt in.

These requirements have been challenging to us; although mobile and desktop sites use different stylesheets, they share a lot of the same HTML. The HTML of these ten-year-old interfaces were built squarely with desktop in mind.

While some of this work has been adding links to the interface to missing moderation features, the real challenge has been making the legacy interfaces that those pages link to usable on a mobile device.

This work has involved taking complicated legacy interfaces that were built over ten years ago, and trying to mold them into things that are presentable on mobile, while at the same time making sure nobody using the desktop version noticed. While the engineer in me would ideally burn them all to the ground and rewrite them from scratch, we didn’t have enough people to commit to such a redesign. Instead, the goal was to get them into mobile in the quickest possible way “warts and all”.

The RecentChanges moderation tool (which shows edits across the site, so our editors can watch out for vandalism) is a good example. Many assumptions on the desktop site had been made about how the page should look, and those assumptions didn’t help present the same page on mobile. The desktop site used semicolons, ellipses, and brackets to separate content. It did this in raw HTML. This screenshot hopefully illustrates the problem:

While we might imagine further improvements to such a page, remember we were not looking to make too many drastic changes as we didn’t want to touch the desktop site. My company is a non-profit. The goal was to remove a semicolon not to start a revolution.

On mobile we decided to stack this information vertically, like so:

To do that, we had to get creative with CSS to ensure we kept things the same to desktop users while being able to change it based on the display media.

Luckily we found a solution using CSS media queries and CSS pseudo elements to do exactly that.

After such changes the RecentChanges moderation tool was looking good, but it had one nagging problem — a forgotten semicolon lingered in the UI after the page titles.

The follow up task “Stray semicolon in RecentChanges, Watchlist, History and Contributions interface” looked like just a one line change, continuing the work we had already done to remove unwanted “..” and unwanted parentheses.

Inspecting the HTML, I saw that the semicolon was in the raw HTML of the page. It served a purpose in the desktop version of the page by helping separate the content in each of the rows. However in the mobile mode it was redundant. I decided to move the semicolon from the HTML into a desktop only style sheet.

So I picked the bug up, I found where the semicolon was in the HTML and deleted it. Using the class name of the element that followed the semicolon, I added a CSS rule to the associated page. A fellow developer reviewed it, and like me, thought everything looked straightforward and the patch was merged. I focused on other things. As I did, the Earth orbited the sun. Light turned to darkness. Darkness to light.

The patch started to distribute out to our many websites. Then, the first bug report came in. The semicolon was not being translated into our many languages.

For those wondering how semicolons are translatable — this was news to me too — In French, a space must always proceed a semicolon, and here are some places where a semicolon is not the same semicolon we use in English:

In Urdu:
؛;

In Persian:
؛

That was relatively straightforward to fix. We have devised ways to translate pseudo element content. The fix was quickly merged and deployed by a volunteer. I thought I was done; however more bug reports came in.

My semicolon logic was still not right. I had made some bad assumptions about where the semicolon was needed. I had assumed it would always show before a timestamp, but that was not true. The ten-year-old code was badly organized and badly documented. There were various edge cases in our complicated ecosystem. I sat down and wrote a spec and some very explicit quality assurance steps to make sure the undocumented behavior was documented.

My final patch, with the help of retroactively defining its behavior with bug reports from editors, was mostly an extensive documentation block to explain the behavior to the poor soul that would work on this code in future. The best thing a developer can do is write documentation for a future user, no matter how obvious it may seem. Developers that complain about documentation blocks need to be slapped with a wet fish.

On Tuesday, the patch had still not been merged due to the Veterans Day holiday. Nobody was around to review and deploy the patch.

Back on the discussion page, another anonymous user, possibly impatiently, asked “why is this still a problem?” The patch rolled out shortly after. Everyone involved was a little exhausted. An editor complained about semicolons that had always been there (I checked with git blame) and I wondered if anyone knew why we needed a semicolon at all? What would Kurt Vonnegut say? I decided those questions were best left for another day.

This exercise for me was a good reminder to never assume things and to treat every bug with caution, no matter how trivial it looks. There is no such thing as an easy bug. All bugs lead to other bugs.

Reflection

Much of the hard work to get our moderation tools available on a mobile phone was spent carefully dissecting legacy code like this. Even though we’ve tried to minimize the disruption to desktop users, there were a few exceptions. The semicolon incident was one of three incidents that got the attention of editors.

There was the Hideous history page incident” when some editors were angry that the touch areas for our form inputs, which had been made larger. While our intentions were to follow accessibility recommendations, some felt that the enlarged form got in the way of their workflow by pushing down other content. It should be “collapsed by default” said one.

There was the User Contributions incident, when, learning from the feedback of hideous history page incident, we collapsed a form by default, but the feedback was that we should not be collapsing the form that was a vital part of someone’s workflow. “Would prefer if it was expanded by default tbh” one user remarked. 🙄

There was also the Alarming “only a preview message” incident when we standardized the styles of a semantically identical component to avoid shipping unnecessary bytes to mobile mode.

The crux of this is that it was impossible to please everyone during this project. but we worked hard to keep a healthy balance. It’s amazing we pulled it off.

Sometimes the things that we work on that make a difference are not newsworthy, retweet-able, technically stimulating or even noticeable. Even with the semicolon incident resolved, there was no pat on the back or thank you. Silence was the best reward. The work we do might not use the latest technologies or be glamorous, but there is an art to making progress without disruption.

I will tell anyone that is available to listen, that being a skilled software engineering is not always about your code. It doesn’t matter what frameworks you use. Ultimately it’s an exercise in problem solving and supportive collaboration between co-conspirators. It’s about being responsible and being honest — a feature can take a lot of time if it’s being done well. It’s about understanding if a semicolon is important or useful. And it’s about being able to have that conversation.

One of our editors abian put it so succinctly when he shared these kind words:

It’s a great idea for users to generate content collaboratively and horizontally, but this model doesn’t favor process efficiency, user experience or innovation, so these areas require skilled professionals who fight against tradition, stagnation, short term and subjectivity.

How does this improve the encyclopedia?

The Broken Windows Theory is a theory that says signs of crime encourage further crime. Similarly, in Wikipedia, bad experiences encourage further bad experiences.

I believe Wikipedia’s mobile experience has suffered because many editors have been unable to use tools inside mobile that they need to carry out their work and have been forced to live in the desktop experience of the site. As a result, they haven’t used mobile. As a result, they haven’t seen the broken windows that readers see. Barely anyone complained about the stray semicolon in the mobile site, but on the other hand, when a semicolon disappeared and appeared in places on the desktop site, seven bug reports get opened and everyone seemed to care about it.

The semicolon incident gives me hope, that if broken windows are being noticed in desktop, they will soon be noticed in mobile. It was pretty cool that our editors got this riled up about a semicolon. Imagine what happens when they get riled up about an experience that is unusable on a mobile phone. There’s no longer an excuse to be an editor using the desktop site on a mobile phone³.

I have nothing to concrete to point to you except several artifacts. A semicolon has gone; a form now has input fields that are actually big enough to be touchable on a mobile screen; certain forms are now collapsible and get out of your way when you no longer need them; there are links to certain features that previously were not there.

The amount of design and product wrangling⁴ that’s gone into this project is staggering. While to the outsider observer, we’ve simply added a bunch of links to the mobile experience, we’ve done it in a remarkably elegant way that doesn’t overload the user with information. Menu hierarchies have been revisited, and we’ve frequently had to make trade-offs between beautiful and functional.

My team, with the support of some wonderful volunteers providing feedback and triaging support⁴, has been making minor tweaks to HTML and CSS across the site, and these small things make a difference and should be celebrated. The advanced mobile contributions project has been about that. It’s been about small tweaks here there and everywhere.

The mountain of small thankless changes at times have felt repetitive and uninspiring, but we churned our way through them. On their own as bug fixes they are meaningless. The end results look unremarkable. However, when combined they build something incredible but less tangible — a better mobile experience.

Of course, our data, does give something more tangible. We're seeing an increase of 47% of moderation actions on mobile. So clearly, every semicolon removal makes a difference.

¹ Thanks in particular to Volker, Jan, Zoran, Xaosflux and Huji for your help in resolving the semicolon matter.

² Logged in users get more from Wikipedia. Logged in users can also opt into additional parts of the experience using our settings page

³ If there is a reason you feel you still cannot edit from a mobile device please let us know on the advanced mobile contributions talk page:

⁴ Hat tip to Alex and Volker for juggling all the different design pieces and Olga for making sure we didn’t get stuck in the weeds (sometimes you need someone to stop you going into Semicolon swamps).

⁵ Thanks to Ammarpad, Masumrezarock100, stjn, Zoran and Pelagic and everyone else who has been involved for your input,codeand guidance to our team during these UX and standardisation efforts.

While I walked through airport security the lady behind me asked me if I was an engineer.

I was a little surprised. I didn’t know this person and somehow they knew my profession. What had given it away? Was it because I was wearing a hoodie? (surely not). Was it the stickers on my laptop? (no my laptop was upside down). Was I wearing a nerdy t-shirt? (no). How did you know?

She pointed to her little finger on her right thing. I saw you are wearing a ring on your little finger, she said.

I laughed. Oh that’s the only finger my ring fits on, I told her. I got it when I was 15 from my mother and it’s been slowly moving down my hand. I spent 5 years with it on my right ring finger and everyone thought I was married. Traditionally, in the United Kingdom, we wear the wedding finger on the left hand and I had no idea it was the opposite for the United States. I sometimes ponder whether that was why I had so many bad dates when I dated in San Francisco.
How funny, she said. Maybe the engineer ring is an American and Canadian thing.

Our conversation was interrupted as I was ushered to go through the x-ray scanner. I pushed down my jeans and lifted my arms. My whole body scrutinised by thousands of invisible rays.

I bet she’s a real engineer, I thought as I stood in there. Creating things that last. Things that are useful to humanity.

My bag was marked for a second check, so we were able to continue our conversation on the other side of the scanner.

Is it any kind of engineering I ask? Or only specific disciplines that wear the ring?
“Any,” she said. I feel vindicated.
“What kind of engineer are you?”
A chemical engineer.
See! A real kind of engineer!
The ring is supposed to signal a commitment to ethics. The idea is that if anything has passed through your hands it’s of good quality and ethically made.
I like that, I said. I’m happy this ring landed on my finger. It’s not an engineering ring, but I’ll think of it as one from now on.

As I sat waiting for my flight, I looked up the history of the ring. Its roots seem to be traceable back to the 1907 Quebec Bridge collapse which killed 75 of the 86 workers due to the design of the bridge insufficiently supporting its own weight. Basically something bad happened that caused engineers to take a step back. My web searches led me to the The Order of the Engineer manual.

The manual describes the symbolism and the ceremony for receiving a ring. Essentially engineers make a vow and get a ring to symbolise their commitment to that vow — to that discipline— very much how we take a vow in a marriage to honour a relationship. The words the receiver recites were apparently written by Rudyard Kipling¹(although I cannot find the source).

In the ceremony the receiver of the ring recites the following:

I AM AN ENGINEER. IN MY PROFESSION I TAKE DEEP PRIDE.

TO IT I OWE SOLEMN OBLIGATIONS.

Software engineering should be an engineering discipline to be proud of. On our best days, we make systems that humans interact with. Systems that make humans more efficient. We free up human time to do the things that matter most.

On those days we are actually no different from engineers who help build bridges that connect people, or engineers that design medicines that heal and give people more time to spend with one another.

I think the reason I felt the way I did in the x-ray scanner was I don’t see enough of this. A lot of the systems we are building are trash. We (humanity) are building systems that waste people’s time, that make people miserable, that lock people into services they can’t escape.

It doesn’t have to be this way. We need to work on things that matter and that impact humanity for the other, not yet another GIF sharing site or a site to create zombies shouting LIKESSS LIKEESSS.

The vow continues:

AS AN ENGINEER, I PLEDGE TO PRACTICE INTEGRITY AND FAIR DEALING, TOLERANCE AND RESPECT; AND TO UPHOLD DEVOTION TO THE STANDARDS AND THE DIGNITY OF MY PROFESSION, CONSCIOUS ALWAYS THAT MY SKILL CARRIES WITH IT THE OBLIGATION TO SERVE HUMANITY BY MAKING THE BEST USE OF THE EARTH’S PRECIOUS WEALTH

I like this. Software development needs more of this. More inclusiveness and more respect in the environments we build this. We need to build systems that uphold to standards — whether it be performance or Making sure we build systems that last rather than systems that fall apart in a year.

The vow finishes with:

AS AN ENGINEER, I SHALL PARTICIPATE IN NONE BUT HONEST ENTERPRISES. WHEN NEEDED, MY SKILL AND KNOWLEDGE SHALL BE GIVEN WITHOUT RESERVATION FOR THE PUBLIC GOOD.IN THE PERFORMANCE OF DUTY AND IN FIDELITY TO MY PROFESSION, I SHALL GIVE MY UTMOST.

As I read this I think of companies supporting unethical practices, and the engineering friends that call it out.

I read through these words and think they make a lot of sense for software engineers. I decide the ring on my rightmost finger is an engineering ring and I think more software engineers should wear engineering rings.

As we type on keyboards whether it be alone or during pair programming sessions they can serve as a visual reminder to ourselves to not cut corners and to write good code that lasts, is easier for the next poor software engineer to maintain and has extensive documentation.

As we eat lunch together or wave together in Google Meets, we can remind ourselves to support one another, that together we are in a profession of great pride and great responsibility.

In meetings where bad decisions and are being made and ethics are being questions we can use the ring as a signal that we need to speak our minds and we should feel empowered to do so.

Why we work on things matters. How we work together matters. As does what we build, how we build it, and who we build it for. Sometimes it’s easy to forget this an I think there is much room for improvement here. Possibly a ring and a vow might be a small thing we could all to get us there.

¹ While I’ve always admired Kipling’s ability with words, given the context of this blog post, I should note that it’s unfortunate that Kipling was likely an imperialist and racist.

Connecting with the people your software serves is more important than anything else you could do.

The Wikimedia movement is an international movement and a Wikimedian is someone who contributes software or free knowledge to one of its many projects (the most famous of which is Wikipedia). Wikipedians work tirelessly to create, nurture and protect content as well as provide the software that wraps it up and delivers it to readers. Working across multiple timezones in different languages in different countries with different skill-sets these people support the largest website in the world run by a non-profit.

It’s important for such people to get together to share knowledge, to share time and to share ideas. Various hackathons and editathons happen across the world every year and are open to all — whether you have no idea what we do or if you have been around for years — and financial support is available for those who might not otherwise be able to attend. I get the privilege of attending many of these events and this weekend I’m in Prague for the European hackathon.

Thursday

The event begins with a social event. I spy TheDJ one of our most prolific and longest running editors who is also a software developer. I don’t think I’ve been to a hackathon where TheDJ has not been there. I give him the same hug I’ve given him in France and in Israel, ask him how he’s doing and then give him a demo of the latest project I’m working on at the Wikimedia Foundation. I am reassured by the feedback he gives as he explores the prototype, and it gives me affirmation that we’ve building something that’s going to have a positive impact to users when finally deployed.

Next to TheDJ is Sage. Sage is also a long term editor, but I haven’t seen him for four years. Back then, he provided community input for a mobile web upload photo feature that eventually had to be shut down due to an influx of selfies that the community was unable to moderate. One good thing that came out of that project was a photo of a very rare disease (diphtheria) and the Wikipedia selfie article which Sage created. I ask him if he’s edited the article recently and we look at it together, scrolling through it on his mobile phone. He feels the image selections could be improved, and I have a feeling he’s going to be editing the page later.

Outside, I talk to Andre, the Wikimedia bug wrangler who lives in Prague. We talk about the city he knows so well and then as I’m a little jet-lagged, I make my way to the hotel as I know very well that many of these old friends at this weekend’s hackathon have lots to catch up on and are likely to be out all night (and I’m too old to do that these days).

At the tram stop, I bump into one of my colleagues in Brazil who’s attending his first hackathon. I’ve seen his name on bug tickets and in emails but never in person so as we make our way back to hotel we talk about the maps infrastructure he supports.

I’m quite excited about the weekend. I work in a remote team with team members in Spain, England, Poland and the United States. If there’s one thing I’ve learned well through this experience, it’s that some things get done better when they are done face to face and some things are done better asynchronously. I’m curious what will come out of this hackathon.

Friday

The next morning, we are up bright and early and everyone is trying to decide what they want to work on during the hackathon. During the introduction session, I learn that the majority of the hackathon attendees are volunteers and a high percentage are Czech. The Wikimedia Foundation’s staff are a small minority of the people here. A few possibilities are pitched and a session to match mentors to newbies is planned.

After the introduction I meet Simon who’s been editing Wikipedia but wants to contribute code. There’s a few bugs in the Wikipedia feature to view images on the desktop site that he’s keen to work on, so we sit down and discuss solutions and I walk him through the code. He’s a smart man and he’s up and running in no time whatsoever. I’m hopeful we can integrate him more into the movement as I feel he would have a great impact.

I’m still not sure what to work on, other than helping enabling people to do other things, but I come across a group of editors who also use the iNaturalist mobile app. iNaturalist is a citizen science project and online social network allowing people to share observations of biodiversity. The Wikimedians here in Prague are interested in exploring ways to share its own freely licensed content with our own projects.

The iNaturalist Wikimedians are planning a photo walk. I decide it will be a good opportunity to spend time with editors and understand what they do better and hopefully find opportunities for my hackathon project to help them do that more efficiently. Inspired by Jimmy Wales’s fundraising banners, I walk around the room with a big sheet of paper saying “come on a walk and learn about iNaturalist” and tell people “the banner campaign will only stop when I have enough people” which turns out to be surprisingly effective.

I’ve been using iNaturalist for some time, but there’s lots of people in the group that don’t know how to use it, thanks to the human billboard banner campaign, so everyone has questions and knowledge to share. I learn something new that iNaturalist defaults to a non-permissive license that doesn’t allow my photos to be used on Wikipedia. The group helps me change that and retroactively update the licenses on my existing photos. Eventually we reach a park where we take some photos using the app. Our plan is to later find ways to use these in Wikipedia.

TheDJ is migrating some legacy code to Webpack and hitting some problems so the two of us sit down to pair program and get things working. He just has one bug left to fix, but he’s unblocked now.

Saturday

I join the group session with the iNaturalist Wikimedians. Apparently lots of Wikipedia articles on biodiversity do not have images, yet there are images inside iNaturalist that can be used! A Wikimedian in the group describes their workflow for transferring images to Wikipedia. It involves manually downloading the image and uploading it via a form. I’m pretty convinced I can make a slightly better workflow so get to work immediately on doing just that.

Soon I have a raw and hacked together prototype that shows me images from iNaturalist for Wikipedia articles. I just need some way to identify which articles do not have images and upload those photos. I ask an API expert how to do the former and discover the Wikidata query service and ask the image upload expert how to do the latter and am pointed at some badly documented parameters to a Wikipedia page that allows you to pre-fill upload forms. I combine everything to get a minimal viable product.

Simon is making good progress with his changes. We review and merge his first ever patch to our codebase. There’s a glow to him that’s shared by most new contributors and it’s nice to see that in person. Someone has already added him to a list of trusted contributors. Should I say thank you? he asks. There’s never any harm in saying thank you, I grin.

Sunday

Playing with my iNaturalist prototype, I realize that most of the articles do not have photos on iNaturalist. I soon realize that I’m only getting Wikipedia articles of endangered species, so it’s no surprise that a citizen science project is also lacking in those images. I need to change my query to exclude endangered species, and for that I need to learn SPARQL, a RDF query language for databases. I make a note to myself to learn that later, but in the meantime I need to find the SPARQL expert and of course, there’s one in the room who very generously gives me ten minutes of his time to correct my SPARQL syntax.I now have a working app! I upload several images from iNaturalist effortlessly to Wikimedia Commons and include them on their corresponding articles. My contributions include a photo of a damselfly. I share the link with my new group of friends.

I look around the room and feel grateful to be part of such a rich movement of inspired dedicated individuals. Everywhere I look, someone is helping somebody solve the latest problem, or sharing a laugh, or furiously coding to wrap up their project so they have something to show. At the showcase a wide range of projects are demoed to cheers and claps. There’s an API documentation website that has been worked on alongside a remote attendee who couldn’t make it to Prague ISBN scanners; text recognition services; and an app that let’s you tell Siri to edit Wikipedia. It’s invaluable to see where everyone’s passions lie; to learn what’s possible and to find ways to make things better. I leave this hackathon energized and inspired and be part of something so unique and proud of my organization for helping make this happen.

I get a message from Dario, an ex-work colleague (when you leave Wikimedia Foundation you don’t necessary leave the movement), who’s seen my iNaturalist prototype. Apparently my prototype has been enthusiastically shared by a hackathon participant and is now going viral with users uploading images. He has lots of user interface improvement suggestions. I guess I know what my next hobby project is going to be…

On the train, as I head to Budapest, I reflect on what makes a Wikimedia hackathon so special. I can only conclude that our volunteers are our customers and there’s nothing more valuable than getting face time with the people your software serves. Software doesn’t solve problems, people using the software solve problems. It’s easy to forget that sometimes.

A hackathon story was originally published in Down the Rabbit Hole on Medium, where people are continuing the conversation by highlighting and responding to this story.

A big challenge for any software engineer is explaining an important technical change to an audience who may not necessarily have the appropriate context. “Technical debt” is a phrase that will perplex a product owner if not articulated correctly. If you work in any product-centric team, you’ll likely find many technical tasks may be brushed aside in exchange for more tangible, visible outputs. That said, technical debt is very real and needs to be addressed.

Back in January 2018, Wikimedia engineer Joaquin Hernandez and I pitched a one year project to pursue the scoped and measurable goal of increasing code coverage of our codebase and investing in our ageing but strategically important codebase. We did so with the belief that this could lead to working with code we liked much more.

Often developers, designers and product owners speak different languages and have very different and conflicting desires. Taking the time out to write the project proposal and talk together about problems, solutions and benefits was well worth the time and resulted in a shared mutual understanding. I believe that because of this work and new shared understanding, the project was easier to get onto our annual goals.

The problem statement

Our “stack” is a LAMP stack, with jQuery and some in-house (but open sourced) libraries you have likely never heard of. Given the age of our project, we still have several libraries that we can’t easily remove from our stack without breaking crucial and complicated tooling made by developers long gone that editors still use to keep Wikipedia running. For example, much of our site is built on jQuery UI which was released back in 2007. Our technical community has made good progress in pulling us away from these libraries by getting them off the critical path (JavaScript that is loaded without the user interacting with anything) and reducing redundancy.

Possibly the biggest example of success with our modernization of the front-end is the mobile website, which runs on a separate domain, and runs 60% fewer bytes of JavaScript than the desktop site, despite being more reliant on JavaScript for many of its workflows, notably editing and searching.

Our past front-end decisions continue to haunt the developers that work on our codebase and are unlikely to disappear from our stack entirely any time soon. Any new library being introduced to our stack is quite rightly met with suspicion. Any attempts to adopt modern JavaScript libraries such as Vue.js and React are quite rightly slower with the scrutiny you’d expect from a decade old web veteran with limited resources and time like Wikipedia that’s seen it all from its rocking chair on the porch¹. We’ve learned that bad technical decisions are hard to undo.

We’ve likely had every discussion every developer in every other organization has had, whether it’s been “Mustache VS Handlebars”, “Should we use TypeScript?” or “Which is better? Vue.js or React.js?”. What we’ve found is it’s really hard to get consensus on these big questions in an open source community with no benevolent dictator to make the decisions.

This aside, mobile usage is growing. We are interested in adopting technologies such as service workers which provide offline support and better availability, but enabling one in a codebase such as ours is risky business.

The solution

After discussion within our engineering team, we focused on a solution to, and have provided proof of concepts for, iteratively refactoring and revising the existing code. Our project proposal, which is public, explains why we feel strongly that we invest in mobile’s frontend architecture.

We drew our proposal from experience in a recent experiment with our page previews feature and a write up of the experience. Before the project, our front-end assets were managed with a MediaWiki-specific system called ResourceLoader and our proposal was to move off it and lean more on modern-widely utilized front-end tooling, for example — but not necessarily — Webpack. With this achieved we would refactor, improve and modernize our neglected component library.

Inside the project proposal, we importantly made no specific promises; instead, we presented tangible problems and measurable outcomes, which after six months we have now partially achieved. Those include:

• Increased test coverage (our code coverage was less than 50% and more alarmingly, 45 of our 81 files had 0% coverage.)
• Fewer regressions (we’ve noticed every JavaScript regression and the majority have not been related to our refactor project)
• Code built with modern tooling (all our JavaScript code is now built via Webpack which has made it much easier to work with)
• Possible performance improvements (so far we’ve not seen any change here but we see the potential for change)
• More reusable standardized components (we’ve started making headway on this!)

And less measurable (but hopefully outcomes that could later be recognized):

• Quicker on-boarding of new hires
• Quicker development cycles and estimations for product work
• More future proof code

Essentially, for a project with limited time and resources (Wikipedia’s mobile site is maintained by a team of just 6 paid people), we pitched a refactor not a rewrite. Rather than replacing a complicated system, we would slowly and iteratively improve it. This was important as it promised iterative development while keeping the site up and running (and improving at all times). While this slowed down development it kept our work visible and kept us accountable to our product team, meaning it was impossible for our development team to hold our product owners hostage by telling them we couldn’t build new things until the refactor had completed. It also allowed us to work on new products in parallel to this important work (we shipped several projects during this time) as well as guaranteeing that we would achieve something!

If you are interested, there is a technical write up of what we have done and what we have achieved so far on our internal blog: Migrating code from ResourceLoader to Webpack.

Lessons learned

Halfway through this project, my team has made great headway, and I feel I am at a good point to reflect on what’s worked well for us.

1. Stop coding every now and again

If your engineering team is going from sprint to sprint without stopping, this is likely a problem. Our team found respite from coding during our company all-hands and used this valuable time to talk strategically and write up the project proposal over the course of three days. This activity wasn’t all talking and writing— we built an important proof of concept!

Make sure your team is making time where everyone can stop coding so that the team is free to think and describe problems they are facing. Stop coding and talk to your team and find out what drives them mad and make sure you collect around a single problem statement.

In my opinion, a team offsite, in a foreign city, in the best environment for this to happen.

2. Be problem focused not solution-driven

It really helped my team to keep the project problem-focused rather than solution-driven. While many of us were tempted to be more ambitious and say we wanted to use React.js or TypeScript, being problem-focused allowed us the flexibility to do whatever we felt was important for the project at any given time to be seen as a success.

One unexpected output of our work, which I wrote about recently, was exposing JavaScript errors. Before we began, the rewrite had us a little nervous and our infrastructure is lacking a system for capturing JavaScript errors. Thanks to being a problem-focused project, we had the flexibility to incorporate this into the pproject. After some discussion, we decided to build the smallest possible thing we could using existing infrastructure — a client-side error counter. While not ideal and not allowing us to understand what the errors were, this allowed us to get a sense of whether changes to the site are introducing bugs to users and provides us with a lovely graph to monitor with motivating numbers. It also gave us data for a future technical project to pitch to our product team (I can’t wait for our next offsite)!

In addition to this, while we’ve been forced to look at the code, we’ve been noticing ways to improve it and prepare it for a modern future. For instance, we’ve been reducing our reliance on jQuery. While we’re not removing jQuery from our stack just yet, we’ve found inspiration in other efforts to do this such as Github to at least make this a real possibility.

3. Consider refactoring rather than rewriting

Many rewrites have been proposed and attempted at Wikimedia, in particular for the mobile site — proof of concepts exist in the form of a React mobile-clone Weekipedia and project Marvin but the results of these have never materialized in production.

A legacy system is a well battle-tested system. While sometimes a rewrite is necessary, (bridges are a good example!), refactoring is a great way to ensure that something good comes of the work you do and that product sees the value of what you are working on.

We’ve been working on a refactor in a living, breathing codebase, for six months now, and that project is still running and hopefully, you have noticed no difference. If I don’t say so myself, I think that’s a pretty remarkable achievement of my team.

4. Make talking a routine

The most important output for me was the conversations my development team had. Having a big technical project and the time and freedom to oversee it empowered the team to justify a dedicated hour every week to talk about the problem statement.

This project, while not giving us shiny technologies, has allowed us to have many conversations about many of the patterns popular libraries encourage such as composition over inheritance, Higher Order Components and dumb components to name a few. It allowed us to understand the history of the project; why things are the way they are; talk about what we like in other codebases and what we’d love to learn. It was essentially a focused brown-bag session that gave us all ideas of where we were heading and if we wanted to achieve things how we might get there.

If you work in a technical team and you are spending most of your time coding in some form, I highly encourage you to find at least one hour a week to talk about the higher-level goals of your work as a team. What one person is struggling with or excited about is likely a larger topic in need of a discussion.

Technical debt is a real problem — recognize it!

Much of Wikipedia’s code is not the most appealing of technology stacks for front-end developers. We don’t use any well-known frameworks. My experience has shown that new hires can take many months to become fully effective, and existing hires are susceptible to getting frustrated with it.

That said, I believe strongly that in a decade from now, much of software engineering is going to be about fixing codebases that were built in this era by teams understandably building irresponsibly in order to hit deadlines or create that first minimal viable product, so I think getting exposure to mature codebases and finding ways to refactor them responsibly is an enlightening and rewarding experience.

While hard, it’s our duty as engineers and product owners to explain and understand why cutting a corner is a bad idea and technical debt is a problem.

We should all talk more — to our existing team members and future team members. Good engineers should be explaining the problems they are solving before building (it might have already been solved!) and take time to document complicated or hacky code in great detail! We shouldn’t be afraid to write long and lengthy commit messages about what we are doing, no matter how obvious it might seem at the time.

We should all strive to ask each other for help and opinions in code reviews and outside code reviews. We should communicate about what we don’t like and what we do like. We should listen to the frustrated junior developers in the team and the conservative senior engineers that have seen it all before and find problems and solutions.

Someone is going to have to live in our code and suffer our exact same problems when we are long gone and ultimately it's the user who suffers from that. Let's remember that next time we decide to speed up a software delivery.

Footnotes

If you want to work with a codebase different from what you may be used to consider cloning our codebase and submitting a patch or two.

If you want to learn more about what we did with Webpack and what we were doing for, please read my other article Migrating code from ResourceLoader to Webpack

¹ [For the record, adoption of new libraries is not impossible (in fact a PHP Vue templating library has just been created with plans for adoption in Wikidata.org)!]

How we tackled technical debt at Wikipedia was originally published in Down the Rabbit Hole on Medium, where people are continuing the conversation by highlighting and responding to this story.

What happens when you have a bunch of JavaScript developers frustrated by not having any JavaScript error reporting in production? They find an unexpected workaround using the tooling they have available.

Coding in the dark

Wikipedia and its sister sites currently have no JavaScript error reporting. We have been using logstash for tracking server errors, but due to the fact we are a small volunteer-based non-profit supporting a top-10 website, some essential parts of the infrastructure end up being de-prioritized. As a result, every change we make has a high risk of causing errors. We lean heavily on JavaScript unit tests, Selenium integration tests, and Jenkins automation servers to prevent this as much as possible, but we can never be 100% sure that what we ship is bug-free.

Our team is currently in the process of investing in the front-end architecture of our mobile sites. The work is based on some tooling experiments that the same team performed earlier during the implementation of the page previews feature, nicely summarized by Joaquin Hernandez. While many of the ideas and lessons learned in that project apply here, there was one big difference that concerned us — rather than building something new we were refactoring a large codebase with technical debt that already lived in production.

The lack of client-side error reporting was concerning us, as the changes we were making were quite ambitious, and we were concerned that while making those changes, the likelihood of bugs would increase. Sentry was identified as a potential solution to this problem back in June 2015, but the work involved in that is non-trivial and spans many teams of various disciplines and we haven’t yet been able to find the bandwidth to work on this.

Rather than being put off by this, our team realized we could use our existing infrastructure to somewhat help with this situation. We currently use statsv (an HTTP beacon endpoint) to send data to Graphite (a real-time graph) as a lightweight way of collecting interesting data about our clients. We use this to plot Grafana graphs that provide at a glance metrics to allow monitoring of things such as opting into our mobile beta and errors with A/B test instrumentation. We check these often using a chore rota. We figured, that at the very least, we could count JavaScript errors and use that as a metric for the quality of our output. On the long-term, having such data would allow us to put together a case study detailing the importance of such tooling in our infrastructure, by linking it back to our users and the initiatives our donors were kindly funding.

The code was simple — if an error happens on the client, increment a counter.

Coding with the statsv light on

Turning the error logging on was a little daunting, as we had no idea how buggy our mobile sites were. We limited our change to the mobile site only (we run a separate mobile site, so were able to separate this change from traffic for our desktop users) and added an off switch in case the situation was direr than our optimism hoped.

We enabled error counting on the 18th October and the errors starting rolling in (30,000 in the first five minutes). Our page views tend to fluctuate during a week, so we waited to collect a week's worth of data before drawing any conclusions. Within a week, the errors had peaked at 50,000 in a five minute period, so we now had a baseline. While high our site receives billions of monthly mobile page views so that’s not too bad.

However, then, on the 26th October, we had a new problem to solve.

On the 26th, the errors spiked. As part of our team chores, we opened a new bug ticket to track the problem. On average, we were now seeing 50,000 every five minutes — several magnitudes more than before. It was clear that something had broken, but we had no idea what had broken. After all, all we were doing was counting errors. We had no knowledge of the stack traces which would help us identify what code was causing the error and didn’t even know the language, URL or web browser it was happening on! Obviously, we knew this before enabling this metric and supposedly had made peace with this, only there was one thing we hadn’t counted on. We were developers...

Using the tooling we had to track down the error

For some reason, whenever hit with a problem, with the odds stacked against me, I’m always drawn to the childhood classic “We’re going on a bear hunt” by Michael Rosen (adapted from an American folk song). That book instilled into me at an early age, that any problem can be overcome by thinking outside the box, even the unusual practice of hunting bears. If you’re a parent of young children, I thoroughly recommend you grab a copy.

When I became a developer, I would often channel Michael Rosen during conversations about scope creep or edge cases. If you’ve ever worked with a team of developers, especially if you are a product owner, you will have noticed that we are a curious bunch, and if you distract us with a shiny problem, we’ll bite, and even if you don’t want us to… we’ll find the reason.

So, hit with the problem of an additional 30,000 client-side errors over the period of five minutes, even though early on, we recognized that counting errors would only tell us “when” an error occurred rather than “why”, it seemed worthy of further investigation. The only question was how.

A work colleague outside our team in a casual conversation pointed out that statsv endpoints were GET requests, and thus went through our infrastructure to count page views. Armed with this information, it seemed we’d be able to glean some useful information about the errors we were seeing, based on data we were collecting for the purposes of monitoring site traffic.

Although we were only counting client-side errors, each of the requests being made to our statsv beacon to count these errors had a host of meta-data associated — notably, the Wikipedia language and domain that was being used; the referrer URL and most importantly the user browser agent.

By looking at the client side errors for a single day, we could run a series of Apache Hive queries to give more context on the errors we were collecting for a single day. I did just that, to see if I could glean any further information. I ran the following Hive queries:

#ALL ERRORS grouped by project:

select uri_host,namespace_id,count(*) from webrequest where day = 30 and month = 10 and year = 2018 and uri_path LIKE '%beacon%' and uri_query LIKE "%WebClientError%" group by uri_host,namespace_id;

# all errors grouped by user agent:
select uri_host,user_agent_map['os_family'],user_agent_map['browser_family'], count(*) from webrequest where day = 30 and month = 10 and year = 2018 and uri_path LIKE '%beacon%' and uri_query LIKE "%WebClientError%" group by uri_host,user_agent_map['os_family'], user_agent_map['browser_family'];

# all errors grouped by referrer:
select uri_host,referer, count(*) from webrequest where day = 30 and month = 10 and year = 2018 and uri_path LIKE '%beacon%' and uri_query LIKE "%WebClientError%" group by uri_host,referer;

Thankfully, these queries did help. While, grouping the errors by project did not shed much light, it ruled out the possibility that the bug was based on language or language-specific customizations. Likewise, grouping by referrer (e.g. the URL the user was on at the time they experienced the bug) showed no real trend — there were no problematic pages triggering more errors than the other. The user agent was the most valuable, the query helped me notice that the majority of the issues were coming from iOS (the iPhone), so I drilled down further looking at iOS by running the following Hive query:

select uri_host,user_agent_map['os_family'],user_agent_map['browser_family'],user_agent_map['browser_major'], user_agent_map['os_major'], count(*) from webrequest where day = 30 and month = 10 and year = 2018 and user_agent_map['browser_family'] = 'Mobile Safari' and uri_path LIKE '%beacon%' and uri_query LIKE "%WebClientError%" group by uri_host,user_agent_map['os_family'], user_agent_map['browser_family'], user_agent_map['browser_major'], user_agent_map['os_major'];

This Hive query showed me clearly that the bug was most prominent in iOS Safari (in fact 70% of all our errors were coming from this browser). Using browserstack (a tool that allows cross-browser testing), and my knowledge of the mobile stack (and code that had recently changed in the deploy that led to the bug), I took a look at one of the most common referrers and quickly I was able to replicate the bug quite quickly (on page load) — and thus hone in on the bug — an issue with some JavaScript templating code.

A happy ending

The bug was squashed promptly and the client side errors reverted back to its “normal” baseline shortly after we deployed the fix. We all sighed in relief.

One of the outcomes of our mobile site investment project is to improve test coverage. Not only had we shown that our error counting could identify new bugs, but we’d shown the impact of identifying and fixing existing bugs.

Since that issue in October, we’ve also identified two other issues via the same debugging method. In January we debugged and squash a much more impactful issue related to cached HTML as well as an issue with a volunteer-run banner campaign.

Hopefully, sometime soon, I can write about how Wikipedia has zero client-side errors.

Takeaways

Obviously, counting errors with statsv is not a sustainable way to run a website, and if you are being sent to this URI being told that you need to install statsv to count errors immediately, please kindly point that person to this section. If you want to count errors and have the resourcing to do such a thing, I highly recommend Sentry and/or its client-side SDK. The Wikimedia Foundation needs to get Sentry up and running, and we will catch up with doing that at some point in the future as soon as we can herd the correct team of people together.

So, what is the point of this article, if not to tell you about how to debug errors with statvs? Maybe it’s a reminder that engineers are paid to solve problems and that sometimes, being an engineer involves thinking outside the box and using the tooling you already have in creative unexpected ways. When we set out to count errors, as engineers we were not satisfied that it was the right way forward, but given it was low effort, we’ve already justified that work and more.

In this particular case, doing something unexpected with inappropriate tooling, was a great way to help non-technical people understand why JavaScript errors were something we should be capturing by making a hidden and unknown problem visible.

As anyone who has gone on a bear hunt will tell you, mountain-like problems can always be climbed given enough time and in absence of a bridge through the middle, while it’s tempting to give up altogether, sometimes we need to find other ways to get our message through.

We’re going on a bug hunt (we’re not scared!) was originally published in Down the Rabbit Hole on Medium, where people are continuing the conversation by highlighting and responding to this story.

MediaWiki, the software that powers Wikipedia, is afflicted with the beautiful dichotomy that plagues many open source projects — it was built over a decade ago with lots of enthusiasm and many fantastic ideas, but with fewer concerns about code quality and long-term maintenance.

It should not come as a surprise that MediaWiki, while powering one of the most important resources on the web, has also accumulated significant amounts of technical debt. To cope with this, my team at the Wikimedia Foundation — the organization now charged with stewarding MediaWiki — found a ritual that allowed us to decrease the numbers of errors and regressions, improve our efficiency, maintain a certain level of quality, and evolve and bond as a team. Here’s our story.

Working at scale

My core team consists of a tech lead (myself), a product owner, an engineering manager, a designer, and four other developers. The eight of us maintain the entire Wikipedia mobile website, the page previews feature that we shipped to desktop several months ago, and other bits of critical but relatively stable infrastructure. We’re actually spread quite thin and keeping on top of problems can be pretty difficult.

Even so, I think we do an incredible job.

Infrastructure and tooling

Since Wikipedia’s conception in 2001 (17 years ago!), the supporting infrastructure has gone through many positive transformations. It was only in 2009, with a new strategic plan and growth of the Foundation, that the infrastructure to support our MediaWiki based production started to become fleshed out. Integration tests (Selenium) were added (conceived in 2012); a production-like beta cluster was added (2012); Logstash was added for server-side error reporting; a performance team was created to set up infrastructure for monitoring and improving performance (WebPageTest and tooling to collect NavigationTiming data from real users); and we switched from Bugzilla to Phabricator to improve how we curate bug reports.

This tooling has been essential for us to measure the health of the site and keep the team accountable. The tooling is by no means complete. Notably, there is no infrastructure for collecting of client-side errors, although Sentry has been proposed.

Infrastructure is only good if you look at it

Despite the fact we have all these tools to aid us to maintain quality, the tooling is only good if you use it. In the early days, if browser tests failed, emails would be sent to point people hoping that they would deal with them; Logstash errors would only go noticed if they caused trouble during deployments or someone had taken the initiative to look at them; it was up to bug wranglers to make sure bug reports got to the right people; and up to the performance team to tell product teams that their work was causing unacceptable performance regressions.

It would be great if failing browser tests blocked deployments, but they don’t.

It would be great if error spikes in server-side errors automatically created a bug and routed it to the right person, but they don’t.

When you work for a non-profit maintaining a site at scale and trying to live up to its values of openness and transparency, one of the big downsides is that you can’t afford every single tool offered to you and even if you could you don’t necessarily have enough people to stand it up and maintain it.

These kind of problems were causing a lot of stress to people in my team as our team cares a lot about our product and our output.

For a period of time my team often felt:

• vexed when nobody had noticed a regression in production caught by our browser tests.
• concerned by any unusual and un-investigated errors in Logstash.
• inadequate when we began drowning in triaging tickets from community members
• upset about all the unknown client-side errors that our users were surely experiencing but not able to tell us.

During one of our team retrospectives, (which we have every two weeks), we shared these frustrations and why we cared about them. Finally, we talked about ways of sharing that burden.

Enter the chores list!

Our scrum master, Max Binder, proposed a solution back in December 2016 (2 years ago!) that had been enjoying success in our iOS team (originally conceived by Josh Minor).

The solution was to define a list of things that were important to the team, which we named the Chore list. The plan was each team member would take turns to go through that list daily to check things in the list were of the standard we expected. A time-box was set up to set the expectation that we shouldn’t spend too long going through that every growing list and to help us prioritise what needed to be looked at at any given time.

Executing a chores list

When we began the experiment, we used a mailing list to report on the current state. Many of these emails never got replies, so the sender was never clear if someone read and digested them, but when we brought this up in our team retrospectives what we actually found was people were using them and finding them useful!

When there were conversations on the emails, these conversations were quite useful, even if it was just an opportunity for our product owner to understand the severity of the latest regression and understand its priority or to explain a spike in a graph we were seeing. Sometimes the chores subject was something as simple as “Chores 13th July — everything is awesome EOM”.

The doing of chores, like any list of chores, has required discipline and engagement from the team.

There have been many days where people “forgot” or “didn’t have time”. We have found that flagging (without blame!) this kind of neglect in team rituals such as retrospectives and stand-ups has helped remedy the problem quite quickly. The important thing is that as a whole the team sees the benefit in this ritual and is keen to keep the ritual alive.

It’s been really important for the team to adapt as our responsibilities have shifted and we’ve encountered new experiences. On one occasion our analytics team flagged to us that we were causing a large chunk of errors that was clogging up their analytics pipeline. They pointed us to a graph measuring errors that we didn’t know existed. After fixing this problem, we added a new chore to the list to check that exact graph to make sure it didn’t happen again.

Even our chores themselves have got maintenance — in particular, many of the queries we use to identify tasks we are responsible for working on; identify code that needs reviewing and identify errors that we are on the hook for fixing have been tweaked and improved for the next person.

The process also got maintenance. The email format we used changed in various ways over time until eventually we settled on a wiki page.

With email, we found several benefits:

• it was useful to pull in people from other teams
• we got to send each other GIFs

However we faced challenges with some basic human problems.

• We struggled to agree on whether the subject header date format should be “Chores: Thursday, 16th March”, or “Chores 16th March” or “Chores 16/03/18” (you get the idea).
• The lack of a consistent subject header, made it hard for us to access historical information
• This also made it hard for us to set up the appropriate email filters for a daily email.
• Half the team really liked emails laid out in rich text format tables while the other half wanted plain text.

These small problems, aside, more crucially, we were noticing larger problems.

• We were duplicating efforts in debugging the same bugs our browser tests had uncovered
• Certain chores were being neglected for weeks without us realizing

We kicked ourselves a little, when we realized that switching to a wiki format removed all of those issues. We maintained a single email thread to allow the benefit of side chat and GIF usage (very important!) and in addition to this we got to dog food our products.

Why every team should have its own chore list

One of the greatest benefits of sharing this load has been the shared responsibility and awareness of what quality means in our team.

Two years later, we still use this list, but the list like any good wiki page is no reflection of its original state, having had over 100 edits from a variety of team members.

Some of the measurable benefits of having a chores list include:

• When we started out there were hundreds of server-side errors being experienced by users in obscure parts of our interface. Now, two years later, it’s quite rare for one to show up there, which feels good.
• Bugs are rarely making it to production (we’ve experienced less emergency hot fixing)
• We’ve become more aware of how banners from our fundraising team and volunteers severely impact our performance (e.g. this bug)
• Our backlog has shrunk from around 400 to 153 open bugs thanks to a shared responsibility in triaging.

While less measurable, the shared responsibility has had a positive impact on the team dynamic.

It’s been beautiful to watch the team self-organize around defining code quality.

Our designer, added a chore to help prioritizes and organize design the backlog. After we launched a PDF rendering service, one of our engineers added a chore to monitor npmjs.com for new package releases. After several rounds of vacations, meaning chores were not being done, the team in retro decided to add a chore to inform the group who was next. This made it transparent not only that chores were not being done, but who was not doing them to the rest of the team.

If your software development team doesn’t have one already, I think a chores list is a great and inexpensive way to get your team thinking constantly about quality and constantly striving to improve. Having one has sure helped mine.

Footnotes

Our chores list is entirely public, so feel free to dig into all the interesting data we monitor, including the grafana dashboard. Let me know in the comments if your team does (or is thinking of doing) something similar. I’d love to hear from you!

Why all developers should do their chores was originally published in Down the Rabbit Hole on Medium, where people are continuing the conversation by highlighting and responding to this story.

On the 1st day of Brexmas
Theresa May gave to me:
an un-recoverable economy

On the 2nd day of Brexmas
Theresa May gave to me:
Two independence referendums
and an un-recoverable economy

On the 3rd day of Brexmas
Theresa May gave to me:
Free movement gone
Two independence referendums
and an un-recoverable economy

On the 4th day of Brexmas
Theresa May gave to me:
False unkept promises
Free movement gone
Two independence referendums
and an un-recoverable economy

On the 5th day of Brexmas
Theresa May gave to me:
High import taxes
False unkept promises
Free movement gone
Two independence referendums
and an un-recoverable economy

On the 6th day of Brexmas
Theresa May gave to me:
Sexy blue and gold passports
High import taxes
False unkept promises
Free movement gone
Two independence referendums
and an un-recoverable economy

On the 7th day of Brexmas
Theresa May gave to me:
Severed ties with Brussels
Sexy blue and gold passports
High import taxes
False unkept promises
Free movement gone
Two independence referendums
and an un-recoverable economy

On the 8th day of Brexmas
Theresa May gave to me:
Hate speech a rising
Severed ties with Brussels
Sexy blue and gold passports
High import taxes
False unkept promises
Free movement gone
Two independence referendums
and an un-recoverable economy

On the 9th day of Brexmas
Theresa May gave to me:
Nine factories closing
Hate speech a rising
Severed ties with Brussels
Sexy blue and gold passports
High import taxes
False unkept promises
Free movement gone
Two independence referendums
and an un-recoverable economy

On the 10th day of Brexmas
Theresa May gave to me:
Ten Downing weeping
Nine factories closing
Hate speech a rising
Severed ties with Brussels
Sexy blue and gold passports
High import taxes
False unkept promises
Free movement gone
Two independence referendums
and an un-recoverable economy

On the 11th day of Brexmas
Theresa May gave to me:
Eleven politicians a sniping
Ten Downing weeping
Nine factories closing
Hate speech a rising
Severed ties with Brussels
Sexy blue and gold passports
High import taxes
False unkept promises
Free movement gone
Two independence referendums
and an un-recoverable economy

On the 12th day of Brexmas
Theresa May gave to me:
Twelve plus rebelling Tories
Eleven politicians a sniping
Ten Downing weeping
Nine factories closing
Hate speech a rising
Severed ties with Brussels
Sexy blue and gold passports
High import taxes
False unkept promises
Free movement gone
Two independence referendums
and an un-recoverable economy.