google reader refugee.
1798 stories

What Am Container


Perhaps you are a software developer.

Perhaps, as a developer, you have recently become familiar with the term "containers".

Perhaps you have heard containers described as something like "LXC, but better", "an application-level interface to cgroups" or "like virtual machines, but lightweight", or perhaps (even less usefully), a function call. You've probably heard of "docker"; do you wonder whether a container is the same as, different from, or part of an Docker?

Are you are bewildered by the blisteringly fast-paced world of "containers"? Maybe you have no trouble understanding what they are - in fact you might be familiar with a half a dozen orchestration systems and container runtimes already - but frustrated because this seems like a whole lot of work and you just don't see what the point of it all is?

If so, this article is for you.

I'd like to lay out what exactly the point of "containers" are, why people are so excited about them, what makes the ecosystem around them so confusing. Unlike my previous writing on the topic, I'm not going to assume you know anything about the ecosystem in general; just that you have a basic understanding of how UNIX-like operating systems separate processes, files, and networks.1

At the dawn of time, a computer was a single-tasking machine. Somehow, you'd load your program into main memory, and then you'd turn it on; it would run the program, and (if you're lucky) spit out some output onto paper tape.

When a program running on such a computer looked around itself, it could "see" the core memory of the computer it was running on, any attached devices, including consoles, printers, teletypes, or (later) networking equipment. This was of course very powerful - the program had full control of everything attached to the computer - but also somewhat limiting.

This mode of addressing hardware is limiting because it meant that programs would break the instant you moved them to a new computer. They had to be re-written to accommodate new amounts and types of memory, new sizes and brands of storage, new types of networks. If the program had to contain within itself the full knowledge of every piece of hardware that it might ever interact with, it would be very expensive indeed.

Also, if all the resources of a computer were dedicated to one program, then you couldn't run a second program without stomping all over the first one - crashing it by mangling its structures in memory, deleting its data by overwriting its data on disk.

So, programmers cleverly devised a way of indirecting, or "virtualizing", access to hardware resources. Instead of a program simply addressing all the memory in the whole computer, it got its own little space where it could address its own memory - an address space, if you will. If a program wanted more memory, it would ask a supervising program - what we today call a "kernel" - to give it some more memory. This made programs much simpler: instead of memorizing the address offsets where a particular machine kept its memory, a program would simply begin by saying "hey operating system, give me some memory", and then it would access the memory in its own little virtual area.

In other words: memory allocation is just virtual RAM.

Virtualizing memory - i.e. ephemeral storage - wasn't enough; in order to save and transfer data, programs also had to virtualize disk - i.e. persistent storage. Whereas a whole-computer program would just seek to position 0 on the disk and start writing data to it however it pleased, a program writing to a virtualized disk - or, as we might call it today, a "file" - first needed to request a file from the operating system.

In other words: file systems are just virtual disks.

Networking was treated in a similar way. Rather than addressing the entire network connection at once, each program could allocate a little slice of the network - a "port". That way a program could, instead of consuming all network traffic destined for the entire machine, ask the operating system to just deliver it all the traffic for, say, port number seven.

In other words: listening ports are just virtual network cards.

Getting bored by all this obvious stuff yet? Good. One of the things that frustrates me the most about containers is that they are an incredibly obvious idea that is just a logical continuation of a trend that all programmers are intimately familiar with.

All of these different virtual resources exist for the same reason: as I said earlier, if two programs need the same resource to function properly, and they both try to use it without coordinating, they'll both break horribly.2

UNIX-like operating systems more or less virtualize RAM correctly. When one program grabs some RAM, nobody else - modulo super-powered administrative debugging tools - gets to use it without talking to that program. It's extremely clear which memory belongs to which process. If programs want to use shared memory, there is a very specific, opt-in protocol for doing so; it is basically impossible for it to happen by accident.

However, the abstractions we use for disks (filesystems) and network cards (listening ports and addresses) are significantly more limited. Every program on the computer sees the same file-system. The program itself, and the data the program stores, both live on the same file-system. Every program on the computer can see the same network information, can query everything about it, and can receive arbitrary connections. Permissions can remove certain parts of the filesystem from view (i.e. programs can opt-out) but it is far less clear which program "owns" certain parts of the filesystem; access must be carefully controlled, and sometimes mediated by administrators.

In particular, the way that UNIX manages filesystems creates an environment where "installing" a program requires manipulating state in the same place (the filesystem) where other programs might require different state. Popular package managers on UNIX-like systems (APT, RPM, and so on) rarely have a way to separate program installation even by convention, let alone by strict enforcement. If you want to do that, you have to re-compile the software with ./configure --prefix to hard-code a new location. And, fundamentally, this is why the package managers don't support installing to a different place: if the program can tell the difference between different installation locations, then it will, because its developers thought it should go in one place on the file system, and why not hard code it? It works on their machine.

In order to address this shortcoming of the UNIX process model, the concept of "virtualization" became popular. The idea of virtualization is simple: you write a program which emulates an entire computer, with its own storage media, network devices, and then you install an operating system on it. This completely resolves the over-sharing of resources: a process inside a virtual machine is in a very real sense running on a different computer than programs running on a different virtual machine on the same physical device.

However, virtualiztion is also an extremly heavy-weight blunt instrument. Since virtual machines are running operating systems designed for physical machines, they have tons of redundant hardware-management code; enormous amounts of operating system data which could be shared with the host, but since it's in the form of a disk image totally managed by the virtual machine's operating system, the host can't really peek inside to optimize anything. It also makes other kinds of intentional resource sharing very hard: any software to manage the host needs to be installed on the host, since if it is installed on the guest it won't have full access to the host's hardware.

I hate using the term "heavy-weight" when I'm talking about software - it's often bandied about as a content-free criticism - but the difference in overhead between running a virtual machine and a process is the difference between gigabytes and kilobytes; somewhere between 4-6 orders of magnitude. That's a huge difference.

This means that you need to treat virtual machines as multi-purpose, since one VM is too big to run just a single small program. Which means you often have to manage them almost as if they were physical harware.

When we run a program on a UNIX-like operating system, and by so running it, grant it its very own address space, we call the entity that we just created a "process".

This is how to understand a "container".

A "container" is what we get when we run a program and give it not just its own memory, but its own whole virtual filesystem and its own whole virtual network card.

The metaphor to processes isn't perfect, because a container can contain multiple processes with different memory spaces that share a single filesystem. But this is also where some of the "container ecosystem" fervor begins to creep in - this is why people interested in containers will religiously exhort you to treat a container as a single application, not to run multiple things inside it, not to SSH into it, and so on. This is because the whole point of containers is that they are lightweight - far closer in overhead to the size of a process than that of a virtual machine.

A process inside a container, if it queries the operating system, will see a computer where only it is running, where it owns the entire filesystem, and where any mounted disks were explicitly put there by the administrator who ran the container. In other words, if it wants to share data with another application, it has to be given the shared data; opt-in, not opt-out, the same way that memory-sharing is opt-in in a UNIX-like system.

So why is this so exciting?

In a sense, it really is just a lower-overhead way to run a virtual machine, as long as it shares the same kernel. That's not super exciting, by itself.

The reason that containers are more exciting than processes is the same reason that using a filesystem is more exciting than having to use a whole disk: sharing state always, inevitably, leads to brokenness. Opt-in is better than opt-out.

When you give a program a whole filesystem to itself, sharing any data explicitly, you eliminate even the possibility that some other program scribbling on a shared area of the filesystem might break it. You don't need package managers any more, only package installers; by removing the other functions of package managers (inventory, removal) they can be radically simplified, and less complexity means less brokenness.

When you give a program an entire network address to itself, exposing any ports explicitly, you eliminate even the possibility that some rogue program will expose a security hole by listening on a port you weren't expecting. You eliminate the possibility that it might clash with other programs on the same host, hard-coding the same port numbers or auto-discovering the same addresses.

In addition to the exciting things on the run-time side, containers - or rather, the things you run to get containers, "images"3, present some compelling improvements to the build-time side.

On Linux and Windows, building a software artifact for distribution to end-users can be quite challenging. It's challenging because it's not clear how to specify that you depend on certain other software being installed; it's not clear what to do if you have conflicting versions of that software that may not be the same as the versions already available on the user's computer. It's not clear where to put things on the filesystem. On Linux, this often just means getting all of your software from your operating system distributor.

You'll notice I said "Linux and Windows"; not the usual (linux, windows, mac) big-3 desktop platforms, and I didn't say anything about mobile OSes. That's because on macOS, Android, iOS, and Windows Metro, applications already run in their own containers. The rules of macOS containers are a bit weird, and very different from Docker containers, but if you have a Mac you can check out ~/Library/Containers to see the view of the world that the applications you're running can see. iOS looks much the same.

This is something that doesn't get discussed a lot in the container ecosystem, partially because everyone is developing technology at such a breakneck pace, but in many ways Linux server-side containerization is just a continuation of a trend that started on mainframe operating systems in the 1970s and has already been picked up in full force by mobile operating systems.

When one builds an image, one is building a picture of the entire filesystem that the container will see, so an image is a complete artifact. By contrast, a package for a Linux package manager is just a fragment of a program, leaving out all of its dependencies, to be integrated later. If an image runs on your machine, it will (except in some extremely unusual circumstances) run on the target machine, because everything it needs to run is fully included.

Because you build all the software an image requires into the image itself, there are some implications for server management. You no longer need to apply security updates to a machine - they get applied to one application at a time, and they get applied as a normal process of deploying new code. Since there's only one update process, which is "delete the old container, run a new one with a new image", updates can roll out much faster, because you can build an image, run tests for the image with the security updates applied, and be confident that it won't break anything. No more scheduling maintenance windows, or managing reboots (at least for security updates to applications and libraries; kernel updates are a different kettle of fish).

That's why it's exciting. So why's it all so confusing?5

Fundamentally the confusion is caused by there just being way too many tools. Why so many tools? Once you've accepted that your software should live in images, none of the old tools work any more. Almost every administrative, monitoring, or management tool for UNIX-like OSes depends intimately upon the ability to promiscuously share the entire filesystem with every other program running on it. Containers break these assumptions, and so new tools need to be built. Nobody really agrees on how those tools should work, and a wide variety of forces ranging from competitive pressure to personality conflicts make it difficult for the panoply of container vendors to collaborate perfectly4.

Many companies whose core business has nothing to do with infrastructure have gone through this reasoning process:

  1. Containers are so much better than processes, we need to start using them right away, even if there's some tooling pain in adopting them.
  2. The old tools don't work.
  3. The new tools from the tool vendors aren't ready.
  4. The new tools from the community don't work for our use-case.
  5. Time to write our own tool, just for our use-case and nobody else's! (Which causes problem #3 for somebody else, of course...)

A less fundamental reason is too much focus on scale. If you're running a small-scale web application which has a stable user-base that you don't expect a lot of growth in, there are many great reasons to adopt containers as opposed to automating your operations; and in fact, if you keep things simple, the very fact that your software runs in a container might obviate the need for a system-management solution like Chef, Ansible, Puppet, or Salt. You should totally adopt them and try to ignore the more complex and involved parts of running an orchestration system.

However, containers are even more useful at significant scale, which means that companies which have significant scaling problems invest in containers heavily and write about them prolifically. Many guides and tutorials on containers assume that you expect to be running a multi-million-node cluster with fully automated continuous deployment, blue-green zero-downtime deploys, a 1000-person operations team. It's great if you've got all that stuff, but building each of those components is a non-trivial investment.

So, where does that leave you, my dear reader?

You should absolutely be adopting "container technology", which is to say, you should probably at least be using Docker to build your software. But there are other, radically different container systems - like Sandstorm - which might make sense for you, depending on what kind of services you create. And of course there's a huge ecosystem of other tools you might want to use; too many to mention, although I will shout out to my own employer's docker-as-a-service Carina, which delivered this blog post, among other things, to you.

You shouldn't feel as though you need to do containers absolutely "the right way", or that the value of containerization is derived from adopting every single tool that you can all at once. The value of containers comes from four very simple things:

  1. It reduces the overhead and increases the performance of co-locating multiple applications on the same hardware,
  2. It forces you to explicitly call out any shared state or required resources,
  3. It creates a complete build pipeline that results in a software artifact that can be run without special installation or set-up instructions (at least, on the "software installation" side; you still might require configuration, of course), and
  4. It gives you a way to test exactly what you're deploying.

These benefits can combine and interact in surprising and interesting ways, and can be enhanced with a wide and growing variety of tools. But underneath all the hype and the buzz, the very real benefit of containerization is basically just that it is fixing a very old design flaw in UNIX.

Containers let you share less state, and shared mutable state is the root of all evil.

  1. If you have a more sophisticated understanding of memory, disks, and networks, you'll notice that everything I'm saying here is patently false, and betrays an overly simplistic understanding of the development of UNIX and the complexities of physical hardware and driver software. Please believe that I know this; this is an alternate history of the version of UNIX that was developed on platonically ideal hardware. The messy co-evolution of UNIX, preemptive multitasking, hardware offload for networks, magnetic secondary storage, and so on, is far too large to fit into the margins of this post. 

  2. When programs break horribly like this, it's called "multithreading". I have written some software to help you avoid it. 

  3. One runs an "executable" to get a process; one runs an "image" to get a container. 

  4. Although the container ecosystem is famously acrimonious, companies in it do actually collaborate better than the tech press sometimes give them credit for; the Open Container Project is a significant extraction of common technology from multiple vendors, many of whom are also competitors, to facilitate a technical substrate that is best for the community. 

  5. If it doesn't seem confusing to you, consider this absolute gem from the hilarious folks over at CircleCI. 

Read the whole story
1 day ago
Melbourne, Australia
Share this story

#913: “You don’t have to quit your day job right this second to follow a creative dream.”

1 Share

Dear Captain,

I’m in my early 30’s. Having spent my 20’s doing the ‘right’ things (college -> law school -> office job), I have now recognised what was clear all along, namely that this is not for me at all, and that maybe that’s okay. I’ve also realised that maybe it’s okay to not be making the maximum amount I possibly could be, and okay to say ‘no thank you’ to the budding career I have zero interest in in favour of pursuing my passions.

Passions, of course, don’t pay very well, certainly not at first and possibly never. If I quit right now, today, I would be living hand to mouth with virtually no safety net. If I hold on for another 22 months, then I would have a very substantial safety net, enough to cushion me for a decade or more to come (I would still need to work to feed myself, but I would be able to absorb a good number of unexpected financial blows before going into crisis mode), plus put me in a better position in old age. There is no in between here.

Herein lies the quandary: I could die in six months’ time, in which case I would rather quit now and take my chances. On the other hand, if I did quit now and then didn’t die shortly afterwards, 2020!Me’s life is likely to be significantly more precarious and uncomfortable than it would be if Present!Me stays for the 22 months. I should add here that I tend to be a lot more productive in the creative area I want to pursue when I feel immediately secure, so this isn’t even just about my own comfort, but potentially impacts the quality of the work I want to do (and of course the point of this whole exercise is to give myself a better chance of producing quality work).

Knowing myself, I will probably do the 22 months. I don’t hate my job, have no reason to expect my imminent demise beyond the fact that it could happen to anyone, and the job comes with a fixed end date at the end of the period, so I’m less likely to fall into the trap of just putting it off and putting it off until it becomes too late. I’m just having a hard time reconciling myself to the ‘what if’ part right now right now. Help please?

My recommendation is that you work at your lucrative career for 22 more months and create the financial cushion that will give you the maximum amount of choices and freedom to do the thing you want to do. You will be setting yourself up to be maximally creative and happy if you have that cushion. We romanticize artistic struggle, but the plain truth is: POVERTY SUCKS. It is tedious and draining and boring and awful and limiting. It’s not cooler or liberating or inspiring. Don’t choose it if you don’t have to.

Since your email subject line was “I could die tomorrow, or I might not die for 50 years,” that’s not my only recommendation. You are thinking in terms of LIFE and DEATH and PRECIOUS TIME YOU’LL NEVER, EVER GET BACK. You need to get connected to your passions right now, TODAY.

Some ways to do that, listed in no particular order of importance:

  • Collect your heroes. Read Steal Like An Artistand choose your hall of virtual mentors. Whose art makes you want to make art? Dig into their work and read, watch, listen intently.
  • Create rituals. Look into The Artist’s Way by Julia Cameron, and whether or not the “spiritual” language works for you, think about trying out her daily practices (free-writing three pages every morning) and weekly practices (The Artist’s Date, where you spend 1-2 hours by yourself exploring and giving yourself permission to daydream and absorb and create).
  • Get social. What is a professional group, MeetUp group, online community or other social network of people who do what you want to do? Join one of those and practice saying out loud to other people what it is that you want to do. “My name is ______ and I’m an attorney but I also _______ and want to be a ______.
  • Find the local scene. Chances are that where you live someone is doing the thing you want to be doing with your life. Subscribe to the local theater company. Go to the local indie film screening. Get on the list for art openings at galleries, readings, humanities festivals. Sit on boards, volunteer behind the scenes if you can. In a careerist sense this is “building your network.” In a creative sense this is “filling the well.”
  • Give yourself permission to be a beginner. Is there an evening or weekend class in what you want to be doing that you could take? Try something out. Don’t put the pressure on yourself to be immediately supporting yourself with this creative pursuit. Experiment and play.
  • Use your breaks. Do you get vacation time at this nice law job of yours? Can you plan to take some of it specifically around a film festival or artists’ retreat or concert or play you want to see? Plan out your vacation time over the next 22 months and make sure you’re giving yourself regular intervals to recharge and soak up what you want to do.
  • Use your hard-won education and skills in service of your future field. There’s a great organization in Chicago called Lawyers For The Creative Arts. Is there one of those where you live? Creative folks need contracts and all the “boring” paperwork that you’re trying to flee from. This is a way you can make yourself valuable, meet people, and start to transition your life toward where you want it to be.
  • Be vigilant about your finances. Maximize retirement contributions and anything your firm matches. Sock away as much money as you can in your FU fund. Shrink your living expenses so that they’ll be manageable during the transition time when you might not be earning quite as much. If you’ve got fancy health insurance, take advantage of it and get every nice thing you might want done now.

If you can start to connect to the work you want to be doing, I predict one of two things will happen:

a) It will make the next 22 months fly by, and you’ll be able to get through the boring days more easily because you know it’s ending soon and because you have good, creative, fun, nourishing stuff to think about.

b) Work will become even more unbearable by comparison and you’ll start hearing klaxons saying “GET THE FUCK OUT RIGHT NOW” so you will in order to save your own life. You always have that option any time, right?

Try one of the above suggestions, or all, or none as it makes sense for your life. Above all: Start the work. If there is a piece of work that is screaming inside you because it wants to be made, then get started, somehow, some way. That feeling you have right now that says “I have wasted so much time doing things I don’t care about already, how can I waste one second more?” is valuable, so use it, and don’t Don’t wait for 22 months to pass. But also, don’t eff up your finances and make life artificially harder for yourself because you think that “being a creative person” is some magical black & white special category that you have to burn your life down to enter. You’re already that person if you want to be. It’s not either/or, now/never. Just, start. Start in some small way to do the work you want to do. either/or.

Read the whole story
1 day ago
Melbourne, Australia
Share this story

Part-time Power


Background: Y Combinator (YC) is an influential seed accelerator and VC firm founded by Paul Graham and run by Sam Altman. Sam may remember me from the time I counted how many women he follows on Twitter. One of YC’s part-time partners is Peter Thiel, who spoke at the Republican National Convention. He also donated $1.25 million to Trump’s presidential campaign in mid-October after more than a dozen women accused the candidate of sexual assault and Trump once again repeated his calls for imprisonment of five innocent black men. For more details, see Project Include’s post on the topic, or Erica Baker, Nicole Sanchez, and Maciej Cegłowski’s numerous and wise tweets around it.

One of the things I teach in the Ally Skills workshop is a concept in moral philosophy called the Paradox of Tolerance – in short, the one thing a tolerant society must be intolerant of is intolerance. It’s really helped me frame how I’ve been thinking about this situation – to consider whether or not Thiel’s support of Trump puts him into the “intolerable intolerance” camp or not. It wasn’t a particularly tough call for me – were I in Altman’s shoes, I’d ask for Thiel’s resignation. But there’s part of the situation that I haven’t seen addressed anywhere.

When you bring someone into your organization as an advisor/mentor/office-hour-holder (which is what Thiel’s role at YC seems to consist of), you are doing three things:

  • Giving them power over the people in your organization that they are tasked with advising
  • Endorsing their advice as being something that people in your organization should follow
  • Sharing your social capital with them

Now, obviously, Thiel has those first two powers in droves in his various other capacities, but in keeping him on as a “part-time partner”, YC is both saying that they value the advice he can give their founders as well as implicitly giving him a position of power over them – the power of making introductions or not, writing letters of recommendation or not, and so on – the power of a sanctioned mentoring role.

They are also saying that they trust him to not discriminate against the people they are giving him power over – the founders in their program – in ways that are not aligned with YC’s values. Thiel has made it clear through decades of public writing and actions what his values are. He wrote a book called “The Diversity Myth”, for starters. Thiel also considers women having the vote to have “rendered the notion of ‘capitalist democracy’ an oxymoron“. This hits me particularly hard as I can’t vote right now – I am in the US on a visa, not yet a citizen, and as a non-resident can no longer vote in Canada.

One last thing: I stressed for two days about writing this post, knowing that Thiel is willing to fund multi-million dollar lawsuits against his critics. I have no connection to him and he has no other power over me. Imagine how it would feel should any of his mentees need to criticize him.

We all get to make a choice as to what constitutes “intolerable intolerance”. YC has made it clear that Thiel’s actions and words are tolerable enough to them to continue to give him power over people in their organization, and I find this unconscionable.

Read the whole story
1 day ago
Melbourne, Australia
8 days ago
Washington, DC
Share this story

Why I won’t be attending Systems We Love

2 Comments and 3 Shares

Systems We Love is a one day event in San Francisco to talk excitedly about systems computing. When I first heard about it, I was thrilled! I love systems so much that I moved from New Mexico to the Bay Area when I was 23 years old purely so that I could talk to more people about them. I’m the author of the Kernel Hacker’s Bookshelf series, in which I enthusiastically described operating systems research papers I loved in the hopes that systems programmers would implement them. The program committee of Systems We Love includes many people I respect and enjoy being around. And the event is so close to me that I could walk to it.

So why I am not going to Systems We Love? Why am I warning my friends to think twice before attending? And why am I writing a blog post warning other people about attending Systems We Love?

The answer is that I am afraid that Bryan Cantrill, the lead organizer of Systems We Love, will say cruel and humiliating things to people who attend. Here’s why I’m worried about that.

I worked with Bryan in the Solaris operating systems group at Sun from 2002 to 2004. We didn’t work on the same projects, but I often talked to him at the weekly Monday night Solaris kernel dinner at Osteria in Palo Alto, participated in the same mailing lists as him, and stopped by his office to ask him questions every week or two. Even 14 years ago, Bryan was one of the best systems programmers, writers, and speakers I have ever met. I admired him and learned a lot from him. At the same time, I was relieved when I left Sun because I knew I’d never have to work with Bryan again.

Here’s one way to put it: to me, Bryan Cantrill is the opposite of another person I admire in operating systems (whom I will leave unnamed). This person makes me feel excited and welcome and safe to talk about and explore operating systems. I’ve never seen them shame or insult or put down anyone. They enthusiastically and openly talk about learning new systems concepts, even when other people think they should already know them. By doing this, they show others that it’s safe to admit that they don’t know something, which is the first step to learning new things. They are helping create the kind of culture I want in systems programming – the kind of culture promoted by Papers We Love, which Bryan cites as the inspiration for Systems We Love.

By contrast, when I’m talking to Bryan I feel afraid, cautious, and fearful. Over the years I worked with Bryan, I watched him shame and insult hundreds of people, in public and in private, over email and in person, in papers and talks. Bryan is no Linus Torvalds – Bryan’s insults are usually subtle, insinuating, and beautifully phrased, whereas Linus’ insults tend towards the crude and direct. Even as you are blushing in shame from what Bryan just said about you, you are also admiring his vocabulary, cadence, and command of classical allusion. When I talked to Bryan about any topic, I felt like I was engaging in combat with a much stronger foe who only wanted to win, not help me learn. I always had the nagging fear that I probably wouldn’t even know how cleverly he had insulted me until hours later. I’m sure other people had more positive experiences with Bryan, but my experience matches that of many others. In summary, Bryan is supporting the status quo of the existing culture of systems programming, which is a culture of combat, humiliation, and domination.

People admire and sometimes hero-worship Bryan because he’s a brilliant technologist, an excellent communicator, and a consummate entertainer. But all that brilliance, sparkle, and wit are often used in the service of mocking and humiliating other people. We often laugh and are entertained by what Bryan says, but most of the time we are laughing at another person, or at a person by proxy through their work. I think we rationalize taking part in this kind of cruelty by saying that the target “deserves” it because they made a short-sighted design decision, or wrote buggy code, or accidentally made themselves appear ridiculous. I argue that no one deserves to be humiliated or laughed at for making an honest mistake, or learning in public, or doing the best they could with the resources they had. And if that means that people like Bryan have to learn how to be entertaining without humiliating people, I’m totally fine with that.

I stopped working with Bryan in 2004, which was 12 years ago. It’s fair to wonder if Bryan has had a change of heart since then. As far as I can tell, the answer is no. I remember speaking to Bryan in 2010 and 2011 and it was déjà vu all over again. The first time, I had just co-founded a non-profit for women in open technology and culture, and I was astonished when Bryan delivered a monologue to me on the “right” way to get more women involved in computing. The second time I was trying to catch up with a colleague I hadn’t seen in a while and Bryan was invited along. Bryan dominated the conversation and the two of us the entire evening, despite my best efforts. I tried one more time about a month ago: I sent Bryan a private message on Twitter telling him honestly and truthfully what my experience of working with him was like, and asking if he’d had a change of heart since then. His reply: “I don’t know what you’re referring to, and I don’t feel my position on this has meaningfully changed — though I am certainly older and wiser.” Then he told me to google something he’d written about women in computing.

But you don’t have to trust my word on what Bryan is like today. The blog post Bryan wrote announcing Systems We Love sounds exactly like the Bryan I knew: erudite, witty, self-praising, and full of elegant insults directed at a broad swathe of people. He gaily recounts the time he gave a highly critical keynote speech at USENIX, bashfully links to a video praising him at a Papers We Love event, elegantly puts down most of the existing operating systems research community, and does it all while using the words “ancillary,” “verve,” and “quadrennial.” Once you know the underlying structure – a layer cake of vituperation and braggadocio, frosted with eloquence – you can see the same pattern in most of his writing and talks.

So when I heard about Systems We Love, my first thought was, “Maybe I can go but just avoid talking to Bryan and leave the room when he is speaking.” Then I thought, “I should warn my friends who are going.” Then I realized that my friends are relatively confident and successful in this field, but the people I should be worried about are the ones just getting started. Based on the reputation of Papers We Love and the members of the Systems We Love program committee, they probably fully expect to be treated respectfully and kindly. I’m old and scarred and know what to expect when Bryan talks, and my stomach roils at the thought of attending this event. How much worse would it be for someone new and open and totally unprepared?

Bryan is a better programmer than I am. Bryan is a better systems architect than I am. Bryan is a better writer and speaker than I am. The one area I feel confident that I know more about than Bryan is increasing diversity in computing. And I am certain that the environment that Bryan creates and fosters is more likely to discourage and drive off women of all races, people of color, queer and trans folks, and other people from underrepresented groups. We’re already standing closer to the exit; for many of us, it doesn’t take much to make us slip quietly out the door and never return.

I’m guessing that Bryan will respond to me saying that he humiliates, dominates, and insults people by trying to humiliate, dominate, and insult me. I’m not sure if he’ll criticize my programming ability, my taste in operating systems, or my work on increasing diversity in tech. Maybe he’ll criticize me for humiliating, dominating, and insulting people myself – and I’ll admit, I did my fair share of that when I was trying to emulate leaders in my field such as Bryan Cantrill and Linus Torvalds. It’s gone now, but for years there was a quote from me on a friend’s web site, something like: “I’m an elitist jerk, I fit right in at Sun.” It took me years to detox and unlearn those habits and I hope I’m a kinder, more considerate person now.

Even if Bryan doesn’t attack me, people who like the current unpleasant culture of systems programming will. I thought long and hard about the friendships, business opportunities, and social capital I would lose over this blog post. I thought about getting harassed and threatened on social media. I thought about a week of cringing whenever I check my email. Then I thought about the people who might attend Systems We Love: young folks, new developers, a trans woman at her first computing event since coming out – people who are looking for a friendly and supportive place to talk about systems at the beginning of their careers. I thought about them being deeply hurt and possibly discouraged for life from a field that gave me so much joy.

Come at me, Bryan.

Note: comments are now closed on this post. You can read and possibly comment on the follow-up post, When is naming abuse itself abusive?

Tagged: conferences, feminism, kernel
Read the whole story
4 days ago
This is painful but I think necessary: I've only met Bryan briefly at a couple of events but he strikes me as someone who internalized a lot of 90s engineering culture, which along with many good things encouraged a lot of arrogance, testing disagreement as conflict, and other general lack of empathy, all of which makes me wonder how different things would be if that culture hadn't excused or encouraged this kind of behavior. I think many of us are trying to unlearn those patterns but it's hard to shake years of habit.
Washington, DC
1 day ago
Total agreement. While I've never been so successful as to be *that* arrogant, I'm working on being okay with not knowing "simple" things or just plain being wrong. Fostering a culture where everyone feels comfortable speaking up and being wrong is difficult, perhaps impossible without management buy-in.
1 day ago
Melbourne, Australia
Share this story
1 public comment
1 day ago
Thanks Valerie.
Brooklyn, New York

Work at different management levels


I remember working as a developer at a company and complaining that I had no idea what the bosses did all day. It felt like while we engineers were working hard and shipping stuff, managers just talked to a lot of people all the time, or sat in their offices behind closed doors, and I had no idea what their work looked like.1

I find that when someone switches from individual contributor (IC) work to management, it’s a huge shock to the system. Managering is so weirdly different: a different skill set to grow, a less tangible way of measuring success, and a kind of work that’s often so intangible that it doesn’t feel like “real” work.

I’ve had the privilege of experiencing a few different management levels (responsibilities? jobs?) at Etsy since I’ve joined. At each stage, I felt like the job of being a manager totally changed. What I did day-to-day changed, what was hard about it changed, how I measured my own success changed, and though I feel like the experiences built on one another, it continues to be an enormous shift in brainpower each time the gig changes a bit.

Given how intangible (and often hidden) management work can be, I’ve outlined some highlights of what my work has been like as a manager over the last four years. (Obvious, major caveat: this is just my experience, and there’s lots in here that is unique to this particular work environment, hierarchy, requirements, and challenges!)

1 One thing I've found helpful to combat this as a manager is to be super transparent about what my days and weeks look like to my teammates. During a show-and-tell meeting, I'll talk about what's on my mind that week, or show a Google doc I'm writing, or even just a calendar invite I made. It helps to demystify what the role is, and if I'm lucky, it does something to help folks value the different kind of work managers do.

Managing a team of individual contributors

My first two roles at Etsy were managing a single team of developers, ranging between three and seven engineers on the team. The first team was started from scratch before I got there, and the second team (which I started managing after the first one disbanded, for strategic reasons) I inherited from another manager who was moving into his next role - growing a new team.

IC work: During this time, I still shipped a little production code here and there.

1:1s: I had hour-long 1:1s with my direct reports weekly.

New skills acquired:

  • I’d hired folks at a previous job, but working with recruiters and more process for onsite interviews was new to me.
  • How to write a performance improvement plan. And how to help a direct report improve when they don’t immediately see what the problem is.
  • How to terminate an employee, and through this process, also learned how to roleplay these kinds of difficult conversations (which is a skill I’ve used a LOT to help other managers through this).
  • How to have less stilted one-on-ones.
  • How to tweak my management style to better-fit a direct report’s personality or management needs.
  • How to navigate politics between different organizations, and how to effect some change without having a lot of authority.

What I was accountable for: my reports’ growth, and the visible output of the team.

My favorite responsibility: During this time, I started coaching new managers, and implemented someone else’s idea (that they didn’t have time to do, and were cool with me running with it) to create a roundtable for new managers at Etsy to talk to each other.

Hardest part of the job: Learning how to emotionally survive incredibly challenging stretches of time with a particular report.

Managing a team of ICs, and also some managers

Eventually, in addition to managing a team of engineers, I started to manage a manager or two of related teams. The sizes of these teams were all generally still relatively small (fewer than 5 engineers per team).

IC work: I wrote some code that made it to internal dashboards once in a while.

1:1s: Weekly, hour-long 1:1s with everyone who reported to me. Skip-level 1:1s for an hour every two weeks with people reporting to the managers who reported to me.

New skills acquired:

  • Managing managers. It’s entirely different than managing ICs, to me. Especially managing people who are new to management.
  • Iterating on team meetings. I learned a lot about how to routinely check in on and improve standing meetings to keep them interesting, make sure they’re still valuable, or just proactively kill them.
  • Taking better control of my calendar. I noticed that my brain was being way too drained when I switched back and forth between talking with ICs about their work, and talking with managers. So I started to group stuff together better, or at least gave myself more space to breathe between context switching.
  • Giving away my Legos.” Gosh, I loved managing the Performance team. I loved building the device lab. I loved running the new manager roundtable. I learned how to step away from the things I had grown and adored, to create more leadership leaderhsip opportunities for others. To help them have space to make it their own, and make it even better.
  • Being vulnerable with people who don’t report to you, like in skip-level 1:1s. Figuring out what it means to be transparent and honest and human while also trying to be a leader.
  • Trying out new leadership styles that aren’t the one I was born with. I started to watch and try on styles of other people I work with - and then I started to figure out the situations in which those alternate styles are more effective.

What I was accountable for: my direct reports’ growth, my direct team’s visible output, and checks/balances so that the managers who reported to me were keeping their teams healthy and delivering.

My favorite responsibility: Running show-and-tell for more than just the people who report directly to me. I started to realize the huge value of having face time with everyone in one room. Bootcampers, people from other teams, and sometimes senior leaders would join occasionally. Also, people laughed a lot during this meeting; it felt like we were really building something amazing together.

Hardest part of the job: That context switching between managing ICs and managing managers. For one, your brain is focused on team context; you’re talking about code debugging or how to better surface the work that the team is doing to the large org. For the other, your brain is focused not on team mission or delivering on that, but much much more about the human stuff that you only tangentially have an impact on – you need to coach this manager through it, and help them do it on their own. I found switching between the two brains really hard.

Managing just managers, and eventually, an identifiable group of teams

“Identifiable group of teams” happened for me when I had a handful of teams who reported in to me that felt closely aligned in their missions (Performance, Front End Infrastructure, and the App Training team). We originally called this “lhogan-org” for convenience, and then as it grew it got a much better name, “UX Infrastructure”.

IC work: hahaha

1:1s: Hour-long weekly 1:1s with my direct reports, all managers. Hour-long skip level 1:1s with their reports, at first every two weeks, then every three, then every four as the teams grew. At some point I think I also shortened them to a half-hour for skip levels. By the time we hit 30+ people in the organization due to inheriting some teams from folks who left, I switched skip-levels to be every eight weeks.

New skills acquired:

  • Strategy and vision-setting. I still remember how much of a challenge and shift this was from me. I had felt so strong about my management skill sets, executing other people’s visions. It was super hard to try and develop my own thoughts on strategy.
  • Coaching managers through their own iterating on team meetings and other processes.
  • Being okay with not being the go-to Performance person anymore. Being okay with turning down speaking opportunities and other thought leadering (heh) because my day job wasn’t even close to the work that I was known for anymore.
  • Eventually, strategy and vision-setting that saw past 6 months. Coaching teams within my organization to do their own strategy and vision-setting. I started to be able to see the forest for the trees.
  • Reorgs. The lessons in this probably warrant their own post.
  • Developing talking points for my reports to share. I started to rely more than ever on managers to communicate to everybody, and it’s really hard to get folks to say effectively the same things in a way that effectively everyone hears them similarly.
  • Getting comfortable talking in terms of my own failure. Previously, I used language about failure that put a more positive spin on it - I couldn’t help but see only “opportunities for improvement” or other silver linings. But sometimes, failure is just failure. I got more comfortable with this idea.

What I was accountable for: all the above, plus: strategy and vision for the group of teams, pitching in with other related orgs’ strategy and vision needs.

My favorite responsibility: There was something magical about the moment when I realized there was a cohesive idea surrounding this group of teams - the moment it really felt like an organization. It felt like I was helping to support this group of people in doing their strong work, all together. I started to feel much more like I had moved into a facilitation role, where my work was to create a space so that folks in the org could kick ass, have an impact, get recognition. (It’s tricky to talk about this shift; I’d previously been known for doing A Thing whether it was perf or the new manager roundtable or whatever. But at this stage, my work became so much less visible, and I think that’s right.)

Hardest part of the job: At this point, I found myself in new kinds of meetings, meetings with people more senior to me who were going through challenges I hadn’t seen before. It was really hard to figure out how to contribute or help without looking like a total n00b. This more than anything in recent memory pushed me well out of my comfort zone and forced me to grow up a lot as a leader.

Further growth

More recently, I’ve started managing a group of teams that contains groups of teams. (“Product Infrastructure” is made up of eight teams, between “Pipeline Engineering” and “UX Infrastructure”.) These days, I rely heavily on communication with all my managers at once (whether via email or in a meeting). I also rely heavily on the folks running those orgs within the larger org - talking with them way more often.

I think at each stage, I’ve “sat at the table” differently in large meetings with senior managers+. I remember very clearly moments of raising my voice differently than in previous roles - more loudly, or more confidently, or more confidently saying that I don’t at all understand what’s happening. I’m more comfortable vocally disagreeing in real time (rather than doing a lot of side chats or iterating on my words before gaining the confidence to speak them).

I’ve also found myself more often saying “no” to second-shift diversity work when I don’t have the energy for it.

A really cool thing about these stages is that getting out of the way, giving away your legos, means that you get to see some really fucking amazing people nail it. You see the whole thing - from the time when they sucked at a part of the job through the time when they are the go-to person for it. You have the privilege of talking to them behind closed doors about how they’re feeling and what a challenge it is they’re dealing with. No one else gets to see how far they’ve come, really, because they can’t see all those moments of defeat and struggle and sometimes anger and tears and all. I can’t describe what an honor this is. To be trusted with someone’s super honest moments and then to see them totally succeed, all on their own. Most of my job these days is just listening to people and asking them open questions to help them figure out what they already know to do, deep inside. It’s unreal.

The hardest bit of all? People say that growth is beautiful. We all want to grow! We use such positive language about it! But in the last year or so, my leadership coach and I started to talk a lot about how growth is awful and painful and hard. Caterpillars are gross and goopy in that cocoon before they emerge as butterflies. I’ve been in and out of that gross painful goopy stage a lot this year. I’m looking forward to a lot more of this.

Read the whole story
8 days ago
Melbourne, Australia
Share this story

Operations for software developers for beginners

1 Share

I work as a software developer. A few years ago I had no idea what “operations” was. I had never met anybody whose job it was to operate software. What does that even mean? Now I know a tiny bit more about it so I want to write down what I’ve figured out.

operations: what even is it?

I made up these 3 stages of operating software. These are stages of understanding about operations I am going through.

Stage 1: your software just works. It’s fine.

You’re a software developer. You are running software on computers. When you write your software, it generally works okay – you write tests, you make sure it works on localhost, you push it to production, everything is fine. You’re a good programmer!

Sometimes you push code with bugs to production. Someone tells you about the bugs, you fix them, it’s not a big deal.

I used to work on projects which hardly anyone used. It wasn’t a big deal if there was a small bug! I had no idea what operations was and it didn’t matter too much.

Stage 2: omg anything can break at any time this is impossible

You’re running a site with a lot of traffic. One day, you decide to upgrade your database over the weekend. You have a bad weekend. Charity writes a blog post saying you should have spent more than 3 days on a database upgrade.

I think if in my “what even is operations” state somebody had told me “julia!! your site needs to be up 99.95% of the time” I would have just have hid under the bed.

Like, how can you make sure your site is up 99.95% of the time? ANYTHING CAN HAPPEN. You could have spikes in traffic, or some random piece of critical software you use could just stop working one day, or your database could catch fire. And what if I need to upgrade my database? How do I even do that safely? HELP.

I definitely went from “operations is trivial, whatever, how hard can keeping a site up be?” to “OMG THIS IS IMPOSSIBLE HOW DOES ANYONE EVER DO THIS”.

Stage 2.5: learn to be scared

I think learning to be scared is a really important skill – you should be worried about upgrading a database safely, or about upgrading the version of Ruby you’re using in production. These are dangerous changes!

But you can’t just stop at being scared – you need to learn to have a healthy concern about complex parts of your system, and then learn how to take the appropriate precautionary steps and then confidently make the upgrade or deploy the big change of whatever the thing you are appropriately scared of is.

If you stop here then you just end up using a super-old Ruby version for 4 years because you were too scared to upgrade it. That is no good either!

Stage 3: keeping your site up is possible

So, it turns out that there is a huge body of knowledge about keeping your site up!

There are people who, when you show them a large complicated software system running on thousands or tens of thousands of computers, and tell them “hey, this needs to be up 99.9% of the time”, they’re like “yep, that is a normal problem I have worked on! Here’s the first step we can take!”

These people sometimes have the job title “operations engineer” or “SRE” or “devops engineer” or “software engineer” or “system administrator”. Like all things, it’s a skillset that you can learn, not a magical innate quality.

Charity is one of these people! That blog post (”The Accidental DBA”)) I linked to before has a bunch of extremely practical advice about how to upgrade a database safely. If you’re running a database and you’re scared – you’re right! But you can learn about how to upgrade it from someone like Charity and then it will go a lot better.

getting started with operations

So, we’ve convinced ourselves that operations is important.

Last year I was on a team that had some software. It mostly ran okay, but infrequently it would stop working or get super slow. There were a bunch of different reasons it had problems! And it wasn’t a disaster, but it also wasn’t as awesome as we wanted it to be.

For me this was a really cool way to get a little bit better at operations! I worked on making the service faster and more reliable. And it worked! I made a couple of good improvements, and I was happy.

Some stuff that helped: helped:, so that people wouldn’t get paged for it

  • work on a dashboard for the service that clearly shows its current state (this is surprisingly hard!)
  • move some complicated code that did a lot of database operations into a separate webservice so we could easily time it out if something went wrong
  • do some profiling and remove some unnecessarily slow code

The most cool part of this, though, is that a much more experienced SRE later came in to work with the team on making the same service operate better, and I got to see what he did and what his process for improving things looked like!

It’s really helped me to realize that you don’t turn into a Magical Operations Person overnight. Instead, I can take whatever I’m working on right Right now, and make small improvements to make it operate better! That makes me a better programmer.

you can make operations part of your job

As an industry, we used to have “software development” teams who wrote code and threw it over the wall to “operations teams” who ran that code. I feel like we’ve collectively decided that we want a different model (“devops”) – that we should have teams who both write code and know how to operate it. And there are a lot of details of how that works exactly (do you have “SRE”s?)

But as an individual software engineer, what does that mean for you? I thiiink it means that you get to LEARN COOL STUFF. You can learn about how to deploy changes safely, and observe what your code is doing. And then when something has gone wrong in production, you’ll both understand what the code is doing (because you wrote it!!) and you’ll have the skills to figure it out and systematically prevent it in the future (because you are better at operations!).

I have a lot more to say about this (how I really love being a generalist, how doing some operations work has been an awesome way to improve my debugging skills and my ability to reason about complex systems and plan how to build complicated software). And I need to write in the future about super useful Ideas For Operating Software Safely I’ve learned about (like dark reads and circuit breakers). But I’m going to stop here for now. If you want more reading The Ops Identity Crisis is a good post about software developers doing operations, from the point of view of an ops person.

This is my favorite paragraph from Charity’s “WTF is operations?” blog post (which you should just go read instead of reading me):

The best software engineers I know are the ones who consistently value the impact and lifecycle of the code they ship, and value deployment and instrumentation and observability. In other words, they rock at ops stuff.

Read the whole story
10 days ago
Melbourne, Australia
Share this story
Next Page of Stories