Matt Kopala

Software Development, Technology, Travel

Coders at Work

| Comments

Earlier this year I spent several weeks reading through Coders at Work by Peter Seibel.

I had a paper copy of the book, and as I read it, I highlighted parts that I liked. I then went back through the book again, cover to cover, and pulled out text to compile this list of quotes that I particularly liked. Someone else’s highlights can be viewed on Wikipedia.

The quotes are listed by interviewee.

This is one of my favorite books about programming. I highly recommend it. Pick up a copy.

The Interviewees

Here are the Wikipedia pages for most of them. There are excerpts on the book’s website for each as well.

Jamie Zawinksi

… it was just nice to be in that kind of environment where someone says, “That’s nonsense,” or “We should do it this way,” you can just take their word for it, believe that they know what they were talking about.
referring to the group of highly talented people he was working with at the time

… vi users quit and restart vi all the time.

“What the fuck did you check in; that’s complete bullshit — you can’t do it that way. You’re and idiot.” And you’d say, “Fuck off!” and go look at it and fix it and check it in.

There’s never a time to start over and rewrite it. And it’s never a good idea to start over and rewrite it.
referring to how they dealt with decreasing code quality at Netscape

this always feels wrong to me, but when you inherit code from someone else, sometimes it’s faster to write your own than to reuse theirs.

I find that getting something on the screen as soon as possible really helps focus the problem for me.

you never know what the design is until the program is done.

Given a leisurely development pace, that’s certainly the way to go.
in answer to the question about whether development would have been easier or faster if they had been more disciplined about testing

You’ve got to say in the comment something that’s not there already.

there’s got to be agreement about what the interface between modules is

I was kind of freaked out when I realized that there are people graduating with CS degrees who’d never written C.

you’re trying to express a concept to a very dumb person – the computer – who has a limited vocabulary.

Brad Fitzpatrick

I’ve also done a lot of testing since LiveJournal. Once I started working with other people especially. And once I realized that code I write never fucking goes away and I’m going to be a maintainer for life.

It’s always easier to do something right the first time than to do a migration with a live service. That’s the biggest pain in the ass ever.

Anything that’s halfway performant is in C++ there.
referring to Google

I think it’s really important to know the whole stack even if you don’t operate within the whole stack
in answer to the question about whether programmers still need to know assembly and how chips work

In practice, nothing works. There are all these beautiful abstractions that are backed by shit.

if you’re the one responsible for the cost of buying servers … it helps to actually know what’s going on under the covers and not trust everyone else’s libraries, and code, and interfaces.

I start with interfaces between things.
answer to question “How do you design software?”

I’ve started with the hard bits or the parts I was unsure of, and tried to implement those parts first.

Always try to do something a little harder, that’s outside your reach. Read code. I heard this a lot, but it didn’t really sink in until later.

First step, take a virgin tarball or check out from svn, and try to get the damn thing to build.
how to make a change to something in someone else’s code

Google with really strict style guidelines in all languages. …. name variables … spacing and indentation … patterns and conventions

Once a project is mature and has a lot of people hacking on it, they’ll have a style guide.

The way it works within Google is that it’s one massive source tree, one root, and one unified build system across all of it.

To check in you need three conditions met: You need someone to review it … need to be certified in the language … need the approval above from somebody in the owner’s file
how code check-ins work at Google

most of the things I work on are very “When it’s done, it’s done.”

You want to write stuff people use, which comes down to web apps.
when asked if he has any interest in writing better desktop software

So every six months I try out one of them, Eclipse of IntelliJ. And the damn thing just sits there spinning forever, consuming memory and maybe crashes in the middle of me typing
said in reference to his coworkers telling him that these tools do something for them automatically.

I’ve always found — at least for your editor — it always pays off.
in regards to time spent learning your IDE/editor being worthwhile

Stop. Try to figure out what’s going on. Learn how to write things incrementally so that at each stage you could verify it.

I’ll go out of my way to write something in a language that I would rather not write it in … because I know I’ll be better in the end.

Douglas Crockford

One of the things that was good for me in being at Electric Communities is it taught me to think in terms of closures.

The worst feature of JavaScript, without question, is its dependence on a global object.

The whole Web is built on one mistake after another.

Where we do have a problem there is that there are too many Ajax libraries.

One of the things I’ve been pushing is code reading.

Having the people you work with helping to keep your code clean is a huge service – you find defects you never would’ve found on your own.

I think an hour of code reading is worth two weeks of QA.

I now believe that code reading should be happening all the time throughout the life of the project.

The simplest is just being consistent in the presentation so you always indent everything properly
in answer to “What makes code readable for you?”

I have never seen a piece of code that I could not improve by taking the continue out.

By cleaning it.
“How do you read code you didn’t write?”

Readability of code is now my first priority.

Ultimately you have to embrace the house style and you hope that the people who put the hourse style together knew what they were doing.

Generally the team knows when it’s time.
“How do you know when, if ever, it’s time for a big rewrite?”

be very disciplined about only reimplementing what you already understand
paraphrased by Peter Seibel

In my opinion, anybody who calls himself a professional programmer should have read Knuth’s books or at least should have copies of his books.

The place where I found that to be the most effective was taking testing, sort of, to the ultimate: going to visit customers.

I want people who can write, because we spend a lot of time writing to each other.

Yeah, read a lot. There are good books out there. Find the good ones and read those. And if you’re doing web development, find the best sites and look at their code.
advice to self-taught programmers.

Brendan Eich

I think what I’ve learned with Mozilla and Firefox has been about more test-driven development, which I think is valuable.

I know a lot of JavaScript programmers who are clever programmers, and the best ones have a good grasp of the economics. They benchmark and they test as they go and they write tight JavaScript.

At SGI, the kernel, of course, was where the real programmers with chest hair went

Most people are lazy. Larry Wall was right. Laziness should be a virtue.

But in the real world we’re all in debuggers and they’re pieces of shit from the ‘70s like GDB.

I have this big allergy to ivory-tower design and design patterns. Peter Norvig … did this paper about how design patterns are really just flaws in your programming language.

I just realize that we don’t always get the luxury of revisiting our deepest design decisions.

I’m skeptical of rewrites. I think it’s rare to find that kind of alignment of interests and get enough funding to live through it and not miss the market.

I do go back and maintain comments – it’s a real pain and sometimes I don’t do it and then I regret it because somebody gets a bum steer.

It’s at the bigger level, the big monster function or the module boundary, that you need docs.

Joshua Bloch

For whatever reason, I realized even back then that a program has to be readable.
regarding what aspects of his programming style have stayed the same

Basically I’ve become harder on myself over the years – that’s that it takes to write good programs. You really can’t accept bad habits from yourself.

The more things you learn and the younger you learn them, the better off you are.

If you can’t write precise, coherent, readable specs, nobody is going to be able to use your stuff.

The older I get, the more I realize it isn’t just about making it work; it’s about producing an artifact that is readable, maintainable, and efficient.

You want to use the right tool for the job, and what would be the right tool in one language may not be the right one in another.

That’s one of the reasons that sometimes languages that are, on paper, better than other languages don’t win – because they just haven’t built the right communities around themselves.

I get infuriated at web sites when they drive me to do the wrong thing. It’s clear that someone just hasn’t thought about how someone approaching this thing will deal with it.

The importance of requirements analysis can’t be overstated. … Not only is it a negotiation but it’s a process of understanding. Many customers won’t tell you a problem; they’ll tell you a solution. … Then you have to ask ‘Why?…’ … You go back and forth until you figure out what all the customer really needs the software to do. … It’s OK if you spend a lot of time getting it reasonably close to right, because if you get it wrong, you’re already dead.

the fundamental rule is, write the code that uses the API before you write the code that implements it.

Good programmers think in terms of pieces that make sense in isolation

If you write a monolithic system and, when it gets too big, you tear it into pieces, there will likely be no clear boundaries, and you’ll end up with unmaintainable sewage.

The fundamental theorem of API design is, when in doubt, leave it out.

“Simple ain’t easy.”

I don’t think tests are even remotely an acceptable substitute for documentation.

For all of the fancy debugging tools at our disposal, there’s nothing that can match the power of simply stepping through a program, in a debugger or by reading it and mentally executing the code.

Yes, students should learn low-level languages. In fact, they should learn assembly language, and even chip architecture.

a lot of engineering is from the gut.

If you’re not solving real problems for real users … then you shouldn’t add the feature.

We’re just fooling ourselves if we think our programs are, by and large, free of bugs.

So when you’re wrestling with different approaches or even different features … you just have to bounce it off other people.

I think it’s unrealistic to expect everyone to own all the code.

I’ve noticed that people who are too smart, in a certain dimension anyway, make the worst code. Because they can actually fit the whole thing in their head they can write these great reams of spaghetti code.

we didn’t understand that what we were doing wasn’t pure engineering. We weren’t really thinking that the most important thing we could do was solve real problems for real customers. The moment you lose sight of that and who your customers are, you’re dead meat.
referring to the failure of the first company he was involved with

if you lack empathy or emotional intelligence, then you shouldn’t be designing APIs or GUIs or languages.

Joe Armstrong

I got to be very good at debugging. If all else failed, I would debug people’s programs. The standard debugging was one beer.

I could see how to do things in a few lines and they’d written tens of lines and I’d sort of wonder why they didn’t see the simple way. I got quite good at that.

Being a young programmer today must be awful – you can choose 20 different programming languages, dozens of frameworks and operating systems and you’re paralyzed by choice.

you can no longer understand the whole system from top to bottom.

Over the years I’ve kind of made a generic mistake and the generic mistake is to not open the black box. … I can’t say beginner programmers should open up all these abstractions. But what I’m saying is you should certainly consider the possibility of opening them.

One that’s tricky is slight spelling errors in variable names. So I choose variable names that are very dissimilar, deliberately, so that error won’t occur.

I’ve gone over to this writing test cases first, now. Unit Testing.

it’s incredibly difficult to get people to use new and better stuff.

Big mistake: don’t tell people how fast something is going to be before you’ve implemented it.

Things you don’t do are difficult and things you’ve done are easy. So you don’t even try. And I think that’s a mistake.

You fix your car and it goes wrong – it’s the last thing you did. You changed something – you just have to remember what it was. It’s true with everything.
regarding debugging technique

Often when you explain things then you understand them better.

I’ve always taken the view of system design, you solve the hard problems first. … I think with very difficult problems I quite often start by writing the documentation. … if you really want to understand it then I think writing the documentation is an essential step.

“I always spend a day a week learning new stuff. That means I spend 20 percent more of my time than my colleagues learning new stuff. Now 20 percent at compound interest means that after four and a half years I will know twice as much as them.”
quoting Hamming

… what should you do to become a better programmer? Spend 20 percent of your time learning stuff.

if you get to the point where if you were to remove anything more it would not work any more – at this point it is beautiful.

Simon Peyton Jones

Today “where things are at” is that big companies are pouring immense resources into ecosystems and editors and profilers and tools and programmers and skills and all that kind of stuff. The mainstream is, by definition, deeply practical.

One of the nice things about working in research is that instead of some manager standing over me saying, “This has to be done this week – just get it done,” I can sit and look at something and say, “There must be a right way to do this.”

For me, part of what makes programming fun is trying to write programs that have an intellectual integrity to them.

That’s the worst thing about long-lived programs … that they gradually become ugly. So there’s no moment at which they become disfigured but nevertheless after a while they just become crappy.

The most depressing thing about life as a programmer, I think, is if you’re faced with a chunk of code that either someone else wrote, or worse still, you wrote yourself but you no longer dare to modify. That’s depressing.

Peter Norvig

Another difference between now and then is it seems like it’s much more assembling pieces now rather than writing everything from scratch.

it’s all scripting to put together these pieces rather than writing everything from scratch. So I think understanding interfaces and how they go together is more important than all the details of the insides of these packages.

I think to really be a good programmer, you can’t just [go ahead and do it]. You have to understand a little bit more, and say, “Is it safe, what I’m doing here? Or what are the failure cases? Sure, I tried it once and it worked, but is it always going to work? How do I write test cases to show that and to understand it a little better, and then once I’ve done that, can I extract what I’ve done and publish a new tool that other people can use because I’ve put these pieces together in a certain way.”

If you have two good programmers, it’s better for them to work independently and then debug each other’s work afterwards … I think it is important to get together when you’re figuring out what it is you want to do both in terms of brainstorming what the problem is and what is the functionality going to be here? … You want feedback, so I think you should require every piece of code to be reviewed by another set of eyes, but it doesn’t have to be real-time, when you’re writing it.
commenting on XP pair-programming

I think the best part of the apprentice approach is that you get to watch the master, and I would like to see more of that. So I guess that’s another use of pair programming.

You need to know your language well. Read the reference.

I think one of the most important things is being able to keep everything in your head at once. If you can do that you have a much better chance of being successful.

you can test all you want and if you don’t know how to approach the problem, you’re not going to get a solution.

I think you want to have good documentation at the level of overall system design.

One of the most important things for having a successful project is having people that have enough experience that they build the right thing.

you want to think about what are the difficult choices vs. what are the easy ones. What’s going to come back to really screw you if you make the wrong architectural choice

They should write lots of tests. They should think about different conditions. And I think you want to have more complex regression tests as well as the unit tests. And think about failure modes
talking about how programmers should think about testing

Then the launch process is the most formal of them all. Then, there is a checklist – it’s very formal in terms of security issues. you need to be certified before you can start contributing code. Every check-in has to be reviewed by somebody else and verified.
talking about the process used at Google

Look at all the software bugs on your computer. There are millions of them. But most of the time it doesn’t crash.

If giving them a bigger monitor increases productivity by such and such a percent, then you should probably do it.

One of the aspects of the modern style of programming … is that programmers have to absorb things quickly.

People don’t do that anymore. They don’t want to read a book. They want an index so they can say, “What’s the least amount of this book that I have to read? …”

Guy Steele

That, in itself, is an eye-opener – the fact that a program can be so organized and so documented, so indexed, that you can find something quickly.
speaking about TeX: The Program

one would hope that a good programmer, a really good programmer, will have given thought to conveying [the story & context of the program] in addition to the story of what the program actually does.
referring to code comments

It’s hard to find good code that’s worth reading.

If you just keep adding one thing in the most obvious way, eventually you end up with a big pile of kludges that barely holds together.

I’ve become much more interface-oriented that I used to be.

I think that the programmers of today are up against a more difficult environment

I’m constantly worrying about ways in which the reader might misinterpret what I’ve written. So I’ve actually spent a lot of time consciously crafting the mechanics of my prose style to use constructions that are less likely to be misinterpreted.

when we are describing processes to be carried out, little details do matter because a change in a small detail can affect the gross outcome of the process.

This forced me to go 100 percent to a regimen of unit testing. So I proceeded to construct thorough unit tests for each of my subprocedures. And this turned out to be a very good discipline.
referring to a project he worked on in Haskell

Dan Ingalls

Who were the big Fortran users? They’re big scientific computation users. And who are they all working for? They’re all working for the government. Do they care how efficient their program is? Not really. What they really want to do is to show that the computer is overloaded and that they need a new computer and more money.

when trying to do something new, I just think of what would be the first piece of success that’s achievable.

Is the framework factored so that someone else coming along will see that these are the tests? That’s something that I have not typically done. This may just be the luxury of my generation – nobody would get away with that these days.

Either you agree on some interface, or often I’ll build some framework that’s incomplete but that works for one example and then it’s obvious to someone else where their stuff would fit in.

I think you have to stand back and say, “What boundaries is this going to cross?”

whenever I have a computing environment to play with, I like to try new things in it.

The first thing is being clear about what you’re trying to do. Trust is part of it, trust for the people that you’re working with … I think the kind of thing that makes for bad micromanagement is you’re worried and you’re insecure, and so you’re feeling like you have to nail everything down.
when asked about tips for being a good technical leader

I don’t know of anybody who does that if they have the choice of using a good debugger.
in answer to “Do you ever resort to print statements?”

I think it’s valuable to have thing be as absolutely simple as possible

When I first write stuff, there are no comments. As soon as I have it working, I’ll write some comments. … I kind of feel like the better a language is, the less you need comments.

L Peter Deutsch

I did that the way I actually have most of my programming, which is to do the data structures first.

I think what intuition is, really, is an unconscious process for synthesizing a solution out of a large amount of data.

How we go about programming now, we don’t have anything like [mathematics foundation of 5,000 or 10,000 years] to build on. Which is one of the reasons why so much software is crap: we don’t really know what we’re doing yet.

I do know that the further down in the plumbing the software is, the more important it is that it be built by really good people.

There has to be something a little wrong with you for you to be a really good programmer. … but the qualities that make somebody a well-functioning human being and the qualities that make somebody a really good programmer – they overlap but they don’t overlap a whole heck of a lot.

The physical properties of matter are such that 99.9 percent of the time you can understand it in aggregate. … To a great extent, that is not true in the world of software.

Software is a discipline of detail, and that is a deep, horrendous fundamental problem with software. Until we understand how to conceptualize and organize software in a way that we don’t have to theink about how every little piece interfaces with every other piece, things are not going to get a whole lot better.

the issues about interfaces, which really become paramount as programs get big …

As I built larger and larger systems, I found that when sitting down to write any piece of code, more and more the question I would ask myself first is, “OK, what’s the interface between this and everything around it going to look like? What gets passed in? What gets passed out? …”

When I was at what I would consider the peak of my abilities, I had extremely trustworthy intuition. I would do things and they would just turn out right.

The thing that he said that made a profound effect on me was how important it is to measure things; that they’rell be times – maybe more times than you think – when your beliefs or intuitions won’t be right, so measure things.
referring to Jerry Elkind, who was manager of the Computer Science Lab at PARC for a while

I seriously question how well software not built from an engineering point of view lasts.

The reason I don’t program in Lisp anymore: I can’t stand the syntax. It’s just a fact of life that syntax matters.

Well, my description of Perl is something that looks like it came out of the wrong end of a dog. I think Larry Wall has a lot of nerve talking about language design – Perl is an abomination as a language.

it’s just that seeing anything around me that’s being done badly has always offended me mightily, so I thought I could do better. That’s the way kids think.

Ken Thompson

I’ve never been a lover of existing code. Code by itself almost rots and it’s gotta be rewritten.

When it’s hard to work on. I do it much quicker than most people do. I’ll throw away code as soon as I want to add something to it and I get the feeling that what I have to do to add it is too hard.
response to how he decides code needs to be thrown away

Documenting is an art as fine as programming. It’s rare I find documentation at the level I like. Usually it’s much, much finder-grained than need be. It contains a bunch of irrelevancies and dangling references that assume knowledge not there. Documenting is very, very hard; it’s time consuming. To do it right, you’ve got to do it like programming. You’ve got to deconstruct it, put it together in nice ways, rewrite it when it’s wrong. People don’t do that.

Some people write fragile code and some people write very structurally sound code, and this is a condition of people. … My definition of fragile code is, suppose you want to add a feature – good code, there’s one place where you add that feature and it fits; fragile code, you’ve got to touch ten places.

I’ll do it as simply as possible the first time and very often that suffices for all time.

When I first came here [to Google] I did it just to try and get the feel of the place. You’ve got to. There’s so much unsaid that you’ve got to know.
when asked if he reads code for fun

If you interview somebody coming out of the computer science right now, they don’t understand the underlying computing at all. It’s really, really scary how abstract they are from what a computer is or even the theory of computing. They just don’t understand it.

Fran Allen

They way I would approach understanding a new language or a new implementation of some very complex problem would be to take a program from somebody that I knew was a great programmer, and read it.

Yes. And in fact, that is true – I very much believe that.
referring to Fred Brook’s statement “Build one to throw away because you’re going to.”

There were about 10 or 12 people and we divided the work up so that each person really had ownership of a piece of it.

I said, “No, we can’t do that. We don’t know how to do that.” Yadda, yadda, yadda. The next Monday I came in and they had done it. They had torn down the system and rebuilt it with hashing. it worked, and it was much faster. So that was a big lesson to me. I should be more open to some brand-new ideas.

It can be very painful to sit there every Friday and do code reads with people explaining why they’re doing what they’re doing and finding other people’s errors. … Absolutely worthwhile.

Yes, as long as he doesn’t think he’s good at the technology, and is able to distinguish between who is and who isn’t in the people that work for him.
reponse to “Is it possible to have a great manager who’s actually not very technically skilled”

I wasn’t a very good programmer. I tended to make quite a few mistakes … I tended to be kind of disinterested in getting all the details right and I was much more interested the in the way systems work.

These days the conventional wisdom is that it’s men who have a bizarre ability to focus on things, usually to the detriment of everything else, and that’s why most programmers are men.

Working for IBM Research was one of the most fortunate things that ever happened to me because IBM Research sits between industry and academia.

Bernie Cosell

I did something that they hadn’t done – it was one of the things that I got known for around BBN, which is, I made things work.

Really what I was was a very careful programmer with the arrogance to believe that very few computer programs are inherently difficult. I would take some piece of code that didn’t look like it was working and I try to read it. And if I could understand, then I could usually see what was wrong or poke around with it and fix it.

But sometimes I would get a piece of code … and I would say, “This is way too complicated.” So I would think through what it was supposed to do, throw it away, and write it again from scratch.

if they’d asked, “How did you fix the bug?” my answer would have been, “I couldn’t understand the code well enough to figure out what it was doing, so I rewrote it.”

Classes can give you a lot of stuff, but in the end programming is a craft you have to perfect by plying it.

I have all the dynamics of the system in my head. I know what order things are supposed to happen.

these days, I always point out that you’ve got to make programs so that they are testable.

The thing that made his design reviews so scary is that he knew when you were bluffing. … He had an instinct, and it was abetted by having a good crew in there, of catching you when you were bluffing, catching you when you hadn’t thought it through. The parts that you did absolutely fine hardly got a mention.
referring to his manager at BBN

What you wanted to do with a design review was double-check that the parts that he had thought he had right he did have right and potentially give him some insight on the parts that he didn’t. … such an obvious good use of the senior talent doing the review.

If you’re looking at a piece of code and it looks very hard – if you can’t understand what this thing is supposed to be doing – that’s almost always an indication that it was poorly thought through.

The other rule is to realize that programs are meant to be read.

I always tried to make the whole program better and that would often mean that even though there was one little problem, I would try to understand the whole program.

When you finish with a routine I want every routine you work on to look as if it was just written. … I was to see code that looks like through some divine inspiration you got it exactly right the first time.

Don’t just fix it in the shortest way. Don’t just fix it in the way that fits, but move it toward the other model so that over several months instead of the program getting more and more mired in patches fixing up the stuff that was old and wrong, all the critical parts of the program all of a sudden look like they’re the new way of doing things.

So you make your estimate someplace between [the shortest way & a complete rewrite] and then every time you get assigned a task you have a little bit of extra time available to make the program better.

Part of what I call the artistry of the computer program is how easy it is for future people to be able to change it without breaking it.

Code it as lucidly, as easy to read, as crystal-clear as you can. Do it the simple way. And then if it needs to be sped up, we’ll deal with that later.

The idea was for you to be stunned at how hard it was to remember whatever it was you thought you understood perfectly clearly just six months ago.
referring to a two-semester course in college where they had you work on a program at the beginning, and then without warning, again months later

Each generation of programmers gets farther and farther away from the low-level stuff and has fancier and fancier tools for doing things. … The trouble is that these baselines are getting more and more complicated.

Donald Knuth

I found more than 100 bugs in my 100-line program.
referring to an early program that he wrote

One of the things I learned was how much software occupies the brain. … I could teach classes full-time and write a book full-time, but software required so much attention to detail.

Yeah, right. That’s a very good corollary.
Seibel asked “you said that it’s impossible to estimate how long it will take to write books. Does that then mean that it’s even hard to estimate how long programming will take?”

I think it’s always going to be true that a person who manages programmers should not expect it to be predictable.

The first rule of writing is to understand your audience – the better you know your read the better you can write

Simplified it is like this: only two percent of the world’s population is born to be super programmers. And only two percent is bron to be super writers. And Knuth is expecting everybody to be both.

I can read my program a year later and know exactly what I was thinking.
referring to literate programming

I’m not saying black boxes are useless. I’m saying that if I’m not allowed to open ‘em up – if I have to do everything with a library or something like this, I would come up with much, much worse results and much slower.

We’re dealing with … things that are on the edge of what human beings can handle and more complicated than have been done before.

It’s very important to be able to get inside of somebody else’s way of thinking, to decode their vocabulary, their notation.