isn't quite ashamed enough to present

jr conlin's ink stained banana

:: The Streetcar Problem

Streetcar-CrashProgramming nerds love theoretical technical problems like The Prisoner's Dilemma, or The Dining Philosophers. Here's a fun mental exercise for your afternoon: What would happen to your favorite library/service/product if the principal engineer was suddenly killed by a streetcar? For an astonishing number of services, the answer will probably not be "continue to work seamlessly".

(i'll note that i decided to break this out of a different post about Being Nice for Selfish Reasons: Work Edition. i decided that it's a question that needed it's own post.)

Most projects, like most movements or causes, tend to have a singular leader. There's a lot of reasons for that, including the facts that running a successful project kinda requires someone to make hard decisions, say "No, we shouldn't do that" a fair bit, and "We're going to go in this direction, in spite of your personal interests". There are ways to present all of those statements, sometimes in friendly, happy ways, other times like a overbearing jerk, but there are ways. The problem is also the Achilles Heel of Open Software Projects: Getting someone else to give a damn about what you're building. You have to be nice enough or provide enough value that people want to help you and deal with your pushy ideas on how to do things.

There's a weird psychological edge to successful software development that many folks don't think about. If you even think about the tools and services you use, you probably associate them with one or a very small group of lead folks. These are the captains of the project and in some respects are cult leaders for their ideas, even if they aren't in direct control. Python has Guido; jQuery has jresig; PhP has Rasmus; etc. Larger projects like Webkit require a large, organizational structure to deal with the huge, complicated code base. However, even these projects have a small group or even an individual who is the acknowledged leader. This is pretty much how humans set things up.

So, what happens when such an individual leaves a project? Usually, it's "not good things". In the sudden vacuum, there can be a power struggle as folks fight to become the replacement, or things drift as the next person in line has a slightly different view of how to do things. In rare instances, it's possible for the project to continue or improve, but that's usually because one of the goals of the individual at the helm was to ensure that he or she is replaceable. Yeah, not a lot of folks think that way. It's very uncomfortable for people to confront their lack of permanence.

All of this does impact you, however. You invest in a given resource. Whether you like it or not, you become dependent on that resource more and more. When the streetcar comes, you're also subject to the repercussions. If you're lucky, skilled and have the time, you can fork the project and keep it alive long enough to find a suitable replacement. If not, well, you're dorked.

It can also be exceptionally difficult to keep your product "Streetcar proof". Sure, there's principals of abstraction, but the big problem is that there are reasons you've picked a technology or service over it's competitors, and those reasons will be reflected in the core of your product's design. Replacing or degrading those will not be easy, or potentially possible for whoever is maintaining your code.

Yeah, that's right. You're also subject to Streetcars.

So, when your personal streetcar arrives, how will folks deal with it?

  1. callous
    2014-07-23 10:11:03

    Oh, this is so delightfully well-aligned for expansion into the Trolley Problem. I anxiously await your follow up post.


Wanna join in?

Advertisers! Be sure to read our
Advertisement Policy!

:: Building A Worse Toaster

Sadly, i'm rolling misses on my Google Fu this morning, but i believe that C.J. Cherryh wrote a short story about an elderly woman who wanted a new toaster. It being the future, the device delivered itself and promptly started doing the laundry, fixing the car, reprogramming the TV and all sorts of other wondrous tasks. The Matron, tutted, jabbed a fork deep into the machine. It sputtered, fell over, sparked and finally a warmed, lightly browned piece of toast popped out of a slot.

While the story is funny, it's also really useful to remember when building a tool.

Ultimately, you want your tool to be as simple as possible. This is the general theory of Unix. Little things that work well together. The problem is that building simple things tends to not be very popular.

As a software engineer, i am both subject to, and constant advocate of, feature creep. i want things that just do a bit more. i mean, sure i could have something that's just a front end to a database that allows me to enter a blob of text and then later display it,… but it could also have an inline editor, ooh, and a search function, and it'd be neat to have plugins… And, well, you wind up with WordPress.

Same with things like jquery, which started off as fairly simple and small, then became, well, jquery1. Even protocols go through that sort of progression (mind you, i already ranted a bit about "human vs. machine readable"). Darn near everything tends to start simple, but get increasingly more complicated.

As is usual, a number of things tweaked me about this point. One was an old rant about how even though people spend most of their lives in front of a computer now, most don't know how to use one. The other post talks about developer inequality and a longing for Hypercard. Both pretty much highlight the same thing, and i could argue against both equally.

For the first, i'll note that computers are media delivery devices and people tend to prefer to consume more than create, and that's ok. A fair number of people have no idea how to maintain their auto or understand the mechanics of municipal transit but use it to come and go. Granted, if they're taking an intro course on how to do car repair, probably a good idea to show them how to change the oil and rotate the tires, much like an intro class to computers ought to discuss programming.

Thing is, you can teach programming to kids. You can teach auto repair to kids. Hell, you can teach quantum mechanics to kids. You just have to make them want to learn about it. Javascript isn't hard. Packing it full of complex crap, frameworks and other stuff to make it "easy" makes it hard, but by-and-large most kids can grok the fundamentals pretty darn fast. No, they're not going to be able to understand using Promises to trigger component events in the Shadow DOM by the end of the week, but i wouldn't expect that.

Give 'em two weeks.

Likewise straight HTML is delightful because it's actually nearly readable. Pretty print it and it's pattern matching you can easily wrap your head around. Heck, explain to someone that The Web® really is basically a bunch of fancy telnet calls, and it's like watching a light bulb go off.

Sadly, that may go away with HTTP22 or SPDY, s/telnet/curl/ and you're basically back in the game again.

The final thing to note about all of this is that folks will surprise the hell out of you. They're not going to follow your arbitrary rules and will pretty much use whatever you hand them as a hammer. Done right, that's not a problem.

In fact, it could be a huge benefit.

Good Sir Tim Berners-Lee never expected people to use his protocol to do face to face video chats. Heck, he wasn't all that hot on documents having things other than links, but he was willing to let folks be creative with his toy. Heck, i've built stuff, given folks instructions on how to properly use it, and they've blissfully ignored me again and again.

This is typical (regardless of my continually trying to either ignore or change it), and should simply be part of any service you build. If anything, that sort of abuse should be lauded.

So, when you're building your next toaster (or library, app, interface, device, whatever), ask yourself:

  • Is this doing the least it can to be useful?
  • Can it talk to other things easily?
  • Does this tool make zero presumptions about the user or the way it will be used?

Answering those questions will pretty much tell you how successful your thing will be.
Otherwise, you're just making a worse toaster.

1i'm not knocking jQuery as bad, i'm just saying that people have decided that jQuery is how you write javascript. Javscript has evolved and adopted a lot of the things that jQuery started. The problem is that folks immediately turn to jQuery $(".foo") rather than use local commands document.getItemsByClassName("foo"). There are good reasons to use jQuery. There are good reasons not to. Ultimately, though, you should balance the decision on what you need to do rather than knee jerk use a tool.

2It's funny how long HTTP2

:: Rules of Deployment

The following are JR's Rules of Deployment:

Note: The following may be sweary.

1) Server, then Client, then Marketing

Everyone's job is hard. So why the hell would you make it even harder by forcing folks to work on highly unstable things? First, build the back end server your thing is going to talk to. Get it to the point where it's stable, reliable, and predictable. It supports all the API calls properly, has reasonable security, sufficient redundancy and nobody is going to ask why the box is down.

Once the server is stable, then start building the client for real. You've got a reliable box and test cases you can use. Yes, there may be some additional tweaks to the server needed, but the core API should be fine enough to do productive work. Get it working to the point it's minimally viable (hell, feel free to roll it out as alpha to get some real usage models and proof the load support).

Once you've got a working client and sufficient knowledge, then you can start marketing and planning on your release, and not a god damn second before then. Yes, this means risk. It also means that customers/vendors/whoever won't be screwed because you're going to have to slip delivery dates because you forgot internationalization, or that we live in a base 10 world.

2) Know What You're Building

But, how will you be able to build a server, then client like that? Easy. You write up requirements. Not "User Stories" because i have yet to see anyone figure out how to build Integration tests off of "Bob wants a sandwich, so he presses a button on his phone and a sandwich arrives." You don't have to generate MILSPEC-25 compliant requirements that detail every action of every menu choice you're ever going to need, but you had better figure out the overall functions, interactions and compromise points before you even fire up your editor.

NASA doesn't send probes to Uranus by chucking cellphones "Somewhere. In that direction. i guess". The think that crap out because they DON'T have a huge budget and unlimited resources.

If you're a developer, demand to see the competitive product feature comparison chart. You're going to be building stuff that uses it, it's every bit a required resource as a man page. Yes, this means you can't be "agile". It means you know what the hell you're building and lets you recognize if you're making a mistake.

You will make mistakes. You will toss code, ideas and work out the window mercilessly. If you find you're incrementally redesigning your entire project, stop that project and design a proper one.

3) Know Your Critical Path

If you have a delivery date. Write it in large type on a very visible wall. You know what you're building, and the order it needs to be built in. You know that you'll need to do at least two weeks of systems testing, plus at least two weeks for patches, two weeks for deployment. Those are not things that can be done in parallel, so that's a month and a half of blocked time. You'll also need to build the damn thing, so figure your time, accounting for one additional month for every "cutting edge" or unfamiliar tech you're going to use. That IS being optimistic. Reality will probably demand more time because "cutting edge" means "broken and unsupported features" and unfamiliar generally means "you have no idea what it really does". These are common words you don't like hearing from your doctor or car mechanic so why should you want to hear them from your own team?

Lay out the critical path. Know what impacts what lays beyond it. Recite the critical path every morning in unison in the parking lot. Whatever it takes so that folks know that having a problem isn't what's bad, it's not communicating that there's a problem.

4) Build for One or Infinite

This gets into design and deployment bit again, but basically, you either need one of something, or you need an infinite amount of them. If you only need one, focus on that, and don't bother adding in clever crap. If you need infinite, build it as a service. (Again this is where "knowing what you're building" really helps.)

5) Make It Safe From Yourself

As a VERY smart paranoid security person once said: "Hackers have infinite time and resources. If there's a weakness, they will find it. If you, the person that wrote the code, can't figure out how to break it and do anything other than what was intended, hackers will have that much harder a time."

i constantly look for exploits against my own code. i sandbox anything the user has access to whenever possible. i always ensure that the owner of the server has the least amount of access possible.

6) Build Simple.

More moving parts == more things that can break.

If you're bringing in hundreds of libraries and support frameworks, you had damn well better need every line of code from them. Libraries are feature and code sinks. Very seldom are they pruned, and honest profiling can show how bad a lot of those "easy-*" or "simple-*" things really are. Be brutal about how little you need. If you're using a 10 line function out of a 2,000 line library, yank those 10 lines (and credit in the comments). If nothing else, this will let you see how crapped up those 10 lines are and if you can't do it simpler yourself.

i mean, it'll all show up in the profiler anyway.

You're running a profiler, right?

7) Document

Code is self-documenting if you never want someone else to deal with it. You don't need API guides if you never plan on someone else reusing your code. There are a lot of corners you can cut if the code is disposable.

If, however, you want to do other things with your life than support a 6 year old XML parsing library, write some damn documentation. If you're writing in some edge language, give people a reason to want to learn it so that they can help pitch in. You could write the most amazing thing ever in heavily obfuscated Perl that generates Pentium byte code and people would treat it like toxic waste unless you explained why the hell you needed to do that.

Granted, if you did that for "fun" you're an idiot and deserve to have the project rewritten on you, but that's beside the point.

Your documenting for your QA, Ops, NOC, and Customer Support groups. Even if those folks are you 2 years from now when things break due to expired dependencies.

Before you can tell someone to RTFM, you have to WTFM.

There are other rules, and if need be, i'll define those as well.
These are the core.

You're welcome.

:: Javascript, the New PHP

A couple of Employers ago, i wrote a LOT of PHP. To be fair, i also wrote an inordinate amount of Perl and Javascript so feel free to make whatever assessment regarding my sanity you feel like. One benefit of being in a fairly heavy PHP shop right around the time that it was just starting to get commercially popular, was that said employer hired the guy that came up with it.

This was incredibly enlightening, not because he was as surly and jaded as myself (as well as a phenomenal security Paranoid), but because he openly professed that PHP was a server side layout language. If you were doing more with it than just simply laying out pages, you were doing it wrong.

Of course, you'd be forgiven for laughing at that. If you have even passing familiarity with projects like WordPress, Cake or the countless projects on Github or Sourceforge that are PHP based. The problem is that PHP is surprisingly easy to program in. It's exceptionally forgiving and has lots of capabilities that really lower the bar for new programmers. What that means is that folks who may not have years of experience or a firm understanding of the dangers to watch out for are trying to roll out very sophisticated applications.

There are loads of examples of this too. A perfect one is WordPress (yes, the very blog software i use right here). It's gotten FAR better, but even here there are lots of security gotcha's i tend to be concerned about. One very large one is that WordPress expects to be able to write to the directory it runs in. This is a HUGE "Not Happening" on any system i work on.

A well secured system runs any display server (that includes the web server) in a protected environment with limited privileges. On my system, the app server can read files. It will never be able to write, because when there's an exploit in the software, Bad Guys can't use it to alter files or execute scripts as me. This is how most sites fall, by the way.

So, how does all of this relate to the link bait title? Put simply, one of the biggest problems of PHP is that anyone can program in it. i don't want to sound elitist, but imagine if you could build a free car out of parts someone gave you. A bit of fidgeting, a little swearing and you've got something that rolls down the road. Chances are REALLY good, though, that you've not built a high end race car with proper safety considerations and navigation systems. Those take skill and knowledge that you may not have yet. You can certainly acquire it, but it's going to take time, experience, and learning from mistakes.

Same is true with software. It's not hard to put together a simple program that does some fun things. It's harder if you want other folks to use it. It's harder still if you want to build something that, like your car, won't actively try to kill you or anyone within 10 feet if you do something wrong.

PHP made the tools readily available and easy for folks with limited knowledge and skill to use. That's a significant accomplishment, but it does mean that there's a lot of code out there with no tests, no reviews and no support.

Javascript is clearly headed the same way. It's incredibly accessible, and products like node.js allow you to run Javascript on a server. The problem is that there are still going to be hard lessons not learned. Folks will still do silly things like open files based on parameters passed from outside, or use unescaped values in database queries, or even do uncached hits against remote servers. There are lots of anti-patterns, and even professionals make mistakes.

What's worse is that, like the PHP stuff out there, there's no way for folks to determine which libraries are "Safe" or not. Right now, folk often have no idea that many browser extensions are able to read and write to your disk or web pages directly, and have the ability to report those elsewhere. Libraries are like articles you read on the web. Sometimes the crazy is so deep it's nearly impossible for you to spot it, when you bother to actually read it.

We old, cynical, battle hardened coders need to do a better job teaching young coders how to write stuff. Sites like Stack Exchange help, but even those tend to have less than beneficial suggestions. Partly due to the fact that writing for devices isn't like writing web servers which isn't like writing web-apps, which isn't like writing databases, etc.

It's also noteworthy that there's a tremendous amount of broken things on the web. OpenSSL is a good example of this. Not because the underpaid and overworked devs are to blame, but because it's hard to get folks excited about working on infrastructure. Honestly, while i appreciate the fact that OpenSLL is finally getting rebuilt from scratch, i can only hope that the new code is thoroughly audited and vetted.

And that it's maintained and regularly updated.

So, what about those javascript frameworks that will be coming? i wonder what the chances are about those?

A bit more on this post since this has gotten attention.

i'm not bemoaning either PHP or JS. i actually like both of them. i agree that education is key, but the problem is that people generally don't like or realize they need education. Folks don't want to read several long, example strewn pages of well thought out prose. The fear i have is that since Javascript is becoming even more available than PHP ever was, the problem is going to get far worse.

So, maybe we need something that folks want to look for, like Goofus & Gallant for Coders, or Where's Waldo's Bug or something. Heck, maybe an Everything Wrong with This Code in 90 Seconds Youtube channel might work.

i help out a few friends that have websites. One of them grabbed a highly rated template for his blog. The template had a JS snippet at the bottom that collected a bunch of stats and did a remote execution. Mind you, there are lots of innocent reasons for something like that, but in essence it opened his site up for an easy XSS attacks. i'd wager that the folks that built that site consider themselves "experts". How do we convince folks like that they still need education?

That's the thing that i'm trying to get my head around.

And yes, i'm not going to propose a solution here. This is more a sketchpad for any future ideas i might have. Conversations are useful, even if they're just with myself.

:: Cost Effective Privacy

Joel Pett created a great editorial cartoon that shows an angry gentleman standing up at a climate summit yelling "What if it's a big hoax and we create a better world for nothing?"

i'm here to argue that having a solid User Privacy policy is one of the best way to reduce and manage costs you can have.

Why?

Well, let's look at a few points.

First off, let me note that i'm a server guy with a couple of decades of working with data and back ends. i can code in javascript, but you really don't want to see that sort of thing, unless you're a fan of horror movies. Nope, the crap i've dealt with generally is around taking data, transmogrifying it in odd ways, and puking it back up. For me, data is data, and there are certain rules one has to think about, the same way that a carpenter or architect thinks about building a house.

Scaling

One thing you NEVER want to tell someone like me is that "Costs don't matter." Yes, yes they do matter. Unless you have an endless supply of money and manpower, they matter. Otherwise i'm going to set up a fully redundant, highly secure system of tiered databases, web heads and monitoring boxes that runs in at least 3 colos on 8 core boxes with max GBs of storage (both online and off). It'll be the most aweseome blog server you've ever seen, can withstand the combined attention of every social media site on the planet simultaneously, and will make your CFO reach escape velocity by soiling himself.

Even Google considers costs.

Where you make your tradeoffs matters, and one of the principal ways one does this is by managing the data you maintain. Storing everything is easy to think of, but absolutely ruinous to do. If you store to flat files, you'll pay in search costs later. If you store to Databases, you'll pay indexing costs now. What you want to do is store the absolute least amount of data possible for anything. Don't try to justify why you should cut something, justify why you need to keep anything. Hard. If there's any reason to pitch it, do it otherwise you're going to be drowning in a sea of noise looking for the few bits of signal you can.

Lifespan

This is the hardest thing for a lot of folks to get. One of the running statements for most products is "Ship It!". Go on reddit or stackoverflow and you'll see folks saying "Doesn't matter, ship it!" or some other term.

i hate that.

Let me explain. In my job. "Ship it" is the very first step. What follows is 5-10 years of "keep it running" / "can we improve it?" / "how do we fix that critical bug?" / "how do we transition users to the new service?" / "blah, blah, blah…" If you're lucky, your product runs on a platform that you can update, otherwise what got "shipped" is what's going to be live. Period, and anything wrong is your problem later.

This point is generally compounded by the folks who yell "Ship It" and send out their farewell notes a week after it ships. This leaves you with a pile of shippy code from the ship head and some angry users that have just been shipped on, and you holding the shovel to dig yourself and your future paychecks out of this pile of ship.

Privacy requires documentation. Clear documentation. Clear, understandable documentation with histories that outline issues and resolutions. Clear, understandable documentation in well known, easy to find locations. One should be annoyed by how much documentation exists for the code and how often you see references to it from within the code.

You also need that for ongoing maintenance, QA testing, release management, Load testing, etc.

So, if you have a product that doesn't have that already, you can tell the guys full of ship that they need to rein it in before they ship themselves and you wind up in deep ship.

(well, unless you're making a one use product that nobody cares about. In that case, you need to seriously re-evaluate your life choices.)

Johnny Law

Let's look at US law for the moment. As it currently stands, the Feds can show up at any time with a love note that basically says "Give us every scrap of all your user's data, and you can't tell anyone because Ter'rsts!" This bit of information will be disclosed at the least opportune time with the worst possible framing. It may have lead to Hitler bin Badguy eventually getting that parking citation on his Suppress-o-tron 2000 Battle Tank, but in the mean time, you will have shattered user trust and whatever good faith they may have had in your company. Folks will switch to other services or stop sharing and recommending your product. The paychecks will stop, and you'll be milking orphan tears for the Suppress-o-tron 3000.

Since the Long Arm of the Law is a bit grabby, it's better to know absolutely nothing about what your user is doing. When Agent Smith shows up with his pen register, he gets back client side encrypted blocks of useless crap. Ideally, blocks of crap that will take him well past the heat death of the Universe to decode.

Needless to say, if you're not in the US or plan on doing business outside of the US, you're going to have to deal with a whole host of privacy laws that will make you and your lawyer cry. In some areas, you'll have to tender any encryption method you have to the government, which means that even if you only see a key in passing, you'll have to hand it over.

Feel free to substitute "Ev1l Haxxorz" or other malicious, non-authorized parties, since if you do have to install a "secret back door" for "only authorized law enforcement use", understand that the credentials you provide will probably be stuck on a post-it on a monitor and flashed across Fox News during some agency promotional piece. Well, that or distributed by a disgruntled former contractor. Feel free to use whatever mechanism you like, because it will probably happen. Do NOT presume that governments have better security than you do.

TL;DR

In short, keeping user data to a bare minimum, and well protected, not only is a good idea, it can save you serious money and hassle. It's not just good for users, it's damn good for you.

Blogs of note
personal Christopher Conlin USMC memoirs of hydrogen guy rhapsodic.org Henriette's Herbal Blog
geek ultramookie

Powered by WordPress
Hosted on Dreamhost.