isn't quite ashamed enough to present

jr conlin's ink stained banana

:: Why The Commenters on Hacker News Are Terrible Engineers

CNN published an article a bit ago talking about Silicon Valley’s Poor. It’s a look at the stark reality of where i live and how hard it is for folks who don’t happen to be fortunate enough to be working in the current, hot industry. The article is really well done, and while i don’t agree that rent subsidies are the right approach to the problem, the bulk of the article makes a stirring case.

So, it was no huge surprise that it showed up on Hacker News, and the comments are truly depressing. Not just because they’re full of the usual self centered, overly superior viewpoints that make reading comments there sad making (really, if you don’t want to get into the same sort of spittle fueled rage i’m in, don’t read the comments), but because they fail for one very simple spectacular reason.

They show how bad these guys are at even understanding complex systems.

First off, let’s run a few numbers, shall we? i mean, i’m an engineer, so before i start a conversation about a given problem, i want to make sure i understand the scope and scale. That’s what being an engineer is about.

First off, let’s look at the population. Silicon Valley comprises of the southern portions of the Bay Area. The counties are Santa Clara (from Palo Alto south to encompass Gilroy and east to cover Milpitas) and San Mateo (up the peninsula to San Francisco. i could easily argue that it includes portions of Santa Cruz (to the south toward the Ocean) & Almeda (which covers the eastern Bay Area), but those are partial at best. i could also argue that San Francisco is part of this as well, since a vast majority of folks live there and commute, but those numbers are immaterial to the conversation at this point.

The population of those two counties works out to be 2,699,483. While the total population is not employed, it’s fair to say that because of the high number of bachelors, it’s fair to factor out 30% of the population as being non-job holders (retired, children, infirm, etc.) It’s also fair to say that the Census is not fully accurate either, so there’s some fuzz here, for sanity, we’ll say the working population is two million.

Working off the number of employees for the top companies in Santa Clara (plus google and facebook because, really? they’re missing?) gives about 756,000. Mind you, that includes big hire numbers like HP, Intel and Lockheed Martin which do NOT have their entire workforce located here, but should cover the countless “startups” and smaller companies like SmugMug, Imgur & NetApp. So, figure 40% of the employed here work for high-tech. Figure another 200,000 folks work in equally high pay positions like patent attorneys, doctors, financials and you’ve got about 50% of the population working in “High Pay” fields.

Monthly rent in the two counties is about $2,650 a month, meaning $31,800 a year, just for housing expenses. If we say that weekly living expenses are $100 (for things like food, clothing, internet access, fuel, utilities, commuting, etc.) are a conservative $200 a week, that means that total cost of living is about $42,000. Now add tax to that (~30%) and you’ve got to earn a minimum of $55,000. That means that the minimum wage for the county has to be around $26.50 an hour. (Living Wage notes the number at $12.01 for Santa Clara, so i’m being VERY generous here.)

“No big deal”, you say. “i earn more than that.”

That’s because you’re special, but remember that you’re less than half the work force. There are more folks cleaning yards, checking your groceries, picking up your trash, teaching your kids, cooking and serving your meals at restaurants, driving trucks to supply businesses and all the myriad of things you need for a society to function. They’re not making $26.50 an hour. They’re making a lot less. i’ll remind you that minimum wage in Santa Clara is $8 an hour.

“So, they should move out of the county!”

Ok, sport. Let’s say they do. Well, San Benito County to the south has Hollister, where rent for a 2BR apartment is $1,200 a month, plus a 1+ hour commute. Tracy, out in the valley is also about $1,200 with a 1+ hour commute. (These commutes go up SIGNIFICANTLY during rush periods, so we’ll double the commute time to 2 hours one way.) That means that folks have to spend 4 hours a day in their car just getting to or from work. Mind you, a lot of folks are doing just that, so there’s not a lot of jobs out in those parts, so yeah, commuting isn’t really avoidable. So, your fuel costs just went up, as well as car maintenance costs. After all, car breaks down and can’t go to work? Well, you’re not getting paid that day, bucko.

Oh, yeah, and God forbid you have kids. It’s not like you can lock them in their cage for the day and call the neighbors to go feed them if you’re running late. (Granted, we’re talking about valley engineers here, so breeding is probably not going to be a major concern.)

So much for saving money, huh?

Granted, i’m not a social engineer either, so i have no clear answer to this. i will note that there are dozens of empty office parks in Sunnyvale with new ones going up all the damn time. Getting some of that zoned as Commercial/Residential would help increase available housing driving the rents down. Likewise, putting some of the acquired wealth in the Valley toward improved infrastructure and better mass transit would make the place a helluva lot better than putting another damn McLaren or Telsa on the roads around here. i also think it’s high time we up the minimum wage around here to $12 and put in rent caps. i’ve seen that apartment that’s “a deal” at $2,000 a month. You painted it last century. You’ve made enough friggin’ money you greedy bastard, and yeah, $12 an hour means your latte’s getting pricier. As you said before, you can afford it. If not, maybe you should move.

But, again, i’m not a civic engineer. Still, if this is the way that these apes approach a complex problem, no wonder they all fell in love with Mongo.

:: Bias Confirmation

Over the past few weeks, i’ve been working on creating “demo” apps for SimplePush integration into third party message transport systems. If i’ve lost you in the previous sentence’s word salad, feel free to bail on this post, because it’s going to get a wee bit cranky, and a whole lot geeky. Actually, strike that, reverse it.

Background Info

Wherein i talk about why i walked into the tarpit

Right, so let’s start from the beginning. Mozilla is rolling out a messaging system for programmers. There’s a lot of these sort of things out there, but mostly they’re like email was back in the bad ol’ days of Prodigy, AOL and Compuserve. Folks who used those systems couldn’t send email to folks who were on other systems, or at least, couldn’t do so easily. Then came some discussion, standards meetings, and the Simple Mail Transport Protocol (SMTP) was introduced. Now, various folks could send mail to other various folks, and after copious arm twisting, customer demands, and folks leaving to use The Internet (because that didn’t have weird restrictions like the older platforms did), the companies caved and converted to SMTP. Much joy was to be had. Well, for a little while at least.

Fast forward a few decades and we’re back in the same spot again, only this time, it’s with how servers can send messages to your devices. Turns out, that’s kinda tricky. Your laptop, phone, tablet, watch, belt buckle, shoelaces, and fridge can pretty easily connect up to sites like Facebook, Google, and RedTube (your belt, sadly, is REALLY into bondage for some reason). That’s because those sites pay money to be at well known locations on the internet. You, however, don’t. You move around. One minute you’re at home, then 30 minutes later you’re at work, or headed down the freeway. Turns out that you moving around so much makes it hard to get something to you. (Imagine that your cellphone number changed between locations, and a few times while you’re between locations. That’d make it a bit hard for folks to get in touch with you.) Turns out exactly that happens with your various gadgets. That means that if you wanted to get stuff like chat messages, or surf reports, or a lot of other things, that could lead to all sorts of problems.

Turns out there are a bunch of ways that folks have tried to solve this problem, and have been trying to solve it for a while. Apple, & Google are the big players in this area (much like Prodigy, AOL and CompuServe were with email in the very early days of the internet). Sadly, none of these play nice together. You can’t use the same system to talk to Google that you do for Apple. Heck, you can’t even send the same sort of data to Google that you can to Apple.

Thus, very smart folk have sat down and created a standard for all of this. This standard is still in the works, but there’s a prototype that i’ve been working with for a while. That’s SimplePush, and much like email, it tries to make things easier for sites you want to hear from to send you things you’re actually interested in getting. Now, because those big systems are big systems, and because those big systems are VERY TIGHTLY INTEGRATED into devices, it makes sense that we try to use those notification systems to carry some of the data we want to send.

For some systems, this can be interesting.

And then there’s Apple.


In which i note that i’m not a fan…

i’m not a fan of Apple. i’ve a long, and somewhat checkered history with the company, and while i appreciate that they have an excellent eye for product design that has lead to a good deal of improvement in tech, i’ll note that Disney has had an equally large impression on amusement park theme rides, but i’m rather happy with my car not following a rail and limited to 15mph.

Fortunately, i’ve had little need to deal with Apple products. i get my music from Amazon, watch movies on Netflix, generally work on Unix systems (i tend to run Windows as the UI client, but most of the apps i run there are Open Source.) i have no beef with other folks using Apple products and, frankly, the office here is pretty much filled with Apple laptop chargers because of the enormous number of Macs in the building.

Suffice to say that i’m a bit biased against the legacy of Steve, but still reasonably polite to the odd Apple recruiter that called.

The Procession of Ducks

In which we learn it is better to be eaten whole by alligators, than nibbled to death by ducks

Right, so first i needed to build a version of SimplePush that used Google Cloud Messaging as it’s carrier. i grabbed a copy of Android Studio, fired it up, and in about 6 days, used Java to write a simple demo app that let me connect to the server, and fire messages that were then echoed back to the app over GCM. i’m not going to claim that it’s a masterpiece of elegant code or anything, but it works well enough to test out if things on the server are working.

About the only really annoying thing is that the key bindings in Android Studio can’t be mapped to Vim, so i would occasionally write :w! in the code. i knew Java from a previous job, and it’s bloody java so it’s not like it’s hard to find things. If you like, you can play with the demo using the built in emulator or “side load” it to any device you’d like.

That done, i turned my attention to Apple Push Notification System (APNS). And there in lies my tale of woe.

Apparently, to build iOS apps of any sort, i need to get xCode (an compiler environment for Macs), an iOS device, and a $99 Apple iOS developmental license. i was able to borrow a mac from work, i had my wife’s old iPad2 as the device, but had to pony up the $99+tax fee out of pocket. If i were to go the cheapest possible route, i’d have to pay $500 for a mac mini + $250 for an ipad mini + the $100 iOS development license. Why do i need a real device? Because the APNS requires a real device. It doesn’t work in an emulator. So, close to a grand out of pocket vs what i would have had to spend for GCM, which was $0.

Receipts filed for next years tax season, i decided to dig in. And that’s when i encountered ObjectiveC.
[[intent alloc] withSarcasmAndSlightMalice: @Seriously];?
Granted, i could have used Swift, but in either case, this is a language specific to writing apps for a specific platform. That’s a bit like having to learn Esperanto to order a lunch. i won’t get into the oddities of ObjectiveC (and there are a lot of them), but i’ll note that you really do need to use xCode to write the app. This is because there are whole portions of the app that can only really be built by dragging connectors around like you’re running a 1920’s era switchboard.

Then there’s the certificates. God, the certificates. i get TLS. It’s wonderful for security because it’s using really large bits of math in order to prove you are you and i am i, and you get it for free with browsers. Granted, for this, i need a certificate saying who i am. i then need another certificate that is signed against the first certificate to indicate the app is my app (ok, that seems fair). i then need to export the certificate parts, convert them to a different format, so i can turn them into another format and store them on my server for the connection back to the APNS server which (obviously) needs to have the certificate it knows about me.

This is why OAuth is popular.

Oh, and in addition, i’ll need to get the long, complicated device id off of the iOS device using a completely program that tries to sell you Kanye West and Pitbull songs so that it can be one of the 1000 devices you’ve specifically blessed to run this app outside of the closely guarded and completely arbitrarily governed sanctum they’ve established. Because, obviously, why would anyone ever want to build testing or demo apps?

Anyway, back to the demo app. Once you’ve got the crux of things (hopefully) working, you then have to try sending the actual notification. This is done using a binary format. Well, actually, one of two. There’s the oldest format which consists of a “0” + the length of a token + the token + the length of the message + the message in JSON format. The message JSON needs to be constructed according to a reasonably simple structure, and after you feed the beast, you get a response back indicating if the message was accepted. Most examples will refer to this format.

Then there’s the new format, which has a 2 + the length of all the items combined + the items [ 1 + length of the token + token, 2 + length of the message + the message, 3 + length of an id + the id, 4 + length of the expiration time + expiration time, 5 + length of the priority flag + priority flag]. This information is bundled up and fed to the beast and you get a response back indicating if the message was rejected. You get nothing back if it was accepted. Mind you, it could take a second or two to process the data, so you ought to hang around a bit waiting, just in case. Oh, and after the third message or so, the server will drop your connection, so you’ll have to reconnect. Golly, and because i nearly forgot, you’ll also want to hang onto messages you’ve sent recently because there’s no ACK and you’ll need to resend them later if they’re requested. Because i’m a nice guy, here’s a PHP script built off the most common example that uses either v0 or v2 APNS.

Oh, and also? Even if the server accepts the message and says there’s no problem. There may be a problem and the service will just silently eat the message. So, if you do something silly and screw up the length of a field, or don’t specify the expiration with just enough buffer, or maybe have a typo in the JSON data, the message is helpfully shot into the void never to be heard from again. This makes debugging a bit of a challenge, but hey! It’s apple! It just works! Unless it doesn’t, in which case it’s your fault and you should fix it, you moron.

(Honestly, it would be lovely to have a service set up that would actually verify the data you’re sending in to it and see if it’s correct. i should write one.)

The Sorrow

In which the hero walks to the sunset behind La Brea

Much swearing and twiddling later, it is possible to use APNS to send data to apps. i’m still missing a few additional things, and will have to create them before this goes fully live, but it’s good enough for early prototyping.

So, in theory, i’m over the worst of it. i should be happy, right?

Well, no. You see, having built this, i recognize that this is more than a one or two week process. i’m going to need to have a stable mac environment to work in for a while. That means i’m going to have to swap the loaner macbook for a mini. i’m also going to have to get a mini for my home because there will be days i’ll want to work from home, and i’ve discovered that remote desktop services on macs (or at least on the loaner) is… problematic. Fortunately, i’ll be able to toss the little crapbox into the garage where it can spin it’s fans up and not keep the rest of the house awake. It also means that i’ll be hanging onto the wife’s old iPad for a while, mostly for the same reasons (hopefully, i won’t have to get a second one of those).

Apple, the craft glitter of computing.

Once that i am actually DONE with this crap, i might see if i can put Linux on the mini and use it as a NAS or something. Having hot plugged a USB 3.0 NTFS drive into the mac and not seeing it show up does not bode well, but that could well be a FUSE solvable problem.

All i can say is that this experience has not endeared me to apple at all. (Seriously? Damn near every platform out there has used Ctrl as the principal modifier key, but apple decides to use it’s own, and has the key sit right next to the spacebar? No wonder they want you to use the mouse for everything.) Frankly, sometimes it seems that Apple exists just to ebb joy out of my life.

But, people love them, and are happily effervescent about the Emperor’s wardrobe choices. Even if he’s standing on his thrown, gyrating his hips, and making helicopter noises.

:: Cutting Out the Brain Candy

i’ve come to realize that i’ve been a horrible slacker.

Not just here (this blog exists pretty much for my own evil purposes as well as my private well to scream down), but in a lot of things. i’ve become a consumer. i’ve come to realize that when i’m bored, i simply consume. i dine on a regular digital diet of fatty brain food like reddit, twitter, and crappy youtube videos.

i’ve come to realize that i’ve started consuming more than i create, by quite a lot, really. i’m not sure when (although i have an inkling) and i’m not sure why (again, inkling), but i’ve let myself stop creating things. To me, at least, this is horrible.

i used to drive myself constantly to make something. Heck, i used to post inanity to this blog daily. i gave myself a strict schedule to write a new chapter of a serial thing every two weeks (with a minimum of five pages). i hacked on damn near anything i could get my hands on and have bins filled with bits an pieces to dozens of unfinished projects. i would never give myself the opportunity to be bored.

Yeah, that’s no longer a problem.

Which is a problem.

As an engineer, i hate problems, so i’m going to solve that particular problem.

i’m making it a point to make at least one something every month. That includes conceptualizing, building and documenting whatever in however long the month is. Big things may get divided into littler things, but each thing needs to be able to stand alone as a thing. If i don’t finish one thing in a month, i have to finish two the next month. Oh yeah, and blog posts don’t count (well, not as just a blog post, gotta be more to it than that. Just writing a post is too easy.)

i’ve tried being bored. It sucks and makes my brain fat. i’m not going to do it any more.

:: F*ck You, Brain.

i feel fortunate that i am reasonably mentally healthy. There’s a lot of people who actually have depression, bi-polar disorder, ADD and other conditions and those people have my respect.

That said, i can understand a tiny fraction of what they go through. i tend to be very hard on myself. i tend to favor criticism more than complements. This is normal, of course, and chances are exceptionally good you do the same thing. i know this because a few years ago when i felt i was at a very low point i wanted to understand what the hell was wrong with me. It was a pattern i found myself falling into and since i didn’t enjoy it, i wanted to know what i could do to solve the problem.

Turns out, little, since we as a species are pretty firmly hardwired for it. i found that at least for me, understanding what was going on did help. Later on, finding books like You Are Not So Smart really helped me understand even more of this sort of self inflicted crap.

Part of this is also my industry of choice. Long ago, i consigned myself to be a tradesman in the midst of artists. There are programmers that craft things of magic and joy. These are wonderful creations that improve life in this veil of tears immeasurably. Granted, those things often need to be maintained and supported. At times, bits need to be built out of things stronger than pixie dust and gossamer, and i take a bit of joy trying to make sure that folks don’t know that some of the “magic” is now rebar.

i’ll also admit that i’m not the best at what i do. Even after working a lifetime in the industry. There are others that i work with that make me look like an amateur. These are people i can greedily learn from, and i’m damn thankful of the opportunity, but it also means that in a culture that spotlights “Rock Stars” and “Ninjas”, i’ll be the guy near the back of the auditorium clapping.

All of this can be a bit… i don’t want to say depressing, but it’s certainly draining. i’ve had numerous, very dark thoughts, but generally can dismiss them. There are days where i feel reasonably good about myself and completely understand that in very short order, my mood will be completely reversed by something. i’ve learned that this swing happens in both directions, and frankly it’s best to simply not let the pendulum swing that hard in any direction.

Take, for instance, right now. At this time, i’m feeling very low. i was responsible for a bit of infrastructure. i drafted the design, implemented code, and got it working, and felt reasonably good about myself. Projects started relying on it. Outside groups started playing with it. There was a significant challenge, i needed a bit of help, and we met the challenge with some tweaking.

The code has since been rewritten by someone far younger than me and is, in many respects, far better than what i originally built. It’s lighter weight, more responsive and probably more maintainable than what i had built. i already see ways that it could be improved and expanded.

It’s also probably going to be a failure point because of things outside of my control, which may cause the projects that were relying on it to also fail. None of the breaking points are my fault, and i’ve noted what the problems are, how to address them, and what actions are available to all. When i dismiss fault, i am being very clear. The failure is due to a behavior in a system i did not code for nor did we clearly understand at the onset of the project. If this system is removed, the fault is also removed, however this system is required for it’s own reasons. It’s a bit like saying “Well, the crop harvest failed due to the dam break.”

Still, it’s damn hard for me to shake the “This is your fault, and you suck as a human being” mindset. The program was written in a language and construct i wasn’t fully familiar with. The younger engineer is, so that’s why the code is better after his attention. The failing system is one that i am also unfamiliar with, and the subsystem that is an issue came as a surprise to a good many folk. i view all of these as being “excuses”. My psyche demands that excuses don’t matter, only results. (Yes, i was brought up in a strict, military household, why do you ask?)

People talk about failure as a benefit. It’s how one learns. It’s inevitable, and constant, and what makes success so remarkable is that it’s uncommon enough to be remarkable. This does not make failure any less pleasant. Culturally, and personally, it’s a stigma. Feeling that way is irrational, but very common.

At this point, i’ll also think about the various other projects that i feel the need to accomplish, and the sorry state that they’re in, the 60+ articles moldering in my unread queue and a thousand other reasons i have to beat myself up. i should be accomplishing more. i’m not. i should be more creative. i’m not. “Nobody else knows how big a screw up you are” as Mr. Savage points out, but i am acutely aware of it.

Like i said at the top, there are folks out there with serious medical issues who are constantly struggling with far bigger demons than me. i admire every day that they succeed in beating those demons and they have every right to scoff at me and my personal pity party. If you’re one of those, i welcome your well earned derision.

Possibly, you may feel the same way i do. Here’s to letting you know you’re not alone.

And thanks for reading my bit of personal therapy.

:: MacGuffin!

Last night, a bunch of friends watched a show called Scorpion. i had no idea it was on, nor did i really have any interest in watching it, but they took to twitter to comment about how bad it was. From what i understand (again, having little interest in actually watching the show) it was the usual Techno-Crime thriller about a group of “geniuses” who use Technology to battle Crime. The show featured such techno crime fighting as driving a Porche behind a plane so that the heroes could grab a network cable dangling beneath, apparently because nobody on the plane had a cellphone or anything that could do wifi, and tossing a few USB sticks would have probably been out of the question as well.

Honestly, it reminded me of something my Sister in law (who is in the entertainment industry) yelled at me while watching a similar sort of show “It’s Entertainment! It doesn’t need facts!”

That flash of insight into the mind of what’s making this stuff really helped me understand quite a bit. i mean, how does one possibly counter something like that? It did, however, make me understand that there’s a huge opportunity there for folks.

i realized that what was sorely lacking was a show that treated the rest of the planet the way that technology is horribly abused. i’d call it “MacGuffin“!

It starts off with a sweeping shot of the city that’s central to the story:

We see our hero, Jack MacGuffin engaged in a chase with his partner Alan Smithee.

Smithee: “Jack, they’re getting away, we need to go faster.”
MacGuffin: “Alan! Quick, grab the wheel and start driving!”

Alan grabs the wheel and starts pounding on the brake and gas pedals of the car. With a cloud of tire smoke, the car lunges ahead and quickly overtakes the van containing the enemies, a group of 80 year old women who had once seen a crime drama and were acting out the overwhelming influence of that corrupting medium.

MacGuffin and Smithee soon learn of a gang that plans on flooding the streets of Phoenix with drugs and white slavery purchased using counterfeit Monopoly money.

Smittee: “Jeez, imagine the number of houses you could buy with that kind of money”
MacGuffin: “Houses? you could fill the block with hotels.”

They reach their biggest break when they spot a “Classified Ad” in a local “newspaper” that they quickly correlate to a “phone number” using a copy of “Yellow Pages” and with that phone number in hand, race to the abandoned warehouse of the gang.

Before they go in, they quickly construct some guns out of some tinker toys and sheet metal they find outside, before bursting in and taking out the gang with a 2 X machine.

Of course, my biggest fear would be that it actually would be made.

Blogs of note
personal Christopher Conlin USMC memoirs of hydrogen guy Henriette's Herbal Blog
geek ultramookie

Powered by WordPress
Hosted on Dreamhost.