jr conlin's ink stained banana

So, flickr got a redesign.

A lot of people love it. i'm not one of those.

Mind you, what i liked about Flickr was that it presented me images that were meant for public display. They were easy to find and really took effort to show them off. This meant that a lot of folks saw and remarked on your images. Folks who wanted to share them could use Creative Commons licenses and offer them up for general consumption. i'm astounded that a few of my images were used that way. i'm also sure that a lot of images that weren't marked as CC were also used and i'm positive some of mine were abused as well.

It doesn't really matter, since that's what that site was about. Heck, they even had a page full of images considered to be Most Interesting.

So, with the latest image focused iteration, i should be over the moon right? Images are not only on display, but smooshed together in one big firehose of imagery goodness pointed at my optical nerves.

Well, no, i'm not.

See, there's this thing called "White Space" that lets you appreciate what you're looking at. It's why the Mona Lisa is displayed like this

and not like this

Granted, there have been a few museums i've been to that do something similar to this. That doesn't mean that it's a good idea, and generally, there are other than aesthetic concerns for why they may do that. i'll add that "white space" isn't always "white". It means uncluttered areas that don't distract. The much maligned windows8 interface even follows that practice by having very simple graphics, all smooshed together like a cubist version of Twister.

So, the layout thing isn't my cup of tea. So, there's the 1TB of image storage. That's worth something, right?

Eh, no. Not really.

Much like how Y!Mail once touted that it offered 1GB of mail storage, there are other services that offer a lot more. Imgur and SmugMug offer fairly unlimited image storage, and have for years. Likewise, while kind of the opposite of flickr in the discovery and usability side, Facebook and Instagram also offer loads of storage as part of their service. In some respects, flickr's announcement could easily be interpreted as "We're limiting the amount of storage to only 1TB".

i'll also note that you still can't pull images via RSS, the price of not bugging visitors with ads has doubled, you're feedback and comments are less important, as are details like how the photo was taken if you're interested in photography at all. Some of the reasons that Flickr was awesome for folks who want to share their photos and develop as photographers. Heck, i can't even learn a way to say "Hello" in Kurdish unless i carefully hover over my icon.

That's the other thing, is it me or does this layout look oddly familiar? i know how long it takes to roll out products, but seriously, did the design teams all go to the same talk or something?

So, yeah, not really as impressed as i could be, sorry. i'm sure i'm in the vast majority, and not 100% on board with this whole Flat Design thing that has everyone's knickers in knots, kind of like how i wasn't awestruck by Aqua or Aero either, and heaven knows those aged well.

i'm working on a project that will require Websockets. Actually, it'll require quite a few of them. i suspect that, by the time i'm done, i may need to have servers holding open 400,000,000 of the little darlings, with about 20% churn as connections are dropped and reopened. Since The Cloud doesn't really exist for me (this will probably be a ranty follow up post, but let's just say that i'm working at the level that most folks call "The Cloud", so i have to actually make stuff work), i need to figure out how to make this happen.

Somewhat surprisingly, there's very little information online about this. It could be that the folks that have figured this out have decided that it's proprietary information that they can use for whatever makes them happy, but since i don't have that problem, i'm going to happily post what i've found out here.

By the way, i would be thrilled if someone were to point out my plate ignorance and provide a set of test numbers and profiles that correct whatever errors i've made. i'm publishing this because i couldn't find anything like this, not because i consider it definitive.

Knowing what you know
i'll note that a fair bit of this was done with the able assistance of Ryan Tilder. One of the problems was determining what to use to actually test things out. There are a lot of technologies out there that promise a good deal, but as i learned at the previous employer, promises are nothing against real numbers. Node.js and Python Tornado both promise that they hold open a phenomenal number of sockets, and a quick mock program in each showed that they were reporting lots of socket connections. The problem was that when we did a netstat, we didn't see anywhere near the same number of active sockets open. We're not 100% sure why this was, but they were both fairly quickly eliminated from testing.

Eventually we settled on two candidate applications. For the socket client app, we picked Go, which not only reported back the same number of active sockets as netstat did, but also kept a fairly large number of sockets open at any time. The other candidate server was a simple Java/Netty demo application. This also just echoed back whatever was tossed at it.

For test instances we picked an AWS Small, Medium and Large as servers. For clients we spun up a number of Smalls. (We very specifically did not pick Micros because they are the instance time most subject to external effect. Such as, say, being a micro on one of the boxes when we were doing a test.)

We applied the following tweaks to the boxes to try and boost the available number of handles:


echo "
## Increase the file handle space
# Increase the ipv4 port range:
sysctl -w net.ipv4.ip_local_port_range = 1024 65535
# General gigabit tuning:
net.core.rmem_max = 16777216
net.core.wmem_max = 16777216
net.ipv4.tcp_rmem = 4096 87380 16777216
net.ipv4.tcp_wmem = 4096 65536 16777216
net.ipv4.tcp_syncookies = 1
#
# This gives the kernel more memory for tcp which you need with many (100k+) open socket connections
net.ipv4.tcp_mem=50576 64768 98152
net.core.netdev_max_backlog=2500
" >> /etc/sysctl.conf
# AWS sets the ulimit to "unlimited". This is optional:
# ulimit -n 999999
# modify /etc/security/limits.conf
echo "
* soft nofile 50000
* hard nofile 50000
" >> /etc/security/limits.conf
# modified /etc/sshd.conf
echo "
UsePrivilegeSeparation no
" >> /etc/sshd/sshd_config


i then built a very simple Go based "pounder" that opens up sockets on whatever server you want, and then sends data across the port every specified period. For the servers, we either ran a netty demo server, or the craptastic "srv.go" in go_pound that just echoed back what it got. We don't really care about it doing much more than writing and clearing the socket data. We wanted it to do more than "just exist" since the actual program will also be doing more than that.

Knowing what you don't know

You'll also note that we're not going through the ELB at this point. We were trying to reduce the number of potential variables, although i will note that for websockets, you have to use "TCP" as the protocol, not HTTP. ELB aggressively terminates HTTP socket connections.

We also knew that since sockets are effectively file handles, we wanted to eliminate as many open files or extra connections we could. i won't go into details about what we did, but basically we got things to under 100 active file connections which we felt was a reasonable number of system level handles.

We did encounter some unusual things. One being that smalls are VERY UNHAPPY when you use more than about 24K sockets. (24,500 is kind of pushing things.) At that number, SSH becomes fairly unstable and you may have to reboot the box externally. This may be an issue with how we set net.ipv4.tcp_[rw]mem and how AWS actually provisions memory. i'll admit that the high number i picked there was a bit aggressive and potentially optimistic, but since we were creating fairly specialized boxes and were hitting max, i'm not really concerned. If you're planning on sticking to smalls, you may want to do a tad more homework and pick a more appropriate number.

Knowing what you didn't know
So… the numbers.

Well, here's what we saw (all numbers rounded down)

Notes
¹Go topped out at 120K when being pounded by requests every 8s. When we increased this number to 20s, it was able to handle the load. >20s between transactions is a more realistic number for our project, but i like being aggressive for testing like this.
²Larges also seemed to top out at just over 200K connections. Again, this may be something related to the tcp_mem setting. We may need to do additional testing on this.
³Netty was able to handle both 8s and 20s transaction, with CPU being the more limiting factor.

Not Knowing the Unknowns
Is this a complete win for Netty? Possibly. There are a few other factors that i'll have to consider before switching over to just that (logging, operational and developmental familiarity, etc.) Likewise, running enough "larges" to handle the problem may solve the issue to the extend that building out the Java server isn't cost effective.

One thing Ryan points out is that when you start dealing with Very Large numbers of concurrent connections, you start to seriously tax your CPUs. If you have a lot of very active channels, that tax goes up a great deal more. It may make sense to opt for CPU friendly configurations, more so than memory heavy ones. i'll try to remember to post up what sort of configurations we found work best for us if we stay with AWS as a solution.

Still, these are the numbers that we came up with. Hopefully, they'll be reasonably useful to you.

i wrote A Quick Refresher seven years ago, and i think it needs a bit of correction:

Please note:

The following rules need not only apply to the internet.

1. Don't be an asshole.
   1. An asshole is someone who thinks of themselves first, and without consideration of anything else.
   2. An evil bastard is someone who actively seeks not only to be an asshole, but to inflict pain and suffering on others.
   3. Nobody likes an asshole, and everyone hates evil bastards.
   4. Assholes should be ignored or treated like infants (at best). Evil bastards should be opposed.
2. If you do something nice (like link to someone else's page, hold the door open, upvote a post), never expect to see the favor returned.
3. People who ask/plead/demand your generosity, should be avoided. Their actions should be enough to warrant your contributions.
4. It is possible that someone else may share your opinion. This neither validates that opinion nor implies that they agree with you in all other aspects.
   1. Your point of view is only valid because you have no idea what the person who opposes your view has experienced.
   2. Their point of view is equally valid, unless, of course, they're just being an asshole or evil bastard.
   3. Correcting a person to their face is always better than scolding in public. Doing the latter without trying the former makes you someone who is intentionally inflicting harm on someone else so that you look better. (See "evil bastard")
   4. Tragedy and comedy are both sides of the same coin. This is why folks find slapstick funny. i'll come over and help you back up, but don't be offended if i'm giggling.
   5. If i do something stupid, i'm ok with you laughing while you help.
5. Always presume the worst until pleasantly suprised otherwise.
6. Nobody actually rolls on the floor, laughing out loud.
7. Most of the stuff you see is done by a lot less folks than you think.
   1. Stuff breaks.
   2. When it breaks, they want to fix it
   3. It may take them a while to fix it because of other things that are even more broken that you don't know about
   4. It's not that they don't like you, it's just that it's hard for five people to respond to several million messages a day.
8. There are people smarter than you.
9. There are people dumber than you.
10. On an average day, half of the people you meet will be in each category.
11. Nobody expects courtesy, it confuses them. Use it with abandon just to catch them off-guard.
12. The best reason to be nice is that it means the person is less likely to screw you over.
    1. Not that there's any reason for them not to try anyway.
13. Friendship and trust must be earned every day.
14. Be open about what you accept, but strict about what you provide.
15. Good and Evil are relative ideas and need context.
16. Never agree to someone's protocol/ideology without really understanding what that protocol/ideology is.
17. Always understand that every protocol/ideology is subject to corruption. If you can't get it from the original source, it's probably invalid.
18. Read the documentation.
19. Never blame the user for a failure of the protocol. Simply report the error as clearly as you can and move on.
20. Errors are errors and no amout of argument will change them. Work around them if you need to get something done or consider using another system.

Thank you, please pull ahead to the second window, and have a nice day.

Ah St. Paddy's Day. A day when everyone is a bit Irish and tuck into a taste of the Emerald Isle.

Well, no, not really. No sane, sober person would, but let me explain a little about why i tend to enjoy Pasta or Thai instead of Corned Beef and Cabbage.

First off, let me introduce you to my Grandfather. That's him, tending the bar that he ran for a few years in Brooklyn. Sadly, the pub he ran closed, partly due to his tendency to be more generous than business savvy, but that did have one interesting side-note.

Far more often than my Grandmother appreciated, he'd come home with someone he'd met while at the pub. His arms would be filled with unusual ingredients and he'd usher his wife from the kitchen and away from the shortbreads and creamed tripe stew, and soon the apartment was filled with exotic scents and tastes from strange and wondrous lands, like Germany or Italy. My sainted Grandmother generally hated whenever he did this, but my Mom and her siblings didn't.

Irish cooking tends to "use everything but the grunt", and they pretty much mean it. Traditional Irish country food tends to use a, frankly, terrifying amount of pork, cabbage and potatoes, most of it boiled in pots above peat fires. One need remember that it was the loss of the Potato that pretty much devastated the country and lead to massive starvation. This was not a country with a wide and varied diet. Special occasions might include a bit of roasted lamb, but that tended to (literally) eat into the export business.

This might also explain why folks drank, but only a wee bit.

Mind you, modern Irish cuisine is a helluva lot better now and has even included things like fish and beef. i'll also note that Irish have taken to, ahem, "embracing the foods of other cultures" with a surprising enthusiasm to those not familiar with traditional fare.

So, aye, i'll not be partaking of the bacon knot and cabbage this year (or the next), and i prefer my corned beef peppered and sliced thin from the deli counter. i'm not exactly sure how many shepherds kept meat grinders in the dairy, either.

Still, raise a glass and a toast. To my ancestors from Roscommon, Tabhair dom ar ais ar mo dhia damanta caorach, bastaird tú!