A lot of people love it. i'm not one of those.

Mind you, what i liked about Flickr was that it presented me images that were meant for public display. They were easy to find and really took effort to show them off. This meant that a lot of folks saw and remarked on your images. Folks who wanted to share them could use Creative Commons licenses and offer them up for general consumption. i'm astounded that a few of my images were used that way. i'm also sure that a lot of images that weren't marked as CC were also used and i'm positive some of mine were abused as well.

It doesn't really matter, since that's what that site was about. Heck, they even had a page full of images considered to be Most Interesting.

So, with the latest image focused iteration, i should be over the moon right? Images are not only on display, but smooshed together in one big firehose of imagery goodness pointed at my optical nerves.

Well, no, i'm not.

See, there's this thing called "White Space" that lets you appreciate what you're looking at. It's why the Mona Lisa is displayed like this

and not like this

Granted, there have been a few museums i've been to that do something similar to this. That doesn't mean that it's a good idea, and generally, there are other than aesthetic concerns for why they may do that. i'll add that "white space" isn't always "white". It means uncluttered areas that don't distract. The much maligned windows8 interface even follows that practice by having very simple graphics, all smooshed together like a cubist version of Twister.

So, the layout thing isn't my cup of tea. So, there's the 1TB of image storage. That's worth something, right?

Eh, no. Not really.

Much like how Y!Mail once touted that it offered 1GB of mail storage, there are other services that offer a lot more. Imgur and SmugMug offer fairly unlimited image storage, and have for years. Likewise, while kind of the opposite of flickr in the discovery and usability side, Facebook and Instagram also offer loads of storage as part of their service. In some respects, flickr's announcement could easily be interpreted as "We're limiting the amount of storage to only 1TB".

i'll also note that you still can't pull images via RSS

, the price of not bugging visitors with ads has doubled, you're feedback and comments are less important, as are details like how the photo was taken if you're interested in photography at all. Some of the reasons that Flickr was awesome for folks who want to share their photos and develop as photographers. Heck, i can't even learn a way to say "Hello" in Kurdish unless i carefully hover over my icon.

That's the other thing, is it me or does this layout look oddly familiar? i know how long it takes to roll out products, but seriously, did the design teams all go to the same talk or something?

So, yeah, not really as impressed as i could be, sorry. i'm sure i'm in the vast majority, and not 100% on board with this whole Flat Design thing that has everyone's knickers in knots, kind of like how i wasn't awestruck by Aqua or Aero either, and heaven knows those aged well.

When i joined Company, Inc. a year ago, i signed up to work with fantastic folks who wanted to change the world. When i arrived, Company, Inc was just starting a plan to rid hunger from the third world by creating self-piloting food delivery systems to ensure against corruption. During that time i saw the release of several projects that i knew would bring about great things. It is with much sadness that i am writing this note to tell you that i'm leaving.

You see, while i appreciate that whole "change the world" thing, it's a lot of work. It means being personally invested in actually seeing things through. That means that i have to battle not only outside the safe company walls to get folks to see the value of what i'm doing, but internally to convince others why they need to stay on track. Fighting battles are like, well, fighting battles, and there's a reason that people don't do that sort of thing on weekends or on vacations.

What really made the difference for me was when that truckload of money appeared in my driveway. Mind you, that was one of the "great things" that i was talking about earlier. Company, Inc. looks fan-fucking-tastic on resumes and let me tell you, launching the WidgetTron 3000 tripled the number of recruiting emails i got.

Sure, i could have fought to make sure that the WidgetTron 3000 we launched this quarter actually did allow near universal document parsing like it was originally intended, but once we moved it to the cloud and added Facebook Integration and weather updates, well, you know how hard it can be to stop linking paperclips together? Yeah, adding features can be like that. As a bonus, i managed to grow my tiny empire from just myself to a team of over 200. The WidgetTron consumed nearly every available resource in the company. Most folks working on the project had little idea what the actual use case of the product was ("use cases will come later" Ha-ha!) or that the key component was the burning eye of evil we hitched up in a remote colo. It's really amazing the sorts of things you can get developers to do if you lure them with "sexy" technologies like node.js and mongo.

Oh sure, WidgetTron 3000 is powered by puppy souls and i still resoundingly dispute those CDC reports about it causing the undead to walk among us, but the good news is that it has a skeleton staff (in this case, literally) and will be forgotten about in a quarter or two.

So, truly, it is with sadness that i write this contractually required goodbye as i leave you all to deal with the Class 1 superfund mess i leave behind. i'm off to live a life that makes Scrooge McDuck look like a skid-row bum.

If you'd like to stay in touch, my email is
IWillNeverReadThis@yahoo.com

Best of luck in the future!

(suckers)

i'm working on a project that will require Websockets. Actually, it'll require quite a few of them. i suspect that, by the time i'm done, i may need to have servers holding open 400,000,000 of the little darlings, with about 20% churn as connections are dropped and reopened. Since The Cloud doesn't really exist for me (this will probably be a ranty follow up post, but let's just say that i'm working at the level that most folks call "The Cloud", so i have to actually make stuff work), i need to figure out how to make this happen.

Somewhat surprisingly, there's very little information online about this. It could be that the folks that have figured this out have decided that it's proprietary information that they can use for whatever makes them happy, but since i don't have that problem, i'm going to happily post what i've found out here.

By the way, i would be thrilled if someone were to point out my plate ignorance and provide a set of test numbers and profiles that correct whatever errors i've made. i'm publishing this because i couldn't find anything like this, not because i consider it definitive.

Knowing what you know
i'll note that a fair bit of this was done with the able assistance of Ryan Tilder. One of the problems was determining what to use to actually test things out. There are a lot of technologies out there that promise a good deal, but as i learned at the previous employer, promises are nothing against real numbers. Node.js and Python Tornado both promise that they hold open a phenomenal number of sockets, and a quick mock program in each showed that they were reporting lots of socket connections. The problem was that when we did a netstat, we didn't see anywhere near the same number of active sockets open. We're not 100% sure why this was, but they were both fairly quickly eliminated from testing.

Eventually we settled on two candidate applications. For the socket client app, we picked Go, which not only reported back the same number of active sockets as netstat did, but also kept a fairly large number of sockets open at any time. The other candidate server was a simple Java/Netty demo application. This also just echoed back whatever was tossed at it.

For test instances we picked an AWS Small, Medium and Large as servers. For clients we spun up a number of Smalls. (We very specifically did not pick Micros because they are the instance time most subject to external effect. Such as, say, being a micro on one of the boxes when we were doing a test.)

We applied the following tweaks to the boxes to try and boost the available number of handles:


echo "
## Increase the file handle space
# Increase the ipv4 port range:
sysctl -w net.ipv4.ip_local_port_range = 1024 65535
# General gigabit tuning:
net.core.rmem_max = 16777216
net.core.wmem_max = 16777216
net.ipv4.tcp_rmem = 4096 87380 16777216
net.ipv4.tcp_wmem = 4096 65536 16777216
net.ipv4.tcp_syncookies = 1
#
# This gives the kernel more memory for tcp which you need with many (100k+) open socket connections
net.ipv4.tcp_mem=50576 64768 98152
net.core.netdev_max_backlog=2500
" >> /etc/sysctl.conf
# AWS sets the ulimit to "unlimited". This is optional:
# ulimit -n 999999
# modify /etc/security/limits.conf
echo "
* soft nofile 50000
* hard nofile 50000
" >> /etc/security/limits.conf
# modified /etc/sshd.conf
echo "
UsePrivilegeSeparation no
" >> /etc/sshd/sshd_config


i then built a very simple Go based "pounder" that opens up sockets on whatever server you want, and then sends data across the port every specified period. For the servers, we either ran a netty demo server, or the craptastic "srv.go" in go_pound that just echoed back what it got. We don't really care about it doing much more than writing and clearing the socket data. We wanted it to do more than "just exist" since the actual program will also be doing more than that.

Knowing what you don't know

You'll also note that we're not going through the ELB at this point. We were trying to reduce the number of potential variables, although i will note that for websockets, you have to use "TCP" as the protocol, not HTTP. ELB aggressively terminates HTTP socket connections.

We also knew that since sockets are effectively file handles, we wanted to eliminate as many open files or extra connections we could. i won't go into details about what we did, but basically we got things to under 100 active file connections which we felt was a reasonable number of system level handles.

We did encounter some unusual things. One being that smalls are VERY UNHAPPY when you use more than about 24K sockets. (24,500 is kind of pushing things.) At that number, SSH becomes fairly unstable and you may have to reboot the box externally. This may be an issue with how we set net.ipv4.tcp_[rw]mem and how AWS actually provisions memory. i'll admit that the high number i picked there was a bit aggressive and potentially optimistic, but since we were creating fairly specialized boxes and were hitting max, i'm not really concerned. If you're planning on sticking to smalls, you may want to do a tad more homework and pick a more appropriate number.

Knowing what you didn't know
So… the numbers.

Well, here's what we saw (all numbers rounded down)

Notes
¹Go topped out at 120K when being pounded by requests every 8s. When we increased this number to 20s, it was able to handle the load. >20s between transactions is a more realistic number for our project, but i like being aggressive for testing like this.
²Larges also seemed to top out at just over 200K connections. Again, this may be something related to the tcp_mem setting. We may need to do additional testing on this.
³Netty was able to handle both 8s and 20s transaction, with CPU being the more limiting factor.

Not Knowing the Unknowns
Is this a complete win for Netty? Possibly. There are a few other factors that i'll have to consider before switching over to just that (logging, operational and developmental familiarity, etc.) Likewise, running enough "larges" to handle the problem may solve the issue to the extend that building out the Java server isn't cost effective.

One thing Ryan points out is that when you start dealing with Very Large numbers of concurrent connections, you start to seriously tax your CPUs. If you have a lot of very active channels, that tax goes up a great deal more. It may make sense to opt for CPU friendly configurations, more so than memory heavy ones. i'll try to remember to post up what sort of configurations we found work best for us if we stay with AWS as a solution.

Still, these are the numbers that we came up with. Hopefully, they'll be reasonably useful to you.

i wrote A Quick Refresher seven years ago, and i think it needs a bit of correction:

Please note:

The following rules need not only apply to the internet.

1. Don't be an asshole.
   1. An asshole is someone who thinks of themselves first, and without consideration of anything else.
   2. An evil bastard is someone who actively seeks not only to be an asshole, but to inflict pain and suffering on others.
   3. Nobody likes an asshole, and everyone hates evil bastards.
   4. Assholes should be ignored or treated like infants (at best). Evil bastards should be opposed.
2. If you do something nice (like link to someone else's page, hold the door open, upvote a post), never expect to see the favor returned.
3. People who ask/plead/demand your generosity, should be avoided. Their actions should be enough to warrant your contributions.
4. It is possible that someone else may share your opinion. This neither validates that opinion nor implies that they agree with you in all other aspects.
   1. Your point of view is only valid because you have no idea what the person who opposes your view has experienced.
   2. Their point of view is equally valid, unless, of course, they're just being an asshole or evil bastard.
   3. Correcting a person to their face is always better than scolding in public. Doing the latter without trying the former makes you someone who is intentionally inflicting harm on someone else so that you look better. (See "evil bastard")
   4. Tragedy and comedy are both sides of the same coin. This is why folks find slapstick funny. i'll come over and help you back up, but don't be offended if i'm giggling.
   5. If i do something stupid, i'm ok with you laughing while you help.
5. Always presume the worst until pleasantly suprised otherwise.
6. Nobody actually rolls on the floor, laughing out loud.
7. Most of the stuff you see is done by a lot less folks than you think.
   1. Stuff breaks.
   2. When it breaks, they want to fix it
   3. It may take them a while to fix it because of other things that are even more broken that you don't know about
   4. It's not that they don't like you, it's just that it's hard for five people to respond to several million messages a day.
8. There are people smarter than you.
9. There are people dumber than you.
10. On an average day, half of the people you meet will be in each category.
11. Nobody expects courtesy, it confuses them. Use it with abandon just to catch them off-guard.
12. The best reason to be nice is that it means the person is less likely to screw you over.
    1. Not that there's any reason for them not to try anyway.
13. Friendship and trust must be earned every day.
14. Be open about what you accept, but strict about what you provide.
15. Good and Evil are relative ideas and need context.
16. Never agree to someone's protocol/ideology without really understanding what that protocol/ideology is.
17. Always understand that every protocol/ideology is subject to corruption. If you can't get it from the original source, it's probably invalid.
18. Read the documentation.
19. Never blame the user for a failure of the protocol. Simply report the error as clearly as you can and move on.
20. Errors are errors and no amout of argument will change them. Work around them if you need to get something done or consider using another system.

Thank you, please pull ahead to the second window, and have a nice day.

Well, no, not really. No sane, sober person would, but let me explain a little about why i tend to enjoy Pasta or Thai instead of Corned Beef and Cabbage.

First off, let me introduce you to my Grandfather. That's him, tending the bar that he ran for a few years in Brooklyn. Sadly, the pub he ran closed, partly due to his tendency to be more generous than business savvy, but that did have one interesting side-note.

Far more often than my Grandmother appreciated, he'd come home with someone he'd met while at the pub. His arms would be filled with unusual ingredients and he'd usher his wife from the kitchen and away from the shortbreads and creamed tripe stew, and soon the apartment was filled with exotic scents and tastes from strange and wondrous lands, like Germany or Italy. My sainted Grandmother generally hated whenever he did this, but my Mom and her siblings didn't.

Irish cooking tends to "use everything but the grunt", and they pretty much mean it. Traditional Irish country food tends to use a, frankly, terrifying amount of pork, cabbage and potatoes, most of it boiled in pots above peat fires. One need remember that it was the loss of the Potato that pretty much devastated the country and lead to massive starvation. This was not a country with a wide and varied diet. Special occasions might include a bit of roasted lamb, but that tended to (literally) eat into the export business.

This might also explain why folks drank, but only a wee bit.

Mind you, modern Irish cuisine is a helluva lot better now and has even included things like fish and beef. i'll also note that Irish have taken to, ahem, "embracing the foods of other cultures" with a surprising enthusiasm to those not familiar with traditional fare.

So, aye, i'll not be partaking of the bacon knot and cabbage this year (or the next), and i prefer my corned beef peppered and sliced thin from the deli counter. i'm not exactly sure how many shepherds kept meat grinders in the dairy, either.

Still, raise a glass and a toast. To my ancestors from Roscommon, Tabhair dom ar ais ar mo dhia damanta caorach, bastaird tú!

i suppose you could also declare these as "Old Man Shouts at iCloud" stories as well.

As i get older (and one supposes, progressively more jaded), i've been increasingly frustrated by just how often folks build things that they haven't really thought out. Mind you, twenty years ago, when we were first building this crap, you'd be excused for not knowing that there's little difference between "Success" and "DDoS Attack" or that cracking SHA1 would be a simple matter of harnessing the power of an internal graphics processing unit, but you can't really say that anymore.

Technology, at least in theory, is supposed to build off of itself. We're supposed to be at that stage of the game where we've determined reasonable, objective facts about how to create and provide services. There will always be new tools introduced, but fundamentally the principles of data exchange and processing should really be kind of A Thing™ by now.

There are supposed to be a few Golden Rules that all engineers follow (or at least know of):

1. Be lenient about what you accept, strict about what you create.
   Other people will screw up, try to work with what you've got so that things continue to work reasonably well. File bugs against the crappy providers, but don't reject them unless it's total jibberish. That said, only output stuff in strict, predictable form that adheres to published standards so that everyone else isn't trying to accommodate you.
2. Play with the new, work with the old.
   Play and explore new technology. Revel in discovery and insight, but when it comes time to provide a service for more than a few hundred people, build it using boring, well tested, reliable tools and technologies. If the new stuff is good, eventually it too will become boring, well tested and reliable. Every project should dream of becoming "useful, but not hip" because that means you're now key to infrastructure.
3. Bad guys have infinite time and infinite resources
   If there's a flaw in your code, bad guys will find and exploit it. Protect your systems from yourself. If you can break it, even with some attack that is so remote that "nobody will ever do that", someone else will do that attack.
4. Don't fall in love with your code
   You may have come up with a good solution. Someone else may have come up with a better one. Always be willing to admit that someone else's answer may be better than yours (again, metrics beat statements), and be ready to move to it.
   Yes, i added this one after i wrote the original.
5. There is no One True Tech
   No technology is perfect and all applicable. Everything sucks for some reason. Only a fool uses one tool to solve all problems. The first question any maker of hammers should ask is, "Do i need a screwdriver?"
6. Open is better than closed
   Being open is like growing dozens or millions of extra brains. Suddenly, it's not just you or a few folks working on a problem, it's lots and lots of folks, constantly working on a problem. Of course, the Achilles Heel of any open project is getting someone else to care about it, but that's where the rest of the rules come into play.
7. Write your documentation for five year olds
   There are people smarter than you. They have different areas of expertise and interest than you do. Not everyone has spent the past decade focused on AES encryption wrapper math, phoneme pattern analysis, or efficient regular expression syntax. By all means, encourage folks to explore the wonders of elliptical compression algorithms, but understand that the reason they are looking at your library/code/application is because it's a small part of the larger problem they're trying to solve. Spending hours to learn interesting tricks in 64bit binomial storage so that they know to call your function with a proper set of prime Fibonacci numbers isn't going to make them love you.
8. Teaching makes you smarter
   You only really understand something when you can explain it to someone else. This ensures that you understand not only one aspect, but the various effects and inputs so that you can answer their questions. Teaching someone else will make you say "i don't know" a surprising number of times.
9. You are not alone
   This is a big one. If you're pounding on a problem, chance are, so is someone else. Or they've worked a problem similar to yours in some fashion. Ask and answer questions. Share knowledge and findings. Humans are capable of sharing information and that's what made us more successful than Bonobos. Use and contribute to that advantage every chance you can.

And, yet, every goddamn day, i see folks gleefully ignore those rules. So i wind up posting cranky notes, laughing at other people's completely avoidable failures, and otherwise turning into an angry old coot standing on his digital porch yelling at the kids these days.

Honestly, this is why the jocks and douchebags became Brogrammers who write crappy code. There's nothing preventing them from doing it and no tools for pointing out the obvious flaws.

Allow me to explain. Say, you've got two computers that want to talk to each other (chances are very good that this is the case). You have infinite ways that they can go about doing this, but a rather attractive way is to have something that you can read.

so instead of getting something like ^A<long stream of binary cruft>^D you get something more like:
HELO
FROM potamus.p@example.com
TO birdman.h@bar.com
SUBJ That thing i sent you
 
Did you get that thing i sent you?
 
.
It's nice and readable, with the added benefit that if you're ever stuck somewhere without a mail client, you can telnet to port 25 and send out an email. Provided you remember the order of the steps. And get the format of the lines right (pretty sure i didn't just now). And don't make spelling mistakes, etc…

There are lots of protocols that have Human Readable Formats. FTP, SMTP (e-mail), HTTP (web), etc. Some of these make sense, particularly if they're for short, on then off right away sorts of things. The problem is that if you want to do something really complicated, there's a lot of work trying to figure out what "HELO\r\n" means, vs. two characters. It becomes even more of an issue when you're talking about small devices that pay per KB (Hi there mobile devices!)

That's when looking at something like
<message to="alice@wonderland.lit" id="if00lu" >
<body>Hi you</body>
<body xml:lang="cy">Prynhawn da</body>
<nick xmlns="http://jabber.org/protocol/nick">Alice</nick>
<active xmlns="http://jabber.org/protocol/chatstates"/>
</message>

becomes absolutely terrifying. (really, i need to fully specify the namespace of a nick with a URL that doesn't resolve just so that the consumer knows what i mean?)

It's at this point that i cry out that anyone that proposes such a protocol is forbidden from using a client in order to use it.

There's a good reason that TCP is not human readable. It's never meant to be displayed directly to humans. Likewise, things like Finger and HTTP are meant to be displayed to humans. Yeah, yeah, you're probably not going to understand the stream of crap that adds up to a JPG image or movie, and depending on how aggressively a server compresses outbound data or collects additional content you might have issues figuring out what it all says, but a minimal HTML page is:
<html>
<head>
<title>This is a page</title>
</head>
<body>
No, really, this is valid HTML and will render just fine.
</body>
</html>


If you didn't have a browser you could just enter the command:
wget http://jrconlin.com/simple.html

and be able to mostly read what you got back.

Granted, most folks (as in > 99.999% folks) don't do that, and thus we have things like SPDY, which is optimized for Machines and is generally faster.

Really, the question becomes, how realistic is it that J. Random User is going to get a screen full of raw protocol they have to act on? If the answer is "Not often", do you really need to have it be "human readable", or can you just build a tool for the less than 1% of developers and systems folks that need to see what's going on?

i'm on fairly good terms with my Real Estate agent. He's smart, has a great contact list and unlike a lot of folks in his profession (sadly), he is a professional. That means that he thinks about this as a career, not a means to an end. i got a chance to talk with him today and we both keyed on a rather interesting point.

i'd been thinking about how there's a weird diminishing return on my reviews of places, while a lot of folks are (frankly) abusing things to get unwarranted favorable treatments.

He noted that with sites like AirB&B, it's not only important for you to look at the reviews for where you're staying, but for the folks that are opening their home to you to know about your history as well. That's when it hit me that i can easily see a future where you have a Reputation Score to maintain, kind of like you have a Credit Score now.

Better people have better scores, and worse ones have worse scores. The score could be generated from things like review feedback, vendor responses and other aspects of life like the way you drive. Yes, there will be ways to "game" the system, as well as ways to avoid it, just like there are ways to game and avoid your credit score. Want to be a F*ckwad? Be anonymous, but understand that if you're a raging F*ckwad, folks might take the extra effort to tie it back to you. Kind of like how a criminal record will effect your credit rating.

Being a privacy wonk, i'm on the fence about this. Sure, i'd prefer if this sort of thing didn't come to pass, but i'd be a moron if i didn't think we weren't on the road to it happening. On the other hand, if this information was accessible, standardized and there was an effective challenge channel in place, it would make these sort of personal tracking that's already being done more transparent to you.

Heh, i have seen the Big Brother, and He is Us.

Hey!

Oh, Hi! How are you doing?

Crap, i can't remember this guy's name. i haven't seen him in over a month and i swear my brain keeps info for less time that an Air Traffic controller at O'Hare. Think, think, who the hell is this again?

Good seeing you. i was down in the area again and was just walking around.

Dammit, ok, so he's a remotie. Perfect. That means that i may have only actually met him twice in my life. Well, go for the general small talk route, i guess.

Wait, he's talking to the guy that wrote Javascript. Ok, this is someone more important. Wait, they're talking about FuncLib? Oh Holy Hell, He is important and there are things i need to talk to him about. But he's smart, like REALLY smart. Like the kind of person who can probably focus their mind enough to fry ants with theoretical physics or something. Think, you stupid simian, what's this guy's name!?!

Wait, so the FuncLib stuff is all solid and working? That's awesome. You've been working on that for years!

Yep, really happy about it. Got a few smaller issues to deal with, but otherwise it's great. Want to see a demo?

Of course i do! Wow, this is fantastic, have you heard about some of the stuff that the XYZ team is working on that could really use this?

Wow, this IS awesome. The XYZ team would cream themselves to have this. So would Ryan. Ok, so now we know what he's been working on. Neato! There's only a few names tied to that project.

… i'm an idiot.

i'd love to help out, but there's another guy named Ryan who would be better suited to pitch in.

Fantastic! i'd love the help. Send me an email with his contact info.

Shit

Great talking with you as always. i've got to get to a meeting, but we'll catch up, right!

You bet. i'll send out that email as soon as i can…

…get into the company phone book and remember your name. Please don't fry me with your massive focused intellect. i can feel my schoolbus losing seats. Bob? John? Mike? Think you sponge, think.

Oh yeah, and that other guy you work with that has a bunch of monogrammed shirts? Thank him the next time you see him.

This is interesting because many of those said gizmos have wifi and cameras built in. In addition, you can get real wireless cameras for a lot less than you could a few years ago, so there's that.

Here's how i set up my configuration (or at least the bits that are interesting):

What you'll need:
N Wireless
a Linux server with a fair bit of space on it (Pics and videos take space)
IP cameras.

1) The pre-install.
Technically, N wireless does 300Mbps per channel, with bursting up to 450Mpbs. A lot of devices top at 100Mbps. i'd recommend getting a dual band router and keep the devices on the broader 2.4Ghz channel.

The linux box is also kinda key. It doesn't have to be spectacular (i run a dual core franken-box i built about 7 years ago), and has Ubuntu server loaded on it. It runs headless, so there's no need for anything fancy.

2) The real webcameras
i picked up some FOSCAMs that were on sale on Amazon. These are nice little cameras that have built-in wifi, as well as remote movement controls. They also have the ability to do motion detection and have a pretty powerful set of IR lights. They're not without a few problems, though.

For one thing, getting the camera on the network. The easiest way to do that is to plug it into the router, find the IP in the attached device list and then http to that. Once you're there you can set the network, IP address, port, and all the other settings you'll need. Fortunately, once they're configured, they're pretty autonomous.

Another issue is that they reboot every so often. When they do, they turn on the IR lights. The problem with that is IR reflects amazingly well off of glass which blinds the camera. Sadly, there's no way have the camera remember to not turn those on, but you can use a remote command to force the IR off.

/usr/bin/curl -n --silent http://$CAMERA_ADDR/decoder_control.cgi?command=94 > /dev/null


i have a crontab entry that sends that command every so often to make sure the damn lights stay off.

3) The more disposable webcams
Like i said, depending on where you are and how tech savvy your community happens to be, you can get things like first generation ipod touches for as low as $1. Old Android phones can also be had for darn cheap, so it can be fairly easy to load up just by visiting craigslist. Once you have your handfuls of junk cameras, you can then load them with ipCam for iOS or IPWebcam for Android. Both of these apps let you route the camera to the local LAN, and that's a delightful thing, as we'll discover in a second.

4) Setting things in Motion
Now comes the fun bit…

On the linux box, grab a copy of motion. Motion will pull the cameras for images and do some quick analytics to detect motion. It will also capture snapshots and compile a stream of activity for the day.

Be sure to point motion at a directory with "ample space". In my case, i record about 2GB of data for 4 days. i have a script that goes through the directories and purges files older than that, but that's me. Granted, in an era of sub $100 2 TB drives, this is probably being overly aggressive. Here's the cleanup code:
#!/bin/sh
/usr/bin/find /d2/media/ipcam -type f -ctime +2 -delete
for i in `ls -1` ; do
/usr/bin/find /d2/media/ipcam/$i/jpg -type d -empty -delete
/usr/bin/find /d2/media/ipcam/$i/swf -type d -empty -delete
done;


And for those that are interested, my motion.conf file is below. i've trimmed out the comments and blank lines, but this shows the values.
process_id_file /var/run/motion/motion.pid
setup_mode off
videodevice /dev/video0
v4l2_palette 8
input 8
norm 0
frequency 0
rotate 0
width 320
height 240
framerate 2
minimum_frame_time .5
netcam_tolerant_check off
auto_brightness off
brightness 0
contrast 0
saturation 0
hue 0
roundrobin_frames 1
roundrobin_skip 1
switchfilter off
threshold 1500
threshold_tune off
noise_level 32
noise_tune on
despeckle EedDl
smart_mask_speed 0
lightswitch 0
minimum_motion_frames 1
pre_capture 0
post_capture 0
gap 60
max_mpeg_time 0
output_all off
output_normal on
output_motion off
quality 75
ppm off
ffmpeg_cap_new on
ffmpeg_cap_motion off
ffmpeg_timelapse 0
ffmpeg_timelapse_mode daily
ffmpeg_bps 600000
ffmpeg_variable_bitrate 0
ffmpeg_video_codec swf
ffmpeg_deinterlace off
snapshot_interval 0
text_right %Y-%m-%d\n%T-%q
text_changes off
text_event %Y%m%d%H%M%S
text_double off
target_dir /tmp/motion
webcam_quality 70
webcam_motion off
webcam_maxrate 1
webcam_localhost off
webcam_limit 0
control_port 8080
control_localhost off
control_html_output on
track_type 0
track_auto off
track_motorx 0
track_motory 0
track_maxx 0
track_maxy 0
track_iomojo_id 0
track_step_angle_x 10
track_step_angle_y 10
track_move_wait 10
track_speed 255
track_stepsize 40
quiet on
thread /etc/motion/lr.conf
thread ...

And the main camera config (remember, this replaces the values from motion.conf)
lightswitch 0
pre_capture 3
width 640
height 480
target_dir /d2/media/ipcam/lr
snapshot_filename jpg/%Y%m%d/s-%H%M%S-%q
jpeg_filename jpg/%Y%m%d/%H%M%S-%q
movie_filename swf/%Y%m%d/%H%M%S
timelapse_filename tl/%Y%M%d
ffmpeg_timelapse 5
text_left LivingRoom
webcam_port 8081
netcam_url http://192.168.2.26/videostream.cgi?rate=11
netcam_userpass ##:##
netcam_http 1.1
auto_brightness on
text_left LivingRoom
webcam_port 8081
on_picture_save /home/jrconlin/bin/newPic.sh %f

The "on_picture_save" bit stores a copy of the latest picture to my remote server. i have something else pull movies, but i could have also had motion push them just as easily. It's also possible to have motion monitor just parts of an image and fire a script when it sees activity at that position. i've thought about rigging that up for Halloween one year to scare the pants off of kids coming up the driveway.

That's it! Happy monitoring.