Recently, i needed a way to sync data for my Firefox add-on between clients. Fortunately, there's a nifty way to do that built into the current versions of Firefox called Sync. Unfortunately, while the documentation is great if you want to learn how sync operates, it's kinda poop if you want to quickly get started using Sync.

Well, "Some see problems, others see opportunities" so as my first go to fix that problem, i'm going to write down how i got Sync working.

The Guts

Ok, before we start delving into code, let's take a moment to talk about what's going on. Sync is a tool that securely exchanges data between two clients. That means you can use it to exchange data between anything you own that can run Firefox. (There are a couple of other browsers out there that can also play along, but for simplicity, let's stick to just firefox for now). It does this by stuffing your data into a well known chunk (marshalling), encrypting it, and sending it via a well known server.

Note, that the server isn't doing much more than relaying your data. That's because it's encrypted and can't really do anything with it right now. (Mozilla is working on making a more "durable" storage tool, but it's not there yet. This is why Mozilla STRONGLY RECOMMENDS you don't think of Sync as a backup service. It is, kind of like the shelf at the ATM is somewhere you can put your wallet. It's useful while the main bit is used, but not a really good long term idea.

Ok, so we've got records that are being exchanged. Those records have a little bit of information associated with them that's useful for syncing, but otherwise they're very simple. Those records go into a "Storage" object that just collects and manages the records, are watched by a "Tracker", and the storage elements get exchanged by an "Engine" that does the actual exchange. The engine watches both for local changes and remote requests for updates. It's worth noting that the exchange isn't instant, but near enough for most usages (e.g. within a few seconds).

Got it? The Tracker notices a change, and asks the Storage element to update the associated Records, then calls the Engine to deliver them.

There, now you understand Sync.

Spelunking

Now that you get Sync (from a tool point of view, at least), i can point out where some of the code lives. You don't need to look at this, but it might be useful if you like to play along at home.
First off, here's where the sync code lives. Most of what you want is in the sync/modules branch.

If you want to see the code i built, you can find the latest version here.

The ground up…

The addon i'm modifying is built off of the Addons SDK. The nice thing is that this means the addon is restartless, the bad thing is that it means i have to do some extra work in order to get things running. In this case, i need to call into Mozilla Core code. To do that, though, you just need to call:

const {Cu} = require('chrome'); // get the Components.utils hook
Cu.import('resource://services-sync/engines.js');
Cu.import('resource://services-sync/record.js');
Cu.import('resource://services-sync/main.js');
Cu.import('resource://services-sync/util.js');

This automagically drags a host of objects into the current namespace. Chances are, if you're wondering where an object is defined, it's within one of these files.

Now, let's work from the ground up, that means building a Record object. Like i said before, a Record is just a marshalling container. The only real restriction is that it should be JSON storable (so fancy JS pointer hacks or methods are not really a good idea).

So, something simple:
function FooRecord(moduleName, recordId) {
CryptoWrapper.call(this, moduleName, recordId);
}
FooRecord.protoType = {
__proto__: CryptoWrapper.prototype; // subclass from CryptoWrapper
_logName: "Record.FNCrypto; // What to use for the log messages
};
Utils.deferGetSet(FooRecord, "cleartext", ["value"]);

As you can see, this subclasses CryptoWrapper (via janky JS subclassing), and then calls a utility function to autobuild the Getter/Setter method, which will store FooRecord.value into "this.cleartext.value". Ok, you probably didn't see that. You'll have to take my word for it. Now, if you had a very complex item where you may not want to exchange every bit of info, you could define a bunch of items in that (where "value" is, and probably with a bit more descriptive labels). This way, you'd send just the bits you need so that things are pleasantly zippy. Since my records are small and not really something you'd want to split up anyway, i opted for a single value.

Now that we have a Record, we need something that can hold them. Let's define the Store. In many respects, this is another Controller layer, in that it calls to whatever you're using to actually store your data (the Model). In this case, think of DB as the model store. (DB is the persistent storage that is available to Add-ons, and is pretty cool too.)


function FooStore(moduleName) {
Store.call(this, moduleName);
}
FNSyncStore.prototype = {
__proto__: Store.prototype,
self: this,
itemExists: function(recordId) {
return DB[recordId] != undefined;
},
createRecord: function(recordId, moduleName) {
var record = new FNSyncRecord(moduleName, recordId);
if(DB[recordId]) {
/* Again, if we had multiple fields, make sure you set them here.
*/
record.keyBundle = DB[recordId];
return record;
}
return undefined;
},
changeItemID: function(oldId, newId) {
DB[newId] = DB[oldId];
delete DB[oldId];
},
getAllIDs: function() {
/* It's important that this return an Object (a Dict/Hash)*/
var recordIds = {};
for (var key in DB) {
/* Only return keys that are actually pointing to values.
* DB is a JS object, so there can be all kinds of cruft in there.
*/
if (key.indexOf('key:') === 0) {
/* The value stored is arbitrary. Only the key name is important.
*/
recordIds[key]=true;
}
}
return recordIds;
},
wipe: function() {
for (var i in self.getAllIDs()){
delete DB[i];
}
},
/* These are meta function calls to normalize data sets
* between this machine and some other.
*/
create: function(record) {
DB[record.id] = record.payload;
},
update: function(record) {
DB[record.id] = record.payload;
},
remove: function(record) {
delete DB[record.id];
}
}

Again, very simple, but a few caveats in there. One thing that can be confusing is that "createRecord" is different than "create". Create Record is more of a "write" function, in that it's called when a record has changed and it needs to be written out to some other device. "Create" is the opposite, that's where a new record needs to be imported onto the local machine. That's pretty much the role of Store.

Ok, we've got the transfer record, we've got the Storage/controller stuff, now we need to look at the Tracker to spot local changes. That's the role of Tracker.
function FNSyncTracker(moduleName){
Tracker.call(this, moduleName);
trackerInstance = this;
}
FNSyncTracker.prototype = {
__proto__: Tracker.prototype,
track: function(recordId){
/* Add the record to the list of items that have changed.
*/
this.addChangedID(recordId);
/* Any dirty records need to be propagated as soon as possible.
* thus the immediate "100"
*/
this.score = 100;
}
}

Really. That's about it. Basically, once data has changed on your side, you indicate to the Tracker "Hey, this changed!". You'll also want to set the score for how important it is to sync this data (0 == ignore this, 100 == OMFG THIS NEEDS TO BE DONE LIKE AN HOUR AGO!). You can set observers to help do this, or just call directly. i took the second option because i know exactly when i need to update and it's kind of important to do it after new record creation.

And finally, we're at the top level, the Engine:

function FooEngine() {
// Defining "moduleName" here so that it's easy to figure out where
// it comes from. This is case insensitive.
var moduleName = "Foo";
Weave.SyncEngine.call(this, moduleName);
// turn the engine on
this.enabled = true;
}
FNSyncEngine.prototype = {
__proto__: Weave.SyncEngine.prototype,
_storeObj: FooStore,
_recordObj: FooRecord,
_trackerObj: FooTracker,
version: 1
};

Hey, so that's where the Module name comes from! Module name is used for a number of things, including logging and tracking. Unfortunately, it doesn't show up on the list of active sync engines on the options panel, yet. It does mean that it has to be a fairly unique, yet human readable string, so "Foo" is probably sub-optimal. The other thing you'll need to do is make sure that the engine is enabled (this caught me hard).

You'll also need to define links to the objects you've built, and the version you're using.

And, you're done!

That's it. Granted, you'll want to test this, and debug it, and all the other nonsense, but from a code sense, that's all you'll really need.

By the way, since i mentioned debugging, i want to chime in on the built in tools for Aurora and Nightly. Sadly, there's still no step-debugger, but here's a stupid trick that is INCREDIBLY useful:

1. go to about:config
2. Open a Web Console [Ctrl+Shift+K]

You now have a web console that can run Javascript with the Browser Chrome. That means that you can run things like Components.util.import('resource://services-sync/utils.js'); Utils.sha1('foo'); and get a pretty SHA1 hash of foo on your screen. And yeah, you can do the same trick with any other Mozilla core function you want to play with.

So, now you know, and knowing is half the battle.
The other half involves blue and red lasers and guys with stupid nicknames.

(i'll beg your forgiveness if the field names are not correct. The therapy helps blot that crap from my mind.)

What's important here is that x500 allows fairly arbitrary fields to be inserted into a given record. The MTA only has to pay attention to the ones that it recognizes, so you could build one that only responds to, say, "Company, FamilyName" if you're a place with a dozen, unrelated employees. The rest of the spec is all optional. This lead to all sorts of interesting abuses, including one of the things i was working on that involved storing a cryptographic key inside of the record that tied back to the authorizing provider, allowing for a verification path to the central root authority, meaning that your headers were often far larger than the actual content of the message and that had to be magically stuffed through a 300baud modem in a timely manner.

Still, flexibility was a strong selling point for x500. So much so, that the example included the frivolous category of "FavoriteDrink", and provided a helpful collection of alcoholic concoctions for the various fictional individuals to imbibe. HaHa, what merriment shall be had!

Only, it was in a spec, so that it became codified.

Now, a few decades later, i fully expect that there are more than a few unit test cases for determining validity of records based on "drink", and some very confused intern wondering what the hell that field has to do with figuring out how to partition their machines. i have been that intern. It was not fun.

This is why i tend to be a little cautious when folks talk about "loose data definitions" in applications, unless they REALLY MEAN IT. As in, they define methods for how data can be stored, but do not require specific elements to be defined. The application is therefore required to determine if a record can be used or not. That tends to shoot a lot of holes into peoples designs, and i'm ok with that. You can't realistically expect anyone else to use something where you hand wave over something as important as "The Data You Use".

Well, unless you're also providing all the alcohol that they'll need to deal with that stuff.

Bad jokes aside, when i heard the report, the first thing i thought of was "So, what happens when Twitter goes away?"

i get the concept at hand here, and what they are talking about. "Tweeting" means posting short messages. It's what folks understand. Like xeroxing a kleenex you hoovered up, still, there are a growing number of services that provide that sort of messaging with little intent of being used by a wider audience. Why? Well, aside from the OAuth handshake, there's something innately appealing about the idea of being able to post info to a simple URL for yourself.

That's definitely the idea behind the "Internet of Things" you occasionally hear about. A web of devices talking to each other (and you). People are thinking about using sites like Twitter and Facebook as the channels of communication, but i'm not really comfy with the idea of those companies knowing my energy consumption patterns and glucose levels. Or, for that matter, my credit and insurance company getting a hold of them without my approval.

So that's kind of the idea behind the stuff Jeff and i are working on, Notifications. In short, it's a simple way for sites (or devices) to post short messages to a URL and have them get to you. What's more, you have control over those messages, and can silence or drop a site easily. Devices also have it pretty easy and can either post stuff just to a URL, or pass them through an encryption filter that prevents the carrier from being able to read the contents.

Internally, i've been kinda selling Notifications as being "Twitter for Mozilla", but i'm not sure that's right. Really, it's "Send and forget" for stuff that needs to talk to other stuff. Even better, is that you can run your own server. Heck, you can even run your own client that blasts these messages to Facebook or Twitter as well. It's not tied to Firefox at all, and that's the beauty of it.

It's simple, secure, and under your complete control. i think that's kind of cool, and hopefully you will as well.

i greeted the gent and he asked if we had family from a certain city. i replied that i wasn't going to answer that question. He pressed and i replied "Sorry, but i don't give information like that over the phone."

"Why not?" he asked, somewhat concerned. "Who'd want to know?"

"Well, i could give you a few hundred reasons why i wouldn't give someone info about my family." i replied helpfully.

"Look, i'm a lawyer from Boston…"

"… and i'm a rocket scientist from Florida. i have no way to prove what you're saying as you have no way to prove what i'm saying." (Frankly, if this guy was a lawyer, he's a pretty bad one.)

"i'm just trying to find my sister…" and then he gave me a rather long winded story about himself, his sister and looking for family. i listened politely, informed him that i could offer no help, and then offered a few additional things that might help him out.

He yelled at me to calm down. i know i can speak quickly, but i was absolutely calm. Heck, i was in complete control of the situation and happy to assist in any manner that would not compromise myself or my family. There's lots of Conlins in Michigan, if he was estranged from his folks, chances are someone up there might have more history he could chase down.

Eventually he hung up, and i did a bit of research on my own. i couldn't find any lawyer in current practice or retired in Boston that matched his name. The cellphone number he provided is less than a year old (some idiot posted it on Facebook about a year ago) and there's scant information that matched any other part of his story. (e.g. high school or college records that match his or his "sister's" name for the approximate years they might have attended. Hell, he even managed to miss on the name of the high school he was supposed to have attended.) For what it's worth, i'm chalking it up as a scammer.

See, that's the thing about personal information. If my wife wasn't married to a paranoid info junkie like me, she'd probably happily give this guy all sorts of information about my family, that he could then use to commit all sorts of dastardly things.

People's first action is to try and help. It's someone looking for family, meaning they might be a long lost great cousin. As the monkey said, "Why would anyone care who wasn't family?"

All that aside, even if he was legit, trying to find his sister by cold calling people in California might explain why the family would have nothing more to do with him.

Setting that aside, however, Twitter's recent announcement of the Innovators Patent Agreement kinda leaves me feeling odd. In many respects, they're "Doing It Right®". They're establishing a patent policy where folks creating the patents own the ideas, the patents can only be used defensively, the policy remains in effect even if the patents are later sold or transferred, and they're even hashing all this out on github.com. Considering the crapola state of software patents and how they're stifling innovation, it's a good step toward solving this sort of crap.

But, then there's part of my brain that speaks up. You see, you can't patent things that are in the public domain. In fact, you can modify, enhance or transform a concept, but the patent only applies to that modification (the same is true if you take an existing patent). Even then, your idea has to be a substantial improvement over the previous idea, and not just painting it blue or something.

So, why patent it at all?

Patents exist for one reason. To prevent you from doing something. If i can patent something, i can do whatever for the period of the patent, but if you want to do it, i have to let you. Most times, i'll let you if you pay me enough, but that's not always the case. Sometimes i won't grant you the right regardless of offers of compensation just out of spite. Right now, companies are being bought and sold just for their portfolio, which is immediately used against companies that are doing the same thing. It's an arms race.

If twitter, or an engineer were to release an idea into the public domain, it becomes prior art. That's why so many mobile devices have multi-touch (since the idea came out in 1985). Apple owns some additional aspects of how that interface is used within their UI, but that's about it.

i'm honestly curious about this. Part of this strikes me as painting flowers on warheads. These are still patents. They still exist to prevent you from doing things. If i wanted to create a library that uses concepts contained in these patents, that library is still subject to the holder enforcing those patents and either requesting my library to be removed or filing damages. That's what patents do. Yeah, i could make a very nice looking oven out of that crate of bullets, but i wouldn't want to make a pizza in it. i'd still want to use ideas and methods that are in the public domain so i could avoid those issues. It's why Ogg, bzip and png exist.

i'm just not sure what to make of this. It's a bit like declaring a Mutual Assured Destruction policy after Mad Max has been driving around for a few decades.

If you want this stuff to be freely used, why not make it free?

Yeah, that surprised me too.

You see, when i think of a "picture platform" i think of sites like Flickr. Go to flickr.com and feel free to waste an hour just looking at pictures. There are billions. Heck, there's even a page that shows photos that the community feels are interesting. What's more, you don't even have to have an account. That's really important. It means that anyone can view your photos because you're sharing them with the world.

That's kinda what photo sharing means to me.

Now, feel free to go to sites like instagram.com and do the same thing. Oh, wait, you can't. You have to download their app, create an account and find people who take good photos before you can see them. Well, at least there's facebook, right? Let me do a search for photos on facebook and… oh. Pretty much the same thing. i mean, i can do a search on google for facebook images, i suppose, but that's not really useful. Even if i were publish a portfolio of photos that were free to use, you wouldn't be able to find them unless you knew i shared them and had a Facebook account.

See, this is where i say that both of these services are blackholes. They suck information in, but you're not really expected to get anything back out. Indeed, one could think of these services a bit like religions (albeit, not the better portions thereof):

• You can't do anything unless you're a member.
• Members are always trying to convert the "unfaithful".
• There are aspects that are accepted without proof.
• One community may use them in a way that is quite different than another.
• Changes are met with cries of "Heresy"!

These are walled gardens and you're not allowed in unless you genuflect. For all i know, both of these companies are making up numbers and statistics because there's no way for you to know otherwise.

You know what? Sure thing. i've also got an amazing platform with zillions of photos loaded up! Why here's frequent visitor Max Foucault (who is absolutely not a name i made from hitting random twice on wikiquotes) talking about how wonderful the service is:

[It's] fantastic! Couldn't live without it. In fact i wouldn't exist if it didn't!

And look here's a photo from him that is in no way two images i grabbed off of pinterest and applied random filters to:

Don't believe me? That's because you're not part of the exclusive set of pre-alpha invitees and my crack team of imaginary developers, and there's no way for you to prove otherwise.

What's more, i'll even take .01% of what was offered for Instagram, because i believe in taking complete advantage of suckers that the service is such a benefit to humanity.

Isn't that right, noted Chinese Blogger Xū Rén:

Yes! Your service is the only one i use in China with all my Chinese friends who are Chinese!

Odd that he didn't write that out.

There you go, Mr. Conlin, you're all set with your new Smartphone.

Thanks, and…[CLANK!] there you go, you're all set now too.

Uh, Mr. Conlin, what is that?

That? Oh, well, i want to make sure that your company is around a long time. That means your money should be safe, so that's my free security system for your cash register.

[SCREEEEEEEEEEECH!]

That… is a monkey.

Well, yes. Officially it's a Atelinae Ateles, or a "spider monkey". They're a new world monkey with very long limbs and a prehensile tail that…

… yes, but why is he attached to our cash register?

Because he's your new, free security system, duh! No worries about that monkey getting stolen, that leash is made out of woven titanium. Granted, he's a little "bitey" so i had to go for that to keep him from chewing through it too…

[SCREEEEEEECH!]

Mr. Conlin, i appreciate your offer but,…

Offer? No, quite the contrary! This bitey, angry monkey is my gift to you! It's free! And make sure you feed him and keep him up on his shots otherwise the ASPCA will probably not like you. Come to think of it, they may not like you having a monkey chained to your cash register either.

Mr. Conlin! We can't have that thing here! We can't get to the registers!

Well, then they're secure then aren't they? Besides, you'll love it! Kids love the cute little hat he's wearing. Was wearing. Is filling up with his poop.

Get that damn monkey out of my store right now!

Sure thing, after you tell me how i can get these apps off my phone.

Apps?

Yeah, like this Verizon Maps, NFL Football and Netflix. i appreciate the offer, but i don't use any of them and they keep demanding that i update them or use them. i can't remove them or disable them, so they're a bit like Cuddles, there.

[SCREEEEEEEEEEEEEECH!]

Sorry Cuddles, Daddy doesn't love you anymore. Maybe Uncle… Steve right? Steve, will love you. Make sure you wear protective gear if you hug him. Cuddles is strong and goes right for the jugular.

You can't bolt an angry monkey to our cash registers!

You're just jealous that you didn't think of it first. Now, about removing these apps…

Well, what about this MotoActive app? We didn't force that one on you.

True. Motorola also had concerns about their new Howler Monkey music gift i dropped off on their campus, but i'm sure they'll grow to love it.

i'm doing my best not to start swearing like a sailor right now.

This is like trying to teach kids how to repair cars by getting them snap-tite models.

Yes, the iPad is nifty and intuitive and makes the web more portable than it's been in a while. There are professionals that have started using them as a replacement for laptops for taking notes, doing presentations and writing documents. While very expensive, some colleges have moved their texts to eBooks meaning that students don't have to worry about forgetting their tomes.

But as an actual tool for learning technology, it's a piece of crap. There's no native editor that lets you build web pages. There's no way to examine and alter source of a live page. There's no local compiler so you can build and play with programming. You lose half your screen real-estate to the friggin "keyboard".

Tossing iPads at kids and hoping that they learn something more than ballistic trajectory patterns of cartoon fowl just points out how clueless you are about how to use technology.

You want to give your kids an edge in technology? Let them break stuff. Give them tools that let them see computers aren't magical sealed boxes they're not to play with, but bundles of commands and handles that will do their bidding provided you ask them correctly. Point them at Hackasaurus.org and let them dismantle and web page and learn javascript. Teach them python and Java. Give them Virtual Machines they can play with. Show them data stores like Cassandra and MySQL and let them see how they're both great and crap depending on how you use them.

Want your kids to be tech smart? Get them hacking. Get them excited about technology. Let them teach other kids so that the lessons really sink in.

Anything else, and you're basically buying your kids slot car racers and hoping that they'll win the Indy 500.

Like far too many of these sorts of things, it was both sudden and expected, and while i appreciate your kind thoughts and words, can i ask you to do something instead?

My family doesn't have a great deal of recorded history. This is a bit of an annoyance being a fan of history (heh, something i inherited from my father). While we have great stories as a family, a good deal of it is lost to the ages. Since he got sick, i'd been pushing my Dad to write down some of the stories he remembered growing up. Granted, he was a humble man, and would argue that nobody would ever care about that stuff.

And we wonder why we didn't have a lot of family history.

Please don't make that mistake. Start a journal. It doesn't matter if you're not Hemmingway or all you ever eat for lunch is peanut butter and jelly. If you never had an exceptional moment in your life, that's exceptional and should be recorded somewhere. Write it down. People will want to read it. If not your family, then the fine folks at the Library of Congress and National Archives will happily take a copy and some future scholar may use it in his thesis. Or possibly they actually will make that movie about you.

Once your gone, so are your memories, unless you make the effort to tell them to someone.

Take the effort.

i am not a math whiz. i get math, but not to the level that it's effortless music. So i'd appreciate ye far smarter folk double checking my thinking here.

The Problem
For the Notifications stuff i'm working on, i need a token that is Really Hard to Guess. That means Lots of Entropy stuffed into a token.

Because this is going through SMTP, i have a hard limit of 64 characters. While also not part of the RFC (or anything else, experience with various MTAs have shown that you're limited to a through z and 0 through 9. Using a handy Entropy Bit Calculator, that shows that:

log₂(36)*64 ~= 331 Bits of Entropy.

That means you have a 1 in 2³³¹ (or 4,374,501,449,566,023,848,745,004,454,235,242,730,706,338,861,700,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000) chance of guessing it. (In case you're curious, that's 4 duotrigintillion)

That's great, and all, but does lead one to wind up getting a token that is 64 characters of random crap, like:
wgmrbj3d9pcv1op6opprw7hvjgzimzonakq3m02egtjh8zs2bzyuxnowfx5e9fxu

So i was asked to research how good/bad an idea it might be to make the token a little less random.

The Solution

Specifically, i was asked if i could create a token based on words. That means you'd get a token like:
faucet.variorum.hambling.baiters.harmed.stoa.haysel.glandular.61
i arbitrarily decided to separate the words with periods and back-fill with numbers.

That leads to some interesting math.

i pulled the English Open Word List and collected all the words into a single file (dropping the words that have non-ascii characters). That produced a catalog of 128,766 candidate words. A crappy token generator script later, i found that the mean token consisted of 8 words and was about 61 characters long. By my guess, each token would consist of
[a-z]{2} + // shortest word length was 2 characters
[a-z\.]{59} + // the bulk of the token would be character or a period
[a-z0-9\.]{3} // the remainder would be character, number or period

This would lead me to think that the base entropy would be something like:
(log₂(26)*2) + (log₂(27)*59) + (log₂(37)*3)
or about 306 bits of entropy. (1 in 130 novemvigintrillion odds)

The real problem

But i'm not sure that's true.

That's supposing that the core of the characters being used are taken randomly from that pool of 27 available characters. They're not. These are using words in the English language which means that there are certain character distributions which diminish the pool of potential entropy. Granted, that's masked somewhat by the varying word lengths, but that still means that there aren't a hell of a lot of words that have J, Q, X or Z in them, and that the default letter set that Wheel of Fortune picks for you sucks.

If one takes only the letters that have a greater than 10% chance in 1000 of appearing, then you wind up with about 15 meaning you've got:

log₂(26)*2 + log₂(27)*59 + log₂(37)*3 = 30 bits of entropy, or 1 in 1,073,741,824

That number is no where near good enough. It's even pronounceable.

So, really, the question comes down to: "How much entropy am i losing by using real words?"

Thanks, Lazyweb!

i'll just keep refreshing the page until you answer.