i've been informed that Sync is due for a change and that what you read here may not be outdated in a few months. (grumble.)
Be forewarned.
Recently, i needed a way to sync data for my Firefox add-on between clients. Fortunately, there's a nifty way to do that built into the current versions of Firefox called Sync. Unfortunately, while the documentation is great if you want to learn how sync operates, it's kinda poop if you want to quickly get started using Sync.
Well, "Some see problems, others see opportunities" so as my first go to fix that problem, i'm going to write down how i got Sync working.
The Guts
Ok, before we start delving into code, let's take a moment to talk about what's going on. Sync is a tool that securely exchanges data between two clients. That means you can use it to exchange data between anything you own that can run Firefox. (There are a couple of other browsers out there that can also play along, but for simplicity, let's stick to just firefox for now). It does this by stuffing your data into a well known chunk (marshalling), encrypting it, and sending it via a well known server.
Note, that the server isn't doing much more than relaying your data. That's because it's encrypted and can't really do anything with it right now. (Mozilla is working on making a more "durable" storage tool, but it's not there yet. This is why Mozilla STRONGLY RECOMMENDS you don't think of Sync as a backup service. It is, kind of like the shelf at the ATM is somewhere you can put your wallet. It's useful while the main bit is used, but not a really good long term idea.
Ok, so we've got records that are being exchanged. Those records have a little bit of information associated with them that's useful for syncing, but otherwise they're very simple. Those records go into a "Storage" object that just collects and manages the records, are watched by a "Tracker", and the storage elements get exchanged by an "Engine" that does the actual exchange. The engine watches both for local changes and remote requests for updates. It's worth noting that the exchange isn't instant, but near enough for most usages (e.g. within a few seconds).
Got it? The Tracker notices a change, and asks the Storage element to update the associated Records, then calls the Engine to deliver them.
There, now you understand Sync.
Spelunking
Now that you get Sync (from a tool point of view, at least), i can point out where some of the code lives. You don't need to look at this, but it might be useful if you like to play along at home.
First off, here's where the sync code lives. Most of what you want is in the sync/modules branch.
If you want to see the code i built, you can find the latest version here.
The ground up…
The addon i'm modifying is built off of the Addons SDK. The nice thing is that this means the addon is restartless, the bad thing is that it means i have to do some extra work in order to get things running. In this case, i need to call into Mozilla Core code. To do that, though, you just need to call:
const {Cu} = require('chrome'); // get the Components.utils hook
Cu.import('resource://services-sync/engines.js');
Cu.import('resource://services-sync/record.js');
Cu.import('resource://services-sync/main.js');
Cu.import('resource://services-sync/util.js');
This automagically drags a host of objects into the current namespace. Chances are, if you're wondering where an object is defined, it's within one of these files.
Now, let's work from the ground up, that means building a Record object. Like i said before, a Record is just a marshalling container. The only real restriction is that it should be JSON storable (so fancy JS pointer hacks or methods are not really a good idea).
So, something simple:
function FooRecord(moduleName, recordId) {
CryptoWrapper.call(this, moduleName, recordId);
}
FooRecord.protoType = {
__proto__: CryptoWrapper.prototype; // subclass from CryptoWrapper
_logName: "Record.FNCrypto; // What to use for the log messages
};
Utils.deferGetSet(FooRecord, "cleartext", ["value"]);
As you can see, this subclasses CryptoWrapper (via janky JS subclassing), and then calls a utility function to autobuild the Getter/Setter method, which will store FooRecord.value into "this.cleartext.value". Ok, you probably didn't see that. You'll have to take my word for it. Now, if you had a very complex item where you may not want to exchange every bit of info, you could define a bunch of items in that (where "value" is, and probably with a bit more descriptive labels). This way, you'd send just the bits you need so that things are pleasantly zippy. Since my records are small and not really something you'd want to split up anyway, i opted for a single value.
Now that we have a Record, we need something that can hold them. Let's define the Store. In many respects, this is another Controller layer, in that it calls to whatever you're using to actually store your data (the Model). In this case, think of DB as the model store. (DB is the persistent storage that is available to Add-ons, and is pretty cool too.)
function FooStore(moduleName) {
Store.call(this, moduleName);
}
FNSyncStore.prototype = {
__proto__: Store.prototype,
self: this,
itemExists: function(recordId) {
return DB[recordId] != undefined;
},
createRecord: function(recordId, moduleName) {
var record = new FNSyncRecord(moduleName, recordId);
if(DB[recordId]) {
/* Again, if we had multiple fields, make sure you set them here.
*/
record.keyBundle = DB[recordId];
return record;
}
return undefined;
},
changeItemID: function(oldId, newId) {
DB[newId] = DB[oldId];
delete DB[oldId];
},
getAllIDs: function() {
/* It's important that this return an Object (a Dict/Hash)*/
var recordIds = {};
for (var key in DB) {
/* Only return keys that are actually pointing to values.
* DB is a JS object, so there can be all kinds of cruft in there.
*/
if (key.indexOf('key:') === 0) {
/* The value stored is arbitrary. Only the key name is important.
*/
recordIds[key]=true;
}
}
return recordIds;
},
wipe: function() {
for (var i in self.getAllIDs()){
delete DB[i];
}
},
/* These are meta function calls to normalize data sets
* between this machine and some other.
*/
create: function(record) {
DB[record.id] = record.payload;
},
update: function(record) {
DB[record.id] = record.payload;
},
remove: function(record) {
delete DB[record.id];
}
}
Again, very simple, but a few caveats in there. One thing that can be confusing is that "createRecord" is different than "create". Create Record is more of a "write" function, in that it's called when a record has changed and it needs to be written out to some other device. "Create" is the opposite, that's where a new record needs to be imported onto the local machine. That's pretty much the role of Store.
Ok, we've got the transfer record, we've got the Storage/controller stuff, now we need to look at the Tracker to spot local changes. That's the role of Tracker.
function FNSyncTracker(moduleName){
Tracker.call(this, moduleName);
trackerInstance = this;
}
FNSyncTracker.prototype = {
__proto__: Tracker.prototype,
track: function(recordId){
/* Add the record to the list of items that have changed.
*/
this.addChangedID(recordId);
/* Any dirty records need to be propagated as soon as possible.
* thus the immediate "100"
*/
this.score = 100;
}
}
Really. That's about it. Basically, once data has changed on your side, you indicate to the Tracker "Hey, this changed!". You'll also want to set the score for how important it is to sync this data (0 == ignore this, 100 == OMFG THIS NEEDS TO BE DONE LIKE AN HOUR AGO!). You can set observers to help do this, or just call directly. i took the second option because i know exactly when i need to update and it's kind of important to do it after new record creation.
And finally, we're at the top level, the Engine:
function FooEngine() {
// Defining "moduleName" here so that it's easy to figure out where
// it comes from. This is case insensitive.
var moduleName = "Foo";
Weave.SyncEngine.call(this, moduleName);
// turn the engine on
this.enabled = true;
}
FNSyncEngine.prototype = {
__proto__: Weave.SyncEngine.prototype,
_storeObj: FooStore,
_recordObj: FooRecord,
_trackerObj: FooTracker,
version: 1
};
Hey, so that's where the Module name comes from! Module name is used for a number of things, including logging and tracking. Unfortunately, it doesn't show up on the list of active sync engines on the options panel, yet. It does mean that it has to be a fairly unique, yet human readable string, so "Foo" is probably sub-optimal. The other thing you'll need to do is make sure that the engine is enabled (this caught me hard).
You'll also need to define links to the objects you've built, and the version you're using.
And, you're done!
That's it. Granted, you'll want to test this, and debug it, and all the other nonsense, but from a code sense, that's all you'll really need.
By the way, since i mentioned debugging, i want to chime in on the built in tools for Aurora and Nightly. Sadly, there's still no step-debugger, but here's a stupid trick that is INCREDIBLY useful:
- go to about:config
- Open a Web Console [Ctrl+Shift+K]
You now have a web console that can run Javascript with the Browser Chrome. That means that you can run things like Components.util.import('resource://services-sync/utils.js'); Utils.sha1('foo'); and get a pretty SHA1 hash of foo on your screen. And yeah, you can do the same trick with any other Mozilla core function you want to play with.
So, now you know, and knowing is half the battle.
The other half involves blue and red lasers and guys with stupid nicknames.
