I recently switched to using a Chromebook Pixel as my primary device, for both personal and work tasks. Having mentioned this on Twitter, quite a few people asked me to elaborate on my experiences as the Pixel is an enticing device and Chrome OS is largely unknown to many people. Professionally, I am an Systems/Infrastructure/Devops Engineer and my tools mostly consist of a terminal and a browser. I do not generally spend a lot of time in Photoshop, AutoCAD, or Eclipse. If I were a heavy user of such an app, I'd seriously reconsider switching to Chrome OS.

I'm going to focus on Chrome OS in this post, as several other blogs and sites have done in-depth reviews of the Pixel hardware, but I did notice a few things that other reviews seem to have glossed over...

The Pixel is easily one of the best laptops I've ever owned. Coming from a 13" Macbook Air, the size is roughly the same (a bit different due to the 3:2 aspect ratio), it doesn't weigh too much, the battery lasts just about as long, and the display is unbelievably pretty. The top-row of "function" keys are harder to press and have less travel than the rest of the keyboard, with the Esc key being the most annoying for a heavy vim user. The only Apple-patented feature I miss is the magsafe adapter. The barrel plug with the Pixel has a tendency to come loose a bit too easy, but is otherwise acceptable. Overall, it's an incredibly solid feeling device that lacks the cheap plasticy feel of most PCs.

Immediately after I ordered the Chromebook Pixel, I spent some time cleaning up my Google account and switched to an all-Chrome based workflow on my Macbook Air. I had cycled through an Google Apps email, Gmail, and a Google-independent email address over the years, all of which had separate Google accounts. An hour of futzing around with various settings and verifying addresses, I had everything on one unified account. I have a feeling that if I hadn't taken the time to do this, my experience would've involved quite a bit more pain as *everything* on Chrome OS is tied to your Google account.

Upon opening the Pixel, I was greeted with a relatively simple setup wizard that involved connecting to my wireless network (which worked on the second try, dunno why it didn't work the first time), waiting about five minutes for updates to download and install, then logging into my Google account. Once logged in, I was greeted with a set of animations and videos demonstrating basic features of the OS and hardware. The most striking of these demos was "TimeScapes", an HD video that showed off the color depth and resolution of the Pixel's display. Sadly, this is a rare experience. Once you open a Chrome browser, everything you see is scaled to 2x resolution, similar to the way the retina iPad and Macbooks. Type and vector graphics are rendered natively and have nice sharp edges, but most images and videos are disappointingly blurry. I'm sure this will improve in time, as more people own high DPI devices and bandwidth available to the average user increases. Until then, finding high resolution content is a rare and exciting treat.

The tutorial on multi-touch gestures for the trackpad was quick and informative, but I immediately became aware of how few gestures are supported compared to Mac OSX. Scrolling, clicking, and dragging work as expected, and a two-finger tap to emulate a right-click is a welcome addition. Macs might support this, but I was never aware of it, I always just Ctrl-tap. There is no swipe left/right to switch apps as I was used to with Expose/Spaces and pinch to zoom is not supported outside of Google Maps as far as I can tell.

On the subject of multi-touch gestures, the Pixel's display is a capacitive touch surface with Gorilla Glass, and feels very much like a Tablet (except that it's vertical). Scrolling and tapping work as expected with no perceptible lag. Again, pinch-to-zoom and even double-tap-to-zoom aren't supported outside of Google Maps. In it's current state, the touchscreen on the Pixel is mostly a novelty. Reaching up from the keyboard to tap or scroll feels somewhat unnatural and excessive when the touchpad is literally at your fingertips. I occasionally use the touchscreen to scroll through long pages when I'm sitting back and reading, but otherwise only use the touchscreen when showing off the device to other people. I suspect that Google has plans for the touchscreen, but for now it just feels like it was tacked on because why not?

Once I had finished with the tutorials and demos, I was surprised to find that all of the apps and bookmarks I had installed in Chrome on my Mac had been sync'd over and installed automatically, without any visual indication. This saved me a bit of time, but some of the apps didn't sync their settings, so I had to poke around to get things setup the way I like. The most obvious one here was the SSH client. None of my connection settings or keys had been saved, which at first sounds like an inconvenience but is actually a feature. I do not want my keys and credentials anywhere near anybody's servers but my own. Google made the right decision here to require a bit more user interaction to improve security.

Getting my SSH keys onto the Chromebook proved a bit more difficult than I had anticipated. I keep a copy of my private keys on an IronKey USB drive, which requires a userspace program to "unlock" and decrypt the data on it. When plugged in, it mounts a read-only FAT32 filesystem with Windows, Mac, and Linux unlocking utilities. I double-clicked on the Linux binary in the "Files" window, but was informed that it was not supported by Chrome OS. I suspect that I could have switched to Developer Mode at that point and gotten it working in a shell, but I wanted to stay in normal user mode for a few days to really understand the limitations of a fully signed and trusted OS.

I admit that I caved at this point. I opened up my Macbook Air, copied my private key to a web server, and downloaded it on the Pixel from there. This is far from an ideal solution, and would have made me much more nervous if I didn't have an authenticated web server handy. I suppose I could also have copied the key to an unencrypted USB drive, but none of these options were really satisfying. The native SSH client app currently does not support generating new keys either, although based on what I have seen in Chromium bug reports, this is an often-requested feature and will likely be added soon.

Keys imported, I was able to connect to my personal server's tmux session, where I run irssi and bitlbee, as well as an ssh-agent with my credentials for accessing several other sites and servers. I immediately noticed that Ctrl-N and Ctrl-P keybindings in irssi did not behave as expected and would cause Chrome to open a new window or open the print dialog. A few more Google searches later, I learned that I could right-click on the SSH client app in the launcher and set it to always open in it's own window rather than a Chrome tab, which removes these keybindings and allows them to pass through to irssi unharmed. Initially, I thought I'd get tired of having to switch windows to get to my terminal, but this ended up being a non-issue, as Alt-Tab works as expected and fits nicely with the Ctrl-Tab binding that switches tabs in Chrome.

As soon as I started to feel comfortable with this workflow, some minor emergency caused me to login to some servers for work, patch some code, commit it, and deploy it. As I was working on this issue, I didn't feel Chrome OS getting in my way too much. On OSX, I was used to having four to six terminal windows open in a tiled arrangement, with a browser covering 50% of the screen and overlapping a few of the terminals. Due to the scaling of fonts in Chrome OS, only a single 80x24 terminal fits on the screen at once, making it more efficient to just run the terminal full-screen and rely on tmux's pane splitting. This took a little getting used to, but hasn't impeded my workflow much. I've yet to find a way to make the fonts in the Chrome native terminal smaller, but a cursory look at the Chromium hterm code leads me to believe that the formatting of text in ther terminal is solely based on a CSS stylesheet embedded in the extension. I'll likely need to build a custom version of the extension to make the font smaller, but for now it's easy enough to work with.

I was able to change the size of the fonts inside Chrome itself, as everything felt a bit large with the defaults. I feel like this is probably overcompensation for the high native resolution of the display and Chrome OS' pedigree as an OS for netbooks with smaller, lower resolution displays.

The next day, I took the Pixel to work with me, not only to show it to curious co-workers, but also to see if I could truly use it as my primary device. Overall, the answer is yes. I spent several hours writing code, diagnosing problems on production servers, reading logs, poking at Jenkins, reviewing GitHub pull requests, trolling IRC, and responding to things with Gmail. The Google Drive integration is a bit awkward as I'm logged into Chrome OS with my personal account, but have a browser tab logged into my work Google Apps account to edit documents there, but I surprisingly didn't have any issues with other Google services getting confused about which user I was, everything seemed to *stick* to whichever account I had used most recently for that service.

Later that night, I decided it was time to try to run Minecraft on the Pixel. Minecraft is written in Java, uses the LWJGL library, and is supported on Windows, Mac, and Linux using the Sun JRE. However, Java is most definitely not supported on Chrome OS and attempting to run Minecraft as a Java applet in the browser was met with a lovely error about unsupported plugins. We must go deeper.

Armed with the knowlege that Chrome OS is really just a polished Linux system, I found instructions for turning on the "Developer Switch" on the Pixel. The Developer Switch was a physical hardware switch in previous Chromebooks, but has been implemented as a part of the BIOS on the Pixel. Boot into recovery mode with Esc-Refresh-Power, hold Ctrl-D at the right moment, confirm that you want to wipe the stateful partition, and away you go. Yes, switching to Developer Mode requires deleting all non-OS data on the Chromebook. This was a bit disappointing, as I had downloaded a few documents and things already, but it didn't take me too long to copy everything to Google Drive so that I could safely wipe the system. Once again, I had to fool around getting my SSH key imported, but it didn't take me very long this time around, because I immediately jumped to the web server approach.

At first glance, Developer Mode looks no different than the normal mode. If you didn't know any better, you'd say it was exactly the same system. The real difference is in the "crosh" terminal (Ctrl-Alt-T), which originally just had simple commands like ping. With Developer Mode enabled, a "shell" command spawns a familiar bash shell on a Linux system. The shell starts logged in as the "chronos" user, which contains all of Chrome's user data in it's home directory. Reading and understanding the structure of mountpoints and the encrypted loopback devices in /mnt/stateful_partition is incredibly informative and goes a long way toward understanding how Google has hardened Chrome OS against malicious code.

Basically, only the rootfs "/" mountpoint is mounted without the "noexec" option, meaning that files under any other mountpoint are restricted to reading and writing data only, and files stored there cannot be executed as programs. Only the root user can edit files in the rootfs, so this means no executables can be changed without root privileges. Developer mode is a chink in this armor, as it allows the chronos user to use sudo to escalate to root and run any command.

Chrome OS does not include a package manager like apt or yum in it's default install as most other modern Linux distributions do. Instead, package installation is done as a part of building the OS image, which is almost always done by Google themselves. You're free to compile and build your own Chrome OS image, but this requires a fair bit of effort.

I decided to try setting up a minimal Ubuntu chroot jail to run on the Chrome OS system. It is worth mentioning that just wiping Chrome OS and installing a normal Linux distribution on the Pixel may be less trouble at this point. I haven't had any issues with Chrome OS for performing normal tasks, so I wanted to see how much I could do without replacing the OS.

Chrome OS uses a recent Linux kernel, on a common CPU architecture, with a userspace that includes the chroot command. It's certainly possible to work around all of those requirements, but taking advantage of the fact that Chrome OS is a mostly-standard Linux system, we can use some mostly-standard tricks to get a new userspace running.

On another Ubuntu box in a cloud somewhere, I ran this:

mkdir dist
debootstrap precise dist
cd dist
tar -czpvf ../precise.tar.gz .

Then downloaded the resulting precise.tar.gz on the Pixel. Given the noexec flag on most of the mountpoints, and not wanting to trample the Chrome OS install, I built a sparse 8GB loopback image on the stateful partition. This could theoretically be larger, but 8GB is enough for my purposes, and makes it easy to copy the image to a USB drive later if I want a backup.

dd if=/dev/zero of=/mnt/stateful_partition/precise.img bs=1M count=0 skip=8192
losetup /dev/loop1 /mnt/stateful_partition/precise.img
mkfs.ext4 /dev/loop1
mkdir /mnt/stateful_partition/precise
mount /dev/loop1 /mnt/stateful_partition/precise
cd /mnt/stateful_partition/precise
tar -xzvpf /home/chronos/Downloads/precise.tar.gz
cp /etc/resolv.conf /mnt/stateful_partition/precise/etc/
umount /mnt/stateful_partition/precise

Theoretically all of those operations could have been performed on any Linux box, then the resulting image copied to the Pixel, but I did it locally for some reason. Shouldn't matter. Now on to the mounting and chrooting...

mount -o loop /mnt/stateful_partition/precise.img /mnt/stateful_partition/precise
mount -t proc none /mnt/stateful_partition/precise/proc
mount -o bind /dev /mnt/stateful_partition/precise/dev
chroot /mnt/stateful_partition/precise /bin/bash

Those mounts will have to be re-mounted every time the laptop is rebooted, so you may want to shove that in a script somewhere, or add them to /etc/fstab. At this point, it's a mostly functional Ubuntu chroot with very few things installed, no upstart daemon, and full access to all of the host's hardware. Needless to say, this is not secure at all. I haven't played with cgroups and namespaces on Chrome OS yet, but I see no reason why you wouldn't be able to build a more comprehensive jail if you wanted. For my purposes, this is good enough.

A bit of messing about with locales (hint: locale-gen en_US.UTF-8; dpkg-reconfigure locales) and other minor bits of configuration, I installed the Sun JVM and minecraft.jar inside the chroot and bind-mounted /home/chronos into the jail as well, in order to make it find the .Xauthority file. The Minecraft launcher started up (in native eye-bleedingly small resolution), but upon logging in and trying to initialize OpenGL, the screen went blank and X restarted. Sadly, I've not been able to get Minecraft to start without crashing X.

The next day, this jail came in handy as I needed to use a serial console on a router at work. I plugged in my USB-Serial adapter, it showed up as /dev/ttyUSB0 as it does on most Linux boxes, installed picocom in my jail, and jumped on the console, all within a crosh tab.

Over the last week, having an Ubuntu jail handy has become quite useful for even basic tasks like traceroute, nmap, and tcpdump which are not included in the stripped-down Chrome OS install. If you're doing any sort of network or security work, access to these tools is indispensible.

This post is getting a bit long, so I should wrap it up with some conclusions. After a week of using Chrome OS, I've found it to be more than adequate for both work and personal use cases.

As a terminal and browser combination, Chrome OS is perfect. The browser is fast and the terminal works as expected in every scenario I've needed it. Developer mode is nice to have available for when your world doesn't fit inside the sandboxes you're given.

As a media player, I found that Spotify's web player is not available to US accounts yet but Rdio, Amazon Cloud Player, and Grooveshark are viable alternatives. Netflix detects developer mode and refuses to stream any video. The lack of a serious content catalog on Google Play makes it laughable to even mention it. Chrome has libavcodec installed as a plugin, but has certain patent-encumbered codecs disabled and is not associated with all of the Content-types that it actually supports. For example, if you rename a .mkv to .mp4, it'll demux just fine (likely because WebM uses Matroska containers), but some codecs you'd expect to work, like AC3, are missing. For gaming, Chrome OS has a long way to go.

As a gaming platform, few games have been ported to NaCL, and those that have been are more like what you'd find on a tablet device than Steam (think Angry Birds), probably because NaCL only exposes OpenGLES, even though there's an Intel HD 4000 GPU underneath.

Overall, Chrome OS is a really solid attempt to bring a simplified, stripped down computing experience to the masses. As with many other new platforms, you have to buy into the world view of the people that built it. In this case, that means storing the majority of your data in the "cloud" and doing nearly everything in the context of a web browser. It's still a bit rough around the edges but is evolving quickly and there are reasonable workarounds for most issues. If you're tied to a specific application or environment native to an existing platform (Office, Xcode, Adobe), then Chrome OS is not ready for you yet.

I'm pleased with my decision to give Chrome OS a try. While I can't say with certainty that I'll use it forever, I'm happy for now and will continue to use it on a daily basis, supplemented by a Mac Mini for Ableton Live and a Linux box for 3D printing.

Startups iterate quickly. As a devops professional (apparently that's what they call me now), I've found that the ability to turn over code deploys, new services, and configuration changes very fast is critical to keeping your customers (the engineering team) happy. This process is generally referred to as Continuous Integration or CI. A lot of new tools have popped up over the last few years to get code from VCS to prod servers faster along the lines of Capistrano, Fabric, and... probably some other stuff. These tools are great for creating quick-and-dirty, reproducible-ish deployments without a ton of thought. However, this approach is completely wrong.

There is quite a bit of prior art in the area of code deployment in the form of operating system packages. Everything on a fresh install of a server probably came from a system package. Kernels, compilers, editors, network utilities, were all packaged by the OS maintainers and installed by an automated system. Your code is not special, so why treat it's deployment specially?

The problem with OS packaging is that it's a bit of an arcane art. Systems like RPM, Apt, and Ebuild are fraught with outdated documentation, links to dead blogs, buried mailing list posts, and "just ask the dude on IRC, he's pretty cool". But you can't ignore that all of these systems have essentially solved every type of code deployment problem you're likely to face, and as lazy devops/sysadmin people, we hate reinventing the wheel, so here we go.

For the sake of brevity, sanity, and inanity, I'm going to assume that you're deploying a Python web app in a Git repo and deploying to a recent Debian or Ubuntu system. Our goal is to be able to install a package using Puppet, Chef, CFengine, or something else that puts all of your code in a standard location every time, on every server.

Ok, the basics. Debian packages are .deb files. A .deb file is basically a compressed tar archive containing a built binary of your app with some extra metadata about versioning and dependencies. The dpkg-deb utility allows us to examine and unpack .deb files. Take a look at an existing package's metadata with dpkg-deb -I /var/cache/apt/archives/something_1.2.3-1.deb. Anytime you apt-get install a package, it caches the downloaded archive in /var/cache/apt/archives. If you've installed a lot of things, you'll have a lot of .deb files here. Feel free to poke around, take a look at man dpkg-deb for more options and tools for playing with .deb archives.

So, why can't we just pack up our source directory into a .deb file and call it a day? That'll get our files onto the server, and it'll work the same every time, right? Sure, but then you'd be skipping a lot of the really useful and important parts of the Debian packaging system. As a rule, nothing makes it into upstream Debian unless it can be built from a special type of package called a source package. Source packages provide all of the instructions, dependencies, and versioning information required for the build system to generate a .deb for (theoretically) any architecture or distribution using Debian-style packaging from nothing but source code.

Great, so how do you create a source package? First, you'll need some tools.

sudo apt-get install devscripts pbuilder git-buildpackage dh-make apt-utils

Ok, so what are these things? pbuilder will create a chroot jail (google it, I'll wait) around a completely clean and sterile, minimal Debian install, and build your package inside the jail using a bunch of tools, rather than directly on your system with god-knows-what installed on it. This ensures that your final built package is reproducible, that some dude in Siberia that wants to hack on your code or port it to a different system gets the same resulting binaries that you did. git-buildpackage wraps around pbuilder to provide intelligent management of package sources within a git repository, and dh-make generates boilerplate for new source packages.

Let's start packaging. This is a Python app, so we'll package it the standard Python way, using distutils or setuptools. A simplistic setup.py is enough to get started: http://docs.python.org/distutils/introduction.html#a-simple-example

The Debian maintainers have spent a LOT of time figuring out the best way to get Python code onto a system in sys.path without resorting to patching every upstream package drastically. All you need to do to get a source package is to add a debian/ directory at the root of your repository. Of course, there are already several tools that will setup the boilerplate debian/ for you, the most common of which is called dh_make. There are other similar tools specific to different languages, feel free to Google them.

~/myfirstwebapp-0.1 $ export DEBEMAIL="jeremy@example.com"
~/myfirstwebapp-0.1 $ export DEBFULLNAME="Jeremy Grosser"
~/myfirstwebapp-0.1 $ dh_make --createorig
~/myfirstwebapp-0.1 $ rm debian/*.{ex,EX}

First, we set a couple environment variables to tell the scripts what to fill in for the maintainer and author fields. You probably want to just put this in your .bashrc. dh_make is kinda old and janky, so your source directory has to be named - or it'll whine about it. You also have to pass --createorig to generate an .orig.tar.gz (more on this later), which is pretty irrelevant when using git-buildpackage, but again we do it to keep dh_make from complaining. When dh_make asks you what kind of package to create, just type "s" and press enter for now, the other types are designed for different packaging scenarios, like kernel modules. You may want to try CDBS if you're packaging an Ant library or some other standardish build scripts. We also deleted a bunch of the example files that dh_make created because for simple packages they're completely unnecessary. Feel free to browse around the examples if you've got more complicated projects in mind, they are well commented.

At this point, you have something vaguely resembling this:

~/myfirstwebapp-0.1 $ ls -1 debian/
changelog
compat
control
copyright
docs
README.Debian
README.source
rules
source

Of these, changelog, compat, control, and rules are absolutely required. The rest are technically optional, but you probably don't want to get rid of them.

debian/changelog is exactly what it sounds like, a changelog of what you've done with this source package. Each section of this file represents a package version. Note that all of the spacing and formatting in this file is very specific and your build will break if you change spaces to tabs, for example. Let's take a closer look at the version number: (0.1-1). Version numbers in packages hold a lot of meaning to the system. The first part 0.1 is the upstream package version. If you had downloaded this package from some website, this is the upstream author's version. The second part -1 is the Debian version. Occasionally it's useful to be able to revise the packaging metadata or build scripts for a package before the upstream maintainer has tagged a new release. In those cases, this number simply gets incremented. There are a lot of different ways to represent a version here, they're all enumerated in the completely difficult to comprehend Debian Policy Manual, section 5.6.12 (http://www.debian.org/doc/debian-policy/ch-controlfields.html#s-f-Version). All you really need to know is that a version with a "-" in it came from an upstream source (eg. you didn't write it), and anything that's just a version number like "2.5.1" with no "-" part, is known as a native package. In most cases, you'll wanna stick with a non-native package and keep the debian version separate from the upstream version.

debian/compat basically just tells apt that you're using a not ancient spec for package layout. You will probably never need to change this.

debian/control provides a lot of the metadata that's used for managing how your package appears to the user and the rest of the system. Let's dive a bit deeper...

Source: myfirstwebapp
Section: unknown
Priority: extra
Maintainer: Jeremy Grosser <jeremy@example.com>
Build-Depends: debhelper (>= 8.0.0)
Standards-Version: 3.9.2
Homepage: <insert the upstream URL, if relevant>
#Vcs-Git: git://git.debian.org/collab-maint/myfirstwebapp.git
#Vcs-Browser: http://git.debian.org/?p=collab-maint/myfirstwebapp.git;a=summary

Package: myfirstwebapp
Architecture: any
Depends: ${shlibs:Depends}, ${misc:Depends}
Description: <insert up to 60 chars description>
 <insert long description, indented with spaces>

As you can see, there are two sections here. The first one describes the source package and the dependencies needed to build it, where the second one describes the resulting binary package and dependencies required to install it. Let me repeat that: there are two dependency lists, one for building and one for installing. The rest of the fields here are fairly self-explanatory and some are even optional (but highly recommended). There are a lot of different things that you can specify in the control file, there's a whole section dedicated to it in the Debian Policy Manual. http://www.debian.org/doc/debian-policy/ch-controlfields.html

So what belongs in your Build-Depends list? debhelper (>= 8.0.0) is listed by default, because you'll basically always want it. Debhelper gives you all kinds of useful shortcuts and anything prefixed with dh_ probably comes from here. Yeah, you want that, it makes your life easier. We're building a python package, so let's add... python! Amazing how that works. If your package had C extensions, you'd also want to include python-all-dev in this list. If you're using setuptools, you'll need that too, it's called python-setuptools. Ideally you'd also specify a minimum version number for each of these things (except python, but that's a whole can of worms, just leave it without a version). I tend to leave explicit versions off of the Build-Depends while getting a new package working, then go back and add them later, once I know exactly what I need and what's available.

In the binary package section, we don't need to change a lot. Unless you're building C extensions or have some other dependency on x86 architecture, change that to "all", which causes the build system to create one binary package that can be installed on any system, like a home router with an ARM chip, for example. If you left this as "any", you'd have to rebuild the binary for each target architecture.

It's worth noting, your binary package name does not have to be the same as your source package name. It's actually fairly common for a source package to be something like "sqlalchemy" that creates a binary named "python-sqlalchemy". In the case of Python packages, the convention is that libraries and module are prefixed with "python-" where services and tools that just happen to be written in Python aren't prefixed.

A single source package can define several binary packages. For example, the "sqlalchemy" source package creates a "python-sqlalchemy-doc" binary, so that you can install the docs separately. Multiple binary packages are outside the scope of this document.

At this point, you should have something like this:

Source: myfirstwebapp
Section: misc
Priority: extra
Maintainer: Jeremy Grosser <jeremy@example.com>
Build-Depends: debhelper (>= 8.0.0), python
Standards-Version: 3.9.2
Homepage: http://example.com/dist/myfirstwebapp-latest.tar.gz
Vcs-Git: https://github.com/synack/myfirstwebapp.git
Vcs-Browser: https://github.com/synack/myfirstwebapp

Package: myfirstwebapp
Architecture: all
Depends: ${shlibs:Depends}, ${misc:Depends}, python-bottle, python-jinja2
Description: My first web app!
 There are many like it, but this one is mine. Accessories sold separately, batteries not included.

debian/copyright lists out who maintains this code, who wrote it, who owns the rights to it, how they've licensed it, etc, etc. Technically you don't need it, but somebody somewhere on the internet cares about this sort of thing. If you ignore it, prepare for an onslaught of trolls.

debian/docs is simply a list of documentation files that come with your package. They'll get installed somewhere under /usr/share/doc and nobody will ever read them before emailing you. How sad. In most cases, you'll just put something like "README.txt" in here and call it a day.

debian/README.Debian and debian/README.source are where you'd put any notes specific to the packaging of this application. If you don't have anything interesting to say, just delete them.

debian/rules is a Makefile that needs to support a bunch of different targets like "build" and "install". This is where the real work of the package comes in. The build target needs to do any sort of compiling, linking, and whatnot to turn your code into something useful, then the install target comes along and lays it all out under debian/tmp/ as close to the FHS standard (http://www.pathname.com/fhs/) as possible. The good news is that you don't have to write any of that! Debhelper is here to... uh... help! The example rules file generated by dh_make is basically all you need:

#!/usr/bin/make -f
# -*- makefile -*-
# Sample debian/rules that uses debhelper.
# This file was originally written by Joey Hess and Craig Small.
# As a special exception, when this file is copied by dh-make into a
# dh-make output file, you may use that output file without restriction.
# This special exception was added by Craig Small in version 0.37 of dh-make.

# Uncomment this to turn on verbose mode.
#export DH_VERBOSE=1

%:
    dh $@

With one slight addition, this will work perfectly for us. Change that last line to dh --with=python2 $@ and you're done. This tells debhelper that there's a setup.py in the root of your source package and it behaves as expected, just like that.

debian/source/format hints to the build system whether this is a native or non-native package.

Native: 1.0

Non-native: 3.0 (quilt)

quilt is a patching system that allows your debian packaging to apply patches to the upstream tarball before building it. We're not using it here, but you have to say that anyway. If you need to edit something from upstream and you can't get a patch submitted there, you can create a debian/patches directory with a bunch of patch files in it, and the build scripts will take care of getting that all squared away during the build. If you need this, go read the quilt manpages.

That's all. You now have a basic Debian source package. As we're managing this repo with git-buildpackage, there are some conventions for where to commit it. The ideal case is that you have an upstream branch and a master branch, where upstream is simply a copy of the latest upstream release, and master has the debian/ directory added to it. When a new upstream release comes out, you merge it into the upstream branch, then merge that to master, update debian/changelog (using git-dch or debchange), make sure the dependencies in debian/control are still accurate, and commit to master. This whole process becomes quite a bit easier if the upstream package maintainer also uses git, or if you use a tool like git-import-orig to pull in new release tarballs. It's good practice to also tag each upstream release, as well as your debian package releases. git-dch and git-import-orig will do this for you, given the right arguments.

We're going to be setting up a brand new pbuilder environment, which means building a new tarball that represents a clean debian install for our architecture and distribution. Every time a build runs, it'll untar this into a chroot jail and run the build there.

pbuilder create

This command will take a while and download all of the packages necessary to build a minimal debian system using a tool called debootstrap. If you wanted to build for a different architecture or distribution than the one you're currently runnning, you'd edit /etc/pbuilderrc or ~/.pbuilderrc to point it at the right mirror, dist, tarball location, build result location, etc, etc. The pbuilderrc(5) lists all of the options you can use there. For now, let's just stick with the defaults.

sudo git-buildpackage --git-builder='pdebuild --debsign-k jeremy@example.com --auto-debsign'

This kicks off the building of our package, using the sources in the current working directory. It needs root permissions in order to create the chroot jail. chroot is a privileged syscall. It goes through a lot of different stages, roughly:

1. Runs debian/rules clean to make sure our source directory isn't dirty
2. Create a source package consisting of a .dsc, .debian.tar.gz, .orig.tar.gz, and .changes files from our upstream and master branches.
3. Setup a new chroot jail and untar the base tarball into it
4. Create and install a virtual package that installs all of our Build-Depends into the jail
5. Copies our source package into the jail
6. Runs debuild inside the jail, which calls dpkg-buildpackage. The manpages for these scripts tell you more about what they do. All you need to know is that they parse debian/control and debian/changelog, call your rules targets and spit out a .deb
7. Signs the resulting package with your GPG key, identified by the debsign-k argument to pdebuild. You may be prompted to enter your key's passphrase at this point.

If you're lucky, you waited a while, saw a lot of text scroll by, and the build finished successfully. Your built package is now in /var/cache/pbuilder/result

$ ls -1 /var/cache/pbuilder/result/myfirstwebapp_0.1*
myfirstwebapp_0.1-1_all.deb
myfirstwebapp_0.1-1_amd64.changes
myfirstwebapp_0.1-1.debian.tar.gz
myfirstwebapp_0.1-1.dsc
myfirstwebapp_0.1.orig.tar.gz

Pretty awesome, right? Now you just need a Debian repository to upload it to.

A Debian package repository is nothing more than a bunch of files arranged in a specific way, with a files like Packages and Release that tell apt what versions of what packages are available and where to find the .deb files to download. You can browse around most official Debian distributions to get a feel for how a large repo is laid out: http://ftp.us.debian.org/debian/. For our purposes, let's keep it a bit simpler. We're just going to generate a Packages and Release file in the same directory as our binaries.

mkdir -p /var/www/mybuntu
cd /var/www/mybuntu
cp /var/cache/pbuilder/result/myfirstwebapp_0.1* .
apt-ftparchive packages >Packages
apt-ftparchive release >Release
gpg --yes --abs -u jeremy@example.com -o Release.gpg Release

You'll need to run this (everything except the mkdir -p part) every time you add a new package to the repository. It's probably a good idea to write a script so you don't forget all that stuff. apt-ftparchive supports much more complicated layouts and configuration files, I encourage you to read the manpages for that so you can do clever things like manage multiple distributions (separating production and development packages is really handy).

Now you just need to add this apt repository to your clients. Edit /etc/apt/sources.list.d/mybuntu.list on some other server or even your repository machine, if so inclined.

$ cat /etc/apt/sources.list.d/mybuntu.list
deb http://packages.example.com/mybuntu/ ./
$ sudo apt-get update
$ sudo apt-get install myfirstwebapp

Ta da! Your webapp is now installed in the python path! Any scripts you defined in your setup.py are in /usr/bin, and everything is happy and reproducible. You can build new releases of your package, put them in the apt repository and now they're available to all of your servers. Combine this with a config management tool like Puppet where you can do things like ensure => latest and your code gets updated on every puppet run. Need to roll back code because you broke everything? apt-get install myfirstwebapp=0.1-1 or just remove it completely: apt-get remove myfirstwebapp

These tools fit quite nicely with existing CI tools like Jenkins, allowing you to do automated testing, building, and deployment of a package every time somebody commits to master.

After seeing Hack-a-Day’s post about a cheap IP webcam that runs Linux I couldn’t resist the urge to dig into yet another embedded system. I don’t really have a definite need for something like this, but thought it sounded like a neat piece of hardware to hack around with, so why not?

When I first plugged the camera in, I didn’t see it grab an IP over DHCP and went digging for the Manual (PDF). It boots up as 192.168.2.3 with a webserver on port 80. Default username and password: admin/1234.

After changing the IP to a more reasonable subnet, it functions as a nifty little IP webcam providing either a MJPEG or MPEG4 stream over RTSP. Not bad for $40.

Cracking the case open was a bit difficult, until I realized there was a screw hidden under the label on the back.

Time for hacking! I’ve found at least two potential ways of getting a shell on this thing, with several more theoretically possible. The easiest approach is the one that Hack-a-Day mentioned: wire up a serial console to some empty pads on the board and rock on. There are four pads on the back side of the device (the side without the camera sensor). In the picture below, from left to right: 3.3v, RX, TX, GND. Please excuse my horrible soldering job.

For the sake of simplicity, I connected this header up to my Bus Pirate and configured the port as 115200 8e1.


Bus Pirate v3a
Firmware v5.10 (r559) Bootloader v4.1
DEVID:0x0447 REVID:0x3043 (24FJ64GA002 B5)

http://dangerousprototypes.com

HiZ>m
1. HiZ
2. 1-WIRE
3. UART
4. I2C
5. SPI
6. 2WIRE
7. 3WIRE
8. LCD
9. DIO
x. exit(without change)

(1)>3
Set serial port speed: (bps)
1. 300
2. 1200
3. 2400
4. 4800
5. 9600
6. 19200
7. 38400
8. 57600
9. 115200
10. BRG raw value

(1)>9
Data bits and parity:
1. 8, NONE *default
2. 8, EVEN
3. 8, ODD
4. 9, NONE
(1)>2
Stop bits:
1. 1 *default
2. 2
(1)>1
Receive polarity:
1. Idle 1 *default
2. Idle 0
(1)>1
Select output type:
1. Open drain (H=Hi-Z, L=GND)
2. Normal (H=3.3V, L=GND)

(1)>2
Ready
UART>(1)
UART bridge
Reset to exit
Are you sure? y


Assuming everything’s wired up correctly, you should now be able to watch the board boot up and interact with it on the console.

Welcome to Proboot 1.0
Prolific Technology Inc. 2003-2006

[General]
Version : 1.0.39
Date : 2009/10/26
CPU : ARM FA526
MemSize : 32 MB
Cache : vIbrSLWCaM
IP : 0.0.0.0
MAC : 00:00:00:00:00:00

[OnBoard NOR Flash]
Type : PPI-AMD
ID : 00a8
Sectors : 71
Size : 4MB

[Memery Layout]
Stack : 0x00400000 - 0x003c0000
Shadow : 0x00600000 - 0x006a4f68
Pages : 0x00004000 - 0x00008000
Buffer : 0x00700000 - 0x00800000
Tags : 0x00002000 - 0x00004000

script 0x19802000 8192 2
Running script (Ctrl-C to cancel)

echo "loading compressed kernel"
loading compressed kernel
load 0x20000 0x220000 786432
echo "execute kernel"
execute kernel
exec 0x220000 "root=/dev/norblock/disc0/disc rootfstype=squashfs nor=b:0x320000@0xe0000,c:8192@0x4000,c:40960@0x6000 prolific.keypad=GPIO_IN(11);GPIO_OE(5,6);"
Uncompressing Linux......................................... done, booting the kernel.
Linux version 2.4.19-pl1029 (root@neo) (gcc version 3.3.4) #219 ?| 7?? 29 10:55:30 CST 2010
CPU: Faraday FA526id(wb) revision 1
ICache:16KB enabled, DCache:16KB enabled, BTB support, IDLE support
Machine: Prolific ARM9v4 - PL1029
Prolific arm arch version 1.0.12
On node 0 totalpages: 8192
zone(0): 8192 pages.
zone(1): 0 pages.
zone(2): 0 pages.
Kernel command line: root=/dev/norblock/disc0/disc rootfstype=squashfs nor=b:0x320000@0xe0000,c:8192@0x4000,c:40960@0x6000 prolific.keypad=GPIO_IN(11);GPIO_OE(5,6);
plser console driver v2.0.0
Calibrating delay loop... 147.56 BogoMIPS
Memory: 32MB = 32MB total
Memory: 30948KB available (1064K code, 297K data, 64K init)
Dentry cache hash table entries: 4096 (order: 3, 32768 bytes)
Inode cache hash table entries: 2048 (order: 2, 16384 bytes)
Mount-cache hash table entries: 512 (order: 0, 4096 bytes)
Buffer-cache hash table entries: 1024 (order: 0, 4096 bytes)
Page-cache hash table entries: 8192 (order: 3, 32768 bytes)
POSIX conformance testing by UNIFIX
PCI: Probing PCI hardware on host bus 0.
Linux NET4.0 for Linux 2.4
Based upon Swansea University Computer Society NET3.039
Initializing RT netlink socket
Prolific addr driver v0.0.4
tts/%d0 at MEM 0x1b000400 (irq = 2) is a PLSER
Starting kswapd
devfs: v1.12a (20020514) Richard Gooch (rgooch@atnf.csiro.au)
devfs: boot_options: 0x1
squashfs: version 3.1 (2006/08/19) Phillip Lougher
i2c-core.o: i2c core module version 2.8.1 (20031005)
i2c-dev.o: i2c /dev entries driver module version 2.8.1 (20031005)
Prolific i2c algorithm module v1.2
Initialize Prolific I2C adapter module v1.2.1
found i2c adapter at 0xd9440000 irq 17. Data tranfer clock is 100000Hz
i2c-proc.o version 2.8.1 (20031005)
pty: 256 Unix98 ptys configured
PL-1029 NOR flash driver, version 0.8.4
NOR flash type: ppi-amd 8x8 64x63
NOR: PPI
NOR flash id = 0xa8
nor interrupt 30 registered
Partition check:
norblocka: unknown partition table
RAMDISK driver initialized: 16 RAM disks of 2048K size 1024 blocksize
PPP generic driver version 2.4.2
PPP Deflate Compression module registered
PPP BSD Compression module registered
usb.c: registered new driver usbdevfs
usb.c: registered new driver hub
usb-ohci-pci.c: usb-00:05.0, PCI device 180d:2300
usb-ohci.c: USB OHCI at membase 0xd8400000, IRQ 9
usb.c: new USB bus registered, assigned bus number 1
usb.c: ### @@@ usb_set_address

usb.c: >>> usb_get_device_descriptor

usb.c: >>> usb_get_configuration

Product: USB OHCI Root Hub
SerialNumber: d8400000
hub.c: USB hub found
hub.c: 4 ports detected
Prolific Real-Time Clock Driver version 1.0.0 (2003-04-02)
Prolific keypad driver v1.0.8
PL UART driver version 1.1.0-1 (2007-02-15)
ov7670.o version 1.0.0.0 (20070316) [ fine tune ]
OV7670AttachAdapter() be called
OV7670Detect() be called
Set 166 registers value
NET4: Linux TCP/IP 1.0 for NET4.0
IP Protocols: ICMP, UDP, TCP
IP: routing cache hash table of 512 buckets, 4Kbytes
TCP: Hash tables configured (established 2048 bind 4096)
NET4: Unix domain sockets 1.0/SMP for Linux NET4.0.
NET4: Ethernet Bridge 008 for NET4.0
Fast Floating Point Emulator V0.9 (c) Peter Teichmann.
VFS: Mounted root (squashfs filesystem) readonly.
Mounted devfs on /dev
Freeing init memory: 64K
pl serial only support even parity
MAC address: 00:1F:1F:CE:C8:25
MAC address: 00:1F:1F:CE:C8:25
init started: BusyBox v1.01 (2009.10.29-06:28+0000) multi-call binary
init started: BusyBox v1.01 (2009.10.29-06:28+0000) multi-call binary
Starting pid 10, console /dev/tts/0: '/etc/rc.d/rcS'
F: applet not found
Enable alignment-fixup function...
$Starting tmpfs:---->

$Starting encoder driver module: ---->
Using /lib/modules/plmedia.o
plmedia version 1.3.0
Hello Grabber!
[Grab] : Preallocation memory space 2097152 bytes
Hello Encoder!
Hello PLMD!
Warning: loading plmedia will taint the kernel: non-GPL license - PLGRAB
See http://www.tux.org/lkml/#export-tainted for information about tainted modules
Using /lib/modules/crypt.o
aes interrupt 12 registered
ic3010_ctrl_init...
Using /lib/modules/ic3005_ctrl.o
Warning: loading ic3005_ctrl will taint the kernel: no license
See http://www.tux.org/lkml/#export-tainted for information about tainted modules
Using /lib/modules/rt3070sta.o
Warning: loading rt3070sta will taint the kernel: no license
See http://www.tux.org/lkml/#export-tainted for information about tainted modules
rtusb init --->
usb.c: registered new driver rt2870
SIOCGIFFLAGS: No such device
Using /lib/modules/sundance.o
sundance.c:v1.30b 7-9-2007 Written by Donald Becker

http://www.scyld.com/network/sundance.html

eth0: IC Plus IP100A Fast Ethernet Adapter at 0xc28bf000, 00:00:00:00:00:00, IRQ 5.
eth0: MII PHY found at address 0, status 0x7849 advertising 01e1.

Starting enet...

****************************************************************
Initializing ipc between Spook and Enet....: Success
****************************************************************

: Success
total enet structure = 5693 (163D) checksum:(enet:532F) (flash:532F)

default MAC address: 00 1F 1F CE C8 25
eth0 Link down, 7849.
neo: netdev_open Reset PHY
neo: netdev_open Reset PHY 2
eth0: Promiscuous mode enabled.
device eth0 entered promiscuous mode
Starting pibr0: port 1(eth0) entering learning state
d 48,br0: port 1(eth0) entering forwarding state
br0: topology change detected, propagating
console /dev/tts/0: '/bin/ash'

BusyBox v1.01 (2009.10.29-06:28+0000) Built-in shell (ash)
Enter 'help' for a list of built-in commands.

/ # NTP thread start.. PID:52
udpsock port: 4322
httpd->port:80

link_status->wlg_ifindex: 0
link_status->lan_ifindex: 2
enet_watchdog_init
eNET main task started
Init eth0: Promiscuous mode enabled.
random number generator with value: 505
System startbr0: port 1(eth0) entering disabled state
up [ v1.0 (Jul 29 2010 10:55:12) ]
enet_watchdog_start
klogd thread started
Set gateway (change the routing table)
add 255.255.255.255/255.255.255.255 to routing table for broadcast
embedded av stream server started PID:58

Starting mpeg4 pipe server for Spook
Waiting for a connection from spook... (video)
video device thread started PID:60
motion detect thread started PID:61
audio device thread started PID:62
embedded av stream controller started 63
SO_SNDBUF:131070 SO_SNDLOWAT:1
set audio capture properties
start audio capture ok
stop video capture
stop video device.....
set video capture properties
Enter normal VGA mode...
set webpush fps to 30
set video capture properties done
set motion detection properties
set motion detection properties done
start video capture
success to open digits.bmp file

Set 166 registers value
Opened /dev/pl_grab successfully
uDenominator:30 uNumerator:1
******************************************************
MPEG4 encoding source: 640 X 480
MPEG4 encoding format: 640 X 480
MPEG4 bitrate: 0
MPEG4 FPS: 30
MPEG4 VBR nQuant: 5
Opened /dev/pl_enc successfully (MPEG4)
******************************************************
skip MPEG4 encoder!! Using MD profile instead
Jan 1 00:00:12 spook[65]: listening on control socket /tmp/spook.sock
Jan 1 00:00:12 spook[65]: unable to open /tmp/spook.conf: No such file or directory
killall: spook: no process killed
generate new spook.conf... (MPEG4)
******************************************************
MJPEG encoding source: 640 X 480
MJPEG encoding format: 640 X 480
MJPEG quality: 4
MJPEG FPS: 30
Opened /dev/pl_enc successfully (MJPEG)
******************************************************
******************************************************
MD encoding source: 640 X 480
MD encoding format: 640 X 480
MD FPS: 30
MD VBR nQuant: 5
Opened /dev/pl_enc successfully (MD)
******************************************************
start video capture ok
httpd thread started PID:69
mmap Y[0]
enet agentd started PID:73
This board has no wireless supported.
encode VOS header...
enet
00 00 01 B0 08 00 00 01 B5 89 13 00 00 01 00 00 00 01 20 00 C4 88 80 0F 51 40 43 C1 46 3F 00 00
first read video at 13:54
mmap Y[2]
eth0: Promiscuous mode enabled.
eth0 Link down, 7849.
neo: netdev_open Reset PHY
neo: netdev_open Reset PHY 2
eth0: Promiscuous mode enabled.
eth0: Promiscuous mode enabled.
br0: port 1(eth0) entering learning state
eth0: Promiscuous mode enabled.
br0: port 1(eth0) entering forwarding state
br0: topology change detected, propagating
E-mail schedule system started (PID:51)
FTP schedule system started (PID:50)
mmap Y[1]
Jan 1 00:00:15 spook[78]: listening on control socket /tmp/spook.sock
Jan 1 00:00:15 spook[78]: listening on tcp port 554
Jan 1 00:00:15 spook[78]: pl1029-video: start_block
Jan 1 00:00:15 spook[78]: pl1029-video: set_device
Jan 1 00:00:15 spook[78]: pl1029-video: set_framesize
Jan 1 00:00:15 spook[78]: pl1029-video: set_bitrate
Jan 1 00:00:15 spook[78]: pl1029-video: set_format
Jan 1 00:00:15 spook[78]: pl1029-video: set_framerate_num
Jan 1 00:00:15 spook[78]: pl1029-video: set_output
Jan 1 00:00:15 spook[78]: pl1029-video: end_block
Jan 1 00:00:15 spook[78]: pl1029-video: device: /tmp/mpeg4Stream
Jan 1 00:00:15 spook[78]: pl1029-video: format: 100
Jan 1 00:00:15 spook[78]: pl1029-video: bitrate: 2048
Jan 1 00:00:15 spook[78]: pl1029-video: width: 640
Jan 1 00:00:15 spook[78]: pl1029-video: height: 480
Jan 1 00:00:15 spook[78]: pl1029-video: fps: eth0: Promiscuous mode enabled.
30
Jan 1br0: port 1(eth0) entering disabled state
00:00:15 spook[78]: pl1029-video: gopsize: 30eth0: Promiscuous mode enabled.
eth0 Link down, 7849.
neo: netdev_open Reset PHY
neo: netdev_open Reset PHY 2
eth0: Promiscuous mode enabled.

Jan 1 00:00:eth0: Promiscuous mode enabled.
br0: port 1(eth0) entering learning state
eth0: Promiscuous mode enabled.
15 spbr0: port 1(eth0) entering forwarding state
br0: topology change detected, propagating
ook[78]: pl1029-video: quality: 12
Accepted a new connection from spook... (video)
Jan 1 00:0br0: port 1(eth0) entering disabled state
0:15 spook[78]: pl1029-video: capture_loop
=============== reinit br0: port 1(eth0) entering learning state
libbrbr0: port 1(eth0) entering forwarding state
br0: topology change detected, propagating
idge ================ ...
Jan 1 00Inconsistent of vop_type (i-vop)!!
:00:15 spook[78]: pl1029-video: set_running
Jan 1 00:00:17 spook[78]: waiting for VOS header...
Set gateway (change the routing table)
add 255.255.255.255/255.255.255.255 to routing table for broadcast
encode VOS header...
enet
00 00 01 B0 08 00 00 01 B5 89 13 00 00 01 00 00 00 01 20 00 C4 88 80 0F 51 40 43 C1 46 3F 00 00
Jan 1 00:00:17 spook[78]: pl1029-video: set_running
Terminated
generate new spook.conf... (MPEG4)
send mpeg4 data to spook pipe failed. aborted..
Waiting for a connection from spook... (video)
Inconsistent of vop_type (i-vop)!!
Jan 1 00:00:18 spook[86]: listening on control socket /tmp/spook.sock
Jan 1 00:00:18 spook[86]: listening on tcp port 554
Jan 1 00:00:18 spook[86]: pl1029-video: start_block
Jan 1 00:00:18 spook[86]: pl1029-video: set_device
Jan 1 00:00:18 spook[86]: pl1029-video: set_framesize
Jan 1 00:00:18 spook[86]: pl1029-video: set_bitrate
Jan 1 00:00:18 spook[86]: pl1029-video: set_format
Jan 1 00:00:18 spook[86]: pl1029-video: set_framerate_num
Jan 1 00:00:18 spook[86]: pl1029-video: set_output
Jan 1 00:00:18 spook[86]: pl1029-video: end_block
Jan 1 00:00:18 spook[86]: pl1029-video: device: /tmp/mpeg4Stream
Jan 1 00:00:18 spook[86]: pl1029-video: format: 100
Jan 1 00:00:18 spook[86]: pl1029-video: bitrate: 2048
Jan 1 00:00:18 spook[86]: pl1029-video: width: 640
Jan 1 00:00:18 spook[86]: pl1029-video: height: 480
Jan 1 00:00:18 spook[86]: pl1029-video: fps: 30
Jan 1 00:00:18 spook[86]: pl1029-video: gopsize: 30
Jan 1 00:00:18 spook[86]: pl1029-video: quality: 12
Accepted a new connection from spook... (video)
Jan 1 00:00:18 spook[86]: pl1029-video: set_running
Jan 1 00:00:18 spook[86]: pl1029-video: capture_loop
Jan 1 00:00:18 spook[86]: waiting for VOS header...
Inconsistent of vop_type (i-vop)!!
encode VOS header...
enet
00 00 01 B0 08 00 00 01 B5 89 13 00 00 01 00 00 00 01 20 00 C4 88 80 0F 51 40 43 C1 46 3F 00 00
Jan 1 00:00:19 spook[86]: pl1029-video: set_running
Inconsistent of vop_type (i-vop)!!
enet link_status thread started PID:81
eth0: Promiscuous mode enabled.
Network initial status: WLAN
WLAN was disabled! link status still was LAN
/ #


At this point, you can poke around a bit. The system has a relatively minimal busybox config installed with two processes making up the webcam interface. “/bin/spook” and “/sbin/enet”. Taking a step back, the web server sends a Server header in it’s responses identifying it as a GoAhead web server. This is a small web server designed to be embedded into another process, in this case, spook. The GoAhead web server has a long list of vulnerabilities attributed to it, including a couple good remote buffer overflows. For someone so inclined, this might provide an alternate method of hacking the RXS-3211 without modifying the hardware.

Getting back to the software itself, I noticed some strange behavior if I attempted to use any program to communicate with the network other than spook. After examining dmesg and thinking carefully about how such a low-powered device can provide streaming video at 30fps, I can make an educated guess about how the system works. When the snoop process starts, it maps a shared memory section with the enet process. snoop instructs the hardware to capture frames, encode them as either MPEG4 or MJPEG and write the result into the shared memory area. The enet process then instructs the ethernet hardware to copy the data out of shared memory and send it out to the network. Assuming that this hypothesis is correct, then doing *anything* that attempts to send data to the network while the enet process is copying data directly to the ethernet chip’s buffers will cause the whole network stack to lock up and break all network connectivity. This makes it exceedingly difficult to copy files off the device or interact with it over the network in anything other than the prescribed manner.

Faced with this setback, I started looking toward the bootloader. Astute readers will have noticed that the bootloader pauses for a couple seconds at startup, prompting for Ctrl-C on the serial console. Doing so drops you into a “proboot>” prompt.


proboot> credit
We fight for the honor of our company.
Prolific bootloader - Proboot - Professional bootloader
by Jun Chen, Hugo Leu, Jedy Wei, CC Yen 2003-2006


It seems to be a fairly sane bootloader with some nice options like loading and dumping images over tftp. Unfortunately, proboot does not recognize the PL1023′s network hardware and expects a Realtek 8139 to be connected. I suspect that this is what the extra set of pads on the I/O board are for: an RTL8139 chip and RJ-45 jack for interacting with the bootloader. What’s left on the board is likely a relic of the development cycle.

Faced with spotty network access in userspace and no viable network interface in the bootloader, I decided to dump out the firmware image to the serial console using the “memdump” command. memdump takes a memory address and length and reads the whole thing out in the common hex/ascii dump format. While this format is intended more for human consumption than machines, I was able to turn it into a binary dump by parsing my terminal emulator’s output log with a few Python scripts.

Based on the default boot script, I was able to deduce the locations of the kernel and three sections of the NOR flash, one of which appears to be the root filesystem… I assume the other two flash sections store non-volatile configuration that is preserved between reboots.

Now that I have access to the firmware, I plan to modify the /etc/rc.d/rcS script in the rootfs to not start spook so that I’ll have a chance to interact with the device over the network unhindered. Eventually, I’d like to build an entirely new rootfs image with some sort of support for writing to flash, currently the system is read-only.

Other observations:

• The reset button does nothing
• There are configuration and firmware files for at least three different wireless chipsets, implying that the same software may be running on other IP webcams with better hardware.
• Without networked access to the bootloader, I’ll have to figure out what sort of “Firmware upgrade” image the webserver expects to be uploaded. Rosewill has not released any upgrade image for this device, so I don’t have a whole lot to go on.


/ # cat /proc/cpuinfo
Processor : Faraday FA526id(wb) rev 1 (v4l)
BogoMIPS : 147.56
Features : swp half

Hardware : Prolific ARM9v4 - PL1029
Revision : 0000
Serial : 0000000000000000
/ # cat /proc/gpio
GPIO_0: Disabled.
GPIO_1: Disabled.
GPIO_2: Disabled.
GPIO_3: Disabled.
GPIO_4: Disabled.
GPIO_5: Disabled.
GPIO_6: Disabled.
GPIO_7: Disabled.
GPIO_8: Disabled.
GPIO_9: Disabled.
GPIO_10: Disabled.
GPIO_11: key = '0x00'
GPIO_12: Disabled.
GPIO_13: Disabled.
GPIO_14: Disabled.
GPIO_15: Disabled.
/ # cat /proc/hardware
ROM Ver: 1970/1/1

[Routing Detail]
Source: 24MHz
clk range: 12M~24MHz
multiply_on: pllm
pllm_range: 100M~300MHz
pllout = xclk_in * 8/1 (n/m)
mem src = independ
pci_bridge = enabled

[Divider and Register]
cfg_fdiv: 0 cfg_hdiv: 1 cfg_pdiv: 1
cfg_ediv: 5 cfg_ddiv: 1 cfg_udiv: 3
External PCI: enabled
PCI Deskew: enabled (20M~100MHz)
PCI Drive: 4mA
ccr: 3a95c848
ccr3: 00000111

[Clock Detail]
XCLK_IN: 24000 kHz
PLL_OUT: 192000 kHz
CPU clock: 192000 kHz
MEM clock: 96000 kHz
PCI clock: 96000 kHz
EPCI clock: 32000 kHz
DEV clock: 96000 kHz
USB clock: 48000 kHz

[Device Clock Detail]
NAND: 0 kHz
I2C: 24000 kHz
AC97: 96000 kHz
GPIO: 48000 kHz
IDE1: 0 kHz
SD: 0 kHz
GPIO2: 96000 kHz
UART: 48000 kHz
MS: 0 kHz
JMP4: 96000 kHz
SPI: 12000 kHz
AES: 96000 kHz
NOR: 48000 kHz
/ # cat /proc/pl_grab
Grabbed Frames : 9405
Output Frames : 9405
Dropped Frames : 0
Grabbed fps : 28.67
Output fps : 28.67
Time Stamp Unit : 90000
Last Frame Squence : 9404
Buffer Allocated Size : 524288
Buffer Actual Size : 460800
Buffer Number : 3
Overflow Times : 1
Free List Empty Times : 33
Ready List Empty Times : 0
Free List Size : 1
Ready List Size : 0
Grabbing Index : 1
Frame Interval : 3000
Is grabbing : 1
Grab Mode : Streaming
Using Catched Frame : Off
Preallocated space : 2097152
Used preallocation : 1388544
FG_CNFG : 0x000F0049
FG_STRIDE : 0x00A000A0
FG_HCROP : 0x01400000
FG_VCROP : 0x01E00000


Via @bswatson on Twitter:

@synack Can you make any recommendations for devs interested in getting in early at a startup?

Right now, the demand for high quality engineers who are willing and able to work for a startup is very strong… Not a day goes by that I don’t see tweets, emails, and random conversations that start with “Anybody know a good X engineer?” where X is the latest must-have technology. However, sometimes it is neither simple or prudent to approach strangers and shove your resume in their face. Here are few things you can do to make your search for an exciting work environment a bit easier.

Don’t trust recruiters

Specifically, recruiting firms and contract recruiters that have no loyalty or stake in the company they’re recruiting for. I’m not saying that these people don’t represent amazing companies, but I’ve found that their motivations are mainly focused on getting the commission.

Example: A firm in San Francisco asked me to come in for an interview. After keeping me waiting for over an hour, they spent five minutes reading my resume, then showed me two potential positions that I was completely unqualified for and uninterested in. I didn’t hear from them again until several months later, when the very same recruiter had no prior knowledge of me or the positions I had already rejected. I’ve had very similar experiences with firms like this at least three times.

Bottom line: This process turned out to be a massive waste of time that could’ve been spent chasing down job leads that I had already vetted.

Meetups are great

“They say it’s all about who you know, and I know crackle!”

-Mitch Hedberg
Startup and tech events, like the regularly scheduled GitHub meetups, are potentially the best place to meet founders, employees, investors, and just plain cool people. With an open mind and some booze money, you can learn an amazing amount about what local startups are working on and what sorts of problems they’re encountering. Stick around long enough, and somebody’s sure to say something like “We’re looking for a good JavaScript/iPhone/Android guru, know anybody?” Even if you don’t, remember who’s looking for what. If you can help out with another startup’s recruiting, that karma can go a long way in your own job search.

Social media is your friend

Yes, it’s a buzzword, but that doesn’t make it irrelevant. Networks like Twitter and Facebook have a ton of power in finding startups that are doing cool things. Figure out who the thought leaders in your areas of expertise are, follow them, and maybe even start a conversation. Keep an eye on who and what they’re talking about… Sooner or later, you’ll see a new company or idea pop up. Find out if there’s an IRC channel and get friendly with it’s residents. If nothing else, you’ll probably learn something.

Write code

This one seems to trip a lot of people up. If you’re looking for a job as a coder, write some code and open source it! I think Eric S. Raymond put it best,

“Every good work of software starts by scratching a developer’s personal itch.”

That is, write something that helps you. A database to organize your music library, a driver for some obscure piece of hardware, a parser for freely available weather data, etc. Find something that’s interesting to you and do it. Don’t be afraid to experiment, this is one of the only places where you can write anything you want, any way you want. Use bleeding edge tools, libraries, and services and get familiar with them. Eventually, you’ll meet somebody that uses the same combination of tools and needs a developer to maintain them.

Do your own thing

If you’re having trouble finding a company that feels like a good fit for you, you might have the curse of the entrepreneur. If you’re one of these poor souls with the insatiable urge to work long, thankless hours to make your idea fly, I bow to you. Most of the above still applies in this case, but rather than poking and prodding people for a job, you’ll be asking for their opinion on your idea. Build a prototype as fast as possible and show it to some people. Listen to them. Iterate. Share it with more people. If it’s a good idea, it’ll snowball into something amazing. If not, pray that you grasp the concept of failure before you lose all of your hair. Lather, rinse, repeat.

Redis is a fast data structures server. You store every piece of data under a unique key. Each key has a type associated with it: string, list, hash, set, or sorted set. Redis provides primitive commands for manipulating the data stored in a given key. For example, to push a new item onto the head of a list, the command you would use is "LPUSH <key> <value>". While these primitives are relatively intuitive and powerful, it is considered bad form to sprinkle direct calls to your database throughout your application. Traditional SQL databases have been using object relational mappers (ORMs) for quite some time. An ORM is a library that allows the programmer to interact with objects that represent data in the database, rather than writing raw queries. This paradigm provides a clean layer of abstraction to the programmer, allowing changes to the underlying storage infrastructure without having to completely rewrite orre-factorthe rest of the application. In this example, I'll demonstrate abstractions for the Set and List Redis types, leaving the others as an exercise to the reader. Let's start with a class called Model, that will be the base for both our SetModel and ListModel

import redis

class Model(object):
    db = redis.Redis()

    def __init__(self, name):
        self.name = name
        self.key = '%s:%s' % (self.__class__.__name__, name)

A static variable "db" is created to store the connection to the database. In a more complete setup, this should be broken out into a separate function or module that configures and maintains a pool of connections to Redis that can be reconnected on demand but in this simple example, stick with the naive implementation. The constructor sets a variable called self.key that will be used to identify this instance's data in Redis. We set self.key to be equal to "Model:foo" if the name were "foo" using introspection of the instance's class. As you subclass Model, the key will change to the name of the subclass, keeping the model types isolated and clearly defined.

    def delete(self):
        self.db.delete(self.key)

    def __str__(self):
        return self.key

    def __repr__(self):
        return '%s(\'%s\')' % (self.__class__.__name__, self.name)

Next, add a delete method to the Model class. In Redis, all keys are deleted the same way, regardless of type, so just go ahead and implement this method on Model directly. The __str__ and __repr__ methods tell the Python interpreter how to represent Models as strings. These methods are not strictly required, but make debugging much easier later on.

class SetModel(Model):
    def __init__(self, name):
        Model.__init__(self, name)
        self.setkey = self.key + ':set'

    def add(self, obj):
        self.db.sadd(self.setkey, obj.key)

    def remove(self, obj):
        self.db.srem(self.setkey, obj.key)

    def __len__(self):
        return self.db.scard(self.setkey)

This is our first implementation of a SetModel. We start by appending ":set" to the key so that it consists of "SetModel:foo:set" for an instance named "foo". This is critical later on, as we build concrete Models that have multiple types. Sets are fairly simple data structures with only two methods for manipulating data: add and remove. These methods each take a single argument of another instance subclassed from Model. If you're worried about malformed input, you need to do some type checking with isinstance here. For now, let's assume you never make mistakes. The __len__ method provides a nice, native-looking way of getting the number of items in the set. This looks great, but is missing a way to retrieve the data. Herein lies a problem; suppose you've instantiated the set, called add() with a few other Model instances and now you want to get a list of those objects back, but the add method didn't serialize any objects to Redis, it just stored references to their keys. We need to convert a list of keys into a list of instances. Define this method at the top of the module, outside any classes.

import inspect
import sys

def objectify(keys):
    '''
    Transform a list of keys into a list of object instances
    '''
    # Get all the model types in this module
    types = dict(inspect.getmembers(sys.modules[__name__], inspect.isclass))

    # Split the keys into typename, name
    keys = [x.split(':', 1) for x in keys]

    # Lookup and instantiate each object
    objects = [types[typename](name) for typename, name in keys]
    return objects

As you can see, there are three steps in this dance. First, we enumerate a dict of all classes in this module so that we end up with {'Model': , ...}, etc. Next we split the keys into a typename and name so that SetModel:foo becomes typename='SetModel', name='foo'. Finally, we retrieve the class reference for each typename from our list of classes and instantiate it with the given name. Let's put this bad boy to work.

    def __iter__(self):
        return objectify(self.db.smembers(self.setkey))

By adding an __iter__ method to SetModel, you can now iterate over a list of objects in the set, dynamically generating new Model instances representing the keys in the set.

    @classmethod
    def intersect(cls, sets):
        return objectify(cls.db.sinter([x.setkey for x in sets]))

In addition to simple manipulation and querying, Redis also supports some more interesting set operations, such as intersection. The intersection of two or more sets is a set containing all of the elements common to both sets. Intersection doesn't operate on any single set specifically, so it is defined as a classmethod of SetModel.

import json

class DictModel(Model):
    def __init__(self, name):
        Model.__init__(self, name)
        self.dictkey = self.key + ':dict'

    def __setitem__(self, key, value):
        self.db.hset(self.dictkey, key, json.dumps(value))

    def __delitem__(self, key):
        self.db.hdel(self.dictkey, key)

    def __getitem__(self, key):
        value = self.db.hget(self.dictkey, key)
        if value is None:
            raise KeyError('Key "%s" does not exist in %s' % (key, self.dictkey))
        else:
            return json.loads(value)

Here's our simple implementation of a DictModel (or HashModel if you prefer) that implements the __setitem__, __delitem__, and __getitem__ methods of Python's dictionary interface. The values that get stored in Redis are actually serialized to JSON so that any arbitrary data structure consisting of serializable objects can be stored safely and transparently. Any serialization mechanism would work here, including pickle, yaml, or even xmlrpclib.dumps! The redis-py library returns None if the given key does not exist, so you must handle that case by raising an exception with a descriptive message, of course.

    def keys(self):
        return self.db.hkeys(self.dictkey, key)

    def __len__(self):
        return self.db.hlen(self.dictkey)

    def items(self):
        items = self.db.hgetall(self.dictkey).items()
        items = [(k, json.loads(v)) for k, v in items]
        return items

Redis hashes also support a number of different commands for manipulating and retrieving stored data, and this implementation can be fleshed out further. It is worth noting that class would be an excellent candidate for UserDict.DictMixin, which adds a complete implementation of the Python dictionary interface by manipulating the output of just a few basic methods. However, in my testing I found that DictMixin can be extremely inefficient when certain methods are not implemented. For example, DictMixin.__len__ actually calls len(self.keys()). While this works, it causes a large amount of unnecessary data to be transferred between the application and Redis. In short, it's better to provide your own implementation of these methods that take advantage of Redis' built-in commands to the fullest extent. Now that you have a DictModel and SetModel, you're finally ready to start constructing an application. Let's say you want to store Servers consisting of a dict of attributes and Tags that consist of both attributes and a set of Servers. In a traditional database, this would represent a foreign key relationship.

class Server(DictModel):
    pass

class Tag(DictModel, SetModel):
    def __init__(self, name):
        DictModel.__init__(self, name)
        SetModel.__init__(self, name)

Let's create some Servers and Tags to demonstrate how we'd use them.

>>> prod = Tag('production')
>>> dev = Tag('development')
>>> web = Tag('webserver')
>>> s1 = Server('flippy')
>>> s1['ip'] = '192.168.1.1'
>>> s2 = Server('flappy')
>>> s2['ip'] = '192.168.1.2'
>>> s3 = Server('floopy')
>>> s3['ip'] = '192.168.1.3'
>>> prod.add(s1)
>>> prod.add(s2)
>>> dev.add(s3)
>>> map(web.add, [s1, s2, s3])
[None, None, None]
>>> list(prod)
[Server('flippy'), Server('flappy')]
>>> list(web)
[Server('flippy'), Server('flappy'), Server('floopy')]
>>> Tag.intersect([prod, web])
[Server('flippy'), Server('flappy')]

Great! Now we can relate objects to other objects... But what if you want to go the other way, and get a list of Tags associated with a Server? Back in the Model class, we need to add a few methods and make delete a bit more complicated.

    def add_reference(self, obj):
        self.db.sadd('%s:refs' % self.key, obj.key)

    def delete_reference(self, obj):
        self.db.srem('%s:refs' % self.key, obj.key)

    def references(self):
        return objectify(self.db.smembers('%s:refs' % self.key))

    def delete(self):
        self.db.delete(self.key)

        keys = self.db.smembers('%s:refs' % self.key)
        refs = objectify(keys)
        for obj in refs:
            if hasattr(obj, '_dereference'):
                obj._dereference(self)

SetModel needs to track object references, so add calls to add_reference and delete_reference and a method called dereference.

    def add(self, obj):
        self.db.sadd(self.setkey, obj.key)
        obj.add_reference(self)

    def remove(self, obj):
        self.db.srem(self.setkey, obj.key)
        obj.delete_reference(self)

    def dereference(self, obj):
        self.remove(obj)

Now you should have relationships going both to and from sets. Let's try it out.

>>> s1.references()
[Tag('production'), Tag('webserver')]
>>> s3.references()
[Tag('development'), Tag('webserver')]

This is far from being a complete ORM solution, but it provides a framework for building interesting and complicated data models on top of Redis without tearing your hair out trying to figure out how to draw a diagram of a relationship between a set key and a hash key. Download orm.py Discussion on Hacker News.