After seeing Hack-a-Day’s post about a cheap IP webcam that runs Linux I couldn’t resist the urge to dig into yet another embedded system. I don’t really have a definite need for something like this, but thought it sounded like a neat piece of hardware to hack around with, so why not?

When I first plugged the camera in, I didn’t see it grab an IP over DHCP and went digging for the Manual (PDF). It boots up as 192.168.2.3 with a webserver on port 80. Default username and password: admin/1234.

After changing the IP to a more reasonable subnet, it functions as a nifty little IP webcam providing either a MJPEG or MPEG4 stream over RTSP. Not bad for $40.

Cracking the case open was a bit difficult, until I realized there was a screw hidden under the label on the back.

Time for hacking! I’ve found at least two potential ways of getting a shell on this thing, with several more theoretically possible. The easiest approach is the one that Hack-a-Day mentioned: wire up a serial console to some empty pads on the board and rock on. There are four pads on the back side of the device (the side without the camera sensor). In the picture below, from left to right: 3.3v, RX, TX, GND. Please excuse my horrible soldering job.

For the sake of simplicity, I connected this header up to my Bus Pirate and configured the port as 115200 8e1.


Bus Pirate v3a
Firmware v5.10 (r559) Bootloader v4.1
DEVID:0x0447 REVID:0x3043 (24FJ64GA002 B5)

(1)>2
Ready
UART>(1)
UART bridge
Reset to exit
Are you sure? y


Assuming everything’s wired up correctly, you should now be able to watch the board boot up and interact with it on the console.

Jan 1 00:00:eth0: Promiscuous mode enabled.
br0: port 1(eth0) entering learning state
eth0: Promiscuous mode enabled.
15 spbr0: port 1(eth0) entering forwarding state
br0: topology change detected, propagating
ook[78]: pl1029-video: quality: 12
Accepted a new connection from spook... (video)
Jan 1 00:0br0: port 1(eth0) entering disabled state
0:15 spook[78]: pl1029-video: capture_loop
=============== reinit br0: port 1(eth0) entering learning state
libbrbr0: port 1(eth0) entering forwarding state
br0: topology change detected, propagating
idge ================ ...
Jan 1 00Inconsistent of vop_type (i-vop)!!
:00:15 spook[78]: pl1029-video: set_running
Jan 1 00:00:17 spook[78]: waiting for VOS header...
Set gateway (change the routing table)
add 255.255.255.255/255.255.255.255 to routing table for broadcast
encode VOS header...
enet
00 00 01 B0 08 00 00 01 B5 89 13 00 00 01 00 00 00 01 20 00 C4 88 80 0F 51 40 43 C1 46 3F 00 00
Jan 1 00:00:17 spook[78]: pl1029-video: set_running
Terminated
generate new spook.conf... (MPEG4)
send mpeg4 data to spook pipe failed. aborted..
Waiting for a connection from spook... (video)
Inconsistent of vop_type (i-vop)!!
Jan 1 00:00:18 spook[86]: listening on control socket /tmp/spook.sock
Jan 1 00:00:18 spook[86]: listening on tcp port 554
Jan 1 00:00:18 spook[86]: pl1029-video: start_block
Jan 1 00:00:18 spook[86]: pl1029-video: set_device
Jan 1 00:00:18 spook[86]: pl1029-video: set_framesize
Jan 1 00:00:18 spook[86]: pl1029-video: set_bitrate
Jan 1 00:00:18 spook[86]: pl1029-video: set_format
Jan 1 00:00:18 spook[86]: pl1029-video: set_framerate_num
Jan 1 00:00:18 spook[86]: pl1029-video: set_output
Jan 1 00:00:18 spook[86]: pl1029-video: end_block
Jan 1 00:00:18 spook[86]: pl1029-video: device: /tmp/mpeg4Stream
Jan 1 00:00:18 spook[86]: pl1029-video: format: 100
Jan 1 00:00:18 spook[86]: pl1029-video: bitrate: 2048
Jan 1 00:00:18 spook[86]: pl1029-video: width: 640
Jan 1 00:00:18 spook[86]: pl1029-video: height: 480
Jan 1 00:00:18 spook[86]: pl1029-video: fps: 30
Jan 1 00:00:18 spook[86]: pl1029-video: gopsize: 30
Jan 1 00:00:18 spook[86]: pl1029-video: quality: 12
Accepted a new connection from spook... (video)
Jan 1 00:00:18 spook[86]: pl1029-video: set_running
Jan 1 00:00:18 spook[86]: pl1029-video: capture_loop
Jan 1 00:00:18 spook[86]: waiting for VOS header...
Inconsistent of vop_type (i-vop)!!
encode VOS header...
enet
00 00 01 B0 08 00 00 01 B5 89 13 00 00 01 00 00 00 01 20 00 C4 88 80 0F 51 40 43 C1 46 3F 00 00
Jan 1 00:00:19 spook[86]: pl1029-video: set_running
Inconsistent of vop_type (i-vop)!!
enet link_status thread started PID:81
eth0: Promiscuous mode enabled.
Network initial status: WLAN
WLAN was disabled! link status still was LAN
/ #


At this point, you can poke around a bit. The system has a relatively minimal busybox config installed with two processes making up the webcam interface. “/bin/spook” and “/sbin/enet”. Taking a step back, the web server sends a Server header in it’s responses identifying it as a GoAhead web server. This is a small web server designed to be embedded into another process, in this case, spook. The GoAhead web server has a long list of vulnerabilities attributed to it, including a couple good remote buffer overflows. For someone so inclined, this might provide an alternate method of hacking the RXS-3211 without modifying the hardware.

Getting back to the software itself, I noticed some strange behavior if I attempted to use any program to communicate with the network other than spook. After examining dmesg and thinking carefully about how such a low-powered device can provide streaming video at 30fps, I can make an educated guess about how the system works. When the snoop process starts, it maps a shared memory section with the enet process. snoop instructs the hardware to capture frames, encode them as either MPEG4 or MJPEG and write the result into the shared memory area. The enet process then instructs the ethernet hardware to copy the data out of shared memory and send it out to the network. Assuming that this hypothesis is correct, then doing *anything* that attempts to send data to the network while the enet process is copying data directly to the ethernet chip’s buffers will cause the whole network stack to lock up and break all network connectivity. This makes it exceedingly difficult to copy files off the device or interact with it over the network in anything other than the prescribed manner.

Faced with this setback, I started looking toward the bootloader. Astute readers will have noticed that the bootloader pauses for a couple seconds at startup, prompting for Ctrl-C on the serial console. Doing so drops you into a “proboot>” prompt.


proboot> credit
We fight for the honor of our company.
Prolific bootloader - Proboot - Professional bootloader
by Jun Chen, Hugo Leu, Jedy Wei, CC Yen 2003-2006


It seems to be a fairly sane bootloader with some nice options like loading and dumping images over tftp. Unfortunately, proboot does not recognize the PL1023′s network hardware and expects a Realtek 8139 to be connected. I suspect that this is what the extra set of pads on the I/O board are for: an RTL8139 chip and RJ-45 jack for interacting with the bootloader. What’s left on the board is likely a relic of the development cycle.

Faced with spotty network access in userspace and no viable network interface in the bootloader, I decided to dump out the firmware image to the serial console using the “memdump” command. memdump takes a memory address and length and reads the whole thing out in the common hex/ascii dump format. While this format is intended more for human consumption than machines, I was able to turn it into a binary dump by parsing my terminal emulator’s output log with a few Python scripts.

Based on the default boot script, I was able to deduce the locations of the kernel and three sections of the NOR flash, one of which appears to be the root filesystem… I assume the other two flash sections store non-volatile configuration that is preserved between reboots.

Now that I have access to the firmware, I plan to modify the /etc/rc.d/rcS script in the rootfs to not start spook so that I’ll have a chance to interact with the device over the network unhindered. Eventually, I’d like to build an entirely new rootfs image with some sort of support for writing to flash, currently the system is read-only.

Other observations:

• The reset button does nothing
• There are configuration and firmware files for at least three different wireless chipsets, implying that the same software may be running on other IP webcams with better hardware.
• Without networked access to the bootloader, I’ll have to figure out what sort of “Firmware upgrade” image the webserver expects to be uploaded. Rosewill has not released any upgrade image for this device, so I don’t have a whole lot to go on.


/ # cat /proc/cpuinfo
Processor : Faraday FA526id(wb) rev 1 (v4l)
BogoMIPS : 147.56
Features : swp half

[Device Clock Detail]
NAND: 0 kHz
I2C: 24000 kHz
AC97: 96000 kHz
GPIO: 48000 kHz
IDE1: 0 kHz
SD: 0 kHz
GPIO2: 96000 kHz
UART: 48000 kHz
MS: 0 kHz
JMP4: 96000 kHz
SPI: 12000 kHz
AES: 96000 kHz
NOR: 48000 kHz
/ # cat /proc/pl_grab
Grabbed Frames : 9405
Output Frames : 9405
Dropped Frames : 0
Grabbed fps : 28.67
Output fps : 28.67
Time Stamp Unit : 90000
Last Frame Squence : 9404
Buffer Allocated Size : 524288
Buffer Actual Size : 460800
Buffer Number : 3
Overflow Times : 1
Free List Empty Times : 33
Ready List Empty Times : 0
Free List Size : 1
Ready List Size : 0
Grabbing Index : 1
Frame Interval : 3000
Is grabbing : 1
Grab Mode : Streaming
Using Catched Frame : Off
Preallocated space : 2097152
Used preallocation : 1388544
FG_CNFG : 0x000F0049
FG_STRIDE : 0x00A000A0
FG_HCROP : 0x01400000
FG_VCROP : 0x01E00000


Via @bswatson on Twitter:

@synack Can you make any recommendations for devs interested in getting in early at a startup?

Right now, the demand for high quality engineers who are willing and able to work for a startup is very strong… Not a day goes by that I don’t see tweets, emails, and random conversations that start with “Anybody know a good X engineer?” where X is the latest must-have technology. However, sometimes it is neither simple or prudent to approach strangers and shove your resume in their face. Here are few things you can do to make your search for an exciting work environment a bit easier.

Don’t trust recruiters

Specifically, recruiting firms and contract recruiters that have no loyalty or stake in the company they’re recruiting for. I’m not saying that these people don’t represent amazing companies, but I’ve found that their motivations are mainly focused on getting the commission.

Example: A firm in San Francisco asked me to come in for an interview. After keeping me waiting for over an hour, they spent five minutes reading my resume, then showed me two potential positions that I was completely unqualified for and uninterested in. I didn’t hear from them again until several months later, when the very same recruiter had no prior knowledge of me or the positions I had already rejected. I’ve had very similar experiences with firms like this at least three times.

Bottom line: This process turned out to be a massive waste of time that could’ve been spent chasing down job leads that I had already vetted.

Meetups are great

“They say it’s all about who you know, and I know crackle!”

-Mitch Hedberg
Startup and tech events, like the regularly scheduled GitHub meetups, are potentially the best place to meet founders, employees, investors, and just plain cool people. With an open mind and some booze money, you can learn an amazing amount about what local startups are working on and what sorts of problems they’re encountering. Stick around long enough, and somebody’s sure to say something like “We’re looking for a good JavaScript/iPhone/Android guru, know anybody?” Even if you don’t, remember who’s looking for what. If you can help out with another startup’s recruiting, that karma can go a long way in your own job search.

Social media is your friend

Yes, it’s a buzzword, but that doesn’t make it irrelevant. Networks like Twitter and Facebook have a ton of power in finding startups that are doing cool things. Figure out who the thought leaders in your areas of expertise are, follow them, and maybe even start a conversation. Keep an eye on who and what they’re talking about… Sooner or later, you’ll see a new company or idea pop up. Find out if there’s an IRC channel and get friendly with it’s residents. If nothing else, you’ll probably learn something.

Write code

This one seems to trip a lot of people up. If you’re looking for a job as a coder, write some code and open source it! I think Eric S. Raymond put it best,

“Every good work of software starts by scratching a developer’s personal itch.”

That is, write something that helps you. A database to organize your music library, a driver for some obscure piece of hardware, a parser for freely available weather data, etc. Find something that’s interesting to you and do it. Don’t be afraid to experiment, this is one of the only places where you can write anything you want, any way you want. Use bleeding edge tools, libraries, and services and get familiar with them. Eventually, you’ll meet somebody that uses the same combination of tools and needs a developer to maintain them.

Do your own thing

If you’re having trouble finding a company that feels like a good fit for you, you might have the curse of the entrepreneur. If you’re one of these poor souls with the insatiable urge to work long, thankless hours to make your idea fly, I bow to you. Most of the above still applies in this case, but rather than poking and prodding people for a job, you’ll be asking for their opinion on your idea. Build a prototype as fast as possible and show it to some people. Listen to them. Iterate. Share it with more people. If it’s a good idea, it’ll snowball into something amazing. If not, pray that you grasp the concept of failure before you lose all of your hair. Lather, rinse, repeat.

It seems like there are a lot of people confused about how to make correct use of the BSD sockets interface without having data disappear in the middle of a session. For building servers, you can make use of the SocketServer module to hide some of the annoyances, but for clients you've either got to deal with sockets or use a third party library, which never ends up being as fun as it sounds.

We'll start with a simple example... Open a socket, send a string, then close it.

from socket import socket

sock = socket()
sock.connect(('1.2.3.4', 1234))
sock.send('Hello world!\n')
sock.close()

This looks like a pretty simple example, but it exemplifies the pitfalls of sockets programming fairly well. The first problem that becomes apparent is that while the socket() constructor defaults to AF_INET, SOCK_STREAM (IPv4 TCP socket) arguments now, this may not be true in the future. The most likely change would be AF_INET6 for IPv6 support. While this call probably won't fail on an IPv6 native system, the IP address passed to the connect method would end up being in the wrong format. The simple solution is to pass DNS names rather than IP addresses and ensure that you catch any exceptions that might arise from nonexistant names.

This leads to the next problem: exception handling. The socket constructor could throw an exception if the system encounters an "out of FDs" condition. While this is fairly uncommon, you'll want to catch it to make your code more robust. The connect method could throw an exception for a number of reasons. If the IP or hostname you pass in cannot be parsed or resolved, or if all available source ports have been exhausted (extremely unlikely, requires at least 65536 established connections). The send method could fail if the connection has been closed or interrupted in some manner, although it's more likely that it simply wouldn't send data. Finally, the close method will only fail if the socket has already been closed. In our example above, there's no other way the socket object could be closed, so we really don't need to worry much about that. Additionally, this line may even be extraneous because Python closes sockets upon garbage collection. But it's best to include it for correctness.

Sending data can be a bit tricky, especially if you're trying to send a lot of it. The socket's send() method is not guaranteed to send all of the data you pass it. Instead, it returns the number of bytes that were actually sent and expects your application to handle retransmission of the unsent portion. However, Python provides a convenience method called sendall() that makes sure all of your data is sent before returning. The right way:

sock.sendall('Hello world\n')

If, for some reason, you need to know exactly what bytes were sent, even under an error condition, you can also do it the old-fashioned way:

buffer = 'Hello world\n'
while buffer:
	bytes = sock.send(buffer)
	buffer = buffer[bytes:]

This approach can be a bit inefficient memory-wise, but will allow you to access the unsent portion of buffer if sock.send throws an exception.

Time to try receiving some data! There are many ways to handle incoming data, depending on your application's needs. For now, I'll illustrate how you might handle a common line-based protocol.

def readlines(sock, recv_buffer=4096, delim='\n'):
	buffer = ''
	data = True
	while data:
		data = sock.recv(recv_buffer)
		buffer += data

		while buffer.find(delim) != -1:
			line, buffer = buffer.split('\n', 1)
			yield line
	return

This method will continuously call recv() until the connection is closed, and supply a generator interface to the caller so that implementing your protocol becomes as simple as:

for line in readlines(sock):
	# Do something with a line of data

The socket object also provides a makefile() method that returns a file object based on the socket's file descriptor, allowing implementations like this:

for line in sock.makefile('r'):
	# Do something with a line of data

The makefile approach is likely more efficient and will perform better, but may not be as compatible on multiple platforms and does not give you as much control over the recv buffer size and delimiter.

Unfortunately, recv() is a blocking call. If you'd like your program to continue doing other things while waiting for data, you have several choices: call setblocking(False) on your socket, causing recv to throw an exception if there's no data available, or use the select module to occasionally poll the socket for data, or make use of platform-specific calls like epoll and kqueue, or just handle sockets in a separate thread or process. It all depends on what your application needs. Let's take a look at a simple select() call, as most of the other methods are variants of this.

from select import select
from socket import socket

sock = socket()
sock.connect(('1.2.3.4', 1234))

while True:
	readable, writable, exceptional = select([sock], [], [], timeout=0)
	if readable:
		data = sock.recv(4096)
		# Handle the incoming data
	# Continue other tasks here

In this case, we're only using select to check the status of a single socket, but as you can see, it supports polling lists of sockets for readable, writable, and exception states. The return value is a three item tuple, containing any sockets that were found to be in the given states. Once the data is cleared from a socket, you can continue to perform other tasks in your application, eventually looping around to the select() call to poll for more data. This is an extremely efficient way to handle a large number of connections and use often used in implementing servers in order to avoid having to spawn new processes or threads for each connection, further consuming system resources.

I hope this post has cleared up some of the confusion around sockets programming. If you have questions or comments, don't be afraid to let me know.

I've been working on some scripts that interact with SNMP devices and after much research and frustration settled on the fact that all of the SNMP libraries available in Python can do everything. The NetSNMP bindings are incomplete, the PySNMP API changes every few months, and both are poorly documented. For these reasons, I started using Scapy to create and parse SNMP packets.

It's not terribly difficult to get some useful code with scapy and if you take a close look, the scapy source often provides concrete examples of what you're trying to do. For the purposes of this article, let's try writing an SNMP walk script.

from scapy import *

def walk(ip, community, oid_prefix):
    nextoid = oid_prefix
	while nextoid.startswith(oid_prefix):
		p = IP(dst=ip)/UDP(sport=RandShort())/SNMP(community=community, PDU=SNMPnext(varbindlist=[SNMPvarbind(oid=nextoid)]))
		r = sr1(p)
		oid = r[SNMPvarbind].oid.val
		if oid.startswith(oid_prefix):
			yield (oid, r[SNMPvarbind].value)
		else:
			break
		nextoid = oid

This is mostly just a re-implementation of scapy's builtin snmpwalk() method, except that it doesn't traverse the tree infinitely. For the most part, this code works. However, it requires root privileges to run because scapy uses raw sockets to send packets and pcap to receive them. Clearly this is not ideal.

The simplest workaround I've found (with help from Jordan) is to create a regular socket object in python, then use scapy to generate and parse the payloads.

from scapy import *
from socket import socket, AF_INET, SOCK_DGRAM

def walk(ip, community, oid_prefix):
	sock = socket(AF_INET, SOCK_DGRAM)
	sock.connect((ip, 161))

	nextoid = oid_prefix
	while nextoid.startswith(oid_prefix):
		p = SNMP(community=community, PDU=SNMPnext(varbindlist=[SNMPvarbind(oid=nextoid)]))
		buf = str(p)
		while buf:
			bytes = sock.send(buf)
			buf = buf[bytes:]

		r = SNMP(sock.recv(4096))
		oid = r[SNMPvarbind].oid.val
		if oid.startswith(oid_prefix):
			yield (oid, r[SNMPvarbind].value)
		else:
			break

By using the standard BSD sockets approach, you can avoid the requirement to have root privileges by giving up control of the IP and transport layers to the OS. This should work equally well for SOCK_STREAM (TCP) and SOCK_DGRAM (UDP) socket types. This approach appears to be much faster than scapy's native pcap based method, because pcap isn't parsing every packet on the interface to find the responses.

Apple's recently released HTTP Live Streaming provides a clear, concise method for streaming audio and video content to browsers and mobile devices without the need for proprietary formats or browser plugins (I'm looking at you, Adobe). Instead, Apple's spec relies upon already common and relatively open formats.

The reference for implementing this functionality is Apple's HTTP Live Streaming Overview. Please read this document before continuing. I'll wait.

Right, so the jist is that you need to serve an m3u8 playlist containing metadata and a pointer to a URL containing a valid MPEG2 transport stream. As my preferred language is Python and the general convention for creating web services in Python is to use WSGI (PEP 333) we'll start with a basic WSGI application.

from flup.server.fcgi import WSGIServer
from threading import Thread
from socket import socket
from select import select
from Queue import Queue
import re

class LiveHTTPServer(object):
	def __init__(self):
		self.urls = [
			('^/stream.m3u8$', self.playlist),
			('^/stream.ts$', self.stream),
		]
		self.urls = [(re.compile(pattern), func) for pattern, func in urls]
		self.queues = []

	def __call__(self, environ, start_response):
		for pattern, func in self.urls:
			match = pattern.match(environ['PATH_INFO'])
			if match:
				return func(start_response, match)
		start_response('404 Not Found', [('Content-type', 'text/plain')])
		return ['404 Not Found']

So, there's not too much magic here. When the server object is instantiated, it compiles a list of regular expressions and maps them to instance methods. The WSGI server will call __call__ which attempts to match each of the regexes on the path of the request, calling the associated method if matched or sending a 404 response if not. Note that Ian Bicking's WebOb library is a much simpler way to perform a lot of these tasks, but doesn't provide an easy way to send chunked responses, which are necessary for the MPEG2 transport stream. Just ignore all the extra imports for now, we'll get to them in a minute.

	def playlist(self, start_response, match):
		start_response('200 OK', [('Content-type', 'application/x-mpegURL')])
		return ['''#EXTM3U
#EXTINF:10,
http://video.example.org/stream.ts
#EXT-X-ENDLIST''']

This method implementes the /stream.m3u8 response as required my Apple's spec. The M3U standard says that the EXTINF attribute should have a value of -1 for ongoing streams or those of unknown length, but the iPhone rejected the playlist given anything other than a positive integer in this field.

	def stream(self, start_response, match):
		start_response('200 OK', [('Content-type', 'video/MP2T')])
		q = Queue()
		self.queues.append(q)
		while True:
			try:
				yield q.get()
			except:
				if q in self.queues:
					self.queues.remove(q)
				return

This is where the tricky part actually happens. We create a Queue that will be filled with the MPEG2 data from another thread and start blocking on it, passing the data as a chunked response as soon as it's available. If anything goes wrong (eg. client disconnect) then we remove this stream's queue from the list and return. If this server were to handle a large number of clients, we might want to set a max queue size to avoid filling up memory with data destined for a slow or unresponsive client. It might also be useful to perform some locking on the queues list, to avoid contention between threads. I'll leave that as an exercise for the reader.

def input_loop(app):
	sock = socket()
	sock.bind(('', 9999))
	sock.listen(1)
	while True:
		print 'Waiting for input stream'
		sd, addr = sock.accept()
		print 'Accepted input stream from', addr
		data = True
		while data:
			readable = select([sd], [], [], 0.1)[0]
			for s in readable:
				data = s.recv(1024)
				if not data:
					break
				for q in app.queues:
					q.put(data)
		print 'Lost input stream from', addr

This method serves as the feeder for all of the client queues. It listens for a single connection on port 9999 and puts any received data into all available client queues. If the feeder stream is lost, it will go back to waiting for a new connection.

if __name__ == '__main__':
	app = LiveHTTPServer()
	server = WSGIServer(app, bindAddress=('', 9998))

	t1 = Thread(target=input_loop, args=[app])
	t1.setDaemon(True)
	t1.start()

	server.run()

Finally we tie it all together by instantiating the WSGI application and server and starting a separate thread for the input loop. The flup.server.fcgi.WSGIServer class will act as a FastCGI server that can act as a backend for any number of web servers. If you'd rather not use FastCGI, you should be able to drop in any other WSGI server as long as it supports multiple concurrent requests, otherwise any client after the first will just block waiting for the transport stream.

For my application, I used gstreamer to connect to the input socket and provide an MPEG2 transport stream. This is trivial to do using gst-launch assuming you've got the proper plugins installed.

gst-launch alsasrc device=hw:0,4 ! ffenc_libmp3lame ! ffmux_mpegts ! tcpclientsink host=video.example.org port=9999