Conrad Kramer

Enhancing Git on OS X

Fri, 18 Jan 2013 08:23:31 -0500

Git is a distributed version control system that has grown to be wildly popular among developers. It was created for use in managing the Linux kernel, and has since expanded to all walks of software development. It has become standard in iOS and OS X development, for use as dependency management, if nothing else. Git is extremely popular in other places, as well. Regardless of what git is used for, there are a number of enhancements that can be made to improve its usability. I outlined a few of them, tailored for OS X, in this article.

Installing Git

There are many ways to install git on OS X, but I am only going to discuss one of them in this blog post, using the Homebrew package manager for OS X. Installing git through Homebrew downloads the latest stable release of git, compiles it, and installs it to the ‘cellar’, usually located in /usr/local/. The advantage in using a package manager like Homebrew or MacPorts to install git is that your machine is always running the latest version of it.

Assuming you have Homebrew installed, git can be installed with one command:

brew install git

Color

Git can be configured to color its output so that it is easier to read and understand. For example, if colors are enabled, changes staged for commit are colored in green in the output of git status, whereas those that are not are colored in red. This allows the user to easily and quickly distinguish between the two. Color is not enabled by default, but can be enabled with one command:

git config --global color.ui true

Bash Completion

When using git in a shell, typing out long commands can often be a hassle. Installing bash completion for git will save you a lot of time, allowing you to tab complete all of your commands. It is very easy to set up, especially if you are using Homebrew. You only need to install the bash-completion package:

brew install bash-completion

and then add the following to your ~/.bash_profile file:

if [ -f $(brew --prefix)/etc/bash_completion ]; then
  . $(brew --prefix)/etc/bash_completion
fi

and voila! Git should now tab-complete all of your commands.

In addition to bash completion, aliases can also be defined to shorten the time required to enter long commands. The only problem with aliases is that they can become embedded in muscle memory, and are only local to your own machine. This means that if you login into an arbitrary server, or use your friends computer, the aliases will not be there to help you. It may only cause frustration, but you also may forget the underlying command behind the alias. Regardless, adding aliases is as simple as one command. For example, to alias checkout to co, you can enter:

git config --global alias.co checkout

Prompt

Git also comes with a handy utility to add information about the current repository to your bash prompt. Having git status in the prompt provides a heads up display of whether the current directory is under version control or not, and can also provide basic information, like which branch is currently checked out. This feature can be easily set up, assuming you installed git and bash-completion with Homebrew, by adding the following to your ~/.bash_profile file:

source $(brew --prefix)/etc/bash_completion.d/git-prompt.sh

The __git_ps1 function is now available in your shell. You can then use it in your prompt like the example below, which adds the current branch to the default prompt in OS X:

PS1="\h:\W \u\$(__git_ps1 \" (%s) \")\$"

Github Integration

It you happen to use Github for hosting your git repositories (and enhancing your git workflow), there is a convenient tool that can integrate some of Github’s functionality into your command line. The tool is called Hub. One feature it adds is short-hand notation for repositories, assuming that the repository is hosted on github. For example, it can shorten

git clone git://github.com/conradev/QuickQR.git

git clone conradev/QuickQR

In addition to short-hand, Hub also adds Github specific functionality to your command line, like the ability to open pull requests. If you are working on a topic branch, and want to open a pull request, you need only type

git pull-request

and you can type up a pull request in your text editor. Installing hub is also a breeze. To install it with Homebrew, you can type

brew install hub

This installs the ‘hub’ command to your PATH. It acts as an intermediary layer on top of git, meaning it uses git to execute the commands you enter, after preprocessing them a bit. To integrate hub seamlessly (as seen in the examples above), you can alias git to hub, by adding the following to your ~/.bash_profile:

eval "$(hub alias -s)"

Graphical Components

When working with git, graphical components in one’s workflow can be of great aid in doing specific tasks. If you have had to conduct an extensive merge manually, for example, you know that it can become a nightmare. Git was created with this limitation in mind. Git allows for external diff and merge tools to act as front-ends to aid in those operations. The guys at Black Pixel created an awesome OS X application for comparing and merging files, named Kaleidoscope. With a couple clicks, it will install a command line tool and add itself to your git configuration as a diff and merge tool. To conduct a complex merge, you can use git mergetool as opposed to git merge, and Kaleidoscope will be opened for resolving any conflicts. There are also tools out there that allow you to interact with a git repository graphically, one of the most popular on OS X being Git Tower. Graphical managers allow you to commit code, manage branches and do anything you want with the click of your mouse.

I personally do not use Git Tower, as I prefer git on the command line. I prefer it because I like to know exactly what I am executing, when I execute it, and because shell is portable. This is why I like Kaleidoscope as a diff/merge tool, because it is invoked from the command line only when needed. Regardless, deciding whether or not to use a graphical manager is a matter of personal preference.

Get Coding!

The hope in making git more usable is that it transforms from a necesary and annoying task into one that is efficient and enjoyable. This way, you can spend more time coding, and less time dealing with version control. So, get coding!

Feel free to follow me on Twitter or check out my software. Additionally, if your company is hiring mobile dev interns this upcoming summer, and has no inhibitions hiring a high school student, be sure to get in touch.

A Quicker QR Code Scanner

Sun, 06 Jan 2013 02:41:00 -0500

QR codes have become very popular in the advertising industry. They are slapped onto flyers, business cards and many other marketing materials with the hope that consumers will scan them. QR codes sound like a great idea in writing, but they are faced with many problems being adopted in the real world. One of these problems is the neglect of QR codes by smartphone manufacturers. There are a plethora of applications available for each given smartphone to scan QR codes, but the fact that they are third party detracts from their value. They don’t come pre-installed, involve hassle to activate, and each one is different, causing fragmentation. Countless times have I seen a QR code only to be deterred by the fact that I must first unlock my iPhone, navigate to a page of rarely used applications, open a QR code scanner, and wait for the camera to initialize. If smartphone manufacturers adopted QR codes and integrated scanning functionality into their software, all of these problems would be solved. In Japan, for example, QR codes are immensely popular and almost all phones have QR code scanning functionality. I don’t know which one caused the other, but it does seem as if those two things go hand in hand. In the US, the only smartphone (that I know of) that has QR code scanning functionality built in is Windows Phone, with Bing Vision search. Google also has a visual search platform, Google Goggles, with apps on iOS and Android, but the functionality has not yet been integrated into Android itself.

Interestingly enough, Apple has adopted 2D barcodes (including QR codes) in its new Passbook service in iOS 6. By doing this, Apple is expressing confidence that 2D barcodes will continue to be relevant for a long time to come. It would only make sense for them to integrate QR code scanning into iOS. As some have suggested, this could be done by making the camera application a bit smarter, to automatically look out for QR codes. That article does get one thing wrong, though, in saying that the idea is “probably not as simple to implement as it seems”. As it turns out, the idea is simple to implement, at least on iOS.

Analyzing The Camera

In order to integrate QR code scanning functionality into the camera, I first needed to understand how the camera works. I did this by reverse engineering Apple’s software. I discussed popular tools that I use to reverse engineer software in my previous blog post. The framework that controls basic camera interactions on iOS is the private framework PhotoLibrary. The framework includes the view controllers that display the Camera Roll and also the controller that is responsible for taking photos, PLCameraController. These controllers are not only used in the Camera application, but are also used to back the public UIImagePickerController class. The PhotoLibrary framework is based on the public AVFoundation framework. Here is the interface for PLCameraController, abridged to only show the parts we care about:

@interface PLCameraController : NSObject {
    AVCaptureDeviceInput *_avCaptureInputFront;
    AVCaptureDeviceInput *_avCaptureInputBack;
    AVCaptureDeviceInput *_avCaptureInputAudio;
    AVCaptureStillImageOutput *_avCaptureOutputPhoto;
    AVCaptureMovieFileOutput *_avCaptureOutputVideo;
    AVCaptureVideoDataOutput *_avCaptureOutputPanorama;
}

+ (instancetype)sharedInstance;

@property(weak, nonatomic) AVCaptureOutput *currentOutput;
@property(weak, nonatomic) AVCaptureDeviceInput *currentInput;
@property(readonly, strong, nonatomic) AVCaptureSession *currentSession;
@property(readonly, strong, nonatomic) AVCaptureVideoPreviewLayer *previewLayer;

@end

As you can see, the basic structure of the camera controller is an AVCaptureSession with a number of inputs and outputs. The microphone, front and back cameras function as inputs, and there are outputs for taking photos, videos, and panorama shots. When I first looked into the PhotoLibrary framework before iOS 6 was released, these observations led me to discover the then unreleased Panorama mode.

Adding QR Code Scanning

Once I knew the basic structure of the PhotoLibrary framework, I was able to work on integrating a QR code scanner into it. The library I used to read QR codes was ZXing, pronounced “zebra crossing”. It is a popular open source library for interpreting 2D barcodes, originally written in Java, and then ported to C++. There are also Objective-C wrappers available in the source distribution (two of them, in fact), but I found both heavy-handed for my purposes. In addition, both Objective-C wrappers process image data on the main thread, which I found to cause noticeable interface lag. I chose instead to write a small class (100 lines) that implements the functionality I need. The class I wrote takes data from a AVCaptureVideoDataOutput and processes it with the ZXing C++ library on a private queue.

Thanks to Apple’s straightforward media capture API in AVFoundation, it is easy to add arbitrary outputs to a capture session. All I needed to do was create an instance of my class, and add the video data output to the session in the camera controller, and just like that, I was recognizing QR codes. The implementation was just as straightforward as the idea. I made something to read QR codes, and quite simply added it into the Camera. Don’t believe me? Check out the source code! I also made a video, demonstrating the tweak in action:

If you are interested in tweaking iOS, be sure to check out my previous blog post for a more in-depth discussion. Feel free to follow me on Twitter or check out my software. Additionally, if your company is hiring mobile dev interns this upcoming summer, and has no inhibitions hiring a high school student, be sure to get in touch.

Discuss on Hacker News here

Integrating Google Cloud Print into iOS

Mon, 17 Dec 2012 09:00:00 -0500

Setting up my family’s Chromebox was easy, until someone needed to print something. Google decided that in Chrome OS, the only method for printing should be through its Cloud Print service. Google Cloud Print is a service that provides a API for printing, both to send print jobs and to receive them. This is an example of forward thinking by Google, and greatly simplifies the experience in Chrome OS. However, this adds complexity for the consumer. I ended up configuring the printer in my house using an open source bridge running on a Linux server. Once I configured it properly, I began to really appreciate the advantages of Cloud Print. I enjoyed being able to print documents to my physical printer, and PDF files to my Google Drive, from anywhere in the world. Being able to print directly from Gmail is also awesome. However, I missed these luxuries on the device that I use the most, my iPhone.

Beginning with iOS 4.2, Apple introduced AirPrint as a method to print over the local network. Since then, Apple’s core applications, and many others, have added print functionality. I wanted to be able to print natively on iOS, but using Google’s Cloud Print service. The problem is, as Marco Arment writes, Apple is in the business of saying “no”. No to customizing the interface, and no to modifying the operating system. This is what I consider the biggest advantage of jailbreaking. Jailbreaking gives you the ability to sidestep Apple, and say “yes”. Not only this, but developing software for jailbroken devices, in the form of “tweaks”, proves to be a fun challenge. This is an account of how I integrated Google Cloud Print into Apple’s existing AirPrint functionality.

Building The Client

The first step to integrate Google Cloud Print into iOS was to implement a client. Google documents the Cloud Print API on their website, so the only thing that I needed to do was write a client for it. There are many ways to approach this, but I wanted to try something I have not tried before. I decided to delve into Core Data with Mattt Thompson’s AFIncrementalStore. AFIncrementalStore is a relatively new project, with the ambitious goal of mapping Core Data to a RESTful web service. I created a data model with Job and Printer entities, and implemented an AFRESTClient subclass to interact with the API. AFIncrementalStore is intended to be transparent, so I need only execute standard Core Data calls to use it. To fetch a list of all printers in alphabetical order, I can execute a standard NSFetchRequest:

NSFetchRequest *fetchRequest = [NSFetchRequest fetchRequestWithEntityName:@"Printer"];
fetchRequest.sortDescriptors = @[ [NSSortDescriptor sortDescriptorWithKey:@"name" ascending:YES] ];
NSArray *printers = [context executeFetchRequest:request error:nil];

To send a job to a given printer, I can insert a new Job entity into the context, and a request will be fired off in the background, transparently. Here is an example of sending a job to a printer:

CPJob *job = [NSEntityDescription insertNewObjectForEntityForName:@"Job" inManagedObjectContext:context];
job.title = @"TestJob";
job.printer = printer;
job.fileData = [NSData dataWithContentsOfFile:filePath];
job.contentType = @"application/pdf”;
[context save:nil];

It is pretty cool to not have to interact with the network layer to use a RESTful API. For my purposes, I have found this approach to be great, but Core Data is not for everything or everyone. It is general knowledge among iPhone developers that Core Data is difficult to get right, and that when things do go wrong, they go horribly wrong. In this project, there are relatively few objects in the store, and the relationships between them are simple. I have yet to come across a serious problem (knocks on wood).

In addition to interacting with the Cloud Print API, the client also needed to authenticate using OAuth2. There are many implementations of OAuth2 on iOS, but I was attracted to AFOAuth2Client in particular, because it is based on AFNetworking, just as AFIncrementalStore is. However, AFOAuth2Client in its current state is terribly outdated and feature incomplete. This led me to give it an overhaul to better conform to the OAuth2 spec, and to add some features. One feature I added was the ability to easily store tokens in the keychain for safe storage.

Diving Into PrintKit

In order to integrate Google Cloud Print into iOS, it was integral to understand how Apple approaches the problem of printing on iOS. All of Apple’s magic lies in a private framework on iOS named PrintKit.framework. To understand this magic, I conducted static and dynamic analysis. For disassembly and static analysis, I have a copy of IDA Starter, which is considered to be the standard in static analysis. I also use class-dump, which parses the Objective-C metadata in a binary, and uses the data to generate readable class definitions. For dynamic analysis, I use Jay Freeman’s cycript, pronounced “sscript”, a hybrid between JavaScript and Objective-C. The interpeter can inject into a running process and provide a shell to do just about anything you want.

From analyzing PrintKit, I determined a few useful pieces of information. PrintKit has two components, the framework itself and its associated daemon. The first component, the framework, is relatively small with only six classes. Each class is a wrapper over a portion of the CUPS API, which is compiled into the framework. You can see this for yourself by looking at the symbols in the framework (you must run this on OS X with the iOS SDK installed in order for it to work):

conradev@air :: ~ » nm $(xcode-select --print-path)/Platforms/iPhoneOS.platform/Developer/SDKs/iPhoneOS6.0.sdk/System/Library/PrivateFrameworks/PrintKit.framework/PrintKit | grep -c cups 
328

The second component, the daemon, is known as printd. It is configured with launchd to listen over a UNIX socket. printd is actually a modified version of the CUPS daemon. This can be seen from its usage page (you must run this on a jailbroken iPhone in order for it to work):

iPhone:~ root# /System/Library/PrivateFrameworks/PrintKit.framework/printd -h
CUPS v1.5.0
Copyright 2008-2010 by Apple Inc.  All rights reserved.
...

The framework and daemon communicate over the UNIX socket the same way they would normally be communicating over port 631. The daemon is started on demand, meaning it is only running when it needs to be.

However, in integrating Google Cloud Print, the daemon is of little importance. The proper layer to inject into and modify is the Objective-C layer, the six classes that serve as the API to the framework. Of these six, three are of particular interest: PKPrinterBrowser, PKPrinter and PKJob. The first, PKPrinterBrowser, functions as its name suggests. It browses for printers using Bonjour. Here is the header for PKPrinterBrowser, abridged to only show the parts we might care about:

@protocol PKPrinterBrowserDelegate
-(void)addPrinter:(PKPrinter *)printer moreComing:(BOOL)coming;
-(void)removePrinter:(PKPrinter *)printer moreGoing:(BOOL)going;
@end

@interface PKPrinterBrowser : NSObject

@property (strong, nonatomic) NSMutableDictionary* printers;

+ (id)browserWithDelegate:(id<PKPrinterBrowserDelegate>)delegate;
- (id)initWithDelegate:(id<PKPrinterBrowserDelegate>)delegate;

@end

Each browser has a printers property, which contains the printers, and a delegate property, which wants to be notified of any changes. A browser object is created when you browse for nearby printers in the AirPrint dialog. The remaining two classes, PKPrinter and PKJob are model objects describing printers and jobs, respectively. Additionally, PKPrinter has methods for submitting jobs.

Putting It All Together

After implementing the Cloud Print client and thoroughly investigating PrintKit, I was now able to work on integrating the two. This was the most challenging part, but also the most fun. This is the point where the code entered the bounds prohibited by Apple. The code written prior, the API client, is independent and can be used anywhere. For example, I wrote an application using the API client for debugging purposes.

For those unfamiliar with jailbreaking, the piece of software that enables one to cleanly modify existing software is CydiaSubstrate. CydiaSubstrate is similar to SIMBL (which dates back to 2003) in principle, but has a more robust implementation. On a jailbroken iOS device, CydiaSubstrate is loaded into every process spawned by launchd. Once loaded into a process, CydiaSubstrate checks a directory for user-installed libraries, called tweaks, and selectively loads them into the process. Once loaded into the process, these libraries can then modify existing functionality.

As I mentioned earlier, the Objective-C layer of PrintKit is the best place to make modifications. It is low enough of a level where I would not need to modify any interface code, but high enough of a level so as to avoid the CUPS API. One of the goals is to display Google Cloud Print printers to the user. To do this, I can insert custom printer objects into PrintKit that represent printers in Google Cloud Print. PKPrinter objects are normally created by PKPrinterBrowser, so this seems like the logical place to insert them. Other objects see which printers PKPrinterBrowser has found by checking its printers property. Thus, if I was to modify the getter of this property, other objects would see my modifications. The advantage of only modifying the getter is that the modifications do not interfere with the internal state of the object, as the instance variable containing the printers is not modified. Here is some code from the project doing just this:

%hook PKPrinterBrowser

- (NSMutableDictionary *)printers {
    NSMutableDictionary *orig = %orig();

    NSMutableDictionary *combined = [NSMutableDictionary dictionaryWithDictionary:orig];
    [combined addEntriesFromDictionary:thePrinters]; // Printers from somewhere

    return combined;
}

%end

Now, if you are familiar with Objective-C, you are probably wondering about some of the funky syntax above: %hook, %orig(), and %end. The code above is written for something called Logos. Logos is a preprocessor, written in Perl, developed by Dustin Howett. What Logos does is it takes the code above, and translates it into Objective-C runtime calls to “hook” that method. It replaces the implementation of the method with a custom one, and makes the original implementation available with the %orig() macro. Here is the above example, ran through Logos, and made human readable:

static NSMutableDictionary * (*originalImplementation)(PKPrinterBrowser*, SEL);
static NSMutableDictionary *replacedImplementation(PKPrinterBrowser* self, SEL _cmd) {
    NSMutableDictionary *orig = originalImplementation(self, _cmd);

    NSMutableDictionary *combined = [NSMutableDictionary dictionaryWithDictionary:orig];
    [combined addEntriesFromDictionary:thePrinters]; // Printers from somewhere

    return combined;
}
static __attribute__((constructor)) void initializeHooks() {
    Class _class = objc_getClass("PKPrinterBrowser");
    Method _method = class_getInstanceMethod(_class, @selector(printers));

    IMP superImplementation = class_getMethodImplementation(class_getSuperclass(_class), @selector(printers));    

    if (_method) {
        originalImplementation = superImplementation;
        if (!class_addMethod(_class, @selector(printers), (IMP)&replacedImplementation, method_getTypeEncoding(_method))) {
            originalImplementation = method_setImplementation(_method, (IMP)&replacedImplementation);
        }
    }
}

As you can see, the first example is orders of magnitude cleaner and easier to maintain than the second. This is why Logos is so incredibly useful for writing hooks. It makes writing hooks almost natural, and removes the need to muck around in the runtime manually and repetitively. This approach of directly replacing implementations is somewhat similar to method ‘swizzling’ that many Cocoa developers are familiar with. One of the key differences between swizzling and directly replacing the implementation is that swizzling involves adding a method to a class via a category. This can only be done when the target class is directly linkable, which is not always the case. Mike Ash wrote a great article outlining both of these methods.

But back to adding printers. Now that I am able to insert custom printer objects, I need to get these printer objects from the cloud to instances of PKPrinterBrowser. This is where the API client comes in. I can simply insert the API client into my tweak, and use it to fetch the printer objects. However, it is not that simple. There are many problems with doing this. The first is that a client would be created for every single application that invokes a print dialog. This is horribly inefficient. The second is that the copies of AFIncrementalStore and AFNetworking within my tweak would clash with the copies in any process that also uses these libraries, and lead to undefined behavior. This is very unstable. Both of these problems can be solved by moving the client out of the tweak, and into its own daemon. Within a daemon, nothing would clash, and there would only be a single client running at once.

The tweak could then communicate with the daemon, and ask it for printer objects. Beginning with OS X Lion and iOS 5, Apple added the XPC Services API. XPC stands for cross process communication, and it is designed to facilitate communication between daemons, or services, and other code. Beginning with OS X Mountain Lion and iOS 6, Apple added the NSXPCConnection API, which is an Objective-C wrapper built on top of the C API. On iOS, both XPC APIs are currently private, but they are exactly the same as on OS X. This means that the documentation is also the same.

I went ahead and added the API client to a daemon. I further modified the PKPrinterBrowser class so that each instance opens a connection to the daemon and retrieves printer objects from it. I chose to use the NSXPCConnection API for convenience. Sadly, this forces me to drop iOS 5 support in my project. One thing I had to do in order to get this approach to work was convert the printer objects from NSManagedObject subclasses into PKPrinter subclasses, so they were compatible with PKPrinterBrowser and independent of the Core Data store. Here is a screenshot of the whole thing in action, working:

I also wrote a preference bundle that allows a user to authenticate using OAuth2 from within the iOS Settings application. The entire project is built using Theos, which is a makefile based build system written by Dustin Howett that eliminates many of the headaches that come with using makefiles. Theos was made expressly for the purpose of compiling projects for jailbroken iOS devices, and has many conveniences. Theos includes the ability to automagically preproccess files using Logos. It also includes support for building Debian packages, because APT is the package manager used on all jailbroken iOS devices.

So What?

While the tweak does not support sending print jobs yet, I have made the project open source under the MIT license for those who want to look through the code, play around with it, or even use parts of it. I do intend to eventually finish it and release it. I set out to integrate Google Cloud Print and iOS in the best way possible and the resulting project came out great. On top of this, I learned a whole lot in the process of building it.

I wanted to write this post to demonstrate a few things. First, that by jailbreaking, the possibilities are endless. Not as a user, but as a developer. Nothing in the userland is off limits, and everything is modifiable. In very few places in the mobile ecosystem is this offered. Android certainly has more freedom with their SDK, but there are still certain things off limits. Even when rooted, Java is a static language, making modification of other software difficult. The same goes for almost every other mobile operating system. The only exception I know of is WebOS because it is open by default, and a lot of it is written in JavaScript. Modification via patches is easy.

The second point I wanted to illustrate is that developing software outside of the sandbox makes you a better iOS developer. I am not saying that those who develop outside of the sandbox are better than those who develop within. Software written for the sandbox is certainly more important, as it impacts a much larger market. What I am saying is that developing software outside of the sandbox is a constant challenge, and you will certainly learn a few things by trying it. I am a strong believer in the idea that you must take something apart to understand how it works. A great way of doing this is by disecting iOS and writing software to co-exist with it. Developing software for jailbroken devices transformed me from an Xcode kiddy into what I am today. I now have a thorough understanding of the Objective-C runtime, reverse engineering, dynamic libraries, et cetera.

Lastly, developing tweaks is just plain fun. I have seen many people exercise their design talent once in a while and create an iOS concept video. While it looks great on video, how awesome would it be to actually create it and try it out? Tweaks do not necessarily have to be big, either. They can be as small as adding seconds to the lockscreen clock. Here are some great examples of small and simple tweaks. Developing for jailbroken devices can be a fun and challenging hobby, so I encourage other iOS developers to jump right in and give it a try!

Feel free to follow me on Twitter or check out my software. Additionally, if your company is hiring mobile dev interns this upcoming summer, and has no problem hiring a high school student, be sure to get in touch.

Discuss on Hacker News here.

Vim, LaTeX and High School

Sat, 06 Oct 2012 20:33:00 -0400

This September, after a summer full of programming, I was forced back into my junior year in high school. Back to writing English essays, lab reports, and projects. Luckily, sometime over the summer I discovered LaTeX. LaTeX has many advantages over a proprietary WYSIWYG word proccesor like Microsoft Word or Apple’s Pages. Its free, it’s portable, and it’s friendly with mathematical and scientific notation out of the box. However, It also has its share of disadvantages. One of these disadvantages is the steep learning curve. Unlike the WYSIWYG editors, it’s not straightforward to set up or use. This is why I am writing this blog post, to share my experiences and help bootstrap someone eager to get started with LaTeX.

The Setup

Installing and setting up LaTeX on your machine varies, depending on what operating system you are on. The LaTeX project website lists some good TeX distributions available for download. I own a MacBook running OS X, so I have downloaded and installed MacTeX. It is a big download (>1GB), but it is comprehensive, including everything you need to get started.

To build TeX files into PDFs, I have found Rubber to be the easiest and most comprehensive solution out there. It abstracts away all the tasks involved with building a TeX document, and works on both Linux and OS X without problem. It is available for download and install in many package managers, including Homebrew on OS X.

If you use Vim, I highly recommend the TeX-PDF plugin. As the name implies, the plugin’s sole purpose is to build TeX into finished PDFs, using whatever is available. It will use Rubber if it finds it on the system. The plugin is well suited for high school projects, because building to PDFs is probably the only thing you need to do. Using Vundle, installation is as simple as adding Bundle "TeX-PDF” to your .vimrc file.

Getting Started

There are many resources available to those starting out with LaTeX. I have chosen to learn as I go, and figure out how to do something when I need to do it. However, I have learned some tips and tricks that would be particulary useful for fellow high school students.

Science

In writing lab reports for science classes, I have found the packages mhchem and siunitx to be very useful. Both of these are included in the TeX Live and MacTeX distributions. The first package, mhchem, makes it very easy to write out chemical formulae. It is extremely comprehensive, and supports all the notation I have needed for AP Chemistry thus far. To write the formula for dichromate, it simplifies

$3\,\mathrm{Cr}_2^{\strut}\mathrm{O}_7^{2-}$

into

\ce{3Cr2O7^2-}

to draw

The other package, siunitx, allows you to typeset physical quantities with ease. For example, you can express the specific heat capacity of water with

\SI{4.18}{\joule\per\gram\per\celsius}

to draw

siunitx supports (allmost) every SI unit under the sun.

English

Writing English essays in LaTeX is very easy to do, thanks to the mla-paper package. It takes care of the MLA header in one line, which is very useful.

and the code:

\documentclass[12pt]{article}
\pagestyle{plain}

\usepackage{fullpage}
\usepackage{ifpdf}
\usepackage{mla}

\begin{document}
\begin{mla}{First}{Last}{Teacher}{Class}{Date}{Title}

One of the great American classics, \emph{The Great Gatsby} by F. Scott Fitzgerald \ldots

\end{mla}
\end{document}

Plotting

In all of my science classes, I have had to plot points on a graph, in one form or another. Yes, there is a package for that. Based on the graphics package pgf, pgfplots can be used to create high quality plots of all kinds. Here is a recent example of a project I did in AP Calculus, graphing a logistic function:

This example has four plots overlaid on top of one another. A scatter plot, the logistic regression, and its first and second derivatives. This example also includes labeled nodes, and a legend. pgfplots, like many of the other packages, is extremely comprehensive and extensible. Here is the code:

\documentclass[12pt]{article}
\pagestyle{plain}

\usepackage{pgfplots}

\begin{document}

\centering

\begin{tikzpicture}
\begin{axis}[
  height=10cm,
  width=\textwidth,
  axis y line=left,
  axis x line=middle,
  title=Spread of the Virus,  
  xlabel=Round (Day),
  ylabel=Number of People Infected]

% Points
\addplot[only marks, forget plot, blue] table[x=Day,y=People]{
Day People
1   1
2   2
3   4
4   8
5   14
6   19
7   24
8   28
9   30
10  30
11  32
};

% Regression
\addplot[smooth,blue,mark=none, domain=0:11,samples=40]{ 31.808 / (1 + (60.9 * e^(-0.756 * x))) };
\addplot[smooth,red,mark=none, domain=0:11,samples=40]{ (1464.45 * e^(0.756 * x))/(60.9+e^(0.756 * x))^2 };
\addplot[smooth,green,mark=none, domain=0:11,samples=40]{ (e^(-0.756 * x) * (67424 * e^(1.512 * x) - 1107.13 * e^(2.268 * x)))/(60.9+e^(0.756 * x))^3 };

% Important points
\node[pin=135:{\shortstack{Inflection Point \\ $(5.435,15.905)$}}] at (axis cs:5.435,15.905) {};
\node[pin=45:{\shortstack{Relative Maximum \\ $(5.435,6.012)$}}] at (axis cs:5.435,6.012) {};
\node[pin=15:{\shortstack{Zero \\ $(5.435,0)$}}] at (axis cs:5.435, 0) {};

% Legend
\addlegendentry{ $f(x)$ }
\addlegendentry{ $f'(x)$ }
\addlegendentry{ $f''(x)$ }

\end{axis}
\end{tikzpicture}

\end{document}

Conclusion

Deciding to switch to using only LaTeX for my schoolwork this year was a fantastic decision. Although I had to learn as I went, often under time constraints, it has certainly paid off. I can now edit my schoolwork with Vim. Mathematical and scientific notation is easy to write, and not a hassle anymore. I don’t have to rely on a proprietary word processor. I plan on taking this skill with me when I eventually head off to college.

Discuss on HN here.

Deploy a website using git in Ubuntu

Mon, 04 Jun 2012 22:41:00 -0400

This is a follow up to my previous post on how to set up a Flask web app with nginx and uWSGI. This post will run down how to set up a git-based deployment system for that web app. Git is pretty popular for deployment, and is used in services like Heroku. Heroku is great, but I prefer to have full control over my server, and roll my own stuff, all while maintaining the same ease of use.

Repository Setup

To continue from my previous example, the website is deployed in /srv/www/helloworld. This tutorial should work with pretty much any website, but it will still focus on Flask based ones.

First, if you have not already done so, install git.

sudo apt-get install git

Next, put your website under version control with git

cd /srv/www/helloworld
git init
git add application.py
git commit -m 'First commit'

A gitignore file is also a good idea. For a Flask app, put the following in .gitignore

env
*.pyc

and to add it to the repository

git add .gitignore
git commit -m 'Added .gitignore file'

Now we are going to create a bare ‘hub’ repository that will act as an intermediary between the active deployment folder and the rest of the world.

sudo mkdir -p /srv/git
sudo chown -R $USER:$GROUP /srv/git
cd /srv/git
git clone --bare /srv/www/helloworld
cd helloworld.git
git remote rm origin

Next, add the hub as a remote for the deployment repository

cd /srv/www/helloworld
git remote add hub /srv/git/helloworld.git

Add the magic

To automatically update the deployment repository when changes are pushed to the hub, we are going to use git hooks. Git hooks are scripts that are executed after certain changes occur within the repository.

To begin, add the following to /srv/git/helloworld.git/hooks/post-update

#!/bin/sh
echo
echo "Pulling changes into deployment repository"
echo
git --git-dir /srv/www/helloworld/.git --work-tree /srv/www/helloworld pull hub master

and make sure that it is executable

chmod 755 /srv/git/helloworld.git/hooks/post-update

This post-update script updates the deployment repository when the hub is updated. Next, we are going to setup the reverse, a hook to update the hub when changes are committed in the deployment directory (this should be rare, but just in case). Add the following to /srv/www/helloworld/.git/hooks/post-commit

#!/bin/sh
echo
echo "Pushing changes to hub"
echo
git push hub

and add make it executable

chmod 755 /srv/www/helloworld/.git/hooks/post-commit

Further Modification

If you went ahead and tested the deployment system, you’d find that the changes aren’t reflected in your browser yet. uWSGI is still using the old compiled python files, and did not notice the changes to the directory. To get uWSGI to notice the changes, you have to reload it every time you push changes. A reload command can be added to the git hook, but it isn’t as easy as it sounds. You need root privileges to reload uWSGI, and when you push changes to the server, you are most likely logging in as your own personal user. The solution to this problem is to create a setuid binary that reloads uWSGI.

First, install the Ubuntu build-essential package

sudo apt-get install build-essential

Next, put the following in ~/reload_uwsgi.c

#include <stdio.h>
#include <sys/types.h>
#include <unistd.h>
#include <stdlib.h>

int main() {
    setuid(0);
    system("/sbin/initctl reload uwsgi");
    return 0;
}

Now to compile, install and configure the binary

cd ~
sudo gcc reload_uwsgi.c -o /usr/local/bin/reload_uwsgi
rm reload_uwsgi.c
sudo chown root:root /usr/local/bin/reload_uwsgi
sudo chmod 4755 /usr/local/bin/reload_uwsgi

And lastly add this executable to our git hooks, which should look like the following

/srv/git/helloworld.git/hooks/post-update:

#!/bin/sh
echo
echo "Pulling changes into deployment repository"
echo
git --git-dir /srv/www/helloworld/.git --work-tree /srv/www/helloworld pull hub master
/usr/local/bin/reload_uwsgi

/srv/www/helloworld/.git/hooks/post-commit:

#!/bin/sh
echo
echo "Pushing changes to hub"
echo
git push hub
/usr/local/bin/reload_uwsgi

Testing it out

Clone the repository on your local machine

git clone user@server:/srv/git/helloworld.git

Make some changes, commit them to master, and push them to the server

vim application.py
git commit -am 'Made some changes!'
git push origin master

and voila! The change should be reflected on the website.

Discuss on Hacker News here

Getting a Flask website up and running in Ubuntu

Sun, 06 May 2012 19:00:00 -0400

This is a guide to get a Flask website up and running on Ubuntu 12.04 LTS using nginx and uWSGI. There are many routes to take when it comes to Python on the web; this just is my personal favorite. Some people enjoy configuring servers, while others view it as a chore. Regardless, this guide should get you up, running, and ready to make something awesome in no time!

Installation

nginx

To install nginx you first need to add the repository. Add the following to /etc/apt/sources.list.d/nginx-lucid.list :

deb http://nginx.org/packages/ubuntu/ lucid nginx
deb-src http://nginx.org/packages/ubuntu/ lucid nginx

You will also want to add the gpg key to the apt keyring:

wget http://nginx.org/keys/nginx_signing.key
sudo apt-key add nginx_signing.key
rm nginx_signing.key

Finally, to install nginx, run:

apt-get update
apt-get install nginx

uWSGI

You can use pip to install the latest version of uWSGI by doing the following:

sudo apt-get install python-dev build-essential python-pip
sudo pip install uwsgi

To configure uWSGI to run as a daemon, you first want to create a separate uwsgi user:

sudo useradd -c 'uwsgi user,,,' -g nginx -d /nonexistent -s /bin/false uwsgi

You then want to create an upstart configuration file, to run uWSGI in the background. Add the following to /etc/init/uwsgi.conf :

description "uWSGI"
start on runlevel [2345]
stop on runlevel [06]

respawn

exec uwsgi --master --processes 4 --die-on-term --uid uwsgi --gid nginx --socket /tmp/uwsgi.sock --chmod-socket 660 --no-site --vhost --logto /var/log/uwsgi.log

To set up logging, you can add a logrotate configuration file at /etc/logrotate.d/uwsgi:

/var/log/uwsgi.log {
    rotate 10
    daily
    compress
    missingok
    create 640 uwsgi adm
    postrotate
        initctl restart uwsgi >/dev/null 2>&1
    endscript
}

and to prime the log file, you can run:

sudo touch /var/log/uwsgi.log
sudo logrotate -f /etc/logrotate.d/uwsgi

It is okay if there is an error with the postrotate script. uWSGI cannot be restarted because it is not currently running.

virtualenv

Using virtualenv is a good idea because it compartmentalizes the environment of your application. To install virtualenv, you can use pip:

sudo pip install virtualenv

Configuration

Flask

Now to set up the website! In this case, the website is going to be called ‘helloworld’ and it is going to be set up in /srv/www/helloworld. Feel free to change it, just adjust these instructions accordingly.

The first step is to create the directory:

sudo mkdir -p /srv/www/helloworld

and a subdirectory for static files, to be hosted by nginx:

cd /srv/www/helloworld
mkdir static

The next step is to create a virtual environment for the application to run in:

virtualenv ./env

and to install Flask in that environment:

source env/bin/activate
pip install Flask
deactivate

Now that that is done, let’s create a sample ‘Hello World’ application, in application.py:

from flask import Flask
app = Flask(__name__)

@app.route('/')
def hello_world():
    return 'Hello World!'

if __name__ == '__main__':
    app.run()

The permissions of the directory now have to be configured. uWSGI needs read permission to read the contents of the scripts, and write permission to save compiled python files:

sudo usermod -a -G nginx $USER
sudo chown -R $USER:nginx /srv/www/helloworld
sudo chmod -R g+w /srv/www/helloworld

Nginx

The final step is to configure nginx. First, remove the default configuration file:

sudo rm /etc/nginx/conf.d/default.conf

Now, add the helloworld configuration file at /etc/nginx/conf.d/helloworld.conf:

server {
    listen       80;
    server_name  localhost;

    location /static {
        alias /srv/www/helloworld/static;
    }

    location / {
        include uwsgi_params;
        uwsgi_pass unix:/tmp/uwsgi.sock;
        uwsgi_param UWSGI_PYHOME /srv/www/helloworld/env;
        uwsgi_param UWSGI_CHDIR /srv/www/helloworld;
        uwsgi_param UWSGI_MODULE application;
        uwsgi_param UWSGI_CALLABLE app;
    }

    error_page   404              /404.html;

    error_page   500 502 503 504  /50x.html;
    location = /50x.html {
        root   /usr/share/nginx/html;
    }
}

Running it

Now that you have successfully configured your website, you can run it using:

sudo service uwsgi restart
sudo service nginx restart

Extensions

Flask

Flask is fantastic because it is very extensible. Check out the documentation to get started.

More Websites!

It is really easy to add more websites to this configuration. Simply create a new application directory, set it up, and create a second nginx configuration. uWSGI is configured to be in virtualhost mode, so it can handle multiple websites at once.

Portability

You can also use uWSGI as an http server, for testing purposes. If you have a copy of your website checked out in the current directory, you can run

uwsgi --http 127.0.0.1:9090 --pyhome ./env --module application --callable app

and uWSGI will create a HTTP server hosting your application on port 9090. This is incredibly useful for development.

Discuss on Hacker News here