michael-mccracken.net

A Weblog

Flashing a Nexus Device With Ubuntu Touch From Mac OS X

| Comments

I’ve been working with Ubuntu Touch recently, using VirtualBox to run Ubuntu Saucy as a guest on my Mac, and while it does work to connect to the device once it’s installed, the process of reflashing is difficult. The manual steps on the Ubuntu Wiki Touch/Install page work for the old way, via phablet-flash cdimage-touch, with the caveat that you may have to manually reboot the device and/or re-connect the USB in between steps.

However, as of this email last week, the official way to flash phones is using the system images via phablet-flash ubuntu-system, which has a different set of files.

Instead of just muddling through and updating the manual instructions, I decided to see if phablet-flash would run natively on the Mac. After a bit of python dependency resolution, it does. Here’s what I did:

  1. Download the Android SDK tools. You only really need the “SDK Tools”, not the whole ADT Bundle, so I clicked on “Use an existing IDE” at the bottom of that page, to get the smaller installer.
  2. Unpack the archive and run ./android. Install the platform-tools (and nothing else).
  3. Add the platform-tools/ directory to your $PATH for convenience.

  4. (Optional) create a virtualenv for the python dependencies, and activate it.

  5. pip install each of configobj pyxdg, and pyliblzma
  6. get bazaar if you don’t already have it.
  7. branch phablet-tools: bzr branch lp:phablet-tools
  8. cd phablet-tools && python setup.py build
  9. At this point, phablet-flash ubuntu-system should work smoothly, with all transfers and reboots and timeouts going as expected.

The only minor hiccup is that it apparently doesn’t get ~/Downloads/ from pyxdg, but it just means that you’ll get the images saved to ~/phablet-flash instead of ~/Downloads/phablet-flash. No big deal.

Links - Simulation, Programming, Crab Cakes and Hockey Player Usage Charts

| Comments

  • DYNAMO Someone has rewritten one of the earliest simulation systems in JavaScript (the fate of all interesting software). Also includes a link to an article about the history of simulation software that sounds very interesting.
  • The Food Lab: The Crabbiest Crab Cakes I love crab cakes, but I’m not sure I really want to try to make them at home. If I do, I’ll use these tips. The Food Lab is fun stuff.
  • Debug It! A review of a book on debugging, which is a topic that I think should be taught right alongside programming. See also “Why Programs Fail”
  • GiveDirectly: introducing a radical new way to give! | GiveDirectly Send cash straight to poor people. If their assertions are true, it’s a really interesting idea, and I can’t believe it hasn’t been done before. It also seems transparently better than microloans.
  • ContinuumIO/Bokeh · GitHub Something to look out for — a Python ggplot that works with HTML5 is a great idea. “Bokeh (pronounced boh-Kay) is an implementation of Grammar of Graphics for Python, that also supports the customized rendering flexibility of Protovis and d3. Although it is a Python library, its primary output backend is HTML5 Canvas. There are many excellent plotting packages for Python, but they generally do not optimize for the particular needs of statistical plotting (easy faceting, bulk application of aesthetic and visual parameters across categorical variables, pleasing default color palettes for categorical data, etc.). The goal of Bokeh is to provide a compelling Python equivalent of ggplot in R.”
  • FitDesk X1 Level up from a standing desk? I’d love to try this for a day.
  • Concurrent Revisions DVCS-like concurrent programming. Interesting sounding research – I haven’t read it yet…
  • Many thanks to @robvollmannhl and the good folks at Hockey Abstract for these great interactive Player Usage Charts: hockeyabstract.com/playerusagecha… Player Usage Charts are fascinating, but I can never figure out why people always change the axes so that the dots fill the space. It makes it impossible to compare two charts, and it’s not obvious, so you end up comparing charts without realizing that it’s meaningless.

Mock Nested Properties With Python-mock

| Comments

Python’s mock library (part of stdlib in 3.3+) is a great tool for writing concise tests. Its documentation is very good, and rewards multiple reads – but I found one thing that wasn’t totally clear, even after looking through a few times. I wanted to use PropertyMock to mock nested Properties. Specifically, I had patched the python Gnome-introspection wrapper for libsoup at the top level Soup objcet, and I also wanted to replace one of its nested constant properties, Soup.MemoryUse.COPY with a sentinel that I controlled, for later comparison.

The idiom for PropertyMock is to assign a PropertyMock to the type object of the Mock object whose property you want control of. What I found is that because Mocks auto-create properties, it’s possible to do nested mocking in one line, like this:

1
2
3
4
import mock
m = mock.Mock()
type(m.a.b.c).d = mock.PropertyMock(return_value = mock.sentinel.my_value)
assert(m.a.b.c.d == mock.sentinel.my_value)

So my soup example looks roughly like this (mixing testing and tested code, and repeating literals for brevity):

1
2
3
4
5
6
7
8
9
10
11
12
13
14
json_body = "{}"
with patch(gi.repository.Soup) as mock_soup:
    from gi.repository import Soup
    type(mock_soup.MemoryUse).COPY = mock.PropertyMock(return_value=mock.sentinel.COPY)

    # tested code:
    message = Soup.Message.new("POST", "http://fake.com/api")
    message.set_request('application/json', Soup.MemoryUse.COPY, json_body, len(json_body))

    # checking:
    assert(mock_soup.mock_calls == [mock.call.Message.new("POST", "http://fake.com/api"),
                                    mock.call.Message.new().set_request('application/json',
                                                                        sentinel.COPY # <--- this was the point
                                                                        "{}", 2)])

It’s often possible to think of a shorter, clearer use of the mock library after revisiting a problem, but so far this still seems good. Let me know in the comments if you have a suggestion for improvements.

Mac IRC Clients

| Comments

At Canonical, we are spread all over and keep in touch via IRC. I’ve tried out a bunch of IRC clients, but nothing is quite right. I’ll explain. Here’s a list of some things I want from a client – it’s probably not complete, but it’ll do for now:

  1. Don’t crash. A bouncer makes crashes recoverable, but I shouldn’t have to use one just to keep the backlog around.

  2. I need to search the backlog. Incremental search is best.

  3. Show multiple channels at once, in one window.

  4. Highlight mentions of my nick so I can glance at a window and see if anyone’s asking me something. Use Growl or notification center, too.

  5. Make it obvious when I’ve disconnected, and reconnect robustly.

  6. Let me hide/fade out join/leave notifications and other admin messages.

  7. Keep my place so I know what messages are new.

  8. Make it obvious which channel I’m looking at, and even more obvious which channel I’m about to type in. (Nothing is great at this)

  9. Spell-check is nice, but autocorrect is awful for IRC and technical discussions. Needs a global switch.

  10. Don’t show unread counts. Some channels are chatty and I’m just in them listening for my name.

Here are the clients I’ve tried and why they aren’t quite perfect:

Colloquy

Colloquy is a nice full-featured open source client. I’ve used it the most. However – it doesn’t show multiple channels in one window at the same time, and doesn’t keep your place. I’ve had some problems with reconnection, where I thought IRC was quiet when I was actually just disconnected and didn’t know it.

I have to turn autocorrect off for each room I’m in, which I only remember after the first time it really annoys me by mangling some jargon.

There’s also some general buggy behavior, and I have a short list of bugs I’d like to report someday, but I don’t have time to fix them.

Years ago I hacked my own version of a place marker that I liked, but I can’t remember why it didn’t make it in, and of course the code is lost now.

The backlog search looks like it should be powerful, but in my experience it just doesn’t work – it never displays any results. I’m not sure what the deal is there.

(As a side note, Colloquy’s iPhone app is the only iPhone IRC client I’ve used. It seems to work fine, and I don’t use it enough to have specific gripes. Just having to have IRC open on my phone is guaranteed to annoy me more than the software itself could.)

Emacs ERC

My current setup is just to run ERC in an emacs window alongside my code editing. This has super flexible window splitting and size management and unbeatable search for your backlog, since it’s just another emacs buffer. There’s another nice benefit – if you’re writing about code, emacs’ tab-completion for strings across all buffers means that you can easily autocomplete SomeReallyLongFunctionName in chats. This saves a lot of typing and cut n’ pasting. When you do need to cut n’ paste, it’s nicer to do it between emacs buffers than between apps.

The defaults are not ideal, as usual with emacs you need to dig around to find a good setup.

There’s a module to colorize nicknames according to a hash of the name, so it’s easier to tell people apart. Other clients have similar features but I think I like emacs’ the best for some reason – maybe the palette is nicer.

There’s a module to mark your place that only works if you left the cursor in the backlog, which is a little weird. It will highlight my nick when mentioned, but not the whole line.

Since it’s open source, I have the same problem as Colloquy, it’s hackable, but I don’t really have time for that. And emacs is fun to hack on, so it’s a potentially dangerous time sink.

Linkinus

This is a nice client with some minor flaws and a little flakiness. I liked it but moved on because of minor annoyances.

It has a kind-of nifty “conversation tracking” feature that tries to highlight messages in a back-and-forth when you hover over it with the mouse (but only in the currently selected channel). It’s not really that useful, but I think it could be if pushed further. I’d like to see more clients try this kind of thing, since separating multiple concurrent conversations is hard on IRC, and it can be hard to avoid them.

Lets you display multiple channels in one window, but that needs some improvement. There’s just one text-entry field so it’s easy to type something to the wrong channel.

The split display has some issues. The split channel views don’t show you the channel name anywhere, and in some cases don’t even show you the channel topic, which may or may not be helpful anyway. So it can take some effort to figure out what channel you’re looking at, especially if there are many of the same people talking on two channels.

It can save a set of channels as a group, but it doesn’t save the relative sizes of the splits in the group, so if you want to flip between groups, you’d better be OK with even splits.

Finally, it only supports vertical splits, so with more than two channels, you just get a couple unreadably long lines per channel.

It has a very barebones backlog search, but it does work.

They’ve also clearly tried to re-design the account setup / prefs experience, but I’m not a big fan of an inspector palette – I wanted to compare settings between two servers and it will only show me one at a time. This was also the biggest point of flakiness, where some edits wouldn’t commit and I couldn’t tell if the settings were being changed.

Textual

Textual is an open-source app (github.com/Codeux/Textual) that is for sale on the Mac app store. This is an interesting approach that I think has some merit – people may still contribute code or bug fixes, but it’s also nice to know that if the app ever gets abandoned, it could be resurrected. That said, I tried building from source to enable a quick hack once but the master branch didn’t build for me, so they’re not trying to make it easy.

It lets me hide some admin messages, has automatic & manual scrollback markers to keep track of what I’ve missed.

This has a setting to “Track conversations using nickname highlighting” but I couldn’t really tell what it was doing.

This also has a pretty barebones search, but again, it works.

I tried this for a few days and liked it, but ultimately switched away, I think mostly because you can’t display multiple channels – there’s just one window and it doesn’t split up.

Mango

I haven’t tried Mango. The screenshots in the App Store and feature listings don’t lead me to believe that it’s anything significantly different from the others. I’d be glad to hear from fans, though.

So, what do you use? Have I missed anything great?

More Out of Date Links

| Comments

Links

| Comments

  • What’s so bad about Template Haskell? – Stack Overflow

    A bunch of good answers to why Template Haskell is occasionally maligned.

  • Which Haskell (GHC) extensions should users use/avoid? – Stack Overflow

    Interesting list of aesthetic judgements of haskell extensions. At press time I don’t know what most of these words mean.

  • Yesod and Template Haskell

    Short post on why Yesod uses template haskell (which has a bad rep)

  • deck.js » Modern HTML Presentations

    Best “Write your presentations in HTML” thing I’ve seen yet, although I’m not sure how great it’d be at complex visuals or builds. Yes sometimes builds are good! SOMETIMES.

  • REDbot

    RED is a robot that checks HTTP resources to see how they’ll behave, pointing out common problems and suggesting improvements. Although it is not a HTTP conformance tester, it can find a number of HTTP-related issues.

  • .@dpp’s Visi looks to be exactly the thing when Soulver isn’t enough and spreadsheets are rigid github.com/visi-lang/visi #emerginglangs

    Should I look at this before posting a link? Nah… But I will look at it, eventually! It looks interesting, I love this kind of stuff – although as usual I wonder if this is something you can already get in Mathematica but no one knows because it’s so expensive.

  • High Scalability – Prismatic Architecture – Using Machine Learning on Social Networks to Figure Out What You Should Read on the Web 

    Probably interesting? I skimmed this, I’m not gonna lie, I don’t remember much.

  • Deslidefied HTML

    Really useful bookmark to make slideshow news articles readable on one page. I am using it all the time!

  • Thinking Functionally with Haskell

    “”In which we explore what modern type systems bring to the table. Imagine an approach to programming where you write down some description of what your code should do, then before running your code you run some automatic tool to see if the code matches the description. That’s Test-driven development, you say! Actually, this is what you are doing when you use static types in most languages too. Types are a description of the code’s inputs and outputs, and the check ensures that inputs and outputs match up and are used consistently. Modern type systems—such as in Haskell or above—are very flexible, and allow these descriptions to be quite detailed; plus they are not too obtrusive in use and often very helpful. One point I’ll investigate here is how advances in types are converging with new ideas on testing, to the point where (I claim) the old distinctions are starting to blur and starting to open up exciting new possibilities—hence my suggestion that we need a new word to describe what we’re doing that is free from preconceptions and out-dated thinking. So put aside your bad experiences from Java, and prepare to be amazed! ””

  • So you want to hack on IMAP…

    “Well, first off, you’re horribly misinformed: you do not actually want to hack on IMAP.” (I agree, IMAP bad.)

  • The Machine That Changed the World: Great Brains – Waxy.org

    Documentary about computing history: “It’s a whirlwind tour of computing before the Web, with brilliant archival footage and interviews with key players — several of whom passed away since the filming. Jointly produced by WGBH Boston and the BBC, it originally aired in the UK as The Dream Machine before its U.S. premiere in January 1992. Its broadcast was accompanied by a book co-written by the documentary’s producer Jon Palfreman.”

A Testing Story

| Comments

Tests are great! This year I’ve started working on a codebase with a significant unit test suite for the first time in a while, and good coverage has definitely come in handy. Now I’m going to share a story where a single test did double duty.

Our filesystem events listening daemon was crashing occasionally on my system, and I didn’t know how to reproduce it. No one else seemed to be getting these crashes, either. The system was saving a backtrace, and it was always the same, but I wasn’t sure I believed its line numbers. As a start, I made a guess at the lines that were really failing, and added a ton of debug dumps to inspect the state. (Since this daemon is run as root using launchd, it’s still easiest to just use the old printf-n’-stare debug method.)

Then I tried a lot of stuff to poke at it, even leaving Spotify on overnight in an attempt to recreate the conditions of the bug. No luck. It seemed like it would only crash when I wasn’t trying to get it to crash – pretty frustrating.

I finally found it, but only after giving up for a while. I checked in again after working for a while on another project, and hey, lots of new crashes! With all my extra debug info, I could see what was going on – a string that couldn’t be encoded in UTF-8 was being handled by some code that assumed it could be. It was a filesystem path with invalid characters.

What was the path that was killing my daemon? It was a temp file written by the test suite for the other project. It was a non-utf8 path, written to test the unicode handling of the GUI, and it had the wonderful (in retrospect) side effect of poking a bug in the daemon too. It’s so satisfying when you find a bug’s cause and it completely explains all the symptoms you were seeing.

One test exercising the unicode handling of multiple projects, now that’s coverage!

Disposable

| Comments

Computing gadgets don’t last long enough. I wanted to write a lot about this, but since I don’t have good answers, I figured I might as well just pose some questions.

All the time, we’re throwing out something that still works, because there’s something new that works better.

Working better is great! Faster chips and better screens and cameras make our lives easier and help us make cool new stuff.

I used to work near people designing processors, so I know it’s plenty hard just to keep that part of the system progressing. And I know that newer stuff will use less energy, which is great. I know that people designing data centers think hard about their efficiency, and upgrade ruthlessly to improve it.

But how carefully are we considering the impact of getting rid of all that old equipment? How do we compare whether it’s really, globally, ethically, better to improve speed & efficiency vs. just keep using old stuff? How do we factor in things like resources mined in conflict zones?

I recently remodeled a house, and among all the advice I read, I remember this best: “The greenest building material is the one that’s already there.” (It was about floors. We put in new bamboo anyway. That old parquet was awful, but I hope the new stuff stays there for a hundred years or more.)

I know that designing e.g., a great modern smartphone, is extremely difficult, even ‘only’ considering functionality, space and power constraints. Designing something that small and integrated that could still be upgraded partially, keeping around parts that still work, sounds near impossible, but wouldn’t it be great? It sounds like a fun challenge.

‘Editor Wars’

| Comments

An idea I’d like to see: Editor Wars, the game of hacking at code.

Hackers compete on how fast they can complete code editing tasks from a variety of languages, with results plotted and dissected on a web leaderboard. The idea is not to evaluate language understanding or design, but simply editing skill and speed in the kind of thing that editor flame wars start over.

The tasks could be simple refactorings, like renaming a function throughout a source tree, or extracting code into a method. I’d expect this would show advantages of specialized refactoring tools in some IDEs. Other tasks might be more complex, like writing a new set of functions, a whole class, or adding functions to a class to conform to a protocol or interface. Good template support and autocomplete might be an advantage here. Or maybe you want to add conditional debug logging around a set of functions, with each call having a separate hardcoded counter value? Surely powerful macros win this task? Naturally, new tasks could be submitted by the public, and voted on. Each task would have a “correct” answer, but if you’re really clever you could always suggest a better correct answer.

Use of extensions and custom macros would be happily encouraged, as long as you can share what you’ve used.

You’d need either an editor plugin or at least something that watches files efficiently to get the split-second timing your contestants will demand. Ideally you’d be able to record keystrokes and grab the source for any macros you call, then the site would be able to show a replay for the viewing public.

It’d be fascinating to learn how other people use your favorite editor by watching the best of the best compete. Not to mention, just imagine the forum threads arguing over the graphs from the vast database of editor timings.

Anyone want to build this?

Selected Links From This April

| Comments

Since my new blogging setup doesn’t automatically pull pinboard links into summary posts like before, I’m going to run through my bookmarks manually. Sound like fun? Let’s go:

  • Our approach to replication in computational science – “bioinformatician makes code and all analyses available. very admirable, and yet a little worrying that this is so rare. It is also rare in computer science, where it is arguably easier to share.”
  • ModSquadHockey Forums – good reviews of equipment with painful forum UI. BTW, yes, a Bauer Concept II full shield will definitely fit a Stealth S19 (2010 model) helmet. You’re welcome.
  • Hockey Helmet Cage/Visor Compatibility Charts – really useful, but sadly not fully complete. An impossible task.
  • Opengazer: open source gaze tracker for ordinary webcams – Presented without comment. Have you used this?
  • Storm Python ORM – Python ORM from Canonical, used in launchpad.
  • What it takes to build great machine learning products – I think the answer was careful definition of the problem domain?
  • A calculator that only shows the answer after you give a suitable estimate – Really cool idea. The magic is in how to adjust the tolerance so it seems fair. Linked to the HN post because the comments are so polarized – it’s interesting to swatch people violently miss the point.
  • GNU Parallel – Well if you don’t know, now you know: “GNU parallel is a shell tool for executing jobs in parallel using one or more computers.” Super useful. You don’t have to use GNU parallel if you like something else that’s similar, but you should probably know at least one tool for this stuff.
  • GNOME + Do – quicksilver for gnome. I left out the part in the page title that says “Crazy Delicious” because come on, people. These are computers.
  • Field visualization software – “An environment for writing code to rapidly and experimentally assemble and explore algorithmic systems” – A really fascinating project. It looks like big ball of squeak, python, and processing that Bret Victor rolled up. I wanted to try it out in April but haven’t had a chance.
  • circa – A programming language that lets you manipulate the AST and do round trips. I forget what I meant by that exactly. I also wrote that ‘state is first class’. You’d better go look at the page if that sounds interesting. Don’t ask me!

That’s it for April. Look for May’s bookmarks sometime in 2014.