More about photos

The last time I wrote something here, it was about how to keep family photos. I was using Picturelife to upload photos from our phones. I used a feature of theirs to host my own photos on Amazon S3, so when their service imploded after an acquisition a while back, I managed to avoid disaster.

There's a great episode about the end of Picturelife on the Reply All podcast - they talk to the eventual owner about what actually happened.

Anyway, I gave up on public services and bought a WD MyCloud Mirror NAS box - a little Linux-based appliance with two redundant disks that is available over the network, so phones can back up to it, and it can back itself up to S3 for extra security. I spent a couple days loading all our phones and flash cards into it, and it's up and running. It's a little slow (there are fancier boxes available for more cash), but it seems to be good enough for now.

However backup is all it does, so now I'm kind of in the opposite place I was before - I feel secure that I'm not going to lose these photos, but they're all just sitting there, unorganized and not easily viewable.

I'd like to find a nice program to organize and de-dup them, without buying into a big online system or some soon-to-be-dead proprietary database format. It seems like maybe I'm in the minority, though. Maybe next fall's blog post will be about what I've found. If you have any recommendations, let me know.

What to do with Photos 2015

It's 2015, and saving and making use of digital photos and videos is too hard, and doesn't seem to be getting better. I still have printed pictures of my parents when they were younger than me, because they were in an album somewhere. But I need to get serious with some kind of digital media strategy, or my kids might not be able to find or access any pictures of themselves as babies.

We have a couple Macs, some iPhones and some cameras with SD cards. We want to back up photos to one place, permanently prune out bad ones, and be able to access all of them in an organized way that lets us e.g. upload them to places to get yearly books printed. This basically means we need the stuff in the cloud, with an easy way to download subsets onto local disks.

So, we need some kind of server-hosted solution, and there are so many it's hard to keep track. I looked into most of them, and read a few summary posts, and so many of them don't quite make it easy to import from everywhere, or they leave me uneasy that I'll have to switch again sometime soon. Big companies abandon their products, and small companies get acquired then abandon their products often enough that almost any for-pay service feels temporary. Free services, or platform-tied things like iCloud, Google plus or Amazon Prime Photos also feel temporary, because they're tied to the success of that platform (and what if I decide to change phones?)

Last summer, I spent a lot of time working with git-annex because I liked the flexibility of not having to keep all the photos in one place but still have them in one logical 'annex' with their location being tracked - and I wanted to control the backup of the data. I even spent a long time on an import script that avoided dupes and tagged photos with git-annex metadata from their GPS info. The neat database-like "view" features built around metadata turned out to be unusably slow with >1K files, though, and so I was looking at more work trying to help fix that, along with still needing to figure out what to do with the iPhones, and it just seemed like too much effort.

So, I looked back at reviews of hosted sites (I thought it was from sweethome.com, but I can't find it there anymore), and saw that Picturelife allows you to use your own bucket on Amazon S3 for photo storage - then you get the Picturelife features for free and only pay Amazon for storage. So if Picturelife dies for some reason, your bucket still has your photos. I thought that sounded pretty good, so I've been trying it out for a while. I like their import features - there's a mac app that I used for the initial bulk upload and ongoing for emptying the SD cards, and the iPhone apps work as well as apps can be expected to (background uploading seems to not work or not be implemented).

I have some UI nits to pick, but overall it is good - lets me look through pictures (I haven't bothered with the editing UI yet), and hide or delete bad ones, tag and write captions. All good so far. There's a map view that's interesting but has equally interesting bugs - I reported one where photos from Portland, OR were sometimes interpreted as being in Portland, Maine for example. And it'd be nice to be able to say "show me this picture on the map", but that direction isn't implemented.

Since I wrote the last paragraph, none of my bugs have been fixed, and Picturelife's incredible journey has been acquired. I'm currently just waiting to see how that changes things - they say it'll continue as a product. At least my photos are in my own bucket.

If only there was an easy obvious choice. What do you do?

Links for June 27, 2014

  • OpenTuner by jansel "OpenTuner is a new framework for building domain-specific multi-objective program autotuners. OpenTuner supports fully customizable configuration representations, an extensible technique representation to allow for domain-specific techniques, and an easy to use interface for communicating with the tuned program. A key capability inside OpenTuner is the use of ensembles of disparate search techniques simultaneously, techniques which perform well will receive larger testing budgets and techniques which perform poorly will be disabled." -- This looks really useful. I started on something like this long ago at LLNL, but since I was so young I focused mostly on fancy plots and designing a language for describing experiments.

  • Paper to Plants on Vimeo A really cute video about a game that seems pretty charming. I'm still not sold on little kids using iPads so much, I'm told it's bad for their eye development.

  • Ergonomic Workspace Planner Tool | ComputingComfort.org Use this to figure out the optimal height of your standing desk

  • pollen Write web based books in racket

  • Finding the perfect house using open data — dealloc.me A guy builds a map of available houses in Portland that match his desires. Seems like a good real estate agent should do this for you - but how do you know you've got a good one? I guess you have to write some code.

  • Drawing Presentable Trees "When I needed to draw some trees for a project I was doing, I assumed that there would be a classic, easy algorithm for drawing neat trees. What I found instead was much more interesting: not only is tree layout an NP-complete problem1, but there is a long and interesting history behind tree-drawing algorithms. I will use the history of tree drawing algorithms to introduce central concepts one at a time, using each of them to build up to a complete O(n) algorithm for drawing attractive diagrams of trees." -- I've always wondered what a good way to do this would be. Knew it had to be a solved problem.

  • Grain-Free Oatmeal Raisin Cookies | Against All Grain - Delectable paleo recipes to eat & feel great If you want oatmeal-raisin but you're trying to avoid oats and sugar, these are really quite good. The coconut makes a great texture.

  • Lev Grossman on his daughter, Lily: How being a father ruined my life and made me a better writer. A really heartfelt story about becoming a dad and getting your act together. Now that I have kids I'm a total sucker for this kind of article.

  • HAL :: [inria-00555588, version 1] A comprehensive study of Convergent and Commutative Replicated Data Types Formal exploration of sync-able data types. Abstract: "Eventual consistency aims to ensure that replicas of some mutable shared object converge without foreground synchronisation. Previous approaches to eventual consistency are ad-hoc and error-prone. We study a principled approach: to base the design of shared data types on some simple formal conditions that are sufficient to guarantee eventual consistency. We call these types Convergent or Commutative Replicated Data Types (CRDTs). This paper formalises asynchronous object replication, either state based or operation based, and provides a sufficient condition appropriate for each case. It describes several useful CRDTs, including container data types supporting both \add and \remove operations with clean semantics, and more complex types such as graphs, montonic DAGs, and sequences. It discusses some properties needed to implement non-trivial CRDTs."

  • What I Wish I Knew When Learning Haskell 2.1 ( Stephen Diehl ) Nice quick article with practical tips for beginners.

  • Not So Much 'New York Poor' as 'Pittsburgh Rich' - Pacific Standard: The Science of Society You can get a lot for your money in Pittsburgh, and lots of other places throughout the US. I'm from Pgh, and have fond memories. I'd consider moving back there if they could fix the weather.

  • Call me maybe: Elasticsearch Part of a series of irreverent but thorough explorations of various popular distributed systems.

  • What Do We Save When We Save the Internet So as you proceed with your protests, I wonder if you might also ask, quietly, to yourself even, what new growth might erupt if we let the Internet as we know it burn. Shouldn't we at least ponder the question? Perhaps we’d be better off tolerating the venial regret of having lost something than suffering the mortal regret of enduring it."

  • Music Animation Machine — "Music Worth Watching" Old stuff, but great. Couldn't believe I hadn't bookmarked it long ago.

  • Moving from Octopress to Nikola | serialized.net This post and a few tweaks got me into nikola without much hassle.

Flashing a Nexus device with Ubuntu Touch from Mac OS X

I've been working with Ubuntu Touch recently, using VirtualBox to run Ubuntu Saucy as a guest on my Mac, and while it does work to connect to the device once it's installed, the process of reflashing is difficult. The manual steps on the Ubuntu Wiki Touch/Install page work for the old way, via phablet-flash cdimage-touch, with the caveat that you may have to manually reboot the device and/or re-connect the USB in between steps.

However, as of this email last week, the official way to flash phones is using the system images via phablet-flash ubuntu-system, which has a different set of files.

Instead of just muddling through and updating the manual instructions, I decided to see if phablet-flash would run natively on the Mac. After a bit of python dependency resolution, it does. Here's what I did:

  1. Download the Android SDK tools. You only really need the "SDK Tools", not the whole ADT Bundle, so I clicked on "Use an existing IDE" at the bottom of that page, to get the smaller installer.
  2. Unpack the archive and run ./android. Install the platform-tools (and nothing else).
  3. Add the platform-tools/ directory to your $PATH for convenience.

  4. (Optional) create a virtualenv for the python dependencies, and activate it.

  5. pip install each of configobj pyxdg, and pyliblzma
  6. get bazaar if you don't already have it.
  7. branch phablet-tools: bzr branch lp:phablet-tools
  8. cd phablet-tools && python setup.py build
  9. At this point, phablet-flash ubuntu-system should work smoothly, with all transfers and reboots and timeouts going as expected.

The only minor hiccup is that it apparently doesn't get ~/Downloads/ from pyxdg, but it just means that you'll get the images saved to ~/phablet-flash instead of ~/Downloads/phablet-flash. No big deal.

Links: Simulation, Programming, Crab Cakes and Hockey Player Usage Charts

  • DYNAMO Someone has rewritten one of the earliest simulation systems in JavaScript (the fate of all interesting software). Also includes a link to an article about the history of simulation software that sounds very interesting.
  • The Food Lab: The Crabbiest Crab Cakes I love crab cakes, but I'm not sure I really want to try to make them at home. If I do, I'll use these tips. The Food Lab is fun stuff.
  • Debug It! A review of a book on debugging, which is a topic that I think should be taught right alongside programming. See also "Why Programs Fail"
  • GiveDirectly: introducing a radical new way to give! | GiveDirectly Send cash straight to poor people. If their assertions are true, it's a really interesting idea, and I can't believe it hasn't been done before. It also seems transparently better than microloans.
  • ContinuumIO/Bokeh · GitHub Something to look out for -- a Python ggplot that works with HTML5 is a great idea. "Bokeh (pronounced boh-Kay) is an implementation of Grammar of Graphics for Python, that also supports the customized rendering flexibility of Protovis and d3. Although it is a Python library, its primary output backend is HTML5 Canvas. There are many excellent plotting packages for Python, but they generally do not optimize for the particular needs of statistical plotting (easy faceting, bulk application of aesthetic and visual parameters across categorical variables, pleasing default color palettes for categorical data, etc.). The goal of Bokeh is to provide a compelling Python equivalent of ggplot in R."
  • FitDesk X1 Level up from a standing desk? I'd love to try this for a day.
  • Concurrent Revisions DVCS-like concurrent programming. Interesting sounding research - I haven't read it yet...
  • Many thanks to @robvollmannhl and the good folks at Hockey Abstract for these great interactive Player Usage Charts: hockeyabstract.com/playerusagecha… Player Usage Charts are fascinating, but I can never figure out why people always change the axes so that the dots fill the space. It makes it impossible to compare two charts, and it's not obvious, so you end up comparing charts without realizing that it's meaningless.

Mock nested properties with python-mock

Python's mock library (part of stdlib in 3.3+) is a great tool for writing concise tests. Its documentation is very good, and rewards multiple reads - but I found one thing that wasn't totally clear, even after looking through a few times. I wanted to use PropertyMock to mock nested Properties. Specifically, I had patched the python Gnome-introspection wrapper for libsoup at the top level Soup objcet, and I also wanted to replace one of its nested constant properties, Soup.MemoryUse.COPY with a sentinel that I controlled, for later comparison.

The idiom for PropertyMock is to assign a PropertyMock to the type object of the Mock object whose property you want control of. What I found is that because Mocks auto-create properties, it's possible to do nested mocking in one line, like this:

import mock
m = mock.Mock()
type(m.a.b.c).d = mock.PropertyMock(return_value = mock.sentinel.my_value)
assert(m.a.b.c.d == mock.sentinel.my_value)

So my soup example looks roughly like this (mixing testing and tested code, and repeating literals for brevity):

json_body = "{}"
with patch(gi.repository.Soup) as mock_soup:
    from gi.repository import Soup
    type(mock_soup.MemoryUse).COPY = mock.PropertyMock(return_value=mock.sentinel.COPY)

    # tested code:
    message = Soup.Message.new("POST", "http://fake.com/api")
    message.set_request('application/json', Soup.MemoryUse.COPY, json_body, len(json_body))

    # checking:
    assert(mock_soup.mock_calls == [mock.call.Message.new("POST", "http://fake.com/api"), 
                                    mock.call.Message.new().set_request('application/json', 
                                                                        sentinel.COPY # <--- this was the point
                                                                        "{}", 2)])

It's often possible to think of a shorter, clearer use of the mock library after revisiting a problem, but so far this still seems good. Let me know in the comments if you have a suggestion for improvements.

Don't Snooze

Mailbox is a new iPhone email app that grabbed a lot of attention recently, partly because of their waiting list for using it. It's a gmail-only system that stores copies of your email on Mailbox's servers, so they can do new features beyond the standard mail client. Even though I got through the waiting list a week ago, I haven't tried it myself, because I wasn't

I have a few religious beliefs about email. One is that processing incoming email is different from

that you should only process a message once - when you first see it, you should decide what to do next with it, even if that's "read it again later",

I call it a religion because I believe it, but I don't always practice it. I have an inbox full of things I've looked at once and am procrastinating on, partly because it's hard to move things to the todo list that I use from Gmail. It annoys me every time I look at my email, and I've forgotten important things that I left sitting there. This is exactly why I think it's important that our tools encourage good habits, instead of encoding bad ones.

Mac IRC clients

At Canonical, we are spread all over and keep in touch via IRC. I've tried out a bunch of IRC clients, but nothing is quite right. I'll explain. Here's a list of some things I want from a client - it's probably not complete, but it'll do for now:

  1. Don't crash. A bouncer makes crashes recoverable, but I shouldn't have to use one just to keep the backlog around.

  2. I need to search the backlog. Incremental search is best.

  3. Show multiple channels at once, in one window.

  4. Highlight mentions of my nick so I can glance at a window and see if anyone's asking me something. Use Growl or notification center, too.

  5. Make it obvious when I've disconnected, and reconnect robustly.

  6. Let me hide/fade out join/leave notifications and other admin messages.

  7. Keep my place so I know what messages are new.

  8. Make it obvious which channel I'm looking at, and even more obvious which channel I'm about to type in. (Nothing is great at this)

  9. Spell-check is nice, but autocorrect is awful for IRC and technical discussions. Needs a global switch.

  10. Don't show unread counts. Some channels are chatty and I'm just in them listening for my name.

Here are the clients I've tried and why they aren't quite perfect:

Colloquy

Colloquy is a nice full-featured open source client. I've used it the most. However - it doesn't show multiple channels in one window at the same time, and doesn't keep your place. I've had some problems with reconnection, where I thought IRC was quiet when I was actually just disconnected and didn't know it.

I have to turn autocorrect off for each room I'm in, which I only remember after the first time it really annoys me by mangling some jargon.

There's also some general buggy behavior, and I have a short list of bugs I'd like to report someday, but I don't have time to fix them.

Years ago I hacked my own version of a place marker that I liked, but I can't remember why it didn't make it in, and of course the code is lost now.

The backlog search looks like it should be powerful, but in my experience it just doesn't work - it never displays any results. I'm not sure what the deal is there.

(As a side note, Colloquy's iPhone app is the only iPhone IRC client I've used. It seems to work fine, and I don't use it enough to have specific gripes. Just having to have IRC open on my phone is guaranteed to annoy me more than the software itself could.)

Emacs ERC

My current setup is just to run ERC in an emacs window alongside my code editing. This has super flexible window splitting and size management and unbeatable search for your backlog, since it's just another emacs buffer. There's another nice benefit - if you're writing about code, emacs' tab-completion for strings across all buffers means that you can easily autocomplete SomeReallyLongFunctionName in chats. This saves a lot of typing and cut n' pasting. When you do need to cut n' paste, it's nicer to do it between emacs buffers than between apps.

The defaults are not ideal, as usual with emacs you need to dig around to find a good setup.

There's a module to colorize nicknames according to a hash of the name, so it's easier to tell people apart. Other clients have similar features but I think I like emacs' the best for some reason - maybe the palette is nicer.

There's a module to mark your place that only works if you left the cursor in the backlog, which is a little weird. It will highlight my nick when mentioned, but not the whole line.

Since it's open source, I have the same problem as Colloquy, it's hackable, but I don't really have time for that. And emacs is fun to hack on, so it's a potentially dangerous time sink.

Linkinus

This is a nice client with some minor flaws and a little flakiness. I liked it but moved on because of minor annoyances.

It has a kind-of nifty "conversation tracking" feature that tries to highlight messages in a back-and-forth when you hover over it with the mouse (but only in the currently selected channel). It's not really that useful, but I think it could be if pushed further. I'd like to see more clients try this kind of thing, since separating multiple concurrent conversations is hard on IRC, and it can be hard to avoid them.

Lets you display multiple channels in one window, but that needs some improvement. There's just one text-entry field so it's easy to type something to the wrong channel.

The split display has some issues. The split channel views don't show you the channel name anywhere, and in some cases don't even show you the channel topic, which may or may not be helpful anyway. So it can take some effort to figure out what channel you're looking at, especially if there are many of the same people talking on two channels.

It can save a set of channels as a group, but it doesn't save the relative sizes of the splits in the group, so if you want to flip between groups, you'd better be OK with even splits.

Finally, it only supports vertical splits, so with more than two channels, you just get a couple unreadably long lines per channel.

It has a very barebones backlog search, but it does work.

They've also clearly tried to re-design the account setup / prefs experience, but I'm not a big fan of an inspector palette - I wanted to compare settings between two servers and it will only show me one at a time. This was also the biggest point of flakiness, where some edits wouldn't commit and I couldn't tell if the settings were being changed.

Textual

Textual is an open-source app (github.com/Codeux/Textual) that is for sale on the Mac app store. This is an interesting approach that I think has some merit - people may still contribute code or bug fixes, but it's also nice to know that if the app ever gets abandoned, it could be resurrected. That said, I tried building from source to enable a quick hack once but the master branch didn't build for me, so they're not trying to make it easy.

It lets me hide some admin messages, has automatic & manual scrollback markers to keep track of what I've missed.

This has a setting to "Track conversations using nickname highlighting" but I couldn't really tell what it was doing.

This also has a pretty barebones search, but again, it works.

I tried this for a few days and liked it, but ultimately switched away, I think mostly because you can't display multiple channels - there's just one window and it doesn't split up.

Mango

I haven't tried Mango. The screenshots in the App Store and feature listings don't lead me to believe that it's anything significantly different from the others. I'd be glad to hear from fans, though.

So, what do you use? Have I missed anything great?

More out of date links

Links

  • What's so bad about Template Haskell? - Stack Overflow

    A bunch of good answers to why Template Haskell is occasionally maligned.

  • Which Haskell (GHC) extensions should users use/avoid? - Stack Overflow

    Interesting list of aesthetic judgements of haskell extensions. At press time I don't know what most of these words mean.

  • Yesod and Template Haskell

    Short post on why Yesod uses template haskell (which has a bad rep)

  • deck.js » Modern HTML Presentations

    Best "Write your presentations in HTML" thing I've seen yet, although I'm not sure how great it'd be at complex visuals or builds. Yes sometimes builds are good! SOMETIMES.

  • REDbot

    RED is a robot that checks HTTP resources to see how they'll behave, pointing out common problems and suggesting improvements. Although it is not a HTTP conformance tester, it can find a number of HTTP-related issues.

  • .@dpp’s Visi looks to be exactly the thing when Soulver isn’t enough and spreadsheets are rigid github.com/visi-lang/visi #emerginglangs

    Should I look at this before posting a link? Nah... But I will look at it, eventually! It looks interesting, I love this kind of stuff - although as usual I wonder if this is something you can already get in Mathematica but no one knows because it's so expensive.

  • High Scalability - Prismatic Architecture - Using Machine Learning on Social Networks to Figure Out What You Should Read on the Web 

    Probably interesting? I skimmed this, I'm not gonna lie, I don't remember much.

  • Deslidefied HTML

    Really useful bookmark to make slideshow news articles readable on one page. I am using it all the time!

  • Thinking Functionally with Haskell

    ""In which we explore what modern type systems bring to the table. Imagine an approach to programming where you write down some description of what your code should do, then before running your code you run some automatic tool to see if the code matches the description. That’s Test-driven development, you say! Actually, this is what you are doing when you use static types in most languages too. Types are a description of the code’s inputs and outputs, and the check ensures that inputs and outputs match up and are used consistently. Modern type systems—such as in Haskell or above—are very flexible, and allow these descriptions to be quite detailed; plus they are not too obtrusive in use and often very helpful. One point I’ll investigate here is how advances in types are converging with new ideas on testing, to the point where (I claim) the old distinctions are starting to blur and starting to open up exciting new possibilities—hence my suggestion that we need a new word to describe what we’re doing that is free from preconceptions and out-dated thinking. So put aside your bad experiences from Java, and prepare to be amazed! ""

  • So you want to hack on IMAP…

    "Well, first off, you’re horribly misinformed: you do not actually want to hack on IMAP." (I agree, IMAP bad.)

  • The Machine That Changed the World: Great Brains - Waxy.org

    Documentary about computing history: "It's a whirlwind tour of computing before the Web, with brilliant archival footage and interviews with key players — several of whom passed away since the filming. Jointly produced by WGBH Boston and the BBC, it originally aired in the UK as The Dream Machine before its U.S. premiere in January 1992. Its broadcast was accompanied by a book co-written by the documentary's producer Jon Palfreman."