Making the Nordics programmable

That’s the slogan for the on-going tour Nordic APIs with Andreas from Dopter and Travis from Twobo in the lead. Their intent is to

create a community to bring API developers together to share best practices on all aspects of API development.

Yesterday, @fohlin and I attended the Copenhagen half-day “conference”. All talks are (or will be made) available on the Nordic API website. 

Overall I’m impressed with how much content you can squeeze into a half a day. The 25 minute presentations were long enough to convey details, but short enough to not get bored during the less interesting talks. Presentation quality was generally high with @mitraman definitely providing both the most interesting content and with excellent delivery. In his “journey into the terrifying world of hypermedia” he shares some thoughts about why hypermedia is hard and why businesses are not adopting it. 

Although perfectly reasonable, I disagree with the default outset that consumers of APIs are end-user clients. Most, if not all (?), talks assumed the API consumer is some sort of user interface: a mobile app, mobile website, or website. For example, both @mitraman and @madsenevoldsen (who also talked about HATEOAS hypermedia) discussed various levels of implementations of relations between resources where a lot of focus was placed either on 1) the developer using the API to develop a UI, or 2) automatically generated or adaptive UI clients.  

I think we’re missing a large, and possibly relevant, discussion about hypermedias place in machine-to-machine APIs. In my current work, which is heavily related to the Internet of Things, consumers of APIs are not necessarily end-user applications. There’s an additional level somewhere in between the device (or service) and the client where lower-level functionality is sublimed by additional services. This is not only present in the IoT context. Another example that comes to mind is building, for example, recommendation systems which may operate on data from multiple sources. 

For next time I would like to see the following:

  • less focus on end-user clients and more about service-to-service, machine-to-service, or machine-to-machine APIs
  • 10 or so years ago it was all about SOAP and WS-*, today it appears to be all about REST. What about Push and streaming APIs?
  • better air circulation in the room when it’s packed with brilliant people :) 
Personally I find it hard to differentiate between solid experience and best-effort guessing. Not to mention gut-feelings, which make the whole situation even more complex. A friend of mine, Tristan, linked this wonderful website the other day: YourLogicalFallacyIs.com, and I thought “Hey that’s a great tool for learning when to use data.” Related to my two earlier posts on using data to drive the process of making decisions this tool may help you spot the occasions when belief or personal values are getting in the way. For example, the anecdotal fallacy, concerns:  [using] a personal experience or an isolated example instead of a sound argument or compelling evidence. Anecdotes have their place in story-telling. Use them there, or possibly when explaining a complex argument, but not as the single part of a decision. There are more sides to the story.  Note here that I’m not arguing there is no value in experience or gut feelings. There’s certainly room for that too. Perhaps sometimes it is the only “data” available (although I have a hard time believing that…). Experience, in my opinion, is extremely useful when evaluating and interpreting data. Print (or buy for that matter) the logical fallacy poster and stick it on a wall close to you!
Pop-upView Separately

Personally I find it hard to differentiate between solid experience and best-effort guessing. Not to mention gut-feelings, which make the whole situation even more complex.

A friend of mine, Tristan, linked this wonderful website the other day: YourLogicalFallacyIs.com, and I thought “Hey that’s a great tool for learning when to use data.” Related to my two earlier posts on using data to drive the process of making decisions this tool may help you spot the occasions when belief or personal values are getting in the way.

For example, the anecdotal fallacy, concerns: 

[using] a personal experience or an isolated example instead of a sound argument or compelling evidence.

Anecdotes have their place in story-telling. Use them there, or possibly when explaining a complex argument, but not as the single part of a decision. There are more sides to the story. 

Note here that I’m not arguing there is no value in experience or gut feelings. There’s certainly room for that too. Perhaps sometimes it is the only “data” available (although I have a hard time believing that…). Experience, in my opinion, is extremely useful when evaluating and interpreting data.

Print (or buy for that matter) the logical fallacy poster and stick it on a wall close to you!

(Continued) Stop saying “I think…” with Statwing

In my last post I ranted about the importance of using data, or more specifically, the need to stop making decisions on uneducated guesses or ill-founded feelings. I suggested a three-step process to address the issue. Today I want to follow up with a few thoughts on what to do once you’ve gone through those initial three steps. In other words, how do you extracting meaning from the data. 

There are plenty of ways for this and there’s generally no need need to complicate things. Avid Excel users will make wonders there. Some may calculate the numbers on the back of a napkin, and then there are some online tools to help.  

Enter Statwing. A YC-backed product to “Turn data into insight. In seconds.” You upload the oh-so-familiar spreadsheets and Statwing enables you to describe and relate the data contained in its columns to each other. 

For example, an automatically generated description of age-data:

I extracted some of the registration data from Lägr1 to explore how well we are reaching our target groups: 

  • Participants in the ages 15-25 years old.
  • Staff, preferably over 25, but at least 18 years old. 
  • and, are the registrations balanced across the five Scout associations in Sweden? (Yes, I know nowadays there’s only one, but that’s only true on paper so far)

Creating the relations is straightforward and happens almost instantly on smaller datasets. Mine contains about 300 data points. The figure below indicates an answer to the first two questions by relating Type (staff or participant) with Age.  

Surely these are just two simple histograms. I could probably have made those with Excel, gnuplot, matplotlib, or by hand for that matter. The big-win, however, is that this only took me about five minutes from logging in the first time to when I was having the result. And I had never used Statwing before that!

Further I wanted to see the distribution across the five associations. Relating the association name with type I quickly yielded the following figure. 

Skewed! We know Equmenia is organising their own national Jamboree thus we expect lower participation from their side. However, that KM and NSF are as low is somewhat surprising. We certainly have to do some work there. 

Data quality will impact the usefulness of the description and relations. Some data extracted from the registrations is simply too diverse to be of any good use. Or, put it another way, require other methods not (yet?) available in Statwing. For example, mapping participants to a geographical area, i.e, making some form of heatmap, would be nice. If I could overlay that with census data from the Swedish Scouts I could see in which areas  our marketing efforts have had the most impact. 

In conclusion, Statwing is still in its infancy but it is refreshing to see tools like this emerge on the market. There is nothing revolutionary about the graphs above, part from one simple fact: they were dead simple to generate. Tools like Statwing are surely going to influence the way we analyse and use data. And (hopefully!) it will push more people to base decisions on data rather than vague beliefs. 

Stop saying “I think…”

There is one sentence I’ve heard way too often lately while working with all of the projects I’m involved in: “I think it is because this…” or “so you believe it is because of…” Basically relying on, in the best case, guesses but more often on feelings. We need less guessing and feeling and more data to drive our decision-making. 

Data-driven decision-making isn’t anything new. Equally, data cannot stand on its own. There’s certainly a space for intuition too. My argument, however, is that we need to become better at finding and extracting data that is available to us. 

Get data, any data. In one of the projects, the scout camp Lägr1, we’re using easy, quantitative, metrics such as the number of participants who have registered, how many Facebook likes we have, how each of those numbers develop over time, from where the participants are, age groups, demographics from census data, number of emails received and sent to our info-mail, number of opens of mass mails, downloads of our posters, and more. Each of these metrics enables us to attain an objective view on our marketing efforts. It also hints at where we need to improve.

However, quantitative data says little about how we can improve. Qualitative data may provide more insight in those cases. We have, for example, talked to professionals in Scouting, asked for opinions from participants on specific activities, and talked ambassadors within the right age group. These people provide insight from an external perspective that, although it may be “guessable”, is invaluable data to help drive our decisions and future efforts. 

So far there’s nothing new here. Right and wrong. Right because there is data all around all the time. Wrong because to think that people are doing this is not true (according to my own data collection during the past few weeks). We’re simply either too ignorant or too lazy to go looking for data to support our decision-making. Guessing and feeling is the cheap way out. 

Thus, I’m trying to synthesise a three-step process which I run through my own head when subjectivity goes through the roof. At the moment it looks like this: 

  1. What am I trying to answer?
  2. Who or what knows more than me about this?
  3. Is there a correlation between the data and what I’m trying to answer? 

Figure out where to get the data. Start by what you have already, if you don’t have any, postpone or do some simple tests. For example you can set up tracking of the number of incoming e-mails (this was essentially Lägr1’s first metric). And remember that it doesn’t have so darn complicated. Data rules best-effort guessing any day! 

“I think this blog post will spread through the Internetz like wildfire…” 

Or not.

As part of a pre-incubator process called Wrap-it I’m going to assist a team in prototyping their product under the working name Kunskapsplattformen. They’ve been toying with the idea of using game mechanisms to increase stimulation with the core idea of the product.

I haven’t worked explicitly with gamification before, so I thought it would be appropriate to do a small (and not so scientific) survey on popular examples online. 

We’re not yet sure gamification is the “right thing to do” (tm), but that’s part of the prototyping process to figure out. It appears crucial, however, that gamification should not be relied upon and, if applied, giving users early recognition and reward is important.

Got more great examples? 

Analytics for personal finances: testing Qapital Beta

Internet banking sucks. I happen to use Swedbank and its Internet bank is no better than any of its competitors. There is little support for browsing history. All entries are limited in size and usually extremely cryptic (probably shouldn’t blame the bank for this but it does not appear to be doing anything about the problem either). Categorising and grouping expenses is close to impossible although I’m paying by card 98% of the time. To see evolution over time I need to export last year’s transactions to excel and plot them myself. 

This is about as much detail I can get out of Swedbank, except for the obvious “how much money is in my account.” In short, Internet banking sucks.

Qapital is hoping to change that. They are the Swedish equivalent of Mint, a personal finance tool for tracking and analysing your income and expenses. It’s in early beta still, but it’s looking damn promising! 

Key features so far include: 

  • Getting an overview of all your accounts from multiple banks (if you have that)
  • Calculating trends with averages of savings (hopefully) over time. 
  • Automatic categorisation of expenses. Those it doesn’t recognise you can help categorise, and I presume there’s some learning involved (otherwise I’d be interested in helping out with that).
  • Breakdowns of each category down to the actual transactions.  
  • Graphing expenses and income over time with a multitude of time intervals to choose from.

By the time I had logged in and connected it to my bank it had already categorised over 75% of my expenses from today back to over a year ago. Getting that ratio up to 95% took around 30 minutes and most of that time was me trying to figure out where I was and what I did at the time of the transaction (and that was due to the messages being encrypted with bank-grade complexity). If it finds that you have more than one expense at the same place it automatically tags the rest. For February (shown below) there’s only one transaction not categorised. I suspect it is a lunch out with colleagues… 

The screenshot above shows high-level categories and I could opt between a good standard selection as well as create my own sub-categories.

There’s one problem I foresee. Several of my expenses per month are covered by other organisations, and hence at the end of each month I usually reclaim that money. Separating both the expenses and the claims from the other transactions isn’t straightforward and is something I would gladly like to see being supported in the future. Or perhaps I need to tweak or work with the categories somehow. There seems to be a category called “exclude” which may be an option. 

Not entirely surprising, my biggest expense is rent, closely followed by travels, and on third place, drinks and food. I look forward to drilling down into the details to see if I can get actionable information out. In the end, this is where change happens and Qapital is definitely miles ahead of any bank I know.

Lets just hope the banks doesn’t shut them down or anything stupid like that. Keep it up! 

Programming shouldn’t be a part of the curriculum! With arguments like “Not everyone posses the talent to become programmers” or “Programming isn’t as important as math, history or biology” it certainly will never become a part of it either. The fact remains though, as Code.org’s video clearly point out, that the world depends on computers and the programmers who make the computers do what we want.  

In Sweden, programming is not available as part of the curriculum until you’re 15 or 16 years old. About four years after I started tinkering with computers myself. If, however, programming was a part of the curricula then we may become better at logical thinking and tackling complex problems. Teacher James explains in the video that:

Students were able to push through problems. It really builds critical thinking, it builds problem solving, it then something they can apply to math in the classroom or the reading skills.

In my mind, programming can as a first step be integrated with the natural sciences and maths curricula. We had a course in 7th or 8th grade on technology. Building paper-roll towers, toothpick bridges and other tasks that improved our understanding of construction. These efforts need to be expanded into the software realm and pushed down to younger ages. For example, with the use of tools like Lego Mindstorms or Arduino or some other graphical programming.

There are out-of-school activities which encourage programming too. A great example is the several hundred CoderDojos that takes place in local communities around the world. Or the makers communities which are great at inviting kids to play with technology. I had the possibility through scouting to explore technology too. We’ve organised LAN-parties to play games and tinker with computers. Another time we made parachutes for eggs and dropped them from the third floor to see if they landed safely. Of course they didn’t at first. But through trial-and-error and given time to experiment we figured out the necessary components for a safe flight.

Programming to me works a lot in the same way. You need to spend time with a problem, try different angles of attack, fail, and eventually figure it out. Learning this, however, isn’t done in a fortnight. It takes practice. A lot of practice.

Making programming a part of our standard education would significantly enhance our chances to become programmers. The ones who will be “the wizards of the future.” 

As the internet already know by now, Posterous is shutting down. I’ve been using it on and off for the past three years and was fairly happy with it. Since Twitter acquired it it soon became obvious that its future death was imminent. Thus, I knew the day would come when I had to move.  These are some alternatives that I opted between and why I didn’t go for them:  Wordpress.com: On the plus side is easy integration from Posterous, its extremely wide usage and popularity. However, Wordpress is too bloated for my needs. There are a gazillion of widgets and I’ve got no interest in messing around with plugins. None of the default themes immediately came to my liking.  AWS EC2 Micro + Jekyll: I’d been toying with the idea for a while to use Amazon Web Services and a static HTML based site to get more practical experience with AWS. Running a micro-instance is for my blogging needs more expensive than I’m prepared to pay. And messing around with HTML (although Markdown-support is possible) didn’t rank high on my list. Heroku + Jekyll: Lower costs, but there was still a nagging thought about static HTML.  Tumblr: long-time competitor to Posterous that “made it”. Sleek posting interface and Markdown support. Adequate defaults, different post-types and support for custom domains made me opt for it. And thanks to justmigrate.com moving all posts from Posterous to Tumblr took no time at all.  Last weekend I also kicked-off a Tumblr blog for the scout camp that I’m helping out with. If you’re a Scout, you should check it out!  Hey Tumblr! 
Pop-upView Separately

As the internet already know by now, Posterous is shutting down. I’ve been using it on and off for the past three years and was fairly happy with it. Since Twitter acquired it it soon became obvious that its future death was imminent. Thus, I knew the day would come when I had to move. 

These are some alternatives that I opted between and why I didn’t go for them: 

  • Wordpress.com: On the plus side is easy integration from Posterous, its extremely wide usage and popularity. However, Wordpress is too bloated for my needs. There are a gazillion of widgets and I’ve got no interest in messing around with plugins. None of the default themes immediately came to my liking. 
  • AWS EC2 Micro + Jekyll: I’d been toying with the idea for a while to use Amazon Web Services and a static HTML based site to get more practical experience with AWS. Running a micro-instance is for my blogging needs more expensive than I’m prepared to pay. And messing around with HTML (although Markdown-support is possible) didn’t rank high on my list.
  • Heroku + Jekyll: Lower costs, but there was still a nagging thought about static HTML. 
  • Tumblr: long-time competitor to Posterous that “made it”. Sleek posting interface and Markdown support. Adequate defaults, different post-types and support for custom domains made me opt for it. And thanks to justmigrate.com moving all posts from Posterous to Tumblr took no time at all. 

Last weekend I also kicked-off a Tumblr blog for the scout camp that I’m helping out with. If you’re a Scout, you should check it out! 

Hey Tumblr! 

Loading and running Python tests from code

Loading and running tests from code is possible, but it is not immediately clear from the docs of unittest. This is how I did it:

In the project which I’m currently involved with I’m using an early version of a simulator for connected devices. The first version of the simulator used a rather hairy bash script to set up environment variables, launch a web interface, load the simulator and run the specified simulation. Since the simulator is mostly written in Python though, I wanted to remove the dependencies on bash and also the cryptic environment variables. 

Python has an awesome library called argparse. Using argparse and some regular Python magic I substituted the original bash script with a Python one both reducing code size (by half) and increasing readability. 

Additionally, I created a set of sub-commands to run and test simulations using the code above. Now it is only a matter of running: 

./simulator test projectname

Happy hacking! 

Part 1: Unboxing the Ninja Blocks kit

One of the best things at work is that I get to test and play with a lot of technology. So when the Ninjas down under announced at Le Web that they’re opening up a pre-order of 1000 kits we jumped on the train. The kit, after having had some issues clearing customs, arrived this week. 

2013-02-12_08
Opening packages is always a bit like Christmas. The Ninja Blocks kit is nicely packaged and comes with the following devices:
  • a Ninja Block
  • a button (it is in fact a door bell without the soundy function)
  • a PIR sensor which detects movement
  • a temperature and humidity sensor with its own display
  • a magnetic switch
  • TP and power cable for the Ninja Block
  • and, superbly, a micro-full SD converter for plugging the memory card in your computer

What’s worth noting is that it is only the Ninja Block itself which is custom made, all the other devices are off-the-shelf products that have been made to work with the Ninja Block. To me this is a sound approach. First, the guys don’t have to spend all their time and money building devices that connect with the Block. Second, there’s already a plethora of devices out there which does most things you can imagine. And third, they are able to leverage the community to create support for devices for the Blocks since the kit is open source. If for some reason there’s a sensor or actuator missing on the market, build your own! There’s a thread in the Ninja Blocks forum (and another one) reporting other devices that works out of the box too. 

2013-02-12_08 2013-02-12_08 2013-02-12_08 2013-02-12_08 2013-02-12_08
See the full gallery on Posterous

Even the Block isn’t fully custom made, it packs an Arduino and a Beagleboard to run communication between the devices (RF 433Mhz) and “the cloud”. That in turn means support and strength from two other huge communities.

Getting everything up and running is almost dead simple. The one exception being that you have to open the Block to read its serial number which is required to pair the Block with your Ninja account. However, the instructions are awesome and easy to follow. Afterwards, pairing devices with your Block is done mostly automatically where you simply have to name them. 

2013-02-14-09 Screen_shot_2013-02-14_at_9 Screen_shot_2013-02-14_at_9
See the full gallery on Posterous
The fact that Ninja Blocks builds on Arduino is one of the reasons we got a kit. Arduino is being used heavily in research at Medea and one of its co-founders also happens to work here. Thus, there’s a lot of hardware hacking going on and it’ll be interesting to see what they may do with with the kit while I’ll stick to investigate the software side. More on that in a future post. 

Happy hacking!