Open Source Software is Important for Modern Science

A Call for Open Crowdsourcing Platforms

Open Source Spot

Humanity is gaining new abilities that are opening frontiers for scientific research. The proliferation of smartphones allows us to create ad-hoc distributed data collection networks that we can use to solve problems we previously thought to be unsolvable. We can now crowdsource live data through active user inputs as well as connected sensors, and we can do this on a previously unobtainable scale, both locally and globally.

However, we are simultaneously facing grave dangers that jeopardize the stability of our global civilization. Our desire for a rapidly increasing quality of life has caused runaway problems like global climate change – we are not currently on a sustainable path to the future. Not coincidentally, our greed has caused serious problems and, potentially, some of the solutions.

There are many problems that we might solve with the large-scale crowdsourcing of data obtained through smartphones. Waze has made significant progress collecting live traffic data which is used to help users avoid traffic, thus potentially lessening the overall negative effects of heavy traffic and therefore pollution. Some researchers are working to build live Earthquake detection and response technology by using accelerometers in stationary smartphones to detect seismic wave signatures. Others are working on crowdsourcing noise data to build noise pollution maps. I’m developing a network of atmospheric pressure sensors to better understand Earth’s atmosphere, with a direct goal of building significantly improved weather models: this is pressureNET. The exciting new abilities that we gain from smartphones seems limited only by our imaginations.

Some of the problems that we’re facing currently, as a species, are urgent. Global climate change, for example, is a very difficult problem that’s already getting away from us. Learning how our actions affect our planet is only one step, and it’s a difficult one. Even more difficult is changing our behaviours so that we can live sustainably on Earth and elsewhere.

So far, we have been too slow. We took too long to figure out how our pollution was changing the atmosphere. We’re taking too long communicating these concepts to everyone. We’re taking too long developing technologies to minimize our future impact. Every year that goes by, our problems get worse and even though the work we’re doing may be good, so far it’s not good enough. Speed matters.

This is why I build open source software, and specifically why pressureNET is open source. Sharing the work that we do, especially in large-scale, crowdsourced data networks, can enable others to work on newer and harder problems. Forcing everyone to reinvent the wheel is not a fast way to get to the future, especially when there are an unknown – but large – number of problems that may be solved with the solution. There are some merits to keeping source code closed; for example, proprietary solutions may result in greater diversity of potential solutions since everyone is forced to solve the same problem their own way. However, this does not outweigh the positives that result from sharing your work, given the low barriers to entry in software development that enable both code to be shared and solutions to be reinvented.

This concept doesn’t end with source code and applications. We should also be sharing entire platforms for data collection and analysis to further reduce the duplication of work that is prerequisite to science research. To my knowledge there does not yet exist a framework for software developers to “plug and play” crowdsourced data networks. The most promising project seems to be Code in the Air, developed by MIT’s Networks and Mobile Systems group. However, CITA doesn’t yet solve all the problems needed to make it simple for software developers to build arbitrary crowdsourcing applications.

I wish I had understood the need for a general purpose, open source crowdsourcing platform sooner, because I would have built it. Instead, I’ve written a more narrowly focused software development kit that makes it simple to collect atmospheric pressure data on Android. This project is the pressureNET SDK which we use to build the core of the pressureNET app. The code is designed to make it simple for developers to integrate atmosphere sensor crowdsourcing into their own apps, both to contribute to the global data collection effort and to pull results from it.

It’s clear now that we must expand the scope of the pressureNET SDK, one step at a time, to make it into a general purpose smartphone crowdsourcing platform. The goal is that it should be a simple task for any developer (on any platform) to build a sensor- or user-based crowdsourcing app. It should be even simpler for these networks to connect together and for the results to be shared. This is a big project and I’m pretty busy right now, so it might take me too long to build this. If these ideas excite you, I invite you to either check out pressureNET’s SDK and expand it to a greater scope, or to build your own from scratch – either way, I urge you to make your solution open source so we can all share the benefits of faster scientific progress.

Locally crowdsourced weather notifications to improve model accuracy

The weather is sometimes extremely localized: it might be snowing where you are, but two blocks away it might be sunny. This level of granularity is rarely captured by weather forecasts, since obtaining such high-resolution data and making forecasts at this micro scale is extremely difficult.

I’m hoping to solve this problem with pressureNET, and we’ve just shipped a new feature that I’m really excited about. We’ve improved our current conditions feature by adding a new widget to quickly submit current conditions as well as a notification if someone nearby reports a condition.

Nearby snowWeather over North AmericaNearby snow notification

You’ll get an alert about the nearby weather and will be prompted to report what’s it like outside where you are. We’ll combine this data with our live stream of atmospheric pressure data to create very high-resolution, personalized weather forecasts that are validated against reality. This will take us some time, so for now, enjoy looking at the map and watching the weather change block-by-block in your city (if enough people contribute)!

Soon we’ll add a feature to show animations of these conditions over time – it’ll be really cool to watch storms roll across cities, and this data will help validate the forecasts we build.

pressureNET 4.0 Beta: testing new methods of crowdsourcing weather data

Yesterday, Cumulonimbus launched pressureNET 4.0 Beta. The update has a lot of new features like regional graphs, location search, and a host of new options and preferences. We’ve added new units, as well, and support for addition sensors beyond the barometer, such as the hygrometer and thermometer aboard the Samsung Galaxy S4.

Primary map view pressureNET 4 beta Device graph in pressureNET 4 beta Regional graph in pressureNET 4 betaSettings in pressureNET 4 beta

We’re testing more than just a new version of our app; we’re testing a new paradigm in smartphone sensor data collection. Our open source pressureNET SDK is a simple library project that Android developers can add to their apps: when started with a simple Android Intent, it will automatically collect and send sensor readings to our (and your) servers. It has minimal impact on battery life and the data is sent directly to atmospheric science researchers who are working with us to improve severe weather forecasting.

If you’d like to help out (and also be a part of this cutting-edge live atmosphere data collection), I appreciate everyone who can help, both in testing pressureNET 4 beta (join this Google+ community, or download directly here) and in testing our SDK – if you’re an Android developer and you’re curious about integrating pressureNET into your apps, check out the project and send me an email. Thanks!

 

Imagining Success for Today’s Startups

Imagine 20 to 30 years out and pretend your favourite startups today have been wildly successful. What kind of world will they have created? Google seems to be on a quest to develop AI and provide it to everyone for free. SpaceX is aiming for a thriving Martian colony by using rapidly and completely reusable rockets and spaceships. Tesla is working towards most ground transportation being fast, electric and solar. Planetary Resources will mine asteroids for rocket fuel to build deep space gas stations, and for precious metals to return to Earth for manufacturing.

If each of these endeavours is successful, the 2030s are going to be an incredible time: commonplace AI, spaceships, renewable energy and cheap, clean transportation. What kind of society will we be? Will those technological changes shape humanity as a group?

I took a pretty narrow focus in picking the examples above; what startups do you imagine 20 years out, and what happens when they succeed?

Crowdsourced Weather Forecasting with Google Glass

Google Glass creates more opportunities for live crowdsourced weather data collection. An app built for Glass could make use of the accelerometer and camera to automatically take a photo of the sky when the wearer is outside and tilts their head upward. The data would be collected and processed and added to existing weather data collection efforts. Algorithms could automatically detect weather features such as sky colour  cloud types, precipitation type and amount, maybe even wind speed. If this app were installed by a significant number of Glass wearers, the amount of data could be significant.

From the Developer Guides it doesn’t seem this app is possible with the Explorer Edition. However, I would expect that the right APIs would be enabled in the near future, potentially for the initial public release.

In their initial video for Project Glass “One Day…”, Google shows the trigger of looking at the sky to display weather information. Perhaps they are already thinking about snapping a photo at the same time, but it hasn’t been built yet as far as I can see.

Since Glass is an Android device, it will be very simple to integrate with pressureNET. Our data collection and analysis system is already in place at Cumulonimbus, so we could provide the Glass wearer with additional data to improve their personal forecasts.

If Google would like to pass me some Glass I’d love to build a prototype. I promise not to name it SkyNET.

Living in the Future: Startup Ideas

As of yesterday I’m working full time on Cumulonimbus, my weather data startup that’s seen fantastic growth and media attention recently. We’re collecting about 300,000 atmospheric pressure measurements per day from Android devices with pressureNET and intend to use the data to improve short term severe weather forecasts. I’m currently focused on Cumulonimbus (product development, growth, and looking for funding!) but I have a lot of ideas for future startups. What do you think?

1. Autonomous air delivery

Build a fleet of small solar-powered airplanes and helicopters that will make long-distance air deliveries autonomously. Monetize by offering lower rates for package delivery; a lower cost structure should exist for the delivery network if the transportation uses no fuel and has little human involvement.

This is a very difficult problem but fast progress is being made in the autonomous flight of small vehicles. The initial work for this startup should probably start in 2014 and it should go to market in 2017-2018. At that time, much of the lower-level functions and engineering problems will be solved and the business can focus on optimizing. There are already some existing projects in this space; very interesting ones are Tacocopter and Blizzard (pdf). Recently a Star Trek Into Darkness promotion involved networked, coordinated hovering quadrocopters forming an emblem in the sky.

I see two ways to get this off the ground:

The first is to test an expensive, beautiful small package delivery in a single area. Make the service very sleek and charge a high price at first, then refine the technology, and expand reach and scale to a global package delivery service. The initial package delivery could be a document courier: to send a small document, request a pickup from a quadrocopter and it will notify you when to come outside for document attachment. Attach the package and it will fly to its destination, where the package drop could include a dramatic parachute from a small height, or a direct package landing in the recipient’s hands from feet above. Most vehicles should be able to land in order for calm deliveries and package attachment.

The second way to get this started is to deliver medicine and lightweight aid to remote locations. Set up an office in a big city of a developing country and fly to remote areas with medicine or other lightweight packages like e-readers. Expand to neighboring countries targeting global reach. The initial approach of sending small aid packages to remote areas is exciting because this is normally accomplished with significant human involvement which is often dangerous. Dropping aid from small autonomous flying robots could immensely reduce the costs currently associated with remote aid distribution.

2. Autonomous flying atmosphere sensors

Build the same fleet of aircraft I just described but pack them with weather sensors and send them away from people. This fleet would gather and send live atmosphere data from over the oceans and remote land areas. This would be an excellent extension to pressureNET and it may even be in Cumulonimbus’ future.  However, it is a very different undertaking than a smartphone network so we’re not working on it just yet. This should also be done parallel and integrated with the autonomous air delivery system described above. There are many paths to monetization, such as early and accurate warning systems for tropical storms. Additionally, the data could be made completely open, so that other innovations could arise from researchers and engineers everywhere.

I think that in the future it may be possible for extremely large swarms of autonomous, solar powered airplanes and helicopters may be sent to forming storms to alter their formation and gently nudge them away from populated areas or towards wind farms built for tropical storm speed winds.

3. Improve Collaborative Document Editing

Compete with Google Drive. Drive is close to my ideal productivity suite but it has major flaws that annoy me on a daily basis. Primarily, I have issues with the current offline model and the poor state of mobile document editing. Is Google actively improving and working on these issues? Or will years pass where Google rides on their excellent real-time collaboration features and makes few changes? I’d also like to experiment with building a git UI into document editing. A large flaw in current Drive workflow is the primitive revision history view and no real branching mechanism.

I also think that there may be a very important hardware decision to make in this process. Form factors are changing rapidly and it may be the case that a certain form factor is found to be the best. I’m writing this blog post on my Xoom tablet, with the wireless bluetooth keyboard, using Google Drive on Android. I’m distinctly aware of the software annoyances and what I want to fix, but hardware-wise I’m not sure where to begin. Having a separate screen and keyboard is pretty fantastic and allows for a lot of flexibility, but sure it’s not the best possible solution to mobile document editing from a hardware perspective. What else can be improved? Are those improvements specific to writing and editing documents or can they be ported elsewhere?

4. Galactic Exploration Game

I’ve wanted to play a very specific game for long time but I’ve never been satisfied with anything that’s been made. It looks like I’ll have to make it myself:

Build a multiplayer online game about space exploration. The game scales from single player to an entire galaxy of inhabited star systems played by millions of people.

Each player begins on an Earth-like planet as an alien intelligence. Initially most players are separated a few star systems apart. Players grow their civilization, starting on a single planet and expanding into rest of their system if they choose. Gameplay would involve creating a growing civilization by good management, learning about the cosmos, and examining the morality of species expansion among other things. This exploration continues into star systems nearby the origin planet by building fast ships, at which point civilizations will interact as they encounter each other. Civilizations may interact far sooner, in fact, if they choose to build powerful telescopes and communicate by radio early in the game. Each player would experience a First Contact moment – if they’re listening.

I expect to see a pattern in the livelihood of civilizations when they choose to explore compared to when they do not. Additionally, poor resource management will result in unstable civilizations who will have to make careful decisions in order to survive. To take this further, I’d envision that a player ought to only play once: if their civilization is wiped out, they’re not welcome in that galaxy any more. Increased connection with reality is a bonus: early-stage gameplay could model optimistic predictions about human reality in the near future. For example, poor decisions may lead to complex and out-of-control global climate change; asteroid mining projects may be the most likely route for players to take in starting their space exploration; unexpected events occur especially for the under-prepared civilizations.

Galactic simulation takes place on servers running open source galaxy simulation code. Star systems are procedurally generated and created on-the-fly. The game could be set in the Milky Way Galaxy but significantly far from Earth. This would allow for the procedural generation of early planets to meld with real science data about extrasolar planets, though only if the game grew sufficiently large as to intersect the areas that we have mapped in detail.

This game must be free and open source to build a community and allow for organic growth. This could be monetized by charging for next-level graphics and simulations. The core game and logic will be free and open source and free to play and modify forever. MVP with minimal graphics and build them out from there, always charging a premium to access the next-level graphical appearance.

5. Mobile IDE

I’m really excited about Android IDE (AIDE) and LightTable because of their fantastic ideas and excellent execution. Those projects make me optimistic about the future of the user experience of software development but there are lots of ideas left unexplored so far. Similar to the ideas above about collaborative document editing, I wonder if there will be a convergence of form factors towards something we agree as ideal? What would this look like, and what would it mean for the software?

What kind of code editing can be done in an extremely mobile environment? There are certainly some coding shortcuts that we can create to enable faster mobile development. In the near future, smart watches, glasses, etc will be gaining traction and combining these with minimal artificial intelligence constructs, I think that the software development experience could be radically altered to focus more on hard problems and less on typing code.

There’s also Google’s new language Go. I recently attended an introductory talk on Go at Google in Montreal and I’m now very interested. Since the language is so new it would be excellent to begin testing and prototyping mobile-focused IDEs for Go. Focus must also include collaborative code editing, perhaps with new additions to a git-like ecosystem for real-time tracking of code.

pressureNET: An Android app to create a worldwide network of user-submitted barometer readings

Two weeks ago Phil Jones and I published the first version of our ambitious Android application on the market. It’s called pressureNET and it lets users submit barometric readings from their Android devices to our database. Users can see a map of the data and browse around, looking at readings submitted by other users. While this data is currently not very informative on its own, we are rapidly gaining users we will be developing tools and analytics to better process this wealth of data. We hope that we can build a large network of barometers that will then enable us to chart and graph weather patterns on a global scale. We have hopes to improve weather prediction on a short-term, local scale as well, perhaps by collaborating with other weather prediction organizations.

Pretty soon we’ll be bringing the map to the web, so that everyone can enjoy our results. Currently, pressureNET supports all Android 3.1+ devices, but so far only the Motorola Xoom and Samsung Galaxy Nexus have barometers.  There are lots of details about our plans (as well as the app’s source code!) on our blog at cumulonimbus.ca.

Binary Search in Haskell

I have just written a function in Haskell that will perform a binary search on a list of integers. Pass it your list, the value you’re searching for, default low and high (0 and n-1) and it will return for you the position of your value. I’m sure there are plenty of experienced Haskell coders that will tell me there’s a better way, but I haven’t heard from one yet.

binsearch :: [Int] -> Int -> Int -> Int -> Int -- list, value, low, high, return int
binsearch xs value low high
   | high < low       = -1
   | xs!!mid > value  = binsearch xs value low (mid-1)
   | xs!!mid < value  = binsearch xs value (mid+1) high
   | otherwise        = mid
   where
   mid = low + ((high - low) `div` 2)