Visually Programming Hardware

TL;DR A visual programming environment that follows Imitation as a metaphor for Interaction design. This project brings a new programming paradigm where users don’t have to follow/understand any of the usual programming vocabularies, whereas it will enable it’s users to program hardware in an easy to understand, intuitive and safe virtual environment, by presenting programming hardware in an event drive timeline based format. It includes an Arduino based hub, which can be connected to multiple small hardware modules and a web- based application.

 
Screen Shot 2018-10-15 at 11.56.53 PM.png

Project Mentor:
Mr. Rishikesh Joshi, (rishikesh.joshi@sid.edu.in)
Symbiosis Institute of Design

Introduction

The personal computer era began in the early 80s, and by the late 90s, they were everywhere, thanks to the internet to quadruple their functionality. The graphic user interface changed how a computer was perceived, by enabling everyone to use it. Programmers could now create programs for children, artists, housewives etc.

Democratization of content creation has been the cornerstone of these two decades of the 21st century. There is no everyday joe now, joe could be a youtube star or be running his own little online home cake shop. New industries have been created, some changed and some became obsolete. What has remained the same over the year is how we create computer programs. Yes, the computer industry has simplified the world but has somehow still remained stagnant on how we create new programs.

Programing VS Coding

The terms Programming and Coding are often used interchangeably, but we argue that they are not the same. Programming is the process ‘to program a computer to perform a certain task’, whereas coding is the process of writing that program in terms of code. Programming is independent of the medium, and coding is a method to program. When we say that programming hasn’t changed much from 1986 to 2016, we mean it by purely an Interaction Design perspective, of course it’s much more powerful and probably much much simpler but you’re still writing code.

The concept of Visual Programming has been around since the early 90s, LOGO, PureDate, Alice, Scratch are some of the most popular avatars. What sticks about these platforms is all of these still confide to old notions of programming and use the same vocabulary. Most of these were developed to help kids learn to program and were considered as a stepping stone to coding.

Problems with Coding as a medium to Program

  • High dependency in Language limiting it to only English speakers.

  • Steep learning curve for casual learners and makers.

  • High cognitive load.

  • Prone to errors.

Problem Statement

The problem of coding as the only effective medium to program hardware is affecting people who are casual learners, creators or simply inquisitive about the maker revolution. The impact of which is that the world of software creation is limited to only trained computer programmers. A successful solution will provide a universal programming platform that can work across boundaries, regions and is not bound to any language, uses visual interactions to let people create short and sweet Arduino programs with the minimum learning curve.

Project Scope

With the Maker Movement stirring things up all over the world and for the sake of timelines given the scope of the project was narrowed down to Arduino Programming with our target audiences being Makers, Artists, Students, aged between 14-25 Yrs.

Exploration 1

User Research

Contextual Inquiry -
Contextual Inquiry for this project was conducted at a primary school in village Mulshi, Maharashtra and at a computer training center in Pune. The students at the village school were between 6-10 years of age and had could only speak Marathi and were practically alien to verbal English. They were taught only basic computer usage. The students at the computer training center were aged between 13-21 and mostly hailed from lower-middle class families and were learning to program to secure better jobs.
While the kids at the village school were mostly self-taught and at the computer center had regular classes, dependency on the English language was a barrier. Reliance on Memory, more that intuition was high for both.

User Interviews -
After multiple User Interviews with various groups of code enthusiasts and programmers, we were able to establish their goals, pain points, and motivations and sort them in User Personas.

User Needs

Analyzing the research, we formulated our User needs, which will help us in better conceptualizing our product.

  • Lower reliance on memory.

  • Step by step guides.

  • Freedom to experiment.

  • Lower reliance on Language.

Conceptualization

Our User Needs called for a strong GUI driven Visual Programming Platform that can let people program without the use of a keyboard and enables easy simulation, sharing, and execution of ideas.

After multiple iterations we finalized on a model where you first assemble hardware pieces in a virtual space, set small conditions and simulate the process before exporting. Such a model allows users to play around with hardware and setting conditions are easy as all the available options are visible to the user.

Information Architecture

The product required an IA with easy navigation and browsing through various hardware modules and comprehensive settings for each hardware module. The system was divided into three sections, one the virtual workspace/ the canvas, second the properties bar and the third the settings panel, featuring the profile, settings and sharing options.

Wireframes

Exploration 2

Design Metaphor

The word “metaphor,” like the word “affordance,” is very familiar to HCI researchers and practitioners. Metaphors are used in HCI like analogies, to help the users understand complex interactions easily, by relating them to simple everyday tasks and things. “Metaphors are the tools we use to link highly technical, complex software with the user’s everyday world.”. Though the use of metaphors has mostly been discussed in the context of UI design and visual styling of UI elements, we’ve tried to link the entire functioning of the program and the process of programming to a metaphor.

The process of programming today requires one to explicitly mention each and every step you want the computer to perform, in extreme detail. This is opposite of how humans like to and are used to interacting. What we propose is an interaction model where we design a program to behave like a child, eager to learn, that imitates you as you perform the steps you wish it perform. Let’s take a simple example from our daily lives to understand this, imagine how would you instruct a child to do a simple task like brushing his/her teeth. Would you in detail specify and explain him how to move his toothbrush, define the circular axis or would you just perform the action ones and expect him/her to imitate it and later fix his errors. Imitation as an Interaction design metaphor will not only help us create much smarter and intelligent machines, that can easily learn from their instructors but will also simplify the communication between the human and the machine.

Conceptualization

Upon reflection on our first exploration we realized to achieve our goals, we need to let go of all our notions about programming. While the last version would’ve been a huge improvement, it was still using the same vocabulary of programming, creating conditions, using breadboards, tricky wiring etc. This time, we visualized programming as a series of events, that we could tab on a timeline-like interface. Following our Imitation metaphor the final concept is a timeline backed simulation environment where any change in properties of a hardware module is considered an event. All input changes registered as cascading events, and output events are shelled inside input events as their outcomes. The timeline interface helps in adjusting timing, grouping and moving around events.

We also decided to redesign the Arduino and our hardware modules, instead of sticking to generic components we created our own custom modules that are self-sufficient and reduce the clutter.

The Product - 'Mime'

‘Mime’ is a visual programming platform that enables it’s users to program hardware in an easy to understand, intuitive and safe virtual environment. Mime can be used to program an Arduino based hub, which can be connected to multiple small hardware modules that are easily made from off the shelf components. Our goal towards conceptualizing mime is to create a new paradigm of programming that makes it easier to create interactive digital content.

The Mime Hardware -
The core of Mime is an Arduino based ‘hub’, to which other modules can be connected. The hub was built using an Arduino Mega and a custom shield designed and built by us. The is connected to the computer via a USB cable and up to 6 hardware modules can be connected to the hub at a time. Each hardware module has generic components inside and can be identified by the hub for which is which. When a module is connected to the hub, it automatically shows up on the canvas.

The Mime Interface -

  1. Canvas -
    The canvas is the virtual workspace. When you connect a hardware module to the hub it shows up the canvas.

  2. Properties Tab -
    The properties tab show the actions and properties of the selected module of the current state. Every change in the property is recorded on the timeline as an event, all input events are major events and output events are minor events shelled inside input events as their subsequent outcomes.

  3. Timeline -
    The timeline is the core to the Mime experience. The timeline records all changes in properties as events and allows for easy navigation and restructuring of the project. Events can be moved around easily, grouped together, delayed and looped.

How to Use Mime

Assembling the hardware

Step 1 - Place the Mime shield on top of the Arduino board (Images 1-7)
Step 2 - Connect modules to the shield (Images 8-12)
Step 3 - Connect the Arduino to your computer using a USB cable and load up Mime.
 

Using the interface

Project Status

The project is currently under development and the designs plus the source code will soon be released online under the open source license. Connect to know more.