Building a Cyber-Physical System for Developing IoT Apps
The Internet of Things (IoT) is a network of smart devices connected through the Internet that can sense and react to changes in the physical world at different levels. By attaching or embedding such devices to “things” around us, the environment we live in can be turned into a giant, intelligent system that helps to improve our quality of life. As a cornerstone technology that ushers in the next industrial revolution known as Industry 4.0, IoT is key to the economy. It engenders numerous opportunities to digitalize our enterprises, boost their productivity, and grow their businesses. As the rollout of the 5G wireless networks begins to remove the roadblocks to fast and ubiquitous connectivity, the tremendous potential of IoT in transforming our industries and societies starts to unfold. It has been recently predicted by Forbes that the IoT market would double by 2021, reaching $520 billion. The world is expected to demand millions of IoT-ready workers in the coming decades. Our education systems must respond to this trend and take immediate action to inspire and prepare students for this exciting future.
The learning curve of IoT is steep
One of the best ways to master a new thing is to learn it by doing it. Project-based learning, whenever feasible and appropriate, is essential to gaining firsthand experiences and deep insights. However, for the time being at least, doing a simple IoT project appears to be way more complicated than coding a simple animation or game (a good starting point to learn programming in the eras of the personal computer and the Internet). Many beginners are intimidated at first glance by the complexity of even a simple IoT application, which may require knowledge and skills across electronics, mechanics, computer science, and other sciences in order to work with the numerous heterogeneous physical and cyber layers in an IoT system. For example, to build a simple IoT weather station that collects data in a campus and displays the results in a cafe, students must learn:
- What physical properties may characterize the weather condition?
- What sensors can be used to measure those properties reliably?
- How can sensor data be collected and calibrated by a local device?
- How can the data be sent to a server in the cloud for processing?
- What should a server do with the data upon receiving them?
- Which actuator should the result gleaned from the data be sent to?
- How should the information be presented to the target audience?
The list of tasks can grow even longer if students want the system to stop the lawn sprinklers on a rainy day. What makes this list formidable is that each of the steps may require specific knowledge and skills completely new to students (just open the product specs sheet that comes with a sensor and you would know what I mean). As a result, only a small fraction of dedicated learners would be able to overcome the initial barrier. The majority would probably end up feeling defeated.
The risk of being left behind in this crucial wave of innovations is presumably too great for a student as IoT is predicted to be ubiquitous in future industries and societies. A mindset that can think in IoT will guide people to use technology to find more opportunities in businesses or better ways of living in an increasingly smart and connected world. How to support all students to get a sense of IoT and ready them for the future of work becomes a priority in my research lab since the beginning of this year. In this article, I introduce our plans and demonstrate some initial work.
Our vision
Our goal is to develop an easy-to-use educational platform that supports everyone to learn IoT knowledge and skills by creating IoT apps that solve practical problems. As IoT is rooted in everyday life, there are a vast number of interesting problems waiting to be solved in a novel way enabled by an innovative way of thinking. To some extent, thinking in IoT is a brand new ability that our research aims to define, promote, and measure.
This platform is not intended to cover everything in IoT, but it strives to give learners basic ideas needed to develop an ability to think in terms of IoT and, for those who are hooked, a stepping stone to more professional tools. As an IoT system is a cyber-physical system, this platform must have the capabilities of designing and integrating both cyber and physical parts — which also make itself a cyber-physical system. This capacity is useful in teaching critical concepts in the IoT world, such as physical computing, cloud computing, fog computing, edge computing, swarm intelligence, and more (Figure 1).
Recognizing that most people who are interested in learning about IoT do not have a lot of time for tinkering and debugging numerous hardware and software problems that they are bound to run into, we use (but will not be limited to) the following simplification strategies:
- Select versatile microcontrollers such as Raspberry Pi (a $35 quad-core computer so powerful that it can even run my computational fluid dynamics software) for reducing the number of parts and reusing across different apps.
- Favor the use of printed circuit boards (PCBs) such as hardware attached to top (HAT), breakouts, shields, and feathers to save time and avoid problems in designing, wiring, programming, and diagnosing low-level electronic and mechanical components.
- Abstract as many concepts and processes as possible to the cyber part of the platform through digital twinning, to the point that an IoT system can be virtually prototyped before setting up the hardware and fully synchronized when it becomes operational.
- Provide a visual programming environment similar to Grasshopper (an impressive visual programming tool popular in the fields of architectural design and industrial design that bear some resemblance to both block programming and circuit design) to support the design of the computing logic for the target IoT system.
- Use common cloud services such as real-time databases (e.g., Firebase) and machine learning libraries (e.g., TensorFlow) to simplify network connection, data analysis, and decision making.
My initial attempt with Android Things
As a beginner, I decided to choose Android Things as a starting point to build the platform, largely because I trusted its affiliation with Android — the planet’s most popular operating system (OS). In particular, Google once touted Android Things as its general-purpose OS for IoT apps. However, on February 12, 2019, Google abruptly announced in its developer blog that it would “refocus” Android Things on supporting their OEM partners to build smart speakers and displays, effectively terminating the technology as a universal IoT OS. As a result, I had to find an alternative platform to restart my development work and secure my investment. Luckily, I have decided to go with Raspberry Pi, which is supported by the Raspbian OS that bundles the Java Standard Edition. There is also a great open-source Java I/O library for Raspberry Pi called Pi4J. So migrating from Android’s Java to Open JDK didn’t take long for me. I just needed to write my own drivers that talk to the GPIO pins through the Pi4J library (or find open-source solutions if any). Fortunately, there are also a lot of driver code written in Processing for Arduino or Python for Raspberry Pi that can be ported to Java, especially those written in Processing (which is a subset of Java).
The Rainbow HAT driven by Java
Our project began with supporting a small set of boards that can be readily ordered online. The first such board is the Rainbow HAT (again, HAT stands for “hardware attached on top”), mostly an educational device developed by Pimoroni in collaboration with Google (Figure 2). The board comes with three momentary touch buttons, three monochromatic LED lights, seven APA102 trichromatic LED lights, a temperature sensor, a barometric pressure sensor, a piezo buzzer, and four 14-segment alphanumeric displays. An array of GPIO pins, such as PWM, UART, SPI, and I²C interfaces, is also available on the left side of the board, making it extensible beyond the provided sensors and actuators. Although it has been advertised as a board for learning the ill-fated Android Things, it can also be easily controlled using Python or Java — independent of Android Things. If you happen to have bought a Rainbow HAT and are interested in driving it with POJO code instead of Android Things, please leave me a comment below and I will find some time to clean up my source code at GitHub.
On the Raspberry Pi, I wrote a simple Java app that controls the Rainbow HAT. The app supports both headful and headless modes (i.e., mode that uses graphics and mode that doesn’t). In the headful mode, it shows a simple Swing graphical user interface (GUI) for actions such as changing the color of the trichromatic LED lights and a button to gracefully exit the app. Figure 3 shows a result of my code in controlling the board.
Connecting edge devices through cloud services
The standalone Java app that runs on the Raspberry Pi is not an IoT app because it does not communicate with any other device. This means that no one else outside my lab can receive the sensor data or control the lights. But if we connect the app with other apps running on other devices through the Internet, then we can send the data out and have anyone turn the lights on and off remotely. The key to make this over-the-air magic happen is a critical cloud technology based on real-time databases (Figure 4).
To realize the IoT system depicted in Figure 4, I had to write two other apps that set up the connections (devices cannot automatically talk to one another without driver apps). One is a Web-based app that runs within a browser (Figure 5). The other is a mobile app that runs on an Android smartphone or tablet (Figure 6). So we now have three edge devices — a Raspberry Pi (with the Rainbow HAT), a smartphone, and a laptop computer — that are ready to form a small IoT system. Remember, all these apps are only experimental, so bear with the terrible graphics. The important thing is that all the three apps are synchronized to the same state through Firebase in real time. For example, if someone presses a button on the Rainbow HAT to turn on a LED light, its cyber counterparts in the mobile and Web apps will immediately show the “on” state as well. On the other hand, if someone presses a virtual button in the mobile or Web app, the LED light on the Rainbow HAT will be turned on. In other words, this cyber-physical system is ensured to have one and only one state at all time (given some delay, often unnoticeable, due to the latency of communication over the Internet). This feature is vital to our project as it lays a technical and theoretical foundation for creating virtual models that can be readily cast into physical systems.
Notice that in Figure 5, we also show a line graph generated using the data stream from the temperature sensor. The creation of this kind of custom features will be supported by the visual programming environment. The images of electronic components shown in Figures 5 and 6 may eventually be replaced by the real board layout in the HAT (of course, the flexible standalone options will be available for use with breadboards).
The follow YouTube video shows how this IoT system works (without the Web app).
Digital twins
What I show above is ultimately related to the important idea of digital twin. Digital twinning is a technology that maps a physical system to a cyber model. It is an emerging technology that will be pervasive in Industry 4.0. From design to manufacturing and then to operation, digital twins will play a critical role in the industry. The integration of sensors and simulations through IoT technologies will allow engineers to design better products and predict their maintenance needs. Figure 7 shows the realization of a digital twin for the Rainbow HAT. This digital twin can be used as an emulator to support students to program the behavior of the device.
Summary
To summarize, I have made a bit of progress towards the ambitious goal of building a cyber-physical system that supports learners to explore IoT concepts and applications with less frustrations. The work presented here is mostly a recap of my own journey of learning. In the months to come, I hope to be able to report some substantial progresses.
