Announcement: NNStreamer Releases version 2.0

NNStreamer released its version 2.0 last October. It is a set of GStreamer plugins supporting easier and more efficient adoption of neural network models for GStreamer developers, adopting neural network models and neural network developers managing neural network pipelines and their filters.

There are a variety of improvements added in version 2.0. Through Edge-AI capabilities, users may connect independent and remote pipelines with NNStreamer-provided protocols designed for AI data streams. In other words, a lightweight IoT device may offload its AI workload to its neighbor devices with GStreamer/NNStreamer pipeline descriptions, and an AI service may easily publish its output data streams for other AI pipelines.

NNStreamer as Open Source

NNStreamer is a project of the Linux Foundation AI & Data (LF AI & Data), and is currently in the incubation stage. It was contributed by Samsung last year after said project was open sourced in 2018.

The LF AI & Data Foundation is an organization created to establish an ecosystem for open source innovation in the fields of AI, machine learning, and deep learning. Recently, it has announced its Principles for Trusted AI, which include Explainability, Transparency, Robustness, and 5 others. As seen below [1], there are various open source projects aside from the NNStreamer project. Additionally, there are also other open source projects relating to data stream processing.

Achievement of CII Best Practices "passing" badge

As part of the effort to advance the project from Incubation to the Graduate level this year, NNStreamer achieved a Core Infrastructure Initiative Best Practice (CII BP) Badge [2], as well as documented and structured their open governance processes.

A Linux Foundation (LF) CII BP Badge is a way for Open Source projects to show that they are following best practices. The badges let others quickly assess which projects are following best practices and hence are more likely to produce higher-quality secure software. It also helps open source projects find areas where they can improve. A CII BP Badge shows that an open source project is implementing a large number of good practices to keep the project sustainable, addressing vulnerabilities when found, and is helping improve that security. There are three badge levels: passing, silver, and gold. Each level requires that an open source project meet a set of criteria; and for silver and gold this includes meeting the previous level. The “passing” level captures what well-run open source projects typically already do, and has 67 criteria grouped into six categories such as quality, security, etc.

Other Improvements in the 2.0 Release

In addition to the improvements in NNStreamer’s Edge AI capabilities, the security vulnerabilities discovered using the LGTM tool were resolved and improved in the NNStreamer 2.0 release. This is more meaningful as the code enhancement is done by a contributor who, while is a new participant in the NNStreamer project, collaborated and contributed to it.

Interview with NNStreamer Project Pioneers

We are pleased to announce the NNStreamer 2.0 release with code maturity and quality enhancement above. To celebrate it more, we arranged an interview with Maintainer MyungJoo Ham, Committers Jaeyun Jung and Taras Drozdovsky. Kindly read through the interview.

First, we congratulated them on the release of version 2.0 of NNStreamer, and asked them to tell us the background story of the project’s birth.

“In 2018, we were working on a project on autonomous driving vehicles. In this project, we have seen the vehicle as a high-end and real-time on-device AI system that is required to execute a lot of neural networks interconnected with each other and to fetch high-bandwidth data streams from various sensors, which has caused enough pain to consider writing a new software framework or approach for the whole system. Another major pain usually neglected by many developers was that, with a lot of developers involved, there are a large number of changes in the software system that change relations between components, which affect the overall architecture of system, causing a lot of effort and cost.

A small group of developers was tasked to mitigate this issue by applying a new architecture or software framework. After a series of discussions, research, and prototyping, we have determined that a pipe-and-filter architecture is suitable for such systems and complex on-device AI systems in general and we have decided to use GStreamer as the basis of such systems. This was the beginning of the project, NNStreamer, although the NNStreamer group was soon detached from autonomous driving project and started working for different on-device AI systems of conventional embedded devices.

It has started with a single developer (the founder) and soon garnered attention of more developers. Within a few months, the NNStreamer group has grown to have about 10 developers and declared to be an open source project. The first commercialization has occurred within a year after starting the project, and it has soon become the standard machine learning framework of Tizen for various consumer electronic devices.” MyungJoo Ham answered.

Then, we asked, “What was the most impressive thing during the past three years when contributor strength has increased and the project has grown, as shown in the graph below? [3]”

“It’s definitely contributions from other affiliations, including other companies, students, and hobbyists. It’s especially impressive to see other affiliations (e.g., NXP Semiconductors and Collabora) to contribute to the project and hearing how they are going to use NNStreamer for their own projects.” he said about collaboration.

“Supporting multiple platforms became important as each project required a different environment to run the NNStreamer application. So, we had improved our project implementing one code-set to support various platforms and neural network frameworks.” a committer, Jaeyun Jung added this impressive moment too.

We can find more details about the multiple platforms that he talked about in GitHub NNStreamer. You can also track all the related issues including the new release if you want. We then talked about Edge AI on the recent release, by asking them on what they think the role of Edge Intelligence in the future, as well as on how NNStreamer will be developed in relation to Edge Intelligence.

“I view Edge AI or Among-Device AI as a distributed intelligence system with physically enclosed environments (e.g., in-home or in-building), where personal data may be used by intelligence systems while such data are not exposed out of the enclosed environments. As becoming among-device AI systems, any device, regardless of its computing power or sensor components, may provide consistent intelligence services as if every device interfacing users has high computing power and various sensor components by sharing such resources connected in the enclosed environments.

We are extending the capability of NNStreamer so that it can not only provide on-device AI capability but also provide among-device AI capability by implementing inter-pipeline stream connections that are easy-to-use and efficient-to-transmit. Another set of capabilities is to deploy and publish AI services atomically so that other among-device AI entities may subscribe. We will also provide API sets that can be used by among-device AI applications, either based on NNStreamer or not, to connect with NNStreamer pipelines as AI services for among-device AI capability. All the related components will be available as open source as it has always been. With the capability to make non-NNStreamer software connect with NNStreamer pipelines as AI services will further enable different systems to be connected for the given among-device AI systems.

In conclusion, by allowing any party to publish AI services for NNStreamer’s among-device AI (a.k.a. Edge AI) systems and to subscribe to such AI services even if the corresponding devices do not have NNStreamer running, we expect that we can create an intelligent IoT device ecosystem that is open to anyone and any system. Ultimately, we would like to provide such among-device AI streaming mechanisms to IoT standards including Matter and SmartThings so that more devices and more people may enjoy the capability of NNStreamer.” a maintainer, MyungJoo Ham explaining the vision of the project.

Jaeyun Jung also shared his opinions on expanding the AI ecosystem.

“Suppose you create AI services using NNStreamer. If several models need to be linked or the models continue to be changed during development, application developers sometimes have to modify the pipeline or change the code whenever these changes occur. Application developers have some requirements to minimize code modification even if these changes occur.

Therefore, we will provide an ML API that application developers can use to facilitate code maintenance regardless of changes in the model or pipeline as the work of changing the model will be registered as a separate service.” he said

Finally, it is more meaningful that Taras Drozdovsky, who is participating in several open source projects as a security expert to acquire this CII BP Badge, participated in the NNStreamer project to collaborate and contribute. He shared his experience in participating as a new committer.

“Like every developer, before the moment, you create your first commit in a project, I go the way from the first familiarity with the product to implementing a good idea. Therefore, any Open Source project should create an atmosphere that will not leave any developer indifferent, inspire him to improve it, and answer the main questions: What kind of project is this? What is it for? How and where can it be used? When I visited the GitHub NNStreamer project and saw the “Applications” part [4], all the answers were obtained!

I understood that I wanted to put a piece of myself to create something incredible! Summing up this part, I can say that the structure and description of the project make an attractive force, and I want to do something to improve it.

The next step is more detailed familiarization with the project build and launch of examples. And here, everything was at the highest level. For a few hours, I was able to build a project and run a few examples. A miracle happened in front of my eyes! I once again made sure that quality documentation was significant.

I started to think about what contribution I could make, and I analyzed the project using my experience in security. I did not find serious vulnerabilities but found that it can make the product more protected and efficient. Opening the documentation again at the “Contributing” part and following the instructions, I created my first commit and pull request and began to wait for the maintainers’ reaction. The discussion with them was rapid and high-quality (they are real professionals). So the first step was done! And now I am sure that I will not stop!

Result: The CII Best Practices “Passing” badge was obtained! And now, every developer seeing it can be sure that he will face high-quality documentation and safe source code.” he vividly said.

Your contribution is always welcomed!

Looking at the proportion of the total number of commits, the NNStreamer project is mainly operated by Samsung developers. However, others such as NXP also have participated and collaborated. In addition, LF TSC regular meetings and webinars are held, and external outreach activities are steadily carried out. We invite you to visit the GitHub where NNStreamer and its subprojects are developed. Anyone can participate in its development and contribute to the project according to open source governance.